Title: NetHack is Hard to Hack
Venue: NeurIPS 2023
Accepted: yes
Keywords: imitation learning

NetHack
Abstract: Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in long-horizon tasks, especially in open-ended environments with multi-modal observations, such as the popular dungeon-crawler game, NetHack. Intriguingly, the NeurIPS 2021 NetHack Challenge revealed that symbolic agents outperformed neural approaches by over four times in median game score. In this paper, we delve into the reasons behind this performance gap and present an extensive study on neural policy learning for NetHack. To conduct this study, we analyze the winning symbolic agent, extending its codebase to track internal strategy selection in order to generate one of the largest available demonstration datasets. Utilizing this dataset, we examine (i) the advantages of an action hierarchy; (ii) enhancements in neural architecture; and (iii) the integration of reinforcement learning with imitation learning. Our investigations produce a state-of-the-art neural agent that surpasses previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, we also demonstrate that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players.

NetHack is Hard to Hack

Abstract

Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in long-horizon tasks, especially in open-ended environments with multi-modal observations, such as the popular dungeon-crawler game, NetHack. Intriguingly, the NeurIPS 2021 NetHack Challenge revealed that symbolic agents outperformed neural approaches by over four times in median game score. In this paper, we delve into the reasons behind this performance gap and present an extensive study on neural policy learning for NetHack. To conduct this study, we analyze the winning symbolic agent, extending its codebase to track internal strategy selection in order to generate one of the largest available demonstration datasets. Utilizing this dataset, we examine (i) the advantages of an action hierarchy; (ii) enhancements in neural architecture; and (iii) the integration of reinforcement learning with imitation learning. Our investigations produce a state-of-the-art neural agent that surpasses previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, we also demonstrate that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players.

1 Introduction

Reinforcement Learning (RL) combined with deep neural policies has achieved impressive results in control problems, such as short-horizon simulated locomotion tasks [55, 7]. However, these methods struggle in long-horizon problem domains, such as NetHack [34], a highly challenging grid-world game. NetHack poses difficulties due to its vast state and action space, multi-modal observation space (including vision and language), procedurally-generated randomness, diverse strategies, and deferred rewards. These challenges are evident in the recent NetHack Challenge [23], where agents based on hand-crafted symbolic rules outperform purely neural approaches (see Figure 1), despite the latter having access to high-quality human demonstration data [24] and utilizing large-scale models.

Wepropose three reasons for the poor performance of large-scale neural policies compared to symbolic strategies. First, symbolic strategies implement hierarchical control schemes, which are generally absent in neural policies used for NetHack. Second, symbolic models use hand-crafted parsers for multi-modal observations, suggesting that larger networks could enhance representations extracted from complex observations. Third, symbolic strategies incorporate error correction mechanisms, which could be crucial for improving neural policies if integrated with RL based error correction.

In this work, we conduct a comprehensive study of NetHack and examine various learning mechanisms to enhance the performance of neural models. We bypass traditional RL obstacles, such as sparse rewards or exploration challenges, by focusing on imitation learning. However, we find that existing datasets lack crucial information, such as hierarchical labels and symbolic planning traces. To address this, we augment the codebase of AutoAscend , the top-performing symbolic agent in the 2021 NetHack Challenge, and extract hierarchical labels tracking the agent's internal strategy selection in order to construct a large-scale dataset containing 10 9 actions.

CDGPT5

AutoAscend

NHC 3rd Place (Neural)

NHC 1st Place (Overall, Symbolic)

NetHackLearning Environment(NLE)

BestPurely Data-Driven Neural Agent

BestOverallAgent

message

neural-symbolic gap

You find an antimagjic field.

Policy Architecture

ego-centricpixelview

^f1

#

BC

LSTM

@

$1

一#

Transformer+LSTM

(sino)

HierarchicalLSTM

"bottomline"statistics

RL+ BC

(suno)

①v1:1 $:2 HP : 15(15) Pw:2(2) AC:8 xp:1/0

St: 18/01 Dx:15 Co:18 Ie:9 vi:8 ch:6 Chaotie S: 2

5·102

1 · 104

mean score (NLE)

Using this dataset, we train a range of deep neural policies and investigate: (a) the advantages of hierarchy; (b) model architecture and capacity; and (c) fine-tuning with reinforcement learning. Our main findings are as follows:

Hierarchical behavioral cloning (HBC) significantly outperforms BC and baseline methods, provided that the model has adequate capacity.

Large transformer models exhibit considerable improvements over baselines and other architectures, such as LSTMs. However, the power-law's shallow slope indicates that data scaling alone will not suffice to solve the game.

Online fine-tuning with RL further enhances performance, with hierarchy proving beneficial for exploration.

The combined effects of hierarchy, scale, and RL lead to state-of-the-art performance, narrowing the gap with AutoAscend but not eliminating it.

We validate the statistical significance of our findings with low-sample hypothesis testing (see Appendix H). Additionally, we open-source our code, models, and the HiHack repository 2 , which includes (i) our 10 9 dataset of hierarchical labels obtained from AutoAscend and (ii) the augmented AutoAscend and NLE code employed for hierarchical data generation, encouraging development.

Though we base our experiments in NetHack, we take measures to preserve the generality of insights yielded by our investigations of neural policy learning. Specifically, we intentionally forgo the addition of any environment-specific constraints in the architectural design or training setup of all models explored in this paper. This is in contrast to NetHack agents RAPH and KakaoBrain, the first and second place winners of the neural track of the NHC Competition, respectively, which incorporate augmentations such as hand-engineered action spaces, role-specific training, and hard-coded symbolic sub-routines [23]. While this choice prevents us from achieving absolute state-of-the-art performance in NLE in this paper, we believe it to be crucial in preserving the general applicability of our insights to neural policy learning for general open-ended, long-horizon environments.

2 Related Work

Our work builds upon previous studies in the NetHack environment, imitation learning, hierarchical learning, and the use of transformers as policies. In this section, we briefly discuss the most relevant works.

NetHack Following the introduction of the NetHack Learning Environment (NLE) [34], the NetHack Challenge (NHC) competition [23] enabled comparisons between a range of different agents. The best performing symbolic and fully data-driven neural agents were AutoAscend and Chaotic-Dwarven-GPT-5 ( CDGPT5 ), respectively, and we base our investigations on them, as well as on the NetHack Learning Dataset [24].

Several notable works make use of the NetHack environment. Zhong et al. [61] show how a dynamics model can be learned from the Nethack text messages and leveraged to improve performance; on account of their utility, we also encode the in-game message but instead use a model-free policy to pick actions. Bruce et al. [9] show how a monotonic progress function in NetHack can be learned from human play data and then combined with a reinforcement learning (RL) reward to solve long-range tasks in the game. This represents a complementary way of employing NetHack demonstration data without direct action imitation.

Imitation Learning Pomerleau [47] demonstrated the potential of driving an autonomous vehicle using offline data and a neural network, which has since become an ongoing research topic for scalable behavior learning [3, 6, 50]. Formally, behavior learning is a function approximation problem, where the goal is to model an underlying expert policy mapping states or observations to actions, directly from data. Approaches to behavior learning can be categorized into two main classes: offline RL [21, 32, 33, 58, 36, 20], which focuses on learning from mixed-quality datasets with reward labels; and imitation learning [42, 44, 45, 26], which emphasizes learning behavior from expert datasets without reward labels. Our work primarily belongs to the latter category as it employs a behavior cloning model. Behavior cloning, a form of imitation learning, aims to model the expert's actions given the observation and is frequently used in real-world applications [60, 62, 60, 48, 18, 59].

Since behavior cloning algorithms typically address a fully supervised learning problem, they are often faster and simpler than offline RL algorithms while still yielding competitive results [20, 22]. A novel aspect of our work is the use of hierarchy in conjunction with behavioral cloning, i.e. supervision at multiple levels of abstraction, a topic which has received relatively little attention. Recent efforts to combine large language models with embodied agents use the former to issue a high-level textual 'action' to the low-level motor policy. Approaches such as Abramson et al. [1] have shown the effectiveness of hierarchical BC for complex tasks in a simulated playroom settings.

Hierarchical Policy Learning Hierarchical reinforcement learning (HRL) based techniques [5, 4] extend standard reinforcement learning methods in complex and long-horizon tasks via temporal abstraction across hierarchies, as demonstrated by Levy et al. [37] and Nachum et al. [40][41]. Similarly, numerous studies have concentrated on showing that primitives [17, 54, 53] can be beneficial for control. These concepts have been combined in works such as Stochastic Neural Networks by Florensa et al. [19], where skills are acquired during pretraining to tackle diverse complex tasks. Likewise, Andreas et al. [2] learn modular sub-policies for solving temporally extended tasks. However, most prior work focus on learning both levels of the hierarchy, i.e. a decomposition across primitives, skills, or sub-policies as well as the primitives, skills, or sub-policies themselves, which makes training complex. Correspondingly, the resultant approaches have had limited success on more challenging tasks and environments. Le et al. [35] explores the interaction between hierarchical learning and imitation and finds benefits, albeit in goal-conditioned settings. In contrast, our work explores the benefits of access to a fixed hierarchy chosen by the domain-expert designer of AutoAscend , which simplifies our study of overall learning mechanisms for NetHack.

Transformers for RL The remarkable success of transformer models [57] in natural language processing [15, 8] and computer vision [16] has spurred significant interest in employing them for learning behavior and control. In this context, [11, 29] apply transformers to RL and offline RL, respectively, while [12, 14, 38] utilize them for imitation learning. Both [14, 38] primarily use transformers to summarize historical visual context, whereas [12] focuses on their long-term extrapolation capabilities. More recent work have explored the use of multi-modal transformers [27] to fit large amounts of demonstration data [49, 52, 56]. To enable transformers to encode larger context lengths, recurrent transformer models have been proposed [13, 10]. Our work draws inspiration from these use cases, employing a transformer to consolidate historical context and harness its generative abilities in conjunction with a recurrent module.

AutoAscend

global controller

inventory

eatfrom

disease

cure

engulfed

fight

explore,

identify

gather,

sokoban

solve

guard

follow

unexpected

waitout

state

eatcorpses

ground

from

fight

emergency

corpses

offer

at

3 Data Generation: Creating the HiHack Dataset

3.1 Extending the NetHack Learning Environment

The NetHack Learning Environment (NLE) is a gym environment wrapping the NetHack game. Like the game itself, the action and state spaces of NLE are complex, consisting of 121 distinct actions and ten distinct observation components. The full observation space of NLE is far richer and more informed than the view afforded to human players of NetHack, who observe only the more ambiguous 'text based' components of NLE observations, denoted as tty_chars , tty_colors , and tty_cursor . This text based view corresponds also to the default format in which both NetHack and NLE gameplay is recorded, loaded, and streamed via the C based ttyrec library native to the NetHack game.

The popular NetHack Learning Dataset (NLD) offers two large-scale corpuses of NetHack gameplay data, NLD-AA , consisting of action-labeled demonstrations from AutoAscend , and NLD-NAO , consisting of unlabeled human player data [24]. NLD adopts the convention of recording only tty ∗ components of NLE observations as a basis for learning, hence benefiting from the significant speedups in data operations offered via integration with the ttyrec library. We adhere to this convention with the hierarchical HiHack Dataset introduced in this paper. Thus, in order to generate our dataset, we extend the ttyrec library to store hierarchical strategy or, equivalently, goal labels alongside action labels. We further integrate this extension of ttyrec with NLE, modifying the gym environment to accept an additional hierarchical label at each step of interaction. This input hierarchical label does not affect the underlying state of the environment, and is instead employed strictly to enable the recording of hierarchically-informed NetHack game-play to the ttyrec data format.

3.2 AutoAscend: A Hierarchical Symbolic Agent

An inspection of the fully open-source code base underlying the AutoAscend bot reveals the internal structure of the bot to be composed of a directed acyclic graph of explicitly defined strategies . The bot's underlying global controller switches between strategies in an imperative manner via sets of strategy-specific predicates, as visualised in Figure 2. Among these strategies are hand-engineered routines for accomplishing a broad range of goals crucial to effective survival in the game and successful descent through the NetHack dungeons. These include routines for fighting off arbitrary monsters, selecting food that is safe to eat from an agent's inventory, and efficiently exploring the dungeon while gathering and identifying valuable items, among many others. The various strategies are supported in turn by shared sub-strategies for accomplishing simpler 'sub-goals' (see Appendix B for the full graph).

We exploit this explicit hierarchical structure in the generation of HiHack , extending the AutoAscend codebase to enable per-step logging of the strategy responsible for yielding each action executed by the bot, as supported by our modifications to the C based ttyrec writer library and NLE.

3.3 The HiHack Dataset

Our goal in generating the HiHack Dataset ( HiHack ) is to create a hierarchically-informed analogue of the large-scale AutoAscend demonstration corpus of NLD, NLD-AA . Thus, as previously described, HiHack is composed of demonstrations recorded in an extended version of the ttyrec format, consisting of sequences of tty ∗ observations of the game state accompanied by AutoAscend action and strategy labels. HiHack contains a total of 3 billion recorded game transitions, reflecting more than a hundred thousand AutoAscend games. Each game corresponds to a unique, procedurallygenerated 'seed' of the NetHack environment, with AutoAscend playing as one of thirteen possible character 'starting roles' across a unique layout of dungeons.

We verify that the high-level game statistics of HiHack match those of NLD-AA in Table 1. Indeed, we find a high degree of correspondence across mean and median episode score, total number of transitions, and total number of game turns. We attribute the very slightly diminished mean scores, game transitions, and turns associated with HiHack to a difference in the value of the NLE timeout parameter employed in the generation of the datasets. This parameter regulates the largest number of contiguous keypresses failing to advance the game-state that is permitted before a game is terminated. The value of the timeout parameter was set to 1000 in the generation of HiHack .

4 Hierarchical Behavioral Cloning

Our first set of experiments leverage the hierarchical strategy labels recorded in HiHack for offline learning with neural policies, via hierarchical behavior cloning (HBC).

Method Mimicking the imperative hierarchical structure of AutoAscend , we introduce a bilevel hierarchical decoding module over a popular NetHack neural policy architecture, namely the ChaoticDwarvenGPT5 ( CDGPT5 ) model. This model achieved 3rd place in the neural competition track of the NeurIPS 2021 NetHack Challenge when trained from scratch with RL, making it the top-performing fully data-driven neural model for NetHack [23].

The CDGPT5 model consists of three separate encoders: a 2-D convolutional encoder for pixelrendered visual observations of the dungeon o t , a multilayer perceptron (MLP) encoder for the environment message m t , and a 1-D convolutional encoder for the bottom-line agent statistics b t . These three observation portions are extracted from the tty ∗ NLE observations of HiHack . The core module of the network is an LSTM, which is employed to produce a recurrent encoding of an agent's full in-game trajectory across what may be hundreds of thousands of keypresses, both in training and at test-time. The core module may also receive a one-hot encoding of the action executed at the previous time-step a t -1 as input.

Our hierarchically-extended version of this LSTM based policy is shown in Figure 3(left). We replace the linear decoder used to decode the LSTM hidden state into a corresponding action label in the CDGPT5 model with a hierarchical decoder consisting of (i) a single 'high level' MLP responsible for predicting the strategy label g t given the environment observation tuple { m t , o t , b t } , and (ii) a set of 'low level ' MLPs, one for each of the discrete strategies in the AutoAscend hierarchy (see Figure 2), with a SoftMax output over discrete actions. The strategy prediction g t selects which of these low-level MLPs to use.

BCResults

LSTM Policies

gumbel softmax

1000

6

mean score (NLE)

800

lineardecoders

600

ht

400

ht-1

LSTM

convolutional

200

+linear encoders

at-1

mt

40

bt

LSTM

LSTM +

Hierarchical

LSTM

We employ a simple cross-entropy loss to train both the baseline non-hierarchical LSTM CDGPT5 policy, as well as our Hierarchical LSTM policy, aggregating gradients across the bilevel decoders of the latter via the Gumbel-Softmax reparameterization trick [28].

Training and evaluation details We train all policies on a single GPU for 48 hours with the full 3 . 2 B HiHack Dataset. As with all experiments in the paper, a total of 6 seeds are used to randomize dataloading and neural policy parameter initialization. We employ mean and median NLE score on a batch of withheld NLE instances ( n = 1024 ) as our central metrics for evaluating model performance and generalization at the conclusion of training, following the convention introduced in the NetHack Challenge competition [23]. Reported performance is aggregated over random seeds. Further details of architectures as well as training and evaluation procedures can be found in Appendices D, E, and F of the supplemental material.

Results We find that the introduction of hierarchy results in a significant improvement to the test-time performance of LSTM policies trained with behavioral cloning, yielding a 40% gain over the baseline in mean NLE score as shown in Figure 3(right), and 50% improvement in median score across seeds as shown in Table 2. Additionally, to verify that this improvement in performance is indeed due to hierarchy and not simply a result of the increased parameter count of the hierarchical LSTM policy decoder, we run ablation experiments with a modified, large-decoder version of the baseline (non-hierarchical) policy architecture. The results, shown in Figure 3(right), show that increasing the size of the LSTM decoder, without the introduction of a hierarchy, does not result in performance improvements over the baseline.

5 Architecture and Data Scaling

Despite the benefits of introducing hierarchical labels, the performance of the Hierarchical LSTM policy trained with HBC remains significantly behind that of symbolic policy used to generate the HiHack demonstrations in the first place, AutoAscend . This finding prompts us to explore scaling: perhaps increasing the quantity of demonstration data or the model capacity may close the observed performance gap.

Method To test this new hypothesis, we conduct a two-pronged investigation: (i) to explore model capacity, we develop a novel base policy architecture for NetHack that introduces a transfomer module into the previous CDGPT5 based architecture; and (ii) for data scaling, we run a second set of 'scaling-law' [30] experiments that use subsets of the HiHack Dataset to quantify the relationship between dataset size and the test-time performance of BC policies.

BCResults

LSTMvs.TransformerPolicies

1250

lineardecoder

causaltransformer

mean score (NLE)

1000

embedding

750

+positionalencoding

at-1

mt

Ot

(frozen)

500

LSTM

recurrentmodule

250

0

LSTM

Hierarchical

Flat

Transformer

Hierarchical

LSTM

Transformer

+ LSTM

Transformer

+ LSTM

Our novel base policy architecture is visualized in Figure 4 (left). This architecture features two copies of the encoders employed in CDGPT5 . One set is kept frozen and employed strictly to yield a recurrent encoding of the complete trajectory up to the current observation step via a pre-trained frozen LSTM, while the second is kept 'unlocked' and is employed to provide embeddings of observations directly to a causal transformer, which receives a fixed context length during training.

Training and evaluation details The training and evaluation procedures employed here echo those of section 4. In our data-scaling experiments, the subset of sampled HiHack games seen in offline training is randomized over model seeds. The causal transformer component of our transformer-LSTM models is trained with the same fixed, context-length ( c = 64 ) in all experiments. Hyperparameter sweeps were employed to tune all transformer hyperparameters (see Appendix D).

Results The architecture experiments in Figure 4(right) show that both variants of our combined transformer-LSTM policy architecture yield gains eclipsing those granted solely by the introduction of hierarchy in the offline learning setting.

ModelSizevs.Transformer-LSTM

DatasetSizevs.PolicyPerformance

PolicyPerformance

105

106

107

108

109

1500

98.9Mparams

50.8Mparams

+

transitions

(37N)

1250

(N)

1500

一

8 = 850 10g10(g)0.179

PolicyArchitecture

score

1000

meanscore

1000

LSTM

mean：

750

+ LSTM (large)

Transformer

500

500

250

= 184 l0og1o(g)0.819

0

Transformer

Transformer

0

101

T

102

T

103

104

T

105

+ LSTM (large)

+LSTM

games

We find the LSTM recurrent module of our transformer-LSTM models to be a crucial component of the architecture. When this module is ablated to yield a flat transformer , we observe a severe decline in policy performance (Figure 4(right) and Appendix G, Figure 14).

Probing further, in Figure 5(left), we compare the performance of two variants of our transformerLSTM model, one with 3 layers (50.8M parameters) and another 'scaled-up' model with 6 layers (98.9M parameters). The larger model can be seen to perform worse in evaluation than the smaller one, suggesting over-fitting and an inverse relationship between parameter count and generalization in NetHack for this model class (see also Appendix G, Figure 12). This finding indicates that scaling of model capacity alone may not be sufficient to close the neural-symbolic gap.

In Figure 5(right), we now explore the effect of training set size on mean NLE test score. We perform BC training for the LSTM baseline [23] and our largest 6 layer transformer-LSTM model (98.9M params) for 10 up to 100 , 000 games, subsampled from the HiHack Dataset. For both architectures, we observe a sub log-linear dependence on training set size, asymptoting at a mean score of approximately 1000. Thus, brute force scaling of the dataset alone cannot viably close the gap to symbolic methods (score of 8500).

Though our architecture and data scaling experiments are compute-time constrained, we find the test-time performance of all tested models to saturate on the given computational budget (Appendix G, Figures 11 and 12).

6 Combining Imitation with Reinforcement Learning

Given that hierarchy and scaling are insufficient to bridge the performance gap with AutoAscend , we now explore the impact of incorporating an online learning component to neural policy training, via RL.

Method In this set of experiments, we build on results from Hambro et al. [24], which suggested that in the presence of action-labeled data, behavioral cloning coupled with reinforcement learning is superior to RL training from scratch in NetHack. As in Hambro et al. [24], we employ the asynchronous moolib distributed-RL library to train our models with a combination of BC and asynchronous proximal policy optimization (APPO) [39, 46, 51]. At each time-step of training, the overall loss used to perform model parameter updates is a weighted combination of BC and RL losses, i.e. the cross-entropy loss of a batch of demonstrations from HiHack plus an RL loss over a batch of rollouts of the current policy in NLE. For hierarchical models, we use RL only to update the low-level strategies; that is, the strategy selection policy is trained with HBC alone and is not updated with RL.

NLEScoreDistribution(t)vs.StartingRoleAcrossNeuralPolicyClasses

(BC Policies)

LSTM

Hierarchical LSTM

Transformer + LSTM

HierarchicalTransformer+LSTM

15000

15000

15000

15000

10000

score (NLE)

10000

()

10000

() S)

10000

5000

5000

5000

5000

0

Arc

Cav

i

Kni

Mon

!d

Ran

if

no

ZIM

Arc

Cav

Kni

Mon

早

!

Ran

ues

noL

Val

Wiz

0

Arc

Bar

Cav

T

Kni

Mon

Ran

ues

Val

Wiz

0

Bar

Cav

日

Kni

Mon

自

Ran

noL

ZIM

(APPO +BC Policies)

15000

LSTM

15000

Hierarchical LSTM

15000

Transformer+LSTM

15000

HierarchicalTransformer+LSTM

00:000

score (NLE)

10000

score (NLE)

10000

score (NLE)

10000

10000

00 eods

5000

0

品

正

Cav

Kni

Mon

d

工

Ran

i

0

noL

正+

ZIM

1

5000

0

Ba

Kni

早

Mon

d

自

Ran

早

ZIM

5000

0

Kni

Mon

i+

Ran

正

noL

ZIM

5000

D

-i

Kni

早

Mon

自

!d

Ran

正

noL

Training and evaluation details Our high-level training procedure here mirrors that of our hierarchical behavioral cloning experiments: we evaluate model performance under the constraint of computation time, training all policies for exactly 48 hours on a single GPU, using 6 random seeds to randomize data loading and environment seeding only. We test the effect of warm-starting from a checkpoint pre-trained with BC or HBC alone (via the procedure delineated in section 4) for all model classes. We report APPO + BC results with warm-starting for all model classes where we find it to be beneficial in improving stability and/or evaluation policy performance.

Results Table 2 summarizes the performance of all our models. A number of observations can be made: (a) In the offline setting, our best performing model (non-hierarchical transformer-LSTM) outperforms the vanilla CDGPT5 model baseline by 100% in mean NLE score and 127% in median NLE score; (b) RL fine-tuning offers a clear and significant performance boost to all models, with gains in the test-time mean NLE score associated with all model classes; (c) The overall best performing approach is APPO + BC using the hierarchical LSTM model. The mean score of 1551 represents a new state-of-the-art for neural policies in NLE, beating the performance of the vanilla CDGPT5 model when trained with APPO + BC by 29% in mean NLE score and 25% in median NLE score; (d) The transformer-LSTM models, which are slower and more unstable to train with APPO + BC than LSTM models (see training curves in Appendix G), see a smaller margin of improvement during RL fine-tuning; (e) Other metrics commonly employed to evaluate meaningful NetHack gameplay in previous work, such as dungeon level reached and lifetime of the agent in game turns [34, 23], show a broadly similar pattern to the chief median and mean score metrics employed in the NetHack Challenge [23]; and (f) Lastly, for BC, hierarchy seems to hurt performance for the larger Transformer-RL models, though this gap is closed once APPO fine-tuning is applied. See Appendix H for low-sample hypothesis tests validating the statistical significance of these findings.

In NetHack, the player can choose from 13 distinct roles (barbarian, monk, wizard, etc.), each of which require distinctive play-styles. In NLE, starting roles are randomized, by default. Figure 6 shows a score distribution breakdown across role for different neural policy classes trained with BC and APPO + BC. In general, we observe that fine-tuning with RL improves the error-correction capability of models of all classes (as indicated by positive shifts in NLE score distributions) over their purely offline counterparts.

7 Conclusion and Discussion

In this work, we have developed a new technique for training NetHack agents that improves upon prior state-of-the-art neural models by 127% in offline settings and 25% in online settings, as evaluated by median NLE score. We achieve this by first creating a new dataset, the HiHack Dataset ( HiHack ), by accessing the best symbolic agent for NetHack. This dataset, combined with new architectures, allows us to build the strongest purely data-driven agent for NetHack as of the writing of this paper.

More importantly, we analyze several directions to improve performance, including the importance of hierarchy, the role of large transformer models, and the boosts that RL could provide. Our findings are multifaceted and provide valuable insights for future progress in training neural agents in open-ended environments and potentially bridging the gap to symbolic methods, as well as to the very best human players.

Hierarchy can improve underfitting models. Prior LSTM based models severely underfit on the HiHack Dataset (see training curves in Appendix G). The addition of hierarchy improves such models, whether trained offline or online (Figure 3(right) and Table 2).

Hierarchy can hurt overfitting models. Transformer based models are able to overfit on the HiHack Dataset (see training curves in Appendix G). Hierarchy hurts this class of models when trained offline at test-time (Figure 4(right) and Table 2).

Reinforcement learning provides larger improvements on light, underfitting models. We obtain little and no improvement with RL fine-tuning on our hierarchical transformer-LSTM and non-hierarchical transformer-LSTM models, respectively. However, the underfit LSTM models enjoy significant gains with RL, ultimately outperforming transformer based models under a compute-time constraint (Table 2).

Scale alone is not enough. Our studies on increasing both model and dataset size show sub-log-linear scaling laws (Figure 5). The shallow slope of the data scaling laws we observe suggests that matching the performance of AutoAscend with imitation learning (via games played by bot) will require more than just scaling up demonstration count.

Possible avenues for future exploration include: (a) alternate methods for increasing the transformer context length to give the agent a more effective long-term memory; (b) explicitly addressing the multi-modal nature of the demonstration data (i.e. different trajectories can lead to the same reward), which is a potential confounder for BC methods. Some forms of distributional BC (e.g. GAIL [26], BeT [52]) might help alleviate this issue.

Finally, we hope that the hierarchical dataset that we created in this paper may be of value to the community in exploring the importance of hierarchy and goal-conditioning in long-horizon, open-ended environments.

References

J. Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, S. Clark, A. Dudzik, P. Georgiev, A. Guy, T. Harley, F. Hill, A. Hung, Z. Kenton, J. Landon, T. P. Lillicrap, K. W. Mathewson, A. Muldal, A. Santoro, N. Savinov, V. Varma, G. Wayne, N. Wong, C. Yan, and R. Zhu. Imitating interactive intelligence. arXiv , abs/2012.05672, 2020. 3

J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches, 2016. 3

B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems , 57(5):469-483, 2009. 3

P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence , 2017. 3

A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems , 13(1-2):41-77, 2003. 3

A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Survey: Robot programming by demonstration. Handbook of robotics , 59(BOOK_CHAP), 2008. 3

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, S. Sidor, I. Sutskever, and R. S. Zemel. Openai gym. arXiv preprint arXiv:1606.01540 , 2016. 1

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020. 3

D. Bruce, E. Hambro, M. Azar, and M. G. Bellemare. Learning about progress from experts. In International Conference on Learning Representations , 2023. 3

A. Bulatov, Y. Kuratov, and M. Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems , 35:11079-11091, 2022. 3

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. CoRR , abs/2106.01345, 2021. URL https://arxiv.org/abs/2106.01345 . 3

H. M. Clever, A. Handa, H. Mazhar, K. Parker, O. Shapira, Q. Wan, Y. Narang, I. Akinola, M. Cakmak, and D. Fox. Assistive tele-op: Leveraging transformers to collect robotic task demonstrations. arXiv preprint arXiv:2112.05129 , 2021. 3

Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 , 2019. 3

S. Dasari and A. Gupta. Transformers for one-shot visual imitation. arXiv preprint arXiv:2011.05970 , 2020. 3

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , pages 4171-4186, 2018. doi: 10.18653/v1/N19-1423. 3

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 3

B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. CoRR , abs/1802.06070, 2018. URL http://arxiv.org/abs/ 1802.06070 . 3

P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters , 5(2):492-499, 2019. 3

C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning, 2017. 3

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020. 3

S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1587-1596. PMLR, PMLR, 2018. 3

C. Gulcehre, Z. Wang, A. Novikov, T. Le Paine, S. Gomez Colmenarejo, K. Zolna, R. Agarwal, J. Merel, D. Mankowitz, C. Paduraru, et al. Rl unplugged: Benchmarks for offline reinforcement learning. arXiv e-prints , pages arXiv-2006, 2020. 3

E. Hambro, S. Mohanty, D. Babaev, M. Byeon, D. Chakraborty, E. Grefenstette, M. Jiang, J. Daejin, A. Kanervisto, J. Kim, S. Kim, R. Kirk, V. Kurin, H. Küttler, T. Kwon, D. Lee, V. Mella, N. Nardelli, I. Nazarov, N. Ovsov, J. Holder, R. Raileanu, K. Ramanauskas, T. Rocktäschel, D. Rothermel, M. Samvelyan, D. Sorokin, M. Sypetkowski, and M. Sypetkowski. Insights from the neurips 2021 nethack challenge. In Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track , pages 41-52, 2022. 1, 2, 3, 4, 5, 6, 7, 8, 9, 19, 21, 25

E. Hambro, R. Raileanu, D. Rothermel, V. Mella, T. Rocktäschel, H. Küttler, and N. Murray. Dungeons and data: A large-scale nethack dataset. In Advances in Neural Information Processing Systems , 2022. 1, 3, 4, 8, 17, 18

D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016. 19

J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems , volume 29, pages 4565-4573, 2016. 3, 10

A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 4651-4664. PMLR, 2021. 3

E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. 6, 20

M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34, 2021. 3

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv , abs/2001.08361, 2020. 6

D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 18

A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019. 3

A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020. 3

H. Kuttler, N. Nardelli, T. Lavril, M. Selvatici, V. Sivakumar, M. G. Bellemare, R. Munos, A. Graves, and M. G. Bellemare. The nethack learning environment. In Advances in Neural Information Processing Systems , 2020. 1, 3, 9, 15, 20

H. Le, N. Jiang, A. Agarwal, M. Dudík, Y. Yue, and H. Daumé III. Hierarchical imitation and reinforcement learning. In International conference on machine learning , pages 2917-2926. PMLR, 2018. 3

S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020. 3

A. Levy, G. Konidaris, R. Platt, and K. Saenko. Learning multi-level hierarchies with hindsight, 2017. 3

Z. Mandi, F. Liu, K. Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. arXiv preprint arXiv:2110.13423 , 2021. 3

V. Mella, E. Hambro, D. Rothermel, and H. Küttler. moolib: A Platform for Distributed RL. 2022. URL https://github.com/facebookresearch/moolib . 8, 18

O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. CoRR , abs/1805.08296, 2018. URL http://arxiv.org/abs/1805.08296 . 3

O. Nachum, S. Gu, H. Lee, and S. Levine. Near-optimal representation learning for hierarchical reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. 3

T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.06711 , 2018. 3

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32:8026-8037, 2019. 18

X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACMTransactions on Graphics (TOG) , 37(4):1-14, 2018. 3

X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG) , 40(4):1-20, 2021. 3

A. Petrenko, Z. Huang, T. Kumar, G. Sukhatme, and V. Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learning , pages 7652-7662. PMLR, 2020. 8

D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988. 3

R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3758-3765. IEEE, 2018. 3

S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022. 3

S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences , 3 (6):233-242, 1999. 3

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. 8

N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems , 35:22955-22968, 2022. 3, 10

T. Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations , 2020. URL https: //openreview.net/forum?id=rkgHY0NYwr . 3

A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. CoRR , abs/1907.01657, 2019. URL http://arxiv.org/abs/1907. 01657 . 3

Y. Tassa, D. Silver, J. Schrittwieser, A. Guez, L. Sifre, G. v. d. Driessche, S. Dieleman, K. Greff, T. Erez, and S. Petersen. Deepmind control suite. arXiv preprint arXiv:1801.01294 , 2018. 1

A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608 , 2023. 3

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. 3, 18

Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019. 3

A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning , 2020. 3

T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA , pages 5628-5635. IEEE, 2018. 3

Y. Zhong, E. Hambro, E. Grefenstette, T. Park, M. Azar, and M. G. Bellemare. Improving policy learning via language dynamics distillation. In Advances in Neural Information Processing Systems , 2022. 3

Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564 , 2018. 3

A NetHack Learning Environment

Both NetHack and the NetHack Learning Environment (NLE) feature a complex and rich observation space. The full observation space of NLE consists of many distinct (but redundant) components: glyphs , chars , colors , specials , blstats , message , inv_glyphs , inv_strs , inv_oclasses , screen_descriptions , tty_chars , tty_colors , and tty_cursor [34].

The HiHack Dataset ( HiHack ), as well as all RL experiments in NLE conducted in this paper, consist of and rely solely upon the tty ∗ view of the game.

B Details on AutoAscend

followguard

arrangeitems

search neighbors

check altar

fortraps

solve sokoban

untrap traps

explore,gather,

identify

check containers

open neighbor

s

identifyitemson

pickup and drop

altar

openvisit search

items

cure disease

eat frominventory

dipfor excalibur

gatheritems

wearbeststuff

global controller

AutoAscend

unexpected state

waitout

gotolevel

gotoleast

go to unchecked

containers

exploredlevel

eat corpsesfrom

followlevelpath

punoub

explore stairs

offercorpses

wandengrave

identify

emergency

initialization

gotoitemfor

pickup

fight

write-error

checkitems

We include a comprehensive visualization of the full internal structure of AutoAscend in Figure 7. As indicated in the summary visualization of high-level AutoAscend strategies provided in Figure 2 of the main paper, the bot features 11 explicit, hard-coded strategy routines. These interface with other low-level sub-strategy routines, some which are re-used by multiple strategies or even multiple sub-strategies. One example of such a subroutine is the 'arrange items' sub-strategy, which is called both by 'follow guard' and by 'check altar,' which is itself a subroutine of the 'explore' sub-strategy. When factorized across strategies and sub-strategies, the full structure of AutoAscend is a directed acyclic graph (DAG) with a maximal depth of 5 from the 'root,' i.e. the AutoAscend global controller 'node' indicated in dark gray above, which re-directs global behavioral flow across strategies via a predicate-matching scheme.

The HiHack Dataset includes a hierarchical strategy label for each timestep of AutoAscend interaction. As a result, alongside the 11 explicit strategies of the bot, there are two additional labels present in the dataset, which account for extra-hierarchical behavior in ttyrec game records yielded by the augmented ttyrec writer employed for HiHack generation and loading. These are visualized in light gray in Figure 7. The first of these corresponds to the hard-coded initialization routine employed by AutoAscend , effectively serving as a twelfth (albeit implicit) strategy, while the second covers ttyrec timestep records with missing strategy values. Missing strategy values may reflect ttyrec writer errors, or advancement of the underlying NetHack state by NLE rather than by agent, which occurs e.g., during NLE's timeout based termination of games [34]. Empirically, 'write-error' strategy labels occur with very low frequency in HiHack , representing less than ≈ 0 . 05% of all transitions.

NLD-AA (n =109,545)vs.HiHack (n =109,907)

(Log-ScoreDistributions)

NLD-AA

HiHack

0

16

256

4096

65536

1048576

0

16

256

4096

65536

1048576

score

score

0000

10000

7500

7500

5000

5000

2500

2500

0

0

4

8

12

16

20

4

8

12

16

20

log2(score)

log2(score)

(Log-TurnsDistributions)

NLD-AA

HiHack

1

16

256

4096

65536

1

16

256

4096

65536

25000

turns

25000

tums

20000

20000

15000

10000

10000

5000

5000

0

0

4

log2(turns)

8

12

16

4

log2(turns)

8

12

16

C Details on HiHack

Generation All games in the HiHack dataset were recorded by running an augmented version of AutoAscend in NLE v 0.9.0 . This augmented version of the AutoAscend source features the introduction of only a dozen extra lines of code that enable step-wise logging of the strategy trace behind each action executed by the bot. This strategy trace is recorded directly to game ttyrec s at each timestep via the addition of an extra channel to the C based ttyrec -writer in the NetHack source code. Each game was generated via a unique NLE environment seed.

Game Statistics The comparison of the full log-score and log-turns distributions across NLD-AA and HiHack made in Figure 8 further supports the claim of high correspondence between the datasets made in Section 3.3. Figure 9 shows the distribution of strategies across a sample consisting of ≈ 10 8 unique game transitions from HiHack . We observe coverage of all but the least frequent explicit strategy executed by AutoAscend : the 'solve sokoban' routine, employed a means to gain yet more experience exclusively in highly advanced game states.

Distribution of Strategies in HiHack (T  108)

explore,gather,identify

fight

eat corpses from ground

waitoutunexpectedstate

eat frominventory

emergency

initialization

offercorpses

write-error

cure disease

follow guard

solve sokoban

104

105

106

107

108

transitions

D Training Details

Hyperparameters All relevant training hyperparameter values, across model families as well as BC vs APPO + BC experiment variants, are displayed in Table 3.

To kick-start all experiments, we employed the training hyperparameter values reported in Hambro et al. [24], used to train the CDGPT5 , LSTM based baseline across experiments. For several hyperparameters, however, additional tuning was conducted. These hyperparameters are indicated in bold in Table 3. Tuning across these hyperparameters was performed once for the 'default' representative policy class from each of the LSTM and transformer-LSTM model families for all but the Model Parameter Scaling experiments 3 . After tuning was complete, hyperparameter configurations were

fixed across all proceeding offline and combined offline + online experiments. Specifications of hyperparameter values swept over during tuning are provided in Table 4.

All models were trained with the Adam optimizer [31] and a fixed learning rate. We experimented with the introduction of a learning rate schedule for transformer-LSTM models, but we found no additional improvements in policy prediction error or evaluation performance at the conclusion of training to be yielded by such a schedule.

Random Seeds For each of the Hierarchical Behavioral Cloning, Model Parameter Scaling, Data Scaling, and combined Imitation and Reinforcement Learning experiments described in Sections 4, 5, and 6 of this paper, a total of 6 random seeds were run across all relevant policy classes. Randomized quantities included: policy parameter values at initialization, data loading and batching order, HiHack Dataset subsampling (in data scaling experiments only), and initial environment seeding (in APPO + BC experiments only).

Training Infrastructure and Compute The RPC based moolib library for distributed, asynchronous machine learning was employed across all experiments [39]. All data loading and batching was parallelized. Our model training code builds heavily upon the code open-sourced by Hambro et al. [24].

Experiments were run on compute nodes on a private high-performance computing (HPC) cluster equipped either with a NVIDIA RTX-8000 or NVIDIA A100 GPU, as well as 16 CPU cores. All policies were trained for a total of 48 hours. We detected no substantial differences in training framesper-second (FPS) rates for both offline and online experiments across compute nodes in 'speed-run' tests, provided nodes were under no external load. When running experiments, we did detect some variance in total optimization steps completed under the 48-hour constrained computational budget across seeds belonging to single policy classes, which we attribute to variance in external HPC cluster load during these runs.

E Model Architectures

A description of all model components and policy architectures is given in Tables 5 and 6, separated across LSTM and transformer-LSTM model families. The PyTorch library was used for to specify all models, loss functions, and optimizers [43].

Additional Transformer Specifications All transformer modules tested in this paper consist purely of 'Transformer-Encoder' layers. Each layer is configured with 16 attention heads per attention mechanism, and layer normalization is applied prior to all attention and feed-forward operations. A dropout of 0 . 1 is used during training [57]. Unlike the rest of the modules we employ, which use set of values tested, the same values were found to be optimal for this model configuration as for the default transformer-LSTM policy; hence, only a single set of hyperparameter values for the model family is reported here.

Exponential Linear Unit ( ELU ) activation functions as per the original CDGPT model architecture [23], our transformer modules employ Gaussian Error Linear Unit ( GeLU ) activations [25]. Doubling the number of attention heads and the hidden dimension of transformer encoder layers in addition to the layer count of transformer-LSTM models resulted only in equivalent or additional policy performance degradation over that observed for the six layer model whose specifications are reported in Table 6; thus, we include only results for the latter here.

Context Length Models belonging to all policy classes from the LSTM family are trained by sequentially 'unrolling' batched-predictions. The length of this 'unrolled' sequence is held fixed throughout training, and is specified by the value of the ' ttyrec unroll length' hyperparameter in Table 3, also referenced as c in the main body of this paper.

A fixed context length is also used to train the core transformer modules of policy classes from the transformer-LSTM family, similarly specified via the ' ttyrec unroll length' hyperparameter. We found causally masking context in transformer attention mechanisms to be greatly beneficial towards improving the generalization capability of models. The pre-trained frozen LSTM 'recurrent encoder' module of these networks provides a very cheap and simple means of dramatically extending the effective context length to cover full NetHack games (which may span hundreds of thousands of keypresses) without substantially slowing model training. As delineated in Section 5 and Appendix G, we find the recurrent module to be a crucial component of these models, with its ablation resulting in a substantial performance decline.

Hierarchical Policy Variants As alluded to in Tables 5 and 6, as well as in Section 4 of the main paper, all hierarchical policy variants are equipped with two sets of decoders: one high-level strategy decoder trained to predict the thirteen possible strategy labels in HiHack ; as well as thirteen low-level action decoders , trained to predict actions corresponding to a single HiHack strategy across the 121dimensional NLE action space [34]. Our hierarchical policies thus mimic the hierarchical structure of Autoascend , with an action decoder corresponding to each of the eleven, explicit strategies executed by the symbolic bot as well as two additional action decoders corresponding to the bot's 'initialization' routine (an implicit twelfth strategy) and 'write-errors,' representing missing strategy labels 4 , respectively, as introduced in Figure 7. A full, diagrammatic illustration of the Hierarchical LSTM policy architecture is provided in Figure 3.

In all Hierarchical Behavioral Cloning (HBC) experiments, a BC loss was computed for the strategy decoder via ground-truth, batched HiHack strategy labels. A separate BC loss was computed for a single action decoder over batched HiHack action labels, with this action decoder 'selected' in an end-to-end fashion by the strategy decoder. The action decoder 'selection' procedure was executed across batches by sampling predicted strategy indices from the strategy decoder with Gumbel-Softmax re-parameterization [28], thus preserving gradient flow across the bi-level hierarchical structure of policies during training. An illustration of the full Hierarchical LSTM policy architecture is provided in Figure 3.

The strategy-specific BC loss component was re-weighted (via the 'strategy loss coefficent' hyperparameter, introduced in Table 3) and recombined with low-level action decoder losses to produce a single, overall HBC loss.

We resolved the presence of ttyrec transitions with missing strategy values, represented via the 'write-error' hierarchical label in HiHack , by extending the action-space of the hierarchical strategy decoder and adding an additional hierarchical action decoder copy corresponding to this class of labels. It is possible that the performance of hierarchical policy variants can be further improved by instead filtering out all transitions with this property. We leave this evaluation for future work.

NLESeed Counts(t)vs.Starting Role(n=3402)

AutoAscendNLEScore(t)vs.StartingRole

50000

300

score (NLE)

counts

200

25000

100

meanAutoAscendscore

medianAutoAscendscore

0

以

F Evaluation Details

For all policy classes belonging to the LSTM model family, we observe monotonic improvements in the performance of models on withheld instances of NLE as a function of training samples; as a result, we employ the final training checkpoint of these policies when running evaluations across policy seeds. In contrast, the generalization capabilities of models in the transformer-LSTM family do not monotonically improve as a function of training samples, which we interpret as an indicator of overfitting. Thus, evaluations are conducted only for the 'best' checkpoints corresponding to each policy seed, as evaluated on the basis of the rolling NLE score proxy metric. An in-depth description of this metric, as well as experiment training curves supporting the claims of over- and underfitting across model classes, can be found in Appendix G.

Two classes of evaluations are conducted in this paper for such checkpoints: a 'standard evaluation' of policy NLE score across randomly sampled and withheld instances of environment, and an 'indepth evaluation,' recording all metrics of game-play and employing precisely the same set of seeded environment instances to evaluate all policies.

The former policy evaluation procedure mirrors the one conducted during the NeurIPS 2021 NetHack Challenge Competition [23]. This is the procedure we employ to compute the mean and median NLE scores associated with policy seeds for all experiments in this paper as well as to compute the estimates of AutoAscend mean and median NLE score in Table 2, producing our core results.

The latter policy evaluation procedure yields a suite of more fine-grained metrics for informed and 'human-like' game-play in NetHack, such as maximal dungeon level reached and the total life-time of the agent. We run this evaluation procedure for each of the best-performing 5 seeds from each neural policy class, as well as for AutoAscend . Metrics computed with this procedure are denoted via the ( † ) symbol throughout the paper.

Standard Evaluation Policies are evaluated on a randomly seeded batch of 1024 (withheld) NLE games. Only the final NLE scores at the end of game-play are recorded.

In-Depth Evaluation Policies are evaluated across precisely the same seeded batch of 3402 (withheld) NLE games, i.e. all agents play precisely the same set of starting roles across the same NLE dungeon configurations, none of which are covered in HiHack . All games are recorded to the ttyrec data format, and can be streamed post-facto .

A visualization of the distribution of starting roles covered in this evaluation, as well as the corresponding AutoAscend score distributions (factorized by role across game instances), are shown in Figure 10.

Model Parameter Scaling:From 50.8M to 98.6M Parameter (Transformer + LSTM) Policies

PolicyPerformancevs.TrainingSamples

3000

1.0

rolling score (NLE)

PolicyArchitecture

0.8

2000

loss

0.6

Transformer

BC

+LSTM

1000

0.4

+LSTM (large)

Transformer

0.2

0

0.0

0

2

4

6

0

2

4

6

BCsamples(1e8)

BCsamples(1e8)

G Training Curves

We provide training curves reflecting all conducted experiments. In Figures 11 and 12, we display both rolling NLE scores as well as BC loss curves as a function of training samples, across all model and data scaling experiments presented in Section 5. In Figure 13, we display aggregate rolling NLE scores as a function of training samples for all remaining BC and APPO + BC experiments, separated according to model family.

Rolling NLE Score The rolling NLE score metric introduced and displayed in the figures discussed here reflects an evaluation of policy performance on withheld NLE instances conducted continually during model training in a 'rolling' fashion via a fixed number of CPU workers. As such, this metric is biased towards shorter-length games, with the value of smoothed rolling score as a function of

DatasetScaling:From10 to100,000AutoAscendGames

PolicyPerformancevs.TrainingSamples

LSTM

Transformer+LSTM

3000

3000

rolling score (NLE)

rolling score (NLE)

2000

2000

1000

1000

Dataset Size

(games)

0

1e1

0

20

40

0

2

4

6

BC samples (1e8)

BC samples (1e8)

1e2

(Lossvs.TrainingSamples)

1e3

LSTM

Transformer+LSTM

1.00

1.00

1e4

0.75

0.75

1e5

BCloss

BC loss

0.50

0.50

0.25

0.25

0.00

0.00

0

20

40

0

2

4

6

BCsamples(1e8)

BC samples (1e8)

training sample confounded by the policy-specific relationship between NLE score and total game turns. Rolling NLE score is thus not interchangeable with the large-batch 'standard evaluations' employed elsewhere in this paper and presented in-depth in Appendix F. However, unlike BC loss, it serves as an efficient and useful (if noisy) proxy measure of policy generalization over the course of training.

Model Parameter Scaling Training Curves As shown in Figure 11, the generalization capability of transformer-LSTM policies (approximated via rolling NLE score) peaks soon after the start of training, decaying and flattening out as training proceeds despite continued improvement in BC loss. This observation supports the claim made in Section 7 that transformer based models overfit to HiHack .

Despite the aforementioned noisy nature of rolling NLE score, the 'Policy Performance vs. Training Samples' curves on the left of Figure 11 allude to our large-scale 'standard evaluation' finding from Section 5; namely, that policy performance does not increase when model parameter count is scaled up, even after training hyperparameters are tuned. Similarly, the 'Loss vs. Training Samples' curves on the right of this figure indicate nearly identical training errors across both models as a function of BC training samples.

Dataset Scaling Training Curves In Figure 12, we note that for datasets consisting of, or exceeding, 1000 AutoAscend games, LSTM policies do not appear to overfit over the course of our BC experiments, with rolling NLE score consistently monotonically increasing as a function of BC training samples for all such policies. However, a positive relationship between dataset size and maximal rolling NLE score over training persists, indicating that the addition of more data does lead to measurable (if sub log-linear) improvements in policy generalization. An inspection of

LSTM policy loss curves suggests a similar story. Losses across policy seeds trained on 10 and 100 AutoAscend games drop swiftly to values near zero, supporting this overfitting hypothesis.

The transformer-LSTM policy loss curves in Figure 12 also reveal harsh overfitting for policies trained with 10 and 100 games. Interestingly, the generalization capability of these policies, again indicated by rolling NLE score over training, is vastly superior to that of their LSTM counterparts. Indeed, we observe that a transformer-LSTM policy trained on just 100 games vastly outperforms a pure LSTM policy trained on 10x as many games. We attribute this gap to the frozen nature of the pre-trained LSTM component of the transformer-LSTM policies employed as a recurrent encoder in these policies, and hypothesize that it is the static quality of the recurrent representation output by this encoder which bolsters the generalization capability of the resultant models in exceptionally low-data regimes.

Aggregate Policy Performance vs. Training Samples

(BC Policies)

LSTMvs.HierarchicalLSTM

Transformer+LSTMvs.HierarchicalTransformer+LSTM

3000

3000

rolling score (NLE)

score (NLE)

2000

2000

1000

rolling

1000

PolicyArchitecture

0

LSTM

0

20

40

0

2

4

6

8

LSTM +

BC samples (1e8)

BC samples (1e8)

(APPO + BC Policies)

Hierarchical

LSTM

LSTMvs.HierarchicalLSTM

Transformer+LSTMvs.HierarchicalTransformer+LSTM

Transformer+

0

5

10

15

0

2

4

6

8

LSTM

3000

3000

Hierarchical

rolling score (NLE)

RL samples (1e8)

roling score(NLE)

RL samples (1e8)

Transformer

+ LSTM

2000

2000-

1000

1000

0

0

0

5

10

15

0

2

4

6

8

BC samples (1e8)

BC samples (1e8)

Aggregate BC and APPO + BC Training Curves The 'Aggregate Policy Performance vs Training Samples' curves of Figure 13 align with the general model family training trends previously observed in the scaling experiments. Notably, we find once again that LSTM based policies' rolling NLE scores improve monotonically with training samples whether training is conducted with BC or APPO + BC, while this is not the case for transformer-LSTM based models. Indeed, the generalization properties' of policies belonging to this model family improve at the start of training before worsening as training proceeds across BC experiments. We interpret continued demonstration of these trends as further support for the claims of LSTM underfitting and transformer-LSTM overfitting made in Section 7 of the main body of the paper as well as in Appendix F.

Moreover, we note that the introduction of an RL loss induces a particularly large amount of volatility in the rolling NLE score associated with transformer-LSTM based models, disrupting the monotonically decreasing relationship between score and samples previously observed for these policies following ≈ 2 · 10 8 training samples in the pure BC experiments (i.e. the eventual overfitting behavior). This observation suggests that the performance of policies belonging to this model family is bottlenecked by an insufficient total throughput of 'on-policy' or interactive data.

FlatTransformer andTransformer+LSTMPolicyPerformancevs.TrainingSamples

AllTransformerPolicies

FlatTransformersOnly

PolicyArchitecture

3000

200

Transformer+

LSTM

(7N)

score (NLE)

150

Hierarchical

2000

Transformer

score

+ LSTM

100

FlatTransformer

rolling

1000

rolling

(c =64)

50

FlatTransformer

(c =128)

0

0

FlatTransformer

0

2

4

6

8

0

1

2

3

4

(c=256)

BC samples(1e8)

BC samples(1e8)

Ablating the LSTM Component of transformer-LSTM Policies To establish and confirm the importance of the recurrent LSTM component of the transformer-LSTM policies introduced in this paper, we perform a set of ablation experiments with policies featuring a transformer core module only, meaning all input is confined strictly to the transformer's context window at each iteration of training and inference. We refer to models with such an architecture as flat transformers . As demonstrated in Figure 14, we find flat transformers to perform substantially worse in evaluation when trained with behavioral cloning on AutoAscend demonstration data than either their pure LSTM or transformer-LSTM counterparts. The context-length parameter, denoted as c in the legend of Figure 14, reflects the number of contiguous observations employed for action prediction. We perform a small sweep over values of this parameter in an effort to establish the relationship between observation history and model generalization. Reported results reflect the best performing flat transformer architecture after conducting hyperparameter tuning across transformer encoder layer size, layer count, and attention head count.

We make two key observations: (1) the number of samples seen during the 48-hour training window is inversely proportional to the context length employed in training; (2) the rate of BC policy improvement does not increase with context length, across the set of context length values tested here. It is on this basis of (2), coupled with the fact that the average AutoAscend demonstration in our HiHack dataset consists of 27000 keypresses (Table 1), several orders of magnitude beyond the longest context length, c = 256 , we find feasible to test for flat transformers, that we hypothesize that an encoding of full game history is important for imitation in our setting.

H Low-Sample Hypothesis Testing

To establish the statistical significance of the core BC and APPO + BC empirical results reported in the main body of this paper, we perform a series of two sample t-tests with a decision threshold of α = 0 . 05 . The results of these t-tests are summarized in Tables 7, 8, and 9. We confirm all reported findings.

I Distributional Visualizations of Evaluation Results

Max Dungeon Level Reached vs. Total Turns In Figure 15, we supplement the absolute mean and median max-dungeon level and agent life-time (in game turns) statistics introduced in Table 2 with 2-D distributional visualizations of both the raw values of these metrics as well as their mutual inter-relationship, evaluated via n = 3402 seeded NLE games run individually for all policy class representatives in our 'in-depth evaluation.'

The AutoAscend 2-D contour plot on the right of this figure reveals an interesting emergent property of the bot's behavior, which is confirmed by study of individual games: the high-level 'descent behavior' of AutoAscend appears to fall along one of two modes. In the first of these modes, the bot spends a very large amount of turns on dungeon level 1, and avoids descending further into the dungeon before the end of the game. In the second mode, the bot begins to rapidly descend fairly deep into the dungeon, reaching as far as level 11 in the sample of game seeds tested here. The overall

NLE Max Dungeon Level Reached (t) vs. Total Turns (t)Across Neural Policy Classes

(BC Policies)

LSTM

Hierarchical LSTM

3.7e-04

2.9e-04

4

3.1e-04

4

2.7e-04

2.7e-04

2.4e-04

2.2e-04

1.8e-04

(N I

3

1.7e-04

2.0e-04

1.4e-04

1.3e-04

9.9e-05

2

9.4e-05

6.1e-05

6.0e-05

2.6e-05

2.9e-05

10000 2000 30000 40000

5.5e-06

0

10000 200030000 40000

5.2e-06

(3N) SU24

turns (NLE)

Transformer + LSTM

Hierarchical Transformer+LSTM

1.6e-04

1.4e-04

1.2e-04

4

1.2e-04

9.7e-05

9.9e-05

7.8e-05

8.2e-05

5.8e-05

6.5e-05

4.1e-05

4.7e-05

(AutoAscend)

3.0e-05

0-0

1

1.9e-05

8.9e-06

2.2e-06

8.9e-06

1.9e-05

2.1e-06

12

1.6e-05

10000200003000040000

0

10000200003000040000

10

1.3e-05

(N) SUn,

(N) SUn,

9.8e-06

(APPO + BC Policies)

(3TN) I∧IP

8

7.6e-06

6

6.1e-06

LSTM

Hierarchical LSTM

4.8e-06

2.5e-04

2.0e-04

4

3.7e-06

4

2.2e-04

1.8e-04

2.0e-04

1.5e-04

2.5e-06

(N

1.4e-04

1.7e-04

司

3

1.1e-04

1.3e-04

2

1.4e-06

2

1.1e-04

8.2e-05

2

8.5e-05

6.0e-05

0

3.4e-07

5.2e-05

3.8e-05

0

20000

40000

1

2.5e-05

3.8e-06

1.6e-05

2.9e-06

turns (NLE)

10000 2000 30000 40000

10000200003000040000

(N) sSUn

turns (NLE)

Transformer + LSTM

HierarchicalTransformer+LSTM

1.4e-04

1.2e-04

1.3e-04

1.1e-04

1.1e-04

9.2e-05

(NAP

3

9.2e-05

(37N) I

3

7.3e-05

7.2e-05

5.4e-05

3.3e-05

5.2e-05

2

3.9e-05

2.8e-05

2.1e-05

1.8e-05

1.0e-05

8.6e-06

1

2.2e-06

1

2.0e-06

0

10000

200003000040000

10000200003000040000

(N) SUN

(3N) SU2,

relationship between AutoAscend 's total in-game life-time and max-dungeon level reached appears to be roughly quadratic.

In contrast, none of the evaluated neural policy class representatives comes close to achieving in-game progress resembling AutoAscend 's secondary behavior mode. A large majority of games for all neural policy classes appear to end on the first level of the dungeon, with policies very rarely surviving as long on this level as AutoAscend . This suggests that neural policies may be failing to master the long-range, very low-level behaviors of the bot, even when these behaviors are factorized across AutoAscend strategies, as in the case of hierarchical policy variants.

Nevertheless, the qualitative performance of hierarchical policies clearly improves upon that of nonhierarchical models, with the hierarchical transformer-LSTM policy trained with BC both surviving longer of dungeon level 1 and descending with higher frequency than other BC-trained policy representatives. Furthermore, this qualitative behavior appears to be strengthened when interaction is added into the mix, with the mode centered on 'dungeon level 2' increasing in density for the APPO + BC variant of the hierarchical transformer-LSTM representative policy.

Taken together, these sets of observations lead us to hypothesize that the quality of neural policies trained with imitation learning on extremely complex, long-horizon tasks like NetHack may be

further improved with increases in the scale of hierarchically-informed, fixed behavioral factorization, beyond that explored in this paper. We consider this to be a very exciting direction for future work.

Log-Score Distribution (t) Across Neural Policy Classes

(BC Policies)

LSTM

Hierarchical LSTM

0

16

256

4096

65536

0

16

256

4096

65536

300

300

counts

200

counts

200 十

100

100

8

12

16

8

12

16

log2(score)

log2(score)

Transformer+LSTM

HierarchicalTransformer+LSTM

0

16

256

4096

65536

0

16

256

4096

65536

(AutoAscend)

300

300

0

16

256

4096

65536

200

200

100

100

300

0

8

12

16

8

12

16

counts

200

log2(score)

log2(score)

(APPO+BC Policies)

100

LSTM

Hierarchical LSTM

0

16

256

4096

65536

0

16

256

4096

65536

0

300

300

4

8

12

16

log2( score)

counts

200

nos

200

100

100

4

log2(score)

8

12

16

log2(score)

12

16

Transformer + LSTM

Hierarchical Transformer+LSTM

0

16

256

4096

65536

0

16

256

4096

65536

300

300

200

counts

200

100

100

0

log2(score)

8

12

16

8

12

16

log2(score)

Log-Score Distributions We conclude this supplementary analysis with a visualization of log-score distributions across neural policy classes in Figure 16, computed over the 'in-depth evaluation' seeded games. The trends displayed in this figure align with those described and demonstrated previously. Improvements in model architecture as well as the introduction of hierarchy and interactive learning lead to a re-distribution of 'mass' between left and right tails of distribution, with the overall counts of 'low score' games decreasing and 'high score' games increasing when these improvements are applied. The gap to AutoAscend is significantly reduced, but not bridged.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to study and better understand the large performance gap between neural and symbolic agents in the NeurIPS 2021 NetHack Challenge. The main hypothesis is that symbolic agents advantage derives from hierarchical reasoning, which was not an element in participating neural agents. To test this hypothesis, a new dataset is generated using the winning symbolic agent that records both actions and higher-level strategic labels and a neural behavior cloning agent was trained with this augmented data. Beyond this, the paper also explores the impact of increased model size and dataset size, changes to the neural architecture, and the addition of a policy fine-tuning step using a reinforcement learning algorithm. The results suggest that hierarchical training improves the neural agent's performance significantly more than increased model capacity but that more powerful model architectures (i.e. a transformer-based model) could overfit to the augmented data.\n\nSTRENGTHS:\nQuality/Soundness\n\nThe hypotheses and claims of the paper were laid out clearly and the experiments were well-designed to evaluate them. \n\nClarity/Presentation\n\nI found the paper to be clearly presented and well-written. At each stage I had a clear understanding of the question under investigation and the methodology for studying it.\n\nOriginality/Contribution\n\nThe paper is explicit that its main contribution is not algorithmic but scientific. Since the scientific questions posed here are grounded in the performance of agents from a specific competition that happened last year, I expect that this analysis is original.\n\nSignificance/Contribution\n\nIncreasing understanding of the performance gap in the NetHack problem, as a proxy for complex, long-horizon problems in general, could be important. Symbolic approaches make use of quite a lot of domain expertise applied to constructing the symbolic structure, making them effective in their target problem but inflexible. Neural networks seem to be quite flexible and capable of learning without a lot of structure engineering, but don't seem to be able to take advantage of structure in the environment. It seems sensible to study the primary factors preventing neural approaches from building the same long-range structures.\n\nI think it is worthwhile to see the comparison between this structural change and the alternative interventions of more data, more parameters, or more sophisticated/expensive architecture. The finding that, in a time-constrained setting (true of many decision-making problems), model structure may be more important than model capacity seems likely to at least spark interesting conversations within the community. The fact that there is plenty of performance gap still to cover, even with this built-in domain knowledge may also inspire further investigation into what measures could come closer to closing the gap and how those insights might be applied to more general practice.\n\nOverall\n\nOverall, I find this to be a clearly written paper with a reasonable scientific question and sound methodology to address it (modulo some missing statistical analyses). The findings are not revolutionary but, to my eyes, they do provoke further questions about neural approaches to learning in complex, long-horizon problems and may inspire follow-up work either in the NetHack testbed specifically or in studying these questions more generally.\n\nWEAKNESSES:\nQuality/Soundness\n\nMy main concern in this area is the small number of independent samples per model class (6). I do understand that these results are generated at great expense and that it may not be feasible to generate more trials, but the small number of samples diminishes the statistical power of these analyses. I can see that the error bars are quite small; assuming those are showing the standard error, that's encouraging. Nevertheless, whether more samples can be generated or not, I think it's important that the paper include the findings of low-sample hypothesis testing (e.g. t-test) on these results; without that, we can't confidently distinguish between noise and meaningful differences.\n\nOriginality/Contribution\n\nThe paper acknowledges that behavior cloning, hierarchical policies, and transformer-represented policies have all been studied in prior work. \n\nSignificance/Contribution\n\nNetHack is not, in and of itself, an intrinsically important problem to solve. \n\nThe results presented here are not conclusive or enormously surprising. The main result is fairly predictable: adding explicit supervision about high-level strategy and explicit hierarchical structure in the model helps the model take advantage of hierarchical structure in the environment.\n\nOverall\n\nOverall, I find this to be a clearly written paper with a reasonable scientific question and sound methodology to address it (modulo some missing statistical analyses). The findings are not revolutionary but, to my eyes, they do provoke further questions about neural approaches to learning in complex, long-horizon problems and may inspire follow-up work either in the NetHack testbed specifically or in studying these questions more generally.\n\n---After discussion---\n\nI have considered the other reviews and the authors' responses. I continue to feel confident about my overall assessment.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nSince the paper does not propose significantly new algorithmic ideas, the main source of limitations would be in the methodology and analysis. I generally found the paper to avoid overclaiming. I've already discussed one area where this aspect of the paper could be improved: acknowledgement of the small sample sizes and proper statistical analysis to inform the conclusions. The other area might be in the conclusions where perhaps the summary of the findings might be a bit too general and could be toned down and/or stated clearly as hypotheses (e.g. \"Hierarchy hurts overfitting models\" is an overly broad conclusion from a limited set of experiments but seems like a reasonable hypothesis given these results).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Problem Statement\nThe paper addresses the challenge of neural policy learning methods struggling in long-horizon tasks, particularly in open-ended environments with multi-modal observations, such as the game NetHack. It was observed that symbolic agents significantly outperformed neural approaches in the NeurIPS 2021 NetHack Challenge.\n\n# Main Contribution\nThe paper's main contribution is an extensive study on neural policy learning for NetHack. The authors analyzed the winning symbolic agent and extended its codebase to generate one of the largest available demonstration datasets. They examined the advantages of an action hierarchy, enhancements in neural architecture, and the integration of reinforcement learning with imitation learning. Their investigations resulted in a state-of-the-art neural agent that surpassed previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, they also demonstrated that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players.\n\n# Methodology and Experiments\n\n## The Hierarchical HiHack Dataset\nThe authors create the HiHack dataset, which is a hierarchically-informed version of the NetHack Learning Dataset (NLD-AA), containing 3 billion recorded game transitions from over a hundred thousand games played by the AutoAscend agent.\n\n## Hierarchical Behavioral Cloning\nThe authors extend the ChaoticDwarvenGPT5 (CDGPT5) model, a top-performing open-source neural model for NetHack, by introducing a hierarchical decoding module. The model consists of three separate encoders for different types of observations and an LSTM core module. The hierarchical extension replaces the linear decoder of the CDGPT5 model with a hierarchical decoder that predicts the strategy label and selects the appropriate low-level MLP for action prediction. The hierarchical LSTM policy and the baseline non-hierarchical LSTM CDGPT5 policy are trained using a simple cross-entropy loss. The results show that the introduction of hierarchy significantly improves the performance of LSTM policies trained with behavioral cloning, yielding a 40% gain over the baseline in mean NLE score and a 50% improvement in median score across seeds. The authors confirm that this improvement is due to hierarchy and not simply a result of the increased parameter count of the hierarchical LSTM policy.\n\n## Architecture and Data Scaling\nThe authors explored scaling as a potential solution to improve the performance of the model, which was significantly behind the symbolic policy used to generate the HiHack demonstrations. They developed a novel base policy architecture for NetHack that introduces a Transformer module into the previous CDGPT5-based architecture. They also conducted data scaling experiments using subsets of the HiHack dataset to examine the relationship between dataset size and the test-time performance of BC policies. The results showed that both the non-hierarchical and hierarchical variants of the combined transformer-LSTM policy architecture yielded gains, but the larger model performed worse than the smaller one due to overfitting. This suggested that scaling of model capacity alone would not be sufficient to close the neural-symbolic gap. Additionally, brute force scaling of the dataset alone could not viably close the gap to symbolic methods.\n\n## Combining Imitation with Reinforcement Learning\nThe authors explored combining imitation learning with reinforcement learning (RL) to bridge the performance gap with AutoAscend. They used a combination of behavioral cloning (BC) and asynchronous proximal policy optimization (APPO) for training. The results showed that RL fine-tuning significantly improved the performance of all models. The best-performing approach was APPO + BC using the hierarchical LSTM model, which achieved a new state-of-the-art for neural policies on NLE, surpassing the previous best result by 48% in mean NLE score and 25% in median NLE score. The Transformer-LSTM models performed worse due to their slower training speed and the fixed training time budget. The authors also observed that fine-tuning with RL improved the error-correction capability of models across all model classes compared to their purely offline counterparts.\n\nSTRENGTHS:\n# Originality\nThe problem is interesting and the approaches are insightful.\n\n# Quality\nThe analysis and experiments are comprehensive.\n\n# Clarity\nThe article is overall well written and clear.\n\nWEAKNESSES:\n1. The current focus of the study is quite narrow, being primarily centered on the application of imitation learning for NetHack, limiting its influence. In the context of mastering the game, while this approach is interesting, it is unlikely to exceed the performance of experts that generate demonstrations, not to mention that the experts are already algorithms that can scale well. Furthermore, NetHack, despite being an excellent game, is somewhat niche and its real-world implications are relatively minimal. The techniques proposed in this study are specifically tailored for this game, which limits their potential for inspiring more universally applicable methods that could have a broader impact.\n  - The availability of hierarchical labels is a strong assumption that does not often hold, which further limits the applicability of the proposed methods.\n\n2. Even just for bridging the performance gap between neural models and AutoAscend, there is no promising direction revealed by the work as the various augmenting components seem to contradict each other.\n\nQUESTIONS:\n1. When introducing Transformer to augment the capacity of the neural model, why did authors choose the architecture as shown in the article? Specifically, transformers are best known for their NLP and CV capacity, which could make them good replacement for the CNN and MLP encoders.\n2. Why do the authors enforce the 48 hour training time cap instead of training all models till convergence? Given that this study does not appear to prioritize data efficiency or training efficiency, the necessity of such a computational time constraint is unclear. It would be beneficial to understand the rationale behind this choice, as it may not directly align with the study's primary objectives.\n\nLIMITATIONS:\nThe authors note that possible avenues for future exploration include: (a) methods for increasing the Transformer context length to give the agent a longer memory to aid exploration; (b) addressing the multi-modal nature of the demonstration data (i.e. quite different trajectories can lead to the same reward), which is a potential confounder for BC methods. Some forms of distributional BC (e.g. GAIL, BeT) could help alleviate this issue.\n\nThe aforementioned two points do not address the limitations raised in the \"Weakness\" section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper improves the existing solutions in the NetHack Learning Environment (NLE). This is done by taking earlier solutions from a competition around NLE, collecting more data with the best available (symbolic) agent, and using that data to improve a neural only solution. The paper provides experiments with imitation learning (with or without RL tuning), larger models, hierarchical memory setup (LSTM + Transformers) as hierarchical behavioural cloning setup, using labels of the newly collected dataset. While there are improvements, it is still below the demonstrator results, which is then studied by scaling the model sizes and amount data collected. Paper concludes by providing the state of the art results in the task, but also noting that scaling alone is not enough to reach the expert demonstrator level (symbolic agent).\n\nSTRENGTHS:\n- Provides more detailed dataset than the previous works (with hierarchical action labels)\n- Sets an interesting premise/task for trying to reach the demonstrators' (AutoHack agent) performance with neural solutions.\n- Different ablations to try to answer questions (data/model scaling, model architecture with hierarchy)\n- Proposed hierarchical approach to imitate the demonstrator agent.\n\nWEAKNESSES:\nWhile I enjoyed reading the paper, overall I think the results are interesting or applicable to most of the NeurIPS audience, even in the limited scope. The paper presents many results and provides some explanations for them, but does not verify these explanations with further experiments. I think proper answers to these issues would be insightful to many, and others could then use these insights in their work (e.g., where the trained agent failed to imitate the demonstrator? What was the cause of poorer performance? Why did bigger model perform worse?). Creating such insight in one environment would be sufficient, as by focusing on a single environment, you can create very specific scenarios to tease out these answers. \n\n- Limited scope of the work: experiments done in a single environment. Most of the paper is framed in a way that this is not a huge issue (e.g., ablations), but proposing new method just for playing NLE has limited impact. If a new method is proposed to generally improve RL/IL performance, it should be tested at least in two distinct environments.\n- Limited improvement in the context of SOTA solutions: 2x over the baseline used in the paper with RL and proposed architecture included, but other neural agents in the NetHack Challenge had higher score. To be interesting in terms of performance, it should at least outperform the NetHack Challenge Neural solutions.\n- Proposed method is limited in novelty, as evident by the previous work listed in the paper. If the hierarchical BC figured out the hierarchy automatically (or, if it was an emergent behaviour of the model), that would be more interesting.\n- Paper outlines some assumptions on why things failed (e.g., \"model overfitted\" or \"learned to self-correct\"), but these claims were not verified with results. The paper would be much stronger if you can give solid, verified answer that indeed, overfitting was to blame or that RL trained the model to \"self-correct\".\n\nQUESTIONS:\nQuestions:\n1) In multiple occasions paper says that the lower performance of bigger model is due to overfitting (e.g., line 229). However there are no results/experiments to show that this indeed was the case. A simple way to find this out is to do train-validation (or even train/validation/test) split, and testing on held out data as training progresses.\n2) Regarding data scaling experiments: did you change any other settings of the training setup when increasing data amount? Previous work has demonstrated that the optimal model size and/or training compute depends on the amount of data (Hoffmann et al. 2020).\n3) Regarding model scaling experiments: I assume only the number of layers in the transformer was changed? The bottleneck of the network may be elsewhere, e.g., one of the input layers or output layers. I would recommend scaling the whole network, similar to what OpenAI VPT work did, where ResNet blocks were \"widened\" in terms of filters, as well as increasing transformer size (Baker et al. 2022). Also, Hoffmann et al. (2020) changed number of layers, number of attention heads and transformer dimensionality when scaling models. This might be something you want to try.\n4) Instead of LSTM + Transformer model, did you experiment with transformer model only? E.g., akin to VPT work (Baker et al. 2022), embed all inputs into one vector, stack vectors over timesteps, apply causal transformer, and predict actions from the transformer outputs. This type of model might scale better, as it reduces the amount of components that might interfere.\n\n#### Comments (not questions)\n- Fig1 right: weird scale. Any chance to get more points?\n- Line 205: grammar error at the start of the line\n- Explain/rename \"Dlvl\" and why \"Turns\" is good metric\n- Figure 3: \"LSTM + XXL Dec\" is bit confusing naming, since \"decoder\" is not commonly used term in the paper. I'd recommend using something like \"LSTM (bigger)\" to simply reflect that it is the LSTM baseline but with bigger network\n- Figure 3 (and others): add explanation to caption what is the error bar of the bar plots. Is it standard deviation or standard error (or something else)?\n- Table 2 caption: starts with weird \"[V4]\"\n\n#### References\n\n- Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas et al. \"Training compute-optimal large language models.\" arXiv preprint arXiv:2203.15556 (2022).\n- Baker, Bowen, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. \"Video pretraining (vpt): Learning to act by watching unlabeled online videos.\" Advances in Neural Information Processing Systems 35 (2022): 24639-24654.\n\nLIMITATIONS:\nNo explicit sections for limitations or broader/societal impact was given. Authors bring up the future work ideas in the conclusion. While I think the work does not require societal impact section (no immediate impact), I urge authors still think through of any cases where the work or the insights could impact others. Or alternatively, what impact would _not_ including some results do (e.g., skipping some analysis).\n\n\n## Rebuttal acknowledgement\n\nI have read authors' rebuttal which did address my concerns, and I increased my rating from 4 to 7 to signal my vote to accept this paper (change was done before discussion period closed).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is an emergency review, and I regret that the paper is out of my expertise, which is why my review will rather stay at the surface level.\n\nThe paper is concerned with the NetHack challenge, a complex AI challenge that in 2021 reached headlines, because symbolic agents considerably outperformed neural agents. I see three main contributions in the paper:\n - The construction of a large-scale dataset, based on the best symbolic agent and its policy choices, that can enable training better neural agents\n - The training of better neural agents based on this dataset, and other improvements\n - A systematic analysis of the effect of different technical improvements (hierarchical BC, larger Transformer models, larger datasets, online fine-tuning with RL), notably finding that scaling training sets or model size alone will not bridge the gap to the best symbolic agent.\n\nThe problem is of very high interest to the AI community, and the technical investigation, results, and discussion appear thorough and insightful. The dataset might also enable further research. I find especially the results regarding scaling interesting, i.e., that performance increases logarithmic, and so more data or bigger models alone will not enable achieving parity with the symbolic approach.\n\nQuality of writing is very good, and so the paper is easy to follow (subject to my lack of technical background).\n\nMinor notes:\n - The paper appears to be missing a link to the dataset\n - The related work is not easy to access for someone not close to the field. E.g., paragraphs on \"imitation learning\" and \"hierarchical policy learning\" give too little detail about the basic ideas (do not start with descriptions of what they are for, but what they do)\n - \"The full observation space of NLE is far richer and more informed than the view afforded to human players of NetHack, who observe only the more ambiguous “text-based” components of NLE observations\" - I do not fully understand this sentence, please expand. What can systems observe in NLE, that humans don't receive in the original interface? Or do you mean that NLE aggregates the Ascii terminal characters into something more high-level?\n - Showing an excerpt from the dataset would be helpful, especially, as it is not quite clear what is added there, both strategies and substrategies? Or the more specific one only?\n\nSTRENGTHS:\nSee above.\n\nWEAKNESSES:\nSee above.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes, the authors critically discuss that scaling alone will not bridge the gap to symbolic agents on this challenge.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores reasons for this performance gap between neural and symbolic methods in NetHack:\nSymbolic agents use hierarchical policies and parsers to extract high-level features\nSymbolic agents have handcrafted heuristics and error correction\nNeural agents lack inductive biases like hierarchy that may be needed for sparse rewards\nExperiments show hierarchy, scale, and combining imitation and RL help improve neural agents:\nHierarchical behavior cloning improves over flat BC\nLarger Transformer-based architectures improve over LSTMs\nRL fine-tuning provides gains, especially for underfitting models\nBut significant gaps to symbolic agents remain\n\nSTRENGTHS:\nThe experimental design is very clever, the chart is very clear, and the experimental effect is obvious. The paper explores a novel problem domain of applying neural networks to master the game NetHack, where current methods struggle compared to symbolic AI. The authors introduce a new large-scale dataset of NetHack demonstrations called HiHack to facilitate this analysis. The idea of using demonstrations to help neural networks learn better policies in sparse, long-horizon environments like NetHack is creative.The methods are detailed appropriately to replicate experiments. Results are presented logically and incorporate useful visualizations. The conclusion summarizes takeaways concisely.Mastering complex environments like NetHack with sparse rewards and long time horizons remains an open challenge for deep RL. This paper provides significant evidence and analysis characterizing the limitations of current neural network methods in these settings, and points the way towards progress, whether via incorporating stronger inductive biases like hierarchy or combining neural and symbolic approaches. The insights will broadly impact research in sparse reward RL, imitation learning, and integrating neural and classical AI.\n\nWEAKNESSES:\nThis model is based on the nethack, and the results hold up on the above models, and whether the above results can still hold up on the other models。The authors recognize the limited generality so far of methods tested on NetHack to other complex environments.No obvious harmful biases or problematic data sources are introduced in this work. The NetHack environment itself seems relatively innocuous.\n\nQUESTIONS:\nCan you add some experiments, add some theoretical derivation, whether the contribution of this article is more.\n\nLIMITATIONS:\nThe model is not so representative, can switch a more popular model。Overall, the authors demonstrate good care and thoughtfulness regarding the limitations and potential negative impacts of this research direction. The discussion seems sufficient without being overreaching or distracting from the primary technical contributions. I do not have any major suggestions for improvement.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}