SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ulyana Piterbarg* NYU
Lerrel Pinto NYU
Rob Fergus NYU
Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in long-horizon tasks, especially in open-ended environments with multi-modal observations, such as the popular dungeon-crawler game, NetHack. Intriguingly, the NeurIPS 2021 NetHack Challenge revealed that symbolic agents outperformed neural approaches by over four times in median game score. In this paper, we delve into the reasons behind this performance gap and present an extensive study on neural policy learning for NetHack. To conduct this study, we analyze the winning symbolic agent, extending its codebase to track internal strategy selection in order to generate one of the largest available demonstration datasets. Utilizing this dataset, we examine (i) the advantages of an action hierarchy; (ii) enhancements in neural architecture; and (iii) the integration of reinforcement learning with imitation learning. Our investigations produce a state-of-the-art neural agent that surpasses previous fully neural policies by  $127\%$  in offline settings and  $25\%$  in online settings on median game score. However, we also demonstrate that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players.
Reinforcement Learning (RL) combined with deep neural policies has achieved impressive results in control problems, such as short-horizon simulated locomotion tasks [55, 7]. However, these methods struggle in long-horizon problem domains, such as NetHack [34], a highly challenging grid-world game. NetHack poses difficulties due to its vast state and action space, multi-modal observation space (including vision and language), procedurally-generated randomness, diverse strategies, and deferred rewards. These challenges are evident in the recent NetHack Challenge [23], where agents based on hand-crafted symbolic rules outperform purely neural approaches (see Figure 1), despite the latter having access to high-quality human demonstration data [24] and utilizing large-scale models.
We propose three reasons for the poor performance of large-scale neural policies compared to symbolic strategies. First, symbolic strategies implement hierarchical control schemes, which are generally absent in neural policies used for NetHack. Second, symbolic models use hand-crafted parsers for multi-modal observations, suggesting that larger networks could enhance representations extracted from complex observations. Third, symbolic strategies incorporate error correction mechanisms, which could be crucial for improving neural policies if integrated with RL based error correction.
In this work, we conduct a comprehensive study of NetHack and examine various learning mechanisms to enhance the performance of neural models. We bypass traditional RL obstacles, such as sparse rewards or exploration challenges, by focusing on imitation learning. However, we find that existing datasets lack crucial information, such as hierarchical labels and symbolic planning traces. To address this, we augment the codebase of AutoAscend, the top-performing symbolic agent in the 2021 NetHack Challenge, and extract hierarchical labels tracking the agent's internal strategy selection in order to construct a large-scale dataset containing  $10^{9}$  actions.
*Correspondence to up2021@cims.nyu.edu.
Figure 1: Left: The per-step observation for NetHack agents consists of an ego-centric pixel image (blue) and two text fields containing in-game messages (green) and statistics (pink). Right: Selected results from the NeurIPS 2021 NetHack Challenge (NHC) [23] showing game score on a log-scale. Error bars reflect standard error. Neural baseline models (grey) trained with BC perform poorly, but are somewhat improved when fine-tuned with RL. We find that the introduction of hierarchy and changes in model architecture yield significant improvements (light/dark blue), resulting in state-of-the-art performance for a neural model. However all neural approaches are significantly worse than AutoAscend, a hand-crafted symbolic policy. Our paper explores this performance gap.
Using this dataset, we train a range of deep neural policies and investigate: (a) the advantages of hierarchy; (b) model architecture and capacity; and (c) fine-tuning with reinforcement learning. Our main findings are as follows:
- Hierarchical behavioral cloning (HBC) significantly outperforms BC and baseline methods, provided that the model has adequate capacity.
- Large transformer models exhibit considerable improvements over baselines and other architectures, such as LSTMs. However, the power-law's shallow slope indicates that data scaling alone will not suffice to solve the game.
- Online fine-tuning with RL further enhances performance, with hierarchy proving beneficial for exploration.
- The combined effects of hierarchy, scale, and RL lead to state-of-the-art performance, narrowing the gap with AutoAscend but not eliminating it.
We validate the statistical significance of our findings with low-sample hypothesis testing (see Appendix H). Additionally, we open-source our code, models, and the HiHack repository<sup>2</sup>, which includes (i) our  $10^{9}$  dataset of hierarchical labels obtained from AutoAscend and (ii) the augmented AutoAscend and NLE code employed for hierarchical data generation, encouraging development.
Though we base our experiments in NetHack, we take measures to preserve the generality of insights yielded by our investigations of neural policy learning. Specifically, we intentionally forgo the addition of any environment-specific constraints in the architectural design or training setup of all models explored in this paper. This is in contrast to NetHack agents RAPH and KakaoBrain, the first and second place winners of the neural track of the NHC Competition, respectively, which incorporate augmentations such as hand-engineered action spaces, role-specific training, and hard-coded symbolic sub-routines [23]. While this choice prevents us from achieving absolute state-of-the-art performance in NLE in this paper, we believe it to be crucial in preserving the general applicability of our insights to neural policy learning for general open-ended, long-horizon environments.
Our work builds upon previous studies in the NetHack environment, imitation learning, hierarchical learning, and the use of transformers as policies. In this section, we briefly discuss the most relevant works.
$^{2}$ Code is available at https://github.com/upiterbarg/hihack.
NetHack Following the introduction of the NetHack Learning Environment (NLE) [34], the NetHack Challenge (NHC) competition [23] enabled comparisons between a range of different agents. The best performing symbolic and fully data-driven neural agents were AutoAscend and Chaotic-Dwarven-GPT-5 (CDGPT5), respectively, and we base our investigations on them, as well as on the NetHack Learning Dataset [24].
Several notable works make use of the NetHack environment. Zhong et al. [61] show how a dynamics model can be learned from the Nethack text messages and leveraged to improve performance; on account of their utility, we also encode the in-game message but instead use a model-free policy to pick actions. Bruce et al. [9] show how a monotonic progress function in NetHack can be learned from human play data and then combined with a reinforcement learning (RL) reward to solve long-range tasks in the game. This represents a complementary way of employing NetHack demonstration data without direct action imitation.
Imitation Learning Pomerleau [47] demonstrated the potential of driving an autonomous vehicle using offline data and a neural network, which has since become an ongoing research topic for scalable behavior learning [3, 6, 50]. Formally, behavior learning is a function approximation problem, where the goal is to model an underlying expert policy mapping states or observations to actions, directly from data. Approaches to behavior learning can be categorized into two main classes: offline RL [21, 32, 33, 58, 36, 20], which focuses on learning from mixed-quality datasets with reward labels; and imitation learning [42, 44, 45, 26], which emphasizes learning behavior from expert datasets without reward labels. Our work primarily belongs to the latter category as it employs a behavior cloning model. Behavior cloning, a form of imitation learning, aims to model the expert's actions given the observation and is frequently used in real-world applications [60, 62, 60, 48, 18, 59].
Since behavior cloning algorithms typically address a fully supervised learning problem, they are often faster and simpler than offline RL algorithms while still yielding competitive results [20, 22]. A novel aspect of our work is the use of hierarchy in conjunction with behavioral cloning, i.e. supervision at multiple levels of abstraction, a topic which has received relatively little attention. Recent efforts to combine large language models with embodied agents use the former to issue a high-level textual "action" to the low-level motor policy. Approaches such as Abramson et al. [1] have shown the effectiveness of hierarchical BC for complex tasks in a simulated playroom settings.
Hierarchical Policy Learning Hierarchical reinforcement learning (HRL) based techniques [5, 4] extend standard reinforcement learning methods in complex and long-horizon tasks via temporal abstraction across hierarchies, as demonstrated by Levy et al. [37] and Nachum et al. [40][41]. Similarly, numerous studies have concentrated on showing that primitives [17, 54, 53] can be beneficial for control. These concepts have been combined in works such as Stochastic Neural Networks by Florensa et al. [19], where skills are acquired during pretraining to tackle diverse complex tasks. Likewise, Andreas et al. [2] learn modular sub-policies for solving temporally extended tasks. However, most prior work focus on learning both levels of the hierarchy, i.e. a decomposition across primitives, skills, or sub-policies as well as the primitives, skills, or sub-policies themselves, which makes training complex. Correspondingly, the resultant approaches have had limited success on more challenging tasks and environments. Le et al. [35] explores the interaction between hierarchical learning and imitation and finds benefits, albeit in goal-conditioned settings. In contrast, our work explores the benefits of access to a fixed hierarchy chosen by the domain-expert designer of AutoAscend, which simplifies our study of overall learning mechanisms for NetHack.
Transformers for RL The remarkable success of transformer models [57] in natural language processing [15, 8] and computer vision [16] has spurred significant interest in employing them for learning behavior and control. In this context, [11, 29] apply transformers to RL and offline RL, respectively, while [12, 14, 38] utilize them for imitation learning. Both [14, 38] primarily use transformers to summarize historical visual context, whereas [12] focuses on their long-term extrapolation capabilities. More recent work have explored the use of multi-modal transformers [27] to fit large amounts of demonstration data [49, 52, 56]. To enable transformers to encode larger context lengths, recurrent transformer models have been proposed [13, 10]. Our work draws inspiration from these use cases, employing a transformer to consolidate historical context and harness its generative abilities in conjunction with a recurrent module.
Figure 2: A diagrammatic visualization of the internal structure of AutoAscend. The bot is composed of eleven goal-directed, high-level strategies. The "global controller" underlying AutoAscend employs a complex predicate based control flow scheme to determine which strategy to query for an action on a per-timestep basis [23].
3 Data Generation: Creating the HiHack Dataset
3.1 Extending the NetHack Learning Environment
The NetHack Learning Environment (NLE) is a gym environment wrapping the NetHack game. Like the game itself, the action and state spaces of NLE are complex, consisting of 121 distinct actions and ten distinct observation components. The full observation space of NLE is far richer and more informed than the view afforded to human players of NetHack, who observe only the more ambiguous "text based" components of NLE observations, denoted as tty_chars, ttycolors, and tty_cursor. This text based view corresponds also to the default format in which both NetHack and NLE gameplay is recorded, loaded, and streamed via the C based ttyrec library native to the NetHack game.
The popular NetHack Learning Dataset (NLD) offers two large-scale corpuses of NetHack gameplay data, NLD-AA, consisting of action-labeled demonstrations from AutoAscend, and NLD-NAO, consisting of unlabeled human player data [24]. NLD adopts the convention of recording only tty* components of NLE observations as a basis for learning, hence benefiting from the significant speedups in data operations offered via integration with the ttyrec library. We adhere to this convention with the hierarchical HiHack Dataset introduced in this paper. Thus, in order to generate our dataset, we extend the ttyrec library to store hierarchical strategy or, equivalently, goal labels alongside action labels. We further integrate this extension of ttyrec with NLE, modifying the gym environment to accept an additional hierarchical label at each step of interaction. This input hierarchical label does not affect the underlying state of the environment, and is instead employed strictly to enable the recording of hierarchically-informed NetHack game-play to the ttyrec data format.
3.2 AutoAscend: A Hierarchical Symbolic Agent
An inspection of the fully open-source code base underlying the AutoAscend bot reveals the internal structure of the bot to be composed of a directed acyclic graph of explicitly defined strategies. The bot's underlying global controller switches between strategies in an imperative manner via sets of strategy-specific predicates, as visualised in Figure 2. Among these strategies are hand-engineered routines for accomplishing a broad range of goals crucial to effective survival in the game and successful descent through the NetHack dungeons. These include routines for fighting off arbitrary monsters, selecting food that is safe to eat from an agent's inventory, and efficiently exploring the dungeon while gathering and identifying valuable items, among many others. The various strategies are supported in turn by shared sub-strategies for accomplishing simpler "sub-goals" (see Appendix B for the full graph).
We exploit this explicit hierarchical structure in the generation of HiHack, extending the AutoAscend codebase to enable per-step logging of the strategy responsible for yielding each action executed by the bot, as supported by our modifications to the C based ttyrec writer library and NLE.
3.3 The HiHack Dataset
Our goal in generating the HiHack Dataset (HiHack) is to create a hierarchically-informed analogue of the large-scale AutoAscend demonstration corpus of NLD, NLD-AA. Thus, as previously described, HiHack is composed of demonstrations recorded in an extended version of the ttyrec format, consisting of sequences of tty* observations of the game state accompanied by AutoAscend action and strategy labels. HiHack contains a total of 3 billion recorded game transitions, reflecting more than a hundred thousand AutoAscend games. Each game corresponds to a unique, procedurally-generated "seed" of the NetHack environment, with AutoAscend playing as one of thirteen possible character "starting roles" across a unique layout of dungeons.
We verify that the high-level game statistics of HiHack match those of NLD-AA in Table 1. Indeed, we find a high degree of correspondence across mean and median episode score, total number of transitions, and total number of game turns. We attribute the very slightly diminished mean scores, game transitions, and turns associated with HiHack to a difference in the value of the NLE timeout parameter employed in the generation of the datasets. This parameter regulates the largest number of contiguous keypresses failing to advance the game-state that is permitted before a game is terminated. The value of the timeout parameter was set to 1000 in the generation of HiHack.
Table 1: A comparison of dataset statistics between NLD-AA [23] and our generated HiHack Dataset, produced by running AutoAscend in NLE v0.9.0 with extended tty* observations.
NLD-AA	HiHack
Total Episodes	109,545	109,907
Total Transitions	3,481,605,009	3,244,729,367
Mean Episode Score	10,105	8,166
Median Episode Score	5,422	5,147
Median Episode Game Transitions	28,181	27,496
Median Episode Game Turns	20,414	19,991
Hierarchical Labels	X	✓
4 Hierarchical Behavioral Cloning
Our first set of experiments leverage the hierarchical strategy labels recorded in HiHack for offline learning with neural policies, via hierarchical behavior cloning (HBC).
Method Mimicking the imperative hierarchical structure of AutoAscend, we introduce a bilevel hierarchical decoding module over a popular NetHack neural policy architecture, namely the ChaoticDwarvenGPT5 (CDGPT5) model. This model achieved 3rd place in the neural competition track of the NeurIPS 2021 NetHack Challenge when trained from scratch with RL, making it the top-performing fully data-driven neural model for NetHack [23].
The CDGPT5 model consists of three separate encoders: a 2-D convolutional encoder for pixelrendered visual observations of the dungeon  $o_t$ , a multilayer perceptron (MLP) encoder for the environment message  $m_t$ , and a 1-D convolutional encoder for the bottom-line agent statistics  $b_t$ . These three observation portions are extracted from the tty* NLE observations of HiHack. The core module of the network is an LSTM, which is employed to produce a recurrent encoding of an agent's full in-game trajectory across what may be hundreds of thousands of keypresses, both in training and at test-time. The core module may also receive a one-hot encoding of the action executed at the previous time-step  $a_{t-1}$  as input.
Our hierarchically-extended version of this LSTM based policy is shown in Figure 3(left). We replace the linear decoder used to decode the LSTM hidden state into a corresponding action label in the CDGPT5 model with a hierarchical decoder consisting of (i) a single "high level" MLP responsible for predicting the strategy label  $g_{t}$  given the environment observation tuple  $\{m_t, o_t, b_t\}$ , and (ii) a set of "low level" MLPs, one for each of the discrete strategies in the AutoAscend hierarchy (see Figure 2), with a SoftMax output over discrete actions. The strategy prediction  $g_{t}$  selects which of these low-level MLPs to use.
Figure 3: Left: Hierarchical LSTM policy, where  $g_{t}$  is the high-level strategy prediction (purple) that is used to select over the  $k$  low-level policies (yellow). Figure 1 shows the three different components of the input observation: message  $m_{t}$ , ego-centric pixel view of the game  $o_{t}$ , and "bottom line" statistics  $b_{t}$ . Right: Mean score for baseline LSTM model [23] (grey), our hierarchical model (blue) at the conclusion of training. The addition of hierarchical labels in decoding provides a significant performance gain, not matched by an (extra) large-decoder version of the baseline (dark grey). Error bars indicate standard error.
We employ a simple cross-entropy loss to train both the baseline non-hierarchical LSTM CDGPT5 policy, as well as our Hierarchical LSTM policy, aggregating gradients across the bilevel decoders of the latter via the Gumbel-Softmax reparameterization trick [28].
Training and evaluation details We train all policies on a single GPU for 48 hours with the full  $3.2B$  HiHack Dataset. As with all experiments in the paper, a total of 6 seeds are used to randomize data loading and neural policy parameter initialization. We employ mean and median NLE score on a batch of withheld NLE instances  $(n = 1024)$  as our central metrics for evaluating model performance and generalization at the conclusion of training, following the convention introduced in the NetHack Challenge competition [23]. Reported performance is aggregated over random seeds. Further details of architectures as well as training and evaluation procedures can be found in Appendices D, E, and F of the supplemental material.
Results We find that the introduction of hierarchy results in a significant improvement to the test-time performance of LSTM policies trained with behavioral cloning, yielding a  $40\%$  gain over the baseline in mean NLE score as shown in Figure 3(right), and  $50\%$  improvement in median score across seeds as shown in Table 2. Additionally, to verify that this improvement in performance is indeed due to hierarchy and not simply a result of the increased parameter count of the hierarchical LSTM policy decoder, we run ablation experiments with a modified, large-decoder version of the baseline (non-hierarchical) policy architecture. The results, shown in Figure 3(right), show that increasing the size of the LSTM decoder, without the introduction of a hierarchy, does not result in performance improvements over the baseline.
5 Architecture and Data Scaling
Despite the benefits of introducing hierarchical labels, the performance of the Hierarchical LSTM policy trained with HBC remains significantly behind that of symbolic policy used to generate the HiHack demonstrations in the first place, AutoAscend. This finding prompts us to explore scaling: perhaps increasing the quantity of demonstration data or the model capacity may close the observed performance gap.
Method To test this new hypothesis, we conduct a two-pronged investigation: (i) to explore model capacity, we develop a novel base policy architecture for NetHack that introduces a transformer module into the previous CDGPT5 based architecture; and (ii) for data scaling, we run a second set of "scaling-law" [30] experiments that use subsets of the HiHack Dataset to quantify the relationship between dataset size and the test-time performance of BC policies.
Figure 4: Left: Transformer based architecture (non-hierarchical version). The LSTM encoder (grey) is used to provide a long temporal context  $h_t$  to the transformer. Right: The transformer-LSTM models (light blue and green) outperform pure LSTM models with & without hierarchy (see Section 4 and [23] respectively). Ablating the LSTM encoder component from transformer-LSTM models to yield a flat transformer policy architecture (orange) causes a substantial decline in policy performance. Error bars indicate standard error.
Our novel base policy architecture is visualized in Figure 4 (left). This architecture features two copies of the encoders employed in CDGPT5. One set is kept frozen and employed strictly to yield a recurrent encoding of the complete trajectory up to the current observation step via a pre-trained frozen LSTM, while the second is kept "unlocked" and is employed to provide embeddings of observations directly to a causal transformer, which receives a fixed context length during training.
Training and evaluation details The training and evaluation procedures employed here echo those of section 4. In our data-scaling experiments, the subset of sampled HiHack games seen in offline training is randomized over model seeds. The causal transformer component of our transformer-LSTM models is trained with the same fixed, context-length  $(c = 64)$  in all experiments. Hyperparameter sweeps were employed to tune all transformer hyperparameters (see Appendix D).
Results The architecture experiments in Figure 4(right) show that both variants of our combined transformer-LSTM policy architecture yield gains eclipsing those granted solely by the introduction of hierarchy in the offline learning setting.