NetHack is Hard to Hack
Authors: Ulyana Piterbarg
Lerrel Pinto
Rob Fergus
Venue: NeurIPS 2023 poster (2023)
Keywords: imitation learning
NetHack

Abstract: Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in long-horizon tasks, especially in open-ended environments with multi-modal observations, such as the popular dungeon-crawler game, NetHack. Intriguingly, the NeurIPS 2021 NetHack Challenge revealed that symbolic agents outperformed neural approaches by over four times in median game score. In this paper, we delve into the reasons behind this performance gap and present an extensive study on neural policy learning for NetHack. To conduct this study, we analyze the winning symbolic agent, extending its codebase to track internal strategy selection in order to generate one of the largest available demonstration datasets. Utilizing this dataset, we examine (i) the advantages of an action hierarchy; (ii) enhancements in neural architecture; and (iii) the integration of reinforcement learning with imitation learning. Our investigations produce a state-of-the-art neural agent that surpasses previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, we also demonstrate that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players.

NetHack is Hard to Hack
Ulyana Piterbarg" NYU
Lerrel Pinto NYU
Rob Fergus NYU
Abstract
Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in long-horizon tasks, especially in open- environments with multi-modal observations, such as the popular mangeon-crawlae game, NetHack. Intriguingly, the NeurlPS 2021 NetHack Challenge revealed that symbolic agents outperformed neural approaches by over four times in median game score. this paper, we delve into the reasons behind this performance gap and present an extensive study on neural policy learning for NetHack. To conduct this study, we analyze the winning symbolic agent, extending its codebase to track internal strategy selection in order to generate one of the largest available demonstration datasets. Utilizing this dataset, we examine (i) the advantages of an action hierarchy; (ii) enhancements in neural architecture; and (iii) the integration reinforcement learning with imitation learning. Our investigations produce state-of-the ooffth--ann art neural agent that surpasses previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, also demonstrate that mere scaling is insufficient to bridge the performance gap with the best symbolic models the top human players.
Introduction
Reinforcement Learning (RL) combined with deep neural policies has achieved impressive results in control problems, such short-horizon simulated locomotion tasks [55, 7J. However, these methods struggle in long-horizon problem domains, such as NetHack [34], a highly challenging grid-world game. NetHack poses difficulties due its vast state and action space, multi-modal observation space (including vision and language), pnncedurally-genenae rewards. These challenges are evident in the recent NetHack Challenge [23], where agents based hand-crafted symbolic rules outperform purely neural approaches (see Figure 1), despite the latter having access to high-quality human demonstration data [24) and utilizing large-scale models.
randomness, diverse strategies, and deferred
We propose three reasons for the poor performance of large- neural policies compared to symbolic strategies. First, symbolic strategies implement hierarchical control schemes, which are generally absent in neural policies used for NetHack. Second, symbolic models use hand-crafted parsers for multi-modal observations, suggesting that larger networks could enhance representations extracted from complex observations. Third, symbolic strategies incorporate eгтоr correction mechanisms, which could be crucial for improving neural policies if integrated with RL based error correction.
In this work, we conduct comprehensive study of NetHack and examine various learning mecha- nisms to enhance the performance of neural models. We bypass traditional RI. obstacles, such as sparse rewards or exploration challenges, by focusing on imitation learning. However, we find that existing datasets lack crucial information, such hierarchical labels and symbolic planning traces. To address this, we augment the codebase of AutoAscend, the oop-erfforming symbolic agent in the 2021 NetHack Challenge, and extract hierarchical labels tracking the agent's 'ss internal strategy selection in order to construct a large-scale dataset containing 10 actions.
"Correspondence to up20218cims. nyu. edu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

CDGPTS
Autonscend
NHC 31 Place Plunn) Best Purely Data Driin Noura Agent
NHG 1 Place Divend, Sambblii
NetHack Learning Environment (NLE)
Beet Overal Agant
neural -ymmbbllc gli
Paiicy Archilacture
ogo centric piuel view
BC
Transformer LSTM (sun))
Hararchical LSTM (ouns)
RL
5-10P
100
(NLE)
Figure I: Left: The per-step observation for NetHack agents consists of an ego- centeentiic pixel image (blue) and two text fields containing ga-e messages (green) and statistics (pink). Right: Selected results from the NeurlPS 2021 NetHack Challenge (NHC) [23] showing game score on a log-scale. Error bars reflect standard error. Neural baseline models (grey) trained with BC perform poorly, but are somewhat improved when fine-tuned with RL. We find that the introduction of hierarchy and changes in model architecture yield significant improvements (light/dark blue), resulting in state-of-the-an performance for a neural model. However all neural approaches are significantly worse than AutoAscend, hand-crafted symbolic policy. Our paper explores this performance gap.
Using this dataset, we train a range deep neural policies and investigate: (a) the advantages of hierarchy; (b) model architecture and capacity; and (c) fine-tuning with reinforcement learning. Our main findings are as follows:
Hierarchical behavioral cloning (HBC) significantly outperforms BC and baseline methods, provided that the model has adequate capacity.
Large transformer models exhibit considerable improvements over baselines and other architectures, such as LSTMs. However, the power-law's 's shallow slope indicates that data scaling alone will not suffice to solve the game.
Online fine-tuning with RL further enhances performance, with hierarchy proving beneficial for exploration.
The combined effects of hierarchy, scale, and RL lead to state-of-the-an performance, narrowing the gap with AutoAscend but not eliminating it.
We validate the statistical significance of our findings with low-sample hypothesis testing (see Appendix H). Additionally, we open-source our code, models, and the HiHack repository?. which includes (i) our 10 dataset of hierarchical labels obtained from AutoAscend and (ii) the augmented AutoAscend and NLE code employed for hierarchical data generation, encouraging development.
Though we base our experiments in NetHack, we take measures preserve the generality of insights yielded by our investigations of neural policy learning. Specifically, we intentionally forgo the addition of any environmeet-sspccfic constraints in the architectural design or training setup of all models explored this paper. This is in contrast to NetHack agents RAPH and KakaoBrain, the first and second place winners of the neural track of the NHC Competition, respectively, which incorporate augmentations such as hand-engineered action spaces, role-specific training, and hard-coded symbolic sub-routines [23]. While this choice prevents from achieving absolute state-of of-the-art performance in NLE in this paper, we believe it to crucial in preserving the general applicability our insights to neural policy learning for general open- open-ended, long-horizer environments.
2 Related Work
Our work builds upon previous studies in the NetHack environment, imitation learning, hierarchical learning, and the use of transformers policies. In this section, briefly discuss the most relevant works.
Code available https: https://githhie htt.iuu...o.mmiitttieteteee comm/upittrbbrr/iaaackk

---

NetHack Following the introduction of the NetHack Learning Environment (NLE) [34], the NetHack Challenge (NHC) competition [23] enabled comparisons between a range of different agents. The best performing symbolic and fully data-driven neural agents were AutoAscend and Shaotic-Dwarven-GTT-5 (CDGPT5), respectively, and we base our investigations on them, as well as on the NetHack Learning Dataset [24].
Several notable works make use of the NetHack environment. Zhong et al. [61] show how a dynamics model can be learned from the Nethack text messages and leveraged to improve performance; on account of their utility, we also encode the in- in-amm message but instead use model-free policy to pick actions. Bruce et al. [9] show how monotonic progress function in NetHack can be learned from human play data and then combined with a reinforcement learning (RL) reward to solve rangrange tasks in the game. This represents complementary way of employing NetHack demonstration data without direct action imitation.
Imitation Learning Pomerleau [47] demonstrated the potential of driving an autonomous vehicle using offline data and neural network, which has since become ongoing research topic for scalable behavior learning [3, 6, 50]. Formally, behavior learning is function approximation problem, where the goal is to model an underlying expert policy mapping states or observations to actions, directly from data. Approaches to behavior learning can be categorized into two main classes: offline RL [21, 32, 33, 58, 36, 20], which focuses on learning from mixed-qualin datasets with reward labels; and imitation learning [42, 44, 45, 26), which emphasizes learning behavior from expert datasets without reward labels. Our work primarily belongs to the latter category as it employs behavior cloning model. Behavior cloning, a form of imitation learning. aims to model the expert's 's actions given the observation and is frequently used in real-world applications [60, 62, 60, 48, 18, 59].
Since behavior cloning algorithms typically address a fully supervised learning problem, they are often faster and simpler than offline RL algorithms while still yielding competitive results [20, 22]. A novel aspect of our work is the use of hierarchy in conjunction with behavioral cloning, i.e. supervision at multiple levels of abstraction, topic which has received relatively little attention. Recent efforts to combine large language models with embodied agents use the former to issue a high-level textual "action" to the low-level motor policy. Approaches such as Abramson et al. [1] have shown the effectiveness of hierarchical BC for complex tasks in a simulated playroom settings.
Hierarchical Policy Learning Hierarchical reinforcement learning (HRL) based techniques [5, 4] extend standard reinforcement learning methods in complex and long-horizon tasks via temporal abstraction across hicrarchies, as demonstrated by Levy et al. [37] and Nachum et al. [40][41]. Similarly, numerous studies have concentrated on showing that primitives [17, 54, 53] can be beneficial for control. These concepts have been combined in works such as Stochastic Neural Networks by Florensa al. [19], where skills are acquired during pretraining to tackle diverse complex tasks. Likewise, Andreas et al. [2] learn modular sub-policies for solving temporally extended tasks. However, most prior work focus learning both levels of the hierarchy, i.e. decomposition across primitives, skills, or sub-policies as well as the primitives, skills, or sub-policies themselves, which makes training complex. Correspondingly, the resultant approaches have had limited success on more challenging tasks and environments. Le et al. [35] explores the interaction between hierarchical learning and imitation and finds benefits, albeit in goal-conditioned settings. In contrast, our work explores the benefits of access to a fixed hierarchy chosen by the domain designer of AutoAscend, which simplifies our study of overall learning mechanisms for NetHack.
Transformers for RL The remarkable success of transformer models [57] in natural language processing [15, 8] and computer vision [16] has spurred significant interest in employing them for learning behavior and control. In this context, [11, 29] apply transformers RL and offline RL, respectively, while [12, 14, 38] utilize them for imitation learning. Both [14, 38] primarily use transformers summarize historical visual context, whereas [12] focuses on their long-term extrapolation capabilities. More recent work have explored the use of multi-modal transformers [27] to fit large amounts demonstration data [49,52 56]. To enable transformers to encode larger context lengths, recurrent transformer models have been proposed [13, 10). Our work draws inspiration from these use cases, employing a transformer to consolidate historical context and harness its generative abilitics in conjunction with a recurrent module.
3

---

Ascend global contralerr
Ireentory
disease
enguiieed
expiorn,
iduntiy
sokoban
Meilt out unospected
eSt cotoooeee
from
Rghh
emergency
ofter
Rt
Figure 2: A diagrammatic visualization of the internal structure of AutoAscend. The bot is composed of eleven goal-directed, high-level strategies. The "global controller" underlying AutoAscend employs complex predicate based control flow scheme determine which strategy query for an action on a per-timestep basis [23].
3 Data Generation: Creating the HiHack Dataset
3.1 Extending the NetHack Learning Environment
The NetHack Learning Environment (NLE) is a gym environment wrapping the NetHack game. Like the game itself, the action and state spaces of NLE are complex, consisting of 121 distinct actions and distinct observation components. The full observation space of NLE is far richer and more informed than the view afforded to human players of NetHack, who observe only the more ambiguous "text based" components of NLE observations, denoted as tty__harrs and tty_ttyyueettt tty_cursor. cursor. This text based view corresponds also to the default format in which both NetHack and NLE gameplay recorded, loaded, and streamed via the C based ttyrec library native the NetHack game.
The popular NetHack Learning Dataset (NLD) offers two large-scale corpuses of NetHack game- play data, NLD-AA, consisting of action-labeled demonstrations from AutoAscend, and NLD-NAD, consisting of unlabeled human player data [24]. NLD adopts the convention of recording only tty* components of NLE observations as a basis for learning, hence benefiting from the significant speedups in data operations offered via integration with the ttyrec library. We adhere this convention with the hierarchical HiHack Dataset introduced in this paper. Thus, in order to generate our dataset, we extend the ttyrec library store hierarchical strategy or, equivalently, goal labels alongside action labels. We further integrate this extension of ttyrec with NLE, modifying the gym environment to accept an additional hierarchical label at each step of interaction. This input hierarchical label does not affect the underlying state of the environment, and is instead employed strictly enable the recording of hierarchialll--inforee NetHack game-play to the ttyrec data format.
3.2 AutoAscend: A Hierarchical Symbolic Agent
An inspection of the fully open-source code base underlying the AutoAscend bot reveals the internal structure of the bot to be composed directed acyclic graph explicitly defined strategies. The bot's underlying global controller switches between strategies in imperative manner via sets of strategy-speciffe predicates, as visualised in Figure Among these strategies are hand-engineered routines for accomplishing broad range of goals crucial to effective survival in the game and successful descent through the NetHack dungeons. These include routines for fighting off arbitrary monsters, selecting food that is safe to eat from an agent's 's inventory, and efficiently exploring the dungeon while gathering and identifying valuable items, among many others. The various strategies supported in turn by shared sub sub-strategies for accomplishing simpler "sub-goals" (see Appendix for the full graph).
We exploit this explicit hierarchical structure in the generation of HiHack, extending the AutoAscend the bot, supported by our modifications the C based ttyrec writer library and NLE.
codebase to enable per-step logging of the strategy responsible for yielding each action executed by

---

3.3 The HiHack Dataset
Our goal in generating the HiHack Dataset (HiHack) is to create a cierarchically-infomme analogue of the large-scale AutoAscend demonstration corpus of NLD, NLD-AA. Thus, as previously described, HiHack is composed of demonstrations recorded in an extended version of the ttyrec format, consisting of sequences of tty* observations of the game state accompanied by AutoAscend action and strategy labels. HiHack contains a total of 3 billion recorded game transitions, reflecting more than hundred thousand AutoAscend games. Each game corresponds to a unique, procedurally- generated "seed" of the NetHack environment, with AutoAscend playing as one of thirteen possible character "starting roles" across a unique layout of dungeons.
We verify that the bigh-level game statistics of HiHack match those of NLD-AA in Table 1. Indeed, find high degree of correspondence across mean and median episode score, total number of transitions, and total number of game turns. We attribute the very slightly diminished mean scores, game transitions, and turns associated with HiHack to difference in the value of the NLE timeout parameter employed in the generation of the datasets. This parameter regulates the largest number of contiguous keypresses failing advance the game-state that is permitted before game is terminated. The value of the timeout parameter was set to 1000 in the generation of HiHack.
Table I: A comparison of dataset statisties between NLD- AA [23] and our generated HiHack Dataset, produced by running AutoAscend in NLE v0.9.0 with extended tty* observations.
NLD-AA
HiHack
Total Episodes
109,545
109,907
Total Transitions
3,481,600 1,481,6055000 ,605,009
3,244,729,367
Mean Episode Score
10,105
8,166
Median Episode Score
5.422
5.147
Median Episode Game Transitions
28,
27,496
Median Episode Game Turns
20,414
19,991
Hierarchical Labels
X
Hierarchical Behavioral Cloning
Our first set of experiments leverage the hierarchical strategy labels recorded in HiHack for offline learning with neural policies, via hierarchical behavior cloning (HBC).
Method Mimicking the imperative hierarchical structure of AutoAscend, we introduce a bilevel hierarchical decoding module over popular NetHack neural policy architecture, namely the Chaotic slaaticDDarvenGP155 (CDGPT5) model. This model achieved 3rd place in the neural competi- tion track of the NeurIPS 2021 NetHack Challenge when trained from scratch with RL, making it the top-performing fully data-driven neural model for NetHack [23].
The CDGPT5 model consists of three separate encoders: 2-D convolutional encoder for pixel- rendered visual observations of the dungeon o₁, multilayer perceptron (MLP) encoder for the environment message m, and a 1-D convolutional encoder for the bottom-line agent statistics b, These three observation portions are extracted from the tty* NLE observations of HiHack. The core module of the network is an LSTM, which is employed produce recurrent encoding of an agent's full in-game trajectory across what may be hundreds of thousands of keypresses, both in training and at test-time. The core module may also receive one-hot encoding of the action executed at the previous time-step d as input.
Our hierarchically-extere extended version of this LSTM based policy is shown in Figure 3(lftt). We replace the linear decoder used to decode the LSTM hidden state into corresponding action label in the CDGPT5 model with a hierarchical decoder consisting of (i) single "high level" MLP responsible for predicting the strategy label St given the environment observation tuple (m, or, by and (ii) a set of "low level MLPs, one for each of the discrete strategies in the AutoAscend hierarchy (see Figure 2), with SoftMax output over discrete actions. The strategy prediction ge selects which of these low-level MLPs to use.
5

---

BC Results
LSTT Polciis
gumbel sottmex
1000
800
mean score
600
LSTM
convolultonal ihear encodara
200
L.5TM
LSTM
Hararchical LSTM
XXL dec
Figure 3: Left: Hierarchical LSTM policy, where It is the high-level strategy prediction (purple) that used to select over the l low-level policies (yellow). Figure shows the three different components of the input observation: message m,, ego-centric pixel view of the game os, and "bottom line" statistics b, Right: Mean score for baseline LSTM model [23] (grey), our hierarchical model (blue) the conclusion of training. The addition of hierarchical labels in decoding provides significant performance gain, not matched by an (extra) large-decoder version of the baseline (dark grey). Error bars indicate standard error.
We employ a simple cross-entropy loss to train both the baseline non-hierarchical LSTM CDGPT5 policy, as well as our Hierarchical LSTM policy, aggregating gradients across the bilevel decoders of the latter via the Gumbel-Softmax reparameteriiztiion trick [28].
Training and evaluation details We train all policies on single GPU for 48 hours with the full 3.2B HiHack Dataset. As with all experiments the paper, total of 6 seeds are used to randomize dataloading and neural policy parameter initialization. We employ mean and median NLE score on batch of withheld NLE instances (n 1024) as our central metries for evaluating model performance and generalization at the conclusion of training. following the convention introduced in the NetHack Challenge competition [23]. Reported performance aggregated over random seeds. Further details of architectures as well as training and evaluation procedures can be found in Appendices D, E, and F of the supplemental material.
Results We find that the introduction of hierarchy results in significant improvement to the test-time performance of LSTM policies trained with behavioral cloning, yielding a 40% gain over the baseline mean NLE score as shown in Figure 3rright), and 50% improvement in median score across seeds as shown in Table 2. Additionally, to verify that this improvement in performance is indeed due hierarchy and not simply result of the increased parameter count of the hierarchical LSTM policy decoder, we run ablation experiments with modified, large-decoder version of the baseline )in-hierarrh policy architecture. The results, shown in Figure 3(right), show that increasing the size of the LSTM decoder, without the introduction of a hierarchy, does not result in performance improvements over the baseline.
5
Architecture and Data Scaling
Despite the benefits of introducing hierarchical labels, the performance of the Hierarchical L.STM policy trained with HBC remains significantly behind that of symbolic policy used to generate the HiHack demonstrations in the first place, AutoAscend. This finding prompts to explore scaling: perhaps increasing the quantity of demonstration data or the model capacity may close the observed performance gap.
Method To test this new hypothesis, we conduct two-pronged investigation: (i) to explore model capacity, we develop a novel base policy architecture for NetHack that introduces transfomer module into the previous CDGPT5 based architecture; and (ii) for data scaling, we run second set of "scaling-law" [30] experiments that use subsets the HiHack Dataset to quantify the relationship between dataset size and the test-time performance of BC policies.

---

Results
LSTM vs. Transformar Policis
1250
decoder
causal transformer
moan score (NLE)
1000
embedding
750
positional encoding
(rooer))
500
LSTM
recumeett module
250
Transformer
LSTM
Tranfformer
LSTM
Figure 4: Left: Transformer based architecture mnon-hierarchical version). The LSTM encoder (grey) is used to provide long temporal context he to the transformer. Right: The transformer-LSTM models (light blue and green) outperform pure LSTM models with without hierarchy (see Section 4 and [23] respectively). Ablating the LSTM encoder component from transformer-LSTH models to yield a flat transformer policy architecture (orange) causes a substantial decline in policy performance. Error bars indicate standard error.
Our novel base policy architecture is visualized Figure 4 (left). This architecture features two copies of the encoders employed in CDGPT5. One set kept frozen and employed strictly to yield a recurrent encoding the complete trajectory up to the current observation step via pre-trained frozen LSTM, while the second is kept "unlocked" and is employed to provide embeddings of observations directly to a causal transformer, which receives a fixed context length during training.
Training and evaluation details The training and evaluation procedures employed here echo those of section 4. In our data-scaling experiments, the subset of sampled HiHack games seen in offline training randomized over model seeds. The causal transformer component our transformer-LSTM models is trained with the same fixed, context-length (c 64) in all experiments. Hyperparameter sweeps were employed to tune all transformer hyperparameters (see Appendix D).
Results The architecture experiments in Figure 4(right) show that both variants of combined transformer-LSTH policy architecture yield gains eclipsing those granted solely by the introduction of hierarchy in the offline learning setting.
Model Size vs. Transforme-LLSMM
Policy Performance
Policy
10a
10
1
10P
98.0M parama
50.00 parera
transitions
moan score (NLEE)
moan score (NLE)
850 Jos 179
Pollcy Architecture
1000
1000
LSTM
750
Jargei
500
500
250
184 loj
Transformar LSST (arge)
Transformer LSTM
10a
10a
10a
10
10
games
Figure Left: Model capacity versus mean score for our transformer-LSTM model. The larger model performs Error bars indicate standard error. Right: Dataset scaling experiments showing diminishing returns in mean NLEE score & achieved during training with BC as the number training games reaches 105. We employ non-linear reast-squares relating log-game count and mean policy score outside of the low-data regime (g > 102). Curves of best fit are displayed. Errors bars indicate mean score ranges across seeds. Collectively these two plots show that scaling of data and model size are not sufficient to close the performance gap to symbolic models.
estimate the power law coefficients

---

We find the LSTM recurrent module of our transformer-Lof LSTM models to be crucial component of the architecture. When this module is ablated to yield a flat transformer, we observe a severe decline in policy performance (Figure 4(right) and Appendix G, Figure 14).
Probing further, in Figure 5(left), compare the performance of two variants of our transformer- LSTM model, one with 3 layers (50.8M parameters) and another "scaled-up" model with 6 layers (98.9M parameters). The larger model can be seen to perform worse in evaluation than the smaller one, suggesting over-fitting and an inverse relationship between parameter count and generalization in NetHack for this model class (see also Appendix G, Figure 12). This finding indicates that scaling of model capacity alone may not be sufficient to close the neural-symbolic gap.
In Figure 5(right), we now explore the effect of training set size mean NLE test score. We perform BC training for the LSTT baseline [23] and our largest layer transformer-L.ST model (98.9M params) for 10 up to 100, 000 games, subsampled from the HiHack Dataset. For both architectures, we observe a sub log-linear dependence on training set size, asymptoting at a mean score of approximately 1000. Thus, brute force scaling of the dataset alone cannot viably close the gap to symbolic methods (score of 8500).
Though our architecture and data scaling experiments compute-time constrained, we find the test-time performance of all tested models to saturate on the given computational budget (Appendix G, Figures 11 and 12).
Table 2: Evaluating the impact of hierarchical labels and architectural improvement on the performance of policies trained both with behavioral cloning, well with combined behavioral cloning and asynchronchronous proximal policy optimization. All policies were trained for 48 hours on single GPU. Metrics annotated with (f) were computed only for the top-scoring neural policy seed (out of 6) across each model class.
Score
Dlyl (f)
Turns (f)
Hierarchy
Mean
Median
Mean
Mean
Median
BC
LSTM [23]
658 41
403
1.11 0.01
5351 + 76
4111
BC
I.STM
931 42
614
1.09 0.01
6983 + 84
5981
BC
Transformer I.SM
X
1318 = 38
914
1.36 0.01
6088 + 75
5121
BC
Transfomer-I.TM.
1151 = 43
731
1.26 + 0.01
7568 # 99
6242
APPO + BC
LSTM [23]
X
1204 138
779
1.07 0.01
8712 + 112
7376
APPO BC
LSTM
V
1551 = 73
972
1.01
11435 + 134
9849
APPO BC
Transfomer-LSTM
X
1326 28
887
1.25 1.05
7924 + 99
6788
APPO BC
Transfomer-LSTM
1346
894
1.32 0.01
7874 = 101
6769
Symbolic
AutoAscend
8556 187
4918
3.10 0.04
19586 + 171
19710
6 Combining Imitation with Reinforcement Learning
Given that hierarchy and scaling are insufficient to bridge the performance gap with AutoAscend, we now explore the impact of incorporating an online learning component to neural policy training, via RL.
Method In this set of experiments, we build on results from Hambro et al. [24], which suggested that in the presence of action-labeled data, behavioral cloning coupled with reinforcement learning superior to RL training from scratch in NetHack. As Hambro et al. [24], we employ the asynchronous moolib distributed- RL. library to train our models with combination of BC and asynchronous proximal policy optimization (APPO) [39, 46, 51]. At each time-step of training. the overall loss used to perform model parameter updates is a weighted combination of BC and RL losses, i.e. the cross-entropy loss of a batch of demonstrations from HiHack plus an RL loss over a batch of rollouts of the current policy in NLE. For hierarchical models, we use RL only to update the low-level strategies; that is, the strategy selection policy is trained with HBC alone and is not updated with RL.
8

---

NLE Score Distribution (f) vs. Starting Role Across Noural Policy Classes
Policias)
1.TMM
Hanarchical LSTM
1G000
Transfommr LBTM
LG000
Hararciic Transfomer- LSTM
200
5000
Arc
(APPO BC Policies)
12000
L.5MM
1200
Haranhool LSTM
1500
Tranoforarr + LBTM
15000
Haranchic Transfommr- LSTM
100N PALET
An
16-
11
101
10
Figure 6: Aggregate NLE score breakdown versus player role. Our model refinements (hierarchy, transformer, RL fine-tuning) show gains over the LSTM based CDGPT5 baseline [23] across all roles. As in Table 2, we employ (t) to confer that these score distributions were computed only for the hop-perforning neural policy seed (out of 6) across each model class.
Training and evaluation details Our high-level training procedure here mirrors that of our hier- archical behavioral cloning experiments: we evaluate model performance under the constraint of computation time, training all policies for exactly 48 hours on a single GPU, using 6 random seeds to randomize data loading and environment seeding only. We test the effect of warm-starting from a checkpoint pre-trained with BC or HBC alone (via the procedure delineated in section 4) for all model classes. We report APPO + BC results with warm-starting for all model classes where find it to be beneficial in improving stability and/or evaluation policy performance.
Results Table 2 summarizes the performance of all our models. A number of observations can be made: (a) In the offline setting, our best performing model (non-hierarchical transformer-LSTM outperforms the vanilla CDGPT5 model baseline by 100% in mean NLE score and 127% in median scor; (b) RL fine-tuning offers a clear and significant performance boost to all models, with gains in the test-time mean NLE score associated with all model classes; (c) The overall best performing approach is APPO + BC using the hierarchical LSTM model. The mean score of 1551 represents a new nate-of-thheart for neural policies in NLE, beating the performance of the vanilla CDGPT5 model when trained with APPO + BC by 29% in mean NIEE score and 25% in median NLE score; (d) The transformer-LSSTM models, which are slower and more unstable to train with APPO + BC than LSTM models (see training curves in Appendix G), see smaller margin of improvement during RL fine-tuning; (e) Other metries commonly employed to evaluate meaningful NetHack gameplay in previous work, such as dungeon level reached and lifetime of the agent in game turns [34, 23], show a broadly similar pattern to the chief median and mean score metrics employed in the NetHack Challenge [23]; and (f) Lastly, for BC, hierarchy seems to hurt performance for the larger Transformer- RL models, though this gap is closed once APPO fine-tuning is applied. See Appendix H for lample hypothesis tests validating the statistical significance of these findings.
In NetHack, the player can choose from 13 distinct roles (barbarian, monk, wizard, etc.), each of which require distinctive play-styles. In NLE, starting roles are randomized, by default. Figure 6 shows a score distribution breakdown across role for different neural policy classes trained with BC and APPO + BC. In general, we observe that fine-tuning with RL improves the error-correction capability of models of all classes (as indicated by positive shifts in NLE score distributions) over their purely offline counterparts.
7 Conclusion and Discussion
In this work, we have developed a new technique for training NetHack agents that improves upon prior state-of-the-an neural models by 127% in offline settings and 25% in online settings, as evaluated by median NLE score. We achieve this by first creating new dataset, the HiHack Dataset (HiHack), by accessing the best symbolic agent for NetHack, This dataset, combined with new architectures, allows us to build the strongest purely data-driven agent for NetHack as of the writing of this paper.

---

More importantly, we analyze several directions to improve performance, including the importance of hierarchy, the role of large transformer models, and the boosts that RL could provide. Our findings are multifaceted and provide valuable insights for future progress in training neural agents in open-ended environments and potentially bridging the gap to symbolic methods, as well as to the very best human players.
Hierarchy can improve underfitting models. Prior LSTM based models severely underfit on the HiHack Dataset (see training curves in Appendix G). The addition of hierarchy improves such models, whether trained offline or online (Figure 3(right) and Table 2).
Hierarchy can hurt overfitting models. Transformer based models are able to overfit on the Hillack Dataset (see training curves in Appendix G). Hierarchy hurts this class of models when trained offline at test-time (Figure 4(right) and Table 2).
Reinforcement learning provides larger improvements on light, underfitting models. We obtain little and no improvement with RL. fine-tuning on our hierarchical transformer-LSTIH models, respectively. However, the underfit LSTM models enjoy significant gains with RL, ultimately outperforming transformer based models under a compute-time constraint (Table 2).
and non-hierarchical transformer-LSTT
Scale alone is not enough. Our studies on increasing both model and dataset size show sub-log-lineas scaling laws (Figure 5). The shallow slope of the data scaling laws we observe suggests that matching the performance of AutoAscend with imitation learning (via games played by bot) will require more than just scaling up demonstration count.
Possible avenues for future exploration include: (a) alternate methods for increasing the transformer context length to give the agent more effective long-term memory: (b) explicitly addressing the multi-modal nature of the demonstration data (i.e. different trajectories can lead to the same reward), which is a potential confounder for BC methods. Some forms of distributional BC (e.g. GAIL. [26]. BeT [52]) might help alleviate this issue.
Finally, we hope that the hierarchical dataset that we created in this paper may be of value to the community in exploring the importance of hierarchy and goal-conditionig in long-horizon, open-ended environments.
Acknowledgements
We would like to thank Alexander N. Wang, David Brandfonbrener, Nikhil Bhattasali, and Ben Evans for helpful discussions and feedback. Ulyana Piterbarg is supported by the National Science Foundation Graduate Research Fellowship Program (NSF GRFP). This work also supported by grants from Honda and Meta as well as ONR awards N00014-21-1-2758 and N00014-22-1-2773
References
[1] Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, Clark, A. Dudzik, Georgiev, A. Guy. T. Harley, F. Hill, A. Hung. Z. Kenton, J. Landon, T. P. Lillicrap, K. W. Mathewson, A. Muldal, A. Santoro, N. Savinov, V. Varma, G. Wayne, N. Wong, C. Yan, and R. Zhu. Imitating interactive intelligence. arXiv, abs/2012.05672 2020. 3
[2] J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches, 2016. 3
[3] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009. 3
[4] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Thirty-First AAAI Conference Artificial Intelligence, 2017. 3
[5] A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcementt learning, Discrete event dynamic systems, (3(1-)):41-77, 2003.
[6] A. Billard, S. Calinon, R. Dillmann, and Schaal. Survey: Robot programming by demonstra- tion. Handbook of robotics,
2008. 3
10

---

[7] G. Brockman, V. Cheung, L. Pettersson, J. Schneider. J. Schulman, S. Sidor, I. Sutskever, and R. S. Zemel. Openai gym. arXiv preprint wrXiv:1606.015000 2016. 1
[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprini arXiv:2005. 14165, 2020. 3
[9] D. Bruce, E. Hambro, M. Azar, and M. G. Bellemare. Learning about progress from experts. In International Conference on Learning Representations, 2023.
[10] A. Bulatov, Y. Kuratov, and M. Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35: 11111117991999 2022. 3
2022.
[11] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. CoRR, abs/2106.01344, 2021. URL pectps://arxiie wwwp:iiiiorro/ooooooo largg/abss. aas///bs////0 01345. 3
[12] H. M. Clever, A. Handa, H. Mazhar, K. Parker, O. Shapira, Q. Wan, Y. Narang, I. Akinola, M. Cakmak, and D. Fox. Assistive tele-op: Leveraging transformers to collect robotic task demonstrations. arXiv preprint arXiv: 22t.:::222..011299 2021.
[13] Z. Dai, Z. Yang. Y. Yang, J. Carbonell, Q. V. Le. and R. Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint Stt::1900102866 2019.
[14] S. Dasari and A. Gupta, Transformers for one-shot visual imitation. arXiv preprint arXiv:201..09970 2020. 3
[15] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04888 pages 4171 -116, 2018. doi: 10. 111865/31/N119 1423. 3
[16] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1992,, 2020. 3
[17] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. CoRR, abs/1802.06070, 2018. URL http: //::://aaa /arxiv.vvva org/orgg//a. / 1802. 02070.000. 3
[18] Florence. L. Manuelli, and Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019. 3
[19] C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning, 2017. 3
[20] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint urXiv:2004.0722 2020. 3
[21] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR, PMLR, 2018.
[22] C. Gulcehre, Z. Wang, A. Novikov, T. Le Paine, S. Gomez Colmenarejo. K. Zolna, R. Agarwal, J. Merel, D. Mankowitz, C. Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning, arXiv e-prints, pages arXiv-2006, 2020. 3
[23] E. Hambro, S. Mohanty, D. Babaev, M. Byeon, D. Chakraborty, E. Grefenstette, Jiang, J. Dae- jin, A. Kanervisto, J. Kim, S. Kim, R. Kirk, V. Kurin, H. Küttler, T. Kwon, D. Lee, V. Mella, Rocktäschel,
N. Nardelli, I. Nazarov, N. Ovsov, Holder, R. Raileanu, K. Ramanauskas, D. Rothermel, M. Samvelyan, D. Sorokin, M. Sypetkowski, and M. Sypetkowski. Insights from the neurips 2021 nethack challenge. Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, pages 41-52, 2022. 1, 2, 4, 5, 6, 7, 8, 9, 19, 21, 25

---

[24] E. Hambro, R. Raileanu, D. Rothermel, V. Mella, T. Rocktäschel, H. Küttler, and N. Mur- ray. Dungeons and data: surge-scale nethack dataset. Advances Neural Information Processing Systems, 2022. 1,3, 4, 8, 17, 18
[25] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprini
arXiv:1606.080 2016. 19
[26] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, volume 29, pages 4565-4573, 2016. 3, 10
[27] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages PMLR, 2021.
[28] E. Jang, S. Gu, and B. Poole. Categorical reparameterizationn with gumbel-softmax. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. 6,
[29] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 34, 2021.
[30] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford,
J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv, abs/2001.0883 2020.
6
[31] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014. 18
[32] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[33] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[34] H. Kuttler, N. Nardelli, T. Lavril, M. Selvatici, V. Sivakumar, M. G. Bellemare, R. Munos, A. Graves, and M. G. Bellemare. The nethack learning environment. In Advances in Neural Information Processing Systems, 2020. 1, 3, 9, 15, 20
[35] H. Le, N. Jiang, A. Agarwal, M. Dudik, Y. Yue, and H. Daumé III. Hierarchical imitation and reinforcement learning. In International conference on machine learning, pages 2917-2926. PMLR, 2018. 3
[36] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020. 3
[37] A. Levy, G. Konidaris, R. Platt, and K. Saenko. Learning multi-level hierarchies with hindsight, 2017. 3
[38] Z. Mandi, F. Liu, Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. arXiv preprint 10. 2021. 3
[39] V. Mella, E. Hambro, D. Rothermel, and H. Küttler. moolib: A Platform for Distributed RL. 2022. URL https:////// //gggtttubbu www./aaceeboorreeerrr//oooooooo
8, 18
[40] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficieet hierarchical reinforcement learning. CoRR, htbs/180500829. 2018. URL UuRL.attpp//axxx..oor///aa
[41] O. Nachum, S. Gu, H. Lee, and S. Levine. Near-optimal representation learning for hierarchical reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 3
[42] Osa, J. Pajarinen, G. Neumann, A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective imitation learning, arXiv preprint arXiv:1811.06711, 2018. 3
12

---

[43] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, pigf--erformance deep learning library. Advances in neural information processing systems, 32:8026-800 8037, 2019. 18
[44] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
[45] X. B. Peng. Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG), 40(4):1-20, 2021.
[46] A. Petrenko, Z. Huang, T. Kumar, G. Sukhatme, and V. Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learning, pages 7652-7662. PMLR, 2020. 8
[47] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988. 3
[48] R. Rahmatizadeh. P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE international conference on robotics and automation (ICRA), pages 3758-3765. IEEE, 2018. 3
[49] S. Reed, K. Zolna, E. Parisotto, G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06115, 2022.
[50] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233 (6,2233-242, 1999. 3
[51] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arxivv:1777: 1700::047777777 2017. 8
[52] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L.. Pinto. Behavior transformers: Cloning k modes with one stone, Advances in neural information processing systems, 35:22955-22968, 2022. 3, 10
[53] T. Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference Learning Representations, 2020. URL https: / //openreview,
3
[54] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. CORR, 00///907701657 2019. URL http://arxxv..rggtbbee 01657. 3
[55] Y. Tassa, D. Silver, J. Schritwieser, A. Guez, L. Sifre, G. d. Driessche, S. Dieleman, K. Greff, T. Erez, and S. Petersen. Deepmind control suite. arXiv preprint 100Xiv:/001.01/1 2018.
[56] A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg. adaptation in an open-ended task space.
M. Chang, N. Clay, A. Collister, et al. arXiv preprint ar 20011:230,,07608 2023. 3
[57] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and
I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1998-6008, 2017. 3, 18
[58] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1361, 2019. 3
[59] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, 2020.
13

---

[60] T. Zhang. Z. McCarthy, O. Jow. D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, pages IEEE, 2018. 3
[61] Y. Zhong, E. Hambro, E. Grefenstette, T. Park, M. Azar, and M. G. Bellemare. Improving policy learning via language dynamics distillation. In Advances in Neural Information Processing Systems, 2022. 3
[62] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills, arXiv preprint arXiv; 888.00004444 2018. 3
14

---

A NetHack Learning Environment
Both NetHack and the NetHack Learning Environment (NLE) feature complex and rich ob- servation space. The full observation space of NLE consists of many distinct (but redundant) components: glyphs, chars, colors, specials, blstats, message, inv-glyphs, inv_ inv inv_oclasses, screen_descriptionn
chars, tty_colors, and tty_cursor [34].
The HiHack Dataset (HiHack), as well as all RI experiments in NLE conducted in this paper, consist of and rely solely upon the tty* view of the game.
B Details on AutoAscend
Ishow pard
urrnnge itera
reighturs trage
sotes sokoban
untrup tripo
identity
eGp lone
reighbor doors
chock conbennrr
orgulted fight
idenvily leens on
open vilit search
Iurra
fetirun invertory
ep for canallter
gither iomn
Autooaccand
wall out
go
go lssat
unchecked cpntainers
from ground
ow level path
explore stalts
wand engrase Identity
emargency
to Item for
Right
wilt-eeror
check Hamm
Figure 7: Full control flow structure of AutoAscend, separated across explicit strategy and sub- strategy routines. There are 13 possible "strategy" or hierarchical labels in HiHack, 11 representing the explicit strategies employed by the bot (in magenta) and two additional labels handle extra- hierarchical behavior (in light-grey).
We include a comprehensive visualization of the full internal structure of AutoAscend in Figure 7. As indicated in the summary visualization high-level AutoAscend strategies provided in Figure 2 of the main paper, the bot features 11 explicit, hard-coded strategy routines. These interface with other low-level sub-strategy routines, some which are re-used by multiple strategies or even multiple sub-strategies. One example of such subroutine is the "arrange items" sub-strategy, which called both by "follow guard" and by "check altar," which itself subroutine of the "explore" sub- strategy. When factorized across strategies and sub- uub-strategies, the full structure of AutoAscend a directed acyclic graph (DAG) with maximal depth of from the "root," i.e. the AutoAscend globul controller "node" indicated in dark gray above, which re-directs global behavioral flow across scheme.
strategies via a predicate-matchin
The HiHack Dataset includes hierarchical strategy label for each timestep of AutoAscend interac- tion. As result, alongside the 1l explicit strategies of the bot, there in the dataset, which account for extra-hierarchical behavior in ttyrec game records yielded by the augmented ttyrec writer employed for HiHack generation and loading. These are visualized in light gray in Figure 7. The first of these corresponds to the hard-coded initialization routine employed by AutoAscend, effectively serving as twelfth (albeit implicit) strategy, while the second covers ttyrec timestep records with missing strategy values. Missing strategy values may reflect ttyrec writer errors, or advancement the underlying NetHack state by NLE rather than by agent, which occurs e.g., during NLE's timeout based termination of games [34]. Empirically, "write-error" strategy labels occur with very low frequency in HiHack, representing less than 0.05% of all transitions.
additional labels present
15

---

NLD-AA 105) va. HiHackk (n 109, 107)
[Log Score Distributiore
NLD
10000
10000
7300
7503
beg scora]
(Log-Tumss Distribution)
NLD-AA
HiHock
276
40%6
63336
22000
20000
00.n5
350.0
30030
logo tumss
Junyl tume
Figure 8: Distributional comparisons of basic AutoAscend game statistics in NLD vs HiHack. Median and mean values (as reported in Table are indicated respectively by vertical dashed-blue and solid-pink lines in each figure. Top: Log-score vs game counts in NLD- AA and HiHack. Bottom: Log-turns vs game counts in NLD-AA and HiHack.
C Details on HiHack
Generation All games in the HiHack dataset were recorded by running an augmented version of AutoAscend in NLE v 0.9.0. O. This augmented version of the AutoAscend source features the introduction of only dozen extra lines of code that enable step-wise logging of the strategy trace behind each action executed by the bot. This strategy trace is recorded directly to game ttyrecs each timestep via the addition of an extra channel the based ttyrec- cerierr in the NetHack source code. Each game was generated via a unique NL.E environment seed.
Game Statistics The comparison of the full log-score and log-turns distributions across NLD AA and HiHack made in Figure 8 further supports the claim of high correspondence between the datasets made in Section 3.3. Figure 9 shows the distribution of strategies across sample consisting of 10 unique game transitions from HiHack. We observe coverage of all but the least frequent explicit strategy executed by AutoAscend: the "solve sokoban" routine, employed a means to gain yet more experience exclusively in highly advanced game states.
Distribution of Strategies in Hillack (7 10")
esplom, gather, identty
corpaes
unacpeedied
from invertory
wreegency
intiaization
ofter corpaes
write-mur
engulledd fight
aure disease
follww guord
sohv solvotur
Figure 9: A visualization of the distribution strategies across sample of 4300 HiHack games, containing total of as 10 transitions.
16

---

Table 3: Training hyperparameter configurations across all BC and APPO + BC experiments. Hyperparameters are listed in alphabetical order. We employ ( to indicate hyperparameters only relevant for the corresponding hierarchical policy variants. The presence of the symbol in lieu of a parameter value reflects the parameter's irrelevance for offline, BC experiments. All bolded hyperparameters were tuned. After tuning was complete, precisely the same sets of hyperparam- eters were employed to train models across individual policy classes belonging to the LSTM and transformer-LSTW based model families explored in this paper, across all BC and APPO + BC experiments. Note that the abbreviation 'CE' denotes the cross-entropy loss function.
BC
APPO BC
Hyperparameter
LSTM
transformer-LSTM
LSTM
transformer-LSTM
actor batch size
512
256
adam betal
0.9
0.9
0.9
0.9
adam beta2
0,999
0.999
0,999
0,999
adam eps
1.00E-07
1.00E-07
1.00E-07
1.00E-07
adam learning rate
0.0001
0.0002 1
0,0001 1
0.0001
appo clip baseline
appo clip policy
0.1
0.1
baseline cost
1
1
1
crop dim discount factor entropy cost cnv max episode steps env name fn penalty step grad norm clipping inference unroll length
18
18
18
18
0.999
0.999
0.001
0.001
100000
100000
challenge
challenge
constant
constant
4
loss function normalize advantages normalize reward actor batches num actor epus penalty step penalty time
CE
CE
CE
/
x
x
2
10
10
0
0
0
pixel size
6
reward clip
10
10
reward scale RL. loss coeff strategy loss coeff (t) supervised loss coeff
0.001
1
0.001
ttyrec batch size
512
512
256
ttyrec cores
12
12
12
12
ttyrec envpool size
4
6
4
ttyrec unroll length
32
64
32
64
prev action
/
virtual batch size
512
1024
512
D Training Details
Hyperparameters All relevant training hyperparameter values, across model families as well as BC vs APPO + BC experiment variants, are displayed in Table 3.
To kick-start all experiments, we employed the training hyperparameter values reported in Hambro al. [24], used train the CDGPT5, LSTM based baseline across experiments. For several hyperpa- rameters, however, additional tuning was conducted. These hyperparameters are indicated bold in Table 3. Tuning across these hyperparameters
performed once for the "default" representative policy class from each of the LSTT and transformer- L.TM model families for all but the Model Parameter Scaling experiments". After tuning was complete, hyperparameter configurations were
'Prior to the start of these experiments, additional tuning of the "adam learning rate" and "ttyrec batch size" hyperparameters conducted for the transformer-LSTM (large) policy class. However, across the
17

---

fixed across all proceeding offline and combined offline + online experiments. Specifications of hyperparameter values swept over during tuning are provided Table
All models were trained with the Adam optimizer [31] and a fixed learning rate. We experimented with the introduction of learning rate schedule for transformer-LSTTM models, but we found no additional improvements in policy prediction error or evaluation performance at the conclusion of training be yielded by such a schedule.
Random Seeds For each of the Hierarchical Behavioral Cloning, Model Parameter Scaling, Data Scaling, and combined Imitation and Reinforcement Learning experiments described in Sections 4, 5, and 6 of this paper, a total of 6 random seeds were across all relevant policy classes. Randomized quantities included: policy parameter values at initialization, data loading and batching order, HiHack Dataset subsampling (in data scaling experiments only), and initial environment seeding (in APPO + BC experiments only).
Training Infrastructure and Compute The RPC based moolib library for distributed, asyn- chronous machine learning was employed across all experiments [39]. All data loading and batching was parallelized. Our model training code builds heavily upon the code open- by Hambro et al. [24].
Experiments were run on compute nodes a private tigh-peeformance computing (HPC) cluster equipped either with a NVIDIA RTX-8000 or NVIDIA A 100 GPU, as well as 16 CPU cores. All policies were trained for total of 48 hours. We detected no substantial differences in training frames- per-second (FPS) rates for both offline and online experiments across compute nodes in "speed-run" tests, provided nodes were under no external load. When running experiments, we did detect some variance in total optimization steps completed under the 48-hour constrained computational budget across seeds belonging to single policy classes, which we attribute to variance in external HPC cluster load during these runs.
Table 4: Training hyperparameter tuning sweeps. We specify the hyperparameter values tested dur- ing tuning sweeps conducted for the "default" representatives of each model class. Full specifications of final hyperparameter values employed in experiments are included in Table 3.
Sweep Range
adam learning rate discount factor
{0.0001, 0.0002, 0.0005, 0.01)
{0.9, 0.99, 0.999, 0.9999) {0.1, {0.001, 0.01, 1} (0.001, 0.01, 10} (0.001, 0,01,
grad norm clipping
RL loss coeff strategy loss coeff supervised loss coeff ttyrec envpool size ttyrec unroll length virtual batch size
{3, 4,6) {16, 32, 64, 128]
{128, 256, 512, 1024}
E Model Architectures
A description of all model components and policy architectures is given in Tables 5 and 6, separated across LSTM and transformer-LSTM model families. The PyTorch library was used for to specify all models, loss functions, and optimizers [43].
Additional Transformer Specifications All transformer modules tested in this paper consist purely of Seanssforrr-EEnnodee layers. Each layer configured with 16 attention heads per attention mechanism, and layer normalization is applied prior to all attention and feed-forward operations. A dropout of 0.1 is used during training [57]. Unlike the rest of the modules we employ, which use
set of values tested, the same values were found to be optimal for this model configuration as for the default Merssformrr-LTTM policy; hence, only single set of hyperparameter values for the model family is reported here.
18

---

Table 5: LSTM model family architectural details. The final three columns indicate the presence (or absence) of each component across the relevant policy classes, whether trained with BC or APPO + BC.
Policy Class
Hidklen Dim
Class
Type
Module(s)
Layers
Activ.
Copies
LSTM
XXL dec
LSTM
Enc
Message
MLP
ELU
Enc
Blstats
Conv 1Da MLP
128
ELU
Enc
Pixel Obs
Conv-2D, MLP
512
ELU
Enc
Action Hist
one-hot
128
/
/
Core
LSTM
512
/
/
/
Dec
Default
MLP
512
/
x
x
Dec
XXL
MLP
1024
ELU
X
X
Dec
Hier Strat
MLP
X
X
Dec
Hier Action
MLP
256
ELU
13
X
X
Table 6: transformer-LSTM model family architectural details. As in Table 5, the final three columns indicate the presence (or absence) of each component across the relevant policy classes, whether trained with BC or APPO BC.
Policy Class
Trnsfrnr
Hier
Hidden Dim
Trasfrmr LSTM
LSTM (lrgg)
Trasfmr + LSTM
Class
Type
Modulc(s)
Layers
Activ.
Copies
Enc
Message
128
2
Enc
Blstats
Conv-1D, 1D, MLP
128
4
ELU
Enc
Pixel Obs
Conv-2D, MLP
512
5
ELU
Enc
Action Hist
one-hot
128
Enc
Recurrent
LSTM (frozen)
512
Core
Default
Trnsfrer
1408
3
GeLU
Core
Large
Trnsfrer
1408
GeLU
Dee
Default
MLP
512
1
Dee
Hier Strat
MLP
512
1
Dec
Hier Action
MLP
512
2
ELU
13
Exponential Linear Unit (ELU) activation functions as per the original CDGPT model architecture [23], our transformer modules employ Gaussian Error Linear Unit (GeLU) activations [25]. Doubling the number of attention heads and the hidden dimension of transformer encoder layers in addition
models resulted only in equivalent or additional policy performance degradation over that observed for the six layer model whose specifications are reported in Table 6; thus, we include only results for the latter here.
to the layer count transformer-L.STI
Context Length Models belonging to all policy classes from the LSTM family are trained by sequentially "unrolling" batched-pree sanched-predictionn.. The length of this "unrolled" sequence held fixed throughout training, and is specified by the value of the "ttyrec unroll length" hyperparameter in Table 3, also referenced as c in the main body this paper.
A fixed context length is also used to train the core transformer modules of policy classes from the transformer-LSTM family, similarly specified via the "ttyrec unroll length" hyperparameter. We found causally masking context in transformer attention mechanisms to be greatly beneficial towards improving the generalization capability models. The pre-trained frozen LSTM "recurrent encoder" module of these networks provides very cheap and simple means of dramatically extending the effective context length to cover full NetHack games (which may span hundreds of thousands of keypresses) without substantially slowing model training. As delineated in Section 5 and Appendix G, we find the recurrent module be crucial component of these models, with its ablation resulting in substantial performance decline.
19

---

Hierarchical Policy Variants As alluded to in Tables 5 and 6, as well as in Section 4 of the main paper, all hierarchical policy variants are equipped with two sets of decoders: one high-level strategy decoder trained to predict the thirteen possible strategy labels in HiHack; as well as thirteen low-level action decoders, trained to prediet actions corresponding to a single HiHack strategy across the 121- dimensional NLE action space [34]. Our hierarchical policies thus mimic the hierarchical structure of Autoascend, with an action decoder corresponding to each of the eleven, explicit strategies executed by the symbolic bot as well as two additional action decoders corresponding to the bot's s "initialization" routine (an implicit twelfth strategy) and "write-errors, representing missing strategy labels", respectively, as introduced in Figure 7. A full, diagrammatic illustration of the Hierarchical LSTM policy architecture is provided in Figure 3.
In all Hierarchical Behavioral Cloning (HBC) experiments, a BC loss was computed for the strategy decoder via ground-truth, batched Hick strategy labels. A separate BC loss was computed for a single action decoder over batched HiHack action labels, with this action decoder "selected" in an end-to-end fashion by the strategy decoder. The action decoder "selection" procedure was executed across batches by sampling predicted strategy indices from the strategy decoder with Gumbel-Softmaa re-parameterization [28], thus preserving gradient flow across the bi-level hierarchical structure of policies during training. An illustration of the full Hierarchical LSTM policy architecture is provided in Figure 3.
The
BC loss component was re-weighted (via the "strategy loss coefficent" hyperpa- rameter, introduced in Table 3) and recombined with low-level action decoder losses produce a single, overall HBC loss.
We resolved the presence of ttyrec transitions with missing strategy values, represented via the "write-error" hierarchical label in HiHack, by extending the action-space of the hierarchical strategy decoder and adding an additional hierarchical action decoder copy corresponding labels. It is possible that the performance of hierarchical policy variants can be further improved by instead filtering out all transitions with this property. We leave this evaluation for future work.
this class of
NLE Seed Counts (f) va. Starting Role (n 3402)
AutoAscend NLE Score (1) vs. Starting Role
50000
5000 (NLE)
counts
200
25000
100
100
mean AutoRscend score median Autohscend score
AV
But
CAv
Hea
Krd
Mon
PH
Ran
Roo
Sum
Val.
Wi
Bay
Cav
Hea
Krd
Mon
Pri
Ran
Rog
Swm
Toa
Val
Wiz
Figure 10: Left: The distribution of starting roles across the large-scale "in-depth evaluation." As described above, this evaluation run for the pop-erfornnnn neural policy seeds (out of 6) across model class. We observe a uear-uniform distribution of possible NLE roles across random seeds. Right: Autoascend NLE Score distribution vs. starting role in the "in-depth evaluation." " This figure is companion to the visualizations of neural policy NLE scores across role in Figure 6. As in Figure 8, we indicate absolute mean and median values of AutoAscend NLE score in the "in-depth evaluation" " with dashed-blus -lue and solid-pink lines.
F Evaluation Details
For all policy classes belonging to the LSTM model family, we observe monotonic improvements in the performance of models on withheld instances of NLE as function of training samples; as a result, we employ the final training checkpoint of these policies when running evaluations across policy seeds. In contrast, the generalization capabilities of models in the transformer-LTMM family do not monotonically improve as function of training samples, which we interpret as an indicator of overfitting. Thus, evaluations are conducted only for the "best" checkpoints corresponding to each policy seed, as evaluated on the basis of the rolling NLE score proxy metric. An in-depth description of this metric, as well as experiment training curves supporting the claims of over- and underfitting across model classes, can be found in Appendix G.
'Please refer to our earlier discussion in Appendix for more details.
20

---

Two classes of evaluations are conducted in this paper for such checkpoints: a "standard evaluation" of policy NLE score across randomly sampled and withheld instances of environment, and an "in- depth evaluation," recording all metrics of game-play and employing precisely the same set of seeded environment instances to evaluate all policies.
The former policy evaluation procedure mirrors the one conducted during the NeurIPS 2021 NetHack Challenge Competition [23]. This the procedure we employ compute the mean and median NLE scores associated with policy seeds for all experiments in this paper as well as to compute the estimates of AutoAscend mean and median NLE score in Table 2, producing our core results.
The latter policy evaluation procedure yields a suite of more fine-grained metrics for informed and "human-like like" game-play in NetHack, such as maximal dungeon level reached and the total life-time of the agent. We run this evaluation procedure for each of the best-performing' seeds from each neural policy class, well for AutoAscend. Metrics computed with this procedure are denoted via the (t) symbol throughout the paper.
Standard Evaluation Policies are evaluated on randomly seeded batch of 1024 (withheld) NLE games. Only the final NLE scores at the end of game-play are recorded.
In-Depth Evaluation Policies are evaluated across precisely the same seeded batch of 3402 (withheld) NLE games, i.e. all agents play precisely the same set of starting roles across the same NLE dungeon configurations, none of which are covered in HiHack. All games are recorded to the ttyrec data format, and can be streamed post-facto.
visualization of the distribution of starting roles covered this evaluation, as well as the corre- sponding AutoAscend score distributions (factorized by role across game instances), are shown in Figure 10.
Model Parameter Scaling: From 50.8M 98.6M Parameter (Transformer LSTM) Policies
Policy Performance vs. Training Samples
Loss vs. Training Samples
3000
1.0
roling soore (NLE)
Pollcy Architecture
0.8
2000
1086
Transformer LBTM
1000
0.4
Transformer LSTM Jarge)
0.0
0
BC samples (1e@)
BC samples (1 66)
Figure 11: Model Parameter Scaling experiment training curves. In each plot, solid lines reflect point-wise averages across 6 random seeds, while shaded regions reflect point-wise min-to-max max value ranges across seeds. Left: Rolling NLE evaluation score vs total BC training samples, for "default" and 2x deeper transformer-LSTY, policies. Right: BC loss vs total BC training samples.
G Training Curves
We provide training curves reflecting all conducted experiments. In Figures 11 and 12, display both rolling NLE scores as well as BC loss curves as a function of training samples, across all model and data scaling experiments presented in Section 5. In Figure 13, we display aggregate rolling NLE scores as a function of training samples for all remaining BC and APPO + BC experiments, separated according to model family.
The rolling NLE score metric introduced and displayed in the figures discussed here reflects an evaluation of policy performance on withheld NLE instances conducted continually during model training in "rolling" fashion via fixed number of CPU workers. As such, this metric biased towards shorter-length games, with the value smoothed rolling score as a function of As indicated by overall mean NLE score in the "standard evaluation" procedure.
Rolling NLE Score
21

---

Dataset Scaling: From 10 to 100,000 AutoAscend Games
Policy Performance Training Samples
Transtormer LSTM
3000
3000
rolling Score (NLE)
2000
2000
rolling
1000
Dataset Size (garne))
0
1e1
20
BC samples (1e6)
BC samples (1e6)
1e2
(Loss vs. Training Samples)
1e5
LSTM
Transformer LSTM
1.00
1.00
0.75
1e5
0.50
0.50
0.00
0.00
BC samples (1e8)
BC samples (1e88)
Figure 12: Dataset scaling experiment training curves. All experiments for a given dataset size were trained on dataset sub- sampled without replacement from HiHack. The sub-sampling procedure was seeded. Training hyperparameters and model architectures were identical across all runs belonging
single policy class. As in Figure 12. solid lines reflect point-wise averages across 6 random seeds, while shaded regions reflect point-wise min-to-max value ranges across seeds in all plots. Top: Rolling NLE evaluation score vs total BC training samples, across dataset sizes for the "LSTM"" and "transformer-SSTW policy classes. Bottom: BC loss vs total BC training samples, across dataset sizes for the "LSTM" and "transformer-LSTTW policy classes.
training sample confounded by the speciic relationship between NLE score and total game turns. Rolling NLE score is thus not interchangeable with the large-batch "standard evaluations" employed elsewhere this paper and presented in-depth in Appendix F. However, unlike BC loss, serves efficient and useful (if noisy) proxy measure of policy generalization over the course of training.
Model Parameter Scaling Training Curves As shown in Figure 11, the generalization capability of transformer-LSTM policies (approximated via rolling NLE score) peaks soon after the start of training, decaying and flattening out training proceeds despite continued improvement in BC loss. This observation supports the claim made Section that transformer based models overfit to HiHack.
Despite the aforementioned noisy nature of rolling NLEE score, the "Policy Performance vs. Training Samples" curves on the left of Figure 11 allude our large-scale "standard evaluation" finding from Section 5; namely, that policy performance does not increase when model parameter count scaled up, even after training typerparameters are tuned. Similarly, the "Loss Training Samples" curves on the right of this figure indicate nearly identical training errors across both models as a function of BC training samples.
Dataset Scaling Training Curves In Figure 12, we note that for datasets consisting of, or ex- ceeding, 1000 AutoAscend games, LSTM policies do not appear to overfit over the course of our BC experiments, with rolling NLE score consistently monotonically increasing as a function of BC training samples for all such policies. However, positive relationship between dataset size and maximal rolling NLE score over training persists, indicating that the addition of more data does lead measurable (if sub linearr improvements in policy generalization. An inspection of
22

---

LSTM policy loss curves suggests a similar story. Losses across policy seeds trained on 10 and 100 AutoAscend games drop swiftly to values near zero, supporting this overfitting hypothesis.
The transformer-LSSTI policy loss curves in Figure 12 also reveal harsh overfitting for policies trained with 10 and 100 games. Interestingly, the generalization capability of these policies, again indicated by rolling NLE score over training, is vastly superior to that of their LSTM counterparts. Indeed, we observe that transformer-LSTE
policy trained on just 100 games vastly outperforms pure LSTM policy trained on 10x as many games. We attribute this gap to the frozen nature of the pre-trained LSTM component of the ransformer-LSTWW policies employed as a recurrent encoder in these policies, and hypothesize that it is the static quality of the recurrent representation output by this encoder which bolsters the generalization capability of the resultant models in exceptionally low-data regimes.
Aggregate Policy Performance vs. Training Samples
Policies)
va. Hierarchical LSTM
Transformer LSTM Hierarchical Transformer LSTM
3000
3000
rolling score (NLE)
2000
2000
rolling score
1000
1000
Policy Architecture
0-
LSTM
20
40
LSTM XXL dec
BC samples (1e8)
BC samples (1e8)
(APPO Policies)
Haaaarchicl LSTM
LSTM vs. Hierarchical LSTM
Transformer LSTM vs. Hierarchical Transfommer + LSTM
Transformer +
10
8
LSTM
roling score (NLE)
3000
RL samples (1e6)
rolling score (NLE)
3000
AL samples (1e8)
Hierarchical LSTM
2000
2000
1000
1000
10
15
BC samples (1o8)
BC samples (1o8)
Figure 13: Aggregate rolling NLE evaluation score curves for the central BC and APPO + BC experiments discussed in this paper. Model parameter and dataset scaling training curves are omitted here on account of being visualized in Figures 11 and 12, respectively. For APPO + BC experiments, batch accumulation was employed to ensure that the ratio of RL to BC samples "seen" during training was 1:1. This property is indicated via the secondary x-axes of APPO + BC figures here, which show RL sample quantities. Top: Rolling NLE evaluation score vs total BC training samples in pure BC experiments, across mon-hierarchica and hierarchical LSTM and transformer- LSTM based policy classes. Bottom: Rolling NLE evaluation score vs total BC training samples in APPO BC experiments, across non-hierarchical and hierarchical LSTM and transformer-LSTM based policy classes.
Aggregate BC and APPO + BC Training Curves The "Aggregate Policy Performance vs Training Samples" curves of Figure 13 align with the general model family training trends previously observed in the scaling experiments. Notably, we find once again that LSTM based policies' rolling NLE scores improve monotonically with training samples whether training conducted with BC or APPO + BC. while this is not the case for transformer-LSTH based models. Indeed, the generalization properties' of policies belonging to this model family improve at the start of training before worsening as training proceeds across BC experiments. We interpret continued demonstration of these trends as further support for the claims of LSTM underfitting and transformer-L. overfitting made in Section 7 of the main body of the paper well as in Appendix F.
23

---

Moreover, we note that the introduction of an RL loss induces particularly large amount of volatility in the rolling NLE score associated with transformer-LSTH. based models, disrupting the monotonically decreasing relationship between score and samples previously observed for these policies following
10 training samples in the pure BC experiments (i.e. the eventual overfitting behavior). This observation suggests that the performance of policies belonging to this model family bottlenecked by an insufficient total throughput of "on-policy" or interactive data.
Flat Transformer and Transformer + LSTM Policy Performance vs. Training Samples
Transtomer Policies
Flat Transtomers Onl
Pollcy Architecture
3000
200
Transformer LSTM
ralling score (NLE)
(NLE)
150
Hierarehical Transforer LSTM
2000
100
Transformer (o
1000
Fiat Transformrr (c 128)
Transformer
BC samples (1e6)
BC samples (1e@)
curves for flat transformer policies vs transformer-LSTI and hierarchical transformer-LSTM policies during BC training. Flat transformer policies have similar architecture to transformer-LSTMs with exceptions: (1) the pretrained, frozen LSTM component of the latter is ablated from the former; and (2) the core modules of tested flat transformer policies consist of twice as many (i.e. 6) transformer encoder layers. We perform a sweep over values of the context-length parameter c, corresponding to the number of consecutive timesteps included in the flat transformer context length per sample during training with BC. A total of six random seeds for each model class were employed to test flat transformer policies. Training for each random seed was conducted on a single GPU for hours.
Figure 14: Aggregate rolling NLE evaluation
Ablating the LSTM Component of transformer-LsT Policies To establish and confirm the importance of the recurrent LSTM component the transformer-LSSMM policies introduced this paper, we perform set of ablation experiments with policies featuring transformer core module only, meaning all input is confined strictly to the transformer's context window at each iteration of training and inference. We refer models with such architecture as flat transformers. As demonstrated in Figure 14, we find flat transformers to perform substantially worse evaluation when trained with behavioral cloning Autenscend demonstration data than either their pure LSTM or transformer LSTM counterparts. The context-length parameter, denoted as in the legend of Figure 14, reflects the number of contiguous observations employed for action prediction. We perform a small sweep over values this parameter in an effort to establish the relationship between observation history and model generalization. Reported results reflect the best performing flat transformer architecture after conducting hyperparameter tuning across transformer encoder layer size, layer count, and attention head count.
We make two key observations: (I) the number of samples seen during the 48-hour training window is inversely proportional to the context length employed in training; (2) the rate of BC policy improvement does not increase with context length, across the set of context length values tested here. It is this basis of (2), coupled with the fact that the average AutoAscend demonstration in our HiHack dataset consists 27000 keypresses (Table 1). several orders of magnitude beyond the longest context length, 256, find feasible to test for flat transformers, that hypothesize that an encoding of full game history is important for imitation in our setting.
H Low-Sample Hypothesis Testing
To establish the statistical significance of the core BC and APPO BC empirical results reported in the main body of this paper, we perform series of two sample t-tests with decision threshold of 0.05. The results of these t-tests are summarized in Tables 8, and We confirm all reported findings.
24

---

Table 7: Statistical significance of policy comparisons across BC experiments: two-sample t-tests on mean policy NLE scores in evaluation, across six random seeds (df 5). We compute p-values associated with three different null hypotheses, testing for improvement in performance/generalizaati over the baseline LSTM (i.e. CDGPT5) [23] and hierarchical LSTM policies, as well as a decline in perroomanceeggnnraaizziiio over the 3-layer non-hierarchical transformer-LSTM for models trained with BC.
Ho : 32 S 3LSTM
Hu 3x S SherLLIM
Hu 3g 2 Bundami-LSM
Hierarchy
p-value
reject Ho
p-value
reject Ho
p-value
reject Ho
LSTM (XXLddee) LSTM
.302255
.000017
Sunsformer-I.ST..
X
< .00001
.0001 25
Transformer-I. L.M (large)
X
.000014
.001646
.001792
Transformer-L.TT
.000063
.002549
.0160355
Table 8: Statistical significance of policy comparisons across APPO + BC experiments: two- sample t-tests on mean policy NLE scores in evaluation, across six random seeds (df 5). As in Table 7, we compute p-values associated with three different null hypotheses, but across model classes trained with APPO BC.
Ho : By S FLSTM
Ho : 8y 8hie-11TMM
Ho : By 2 81ms/m--1.STM
Hierarchy
p-value
Reject Ho
p-value
Reject Ho
p-value
Reject Ho
LSTM (XXL doo)
X
470427
x
LSTM
/
.013977
Transfoma-LLTMM
X
042154
318223
x
Transforme-LSTM
/
.066776
.362569
X
.122558
Table 9: Statistical significance of gains in policy performance from APPO + BC vs pure BC: two-sample t-tests on mean policy NLE score 3 in evaluation, across six random seeds (df = 5). We compute p-values associated with the null hypothesis that policy performance yielded by APPO + BC training is less than or equal to that yielded by pure BC. We reject this null hypothesis for all but hinrhheracc transformer-LSTM models, for which we see no statistically significant improvement in policy generalization with RL fine-tuning.
Ho 8 BC) S (T(
Hicrarcby
p value
Reject Ho
LSTM [23]
X
000121
LSTM (XXL doe)
X
000017
LSTM
V
.000066
Transforma-LLTM
X
.416951
X
Transformer
.003269
I Distributional Visualizations of Evaluation Results
Max Dungeon Level Reached vs. Total Turns In Figure 15, we supplement the absolute mean and median max-dungcon level and agent life-time (in game turns) statistics introduced in Table with 2-D distributional visualizations of both the raw values of these metrics well as their mutual inter-relationship. evaluated via n = 3402 seeded NLE games run individually for all policy class representatives in our "in-depth evaluation."
The AutoAscend 2-D contour plot on the right of this figure reveals an interesting emergent property of the bot's behavior, which is confirmed by study of individual games: the high-level "descent behavior" of AutoAscend appears to fall along one of two modes. In the first of these modes, the bot spends very large amount of turns on dungeon level 1, and avoids descending further into the dungeon before the end of the game. In the second mode, the bot begins to rapidly descend fairly deep into the dungeon, reaching far as level 11 in the sample of game seeds tested here. The overall
25

---

NLE Max Dungeon Level Reached (f) vs. Total Turns (1) Across Neural Pollcy Classes
(BC Policies) Herwchical
LSTM
27p61
12:00
95-45
52x
tam (LE)
Transformor LSTM
Heerorhiical Tranffrmer e LSTM
1reck
1de44
7ied5
83x40
Liet0
4.5 05
(AutoAscend)
Cheds
12
16c-05
100E
2:00
2001
1030
2001
3000
use
10
1.3e-05
tumss (A.
9.8e-06
7.6c-06
(APPO BC Policies)
L8TM
Hicrarshical LSTM
4.8e-06
33c-3
3.7e-06
2
2.5e-06
14e-06
3.4e-07
20000
40001
15
tums (NLE)
Transformer - -STM
Harwchod Transformer + LSTM
Losil
130k
Elem
25
2.006
23000 3030
hume (u.1)
Figure 15: Max dungeon level reached vs. total turns across "in-depth evaluation" games, visual- ized via 2-D contour density plots. Contour densities are indicated by the color-bars accompanying each subplot. Mean quantity values (computed dimension evaluation" batch are indicated by white 'X' symbols in each subplot. The symbolic bot's vastly superior ability to play longer and descend much further into the dungeon creates separation in scale between it and its neural counterparts; as a result, for clarity, we indicate the max dungeon level vs turns subspace displayed in the neural policy contours with bolded black rectangle in visualization of AutoAscend's 's behavior. Top Left: Best-performing BC neural policy seeds. Bottom Left: Best-perf APPO BC neural policy seeds. Right: AutoAscend.
for all policies) across the "in-depth
relationship between AutoAscend's total in-game life-time and max-dungeon level reached appears be roughly quadratic.
In contrast, none the evaluated neural policy class representatives comes close to achieving in in-game progress resembling AutoAscend's secondary behavior mode. A large majority of games for all neural policy classes appear to end on the first level of the dungeon, with policies very rarely surviving long on this level AutoAscend. This suggests that neural policies may be failing to master the long-range, very low-level hehaviors of the bot, even when these behaviors are factorized across AutoAscend strategies, as the case of hicrarchical policy variants.
Nevertheless, the qualitative performance of hicrarchical policies clearly improves upon that of non- hierarchical models, with the hierarchical LrAM policy trained with BC both surviving longer dungeon level and descending with higher frequency than other BC-trained policy representatives. Furthermore, this qualitative behavior appears be strengthened when interaction is added into the mix, with the mode centered on "dungeon level 2" increasing in density for the APPO + BC variant of the hierarchical transformer-LSTTM representative policy.
Taken together, these sets of observations lead us to hypothesize that the quality of neural policies trained with imitation learning on extremely complex, long- morizon tasks like NetHack may be
26

---

further improved with increases in the scale of peturchiccall-nnffrme fixed behavioral factorization, beyond that explored this paper. We consider this be a very exciting direction for future work.
Log-Score Distribution (i) Across Neural Policy Classes
(BC Policies)
LSTM
Hirr Horarchical LSTM
4009
40%
00URR3
100
100
logg (econs
Transformer LSTM
Horrarhhoaa Transformor
650m
6326
(AutoAscend)
0
16
256
4096
65336
0ounts
counts
300
counts
200
lotty (spore)
(ncore
(APPO Policies)
100
LSTM
Herarchical LSTM
20c
0146
254
300
logg score)
counts
DoUms
100
100
12
12
12
36
long (score)
Incy(acore)
Transformer
LSTM
1 lererchical Transformer + LSTM
Zb6
283
05006
1n
4230
53006
00UP55
counts
koty (Rcom))
(ncore
Figure 16: Log-score distribution across "in-depth evaluation" games. As in Figures and 10, we indicate absolute median and mean values of policies' NLE scores in the "in-depth evaluation" with dashed-blue and solid-pink lines. The introduction an RL loss reduces the "mass" associated with the left-tail of log-score and increases the "mass" associated with the right-tail across all neural policy classes, inducing right- ward shifts in median and mean scores, though difficult to perceive in this figure account of the log-scale of the x-axis. We refer the reader to Table in the main paper for aggregate absolute numerical values of NLE score. Top Left: Best-performing BC neural policy seeds. Bottom Left: Best-performing APPO BC neural policy seeds. Right: AutoAscend.
Log-Score Distributions We conclude this supplementary analysis with a visualization of log-score distributions across neural policy classes in Figure 16, computed over the "in-depth evaluation" seeded games. The trends displayed in this figure align with those described and demonstrated previously. Improvements in model architecture as well as the introduction of hierarchy and interactive learning lead to re-distribution of "mass" between left and right tails of distribution, with the overall counts of "low score" games decreasing and "high score" games increasing when these improvements are applied. The gap to AutoAscend is significantly reduced, but not bridged.
27


Reviews:
[{"review_id": "88GRCLPmV5", "reviewer": "Reviewer_eWjQ", "summary": "The paper explores reasons for this performance gap between neural and symbolic methods in NetHack:\nSymbolic agents use hierarchical policies and parsers to extract high-level features\nSymbolic agents have handcrafted heuristics and error correction\nNeural agents lack inductive biases like hierarchy that may be needed for sparse rewards\nExperiments show hierarchy, scale, and combining imitation and RL help improve neural agents:\nHierarchical behavior cloning improves over flat BC\nLarger Transformer-based architectures improve over LSTMs\nRL fine-tuning provides gains, especially for underfitting models\nBut significant gaps to symbolic agents remain", "questions": "Can you add some experiments, add some theoretical derivation, whether the contribution of this article is more.", "limitations": "The model is not so representative, can switch a more popular model。Overall, the authors demonstrate good care and thoughtfulness regarding the limitations and potential negative impacts of this research direction. The discussion seems sufficient without being overreaching or distracting from the primary technical contributions. I do not have any major suggestions for improvement.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The experimental design is very clever, the chart is very clear, and the experimental effect is obvious. The paper explores a novel problem domain of applying neural networks to master the game NetHack, where current methods struggle compared to symbolic AI. The authors introduce a new large-scale dataset of NetHack demonstrations called HiHack to facilitate this analysis. The idea of using demonstrations to help neural networks learn better policies in sparse, long-horizon environments like NetHack is creative.The methods are detailed appropriately to replicate experiments. Results are presented logically and incorporate useful visualizations. The conclusion summarizes takeaways concisely.Mastering complex environments like NetHack with sparse rewards and long time horizons remains an open challenge for deep RL. This paper provides significant evidence and analysis characterizing the limitations of current neural network methods in these settings, and points the way towards progress, whether via incorporating stronger inductive biases like hierarchy or combining neural and symbolic approaches. The insights will broadly impact research in sparse reward RL, imitation learning, and integrating neural and classical AI.", "weaknesses": "This model is based on the nethack, and the results hold up on the above models, and whether the above results can still hold up on the other models。The authors recognize the limited generality so far of methods tested on NetHack to other complex environments.No obvious harmful biases or problematic data sources are introduced in this work. The NetHack environment itself seems relatively innocuous.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\n\nWe thank you for your feedback and comments on our submission. We are glad that you found our paper to be clear and insightful.\n\n\n> The model is not so representative, can switch a more popular model\n\n\nThe transformer-LSTM policy architecture that we employ in this paper is indeed novel, and was developed by us as a result of the limited generalization and expensive gradient updates that we observed when experimenting with more standard transformer-only architectures. \n\n\nA long context length appears to be greatly beneficial to successful imitation of AutoAscend. We found augmentation of a causal transformer-core with an LSTM recurrent module to offer a simple and, importantly, highly lightweight solution to this issue. \n\n\nPlease take a look at **section (ii)** of our general rebuttal above as well as the attached PDF for additional reasoning and figures supporting these claims.\n\n\n> This model is based on the nethack, and the results hold up on the above models, and whether the above results can still hold up on the other models\n\n\nAs above, please refer to **section (i)** of our general rebuttal for our response to this point. We will add this additional discussion to the paper. \n\n\nThank you once again for the time you have taken to review our paper, and please let us know if you have any outstanding concerns that stand between us and a strong recommendation for acceptance.\n"}, {"review_id": "DOYgFkP1j5", "reviewer": "Reviewer_Ub8t", "summary": "This is an emergency review, and I regret that the paper is out of my expertise, which is why my review will rather stay at the surface level.\n\nThe paper is concerned with the NetHack challenge, a complex AI challenge that in 2021 reached headlines, because symbolic agents considerably outperformed neural agents. I see three main contributions in the paper:\n - The construction of a large-scale dataset, based on the best symbolic agent and its policy choices, that can enable training better neural agents\n - The training of better neural agents based on this dataset, and other improvements\n - A systematic analysis of the effect of different technical improvements (hierarchical BC, larger Transformer models, larger datasets, online fine-tuning with RL), notably finding that scaling training sets or model size alone will not bridge the gap to the best symbolic agent.\n\nThe problem is of very high interest to the AI community, and the technical investigation, results, and discussion appear thorough and insightful. The dataset might also enable further research. I find especially the results regarding scaling interesting, i.e., that performance increases logarithmic, and so more data or bigger models alone will not enable achieving parity with the symbolic approach.\n\nQuality of writing is very good, and so the paper is easy to follow (subject to my lack of technical background).\n\nMinor notes:\n - The paper appears to be missing a link to the dataset\n - The related work is not easy to access for someone not close to the field. E.g., paragraphs on \"imitation learning\" and \"hierarchical policy learning\" give too little detail about the basic ideas (do not start with descriptions of what they are for, but what they do)\n - \"The full observation space of NLE is far richer and more informed than the view afforded to human players of NetHack, who observe only the more ambiguous “text-based” components of NLE observations\" - I do not fully understand this sentence, please expand. What can systems observe in NLE, that humans don't receive in the original interface? Or do you mean that NLE aggregates the Ascii terminal characters into something more high-level?\n - Showing an excerpt from the dataset would be helpful, especially, as it is not quite clear what is added there, both strategies and substrategies? Or the more specific one only?\n", "questions": "See above.", "limitations": "Yes, the authors critically discuss that scaling alone will not bridge the gap to symbolic agents on this challenge.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "See above.", "weaknesses": "See above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\n\nWe are very grateful for the time you have taken to provide thoughtful feedback and comments on our paper.\n\n\n> The paper appears to be missing a link to the dataset.\n\n\nWe will publicly release the HiHack dataset upon revision of our submission, at which time we will include a link to the dataset in the main body of the paper.\n\n\n> The related work is not easy to access for someone not close to the field. E.g., paragraphs on \"imitation learning\" and \"hierarchical policy learning\" give too little detail about the basic ideas (do not start with descriptions of what they are for, but what they do)\n\n\nThank you for these suggestions, we will increase the level of detail employed in the related work section of the final version of the paper.\n\n\n> \"The full observation space of NLE is far richer and more informed than the view afforded to human players of NetHack, who observe only the more ambiguous “text-based” components of NLE observations\" - I do not fully understand this sentence, please expand. \n\n\nThe full observation space of NLE includes not only the “text-based” view of NetHack (i.e. the tty characters observed by human players of the game), but also *glyphs* representing the exact identities of all monsters and objects currently in view as well as all items in the player’s inventory [1]. Glyphs are not (directly) observable to human players of the game in standard versions of NetHack. Our HiHack dataset follows the convention set by Hambro et al. [2], storing the text-based, *tty*-view of NetHack observations only.  \n\n\n> Showing an excerpt from the dataset would be helpful, especially, as it is not quite clear what is added there, both strategies and substrategies? Or the more specific one only?\n\n\nIn its current form, the HiHack dataset reflects high level strategy information from AutoAscend only, as detailed in sections B and C of the supplementary materials accompanying the paper. All hierarchical policies in the paper are trained against these high level strategy labels only. We will add excerpts previewing the dataset to the supplementary materials during revision.\n\n\nGiven that there is already interest, we will release an additional version of the dataset featuring both high-level and sub-strategy labels upon revision, reflecting the full structure of AutoAscend displayed in Figure 7 of the supplement.\n\n\nThank you once again for your review. Please do let us know if you have any outstanding concerns or questions about our work.\n\n\n**Citations**\n\n\n[1] Küttler, Heinrich, et al. \"The nethack learning environment.\" Advances in Neural Information Processing Systems 33 (2020): 7671-7684.\n\n\n[2] Hambro, Eric, et al. \"Dungeons and Data: A Large-Scale NetHack Dataset.\" Advances in Neural Information Processing Systems 35 (2022): 24864-24878.\n"}, {"review_id": "jedOyCgWWQ", "reviewer": "Reviewer_PtAe", "summary": "The paper improves the existing solutions in the NetHack Learning Environment (NLE). This is done by taking earlier solutions from a competition around NLE, collecting more data with the best available (symbolic) agent, and using that data to improve a neural only solution. The paper provides experiments with imitation learning (with or without RL tuning), larger models, hierarchical memory setup (LSTM + Transformers) as hierarchical behavioural cloning setup, using labels of the newly collected dataset. While there are improvements, it is still below the demonstrator results, which is then studied by scaling the model sizes and amount data collected. Paper concludes by providing the state of the art results in the task, but also noting that scaling alone is not enough to reach the expert demonstrator level (symbolic agent).", "questions": "Questions:\n1) In multiple occasions paper says that the lower performance of bigger model is due to overfitting (e.g., line 229). However there are no results/experiments to show that this indeed was the case. A simple way to find this out is to do train-validation (or even train/validation/test) split, and testing on held out data as training progresses.\n2) Regarding data scaling experiments: did you change any other settings of the training setup when increasing data amount? Previous work has demonstrated that the optimal model size and/or training compute depends on the amount of data (Hoffmann et al. 2020).\n3) Regarding model scaling experiments: I assume only the number of layers in the transformer was changed? The bottleneck of the network may be elsewhere, e.g., one of the input layers or output layers. I would recommend scaling the whole network, similar to what OpenAI VPT work did, where ResNet blocks were \"widened\" in terms of filters, as well as increasing transformer size (Baker et al. 2022). Also, Hoffmann et al. (2020) changed number of layers, number of attention heads and transformer dimensionality when scaling models. This might be something you want to try.\n4) Instead of LSTM + Transformer model, did you experiment with transformer model only? E.g., akin to VPT work (Baker et al. 2022), embed all inputs into one vector, stack vectors over timesteps, apply causal transformer, and predict actions from the transformer outputs. This type of model might scale better, as it reduces the amount of components that might interfere.\n\n#### Comments (not questions)\n- Fig1 right: weird scale. Any chance to get more points?\n- Line 205: grammar error at the start of the line\n- Explain/rename \"Dlvl\" and why \"Turns\" is good metric\n- Figure 3: \"LSTM + XXL Dec\" is bit confusing naming, since \"decoder\" is not commonly used term in the paper. I'd recommend using something like \"LSTM (bigger)\" to simply reflect that it is the LSTM baseline but with bigger network\n- Figure 3 (and others): add explanation to caption what is the error bar of the bar plots. Is it standard deviation or standard error (or something else)?\n- Table 2 caption: starts with weird \"[V4]\"\n\n#### References\n\n- Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas et al. \"Training compute-optimal large language models.\" arXiv preprint arXiv:2203.15556 (2022).\n- Baker, Bowen, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. \"Video pretraining (vpt): Learning to act by watching unlabeled online videos.\" Advances in Neural Information Processing Systems 35 (2022): 24639-24654.", "limitations": "No explicit sections for limitations or broader/societal impact was given. Authors bring up the future work ideas in the conclusion. While I think the work does not require societal impact section (no immediate impact), I urge authors still think through of any cases where the work or the insights could impact others. Or alternatively, what impact would _not_ including some results do (e.g., skipping some analysis).\n\n\n## Rebuttal acknowledgement\n\nI have read authors' rebuttal which did address my concerns, and I increased my rating from 4 to 7 to signal my vote to accept this paper (change was done before discussion period closed).", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Provides more detailed dataset than the previous works (with hierarchical action labels)\n- Sets an interesting premise/task for trying to reach the demonstrators' (AutoHack agent) performance with neural solutions.\n- Different ablations to try to answer questions (data/model scaling, model architecture with hierarchy)\n- Proposed hierarchical approach to imitate the demonstrator agent.", "weaknesses": "While I enjoyed reading the paper, overall I think the results are interesting or applicable to most of the NeurIPS audience, even in the limited scope. The paper presents many results and provides some explanations for them, but does not verify these explanations with further experiments. I think proper answers to these issues would be insightful to many, and others could then use these insights in their work (e.g., where the trained agent failed to imitate the demonstrator? What was the cause of poorer performance? Why did bigger model perform worse?). Creating such insight in one environment would be sufficient, as by focusing on a single environment, you can create very specific scenarios to tease out these answers. \n\n- Limited scope of the work: experiments done in a single environment. Most of the paper is framed in a way that this is not a huge issue (e.g., ablations), but proposing new method just for playing NLE has limited impact. If a new method is proposed to generally improve RL/IL performance, it should be tested at least in two distinct environments.\n- Limited improvement in the context of SOTA solutions: 2x over the baseline used in the paper with RL and proposed architecture included, but other neural agents in the NetHack Challenge had higher score. To be interesting in terms of performance, it should at least outperform the NetHack Challenge Neural solutions.\n- Proposed method is limited in novelty, as evident by the previous work listed in the paper. If the hierarchical BC figured out the hierarchy automatically (or, if it was an emergent behaviour of the model), that would be more interesting.\n- Paper outlines some assumptions on why things failed (e.g., \"model overfitted\" or \"learned to self-correct\"), but these claims were not verified with results. The paper would be much stronger if you can give solid, verified answer that indeed, overfitting was to blame or that RL trained the model to \"self-correct\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\n\nWe thank you for your detailed and thoughtful feedback on our paper, which has greatly helped to further strengthen our work. We hope to address the concerns and questions that you raise in your review below.\n\n\n> If a new method is proposed to generally improve RL/IL performance, it should be tested at least in two distinct environments.\n\n\nOur goal in this work is to conduct a deep, systematic, and scientific investigation of performance gaps between data-driven neural policy learning and symbolic methods employing NetHack as a testbed, rather than to introduce a new model achieving SOTA RL/IL performance in NLE. As discussed in **section (i)** of the general response above, we respectfully disagree that the insights revealed by our investigations in this paper are limited to NLE.\n\n\n> To be interesting in terms of performance, it should at least outperform the NetHack Challenge Neural solutions.\n\n\nThe two neural policies outperforming the CDGPT5 baseline from the NetHack Challenge (NHC) that you are referring to rely on hand-engineered augmentations including separated action spaces, role-specific training, and even hard-coded subroutines. CDGPT5 is not only the best open-sourced neural model from the NHC, but it is also the best purely data-driven model [1].\n\n\nAs stated above, our goal in this study is to probe the general capability of neural approaches at learning complex, flexible behaviors directly from multimodal data; hence, we refrain from reliance on any NLE-specific policy augmentations. Respectfully, we believe that this choice makes our analysis more interesting, both from a scientific perspective and to the RL/IL community at large, rather than less.\n\n\n> If the hierarchical BC figured out the hierarchy automatically (or, if it was an emergent behaviour of the model), that would be more interesting.\n\nWe thank the reviewer for this suggestion. \n\nTo our knowledge, our paper is the first to demonstrate that hierarchical labels can offer a stable and powerful mechanism for embedding behavioral priors into policies during pre-training.\n\nAs observed by reviewer qyRC, there are few complex environments where expert demonstrations are accompanied by hierarchical labels, though such labels are in-principle no more difficult to collect than demonstrations. We are hopeful that our paper may spur the creation of more datasets with such labels as well as further investigation of how hierarchical behavioral priors may best be exploited for policy generalization, both in NetHack and beyond.\n\n> In multiple occasions paper says that the lower performance of bigger model is due to overfitting (e.g., line 229). However there are no results/experiments to show that this indeed was the case.\n\n\nWe provide detailed support for our claims of model overfitting and underfitting in Section G of the supplementary materials accompanying our paper. In Figures 11, 12, and 13 of this section, we include visualizations of policy performance via “rolling NLE score” [2] on held-out validation seeds of the NetHack Learning Environment (NLE) across training iterations, for all experiments conducted. We will include explicit references to these supplemental figures to help guide readers in the final version of this paper.\n\n\n> ...did you change any other settings of the training setup when increasing data amount? \n\n\nOn optimal training compute: The results of the data scaling experiments visualized in Figure 5(right) of the main paper reflect the aggregate, large-scale evaluation performance of “best” model checkpoints across dataset subsample sweeps, selected on the basis of the “rolling NLE score” metric [2]. Consequently, our results in these experiments do reflect provisions for optimal training compute, subject to the 48-hour compute budget that we employ as a basis for fair comparison of policies across model classes. For more details on evaluation procedures, please refer to Section F of the supplementary materials.\n\n\nOn optimal model size: The goal of this particular set of experiments was to test the effects of data scaling and model parameter scaling in a decoupled setting; as a result, the model architectures tested in the data scaling experiments were held fixed, with only the size of the dataset varied. Please refer to Section E of the supplementary materials for detailed description of model architectures, with all employed model hyperparameter values included in Tables 5 and 6.\n\n\n> I assume only the number of layers in the transformer was changed? \n\nIndeed, only the number of layers in the transformer was changed here. The architecture of the default transformer-LSTM model (i.e. with 3 layers) was tuned for input and output layer width when these experiments were launched. We will include a version of these experiments with all layer parameters scaled as you suggest in the final version of this submission.\n\n> ...did you experiment with transformer model only?\n\nPlease see **section (ii)** of the general response as well as the attached PDF.\n\n> Is it standard deviation or standard error (or something else)?\n\nThank you for pointing out this omission. Indeed, all error bars on visualizations of experimental results reflect standard error.\n\nWe also appreciate your comments identifying typos, and will make according adjustments.\n\nThank you, once again, for the time you have taken to provide thorough feedback on our submission. We would be grateful if you would notify us whether you have any additional concerns or questions preventing a recommendation for acceptance.\n \n**Citations**\n\n\n[1] Hambro, Eric, et al. \"Insights from the neurips 2021 nethack challenge.\" NeurIPS 2021 Competitions and Demonstrations Track. PMLR, 2022.\n\n\n[2] Hambro, Eric, et al. \"Dungeons and Data: A Large-Scale NetHack Dataset.\" Advances in Neural Information Processing Systems 35 (2022): 24864-24878.\n\n"}, {"review_id": "OQtwCFUdVf", "reviewer": "Reviewer_qyRc", "summary": "# Problem Statement\nThe paper addresses the challenge of neural policy learning methods struggling in long-horizon tasks, particularly in open-ended environments with multi-modal observations, such as the game NetHack. It was observed that symbolic agents significantly outperformed neural approaches in the NeurIPS 2021 NetHack Challenge.\n\n# Main Contribution\nThe paper's main contribution is an extensive study on neural policy learning for NetHack. The authors analyzed the winning symbolic agent and extended its codebase to generate one of the largest available demonstration datasets. They examined the advantages of an action hierarchy, enhancements in neural architecture, and the integration of reinforcement learning with imitation learning. Their investigations resulted in a state-of-the-art neural agent that surpassed previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, they also demonstrated that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players.\n\n# Methodology and Experiments\n\n## The Hierarchical HiHack Dataset\nThe authors create the HiHack dataset, which is a hierarchically-informed version of the NetHack Learning Dataset (NLD-AA), containing 3 billion recorded game transitions from over a hundred thousand games played by the AutoAscend agent.\n\n## Hierarchical Behavioral Cloning\nThe authors extend the ChaoticDwarvenGPT5 (CDGPT5) model, a top-performing open-source neural model for NetHack, by introducing a hierarchical decoding module. The model consists of three separate encoders for different types of observations and an LSTM core module. The hierarchical extension replaces the linear decoder of the CDGPT5 model with a hierarchical decoder that predicts the strategy label and selects the appropriate low-level MLP for action prediction. The hierarchical LSTM policy and the baseline non-hierarchical LSTM CDGPT5 policy are trained using a simple cross-entropy loss. The results show that the introduction of hierarchy significantly improves the performance of LSTM policies trained with behavioral cloning, yielding a 40% gain over the baseline in mean NLE score and a 50% improvement in median score across seeds. The authors confirm that this improvement is due to hierarchy and not simply a result of the increased parameter count of the hierarchical LSTM policy.\n\n## Architecture and Data Scaling\nThe authors explored scaling as a potential solution to improve the performance of the model, which was significantly behind the symbolic policy used to generate the HiHack demonstrations. They developed a novel base policy architecture for NetHack that introduces a Transformer module into the previous CDGPT5-based architecture. They also conducted data scaling experiments using subsets of the HiHack dataset to examine the relationship between dataset size and the test-time performance of BC policies. The results showed that both the non-hierarchical and hierarchical variants of the combined transformer-LSTM policy architecture yielded gains, but the larger model performed worse than the smaller one due to overfitting. This suggested that scaling of model capacity alone would not be sufficient to close the neural-symbolic gap. Additionally, brute force scaling of the dataset alone could not viably close the gap to symbolic methods.\n\n## Combining Imitation with Reinforcement Learning\nThe authors explored combining imitation learning with reinforcement learning (RL) to bridge the performance gap with AutoAscend. They used a combination of behavioral cloning (BC) and asynchronous proximal policy optimization (APPO) for training. The results showed that RL fine-tuning significantly improved the performance of all models. The best-performing approach was APPO + BC using the hierarchical LSTM model, which achieved a new state-of-the-art for neural policies on NLE, surpassing the previous best result by 48% in mean NLE score and 25% in median NLE score. The Transformer-LSTM models performed worse due to their slower training speed and the fixed training time budget. The authors also observed that fine-tuning with RL improved the error-correction capability of models across all model classes compared to their purely offline counterparts.", "questions": "1. When introducing Transformer to augment the capacity of the neural model, why did authors choose the architecture as shown in the article? Specifically, transformers are best known for their NLP and CV capacity, which could make them good replacement for the CNN and MLP encoders.\n2. Why do the authors enforce the 48 hour training time cap instead of training all models till convergence? Given that this study does not appear to prioritize data efficiency or training efficiency, the necessity of such a computational time constraint is unclear. It would be beneficial to understand the rationale behind this choice, as it may not directly align with the study's primary objectives.", "limitations": "The authors note that possible avenues for future exploration include: (a) methods for increasing the Transformer context length to give the agent a longer memory to aid exploration; (b) addressing the multi-modal nature of the demonstration data (i.e. quite different trajectories can lead to the same reward), which is a potential confounder for BC methods. Some forms of distributional BC (e.g. GAIL, BeT) could help alleviate this issue.\n\nThe aforementioned two points do not address the limitations raised in the \"Weakness\" section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Originality\nThe problem is interesting and the approaches are insightful.\n\n# Quality\nThe analysis and experiments are comprehensive.\n\n# Clarity\nThe article is overall well written and clear.", "weaknesses": "1. The current focus of the study is quite narrow, being primarily centered on the application of imitation learning for NetHack, limiting its influence. In the context of mastering the game, while this approach is interesting, it is unlikely to exceed the performance of experts that generate demonstrations, not to mention that the experts are already algorithms that can scale well. Furthermore, NetHack, despite being an excellent game, is somewhat niche and its real-world implications are relatively minimal. The techniques proposed in this study are specifically tailored for this game, which limits their potential for inspiring more universally applicable methods that could have a broader impact.\n  - The availability of hierarchical labels is a strong assumption that does not often hold, which further limits the applicability of the proposed methods.\n\n2. Even just for bridging the performance gap between neural models and AutoAscend, there is no promising direction revealed by the work as the various augmenting components seem to contradict each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\n\nWe thank you for your thoughtful and detailed feedback on our submission. We look to address your remaining concerns about our paper below.\n\n\n> ...while this approach is interesting, it is unlikely to exceed the performance of experts that generate demonstrations, not to mention that the experts are already algorithms that can scale well.\n\nRather than to achieve SOTA in NetHack, our goal in this paper is to employ NLE as a testbed for investigation of the general performance gaps between neural policies and symbolic agents in long-horizon, open-ended environments. \n\nPlease also note that while AutoAscend is the SOTA artificial agent in NLE, the median in-game score it achieves is still several orders of magnitude short of a typical expert player’s “ascension” score. The game remains unsolved by any artificial agent [1].\n\n\n>  The techniques proposed in this study are specifically tailored for this game, which limits their potential\n\nPlease take a look at **section (i)** of our general rebuttal.\n\n\n> The availability of hierarchical labels is a strong assumption that does not often hold, which further limits the applicability of the proposed methods.\n\nHierarchical labels have been gathered for other settings, such as in the 3D DeepMind Playhouse environment, where human annotators were asked to provide natural language instructions or questions labeling manipulation \"subtasks\" [2]; thus, they are not impractical to gather. In general, we hope that the impact of hierarchical priors via structured labels revealed by our work might spur the future collection of more demonstration data with this property in open-ended environments.\n\n\n> ...there is no promising direction revealed by the work as the various augmenting components seem to contradict each other.\n\nOur experimental results suggest that the introduction of structured labels is strongly beneficial to the improved performance of lighter-weight, data-limited LSTM policies, both in the BC and BC + RL settings. \n\nHowever, in the pure BC setting, we find that architectural improvements and scaling make these beneficial effects of structured labels obsolete.\n\nFurthermore, we find a “no free lunch” effect to hold for large transformer-based policies in our BC + RL experiments: they are far more unwieldy to finetune with RL than their smaller LSTM counterparts, with the increased cost of gradient updates dealing a large blow to sample-hungry RL under the compute-time cap. As a result, while we do find interaction to improve the generalization performance of both hierarchical and non-hierarchical transformer-LSTM policies, the performance of these policies on withheld NLE instances is superseded by that of hierarchical LSTMs in this case.\n\nEvaluation data supporting the above is aggregated in Table 2 of the paper.\n\nThere are several directions for future work that we find compelling:\n\n(1) Deeper explorations of the impacts of structured labels, both in offline and online settings. Do performance improvements grow as the granularity of hierarchical priors used in pre-training increases? Can we further exploit hierarchical behavioral priors for structured exploration?\n\n(2) Investigation of even better transformer-based policies. If we are able to eliminate the overfitting effects seen in pre-training, does these models’ performance benefit from structured labels too?\n\n\n> When introducing Transformer to augment the capacity of the neural model, why did authors choose the architecture as shown in the article?\n\nPlease refer to **section (ii)** of our general rebuttal. As stated above, we will amend our submission to include a discussion of the LSTM ablation upon revision.\n\n> Why do the authors enforce the 48 hour training time cap instead of training all models till convergence? \n\nConstrained model comparisons are common in the computer vision and reinforcement learning literature [3, 4, 5].\n\nWe choose to employ a 48-hour training time cap in our study for two reasons.\n\n(1) A compute-time cap **provides a basis for comparison of different neural policy architectures on equal footing**, revealing, for instance, the degree of the performance advantage that faster gradient updates grant LSTM models against bulkier transformer-LSTMs, both in the presence and absence of structured labels. As noted by reviewer 83YB, our finding that in the “...time-constrained setting (true of many decision-making problems), model structure may be more important than model capacity seems likely to at least spark interesting conversations within the community.” Additionally, we believe that a compute-time cap increases the broader impact of our work by making our insights more relevant to researchers constrained by cost, e.g. in academic research settings.\n\n(2) From a practical perspective, employing a training time cap as our comparison basis also **enables us to expand the number of random seeds we employ to report results** across the numerous experiments within the paper, strengthening the statistical significance of our findings.\n\nThank you once more for the time you have taken to review our paper. Please let us know if you have any outstanding comments, questions, or concerns about our work.\n\n\n\n\n**Citations**\n\n\n[1] Hambro, Eric, et al. \"Insights from the neurips 2021 nethack challenge.\" NeurIPS 2021 Competitions and Demonstrations Track. PMLR, 2022.\n\n[2] Team, DeepMind Interactive Agents, et al. \"Creating multimodal interactive agents with imitation and self-supervised learning.\" arXiv preprint arXiv:2112.03763 (2021).\n\n\n[3] Zhai, Xiaohua, et al. \"Scaling vision transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n\n[4] Yarats, Denis, et al. \"Mastering visual continuous control: Improved data-augmented reinforcement learning.\" arXiv preprint arXiv:2107.09645 (2021).\n\n\n[5] Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" arXiv preprint arXiv:2212.09748 (2022).\n"}, {"review_id": "tg1sOXFXjz", "reviewer": "Reviewer_83YB", "summary": "This paper seeks to study and better understand the large performance gap between neural and symbolic agents in the NeurIPS 2021 NetHack Challenge. The main hypothesis is that symbolic agents advantage derives from hierarchical reasoning, which was not an element in participating neural agents. To test this hypothesis, a new dataset is generated using the winning symbolic agent that records both actions and higher-level strategic labels and a neural behavior cloning agent was trained with this augmented data. Beyond this, the paper also explores the impact of increased model size and dataset size, changes to the neural architecture, and the addition of a policy fine-tuning step using a reinforcement learning algorithm. The results suggest that hierarchical training improves the neural agent's performance significantly more than increased model capacity but that more powerful model architectures (i.e. a transformer-based model) could overfit to the augmented data.", "questions": "n/a", "limitations": "Since the paper does not propose significantly new algorithmic ideas, the main source of limitations would be in the methodology and analysis. I generally found the paper to avoid overclaiming. I've already discussed one area where this aspect of the paper could be improved: acknowledgement of the small sample sizes and proper statistical analysis to inform the conclusions. The other area might be in the conclusions where perhaps the summary of the findings might be a bit too general and could be toned down and/or stated clearly as hypotheses (e.g. \"Hierarchy hurts overfitting models\" is an overly broad conclusion from a limited set of experiments but seems like a reasonable hypothesis given these results).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Quality/Soundness\n\nThe hypotheses and claims of the paper were laid out clearly and the experiments were well-designed to evaluate them. \n\nClarity/Presentation\n\nI found the paper to be clearly presented and well-written. At each stage I had a clear understanding of the question under investigation and the methodology for studying it.\n\nOriginality/Contribution\n\nThe paper is explicit that its main contribution is not algorithmic but scientific. Since the scientific questions posed here are grounded in the performance of agents from a specific competition that happened last year, I expect that this analysis is original.\n\nSignificance/Contribution\n\nIncreasing understanding of the performance gap in the NetHack problem, as a proxy for complex, long-horizon problems in general, could be important. Symbolic approaches make use of quite a lot of domain expertise applied to constructing the symbolic structure, making them effective in their target problem but inflexible. Neural networks seem to be quite flexible and capable of learning without a lot of structure engineering, but don't seem to be able to take advantage of structure in the environment. It seems sensible to study the primary factors preventing neural approaches from building the same long-range structures.\n\nI think it is worthwhile to see the comparison between this structural change and the alternative interventions of more data, more parameters, or more sophisticated/expensive architecture. The finding that, in a time-constrained setting (true of many decision-making problems), model structure may be more important than model capacity seems likely to at least spark interesting conversations within the community. The fact that there is plenty of performance gap still to cover, even with this built-in domain knowledge may also inspire further investigation into what measures could come closer to closing the gap and how those insights might be applied to more general practice.\n\nOverall\n\nOverall, I find this to be a clearly written paper with a reasonable scientific question and sound methodology to address it (modulo some missing statistical analyses). The findings are not revolutionary but, to my eyes, they do provoke further questions about neural approaches to learning in complex, long-horizon problems and may inspire follow-up work either in the NetHack testbed specifically or in studying these questions more generally.", "weaknesses": "Quality/Soundness\n\nMy main concern in this area is the small number of independent samples per model class (6). I do understand that these results are generated at great expense and that it may not be feasible to generate more trials, but the small number of samples diminishes the statistical power of these analyses. I can see that the error bars are quite small; assuming those are showing the standard error, that's encouraging. Nevertheless, whether more samples can be generated or not, I think it's important that the paper include the findings of low-sample hypothesis testing (e.g. t-test) on these results; without that, we can't confidently distinguish between noise and meaningful differences.\n\nOriginality/Contribution\n\nThe paper acknowledges that behavior cloning, hierarchical policies, and transformer-represented policies have all been studied in prior work. \n\nSignificance/Contribution\n\nNetHack is not, in and of itself, an intrinsically important problem to solve. \n\nThe results presented here are not conclusive or enormously surprising. The main result is fairly predictable: adding explicit supervision about high-level strategy and explicit hierarchical structure in the model helps the model take advantage of hierarchical structure in the environment.\n\nOverall\n\nOverall, I find this to be a clearly written paper with a reasonable scientific question and sound methodology to address it (modulo some missing statistical analyses). The findings are not revolutionary but, to my eyes, they do provoke further questions about neural approaches to learning in complex, long-horizon problems and may inspire follow-up work either in the NetHack testbed specifically or in studying these questions more generally.\n\n---After discussion---\n\nI have considered the other reviews and the authors' responses. I continue to feel confident about my overall assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\n\nThank you for the time and effort that you have taken to review our submission. We are delighted that you found our paper to be well-written and the insights generated by our experiments to be valuable as a basis for future work on neural approaches to learning in complex, long-horizon environments.\n\n\nWe hope to address your remaining concerns and questions below.\n\n\n> My main concern in this area is the small number of independent samples per model class (6).\n\n\nThank you for pointing out our omission of a clarification on the meaning of the error bars plotted in the results figures of our draft submission; indeed, the error bars in Figures 3, 4, and 5 all indicate one standard error across policy random seeds. The absolute numerical values of evaluation metrics' standard errors are also reported in Table 2, which aggregates all of the experimental results of the paper. We will remedy this omission in the revised version of our submission.\n\n\nAs you note, due to the great computational expense associated with trials, we regret that we are unable to increase the number of independent trials run per model and experiment class in this paper. \n\n\nWe agree that a thorough verification of the statistical significance of our results is important. We will include the low-sample hypothesis tests that you suggest in the revised version of this paper. \n\n\nIn the interim, please note that the values of the key metric we employ to compare model performance on held-out seeds of NLE (mean score across a fixed set of 1024 unique, procedurally-generated NetHack games) are separated by two standard errors for all model classes in both our BC and APPO + BC sets of experiments.\n\n\n> NetHack is not, in and of itself, an intrinsically important problem to solve.\n\n\nWe concur that NetHack may not be an intrinsically important problem to solve. Nevertheless, we respectfully contend that this does not diminish the importance of work employing this environment as a general testbed for studying flexible policy learning in long horizon, open-ended environments. \n\n\nPlease also take a look at **section (i)** of our general rebuttal.\n\n\n> The main result is fairly predictable: adding explicit supervision about high-level strategy and explicit hierarchical structure in the model helps the model take advantage of hierarchical structure in the environment.\n\n\nTo our knowledge, our paper is the first to demonstrate that hierarchical labels can offer a stable and powerful mechanism for embedding behavioral priors into policies during pre-training.\n\n\nAs observed by reviewer qyRC, currently, there are few complex environments where expert demonstrations are accompanied by hierarchical labels, though such labels are in-principle no more difficult to collect than the demonstrations themselves from annotators.\n\n\nThough we agree that our findings concerning the power of hierarchy may not be surprising from a theoretical perspective, we are hopeful that our paper may spur the creation of more datasets with this property as well as further investigations of how hierarchical labels may be best exploited for offline and online learning of flexible neural policies, both in NetHack and beyond.\n\n\n> The other area might be in the conclusions where perhaps the summary of the findings might be a bit too general and could be toned down and/or stated clearly as hypotheses.\n\n\nThank you for these suggestions, we will tone down the language employed in this section of the paper accordingly.\n\n\nOnce again, we are sincerely grateful for your feedback on our submission. Please let us know if you have any outstanding comments, questions, or concerns preventing a stronger recommendation for acceptance.\n\n\n"}]

Author rebuttal:
We thank the reviewers for their constructive and insightful feedback. We are glad that you found our analysis to be comprehensive (qyRc), our experimental insights impactful (eWjQ, Ub8t, 83YB), and our submission to be well-written (Ub8t, qyRc, 83YB).

However, in this general response we would like to address two concerns raised by several reviewers: the applicability of this work outside of the NetHack Learning Environment (eWjQ, PtAe, qyRc) and transformer-only architectures (eWjQ, PtAe). 

**(i) On the limited applicability of this work outside of NLE**

While we understand the sentiment behind this statement, we respectfully disagree that our experimental insights are limited to NLE. Performance gaps between neural and symbolic methods are ubiquitous in open-ended environments including task and motion planning (TAMP) settings, and their underlying causes have received little study [1, 2].  Our study fills this gap in the neural policy learning literature by conducting a scientific investigation of the failure modes of popular neural approaches at mastering complex, generalizable behaviors directly from multi-modal data.


NLE is well-suited for such an investigation on account of: 
(1) the sheer extent of the performance gap between leading symbolic and neural policies [3, 4]; 
(2) the existence of previously-conducted benchmarks [3, 4]; and
(3) the open-source and hierarchical nature of the symbolic, state-of-the-art agent, AutoAscend [3].


In particular, we believe the last point makes NetHack a singularly compelling testbed over peer environments such as Habitat, MineCraft, or AI2-Thor. In generating the large-scale HiHack dataset, we provide the community with what is, to our knowledge, a unique opportunity to explore the impact of hard-coded hierarchical behavioral priors via structured labels on learning in long-horizon, complex environments.


Further, we take measures to preserve the generality of the insights yielded by our investigations of neural policy learning in this paper. Specifically, we intentionally forgo the addition of any environment-specific constraints in the architectural design or training setup of all models explored in this paper. This contrasts with leading NLE agents RAPH and KakaoBrain that rely on augmentations such as hand-engineered separated action spaces, role-specific training, and hard-coded sub-routines [3]. While this choice prevents us from achieving absolute SOTA in NLE in this paper, we believe it to be crucial in preserving the general applicability of our insights to neural policy learning for general open-ended, long-horizon environments. 


We agree with the reviewers that further probing of the nature of tradeoffs between hard-coded hierarchical priors, model capacity, and interaction in other environments is an important and exciting direction for future work, but we believe it to be beyond the scope of this paper.


**(ii) On transformer-only architectures**

The first iteration of transformer-based models we experimented with in our investigations were structured precisely as several of the reviewers described, featuring a “flat” transformer core module only (i.e. with the LSTM-based recurrent module ablated from the model architecture visualized in Figure 4 (left)), meaning all input is now confined to the transformer’s context window. We found such models to perform substantially worse when trained with behavioral cloning on AutoAscend demonstration data than their pure LSTM or transformer-LSTM counterparts. 


In the PDF attachment, please find visualizations of the flat transformer policies’ behavioral cloning rolling score [4] curves on withheld seeds of NLE over the course of training, contrasted against the corresponding curves for the transformer + LSTM and hierarchical transformer + LSTM policies (included in Figure 13 of our submission’s supplement). 


The parameter “unroll length,” denoted as “URL” in the legend of this figure, reflects the length of the context or observation history employed for action prediction.





A total of six random seeds for each model class were employed to test flat transformer policies. Training for each random seed was conducted on a single GPU for 48 hours. The transformer core modules of these policies have precisely the same architectural hyperparameters as their counterparts in the transformer-LSTMs, but featuring 6 layers.


We make two key observations: (1) the number of samples seen during the 48-hour training window is inversely proportional to the context length employed in training; (2) the rate of BC policy improvement does not increase with context length, across the set of context length values tested here. It is on this basis of (2), coupled with the fact that the average AutoAscend demonstration in our HiHack dataset consists of 27000 keypresses (Table 1 in the paper) – several orders of magnitude beyond the longest context length, URL = 256, we find feasible to test for flat transformers – that we conclude that an encoding of full game history is important for imitation in our setting.


We will update the supplemental section of our paper to include an in-depth discussion of these flat transformer ablation experiments upon revision.




**Citations**


[1] Silver, Tom, et al. "Learning symbolic operators for task and motion planning." 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.


[2] Zhang, Kai, et al. "Task and motion planning methods: applications and limitations." 19th International Conference on Informatics in Control, Automation and Robotics ICINCO 2022). SCITEPRESS-Science and Technology Publications, 2022.


[3] Hambro, Eric, et al. "Insights from the neurips 2021 nethack challenge." NeurIPS 2021 Competitions and Demonstrations Track. PMLR, 2022.


[4] Hambro, Eric, et al. "Dungeons and Data: A Large-Scale NetHack Dataset." Advances in Neural Information Processing Systems 35 (2022): 24864-24878.



Meta-review: After careful consideration, I recommend that the submission be accepted to NeurIPS 2023. The reviewers agree that the paper tackles an important problem and presents a novel and clever solution with thorough and insightful experimentation. I recommend the authors to include the changes suggested by the reviewers in the camera-ready version including experimentation with vanilla transformer and open-source code for the proposed method.
Decision: Accept (poster)