SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: Left: Model capacity versus mean score for our transformer-LSTM model. The larger model performs worse. Error bars indicate standard error. Right: Dataset scaling experiments showing diminishing returns in mean NLE score  $s$  achieved during training with BC as the number of training games  $g$  reaches  $10^{5}$ . We employ non-linear least-squares to estimate the power law coefficients relating log-game count and mean policy score outside of the low-data regime ( $g > 10^{2}$ ). Curves of best fit are displayed. Errors bars indicate mean score ranges across seeds. Collectively these two plots show that scaling of data and model size are not sufficient to close the performance gap to symbolic models.
We find the LSTM recurrent module of our transformer-LSTM models to be a crucial component of the architecture. When this module is ablated to yield a flat transformer, we observe a severe decline in policy performance (Figure 4(right) and Appendix G, Figure 14).
Probing further, in Figure 5(left), we compare the performance of two variants of our transformer-LSTM model, one with 3 layers (50.8M parameters) and another "scaled-up" model with 6 layers (98.9M parameters). The larger model can be seen to perform worse in evaluation than the smaller one, suggesting over-fitting and an inverse relationship between parameter count and generalization in NetHack for this model class (see also Appendix G, Figure 12). This finding indicates that scaling of model capacity alone may not be sufficient to close the neural-symbolic gap.
In Figure 5(right), we now explore the effect of training set size on mean NLE test score. We perform BC training for the LSTM baseline [23] and our largest 6 layer transformer-LSTM model (98.9M params) for 10 up to 100,000 games, subsampled from the HiHack Dataset. For both architectures, we observe a sub log-linear dependence on training set size, asymptoting at a mean score of approximately 1000. Thus, brute force scaling of the dataset alone cannot viably close the gap to symbolic methods (score of 8500).
Though our architecture and data scaling experiments are compute-time constrained, we find the test-time performance of all tested models to saturate on the given computational budget (Appendix G, Figures 11 and 12).
Table 2: Evaluating the impact of hierarchical labels and architectural improvement on the performance of policies trained both with behavioral cloning, as well as with combined behavioral cloning and asynchronous proximal policy optimization. All policies were trained for 48 hours on a single GPU. Metrics annotated with  $(\dagger)$  were computed only for the top-scoring neural policy seed (out of 6) across each model class.
Hierarchy	Score	Dlvl (†)	Turns (†)
Mean	Median	Mean	Mean	Median
BC	LSTM [23]	X	658 ± 41	403	1.11 ± 0.01	5351 ± 76	4111
BC	LSTM	✓	931 ± 42	614	1.09 ± 0.01	6983 ± 84	5981
BC	Transformer-LSTM	X	1318 ± 38	914	1.36 ± 0.01	6088 ± 75	5121
BC	Transformer-LSTM	✓	1151 ± 43	731	1.26 ± 0.01	7568 ± 99	6242
APPO + BC	LSTM [23]	X	1204 ± 138	779	1.07 ± 0.01	8712 ± 112	7376
APPO + BC	LSTM	✓	1551 ± 73	972	1.09 ± 0.01	11435 ± 134	9849
APPO + BC	Transformer-LSTM	X	1326 ± 28	887	1.25 ± 0.01	7924 ± 99	6788
APPO + BC	Transformer-LSTM	✓	1346 ± 16	894	1.32 ± 0.01	7874 ± 101	6769
Symbolic	AutoAscend	✓	8556 ± 187	4918	3.10 ± 0.04	19586 ± 171	19710
6 Combining Imitation with Reinforcement Learning
Given that hierarchy and scaling are insufficient to bridge the performance gap with AutoAscend, we now explore the impact of incorporating an online learning component to neural policy training, via RL.
Method In this set of experiments, we build on results from Hambro et al. [24], which suggested that in the presence of action-labeled data, behavioral cloning coupled with reinforcement learning is superior to RL training from scratch in NetHack. As in Hambro et al. [24], we employ the asynchronous moolib distributed-RL library to train our models with a combination of BC and asynchronous proximal policy optimization (APPO) [39, 46, 51]. At each time-step of training, the overall loss used to perform model parameter updates is a weighted combination of BC and RL losses, i.e. the cross-entropy loss of a batch of demonstrations from HiHack plus an RL loss over a batch of rollouts of the current policy in NLE. For hierarchical models, we use RL only to update the low-level strategies; that is, the strategy selection policy is trained with HBC alone and is not updated with RL.
Figure 6: Aggregate NLE score breakdown versus player role. Our model refinements (hierarchy, transformer, RL fine-tuning) show gains over the LSTM based CDGPT5 baseline [23] across all roles. As in Table 2, we employ  $(\dagger)$  to confer that these score distributions were computed only for the top-performing neural policy seed (out of 6) across each model class.
Training and evaluation details Our high-level training procedure here mirrors that of our hierarchical behavioral cloning experiments: we evaluate model performance under the constraint of computation time, training all policies for exactly 48 hours on a single GPU, using 6 random seeds to randomize data loading and environment seeding only. We test the effect of warm-starting from a checkpoint pre-trained with BC or HBC alone (via the procedure delineated in section 4) for all model classes. We report  $\mathrm{APPO} + \mathrm{BC}$  results with warm-starting for all model classes where we find it to be beneficial in improving stability and/or evaluation policy performance.
Results Table 2 summarizes the performance of all our models. A number of observations can be made: (a) In the offline setting, our best performing model (non-hierarchical transformer-LSTM) outperforms the vanilla CDGPT5 model baseline by  $100\%$  in mean NLE score and  $127\%$  in median NLE score; (b) RL fine-tuning offers a clear and significant performance boost to all models, with gains in the test-time mean NLE score associated with all model classes; (c) The overall best performing approach is APPO + BC using the hierarchical LSTM model. The mean score of 1551 represents a new state-of-the-art for neural policies in NLE, beating the performance of the vanilla CDGPT5 model when trained with APPO + BC by  $29\%$  in mean NLE score and  $25\%$  in median NLE score; (d) The transformer-LSTM models, which are slower and more unstable to train with APPO + BC than LSTM models (see training curves in Appendix G), see a smaller margin of improvement during RL fine-tuning; (e) Other metrics commonly employed to evaluate meaningful NetHack gameplay in previous work, such as dungeon level reached and lifetime of the agent in game turns [34, 23], show a broadly similar pattern to the chief median and mean score metrics employed in the NetHack Challenge [23]; and (f) Lastly, for BC, hierarchy seems to hurt performance for the larger Transformer-RL models, though this gap is closed once APPO fine-tuning is applied. See Appendix H for low-sample hypothesis tests validating the statistical significance of these findings.
In NetHack, the player can choose from 13 distinct roles (barbarian, monk, wizard, etc.), each of which require distinctive play-styles. In NLE, starting roles are randomized, by default. Figure 6 shows a score distribution breakdown across role for different neural policy classes trained with BC and APPO + BC. In general, we observe that fine-tuning with RL improves the error-correction capability of models of all classes (as indicated by positive shifts in NLE score distributions) over their purely offline counterparts.
In this work, we have developed a new technique for training NetHack agents that improves upon prior state-of-the-art neural models by  $127\%$  in offline settings and  $25\%$  in online settings, as evaluated by median NLE score. We achieve this by first creating a new dataset, the HiHack Dataset (HiHack), by accessing the best symbolic agent for NetHack. This dataset, combined with new architectures, allows us to build the strongest purely data-driven agent for NetHack as of the writing of this paper.
NLE Score Distribution  $(\dagger)$  vs. Starting Role Across Neural Policy Classes
More importantly, we analyze several directions to improve performance, including the importance of hierarchy, the role of large transformer models, and the boosts that RL could provide. Our findings are multifaceted and provide valuable insights for future progress in training neural agents in open-ended environments and potentially bridging the gap to symbolic methods, as well as to the very best human players.
- Hierarchy can improve underfitting models. Prior LSTM based models severely underfit on the HiHack Dataset (see training curves in Appendix G). The addition of hierarchy improves such models, whether trained offline or online (Figure 3(right) and Table 2).
- Hierarchy can hurt overfitting models. Transformer based models are able to overfit on the HiHack Dataset (see training curves in Appendix G). Hierarchy hurts this class of models when trained offline at test-time (Figure 4(right) and Table 2).
- Reinforcement learning provides larger improvements on light, underfitting models. We obtain little and no improvement with RL fine-tuning on our hierarchical transformer-LSTM and non-hierarchical transformer-LSTM models, respectively. However, the underfit LSTM models enjoy significant gains with RL, ultimately outperforming transformer based models under a compute-time constraint (Table 2).
- Scale alone is not enough. Our studies on increasing both model and dataset size show sub-log-linear scaling laws (Figure 5). The shallow slope of the data scaling laws we observe suggests that matching the performance of AutoAscend with imitation learning (via games played by bot) will require more than just scaling up demonstration count.
Possible avenues for future exploration include: (a) alternate methods for increasing the transformer context length to give the agent a more effective long-term memory; (b) explicitly addressing the multi-modal nature of the demonstration data (i.e. different trajectories can lead to the same reward), which is a potential confounder for BC methods. Some forms of distributional BC (e.g. GAIL [26], BeT [52]) might help alleviate this issue.
Finally, we hope that the hierarchical dataset that we created in this paper may be of value to the community in exploring the importance of hierarchy and goal-conditioning in long-horizon, open-ended environments.
We would like to thank Alexander N. Wang, David Brandfonbrener, Nikhil Bhattasali, and Ben Evans for helpful discussions and feedback. Ulyana Piterbarg is supported by the National Science Foundation Graduate Research Fellowship Program (NSF GRFP). This work was also supported by grants from Honda and Meta as well as ONR awards N00014-21-1-2758 and N00014-22-1-2773.
[1] J. Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, S. Clark, A. Dudzik, P. Georgiev, A. Guy, T. Harley, F. Hill, A. Hung, Z. Kenton, J. Landon, T. P. Lillicrap, K. W. Mathewson, A. Muldal, A. Santoro, N. Savinov, V. Varma, G. Wayne, N. Wong, C. Yan, and R. Zhu. Imitating interactive intelligence. arXiv, abs/2012.05672, 2020. 3
[2] J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches, 2016. 3
[3] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009. 3
[4] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017. 3
[5] A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1-2):41-77, 2003. 3
[6] A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Survey: Robot programming by demonstration. Handbook of robotics, 59(BOOK_CHAP), 2008. 3
[60] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, pages 5628-5635. IEEE, 2018. 3
[61] Y. Zhong, E. Hambro, E. Grefenstette, T. Park, M. Azar, and M. G. Bellemare. Improving policy learning via language dynamics distillation. In Advances in Neural Information Processing Systems, 2022. 3
[62] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564, 2018. 3
A NetHack Learning Environment
Both NetHack and the NetHack Learning Environment (NLE) feature a complex and rich observation space. The full observation space of NLE consists of many distinct (but redundant) components: glyphs, chars, colors, specials, blstats, message, inv_glyphs, inv_strs, inv_oclasses, screenDescriptions, ttychars, ttycolors, and tty_cursor [34].
The HiHack Dataset (HiHack), as well as all RL experiments in NLE conducted in this paper, consist of and rely solely upon the tty* view of the game.
B Details on AutoAscend
Figure 7: Full control flow structure of AutoAscend, separated across explicit strategy and substrategy routines. There are 13 possible "strategy" or hierarchical labels in HiHack, 11 representing the explicit strategies employed by the bot (in magenta) and two additional labels to handle extra-hierarchical behavior (in light-grey).
We include a comprehensive visualization of the full internal structure of AutoAscend in Figure 7. As indicated in the summary visualization of high-level AutoAscend strategies provided in Figure 2 of the main paper, the bot features 11 explicit, hard-coded strategy routines. These interface with other low-level sub-strategy routines, some which are re-used by multiple strategies or even multiple sub-strategies. One example of such a subroutine is the "arrange items" sub-strategy, which is called both by "follow guard" and by "check altar," which is itself a subroutine of the "explore" sub-strategy. When factorized across strategies and sub-strategies, the full structure of AutoAscend is a directed acyclic graph (DAG) with a maximal depth of 5 from the "root," i.e. the AutoAscend global controller "node" indicated in dark gray above, which re-directs global behavioral flow across strategies via a predicate-matching scheme.
The HiHack Dataset includes a hierarchical strategy label for each timestep of AutoAscend interaction. As a result, alongside the 11 explicit strategies of the bot, there are two additional labels present in the dataset, which account for extra-hierarchical behavior in ttyrec game records yielded by the augmented ttyrec writer employed for HiHack generation and loading. These are visualized in light gray in Figure 7. The first of these corresponds to the hard-coded initialization routine employed by AutoAscend, effectively serving as a twelfth (albeit implicit) strategy, while the second covers ttyrec timestep records with missing strategy values. Missing strategy values may reflect ttyrec writer errors, or advancement of the underlying NetHack state by NLE rather than by agent, which occurs e.g., during NLE's timeout based termination of games [34]. Empirically, "write-error" strategy labels occur with very low frequency in HiHack, representing less than  $\approx 0.05\%$  of all transitions.
NLD-AA  $(n = 109,545)$  vs. HiHack  $(n = 109,907)$
Figure 8: Distributional comparisons of basic AutoAscend game statistics in NLD-AA vs HiHack. Median and mean values (as reported in Table 1) are indicated respectively by vertical dashed-blue and solid-pink lines in each figure. Top: Log-score vs game counts in NLD-AA and HiHack. Bottom: Log-turns vs game counts in NLD-AA and HiHack.
C Details on HiHack
Generation All games in the HiHack dataset were recorded by running an augmented version of AutoAscend in NLE v 0.9.0. This augmented version of the AutoAscend source features the introduction of only a dozen extra lines of code that enable step-wise logging of the strategy trace behind each action executed by the bot. This strategy trace is recorded directly to game ttyrecs at each timestep via the addition of an extra channel to the C based ttyrec-writer in the NetHack source code. Each game was generated via a unique NLE environment seed.
Game Statistics The comparison of the full log-score and log-turns distributions across NLD-AA and HiHack made in Figure 8 further supports the claim of high correspondence between the datasets made in Section 3.3. Figure 9 shows the distribution of strategies across a sample consisting of  $\approx 10^8$  unique game transitions from HiHack. We observe coverage of all but the least frequent explicit strategy executed by AutoAscend: the "solve sukoban" routine, employed a means to gain yet more experience exclusively in highly advanced game states.
Figure 9: A visualization of the distribution of strategies across a sample of 4300 HiHack games, containing a total of  $\approx 10^8$  transitions.
Table 3: Training hyperparameter configurations across all BC and APPO + BC experiments. Hyperparameters are listed in alphabetical order. We employ  $(\ddagger)$  to indicate hyperparameters only relevant for the corresponding hierarchical policy variants. The presence of the symbol  $-$  in lieu of a parameter value reflects the parameter's irrelevance for offline, BC experiments. All bolded hyperparameters were tuned. After tuning was complete, precisely the same sets of hyperparameters were employed to train models across individual policy classes belonging to the LSTM and transformer-LSTM based model families explored in this paper, across all BC and APPO + BC experiments. Note that the abbreviation 'CE' denotes the cross-entropy loss function.
Hyperparameter	BC	APPO + BC
LSTM	transformer-LSTM	LSTM	transformer-LSTM
actor batch size	-	-	512	256
adam beta1	0.9	0.9	0.9	0.9
adam beta2	0.999	0.999	0.999	0.999
adam eps	1.00E-07	1.00E-07	1.00E-07	1.00E-07
adam learning rate	0.0001	0.0002	0.0001	0.0001
appo clip baseline	1	1	1	1
appo clip policy	0.1	0.1	0.1	0.1
baseline cost	1	1	1	1
crop dim	18	18	18	18
discount factor	-	-	0.999	0.999
entropy cost	-	-	0.001	0.001
env max episode steps	-	-	100000	100000
env name	-	-	challenge constant	challenge constant
fn penalty step	-	-
grad norm clipping	4	1	4	1
inference unroll length	-	-	1	1
loss function	CE	CE	CE	CE
normalize advantages	-	-	✓	✓
normalize reward	-	-	×	×
num actor batches	-	-	2	2
num actor cpus	-	-	10	10
penalty step	-	-	0	0
penalty time	-	-	0	0
pixel size	6	6	6	6
reward clip	-	-	10	10
reward scale	-	-	1	1
RL loss coeff	-	-	1	0.001
strategy loss coeff (‡)	1	1	1	1
supervised loss coeff	1	1	0.001	1
ttyrec batch size	512	512	256	256
ttyrec cores	12	12	12	12
ttyrec envpool size	4	6	4	3
ttyrec unroll length	32	64	32	64
use prev action	×	✓	✓	✓
virtual batch size	512	1024	512	512
D Training Details
Hyperparameters All relevant training hyperparameter values, across model families as well as BC vs APPO + BC experiment variants, are displayed in Table 3.
To kick-start all experiments, we employed the training hyperparameter values reported in Hambro et al. [24], used to train the CDGPT5, LSTM based baseline across experiments. For several hyperparameters, however, additional tuning was conducted. These hyperparameters are indicated in bold in Table 3. Tuning across these hyperparameters was performed once for the "default" representative policy class from each of the LSTM and transformer-LSTM model families for all but the Model Parameter Scaling experiments<sup>3</sup>. After tuning was complete, hyperparameter configurations were
Prior to the start of these experiments, additional tuning of the "adam learning rate" and "ttyrec batch size" hyperparameters was conducted for the transformer-LSTM (large) policy class. However, across the
fixed across all proceeding offline and combined offline + online experiments. Specifications of hyperparameter values swept over during tuning are provided in Table 4.
All models were trained with the Adam optimizer [31] and a fixed learning rate. We experimented with the introduction of a learning rate schedule for transformer-LSTM models, but we found no additional improvements in policy prediction error or evaluation performance at the conclusion of training to be yielded by such a schedule.
Random Seeds For each of the Hierarchical Behavioral Cloning, Model Parameter Scaling, Data Scaling, and combined Imitation and Reinforcement Learning experiments described in Sections 4, 5, and 6 of this paper, a total of 6 random seeds were run across all relevant policy classes. Randomized quantities included: policy parameter values at initialization, data loading and batching order, HiHack Dataset subsampling (in data scaling experiments only), and initial environment seeding (in APPO + BC experiments only).
Training Infrastructure and Compute The RPC based moolib library for distributed, asynchronous machine learning was employed across all experiments [39]. All data loading and batching was parallelized. Our model training code builds heavily upon the code open-sourced by Hambro et al. [24].
Experiments were run on compute nodes on a private high-performance computing (HPC) cluster equipped either with a NVIDIA RTX-8000 or NVIDIA A100 GPU, as well as 16 CPU cores. All policies were trained for a total of 48 hours. We detected no substantial differences in training frames-per-second (FPS) rates for both offline and online experiments across compute nodes in "speed-run" tests, provided nodes were under no external load. When running experiments, we did detect some variance in total optimization steps completed under the 48-hour constrained computational budget across seeds belonging to single policy classes, which we attribute to variance in external HPC cluster load during these runs.
Table 4: Training hyperparameter tuning sweeps. We specify the hyperparameter values tested during tuning sweeps conducted for the "default" representatives of each model class. Full specifications of final hyperparameter values employed in experiments are included in Table 3.
Sweep Range
adam learning rate	{0.0001, 0.0002, 0.0005, 0.01}
discount factor	{0.9, 0.99, 0.999, 0.9999}
grad norm clipping	{0.1, 1, 4}
RL loss coeff	{0.001, 0.01, 1}
strategy loss coeff	{0.001, 0.01, 1, 10}
supervised loss coeff	{0.001, 0.01, 1}
ttyrec envpool size	{3, 4, 6}
ttyrec unroll length	{16, 32, 64, 128}
virtual batch size	{128, 256, 512, 1024}
E Model Architectures
A description of all model components and policy architectures is given in Tables 5 and 6, separated across LSTM and transformer-LSTM model families. The PyTorch library was used for to specify all models, loss functions, and optimizers [43].
Additional Transformer Specifications All transformer modules tested in this paper consist purely of "Transformer-Encoder" layers. Each layer is configured with 16 attention heads per attention mechanism, and layer normalization is applied prior to all attention and feed-forward operations. A dropout of 0.1 is used during training [57]. Unlike the rest of the modules we employ, which use
set of values tested, the same values were found to be optimal for this model configuration as for the default transformer-LSTM policy; hence, only a single set of hyperparameter values for the model family is reported here.
Table 5: LSTM model family architectural details. The final three columns indicate the presence (or absence) of each component across the relevant policy classes, whether trained with BC or APPO + BC.
Class	Type	Module(s)	Hidden
Dim	Layers	Activ.	Copies	Policy Class
LSTM	LSTM +
XXL dec	Hier
LSTM
Enc	Message	MLP	128	2	ELU	-	✓	✓	✓
Enc	Blasts	Conv-1D, MLP	128	4	ELU	-	✓	✓	✓
Enc	Pixel Obs	Conv-2D, MLP	512	5	ELU	-	✓	✓	✓
Enc	Action Hist	one-hot	128	-	-	✓	✓	✓
Core	-	LSTM	512	1	-	-	✓	✓	✓
Dec	Default	MLP	512	1	-	-	✓	X	X
Dec	XXL	MLP	1024	2	ELU	-	X	✓	X
Dec	Hier Strat	MLP	128	1	-	-	X	X	✓
Dec	Hier Action	MLP	256	2	ELU	13	X	X	✓