SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: transformer-LSTM model family architectural details. As in Table 5, the final three columns indicate the presence (or absence) of each component across the relevant policy classes, whether trained with BC or APPO + BC.
Class	Type	Module(s)	Hidden
Dim	Layers	Activ.	Copies	Policy Class
Trnsfrmr
+ LSTM	Trnsfrmr
+ LSTM
(large)	Hier
Trnsfrmr
+ LSTM
Enc	Message	MLP	128	2	ELU	-	✓	✓	✓
Enc	Blasts	Conv-1D, MLP	128	4	ELU	-	✓	✓	✓
Enc	Pixel Obs	Conv-2D, MLP	512	5	ELU	-	✓	✓	✓
Enc	Action Hist	one-hot	128	-	-	-	✓	✓	✓
Enc	Recurrent	LSTM (frozen)	512	1	-	-	✓	✓	✓
Core	Default	Trnsfrmr	1408	3	GeLU	-	✓	X	✓
Core	Large	Trnsfrmr	1408	6	GeLU	-	X	✓	X
Dec	Default	MLP	512	1	-	-	✓	✓	X
Dec	Hier Strat	MLP	512	1	-	-	X	X	✓
Dec	Hier Action	MLP	512	2	ELU	13	X	X	✓
Exponential Linear Unit (ELU) activation functions as per the original CDGPT model architecture [23], our transformer modules employ Gaussian Error Linear Unit (GeLU) activations [25]. Doubling the number of attention heads and the hidden dimension of transformer encoder layers in addition to the layer count of transformer-LSTM models resulted only in equivalent or additional policy performance degradation over that observed for the six layer model whose specifications are reported in Table 6; thus, we include only results for the latter here.
Context Length Models belonging to all policy classes from the LSTM family are trained by sequentially "unrolling" batched-predictions. The length of this "unrolled" sequence is held fixed throughout training, and is specified by the value of the "ttyrec unroll length" hyperparameter in Table 3, also referenced as  $c$  in the main body of this paper.
A fixed context length is also used to train the core transformer modules of policy classes from the transformer-LSTM family, similarly specified via the "ttyrec unroll length" hyperparameter. We found causally masking context in transformer attention mechanisms to be greatly beneficial towards improving the generalization capability of models. The pre-trained frozen LSTM "recurrent encoder" module of these networks provides a very cheap and simple means of dramatically extending the effective context length to cover full NetHack games (which may span hundreds of thousands of keypresses) without substantially slowing model training. As delineated in Section 5 and Appendix G, we find the recurrent module to be a crucial component of these models, with its ablation resulting in a substantial performance decline.
Hierarchical Policy Variants As alluded to in Tables 5 and 6, as well as in Section 4 of the main paper, all hierarchical policy variants are equipped with two sets of decoders: one high-level strategy decoder trained to predict the thirteen possible strategy labels in HiHack; as well as thirteen low-level action decoders, trained to predict actions corresponding to a single HiHack strategy across the 121-dimensional NLE action space [34]. Our hierarchical policies thus mimic the hierarchical structure of Autoascend, with an action decoder corresponding to each of the eleven, explicit strategies executed by the symbolic bot as well as two additional action decoders corresponding to the bot's "initialization" routine (an implicit twelfth strategy) and "write-errors," representing missing strategy labels $^{4}$ , respectively, as introduced in Figure 7. A full, diagrammatic illustration of the Hierarchical LSTM policy architecture is provided in Figure 3.
In all Hierarchical Behavioral Cloning (HBC) experiments, a BC loss was computed for the strategy decoder via ground-truth, batched HiHack strategy labels. A separate BC loss was computed for a single action decoder over batched HiHack action labels, with this action decoder "selected" in an end-to-end fashion by the strategy decoder. The action decoder "selection" procedure was executed across batches by sampling predicted strategy indices from the strategy decoder with Gumbel-Softmax re-parameterization [28], thus preserving gradient flow across the bi-level hierarchical structure of policies during training. An illustration of the full Hierarchical LSTM policy architecture is provided in Figure 3.
The strategy-specific BC loss component was re-weighted (via the "strategy loss coefficient" hyperparameter, introduced in Table 3) and recombined with low-level action decoder losses to produce a single, overall HBC loss.
We resolved the presence of ttyrec transitions with missing strategy values, represented via the "write-error" hierarchical label in HiHack, by extending the action-space of the hierarchical strategy decoder and adding an additional hierarchical action decoder copy corresponding to this class of labels. It is possible that the performance of hierarchical policy variants can be further improved by instead filtering out all transitions with this property. We leave this evaluation for future work.
Figure 10: Left: The distribution of starting roles across the large-scale "in-depth evaluation." As described above, this evaluation was run for the top-performing neural policy seeds (out of 6) across model class. We observe a near-uniform distribution of possible NLE roles across random seeds. Right: Autoascend NLE Score distribution vs. starting role in the "in-depth evaluation." This figure is a companion to the visualizations of neural policy NLE scores across role in Figure 6. As in Figure 8, we indicate absolute mean and median values of AutoAscend NLE score in the "in-depth evaluation" with dashed-blue and solid-pink lines.
F Evaluation Details
For all policy classes belonging to the LSTM model family, we observe monotonic improvements in the performance of models on withheld instances of NLE as a function of training samples; as a result, we employ the final training checkpoint of these policies when running evaluations across policy seeds. In contrast, the generalization capabilities of models in the transformer-LSTM family do not monotonically improve as a function of training samples, which we interpret as an indicator of overfitting. Thus, evaluations are conducted only for the "best" checkpoints corresponding to each policy seed, as evaluated on the basis of the rolling NLE score proxy metric. An in-depth description of this metric, as well as experiment training curves supporting the claims of over- and underfitting across model classes, can be found in Appendix G.
4Please refer to our earlier discussion in Appendix B for more details.
Two classes of evaluations are conducted in this paper for such checkpoints: a "standard evaluation" of policy NLE score across randomly sampled and withheld instances of environment, and an "in-depth evaluation," recording all metrics of game-play and employing precisely the same set of seeded environment instances to evaluate all policies.
The former policy evaluation procedure mirrors the one conducted during the NeurIPS 2021 NetHack Challenge Competition [23]. This is the procedure we employ to compute the mean and median NLE scores associated with policy seeds for all experiments in this paper as well as to compute the estimates of AutoAscend mean and median NLE score in Table 2, producing our core results.
The latter policy evaluation procedure yields a suite of more fine-grained metrics for informed and "human-like" game-play in NetHack, such as maximal dungeon level reached and the total life-time of the agent. We run this evaluation procedure for each of the best-performing<sup>5</sup> seeds from each neural policy class, as well as for AutoAscend. Metrics computed with this procedure are denoted via the  $(\dagger)$  symbol throughout the paper.
Standard Evaluation Policies are evaluated on a randomly seeded batch of 1024 (withheld) NLE games. Only the final NLE scores at the end of game-play are recorded.
In-Depth Evaluation Policies are evaluated across precisely the same seeded batch of 3402 (withheld) NLE games, i.e. all agents play precisely the same set of starting roles across the same NLE dungeon configurations, none of which are covered in HiHack. All games are recorded to the ttyrec data format, and can be streamed post-facto.
A visualization of the distribution of starting roles covered in this evaluation, as well as the corresponding AutoAscend score distributions (factorized by role across game instances), are shown in Figure 10.
Model Parameter Scaling: From 50.8M to 98.6M Parameter (Transformer + LSTM) Policies
Figure 11: Model Parameter Scaling experiment training curves. In each plot, solid lines reflect point-wise averages across 6 random seeds, while shaded regions reflect point-wise min-to-max value ranges across seeds. Left: Rolling NLE evaluation score vs total BC training samples, for "default" and  $2\mathrm{x}$  deeper transformer-LSTM policies. Right: BC loss vs total BC training samples.
Policy Architecture
G Training Curves
We provide training curves reflecting all conducted experiments. In Figures 11 and 12, we display both rolling NLE scores as well as BC loss curves as a function of training samples, across all model and data scaling experiments presented in Section 5. In Figure 13, we display aggregate rolling NLE scores as a function of training samples for all remaining BC and  $\mathrm{APPO} + \mathrm{BC}$  experiments, separated according to model family.
Rolling NLE Score The rolling NLE score metric introduced and displayed in the figures discussed here reflects an evaluation of policy performance on withheld NLE instances conducted continually during model training in a "rolling" fashion via a fixed number of CPU workers. As such, this metric is biased towards shorter-length games, with the value of smoothed rolling score as a function of
As indicated by overall mean NLE score in the "standard evaluation" procedure.
Dataset Scaling: From 10 to 100,000 AutoAscend Games
Policy Performance vs. Training Samples
(Loss vs. Training Samples)
Figure 12: Dataset scaling experiment training curves. All experiments for a given dataset size were trained on a dataset sub-sampled without replacement from HiHack. The sub-sampling procedure was seeded. Training hyperparameters and model architectures were identical across all runs belonging to a single policy class. As in Figure 12, solid lines reflect point-wise averages across 6 random seeds, while shaded regions reflect point-wise min-to-max value ranges across seeds in all plots. Top: Rolling NLE evaluation score vs total BC training samples, across dataset sizes for the "LSTM" and "transformer-LSTM" policy classes. Bottom: BC loss vs total BC training samples, across dataset sizes for the "LSTM" and "transformer-LSTM" policy classes.
Dataset Size (games)
training sample confounded by the policy-specific relationship between NLE score and total game turns. Rolling NLE score is thus not interchangeable with the large-batch "standard evaluations" employed elsewhere in this paper and presented in-depth in Appendix F. However, unlike BC loss, it serves as an efficient and useful (if noisy) proxy measure of policy generalization over the course of training.
Model Parameter Scaling Training Curves As shown in Figure 11, the generalization capability of transformer-LSTM policies (approximated via rolling NLE score) peaks soon after the start of training, decaying and flattening out as training proceeds despite continued improvement in BC loss. This observation supports the claim made in Section 7 that transformer based models overfit to HiHack.
Despite the aforementioned noisy nature of rolling NLE score, the "Policy Performance vs. Training Samples" curves on the left of Figure 11 allude to our large-scale "standard evaluation" finding from Section 5; namely, that policy performance does not increase when model parameter count is scaled up, even after training hyperparameters are tuned. Similarly, the "Loss vs. Training Samples" curves on the right of this figure indicate nearly identical training errors across both models as a function of BC training samples.
Dataset Scaling Training Curves In Figure 12, we note that for datasets consisting of, or exceeding, 1000 AutoAscend games, LSTM policies do not appear to overfit over the course of our BC experiments, with rolling NLE score consistently monotonically increasing as a function of BC training samples for all such policies. However, a positive relationship between dataset size and maximal rolling NLE score over training persists, indicating that the addition of more data does lead to measurable (if sub log-linear) improvements in policy generalization. An inspection of
LSTM policy loss curves suggests a similar story. Losses across policy seeds trained on 10 and 100 AutoAscend games drop swiftly to values near zero, supporting this overfitting hypothesis.
The transformer-LSTM policy loss curves in Figure 12 also reveal harsh overfitting for policies trained with 10 and 100 games. Interestingly, the generalization capability of these policies, again indicated by rolling NLE score over training, is vastly superior to that of their LSTM counterparts. Indeed, we observe that a transformer-LSTM policy trained on just 100 games vastly outperforms a pure LSTM policy trained on  $10\mathrm{x}$  as many games. We attribute this gap to the frozen nature of the pre-trained LSTM component of the transformer-LSTM policies employed as a recurrent encoder in these policies, and hypothesize that it is the static quality of the recurrent representation output by this encoder which bolsters the generalization capability of the resultant models in exceptionally low-data regimes.
Aggregate Policy Performance vs. Training Samples
(BC Policies)
(APPO + BC Policies)
Figure 13: Aggregate rolling NLE evaluation score curves for the central BC and APPO + BC experiments discussed in this paper. Model parameter and dataset scaling training curves are omitted here on account of being visualized in Figures 11 and 12, respectively. For APPO + BC experiments, batch accumulation was employed to ensure that the ratio of RL to BC samples "seen" during training was 1:1. This property is indicated via the secondary x-axes of APPO + BC figures here, which show RL sample quantities. Top: Rolling NLE evaluation score vs total BC training samples in pure BC experiments, across non-hierarchical and hierarchical LSTM and transformer-LSTM based policy classes. Bottom: Rolling NLE evaluation score vs total BC training samples in APPO + BC experiments, across non-hierarchical and hierarchical LSTM and transformer-LSTM based policy classes.
Aggregate BC and APPO + BC Training Curves The "Aggregate Policy Performance vs Training Samples" curves of Figure 13 align with the general model family training trends previously observed in the scaling experiments. Notably, we find once again that LSTM based policies' rolling NLE scores improve monotonically with training samples whether training is conducted with BC or APPO + BC, while this is not the case for transformer-LSTM based models. Indeed, the generalization properties' of policies belonging to this model family improve at the start of training before worsening as training proceeds across BC experiments. We interpret continued demonstration of these trends as further support for the claims of LSTM underfitting and transformer-LSTM overfitting made in Section 7 of the main body of the paper as well as in Appendix F.
Moreover, we note that the introduction of an RL loss induces a particularly large amount of volatility in the rolling NLE score associated with transformer-LSTM based models, disrupting the monotonically decreasing relationship between score and samples previously observed for these policies following  $\approx 2\cdot 10^{8}$  training samples in the pure BC experiments (i.e. the eventual overfitting behavior). This observation suggests that the performance of policies belonging to this model family is bottlenecked by an insufficient total throughput of "on-policy" or interactive data.
Flat Transformer and Transformer + LSTM Policy Performance vs. Training Samples
Figure 14: Aggregate rolling NLE evaluation score curves for flat transformer policies vs transformer-LSTM and hierarchical transformer-LSTM policies during BC training. Flat transformer policies have similar architecture to transformer-LSTMs with two exceptions: (1) the pretrained, frozen LSTM component of the latter is ablated from the former; and (2) the core modules of tested flat transformer policies consist of twice as many (i.e. 6) transformer encoder layers. We perform a sweep over values of the context-length parameter  $c$ , corresponding to the number of consecutive timesteps included in the flat transformer context length per sample during training with BC. A total of six random seeds for each model class were employed to test flat transformer policies. Training for each random seed was conducted on a single GPU for 48 hours.
Ablating the LSTM Component of transformer-LSTM Policies To establish and confirm the importance of the recurrent LSTM component of the transformer-LSTM policies introduced in this paper, we perform a set of ablation experiments with policies featuring a transformer core module only, meaning all input is confined strictly to the transformer's context window at each iteration of training and inference. We refer to models with such an architecture as flat transformers. As demonstrated in Figure 14, we find flat transformers to perform substantially worse in evaluation when trained with behavioral cloning on AutoAscend demonstration data than either their pure LSTM or transformer-LSTM counterparts. The context-length parameter, denoted as  $c$  in the legend of Figure 14, reflects the number of contiguous observations employed for action prediction. We perform a small sweep over values of this parameter in an effort to establish the relationship between observation history and model generalization. Reported results reflect the best performing flat transformer architecture after conducting hyperparameter tuning across transformer encoder layer size, layer count, and attention head count.
We make two key observations: (1) the number of samples seen during the 48-hour training window is inversely proportional to the context length employed in training; (2) the rate of BC policy improvement does not increase with context length, across the set of context length values tested here. It is on this basis of (2), coupled with the fact that the average AutoAscend demonstration in our HiHack dataset consists of 27000 key presses (Table 1), several orders of magnitude beyond the longest context length,  $c = 256$ , we find feasible to test for flat transformers, that we hypothesize that an encoding of full game history is important for imitation in our setting.
H Low-Sample Hypothesis Testing
To establish the statistical significance of the core BC and APPO + BC empirical results reported in the main body of this paper, we perform a series of two sample t-tests with a decision threshold of  $\alpha = 0.05$ . The results of these t-tests are summarized in Tables 7, 8, and 9. We confirm all reported findings.
Table 7: Statistical significance of policy comparisons across BC experiments: two-sample t-tests on mean policy NLE scores  $\overline{s}$  in evaluation, across six random seeds (df = 5). We compute p-values associated with three different null hypotheses, testing for improvement in performance/generalization over the baseline LSTM (i.e. CDGPT5) [23] and hierarchical LSTM policies, as well as a decline in performance/generalization over the 3-layer non-hierarchical transformer-LSTM for models trained with BC.
Hierarchy	H0: s? ≤ sLSTM	H0: s? ≤ sHier-LSTM	H0: s? ≥ sTransfmr-LSTM
p-value	reject H0	p-value	reject H0	p-value	reject H0
LSTM (XXL dec)	X	.320265	X	-	-	-	-
LSTM	✓	.000047	✓	-	-	-	-
Transformer-LSTM	X	< .00001	✓	.000125	✓	-	-
Transformer-LSTM (large)	X	.000014	✓	.001646	✓	.001792	✓
Transformer-LSTM	✓	.000063	✓	.002549	✓	.016935	✓
Table 8: Statistical significance of policy comparisons across APPO + BC experiments: two-sample t-tests on mean policy NLE scores  $\overline{s}$  in evaluation, across six random seeds (df = 5). As in Table 7, we compute p-values associated with three different null hypotheses, but across model classes trained with APPO + BC.
Hierarchy	H0: s? ≤ sLSTM	H0: s? ≤ sHier-LSTM	H0: s? ≥ sTransfmr-LSTM
p-value	Reject H0	p-value	Reject H0	p-value	Reject H0
LSTM (XXL dec)	X	.470427	X	-	-	-	-
LSTM	✓	.001397	✓	-	-	-	-
Transformer-LSTM	X	.042454	✓	.348223	X	-	-
Transformer-LSTM	✓	.026776	✓	.362569	X	.102058	X
Table 9: Statistical significance of gains in policy performance from APPO + BC vs pure BC: two-sample t-tests on mean policy NLE score  $\overline{s}$  in evaluation, across six random seeds (df = 5). We compute p-values associated with the null hypothesis that policy performance yielded by APPO + BC training is less than or equal to that yielded by pure BC. We reject this null hypothesis for all but non-hierarchical transformer-LSTM models, for which we see no statistically significant improvement in policy generalization with RL fine-tuning.
Hierarchy	H0: s?(APPO + BC) ≤ s?(BC)
p-value	Reject H0
LSTM [23]	X	.000121	✓
LSTM (XXL dec)	X	.000017	✓
LSTM	✓	.000066	✓
Transformer-LSTM	X	.416954	X
Transformer-LSTM	✓	.003269	✓
I Distributional Visualizations of Evaluation Results
Max Dungeon Level Reached vs. Total Turns In Figure 15, we supplement the absolute mean and median max-dungeon level and agent life-time (in game turns) statistics introduced in Table 2 with 2-D distributional visualizations of both the raw values of these metrics as well as their mutual inter-relationship, evaluated via  $n = 3402$  seeded NLE games run individually for all policy class representatives in our "in-depth evaluation."
The AutoAscend 2-D contour plot on the right of this figure reveals an interesting emergent property of the bot's behavior, which is confirmed by study of individual games: the high-level "descent behavior" of AutoAscend appears to fall along one of two modes. In the first of these modes, the bot spends a very large amount of turns on dungeon level 1, and avoids descending further into the dungeon before the end of the game. In the second mode, the bot begins to rapidly descend fairly deep into the dungeon, reaching as far as level 11 in the sample of game seeds tested here. The overall
(APPO + BC Policies)
Figure 15: Max dungeon level reached vs. total turns across "in-depth evaluation" games, visualized via 2-D contour density plots. Contour densities are indicated by the color-bars accompanying each subplot. Mean quantity values (computed dimension-wise for all policies) across the "in-depth evaluation" batch are indicated by white 'X' symbols in each subplot. The symbolic bot's vastly superior ability to play longer and descend much further into the dungeon creates a separation in scale between it and its neural counterparts; as a result, for clarity, we indicate the max dungeon level vs turns subspace displayed in the neural policy contours with a bolded black rectangle in our visualization of AutoAscend's behavior. Top Left: Best-performing BC neural policy seeds. Bottom Left: Best-performing APPO + BC neural policy seeds. Right: AutoAscend.
relationship between AutoAscend's total in-game life-time and max-dungeon level reached appears to be roughly quadratic.
In contrast, none of the evaluated neural policy class representatives comes close to achieving in-game progress resembling AutoAscend's secondary behavior mode. A large majority of games for all neural policy classes appear to end on the first level of the dungeon, with policies very rarely surviving as long on this level as AutoAscend. This suggests that neural policies may be failing to master the long-range, very low-level behaviors of the bot, even when these behaviors are factorized across AutoAscend strategies, as in the case of hierarchical policy variants.