SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nevertheless, the qualitative performance of hierarchical policies clearly improves upon that of non-hierarchical models, with the hierarchical transformer-LSTM policy trained with BC both surviving longer of dungeon level 1 and descending with higher frequency than other BC-trained policy representatives. Furthermore, this qualitative behavior appears to be strengthened when interaction is added into the mix, with the mode centered on "dungeon level 2" increasing in density for the APPO + BC variant of the hierarchical transformer-LSTM representative policy.
Taken together, these sets of observations lead us to hypothesize that the quality of neural policies trained with imitation learning on extremely complex, long-horizon tasks like NetHack may be
NLE Max Dungeon Level Reached (†) vs. Total Turns (†) Across Neural Policy Classes
further improved with increases in the scale of hierarchically-informed, fixed behavioral factorization, beyond that explored in this paper. We consider this to be a very exciting direction for future work.
Log-Score Distribution (†) Across Neural Policy Classes
Figure 16: Log-score distribution across "in-depth evaluation" games. As in Figures 8 and 10, we indicate absolute median and mean values of policies' NLE scores in the "in-depth evaluation" with dashed-blue and solid-pink lines. The introduction of an RL loss reduces the "mass" associated with the left-tail of log-score and increases the "mass" associated with the right-tail across all neural policy classes, inducing right-ward shifts in median and mean scores, though difficult to perceive in this figure on account of the log-scale of the x-axis. We refer the reader to Table 2 in the main paper for aggregate absolute numerical values of NLE score. Top Left: Best-performing BC neural policy seeds. Bottom Left: Best-performing APPO + BC neural policy seeds. Right: AutoAscend.
Log-Score Distributions We conclude this supplementary analysis with a visualization of log-score distributions across neural policy classes in Figure 16, computed over the "in-depth evaluation" seeded games. The trends displayed in this figure align with those described and demonstrated previously. Improvements in model architecture as well as the introduction of hierarchy and interactive learning lead to a re-distribution of "mass" between left and right tails of distribution, with the overall counts of "low score" games decreasing and "high score" games increasing when these improvements are applied. The gap to AutoAscend is significantly reduced, but not bridged.