SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also utilize the previous achievement for the achievement prediction task in Section 4.1. Given a state-action pair  $(s_t, a_t)$  and its corresponding previous achievement  $g_t^-$ , we concatenate the latent state representation  $\phi_\theta(s_t)$  from the encoder with  $a_t$  and  $\nu_\theta(g_t^-)$ . The resulting vector is then fed into an MLP layer and normalized to obtain the representation  $\psi_\theta(s_t, a_t, g_t^-)$  for the next achievement prediction. This is interpreted as learning forward dynamics in the achievement space.
4.4 Integration with RL training
We integrate the contrastive learning method proposed in Sections 4.1 to 4.3 with PPO training by introducing two alternating phases, the policy and auxiliary phases. During the policy phase, which is repeated several times, we update the policy and value networks using newly-collected episodes and store them in a buffer. During the auxiliary phase, we update the encoder to optimize the contrastive objectives in conjunction with the policy and value regularizers using all episodes in the buffer. We call this auxiliary learning achievement distillation. The illustration and pseudocode are presented in Figure 4 and Algorithm 1, respectively.
Algorithm 1 PPO with achievement distillation
Require: Policy network  $\pi_{\theta}$  ,value network  $V_{\theta}$    
1: for phase  $= 1,2,\ldots$  do   
2: Reset the buffer  $\mathcal{B}$    
3: for iter  $= 1,2,\dots ,N_{\pi}$  do PPO training   
4: Collect episodes  $\mathcal{T}$  using  $\pi_{\theta}$  and add them to  $\mathcal{B}$    
5: for epoch  $= 1,2,\dots ,E_{\pi}$  do   
6: Optimize  $J_{\pi}(\theta)$  and  $J_V(\theta)$  using  $\mathcal{T}$    
7: end for   
8: end for   
9:  $\pi_{\theta_{old}}\gets \pi_{\theta},V_{\theta_{old}}\gets V_{\theta}$    
10: for iter  $= 1,2,\dots ,E_{aux}$  do AChievement distillation   
11: Optimize  $L_{pred}(\theta),R_{\pi}(\theta)$  ,and  $R_V(\theta)$  using  $\mathcal{B}$    
12: Optimize  $L_{match}(\theta),R_{\pi}(\theta)$  ,and  $R_V(\theta)$  using  $\mathcal{B}$    
13: end for   
14: end for
To assess the effectiveness of our method in discovering hierarchical achievements, we train the agent on Crafter for 1M environment steps and evaluate its performance, following the protocol in Hafner [19]. We measure the success rates for all 22 achievements across all training episodes as a percentage and calculate their geometric mean to obtain our primary evaluation score<sup>2</sup>. Note that the geometric mean prioritizes unlocking challenging achievements. We also measure the episode reward, which indicates the number of achievements unlocked within a single episode, and report the average across all episodes within the most recent 100K environment steps. We conduct 10 independent runs using different random seeds for each experimental setting and report the mean and standard deviation. The code can be found at https://github.com/snu-mllab/Achievement-Distillation.
We compare our method with our backbone algorithm PPO and four baseline methods that have been previously evaluated on Crafter in other research work: DreamerV3, LSTM-SPCNN, MuZero + SPR, and SEA [21, 44, 45, 49]. DreamerV3 is a model-based algorithm that has achieved state-of-the-art performance on Crafter without any pre-training. LSTM-SPCNN is a model-free algorithm based on PPO that employs a recurrent and object-centric network to improve performance on Crafter. MuZero + SPR is a model-based algorithm that has demonstrated state-of-the-art performance on Crafter by utilizing unsupervised pre-training. SEA is a hierarchical algorithm based on IMPALA that employs
The score is computed by  $S = \exp \left( \frac{1}{N} \sum_{i=1}^{N} \ln(1 + s_i) \right) - 1$ , where  $s_i \in [0, 100]$  is the success rate of the  $i$ th achievement and  $N = 22$ .
Figure 5: Score and reward curves.
a high-level planner to discover achievements on Crafter. We provide the implementation details and hyperparameters in Appendix C.
5.2 Crafter results
Table 1 and Figure 5 present the Crafter scores and rewards obtained by our method and the baselines. Our method outperforms all the baselines trained from scratch in both the metrics by a considerable margin, with a score of  $21.79\%$  and a reward of 12.60. Notably, our method exhibits superior score performance compared to MuZero + SPR, which utilizes pre-collected exploratory data and employs a computationally expensive tree search algorithm for planning, while achieving comparable rewards.
Figure 6 shows the individual success rates for all 22 achievements of our method and two successful baselines, DreamerV3 and LSTM-SPCNN. Remarkably, our method outperforms the baselines in unlocking challenging achievements. For instance, our method collects iron with a probability over  $3\%$ , which is 20 times higher than DreamerV3. This achievement is extremely challenging due to its scarcity on the map and the need for wood and stone tools. Moreover, our method crafts iron tools with a probability of approximately  $0.01\%$ , which is not achievable by either of the baselines. Finally, our method even succeeds in collecting diamonds, the most challenging task, on individual runs.
Table 1: Scores and rewards. MuZero + SPR† denotes the results replicated from the original paper.
Method	Parameters	Score (%)	Reward
Human Expert	-	50.5 ± 6.8	14.3 ± 2.3
From scratch	Ours	9M	21.79 ± 1.37	12.60 ± 0.31
PPO	4M	15.60 ± 1.66	10.32 ± 0.53
DreamerV3	201M	14.77 ± 1.42	10.92 ± 0.53
LSTM-SPCNN	135M	11.67 ± 0.80	9.34 ± 0.23
MuZero + SPR†	54M	4.4 ± 0.4	8.5 ± 0.1
SEA	1.5M	1.22 ± 0.13	0.63 ± 0.08
Pre-training	MuZero + SPR†	54M	16.4 ± 1.5	12.7 ± 0.4
5.3 Model size analysis
We compare the model sizes between our method and the baselines. As shown in Table 1, our method achieves better performance with fewer parameters. In particular, our method only requires  $4\%$  of the parameters used by DreamerV3. Note that while our method has twice as many parameters as PPO, most of this increase is due to the networks used for the auxiliary learning, which are not utilized during inference. Additionally, we test our method with a smaller model by reducing the channel size to [16, 32, 32] and the hidden dimension to 256, resulting in a total of 1M parameters. Notably, it still outperforms the baselines with a score of  $17.07\%$ .
Figure 6: Individual success rates for all achievements.
5.4 Representation analysis of achievement distillation
To validate whether our method induces the encoder to have better representations for predicting the next achievements, we conduct an analysis of the latent representations of the encoder using the same approach as described in Section 3. Our method achieves a classification accuracy of  $73.6\%$ , which is a  $28.7\%$  p increase compared to PPO. Furthermore, Figure 3 demonstrates that our method produces predictions with significantly higher confidence than PPO, with a median value of 0.752.
5.5 Ablation studies
We conduct ablation studies to evaluate the individual contribution of our proposed method, intra-trajectory achievement prediction (I), cross-trajectory achievement matching (C), and memory (M). Table 2 shows that while intra-trajectory achievement prediction is the most significant contributor, cross-trajectory achievement matching and memory also play important roles in improving the performance of our method.
Table 2: Ablation studies.
I	C	M	Score (%)
X	X	X	15.60 ± 1.66
✓	X	X	19.02 ± 1.65
✓	✓	X	20.36 ± 1.79
✓	✓	✓	21.79 ± 1.37
5.6 Extension to value-based algorithms
In our previous experiments, we use the on-policy policy gradient algorithm, PPO, as our backbone RL algorithm. To assess the adaptability of our method to other RL paradigms, we extend our experiments to the popular off-policy value-based algorithm, QR-DQN, introducing a slight modification [11]. Specifically, we employ Huber quantile regression to preserve the Q-network's output distribution in alignment with the value function optimization in QR-DQN. We train the agent on Crafter for 1M environment steps and evaluate its performance. Notably, our method also proves effective for value-based algorithms, elevating the score from 4.14 to 8.07. The detailed experimental settings and results are provided in Appendix D.
5.7 Application to other environments
To evaluate the broad applicability of our method to diverse environments, we conduct experiments on two additional benchmarks featuring hierarchical achievements: Procgen Heist and MiniGrid [8, 7]. Heist is a procedurally generated environment whose goal is to steal a gem hidden behind a sequence of blue, green, and red locks. To open each lock, an agent must collect a key with the corresponding color. Heist introduces another challenge, given that the color of wall and background can vary between environments, whereas Crafter maintains fixed color patterns for its terrains. Additionally, we create a customized a door-key environment using MiniGrid to evaluate the effectiveness of our method on a deeper achievement graph. Notably, our method significantly improves the performance of PPO in Heist, increasing the score from 29.6 to 71.0. Our method also outperforms PPO in the MiniGrid environment by a substantial margin, elevating the score from 3.33 to 8.04. The detailed experimental settings and results can be found in Appendix E.
Discovering hierarchical achievements in RL One major approach to this problem is model-based algorithms. DreamerV3 learns a world model that predicts future states and rewards and trains an agent using imagined trajectories generated by the model [21]. While achieving superior performance on Crafter, it requires more than 200M parameters. MuZero + SPR trains a model-based agent with a self-supervised task of predicting future states [37, 45]. However, it relies on pre-training with 150M environment steps collected via RND to improve its performance on Crafter [6].
Another approach is hierarchical algorithms, while many of these methods are only tested on gridworld environments [43, 10]. HAL introduces a classifier that predicts the next achievement to be done and uses it as a high-level planner [10]. However, it relies on prior information on achievements, such as what achievement has been completed, and does not scale to the high-dimensional Crafter. SEA reconstructs the Crafter achievement graph using 200M offline data collected from a pre-trained IMPALA policy and employs a high-level planner on the graph [49]. However, it requires expert data to fully reconstruct the graph and has not been tested on a sample-efficient regime.
There are only a few studies that have explored model-free algorithms. LSTM-SPCNN uses an LSTM and a size-preserving CNN to improve the performance of PPO on Crafter [23, 26, 44]. However, it requires 135M parameters and the performance gain is modest compared to PPO with a CNN.
Representation learning in RL There has been a large body of work on representation learning for improving sample efficiency on a single environment [25, 47, 40], or generalization on procedurally generated environments [29, 30, 32]. However, representation learning for discovering hierarchical achievements has little been explored. A recent study has evaluated the performance of the previously developed representation learning technique SPR on Crafter, but the results are not promising [40, 45].
While widely used in other domains, optimal transport has recently garnered attention in RL [12, 15, 27]. However, the majority of the studies focus on imitation learning, where optimal transport is employed to match the behavior of an agent with offline expert data. In this work, we utilize optimal transport to obtain generalizable representations for achievements in the online setting.
In this work, we introduce a novel self-supervised method for discovering hierarchical achievements, named achievement distillation. This method distills relevant information about achievements from episodes collected during policy updates into the encoder and can be seamlessly integrated with a popular model-free algorithm PPO. We show that our proposed method is capable of discovering the hierarchical structure of achievements without any explicit component for long-term planning, achieving state-of-the-art performance on the Crafter benchmark using fewer parameters and data.
While we utilize only minimal information about hierarchical achievements (i.e., the agent receives a reward when a new achievement is unlocked), one limitation of our work is that we have not evaluated the transferability of our method to an unsupervised agent without any reward. A promising future direction would be developing a representation learning method that can distinguish achievements in a fully unsupervised manner and combining it with curiosity-driven exploration techniques [41, 16].
This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00882, SW STAR LAB) Development of deployable learning intelligence via self-sustainable and trustworthy machine learning,  $80\%$ , and No. 2022-0-00480, Development of Training and Inference Methods for Goal-Oriented Artificial Intelligence Agents,  $20\%$ ). This research was supported by a grant from KRAFTON AI. This material is based upon work supported by the Air Force Office of Scientific Research under award number FA2386-23-1-4047. Hyun Oh Song is the corresponding author.
[38] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[40] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In ICLR, 2021.
[41] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, 2020.
[42] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
[43] Sungryull Sohn, Hyunjae Woo, Jongwook Choi, and Honglak Lee. Meta reinforcement learning with autonomous inference of subtask dependencies. In ICLR, 2020.
[44] Aleksandar Stanić, Yujin Tang, David Ha, and Jürgen Schmidhuber. Learning to generalize with object-centric agents in the open world survival game drafter. arXiv preprint arXiv:2208.03374, 2022.
[45] Jacob Walker, Eszter Vértes, Yazhe Li, Gabriel Dulac-Arnold, Ankesh Anand, Théophane Weber, and Jessica B Hamrick. Investigating the role of model-based learning in exploration and transfer. arXiv preprint arXiv:2302.04009, 2023.
[46] Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. Emergence of maps in the memories of blind navigation agents. In ICLR, 2023.
[47] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In ICML, 2021.
[48] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI, 2021.
[49] Zihan Zhou and Animesh Garg. Learning achievement structure for structured exploration in domains with sparse reward. In ICLR, 2023.