Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning
Authors: Seungyong Moon
Junyoung Yeom
Bumsoo Park
Hyun Oh Song
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
hierarchical reinforcement learning
contrastive learning
procedurally generated environments
TL;DR: We develop a new contrastive learning method for discovering hierarchical achievements that can be seamlessly integrated with model-free RL algorithms.

Abstract: Discovering achievements with a hierarchical structure in procedurally generated environments presents a significant challenge.
This requires an agent to possess a broad range of abilities, including generalization and long-term reasoning. Many prior methods have been built upon model-based or hierarchical approaches, with the belief that an explicit module for long-term planning would be advantageous for learning hierarchical dependencies. However, these methods demand an excessive number of environment interactions or large model sizes, limiting their practicality. In this work, we demonstrate that proximal policy optimization (PPO), a simple yet versatile model-free algorithm, outperforms previous methods when optimized with recent implementation practices. Moreover, we find that the PPO agent can predict the next achievement to be unlocked to some extent, albeit with limited confidence. Based on this observation, we introduce a novel contrastive learning method, called achievement distillation, which strengthens the agent's ability to predict the next achievement. Our method exhibits a strong capacity for discovering hierarchical achievements and shows state-of-the-art performance on the challenging Crafter environment in a sample-efficient manner while utilizing fewer model parameters.

Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning
Seungyong Moon', Junyoung Yeom', Bumsoo Park?, Hyun Oh Song!"
Seoul National University, *KRAFTON
(symoon11, yeomjy, lyyunoh]@mlaab @mllab. snu ac kr bumsoo. b park96kkrafton. com
Abstract
Discovering achievements with a hierarchical structure in procedurally generated environments presents a significant challenge. This requires an agent to possess a broad range of abilities, including generalization and long-term reasoning. Many prior methods have been built upon model- based or hierarchical approaches, with the belief that explicit module for long-tenn plamning would be advantageous for learning hierarchical dependencies. However, these methods demand an excessive number of environment interactions large model sizes, limiting their practicality. In this work, we demonstrate that proximal policy optimization (PPO), a simple yet versatile model-free algorithm, outperforms previous methods when optimized with recent implementation practices. Moreover, we find that the PPO agent can predict the next achievement be unlocked to some extent, albeit with limited confidence. Based on this observation, we introduce novel contrastive learning method, called achievement distillation, which strengthens the agent's ability predict the next achievement. Our method exhibits strong capacity for discovering hierarchical achievements and shows saate-of-hh--ss performance on the challenging Crafter environment in a sample-efficient manner while utilizing fewer model parameters.
Introduction
Deep reinforcement learning (RL) has recently achieved remarkable successes solving challenging decision-makinn making problems, including video games, board games, and robotic controls [31 42 However, these advancements are often restricted set of tasks. To successfully deploy RL agents in real-world scenarios, which are constantly changing and open-ended, they should generalize well to new unseen situations and acquire reusable skills for solving increasingly complex tasks via long-tern ong-term reasoning. Unfortunately, many existing algorithms exhibit limitations learning these abilities and tend to memorize action sequences rather than truly understand the underlying structures of the environments 28 24
18 36)
single deterministic environment with narrow
To assess the abilities of agents in generalization and long-term reasoning. we focus the problem of discovering hierarchical achievements in procedurally generated environments with digh-dimensional image observations. In each episode, an agent navigates a previously unseen environment and receives a sparse reward upon accomplishing novel subtask labeled as an achievement. Importantly each achievement is semantically meaningful and can be reused complete more complex achievements. Such setting inherently demands strong generalization and long-tern teg-termmen reasoning from the agent.
Previous work this problem has mainly relied model-based hierarchical approaches, which involve explicit modules for long-term planning. Model-based methods employ a latent world model
that predicts future states and rewards for learning long-term dependencies 20 21
45). While
Conresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

these methods have shown effectiveness in discovering hierarchical achievements, particularly in procedurally generated environments, they are constructed with large model sizes and often require substantial exploratory data, which limits their practicality. Hierarchical methods aim to reconstruct the dependencies between achievements as graph and employ high-level planner on the graph to direct a low-level controller toward the next achievement to be unlocked 433 T0 49]. However, these methods rely on prior knowledge of achievements (e.g., the number of achievements), which is impractical in open- world scenarios where the exact number of achievements cannot be predetermined. Additionally, they necessitate a significant number of offline expert data to reconstruct the graph.
To address these issues, we begin by exploring the potential of proximal policy optimization (PPO), simple and flexible model-free free algorithm, in discovering hierarchical achievements [39]. Surprisingly. PPO outperforms previous model- based and hierarchical methods by adopting recent implementing practices. Furthermore, upon analyzing the latent representations of the PPO agent, we observe that has a certain degree of predictive ability regarding the next achievement, albeit with high uncertainty.
Based on this observation, we propose novel self self-supervised learning method alongside RL training, named achievement distillation. Our method periodically distills relevant information on achievements from episodes collected during policy updates the encoder via contrastive learning (44 Specifically, we maximize the similarity in the latent space between state-action pairs and the corresponding next achievements within a single episode. Additionally, by leveraging the uniform achievement structure across all environments, we maximize the similarity in the latent space between achievements from two different episodes, matching them using optimal transport [5]. This learning can be seamlessly integrated into PPO by introducing auxiliary training phase. Our method demonstrates state-of- the-art performance in discovering hierarchical achievements on the challenging Crafter benchmark, unlocking all 22 achievements with a budget of 1M environment steps while utilizing only 4% of the model parameters compared to the previous state-of-the-art method [9].
2
Preliminaries
2.1 Markoy decision processes with hierarchical achievements
We formalize the problem using Markov decision processes (MDPs) with hierarchical achievements [4)] Let M. represent a collection of such MDPs. Each environment M2 E M is defined by a tuple (S., G. p. r. Ps: 7). Here, S, C S is the image observation space, which has visual variations across different environments, A is the action space, 9 is the achievement graph with a hierarchical structure, p: S x A P(S) is the transition probability function, r S x A S -> IK is the achievement reward function, Ps E PS) is the initial state distribution, and y E [0, 1] is the discount factor.
The achievement graph G = (V,E)) is directed acyclic graph, where each vertex E V represents an achievement and each edge (u, v) E & indicates that achievement v has dependency on achievement To unlock achievement v, all of its ancestors (i.e., achievements in the path from the root to v) must also be unlocked. When an agent unlocks a new achievement, it receives an achievement reward of 1. Note that each achievement can be accomplished multiple times within single episode, but the agent will only receive a reward when unlocking it for the first time. Specifically, let b E (0. 1) 1) be a binary vector indicating which achievements have been unlocked and c: S x x - VU (6) be function determining whether transition tuple results in the completion of an achievement. Then, the achievement reward function is defined as
[1 if 30 E V bbd 0, c(s,, a2 8(41) 02
r(st, ae, 81+1)
0 herwise.
This reward structure provides an incentive for the agent to explore the environments and discover a new achievement, rather than repeatedly accomplishing the same achievements.
We assume that the agent has no prior knowledge of the achievement graph, including the number of achievements and their dependencies. Additionally, the agent has no direct access to information about which achievements have been unlocked. Instead, the agent must infer this information indirectly from the reward signal it receives. Given this situation, our objective is to learn generalizable policy S > P(A) that maximizes the expected return (i.e,, unlocks as many achievements as possible) across all environments of M.
2

---

Collect wood
Place table
Callect sapling
Place plant
Make wood pickaxe
Make wood sword
Eat plant
Collect coal
Collect
Place
Collect drink
Place Turnace
Make stone pickase
Make stone sword
Eat cow
Make iron sword
Collect iron
Defeat zrmbii
Wakk up
4
11111
Make iron pickaxe
Collect diamond
Defeat skeletton
(a) Agent's observation
(b) Achievement graph with a hicrarchical structure.
Figure 1: Overview of Crafter. Each environment procedurally generated and partially observable. The objective is to unlock as many achievements possible within single episode.
2.2 Crafter environment
We primarily utilize the Crafter environment benchmark to assess the capabilities of an agent in solving MDPs with hierarchical achievements [19]. Crafter is an open-world survival game with 2D visual inputs, drawing inspiration from the popular 3D game Minecraft [17]. This is optimized for research purposes, with fast and straightforward environment interactions and clear evaluation metrics. The game consists of procedurally generated environments with varying world map layouts, terrain types, resource placements, and enemy spawn locations, each of which is uniquely determined by an integer seed. An agent can only observe its immediate surroundings depicted in Figure Ia| which makes Crafter partially observable and thus challenging. To survive, the agent must acquire a variety of skills, including exploring the world map, gathering resources, building tools, placing objects, and defending against enemies. The game features set of 22 hierarchical achievements that the agent can unlock by completing specific prerequisites, as illustrated in Figure Tb] For instance, to make wood pickaxe, the agent needs to collect wood, place table, and stand nearby. This achievement structure is designed to require the agent to learn and utilize wide range of skills to accomplish increasingly challenging achievements, such as crafting iron tools and collecting diamonds.
2.3 Proximal policy optimization
PPO is one of the most successful model-free policy gradient algorithms due to its simplicity and effectiveness [39]. PPO learns a policy Ta S - P(A) and a value function Vo S S R, which parameterized by neural networks. During training. PPO first collects a new episodes T using the policy Masa immediately prior to the update step. Subsequently, PPO updates the policy network using these episodes for several epochs maximize the clipped surrogate policy objectives given by
( se) Ae, clip Toas (az st)
(t)) To (at st)
Te(a;
(+) E(EerOe)T
min
e, 1 1,1cc) A,
where A, is the estimated advantage computed by generalized advantage estimate (GAE) [38 PPO simultaneously updates the value network to minimize the value objective given by
(Vo(s)) -v)) v4)
(+) = E 2
where V; - + Volsa (8t) is the bootstrapped value function target.
In bmsge RL, it is common practice to optimize the policy and value networks using shared An image observation is first passed through convolutional encoder Φo : S - Rhth to extract state representation, which is then fed into linear heads to compute the policy and value function. Sharing state representations between the policy and value networks crucial to improving the performance of agents in high- -igeenninnii state spaces. However, relying solely on policy and value optimization to train the encoder can lead to suboptimal state representations, particularly in procedurally generated environments [9] To address this issue, recent studies introduce an auxiliary training phase alongside the policy and value optimization that trains the encoder with auxiliary value or self-supervised objectives 32
network architecture 13 48
3

---

3 Motivation
3.1 PPO is strong baseline for hierarchical achievements
Despite being a simple and ubiquitous algorithm, PPO is less utilized than model-based or hierarchical approaches for solving MDPs with hierarchical achievements. This is due to the fact that PPO does not have an explicit component for long-term planning or reasoning, which is believed to be essential for solving hierarchical tasks. However, recent study has shown that PPO-based algorithm is also successful in solving hierarchical achievement on the Minecraft environment, albeit with the aid of pre-training on human video data
Based on this observation, we first investigate the effectiveness of PPO in solving hierarchical achievements on Crafter without pre- training. We adopt the recent implementation practices proposed in Andrychowicz et al. (2), Baker et al. [4]. Concretely, modify the default ResNet architecture in IMPALA as follows [4]:
Size + Layer + Value
Size Layer Size Default
Network size: We increase the channel size from [16, 32, 32] [64, 128, 128] and the hidden size from 256 to 1024.
Layer normalization: We add layer normalization before each
Score (%)
dense or convolutional layer (3]
Value normalization: We keep moving average for the mean and standard deviation of the value function targets and update the value network to predict the normalized targets.
5
We train the modified PPO Crafter for 1M environment steps and evaluate the success rates for unlocking achievements (please refer to Section [. for the evaluation). Figure 2 shows that the slight modification in implementing PPO significantly improves the performance, increasing the score from 8. 8.17 to 15.60. Notably, this outperforms the current state-of-the-art DreamerV3, which achieves score of 14.77.
0
0.4
0.6
0.8
1.0
Environment steps (M)
Figure 2: Score curves of PPO.
3.2 Representation analysis of PPO
Since Crafter environments are procedurally generated, simply memorizing successful episodes is insufficient for achieving high performance. We hypothesize that the PPO agent acquires knowledge beyond mere memorization action sequences, possibly including information about achievements. To validate this, we analyze the learned latent representations of the encoder, as inspired by Wijmans al. [46. Specifically, we collect a batch of episodes using an expert policy and subsample a set of states for training. For each state in the training set, freeze its latent representation Φe(s) from the encoder. Subsequently, we train a linear classifier using this representation as input to predict the very next achievement unlocked in the episode containing Finally, evaluate the classification accuracy and the prediction confidence of the ground-trun -truh labels on a held-out test set. The detailed experimental settings provided in Appendix A.
Surprisingly, PPO achieves nontrivial accuracy of 44.9% in the 22-way classification. However, Jisurs that the prediction outputs lack confidence with a median value of 0.240. This suggests that the learned representations of the PPO encoder are not strongly correlated with the next achievement to be unlocked and the agent may struggle to generate optimal action sequences towards a specific goal.
Ours
PPO
0.2
Density
0.1
This finding warrants providing additional guidance to the encoder for predicting the next achievements with high confidence. However, since the agent has no access to the achievement labels, is challenging to guide the agent a supervised fashion. Therefore, it is necessary to explore alternative approaches to guide the agent toward predicting the next achievements.
0
0.5 Confidence
Figure 3: Histogram for the confidence next achievement prediction.

---

Policy phase
(Tts 9141)
Policy mo 8t)
Policy optimization
State
Encoder 4
Butfer
Agent
Store episodes
Run PPO
Value Vo(*,)
Value optimization
0g
mpp
8t)
Auxiliary phase Intra-trajectory achievement prediction
shy (8 de. St
Positive
Poloz Anchor
Buffer
( der, (c) Negative
Sample episodes
Current state-action Next achievement
Random state action
Cross-trajectory achievement matching
9:
94
de
(o(o))
91
0.0
0.2
0.10
0.30
0.81
0.05
0.11
0.03
Positive
Buffer
.2
0.90
0.000.30
POT
0.07
0.00
0.31
>
0.5
(u(gh Anchor
me(os)
Sample episodes
Negative
Cost matrix
Soft matching
Hard matching
Figure Illustration of achievement distillation.
4 Contrastive learning for achievement distillation
In this section, we introduce a new self-supervised learning method that works alongside RL training to guide the encoder in predicting the next achievement to be unlocked. This approach distills relevant information about discovered achievements from episodes collected during multiple policy updates into the encoder via contrastive learning. This method consists of two key components:
Intra-trajectory achievement prediction: Within an episode, this maximizes the similarity in the latent space between state-action pair and its corresponding achievement.
Cross-trajectory achievement matching: Between episodes, this maximizes the similarity in the latent space for matched achievements.
For of notation, denote the sequence of unlocked achievements within episode (gi ) and their corresponding timesteps as (t where each achievement 9; is defined by transition tuple (St, de, St,+1). For each timestep 6, we represent the very next achievement as gr = Ju. where min S 42), and the very previous achievement as 9r = gt. where max (i >>> 4...
4.1 Intra-trajecfory achievement prediction
Given state-action pair (st, ar) and its corresponding next achievement 9 within an episode we train the encoder de to produce similar representations for them through contrastive learning [34 22] Specifically, regard g, as the anchor and (st, 0t) as the positive. We also randomly sample another state-acti state-action pair (str, av.) from the same episode serve as the negative. Subsequently, we obtain the normalized representations of the anchor, positive, and negative, denoted as ve "e(g:" ), ( ax), and qhe(Sr, ar) respectively. Finally, we minimize the following contrastive loss to maximize the cosine similarity between the anchor and positive representations while minimizing the cosine similarity between the anchor and negative representations:
exp(vo(s2, a.) TT (o())/A)) exp(v0o a.) To vo(g))/)) )/1) + exp(No(se, ax) vo ((T))/))
Lpred (0) -E JNT (Sgrsa,)) y
log
where À > 0 is the temperature parameter.
To obtain the state-action representation (h, az), we calculate the latent representation of the state
(*(*)) from the encoder and concatenate with action a, using FiLM layer (35). The resulting
vector is then passed through an MIP layer and normalized. To obtain the achievement representation vo(gt we simply calculate the residual the latent representations the two consecutive states
from the encoder and normalize it, as motivated by Nair al. [33].

---

While this contrastive objective encourages the encoder to predict the next achievement in the latent space, it can potentially lead distortions in the policy and value networks due the changes in the encoder. To address this, we jointly minimize the following regularizers to preserve the outputs of the policy and value networks, following the practice in Moon et al. 32
(0) = Esser [DRL. (Fo (: 8e) ma(- 8x))], (0) EsT
[2
(Vo(s)) Voeu (8+))²
where a0, and Vossa are the policy and value networks immediately prior to the contrastive learning, respectively and DKL denotes the KL divergence.
4.2 Cross-trajectory achievement matching
Since Crafter environments are procedurally generated, the achievement representations learned solely from intra-traecctory information may include environment-speeific features that limit generalization. To obtain better achievement representations, we leverage the common achievement structure shared across all episodes.
We first match the sequences of unlocked achievements from two different episodes in an unsupervised fashion. Given two achievement sequences g = (g.) and g' (gr) define the cost matrix M E x as the cosine distance between the achievement representations:
My 1 vo()) T vo(o))-
Subsequently, we regard these two sequences as discrete uniform distributions and compute soft- matching T mg between them using partial optimal transport, which can be solved by
T = arg min (T +
Tij log Tij
T20
=1
subject to 71 T1 1, T 151, 1 1''
where a >> is the entropic regularization parameter [5]. Here, we the total amount of probability mass to be transported to the minimum length of the two sequences for simplicity. However, some unlocked achievements in sequence may not exist in the other sequence, and therefore should not be transported. In this case, the optimal amount of probability mass to be transported should be less than the minimum length. To address this, we compute the conservative hard matching T* from T by thresholding the probabilities less than 0.5 (i.e., 7 1[ > 0.5l). Note that this also encourages each achievement to be matched to most one other achievement. We provide examples of matching results in Appendix B.
We train the encoder to produce similar representations for the matched achievements according to T* through contrastive learning. Specifically, suppose that the vth achievement of the source sequence gi is matched with the kth achievement of the target sequence 9 Then, we consider yi as the anchor and 96. as the positive. We also randomly sample another achievement g/ from the target sequence to serve as the negative. Subsequently, we obtain the normalized representations of these achievements. Finally, we minimize the following contrastive loss to maximize the cosine similarity between the anchor and positive representations while minimizing the cosine similarity between the anchor and negative representations:
cxp(vo(o))Tvoocdt))
)
Lmaatc (0) = -E E 9.008.9/~'""
log exp(ve(g:) (eoe)))v((g))))) ve(g)())) + (x)()) vo(g))/A)
As in Section 4.1 we jointly minimize the policy and value regularizers to prevent distortions.
4.3 Achievement representation memory
We further utilize the achievement representations learned Sections 4.1 and 4.2 as memory for the policy and value networks. Specifically, given state s, and its corresponding previous achievement gr we concatenate the latent state representation Φo (st) with the previous achievement representation vo(g)) The resulting vector then fed into the policy and value heads to output an action distribution and value estimate, respectively.

---

We also utilize the previous achievement for the achievement prediction task in Section 4.1 Given state-action pair (st, az) and its corresponding previous achievement 9: we concatenate the latent state representation (b((s) from the encoder with a, and (g The resulting vector is then fed into an MLP layer and normalized obtain the representation (ha (St, des 9r for the next achievement prediction. This is interpreted as learning forward dynamics in the achievement space.
4.4 Integration with RL training
We integrate the contrastive learning method proposed in Sections 4. to 4.3 with PPO training by introducing two alternating phases, the policy and auxiliary phases. During the policy phase, which repeated several times, we update the policy and value networks using newly-collected episodes and store them in buffer. During the auxiliary phase, we update the encoder to optimize the contrastive objectives in conjunction with the policy and value regularizers using all episodes in the buffer. We call this auxiliary learning achievement distillation. The illustration and pseudocode are presented in Figure 4 and Algorithm respectively.
Algorithm 1 PPO with achievement distillation
Require: Policy network Te. value network Vi
1: for phase = 1, 2. do
2: Reset the buffer B
3: for
. Nr do Collect episodes T using To and add them to B for epoch = 2..... Ex do
b PPO training
4:
5:
6:
Optimize (0) and Jy (0) using T
7:
end for end for
8:
9:
Telid f TTa Viaia f- V/8
for iter= ....... EAux do
Achievement distillation
11:
Optimize Lired (0), Ry (0), and Ry(00) using B Optimize Lmatth (0), R2 (9), and Ry (8) using B
13: end for
end for
5 Experiments
5.1
Experimental setup
To assess the effectiveness of our method in discovering hierarchical achievements, we train the agent on Crafter for 1M environment steps and evaluate its performance, following the protocol Hafner [19]. We measure the success rates for all 22 achievements across all training episodes percentage and calculate their geometric mean to obtain our primary evaluation score] Note that the geometric mean prioritizes unlocking challenging achievements. We also measure the episode reward, which indicates number of achievements unlocked within single episode, and report the average across all episodes within the most recent 100K environment steps. We conduct 10 independent runs using different random seeds for each experimental setting and report the mean and standard deviation. The code can be found at ptttps://githut com/ snu mllab/ Achi -eeiiieeeeetiiieee wissillatiiinnnn..m.m
We compare our method with our backbone algorithm PPO and four baseline methods that have been previously evaluated on Crafter in other research work: DreamerV3, LSTM-SPCNNS MuZero SPR, model-based algorithm that has achieved state-of-the- performance Crafter without any pre-training. LSTM-SPCNN is model-free algorithm based on PPO that employs recurrent and object-centric network to improve performance on Crafter. MuZero + SPR is model-based algorithm that has demonstrated state-of-the-ard performance on Crafter by utilizing unsupervised pre- trainininng... SEA is a hierarchical algorithm based on IMPALA that employs
and SEA 21 44
45 49
DreamerV3
'The score is computed by S = exp( 1 ΣN, In(1 + st)) - 1, where [0, 100] is the success rate of the ith achievement and N 22.

---

25
16
Ours
20
PPO
DreamerV3
Score (%
15
Reward
LSTM-SPCNN
8
10
4
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
Environment steps (M)
Environment steps (M)
Figure Score and reward curves.
a high-level planner to discover achievements on Crafter. We provide the implementation details and hyperparameters in Appendix C.
5.2 Crafter results
Table Figure present the Crafter scores and rewards obtained by our method and the baselines. Our method outperforms all the baselines trained from scratch in both the metrics by considerable margin, with score of 21.79% and a reward of 12.60. Notably, our method exhibits superior score performance compared to MuZero SPR, which utilizes pre-collected exploratory data and employs a computationally expensive tree search algorithm for planning, while achieving comparable rewards.
Figure 6 shows the individual success rates for all 22 achievements of method and two successful baselines, DreamerV3 and LSTM-SPCNN. Remarkably, our method outperforms the baselines in unlocking challenging achievements. For instance, our method collects iron with probability 3%, which is 20 times higher than Dreamer V3. This achievement is extremely challenging due to its scarcity on the map and the need for wood and stone tools. Moreover, method crafts iron tools with probability of approximately 0.01 %, which is not achievable by either of the baselines. Finally, method even succeeds in collecting diamonds, the most challenging task, individual runs.
Table Scores and rewards. MuZero SPR denotes the results replicated from the original paper.
Method
Parameters
Score (%)
Reward
Human Expert
50.5 6.8
14.3 + 2.3
Ours
9M
21.79 = 1.37
12.60 + 0.31
PPO
4M
15.60
1.66
10.32 + 0.53
From scratch
DreamerV3
201M
14.77
1.42
10.92 + 0.53
LSTM-SPCNN
135M
11.67
= 0.80
9.34 + 0,23
MuZero + SPR SEA
54M
4.4 = 0.4
8.5 + 0.1
1.5M
1.22 = 0.13
0.63 + 0.08
Pre-training
MuZero + SPR
54M
16.4 1.5
12.7 + 0.4
5.3
Model size analysis
We compare the model sizes between our method and the baselines. As shown in Table our method achieves better performance with fewer parameters. In particular, our method only requires 4% of the parameters used by DreamerV3. Note that while our method has twice many parameters as PPO, most of this increase due to the networks used for the auxiliary learning, which are not utilized during inference. Additionally, we test our method with smaller model by reducing the channel size to [16, 32, 32] and the hidden dimension to 256, resulting in a total of 1M parameters. Notably, still outperforms the basclines with score of 17.07%.

---

Ours DreamerV3
LSTM-SPCNN
100
Success (%)
0.1
0.01
0.001
Collect ond
Collect dismond
Collect drink
Collect iron
Collect sapligg
Collect stone
Callect wood
Defeat skdeton
Dekeat zuntii
Eal cow
Make iron pickase
Fat plant
Make imn sword
Make stone pickase
Make stone sword
Make wood pickate
Make ward sword
Place furnare
Place plant
Place stone
Place table
Wake up
Figure 6: Individual success rates for all achievements.
5.4 Representation analysis of achievement distillation
To validate whether our method induces the encoder to have better representations for predicting the next achievements, we conduct an analysis of the latent representations of the encoder using the same approach as described in Section 3 Our method achieves a classification accuracy of 73.6%, which is a 28.7% increase compared to PPO. Furthermore, Figure demonstrates that our method produces predictions with significanlly higher confidence than PPO, with a median value of 0.752.
5.5 Ablation studies
Table 2: Ablation studies.
We conduct ablation studies evaluate the individual contribution of our proposed method, intra-trajectory achievement prediction (D,, cross-trajectory achievement matching (C), and memory (M). Table 2 Ishows that while intra-trajectory achievement prediction is the most significant contributor, cross- coos-trraectory achievement matching and memory also play important roles in improving the performance of our method.
C
M
Score (%)
X
X
15.60 + 1.66
/
X
X
19.02 + 1.65
X
20.36 + 1.79
J
21.79 +'t 1.37
5.6 Extension to value-based algorithms
In our previous experiments, we use the on-policy policy gradient algorithm, PPO, as our backbone RL algorithm. To assess the adaptability of our method to other RL paradigms, extend our experiments to the popular off-policy value-based algorithm, QR-DQN, introducing slight modification [[]] Specifically, we employ Huber quantile regression preserve the Q-network's output distribution in alignment with the value function optimization in QR-DQN. We train the agent on Crafter for IM environment steps and evaluate its performance. Notably, our method also proves effective for value-based algorithms, elevating the score from 4.14 to 8.07. The detailed experimental settings and results are provided in Appendix D.
5.7 Application to other environments
To evaluate the broad applicability of our method to diverse environments, we conduct experiments on two additional benchmarks featuring hierarchical achievements: Procgen Heist and MiniGrid 88 77. Heist is procedurally generated environment whose goal is to steal a gem hidden behind a sequence of blue, green, and red locks. To open each lock, an agent must collect a key with the corresponding color. Heist introduces another challenge, given that the color of wall and background can vary between environments, whereas Crafter maintains fixed color patterns for its terrains. Additionally, we create a customized a door-key environment using MiniGrid to evaluate the effectiveness of our method on deeper achievement graph. Notably, our method significantly improves the performance of PPO in Heist, increasing the score from 29.6 to 71.0. Our method also outperforms PPO in the MiniGrid environment by substantial margin, elevating the score from 3.33 8.04. The detailed experimental settings and results can be found in Appendix E.

---

Related work
Discovering hierarchical achievements in RL One major approach to this problem model-bascl algorithms. DreamerV3 learns a world model that predicts future states and rewards and trains an agent using imagined trajectories generated by the model 21. While achieving superior performance on Crafter, it requires more than 200M parameters. MuZero + SPR trains model-based agent with a However, relies on pre-training with 150M Crafter [6]
self-supervised task of predicting future states 37 45
environment steps collected via RND to improve its performance
Another approach is hierarchical algorithms, while many of these methods are only tested on grid- world environments [43 [0]. HAL introduces classifier that predicts the next achievement to be done and uses as high-level planner (1O). However, relies on prior information on achievements, such as what achievement has been completed, and does not scale to the high-dimensional Crafter. SEA reconstructs the Crafter achievement graph using 200M offline data collected from a pre-trained IMPALA policy and employs high-level planner on the graph [9). However, it requires expert data to fully reconstruct the graph and has not been tested on a sample- regime.
There are only a few studies that have explored model-free algorithms. LSTM-SPCNN uses an LSTM and size-preserving CNN improve the performance of PPO Crafter 23 26 44 However, it requires 135M parameters and the performance gain is modest compared to PPO with CNN.
Representation learning in RL There has been a large body of work on representation learning for improving sample efficiency on single environment [25] 47 40). or generalization on procedurally generated environments [29 30 321. However, representation learning for discovering hierarchical achievements has little been explored. A recent study has evaluated the performance of the previously developed representation learning technique SPR on Crafter, but the results are not promising 40 45
While widely used in other domains, optimal transport has recently garnered attention in RL However, the majority of the studies focus on imitation learning, where optimal transport is employed to match the behavior of an agent with offline expert data. In this work, we utilize optimal transport to obtain generalizable representations for achievements in the online setting.
15 27
7
Conclusion
In this work, we introduce novel self-supervised method for discovering hierarchical achievements, named achievement distillation. This method distills relevant information about achievements from episodes collected during policy updates into the encoder and can be seamlessly integrated with a popular model-free algorithm PPO. We show that our proposed method is capable of discovering the hierarchical structure of achievements without any explicit component for long-term planning, achieving tat performance on the Crafter benchmark using fewer parameters and data.
While we utilize only minimal information about hierarchical achievements (i.e., the agent receives a reward when a new achievement is unlocked), one limitation our work that we have not evaluated the transfcrability of our method to an unsupervised agent without any reward. A promising future direction would be developing representation learning method that can distinguish achievements in a fully unsupervised manner and combining it with suriosity-driver exploration techniques 41 16
Acknowledgements
This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00882, (SW STAR LAB) Development of deployable learning intelligence via self-sustainable and trustworthy machine learning, 80%, and No. 2022-0-00480, Development of Training and Inference Methods for Goal- Oriented Artificial Intelligence Agents, 20%). This research was supported by a grant from KRAFTON AI. This material is based upon work supported by the Air Force Office of Scientific Research under award number FA2386-25 FA2386-23-1-4047 Hyun Oh Song is the corresponding author.
10

---

References
[1] Ankesh Anand, Jacob C Walker, Yazhe Li, Eszter Vértes, Julian Schrittwieser, Sherjil Ozair, Theophane Weber, and Jessica B Hamrick. Procedural generalization by planning with self- supervised world models. In ICLR, 2022.
[2] Marcin Andrychowicz, Anton Raichuk, Piotr Starczykk Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In ICLR,, 2021.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 06450, 2016.
[4] Bowen Baker, 1gge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. In NeurlPS, 2022.
[5] Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. Itera- tive bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing, 2015.
[6] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR, 2019.
[7] Maxime Chevalier- Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. arXiv preprint arXiv:2306. 13831, 2023.
[8] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In ICML, 2020.
[9] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In ICML, 2021.
[10] Robby Costales, Shariq Iqbal, and Fei Sha. Possibility before utility: Learning and using hierarchical affordances. In ICLR, 2022.
[11] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI, 2018
[12] Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. ICLR, 2021.
[13] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://g/ thub. com/ http///paaaaaalieeaeeeeeeeeaeeee ines, 2017.
[14] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In JCML, 2018.
[15] Arnadd Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation learning via optimal transport. In ICLR, 2022.
[16] Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Grill, Yunhao Tang, et al. Byol-explore:
Tallec, Alaa Saade, Daniele Calandricllo, Exploration by bootstrapped prediction. In NeurlPS, 2022.
[17] William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. MineRL: A large-scale dataset Minecraft demonstrations. In IJCAI, 2019.

---

[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
[19] Danijar Hafner. Benchmarking the spectrum of agent capabilities. In ICLR, 2022.
[20] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021.
[21] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 2301003300,04044 2023.
[22] Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
[23] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
[24] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 2023,
[25] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In ICML, 2020.
[26] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
[27] Yicheng Luo, zhengyao jiang, Samuel Cohen, Edward Grefenstette, and Mare Peter Deisenroth. Optimal transport for offline imitation learning. In ICLR, 2023.
[28] Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 2018.
[29] Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. In NeurlPS, 2020.
[30] Bogdan Mazoure, Ahmed M Ahmed, R Devon Hjelm, Andrey Kolobov, and Patrick MacAlpine. Crossttray representation learning for zero-shot generalization in RL. In ICLR, 2022.
[31] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[32] Seungyong Moon, Jun Yeong Lee, and Hyun Oh Song. Rethinking value function learning for generalization in reinforcement learning. In NeurlPS, 2022.
[33] Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei, Causal induction from visual observations for goal directed tasks. arXiv preprint arXiv: 2019.
[34] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1807 2018
[35] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
[36] Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 2020.
[37] Julian Schrittwieser, Thomas K Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. In NewrIPS, 2021.
12

---

[38] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprini wrXiv:1707.06444 2017.
[40] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- preficreeietive representations. In ICLR, 2021.
[41] Ramanan Sckar, Oleh Rybkin, Kostas Daniilidis, Picter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, 2020.
[42] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
[43] Sungryull Sohn, Hyunjae Woo, Jongwook Choi, and Honglak Lec. Meta reinforcement learning with autonomous inference subtask dependencies. ICLR, 2020.
[44] Aleksandar Stanić, Yujin Tang, David Ha, and Jürgen Schmidhuber. Learning to generalize with object-centric agents in the open world survival game crafter. arXiv preprint arXiv:2208.0337 03334, 2022.
[45] Jacob Walker, Eszter Vértes, Yazhe Li, Gabriel Dulac-Amold, Ankesh Anand, Théophane Weber, and Jessica B Hamrick. Investigating the role of model-based learning in exploration and transfer. arXiv preprint arXiv:2300. 04009, 2023.
[46] Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. Emer- gence of maps in the memories of blind navigation agents. In ICLR, 2023.
[47] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In ICML, 2021.
[48] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI, 2021.
[49] Zihan Zhou and Animesh Garg. Learning achievement structure for structured exploration in domains with sparse reward. ICLR, 2023.
13


Reviews:
[{"review_id": "7KQ4v5riET", "reviewer": "Reviewer_z85w", "summary": "This paper proposes a contrastive learning approach for representation learning in the problem of hierarchical achievement discovery. The proposed method leverages previous contrastive learning loss and combines it with optimal transport. Empirical results show that the learned representation could improve PPO in the Crafter environment.", "questions": "1.\tThis paper defines MDPs with hierarchical achievements. Has prior work studied this model? If so, please add references.\n2.\tIt is interesting to see PPO is more sample-efficient than model-based methods in this paper’s experiments. Except for implementation practice improvement of PPO, any other insights for this result?", "limitations": "This paper has discussed its limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tIt is interesting that self-supervised representation learning can improve PPO, which significantly outperforms model-based approaches, even regarding sample efficiency. \n2.\tExperimental results show the strong performance of the proposed method in the Crafter environment. \n3.\tPresentation: this paper is generally well-organized and easy to follow.", "weaknesses": "1.\tDomain knowledge: unlike baselines, the proposed method utilizes additional important knowledge, i.e., identifying when a new achievement is unlocked (which can be easily inferred from the observed rewards). This other knowledge can easily explain why the achievement prediction of the proposed method is much better than PPO. \n2.\tGeneralization: this paper only shows its results in the Crafter environment. It is highly recommended to conduct experiments in other environments to show its generality.\n3.\tNovelty: though it is interesting to use optimal transport for matching achievements, the proposed method is a natural application of contrastive learning, which is not quite novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the review's constructive and helpful feedback. We are encouraged by the reviewer’s positive comments (“it is interesting that self-supervised representation learning can improve PPO”, “this paper is generally well-organized and easy to follow”). We would like to address the questions and concerns raised by the reviewer, as detailed below.\n\n---\n\n**Q1: The proposed method utilizes additional important knowledge.**\n\nWe agree with your view that our method utilizes additional information about the reward structure, where each reward signal represents distinct achievements. Nonetheless, we would like to emphasize that we employ the same reward function as the baseline methods without any modification and do not leverage any other information regarding achievements beyond the reward function itself.\n\n**Q2: It is highly recommended to conduct experiments in other environments to show its generality.**\n\nThank you for your recommendation. We conduct experiments on two additional benchmarks that feature hierarchical achievements: Procgen Heist and a custom MiniGrid environment. Please refer to the Global Response Q1 for more details on the benchmarks and our experimental results.\n\n**Q3: This paper defines MDPs with hierarchical achievements. Has prior work studied this model? If so, please add references.**\n\nThe concept of MDPs with hierarchical achievements has been studied in recent prior work [1, 2]. While we have cited these studies in the related work section, we will also include these references in Section 2.1 for further clarification. Thank you for the suggestion.\n\n**Q4: It is interesting to see PPO is more sample-efficient than model-based methods in this paper’s experiments. Except for implementation practice improvement of PPO, any other insights for this result?**\n\nIn general, model-based algorithms exhibit greater sample efficiency than their model-free counterparts, provided that the learned models are accurate. However, training accurate world models on procedurally generated, partially observable environments poses a significant challenge. When reproducing the results of DreamerV3, we find that the losses for the world model gradually increase as training continues. It is worth noting that an agent in Crafter encounters a new environment in each episode. Given this, the world model of DreamerV3, which relies heavily on prior experience from the replay buffer for training, does not transfer well to unseen environments. This challenge has been underscored by a recent study, which further demonstrates that a Dreamer agent struggles to adapt rapidly to the ever-changing environments [3]. In contrast, PPO usually updates the policy and value networks using Monte Carlo methods with recently collected episodes, and does not heavily rely on models trained with prior experience, compared to DreamerV3.\n\nAdditionally, we observe a rapid drop in the policy entropy of DreamerV3, reaching a level of 0.5 at timestep 50K and remaining low thereafter. In contrast, PPO maintains the policy entropy above a level of 1.0 by the end of the training process. This phenomenon limits a DreamerV3 agent's ability to explore environments and unlock new achievements.\n\n---\n\nWe again thank the reviewer for giving constructive suggestions. We hope our explanation above addressed all reviewer's questions.\n\n---\n\n**References**\n\n[1] Robby Costales et al., Possibility Before Utility: Learning And Using Hierarchical Affordances, ICLR 2022. \\\n[2] Zihan Zhou and Animesh Garg, Learning Achievement Structure for Structured Exploration in Domains with Sparse Reward, ICLR 2023. \\\n[3] Isaac Kauvar et al., Curious Replay for Model-based Adaptation, ICML 2023.\n"}, {"review_id": "7aLd7a4fsn", "reviewer": "Reviewer_Tzgb", "summary": "This work introduces achievement distillation, a model-free RL method designed to discover achievements without the need for explicit long-term planning components. The proposed method comprises three primary components: two self-supervised tasks and a memory component. The self-supervised tasks, namely Intra-trajectory achievement prediction and cross-trajectory achievement matching, guide the encoder to predict the next achievement to be unlocked using a contrastive learning loss function with different objectives. The memory component is formed by concatenating the latent state representation from the encoder with the action and the representation of the previous achievement. These components are integrated into the PPO algorithm through two alternating training phases involving policy training and auxiliary self-supervised tasks. The effectiveness of achievement distillation is evaluated in the Crafter environment, where it demonstrates significant performance improvements over strong baselines. Additionally, the authors conducted analyses on model sizes, representations, and the contribution of individual components.", "questions": "1. I would encourage the authors to address the main limitation of their work discussed above by conducting further experimentation in various environments, such as MiniHack and maybe ProcGen.\n\n2. It would greatly enhance the understanding of the contribution of cross-trajectory achieving matching and memory if the authors could conduct additional research. For instance, including the individual success rates for all achievements (Figure 6) while ablating these components could be a valuable option.\n\n3. It would be interesting if the authors could present the results in Table 1 and Figure 5 for a larger range of environment steps (e.g., 5M, 10M). This would clarify whether this method is only more data efficient or remains superior with more steps.\n\n4. In Figure 6, why the Individual success rate is not shown for MuZero+SPR?\n\n5. Do you think your method would work with RL algorithms other than PPO?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The primary strength of the paper lies in the significant results it presents. The main results in Table 1 and Figure 5 provide clear evidence that the proposed achievement distillation method outperforms the baseline methods to a substantial degree. Furthermore, the results in Figure 6 demonstrate that achievement distillation achieves success in several achievements that none of the baselines can accomplish (e.g., making an iron sword), which is impressive.\n2. The authors have done a great job in implementing various relevant strong baselines. Reproducing these baselines are non-trivial given the complexity of them.\n3. While the concept of self-supervised auxiliary tasks and the used self-supervised losses are not novel ideas, the authors have managed to execute them very well, resulting in an agent with remarkable performance.\n4. The paper is well-written, and the ideas are effectively presented and justified.\n", "weaknesses": "1. The primary weakness of this work lies in the limited range of environments in which the proposed method is evaluated. Since the authors only tested it on the Crafter environment, it remains uncertain whether their method would be effective in different settings and whether it has avoided overfitting to the Crafter environment.\n\n2. The ablation study appears to be somewhat superficial. Although the contribution of cross-trajectory achievement matching is evident, the significance of cross-trajectory achieving matching and memory is not as convincing. The authors should provide additional evidence to support the role of these components. Please refer to the next section for suggestions on how to improve this aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We appreciate the encouraging comments (“The authors have done a great job in implementing various relevant strong baselines”, “The paper is well-written, and the ideas are effectively presented and justified”). We would like to address your questions below.\n\n---\n\n**Q1: I would encourage the author to conduct further experimentation in various environments, such as MiniHack and maybe ProcGen.**\n\nThank you for your suggestion. We conduct experiments on two additional benchmarks that feature hierarchical achievements: Procgen Heist and a custom MiniGrid environment. Please refer to the Global Response Q1 for more details on the benchmarks and our experimental results.\n\n**Q2: It would greatly enhance the understanding of the contribution of cross-trajectory achieving matching and memory if the authors could conduct additional research.**\n\nThank you for the suggestion. We first provide the individual success rates for challenging achievements, such as collecting iron, while conducting ablation studies on cross-trajectory matching (C) and memory (M). The table below demonstrates that both cross-trajectory achievement matching and memory play significant roles in unlocking these challenging achievements.\n\n|Achievement|PPO + I|PPO+I+C|PPO+I+C+M|\n|---|---|---|---|\n|Make stone pickaxe|10.92|16.43|22.93|\n|Make stone sword|14.32|20.08|23.35|\n|Collecting iron|1.33|2.70|4.02|\n|Make iron pickaxe|0.01|0.00|0.01|\n|Make iron sword|0.00|0.00|0.02|\n\nHowever, we find that there is no substantial difference in the success rates for easy achievements, such as crafting wooden tools, as shown below.\n\n|Achievement|PPO + I|PPO+I+C|PPO+I+C+M|\n|---|---|---|---|\n|Make wood pickaxe|71.42|71.44|72.69|\n|Make wood sword|67.16|68.68|70.86|\n\nAdditionally, to gain deeper insight into how the cross-trajectory matching works, we conduct an oracle experiment suggested by Reviewer cpyF. In this experiment, we replace the cross-trajectory Wasserstein matching with exact matching using oracle achievement labels and compare its performance with our matching algorithm. Table 2 in the attached file presents the performance of oracle cross-trajectory matching. From this analysis, we anticipate a score increase of 1.8 when the cross-trajectory matching is optimized to its fullest potential. \n\n**Q3: It would be interesting if the authors could present the results in Table 1 and Figure 5 for a larger range of environment steps.**\n\nThank you for your valuable recommendation. We increase the number of environment steps to 10M and evaluate the performance of our method against three baseline: PPO, LSTM-SPCNN, and SEA. It is worth noting that training a DreamerV3 agent with 10M environment steps was not feasible within the constrained time of the rebuttal period. With a single NVIDIA RTX 3090 GPU, it requires approximately 2.5 days to complete just 1M environment steps. As illustrated in Figure 6 of the attached file, our method not only demonstrates higher sample efficiency but also outperforms the baselines with superior final performance.\n\n**Q4: In Figure 6, why the Individual success rate is not shown for MuZero + SPR?**\n\nWe first clarify that the results of MuZero + SPR in this paper have been derived from the original paper due to the unavailability of its source code. Additionally, the original paper only presents the individual success rate for a MuZero + SPR agent with pre-training using 150M exploratory data. Since our primary focus is on training an agent without pre-training, we have opted not to include MuZero + SPR in Figure 6. Afterwards, we plan to reproduce the results of MuZero + SPR and provide its individual success rate once the official source code becomes available.\n\n**Q5: Do you think your method would work with RL algorithms other than PPO?**\n\nThank you for bringing this to our attention. We evaluate our contrastive learning method alongside a popular off-policy value-based algorithm QR-DQN and observe its strong performance. For more details on the experimental settings and results, please refer to General Response 2.\n\n---\n\nOnce again, we really appreciate the reviewer's insightful questions, which greatly help us to to enhance our paper. We hope that our response above addresses all of the reviewer's questions."}, {"review_id": "Xrv4wEK9fs", "reviewer": "Reviewer_nAYF", "summary": "This paper introduces achievement distillation, a representation learning method that is combined with PPO to obtain state-of-the-art results on the 2D crafter benchmark. First, the authors demonstrate that with simple hyper-parameter tweaks, the performance of vanilla PPO can be greatly improved. Next, detail the three components of achievement distillation which leverage achievement labels $g$. Intra-trajectory achievement prediction uses a contrastive objective to maximize the similarity between state action pairs and the next achievement. Cross-trajectory achievement matching maximizes the similarity between state-action pairs of the same achievement across episodes using optimal transport. Finally, they use the achievement representations as memory by concatenting the last achievement representation to the policy and value inputs. This results in a method that achieves high performance on the Crafter benchmark, especially for difficult to reach achievements.\n", "questions": "Major:\nSince crafter is partially observed, do you use a recurrent network? \n\nMinor:\nLine 141: outperforms dreamer: for the same number of environment steps? I would expected model-based to have higher sample efficiency, though it may have lower asymptotic performance.\n", "limitations": "The work is heavily designed for the crafter benchmark. While this is impressive, it would be good if the authors could include a discussion of where achievement distillation would be useful beyond just this setting. Where is it inapplicable? What are the challenges in implementing this?\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper was generally easy to follow.\n\nThe experiments improving the performance of vanilla PPO were exciting! It’s great to see better tuned baselines. The improvements in PPO also naturally flowed into the introduction of achievement distillation. \n\nAchievement distillation is an interesting form of representation learning, and to my knowledge, is novel. \n\nThe authors conducted experiments only on the Crafter environment. The results in this benchmark are compelling as it is extremely challenging, and as the authors highlight, achievement distillation can perform better on very hard to achieve tasks. \n\nThe authors additionally include ablation studies by subtracting components of their method.\n\nI found the ideas in the paper to be easy to follow and straightforward in a good way. \n", "weaknesses": "The approach appears slightly overfit to the chosen Crafter benchmark, and the authors do not test achievement distillation in any other settings. Though the results on Crafter are compelling, it would be great if the authors could provide more examples and discussion of when this could be applied to other environments, and if so, how. \n\nThe comparison to baselines is a bit mis-leading, as other approaches, like DreamerV3, do not make use of the achievement labels necessary for achievement distillation. That being said, even PPO outperforms these baselines! I also found the log-scale on the axes of Figure 6 to be a bit confusing – is a success rate < 0.01% really significant?\n\nI generally recommend acceptance of this work, primary drawbacks being its limited applicability to broader settings and evaluation on only one environment. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. We are encouraged by the reviewer’s positive comments (“the experiments improving the performance of vanilla PPO were exciting”, “the ideas in the paper are easy to follow and straightforward in a good way”). We would like to address the questions raised by the reviewer, as presented below.\n\n---\n\n**Q1: More benchmarks. It would be great if the authors could provide more examples and discussion of when this could be applied to other environments.**\n\nThank you for your suggestion. We conduct experiments on two additional benchmarks that feature hierarchical achievements: Procgen Heist and a custom MiniGrid environment. Please refer to the Global Response Q1 for more details on the benchmarks and our experimental results.\n\n**Q2: DreamerV3 does not make use of the achievement labels.**\n\nWe first clarify that our method operates under the assumption that an agent has information only regarding when a new achievement is unlocked and does not utilize achievement labels, as detailed in Section 2.1 of our main paper. Additionally, this information can be easily retrieved from the reward signal, specifically at timesteps when rewards are 1. The sole distinction from DreamerV3 is our assumption that there exists an underlying hierarchical structure of achievements. We will explain our assumption and the difference between other baselines more clearly in the revised version.\n\n**Q3: I also found the log-scale on the axes of Figure 6 to be a bit confusing. Is a success rate < 0.01% really significant?**\n\nThank you for your helpful comment. We report the individual success rates of achievements using a log-scale, following the practice in DreamerV3 [1]. A success rate of an achievement under 0.01% implies that this is unlocked only in a subset of individual runs, also mentioned in the second paragraph in Section 5.2 of our main paper. Nevertheless, collecting a diamond in an individual run remains noteworthy, as it is particularly challenging due to the scarcity of resources. We will explain this more clearly in the revised version.  \n\n**Q4: Since crafter is partially observed, do you use a recurrent network?**\n\nIn this paper, we do not employ recurrent neural networks such as LSTM and only utilize convolutional neural networks for image processing. LSTM has been widely utilized in RL to address partial observability [2, 3]. However, it has been noted that employing LSTM for PPO on Crafter does not yield a significant performance improvement, resulting in a score increase of only 0.1 [4]. Incorporating our contrastive learning with memory-based policies would be a promising avenue for future research.\n\n**Q5: Does PPO outperform DreamerV3 for the same number of environment steps? I would expect model-based to have higher sample efficiency, though it may have lower asymptotic performance.**\n\nWe evaluate PPO and DreamerV3 with the same number of 1M environment steps and find that PPO outperforms DreamerV3. I agree with your opinion that model-based algorithms generally exhibit greater sample efficiency than model-free algorithms, provided the learned models are accurate.  However, in our attempt to reproduce the results of DreamerV3, we notice that the losses for training a world model (Equation 5 in the original paper) tend to increase as training progresses, leading to imprecise world models. Furthermore, we find a rapid drop in the policy entropy, reaching a level of 0.5 at timestep 50K and remaining low thereafter. This phenomenon hinders sufficient exploration for unlocking new achievements.\n\n---\n\nWe again thank the reviewer for providing constructive feedback, which truly enhances the quality of our paper. We hope that our response adequately addresses all the reviewer's questions.\n\n---\n\n**References**\n\n[1] Danijar Hafner et al., Mastering Diverse Domains through World Models, arXiv 2023. \\\n[2] Matthew Hausknecht and Peter Stone, Deep Recurrent Q-Learning for Partially Observable MDPs, arXiv 2015. \\\n[3] Steven Kapturowski et al., Recurrent Experience Replay in Distributed Reinforcement Learning, ICLR 2019. \\\n[4] Aleksandar Stanic et al., Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter, ToG 2023."}, {"review_id": "zstBkXFtjy", "reviewer": "Reviewer_cpyF", "summary": "This paper focuses on the problem of sequential decision-making within a hierarchical framework, where tasks exhibit a hierarchical decomposition structure, and the agent does not possess any prior knowledge of the task dependency graph. In contrast to prior hierarchical approaches that directly model dependencies and utilize two-level policies for task resolution, it investigates an even more demanding scenario. Specifically, the agent lacks information about the unlocked achievements within each episode.\n\nTo tackle this challenge, the paper employs PPO as its backbone RL algorithm. Interestingly, it is found that the representations learned by PPO possess some ability to predict the next locked achievement, albeit with limited confidence. To further enhance this predictive capability, the paper proposes two contrastive loss mechanisms as representation learning objectives. These two mechanisms help the representation to be able to predict the agent’s next unlocked achievement (intra-trajectory contrastive loss) and also learn a representation of the achievement so that it is not capturing any environment-specific or spurious features (cross-trajectory matching).\n", "questions": "Here are my questions about the paper. Given the limited amount of time for rebuttal, I understand that it may not be possible to address all of the points with comprehensive additional experiments. But I would be happy to raise my score if my following concerns are adequately addressed.\n \n1. **Assumptions of the problem setting**: The paper introduces an additional level of difficulty by assuming that the agent does not possess knowledge of the specific achievements unlocked throughout each episode. This assumption diverges from previous model-based and hierarchical approaches, which rely on such information. It would be helpful if the authors could provide a real-world example or scenario where relaxing this assumption becomes necessary. This would aid in understanding the practical applicability of such an assumption. From my perspective, this assumption appears weak, as it is difficult to envision an agent successfully completing an episode without awareness of the achievements attained during the process.\n\n2. **Backbone RL algorithm**: The paper extensively discusses the effectiveness of the Proximal Policy Optimization (PPO) algorithm as the backbone for addressing tasks with hierarchical structures. The authors demonstrate how the representations learned by the PPO agent exhibit predictive abilities for the next achievement, as outlined in sections 3.1 and 3.2. However, it would be valuable to clarify whether these findings are specific to policy gradient methods or if they can be extended to value-based approaches as well. Understanding if the proposed contrastive loss mechanisms can be applied to value-based reinforcement learning algorithms would contribute to a more comprehensive evaluation of their potential.\n\n3. **Baseline Comparison**: It would be insightful to compare the performance of the two proposed contrastive losses with an \"oracle\" objective, where the agent does possess prior knowledge of its unlocked achievements. For instance, for intra-trajectory achievement prediction, a simple 22-way classification could be employed as the loss, predicting the next achievement based on the current state-action pair. Similarly, for the cross-trajectory objective, minimizing the distance between representations of the same achievements could be considered. Such a comparison would facilitate a better understanding of the impact of relaxing the assumption regarding the agent's knowledge of achievements on the overall algorithm performance.\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The author proposes a novel self-supervised loss within the context of hierarchical decision-making, which could be easily integrated into the existing RL algorithm (PPO).\n2. Good empirical performance boost compared with the previous strongest model-based RL algorithms (Dreamer-v3) on the Crafter environment.\n", "weaknesses": "My concerns is mostly around how relaxing the assumption of the agent's knowledge of its unlocked achievement during an episode. In particular, whether it is realistic in real applications, and how much does this affect the overall performance (in other words comparison to a  baseline method that does assume such prior knowledge). Please see my detailed comments in the Questions section. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We appreciate the encouraging comments (“the author proposes a novel self-supervised loss within the context of hierarchical decision-making”, “good empirical performance boost compared with the previous strongest model-based RL algorithms”). We would like to address the questions and concerns of the reviewer, as presented below.\n\n---\n\n**Q1: Assumption of the problem setting appears weak.**\n\nWe wish to emphasize that our work focuses on developing an agent capable of discovering reusable skills and composing them to solve complex tasks in open-ended environments, as stated in the first paragraph of Section 1 in our main paper. We agree with your view that, for completing specific tasks with pre-defined subtask dependencies (e.g., making pasta with a detailed recipe), leveraging these dependency structures (e.g., providing the entire recipe) and incorporating subtask completion signals (e.g., acknowledging the completion of making the sauce) would be beneficial for training an agent [1, 2]. However, in open-ended environments where the final goal is not clearly defined and an agent faces the challenge of solving a myriad of tasks, pre-defining subtask dependencies for each task could be impractical. In such situations, an agent must explore environments, build a repertoire of skills, and combine these skills to solve tasks in an autonomous way.\n\nOne realistic example is the game of Minecraft, where players are not provided with explicit instructions for survival, such as building shelters or crafting weapons. Instead, they must find their own survival strategies through exploration. Due to the unlimited number of potential survival strategies, defining specific subtask dependencies or providing subtask completion signals becomes infeasible. In this context, it is reasonable to assume that an agent receives a reward if they discover a new survival strategy. This approach aligns neatly with our reward assumption where an agent is granted a reward of 1 when unlocking a new achievement. It is important to note that the objective of Crafter is not solely about collecting diamonds. There is no pre-defined end goal and an agent is rewarded as it continually discovers and develops new survival skills.\n\nThank you for raising this particular concern regarding the assumption. We appreciate your feedback and recognize the need to elucidate realistic scenarios where our assumption holds. We will revise the introduction section to provide a more in-depth explanation of the practical applicability of our assumption. Additionally, we will revisit lines 39 to 41 of the main paper and rewrite them to better articulate the potential limitations of the hierarchical approach in an open-ended world.\n\n**Q2: Application of contrastive learning to value-based methods.**\n\nThank you for bringing this to our attention. We evaluate our contrastive learning method alongside a popular off-policy value-based algorithm QR-DQN and observe its strong performance. For more details on the experimental settings and results, please refer to General Response Q2.\n\n**Q3: It would be insightful to compare the performance of the two proposed contrastive losses with an \"oracle\" objective.**\n \nThank you for your valuable suggestion. We train PPO agents with the oracle objectives on Crafter for 1M environment steps and compare their performance with our proposed contrastive objectives. Following your recommendation, we substitute the intra-trajectory contrastive prediction with a 23-way classification (22 achievements and 1 no achievement) using oracle labels. Additionally, we replace the cross-trajectory Wasserstein matching with the exact matching using oracle labels. To more precisely assess the impact of each objective, we choose not to employ the memory component, which is not necessarily required for implementing the intra- and cross-trajectory objectives.\n\nTables 1 and 2 of the attached file present the performance of the oracle intra- and cross-trajectory objectives, respectively. Notably, the oracle cross-trajectory matching outperforms our approach. When it comes to the intra-trajectory prediction, however, there is no significant difference between the oracle and our method. It is important to highlight that the achievement label itself does not contain detailed information about the achievement. For example, the label indicating crafting a stone pickaxe omits essential information regarding the agent's requirement for wood and stone, as well as proximity to a crafting table. We hypothesize that predicting the next achievement in the latent space of the encoder, whose representations may encompass richer information about object location and inventory states, provides additional benefit over predicting achievement labels.\n\n---\n\nThank you again for your constructive comments, which help us to improve our paper’s quality. We hope that our answers address all the reviewer's points.\n\n**References**\n\n[1] Sungryull Sohn et al., Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies, NeurIPS 2018. \\\n[2] Robby Costales et al., Possibility Before Utility: Learning And Using Hierarchical Affordances, ICLR 2022."}]

Author rebuttal:
We thank all the reviewers for their time and effort in providing valuable feedback. We are especially appreciative of the encouraging comments we received from each of them. To begin our response, we would like to first address some of the common concerns that have been raised by multiple reviewers.

---

**Q1: Application to other environments**

We conduct experiments on two additional benchmarks featuring hierarchical achievements: Procgen Heist [1] and a custom MiniGrid environment [2].

Heist is a procedurally generated “Door Key” environment where the goal is to steal a gem hidden behind a sequence of blue, green, and red locks, as illustrated in Figure 1 of the attached file. To open each lock, an agent must collect a key with the corresponding color. We consider opening each lock and stealing a gem as achievements. It is worth noting that Heist introduces another challenge, given that the color of wall and background can vary between environments, whereas Crafter maintains fixed color patterns for its terrains. Moreover, there is only a single pathway to unlock an achievement in Heist, while multiple routes exist to unlock an achievement in Crafter. For instance, an agent can readily gather wood almost everywhere on the map due to its abundance. To ensure closer alignment with Crafter, we slightly adjust the reward structure so that an agent receives a reward of 2 for opening each lock and a reward of 10 for successfully stealing a gem. We train an agent in the “hard” difficulty mode for 25M environment steps and evaluate its performance in terms of the success rate of gem pilfering and the episode reward.

Additionally, we create a customized "Door Key" environment using the MiniGrid library to assess the effectiveness of our method on a larger achievement graph. The design of this environment takes inspiration by TreeMaze proposed in SEA [3]. An agent must sequentially unlock doors, find keys, and ultimately reach the green square, as depicted in Figure 2 of the attached file. The environment comprises a total of 10 achievements, such as opening doors, collecting keys, and reaching the goal. An agent receives a reward of 1 for unlocking a new achievement, mirroring the reward structure in Crafter. We train an agent for 1M environment steps and evaluate its performance in terms of the geometric mean of success rates and the episode reward, following the same protocol as Crafter.

Figure 3 and 4 of the attached file illustrate the performance of our method and PPO. Remarkably, our method significantly enhances the performance of PPO in Heist, elevating the success rate from 29.6% to 71.0%. Furthermore, our method outperforms PPO in the MiniGrid environment by a substantial margin, increasing the score from 3.33% to 8.04%. These results highlight the broad applicability of our method to diverse environments with hierarchical achievements.

**Q2: Application to value-based RL algorithms**

We evaluate our contrastive learning method in conjunction with a popular off-policy value-based algorithm QR-DQN [4] on Crafter for 1M environment steps and observe its strong performance. Specifically, we apply our contrastive learning to the Q-network encoder at intervals of every 8000 environment steps. We employ Huber quantile regression to preserve the Q-network’s output distribution in a manner congruent with the value function optimization in QR-DQN. Figure 5 of the attached file demonstrates that our contrastive learning method is also effective in value-based algorithms, enhancing the performance of QR-DQN from 4.14 to 8.07.

---

We hope that our response addresses all the questions and concerns of the reviewers. Please let us know if there are any further questions.

---

**References**

[1] Karl Cobbe et al., Leveraging Procedural Generation to Benchmark Reinforcement Learning, ICML 2020. \
[2] Maxime Chevalier-Boisvert et al., Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks, arXiv 2023. \
[3] Zihan Zhou and Animesh Garg, Learning Achievement Structure for Structured Exploration in Domains with Sparse Reward, ICLR 2023. \
[4] Will Dabney et al., Distributional Reinforcement Learning with Quantile Regression, AAAI 2018.

Meta-review: This is a nice paper, and the reviews were all positive. I recommend accepting it.
Decision: Accept (poster)