Title: Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

hierarchical reinforcement learning

contrastive learning

procedurally generated environments
Abstract: Discovering achievements with a hierarchical structure in procedurally generated environments presents a significant challenge.
This requires an agent to possess a broad range of abilities, including generalization and long-term reasoning. Many prior methods have been built upon model-based or hierarchical approaches, with the belief that an explicit module for long-term planning would be advantageous for learning hierarchical dependencies. However, these methods demand an excessive number of environment interactions or large model sizes, limiting their practicality. In this work, we demonstrate that proximal policy optimization (PPO), a simple yet versatile model-free algorithm, outperforms previous methods when optimized with recent implementation practices. Moreover, we find that the PPO agent can predict the next achievement to be unlocked to some extent, albeit with limited confidence. Based on this observation, we introduce a novel contrastive learning method, called achievement distillation, which strengthens the agent's ability to predict the next achievement. Our method exhibits a strong capacity for discovering hierarchical achievements and shows state-of-the-art performance on the challenging Crafter environment in a sample-efficient manner while utilizing fewer model parameters.

Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning

Abstract

Discovering achievements with a hierarchical structure in procedurally generated environments presents a significant challenge. This requires an agent to possess a broad range of abilities, including generalization and long-term reasoning. Many prior methods have been built upon model-based or hierarchical approaches, with the belief that an explicit module for long-term planning would be advantageous for learning hierarchical dependencies. However, these methods demand an excessive number of environment interactions or large model sizes, limiting their practicality. In this work, we demonstrate that proximal policy optimization (PPO), a simple yet versatile model-free algorithm, outperforms previous methods when optimized with recent implementation practices. Moreover, we find that the PPO agent can predict the next achievement to be unlocked to some extent, albeit with limited confidence. Based on this observation, we introduce a novel contrastive learning method, called achievement distillation , which strengthens the agent's ability to predict the next achievement. Our method exhibits a strong capacity for discovering hierarchical achievements and shows state-of-the-art performance on the challenging Crafter environment in a sample-efficient manner while utilizing fewer model parameters.

1 Introduction

Deep reinforcement learning (RL) has recently achieved remarkable successes in solving challenging decision-making problems, including video games, board games, and robotic controls [31, 42, 18, 36]. However, these advancements are often restricted to a single deterministic environment with a narrow set of tasks. To successfully deploy RL agents in real-world scenarios, which are constantly changing and open-ended, they should generalize well to new unseen situations and acquire reusable skills for solving increasingly complex tasks via long-term reasoning. Unfortunately, many existing algorithms exhibit limitations in learning these abilities and tend to memorize action sequences rather than truly understand the underlying structures of the environments [28, 24].

To assess the abilities of agents in generalization and long-term reasoning, we focus on the problem of discovering hierarchical achievements in procedurally generated environments with high-dimensional image observations. In each episode, an agent navigates a previously unseen environment and receives a sparse reward upon accomplishing a novel subtask labeled as an achievement . Importantly, each achievement is semantically meaningful and can be reused to complete more complex achievements. Such a setting inherently demands strong generalization and long-term reasoning from the agent.

Previous work on this problem has mainly relied on model-based or hierarchical approaches, which involve explicit modules for long-term planning. Model-based methods employ a latent world model that predicts future states and rewards for learning long-term dependencies [20, 21, 1, 45]. While

these methods have shown effectiveness in discovering hierarchical achievements, particularly in procedurally generated environments, they are constructed with large model sizes and often require substantial exploratory data, which limits their practicality. Hierarchical methods aim to reconstruct the dependencies between achievements as a graph and employ a high-level planner on the graph to direct a low-level controller toward the next achievement to be unlocked [43, 10, 49]. However, these methods rely on prior knowledge of achievements ( e.g. , the number of achievements), which is impractical in open-world scenarios where the exact number of achievements cannot be predetermined. Additionally, they necessitate a significant number of offline expert data to reconstruct the graph.

To address these issues, we begin by exploring the potential of proximal policy optimization (PPO), a simple and flexible model-free algorithm, in discovering hierarchical achievements [39]. Surprisingly, PPO outperforms previous model-based and hierarchical methods by adopting recent implementing practices. Furthermore, upon analyzing the latent representations of the PPO agent, we observe that it has a certain degree of predictive ability regarding the next achievement, albeit with high uncertainty.

Based on this observation, we propose a novel self-supervised learning method alongside RL training, named achievement distillation . Our method periodically distills relevant information on achievements from episodes collected during policy updates to the encoder via contrastive learning [34]. Specifically, we maximize the similarity in the latent space between state-action pairs and the corresponding next achievements within a single episode. Additionally, by leveraging the uniform achievement structure across all environments, we maximize the similarity in the latent space between achievements from two different episodes, matching them using optimal transport [5]. This learning can be seamlessly integrated into PPO by introducing an auxiliary training phase. Our method demonstrates state-ofthe-art performance in discovering hierarchical achievements on the challenging Crafter benchmark, unlocking all 22 achievements with a budget of 1M environment steps while utilizing only 4% of the model parameters compared to the previous state-of-the-art method [19].

2 Preliminaries

2.1 Markov decision processes with hierarchical achievements

We formalize the problem using Markov decision processes (MDPs) with hierarchical achievements [49]. Let M represent a collection of such MDPs. Each environment M i ∈ M is defined by a tuple ( S i , A , G , p, r, ρ i , γ ) . Here, S i ⊂ S is the image observation space, which has visual variations across different environments, A is the action space, G is the achievement graph with a hierarchical structure, p : S × A → P ( S ) is the transition probability function, r : S × A × S → R is the achievement reward function, ρ i ∈ P ( S i ) is the initial state distribution, and γ ∈ [0 , 1] is the discount factor.

The achievement graph G = ( V , E ) is a directed acyclic graph, where each vertex v ∈ V represents an achievement and each edge ( u, v ) ∈ E indicates that achievement v has a dependency on achievement u . To unlock achievement v , all of its ancestors ( i.e. , achievements in the path from the root to v ) must also be unlocked. When an agent unlocks a new achievement, it receives an achievement reward of 1. Note that each achievement can be accomplished multiple times within a single episode, but the agent will only receive a reward when unlocking it for the first time. Specifically, let b ∈ { 0 , 1 } |V| be a binary vector indicating which achievements have been unlocked and c : S ×A×S → V ∪ {∅} be a function determining whether a transition tuple results in the completion of an achievement. Then, the achievement reward function is defined as

This reward structure provides an incentive for the agent to explore the environments and discover a new achievement, rather than repeatedly accomplishing the same achievements.

We assume that the agent has no prior knowledge of the achievement graph, including the number of achievements and their dependencies. Additionally, the agent has no direct access to information about which achievements have been unlocked. Instead, the agent must infer this information indirectly from the reward signal it receives. Given this situation, our objective is to learn a generalizable policy π : S → P ( A ) that maximizes the expected return ( i.e. , unlocks as many achievements as possible) across all environments of M .

(a) Agent's observation

Collect wood

Place table

Make wood pickaxe

Make wood sword

Collect coal

Collect stone

Place stone

Place furnace

Make stone pickaxe

Make stone sword

Collect iron

Make iron sword

Make iron pickaxe

Collect diamond

Collect sapling

Place plant

Eat plant

Defeat zombie

Defeat skeleton

Collect drink

Eat cow

Wake up

(b) Achievement graph with a hierarchical structure.

2.2 Crafter environment

We primarily utilize the Crafter environment as a benchmark to assess the capabilities of an agent in solving MDPs with hierarchical achievements [19]. Crafter is an open-world survival game with 2D visual inputs, drawing inspiration from the popular 3D game Minecraft [17]. This is optimized for research purposes, with fast and straightforward environment interactions and clear evaluation metrics. The game consists of procedurally generated environments with varying world map layouts, terrain types, resource placements, and enemy spawn locations, each of which is uniquely determined by an integer seed. An agent can only observe its immediate surroundings as depicted in Figure 1a, which makes Crafter partially observable and thus challenging. To survive, the agent must acquire a variety of skills, including exploring the world map, gathering resources, building tools, placing objects, and defending against enemies. The game features a set of 22 hierarchical achievements that the agent can unlock by completing specific prerequisites, as illustrated in Figure 1b. For instance, to make a wood pickaxe, the agent needs to collect wood, place a table, and stand nearby. This achievement structure is designed to require the agent to learn and utilize a wide range of skills to accomplish increasingly challenging achievements, such as crafting iron tools and collecting diamonds.

2.3 Proximal policy optimization

PPO is one of the most successful model-free policy gradient algorithms due to its simplicity and effectiveness [39]. PPO learns a policy π θ : S → P ( A ) and a value function V θ : S → R , which are parameterized by neural networks. During training, PPO first collects a new episodes T using the policy π θ old immediately prior to the update step. Subsequently, PPO updates the policy network using these episodes for several epochs to maximize the clipped surrogate policy objectives given by

where ˆ A t is the estimated advantage computed by generalized advantage estimate (GAE) [38]. PPO simultaneously updates the value network to minimize the value objective given by

where ˆ V t = ˆ A t + V θ old ( s t ) is the bootstrapped value function target.

In image-based RL, it is common practice to optimize the policy and value networks using a shared network architecture [13, 48]. An image observation is first passed through a convolutional encoder ϕ θ : S → R h to extract a state representation, which is then fed into linear heads to compute the policy and value function. Sharing state representations between the policy and value networks is crucial to improving the performance of agents in high-dimensional state spaces. However, relying solely on policy and value optimization to train the encoder can lead to suboptimal state representations, particularly in procedurally generated environments [9]. To address this issue, recent studies introduce an auxiliary training phase alongside the policy and value optimization that trains the encoder with auxiliary value or self-supervised objectives [9, 32].

3 Motivation

3.1 PPO is a strong baseline for hierarchical achievements

Despite being a simple and ubiquitous algorithm, PPO is less utilized than model-based or hierarchical approaches for solving MDPs with hierarchical achievements. This is due to the fact that PPO does not have an explicit component for long-term planning or reasoning, which is believed to be essential for solving hierarchical tasks. However, a recent study has shown that a PPO-based algorithm is also successful in solving hierarchical achievement on the Minecraft environment, albeit with the aid of pre-training on human video data [4].

Based on this observation, we first investigate the effectiveness of PPO in solving hierarchical achievements on Crafter without pretraining. We adopt the recent implementation practices proposed in Andrychowicz et al. [2], Baker et al. [4]. Concretely, we modify the default ResNet architecture in IMPALA as follows [14]:

Network size : We increase the channel size from [16, 32, 32] to [64, 128, 128] and the hidden size from 256 to 1024.

Layer normalization : We add layer normalization before each dense or convolutional layer [3].

Value normalization : We keep a moving average for the mean and standard deviation of the value function targets and update the value network to predict the normalized targets.

We train the modified PPO on Crafter for 1M environment steps and evaluate the success rates for unlocking achievements (please refer to Section 5.1 for the evaluation). Figure 2 shows that the slight modification in implementing PPO significantly improves the performance, increasing the score from 8.17 to 15.60. Notably, this outperforms the current state-of-the-art DreamerV3, which achieves a score of 14.77.

3.2 Representation analysis of PPO

Since Crafter environments are procedurally generated, simply memorizing successful episodes is insufficient for achieving high performance. We hypothesize that the PPO agent acquires knowledge beyond mere memorization of action sequences, possibly including information about achievements. To validate this, we analyze the learned latent representations of the encoder, as inspired by Wijmans et al. [46]. Specifically, we collect a batch of episodes using an expert policy and subsample a set of states for training. For each state s in the training set, we freeze its latent representation ϕ θ ( s ) from the encoder. Subsequently, we train a linear classifier using this representation as input to predict the very next achievement unlocked in the episode containing s . Finally, we evaluate the classification accuracy and the prediction confidence of the ground-truth labels on a held-out test set. The detailed experimental settings are provided in Appendix A.

Surprisingly, PPO achieves a nontrivial accuracy of 44.9% in the 22-way classification. However, Figure 3 shows that the prediction outputs lack confidence with a median value of 0.240. This suggests that the learned representations of the PPO encoder are not strongly correlated with the next achievement to be unlocked and the agent may struggle to generate optimal action sequences towards a specific goal.

This finding warrants providing additional guidance to the encoder for predicting the next achievements with high confidence. However, since the agent has no access to the achievement labels, it is challenging to guide the agent in a supervised fashion. Therefore, it is necessary to explore alternative approaches to guide the agent toward predicting the next achievements.

0

0

.

5

1

0

0.1

0.2

Confidence

Density

Ours

PPO

0

0.2

0.4

0.6

0.8

1.0

0

5

10

15

Environment steps (M)

Score (%)

Size

Default

<latexi sh

1_b

64="

TG

fKY

W

j

2

>A

n

c

VHLS

D

r

p

dug

w

R

y

J

m

I/

Qv

o

X

P

3

7

B

M

5

z

9

+

q

<latexi sh

1_b

64="q

QOuy

U3nw cJf

Y

FoXrE

>A

C

jVHLS

N

D2

p

Zd gk

R

K

m

W

T

I/

v

P

7

B

M

5

z

0

9

G

+

8

<latexi sh

1_b

64="

D

o

X

gZ5U

KE

A0v y8G

Q

>

C

n

cjVHLS

N

F

2

r

k

w

R

B9

M

/

3

T

O

P

Y

m

J

W

+

q

I

<latexi sh

1_b

64="5BU/u

C+RHJ

zjToSkE

FV

r9

>A

c

L

gM

D2O73f pZv

I

m

X

G

Y

8

P

d

Q

y

N

K

w

q

n

W

0

<latexi sh

1_b

64="7jmHvGApMF

KTWLXuy

0k oSU

>

C

3

c

V

N

D2

r

f

Zd

g

R

+

B

JO

5

P

/Q

8

Y

wn

I

E

9

z

q

<latexi sh

1_b

64="XSokyN2Pd

R/Bf

C+

rKQgjcI

>A

0

VHL

FD

p

Z

u

W

U

8

G5

7

q

M

TE

n9

v

w

z

m

3

Y

O

J

<latexi sh

1_b

64="XRJ

r

pjkw

W

G

g+ITFv fK

M

>A

CyH

c

V

LS

N

D2

Zdu

Y0nE

7

o

3

Q

/

B

U

z

q

O

m

P

5

9

8

<latexi sh

1_b

64="wpRJuV0SK

HUM

jd

Z

r

m

A

>

C

3

c

L

N

FD2

f

gk

y

+

B

O

9

8

E

zqn

/o

X

G

T

v

W

5

P

Y

I

Q

7

<latexi sh

1_b

64="KqRCAdmO/7F

p

0

J

3I

>

cjVHLS

N

D2

f

ugk

y

Ww

YU

TP

Qv

+

E

G

M

B

5

9

z

8

o

n

<latexi sh

1_b

64="AQ

UgOHJTWCqf

NP

u

L

2

k

>

n

cjV

S

FD

r

p

Zd

w

R

y

K

9

0

o

7

/

M

G

mX

v

Y

z

5

I

E

8

+

3

B

<latexi sh

1_b

64="

QjZLI0RNX

K

Y

+A8

cfd7

nSw

>

C

3

VH

FD

G

2vqk

o

U

m

T

p

u

r

y

g

E

z59

W

J

P

M

B

O

/

<latexi sh

1_b

64="7

90ujYwH+

fopKT

NOn

g

k

>A

C

c

V

LS

FD2

r

Zd

R

y

U

J

P/Q

M

G

mX

v

z

5

W

I

E

8

3

B

q

<latexi sh

1_b

64="7jmHvGApMF

KTWLXuy

0k oSU

>

C

3

c

V

N

D2

r

f

Zd

g

R

+

B

JO

5

P

/Q

8

Y

wn

I

E

9

z

q

<latexi sh

1_b

64="

9C+

/I2U

0

SjT

7okgAXp

>

yH

c

V

L

N

FD

r

f

Zdu

Q

JK

R

qY

W

v

E

m

G3

M

B

nO

P

w

z

8

5

<latexi sh

1_b

64="wpRJuV0SK

HUM

jd

Z

r

m

A

>

C

3

c

L

N

FD2

f

gk

y

+

B

O

9

8

E

zqn

/o

X

G

T

v

W

5

P

Y

I

Q

7

<latexi sh

1_b

64="AQ

UgOHJTWCqf

NP

u

L

2

k

>

n

cjV

S

FD

r

p

Zd

w

R

y

K

9

0

o

7

/

M

G

mX

v

Y

z

5

I

E

8

+

3

B

<latexi sh

1_b

64="KqRCAdmO/7F

p

0

J

3I

>

cjVHLS

N

D2

f

ugk

y

Ww

YU

TP

Qv

+

E

G

M

B

5

9

z

8

o

n

<latexi sh

1_b

64="

QjZLI0RNX

K

Y

+A8

cfd7

nSw

>

C

3

VH

FD

G

2vqk

o

U

m

T

p

u

r

y

g

E

z59

W

J

P

M

B

O

/

<latexi sh

1_b

64="fCoJNYv jDpU8A

Gdc

KS

g

>

y3

VHL

F

2

r

Z

u

k

0

n

7

QwT

z

/

+

M

E5

O

P

9

X

q

I

W

m

B

R

<latexi sh

1_b

64="

cLHw zPp/RNjV

g

DF

3

GQ

>A

Cyn

S

2

r

f

Zdu

k

K

q

X

J

8

I

EU

5

+

T

0

mo

7

M

v

Y

W

B

9

O

<latexi sh

1_b

64="w+rNUg

CZEXJ8

fz

O5

vkY

c

>A

n

jVHLS

FD2

p

du

R

y

BPq

K

3/o

Q

m

7

M

T

9

I

G

W

0

<latexi sh

1_b

64="mL

75GXY

2E+H0u vCN

D

g

c

>A

y

jV

T

J

F

3U

I

d

kR

W

8

o

f

p

B9zq n/QP

O

rZ

S

w

K

M

<latexi sh

1_b

64="o3fCLK/H

V2

cdqWU

XQzG

>A

y

j

T

J

FD

+I

m

kR

8

Y

u

N

g

E

n

9

S

5

7

0

rZ

w

O

v

p

M

P

B

<latexi sh

1_b

64="7jmHvGApMF

KTWLXuy

0k oSU

>

C

3

c

V

N

D2

r

f

Zd

g

R

+

B

JO

5

P

/Q

8

Y

wn

I

E

9

z

q

<latexi sh

1_b

64="X8S/N

Vu

TmqG

c

YZ

U

>A

3n

j

HL

F

d

B

p

0

2k

JO

v

o

Q

r

z

79

f

w5

+

M

g

R

K

P

W

E

<latexi sh

1_b

64="wpRJuV0SK

HUM

jd

Z

r

m

A

>

C

3

c

L

N

FD2

f

gk

y

+

B

O

9

8

E

zqn

/o

X

G

T

v

W

5

P

Y

I

Q

7

<latexi sh

1_b

64="

n

3zJQRr jEyg

mkT

/

d

o

M

>A

C0X

c

VHLS

N

FD2

pf

Z

u

W

5K

9

Y

I

P

+

vU7

G

w

B

8

Oq

<latexi sh

1_b

64="KqRCAdmO/7F

p

0

J

3I

>

cjVHLS

N

D2

f

ugk

y

Ww

YU

TP

Qv

+

E

G

M

B

5

9

z

8

o

n

<latexi sh

1_b

64="

QjZLI0RNX

K

Y

+A8

cfd7

nSw

>

C

3

VH

FD

G

2vqk

o

U

m

T

p

u

r

y

g

E

z59

W

J

P

M

B

O

/

<latexi sh

1_b

64="Szq8

X

dwk

/W

RZOEn

BK

j

Y

>A

C

c

VHL

N

FD2

r

pf

7

I

Q

Uo

v

y

0

T

3

Pu

9

5M

m

G

g

+

J

<latexi sh

1_b

64="

nqAvwE

cU

k

gjIC

/yr

9

7

>

H

V

LS

N

FD2

p

KW

YB

T

0

f

Q

JO

Z

o

Pu

8

5M

3

dX

R

G

z

m

+

<latexi sh

1_b

64="S

dHB

rw5

Io gM2ZJ

>A

C

X

c

V

L

Q

FD

T3+N

KUu

f

0

G

R

7

y

W

/

9

k

n

z

v

8

p

P

Y

<latexi sh

1_b

64="WV

kv

2g

Q5

S

IM

J

c

3Y

>A

C

n

j

HL

FD

Or/qu

z

K

0

B

m

Z

wy

N

d

Po

R

T

9

X

f

8

G

U

7

E

p

+

<latexi sh

1_b

64="3woUGC

LuvY9/

K

ym

DH

WA

>

X

cjV

S

N

F

2

r

pf

Zd gk

5

I

R

Pz

O

Q

J

n

8

+

M0

TB

7

E

q

<latexi sh

1_b

64="5j

W

m/QfOrAGFD

9u

8

dz

L

>

C

X

c

VH

S

N

2

p

UZ

gk

K

T

y

R

n

wq7

o

+

J

Pv

3

I

B

0

M

E

Y

0.05

0.29

0.19

0.38

0.23

0.10

0.33

0.15

0.42

0.58

0.25

0.06

Intra-trajectory achievement prediction

Cross-trajectory achievement matching

0.81

0.05

0.11

0.03

0.07

0.57

0.05

0.31

0.05

0.03

0.08

0.84

POT

Current state-action

Next achievement

Random state-action

Anchor

Positive

Negative

Anchor

Positive

Negative

Policy phase

Auxiliary phase

Cost matrix

Soft matching

T >

0

.

5

g

1

g

2

g

3

Hard matching

Buffer

ν

θ

(

g

+

t

)

ψ

θ

(

s

t

, a

t

, g

-

t

)

ψ

θ

(

s

t

′

, a

t

′

, g

-

t

′

)

ν

θ

(

g

1

)

Agent

g

-

t

g

′

1

g

′

2

g

′

3

g

′

4

g

′

1

g

′

2

g

′

3

g

′

4

g

′

1

g

′

2

g

′

3

g

′

4

s

t

a

t

g

+

t

g

-

t

′

s

t

′

a

t

′

ν

θ

(

g

′

1

)

ν

θ

(

g

′

3

)

(

r

t

, s

t

+1

)

Encoder

State

s

t

Policy

Value

Policy optimization

V

θ

(

s

t

)

π

θ

(

a

t

|

s

t

)

Value optimization

Store

episodes

Run

PPO

Buffer

Sample

episodes

Buffer

Sample

episodes

φ

θ

a

t

∼

π

θ

(

·

|

s

t

)

4 Contrastive learning for achievement distillation

In this section, we introduce a new self-supervised learning method that works alongside RL training to guide the encoder in predicting the next achievement to be unlocked. This approach distills relevant information about discovered achievements from episodes collected during multiple policy updates into the encoder via contrastive learning. This method consists of two key components:

Intra-trajectory achievement prediction : Within an episode, this maximizes the similarity in the latent space between a state-action pair and its corresponding next achievement.

Cross-trajectory achievement matching : Between episodes, this maximizes the similarity in the latent space for matched achievements.

For ease of notation, we denote the sequence of unlocked achievements within an episode as ( g i ) i m =1 and their corresponding timesteps as ( t i ) i m =1 , where each achievement g i is defined by a transition tuple ( s t i , a t i , s t i +1 ) . For each timestep t , we represent the very next achievement as g + t = g u , where u = min { i | t ≤ t i } , and the very previous achievement as g -t = g l , where l = max { i | t > t i } .

4.1 Intra-trajectory achievement prediction

Given a state-action pair ( s t , a t ) and its corresponding next achievement g + t within an episode τ , we train the encoder ϕ θ to produce similar representations for them through contrastive learning [34, 22]. Specifically, we regard g + t as the anchor and ( s t , a t ) as the positive. We also randomly sample another state-action pair ( s t ′ , a t ′ ) from the same episode to serve as the negative. Subsequently, we obtain the normalized representations of the anchor, positive, and negative, denoted as ν θ ( g + t ) , ψ θ ( s t , a t ) , and ψ θ ( s t ′ , a t ′ ) , respectively. Finally, we minimize the following contrastive loss to maximize the cosine similarity between the anchor and positive representations while minimizing the cosine similarity between the anchor and negative representations:

where λ > 0 is the temperature parameter.

To obtain the state-action representation ψ θ ( s t , a t ) , we calculate the latent representation of the state ϕ θ ( s t ) from the encoder and concatenate it with the action a t using a FiLM layer [35]. The resulting vector is then passed through an MLP layer and normalized. To obtain the achievement representation ν θ ( g + t ) , we simply calculate the residual of the latent representations of the two consecutive states from the encoder and normalize it, as motivated by Nair et al. [33].

While this contrastive objective encourages the encoder to predict the next achievement in the latent space, it can potentially lead to distortions in the policy and value networks due to the changes in the encoder. To address this, we jointly minimize the following regularizers to preserve the outputs of the policy and value networks, following the practice in Moon et al. [32]:

where π θ old and V θ old are the policy and value networks immediately prior to the contrastive learning, respectively and D KL denotes the KL divergence.

4.2 Cross-trajectory achievement matching

Since Crafter environments are procedurally generated, the achievement representations learned solely from intra-trajectory information may include environment-specific features that limit generalization. To obtain better achievement representations, we leverage the common achievement structure shared across all episodes.

Wefirst match the sequences of unlocked achievements from two different episodes in an unsupervised fashion. Given two achievement sequences g = ( g i ) i m =1 and g ′ = ( g ′ j ) n j =1 , we define the cost matrix M ∈ R m × n as the cosine distance between the achievement representations:

Subsequently, we regard these two sequences as discrete uniform distributions and compute a softmatching T ∈ R m × n between them using partial optimal transport, which can be solved by

where α > 0 is the entropic regularization parameter [5]. Here, we set the total amount of probability mass to be transported to the minimum length of the two sequences for simplicity. However, some unlocked achievements in one sequence may not exist in the other sequence, and therefore should not be transported. In this case, the optimal amount of probability mass to be transported should be less than the minimum length. To address this, we compute the conservative hard matching T ∗ from T by thresholding the probabilities less than 0.5 ( i.e. , T ∗ = ✶ [ T > 0 . 5] ). Note that this also encourages each achievement to be matched to at most one other achievement. We provide examples of matching results in Appendix B.

We train the encoder to produce similar representations for the matched achievements according to T ∗ through contrastive learning. Specifically, suppose that the i th achievement of the source sequence g i is matched with the k th achievement of the target sequence g ′ k . Then, we consider g i as the anchor and g ′ k as the positive. We also randomly sample another achievement g ′ j from the target sequence to serve as the negative. Subsequently, we obtain the normalized representations of these achievements. Finally, we minimize the following contrastive loss to maximize the cosine similarity between the anchor and positive representations while minimizing the cosine similarity between the anchor and negative representations:

As in Section 4.1, we jointly minimize the policy and value regularizers to prevent distortions.

4.3 Achievement representation as memory

We further utilize the achievement representations learned in Sections 4.1 and 4.2 as memory for the policy and value networks. Specifically, given a state s t and its corresponding previous achievement g -t , we concatenate the latent state representation ϕ θ ( s t ) with the previous achievement representation ν θ ( g -t ) . The resulting vector is then fed into the policy and value heads to output an action distribution and value estimate, respectively.

We also utilize the previous achievement for the achievement prediction task in Section 4.1. Given a state-action pair ( s t , a t ) and its corresponding previous achievement g -t , we concatenate the latent state representation ϕ θ ( s t ) from the encoder with a t and ν θ ( g -t ) . The resulting vector is then fed into an MLP layer and normalized to obtain the representation ψ θ ( s t , a t , g -t ) for the next achievement prediction. This is interpreted as learning forward dynamics in the achievement space.

4.4 Integration with RL training

We integrate the contrastive learning method proposed in Sections 4.1 to 4.3 with PPO training by introducing two alternating phases, the policy and auxiliary phases. During the policy phase, which is repeated several times, we update the policy and value networks using newly-collected episodes and store them in a buffer. During the auxiliary phase, we update the encoder to optimize the contrastive objectives in conjunction with the policy and value regularizers using all episodes in the buffer. We call this auxiliary learning achievement distillation . The illustration and pseudocode are presented in Figure 4 and Algorithm 1, respectively.

Algorithm 1 PPO with achievement distillation

5 Experiments

5.1 Experimental setup

To assess the effectiveness of our method in discovering hierarchical achievements, we train the agent on Crafter for 1M environment steps and evaluate its performance, following the protocol in Hafner [19]. We measure the success rates for all 22 achievements across all training episodes as a percentage and calculate their geometric mean to obtain our primary evaluation score 2 . Note that the geometric mean prioritizes unlocking challenging achievements. We also measure the episode reward, which indicates the number of achievements unlocked within a single episode, and report the average across all episodes within the most recent 100K environment steps. We conduct 10 independent runs using different random seeds for each experimental setting and report the mean and standard deviation. The code can be found at https://github.com/snu-mllab/Achievement-Distillation .

We compare our method with our backbone algorithm PPO and four baseline methods that have been previously evaluated on Crafter in other research work: DreamerV3, LSTM-SPCNN, MuZero + SPR, and SEA [21, 44, 45, 49]. DreamerV3 is a model-based algorithm that has achieved state-of-the-art performance on Crafter without any pre-training. LSTM-SPCNN is a model-free algorithm based on PPO that employs a recurrent and object-centric network to improve performance on Crafter. MuZero + SPR is a model-based algorithm that has demonstrated state-of-the-art performance on Crafter by utilizing unsupervised pre-training. SEA is a hierarchical algorithm based on IMPALA that employs

0

0.2

0.4

0.6

0.8

1.0

0

5

10

15

20

25

Environment steps (M)

Score (%)

0

0.2

0.4

0.6

0.8

1.0

0

4

8

12

16

Environment steps (M)

Reward

Ours

PPO

LSTM-SPCNN

a high-level planner to discover achievements on Crafter. We provide the implementation details and hyperparameters in Appendix C.

5.2 Crafter results

Table 1 and Figure 5 present the Crafter scores and rewards obtained by our method and the baselines. Our method outperforms all the baselines trained from scratch in both the metrics by a considerable margin, with a score of 21.79% and a reward of 12.60. Notably, our method exhibits superior score performance compared to MuZero + SPR, which utilizes pre-collected exploratory data and employs a computationally expensive tree search algorithm for planning, while achieving comparable rewards.

Figure 6 shows the individual success rates for all 22 achievements of our method and two successful baselines, DreamerV3 and LSTM-SPCNN. Remarkably, our method outperforms the baselines in unlocking challenging achievements. For instance, our method collects iron with a probability over 3%, which is 20 times higher than DreamerV3. This achievement is extremely challenging due to its scarcity on the map and the need for wood and stone tools. Moreover, our method crafts iron tools with a probability of approximately 0.01%, which is not achievable by either of the baselines. Finally, our method even succeeds in collecting diamonds, the most challenging task, on individual runs.

5.3 Model size analysis

We compare the model sizes between our method and the baselines. As shown in Table 1, our method achieves better performance with fewer parameters. In particular, our method only requires 4% of the parameters used by DreamerV3. Note that while our method has twice as many parameters as PPO, most of this increase is due to the networks used for the auxiliary learning, which are not utilized during inference. Additionally, we test our method with a smaller model by reducing the channel size to [16, 32, 32] and the hidden dimension to 256, resulting in a total of 1M parameters. Notably, it still outperforms the baselines with a score of 17.07%.

±

±

Collect coal

Collect diamond

Collect drink

Collect iron

Collect sapling

Collect stone

Collect wood

Defeat skeleton

Defeat zombie

Eat cow

Eat plant

Make iron pickaxe

Make iron sword

Make stone pickaxe

Make stone sword

Make wood pickaxe

Make wood sword

Place furnace

Place plant

Place stone

Place table

Wake up

0.001

0.01

0.1

1

10

100

Success rate (%)

Ours

LSTM-SPCNN

5.4 Representation analysis of achievement distillation

To validate whether our method induces the encoder to have better representations for predicting the next achievements, we conduct an analysis of the latent representations of the encoder using the same approach as described in Section 3. Our method achieves a classification accuracy of 73.6%, which is a 28.7%p increase compared to PPO. Furthermore, Figure 3 demonstrates that our method produces predictions with significantly higher confidence than PPO, with a median value of 0.752.

5.5 Ablation studies

We conduct ablation studies to evaluate the individual contribution of our proposed method, intra-trajectory achievement prediction (I), cross-trajectory achievement matching (C), and memory (M). Table 2 shows that while intra-trajectory achievement prediction is the most significant contributor, cross-trajectory achievement matching and memory also play important roles in improving the performance of our method.

5.6 Extension to value-based algorithms

In our previous experiments, we use the on-policy policy gradient algorithm, PPO, as our backbone RL algorithm. To assess the adaptability of our method to other RL paradigms, we extend our experiments to the popular off-policy value-based algorithm, QR-DQN, introducing a slight modification [11]. Specifically, we employ Huber quantile regression to preserve the Q-network's output distribution in alignment with the value function optimization in QR-DQN. We train the agent on Crafter for 1M environment steps and evaluate its performance. Notably, our method also proves effective for value-based algorithms, elevating the score from 4.14 to 8.07. The detailed experimental settings and results are provided in Appendix D.

5.7 Application to other environments

To evaluate the broad applicability of our method to diverse environments, we conduct experiments on two additional benchmarks featuring hierarchical achievements: Procgen Heist and MiniGrid [8, 7]. Heist is a procedurally generated environment whose goal is to steal a gem hidden behind a sequence of blue, green, and red locks. To open each lock, an agent must collect a key with the corresponding color. Heist introduces another challenge, given that the color of wall and background can vary between environments, whereas Crafter maintains fixed color patterns for its terrains. Additionally, we create a customized a door-key environment using MiniGrid to evaluate the effectiveness of our method on a deeper achievement graph. Notably, our method significantly improves the performance of PPO in Heist, increasing the score from 29.6 to 71.0. Our method also outperforms PPO in the MiniGrid environment by a substantial margin, elevating the score from 3.33 to 8.04. The detailed experimental settings and results can be found in Appendix E.

6 Related work

Discovering hierarchical achievements in RL One major approach to this problem is model-based algorithms. DreamerV3 learns a world model that predicts future states and rewards and trains an agent using imagined trajectories generated by the model [21]. While achieving superior performance on Crafter, it requires more than 200M parameters. MuZero + SPR trains a model-based agent with a self-supervised task of predicting future states [37, 45]. However, it relies on pre-training with 150M environment steps collected via RND to improve its performance on Crafter [6].

Another approach is hierarchical algorithms, while many of these methods are only tested on gridworld environments [43, 10]. HAL introduces a classifier that predicts the next achievement to be done and uses it as a high-level planner [10]. However, it relies on prior information on achievements, such as what achievement has been completed, and does not scale to the high-dimensional Crafter. SEA reconstructs the Crafter achievement graph using 200M offline data collected from a pre-trained IMPALA policy and employs a high-level planner on the graph [49]. However, it requires expert data to fully reconstruct the graph and has not been tested on a sample-efficient regime.

There are only a few studies that have explored model-free algorithms. LSTM-SPCNN uses an LSTM and a size-preserving CNN to improve the performance of PPO on Crafter [23, 26, 44]. However, it requires 135M parameters and the performance gain is modest compared to PPO with a CNN.

Representation learning in RL There has been a large body of work on representation learning for improving sample efficiency on a single environment [25, 47, 40], or generalization on procedurally generated environments [29, 30, 32]. However, representation learning for discovering hierarchical achievements has little been explored. A recent study has evaluated the performance of the previously developed representation learning technique SPR on Crafter, but the results are not promising [40, 45].

While widely used in other domains, optimal transport has recently garnered attention in RL [12, 15, 27]. However, the majority of the studies focus on imitation learning, where optimal transport is employed to match the behavior of an agent with offline expert data. In this work, we utilize optimal transport to obtain generalizable representations for achievements in the online setting.

7 Conclusion

In this work, we introduce a novel self-supervised method for discovering hierarchical achievements, named achievement distillation . This method distills relevant information about achievements from episodes collected during policy updates into the encoder and can be seamlessly integrated with a popular model-free algorithm PPO. We show that our proposed method is capable of discovering the hierarchical structure of achievements without any explicit component for long-term planning, achieving state-of-the-art performance on the Crafter benchmark using fewer parameters and data.

While we utilize only minimal information about hierarchical achievements ( i.e. , the agent receives a reward when a new achievement is unlocked), one limitation of our work is that we have not evaluated the transferability of our method to an unsupervised agent without any reward. A promising future direction would be developing a representation learning method that can distinguish achievements in a fully unsupervised manner and combining it with curiosity-driven exploration techniques [41, 16].

References

Ankesh Anand, Jacob C Walker, Yazhe Li, Eszter Vértes, Julian Schrittwieser, Sherjil Ozair, Theophane Weber, and Jessica B Hamrick. Procedural generalization by planning with selfsupervised world models. In ICLR , 2022.

Marcin Andrychowicz, Anton Raichuk, Piotr Sta´ nczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In ICLR , 2021.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. In NeurIPS , 2022.

Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing , 2015.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR , 2019.

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. arXiv preprint arXiv:2306.13831 , 2023.

Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In ICML , 2020.

Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In ICML , 2021.

Robby Costales, Shariq Iqbal, and Fei Sha. Possibility before utility: Learning and using hierarchical affordances. In ICLR , 2022.

Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI , 2018.

Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. In ICLR , 2021.

Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines , 2017.

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML , 2018.

Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation learning via optimal transport. In ICLR , 2022.

Zhaohan Guo, Shantanu Thakoor, Miruna Pîslar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, et al. Byol-explore: Exploration by bootstrapped prediction. In NeurIPS , 2022.

William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. MineRL: A large-scale dataset of Minecraft demonstrations. In IJCAI , 2019.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML , 2018.

Danijar Hafner. Benchmarking the spectrum of agent capabilities. In ICLR , 2022.

Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR , 2021.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR , 2019.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 1997.

Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research , 2023.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In ICML , 2020.

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurIPS , 2020.

Yicheng Luo, zhengyao jiang, Samuel Cohen, Edward Grefenstette, and Marc Peter Deisenroth. Optimal transport for offline imitation learning. In ICLR , 2023.

Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 2018.

Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. In NeurIPS , 2020.

Bogdan Mazoure, Ahmed M Ahmed, R Devon Hjelm, Andrey Kolobov, and Patrick MacAlpine. Cross-trajectory representation learning for zero-shot generalization in RL. In ICLR , 2022.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Seungyong Moon, JunYeong Lee, and Hyun Oh Song. Rethinking value function learning for generalization in reinforcement learning. In NeurIPS , 2022.

Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal induction from visual observations for goal directed tasks. arXiv preprint arXiv:1910.01751 , 2019.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI , 2018.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 2020.

Julian Schrittwieser, Thomas K Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. In NeurIPS , 2021.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. In ICLR , 2016.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In ICLR , 2021.

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML , 2020.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 2016.

Sungryull Sohn, Hyunjae Woo, Jongwook Choi, and Honglak Lee. Meta reinforcement learning with autonomous inference of subtask dependencies. In ICLR , 2020.

Aleksandar Stani´ c, Yujin Tang, David Ha, and Jürgen Schmidhuber. Learning to generalize with object-centric agents in the open world survival game crafter. arXiv preprint arXiv:2208.03374 , 2022.

Jacob Walker, Eszter Vértes, Yazhe Li, Gabriel Dulac-Arnold, Ankesh Anand, Théophane Weber, and Jessica B Hamrick. Investigating the role of model-based learning in exploration and transfer. arXiv preprint arXiv:2302.04009 , 2023.

Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. Emergence of maps in the memories of blind navigation agents. In ICLR , 2023.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In ICML , 2021.

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI , 2021.

Zihan Zhou and Animesh Garg. Learning achievement structure for structured exploration in domains with sparse reward. In ICLR , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of sequential decision-making within a hierarchical framework, where tasks exhibit a hierarchical decomposition structure, and the agent does not possess any prior knowledge of the task dependency graph. In contrast to prior hierarchical approaches that directly model dependencies and utilize two-level policies for task resolution, it investigates an even more demanding scenario. Specifically, the agent lacks information about the unlocked achievements within each episode.\n\nTo tackle this challenge, the paper employs PPO as its backbone RL algorithm. Interestingly, it is found that the representations learned by PPO possess some ability to predict the next locked achievement, albeit with limited confidence. To further enhance this predictive capability, the paper proposes two contrastive loss mechanisms as representation learning objectives. These two mechanisms help the representation to be able to predict the agent’s next unlocked achievement (intra-trajectory contrastive loss) and also learn a representation of the achievement so that it is not capturing any environment-specific or spurious features (cross-trajectory matching).\n\nSTRENGTHS:\n1. The author proposes a novel self-supervised loss within the context of hierarchical decision-making, which could be easily integrated into the existing RL algorithm (PPO).\n2. Good empirical performance boost compared with the previous strongest model-based RL algorithms (Dreamer-v3) on the Crafter environment.\n\nWEAKNESSES:\nMy concerns is mostly around how relaxing the assumption of the agent's knowledge of its unlocked achievement during an episode. In particular, whether it is realistic in real applications, and how much does this affect the overall performance (in other words comparison to a  baseline method that does assume such prior knowledge). Please see my detailed comments in the Questions section.\n\nQUESTIONS:\nHere are my questions about the paper. Given the limited amount of time for rebuttal, I understand that it may not be possible to address all of the points with comprehensive additional experiments. But I would be happy to raise my score if my following concerns are adequately addressed.\n \n1. **Assumptions of the problem setting**: The paper introduces an additional level of difficulty by assuming that the agent does not possess knowledge of the specific achievements unlocked throughout each episode. This assumption diverges from previous model-based and hierarchical approaches, which rely on such information. It would be helpful if the authors could provide a real-world example or scenario where relaxing this assumption becomes necessary. This would aid in understanding the practical applicability of such an assumption. From my perspective, this assumption appears weak, as it is difficult to envision an agent successfully completing an episode without awareness of the achievements attained during the process.\n\n2. **Backbone RL algorithm**: The paper extensively discusses the effectiveness of the Proximal Policy Optimization (PPO) algorithm as the backbone for addressing tasks with hierarchical structures. The authors demonstrate how the representations learned by the PPO agent exhibit predictive abilities for the next achievement, as outlined in sections 3.1 and 3.2. However, it would be valuable to clarify whether these findings are specific to policy gradient methods or if they can be extended to value-based approaches as well. Understanding if the proposed contrastive loss mechanisms can be applied to value-based reinforcement learning algorithms would contribute to a more comprehensive evaluation of their potential.\n\n3. **Baseline Comparison**: It would be insightful to compare the performance of the two proposed contrastive losses with an \"oracle\" objective, where the agent does possess prior knowledge of its unlocked achievements. For instance, for intra-trajectory achievement prediction, a simple 22-way classification could be employed as the loss, predicting the next achievement based on the current state-action pair. Similarly, for the cross-trajectory objective, minimizing the distance between representations of the same achievements could be considered. Such a comparison would facilitate a better understanding of the impact of relaxing the assumption regarding the agent's knowledge of achievements on the overall algorithm performance.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces achievement distillation, a representation learning method that is combined with PPO to obtain state-of-the-art results on the 2D crafter benchmark. First, the authors demonstrate that with simple hyper-parameter tweaks, the performance of vanilla PPO can be greatly improved. Next, detail the three components of achievement distillation which leverage achievement labels $g$. Intra-trajectory achievement prediction uses a contrastive objective to maximize the similarity between state action pairs and the next achievement. Cross-trajectory achievement matching maximizes the similarity between state-action pairs of the same achievement across episodes using optimal transport. Finally, they use the achievement representations as memory by concatenting the last achievement representation to the policy and value inputs. This results in a method that achieves high performance on the Crafter benchmark, especially for difficult to reach achievements.\n\nSTRENGTHS:\nThe paper was generally easy to follow.\n\nThe experiments improving the performance of vanilla PPO were exciting! It’s great to see better tuned baselines. The improvements in PPO also naturally flowed into the introduction of achievement distillation. \n\nAchievement distillation is an interesting form of representation learning, and to my knowledge, is novel. \n\nThe authors conducted experiments only on the Crafter environment. The results in this benchmark are compelling as it is extremely challenging, and as the authors highlight, achievement distillation can perform better on very hard to achieve tasks. \n\nThe authors additionally include ablation studies by subtracting components of their method.\n\nI found the ideas in the paper to be easy to follow and straightforward in a good way.\n\nWEAKNESSES:\nThe approach appears slightly overfit to the chosen Crafter benchmark, and the authors do not test achievement distillation in any other settings. Though the results on Crafter are compelling, it would be great if the authors could provide more examples and discussion of when this could be applied to other environments, and if so, how. \n\nThe comparison to baselines is a bit mis-leading, as other approaches, like DreamerV3, do not make use of the achievement labels necessary for achievement distillation. That being said, even PPO outperforms these baselines! I also found the log-scale on the axes of Figure 6 to be a bit confusing – is a success rate < 0.01% really significant?\n\nI generally recommend acceptance of this work, primary drawbacks being its limited applicability to broader settings and evaluation on only one environment.\n\nQUESTIONS:\nMajor:\nSince crafter is partially observed, do you use a recurrent network? \n\nMinor:\nLine 141: outperforms dreamer: for the same number of environment steps? I would expected model-based to have higher sample efficiency, though it may have lower asymptotic performance.\n\nLIMITATIONS:\nThe work is heavily designed for the crafter benchmark. While this is impressive, it would be good if the authors could include a discussion of where achievement distillation would be useful beyond just this setting. Where is it inapplicable? What are the challenges in implementing this?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces achievement distillation, a model-free RL method designed to discover achievements without the need for explicit long-term planning components. The proposed method comprises three primary components: two self-supervised tasks and a memory component. The self-supervised tasks, namely Intra-trajectory achievement prediction and cross-trajectory achievement matching, guide the encoder to predict the next achievement to be unlocked using a contrastive learning loss function with different objectives. The memory component is formed by concatenating the latent state representation from the encoder with the action and the representation of the previous achievement. These components are integrated into the PPO algorithm through two alternating training phases involving policy training and auxiliary self-supervised tasks. The effectiveness of achievement distillation is evaluated in the Crafter environment, where it demonstrates significant performance improvements over strong baselines. Additionally, the authors conducted analyses on model sizes, representations, and the contribution of individual components.\n\nSTRENGTHS:\n1. The primary strength of the paper lies in the significant results it presents. The main results in Table 1 and Figure 5 provide clear evidence that the proposed achievement distillation method outperforms the baseline methods to a substantial degree. Furthermore, the results in Figure 6 demonstrate that achievement distillation achieves success in several achievements that none of the baselines can accomplish (e.g., making an iron sword), which is impressive.\n2. The authors have done a great job in implementing various relevant strong baselines. Reproducing these baselines are non-trivial given the complexity of them.\n3. While the concept of self-supervised auxiliary tasks and the used self-supervised losses are not novel ideas, the authors have managed to execute them very well, resulting in an agent with remarkable performance.\n4. The paper is well-written, and the ideas are effectively presented and justified.\n\nWEAKNESSES:\n1. The primary weakness of this work lies in the limited range of environments in which the proposed method is evaluated. Since the authors only tested it on the Crafter environment, it remains uncertain whether their method would be effective in different settings and whether it has avoided overfitting to the Crafter environment.\n\n2. The ablation study appears to be somewhat superficial. Although the contribution of cross-trajectory achievement matching is evident, the significance of cross-trajectory achieving matching and memory is not as convincing. The authors should provide additional evidence to support the role of these components. Please refer to the next section for suggestions on how to improve this aspect.\n\nQUESTIONS:\n1. I would encourage the authors to address the main limitation of their work discussed above by conducting further experimentation in various environments, such as MiniHack and maybe ProcGen.\n\n2. It would greatly enhance the understanding of the contribution of cross-trajectory achieving matching and memory if the authors could conduct additional research. For instance, including the individual success rates for all achievements (Figure 6) while ablating these components could be a valuable option.\n\n3. It would be interesting if the authors could present the results in Table 1 and Figure 5 for a larger range of environment steps (e.g., 5M, 10M). This would clarify whether this method is only more data efficient or remains superior with more steps.\n\n4. In Figure 6, why the Individual success rate is not shown for MuZero+SPR?\n\n5. Do you think your method would work with RL algorithms other than PPO?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a contrastive learning approach for representation learning in the problem of hierarchical achievement discovery. The proposed method leverages previous contrastive learning loss and combines it with optimal transport. Empirical results show that the learned representation could improve PPO in the Crafter environment.\n\nSTRENGTHS:\n1.\tIt is interesting that self-supervised representation learning can improve PPO, which significantly outperforms model-based approaches, even regarding sample efficiency. \n2.\tExperimental results show the strong performance of the proposed method in the Crafter environment. \n3.\tPresentation: this paper is generally well-organized and easy to follow.\n\nWEAKNESSES:\n1.\tDomain knowledge: unlike baselines, the proposed method utilizes additional important knowledge, i.e., identifying when a new achievement is unlocked (which can be easily inferred from the observed rewards). This other knowledge can easily explain why the achievement prediction of the proposed method is much better than PPO. \n2.\tGeneralization: this paper only shows its results in the Crafter environment. It is highly recommended to conduct experiments in other environments to show its generality.\n3.\tNovelty: though it is interesting to use optimal transport for matching achievements, the proposed method is a natural application of contrastive learning, which is not quite novel.\n\nQUESTIONS:\n1.\tThis paper defines MDPs with hierarchical achievements. Has prior work studied this model? If so, please add references.\n2.\tIt is interesting to see PPO is more sample-efficient than model-based methods in this paper’s experiments. Except for implementation practice improvement of PPO, any other insights for this result?\n\nLIMITATIONS:\nThis paper has discussed its limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}