Title: CQM: Curriculum Reinforcement Learning with a Quantized World Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Curriculum Learning

Goal-conditioned RL
Abstract: Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate curriculum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized-variational autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-aware curriculum goals that converges to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations in an uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, in various goal-reaching tasks even with ego-centric visual inputs.

CQM: Curriculum Reinforcement Learning with a Quantized World Model

Abstract

Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate curriculum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized-variational autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-aware curriculum goals that converges to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations in an uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, in various goal-reaching tasks even with ego-centric visual inputs.

1 Introduction

Goal-conditioned Reinforcement Learning (RL) has been successfully applied to a wide range of decision-making problems allowing RL agents to achieve diverse control tasks [42, 1, 30]. However, training the RL agent to achieve desired final goals without any prior domain knowledge is challenging, especially when the desired behaviors can hardly be observed. In those situations, humans typically adopt alternative ways to learn the final goals by gradually mastering intermediate sub-tasks. Inspired by the way humans learn, recent RL studies [29, 10, 6] have solved uninformed exploration tasks by suggesting which goals the agent needs to practice. In this sense of generating curriculum goals, previous approaches proposed various ideas to involve providing intermediate-level tasks [10, 38], quantifying the uncertainty of observations [4, 31, 33, 25, 18], or proposing contextual distance to gradually move away from the initial distribution [15, 6].

However, previous curriculum RL studies are mostly not scalable. Namely, they suffer from serious data inefficiency when they generate curriculum goals in high dimensions. Because of this limitation, they usually rely on the assumption that manually specified goal spaces (e.g., global X-Y coordinates) and clear mappings from high-dimensional observations to the low-dimensional goal spaces are available. Such an assumption requires prior knowledge about observations and the tasks, which remains a crucial unsolved issue that restricts the applicability of previous studies.

In order to design a general curriculum solution without the need for prior knowledge about the observations, defining its own goal space for the curriculum could be an effective scheme. To do so,

TrainVQ-VAE

SpecifyingGoal Space

SuggestingCurriculumGoals

Discretization

Codebook

Bottleneck

C2

Replay

中

Reconstructed

Buffer

Encoder

Decoder

Observation

L={l,l2.….lk}

Codebook

Initial Distribution

e1

Temporal Distance from po

e2

ek

Restore Relations

Close

Uncertainty

Far

Between Landmarks

More Uncertain

the two following operations need to be executed concurrently. (1) composing the semantic goal space which contains vital information for the curriculum process from the arbitrary observation space, and (2) suggesting to the agent which goal to practice over the goal space. Let us consider an agent that tries to explore an uninformed environment with final goal images only. To succeed, the agent needs to specify the semantic goal space from the high-dimensional observation space, and suggest the curriculum goals (e.g., the frontier of the explored area) over the composed goal space to search the uninformed environment. However, most previous studies focused solely on one of these, specifying the low-dimensional goal space without considering how to provide intermediate levels of goals [17, 27], or just suggesting curriculum goals in manually specified semantic goal spaces [10, 21, 6].

The challenge of simultaneously managing (1) specifying the goal space and (2) suggesting curriculum goals is that they are intimately connected to each other. If the agent constructs an ill-formed goal space from the observation space, it would be difficult to propose curriculum goals over it. Conversely, if the method fails to suggest goals to enable the agents to explore the unseen area, it would also be difficult to automatically learn the goal space that covers the uninformed environment based on the accumulated observations. Therefore, it is essential to develop an algorithm that addresses both defining goal space and providing curriculum goals concurrently.

In this paper, we propose a novel curriculum reinforcement learning (RL) method which can provide a general solution for a final goal-directed curriculum without the need for prior knowledge about the environments, observations, and goal spaces. First, our method defines its own semantic goal space by quantizing the encoded observations space through a discretization bottleneck and restoring the temporal relations between discrete goals via a graph. Second, to suggest calibrated guidance towards unexplored areas and the final goals, ours proposes uncertainty and temporal distance-aware curriculum goals that converge to the final goal examples.

The key contributions of our work (CQM: C urriculum RL with Q uantized World M odel) are:

CQMsolves general exploration tasks with the desired examples only, by simultaneously addressing the specification of a goal space and suggestion of curriculum goals (Figure 3).

CQMis the first curriculum RL approach that can propose calibrated curriculums toward final goals from high-dimensional observations, to the best of our knowledge.

CQM is the only curriculum RL method that demonstrates reliable performance despite an increase in the problem dimension, among the 10 methods that we experimented with. (Even state-based → vision-based)

Ours significantly outperforms the state-of-the-art curriculum RL methods on various goal reaching tasks in the absence of a manually specified goal space.

2 Related Works

Curriculum Goal Generation. Although various prior studies [41, 19, 48, 8, 45, 22] have been proposed to solve exploration problems, enabling efficient searching in uninformed environments still

remains a challenge. An effective way to succeed in such tasks with hardly observed final goals is identifying uncertain areas and instructing an agent to achieve the goals in these areas. To identify the uncertain areas and provide the goals sampled from them, previous studies employ uncertaintybased curriculum guidance by the state visitation counts [4, 31], absolute reward difference [37], and prediction model [32, 5]. Other approaches propose to utilize disagreements of ensembles[33, 50, 28] or sample the tasks with high TD errors [18] to generate goals in uncertain areas. An alternative way for solving the exploration tasks is to execute a final goal-directed exploration to propose tailored guidance. To this end, some studies perform successful example-based approaches [25, 6] or propose to minimize the distance between the final goals and curriculum goals [38, 21], measuring it by the Euclidean distance metric. Some studies also employ contextual distance-based metrics to perform final goal-directed exploration away from the initial distribution [15, 6].

However, these methods usually assume that agents have prior knowledge about the observations and unrestricted access to manually specified semantic goal space (e.g. global X-Y coordinates) because they are not scalable to handle the high-dimensional goal spaces. For example, the meta-learning classifierbased uncertainty metrics [25, 6] suffer from distinguishing uncertain areas as the dimension of the goal space increases. Also, some of the methods rely on Euclidean distance metric [38, 21] over the goal space. Moreover, generating curriculum goals [10], employing various prediction models [32, 5, 33, 50], fitting Gaussian mixture models [37], and utilizing disagreements of ensembles-based methods [33, 50] also face difficulty in solving increasingly complex

PointSpiralMaze

PointNMaze

1.0

1.0

_rate

0.8

0.8

success_

0.6

9°0

0.4

0.4

0.2

0°0

口

250

500

750

steps (k)

1000

1250

1500

17502000

0.0

200

steps (k)

400

600

800

000T

1.0

AntUMaze

==OUTPACE[6]

W/Manually

_rate

0.8

==-CURROT[21]

-ALP-GMM[37]-

Specified

success_

0.6

OUTPACE[6]

W/OManually

0.4

CURROT[21]

Specified

0.2

ALP-GMM[37]

0'0

500

1000

steps(k)

1500

2000

2500

000E

Performance Gap

problems in high-dimensional goal spaces. Although there have been attempts to propose a curriculum in high-dimensional observations [36, 13] or include an encoder in their model-based agent [28, 14], unfortunately, these approaches do not incorporate a convergence mechanism to the final goals, which are crucial for efficient curriculum progresses.

Our method incorporates the benefits of the aforementioned methods without manually specified goal spaces: exploring uncertain areas and moving away from the initial distribution while converging to the desired outcome . Although there is a study that has also incorporated these three aspects [6], it retains its performance only in manually specified goal spaces, as other curriculum methods (Figure 2). (We included conceptual comparisons between CQM and more related works in Table 1 in Appendix B.)

Discretizing Goal Space for RL. Vector quantized-variational autoencoder (VQ-VAE) is an autoencoder that learns a discretized representation using a learnable codebook. The use of this discretization technique for learning discrete representations in RL is a recent research topic [17, 27] and has shown an improved sample efficiency. Islam et al. [17] proposes to apply VQ-VAE as a discretization bottleneck in a goal-conditioned RL framework and demonstrates the efficiency of representing the continuous observation spaces into discretized goal space. Also, Mazzaglia et al. [27] utilizes VQ-VAE to discover skills in model-based RL by maximizing the mutual information between skills and trajectories of model states.

Unfortunately, the aforementioned methods require a pre-collected dataset or extra exploration policy, which are not necessary in CQM. Training VQ-VAE with a pre-collected dataset implies that the agent has access to the full information about the task or that it already possesses an agent capable of performing the task well. Although it is possible to obtain a pre-collected dataset through a random rollout policy, this is only in the case where exploring the environments is easy enough to succeed with only random actions.

3 Preliminaries

We consider a Markov Decision Process (MDP) which can be represented as a tuple ( O , A , T , R , ρ 0 , γ ) , where O is an observation space, A is an action space, T ( o t +1 | o t , a t ) is a transition function, ρ 0 is an initial distribution, and γ is a discount factor. Note that the MDP above

does not contain a goal space, since we do not assume that the manually specified goal space is provided. Instead, we consider a discrete low-dimensional discrete goal space G , which is defined by the agent automatically. Also, we assume that the final goal examples o f are provided by the environment, and we denote the projection of these examples into the goal space G as g f . We represent curriculum goal as g c , which is sampled from the goal space G , and the reward function in the tuple can be represented as R : O × A × G → R . Furthermore, we denote actor network as π : O × G → A , and critic network as Q : O × A × G → R . (Thus, Q ( o, a, g ) indicates the goal-conditioned state-action value where goal g , action a , and observation o throughout this paper.)

4 Method

In order to provide a general solution for efficient curriculum learning, our method defines its own goal space and suggests to the agent which goal to practice over the goal space simultaneously. To compose a semantic goal space which reduces the complexity of observation space while preserving vital information for the curriculum process, we first quantize the continuous observation space using a discretization bottleneck (section 4.1) and restore temporal relations in discretized world model via graph (section 4.2). Over the automatically specified semantic goal space, we generate a curriculum goal and guide the agent toward achieving it (section 4.3).

4.1 Specifying Goal Space via VQ-VAE

In order to define a discrete low-dimensional goal space which allows a scalable curriculum with high-dimensional observations, we utilize VQ-VAE as a discretization bottleneck [46, 40, 47] as recently been proposed [17, 27]. VQ-VAE utilizes a codebook composed of k trainable embedding vectors (codes) e i ∈ R D , i ∈ 1 , 2 , ...k , combined with nearest neighbor search to learn discrete representations. The quantization process of an observation o t starts with passing o t through the encoder ϕ . The resulting encoded vector z e = ϕ ( o t ) is then mapped to an embedding vector in the codebook by the nearest neighbor look-up as

The discretized vector z q is then reconstructed into ˆ o t = ψ ( z q ) by passing through the decoder ψ . We closely follow [17] and train quantizer, encoder, and decoder using a vector quantization loss with a simple reconstruction loss. The first term in Eq. 2 represents the reconstruction loss, while the second term represents the VQ objective that moves the embedding vector e towards the encoder's output z e . We update the embedding vectors e using a moving average instead of a direct gradient update [17, 27]. The last term is the commitment loss, and we use the same λ commit ( = 0 . 25 ) across all experiments.

By utilizing VQ-VAE, the RL agent can specify a quantized goal space that consists of discrete landmarks L = { l 1 , l 2 , · · · l m } in two ways. The first approach is to obtain each landmark by decoding each code as l j = ψ ( e j ) . Alternatively, one can obtain the landmarks by passing (encoding, quantizing to the closest embedding vector, and decoding) the continuous observations sampled from the replay buffer through the VQ-VAE. We utilize the first approach as the default, and provide the ablation study to examine the effectiveness of the second approach.

It should be noted that this set of landmarks L = { l 1 , l 2 , · · · l m } only represents discretized observations and does not involve relations among the observations. We describe our approach that can better restore the temporal relations between landmarks in the next section.

4.2 Graph Construction over Quantized Goal Space

In this section, we present the graph construction technique over the quantized goal space to allow the agent to capture the temporal information between the landmarks of the quantized world model. We consider a graph G = ( V , E ) over the goal space where the vertices V represent the landmarks

L = { l 1 , l 2 , · · · l m } obtained from decoding discrete codes of VQ-VAE, and the edges E represent the temporal distance. We utilize Q-value to reconstruct the edge costs, following the method proposed in [49, 24]. If an agent receives a reward of 0 when reaching a goal and -1 otherwise, the timesteps required to travel between landmarks can be estimated using Q-value as (derivation: Appendix A)

Using Eq. 3, we connect vertices with the distance below the cutoff threshold, and the resulting graph restores the temporal relations between the landmarks over a discretized goal space based on the temporal distance. In this way, the agent can calculate geodesic distances between landmarks,

which enables better prediction of temporal distances in environments with arbitrary geometric structures. Also, to incorporate extended supports of the explored area into the graph by creating landmarks in newly explored areas, we periodically reconstructed the graph following [24].

4.3 Uncertainty and Temporal Distance-Aware Curriculum Goal Generation

In the previous sections, we proposed a method for specifying a discretized goal space with semantic information. It is important to provide curriculum goals located in the frontier part of the explored area to expand the graph effectively toward the final goal in an uninformed environment. To achieve this objective, we propose an uncertainty and temporal distance-aware curriculum goal generation method.

To obtain a curriculum goal from graph G = ( V , E ) over the specified goal space, our method samples the landmarks that are considered uncertain and temporally distant from the initial distribution ρ 0 . Thanks to the quantized world model, quantifying uncertainty in a countable goal space is straightforward and computationally light. We quantify the count-based uncertainty of each landmark as η ucert ( l i ) = 1 / ( µ ( l i ) + ϵ ) , based on the empirical distribution µ ( l i ) derived from the recent observations as

where N ( l i ) indicates the number of occurrences of landmark l i in recent episodes and is periodically re-initialized when the graph is reconstructed with a new set of landmarks.

Finally, we deliver the sampled landmarks as curriculum goals to the agent, considering both temporal distance and uncertainty aspects:

where u i is a uniform random variable between 0 and 1 used to perform weighted sampling, and L top -k represents a subset of L that includes the top-k elements with the largest TemporalDist G (Eq. 4) values from initial state distribution. Our method, based on the uncertainty and temporal distance-aware objective (Eq. 6), is capable of providing calibrated curriculum guidance to the agent even in environments with arbitrary geometric structures, without requiring prior knowledge of the environment or a manually specified goal space. Furthermore, the curriculum guidance makes composing the goal space easier, illuminating the unexplored areas and vice versa.

Convergence to the final goal. The curriculum objective in Eq.6 provides a calibrated curriculum towards unexplored areas. In addition to this frontier-directed method, providing final goal-directed guidance can further improve the efficiency of exploration especially when the agent sufficiently explored the environment, i.e., the supports of the final goal and explored area start to overlap [35, 6]. In order to acquire the capability to propose a final goal-directed curriculum, we gradually shift the direction of exploration from the frontier of the explored area to the final goal distribution. To do so,

AntUMaze

Ant2WayMaze

DHRL+

CQM (Ours)

OUTPACE

CURROT

Final Goal

PointNMaze

Initial Distribution

Initial

Curriculum

Initial Distribution

Final Goal

Learning Progress

Curriculum Goals

Changes in

as learning progresses

Final

Curriculum

Initial

Curriculum

Final

Curriculum

Initial

Curriculum

Final

Curriculum

Initial

Curriculum

Final

Curriculum

Curriculum Goals

Curriculum Goals

Curriculum Goals

11

we determine whether to provide the curriculum goal g c ∈ G that is sampled via Eq. 6 or the final goal g f = ϕ ( o f ) ∈ G which the environment originally provided ( ψ ( g c ) ∈ O , o f ∈ O ).

We utilize a mixture distribution of curriculum goals, following the approach proposed in [35],

where p g f is the distribution of the final goal, and p g c is the distribution of curriculum goals. The mixture ratio α measures whether the achieved goal distribution p ag 'covers' the final goal distribution using KL divergence as α = 1 / max ( β + κD KL ( p g f || p ag ) , 1 ) . When the support of achieved goal distribution p ag (= visited state distribution) covers that of the final goal distribution p g f , α produces a value close to 1, and a value close to 0 when the supports of both distributions are not connected.

By combining the curriculum goal objective (Eq. 6) with the mixture strategy (Eq. 7), our approach generates instructive curriculum goals towards unexplored areas and provides the curriculum goals g c ′ to the agent that 'cover' the final goal distribution at the appropriate time when the agent is capable of achieving the final goal.

Planning over the graph As presented above, CQM constructs a graph to restore the temporal relations between landmarks (Section 4.2) and utilizes it to calculate geodesic distances (Eq. 4). In addition to these benefits, we highlight that the graph can also provide the strength of planning, which allows the agent to reason over long horizons effectively [9, 16, 13, 49, 3, 24].

To generate a sequence of waypoints for achieving each goal, we perform shortest path planning (Dijkstra's Algorithm), following the details proposed in the previous graph-guided RL method [24]. Consider a task of reaching a curriculum goal g c ∈ G from the current observation o 0 ∈ O . CQM first adds the encoded observation ϕ ( o 0 ) to the existing graph structure. Then, it finds the shortest path between the curriculum goal and current observation to return a sequence of waypoints ( ϕ ( o 0 ) , w 1 , ..., w n , g c ) where n indicates the number of waypoints in the shortest path. Finally, the agent is guided to achieve each decoded waypoint ψ ( w i ) during TemporalDist( ψ ( w i -1 ) → ψ ( w i )) (Eq. 3) timesteps, rather than achieving the curriculum goal directly. In other words, the RL agent produces goal-conditioned action π ( ·| o t , w i ) , where o t and w i is observation and the waypoint (acting as a goal) respectively. After reaching the final waypoint ψ ( w n ) , the agent receives the original curriculum goal, g c . The only change when the agent attempts to achieve the final goal g f is that g f comes at the end of the sequences, ( ϕ ( o 0 ) , w 1 , ..., w n , g f ) , rather than g c . In this way, the proposed approach not only provides a tailored curriculum for achieving the final goal but also allows the agent to access more elaborate instructions (waypoints) for practicing each curriculum goal.

5 Experiments

The main goal of the experiments is to demonstrate the capability of the proposed method (CQM) to suggest a well-calibrated curriculum and lead to more sample-efficient learning, composing the goal space from the arbitrary observation space. To this end, we provide both qualitative and quantitative

PointNMaze

Point3WayMaze

AntUMaze

Ant2WayMaze

PointSpiralMaze

CQM (Proposed Method)

OUTPACE

DHRL (Does not provide curriculum)

CURROT

GoalGAN

VDS

PLR

DHRL+

SFL

ALP-GMM

20

17.5

25

15.0

20

15

12.5

10

10.0

15

7.5

5

5.0

2.5

0

200

400

steps (k)

600

800

1000

0.0

0

100

200

300

steps (k)

400

500

600

700

0

200

400

steps (k)

600

800

1000

10

6

8

3

1

0

200

400

600

800

1000

0

0

200

400

600

800

1000

steps (k)

steps (k)

PointNMaze

PointSpiralMaze

Point3WayMaze

AntUMaze

Ant2WayMaze

CQM (Proposed Method)

OUTPACE

DHRL

CURROT

GoalGAN

VDS

PLR

DHRL+

SFL

ALP-GMM

1.0

1.0

1.0

0.8

0.8

0.8

success_rate

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

200

400

steps (k)

600

800

1000

0.0

0

100

200

300

steps (k)

400

500

600

700

0.0

O

200

400

steps (k)

600

800

1000

1.0

1.0

success_rate

0.8

success_rate

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0

200

400

steps (k)

600

800

1000

0.0

0

200

400

600

800

1000

steps (k)

results in seven goal-reaching tasks including two visual control tasks, which receive the raw pixel observations from bird's-eye and ego-centric views, respectively. (refer to Appendix C for the detailed configurations of each task.)

We compare our approach with previous curriculum RL methods and previous graph-guided RL methods. We do not provide manually specified goal space in any of the environments; the agent could not map its global X-Y coordinates from the full observation which includes all the state variables for the RL agents (e.g. angle and angular velocity of the joint, position, velocity ...). Also, the results of CQM and the baselines that utilize external reward functions (all the methods except OUTPACE [6]) are obtained by using sparse reward functions. For the baselines that could not be applied in vision-based environments [24, 6], we utilize an extra autoencoder with auxiliary time-contrastive loss [44, 13].

The baselines are summarized below: OUTPACE [6] proposes uncertainty and temporal distanceaware curriculum learning based on the Wasserstein distance and uncertainty classifier. CURROT [21] interpolates the distribution of the curriculum goals and the final goals based on the performance of the RL agent. GoalGAN [10] proposes the goals with appropriate levels of difficulty for the agent Using a Generative Adversarial Network. PLR [18] selectively samples curriculum goals by prioritizing the goals with high TD estimation errors. ALP-GMM [37] selects the goals based on the difference of cumulative episodic reward between the newest and oldest tasks using Gaussian mixture models. VDS [50] proposes the goals that maximize the epistemic uncertainty of the action value function of the policy. DHRL [24] constructs a graph between both levels of HRL, and proposes frontier goals when the random goals are easy to achieve. However, the original DHRL could not generate curriculum goals without the help of the environment. Thus we evaluated a variant of DHRL (DHRL+) with a modified frontier goal proposal module and architecture (Appendix D.3), in

addition to the original DHRL. SFL [13] constructs a graph based on successor features and proposes uncertainty-based curriculum goals. (refer to Appendix D for detailed implementations)

5.1 Experimental Results

First, we visualize the quantitative results to show whether the proposed method successfully and simultaneously addresses the two key challenges: 1) specifying goal space from arbitrary observation space and 2) suggesting a well-calibrated curriculum to achieve the final goal. Figure 3 illustrates the curriculum goals and changes in discrete goal space (graph) of CQM as learning progresses. Each node in the graph consists of the decoded embedding vectors of VQ-VAE, and each edge represents reachability between the decoded embeddings. The graphs of CQM in the figure gradually expand towards unexplored areas as the learning progresses, since the calibrated curriculum goal induces the agent to explore the unexplored area. In the opposite direction as well, the capability of providing proper curriculum goals on arbitrary geometric structures is facilitated by a compact goal space that contains semantic information which enables estimating the uncertainty and temporal distance well. As a result, our method provides tailored curriculum guidance across the environments, while the baselines suffer from the absence of the manually specified goal space.

We also provide the quantitative results in Figures 4 and 5. Figure 4 indicates that the proposed method (CQM) can suggest a tailored sequence of goals that gradually converges to the final goal distributions while instructing the agent to achieve the increasingly difficult goals. Also, as shown in Figure 5, ours consistently outperforms both the prior curriculum method and graph-RL methods. It is noticeable that CQM is the only method that shows robust performance to the variation of the goal dimension, while other methods suffer from serious data inefficiency, especially in the tasks with higher-dimensional goal space (suffering more in Ant (29dims) compared to Point (6dims)).

Curriclum learning and planning in visual control tasks. To validate the performance of the RL agent and the quality of generated curriculum goals in higher dimensional tasks, We conducted two additional vision-based goal-reaching tasks. PointNMazeViz receives only ego-centric view images to reach the goal, while PointSpiralMaze-Viz receives bird'seye view images. Figure 6 visualizes the curriculum goals in the order of the episodes, and how the agent utilizes the benefit of planning over the discrete goal space in order to achieve the curriculum goals. To achieve an

Final Distance from Goal(Loweris better)

CurriculumGoals&PlanningoverQuantized GoalSpace

PointNMave-Viz

O:Waypoints

17.5

1

15.0

distance

12.5

10.0

2

1

7.5

5.0

E

2.5

0

100

200

300400

steps (k)

500

600

700

4

PointSpiralMave-Viz

5

20

distance

15

?

10

7

Start

200

400

steps (k)

600

800

1000

8

CQM

OUTPACE

DHRL+

SFL

Initial Curriculum

Final Curriculum

Ot

b(ek)

image-based final goal ( Goal: 8 ), the agent generates the sequence of images ( {1, 2, 3, ..., 8} ) as waypoints, and tries to achieve the waypoints sequentially.

Interestingly, despite a significant increase in the observation dimension, CQM does not suffer from significant performance degradation in terms of data efficiency, which indicates that CQM effectively reduces the complexity of goal space by constructing a semantic goal space. We emphasize that the performance of our algorithm does not show significant differences between state-based and imagebased environments (Compare PointNMaze in Figures 4 and 6). Another interesting point is that CQM can fully enjoy the advantage of planning over the discretized goal space, even in vision-based control tasks where the agent does not receive information about its global X-Y coordinates explicitly. These results validate that CQM possesses robust performance in terms of the dimensionality of the goal space, and the capability in extracting temporal relations between the discretized landmarks.

5.2 Ablation Studies

Curriculum guidance. First of all, we examine how important curriculum guidance is for an agent to solve goal-conditioned tasks. As shown in Figure 7, when only the final goal is provided without a tailored curriculum ( -w/o Curriculum ), the RL agent has difficulty achieving the final goal directly.

PointSpiralMaze

AntUMaze

PointNMaze

CQM

CQM w/o Planning

CQM w/o Graph

9

8

14

17.5

7

12

15.0

distance

6

distance

10

distance

12.5

5

8

10.0

4

6

7.5

3

4

5.0

2

2

2.5

1

0

200

400

steps (k)

600

800

1000

0

0

200

400

steps (k)

600

800

1000

0.0

0

100

200

300

steps (k)

400

500

600

700

Furthermore, we found that providing curriculum guidance greatly affects the goal space specification module and the absence of a curriculum leads to the ill-formed discrete goal space that barely covers only the observations near the initial distribution. We provide these qualitative results in Figures 13, 14 (Appendix E).

Types of the discrete goal sampling method. The proposed method (CQM) can use two approaches to sample the landmark to form the discrete goal space as introduced in Section 4.1. The first approach is to decode the embedding vectors of the codebook l 1: m = ψ ( e 1: m ) , and the other approach is to sample an observation batch from the replay buffer and pass it through VQ-VAE to quantize it ( -Landmark from Replay Buff. ). As shown in Figure 7, there is no significant difference between them in terms of data efficiency. However, in terms of the stability of learning, utilizing the decoded embeddings of VQ-VAE shows better performance in some environments.

Effect of the goal convergence method. To provide a final goal-directed exploration in addition to the naïve curriculum toward the frontier areas, CQM includes a goal convergence module that guides the agent to practice the final goal after the agent sufficiently explored the environment (Section 4.3). Based on the KL divergence between the achieved goal distribution and the final goal distribution, CQMcalculates the ratio of the mixture between the final goals and the frontier goals (the ratio of providing final goals as learning progresses is presented in Figure 11 in Appendix E). As shown in Figure 7, the absence of the final goal convergence method ( -w/o Goal Convergence ) results in unstable performance, since the agent repeatedly practices unexplored areas instead of converging towards the final goal even after the explored area 'covers' the final goal distribution.

Effect of Graph Construction and Planning. Finally, we examine the effect of constructing graphs and planning on the performance of RL agents. As explained in section 4.2, CQM not only utilizes the decoded embedding vectors from VQ-VAE as a set of discretized observations but also forms a graph by capturing the temporal relations between the discrete observations. First, we evaluated CQM without graph ( -w/o Graph ), which does not construct a graph and measure the distance between the landmarks through naïve temporal distance prediction based on Q values ( TemporalDist ), rather than the geodesic distance over the graph ( TemporalDist G ). Also, we evaluate CQM without planning ( -w/o Planning ) since ours can optionally utilize the benefit of planning and reason over long horizons using the graph. As shown in Figure 7, CQM shows better performance than both CQMwithout a graph and CQM without planning, especially in some long-horizon tasks (AntUMaze and PointSpiralMaze).

6 Conclusions

To solve the complex control tasks without the need for a manually designed semantic goal space, we propose to solve both issues of specifying the goal space and suggesting the curriculum goals to the agent. By constructing the quantized world model using the decoded embedding vectors of the discretization bottleneck and restoring the relations between these, CQM considers both the uncertainty and temporal distance and has the capability of suggesting calibrated curriculum goals to the agent. The experiments show that the proposed method significantly improves performance on various vision-based goal-reaching tasks as well as state-based tasks, preventing the performance drop in the absence of a manually specified goal space.

Limitations and future works. While CQM shows great potential in addressing the limitations of previous studies, more research could further develop it. One area that could be explored is the use of reward-free curriculum learning methods, since CQM still requires minimal human efforts such as defining a success threshold to train agents. Also, this study only used single-code representations with VQ-VAE which would possess a limited capacity of representations, so expanding CQM to include multiple-code representations with discrete factorial representations could be an interesting future direction.

References

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms , pages 1027-1035, 2007.

Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning , pages 521-531. PMLR, 2021.

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. In Proceedings of International Conference on Learning Representations , 2023.

Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , volume 1, pages 539-546. IEEE, 2005.

Ishan Durugkar, Steven Hansen, Stephen Spencer, and Volodymyr Mnih. Wasserstein distance maximizing intrinsic control. arXiv preprint arXiv:2110.15331 , 2021.

Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning , pages 1515-1528. PMLR, 2018.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, and Honglak Lee. Successor feature landmarks for long-horizon goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems , 34:26963-26975, 2021.

Edward S Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. arXiv preprint arXiv:2303.13002 , 2023.

Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. arXiv preprint arXiv:2210.10195 , 2022.

Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. Advances in Neural Information Processing Systems , 32, 2019.

Riashat Islam, Hongyu Zang, Anirudh Goyal, Alex Lamb, Kenji Kawaguchi, Xin Li, Romain Laroche, Yoshua Bengio, and Remi Tachet Des Combes. Discrete factorial representations as an abstraction for goal conditioned reinforcement learning. arXiv preprint arXiv:2211.00247 , 2022.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning , pages 4940-4950. PMLR, 2021.

Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations , 2020.

Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Conference on Machine Learning , pages 2390-2399. PMLR, 2018.

Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning , pages 11341-11358. PMLR, 2022.

Martin Klissarov and Marlos C Machado. Deep laplacian-based options for temporally-extended exploration. arXiv preprint arXiv:2301.11181 , 2023.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning , pages 5639-5650. PMLR, 2020.

Seungjae Lee, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: A graph-based approach for long-horizon and sparse hierarchical reinforcement learning. Advances in Neural Information Processing Systems , 36, 2022.

Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. In International conference on machine learning , pages 6346-6356. PMLR, 2021.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreographer: Learning and adapting skills in imagination. arXiv preprint arXiv:2211.13350 , 2022.

Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems , 34:24379-24391, 2021.

Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In IJCAI , pages 2536-2542, 2017.

Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goalconditioned policies. Advances in Neural Information Processing Systems , 32, 2019.

Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning , pages 2721-2730. PMLR, 2017.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 27782787. PMLR, 2017.

Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning , pages 5062-5071. PMLR, 2019.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning , pages 7750-7761. PMLR, 2020.

Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698 , 2019.

Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning , pages 835-853. PMLR, 2020.

Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems , 32, 2019.

Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics , pages 832-837, 1956.

Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv:1805.11063 , 2018.

Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274 , 2018.

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning , pages 1312-1320. PMLR, 2015.

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929 , 2020.

Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 1134-1141. IEEE, 2018.

Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. arXiv preprint arXiv:2204.02372 , 2022.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Hanwei Wu and Markus Flierl. Variational information bottleneck on vector quantized autoencoders. arXiv preprint arXiv:1808.01048 , 2018.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning , pages 11920-11931. PMLR, 2021.

Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as a graph: Learning latent landmarks for planning. In International Conference on Machine Learning , pages 12611-12620. PMLR, 2021.

Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems , 33:7648-7659, 2020.

A Algorithm and Derivation

A.1 How CQM Works?

Off-Policy RL

Train VQ-VAE

PracticeCurriculumGoals

Replay

Find waypoints w/ Dijkstra's Alg.

Buffer

π(a|s,waypoint)

Encoder

Decoder

中

Env.step

CalculateTemporal

Dist.w/ Q network

Train Q and π

VQ Codebook

Sampling Curriculum

Goals

Construct Graph

Sampled

convergencemodule

Final goal

Calculate

curriculum

VQ Codebook

Uncertainty

Kernel DensityEstimator

Decoder

α = 1/ max(β + kDk(PgrllPag),1)

Calculate

Temp.Dist.

pgc = αpgr +(1-α)pg

A.1.1 How is data collected?

CQMstarts with empty replay buffers.

After performing a step in the environment, the observed transition is stored in the replay buffer. (line 18 in Algorithm 1.)

A.1.2 How is the goal space learned?

Sample batch from replay buffer. (Line 26 in Algorithm 1.)

Train VQ-VAE with the batch via Eq. 2. (Line 27 in Algorithm 1.)

A.1.3 How is the graph constructed?

Decode each vector embedding in VQ-Dictionary (the decoded embeddings are the landmarks).

Using Eq. 3, connect the landmarks with the distance below the cutoff threshold.

A.1.4 How does the agent get the curriculum goal?

Calculate the uncertainty of each node (landmarks) in the graph (by Eq. 5).

Calculate the distance of the landmarks from the initial area (by Eq. 4).

Sample a curriculum goal which is considered temporally distant and uncertain, among the nodes (landmarks) in the graph (by Eq. 6).

Based on the α value from A.1.6, the decision is made whether to provide the agent with a curriculum goal or a final goal. Then provide the selected goal to the agent.

A.1.5 How does the agent utilize the graph?

After the agent obtains the curriculum goal, it can start exploring the environment.

CQM first finds a sequence of waypoints to achieve the curriculum goal (utilizing Dijkstra's algorithm).

The agent is guided to achieve each waypoint, and finally, tries to achieve the curriculum goal.

A.1.6 How does the curriculum goal converge to the final (desired) goal?

At the beginning of the learning, we have some samples of the final goals (e.g. the picture taken at the end of the maze.)

To get p g f , fit a kernel density estimator (KDE) to estimate the distribution of the final goal.

To get p ag , fit a kernel density estimator (KDE) to estimate the distribution of the explored area.

Calculate KL divergence, and then, calculate α in Eq. 7.

Utilize α when we get the curriculum goal (d)

A.2 Algorithm

Algorithm 1 Overview of CQM

A.3 Derivation

Derivation of Eq. 3. Let Q be a state-action value function, l 1: m be landmarks and a ∈ A be an action that an agent executes. If the policy requires n steps to reach l j from l i , the state-action value with discount factor γ can be represented as

Thus, we can recover the temporal distance between l j and l i (= N steps) as γ n -1 = (1 -γ ) Q ( l i , a, l j ) , and we finally get

B Related Works

Table 1 compares our approach and previous curriculum goal generation methods [10, 21, 18, 50, 37, 38, 36, 6], graph-guided RL methods [13, 49, 24], and representation learning methods using VQ-VAE [17, 27]. The characteristics compared include:

Specify Semantic Goal Space: whether the method can extract a compact goal space from the high-dimensional goal space.

Ucert-Aware: whether the curriculum considers uncertainty.

T-Dist-Aware: whether the curriculum considers temporal distance.

Final Goal-Directed Curriculum: Whether the method utilizes target curriculum distribution for curriculum goal convergence.

Curriculum Proposal: The source of the proposed curriculum goals.

Reason over Long Horizon: Whether the method can reason over long horizons.

C Environment Details

PointNMaze: The agent receives the observation consisting of the angle, angular velocity, XY position, and XY velocity of the agent. The agent is initialized in [0, 0] and the final goal is located at [0, 8]. The agent achieves the final goal when it comes within 0.5 from the final goal. Also, the size of the map is 12 by 12. The maximum timestep per episode is 150.

PointSpiralMaze: It shares the other aspects with PointNMaze, but the final goal is located in [8, 16] and the size of the map is 12 by 20. The maximum timestep per episode is 250.

Point3WayMaze: It shares the other aspects with PointNMaze, but the final goal is sampled randomly among [24, -8], [24, 0], and [24, 8]. The map has multiple dead ends and the size of the map is 32 by 24. The maximum timestep per episode is 200.

AntUMaze: Agent receives the observation consisting of joint angle, joint angular velocity, XYZ position, and XYZ velocity of the agent. The agent is initialized in [0, 0] and the final goal is located at [0, 8]. The agent achieves the final goal when it comes within 1.0 from the final goal. Also, the size of the map is 12 by 12. The maximum timestep per episode is 350.

Ant2WayMaze: It shares the other aspects with AntUMaze, but the final goal is sampled randomly among [4, 4] and [-4, -4]. The map has multiple dead ends and the size of the map is 20 by 12. The maximum timestep per episode is 200.

AntUMaze

Ant2WayMaze

PointSpiralMaze

PointSpiralMaze-Viz

(bird's-eye view camera)

PointNMaze

Point3WayMaze

(ego-centriccamera)

5

PointNMaze-Viz: The agent receives the ego-centric camera input. The resolution of the input is 64 x 64. This environment is modified from PointNMaze, by coloring the walls. The maximum timestep per episode is 200.

PointSpiralMaze-Viz: The agent receives the Top-down view camera input. The resolution of the input is 64 x 64. This environment is modified from PointSpiralMaze, by converting the actions of the agent from the polar coordinates to global XY coordinates and increasing the visual size of the agent. The maximum timestep per episode is 250.

D Implementation Details

D.1 Learning VQ-VAE

Replay buffer for VQ-VAE To train the VQ-VAE for quantizing continuous observations, we employ a separated (smaller size) replay buffer which contains recent trajectories compared to the original replay buffer for the RL agent. By employing the separated replay buffer, we can provide VQ-VAE with more recent observations, resulting in a better reflection of the areas recently explored by the RL agent.

Code resampling for VQ-VAE Whenweuse decoded embedding vectors as landmarks to discretize the goal space for CQM, VQ-VAE could ignore some of the codes due to index collapse. Then, the unused codes form landmarks in physically infeasible locations and occupy the capacity of the codes unnecessarily, leading to potential inefficiencies in the model's performance. Thus, we utilize the code resampling method [20] across all the experiments to address such problems of index collapse [20] in training VQ-VAE models.

The code resampling module in CQM first keeps track of inactive codes that have not been used during the previous M rollouts. Subsequently, the inactive codes are re-initialized with the embeddings of the recent observations ϕ ( o ) with the probability of d 2 q ( ϕ ( o )) / Σ o d 2 q ( ϕ ( o )) , where d 2 q ( ϕ ( o )) indicates the Euclidean distance from the closest code ( d q ( ϕ ( o )) = min i ∈{ 1 ,...,k } || ϕ ( o ) -e i || 2 2 ). We refer the reader to [27] for detailed explanations of code resampling.

Point3WayMaze

AntUMaze

PointNMaze

CQM + SAC

1.0

1.0

1.0

0.8

0.8

0.8

rate

0.6

0.6

0.6

success

ccess

success

0.4

suc

0.4

0.4

0.2

0.2

0.2

0.0

0

200

400

steps (k)

600

800

1000

0.0

0

200

400

steps (k)

600

800

1000

0.0

0

100

200

300

steps (k)

400

500

600

700

D.2 Sampling landmarks

We adopt the graph construction module of prior work [49, 24]. Inheriting their implementations, we also employ a landmark sparsification technique which is based on the Greedy Latent(Node) Sparsification algorithm [2, 49]. The detailed algorithm is shown in Algorithm 2

Algorithm 2 Greedy Latent(Node) Sparsification [49, 24] 1: Input: set of states { e 1 , e 2 , ..e k } , sampling number k 2: Selected = [], DistList = [inf, inf, ... inf] 3: for i = 1 to k do 4: Farthes ← argmax ( DistList ) 5: add Farthest to Selected 6: DistFromFarthest ← [TemporalDist( Farthest → ψ ( e 1: m ))] 7: DistList = ElementwiseMin( DistFromFarthest , DistList ) 8: end for 9: return Selected

D.3 State-based Goal-Reaching Tasks

When we train CQM in state-based goal-reaching tasks, we utilize two well-established algorithms (TD3 algorithm [11] for Point- environments and SAC [12] for Ant- environments), following the baselines (TD3: [24], SAC: [21]). We note that CQM can be built on general off-policy RL algorithms [26, 12, 11] and changing the RL agents for CQM does not lead to a performance drop as shown in the figure 10.

Also, following [24], we utilize separate Q-networks for graph construction and policy learning. When the agent is not yet competent to achieve some goals, the experience of failure in the replay buffer can lead to an overestimation of the temporal distance between the landmarks. Thus, prior work utilizes separate Q-networks for graph construction and policy learning. This simple technique maintains two different Q-networks which are trained with different ratios of hindsight experience replay (HER) [1] and prevents the usage of failure trajectories to estimate the temporal distances between the landmarks which can spoil the graph construction.

Baselines. Since the original implementations of some algorithms [13] utilize discrete action policies, we replace them with continuous policies for comparison in the continuous control tasks. Also, for the baseline that could not generate curriculum goals without the help of the environment (DHRL) [24], we modify the frontier goal sampling module of the original baseline. Specifically, we modified the frontier goal-shifting module in the DHRL to provide a curriculum goal in every episode, not only when an easy goal is given from the environment. This modification ensures that the agent can get curriculum goals even when the environment does not provide random goals. Additionally, we empirically found that the high-level agent aggravates the performance since it suffers from providing high-dimensional subgoals. Therefore, we utilize a variant DHRL (DHRL+) with a modified frontier goal-shifting module and without a high-level. For the other baselines, we follow the official implementations (OUTPACE and CURROT) [21, 6] or the implementations from https://github.com/psclklnk/currot (PLR, VDS, GoalGAN, ALP-GMM).

D.4 Vision-based Goal-Reaching Tasks

For the experiments in the vision based-based goal-reaching tasks, the agent needs to be provided with the curriculum goals corresponding to vision inputs. However, we observed that the decoded images from VQ-VAE sometimes have inferior resolution compared to the original images, which could lead to inferior performances. Thus, during the training process for the vision-based control, we utilized the observations that are mapped to each embedding as landmarks, instead of the decoded embeddings as curriculum goals. This technique does not require additional assumptions or computational costs, as the only overhead is saving the original images of each code. Although the mappings from observations to embedding vectors are many-to-one, we empirically found that there is no performance degradation even if we randomly pick one of the observations per embedding vector as a landmark.

Baselines. Since the official implementations of some baselines are unable to solve the visionbased goal-reaching tasks, We experimented with the additional encoder which can encode the high-dimensional observations into compact latent vectors. Considering the insights from prior representation learning research [23, 43], using a naïve autoencoder may not be fair, so we incorporated an auxiliary loss that allows for better representation learning for RL agents. To this end, we utilize an autoencoder with time-contrastive loss [7, 44] as used in SFL [13] which also performed exploration in vision-based goal-reaching tasks. Specifically, we employ the following auxiliary triplet loss to train the encoder for the baselines,

where o a , o p , and o n represent anchor, positive pair, and negative pair respectively. Also, we utilize a margin parameter m = 2 , following [13].

D.5 Computational Resources

Our experiments have been performed using an NVIDIA RTX A5000 and AMD Ryzen 2950X, and the entire training process took approximately 0.5-2 days, depending on the tasks.

E Additional Experimental Results

We provide quantitative results for the ablation study in Figures 13 and 14. As we analyzed in Section 5.2, CQM without each module shows inferior results across the tasks. Without a goal convergence module, the agent often has difficulty in terms of progressing toward the desired final goals. Also, without planning, the agent often has difficulty in achieving long-horizon tasks.

As shown in Section 4.3, our algorithm includes a goal convergence module based on KL divergence between the explored area and the region corresponding to the final goal (Eq. 7) in order to perform final goal-directed exploration. Figure 11 represents the ratio of final goals given to the agent instead of curriculum goals, which is calculated according to the following equation.

As shown in Figure 11, the ratio of providing final goals gradually increases as the learning progresses. This allows the agent to practice the final goal instead of exploring unexplored areas when the agent acquires the capability to pursue the final goals at the end of the curriculum.

AntUMaze

PointSpiralMaze

PointNMaze

1.0

1.0

1.0

rate

0.8

rate

0.8

0.8

10.6

10.6

10.6

10.4

10.4

10.4

0.2

0.2

0.2

0.0

0.0

0.0

0

200

400

600

800

1000

0

200

400

600

800

1000

0

100

200

300

400

500

600

700

steps (k)

steps (k)

steps (k)

17.5

kappa=0.01

17.5

beta=-60

beta=-40

15.0

kappa=0.25

15.0

beta=-25

beta=-20

12.5

kappa=4.0

12.5

nce

kappa=10

beta=0

tand

10.0

istane

10.0

7.5

7.5

云

5.0

5.0

2.5

2.5

0.0

0.0

0

100

200

300

400

500

0

100

200

300

400

500

steps (k)

steps (k)

PointNMaze

PointSpiralMaze

AntUMaze

Ours

w/o  Goal

Convergence

Landmark from

w/o  Planning

Initial

Curriculum

Final

Curriculum

Initial

Curriculum

Final

Curriculum

Initial

Curriculum

Final

Curriculum

Initial

Curriculum

Final

Curriculum

Initial

Curriculum

Final

Curriculum

10

S'/

7.5

15.0

5.0

12.5

2.5

10.0

0.0

7.5

5.0

2.5

17.5

15.0

5.0

12.5

10.0

7.5

5.0

2.5

0.0

10

10.0

10.0

-7.5-5.0-2.50.02.55.0

10.0

10.0

17.5 -

7.5

15.0

5.0

12.5

2.5

10.0

0.0

7.5

2.5

5.0

5.0

2.5

7.5

0.0

10

-10.0

10.0

-7.5-5.0-2.50.02.55.0

7.5

10.0

1/.5

7.5

15.0

12.5

2.5

10.0

0.0

7.5

5.0

2.5

0°0

0.0

2.5

5.0

7.5

10.0

10

10.0

17.

7.5

15.0

5.0

12.5

2.5

10.0

0.0

5

2.5

5.0

5.0

2.5

7.5

0.0

-10.0

10.0

-7.5-5.0-2.50.02.5

0.0

PointNMaze

PointSpiralMaze

AntUMaze

Ours

w/o  Goal

Convergence

Landmark from

w/o  Planning

17.5

7.5

15.0

5.0

12.5

2.5

10.0

0.0

4

7.5

2.5

5.0

-5.0

7.5

2.5

0.0

2

4

6

8

-5

0.0

2.5

5.0

7.5

17.5

7.5

15.0

5.0

12.5

2.5

10.0

0.0

7.5

2.5

5.0

2.5

7.5

0.0

0.0

2.5

5.0

7.5

10

5

0.0

2.5

5.0

7.5

8

17.5

15.0

12.5

4

10.0

0.0

7.5

2

5.0

2.5

口

2

-5

5

0.0

0.0

2.5

5.0

7.5

7.5

17.5

15.0

5.0

12.5

2.5

10.0

0.0

7.5

5.0

2.5

0.0

0.0

2.5

5.0

7.5

5

0.0

2.5

5.0

7.5

17.5

15.0

12.5

10.0

0.0

7.5

2

5.0

2.5

0.0

0.0

2.5

5.0

7.5

10.1

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The proposed method addresses the challenge of learning a curriculum in goal-conditioned policies, which is a significant problem. Previous research on curriculum in goal-conditioned policies has often overlooked the importance of learning the underlying semantic goal space. This paper builds upon a recently proposed method that utilizes VQVAE to learn the semantic goal space and extends it to incorporate the learning of a curriculum. By selecting a sequence of sub-goals, this curriculum-based approach aids in achieving the final goal.\n\n- The paper compares the proposed method to many different SOTA baselines.\n\nSTRENGTHS:\n- The paper is very well-written. \n- The paper effectively compares the proposed method with various baselines, providing a thorough analysis.\n- Additionally, the paper conducts ablations to examine the impact of different design decisions made in this research.\n\nWEAKNESSES:\n- It will be valuable to assess the effectiveness of the proposed method on both manipulation tasks and more intricate navigation tasks like 3D environments (ex. Habitat, AI2-Thor etc)\n- It seems restrictive to the reviewer to use single-code representations with VQ-VAE so would be interesting to see how the results change if using multiple codes like in DGRL. (The reviewer understands this is mentioned as future work, but this seems a severe limitation if the proposed method works only with single code and not multiple codes).\n\n- A minor comment: To enhance clarity, it is recommended to explicitly mention in the introduction that prior research, such as DGRL [1], has proposed and emphasised the significance of learning a semantic space for goal specification. It was only later in the paper that I realised vector quantisation had been proposed in previous work as an abstraction for goal-conditioned RL.\n\n[1] Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning, https://arxiv.org/abs/2211.00247\n\nQUESTIONS:\n- I'm curious about how the performance varies by varying the number of codes in VQ-Dictionary.\n- It will also be useful to study if the performance of the proposed method can be improved by using self-supervised objectives for learning visual representations as done in DGRL.\n\nLIMITATIONS:\nRefer to Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe method works as follows. Graphs are built by (1) quantizing visual observations to create a goal space & (2) creating temporal relations over goal space vectors. Curriculum goals towards a \"user\"-specified goal are made using this graph.\nA VQ-VAE is used to create the goal space where goals are decodings of one of k trainable embedding vectors. The authors also study an ablation where landmarks are decoded from the nearest neighbor goal encodings over replay buffer observations.\nGraphs are derived by connecting edges based on a metric that exploits having rewards by -1 at all states except for the goal state. This is motivated by wanting edges to capture the geodesic between nodes.\nA curriculum for exploring the graph is created by selecting nodes with high uncertainty (or low count via count-based methods) and high temporal distance. Curriculum goals are goals that maximize this sum of this uncertainty and geodesic distance away from the initial state.\nThe method uses Dikstra's algorithm to generate plans towards goals.\n\nThey compare against many baselines in the literature and consistently find a higher success rate. They also visualize the curriculum goals produced by their method.\n\nSTRENGTHS:\noriginality: The method is novel to the best of my knowledge.\nquality: The paper has high quality. The experiments are very though and the authors also do an analysis.\nclarity: The experiments are relatively clear.\nsignificance: The paper seems to produce better performance than many methods in the literature with similar assumptions (e.g. exploiting diikstra's algorithm).\n\nWEAKNESSES:\nThe paper could strongly benefit from some figures that describe the details of the method. The method is relatively complicated (though not more complicated than comparable methods in the literature). Due its complexity, it's hard to understand how the method works from figure 1. I'm somewhat familiar with this literature so I think understand their method section was easier but other readers may find it more challenging. Another figure would strongly improve the clarity of this paper.\n\nThe baseline description method describes how the baselines work but it does not describe what comparing against each baseline (or sets of them) tells us about CQM. Without this, while the results are promising, it's hard to get a takeaway message. For example, OUTPACE also proposes an uncertainty and temporal distance aware curriculum. Comparing against OUTPACE tells us....? It might be the importance of jointly considering uncertainty and temporal distance but that isn't clear in the text. I recommend explicating this a bit more. I think it's fine if baseline methods are grouped together for describing the point of the comparison.\n\nGiven the complexity of this method, it would strongly benefit from an algorithm describing, how is data collected, how is the goal space learned, how is the graph constructed, etc. In particular, which of these are done in tandem and which are done in sequence is not clear.\n\nOverall, this method seems promising, the motivation is clear, and the results are also promising. I think the biggest drawback is clarity right now. It's not that easy to understand the method. Right now I lean towards accept but I strongly encourage the authors to add figures/explicit algorithm/both for the methods section. This would likely push my score to a 7.\n\nQUESTIONS:\nAcross experiments, does the agent encode first-person observations, top-down observations over a map, or both? This seems important to how you quantize the observations. If top-down, this seems like a strong assumption. Can you justify it? Methods like SFL use first-person observations. At least some of the environments that you study permit first-person observations (e.g. AntMaze). I know you do only ego-centric for PointNMave-Viz but it's not clear for the other experiments.\n\nWhat assumptions does this method make over the reward function?\n\nLIMITATIONS:\nThis limitations section is fine.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a curriculum RL approach using a VQ-VAE to learn a goal space, and then construct a graph with the VQ-VAE codes as nodes, and a temporal distance estimate of the Q-value as weighted edges. The curriculum is then constructed by doing frontier-based exploration on this graph, by sampling goals based on their visitation count and temporal distance. In addition, the graph can be used for planning intermediate goals as waypoints for the agent. The paper is well written, and it has proper benchmarks and ablation study. The authors use the term \"world model\" in their paper, but I would argue they don't really train a world model, rather a representation of the observation space. I would omit the use of \"world model\" in a further revision of the manuscript.\n\nSTRENGTHS:\nThe paper proposes a novel curriculum RL method, which learns a goal-space on high-dimensional pixel observations. Turning the VQ-VAE codebook into a graph, weighted by estimated temporal distance from the Q-value is a powerful idea for both curriculum learning as well as planning and long-range credit assignment.\n\nWEAKNESSES:\n- The authors use the term \"world model\", however what they present is a discrete latent representation of the observation space. A world model would entail to e.g. also equip it with an action-conditioned forward dynamics model, as typically used in model-based RL. I would not call this a world model.\n\nQUESTIONS:\n- How to get a distribution over the goal, encoding a set of exemplar goal observations and counting the bins of the VQ-VAE they get encoded in?\n\n- In the Planning section the authors use state s_0 from S, although the previous MDP description uses observation o_0 from O. I assume these are the same or have you shifted to a POMDP setup?\n\n- In the AntUMaze, the performance seems to deteriorate after 600k steps, i.e. the curriculum goals seem to get farther from the goal. Any insight on what is happening there?\n\n- Since the goal space is trained as a VQ-VAE on individual observations, I think it will break when the environment is ambiguous, i.e. if the agent would have to traverse rooms, but 2 rooms would look identical, these observations would map onto the same goal and the graph might collapse?\n\nLIMITATIONS:\n- I think one limitation that is currently not touched upon is that this approach requires that the environment is not ambiguous (i.e. you don't see the same observations at different places in the environment).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Curriculum RL with Quantized World Model (CQM), a novel approach that leverages a VQ-VAE to create a discretized goal space and constructs a graph structure over it. CQM further proposes a curriculum strategy based on uncertainty and temporal distance to guide the learning process. The authors evaluate the effectiveness of CQM through experiments conducted on variants of PointMazes and AntMazes, which serve as benchmarks for Hierarchical Reinforcement Learning.\n\nSTRENGTHS:\n- This paper is well-written and easy to follow up\n- Extensive experiments\n\nWEAKNESSES:\nOne significant weakness of this paper is the lack of clarity regarding why using the representation from VQ-VAE is suitable for graph-building. It is crucial to consider the temporal distances between nodes to adequately cover the visited state space, especially given the limited number of nodes. The proposed goal representation learning scheme by VQ-VAE does not appear to take into account these temporal distances, raising questions about the efficacy of the proposed representation in creating a meaningful semantic goal space. It would greatly benefit the paper to provide a more thorough explanation and justification for the use of VQ-VAE in graph construction and its ability to capture temporal relationships. \n\nFurthermore, the proposed goal representation learning scheme should be compared to prior work on representation learning, such as NORL [1] or LESSON [2]. A comprehensive comparison would help establish the novelty and effectiveness of the proposed approach in relation to existing methods. Additionally, it remains unclear how the proposed goal representation learning scheme outperforms or differs from the simple approach of utilizing farthest point sampling from the replay buffer, which warrants further investigation and comparison. \n\nMoreover, the paper introduces new hyperparameters (\\alpha, \\beta, and \\kappa) for curriculum goal generation. However, it is unclear how these hyperparameters were determined and how the performance varies when these hyperparameters are varied. Additionally, it would be valuable to compare the costs associated with hyperparameter search between CQM and the baseline methods. \n\n[1] Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine, “Near-Optimal Representation Learning for Hierarchical Reinforcement Learning”, ICLR 2019.\n\n[2] Siyuan Li, Lulu Zheng, Jianhao Wang, Chongjie Zhang, “Learning Subgoal Representations with Slow Dynamics”, ICLR 2021\n\nQUESTIONS:\n- How does performance vary when \\alpha and \\beta are changed? Could you provide insights into the selection process for these hyperparameters? Is it possible to set them automatically?\n- Regarding planning over the graph, what happens if an agent is unable to directly reach a specific node w_{i}? Could the system handle a scenario where, after TemporalDist(\\phi(w_{i-1}), \\phi(w_{i})), the agent conditions its behavior on reaching w_{i+1} instead?\n- Could you please explain how the landmarks were sampled from the replay buffer for the \"CQM Landmark from Replay Buff\" depicted in Figure 7? What criteria or process were used to select the landmarks from the replay buffer?\n\nMinor: \n- It appears that there are two different symbols used for \\beta, one in Equation 2 and another in \\alpha = 1 / \\max (\\beta + \\kappa D_{\\text{KL}} (p_{g}^{f} || p_{ag}), 1). I would recommend using distinct symbols to avoid confusion.\n- In Figure 7, the y-label representing the distance is partially hidden.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper proposed a new curriculum reinforcement learning method, CQM, that uses VQ-VAE to learn a quantized goal space, constructs a graph on the quantized goals to propose curriculum goals by distance, and learns a goal-conditional policy.

STRENGTHS:
- The proposed method pioneers in "auto" curriculum RL that learns the goal space and proposes goals all by itself.
- The paper is clearly written and easy to follow.
- Detailed studies on various environments are presented to demonstrate its effectiveness.
- Code is provided for reproduction.

WEAKNESSES:
- $\hat{o}_t$ in Eq. 2 should be $o_t$.
- [Vector Quantized Models for Planning](https://arxiv.org/pdf/2106.04615.pdf) has used VQ-VAE in RL. They did not explicitly generate curricula but it's very similar to the proposed method. The authors should compare and explain the difference.
- $Q(l_i, a, l_j)$ in Eq. 3 is not explained. I guess it's an expectation over the replay buffer?
- The proposed mazes are long but they mostly don't have branched dead ends, which contain unseen states but do not lead to the actual goal. [This work](https://openreview.net/pdf?id=U4r9JNyNZ7) and its experiments in more complicated mazes should be compared to.

QUESTIONS:
- Eq. 6 does not have a weight factor to balance the two terms. How are the two terms distributed in practice? Will one dominate the other?

LIMITATIONS:
The limitations have been adequately addressed.

> After rebuttal

Some of my concerns have been addressed. I appreciate the authors' efforts, but the maze in either their submission or rebuttal doesn't match the complexity of the related work I requested for comparison. Thus I'm holding my score.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-17