CQM: Curriculum Reinforcement Learning with a Quantized World Model
Authors: Seungjae Lee
Daesol Cho
Jonghae Park
H. Jin Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Curriculum Learning
Goal-conditioned RL

Abstract: Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate curriculum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized-variational autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-aware curriculum goals that converges to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations in an uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, in various goal-reaching tasks even with ego-centric visual inputs.

CQM: Curriculum Reinforcement Learning with a Quantized World Model
Seungjae Lee, Daesol Cho, Jonghae Park, H. Jin Kim
Scoul National University
Automation and Systems Research Institute (ASRI)
Artificial Intelligence Institute of Scoul National University (AIIS) (ysz0301, dscho1234, bdfire1234, hjinkim) @snu, ac kr
Abstract
Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate currieulum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized- variizedoaviaainnaall autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-awar curriculum goals that converges to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, various goal-reaching tasks even with ego-centric visual inputs.
Introduction
Goal-conditioned Reinforcement Learning (RL) has been successfully applied to wide range of decision decision-making problems allowing RL agents to achieve diverse control tasks [42] I 30 However, training the RL agent to achieve desired final goals without any prior domain knowledge challenging, especially when the desired behaviors can hardly be observed. In those situations, humans typically adopt alternative ways learn the final goals by gradually mastering intermediate sub-tasks. Inspired by the way humans learn, recent RL. studies [29] 10 6 have solved uninformed exploration tasks by suggesting which goals the agent needs practice. In this sense of generating curriculum goals, previous approaches proposed various ideas to involve providing intermediate- tasks [1O] (38)
quantifying the uncertainty of observations 31 33 25 18], or proposing contextual distance to gradually move away from the initial distribution
15 6.
However, previous curriculum RL studies mostly not scalable. Namely, they suffer from serious data inefficiency when they generate curriculum goals in high dimensions. Because of this limitation, they usually rely the assumption that manually specified goal spaces (c.g., global X-Y coordinates) and clear mappings from high-dimensional observations to the low-dimensional goal spaces are available. Such assumption requires prior knowledge about observations and the tasks, which remains a crucial unsolved issue that restricts the applicability previous studies.
In order to design a general curriculum solution without the need for prior knowledge about the observations, defining its own goal space for curriculum could an effective scheme. To do so,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Train VQ VAE
Specifying Goal Space
Suggesting Curriculum Goals
Rexl Gool
Bemtlneec
Taploy
Enosder
Dasodar
Deservation
Dooade Landmario
Codebook
Inital Distntaution
Tempors Ciegara
Rertore Relations
Figure 1: CQM simultaneously tackles the interrelated problems of specifying the goal space and suggesting which goal the agent needs to practice. CQM trains a VQ-VAE to form a discretized goal space and constructs graph over it, capturing the relations between the discretized observations (landmarks). Concurrently, CQM suggests the agent which goal to practice based on uncertainty and temporal distance.
the two following operations need to be executed concurrently. (1) composing the semantic goal space which contains vital information for the curriculum process from the arbitrary observation space, and (2) suggesting to the agent which goal to practice over the goal space. Let consider an agent that tries to explore an uninformed environment with final goal images only. To succeed, the agent needs to specify the semantic goal space from the high-dimensional observation space, and suggest the curriculum goals (e.g., the frontier of the explored area) over the composed goal space to search the uninformed environment. However, most previous studies focused solely on of these, specifying the low-dimensional goal space without considering how to provide intermediate levels just suggesting curriculum goals in manually specified semantic goal spaces of goals [17 27]
10
21
6
The challenge of simultaneously managing (I) specifying the goal space and (2) suggesting curriculum goals is that they are intimately connected to each other. If the agent constructs an ill-formed goal space from the observation space, it would be difficult to propose curriculum goals over it. Conversely, the method fails to suggest goals to enable the agents to explore the unseen area, it would also be difficult to automatically learn the goal space that covers the uninformed environment based the accumulated observations. Therefore, it is essential to develop an algorithm that addresses both defining goal space and providing curriculum goals concurrently.
In this paper, we propose a novel curriculum reinforcement learning (RI.) method which can provide general solution for a final goal-directed curriculum without the need for prior knowledge about the environments, observations, and goal spaces. First, our method defines its own semantic goal space by quantizing the encoded observations space through discretization bottleneck and restoring the temporal relations between discrete goals via a graph. Second, to suggest calibrated guidance towards unexplored areas and the final goals, ours proposes uncertainty and temporal distance-aware curriculum goals that converge to the final goal examples.
The key contributions of our work (CQM: Curriculum RL with Quantized World Model) are:
CQM solves general exploration tasks with the desired examples only, by simultancously addressing the specification of goal space and suggestion of curriculum goals (Figure 3
CQM is the first curriculum RL approach that can propose calibrated curriculums toward final goals from high-dimensional observations, to the best of our knowledge.
CQM the only curriculum RL method that demonstrates reliable performance despite an increase in the problem dimension, among the 10 methods that we experimented with. (Even state-based - vision-based)
Ours significantly outperforms the state-of-the-ar curriculum RL methods on various goal reaching tasks in the absence of manually specified goal space.
2
Related Works
Curriculum Goal Generation. Although various prior studies 41
19
45 22 have been
proposed to solve exploration problems, enabling efficient searching uninformed environments still

---

remains challenge. An effective way to succeed in such tasks with hardly observed final goals is identifying uncertain areas and instructing an agent achieve the goals in these areas. To identify
the uncertain areas and provide the goals sampled from them, previous studies employ uncertainty-
based curriculum guidance by the state visitation counts 4 31. absolute reward difference [37, and
prediction model [32 5] Other approaches propose to utilize disagreements of ensembles 33 50 28
or sample the tasks with high TD errors
[18] to generate goals in uncertain areas. An alternative way for solving the exploration tasks is to execute final goal-directed exploration to propose tailored
guidance. To this end, some studies perform successful example-based approaches 25 6 or propose to minimize the distance between the final goals and curriculum goals [38 211, measuring it by the Euclidean distance metric. Some studies also employ contextual distance- based metrics to perform
final goal dirl-eirected exploration away from the initial distribution [T5 6
PointSoralMare
PointMMas
However, these methods usually assume that agents have prior knowledge about the observations and un- restricted access to manually specified semantic goal
Succes
space (e.g. global X-Y coordinates) because they are not scalable to handle the high-dimensional goal spaces. For example, the meta-learning classifier- based uncertainty metrics [25, 6 suffer from distin- guishing uncertain areas the dimension of the goal space increases. Also, some of the methods rely
sluge [k]
stage [k]
AntulMaze
w/ Manually
CUBROT [21]
Spectfied
ntee
ALP GMM [37]-
Gaal
OUTPACE
Manually
ELRBET [21]
Specflled
ALP GMM
on Euclidean distance metric [38 21 over the goal space. Moreover, generating curriculum goals [10] employing various prediction models 332
0.08 Performance Sap
33
50].
Figure 2: When the curriculum methods are not scalable to handle the high-dimensional goal; the performance drop in the absence of mamually specified goal space.
fitting Gaussian mixture models [37]. and utilizing
disagreements of ensembles-based methods 33 50
also face difficulty in solving increasingly complex problems in high-dimensional goal spaces. Although there have been attempts propose a currieu- lum in gigh-dimensionll observations [6 [3] or include an encoder in their model-based agent 28 unfortunately, these approaches do not incorporate a convergence mechanism the final goals. which are crucial for efficient curriculum progresses.
Our method incorporates the benefits of the aforementioned methods without manually specified goal spaces: exploring uncertain areas and moving away from the initial distribution while converging to the desired outcome. Although there is a study that has also incorporated these three aspects (6), it retains its performance only in manually specified goal spaces, as other curriculum methods (Figure 2). (We included conceptual comparisons between CQM and more related works in Table T in Appendix B
Discretizing Goal Space for RL. Vector quantized- quantized-variationaa autoencoder (VQ-VAE) is an autoencoder that learns a discretized representation using learnable codebook. The use of this discretization technique for learning discrete representations in RL is a recent research topic [7 27 and has shown improved sample efficiency. Islam et al. [17 proposes apply VQ-VAE as discretization bottleneck in a goal-conditioned RL framework and demonstrates the efficiency of representing the continuous observation spaces into discretized goal space. Also, Mazzaglia et al. [27] utilizes VQ- VAE to discover skills in model- based RL by maximizing the mutual information between skills and trajectories of model states.
Unfortunately, the aforementioned methods require a pre-collected dataset or extra exploration policy, which are not necessary in CQM. Training VAE with pre-collected dataset implies that the agent has access to the full information about the task or that it already possesses an agent capable of performing the task well. Although it is possible to obtain pre-collected dataset through a random rollout policy, this is only in the case where exploring the environments is easy enough to succeed with only random actions.
3 Preliminaries
We consider a Markov Decision Process (MDP) which can be represented as a tuple (O, T, R. 7), where O an observation space, A an action space, T T(o+41 loi; at) transition function, Po is an initial distribution, and y is a discount factor. Note that the MDP above
3

---

does nor contain a goal space, since we do not assume that the manually specified goal space is provided. Instead, we consider discrete now-dimensional discrete goal space G, which is defined by the agent automatically. Also, we assume that the final goal examples of are provided by the environment, and we denote the projection of these examples into the goal space G g¹_ We represent curriculum goal as gf, which is sampled from the goal space G, and the reward function in the tuple can be represented as R. : O A x g + R. Furthermore, we denote actor network as π O x G + A. and critic network Q O A x G + R. (Thus, Q(o, a, g) indicates the goal-conditioned state-action value where goal g, action a, and observation o throughout this paper.)
4 Method
In order to provide general solution for efficient curriculum learning, our method defines its own goal space and suggests to the agent which goal to practice over the goal space simultaneously. To compose semantic goul space which reduces the complexity of observation space while preserving vital information for the curriculum process, we first quantize the continuous observation space using a discretization bottleneck (section 4. and restore temporal relations in discretized world model via graph (section 4.2). Over the automatically specified semantic goal space, we generate curriculum goal and guide the agent toward achieving (section 4.3).
4.1 Specifying Goal Space via VQ-VAE
In order to define a discrete low-dimesiional goal space which allows scalable curriculum with high-dimen nigh-dimensional observations, we utilize VQ-VAE as a discretization bottleneck [46] 40 47 as recently been proposed (17 27). VQ-VAE utilizes a codebook composed of k trainable embedding vectors (codes) G E RD, i 2, ...k, combined with nearest neighbor search to learn discrete representations. The quantization process of an observation or starts with passing o, through the encoder é. The resulting encoded vector 2e = ((o,) is then mapped to an embedding vector in the codebook by the nearest neighbor look-up as
Zq Co. where c argmin, ez112-
The discretized vector 2q is then reconstructed into à ()2g) by passing through the decoder ah. We closely follow [17 and train quantizer, encoder, and decoder using vector quantization loss with simple reconstruction loss. The first term in Eq. 2 represents the reconstruction loss, while the second term represents the VQ objective that moves the embedding vector towards the encoder's output =e. We update the embedding vectors e using a moving average instead of a direct gradient update [17 27). The last term is the commitment loss, and use the same Acommit (= 0.25) across all experiments.
LvQ I[(((())) - oz 112 [|[G[[z]] - -el Acommit || SG[e]ll. (SG stop gradient) (2)
By utilizing VAE, the RL agent can specify quantized goal space that consists of discrete landmarks L (4, 122 | in two ways. The first approach is to obtain each landmark by decoding each code as I = (ee)). ). Alternatively, one can obtain the landmarks by passing (encoding, quantizing to the closest embedding vector, and decoding) the continuous observations sampled from the replay buffer through the VQ-VAE. We utilize the first approach as the default, and provide the ablation study to examine the effectiveness of the second approach.
should be noted that this set of landmarks L {41 .. Im only represents discretized obser- vations and does not involve relations among the observations. We describe our approach that can better restore the temporal relations between landmarks in the next section.
4.2 Graph Construction over Quantized Goal Space
In this section, we present the graph construction technique over the quantized goal space to allow the agent to capture the temporal information between the landmarks of the quantized world model. We consider graph G (V, over the goal space where the vertices V represent the landmarks

---

= (11. 12,...-u obtained from decoding discrete codes of VQ-VAE, and the edges E represent the temporal distance. We utilize Q-value reconstruct the edge costs, following the method proposed in 19 [4] If an agent receives reward of O when reaching goal and -1 otherwise, the timesteps required to travel between landmarks can be estimated using Q-value as (derivation: Appendix A
TemporalDist(4, - 14) = log, (1 + -11--100 (1,,)Q(4,,,,)))
(3)
we connect vertices with the distance below the cutoff threshold, and the resulting graph restores the temporal relations between the landmarks over discretized goal space based on the temporal distance. In this way, the agent can calculate geodesic distances between landmarks,
Using Eq. 3
TimporalDist G (lo + (1, = Ea,+, (Sshortes path(lo-tly TemporalDist((: + 12),
(4)
which enables better prediction of temporal distances in environments with arbitrary geometric structures. Also, incorporate extended supports of the explored area into the graph by creating landmarks in newly explored areas, we periodically reconstructed the graph following
4.3 Uncertainty and Temporal Distance-Aware Curriculum Goal Generation
the previous sections, we proposed a method for specifying discretized goal space with semantic information. It is important to provide curriculum goals located in the frontier part of the explored area to expand the graph effectively toward the final goal in an uninformed environment. To achieve this objective, propose uncertainty and temporal distance- aware curriculum goal generation method.
To obtain a curriculum goal from graph G = (V, E) over the specified goal space, our method samples the landmarks that are considered uncertain and temporally distant from the initial distribution Pu- Thanks to the quantized world model, quantifying uncertainty in countable goal space is straightforward and computationally light. We quantify the count-based uncertainty of each landmark as Jurert (4+) 1/(((4)) + e). based on the empirical distribution μ(l;) derived from the recent observations as
N(4)
p(4)
(5)
where N(4)) indicates the number of occurrences of landmark 2, in recent episodes and is periodically re-initialized when the graph is reconstructed with a new set of landmarks.
Finally, we deliver the sampled landmarks as curriculum goals to the agent, considering both temporal distance and uncertainty aspects:
aLgmax Jucert (4)) U4
(6)
where uy is uniform random variable between 0 and used to perform weighted sampling, and L top-k represents subset of L that includes the top-k elements with the largest TemporalDist" (Eq. values from initial state distribution. Our method, based on the uncertainty and temporal distance-aweer aware objective (Eq. 6). is capable of providing calibrated curriculum guidance to the agent even in environments with arbitrary geometric structures, without requiring prior knowledge of the environment or manually specified goal space. Furthermore, the curriculum guidance makes composing the goal space easier, illuminating the unexplored areas and vice versa.
Convergence to the final goal. The curriculum objective in Eq provides calibrated curriculum towards unexplored areas. In addition to this frontier-directed method, providing final goal- goal-directed guidance can further improve the efficiency of exploration especially when the agent sufficiently explored the environment, i.e., the supports of the final goal and explored area start to overlap 35 6 In order to acquire the capability to propose final gire-did curriculum, we gradually shift the direction of exploration from the frontier of the explored area to the final goal distribution. To do so,
5

---

COM [Ours]
DHRL
OUTPACE
CURROT
Shanges in Goal Space learning poprasus
Ceniosium G0a3
Daniculum Gouls
Curre Goob
AntUMaze
nina Gool
mital Dirniiution
PointMAAze
Final Goal
Duttibution
Learnnng oggres
Figure 3: Left: changes in the discretized goal space of the CQM())r)) as learning progresses. Right: visualization of the curriculum goals proposed by the CQM and baseline algorithms.
we determine whether to provide the curriculum goal وf E G that is sampled via Eq. 6 or final goal g¹ @(o') E & which the environment originally provided (v(g") E O,,o0 E EO).
We utilize a mixture distribution of curriculum goals, following the approach proposed in (5].
por = opg' + (1 - P')p
where par is the distribution of the final goal, and py= is the distribution of curriculum goals. The mixture ratio a measures whether the achieved goal distribution Pag "covers" the final goal distribution using KL divergence as a = 1/ max (3 + (P (Pog), 1). When the support of achieved goal distribution Pag (= visited state distribution) covers that of the final goal distribution pyt a produces a value close to 1, and a value close to O when the supports of both distributions are not connected.
By combining the curriculum goal objective (Eq. 66) with the mixture strategy (Eq. 7p our approach generates instructive curriculum goals towards unexplored areas and provides the curriculum goals g" the agent that "cover" the final goal distribution at the appropriate time when the agent is capable of achieving the final goal.
Planning over the graph As presented above, CQM constructs graph to restore the temporal relations between landmarks (Section 4.2) and utilizes it to calculate geodesic distances (Eq. 4 In addition to these benefits, we highlight that the graph can also provide the strength of planning, which
allows the agent to reason over long horizons effectively [9] 16 13
24].
To generate sequence of waypoints for achieving each goal, we perform shortest path planning (Dijkstra's Algorithm), following the details proposed in the previous graph-guided RL method 24 Consider task of reaching a curriculum goal 9º 9 from the current observation Og O. CQM first adds the encoded observation p(ov) to the existing graph structure. Then, it finds the shortest path between the curriculum goal and current observation return a sequence of waypoints (é(o)), WT. wn. 9d) where n indicates the number of waypoints in the shortest path. Finally, the agent is guided to achieve each decoded waypoint (w;))) during CimporalDsst((((rrr
- (wx)) (Eq. 3 timesteps, rather than achieving the curriculum goal directly. In other words, the RI. agent produces goal-conditioned action π or, w;), where o and W; observation and the waypoint (acting as a goal) respectively. After reaching the final waypoint (w,)), the agent receives the original curriculum goal, g". The only change when the agent attempts to achieve the final goal g¹ is that gf comes at the end of the sequences, (o((v), .... W2, g rather than g". In this way. the proposed approach not only provides a tailored curriculum for achieving the final goal but also allows the agent to access more elaborate instructions (waypoints) for practicing each curriculum goal.
Experiments
The main goal of the experiments is to demonstrate the capability of the proposed method (CQM) to suggest well-calibrated curriculum and lead to more sample-efficint learning, composing the goal space from the arbitrary observation space. To this end, we provide both qualitative and quantitative

---

PaintWMaare
PoirtSpiralldre
Cumta H-Ch DAmaKO
Steus
Sto0s AntUMaze
180
Stecs
AntZWWayMaze
CQM (Proposed Method)
Daloree
OUTPACE
PLR
CURROT
VDS
DHRL (Does not prodde cAr caluro
SFL
DHRL+
ALP-GMM
GoalGAN
5000s
SRcps
Figure 4: (Lower is better) Distance from the curriculum goals to the final goals (PointNMaze, PointSpiralMaze. and AntUMaze). In the 'n-way' environments with multiple goals, provide id /2 distance between the agent and the final goal at the end of the episodes, since calculating the average distance from the curriculum goal to multiple final goals is not possible.
PointWWayMaze
PointNMaze
PointSpiralMaaze
300000
AntZWayMaree
AntUMare
(Proposed Method)
OUTPACE
DHRL+
Figure 5: (Higher is better) Success rates of the results. The curves of baselines not visible in some environments as they overlap each other at zero success rate. Shading indicates a standard deviation across 4 seeds.
results in seven coal-reaching tasks including two visual control tasks, which receive the raw pixel observations from bird's-an bird's-eye and ego-centric views, respectively. (refer to Appendix [oror the detailed configurations of each task.)
We compare our approach with previous curriculum RL methods and previous graph-guided RL methods. We do not provide manually specified goal space in any of the environments; the agent could map its global X-Y coordinates from the full observation which includes all the state variables for the RL agents (e.g. angle and angular velocity of the joint, position, velocity ...). Also, the results of CQM and the baselines that utilize external reward functions (all the methods except OUTPACE (6]) are obtained by using sparse reward functions. For the baselines that could not be applied in vision- based environments 24 6 we utilize an extra autoencoder with auxiliary cine-contrastive
loss 44 13
The baselines summarized below: OUTPACE 6 proposes uncertainty and temporal distance- aware curriculum learning based on the Wasserstein distance and uncertainty classifier. CURROT 2I interpolates the distribution of the curriculum goals and the final goals based on the performance of the RL agent. GoalGAN [10] proposes the goals with appropriate levels of difficulty for the agent Using Generative Adversarial Network. PLR [18] selectively samples curriculum goals by prioritizing the goals with high TD estimation errors. ALP-GMM (37 selects the goals based on the difference of cumulative episodic reward between the newest and oldest tasks using Gaussian mixture models. VDS [50] proposes the goals that maximize the epistemic uncertainty of the action value function of the policy. DHRL 24 constructs graph between both levels of HRL,, and proposes frontier goals when the random goals are easy to achieve. However, the original DHRL could not generate curriculum goals without the help of the environment. Thus we evaluated variant of DHRL (DHRL+) with modified frontier goal proposal module and architecture (Appendix D.3 in
7

---

addition to the original DHRL. SFL [13 constructs a graph based on successor features and proposes uncertainty-basee basee curriculum goals. (refer to Appendix foo detailed implementations)
5.1 Experimental Results
First, we visualize the quantitative results to show whether the proposed method successfully and simultaneously addresses the two key challenges: 1) specifying goal space from arbitrary observation space and 2) suggesting well-calibrated curriculum to achieve the final goal. Figure 3 illustrates the curriculum goals and changes in discrete goal space (graph) of CQM as learning progresses. Each node in the graph consists of the decoded embedding vectors of VQ-VAE, and each edge represents reachability between the decoded embeddings. The graphs of CQM in the figure gradually expand towards unexplored areas the learning progresses, since the calibrated curriculum goal induces the agent to explore the unexplored area. In the opposite direction as well, the capability of providing proper curriculum goals on arbitrary geometric structures is facilitated by a compact goal space that contains semantic information which enables estimating the uncertainty and temporal distance well. As result, our method provides tailored curriculum guidance across the environments, while the baselines suffer from the absence of the manually specified goal space.
We also provide the quantitative results in Figures 4 and S Figure 4 indicates that the proposed method (CQM) can suggest a tailored sequence of goals that gradually converges to the final goal distributions while instructing the agent to achieve the increasingly difficult goals. Also, shown in Figure 5| ours consistently outperforms both the prior curriculum method and graph-RL. methods. is noticeable that CQM is the only method that shows robust performance to the variation of the goal dimension, while other methods suffer from serious data inefficiency, especially in the tasks with higher-dimensional goal space (suffering more Ant (29dims) compared Point (6dims)).
Final Distance from Goal (Lower betteri
Curriculum Goals & Planning over Quantlzed Goal Space
Curriclum learning and planning in visual control tasks. To validate the performance of the RL agent and the quality generated curriculum goals in higher dimensional tasks, We con- ducted two additional vision-based goal-reaching tasks. PointNMaze-- Viz receives only ego-centric view images to reach the goal, while PointSpiralMaze- Viz receives bird's- eye view images. Figure [ visualizes the curriculum goals in the order of the episodes, and how the agent uti- lizes the benefit of planning over the discrete goal space in order to achieve the curriculum goals. To achieve an image-basee based final goal (Goal: 8), the agent generates the sequence of images (f1, 2, 3, ... 8)) as waypoints, and tries to achieve the waypoints sequentially.
Pokettibbove Vit
+
Waypponnt
COM
OUTRACE
De
660
Figure 6: Left: the distance from the agent to the final goals (Lower is better). Right: visualization of curriculum goals and waypoints planning over the graph (CQM).
Interestingly, despite significant increase in the observation dimension, CQM does not suffer from significant performance degradation in terms of data efficiency, which indicates that CQM effectively reduces the complexity of goal space by constructing semantic goal space. We emphasize that the performance of our algorithm does not show significant differences between state-based and image- based environments (Compare PointNMaze in Figures 4 and ()). Another interesting point that CQM can fully enjoy the advantage of planning over the discretized goal space, even in vision-based control tasks where the agent does receive information about its global X-Y coordinates explicitly. These results validate that CQM possesses robust performance in terms of the dimensionality of the goal space, and the capability in extracting temporal relations between the discretized landmarks.
5.2 Ablation Studies
Curriculum guidance. First of all, we examine how important curriculum guidance is for an agent to solve goal-conditioned tasks. As shown in Figure 7 when only the final goal is provided without a tailored curriculum (-w/o Curriculum), the RL agent has difficulty achieving the final goal directly.
8

---

AntUMare
PointSpiralMaaee
PointNMare
17.5
150
assande
distance
10.0
290
603
2000
500
1083
100
200
400
300
600
753
CQM
CCM Landmark from Reploy Buff
CGM w/o Planning
CQM w/o Goal Convergance
COM w/o Curriculum
CGM wafa Graph
Figure 7: (Lower is better) Ablation study: the distance from the agent to the final goals at the end of the episodes.
Furthermore, we found that providing curriculum guidance greatly affects the goal space specification module and the absence of curriculum leads to the ill-formed discrete goal space that barely covers only the observations near the initial distribution. We provide these qualitative results in Figures [3] [Appendix E)
14
Types of the discrete goal sampling method. The proposed method (CQM) can two approaches
to sample the landmark to form the discrete goal space as introduced in Section 4.1 The first approach is to decode the embedding vectors of the codebook 11.m (e1:.)) ), and the other approach is to sample an observation batch from the replay buffer and pass it through VQ- VAE to quantize (-Landmark from Replay Buff.). As shown in Figure 7 there is no significant difference between them in terms of data efficiency. However, in terms of the stability of learning, utilizing the decoded embeddings of VQ-VAE shows better performance in some environments.
Effect of the goal convergence method. To provide a final goal-directed exploration in addition to the naïve curriculum toward the frontier areas, CQM includes a goal convergence module that guides the agent to practice the final goal after the agent sufficiently explored the environment (Section 4.3) Based on the KL divergence between the achieved goal distribution and the final goal distribution, CQM calculates the ratio of the mixture between the final goals and the frontier goals (the ratio of As shown
providing final goals as learning progresses is presented in Figure IT in Appendix in Figure 7 the absence of the final goal convergence method (-w/o Goal Convergence) results in unstable performance, since the agent repeatedly practices unexplored areas instead of converging towards the final goal even after the explored area "covers" the final goal distribution.
Effect of Graph Construction and Planning. Finally, we examine the effect of constructing graphs and planning on the performance of RL agents. As explained in section 4.2, CQM not only utilizes the decoded embedding vectors from VQ-VAE as set of discretized observations but also forms graph by capturing the temporal relations between the discrete observations. First, we evaluated CQM without graph (-w/o Graph), which does not construct a graph and measure the distance between the landmarks through naive temporal distance prediction based on Q values (TemporalDist) rather Also, we evaluate CQM without planning (-w/o Planning) since ours can optionally utilize the benefit of planning and reason over long horizons using the graph. As shown in Figure 7 CQM shows better performance than both CQM without graph and CQM without planning, especially in some long-horizon tasks (AntUMaze and PoimtSpiralMazee.
than the geodesic distance over the graph (TemporalDist
Conclusions
To solve the complex control tasks without the need for manually designed semantic goal space, we propose to solve both issues of specifying the goal space and suggesting the curriculum goals to the agent. By constructing the quantized world model using the decoded embedding vectors of the discretization bottleneck and restoring the relations between these, CQM considers both the uncertainty and temporal distance and has capability of suggesting calibrated curriculum goals to the agent. The experiments show that the proposed method significantly improves performance on various vision-based goal-reaching tasks as well as state-based tasks, preventing the performance drop in the absence of manually specified goal space.

---

Limitations and future works. While CQM shows great potential in addressing the limitations of previous studies, more research could further develop it. One area that could be explored is the use of reward-free curriculum learning methods, since CQM still requires minimal human efforts such as defining a success threshold to train agents. Also, this study only used single-code representations with VQ- VAE which would possess a limited capacity of representations, so expanding CQM to include multiple-code representations with discrete factorial representations could be an interesting future direction.
7 Acknowledgement
This work was supported by Korea Research Institute for defense Technology Planning and ad- vancement (KRIT) Grant funded by Defense Acquisition Program Administration(AAA)) (No. KRIT-CT-23-003, Development of Al researchers based on deep reinforcement learning and estab- lishment of virtual combat experiment environment)
References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[2] David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035, 2007.
[3] Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, pages 521-531. PMLR, 2021.
[4] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information processing systems, 29, 2016.
[5] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 18X00 12894, 2018
[6] Daesol Cho, Seungjae Lee, and H Jin Kim. Dutcome-directed reinforcement learning by uncer- tainty & temporal distance-aware curriculum goal generation. In Proceedings of International Conference on Learning Representations, 2023.
[7] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metrie discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1,, pages $39-546. IEEE, 2005.
[8] Ishan Durugkar, Steven Hansen, Stephen Spencer, and Volodymyr Mnih. Wasserstein distance maximizing intrinsic control. arXiv preprint arXiv:21 10. 15331, 2021.
[9] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Sysiems, 32, 2019.
[10] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
(11) Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International conference on machine learning, pages 1587-1596. PMLR, 2018
[12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
10

---

[13] Christopher Hoang, Sungryull Sohn, Jongwook Choi. Wilka Carvalho, and Honglak Lee. Suc- cessor feature landmarks for long- mang-horzoon conditiond reinforcement learning. Advances in Neural Information Processing Systems, 34:26963- 2021.
[14] Edward S Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. arXiv preprint arXiv:2303. 13002, 2023.
[15] Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. arXiv preprint arXiv:2210.1010 2022.
[16] Zhiao Huang. Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching, Advances in Neural Information Processing Systems, 32, 2019.
[17] Riashat Islam, Hongyu Zang, Anirudh Goyal, Alex Lamb, Kenji Kawaguchi, Xin Li, Romain Laroche, Yoshua Bengio, and Remi Tachet Des Combes. Discrete factorial representations as an abstraction for goal conditioned reinforcement learning. arXiv preprint arXiv:2211.00247.
2022.
[18] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Interna- tional Conference on Machine Learning, pages 4940-4950. PMLR, 2021.
[19] Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in re- inforcement learning with deep covering options. In International Conference on Learning Representations, 2020.
[20] Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Conference Machine Learning, pages 2390-2399. PMLR, 2018.
[21] Pascal Klink, Haoyi Yang. Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum rein- forcement learning via constrained optimal transport. In International Conference on Machine Learning, pages 1134-111388 PMLR, 2022.
[22] Martin Klissarov and Marlos C Machado. Deep laplacian- options for temporally-extendde exploration. arXiv preprint 11X1vv2301.11111, 2023.
[23] Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference Machine Learning, pages 5639-5650. PMLR, 2020.
[24] Seungjae Lee, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: A graph-based approach for long-horizon and sparse hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 36, 2022.
[25] Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty- rewards for outcome-driven reinforcement learning. In International conference on machine learning, pages 6346-6356. PMLR, 2021.
[26] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint arXiv: 050900297. 2015.
[27] Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar, Chore- ographer: Learning and adapting skills imagination. arXiv preprint 1...502211.13350. 2022.
(28) Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379-244 24391, 2021.
[29] Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In IJAA,, pages 2536-2542, 2017.
11

---

[30] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. Advances in Neural Information Processing Systems, 32, 2019.
[31] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count- based exploration with neural density models. In International conference on machine learning, pages 2721-2730 PMLR, 2017.
[32] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[33] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International conference on machine learning. pages 5062-5071. PMLR, 2019.
[34] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825 28380,2-800 2011.
[35] Silviu Pitís, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR, 2020.
[36] Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning, arXiv preprint 1rXIv:1903.03698, 2019.
[37] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pages 835- 853. PMLR, 2020.
[38] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32, 2019.
[39] Murray Rosenblatt. Remarks on some nonparametric estimates of density function. The annals of mathematical statistics, pages 832-837, 1956.
[40] Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint 1rXiv:1805.1106 2018.
[41] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys. Tim- othy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:/810.0222: 2018
[42] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference machine learning, pages 1312-1320. PMLR, 2015.
[43] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement lcarning with self predictedittive representations. arXiv preprint 20Xiv:2007.05529, 2020.
[44] Pierre Sermanet, Corey Lynch. Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised lcarning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141. IEEE, 2018
[45] Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. arXiv preprint arXiv:2204.02377 2022.
[46] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
12

---

[47] Hanwei Wu and Markus Flierl. Variational information bottleneck on vector quantized autoen- coders. arXiv preprint arXiv:1808.01044 2018.
[48] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference Machine Learning, pages 11920-11931. PMLR, 2021.
[49] Lunjun Zhang, Ge Yang, and Bradly Stadie. World model as a graph: Learning latent landmarks for planning. In International Conference on Machine Learning, pages 12611-12622 PMLR, 2021.
[50] Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-7659, 2020.
13

---

A Algorithm and Derivation
A.1 How CQM Works?
Specifying Goal space
Off-Policy RL
Train VQ
Practice Curriculum Goals
Replay Buffer
Find waypaints w/ Dijestra's Aig.
waypsin))
Frender
Deroder
Temporal
Triin & and x
VQ Cociebook
Sampling Curriculum Goals
Construct Graph
Sampled curiculum goal
convergence module
Final goal
VD Cotebook
nty
Denstyy
Decoder
munef + O (hak
PL uppr (1 aby
Figure
The overall diagram of CQM
A.1.1 How is data collected?
1. CQM starts with empty replay buffers.
2. After performing a step in the environment, the observed transition is stored in the replay buffer. (line 18 in Algorithm
A.1.2 How is the goal space learned?
Sample batch from replay buffer. (Line 26 Algorithm I
2. Train VQ-VAE with the batch via Eq. (Line 27 in Algorithm
A.1.3 How is the graph constructed??
1. Decode each vector embedding in VQ-Dictionary (the decoded embeddings the land- marks).
2. Using Eq. connect the landmarks with the distance below the cutoff threshold.
A.1.4 How does the agent get the curriculum goal?
1. Calculate the uncertainty of each node (landmarks) in the graph (by Eq, 5).
2. Calculate the distance the landmarks from the initial area (by Eq. 4
3. Sample a curriculum goal which is considered temporally distant and uncertain, among the nodes (landmarks) in the graph (by Eq. 6
4. Based on the value from A. the decision is made whether to provide the agent with curriculum goal final goal. Then provide the selected goal to the agent.
A.1.5 How does the agent utilize the graph?
1. After the agent obtains the curriculum goal, it can start exploring the environment.
2. CQM first finds sequence of waypoints to achieve the curriculum goal (utilizing Dijkstra's algorithm).
3. The agent is guided to achieve each waypoint, and finally, tries achieve the curriculum goal.
14

---

A.1.6 How does the curriculum goal converge to the final (desired) goal?
1. At the beginning of the learning, we have some samples of the final goals (e.g. the picture taken at the end of the maze.)
2. To get Pof. fit kernel density estimator (KDE) estimate the distribution of the final goal.
3. To get Pag+ fit a kernel density estimator (KDE) to estimate the distribution of the explored area.
4. Calculate KL divergence, and then, calculate in Eq.
5. Utilize a when we get the curriculum goal (d)
A.2 Algorithm
Algorithm 1 Overview of CQM
1: Input: final goal examples 9/ E p/. RL replay buffer B, VQ-VAE VAE replay buffer Bvq- actor network critic network Q, embeddings of VQ-VAE eim. total trainig episodes N, encoder ø, decoder sb, Environment horizon H. graph update cycle A1
2: for iterations 1...N N do
3: sample curriculum goal df from landmarks 11.m = (eemm)) using Eq.
4: Env.r Env.reset()
5: if random. coreommmuuifoommoo = O, high 1) < a then
6:
g f gf
7: else
8:
g t g + random noise
9:
end if
10:
get waypoints (6(s)), W1...., Wn, g) + Dijkstra'salgorithhnff g) for ()... H do
11:
12:
if achieved current waypoint w; or tried more than (emporallDis((((ww--))) - vb(w))) to achieve w, then
13:
current waypoint index +=1 1
end if
15:
ae f T(-[st,w))
Env.step(ar)
17:
end for
B
<- U (so, on, $1.... BvQ + BvQ (s0, 81...
19:
if N%M = 0 then
update a using kernel density estimator (KDE) [3 from Scikit-lear 34 with Gaussian kernel following [35]
21:
Update Graph GVV, E), where the vertices are the landmarks 4.mm = (e(mm) and the costs of the edges are calculated from Eq. (connect each edge if the distance is below the cutoff threshold)
22:
end if
23:
for i= do
24:
Sample minibatch from B Train æ and Q with b
25:
26:
Sample minibatch bvq from BvQ
27:
Train VQ- VAE (encoder o and decoder 4b) with byQ
28:
end for
29:
end for
A.3 Derivation
Let Q be a state-action value function, 11mm be landmarks and a E A be an action that agent executes. If the policy requires steps reach 1, from 4, the state-action value with discount factor can be represented as
Derivation of Eq. 3
(-1)7 + .... (-1)) y
(-1))
(8)
15

---

Thus, we can recover the temporal distance between I and l (= N steps) as 7" - = (1
7)Q(4, a, 1,), and we finally get
TemporalDist(4 - 14) = log, (1 + (14,)
(9)
B
Related Works
Table I: Summarized conceptual comparisons between CQM and the previous works.
Specily
Uncert. -ware
DiDist
Final Goal
Curriculum
Reason over
Goal Space
-wwre
Drect. Curriculum
Proposal
Long Lorizon
GoalGAN 10
X
X
GAN
CURROT21
x
x
x
Unlomm
PL. 18
X
x
x
X
Buffer
X
VDS 50)
x
x
X
Butfer
X
ALP-GMM
x
x
x
X
GMM
X
[GG[38]
x
x
x
Butfer
x
SkewFit 36
X
VAE
1FL
X
Gruph
DHRL
x
X
Graph
L3P[49]
X
X
DGRI, [17
7
x
x
X
X
possible (+HRAC)
Choreographer [2
/
x
x
X
X
OUTPACE
X
Bulter
CQM (ours)
Gruph
Table 1 compares our approach and previous curriculum goal generation methods 10
21 18 50
37 38 36, 6 graph-guided RL methods 13
49 24], and representation learning methods using
VQ-VAE VAE 17 27.. The characteristics compared include:
Specify Semantic Goal Space: whether the method can extract a compact goal space from the nigh-ddmensiooal goal space.
Ucert-Aware: whether the curriculum considers uncertainty.
T-Dist-Aware: whether the curriculum considers temporal distance.
Final Goal-Directed Curriculum: Whether the method utilizes target curriculum distribution for curriculum goal convergence.
Curriculum Proposal: The source of the proposed curriculum goals.
Reason over Long Horizon: Whether the method can reason over long horizons.
C
Environment Details
The agent receives the observation consisting of the angle, angular velocity, XY position, and XY velocity of the agent. The agent is initialized in [0, O] and the final goal is located at [0, 8). The agent achieves the final goal when comes within 0.5 from the final goal. Also, the size of the map is 12 by 12. The maximum timestep per episode is 150.
PointNMaze:
PointSpiralMaze: shares the other aspects with PointNMaze, but the final goal is located in [8. 16] and the size of the map 12 by 20. The maximum timestep per episode is 250.
Point3 Mayn33 It shares the other aspects with PointNMaze, but the final goal is sampled randomly among [24, -8), [24, 0], and [24, 8). The map has multiple dead ends and the size of the map is 32 by 24. The maximum timestep per episode is 200.
AntUMaze:
Agent receives the observation consisting of joint angle,joiin angular velocity, XYZ position, and XYZ velocity of the agent. The agent is initialized in [0, O] and the final goal is located at [0, 8). The agent achieves the final goal when comes within 1.0 from the final goal. Also, the size of the map is 12 by 12. The maximum timestep per episode 350.
AnteWaMMae:: It shares the other aspects with AntUMaze, but the final goal is sampled randomly among [4, 4] and [-4, -4]. The map has multiple dead ends and the size of the map is 20 by 12. The maximum timestep per episode is 200.
16

---

AntUMace
Maaee
PaintSpiaalMaaee
PointSpiraiMaane. Vir (bird's-eye view cameera)
PointNMazee
Poin3WeyMazee
PointNMaaee Viz (ego-centric camera)
SES
Figure 9: Top-down view of each environment used for evaluation. 'n-way" environments have multiple goals, and the final goal of episode is generated randomly among them.
PointNMaee- Viz: The agent receives the ego-centric camera input. The resolution of the input is 64 x 64. This environment is modified from PointNMaze, by coloring the walls. The maximum timestep per episode is 200.
PointSpiralMaze- Viz: The agent receives the Top-down view camera imput. The resolution of the input is 64 64. This environment modified from PointSpiralMaze, by converting the actions of the agent from the polar coordinates to global XY coordinates and increasing the visual size of the agent. The maximum timestep per episode is 250.
D Implementation Details
D.1 Learning VQ-VAE
Replay buffer for VQ-VAE To train the VQ-VAE for quantizing continuous observations, we employ separated (smaller size) replay buffer which contains recent trajectories compared the original replay buffer for the RL agent. By employing the separated replay buffer, we can provide VQ- VAE with more recent observations, resulting in better reflection of the areas recently explored by the RL agent.
Code resampling for VQ-VAE When we use decoded embedding vectors landmarks discretize the goal space for CQM, VQ-VAE could ignore some of the codes due to index collapse. Then, the unused codes form landmarks physically infeasible locations and occupy the capacity of the codes unnecessarily, leading to potential inefficiencies in the model's performance. Thus, we utilize the code resampling method (20 across the experiments address such problems of index collapse 20 in training VQ-VAE models.
The code resampling module CQM first keeps track of inactive codes that have been used during the previous M rollouts. Subsequently, the inactive codes are re-initialized with the embeddings of the recent observations o(o) with the probability of (B(o()))/E,dd (o(o)), where (o(o)) indicates
the Fuclidean distance from the closest code (d)())()) = min (()()) - e, e.ll)) We refer the reader to [27] for detailed explanations of code resampling.
17

---

AntUMaare
WayMaae
PointNMaree
5100250 pate
60
203
500
669
1090
2000
1005
00
143
200
300
590
600
700
COM - TD3
COM SAC
Figure 10: CQM with different RL algorithms.
D.2
Sampling landmarks
Inheriting their implementations, we also employ a landmark sparsification technique which based on the Greedy Latent(Node))
We adopt the graph construction module of prior work 49
24
Sparsification algorithm 2 49]. The detailed algorithm is shown in Algorithm
Algorithm 2 Greedy Latent(Node) Sparsification 49 24
Input: states fe1, .eck..)), sampling number k
2: Selected = []. DistList = [inf, inf, inf]
3: for 1 to 1 do
4: Farthes argmax(DistList)
5: add Farthest to Selected
T + ((em=))
6: DistFromFarthest +- [TemporalDis((arrhhest
7: DistList HemmenwvsssMinOLsserrrrrPrrttts DistList)
8: end for
9: return Selected
D.3 State-based Goal-Reaching Reaching Tasks
When we train CQM in state-based goal-reaching tasks, we utilize two well-established algorithms (TD3 algorithm [I] for Point- environments and SAC [12] for Ant- environments). following the baselines (TD3: [24 SAC: (2 We note that CQM can be built on general off aff-policy RL algorithms 226
12 11 and changing the RL agents for CQM does not lead to performance drop shown in
the figure 10
24 we utilize separate Q-networks for graph construction and policy learning. When the agent is not yet competent to achieve some goals, the experience of failure the replay buffer can lead to an overestimation of the temporal distance between the landmarks. Thus, prior work utilizes separate Q-networks for graph construction and policy learning. This simple technique maintains two different Q-networks which replay (HER) I and prevents the usage of failure trajectories to estimate the temporal distances between the landmarks which can spoil the graph construction.
Also, following
trained with different ratios of hindsight experience
Baselines. Since the original implementations of some algorithms 13 utilize discrete action policies, we replace them with continuous policies for comparison in the continuous control tasks. Also, for the baseline that could not generate curriculum goals without the help of the environment (DHRL) (24), we modify the frontier goal sampling module of the original baseline. Specifically, modified the frontier goal-shifting module the DHRL to provide curriculum goal in every episode, not only when an easy goal is given from the environment. This modification ensures that the agent can get curriculum goals even when the environment does not provide random goals. Additionally, we empirically found that the high-level agent aggravates the performance since it suffers from providing high-dimensional subgoals. Therefore, we utilize a variant DHRL (DHRL+) with modified frontier goal-shifting module and without a high-level. For the other baselines, we follow the official implementations (OUTPACE and CURROT) (2 or the implementations from https:/ / //gg thub. wwt.p/ccwwnn/cccccccccccccc currot (PLR, VDS, GoalGAN, ALP-GN ALP-GMM).
18

---

D.4 Vision-based Goal-Reaching Tasks
For the experiments in the vision based-based goal-reaching tasks, the agent needs be provided with the curriculum goals corresponding to vision inputs. However, we observed that the decoded images from VQ-VAE sometimes have inferior resolution compared to the original images, which could lead to inferior performances. Thus, during the training process for the rision-basee control, we utilized the observations that are mapped to each embedding as landmarks, instead of the decoded embeddings as curriculum goals. This technique does not require additional assumptions or computational costs, as the only overhead is saving the original images of each code. Although the mappings from observations to embedding vectors are many-to-one, we empirically found that there is no performance degradation even if we randomly pick one the observations per embedding vector as a landmark.
Baselines. Since the official implementations of some baselines are unable to solve the vision- based goal-reaching tasks, We experimented with the additional encoder which encode the high-dimensional observations into compact latent vectors. Considering the insights from prior repre- sentation learning research 23 43 using naïve autoencoder may not be fair, so we incorporated an auxiliary loss that allows for better representation learning for RL agents. To this end, we utilize an autoencoder with time-contrastic loss (7
44 used in SFL [13] which also performed exploration in vision- based goal-reaching tasks. Specifically, we employ the following auxiliary triplet loss to train the encoder for the baselines,
||x(0") <(0") < [|@(0") <00")
(10)
where o", o", and o" represent anchor, positive pair, and negative pair respectively, Also, we utilize a margin parameter = 2, following [13].
D.5
Computational Resources
Our experiments have been performed using an NVIDIA RTX A5000 and AMD Ryzen 2950X, and the entire training process took approximately 0.5-2 days, depending on the tasks.
Table 2: Hyperparameters for CQM
# of initial rollouts batch size (state) HER ratio critic Q max graph node critic hidden dim critic hidden depth actor & learning rate interpolation factor (target Q actor update freq VQ-VAE latent dimension
20
HER [ future step batch size (IMG)
150
1024
128
0.8
HER ratio graph Q graph update cycle M discount factor
1.0
300
256
0.99
RL buffer mathcal size
2500000
0.0001
critic Q learning rate target network update freeq
0.001
0,995
2
# of VQ-VAE embeddings
128
(-Viz: 32)
RL optimizer
adam
E
Additional Experimental Results
We provide quantitative results for the ablation study in Figures 13 and 14 As we analyzed in Section 5.2 CQM without each module shows inferior results across the tasks. Without goal convergence module, the agent often has difficulty in terms of progressing toward the desired final goals. Also, without planning, the agent often has difficulty in achieving long-horizon tasks.
As shown in Section 4.3 between the explored area and the region corresponding the final goal (Eq. 7 in order to perform final goal- dire-dire exploration. Tigure[[]epreee represents the ratio of final goals given to the agent instead of curriculum goals, which is calculated according the following equation.
our algorithm includes a goal convergence module based on KL divergence
1/ //x (B KDKL (Pg (Pgr/ 1
(1I)
19

---

Table 3: Task specific hyperparameters for CQM
PointSpiral
PointN
PointSpiral
PointNViz
cutoff threshold for node connection random noise for curriculum goal
7
10
5
4
VQ-VAE buffer ByQ size 3 for mixture ratio a K. for mixture ratio a
le5
le4
le5
lc4
-20
-20
-3
2e-3
2c-3
Wai
AntU
Ant2Way
cutoff threshold for node connection random noise for curriculum goal
10
30
30
VQ- VAE buffer BvQ size 3 for mixture ratio a K for mixture ratio
5e4
le4
le4
-20
-3
2e-3
2e-3
As shown in Figure 11 the ratio of providing final goals gradually increases as the learning progresses. This allows the agent practice the final goal instead of exploring unexplored areas when the agent acquires the capability to pursue the final goals at the end of the curriculum.
AntUMaaee
PointSpiralMaaze
PointNMaze
1.0
1.0
fital goal rete
finil goal rate
0.
Innal
10.0
620
400
500
1003
.00
433
102
Steps
steps
Figure 11: The ratio of providing final goals rather than curriculum goals as learning progresses (Eq.
17.5
kppp
beta- 60
kappa-0.1
40
lappa-0.25
15.0
kappa-1.0
12.5
bota-115
distance
sappa=10
distance
betal00
10.0
kappa-100 100
beta-20
7.5
7.5
5.0
5.0
0.0
0.0
0
100
200
300
400
500
100
200
380
400
500
steps (k)
steps (k)
Figure 12: Ablation study: hyperparameter sensitivity analysis (PointNMaze, Lower is better)
20

---

AntUMaze
PointSpiral
PointNMaze
Final Curriculum
Ours
Initial Curriculum
115
Final Curriculum
Convergence
w/o Goal
C
Initial Curriculum
30-
45
Final Curriculum
w/o Curriculum
E
14-
-50
Initial Curriculum
08
-22
-15
-25
36
Landmark from Replay Buffer
Buffer
Final Curriculum
E
Initial Curriculum
w/o Planning
Final Curriculum
03
Initial Curriculum
90
120
Figure 13: Ablation study: visualization of the curriculum goals proposed by the CQM.
21

---

AntUMaze
PointSpiralMaze
PointNMaze
E
5
12.5
Ours
0.9
3.0
0.0
Convergence
w/o Goal
16.0
1..0
16.0
0.0
7.5
10.0
0.99
2..5
24.53
0.0
2.5
w/O Curriculum
S
19.0
0.5
7.5
2.5
0.0
2.5
5.0
75
Landmark from Replay Buffer
27.5
S
32.5
30.0
0.0
18.0
8.0
-10
w/o Planning
27.00
3.0
15.0
20.0
3.0
5.0
300
Figure 14: Ablation study: changes in the discretized goal space of the CQM(ours) as learning progresses.
22


Reviews:
[{"review_id": "JlFkNwf3YO", "reviewer": "Reviewer_G8ip", "summary": "The paper proposed a new curriculum reinforcement learning method, CQM, that uses VQ-VAE to learn a quantized goal space, constructs a graph on the quantized goals to propose curriculum goals by distance, and learns a goal-conditional policy.", "questions": "- Eq. 6 does not have a weight factor to balance the two terms. How are the two terms distributed in practice? Will one dominate the other?", "limitations": "The limitations have been adequately addressed.\n\n> After rebuttal\n\nSome of my concerns have been addressed. I appreciate the authors' efforts, but the maze in either their submission or rebuttal doesn't match the complexity of the related work I requested for comparison. Thus I'm holding my score.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method pioneers in \"auto\" curriculum RL that learns the goal space and proposes goals all by itself.\n- The paper is clearly written and easy to follow.\n- Detailed studies on various environments are presented to demonstrate its effectiveness.\n- Code is provided for reproduction.", "weaknesses": "- $\\hat{o}_t$ in Eq. 2 should be $o_t$.\n- [Vector Quantized Models for Planning](https://arxiv.org/pdf/2106.04615.pdf) has used VQ-VAE in RL. They did not explicitly generate curricula but it's very similar to the proposed method. The authors should compare and explain the difference.\n- $Q(l_i, a, l_j)$ in Eq. 3 is not explained. I guess it's an expectation over the replay buffer?\n- The proposed mazes are long but they mostly don't have branched dead ends, which contain unseen states but do not lead to the actual goal. [This work](https://openreview.net/pdf?id=U4r9JNyNZ7) and its experiments in more complicated mazes should be compared to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer G8ip\n\n\nWe sincerely appreciate your constructive and insightful comments. We found them extremely helpful. We prepared our response below:\n\n---\n\n**Q1. Differences between ‘Vector Quantized Models for Planning’ [1] and our research.**\n\n**A1.** Thank you for your helpful suggestions on the need for a comparison with the prior work. In this answer, we want to emphasize that there are very few similarities between our research (CQM) and ‘Vector Quantized Models for Planning (VQM)’ [1] except for the use of VQ-VAE.\n\n1. Our algorithm **does NOT require a dataset to pre-train VQ-VAE before learning** an agent, while VQM needs it [1].\n    - Our algorithm enables the RL agent to start learning from an initial distribution without any prior information or dataset about the environment. To do so, we simultaneously learn VQ-VAE and the RL agent with a curriculum.\n    - On the other hand, the mentioned algorithm (VQM) [1] is divided into two stages, where in stage 1, it pre-trains VQ-VAE using a pre-existing dataset. VQM has amassed 101,325,000 episodes to gather the dataset for pre-training VQ-VAE in the Deepmind Lab Env(section 5.2.1 in the VQM paper). To do so, they randomly placed the agents in the environment and utilized A2C algorithm. *It is crucial to consider how strong of an assumption it is to possess such pre-collected data from diverse locations in the environment even before initiating the main policy.*\n1. Ours perform goal-conditioned decision-making, while VQM [1] does not.\n2. Ours perform curriculum learning: we generate the goals autonomously.\n3. Our work is compatible with general off-the-shelf RL algorithms while VQM [1] is based on MuZero (which incorporates MCTS).\n4. VQM [1] corresponds to single-layer RL, while we correspond to multiple-layer RL.\nIn VQM, planning refers to determining each individual action. In contrast, planning in our algorithm carries a higher-level meaning, where nodes are connected by multiple transitions. Thus, the agents execute multiple goal-conditioned actions to facilitate movement between graph nodes in CQM. In other words, our approach can be seen as a hierarchical structure (graph at the high level, and RL agent at the low level).\n\nTo the best of the author’s knowledge, ours is the first VQ-VAE work capable of exploration without relying on pre-training datasets or extra exploration policy.\n\n**Q2. This work [2] and its experiments in more complicated mazes should be compared to (e.g. branched dead ends).**\n\n**A2-1.** We appreciate your comment. However, we would like to inform you that the experiment of our work already includes branched dead ends. (e.g. Point3WayMaze, Ant2WayMaze - the maps of each environment are visualized in the Appendix in the supplementary material.)\n\n**A2-2.** Following your interesting suggestion, we created mazes (*Fig 1.* in the PDF attached to the global response) with a similar geometric shape to the maze used in the prior work [2] and evaluated our algorithm in it. Although the prior work was performed on a low-dimensional state-based environment, we took it a step further and expanded the scope to an image-based setup, thus exploring a more complex and challenging scenario.\n\nThe results are shown in *Fig. 2.* in the PDF. Interestingly, even without further hyperparameter tuning from another setting (PointSpiralMaze-Viz), we achieved remarkable success in this extended setting. We believe that these results demonstrate the capability of CQM to be applied in even more intricate environments.\n\n**Q3. Explanation of $Q(l_i, a, l_j)$ in Eq. 3.**\n\n**A3.** Throughout our manuscript, the Q-function follows the notation provided below.\n\n$Q(\\mathrm{current\\  state}, \\mathrm{action},\\mathrm{goal})$\n\nThus, $Q(l_i, a, l_j)$ indicates the goal-conditioned state-action value where goal, action, and state is $l_i$, $a$, and $l_j$, respectively. (We will incorporate this explanation into the final version.)\n\n**Q4. Questions and minor comments**\n\n**A4-1.** Regarding $\\hat{o}_t$: We appreciate your comment. Once we have the opportunity to make manuscript revisions, we will promptly correct it.\n\n**A4-2.** Regarding the weight factor to balance the two terms in Eq. 6.\n\nActually, there are various ways to implement the curriculum goal sampling part.\n\n- One approach is to implement it by adjusting the balance between the two using a weight factor. (In this case, finding a balance between the two is important.)\n- Selecting the curriculum goal based on one criterion using top-K filtering and then sampling based on the remaining criterion also yields good performance.\n\nWe have found that both of these approaches can yield successful results with appropriate hyperparameters. While the equation in the manuscript corresponds to the former approach, we found that the latter approach tends to be more robust to the hyperparameters. We will include the weight factor & implementation details in the Appendix of the final version.\n\n---\n\nThank you again for your valuable and insightful review.\n\nPlease let us know if our responses have addressed your questions. If anything needs further clarification, please do not hesitate to let us know as soon as possible."}, {"review_id": "4ldJFwUGj1", "reviewer": "Reviewer_paJb", "summary": "This paper introduces Curriculum RL with Quantized World Model (CQM), a novel approach that leverages a VQ-VAE to create a discretized goal space and constructs a graph structure over it. CQM further proposes a curriculum strategy based on uncertainty and temporal distance to guide the learning process. The authors evaluate the effectiveness of CQM through experiments conducted on variants of PointMazes and AntMazes, which serve as benchmarks for Hierarchical Reinforcement Learning.", "questions": "- How does performance vary when \\alpha and \\beta are changed? Could you provide insights into the selection process for these hyperparameters? Is it possible to set them automatically?\n- Regarding planning over the graph, what happens if an agent is unable to directly reach a specific node w_{i}? Could the system handle a scenario where, after TemporalDist(\\phi(w_{i-1}), \\phi(w_{i})), the agent conditions its behavior on reaching w_{i+1} instead?\n- Could you please explain how the landmarks were sampled from the replay buffer for the \"CQM Landmark from Replay Buff\" depicted in Figure 7? What criteria or process were used to select the landmarks from the replay buffer?\n\nMinor: \n- It appears that there are two different symbols used for \\beta, one in Equation 2 and another in \\alpha = 1 / \\max (\\beta + \\kappa D_{\\text{KL}} (p_{g}^{f} || p_{ag}), 1). I would recommend using distinct symbols to avoid confusion.\n- In Figure 7, the y-label representing the distance is partially hidden.\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to follow up\n- Extensive experiments\n", "weaknesses": "One significant weakness of this paper is the lack of clarity regarding why using the representation from VQ-VAE is suitable for graph-building. It is crucial to consider the temporal distances between nodes to adequately cover the visited state space, especially given the limited number of nodes. The proposed goal representation learning scheme by VQ-VAE does not appear to take into account these temporal distances, raising questions about the efficacy of the proposed representation in creating a meaningful semantic goal space. It would greatly benefit the paper to provide a more thorough explanation and justification for the use of VQ-VAE in graph construction and its ability to capture temporal relationships. \n\nFurthermore, the proposed goal representation learning scheme should be compared to prior work on representation learning, such as NORL [1] or LESSON [2]. A comprehensive comparison would help establish the novelty and effectiveness of the proposed approach in relation to existing methods. Additionally, it remains unclear how the proposed goal representation learning scheme outperforms or differs from the simple approach of utilizing farthest point sampling from the replay buffer, which warrants further investigation and comparison. \n\nMoreover, the paper introduces new hyperparameters (\\alpha, \\beta, and \\kappa) for curriculum goal generation. However, it is unclear how these hyperparameters were determined and how the performance varies when these hyperparameters are varied. Additionally, it would be valuable to compare the costs associated with hyperparameter search between CQM and the baseline methods. \n\n[1] Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine, “Near-Optimal Representation Learning for Hierarchical Reinforcement Learning”, ICLR 2019.\n\n[2] Siyuan Li, Lulu Zheng, Jianhao Wang, Chongjie Zhang, “Learning Subgoal Representations with Slow Dynamics”, ICLR 2021\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer paJb\n\nWe sincerely appreciate your constructive comments. We found them extremely helpful. We prepared our response below:\n\n---\n\n**Q1. Lacks clarity on why using the representation from VQ-VAE is suitable for graph-building. VQ-VAE does not appear to take into account temporal distances.**\n\n**A1.** We appreciate your valuable comment. While we agree that explicitly incorporating temporal distance information could potentially lead to better results, we would like to kindly emphasize that our current approach with VQ-VAE already demonstrates impressive coverage of the environment and consistently outperforms the curriculum learning baselines. Also, even though the temporal distance information is not considered in learning VQ-VAE, it eventually becomes incorporated into the graph construction process when generating edges. (It is worth noting that prior studies utilizing VQ-VAE to quantize state space (DGRL[3], Choreographer[4]) have achieved favorable results without explicitly considering the temporal distance.)\n\nFurthermore, quantizing the dimensional space yields clear advantages for graph construction, notably in relation to reducing computational costs, enhancing robustness against Q-value errors, and enabling effective quantification of uncertainty. (more detailed explanations are in **A3.**)\n\n**A1-1.** We present the images of decoded embeddings demonstrating that the VQ-VAE already provides sufficient coverage of the visited state (please refer to *Fig. 3* in the PDF uploaded to the global response).\n\n**A1-2.** Also, we conducted additional experiments to incorporate temporal distance information into VQ-VAE training (*Fig. 4* in the PDF):\n\nTo investigate whether temporal distance information leads to practical performance improvements, we added auxiliary self-supervised loss in learning the latent space of VQ-VAE (CQM+SSL). The auxiliary loss minimizes the distance in the latent space between neighboring observations while enforcing the distance in the latent space between temporally distant observations to become larger than the margin. We found that simply adding this objective could not show improved performance in the current stage, and we conjecture that VQ-VAE already possesses the capability to construct a graph that appropriately covers the environments well. Nevertheless, we agree that testing the effectiveness of injecting temporal distance information into CQM would be valuable.\n\n**Q2. Comparison with prior work on representation learning, such as NORL or LESSON.**\n\n**A2.** Following your suggestion, we replaced the VQ-VAE part of CQM with LESSON [8] (CQM-LESSON) and compared the performance with the original CQM. For this experiment, we modified only the representation learning part while keeping the rest of the modules intact for fair comparisons. (Since the proposed algorithm not only focused on representation learning; it encompasses curriculum goal generation and other parts). The results are provided in the PDF (*Fig. 5*). Through the experiments, we found that the original CQM consistently shows better goal-reaching performances than CQM-LESSON.\n\n**Q3. How the proposed goal representation learning scheme outperforms or differs from the simple approach of utilizing farthest point sampling (FPS) from the replay buffer.**\n\n**A3.** We compared CQM with the simple approach of utilizing farthest point sampling from the replay buffer (CQM-onlyFPS), and the results are provided in the attached PDF (*Fig. 5* and Table 1). Also, we analyzed the factors contributing to the superior performance of origin CQM compared to CQM-onlyFPS:\n\n- Since the computation complexity of FPS is $\\Omega(n^2)$, where n is the number of observations in the initial sample of the FPS algorithm, using only FPS introduces significant computational overhead (Table 1.).\n\n- Additionally, FPS is not robust to temporal distance estimation error and thus, the error affects the graph construction. (This problem has also been mentioned in prior work DHRL [7].)\n\n- VQ-VAE enables count-based uncertainty measurement, but it is hard in onlyFPS option to measure the uncertainty.\n\n**Q4. Regarding the hyperparameters ($\\alpha$, $\\beta$, and $\\kappa$)**\n\n**A4.** Our approach utilizes the mixture distribution of curriculum goals following MEGA[5] and thus, we also utilize a heuristic way to determine it. We conducted a grid search for hyperparameter tuning. *Throughout this process, we made fewer than 20 attempts per environment.* Also, the hyperparameter values found for PointNMaze were directly used without modification for all state-based Point environments, and the values for AntUMaze were directly used for all remaining environments. We provide the performance of CQM under varying hyperparameters in the attached PDF (*Fig. 6*).\n\n**Q5. What happens if an agent is unable to directly reach a specific node $w_{i}$?**\n\n**A5.** If the agent could not achieve the current waypoint $w_{i}$ even after spending TemporalDist($\\psi(w_{i-1}), \\psi(w_{i})$) timestep, CQM updates the current tracking waypoint to the next waypoint $w_{i+1}$ (as shown in line 12 of Algorithm 1 in Appendix A).\n\n**Q6. How the landmarks were sampled from the replay buffer for the \"CQM Landmark from Replay Buff\" depicted in Figure 7?**\n\n**A6**. For \"CQM Landmark from Replay Buff\" depicted in Figure 7 (Ablation study graph), we sampled states from the replay buffer and passed them through VQ-VAE. To do so, we randomly sampled a total of 1000 states, and after passing through VQ-VAE to quantize it. Thus, the states that fall into the same code are merged into a single state. Finally, the decoded states after this process form the nodes of the graph.\n\n---\nThank you again for your valuable and insightful review.\n\nPlease let us know if our responses have addressed your questions. If anything needs further clarification, please do not hesitate to let us know as soon as possible."}, {"review_id": "sZE1eLoUqa", "reviewer": "Reviewer_uT8t", "summary": "This paper proposes a curriculum RL approach using a VQ-VAE to learn a goal space, and then construct a graph with the VQ-VAE codes as nodes, and a temporal distance estimate of the Q-value as weighted edges. The curriculum is then constructed by doing frontier-based exploration on this graph, by sampling goals based on their visitation count and temporal distance. In addition, the graph can be used for planning intermediate goals as waypoints for the agent. The paper is well written, and it has proper benchmarks and ablation study. The authors use the term \"world model\" in their paper, but I would argue they don't really train a world model, rather a representation of the observation space. I would omit the use of \"world model\" in a further revision of the manuscript.", "questions": "- How to get a distribution over the goal, encoding a set of exemplar goal observations and counting the bins of the VQ-VAE they get encoded in?\n\n- In the Planning section the authors use state s_0 from S, although the previous MDP description uses observation o_0 from O. I assume these are the same or have you shifted to a POMDP setup?\n\n- In the AntUMaze, the performance seems to deteriorate after 600k steps, i.e. the curriculum goals seem to get farther from the goal. Any insight on what is happening there?\n\n- Since the goal space is trained as a VQ-VAE on individual observations, I think it will break when the environment is ambiguous, i.e. if the agent would have to traverse rooms, but 2 rooms would look identical, these observations would map onto the same goal and the graph might collapse?", "limitations": "- I think one limitation that is currently not touched upon is that this approach requires that the environment is not ambiguous (i.e. you don't see the same observations at different places in the environment). ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel curriculum RL method, which learns a goal-space on high-dimensional pixel observations. Turning the VQ-VAE codebook into a graph, weighted by estimated temporal distance from the Q-value is a powerful idea for both curriculum learning as well as planning and long-range credit assignment. ", "weaknesses": "- The authors use the term \"world model\", however what they present is a discrete latent representation of the observation space. A world model would entail to e.g. also equip it with an action-conditioned forward dynamics model, as typically used in model-based RL. I would not call this a world model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer uT8t\n\nWe sincerely appreciate your constructive and insightful comments. We found them extremely helpful. We prepared our response below:\n\n---\n\n**Q1. Regarding the term “world model”**\n\n**A1.** Thanks for your valuable comment. We understand your concern regarding the term \"world model\" used in our paper, which may confuse the readers as it is also used in general model-based RL.\n\nWe would like to clarify that the term \"world model\" we used is derived from the paper \"World Model as a Graph: Learning Latent Landmarks for Planning (L3P) [6]\". In the L3P paper, the term \"world model\" refers to a graph-structured environment, since it can also be viewed as a multi-step transition at a higher-level perspective.\n\nAfter considering your input, we understand the need for more careful explanations when using the term \"world model\". We will incorporate additional explanations in the introduction of our paper, providing context and clarifying the specific meaning. If you believe that even with these clarifications, the term \"world model\" might still lead to confusion among readers, please kindly inform us, and we will explore alternative terms to effectively convey the intended meaning.\n\n**Q2. How to get a distribution over the goal, encoding a set of exemplar goal observations and counting the bins of the VQ-VAE they get encoded in?**\n\n**A2.** If you are asking about the goal distributions ($p_{ag}$ and $p_{g^f}$) in the section “Convergence to the final goal”(lines 201-218), we utilize kernel density estimator (KDE) and fit the KDE model to the goal distribution which is sampled from replay buffer. Also, following the prior work (MEGA [5]), we utilize the default Gaussian kernel for KDE.\n\n**Q3. In the Planning section the authors use state s_0 from S, although the previous MDP description uses observation o_0 from O. I assume these are the same or have you shifted to a POMDP setup?**\n\n**A3.** We appreciate your comment. As you mentioned, the symbols $s_0 \\in S$ carry the same meaning as $o_0 \\in O$. Once we have the opportunity to make manuscript revisions, we will promptly correct it.\n\n**Q4. In the AntUMaze, the performance seems to deteriorate after 600k steps, i.e. the curriculum goals seem to get farther from the goal. Any insight on what is happening there?**\n\n**A4.** Thank you for your comment. The performance drop in AntUMaze after 600k is mainly caused by the low $\\alpha$ values after the discovered area covers the final goal area. This problem could be addressed with simple hyperparameter tuning, and we have just adjusted both beta and kappa to x1/2 to get better performance. We would like to share the new graph (we kindly ask you to refer to *Fig. 7.* in PDF uploaded to the global response).\n\n**Q5. Since the goal space is trained as a VQ-VAE on individual observations, it will break when the environment is ambiguous, i.e. if the agent would have to traverse rooms, but 2 rooms would look identical, these observations would map onto the same goal and the graph might collapse?**\n\n**A5.** Similar to the baselines and the prior curriculum learning studies (OUTPACE, SFL, VDS, DHRL, ALP-GMM, GoalGAN, Skew-fit, CURROT, PLR...), our algorithm also does not consider POMDP. (CQM inherits the limitations of the prior curriculum learning algorithms at this point. - We will gladly include this in the limitation part in the final version.)\n\nConsequently, in scenarios involving indistinguishable rooms, additional modules are necessary to differentiate between them. Thus, we think that exploring the curriculum learning methods for partially observable environments is also a valuable and thought-provoking research topic for future direction. We believe that incorporating methodologies that can account for past observations could be a promising direction for addressing these problems.\n\n---\n\nThank you again for your valuable and insightful review.\n\nPlease let us know if our responses have addressed your questions. If anything needs further clarification, please do not hesitate to let us know as soon as possible."}, {"review_id": "shYNMsuyez", "reviewer": "Reviewer_eib8", "summary": "The method works as follows. Graphs are built by (1) quantizing visual observations to create a goal space & (2) creating temporal relations over goal space vectors. Curriculum goals towards a \"user\"-specified goal are made using this graph.\nA VQ-VAE is used to create the goal space where goals are decodings of one of k trainable embedding vectors. The authors also study an ablation where landmarks are decoded from the nearest neighbor goal encodings over replay buffer observations.\nGraphs are derived by connecting edges based on a metric that exploits having rewards by -1 at all states except for the goal state. This is motivated by wanting edges to capture the geodesic between nodes.\nA curriculum for exploring the graph is created by selecting nodes with high uncertainty (or low count via count-based methods) and high temporal distance. Curriculum goals are goals that maximize this sum of this uncertainty and geodesic distance away from the initial state.\nThe method uses Dikstra's algorithm to generate plans towards goals.\n\nThey compare against many baselines in the literature and consistently find a higher success rate. They also visualize the curriculum goals produced by their method.", "questions": "Across experiments, does the agent encode first-person observations, top-down observations over a map, or both? This seems important to how you quantize the observations. If top-down, this seems like a strong assumption. Can you justify it? Methods like SFL use first-person observations. At least some of the environments that you study permit first-person observations (e.g. AntMaze). I know you do only ego-centric for PointNMave-Viz but it's not clear for the other experiments.\n\nWhat assumptions does this method make over the reward function? ", "limitations": "This limitations section is fine.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "originality: The method is novel to the best of my knowledge.\nquality: The paper has high quality. The experiments are very though and the authors also do an analysis.\nclarity: The experiments are relatively clear.\nsignificance: The paper seems to produce better performance than many methods in the literature with similar assumptions (e.g. exploiting diikstra's algorithm).", "weaknesses": "The paper could strongly benefit from some figures that describe the details of the method. The method is relatively complicated (though not more complicated than comparable methods in the literature). Due its complexity, it's hard to understand how the method works from figure 1. I'm somewhat familiar with this literature so I think understand their method section was easier but other readers may find it more challenging. Another figure would strongly improve the clarity of this paper.\n\nThe baseline description method describes how the baselines work but it does not describe what comparing against each baseline (or sets of them) tells us about CQM. Without this, while the results are promising, it's hard to get a takeaway message. For example, OUTPACE also proposes an uncertainty and temporal distance aware curriculum. Comparing against OUTPACE tells us....? It might be the importance of jointly considering uncertainty and temporal distance but that isn't clear in the text. I recommend explicating this a bit more. I think it's fine if baseline methods are grouped together for describing the point of the comparison.\n\nGiven the complexity of this method, it would strongly benefit from an algorithm describing, how is data collected, how is the goal space learned, how is the graph constructed, etc. In particular, which of these are done in tandem and which are done in sequence is not clear.\n\nOverall, this method seems promising, the motivation is clear, and the results are also promising. I think the biggest drawback is clarity right now. It's not that easy to understand the method. Right now I lean towards accept but I strongly encourage the authors to add figures/explicit algorithm/both for the methods section. This would likely push my score to a 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer eib8\n\nWe sincerely appreciate your constructive and insightful comments. We found them extremely helpful. We prepared our response below:\n\n---\n\n**Q1. The paper could strongly benefit from some figures that describe the details of the method. Another figure would strongly improve the clarity of this paper.**\n\n**A1.** Thank you for your helpful suggestions on the need for some figures for CQM. Following your valuable comment, we have prepared a diagram (*Fig 8.* in the PDF attached to the global response) that provides a clearer understanding of each component of our algorithm. We will ensure that these visuals are incorporated into the final version of our paper. If there are any other ideas to further enhance the reader’s understanding, please do not hesitate to let us know. We will be happy to comply.\n\n**Q2. Explanation is required regarding the differences between CQM and the baselines. While the results are promising, it's hard to get a takeaway message.**\n\nWe agree that a detailed comparison with the baselines can provide readers with more insightful takeaway messages. Following your suggestion, we summarized the additional comparisons below. We kindly ask you to refer to Table 1 in Appendix B (in supplementary material) when digesting the explanation below, for better understanding.\n\n**A2-1.** vs GoalGAN, PLR, ALP-GMM, and VDS\n\nFirstly, these methods each employ curriculum learning strategies, but they do not have a convergence mechanism to the final goal distribution. Thus, the agents learned with these methods repeatedly practice unexplored areas instead of converging towards the final goal even after the explored area “covers” the final goal distribution, leading to performance degradation.\n\n**A2-2.** vs CURROT: CURROT can execute curricula directed towards the final goal. However, it requires an assumption that the distance between the samples can be measured by the Euclidean distance metric. Thus, it suffers in the environments such as N-shaped or Spiral-shaped Maze.\n\n**A2-3.** vs DHRL and SFL\n\nThese methods are the RL algorithms employing graphs. Unlike our method, DHRL does not autonomously generate its curriculum goals; rather, it relies on externally supplied goals that are sampled from feasible areas within the environment. Consequently, DHRL's performance heavily depends on the availability of these externally provided random goals.\n\nAlthough SFL allows exploration without external goal input, it does not execute final goal-directed curricula as the baselines mentioned in **A2-1.** Moreover, it solely conducts exploration based on uncertainty, unlike our approach which encompasses both uncertainty and temporal distance considerations.\n\n**A2-4.** vs OUTPACE\n\nIn distinction to the studies above, OUTPACE is the only curriculum goal generation method that considers both uncertainty and temporal distance. However, there are two significant distinctions between OUTPACE and CQM: \n\n(a) OUTPACE does not have a module to learn the goal space while CQM automatically defines the goal space and quantizes it.\n\n(b) OUTPACE predicts uncertainty using the meta-learning-based method Conditional Normalized Maximum Likelihood (CNML) while CQM utilizes count-based uncertainty prediction with quantized state space.\n\nCNML struggles with substantial prediction errors when there is no prior knowledge of a manually specified goal space (since CNML is not scalable to high-dimensional observations). Conversely, our approach employs a straightforward count-based uncertainty mechanism via state space quantization, enabling the generation of suitable curriculum goals even in high-dimensional input environments without a manually specified goal space.\n\n**Q3. Clarification on each part of CQM**\n\n**A3.** To enhance reader comprehension, we have succinctly presented how each module operates in the CQM algorithm. (Due to the word limit, we kindly ask you to refer to our answer (**G-A1**) in the global response). Also, following your comment, we will incorporate this alongside the diagram mentioned in A1 in the final version.\n\n**Q4. About the environments (first-person observation / top-down observation)**\n\n**A4.** Among the two visual input environments we experimented with, PointNViz is a first-person environment, while PointSpiralViz is a third-person environment. While it is true that SFL conducted experiments in a first-person environment similar to ours, it's important to note that they used a discrete action space rather than a continuous action space like ours, making our setting a more challenging problem. (Additionally, SFL utilizes a pre-trained ResNet backbone, whereas our approach does not assume access to a pre-trained network)\n\n**Q5. What assumptions does this method make over the reward function?**\n\n**A5.** The results of CQM and the baselines in this work are obtained by using sparse reward functions. In sparse reward settings, the agent receives a reward of 0 when it succeeds in reaching a goal and -1 otherwise. The criteria for success in each environment are indicated in Appendix C. Environment Details\n\n---\n\nThank you again for your valuable and insightful review. \n\nWe will gladly include the figures and descriptions above in the final version of the manuscript, either in the main body or as an appendix. Also, please let us know if our responses have addressed your questions. If anything needs further clarification, please do not hesitate to let us know as soon as possible."}, {"review_id": "gnEI20jGTy", "reviewer": "Reviewer_vggh", "summary": "- The proposed method addresses the challenge of learning a curriculum in goal-conditioned policies, which is a significant problem. Previous research on curriculum in goal-conditioned policies has often overlooked the importance of learning the underlying semantic goal space. This paper builds upon a recently proposed method that utilizes VQVAE to learn the semantic goal space and extends it to incorporate the learning of a curriculum. By selecting a sequence of sub-goals, this curriculum-based approach aids in achieving the final goal.\n\n- The paper compares the proposed method to many different SOTA baselines. \n\n", "questions": "- I'm curious about how the performance varies by varying the number of codes in VQ-Dictionary.\n- It will also be useful to study if the performance of the proposed method can be improved by using self-supervised objectives for learning visual representations as done in DGRL.", "limitations": "Refer to Weaknesses. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well-written. \n- The paper effectively compares the proposed method with various baselines, providing a thorough analysis.\n- Additionally, the paper conducts ablations to examine the impact of different design decisions made in this research.", "weaknesses": "- It will be valuable to assess the effectiveness of the proposed method on both manipulation tasks and more intricate navigation tasks like 3D environments (ex. Habitat, AI2-Thor etc)\n- It seems restrictive to the reviewer to use single-code representations with VQ-VAE so would be interesting to see how the results change if using multiple codes like in DGRL. (The reviewer understands this is mentioned as future work, but this seems a severe limitation if the proposed method works only with single code and not multiple codes).\n\n- A minor comment: To enhance clarity, it is recommended to explicitly mention in the introduction that prior research, such as DGRL [1], has proposed and emphasised the significance of learning a semantic space for goal specification. It was only later in the paper that I realised vector quantisation had been proposed in previous work as an abstraction for goal-conditioned RL.\n\n[1] Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning, https://arxiv.org/abs/2211.00247\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer vggh\n\nWe sincerely appreciate your constructive and insightful comments. We found them extremely helpful. We prepared our response below:\n\n---\n\n**Q1. It seems restrictive to the reviewer to use single-code representations with VQ-VAE so would be interesting to see how the results change if using multiple codes like in DGRL.**\n\n**A1.** Although we have left utilizing multi-code representation for CQM as future work, we agree that providing some insights into CQM's compatibility with multi-code representation would be highly valuable. To explore this possibility, we conducted experiments where we expanded the single-code representation into a multi-code representation without tuning any other hyperparameters. The numbers of embedding vectors in the VQ-dictionary (i.e. codebook) for each setting we experimented with are as follows:\n\n|  | # of embedding vectors in VQ-dictionary |\n| --- | --- |\n| 1 (single-code repr.) | 128 |\n| 2(multi-code repr.) | 32 |\n| 4(multi-code repr.) | 16 |\n\nFor the experimental results, we kindly ask you to refer to *Fig. 9* in PDF uploaded to the global response. While there is some difference in convergence speed, they eventually exhibited successful performance.\n\n**Q2. I'm curious about how the performance varies by varying the number of codes in VQ-Dictionary.**\n\n**A2.** Following your comment, we attach an additional result with a varying number of embedding vectors (codes) in the VQ-dictionary. (*Fig. 10* in PDF)\n\n**Q3. It will also be useful to study if the performance of the proposed method can be improved by using self-supervised objectives for learning visual representations as done in DGRL.**\n\n**A3.** Thank you for suggesting another interesting experiment. Following your comment, we trained the encoder by adding a self-supervised objective in addition to the VQ-VAE objectives.\n\nThe DGRL [3] algorithm claims to be compatible with any off-the-shelf self-supervised learning method. For this experiment, we utilized a self-supervised representation learning method (LESSON [8]) which is more recently proposed than DeepInfoMax in DGRL.\n\n$$\\min_{\\phi} \\mathbb{E}[L_2||\\phi(s_t)-\\phi(s_{t+1})|| + \\max(0, m-L_2||\\phi(s_t)-\\phi(s_{t+c})||)]$$\n($L_2||\\cdot||$ = L2 distance)\n\nLESSON adopts the technique of triplet loss: it aims to minimize the distance in the latent space between neighboring states while enforcing that the distance in the latent space between states separated by c steps (c=10 in our case) is larger than the margin parameter m.\n\nThe training results (CQM+SSL) are included in the PDF (*Fig. 4*). Through the experiment, we found simply adding Self-Supervised Learning objectives could not show improved performance in the current stage. Nevertheless, we agree that exploring the appropriate SSL objectives for better performance could be a valuable research topic for future direction.\n\n**Q4. It will be valuable to assess the effectiveness of the proposed method on both manipulation tasks and more intricate navigation tasks like 3D environments (ex. Habitat, AI2-Thor etc)**\n\n**A4.** Thank you for your insightful suggestion. In the field of curriculum learning, most of the recent studies (CURROT, OUTPACE, GoalGAN, ALP-GMM, VDS…) have showcased their performance only in state-based RL settings. Thus, it's noteworthy that CQM not only outperforms these prior approaches in state-based scenarios but also demonstrates its efficacy in pixel-based environments. Thus, while we didn’t cover all the complex experiments in this study, we believe that CQM can provide valuable insights to future studies which try to establish effective curriculum learning in such complex settings.\n\n**Q5. To enhance clarity, it is recommended to explicitly mention in the introduction that prior research, such as DGRL**\n\n**A5.** We appreciate your comment. However, we would like to inform you that our manuscript already includes the references of prior studies (including DGRL [3]) in Section 2 - Related Works (lines 109-116). However, we agree that there is room for improvements (e.g. more detailed explanation of the prior works). Therefore, we will take your feedback into account and provide an improved one in the final version.\n\n---\n\nThank you again for your valuable and insightful review. \n\nWe will gladly include these experimental findings in the final version of the manuscript, either in the main body or as an appendix.\nAlso, please let us know if our responses have addressed your questions. If anything needs further clarification, please do not hesitate to let us know as soon as possible."}]

Author rebuttal:

Dear reviewers,

We sincerely appreciate your constructive and insightful comments.

We have prepared our responses at the bottom of each review you provided. This global response includes:

- [Additional Results] In this global response, **we attached a PDF containing the experimental results** for the responses to each reviewer. *(The button labeled "PDF" is at the bottom of this message)*
- [Optional] Also, In case you encounter difficulty in quickly recapping our pipeline, we have included a brief overview of the key components of our algorithm below.

Thank you again for your valuable review.
If anything needs further clarification, please let us know as soon as possible.

---
**G-A1. How CQM works?**

(a) How is data collected?

- CQM starts with empty replay buffers
- After performing a step in the environment, the obtained transition information is stored in the replay buffer. (line 18 in Algorithm 1. (Appendix A.))

(b) How is the goal space learned?

- Sample batch from replay buffer (line 26 in Algorithm 1. (Appendix A.))
- Train VQ-VAE with the batch via Eq. 2. (line 27 in Algorithm 1. (Appendix A.))

(c) How is the graph constructed?

- Decode each vector embeddings in VQ-Dictionary (the decoded embeddings are the landmarks).
- Using Eq. 3., connect the landmarks with the distance below the cutoff threshold.

(d) How the agent gets the curriculum goal

- Calculate the uncertainty of each node (landmarks) in the graph (by Eq. 5).
- Calculate the distance of the landmarks from the initial area (by Eq. 4).
- Sample a curriculum goal which is considered temporally distant and uncertain, among the nodes (landmarks) in the graph.
- Based on the $\alpha$ value from (f), the decision is made whether to provide the agent with a curriculum goal or a final goal. Then provide the selected goal to the agent.

(e) How does the agent utilize the graph?

- After the agent obtains the curriculum goal, it can start exploring the environment. 
- CQM first finds a sequence of waypoints to achieve the curriculum goal (utilizing Dijkstra’s algorithm).
- The agent is guided to achieve each waypoint, and finally, tries to achieve the curriculum goal.

(f) How the curriculum goal converges to the final (desired) goal?

- At the beginning of the learning, we have some samples of the final goals (e.g. the picture taken at the end of the maze.)
- To get $P_{g^f}$, fit a kernel density estimator (KDE) to estimate the distribution of the final goal.
- To get $P_{ag}$, fit a kernel density estimator (KDE) to estimate the distribution of explored area.
- Calculate KL divergence, and then, calculate $\alpha$ in Eq. 7.
- Utilize the $\alpha$ when we get the curriculum goal (d)

**G-A2. References.**

[1] Ozair, Sherjil, et al. "Vector quantized models for planning." *international conference on machine learning*. PMLR, 2021.

[2] Kim, Seongun, Kyowoon Lee, and Jaesik Choi. "Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skills." (2023).

[3] Islam, Riashat, et al. "Discrete factorial representations as an abstraction for goal conditioned reinforcement learning." *arXiv preprint arXiv:2211.00247* (2022).

[4] Mazzaglia, Pietro, et al. "Choreographer: Learning and adapting skills in imagination." *arXiv preprint arXiv:2211.13350* (2022).

[5] Pitis, Silviu, et al. "Maximum entropy gain exploration for long horizon multi-goal reinforcement learning." *International Conference on Machine Learning*. PMLR, 2020.

[6] Zhang, Lunjun, Ge Yang, and Bradly C. Stadie. "World model as a graph: Learning latent landmarks for planning." *International Conference on Machine Learning*. PMLR, 2021.

[7] Lee, Seungjae, et al. "DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning." *Advances in Neural Information Processing Systems* 35 (2022): 13668-13678.

[8] Li, Siyuan, et al. "Learning subgoal representations with slow dynamics." *International Conference on Learning Representations*. 2020.

Meta-review: After extensive discussion with the authors, all reviewers recommend acceptance and the AC agrees. Reviewers had several suggestions for additional experiments, ablations, environments, and hyperparameter analysis that the authors performed during the rebuttal period. Many of these would strengthen the manuscript and provide additional value to readers (either in the main paper or the appendix). Discussion of take-aways (e.g. the reply to eib8) and the easier-to-digest algorithm overview / diagram may also improve readability. Reviewer enthusiasm for more complex environments is likely a positive signal that CQM in image-based settings could find broader impact.
Decision: Accept (poster)