SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: (Lower is better) Distance from the curriculum goals to the final goals (PointNMaze, PointSpiralMaze, and AntUMaze). In the 'n-way' environments with multiple goals, we provide  $l/2$  distance between the agent and the final goal at the end of the episodes, since calculating the average distance from the curriculum goal to multiple final goals is not possible.
Figure 5: (Higher is better) Success rates of the results. The curves of baselines are not visible in some environments as they overlap each other at zero success rate. Shading indicates a standard deviation across 4 seeds.
results in seven goal-reaching tasks including two visual control tasks, which receive the raw pixel observations from bird's-eye and ego-centric views, respectively. (refer to Appendix C for the detailed configurations of each task.)
We compare our approach with previous curriculum RL methods and previous graph-guided RL methods. We do not provide manually specified goal space in any of the environments; the agent could not map its global X-Y coordinates from the full observation which includes all the state variables for the RL agents (e.g. angle and angular velocity of the joint, position, velocity ...). Also, the results of CQM and the baselines that utilize external reward functions (all the methods except OUTPACE [6]) are obtained by using sparse reward functions. For the baselines that could not be applied in vision-based environments [24, 6], we utilize an extra autoencoder with auxiliary time-contrastive loss [44, 13].
The baselines are summarized below: OUTPACE [6] proposes uncertainty and temporal distance-aware curriculum learning based on the Wasserstein distance and uncertainty classifier. CURROT [21] interpolates the distribution of the curriculum goals and the final goals based on the performance of the RL agent. GoalGAN [10] proposes the goals with appropriate levels of difficulty for the agent Using a Generative Adversarial Network. PLR [18] selectively samples curriculum goals by prioritizing the goals with high TD estimation errors. ALP-GMM [37] selects the goals based on the difference of cumulative episodic reward between the newest and oldest tasks using Gaussian mixture models. VDS [50] proposes the goals that maximize the epistemic uncertainty of the action value function of the policy. DHRL [24] constructs a graph between both levels of HRL, and proposes frontier goals when the random goals are easy to achieve. However, the original DHRL could not generate curriculum goals without the help of the environment. Thus we evaluated a variant of DHRL  $(\mathrm{DHRL}+)$  with a modified frontier goal proposal module and architecture (Appendix D.3), in
addition to the original DHRL. SFL [13] constructs a graph based on successor features and proposes uncertainty-based curriculum goals. (refer to Appendix D for detailed implementations)
First, we visualize the quantitative results to show whether the proposed method successfully and simultaneously addresses the two key challenges: 1) specifying goal space from arbitrary observation space and 2) suggesting a well-calibrated curriculum to achieve the final goal. Figure 3 illustrates the curriculum goals and changes in discrete goal space (graph) of CQM as learning progresses. Each node in the graph consists of the decoded embedding vectors of VQ-VAE, and each edge represents reachability between the decoded embeddings. The graphs of CQM in the figure gradually expand towards unexplored areas as the learning progresses, since the calibrated curriculum goal induces the agent to explore the unexplored area. In the opposite direction as well, the capability of providing proper curriculum goals on arbitrary geometric structures is facilitated by a compact goal space that contains semantic information which enables estimating the uncertainty and temporal distance well. As a result, our method provides tailored curriculum guidance across the environments, while the baselines suffer from the absence of the manually specified goal space.
We also provide the quantitative results in Figures 4 and 5. Figure 4 indicates that the proposed method (CQM) can suggest a tailored sequence of goals that gradually converges to the final goal distributions while instructing the agent to achieve the increasingly difficult goals. Also, as shown in Figure 5, ours consistently outperforms both the prior curriculum method and graph-RL methods. It is noticeable that CQM is the only method that shows robust performance to the variation of the goal dimension, while other methods suffer from serious data inefficiency, especially in the tasks with higher-dimensional goal space (suffering more in Ant (29dims) compared to Point (6dims)).
Curriculum learning and planning in visual control tasks. To validate the performance of the RL agent and the quality of generated curriculum goals in higher dimensional tasks, We conducted two additional vision-based goal-reaching tasks. PointNMazeViz receives only ego-centric view images to reach the goal, while PointSpiralMaze-Viz receives bird's-eye view images. Figure 6 visualizes the curriculum goals in the order of the episodes, and how the agent utilizes the benefit of planning over the discrete goal space in order to achieve the curriculum goals. To achieve an
Figure 6: Left: the distance from the agent to the final goals (Lower is better). Right: visualization of curriculum goals and waypoints of planning over the graph (CQM).
image-based final goal (Goal: 8), the agent generates the sequence of images ( $\{1, 2, 3, \dots, 8\}$ ) as waypoints, and tries to achieve the waypoints sequentially.
Interestingly, despite a significant increase in the observation dimension, CQM does not suffer from significant performance degradation in terms of data efficiency, which indicates that CQM effectively reduces the complexity of goal space by constructing a semantic goal space. We emphasize that the performance of our algorithm does not show significant differences between state-based and image-based environments (Compare PointNMaze in Figures 4 and 6). Another interesting point is that CQM can fully enjoy the advantage of planning over the discretized goal space, even in vision-based control tasks where the agent does not receive information about its global X-Y coordinates explicitly. These results validate that CQM possesses robust performance in terms of the dimensionality of the goal space, and the capability in extracting temporal relations between the discretized landmarks.
Curriculum guidance. First of all, we examine how important curriculum guidance is for an agent to solve goal-conditioned tasks. As shown in Figure 7, when only the final goal is provided without a tailored curriculum (-w/o Curriculum), the RL agent has difficulty achieving the final goal directly.
CQM w/o Planning
CQM w/o Graph
Figure 7: (Lower is better) Ablation study: the distance from the agent to the final goals at the end of the episodes.
Furthermore, we found that providing curriculum guidance greatly affects the goal space specification module and the absence of a curriculum leads to the ill-formed discrete goal space that barely covers only the observations near the initial distribution. We provide these qualitative results in Figures 13, 14 (Appendix E).
Types of the discrete goal sampling method. The proposed method (CQM) can use two approaches to sample the landmark to form the discrete goal space as introduced in Section 4.1. The first approach is to decode the embedding vectors of the codebook  $l_{1:m} = \psi(e_{1:m})$ , and the other approach is to sample an observation batch from the replay buffer and pass it through VQ-VAE to quantize it (-Landmark from Replay Buff.). As shown in Figure 7, there is no significant difference between them in terms of data efficiency. However, in terms of the stability of learning, utilizing the decoded embeddings of VQ-VAE shows better performance in some environments.
Effect of the goal convergence method. To provide a final goal-directed exploration in addition to the naive curriculum toward the frontier areas, CQM includes a goal convergence module that guides the agent to practice the final goal after the agent sufficiently explored the environment (Section 4.3). Based on the KL divergence between the achieved goal distribution and the final goal distribution, CQM calculates the ratio of the mixture between the final goals and the frontier goals (the ratio of providing final goals as learning progresses is presented in Figure 11 in Appendix E). As shown in Figure 7, the absence of the final goal convergence method (-w/o Goal Convergence) results in unstable performance, since the agent repeatedly practices unexplored areas instead of converging towards the final goal even after the explored area "covers" the final goal distribution.
Effect of Graph Construction and Planning. Finally, we examine the effect of constructing graphs and planning on the performance of RL agents. As explained in section 4.2, CQM not only utilizes the decoded embedding vectors from VQ-VAE as a set of discretized observations but also forms a graph by capturing the temporal relations between the discrete observations. First, we evaluated CQM without graph (-w/o Graph), which does not construct a graph and measure the distance between the landmarks through naive temporal distance prediction based on Q values (TemporalDist), rather than the geodesic distance over the graph (TemporalDist $^{\text{G}}$ ). Also, we evaluate CQM without planning (-w/o Planning) since ours can optionally utilize the benefit of planning and reason over long horizons using the graph. As shown in Figure 7, CQM shows better performance than both CQM without a graph and CQM without planning, especially in some long-horizon tasks (AntUMaze and PointSpiralMaze).
To solve the complex control tasks without the need for a manually designed semantic goal space, we propose to solve both issues of specifying the goal space and suggesting the curriculum goals to the agent. By constructing the quantized world model using the decoded embedding vectors of the discretization bottleneck and restoring the relations between these, CQM considers both the uncertainty and temporal distance and has the capability of suggesting calibrated curriculum goals to the agent. The experiments show that the proposed method significantly improves performance on various vision-based goal-reaching tasks as well as state-based tasks, preventing the performance drop in the absence of a manually specified goal space.
Limitations and future works. While CQM shows great potential in addressing the limitations of previous studies, more research could further develop it. One area that could be explored is the use of reward-free curriculum learning methods, since CQM still requires minimal human efforts such as defining a success threshold to train agents. Also, this study only used single-code representations with VQ-VAE which would possess a limited capacity of representations, so expanding CQM to include multiple-code representations with discrete factorial representations could be an interesting future direction.
This work was supported by Korea Research Institute for defense Technology Planning and advancement (KRIT) Grant funded by Defense Acquisition Program Administration(DAPA) (No. KRIT-CT-23-003, Development of AI researchers based on deep reinforcement learning and establishment of virtual combat experiment environment)
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[2] David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035, 2007.
[3] Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, pages 521-531. PMLR, 2021.
[4] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[5] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.
[6] Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. In Proceedings of International Conference on Learning Representations, 2023.
[7] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pages 539-546. IEEE, 2005.
[8] Ishan Durugkar, Steven Hansen, Stephen Spencer, and Volodymyr Mnih. Wasserstein distance maximizing intrinsic control. arXiv preprint arXiv:2110.15331, 2021.
[9] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[10] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
[11] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[47] Hanwei Wu and Markus Flierl. Variational information bottleneck on vector quantized autoencoders. arXiv preprint arXiv:1808.01048, 2018.
[48] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
[49] Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as a graph: Learning latent landmarks for planning. In International Conference on Machine Learning, pages 12611-12620. PMLR, 2021.
[50] Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-7659, 2020.
A Algorithm and Derivation
A.1 How CQM Works?
Figure 8: The overall diagram of CQM
A.1.1 How is data collected?
1. CQM starts with empty replay buffers.
2. After performing a step in the environment, the observed transition is stored in the replay buffer. (line 18 in Algorithm 1.)
A.1.2 How is the goal space learned?
1. Sample batch from replay buffer. (Line 26 in Algorithm 1.)
2. Train VQ-VAE with the batch via Eq. 2. (Line 27 in Algorithm 1.)
A.1.3 How is the graph constructed?
1. Decode each vector embedding in VQ-Dictionary (the decoded embeddings are the landmarks).
2. Using Eq. 3, connect the landmarks with the distance below the cutoff threshold.
A.1.4 How does the agent get the curriculum goal?
1. Calculate the uncertainty of each node (landmarks) in the graph (by Eq. 5).
2. Calculate the distance of the landmarks from the initial area (by Eq. 4).
3. Sample a curriculum goal which is considered temporally distant and uncertain, among the nodes (landmarks) in the graph (by Eq. 6).
4. Based on the  $\alpha$  value from A.1.6, the decision is made whether to provide the agent with a curriculum goal or a final goal. Then provide the selected goal to the agent.
A.1.5 How does the agent utilize the graph?
1. After the agent obtains the curriculum goal, it can start exploring the environment.
2. CQM first finds a sequence of waypoints to achieve the curriculum goal (utilizing Dijkstra's algorithm).
3. The agent is guided to achieve each waypoint, and finally, tries to achieve the curriculum goal.
A.1.6 How does the curriculum goal converge to the final (desired) goal?
1. At the beginning of the learning, we have some samples of the final goals (e.g. the picture taken at the end of the maze.)
2. To get  $p_{g^f}$ , fit a kernel density estimator (KDE) to estimate the distribution of the final goal.
3. To get  $p_{ag}$ , fit a kernel density estimator (KDE) to estimate the distribution of the explored area.
4. Calculate KL divergence, and then, calculate  $\alpha$  in Eq. 7.
5. Utilize  $\alpha$  when we get the curriculum goal (d)
A.2 Algorithm
Algorithm 1 Overview of CQM
1: Input: final goal examples  $g_{f} \in p_{g}^{f}$ , RL replay buffer  $\mathcal{B}$ , VQ-VAE replay buffer  $\mathcal{B}_{\mathrm{VQ}}$ , actor network  $\pi$ , critic network  $Q$ , embeddings of VQ-VAE  $e_{i:m}$ , total training episodes  $N$ , encoder  $\phi$ , decoder  $\psi$ , Environment horizon  $H$ , graph update cycle  $M$   
2: for iterations  $= 1 \cdots N$  do  
3: sample curriculum goal  $g^{c}$  from landmarks  $l_{1:m} = \psi(e_{1:m})$  using Eq. 6  
4: Env.reset()  
5: if random.uniform(low = 0, high = 1) <  $\alpha$  then  
6:  $g \gets g^{f}$   
7: else  
8:  $g \gets g^{c} +$  random noise  
9: end if  
10: get waypoints  $(\phi(s_{0}), w_{1}, \ldots, w_{n}, g) \gets Dijkstra'salgorithm(s_{0}, g)$   
11: for  $t = 0 \cdots H - 1$  do  
12: if achieved current waypoint  $w_{i}$  or tried more than TemporalDist  $(\psi(w_{i-1}) \rightarrow \psi(w_{i}))$  to achieve  $w_{i}$  then  
13: current waypoint index  $i += 1$   
14: end if  
15:  $a_{t} \gets \pi(\cdot|s_{t}, w_{i})$   
16: Env.step  $(a_{t})$   
17: end for  
18:  $B \gets B \cup \{s_{0}, a_{0}, s_{1}\ldots\}$ ,  $B_{\mathrm{VQ}} \gets B_{\mathrm{VQ}} \cup \{s_{0}, s_{1}\ldots\}$   
19: if  $N\% M == 0$  then  
20: update  $\alpha$  using kernel density estimator (KDE) [39] from Scikit-learn [34] with Gaussian kernel following [35]  
21: Update Graph  $G(V, E)$ , where the vertices are the landmarks  $l_{1:m} = \psi(e_{1:m})$ , and the costs of the edges are calculated from Eq. 3 (connect each edge if the distance is below the cutoff threshold)  
22: end if  
23: for  $i = 0, 1, \ldots, P$  do  
24: Sample a minibatch  $b$  from  $B$   
25: Train  $\pi$  and  $Q$  with  $b$   
26: Sample a minibatch  $b_{\mathrm{VQ}}$  from  $B_{\mathrm{VQ}}$   
27: Train VQ-VAE (encoder  $\phi$  and decoder  $\psi$ ) with  $b_{\mathrm{VQ}}$   
28: end for  
29: end for
A.3 Derivation
Derivation of Eq. 3. Let  $Q$  be a state-action value function,  $l_{1:m}$  be landmarks and  $a \in \mathcal{A}$  be an action that an agent executes. If the policy requires  $n$  steps to reach  $l_j$  from  $l_i$ , the state-action value with discount factor  $\gamma$  can be represented as
$$
Q \left(l _ {i}, a, l _ {j}\right) = (- 1) + (- 1) \gamma + (- 1) \gamma^ {2} + \dots + (- 1) \gamma^ {n - 1} = - \frac {1 - \gamma^ {n}}{1 - \gamma}. \tag {8}
$$
Thus, we can recover the temporal distance between  $l_j$  and  $l_i$  ( $= N$  steps) as  $\gamma^n - 1 = (1 - \gamma)Q(l_i, a, l_j)$ , and we finally get
$$
\operatorname {T e m p o r a l D i s t} \left(l _ {i} \rightarrow l _ {j}\right) = \log_ {\gamma} (1 + (1 - \gamma) Q \left(l _ {i}, a, l _ {j}\right)). \tag {9}
$$
B Related Works
Table 1: Summarized conceptual comparisons between CQM and the previous works.
Specify Goal Space	Uncert.-Aware	T-Dist.-Aware	Final Goal-Direct. Curriculum	Curriculum Proposal	Reason over Long Horizon
GoalGAN[10]	X	X	X	X	GAN	X
CURROT[21]	X	X	X	✓	Uniform	X
PLR[18]	X	X	X	X	Buffer	X
VDS[50]	X	✓	X	X	Buffer	X
ALP-GMM[37]	X	X	X	X	GMM	X
HGG[38]	X	X	X	✓	Buffer	X
SkewFit[36]	X	✓	X	X	VAE	X
SFL[13]	X	✓	X	X	Graph	✓
DHRL[24]	X	X	✓	X	Graph	✓
L3P[49]	✓	X	X	X	X	✓
DGRL[17]	✓	X	X	X	X	possible (+HRAC)
Choreographer [27]	✓	X	X	X	X	✓
OUTPACE[6]	X	✓	✓	✓	Buffer	X
CQM (ours)	✓	✓	✓	✓	Graph	✓
Table 1 compares our approach and previous curriculum goal generation methods [10, 21, 18, 50, 37, 38, 36, 6], graph-guided RL methods [13, 49, 24], and representation learning methods using VQ-VAE [17, 27]. The characteristics compared include:
- Specify Semantic Goal Space: whether the method can extract a compact goal space from the high-dimensional goal space.
- Ucert-Aware: whether the curriculum considers uncertainty.
- T-Dist-Aware: whether the curriculum considers temporal distance.
- Final Goal-Directed Curriculum: Whether the method utilizes target curriculum distribution for curriculum goal convergence.
- Curriculum Proposal: The source of the proposed curriculum goals.
- Reason over Long Horizon: Whether the method can reason over long horizons.
- PointNmaze: The agent receives the observation consisting of the angle, angular velocity, XY position, and XY velocity of the agent. The agent is initialized in [0, 0] and the final goal is located at [0, 8]. The agent achieves the final goal when it comes within 0.5 from the final goal. Also, the size of the map is 12 by 12. The maximum timestep per episode is 150.
- PointSpiralMaze: It shares the other aspects with PointNMaze, but the final goal is located in [8, 16] and the size of the map is 12 by 20. The maximum timestep per episode is 250.
- Point3WayMaze: It shares the other aspects with PointNMaze, but the final goal is sampled randomly among [24, -8], [24, 0], and [24, 8]. The map has multiple dead ends and the size of the map is 32 by 24. The maximum timestep per episode is 200.
- AntUMaze: Agent receives the observation consisting of joint angle, joint angular velocity, XYZ position, and XYZ velocity of the agent. The agent is initialized in  $[0, 0]$  and the final goal is located at  $[0, 8]$ . The agent achieves the final goal when it comes within 1.0 from the final goal. Also, the size of the map is 12 by 12. The maximum timestep per episode is 350.
- Ant2WayMaze: It shares the other aspects with AntUMaze, but the final goal is sampled randomly among [4, 4] and [-4, -4]. The map has multiple dead ends and the size of the map is 20 by 12. The maximum timestep per episode is 200.
Figure 9: Top-down view of each environment used for evaluation. 'n-way' environments have multiple goals, and the final goal of an episode is generated randomly among them.
- PointNMaze-Viz: The agent receives the ego-centric camera input. The resolution of the input is  $64 \times 64$ . This environment is modified from PointNMaze, by coloring the walls. The maximum timestep per episode is 200.
- PointSpiralMaze-Viz: The agent receives the Top-down view camera input. The resolution of the input is  $64 \times 64$ . This environment is modified from PointSpiralMaze, by converting the actions of the agent from the polar coordinates to global XY coordinates and increasing the visual size of the agent. The maximum timestep per episode is 250.
D.1 Learning VQ-VAE
Replay buffer for VQ-VAE To train the VQ-VAE for quantizing continuous observations, we employ a separated (smaller size) replay buffer which contains recent trajectories compared to the original replay buffer for the RL agent. By employing the separated replay buffer, we can provide VQ-VAE with more recent observations, resulting in a better reflection of the areas recently explored by the RL agent.
Code resampling for VQ-VAE When we use decoded embedding vectors as landmarks to discretize the goal space for CQM, VQ-VAE could ignore some of the codes due to index collapse. Then, the unused codes form landmarks in physically infeasible locations and occupy the capacity of the codes unnecessarily, leading to potential inefficiencies in the model's performance. Thus, we utilize the code resampling method [20] across all the experiments to address such problems of index collapse [20] in training VQ-VAE models.