SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seungjae Lee, Daesol Cho, Jonghae Park, H. Jin Kim
Automation and Systems Research Institute (ASRI)
Artificial Intelligence Institute of Seoul National University (AIIS)
{ysz0301, dscho1234, bdfire1234, hjinkim}@snu.ac.kr
Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate curriculum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized-variational autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-aware curriculum goals that converge to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations in an uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, in various goal-reaching tasks even with ego-centric visual inputs.
Goal-conditioned Reinforcement Learning (RL) has been successfully applied to a wide range of decision-making problems allowing RL agents to achieve diverse control tasks [42, 1, 30]. However, training the RL agent to achieve desired final goals without any prior domain knowledge is challenging, especially when the desired behaviors can hardly be observed. In those situations, humans typically adopt alternative ways to learn the final goals by gradually mastering intermediate sub-tasks. Inspired by the way humans learn, recent RL studies [29, 10, 6] have solved uninformed exploration tasks by suggesting which goals the agent needs to practice. In this sense of generating curriculum goals, previous approaches proposed various ideas to involve providing intermediate-level tasks [10, 38], quantifying the uncertainty of observations [4, 31, 33, 25, 18], or proposing contextual distance to gradually move away from the initial distribution [15, 6].
However, previous curriculum RL studies are mostly not scalable. Namely, they suffer from serious data inefficiency when they generate curriculum goals in high dimensions. Because of this limitation, they usually rely on the assumption that manually specified goal spaces (e.g., global X-Y coordinates) and clear mappings from high-dimensional observations to the low-dimensional goal spaces are available. Such an assumption requires prior knowledge about observations and the tasks, which remains a crucial unsolved issue that restricts the applicability of previous studies.
In order to design a general curriculum solution without the need for prior knowledge about the observations, defining its own goal space for the curriculum could be an effective scheme. To do so,
Figure 1: CQM simultaneously tackles the interrelated problems of specifying the goal space and suggesting which goal the agent needs to practice. CQM trains a VQ-VAE to form a discretized goal space and constructs a graph over it, capturing the relations between the discretized observations (landmarks). Concurrently, CQM suggests the agent which goal to practice based on uncertainty and temporal distance.
the two following operations need to be executed concurrently. (1) composing the semantic goal space which contains vital information for the curriculum process from the arbitrary observation space, and (2) suggesting to the agent which goal to practice over the goal space. Let us consider an agent that tries to explore an uninformed environment with final goal images only. To succeed, the agent needs to specify the semantic goal space from the high-dimensional observation space, and suggest the curriculum goals (e.g., the frontier of the explored area) over the composed goal space to search the uninformed environment. However, most previous studies focused solely on one of these, specifying the low-dimensional goal space without considering how to provide intermediate levels of goals [17, 27], or just suggesting curriculum goals in manually specified semantic goal spaces [10, 21, 6].
The challenge of simultaneously managing (1) specifying the goal space and (2) suggesting curriculum goals is that they are intimately connected to each other. If the agent constructs an ill-formed goal space from the observation space, it would be difficult to propose curriculum goals over it. Conversely, if the method fails to suggest goals to enable the agents to explore the unseen area, it would also be difficult to automatically learn the goal space that covers the uninformed environment based on the accumulated observations. Therefore, it is essential to develop an algorithm that addresses both defining goal space and providing curriculum goals concurrently.
In this paper, we propose a novel curriculum reinforcement learning (RL) method which can provide a general solution for a final goal-directed curriculum without the need for prior knowledge about the environments, observations, and goal spaces. First, our method defines its own semantic goal space by quantizing the encoded observations space through a discretization bottleneck and restoring the temporal relations between discrete goals via a graph. Second, to suggest calibrated guidance towards unexplored areas and the final goals, ours proposes uncertainty and temporal distance-aware curriculum goals that converge to the final goal examples.
The key contributions of our work (CQM: Curriculum RL with Quantized World Model) are:
- CQM solves general exploration tasks with the desired examples only, by simultaneously addressing the specification of a goal space and suggestion of curriculum goals (Figure 3).
- CQM is the first curriculum RL approach that can propose calibrated curriculums toward final goals from high-dimensional observations, to the best of our knowledge.
- CQM is the only curriculum RL method that demonstrates reliable performance despite an increase in the problem dimension, among the 10 methods that we experimented with. (Even state-based  $\rightarrow$  vision-based)
- Ours significantly outperforms the state-of-the-art curriculum RL methods on various goal reaching tasks in the absence of a manually specified goal space.
Curriculum Goal Generation. Although various prior studies [41, 19, 48, 8, 45, 22] have been proposed to solve exploration problems, enabling efficient searching in uninformed environments still
remains a challenge. An effective way to succeed in such tasks with hardly observed final goals is identifying uncertain areas and instructing an agent to achieve the goals in these areas. To identify the uncertain areas and provide the goals sampled from them, previous studies employ uncertainty-based curriculum guidance by the state visitation counts [4, 31], absolute reward difference [37], and prediction model [32, 5]. Other approaches propose to utilize disagreements of ensembles[33, 50, 28] or sample the tasks with high TD errors [18] to generate goals in uncertain areas. An alternative way for solving the exploration tasks is to execute a final goal-directed exploration to propose tailored guidance. To this end, some studies perform successful example-based approaches [25, 6] or propose to minimize the distance between the final goals and curriculum goals [38, 21], measuring it by the Euclidean distance metric. Some studies also employ contextual distance-based metrics to perform final goal-directed exploration away from the initial distribution [15, 6].
However, these methods usually assume that agents have prior knowledge about the observations and unrestricted access to manually specified semantic goal space (e.g. global X-Y coordinates) because they are not scalable to handle the high-dimensional goal spaces. For example, the meta-learning classifier-based uncertainty metrics [25, 6] suffer from distinguishing uncertain areas as the dimension of the goal space increases. Also, some of the methods rely on Euclidean distance metric [38, 21] over the goal space. Moreover, generating curriculum goals [10], employing various prediction models [32, 5, 33, 50], fitting Gaussian mixture models [37], and utilizing disagreements of ensembles-based methods [33, 50] also face difficulty in solving increasingly complex
Figure 2: When the curriculum methods are not scalable to handle the high-dimensional goal; the performance drop in the absence of manually specified goal space.
problems in high-dimensional goal spaces. Although there have been attempts to propose a curriculum in high-dimensional observations [36, 13] or include an encoder in their model-based agent [28, 14], unfortunately, these approaches do not incorporate a convergence mechanism to the final goals, which are crucial for efficient curriculum progresses.
Our method incorporates the benefits of the aforementioned methods without manually specified goal spaces: exploring uncertain areas and moving away from the initial distribution while converging to the desired outcome. Although there is a study that has also incorporated these three aspects [6], it retains its performance only in manually specified goal spaces, as other curriculum methods (Figure 2). (We included conceptual comparisons between CQM and more related works in Table 1 in Appendix B.)
Discretizing Goal Space for RL. Vector quantized-variational autoencoder (VQ-VAE) is an autoencoder that learns a discretized representation using a learnable codebook. The use of this discretization technique for learning discrete representations in RL is a recent research topic [17, 27] and has shown an improved sample efficiency. Islam et al. [17] proposes to apply VQ-VAE as a discretization bottleneck in a goal-conditioned RL framework and demonstrates the efficiency of representing the continuous observation spaces into discretized goal space. Also, Mazzaglia et al. [27] utilizes VQ-VAE to discover skills in model-based RL by maximizing the mutual information between skills and trajectories of model states.
Unfortunately, the aforementioned methods require a pre-collected dataset or extra exploration policy, which are not necessary in CQM. Training VQ-VAE with a pre-collected dataset implies that the agent has access to the full information about the task or that it already possesses an agent capable of performing the task well. Although it is possible to obtain a pre-collected dataset through a random rollout policy, this is only in the case where exploring the environments is easy enough to succeed with only random actions.
We consider a Markov Decision Process (MDP) which can be represented as a tuple  $(\mathcal{O},\mathcal{A},\mathcal{T},\mathcal{R},\rho_0,\gamma)$ , where  $\mathcal{O}$  is an observation space,  $\mathcal{A}$  is an action space,  $\mathcal{T}(o_{t + 1}|o_t,a_t)$  is a transition function,  $\rho_0$  is an initial distribution, and  $\gamma$  is a discount factor. Note that the MDP above
does not contain a goal space, since we do not assume that the manually specified goal space is provided. Instead, we consider a discrete low-dimensional discrete goal space  $\mathcal{G}$ , which is defined by the agent automatically. Also, we assume that the final goal examples  $o^f$  are provided by the environment, and we denote the projection of these examples into the goal space  $\mathcal{G}$  as  $g^f$ . We represent curriculum goal as  $g^c$ , which is sampled from the goal space  $\mathcal{G}$ , and the reward function in the tuple can be represented as  $\mathcal{R}:\mathcal{O}\times \mathcal{A}\times \mathcal{G}\to \mathbb{R}$ . Furthermore, we denote actor network as  $\pi :\mathcal{O}\times \mathcal{G}\rightarrow \mathcal{A}$ , and critic network as  $Q:\mathcal{O}\times \mathcal{A}\times \mathcal{G}\rightarrow \mathbb{R}$ . (Thus,  $Q(o,a,g)$  indicates the goal-conditioned state-action value where goal  $g$ , action  $a$ , and observation  $o$  throughout this paper.)
In order to provide a general solution for efficient curriculum learning, our method defines its own goal space and suggests to the agent which goal to practice over the goal space simultaneously. To compose a semantic goal space which reduces the complexity of observation space while preserving vital information for the curriculum process, we first quantize the continuous observation space using a discretization bottleneck (section 4.1) and restore temporal relations in discretized world model via graph (section 4.2). Over the automatically specified semantic goal space, we generate a curriculum goal and guide the agent toward achieving it (section 4.3).
4.1 Specifying Goal Space via VQ-VAE
In order to define a discrete low-dimensional goal space which allows a scalable curriculum with high-dimensional observations, we utilize VQ-VAE as a discretization bottleneck [46, 40, 47] as recently been proposed [17, 27]. VQ-VAE utilizes a codebook composed of  $k$  trainable embedding vectors (codes)  $e_i \in R^D$ ,  $i \in 1, 2, \ldots, k$ , combined with nearest neighbor search to learn discrete representations. The quantization process of an observation  $o_t$  starts with passing  $o_t$  through the encoder  $\phi$ . The resulting encoded vector  $z_e = \phi(o_t)$  is then mapped to an embedding vector in the codebook by the nearest neighbor look-up as
$$
z _ {q} = e _ {c}, \quad \text {w h e r e} c = \operatorname {a r g m i n} _ {j} \| z _ {e} - e _ {j} \| _ {2}. \tag {1}
$$
The discretized vector  $z_{q}$  is then reconstructed into  $\hat{\sigma}_{t} = \psi (z_{q})$  by passing through the decoder  $\psi$ . We closely follow [17] and train quantizer, encoder, and decoder using a vector quantization loss with a simple reconstruction loss. The first term in Eq. 2 represents the reconstruction loss, while the second term represents the VQ objective that moves the embedding vector  $e$  towards the encoder's output  $z_{e}$ . We update the embedding vectors  $e$  using a moving average instead of a direct gradient update [17, 27]. The last term is the commitment loss, and we use the same  $\lambda_{\mathrm{commit}} (= 0.25)$  across all experiments.
$$
L _ {\mathrm {V Q}} = \left\| \psi \left(\phi \left(o _ {t}\right)\right) - o _ {t} \right\| _ {2} ^ {2} + \left\| \operatorname {S G} \left[ z _ {e} \right] - e \right\| _ {2} ^ {2} + \lambda_ {\text {c o m m i t}} \| z _ {e} - \operatorname {S G} [ e ] \| _ {2} ^ {2}, \quad (\operatorname {S G}: \text {s t o p g r a d i e n t}) \tag {2}
$$
By utilizing VQ-VAE, the RL agent can specify a quantized goal space that consists of discrete landmarks  $L = \{l_1, l_2, \dots, l_m\}$  in two ways. The first approach is to obtain each landmark by decoding each code as  $l_j = \psi(e_j)$ . Alternatively, one can obtain the landmarks by passing (encoding, quantizing to the closest embedding vector, and decoding) the continuous observations sampled from the replay buffer through the VQ-VAE. We utilize the first approach as the default, and provide the ablation study to examine the effectiveness of the second approach.
It should be noted that this set of landmarks  $L = \{l_1, l_2, \dots, l_m\}$  only represents discretized observations and does not involve relations among the observations. We describe our approach that can better restore the temporal relations between landmarks in the next section.
4.2 Graph Construction over Quantized Goal Space
In this section, we present the graph construction technique over the quantized goal space to allow the agent to capture the temporal information between the landmarks of the quantized world model. We consider a graph  $\mathbf{G} = (\mathbf{V},\mathbf{E})$  over the goal space where the vertices  $\mathbf{V}$  represent the landmarks
$L = \{l_{1}, l_{2}, \dots, l_{m}\}$  obtained from decoding discrete codes of VQ-VAE, and the edges  $\mathbf{E}$  represent the temporal distance. We utilize Q-value to reconstruct the edge costs, following the method proposed in [49, 24]. If an agent receives a reward of 0 when reaching a goal and -1 otherwise, the timesteps required to travel between landmarks can be estimated using Q-value as (derivation: Appendix A)
$$
\operatorname {T e m p o r a l D i s t} \left(l _ {i} \rightarrow l _ {j}\right) = \log_ {\gamma} (1 + (1 - \gamma) Q \left(l _ {i}, a, l _ {j}\right)). \tag {3}
$$
Using Eq. 3, we connect vertices with the distance below the cutoff threshold, and the resulting graph restores the temporal relations between the landmarks over a discretized goal space based on the temporal distance. In this way, the agent can calculate geodesic distances between landmarks,
$$
\operatorname {T e m p o r a l D i s t} ^ {\mathbf {G}} \left(l _ {0} \rightarrow l _ {f}\right) = \Sigma_ {\left(l _ {i} \rightarrow l _ {j}\right) \in \text {s h o r t e s t p a t h} \left(l _ {0} \rightarrow l _ {f}\right)} \operatorname {T e m p o r a l D i s t} \left(l _ {i} \rightarrow l _ {j}\right), \tag {4}
$$
which enables better prediction of temporal distances in environments with arbitrary geometric structures. Also, to incorporate extended supports of the explored area into the graph by creating landmarks in newly explored areas, we periodically reconstructed the graph following [24].
4.3 Uncertainty and Temporal Distance-Aware Curriculum Goal Generation
In the previous sections, we proposed a method for specifying a discretized goal space with semantic information. It is important to provide curriculum goals located in the frontier part of the explored area to expand the graph effectively toward the final goal in an uninformed environment. To achieve this objective, we propose an uncertainty and temporal distance-aware curriculum goal generation method.
To obtain a curriculum goal from graph  $\mathbf{G} = (\mathbf{V},\mathbf{E})$  over the specified goal space, our method samples the landmarks that are considered uncertain and temporally distant from the initial distribution  $\rho_0$ . Thanks to the quantized world model, quantifying uncertainty in a countable goal space is straightforward and computationally light. We quantify the count-based uncertainty of each landmark as  $\eta_{\mathrm{ucert}}(l_i) = 1 / (\mu (l_i) + \epsilon)$ , based on the empirical distribution  $\mu (l_i)$  derived from the recent observations as
$$
\mu \left(l _ {i}\right) = \frac {N \left(l _ {i}\right)}{\sum_ {i = 1} ^ {k} N \left(l _ {i}\right)}, \tag {5}
$$
where  $N(l_{i})$  indicates the number of occurrences of landmark  $l_{i}$  in recent episodes and is periodically re-initialized when the graph is reconstructed with a new set of landmarks.
Finally, we deliver the sampled landmarks as curriculum goals to the agent, considering both temporal distance and uncertainty aspects:
$$
\operatorname {a r g m a x} _ {l _ {i} \in L ^ {\mathrm {t o p} - \mathrm {k}}} \left[ \eta_ {\text {u c e r t}} \left(l _ {i}\right) \cdot u _ {i} \right] \tag {6}
$$
where  $u_{i}$  is a uniform random variable between 0 and 1 used to perform weighted sampling, and  $L^{\mathrm{top - k}}$  represents a subset of  $L$  that includes the top-k elements with the largest TemporalDist $^{\mathbf{G}}$  (Eq. 4) values from initial state distribution. Our method, based on the uncertainty and temporal distance-aware objective (Eq. 6), is capable of providing calibrated curriculum guidance to the agent even in environments with arbitrary geometric structures, without requiring prior knowledge of the environment or a manually specified goal space. Furthermore, the curriculum guidance makes composing the goal space easier, illuminating the unexplored areas and vice versa.
Convergence to the final goal. The curriculum objective in Eq.6 provides a calibrated curriculum towards unexplored areas. In addition to this frontier-directed method, providing final goal-directed guidance can further improve the efficiency of exploration especially when the agent sufficiently explored the environment, i.e., the supports of the final goal and explored area start to overlap [35, 6]. In order to acquire the capability to propose a final goal-directed curriculum, we gradually shift the direction of exploration from the frontier of the explored area to the final goal distribution. To do so,
Figure 3: Left: changes in the discretized goal space of the CQM(ours) as learning progresses. Right: visualization of the curriculum goals proposed by the CQM and baseline algorithms.
we determine whether to provide the curriculum goal  $g^{c} \in \mathcal{G}$  that is sampled via Eq. 6 or the final goal  $g^{f} = \phi(o^{f}) \in \mathcal{G}$  which the environment originally provided  $(\psi(g^{c}) \in \mathcal{O}, o^{f} \in \mathcal{O})$ .
We utilize a mixture distribution of curriculum goals, following the approach proposed in [35],
$$
p _ {g ^ {\mathrm {c} ^ {\prime}}} = \alpha p _ {g ^ {f}} + (1 - \alpha) p _ {g ^ {\mathrm {c}}}, \tag {7}
$$
where  $p_{g^f}$  is the distribution of the final goal, and  $p_{g^c}$  is the distribution of curriculum goals. The mixture ratio  $\alpha$  measures whether the achieved goal distribution  $p_{ag}$  "covers" the final goal distribution using KL divergence as  $\alpha = 1 / \max \left( \beta + \kappa D_{\mathrm{KL}}(p_{g^f} || p_{ag}), 1 \right)$ . When the support of achieved goal distribution  $p_{ag}$  (= visited state distribution) covers that of the final goal distribution  $p_{g^f}$ ,  $\alpha$  produces a value close to 1, and a value close to 0 when the supports of both distributions are not connected.
By combining the curriculum goal objective (Eq. 6) with the mixture strategy (Eq. 7), our approach generates instructive curriculum goals towards unexplored areas and provides the curriculum goals  $g^{c'}$  to the agent that "cover" the final goal distribution at the appropriate time when the agent is capable of achieving the final goal.
Planning over the graph As presented above, CQM constructs a graph to restore the temporal relations between landmarks (Section 4.2) and utilizes it to calculate geodesic distances (Eq. 4). In addition to these benefits, we highlight that the graph can also provide the strength of planning, which allows the agent to reason over long horizons effectively [9, 16, 13, 49, 3, 24].
To generate a sequence of waypoints for achieving each goal, we perform shortest path planning (Dijkstra's Algorithm), following the details proposed in the previous graph-guided RL method [24]. Consider a task of reaching a curriculum goal  $g^{c} \in \mathcal{G}$  from the current observation  $o_0 \in \mathcal{O}$ . CQM first adds the encoded observation  $\phi(o_0)$  to the existing graph structure. Then, it finds the shortest path between the curriculum goal and current observation to return a sequence of waypoints  $(\phi(o_0), w_1, \dots, w_n, g^{c})$  where  $n$  indicates the number of waypoints in the shortest path. Finally, the agent is guided to achieve each decoded waypoint  $\psi(w_i)$  during TemporalDist  $(\psi(w_{i-1}) \to \psi(w_i))$  (Eq. 3) timesteps, rather than achieving the curriculum goal directly. In other words, the RL agent produces goal-conditioned action  $\pi(\cdot | o_t, w_i)$ , where  $o_t$  and  $w_i$  is observation and the waypoint (acting as a goal) respectively. After reaching the final waypoint  $\psi(w_n)$ , the agent receives the original curriculum goal,  $g^{c}$ . The only change when the agent attempts to achieve the final goal  $g^{f}$  is that  $g^{f}$  comes at the end of the sequences,  $(\phi(o_0), w_1, \dots, w_n, g^{f})$ , rather than  $g^{c}$ . In this way, the proposed approach not only provides a tailored curriculum for achieving the final goal but also allows the agent to access more elaborate instructions ( waypoints) for practicing each curriculum goal.
The main goal of the experiments is to demonstrate the capability of the proposed method (CQM) to suggest a well-calibrated curriculum and lead to more sample-efficient learning, composing the goal space from the arbitrary observation space. To this end, we provide both qualitative and quantitative