SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The code resampling module in CQM first keeps track of inactive codes that have not been used during the previous M rollouts. Subsequently, the inactive codes are re-initialized with the embeddings of the recent observations  $\phi (o)$  with the probability of  $d_q^2 (\phi (o)) / \Sigma_o d_q^2 (\phi (o))$ , where  $d_q^2 (\phi (o))$  indicates the Euclidean distance from the closest code  $(d_{q}(\phi (o)) = \min_{i\in \{1,\ldots ,k\}}||\phi (o) - e_{i}||_{2}^{2})$ . We refer the reader to [27] for detailed explanations of code resampling.
Figure 10: CQM with different RL algorithms.
D.2 Sampling landmarks
We adopt the graph construction module of prior work [49, 24]. Inheriting their implementations, we also employ a landmark sparsification technique which is based on the Greedy Latent(Node) Sparsification algorithm [2, 49]. The detailed algorithm is shown in Algorithm 2
Algorithm 2 Greedy Latent(Node) Sparsification [49, 24]
1: Input: set of states  $\{e_1,e_2,\dots e_k\}$  , sampling number k   
2: Selected  $= []$  DistList  $=$  [inf, inf, ... inf]   
3: for  $i = 1$  to  $k$  do   
4: Farthes  $\leftarrow$  argmax(DistList)   
5: add Farthest to Selected   
6: DistFromFarthest  $\leftarrow$  [TemporalDist(Farthest  $\rightarrow$ $\psi (e_{1:m}))]$    
7: DistList  $=$  ElementwiseMin(DistFromFarthest, DistList)   
8: end for   
9: return Selected
D.3 State-based Goal-Reaching Tasks
When we train CQM in state-based goal-reaching tasks, we utilize two well-established algorithms (TD3 algorithm [11] for Point- environments and SAC [12] for Ant- environments), following the baselines (TD3: [24], SAC: [21]). We note that CQM can be built on general off-policy RL algorithms [26, 12, 11] and changing the RL agents for CQM does not lead to a performance drop as shown in the figure 10.
Also, following [24], we utilize separate Q-networks for graph construction and policy learning. When the agent is not yet competent to achieve some goals, the experience of failure in the replay buffer can lead to an overestimation of the temporal distance between the landmarks. Thus, prior work utilizes separate Q-networks for graph construction and policy learning. This simple technique maintains two different Q-networks which are trained with different ratios of hindsight experience replay (HER) [1] and prevents the usage of failure trajectories to estimate the temporal distances between the landmarks which can spoil the graph construction.
Baselines. Since the original implementations of some algorithms [13] utilize discrete action policies, we replace them with continuous policies for comparison in the continuous control tasks. Also, for the baseline that could not generate curriculum goals without the help of the environment (DHRL) [24], we modify the frontier goal sampling module of the original baseline. Specifically, we modified the frontier goal-shifting module in the DHRL to provide a curriculum goal in every episode, not only when an easy goal is given from the environment. This modification ensures that the agent can get curriculum goals even when the environment does not provide random goals. Additionally, we empirically found that the high-level agent aggravates the performance since it suffers from providing high-dimensional subgoals. Therefore, we utilize a variant DHRL  $(\mathrm{DHRL}+)$  with a modified frontier goal-shifting module and without a high-level. For the other baselines, we follow the official implementations (OUTPACE and CURROT) [21, 6] or the implementations from https://github.com/psclklnk/currot (PLR, VDS, GoalGAN, ALP-GMM).
D.4 Vision-based Goal-Reaching Tasks
For the experiments in the vision based-based goal-reaching tasks, the agent needs to be provided with the curriculum goals corresponding to vision inputs. However, we observed that the decoded images from VQ-VAE sometimes have inferior resolution compared to the original images, which could lead to inferior performances. Thus, during the training process for the vision-based control, we utilized the observations that are mapped to each embedding as landmarks, instead of the decoded embeddings as curriculum goals. This technique does not require additional assumptions or computational costs, as the only overhead is saving the original images of each code. Although the mappings from observations to embedding vectors are many-to-one, we empirically found that there is no performance degradation even if we randomly pick one of the observations per embedding vector as a landmark.
Baselines. Since the official implementations of some baselines are unable to solve the vision-based goal-reaching tasks, We experimented with the additional encoder which can encode the high-dimensional observations into compact latent vectors. Considering the insights from prior representation learning research [23, 43], using a naive autoencoder may not be fair, so we incorporated an auxiliary loss that allows for better representation learning for RL agents. To this end, we utilize an autoencoder with time-contrastive loss [7, 44] as used in SFL [13] which also performed exploration in vision-based goal-reaching tasks. Specifically, we employ the following auxiliary triplet loss to train the encoder for the baselines,
$$
\left| \left| \phi \left(o ^ {a}\right) - \phi \left(o ^ {p}\right) \right| \right| _ {2} ^ {2} + m <   \left| \left| \phi \left(o ^ {a}\right) - \phi \left(o ^ {n}\right) \right| \right| _ {2} ^ {2}, \tag {10}
$$
where  $o^a$ ,  $o^p$ , and  $o^n$  represent anchor, positive pair, and negative pair respectively. Also, we utilize a margin parameter  $m = 2$ , following [13].
D.5 Computational Resources
Our experiments have been performed using an NVIDIA RTX A5000 and AMD Ryzen 2950X, and the entire training process took approximately 0.5-2 days, depending on the tasks.
Table 2: Hyperparameters for CQM
# of initial rollouts	20	HER [1] future step	150
batch size (state)	1024	batch size (IMG)	128
HER ratio critic Q	0.8	HER ratio graph Q	1.0
max graph node	300	graph update cycle M	5
critic hidden dim	256	discount factor γ	0.99
critic hidden depth	3	RL buffer mathcalB size	2500000
actor φ learning rate	0.0001	critic Q learning rate	0.001
interpolation factor (target Q)	0.995	target network update freq	10
actor update freq	2	# of VQ-VAE embeddings	128
VQ-VAE latent dimension	64 (-Viz: 32)	RL optimizer	adam
We provide quantitative results for the ablation study in Figures 13 and 14. As we analyzed in Section 5.2, CQM without each module shows inferior results across the tasks. Without a goal convergence module, the agent often has difficulty in terms of progressing toward the desired final goals. Also, without planning, the agent often has difficulty in achieving long-horizon tasks.
As shown in Section 4.3, our algorithm includes a goal convergence module based on KL divergence between the explored area and the region corresponding to the final goal (Eq. 7) in order to perform final goal-directed exploration. Figure 11 represents the ratio of final goals given to the agent instead of curriculum goals, which is calculated according to the following equation.
$$
\alpha = 1 / \max  \left(\beta + \kappa D _ {\mathrm {K L}} \left(p _ {g f} \| p _ {a g}\right), 1\right) \tag {11}
$$
Table 3: Task specific hyperparameters for CQM
PointSpiral	PointN	PointSpiralViz	PointNViz
cutoff threshold for node connection	7	10	5	5
random noise for curriculum goal	4	2	-	-
VQ-VAE buffer Bvq size	1e5	1e4	1e5	1e4
β for mixture ratio α	-20	-20	-3	-3
κ for mixture ratio α	1	1	2e-3	2e-3
Point3Way	AntU	Ant2Way
cutoff threshold for node connection	10	30	30
random noise for curriculum goal	2	2	2
VQ-VAE buffer Bvq size	5e4	1e4	1e4
β for mixture ratio α	-20	-3	-3
κ for mixture ratio α	1	2e-3	2e-3
As shown in Figure 11, the ratio of providing final goals gradually increases as the learning progresses. This allows the agent to practice the final goal instead of exploring unexplored areas when the agent acquires the capability to pursue the final goals at the end of the curriculum.
Figure 11: The ratio of providing final goals rather than curriculum goals as learning progresses (Eq. 11)
Figure 12: Ablation study: hyperparameter sensitivity analysis (PointNMaze, Lower is better)
Figure 13: Ablation study: visualization of the curriculum goals proposed by the CQM.
Figure 14: Ablation study: changes in the discretized goal space of the CQM(ours) as learning progresses.