SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\hat {p} ^ {c} (s): | \hat {p} ^ {c} (s) | = K} \sum_ {s _ {i} \in \hat {p} ^ {c} (s), g _ {i} ^ {+} \in \hat {p} ^ {+} (g)} w \left(s _ {i}, g _ {i} ^ {+}\right) \tag {6}
$$
$$
w \left(s _ {i}, g _ {i} ^ {+}\right) := \mathcal {C E} \left(p _ {\text {p s e u d o}} \left(y = 1 \mid s _ {i}; g _ {i} ^ {+}\right); y = p _ {\text {p s e u d o}} \left(y = 1 \mid g _ {i} ^ {+}; g _ {i} ^ {+}\right)\right) \tag {7}
$$
The intuition behind this objective is similar to Eq (4), but it is different in terms of considering conditional quantification, which enables addressing arbitrarily distributed desired outcome examples. Then, we can create a bipartite graph  $\mathbf{G}$  with edge costs  $w$  by considering the sets of nodes  $\mathbf{V}_a$  and  $\mathbf{V}_b$ , which represent achieved states in replay buffer  $\mathcal{B}$  and  $\hat{p}^{+}(g)$ , respectively. We define the bipartite graph  $\mathbf{G}(\{\mathbf{V}_a,\mathbf{V}_b\},\mathbf{E})$  with edge weights  $\mathbf{E}(\cdot ,\cdot) = -w(\cdot ,\cdot)$ . To solve this bipartite matching problem, we employ the Minimum Cost Maximum Flow algorithm [1, 37] to find  $K$  edges with the minimum cost  $w$ . The entire curriculum RL process is shown in Algorithm 2 in Appendix E.
4.4 Conditional classifier-based intrinsic reward
As we have trained conditional classifiers and defined the pseudo probability, we can additionally use this value as a shaped intrinsic reward for enabling the agent to solve the uninformed search problem. As the pseudo probability outputs 0 for the state in the replay buffer  $\mathcal{B}$ , 1 for the desired outcome state, and a value between 0 and 1 for the state where the classifiers disagree (meaning unexplored region), we can define the goal-conditioned intrinsic reward as  $r = p_{\mathrm{pseudo}}(y = 1|s;g)$ . We use this intrinsic reward in all of our experiments. The ablation study for this reward is detailed in Section 5.2.
Table 1: Comparison of our work with the prior curriculum RL methods in conceptual aspects.
Conditional quantification	Target dist. of curriculum	Arbitrary desired outcome dist.	Geometry-agnostic	Without external reward
HGG	X	p+(g)	✓	X	X
CURROT	X	uniform or p+(g)	✓	X	X
PLR	X	X	X	✓	X
VDS	X	X	X	✓	X
ALP-GMM	X	X	X	✓	X
OUTPACE	X	p+(g)	X	✓	✓
Ours	✓	p+(g)	✓	✓	✓
(b) OUTPACE
(c) HGG
Figure 3: Curriculum goal visualization of the proposed method and baselines. First row: Ant Locomotion, Second row: Spiral-Maze.
We conduct experiments on 6 environments that have multi-modal desired outcome distribution to validate our proposed method. We use various maze environments (Point Complex-Maze, Medium-Maze, Spiral-Maze) to validate our curriculum proposal capability, which is not limited to specific geometries. Additionally, we evaluate our method on more complex dynamics or other domains such as the Ant-Locomotion and Sawyer-Peg Push, Pick&Place with obstacle environments to demonstrate its effectiveness in domains beyond the navigation. (Refer to Appendix D for more details.)
We compare our method with several previous curriculum generation approaches, each of which possesses the following characteristics. OUTPACE [5] prioritizes goals that are considered uncertain and temporally distant from the initial state distribution through meta-learning-based uncertainty quantification and Wasserstein-distance-based temporal distance approximation. HGG [37] aims to reduce the distance between the desired outcome state and curriculum distributions, using a value function bias and the Euclidean distance metric. CURROT [18] interpolates between the desired outcome state and curriculum distribution while considering the agent's current capability using the Wasserstein distance. VDS [49] proposes epistemic uncertainty-based goals by utilizing the value function ensembles. ALP-GMM [36] models absolute learning progress score by a GMM. PLR [16] prioritizes increasingly challenging tasks by ranking task levels based on TD errors. We summarize the conceptual comparison between our method and baselines in Table 1.
5.1 Experimental results
First, to validate the quality of curriculum goal interpolation from the initial state distribution to the desired outcome distribution, we qualitatively and quantitatively evaluate the curriculum progress achieved during the training by optimizing Eq (6). For qualitative evaluation, we visualize the curriculum goals obtained by the proposed method and other baselines (Figure 3). The proposed method shows calibrated guidance toward the desired outcome states, even with the multi-modal distribution. But, HGG shows an inappropriate curriculum proposal due to the Euclidean distance metric, and OUTPACE shows curriculum goals collapsed only toward a single direction because it utilizes the Wasserstein distance for biasing curriculum goals into the temporally distant region. Once the agent explores a temporally far region in a specific direction earlier, the curriculum proposal is collapsed toward this area. In contrast, our method does not have such dependence, which enables
(a) Complex-Maze
(b) Medium-Maze
(c) Spiral-Maze
(d) Ant Locomotion
(f) Sawyer Pick&Place
(e) Sawyer Push
Figure 4: The mean distance between the curriculum goals and final goals (Lower is better). The shaded area represents a standard deviation across 5 seeds. The increases of our method at initial steps in some environments are attributed to the geometry of the environments.
(d) Ant Locomotion
Figure 5: Evaluation success rates, using the same seeds and legends as shown in Figure 4. Note that some baselines are not visible as they coincide with a success rate of zero.
(e) Sawyer Push
our method to always propose curriculum goals in any uncertain direction quantified by disagreement between the diversified classifiers.
We also plot the average distance from the proposed curriculum goals to the desired outcome states for quantitative evaluation (Figure 4). As we evaluated with multi-modal desired outcome distribution, the distance cannot be measured by naively averaging the distance between the randomly chosen desired outcome states and curriculum goals. Thus, we measure the distance by bipartite matching with the  $l_{2}$  distance metric, which computes the distance between the desired outcome states and their assigned curriculum goals. As shown in Figure 4, only the proposed method consistently shows superior interpolation results from the initial state distribution to desired outcome distribution, while other baselines show some progress only in a simple geometric structure due to the Euclidean distance metric, or get stuck in some local optimum area due to the lack of or insufficient unexplored region quantification. We also plot the performance of the outcome-directed RL in Figure 5. As expected by the average distance measurement in Figure 4, the proposed method is the only one that consistently and quickly achieves the desired outcome states through the guidance of calibrated curriculum goals, indicating the advantage of the proposed diversified conditional classifiers.
(a) Auxiliary loss weight  $\lambda$
(b) Number of prediction heads
(c) Intrinsic reward & Curriculum
Figure 6: Ablation study in terms of the episode success rates. First row: Spiral-Maze. Second row: Sawyer Push. The shaded area represents the standard deviation across 5 seeds.
5.2 Ablation study
Number of prediction heads & Auxiliary loss weight  $\lambda$ . To investigate the sensitivity to the hyperparameters of our method, we experimented with the different values of the auxiliary loss' weight  $\lambda$  and with the different number of prediction heads of the conditional classifiers. As shown in Figure 6a, 6b, the results are slightly dependent on the environment's characteristics such as the presence of object interaction or complex geometry. However, the overall performance trend is not significantly affected by the number of prediction heads and  $\lambda$ , except in the extreme case, which supports the superiority of our proposed method. More qualitative/quantitative analyses and other ablation studies are included in Appendix F.
Reward type & Curriculum. We evaluate the effectiveness of the proposed intrinsic reward and curriculum proposal by conducting two ablation experiments. First, we replace the intrinsic reward with a sparse reward (Ours w/o IntrinsicReward), which is typically used in goal-conditioned RL problems. Second, we conduct experiments without the curriculum proposal while utilizing the intrinsic reward (Ours w/o Curriculum). As shown in Figure 6c, there is performance degradation without using the proposed intrinsic reward in an environment with complex dynamics since an informative reward signal is crucial. In addition, the absence of a curriculum proposal leads to performance degradation in an environment with complex geometry as it requires carefully crafted guidance for exploration. These results highlight the importance of both proposed components, i.e. intrinsic reward and curriculum proposal.
We propose D2C that 1) performs a classifier diversification process to distinguish the unexplored region from the explored area and desired outcome example and 2) conquers the unexplored region by proposing the curriculum goal and the shaped intrinsic reward. It enables the agent to automatically progress toward the desired outcome states without prior knowledge of the environment. We demonstrate that our method outperforms the previous methods in terms of sample efficiency and geometry-agnostic, desired-outcome-distribution-agnostic curriculum progress, both quantitatively and qualitatively.
Limitation & Broader impacts. Despite the promising results, there is a limitation in the scalability of the proposed method since we use small noise to augment the conditioned goal for numerical stability, making it difficult to scale to high-dimensional inputs such as images. Thus, addressing this point would be an interesting future research direction to develop a more generally applicable method. Also, our work is subject to the potential negative societal impacts of RL, but we do not expect to encounter any additional negative impacts specific to this work.
This work was supported by Korea Research Institute for defense Technology Planning and advancement (KRIT) Grant funded by Defense Acquisition Program Administration(DAPA) (No. KRIT-CT-23-003, Development of AI researchers based on deep reinforcement learning and establishment of virtual combat experiment environment).
[41] Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854, 2019.
[42] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
[43] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407, 2017.
[44] Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. Domain adaptation in computer vision applications, pages 153-171, 2017.
[45] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11359, 2018.
[46] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[47] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Kerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
[48] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100. PMLR, 2020.
[49] Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-7659, 2020.
[50] Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprint arXiv:2103.08107, 2021.
A Training & Experiments details
A.1 Training details
Baselines. The baseline curriculum RL algorithms are trained as follows,
- OUTPACE [5]: We follow the default setting in the original implementation from https: //github.com/jayLEE0301/outpace_official.
- HGG [37]: We follow the default setting in the original implementation from https: //github.com/Stilwell-Git/Hindsight-Goal-Generation.
- CURROT [18]: We follow the default setting in the original implementation from https: //github.com/psclklnk/currot.
- PLR [16], VDS [49], ALP-GMM [36]: We follow the default setting in implementation from https://github.com/psclklnk/currot.
D2C and all the baselines are trained by SAC [13] with the sparse reward except for the OUTPACE which uses an intrinsic reward based on Wasserstein distance with a time-step metric.
Training details. We used NVIDIA A5000 GPU and AMD Ryzen Threadripper 3960X for training, and each experiment took about  $1\sim 2$  days for training. We used small noise from a uniform distribution with an environment-specific noise scale (Table 5) for augmenting the conditioned goal in Eq (5). Also, we used the mapping  $\phi (\cdot)$  that abstracts the state space into the goal space when we use the diversified conditional classifiers (i.e.  $f_{i}(\phi (s);g)$ . For example,  $\phi (\cdot)$  abstracts the proprioceptive states (e.g.  $xy$  position of the agent) in navigation tasks, and abstracts the object-centric states (e.g.  $xyz$  position of the object) in robotic manipulation tasks.
Table 2: Hyperparameters for D2C
critic hidden dim	512	discount factor γ	0.99
critic hidden depth	3	batch size	512
critic target τ	0.01	init temperature αinit of SAC	0.3
critic target update frequency	2	replay buffer β size	3e6
actor hidden dim	512	learning rate for fi	1e-3
actor hidden depth	3	learning rate for Critic & Actor	1e-4
actor update frequency	2	optimizer	adam
Table 3: Default env-specific hyperparameters for D2C
Env name	# of heads	λ	ε	fi update freq (step)	fi # of iteration per update	max episode horizon
Complex-Maze	2	1	0.5	2000	16	100
Medium-Maze	2	1	0.5	2000	16	100
Spiral-Maze	2	1	0.5	2000	16	100
Ant Locomotion	2	2	1.0	4500	16	300
Sawyer-Peg-Push	2	1	0.025	3000	16	200
Sawyer-Peg-Pick&Place	2	1	0.025	3000	16	200
A.2 Environment details
- Complex-Maze: The observation consists of the  $xy$  position, angle, velocity, and angular velocity of the 'point'. The action space consists of the velocity and angular velocity of the 'point'. The initial state of the agent is  $[0, 0]$  and the desired outcome states are obtained from the default goal points [8, 16],  $[-8, -16]$ ,  $[16, -8]$ ,  $[-16, 8]$ . The size of the map is  $36 \times 36$ .
- Medium-Maze: It is the same as the Complex-Maze environment except that the desired outcome states are obtained from the default goal points [16, 16],  $[-16, -16]$ ,  $[16, -16]$ ,  $[-16, 16]$ .
(f) Sawyer Pick&Place
Figure 7: Environments used for evaluation: yellow stars indicate the desired outcome examples. (a)-(c) the agent should navigate various maze environments with multi-modal desired outcome distribution. (d) the ant locomotion environment with multi-modal desired outcome distribution. (e) the robot has to push or pick & place a peg to the multi-modal desired locations while avoiding an obstacle at the center of the table.
- Spiral-Maze: The observation space and actions space and initial state of the agent are the same as in the Complex-Maze environment. The desired outcome states are obtained from the default goal points [12, 16],  $[-12, -16]$ . The size of the map is  $28 \times 36$ .
- Ant Locomotion: The observation consists of the  $xyz$  position,  $xyz$  velocity, joint angle, and joint angular velocity of the 'ant'. The action space consists of the torque applied on the rotor of the 'ant'. The initial state of the agent is [0, 0] and the desired outcome states are obtained from the default goal points [4, 8], [-4, -8]. The size of the map is  $12 \times 20$ .
- Sawyer-Peg-Push: The observation consists of the  $xyz$  position of the end-effector, the object, and the gripper's state. The action space consists of the  $xyz$  position of the end-effector and gripper open/close control. The initial state of the object is [0.4, 0.8, 0.02] and the desired outcome states are obtained from the default goal points [-0.3, 0.4, 0.02], [-0.3, 0.8, 0.02], [0.4, 0.4, 0.02]. The wall is located at the center of the table. Thus, the robot arm should detour the wall to reach the desired goal states. We referred to the metaworld [48] and EARL [40] environments.
- Sawyer-Peg-Pick&Place: It is the same as the Sawyer-Peg-Push environment except that the desired outcome states are obtained from the default goal points  $[-0.3, 0.4, 0.2]$ ,  $[-0.3, 0.8, 0.2]$ ,  $[0.4, 0.4, 0.2]$ , and the wall is located at the center of the table, fully blocking a path for pushing. Thus, the robot arm should pick and move the object over the wall to reach the desired goal states.
Algorithm 1 Overview of D2C algorithm
1: Input: desired outcome examples  $\hat{p}^{+}(g)$ , total training episodes  $N$ , Env, environment horizon  $H$ , actor  $\pi$ , critic  $Q$ , replay buffer  $\mathcal{B}$   
2: for iteration=1,2,...,N do  
3:  $\hat{p}^c \gets$  sample K curriculum goals that minimize Eq (6). We refer to HGG [37] for solving the bipartite matching problem.  
4: for  $i = 1,2,\ldots,K$  do  
5: Env.reset()  
6:  $g \gets \hat{p}^c$   
7: for  $t = 0,1,\ldots,H-1$  do  
8: if achieved  $g$  then  
9:  $g \gets$  random goal (randomly sample a few states near  $s_t$  and measure  $p_{\text{pseudo}}$ . Then select a state with the highest value of  $p_{\text{pseudo}}$ )  
10: end if  
11:  $a_t \gets \pi(\cdot | s_t, g)$   
12:  $s_{t+1} \gets \text{Env step}(a_t)$   
13: end for  
14:  $\mathcal{B} \gets \mathcal{B} \cup \{s_0, a_0, g, s_1\ldots\}$   
15: end for  
16: for  $i = 0,1,\ldots,M$  do  
17: Sample a minibatch b from  $\mathcal{B}$  and replace the original reward with intrinsic reward in Section 4.4 (We used relabeling technique based on [2]).  
18: Train  $\pi$  and  $Q$  with b via SAC [13].  
19: Sample another minibatch b' from  $\mathcal{D}_{\mathrm{S}} \sim \{(B, y = 0), (\mathcal{D}_{\mathrm{G}}, y = 1)\}$  and  $\mathcal{D}_{\mathrm{T}} \sim \mathcal{U}$ .  
20: Train  $f_i$  with b' via Eq. (5)  
21: end for  
22: end for
C.1 Full results of the main script
We included the full results of the main script in this section. We include the visualization of the proposed curriculum goals in all environments in Figure 16. The visualization results of the Sawyer-Peg-Pick&Place are not included as it shares the same map with the Sawyer-Peg-Push environment.
Figure 8: Curriculum goal visualization of the proposed method and baselines in all environments. First row: Complex-Maze. Second row: Medium-Maze. Third row: Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place.
C.2 Additional ablation study results
Full ablation study results of the main script. We conducted ablation studies described in our main script in all environments. Figure 17 shows the average distance from the proposed curriculum goals to the desired final goal states along the training steps, and Figure 18 shows the episode success rates along the training steps. Note that there are no results for the ablation study without a curriculum proposal in Figure 17c (unlike Figure 18c) since there are no curriculum goals to measure the distance from the desired final goal states. As we can see in these figures, we could obtain consistent analysis with the results in the main script in most of the environments.
(c) Intrinsic reward
Figure 9: Ablation study in terms of the distance from the proposed curriculum goals to the desired final goal states (Lower is better). First row: Complex-Maze. Second row: Medium-Maze. Third row: Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place. The shaded area represents a standard deviation across 5 seeds.
(c) Intrinsic reward & Curriculum
Figure 10: Ablation study in terms of the episode success rate. First row: Complex-Maze. Second row: Medium-Maze. Third row: Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place. The shaded area represents a standard deviation across 5 seeds.
Curriculum learning objective type. We conduct additional experiments to validate whether reflecting the temporal distance in a curriculum learning objective (Eq (7)) is required since there are a few works that estimate the temporal distance from the initial state distribution to propose the curriculum goals in a temporally distant region or explore based on this temporal information [5, 37]. To reflect the temporal distance in the cost function (Eq (7)), we modify it as  $w(s_i, g_i^+) \coloneqq \mathcal{C}\mathcal{E}(p_{\mathrm{pseudo}}(y = 1|s_i; g_i^+); y = p_{\mathrm{pseudo}}(y = 1|g_i^+; g_i^+)) - V^\pi(s_0, \phi(s_i)) (\phi(\cdot) \text{ is goal space mapping})$  since the value function itself implicitly represents the temporal distance if we use the sparse reward or custom-defined reward similar to the sparse one. In this case, our proposed intrinsic reward outputs 1 for the desired goal and 0 for the explored states, and it works similarly to the sparse one.
We experimented with this modified curriculum learning objective (+Value), and the results are shown in Figure 19, 20. It shows that there is no significant difference, which supports the superiority of our method in that our method achieves state-of-the-art results without considering additional temporal distance information.
Figure 11: Ablation study in terms of the average distance from the curriculum goals to the final goals (Lower is better). +Value means that we additionally consider the value function bias in the curriculum learning objective to reflect the temporal distance from the initial state distribution.
(a) Complex-Maze
(d) Ant Locomotion
Figure 12: Ablation study in terms of the episode success rates. +Value means that we additionally consider the value function bias in the curriculum learning objective to reflect the temporal distance from the initial state distribution.
Choice of goal candidates in training conditional classifiers. As mentioned in the main script, we also experimented with different choices of the goal candidates when we train the conditional classifiers (Eq (5)). The default setting is  $\mathcal{D}_{\mathrm{G}} = \mathcal{D}_{\mathrm{T}}$ , and we also experimented with  $\mathcal{D}_{\mathrm{G}} = \mathcal{B} \cup p^{+}(g)$ . The results are shown in Figure 21, 22. It shows that there is no significant difference, which means we can even make the problem setting more strict by conditioning the classifier only with the visited states and the given desired outcome examples.
Figure 13: Ablation study in terms of the average distance from the curriculum goals to the final goals (Lower is better). There are no significant differences between the choice of goal candidates to train the conditional classifiers.
(a) Complex-Maze
(d) Ant Locomotion
Figure 14: Ablation study in terms of the episode success rates. There are no significant differences between the choice of goal candidates to train the conditional classifiers.
D Training & Experiments details
D.1 Training details
- OUTPACE [5]: We follow the default setting in the original implementation from https: //github.com/jayLEE0301/outpace_official.