Outcome-directed Reinforcement Learning by Uncertainty \& Temporal Distance-Aware Curriculum Goal Generation
Authors: Daesol Cho
Seungjae Lee
H. Jin Kim
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Curriculum Learning
Outcome-directed RL
Goal-conditioned RL

Abstract: Current reinforcement learning (RL) often suffers when solving a challenging exploration problem where the desired outcomes or high rewards are rarely observed. Even though curriculum RL, a framework that solves complex tasks by proposing a sequence of surrogate tasks, shows reasonable results, most of the previous works still have difficulty in proposing curriculum due to the absence of a mechanism for obtaining calibrated guidance to the desired outcome state without any prior domain knowledge. To alleviate it, we propose an uncertainty \& temporal distance-aware curriculum goal generation method for the outcome-directed RL via solving a bipartite matching problem. It could not only provide precisely calibrated guidance of the curriculum to the desired outcome states but also bring much better sample efficiency and geometry-agnostic curriculum goal proposal capability compared to previous curriculum RL methods. We demonstrate that our algorithm significantly outperforms these prior methods in a variety of challenging navigation tasks and robotic manipulation tasks in a quantitative and qualitative way.

Published as a conference paper at ICLR 2023
OUTCOME-DIRECTED REINFORCEMENT LEARNING BY UNCERTAINTY & TEMPORAL DISTANCE-AWARE CURRICULUM GOAL GENERATION
Daesol Cho", Seungjae Lee", H. Jin Kim
Scoul National University, Automation and Systems Research Institute (ASRI),
Artificial Intelligence Institute Scoul National University (AIIS)
(dacho1234, ysz0301, hjink frinkimm@snu.acc
ABSTRACT
Current reinforcement learning (RL) often suffers when solving a challenging exploration problem where the desired outcomes or high rewards are rarely ob- served. Even though curriculum RI., a framework that solves complex tasks by proposing sequence of surrogate tasks, shows reasonable results, most of the previous works still have difficulty in proposing curriculum due to the absence of mechanism for obtaining calibrated guidance the desired outcome state without any prior domain knowledge. To alleviate it, we propose an uncertainty & temporal distance aware curriculum goal generation method for the outcome- directed RL via solving bipartite matching problem. It could not only provide precisely calibrated guidance of the curriculum the desired outcome states but also bring much better sample efficiency and gcometry-agnosee proposal capability compared to previous curriculum RL methods. We demon- strate that our algorithm significantly outperforms these prior methods in variety of challenging navigation tasks and robotic manipulation tasks in a quantitative and qualitative w
curriculum goal
INTRODUCTION
While reinforcement learning (RL) shows promising results in automated learning of behavioral skills, it still not enough to solve challenging uninformed search problem where the desired behavior and rewards are sparsely observed. Some techniques tackle this problem by utilizing the shaped reward (Hartikainen et al., 2019) or combining representation learning for efficient explo- ration (Ghosh al. 2018). But, these not only become prohibitively time-consuming in terms of the required human efforts, but also require significant domain knowledge for shaping the reward designing the task task-specific representation learning objective. What if could design the algo- rithm that automatically progresses toward the desired behavior without any domain knowledge and human efforts, while distilling the experiences into the general purpose policy?
An effective scheme for designing such an algorithm one that learns on a tailored sequence of curriculum goals, allowing the agent autonomously practice the intermediate tasks. However, fundamental challenge is that proposing the curriculum goal to the agent is intimately connected to the efficient desired outcome-directed exploration and vice versa. If the curriculum generation is ineffective for recognizing frontier parts of the explored and feasible areas, an efficient exploration toward the desired outcome states cannot be performed. Even though some prior works propose to modify the curriculum distribution into uniform one over the feasible state space (Pong et al.) 2019 Klink et al., 2022) or generate a curriculum based on the level of difficulty (Florensa et al., 2018 Sukhbaatar al.) 2017). most of these methods show slow curriculum progress due to the process of skewing the curriculum distribution toward the uniform one rather than the frontier of the explored region or the properties that are susceptible to focusing infeasible goals where the agent's 's capability stagnates in the intermediate level of difficulty.
"Equal contribution.
Code available https://gi thub. com http://ayyl.hhaal///ppppeeeeeeep

---

Published as a conference paper at ICLR 2023
EC
Curriculum goals
Desired outcome
[Unceraintt]
Explored
Uncertain
Desired
CEC
state
[Temporal dist, from initial]
Temporllly
Temporally
Close
Far
Figure 1: OUTPACE proposes uncertainty and temporal distance- curriculum goals to enable the agent to progress toward the desired outcome state automatically. Note that the temporal distance estimation is reliable within the explored region where query the curriculum goals.
Conversely, without the efficient desired outcome-directed exploration, the curriculum proposal could be ineffective when recognizing the frontier parts in terms of progressing toward the de- sired outcomes because the curriculum goals, in general, are obtained from the agent's experiences through exploration. Even though some prior works propose the success-cxample-base (Fu et al. 2018; Singh et al., 2019 Li et al. 2021). these are limited to achieving the only given example states, which means these cannot be generalized to the arbitrary coal-conditioned agents. Other approaches propose to minimize the distance between the curriculum distribution and the de- sired outcome distribution (Ren et al. 2019 Klink et al. 2022), but these require assumption that the distance between the samples can be measured by the Euclidean distance metric, which cannot be generalized for an arbitrary geometric structure of the environment. Therefore, argue that the development of algorithms that simultancously address both outcome-directed exploration and curriculum generation toward the frontier is crucial to benefit from the outcome-directed curriculum RL. In this work, we propose Outcome-directed Uncertainty & TemPoral distance-Awar Curriculum goal gEneration (OUTPACE) to address such problems, which requires desired outcome examples only and not prior domain knowledge nor external reward from the environment. Specifically, the key elements of our work consist of two parts. Firstly, our method addresses desired outcome- directed exploration
approaches
a Bayesian classifier incorporating an uncertainty quantification based on the conditional normalized maximum likelihood (Zhou & Levine, 2021; Li et al., 2021). which enables our method to propose the curriculum into the unexplored regions and provide directed method utilizes Wasserstein distance with
guidance toward the desired outcomes. Secondly, time-step metric (Durugkar et al.) (2021b) not only for a temporal distance-aware intrinsic reward but also for querying the frontier of the explored region during the curriculum learning. By deploying the above two elements, we propose a simple and intuitive curriculum learning objective formalized of calibrated curriculum goals that interpolates between
with bipartite matching generate the initial state distribution and desired outcome state distribution.
To sum up, our work makes the following key contributions.
We propose an outcome-directed curriculum RL method which only requires desired out- come examples and does not require an external reward.
To the best of the author's knowledge, we are the first to propose the uncertainty & tempo- distance-aware curriculum goal generation method for geometry-agnostie progress by leveraging the conditional normalized maximum likelihood and Wasserstein distance.
Through several experiments in goal-reaching environments, we show that our method outperforms the prior curriculum RL methods, most notably when the environment has a geometric structure, and its curriculum proposal shows properly calibrated guidance toward the desired outcome states in a quantitative and qualitative way.
RELATED WORKS
While a number of works have been proposed to improve the exploration problem RL, still remains challenging open problem. For tackling this problem. prior works include state- visitation counts (Bellemare et al. 2016 Ostrovski et al. 2017). curiosity/similaarty--iii explo- ration (aathak al. 2017: Warde-Farley al. 2018), the prediction model's uncertainty-based
2

---

Published as a conference paper at ICLR 2023
exploration (Burda al. 2018, Pathak et al. 2019). mutual information based exploration Ey- senbach et al. 2018: Sharma et al. 2019 Zhao et al.
2021; Laskin et al. 2022). maximizing the
entropy of the visited state distribution (Yarats et al. 2021 Liu & Abbeel 2021a bb. Unfortunately,
these techniques are uninformed about the desired outcomes: the trained agent only knows how to visit frontier states diverse as possible. On the contrary, we consider a problem where the de-
sired outcome can be specified by the given desired outcome examples, allowing for more efficient outcome-directed exploration rather than naive frontier-directed exploration.
Some prior methods that try to accomplish the desired outcome states often utilize the provided suc- cess examples (Fu et al., 20188 Singh et al. 2019 Eysenbach et al. [22T]] [i et al. (02T). However, they do not provide a mechanism for distilling the knowledge obtained from the agent's 's experiences into general-purpont
policies that can be used to achieve new test goals. In this work, we utilize the Wasserstein distance not only for an arbitrary goal-conditioned agent but also for querying the frontier of the explored region during curriculum learning. Although the Wasserstein distance has been adopted by some previous research, they are often limited to imitation learning skill discov-
ery (Dadashi et al.) 2020; Haldar et al.] 2022: Xiao et al. 2019 Durugkar et al. 2021a Fickinger et al. 2021). Another work (Durugkar al. 2021b) tries utilize the Wasserstein distance with
the time-step metric for training the reaching agent, but it requires a stationary goal distribution for stable distance estimation. Our work is different from these prior works in that the distribution
during training is non-stationary for calibrated guidance to the desired outcome states.
Suggesting curriculum also make exploration easier where the agent learns tailored quence of tasks, allowing the agent to autonomously practice the intermediate tasks in training process. However, prior works often require significant amount of samples to measure the curricu- lum's level of difficulty (Florensa et al. 2018 Sukhbaatar et al. 2017). learning progress (Portelas) al. 2020), regret (Jiang et al., 2021p. l.. Curricula are often generated by modifying the goal dis- irbuution into the frontier of the explored region via maximizing a certain surrogate objective, such as the entropy of the goal distribution (Pong al., 2019). disagreement between the value function ensembles (Zhang et al.] 2020), but these methods do not have convergence mechanism to the de- sired outcome distribution. While some algorithms formulate the generation of a curriculum as an explicit interpolation between the distribution of target tasks and a surrogate task distribution (Ren et al. 2019, Klink et al. 2022), these still depend on the Euclidean distance metric when measuring the distance between distributions, which cannot be generalized for arbitrary geometric structure of the environment. In contrast, our method not only provides calibrated guidance to the desired out- come distribution in sample efficient way but also shows gegnetry-agnostic curriculum progress by leveraging the bipartite matching problem with uncertainty & temporal distance-based objective.
3 PRELIMINARY
We consider the Markov decision process (MDP) M (S. 7). where S denotes the state space, AA the action space, G the goal space, T(s'ls, a) the transition dynamics, Po the initial distribution, Pr the state visitation distribution when the agent follows the policy π, and y the dis- count factor. The MDP in our framework is provided without reward function and considers an environment in which only the desired outcome sample { are given, assuming that of is obtained from the desired outcome distribution G+. Therefore, our work employs a trainable in- trinsic reward function S x G x A + R, which is detailed in Section 4.1 Also, we represent the set of curriculum goals as *1 k= and assume that these are sampled from the curriculum goal distribution Ge obtained by our algorithm.
3.1 WASSERSTEIN DISTANCE OVER THE TIME-STEP METRIC
The Wasserstein distance represents how much "work" is required to transport one distribution to another distribution following the optimal transport plan (Villani 2009 Durugkar et al.) 20211b. In this section, we describe the Wasserstein distance over the stme-stp metric and how it can be obtained with a potential function f. Consider metric space (X' d), where X' is set and d is metric on X, and two probability measures μt, v on X. The Wasserstein-p distance for a given metric d is defined follows.
W Wp(,,,) v) == inf E (xx,))=,'d(x,)))) ifgel
[d(X,
sup [Byr Ez [f)]]( (1)
7ell(p.r)
1l 11
3

---

Published as a conference paper at ICLR 2023
where joint distribution Y denotes transport plan, and [[ denotes the set of all possible joint distributions ",s and the second equality is held by the Kantorovich-Rubinsteinn Rubinstein duality with 1-Lipschitz functions (f X + R) (Villami 2009 Arjovsky et al. (017).
we could define the distance metric as (s, sg), which is time-step metric (quasimetric) based on the number of transition steps experienced before reaching the goal &y E for the first time when executing the goal-conditioned policy π (als,, 8g). we could design a temporal-distance aware RL by minimizing the Wasserstein distance Wi (pr, () that gives estimate of the work needed to transport the state visitation distribution Pa to the goal distribution G:
(P. sup [B,,~olf(s Es-o_ [f()]] [1III
Then, the potential function f is approximately increasing along the optimal goal-reaching trajec- tory. That is, if (s) consists of the states optimally reaching toward the goal sg. f(s) increases along the trajectory and f(yy has the maximum value (Durugkar et al. a Adopting these prior works. the I-Lipschitz potential function with respect to d" sg) could be ensured by en- foreing that the difference in values of f on the expected transition from every state is bounded by as follows, (detailed derivations are in Appendix B
sup (ee s()]I s 1. SES
(3)
3.2 CONDITIONAL NORMALIZED MAXIMUM LIKELIHOOD (CNML)
For curriculum learning. our work utilizes conditional normalized maximum likelihood (CNML) (Rissanen & Roos, Fogel Feder 2018) that can perform an uncertainty
classification based on previously observed data by minimizing worst-case regret. Let DD = {(sp: ep) {p))p= be a set of data containing pairs of states 81:--1 and success labels etm-1 E {0, 1}. where '1' represents the occurrence of the desired event. Given a query point 8n, CNML in our framework defines the distribution PCNML (en|s,,) which predicts the probability that the state sn belongs to the desired outcome distribution G (e= 1).
To explain how CNML predicts the label of sn, suppose θ is set of models, where each model B E θ can represent a conditional distribution of labels, po (e11m|/11nd) CNML considers the possibilities that all the possible classes (0,1) will be labeled to the query point Sn, and obtains the models ((e1.n (1nn))
that represent the augmented datasets D DUL (sn,en) well by solving the maximum likelihood estimation (MLE) problem (LHS of Eq. (4). Then, CNML that minimizes the regret over those maximum likelihood estimators can be written as follows (Bibas al.) 2019).
Pa. Pa ilsn)
= arg = -a-gggaaaaaeeeeeeooaaeooeoole 6Ee
PoNML(en = =(ssn)
(4)
Σ) Po, (e: j(ss)
If the query point 8,, is close of the data points in the datasets, CNML will have difficulty in assigning a high likelihood to labels that are significantly different from those of nearby data points. However, if is far from the data points in the dataset, each MLE model 0 will predict the label en as its own class, which leads large discrepancy provides us with normalized likelihoods closer to uniform (RHS of Fq 4)). Thus, by minimizing the regret through labeling all possible classes to the new query data, CNML. can provide a reasonable uncertainty estimate (Li et al. 2021 Zhou & Levine 2021) on the queried s, and classify whether the queried s, similar to the previously observed data either the forms of label O, 1, or out-of- distribution data, which is predicted as 0.5.
the predictions between the models and
However, the classification technique via CNML described above is most cases computationally intractable, as it requires solving separate MLE problems until convergence every queried data. Previous methods proposed some ideas to amortize the cost computing CNML distribution (Zhou
& Levine 2021
Li al. 2021). Following those prior methods, our work adopts MAMI. (Finn
4

---

Published as a conference paper at ICLR 2023
SSS
goat
Gestred
Goal Poter
Righ resurd
Uncertabe
state
Inttlol Point
reward
Figure 2: Visualization of the uncertainty quantification along training progress (left) and trained f (s) (right) in the Point-N-Maze environment. the right figure, high reward means temporally close to the desired outcome states, and low reward means the opposite.
et al. 2017) to address the computational intractability of CNML by training one meta-learner net- work that can quickly adapt to each model 0 rather than training each model separately. As Finn al. (2017) requires samples from G and replay buffer for the inference, the probability should be represented PCNMI((e als; G+, B3,, but we use PCNMI. (e ils) for notational simplicity in this work. More details about the meta-learning-basse classification are included Appendix B
METHOD
For calibrated guidance of the curriculum goals to the desired outcome distribution, we propose to progress the curriculum towards the uncertain & temporally distant area before converging to G+, as it not only the most intuitive way for exploration but also enables the agent to progress without any prior domain knowledge the environment such obstacles. In short, our work tries obtain the distribution of curriculum goals G that are considered (a) temporally distant from and, (b) uncertain and, (c) being progressed toward the desired outcome distribution G+.
4.1 TEMPORAL DISTANCE- RL WITH THE INTRINSIC REWARD
This section details the intrinsic reward for the RL agent well the method of training the parameterized potential function for trained with the data collected by the policy [[ass, sy)s We consider a I-Lipschitz potential function fā whose value increases as the state is far from the initial state distribution and getting close to the goals 8g E 9 proposed by curriculum learning. Then, we can train agent that reaches the goals 89 as few steps possible by minimizing the Wasserstein distance Wi (px,9)
Considering that we can obtain the estimate of W (P 9) by Eq (2. the loss for training the param- eterized potential function fā can be represented as follows DDrugkarr al. 2021b a):
Ls =
(s + A E., ee (z) -s))
(5)
The penalty term with coefficient in Eq 5 from Eq 3 for ensuring the smoothness requirement since consider the Wasserstein distance over the time-step metric. Then, assuming the parameter ó is trained by Eq (5) at every training iteration, we could obtain the supremum of Eq (2). Thus, the
reward can be represented as (s) f8 (sg). which corresponds to - WW((pr
4.2 CURRICULUM LEARNING
As CNML can provide near-uniform prior (prediction of 0.5) for out-of-distribution data given the datasets (Section 3.2), we could utilize it by treating the desired outcome states in G+ as (e 1), and data points in the replay buffer B as (e 0). Then, could quantify the uncertainty of a state s based on CNML as
Tuucet (s (s, = 1 (e 0|s) PONMI 1|s)].
(6)
which is proportional to the uncertainty of the queried data s. However, qucert alone cannot provide curriculum guidance toward the desired outcome states because it only performs uninformed search over uncertainties rather than converging to the desired outcome states. Thus, we modify Eq (6) with an additional guidance term:
5

---

Published as a conference paper at ICLR 2023
Tucore (g,9*) = Es [log (uucert (s.gt) + c. Aguidance (s, (s,g)))
(7)
where 7/guidance (8, 9+) (PCNMI.e 1[s) - 0.5) 1 (PONMI. = 1|8) 2 0.5), and
hyperparameter that adjusts the preference on the desired outcome states. Since the CNML provides near-uniform prior for dustt---istiibution data, qucert provides large values in the uncertain arcas. Also, the guidance term Tiguidance (s, G+ reflects the preference for the states considered to be closer to the desired outcome state distribution.
However, in practice, found the uncertainty quantification itself sometimes has numerical errors, and it makes PCNML erroneously predict the states near the initial states or boundaries of the already explored regions uncertain areas. Therefore, we assume that the curriculum should incorporate the notion of not only the uncertainty but also the temporally distant states from Po for frontier and desired outcome-directed exploration. Thus, we formulate the final curriculum learning objective as follows:
arg max Mucert (9), g 1 WI go
(8)
where the temporal distance bias term with coefficient L is represented by the Wasserstein distance from initial state distribution Po the curriculum goal distribution g And, given parameterized 1-Lipschitz potential function f over the time-step metric d= (s, 8g), we can obtain the estimate of
Wi (Po 9") by RHS of Eq I Also, if we assume ğ to be a finite set of K particles that is sampled from already achieved states the replay buffer B, the objective function aim to maximize can be represented as follows:
Mucert (s", ",, 107(6)) E" sb E p
max G-
enables to propose the currieulum that reflects not only the uncertainty of the states and preference on the desired outcomes but also temporally distant states from Po. while not requiring prior domain knowledge about the environment such as an obstacle.
4.3 SAMPLING CURRICULUM GOAL VIA BIPARTITE MATCHING
Since assume that desired outcome examples from G+ given rather than its distribution, could approximate it by the sampled set Ś+ (1Ś+ = K). Then, solve the currieulum learning problem of Eq (9) we should address the combinatorial setting that requires assigning Go from the entire curriculum goal candidates in the replay buffer to the G+, which addressed via bipartite matching in this work. With the hyperparameter problem with the costs cross-entropy loss (CE) and temporal- the Appendix for the detailed derivation.)
4, we can rearrange Eq (9) minimization
bias (f)) term: (Refer to
min
Σ
w(s+,
(10)
G=:|0=|KK
steg of. e
w(s' CC 1|s)): y= (C = 1lg4t)) (z
(11)
Intuitively, before discovering the desired outcome states in G+, the curriculum goal s* is proposed in region of the state space considered to be uncertain and temporally distant from Pa in order to recognize the frontier of the explored regions. And is kept updated to converge to the desired outcome states for minimizing the discrepancy between the predicted labels of g+ and g=.
Then we can construct bipartite graph G with the cost of the edges Let V. and V, be the sets of nodes representing achieved states in replay buffer and S+ respectively. We define bipartite graph G((V.. a. V&).Er E) with the weight the edges E(...) = -w(.,-) and separated partitions (V, and Vz). To solve the bipartite matching problem, we utilize the Minimum Cost Maximum Flow

---

Published as a conference paper at ICLR 2023
foidel curricutoo
surrictto
curvicufam
Final
(a) OUTPACE(QUIS)
CURROT
(c) HGG
Figure 3: Visualization of the proposed curriculum goals. First row: Ant Locomotion, Second row:
Point-N-Maze
algorithm find K edges with the minimum cost w connecting V. and V (Ahuja et 1993 Ren et al. 2019). The overall training process is summarized in Algorithm in Appendix B
EXPERIMENTS
We include 6 environments to validate our proposed method. Firstly, various maze environ- ments (Point U-Maze, N-Maze, Spiral-Maze) used validate the ecometry-agnossee curricu- lum generation capability. Also, we experimented with the Ant-Locomotion and Sawyer- Push, Pick&Place environments evaluate our method in complex dynamics other domains rather than navigation tasks.
We compare with other previous curriculum goal generation methods, where each method has the following properties. HGG (Ren et al. (201)): Minimize the distance between the curriculum and desired outcome state distributions based on the Euclidean distance metric and value function bias. CURROT (Klink et al. 2022): Interpolate between the curriculum and desired outcome state distri- bution based on the agent's current performance via optimal transport. GoalGAN (Florensa al., 2018): Generate curriculum goals that are intermediate level of difficulty by training GAN (Good- fellow al. 2014). PLR (Jiang et al., 2021 J: Sample increasingly difficult tasks by prioritizing the levels of tasks with high TD errors. ALP-GMM (Portelas et al. 2020): Fit GMM with absolute learning progress score approximated by the absolute reward difference. VDS (Zhang et al. 2020): Prioritize goals that maximize the epistemic uncertainty of the value function ensembles. SkewI it (Pong et al., 2019): Maximize the entropy of the goal distribution to be uniform on the feasible state space via skewing the distribution trained by VAE (Kingma & Welling 2013).
5.1 EXPERIMENTAL RESULTS
Firstly, show how each module our method is trained, we visualized the uncertainty quantifica- tion by CNML, and f values which are proportional to the required timesteps reach from Pp. The uncertainty quantification results (FigureZ) show that the classifier PCNML (-Is) successfully discriminates the queried states as already explored region or desired outcome states, otherwise, un- certain states. Due to the coometry-agnostie property of the classifier PCNML(-|s) (-Is), could propose the curriculum in the arbitrary geometric structure of the environments, while most of the previous curriculum generation methods do not consider it.
We also visualized the values the trained potential function f (s) to show how the intrinsic reward is shaped (Figure 2). As the potential function f (s) is trained to have high values near the desired outcome states due to the Wasserstein distance with the time-step metric, the results show gradual increases of (s) values along the trajectory toward the desired outcome states. That is, high values of f (s) indicate the smaller required timesteps reach the desired outcome state, and this property brings the advantage in identifying the frontier of the explored region.
To validate whether the curriculum goals are properly interpolated from initial states to desired outcome states by combining both objectives for curriculum learning (Eq (8) we evaluated the

---

Published as a conference paper at ICLR 2023
(a) U-Maze
(b) N-Maze
(c) Spiral-Maze
1000
2002
M423
1030
2000
5000
14:23
20.00
(d) Ant Locomotion
Sawyer Push
(l) Sawyer Place
burs)
1600
ALP-CMM
Dew
voS
Figure 4: Average distance from the curriculum goals to the final goals (Lower is better). Our method's increasing tendencies initial steps in environments due the geometric struc- ture of the environments themselves. Shading indicates a standard deviation across seeds.
5000000 066
(a) U-Maze
(b) N-Maze
(c) Spiral-Maze
(d) Ant Locomotion
Sawyer Push
(1 Sawyer Pick&Place
Figure 5: Episode success rates of the evaluation results. The legends and seeds the as Fig Note that the curves of the baselines some environments are not visible as they overlap at zero success rates.
progress of the curriculum goals a quantitative and qualitative way. For quantitative evalua- tion, compare with other previous works described above with respect to the distance from the proposed curriculum goals to - As we can see in Figure 4 our method is the only one that consistently interpolates from initial states G+ training proceeds, while others have difficulty with complex dynamics or geometry of the environments. For qualitative evaluation, we visualized the curriculum goals proposed by our method and other baselines that show somewhat comparable results as training proceeds (Figure 3). The results show that our method consistently proposes the proper curriculum based on the required timesteps and uncertainties regardless of the geometry and dynamics of the various environments, while other baselines have difficulty as they utilize the Eu- clidean distance metric interpolate the curriculum distribution to the G++. We also evaluated the desired outcome-directed RL performance. As we see in Figure 5 our method is able very quickly learn how to solve these uninformed exploration problems through calibrated curriculum goal proposition.
5.2 ABLATION STUDY
Types of curriculum learning cost. We first evaluate the importance of each curriculum learning objective Eq 88. Specifically, we experimented only with umcertainty-relatee objective (only- enml) and timestep-related objective (only-f) when curriculum learning progresses. As
8

---

Published as a conference paper at ICLR 2023
CAITING EISUnD anion COlN
deserce 12
2030
190
Glaatie only.
STHON
(a) Cost type
(b) Reward Goal proposition type
(c) Effect of in Aguddaace
Figure 6: Ablation study in terms of the distance from the proposed curriculum goals to the desired final goal states (Lower is better). First row: Ant I Locomotion. Second row: Sawyer Pick & Place. Shading indicates a standard deviation across seeds.
in Figure 6a both objectives play complementary roles,, which support the requirement of both objectives. Without one of them, the agent has difficulty in progressing the curriculum goals toward the desired outcome states due to the local optimum of fğ or the numerical error of PONML, and more qualitative/quanttttive results and analysis about this and other ablation studies included in Appendix C
Reward type & Goal proposition method. Secondly, we replace the intrinsic reward with the sparse reward, which is typically used in goal-conditioned RL problems, to validate the effects of the timestep-proportionatl
shaped reward. Also, for comparing the curriculum proposition method, we replace the Bipartite Matching formulation with a GAN- based generative model, which is similar to/Florensa et al. 2018). but we label the highly uncertain states positive labels instead of success shaped reward shows consistently
rates. As we can see in Figure 6b the timestep-proportion better results due the more informed reward signal compared the sparse one, and the generative model has difficulty in sampling the proper curriculum goals because GAN shows training instability with the drastic change of the positive labels, while our method is relatively insensitive because curriculum candidates are obtained from the experienced states from the buffer B rather than the generative model.
Effect of in Iguidaacce Lastly, we experiment with different values of hyperparameter e to validate the effect of ?guidance on curriculum learning. When is smaller than the default value 4, it still possible to explore most of the feasible state space except the area near the desired outcome states due to the uncertainty temporal distance-aware curriculum (Figure bc). But, we could verify that "lguidance' 's effect becomes smaller as с decreases and Vguiddance helps to guide the curriculum goals to the desired outcome states precisely. This is consistent with our analysis that the uncertainty & temporal distance themselves provide curriculum goals in the frontier of the explored region while Nguidance can further accelerate the guidance to the desired outcome states.
CONCLUSIONS
In this work, we consider an outcome-directed curriculum RL where the agent should progress toward the desired outcome automatically without the reward function and prior knowledge about the environment. We propose OUTPACE, which performs uncertainty, temporal distance-aware curriculum RL with intrinsic reward, based on the classifier by CNML, and Wasserstein distance with time-step metric. We show that our method can outperform the previous methods regarding sample efficiency and curriculum progress quantitatively and qualitatively. Even though our method shows promising results, there are some issues with computational complexity due to the innate properties the meta-learning inference procedure itself. Thus, it would be interesting future work to find way reduce the inference time for less training wall-clock time.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENT
This work was supported by Al based Flight Control Research Laboratory funded by Defense Acqui- sition Program Administration under Grant UD200045CD. Seungjae Lee would like acknowledge financial support from Hyundai Motor Chung Mong-Kon Koo Foundation.
REFERENCES
R K Ahuja, T L Magnanti, and B Orlin. Network Flows: Theory, Algorithms, and Applications. Prentice Hall, Englewood Cliffs, NJ, 1st edition, 1993.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv: wrxiv:1707.01495, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214 223. PMLR, 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information pro- cessing systems, 29, 2016.
Koby Bibas, Yaniv Fogel, and Meir Feder. Deep pnml: Predictive normalized maximum likelihood for deep neural networks. arXiv preprint arXiv: 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imita- tion learning. arXiv preprint arXiv:2006.0444 04678, 2020.
Ishan Durugkar, Steven Hansen, Stephen Spencer, and Volodymyr Mnih. Wasserstein distance max- imizing intrinsic control. arXiv preprint arXiv:21 10.//331, 2021a.
Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34:8622-8636, 2021b.
Ben Eysenbach, Sergey Levine, and Russ Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. Advances in Neural Information Pro- cessing Systems, 34:1 1541 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint
2018.
Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation learning via optimal transport. arXiv preprint arXiv:2/10.03662 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
Yaniv Fogel and Meir Feder. Universal supervised learning for individual data. arXiv preprint arXiv:1812.09520. 2018
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31, 2018.
10

---

Published as a conference paper at ICLR 2023
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies. arXiv preprint arXiv: 2018.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:140..26111 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learming, pp. 1861-1870. PMLR, 2018.
Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and match: Supercharging imitation with regularized optimal transport. arXiv preprint arXiv:226./55669, 2022.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprint arxiv:1907.08220 2019.
Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum re- inforcement learning using optimal transport via gradual domain adaptation. arXiv preprint arXiv:2210.10195 2022.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference Machine Learning, pp.
PMLR, 2021.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114 2013.
Pascal Klink, Haoyi Yang, Carlo D' Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforce- ment learning via constrained optimal transport. In International Conference on Machine Learn- ing. pp. 11341-11358 PMLR, 2022.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint wrXiv:2202.00111, 2022.
Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-avee aware rewards for outvome-driven reinforcement learning. In International Conference on Machine Learning, pp. 6346-6356. PMLR, 2021.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International
Conference on Machine Learning, pp. 6736 6747. PMLR, 2021a.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 18:114499-188 18473, 2021b.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2734 PMLR, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL httpa////ppppee...aeiipp...oo 9015 otgp:p/cc.....aaaaa........eeeea pdf
librarr.......
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 7778-2787. PMLR, 2017.
11

---

Published as a conference paper at ICLR 2023
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Interaational conference on machine learning, pp. 5062-5071. PMLR, 2019.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.00 1903.03698, 2019.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre Yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Conference on Robot Learning, pp. 835-853, PMLR, 2020.
Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32, 2019.
Jorma Rissanen and Teemu Roos. Conditional nml universal models. In 2007 Information Theory and Applications Workshop, pp. 337-341. IEEE.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware 2019.
unsupervised discovery of skills. arXiv preprint
Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Benchmarking and formalism. arXiv preprint arXiv:2112.09605 2021.
Avi Singh, Larry Yang. Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint 1904.07504 2019.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407, 2017.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
David Warde-Farte FardeeFarley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- non-parametric discriminative rewards. arXiv preprint 1rXiv:1811.1133 2018
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning. arXiv preprint arXiv:/906.08/ 13, 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pp. 11000 PMLR, 2020.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis-
agreement. Advances in Neural Information Processing Systems, 33:7648-7659, 2020.
Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprint 103.v:2103.08 2021.
Aurick Zhou and Sergey Levine. Amortized conditional normalized maximum likelihood: Reliable out of distribution uncertainty estimation. In International Conference Machine Learning, pp- 12803-12812. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
A TRAINING & EXPERIMENTS DETAILS
A.1
TRAINING DETAILS
Baselines.
The baseline curriculum RL. algorithms are trained as follows,
2019) We follow the default setting in the original implementation from https:// g hub. com Stssttilwww Sestlweel-Giin /iinnddiii wendsight-Ceee ght-Goa 1enall-Geerrre r ioon
HGG (Ren et al.
CURROT (Klink et al. 2022): We follow the default setting in the original implementation
from httm/ttts/:://tttt/ wttp:/ccc.pppcccccomoo/mm http:/paaclxnnn/urrrttttt currot
GoalGAN (Florensa et al.
2018).
PLR (Jiang et al. 2021). VDS (Zhang al. 2020 We follow the default setting in implementation from
ALP-GMM (Portelas et al. 2020)
attps://githus:
com wttp://wwoocooooccccccccc
SkewFit (Pong al. 2019): We follow the state-based version of SkewFit. The original implementation was modified and used since only the amage-based version is provided in it. (https://gitte
com/coooiii.ooo berkeley/rkkii
All the baselines are trained by SAC (Haarnoja et al. 2018) with the sparse reward except for the SkewFit as it uses reward based on the conditional entropy. Even though some algorithms' original implementation is based on on-policy algorithms such as TRPO or PPO (Schulman et al., 2015 2017]. for comparing the sample efficiency, we replace the on-policy algorithm with the off off-policy algorithm, SAC, following the referred implementation.
Table 1: Conceptual comparison between our work and the previous curriculum RL algorithms.
Uncert.
Timestep -Aware
Targel
On-
Curriculum proposal
Withoul reward
Withou non- forgetting mechanism
-Aware
curriculum dist.
policy
HGG
B
GoalGAN
X
x
GAN
CURROT
u
PLR
x
8
x
x
B
VDS
x
x
B
x
ALP-GMM
X
8
x
7
GMM
X
itewlit
x
/
VAE
X
OUTPACE (ours)
g
B
We included a conceptual comparison between our work and previous other curriculum or goal generation methods in Table
Uncert.-Aware: whether the curriculum goal proposal process is aware of the uncertainty of the candidate goals.
Twwesee whether the curriculum goal proposal process is aware of the temporal distance from the initial states or from the desired outcome states.
Target curriculum dist: whether there exists a mechanism converge to the target distribution. When there is no target distribution (e.g. just exploring or expanding the curriculum goal distribution as diverse as possible.), we denoted it as X.
the curriculum goals to
Off-policy: whether the off-policy RI can be applied. Some baselines need to measure a kind of difficulty, which means they require repeated trials and on-policy RL algorithms with multi-proccesing such as TRPO, and PPO (Schulman et al. 2015 2017).
Curriculum proposal: where the curriculum goals are proposed from.
Without ext. reward: whether the algorithm requires external environmental reward or not.
Without non-forgetting mechanism: whether the algorithm requires implicit or explicit non-for forgettng mechanisms. Some baselines mix the previously practiced curriculum goals with a fixed or varying ratio or make the curriculum distribution into uniform over the state space cover all possible test goal states.
13

---

Published as a conference paper at ICLR 2023
Training details. To train the potential function fo via Eq. (5), and sg in the buffer should ideally contain all feasible states in the environment. However, until the policy is learned enough to explore the map, obtaining such ideal distribution is difficult. To mitigate this issue, following Durugkar et al. (2021b), we approximate such a distribution with a small replay buffer By containing recent trajectories and the relabelling technique (Andrychowicz does not provide fo with the ideal state distribution covering all feasible states, we empirically found that this assumption works well since OUTPACE only queries fo for the explored area when it generates curriculum goals.
al.) (017). While this approximation
Table 2: Hyperparameters for OUTPACE
Critic hidden dim
512
discount factor Y fo update frequency # of gradient steps for fo update
0.99
Critic hidden depth
3
1000
Critic target т
0.01
10
Critic target update frequency
2
# of ensemble networks for fø
5
Actor hidden dim
512
learning rate for fo
le-4
Actor hidden depth
RL optimizer
adam
Actor update frequency
2
Meta-learner network hidden size
[2048,2048)
RL batch size Init temperature Oiii of SAC Replay buffer B size
512
Meta-learner train sample size
512
0.3
Meta-learner test sample size
2048
3e6
Meta-learner test batch size
2048
Table 3: Env-specific hyperparameters for OUTPACE
Env name
B size
À for Lo
adam for SAC
L.
10000
25
le-2
0.02
Maze-v-
10000
25
le-2
0.02
Point-Spiral-Maze-
20000
50
1e-8
0.02
Ant Locomotion-v0
50000
25
1e-8
0.02
Sawyer-Peg-Push
30000
25
le-2
2
Sawyer-Peg-Pickk&Plaee
30000
25
le-2
0.02
A.2 ENVIRONMENT DETAILS
Point-U-Maze The observation consists of the zy position, angle, velocity, and angular velocity of the 'point'. The action space consists of the velocity and angular velocity of the 'point'. The initial state of the agent is [0, 0) and the desired outcome states are obtained by adding uniform noise the default goal point [0, 8J. The size of the map 12 12.
It is the same as the Point-U-Maze environment except that the desired outcome states are obtained by adding uniform noise to the default goal point [8, 16). and the size of the map is 12 x 20.
Point-N-Maze
is the same as the Point-U-Maze environment except that the desired outcome states are obtained by adding uniform noise to the default goal point [8, -8]. and the size of the map is 20 20.
Point- Saint-SprrllMaae
Ant Locomotion The observation consists of the ryz position, zyz velocity, joint angle, and joint angular velocity of the 'ant'. The action space consists the torque applied on the rotor of the 'ant'. The initial state of the agent is [0, 0) and the desired outcome states are obtained by adding uniform noise to the default goal point (0, 8). The size of the map 12 x 12.
The observation consists of the zyz position the end-effector, the object, and the gripper's state. The action space consists of the zys position of the end- effector and gripper open/close control. The initial state of the object is [0.4, 0.8, 0.02) and the desired outcome states obtained by adding uniform noise the default goal point -0.3, 0.4, 0.02]. We referred to the metaworld (Yu et al. 2020) and EARL (Sharma et al. 2021 environments.
Sawyer-Peg-Pe
14

---

Published as a conference paper at ICLR 2023
ES
(a) U-Maze
(b) N-Maze
(c) Spiral-Maze
(d) Ant Locomotion
(e) Sawyer Manip.
Figure 7: Environments used for evaluation: (a)-(c) the agent must navigate various kinds maze environments. (d) the quadruped ant must navigate the maze to particular location. (e) the robot has to push or pick&place peg to the desired location.
Pawyer-Pog-PPcck&Plee Pick&Place is the same the Sawyer-Peg Peg-Push environment except that the desired outcome states are obtained by adding uniform noise to the default goal point [-0.3, 0.4, 0.2].
B ALGORITHM & DERIVATIONS
B.1 ALGORITHM
Algorithm Overview of OUTPACE algorithm
1: Input: desired outcome examples Ś+, total training episodes N, Env, environment horizon H,
actor network ग, critic network Q. potential function network fg replay buffer B, B,
2: for iteration=1,2....N do
3: G <- sample K curriculum goals that minimize 1[s PCNM1, (e 1|/+t) L. [s)) (Section 4.3
4: for i=1,2.... do
5: Env.reset()
6:
g - 9 pop for - do
7:
8:
if achieved then
9:
g random goal (randomly sample state with high uncertainty measured by PCNML. in a ball (st)). end if
11:
0, + T-|ss,,)) g) St-11 < Env.step(a:)
13:
end for
14:
< BU {80, 4o, 81 J...., J B, +- By U (80, ao, s1...
15:
end for
16:
for i= do
17:
Sample minibatch from B and label reward using f (s:) (Section 4.1 Train æ and Q with b via SAC (Haarnoja et al.) 2018).
18:
19:
Meta-train PCNML with via meta-NML (Algorithm)
20:
Sample minibatch by from By
21: Train fā with by via Eq. 5
22:
end for
23:
end for
15

---

Published as a conference paper at ICLR 2023
Training Goal Sampling Modules
Sampling Curriculum Goals
Uncertainty Aware Module
Bipartite Matching
Replay Buffer
Augment data D u ( (0,1))
Calculate Uncertainty
Calculate Temp. Dist.
PONME
MAML (Fin et al., 2017)
Sample K curriculum goals that minimize Eq.11
Temporal Distance-Awer Aware Module
Practice Curriculum Goals
Curriculum Goals
Train f by minimizing Lg (Eq.5)
Off- Policy RL
Replay Data
Neural Network Model
Single step adaptation
Figure 8: The overall diagram of OUTPACE
Algorithm 2 Meta-NML (Li et al. 2021)
1: Input: desired outcome examples G+, RL buffer B
2: Sample 9- from RL buffer B
3: D < ğ- U (Let the size 'D is n)
4: Create 2n meta-training tasks by relabelling each data points in as 0 and
(D U (xa,y") y') where v; ~ 'D and y {0)
5: Meta-learn with 2n meta-training tasks (Finn et al. 2017
(Let 0 be the parameter of PCNML and L be standard classification loss.
ming Ex, "
U (xi, 3/ v)), 1,00)/))] s.t.0, 0 a VoL(D (2i, v.,
B.2 DERIVATIONS
This section contains the definition of time-step metric d", derivation of Lipschitz smoothness of f, and derivation of Eq. (11). Regarding dº and Lipschitz smoothness of f. we refer the reader Durugkar et al. (202bb) for more detailed explanation.
B.2.1 TIME-STEP METRIC d" LIPSCHITZ SMOOTHNESS OF
Definition B.1. Given state space S, action space A, transition dynamics T - A S, and agent policy n, the time-step metric d" is quasi- metsi- where the distance from E S to 8g E S is the expected number of transitions required with the policy π.
The time-step metric can be expressed by the expectation of the number of transitions taken under the policy म, where T (-|s,, sg) is the probability distribution of the timestep required go from to Sg. d can also be written recursively as
d' (s, = "n
0
ifs = Ea Ee (s otherwise.
(12)
+ E
16

---

Published as a conference paper at ICLR 2023
Lipschitz smoothness of If the difference in values of f the expected transition from every state is bounded by 1, and the policy x can reach the goal $y within finite number of transitions, then f is the 1-Lipschitz function.
Proof. We can write (f(gg) - fsu)) via telescopic sum,
lf(sy) f(s)) = E Ba s0.
(f($411) f(s)))
(13)
EE TT 50.5g)
-(s4+1)
Since B1] [()]] < by Eq 3). we can write
Ba (110018))
Σ
[f(+)11) f(st))
$ E BT (11008))
Σ1
1=0
1=0
(14)
= (-
= (30, $g)
Thus, f is the 1-Lipschitz function with respect to the time-step metric d"
B.2.2 DERIVATION OF EQUATION
By substituting Eq. into Eq 9 and omitting f (sh) which is not related 9%, obtain the following terms,
log {1 (e = PeNMule 1|s')]
max ==|KK
(15)
+ ((poNMI(e= = 1(8*) - - (PCNNM..E 11]|s)) > 0.5) + L. [3)),,, of e
Also, if we use the default value of the hyperparameters e 4, we can express the above terms as follows,
Σ log {1 |1 2P 1|ss)|+
max 6% -=|=K
(16)
4 (( = 1|s²) - 0.5) (( 1|s") > 0.5)) + L- (g())), e
which can be simplified as
K
min G-
log(pCNML (e 1|s²)) - L- f s go
(17)
If could assume that PONML is well trained to classify the desired outcome examples gi. e ②+ from the states in the already explored region, PCNMI.(e 1lg1) approximately equal
to I (pCNML (e = 1lg[.) s 1). Then, the terms inside the above minimization objective can be approximately represented as
-PCNMI. 119 (e= L. (z
(18)
17

---

Published as a conference paper at ICLR 2023
Then, in practical implementation, implement the above terms by cross-entropy loss
wist gf) (E(NNNL 1|s"): y= (E 11g4, (z)s,,
(19)
which Eq (TT). Even though not exactly equivalent to the mathematical definition of the cross-entropy, we can just implement it with cross-entropy loss developed in standard deep learning framework such as PyTorch (Paszke et al. 2019) because choosing to maximize log(pCNML = 1 1ss))) and choosing s close F4. in order to be classified desired outcome examples by the PONMI. (predicted labels of s to be close 1) have the same intuitive meaning.
B.3 A DETAILED DESCRIPTION OF META-NML
Conditional normalized maximum likelihood (CNML) can perform a conservative k-way classi-
fication based on previously seen data (Li al. 2021 Zhou & Levine 202T). Let D Dtrain {( |=] be a set of data containing pairs of inputs Ti:--1 and labels 91-n-1 E {1, k), where k is the number of possible labels, and @ is set of models. Given new input In, CNML defines the distribution PCNML i)an))e(1 R} by minimizing the regret R of the worst-case
label Yn as
PCNML = arg min max R(g, Tims Miens θ)
(20)
Vn
where we define a regret R for label yn of a distribution q and maximum likelihood estimator as
R(g, Tims V1:1 θ) == loger
( 1 (yimm [m1)) log ((Vi.))
(21)
By solving Eq. (20). CNML predicts the distribution of the new label yn, Eq. 2019
22 BBiaas al.
PCNML = mn) = P Pog (y m ml(n)
(22)
Σ/= Patan (y j(
where a model (((")) a model that represents the augmented dataset D Dirain U (Xn, Un = m) well. Thus, the number of total MLE models required is x l since should address each data by augmenting with the label 1, I,.,,,, k, respectively (/)==l))). Since our algorithm utilizes 2-way classification (k= 2), define 2n tasks T; Imn and 7 1:m which constructed by augmenting negative (I;; = 0) and positive (I,, y = 1) labels respectively for each data point (x:).
To amortize the training cost of the tasks (tasks T==1..) we can apply the meta-learning algorithm (Finn al. 2017 Li al. 2021) to this setting and train model 0 which quickly adapt the optimal solution after single step of gradient update with standard classification loss L as
min E,, Be.. (',,)')) y
(23)
0.1.0%
s.1.0 = -0 (x,, 9:),
(24)
where Eq.
24 represent the objective of meta-learning and quick adaptation respec- tively. Training CNML via meta-learning and leveraging CNML are shown in lines and 19 of the algorithm overview (Algorithm
23 and Eq
D. Also, we provide the pseudo-code of meta-nml in Algorithm 2
C MORE EXPERIMENTAL RESULTS
C.1 FULL RESULTS OF THE MAIN SCRIPT
We included the full results of the main script in this section. The uncertainty quantification is visualized in Figure 9 the visualization of the trained f3 (s) is in Figure 10 the visualization of
18

---

Published as a conference paper at ICLR 2023
the proposed curriculum goals is in Figure II
We do not include the visualization results of the Moint-U-Maze, and Pawyer-Peg-Pick&&PPP results as these environments share the same map with the Ant Locomotion, and Sawyer-Peg-Puse environments, respectively.
Desired goo!
Explored state
gooud
U'ncertain
Explored
Dextred
CECC
Uncertaia state
Explored
Destred goul
Explored
Figure 9: Visualization of the uncertainty quantification along training progress. First row: U-Maze (Point, Ant Locomotion), Second row: N-Maze, Third row: Spiral-Maze, Fourth row: Sawyer Peg Push, Pick & Place environments.
High neward
Point
Goal Point
Initial Point
Initial Point
Goal Polnt
roward
(a) Ant Locomotion
(b) N-Maze
(c) Spiral-Maze
(d) Sawyer Push
Figure 10: Visualization of the trained f (s). High reward means temporally close to the desired outcome states (required timesteps are small to reach the goal), and low reward means the opposite (required timesteps are large to reach the goal).
19

---

Published as a conference paper at ICLR 2023
teidel
Fisal curricalur
hnittel curriculam
Final curricalur
huitdel curricufam
CCC
Riaal curricutur
Initia! curriculom
Final curricuiso
(a) OUTPACE(Ours)
(b) CURROT
(c) HGG
Figure 11: Visualization of the proposed curriculum goals. First row: Ant Locomotion (same map size with U-Maze), Second row: Point-N-Maze, Third row: Point-Spiral-Maze, Fourth row: Sawyer Peg Push.
20

---

Published as a conference paper at ICLR 2023
C.2 EVALUATION WITH THE GOALS SAMPLED FROM THE UNIFORM DISTRIBUTION
In some curriculum RI. algorithms, they incorporate mechanism for remembering previously prac- ticed curricula either implicitly or explicitly. For example, GoalGAN (Florensa et al. 2018) mixes the previously generated goals with currently generated goals in specified ratio (c.g. 20 % of pre- viously used goals), and SkewFit (Pong et al. 2019) set the objective as targeting the uniform goal distribution (by maximizing the entropy of the goals H(g)), and CURROT (Klink et al. 2022 also utilizes uniform target curriculum distribution in practice, and so do some of the other baselines. Due to these algorithmic designs, they require many iterations for explicitly practicing previously used curriculum goals, show slow progress of curriculum to match the uniform target distribution.
In contrast, our method is based on an intrinsic reward, which shaped according the required timesteps proportional values fī(s) (a) as described in our main script. Thus, method does not need to explicitly consider the non-forgetiing mechanism when we design the algorithm because the reward is already shaped with respect to the timesteps for reaching the arbitrary goal points along the trajectory that reaches the desired outcome state. Because of this property, our method does not consider uniform target distribution or explicitly mixing the previously practiced curriculum goals, and it enables our method to be much faster and show sample-effictis curriculum progress.
We experimented with the goals sampled from the uniform distribution on the feasible state space for validating the previous hypothesis, and the experimental results are shown in Figure 12 Even though our method does not explicitly consider the previously practiced curriculum goals, it shows success in reaching arbitrary goal points sampled from the uniform distribution. Performance degrades observed in sawyer manipulation environments and these are because we set the uniform distribution areas within the tables in the environment. But the currieulum goals proposed by our method converged before the agent explores the entire state space on the table, thus the agent does not have the opportunity to practice the goals from the entire state space.
STRDS OK)
SCRO6 04)
Staps 04)
(a) U-Maze
(b) N-Maze
(c) Spiral-Maze
1.0
1.0
0.8
00
2.00
2022
382
2003
.5.00
120
O2)
(d) Ant Locomotion
(c) Sawyer Push
(f) Sawyer Pick&Place
(UTPACE(ours)
SkewFit ALP-GMM
CURROT
PLR VDS
HGG
GoalGAN
Figure 12: Episode success rates of the evaluation results with the goals uniformly sampled from the feasible state space.
C.3 ABLATION STUDY
We conducted ablation studies described in our main script in all environments. Figure 13 shows the average distance from the proposed curriculum goals to the desired final goal states along the training steps, and Figure [4] shows the episode success rates along the training steps. As we can in these figures, even though there are not many differences in some environments with simple
21

---

Published as a conference paper at ICLR 2023
dynamics, could obtain consistent analysis with the results in the main script in most of the environments.
We also visualized the curriculum goals obtained by each ablation study training proceeds in Fig- ure T5 As we can see these figures, the curriculum proposal only based on the uncertainty (only- enml) shows progress toward uncertain areas, but it shows unstable curriculum progress, which is depicted by separated curriculum goals despite the same colors or out of order with respect to human intuitive optimal curriculum progress. This is because the meta-learning-basetr inference procedure of PONML. has some numerical errors that could lead to the wrong prediction of the states near the boundaries of the already explored regions as uncertain areas (For example, in Figure there exist some states predicted as uncertain area despite already explored region).
Also, even though the curriculum proposal based on the temporal distance only obtained by fz (only-f) shows some progress that deviates from the initial states, it still has difficulty in most of the environments because Iz is trained to reflect the observed transition data rather than entire state space. That is, the temporal distance estimation is reliable within the explored region. Once the trained f has local optimum before discovering temporally more distant states (Figure T6), the proposed curriculum goals can be stuck in some areas that wrongly predicted to be the most far from the initial states in terms of the temporal distance, and it leads to the ineffective exploration of the agent and recurrent failures.
22

---

Published as a conference paper at ICLR 2023
OUTHACE MAVIE pel.
OUTSACEECUnO Rlin 200 sefeemori stinn GAN
OUTRACE
MW
Adbusce only 2014
aetence
ditaanc
OUTTAIC Flnaro sefhoadd stion, SAN
135
Iolutce only 2011
13.0
distance
distance
20.0
310
Aduace
125
12.
CUT (cctoorr)
disance
disance
disance
FanarO
Fjpuro
seffooad
obttance
OUTTACEE CARSS
OUTTACEICanO sTRO veleward
only
asin
06
1003
20:01 20:33 BC2
20:00 20:01
2150
C4,02 20:00
20:00
12:0
(a) Cost type
(b) Reward Goal proposition type
(c) Effect of in Iggiiaanc
Figure 13: Ablation study terms of the distance from the proposed curriculum goals the desired final goal states. First row: Point-U-Maze. Second row: Point-N-Maze. Third row: Point-Spiral- Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick&Place. Shading indicates a standard deviation across 5 seeds.
23

---

Published as a conference paper at ICLR 2023
1.0
28
Adkesce 3nly.3
stin reflooordd
Vabvice yely 2071
arion JAN
0.4
steds [k]
steps 0K)
steps (8)
3.
CUTTACCFOAOO alon bi0a scffora)) stion
0.0
Ablution only cord
1.
2.
SUccemoo pate
000
stops (K]
P33
1983
5.0
700
4233
stops
1339
stops OK)
1331
OUTTA Fiour
.008
SBC Cecceeoo retee
000
2:00
830
1003
5.5
200
423
1300
204
1300
1.59
1.0
SULEC rete
0.4
ST605 [K]
steps
steps (8)
2.8
DUTPREE AWASS Adkioee J
CUT 113950
stin, Sww seRooord
iakatiee yeay mwmn
arion (0N
1550
1003
1530
2530
2003
15
500
3000
1543
2000
3000
3000
1249
2000
2540
2000
sTeos [K]
steps
steps (K)
siin seferadd
RUCCetoo.ooo rete
pate
1000
19000
7900
2540
1.000
(1)
(a) Cost type
(b) Reward Goal proposition type
(c) Effect of i Vggiidanc
Figure 14: Ablation study in terms of the episode success rates. First row: Point-U-Maze Second row: P-ine-N-Maze. Third row: Point-Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick&Place. Shading indicates standard deviation across seeds.
24

---

Published as a conference paper at ICLR 2023
hnitiel
2S2
curriculam
Fixal curricalurs
Initia! curriculam
Floal curricalur
curriculam
CCC
Final curricauur
Initia!
curriculam
Final curricuison
(a) OUTPACE(ours) Figure 15: Ablation study in terms of curriculum goals visualization. First row: Ant Locomotion,
(b) Ablation: only-cnml
(c) Ablation: only-f
Second row: Point-N-Maze, Third row: Point-Spiral-Maze. Fourth row: Sawyer Manipulation.
L
Temporally
Inttial Potnt
far
Vnttial Potnt
Initial Point
Initial Point
Temponatiy close
(a) Ant Locomotion
(b) N-Maze
(c) Spiral-Maze
(d) Sawyer Push
Figure 16: Visualization the not properly trained fg
when ablation study only-f is performed.
25


Reviews:
[{"review_id": "t96dix97au", "reviewer": "Reviewer_KAqi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a curriculum learning framework which generates training goals and demonstrates empirical sample efficiency compared to existing method, on a range of navigation and manipulation tasks. ", "summary_of_the_review": "The method is overall well-motivated by desired properties of intrinsic reward and desired distribution of training goals, with strong empirical performance compared to prior works. It also offers descent amount of ablations and visualizations for further explanations. ", "strength_and_weaknesses": "Strengths\n* The method is well-motivated to obtain three desired properties: temporal- and uncertainty-awareness, and closeness to the desired outcome distribution. These properties are empirically verified by experiments. \n* The method provides a novel bipartite-matching-based algorithm as an instantiation of the goal sampling process. \n* It is empirically demonstrated that the proposed method achieves better asymptotic performance and sample efficiency compared to prior methods. \n\nWeaknesses\n* On Spiral-Maze and Ant Locomotion environments, none of the baseline methods are able to have non-zero success rate. The paper does not offer an explanation what part of the proposed algorithm results in such performance discrepancy. Is the discrepancy caused by particular attributes of the environments, and are these attributes taken care of by specific components of the proposed method? A further ablation study of the method design on these environments would provide more intuition. \n\nClarification questions\n* In Figure 3, HGG has its final curriculum centered in a region which is far from the designed goal for U-Maze, while still having a good performance. Why is it the case? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "klvSy7GM8GA", "reviewer": "Reviewer_4feY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a method for proposing curriculum goals given a set of desired goal states for curriculum reinforcement learning. The core idea is to find states at the frontier that are similar to the specified goals. The proposed method keeps track of exploration frontiers using a notion of temporal distance and computes state similarity using an estimated wasserstein distance. Experiments in simulated control tasks show the proposed method outperform baselines.\n\n", "summary_of_the_review": "Interesting work on curriculum RL with seemingly strong result, but the presentation of the work can be hard to access.", "strength_and_weaknesses": "Strength:\n-Goal-based curriculum RL is a powerful paradigm of RL that finds policy based on a few examples of goal states by generating gradually challenging goals.\n-This work proposes a curriculum generation method that combines uncertainty measure and a temporal-distance function to explore the frontiers of currently explored states. \nThe intrinsic reward formulation is intuitively and conceptually correct.\n-Experiments in simulated domains demonstrate OUTPACE does outperform other curriculum-RL approaches and the ablation study is extensive and thorough.\n\nWeakness: \n-The presentation of this work in general needs improvement. The language used is not formal and at the same time not easy to understand with grammatical errors. \n-A lot of the technical details are missing from the main text and deferred to the appendix or reference work, which makes understanding the technical contents really difficult. For example, how did you make the CNML estimation tractable? How long or how much longer does the proposed method require compared with baseline methods? How much did the meta learning part help?\n-I also have a hard time understanding what the hyperparameter c represents (how does it adjust the preference?) and how bipartite matching actually applies here. A diagram/visual of different moving parts of the algorithm would be super helpful.\n-More explanation and analysis of the results could be very useful for understanding the performance of the proposed method. In figure5, it seems a lot of baseline data are missing in many task domains but it is unclear whether there is no data or the success rates are zeros for all of them?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WGF5oe08d60", "reviewer": "Reviewer_J8za", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an uncertainty & temporal distance-aware curriculum goal-generation method for various navigation tasks and a manipulation task. Uncertainty quantification is based on a Bayesian classifier obtained via conditional normalized maximum likelihood (Zhou & Levine, 2021; Li et al., 2021), which guides the curriculum into unexplored regions. The temporal distance is represented using a Wasserstein distance with a time-step metric (Durugkar et al., 2021b) to provide intrinsic reward and query the frontier of the explored region. The proposed method is compared against a variety of existing approaches. ", "summary_of_the_review": "Overall, the presented approach is novel and presents thorough comparative and ablative studies. However, it is suggested that authors comment on the generalization capacity of their proposed method across novel settings and of manipulation under collision and other task constraints. ", "strength_and_weaknesses": "Strengths:\n+ Novel approach for curriculum goal generation\n+ Detailed experiments with strong baseline methods\n+ Ablation study demonstrating the role of both uncertainty and temporal distance on performance.\n\nWeaknesses:\n- No evaluation or discussion on the generalization capacity of the proposed method across different environments with unseen start and goals.\n- Does the proposed method scale to Sawyer Pick/Place task in environments with obstacles?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%