Diversify \& Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
Authors: Daesol Cho
Seungjae Lee
H. Jin Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Curriculum learning
Out-of-distribution disagreement
Underspecification
Outcome-directed RL

Abstract: Reinforcement learning (RL) often faces the challenges of uninformed search problems where the agent should explore without access to the domain knowledge such as characteristics of the environment or external rewards. To tackle these challenges, this work proposes a new approach for curriculum RL called $\textbf{D}$iversify for $\textbf{D}$isagreement \& $\textbf{C}$onquer ($\textbf{D2C}$). Unlike previous curriculum learning methods, D2C requires only a few examples of desired outcomes and works in any environment, regardless of its geometry or the distribution of the desired outcome examples. The proposed method performs diversification of the goal-conditional classifiers to identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in a simple and intuitive way. The proposed method then employs bipartite matching to define a curriculum learning objective that produces a sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region. We present experimental results demonstrating that D2C outperforms prior curriculum RL methods in both quantitative and qualitative aspects, even with the arbitrarily distributed desired outcome examples.

Diversify & Conquer: Outcome-directed Curriculum
RL via Out-of-Distribution Disagreement
Daesol Cho, Seungjae Lee, and H. Jin Kim Seoul National University
Automation and Systems Research Institute (ASRI)
Artificial Intelligence Institute of Seoul National University (AIIS) dscho Rscho1234@ens 12342nuu ac.kr, kr, ysz0301 .e..e.enuuu..................... hjinkim@snu ac
Abstract
Reinforcement learning (RL) often faces the challenges uninformed search problems where the agent should explore without access to the domain knowledge such as characteristics of the environment or external rewards. To tackle these challenges, this work proposes new approach for curriculum RL called Diversify for Disagreement & Conquer (D2C). Unlike previous curriculum learning methods, D2C requires only few examples of desired outcomes and works in any environ- ment, regardless its geometry or the distribution of the desired outcome examples. The proposed method performs diversification of the goal-conditional classifiers identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in simple and intuitive way. The proposed method then employs bipartite matching to define curriculum learning objective that produces sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region. We present experimental results demonstrat- ing that D2C outperforms prior curriculum RI. methods in both quantitative and qualitative aspects, even with the arbitrarily distributed desired outcome examples.
Introduction
Reinforcement learning (RL) has great potential for the automated learning of behaviors, but the process learning individual useful behavior can be time-consuming due the significant amount of experience required for the agent. Furthermore, in its general usage, RL often involves solving challenging uninformed search problem, where informative rewards or desired behaviors are rarely observed. While there are some techniques that can alleviate the exploration burden such as reward- shaping [30] or preference- based reward [2] 6), they often require significant domain knowledge or human intervention. This makes it difficult to utilize RL directly for problems that require challenging exploration, especially in many domains practical significance. Thus, becoming increasingly crucial to tackle these challenges from the algorithmic level by developing RL agents that can learn autonomously with minimal supervision.
One potential approach is curriculum learning algorithm. It involves proposing carefully designed sequence of curriculum tasks or goals for the agent to accomplish, with each step building upon the previous one to gradually progress the curriculum. By doing so, enables the agent to automatically explore the environment and improve the capability a structured way. Previous studies primarily involve mechanism to adjust the curriculum distribution by maximizing its entropy within the explored region [35), or taking into account the difficulty level [43] [1], or learning progress of the agent [36]. However, efficient desired
exploration is not available under these
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

UiVateled Singet 5313
Curriculum
states
proposal
(abeld)) ())
Dauuree
Desined
UtVabeled target da13
outcome
Diversify with muttppl claszifiers
Quart uneopbored areas
layers
disgrcomen
Dytadd
Unoolrat
Dolndd
Durget
C.eerokeey gook
Agent
Stati space
Figure 1:
D2C trains set of classifiers with labeled source data while diversifying their outputs on unlabeled target data (red: predicted label O, blue: predicted label 1). Then, it proposes curriculum goals based on the diversified classifier's disagreement and similarity-to-dssii outcome.
frameworks as these approaches do not have converging curriculum objectives, resulting in a naive search for unseen states.
Providing the agent with examples of successful outcomes can make the RL problem more manage- able, as opposed to letting the agent explore without particular objective and with an undirected reward. Such examples can offer considerable guidance on how to achieve a task if the agent can estimate the similarities between the desired outcome examples and visited states. In the realm of cutcome-directed
desired outcome- directed RL, previous studies try to maximize the probability of reaching desired outcomes
8. However, these approaches do not have tools for explicitly quantifying an unexplored region, leading the agent to settle for reaching an initially discovered desired outcome example, rather than continuing to explore further find another example. Other works try to mini- mize the distance between the generated curriculum distribution and the desired outcome distribution to propose intermediate task goals [37 18 But, these approaches have mainly been confined to problems that do not involve significant exploration challenges because they rely on the assumption that the Euclidean distance metric can represent the geodesic interpolation between distributions. This assumption is not universally applicable to all environmental geometries, making these methods less versatile than desired.
Therefore, it is necessary to develop an algorithm that enables the agent to automatically perform the outcome-directen exploration by generating a sequence of curriculum goals, which can be applied to arbitrary geometry and distribution of the given desired outcome examples. To do so, we propose Diversify for Disagreement & Conquer (D2C), which only requires desired outcome examples and does not require prior domain knowledge such as 1) the geometry of the environment 2) or the number of modes or distribution of the desired outcomes, or 3) external reward from the environment. Specifically, D2C involves diversifying the goal-conditioned classifiers to identify the similarities between the visited states and the desired outcome states. This is accomplished by ensuring that the outputs of each classifier disagree on unseen states, which not only allows determining the unexplored frontier region but also provides an arbitrary coal-conditioned intrinsic reward. Based on such conditional classifiers, propose to employ bipartite matching to define straightforward and
curriculum learning objective, which produces a range of well-adjusted curriculum goals that interpolate between the initial state distribution and arbitrarily distributed desired outcome states for enabling the agent to conquer the unexplored region.
To sum up, our work makes the following key contributions.
We propose new outcome-directed curriculum RL method that only needs a few arbitrarily distributed desired outcome examples and eliminates the need for external reward.
To the best of our knowledge, D2C is the first algorithm for curriculum RL that allows for automatic progress in any environment without being restricted by its geometry the desired outcome distribution by proposing diversified conditional classifiers.
In various goal-conditioned RI. experiments, our method consistently outperforms the previous curriculum RI methods through precisely calibrated guidance toward the desired outcome states in both quantitative and qualitative aspects.

---

2
Related Works
Despite various attempts to improve exploration in RL, remains a difficult issue that has not been fully solved. Some previous works for exploration have suggested methods based information
theoretical approaches ( 39 50 20, 17 27], maximizing the state visitation distribution 's entropy
47 25 26], counting the state visitation K 3I), utilizing similarity curiosity [3 45), and
quantifying uncertainty through prediction models 34 Other approaches provide a curriculum
to allow the agent to explore the environment through intermediate tasks. Curricula are usually created by adjusting the distribution of goals to cover new and unexplored areas. It is achieved by considering an auxiliary objective such entropy [35] or disagreement between the model ensembles
[49 15 28 or difficulty level of the curriculum TT 4 regret []]. and learning progress [36]. However, these methods only focus on visiting diverse frontier states or do not provide mechanism to converge toward the desired outcome distribution. In contrast, our method enables more efficient sutcome-directer exploration via curriculum proposal with an objective to converge rather than simply exploring various frontier states, only requiring few desired outcome examples.
Assuming access desired outcome samples or distribution, some prior methods try accomplish the desired outcome states by maximizing the probability of reaching these states 12 8. However, they lack a mechanism for quantifying an under-explored region and synthesizing the knowledge acquired from the agent's experiences into versatile policies that can accomplish novel test goals. Some algorithms generate curricula as an interpolation between the distribution of desired target tasks [8]. but they still rely on the Euclidean distance metric, which is insufficient to handle arbitrary geometric structures. There exists work that addresses geonetry-agnostic curriculum generation using the given desired outcome states, similar to our approach [5]. But, it requires carefully tuned Wasserstein distance estimation that depends on the optimality of the agent. It leads to inconsistent estimation before the convergence, resulting in numerically unstable training, while our method does not have such dependence. Also, it adopts a meta-learring--ase based technique 10 23 that requires gradient computation at every optimization iteration, while our method only requires a single neural network inference, leading to much faster curriculum optimization.
and auxiliary tasks [37]
A core idea behind our method is utilizing classifiers to learn a diverse set of hypotheses that minimize the loss on source inputs but make differing predictions on target inputs 32 22 It is related to ensemble methods 77 19 14] that aggregate the multiple functions' predictions, but the proposed method is distinet in terms of directly optimizing on an underspecified target dataset for enhancing diversity. Even though this diversifying strategy for target data is typically considered from the perspective of dfitt--diitiiuutinn robustness conditions of distribution shift 29 24 38
or domain adaptation 44 46 42). we demonstrate how it can be applied for classifiers to quantify the similarity between the visited states and desired outcome states, and discuss its links to exploration and curriculum generation in RI.
Preliminary
We consider the Markov decision process (MDP) M AS.G. where indicates the state space, G the goal space, A the action space, P(s'ls, a) the transition dynamics, and ? the discount factor. In our framework, the MDP not provided a reward function and we consider a setting where only the desired outcome examples {9, 1k= from the desired outcome distribution p*()) are given. Thus, our method utilizes an intrinsic reward r: : S x G x A -> R. Also, we represent the curriculum distribution obtained by our method as (s))
3.1 Acquiring knowledge from underspecified data
For diversification-bssed curriculum RL, we train a classification model y f(x) in a supervised
learning setting where x E X' is input, and y E y is the corresponding label. The model f is trained with a labeled source dataset Ds ps (zz y) that includes both the given desired outcome examples (y = 1) and the visited states in the replay buffer B of the RL agent (y = 0). We assume that the desired outcome distribution can be modeled as a mixture of outcome distribution, pt (z) = Eoec (x). where each o E O corresponds to a specific outcome distribution (z)). The selection of model from hypothesis class f E F is achieved by minimizing the predictive risk Eps(z.)) (C(f [x)) v)]. where L is a standard classification loss.
3

---

Although f demonstrates good generalization on previously unseen data acquired from the source distribution (z(2, y), it ambiguous to evaluate the model f in distribution shift conditions such as when querying target data obtained from an out-of-distribution (e.g. a state lies in unexplored region). This is because there could be many potential models f that can minimize the predictive risk. To formalize this intuition, we introduce the concept of an e-optimal set defined as F == {f E F (f) < }} (22). where Cp represents the risk associated with a distribution p. and 2 0.
The definition of the s-optimal set implies that the predictions of any two models f1, 12 in the set are almost identical on ps y) when is small. But, using FF as it is has a few drawbacks: 1) there is no criterion to prefer any particular hypotheses of F over another, and 2) the trained model f might not be suitable for evaluating the target data distribution as does not have a mechanism to quantitatively discriminate unseen target data from the labeled source data.
To address this point, we utilize unlabeled target dataset DT ~ pr():) for comparing the functions within FF by examining how their predictions differ on Dr. Since Dr can be viewed as indicating the directions of functional change that are most crucial to the automatic exploration of the RL agent, we set pT( as the uniform distribution between the lower and upper bound of the state space to include all possible candidate states visit. Knowing the state space's 's bounds is commonly utilized assumption in many curriculum learning works [37 18 3 and it does not require being aware of the dynamically feasible areas. Then, our objective is to identify a set of classification models that perform well in pa 3) and disagree in Pr (x( to recogmize the unseen state from the unexplored region by quantifying the similarity between the observed states in B and desired outcome examples.
Such function will lie within F of ps, y), and the model leverages DDr to identify diverse set of functions within the near-optimal set.
Method
For automatic exploration toward the desired outcome distribution via calibrated guidance of the curriculum, the proposed D2C suggests curriculum goals via diversified classifiers that disagree in the unexplored region and enables the agent to conquer this area by exploring through the goal- conditioned shaped intrinsic reward. It allows the agent to make progress without any prior domain knowledge of the environment such obstacles distribution of the desired outcome states and advances the curriculum towards the desired outcome distribution p (g).
4.1 Diversification for disagreement on underspecified data
For the automatic exploration of the RL agent with the curriculum proposal, quantification of whether a queried state is already explored or not is required. To obtain such a quantification, we diversify a collection of classifier functions by comparing predictions for the target dataset while minimizing the training error as briefly described Section 3.T The intuition behind this is that diversifying predictions will produce functions that disagree with data in the ambiguous region 22
Specifically, we use Each head produces prediction for an input x represented as f((z). To ensure that every head has a low predictive risk on Ps y), we minimize the cross-entropy loss for each head using frxe (f+) = Br [CE (ft(z), Ideally, it is desirable for each function to rely on distinctive predictive features of the input to encourage differing predictions. Therefore, we train each pair of heads to generate statistically independent predictions, which implies disagreement in predictions. It could be achieved by minimizing the mutual information between each pair of heads:
neural network with N heads to train diverse multiple functions.
CMI (f..f) EANDY [DKL ff (f(())) @ (f)()
(1)
where the input data is obtained from target dataset Dr. For implementation, we compute empirical estimates of the joint distribution p (f) (x)) and the product of the marginal distributions p (f)) @ p (z)), which can be computed using libraries developed for deep learning (22).
In summary, the overall objective for classifier diversification is represented with a hyperparameter A:
E (ft) + X Σ Cxu (fr
(2)
itj

---

(2) Non-conditional
Conditional
$1
gi
S1
.Cunicuuum candidate
*
$
Agent
Desired outcome
$5
Explored
Unesppee
$3.
* $g.
53.54
02
92
(1)
$1
92
91
54
%
%/
9/
A5
81
0.5
0.4
91
0.4
82
0.5
92
0.9
Select S3.84
Select $3.53
Figure 2: Overview of the curriculum proposal when we select two candidates for curriculum goals of 9, by bipartite matching. (1) The curriculum goals are proposed according to the quantification of the similarity-to-desine in the conditional form, it cannot distinguish the source of the desired outcome example g." resulting in collapsed curriculum proposals, (3) while the conditional classifier enables non-collapsed curriculum proposals even when the agent achieves a specific desired outcome (e.g. 92 first.
outcome (s; with a high probability will be selected). (2) If the classifier
4.2 Quantifying unexplored regions by conditional classifiers
From this section, slightly abuse the notation s instead of æ for the RL setting. Considering the diversification process in Section 4.1 we can quantify how much a queried state is similar to the desired outcome example from p* (g). Specifically, we can define a pseudo probability of the queried point by averaging predictions of each head:
N
Ppscudo (yr = 1(8) =
N Σ f((s)
When the queried state is in proximity to the data within (with the desired outcome example labeled and states from the replay buffer B labeled as 0), becomes challenging for the classifiers to give a high probability to labels that substantially differ from the neighboring data, since each classifier f; is trained minimize the loss for the source data. On the other hand, if the queried state is significantly different from the data in ps. each prediction head will output different values since the classifiers are trained to diversify the predictions on the target data (uniform distribution on the state space) to make their prediction values disagree much as possible. For example, in the case of two heads, one predicts and the other predicts for same queried target data, resulting in pseudo probability of 0.5. This could be interpreted as quantification of how much disagreement exists between classifiers, or uncertainty the queried data, which can be utilized as an unexplored region-aware classification (Figure
Thus, can consider a curriculum learning objective for interpolating the curriculum distribution from the initial state distribution to p (g) represented by the following cross-entropy loss:
Leury Eg-B. (g) CC (Ppseudo (y 1|s): Ppseudo (y/ 1lg*)))
(4)
Intuitively, before discovering the desired outcome examples in p (g), the curriculum goal candidate B that minimizes Eq is proposed in the frontier of the explored regions where classifiers fi disagree. And, as the agent explores and discovers the desired outcome examples, the curriculum candidate is updated to converge to p* (g) minimize the discrepancy between the predicted labels of g* and s.
However, it is not applicable for a case when the desired outcome examples are spread over multi- modal distribution or arbitrarily because the trained classifiers fi not distinguish which pa the desired outcome example is obtained from (Figure 2). In other words, fa will predict value close to
5

---

1 for any desired outcome example g", and the loss in Eq 4 will be close to 0. As the curriculum goal candidates are obtained from the replay buffer B, the curriculum optimization may collapse the agent achieves one of the desired outcome distributions (px) earlier, which is not desirable for achieving all the desired outcome examples regardless of its distribution.
Since we assume that we do not know which Po the desired outcome example is obtained from, nor the number of modes (o) of the desired outcome distributions, we propose to utilize a conditional classifier to address this point while satisfying the assumption. Specifically, we define goal-conditioned classifiers, where each classifier takes input s and conditioned on g. and these classifiers are trained to minimize the following modified objective of Eq (
00] IE, Efaen (Filg + tegg), =
Bg
Es-B
Σ Exem (Ji(s:9),V=
(5)
+ A.
Σ Cau (1(66))))
where is small noise (from uniform distribution around zero or standard normal distribution with a small variance) for numerical stability, and DG can be either Pr B Upppp (g) for training arbitrary goal-conditioned & unexplored region-aware classifiers. We found that there is no significant difference between these choices. Then, the pseudo probability can be represented as Ppseudu (y =
1 ||s; == #; E [((x;,) g) and we can address the arbitrarily distributed desired outcome examples quantification is [5] Because
without curriculum collapse (Figure 2 This proposed conditional classifier- one of the key differences from the previous similar outcome-diredd the previous works require computing gradients of thousands of data for meta-learning-beee based network inference, while our method only requires single feedforward inference without backpropagation which leads fast computation.
RL methods [23]
4.3 Curriculum optimization via bipartite matching
As we assume that we have access to desired outcome examples from (=(g)) instead of their explicit distribution, we approximate it using the sampled formulated by the combinatorial setting that requires finding the curriculum goal candidate set (s) that will be assigned to each sample of g (g), and it can be solved via bipartite matching. With curriculum goal candidates and desired outcome examples, the curriculum learning objective is represented follows:
(g) (p² (g) K). Then, the problem
min (+):lp)))))
Σ
w(s..(")
(6)
(.) (x+
w.s,, 9. = (p (y = 1|s y:y Ppsoudo (11g::9t))
(7)
The intuition behind this objective is similar to Eq (4). but it is different in terms of considering conditional quantification, which enables addressing arbitrarily distributed desired outcome examples. Then, we can create a bipartite graph G with edge costs w by considering the sets of nodes V. and Vz, which represent achieved states in replay buffer B and p (g), respectively. We define the bipartite graph G((Va. Vo J. E) with edge weights E(...) matching problem, we employ the Minimum Cost Maximum Flow algorithm 37 to find K edges with the minimum cost w. The entire curriculum RL process is shown in Algorithm 2 in Appendix
-...)). To solve this bipartite
4.4 Conditional bassiffirr-bsed intrinsic reward
As we have trained conditional classifiers and defined the pseudo probability, we can additionally use this value as shaped intrinsic reward for enabling the agent to solve the uninformed search problem. As the pseudo probability outputs for the state in the replay buffer B, for the desired outcome state, and value between 0 and 1 for the state where the classifiers disagree (meaning unexplored region), we can define the goal-conditioned intrinsic reward as r = Ppscudo (y = ()s We use this intrinsic reward in all of our experiments. The ablation study for this reward is detailed in Section 5.2

---

Table 1: Comparison of our work with the prior curriculum RL methods in conceptual aspects.
guantifiation lastet curticllum Arbitrary desared outcomse
Gooenetry agnostic
reward
HGKI
(g)
CURROT
(g)
PLR
VIS
P-GMM
PUTPACE
P (
Ours
curricufee
orricauu
buitel turritufam
curricuture
(a) Ours
(b) OUTPACE
(c) HGG
Figure Curriculum goal visualization of the proposed method and baselines. First row: Ant Locomotion, Second row: Spiral-Maze.
5
Experiment
We conduct experiments on 6 environments that have multi-modal desired outcome distribution to validate our proposed method. We various maze environments (Point Complex-Maze Medium- Maze, Spiral-Maze) to validate our curriculum proposal capability, which is not limited to specific geometries. Additionally, evaluate our method on complex dynamics other domains such as the Ant-Locomotion and Sawyer-Peg Push, Pick&Place with obstacle environments to demonstrate its effectiveness in domains beyond the navigation. (Refer Appendix D for more details.)
We compare our method with several previous curriculum generation approaches, each of which possesses the following characteristics. OUTPACE (5) prioritizes goals that considered uncertain and temporally distant from the initial state distribution through meta-learning-bart -based uncertainty quantification and pnneerstein-distancee
temporal distance approximation. HGG (37] aims to reduce the distance between the desired outcome state and curriculum distributions, using a value function bias and the Euclidean distance metric. CURROT (18) interpolates between the desired outcome state and curriculum distribution while considering the agent's 's current capability using the Wasserstein distance. VDS 40 proposes epistemic uncertainty-based goals by utilizing the value function ensembles. ALP-GMM [36] models absolute learning progress score by a GMM. PLR [[6] prioritizes increasingly challenging tasks by ranking task levels based on TD errors. We summarize the conceptual comparison between our method and baselines in Table I
5.1
Experimental results
First, to validate the quality of curriculum goal interpolation from the initial state distribution the desired outcome distribution, we qualitatively and quantitatively evaluate the curriculum progress achieved during the training by optimizing Eq (6). For qualitative evaluation,
visualize the
curriculum goals obtained by the proposed method and other baselines (Figure 3). The proposed method shows calibrated guidance toward the desired outcome states, even with the multi-modal distribution. But, HGG shows an inappropriate curriculum proposal due to the Euclidean distance metric, and OUTPACE shows curriculum goals collapsed only toward a single direction because utilizes the Wasserstein distance for biasing curriculum goals into the temporally distant region. Once the agent explores temporally far region in a specific direction earlier, the curriculum proposal collapsed toward this area. In contrast, our method does not have such dependence, which enables

---

diatence
cintance
50000 00
5C009 [k]
(a) Complex-Mo Maze
(b) Medium- Maze
(c) Spiral-Maze
3000
1906
3003
3023
2403
2003
2400
5000
2000
2400
(d) Ant Locomotion
Sawyer Push
Sawyer Plac&
Ours
OUTPACE
CURROT
VD5
The mean distance between the curriculum goals and final goals (Lower is better). The shaded area represents standard deviation across 5 seeds. The increases of our method initial steps in some environments are attributed to the geometry of the environments.
Figure 4:
930000
0.
2005
1230
1200
200 2000
2033 (k)
Da0 205
64
20
282
S0006
(a) Complex- Maze
Mazze
(b) Medium- Mazze
(c) Spiral-Maze
540000 268
(f) Sawyer Pick&Place Figure 5: Evaluation success rates, using the same seeds and legends as shown in Figure 4 Note that some baselines are not visible as they coincide with success rate of zero.
(d) Ant Locomotion
(e) Sawyer Push
our method to always propose curriculum goals in any uncertain direction quantified by disagreement between the diversified classifiers.
We also plot the average distance from the proposed curriculum goals to the desired outcome states for quantitative evaluation (4) As evaluated with multi-modal desired outcome distribution, the distance cannot be measured by naively averaging the distance between the randomly chosen desired outcome states and curriculum goals. Thus, we measure the distance by bipartite matching with the la distance metric, which computes the distance between the desired outcome states and their assigned curriculum goals. As shown Figure 4 only the proposed method consistently shows superior interpolation results from the initial state distribution to desired outcome distribution, while other baselines show some progress only a simple geometric structure due the Euclidean distance metric, or get stuck in some local optimum area due to the lack of or insufficient unexplored region quantification. We also plot the performance of the outcome-directed RL Figure As expected by the average distance measurement in Figure 4 the proposed method is the only one that consistently and quickly achieves the desired outcome states through the guidance of calibrated curriculum goals, indicating the advantage of the proposed diversified conditional classifiers.

---

Dare
SACCCHO0 retee
250
750
1330
1250
1906
1956
2000
250
500
150
D003
1353
1599
1790
3500
280
250
509
1002
1350
1599
1750
1500
3.0
headed
Dara Camiodlu
SU0C0000 petee
23002 (k]
staps
stops 00)
(a) Auxiliary loss weight X
(b) Number of prediction heads
(c) Intrinsic reward & Curriculum
Figure 6: Ablation study in terms of the episode success rates. First row: Spiral-Maze. Second row:
Sawyer Push. The shaded area represents the standard deviation across seeds.
5.2 Ablation study
Number of prediction heads & Auxiliary loss weight A. To investigate the sensitivity to the hyperparameters of our method, we experimented with the different values of the auxiliary loss' weight À and with the different number of prediction heads of the conditional classifiers. As shown in Figure 6a 6b the results are slightly dependent on the environment's 's characteristics such as the presence of object interaction or complex geometry. However, the overall performance trend is not significantly affected by the number of prediction heads and X, except in the extreme case, which supports the superiority of our proposed method. More qualitativelquaniiativy analyses and other ablation studies are included in Appendix
Reward type & Curriculum. We evaluate the effectiveness of the proposed intrinsic reward and curriculum proposal by conducting two ablation experiments. First, we replace the intrinsic reward with sparse reward (Ours w/o Rewrinsi which typically used in goal-conditioned RL problems. Second, we conduct experiments without the curriculum proposal while utilizing the intrinsic reward (Ours w/o Curriculum). As shown in Figure bcc| there is performance degradation without using the proposed intrinsic reward in an environment with complex dynamics since an informative reward signal is crucial. In addition, the absence of curriculum proposal leads to performance degradation in an environment with complex geometry as it requires carefully crafted guidance for exploration. These results highlight the importance of both proposed components, i.e. intrinsic reward and curriculum proposal.
6 Conclusion
We propose D2C that 1) performs a classifier diversification process to distinguish the unexplored region from the explored area and desired outcome example and conquers the unexplored region by proposing the curriculum goal and the shaped intrinsic reward. It enables the agent to automati- cally progress toward the desired outcome states without prior knowledge of the environment. We demonstrate that our method outperforms the previous methods in terms of sample efficiency and seterrd--ttoom---iiiie diie--strrbutiooggnotiie curriculum progress, both quantitatively
woometry-agnoster and qualitatively.
Limitation & Broader impacts. Despite the promising results, there is limitation in the scalability of the proposed method since we use small noise to augment the conditioned goal for numerical stability, making it difficult to scale to high-dimensional inputs such as images. Thus, addressing this point would be an interesting future research direction to develop a more generally applicable method. Also, our work is subject to the potential negative societal impacts of RI., but we do not expect to encounter any additional negative impacts specific to this work.

---

7 Acknowledgement
This work was supported by Korea Research Institute for defense Technology Planning and ad- vancement (KRIT) Grant funded by Defense Acquisition Program Administration(AAPM) (No. KRIT-CT-23-0033 Development of Al researchers based on deep reinforcement learning and estab- lishment of virtual combat experiment environment).
10

---

References
[1] R Ahuja, TL Magnanti, and B Orlin. Network Flows: Theory, Algorithms, and Applications. Prentice Hall, Englewood Cliffs, NJ, edition, 1993.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv:1707.01495 2017.
[3] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810. 2018.
[5] Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directee reinforcement learning by uncertainty temporal distance-awarc curriculum goal generation. arXiv preprint arXiv:2301.1174, 2023.
[6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep rein- forcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[7] Thomas G Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Italy, June 21-23, 2000 Proceedings pages 1-15. Springer, 2000.
[8] Ben Eysenbach. Sergey Levine, Russ Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification. Advances in Neural Information Processing Syxtems, 34: 11541- 11552, 2021.
[9] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint
2018.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pages 1 1126-1135, PMLR, 2017.
[11] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for rein- forcement learning agents. International conference on machine learning, pages 1515-1528. PMLR, 2018.
[12] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang. and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31, 2018.
[13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMIRR, 2018
[14] Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on puttern analysis and machine intelligence, 12(10)9993-001, 1990.
[15] Edward S Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. arXiv preprint 2xxv:2303.13002 2023.
[16] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pages 4940-4950. PMLR, 2021.
[17] Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. arXiv preprint arXiv:2106. 14305, 2021.
[18] Pascal Klink, Haoyi Yang. Carlo D' D'Ermoo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning, pages 11341-11358, PMLR, 2022.
[19] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.

---

[20] Michael Laskin. Hao Liu, Xue Bin Peng, Denis Yarats. Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
[21]
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Fecdback-effiientt interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXi::2106.0000) 2021.
[22] Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data, arXiv preprint arXiv:2202.00 034188 2022.
[23] Kevin Li, Abhishck Gupta, Ashwin Reddy, Vitchyr Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. In Interna- tiomal Conference on Machine Learning, pages 6346-6356. PMLR, 2021.
[24] Evan Z Liu, Behzad Haghgoo. Annie S Chen, Aditi Raghunathan. Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference Machine Learning, pages 6781-6792. PMLR, 2021.
[25] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6736-6747, PMLR, 2021.
[26] Hao Liu and Picter Abbecl. Behavior from the void: Unsupervised active pre- pre-training. Advances in Neural Information Processing Systems, 34:1 18499-88473, 2021.
[27] Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreographer: Learning and adapting skills imagination. arXiv preprint arXiv:2211.33355 2022.
[28] Russell Mendonca, Oleh Rybkin, Kostas Danilidiss, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models, Advonces in Neural Information Processing Systems, 34:24379-24391, 2021.
[29] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased elassifier. Advances in Neural Information Processing Systems, 33:20673-20684, 2020.
[30] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In leml, volume 99, pages 278 287, Citeseer, 1999.
[31] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. International conference on machine learning, pages 2721-2730. PMLR, 2017.
[32] Matteo Pagliardini. Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. arXiv preprint axXiv:2202.04414. 2022.
[33] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[34] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagroement. In International conference on machine learning, pages 5062-500 -0711. PMLR, 2019.
[35] Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.08:: 03698, 2019.
[36] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pages 835-853. PM.R, 2020.
[37] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32, 2019.
[38] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 2019.
[39] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 1907.01677 2019.
[40] Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Benchmarking and formalism. arXiv preprint 222.021 2021.
12

---

[41] Avi Singh. Larry Yang. Kristian Hartikainen, Chelsea Finn. and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering, arXiv preprini arXiv:1904.07854, 2019.
[42] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608 608, 2020.
[43] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula asymmetric self-play arXiv preprint arXiv:1703.0555
2017.
[44] Baochen Sun, Jiashi Feng, Kate Saenko. Correlation alignment for unsupervised domain adaptation. Domain adaptation in computer vision applicaations, pages 153-171, 2017.
[45] David Farde-Farley, Tom Van Wiele, Tejas Kulkarni. Catalin lonescu. Steven Hansen, and Volodymyr Mnih. Unsupervised control through non parametric discriminative rewards. arXiv preprint arXiv:1811.1139, 2018.
[46] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10696 2020.
[47] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforvement learning with prototypical representations. In International Conference Machine Learning, pages 11920-11931, PMLR, 2021.
[48] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-tast and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1 1100. PMLR, 2020.
[49] Yunzhi Zhang, Picter Abbeel, and Lerrel Pinto. Automatic curriculum Icarning through value disagreement. Advances Neural Information Processing Systems, 33:7648-766 7659, 2020.
[50] Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprint arXiv:2103.08/1 080077 2021.
13

---

A Training & Experiments details
A.1 Training details
Baselines.
The baseline curriculum RI. algorithms are trained as follows,
OUTPACE [5]
We follow the default setting in the original implementation from https
//github. com/ Jay http://oopapcccccccciaaaaa official
HGG [37 We follow the default setting in the original implementation from https
//github com/ Stilwell-Git htte/Hiiiniigggtnnnno (ooaa--eneeraion.
CURROT [8]
We follow the default setting in the original implementation from https
7/github.. com httmmpscclnk/ccrrro..
PLR [6] VDS 49 ALP-GMM 36
We follow the default setting implementation
from https: / thub. com pac a lklkkklkkkk.. /urkkrrutttt
with the sparse reward except for the OUTPACE which uses an intrinsic reward based on Wasserstein distance with a time-step metric.
D2C and all the baselines are trained by SAC [3]
Training details. We used NVIDIA A5000 GPU and AMD Ryzen Threadripper 3960X for training, and each experiment took about I~2 days for training. We used small noise from a uniform distribution with environment-speciin noise scale (Table for augmenting the conditioned goal in Eq (5). Also, we used the mapping 6(-) that abstracts the state space into goal space when we use the diversified conditional classifiers (i.e. ((((());)) g). For example, Φ6-) abstracts the proprioceptive states (c.g. zy position of the agent) in navigation tasks, and abstracts the object-centric states (c.g. zyz position of the object) in robotic manipulation tasks.
Table 2: Hyperparameters for D2C
critic hidden dim critic hidden depth
512
discount factor Y batch size
0.99
3
512
critic target
0.01
init temperature Oimitt of SAC
0.3
critic target update frequency
2
replay buffer B size learning rate for fa learning rate for Critic & Actor optimizer
3e6
actor hidden dim actor hidden depth
512
1e-3
3
le-4
actor update frequency
2
adam
Table 3: Default env env-spccific hyperparameters for D2C
Env name
of heads
X
update freq (step)
Ja # of iteration
max episode
per update
horizon
Complex-Maze
2
0.5
2000
16
100
Medium-Mazz
2
0.5
2000
16
100
Spiral-Maze
2
0.5
2000
16
100
Ant Locomotion
2
2
1.0
4500
16
300
Sawyer-Peg-Puse
2
0.025
3000
16
200
Sawyer-Peg-Pick&PPaee
2
0.025
3000
16
200
A.2 Environment details
Complex- The observation consists of the position, angle, velocity, and angular velocity of the 'point'. The action space consists of the velocity and angular velocity 'point'. The initial state of the agent is (0, 0] and the desired outcome states are obtained from the default goal points [8, 16), [-8, - 16), [16, - 1-6,88 The size the map is 36 x 36.
Medium-Maze: It is the same as the Complex-Maze environment except that the desired outcome states are obtained from the default goal points [16, 16), [-66, -16), [16, -16), [-16, 16).
14

---

ae
(a) Complex-Mazz
(b) Medium-Maze
(c) Spiral-Maze
S
(d) Ant Locomotion
(c) Sawyer Push
(f) Sawyer Pick&Place
Figure 7: Environments used for evaluation: yellow stars indicate the desired outcome examples. (a)-(c) the agent should navigate various maze environments with multi-modas desired outcome distribution. (d) the ant locomotion environment with multi-modal desired outcome distribution. (e) the robot has to push or pick place a peg to multi-modal desired locations while avoiding an obstacle at the center of the table.
Spiral-Maze: The observation space and actions space and initial state of the agent are the same in the Complex-Mazz environment. The desired outcome states are obtained from the default goal points [12, 16), |-12, -16]. The size of the map is 28 x 36.
Ant Locomotion: and joint angular velocity of the 'ant'. The action space consists of the torque applied on the rotor of the 'ant'. The initial state of the agent is [0. 0) and the desired outcome states are obtained from the default goal points [4, 8) [-4, 8). The size of the map is 12 x 20.
The observation consists the zyz position, æyz velocity, joint angle,
Sawyer-Peg-Pushee The observation consists of the zyz position of the end-effector, the object, and the gripper's state. The action space consists of the zyz position of the end-effector and gripper open/close control. The initial state of the object is [0.4, 0.8, 0.02] and the desired outcome states are obtained from the default goal points [-0.3, 0.4, 0.02], [-0.3, 0.8, 0.02], [0.4, 0.4, 0.02]. The wall is located at the center of the table. Thus, the robot arm should detour the wall to reach the desired goal states. We referred to the metaworld [48] and EARL [40 environments.
It is the same as the Sawyer-Peg-Pust environment ex- cept that the desired outcome states are obtained from the default goal points -0.3, 0.4, 0.2), [-0.3, 0.8, 0.2), [0.4, 0.4, 0.2]. and the wall is located at the center of the table, fully blocking a path for pushing. Thus, the robot arm should pick and move the object over the wall to reach the desired goal states.
15

---

B
Algorithm
Algorithm 1 Overview of D2C algorithm
1: Input: desired outcome examples p* (g). total training episodes N, Env, environment horizon
H. actor и, critic Q, replay buffer
2: for 1ter.tion=1,2....N do
3: ÿ" < sample K curriculum goals that minimize Eq (6) We refer to HGG 37 for solving the
bipartite matching problem.
4: for i=1,2....KK do
5: Env.reset() 5-
6:
for 1-1 do if achieved then g - random goal (randomly sample a few states near 8, and measure Ppseudo- Then select state with the highest value of Ppscudo.) end if
8:
9:
11:
Ge f (T((s,,)) g)
12:
8:-11 < E.v..tte(aa. end for
13:
14:
+ U {50, 00. 91 81...
15:
end for
16:
for i=0,1....,MM do
17:
Sample minibatch b from B and replace the original reward with intrinsic reward in
Section (We used relabeling technique based on (2).
18:
Train and with b via SAC [[3].
19:
Sample another minibatch b' from Ds ~ {(B, y = 0), (Do, y = 1)) and Dr ~ U.
20:
Train f. with b' via Eq. 5
21:
end for
22:
end for
16

---

C More experimental results
C.1 Full results of the main script
We included the full results of the main script in this section. We include the visualization of the proposed curriculum goals in all environments Figure T6 The visualization results of the Sawyer- Seayer--eg-PPeee Peg-Picc&Place are not included as it shares the same map with the Sewyer-Peg-Putt environment.
Foldel currlcuten
Fiaal curricalun
bultea
curriculsm
curricutor
Fonal
teldel curricutum
Fieal
curicalus
telitel
curriculmm
Fösal
(a) Ours
(b) OUTPACE
(c) HGG
Figure 8: Curriculum goal visualization of the proposed method and baselines in all environments.
First row: Complex-Mazz Maze. Second row: Medium-Maze. Third row: Spiral-Maze. Fourth Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place.
17

---

C.2 Additional ablation study results
Full ablation study results of the main script. We conducted ablation studies described in our main script in all environments. Figure shows the average distance from the proposed curriculum goals to the desired final goal states along the training steps, and Figure [8] shows the episode success rates along the training steps. Note that there are results for the ablation study without a curriculum proposal in Figure 17c (unlike Figure [8c) since there are no curriculum goals to measure the distance from the desired final goal states. As we can see in these figures, we could obtain consistent analysis with the results in the main script in most of the environments.
18

---

13
(eetur hoatee
But
A-01
Parv al Mittadooon
0istance
distance
1350
1250
1900
153
1000
1999
1750
7500
253
1005
3000 Bart
1350
1590
STADs
steps
steps OK)
17.5
17.
15.0
15.0
Dai Matsadoooa
distance
250
1330
1200
1998
2002
253
151
1930
1790
75a
253
153
1962
1939
distance 31
1-05
heade2
12
astence
kw?
detance
22002 (K]
staps
stops (R)
2 (Defhuts head-1
Cara
Gats All Rilisieeeee
A-05
hoaded
aesence
dicance
53005 [k]
stops (2)
stops (8)
Dara
101
Burs win kmies/neemn
ditant
aisternc
aistarce
1209
1550
2530
500
1263
2000
3000
3000
1260
2000
2048
SteDs
steps
steps
A LiDdfall)
14
2.
Bars
Dar Hoinidoond
ditans
cistence
22
1003
1530
2002 2000 3000
5000
3000
1003
2300
2000
3300
3000
1500 2000
2000
3000
(a) Auxiliary loss weight À
(b) Number of prediction heads
Intrinsic reward
Figure 9: Ablation study in terms of the distance from the proposed curriculum goals to the desired final goal states (Lower is better). First row: Complex-Maze Second row: Medium-Maze. Third Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick Place. The shaded area represents a standard deviation across seeds.
19

---

Dart
08
Am01
Dara w/s Curiouuuu
Paiv alo Mitsocoooo
SUCCe00 pare
mm0
250
1300
1250
1900
1758
3000
2.0
153
1068
1999
1950
2500
253
1068 1750 steps (K)
1590
3000
Staps (K]
steps
Daiv aia Marraetooaa
Succetoo retee
400
250
1330
19K
1996
2002
253
150
1793
1930
1990
75a
28
253
1962 1353
1999
1750
3.0
1-0.5
headed
Dara Cumoouuu
km?
SUCCCHOO cete
1
20002 (K]
stops
stops (R)
1.0
3.0
Darn
head-1 hoaded
Oais wlo Hnlisseeeeed
A-05
Dara w/o Caniculun
SUECCIOO cate
22005 [k]
stops (K)
stops (8)
head "(ad-J/De
Data
0.0
1101
hoades
Burs w/n kmies/iieee
Data w/o
1209
1900
2500
00
500
1963
2000
3000
3000
1560
2000
2049
3000
SteDs
steps
steps
- biDdfaul)
Bart
Daro alo Balnaidoonad
(226
5Xcceeto, rete
540
1003
1530 2000
3000
55
530
3000
1349
2300
2000
3000
500
3000
1300
02)
2100
2040
3000
(a) Auxiliary loss weight X
(b) Number of prediction heads
(c) IItrinniic reward Curriculum
Figure 10: Ablation study in terms of the episode success rate. First row: Complex-Mazz Second row: Medium-Maze Third row: Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place. The shaded area represents standard deviation across seeds.
20

---

Curriculum learning objective type. We conduct additional experiments to validate whether reflecting the temporal distance in a curriculum learning objective (Eq ( required since there are a few works that estimate the temporal distance from the initial state distribution to propose the curriculum goals in a temporally distant region or explore based on this temporal informa- tion [ 33.. To reflect the temporal distance in the cost function (Eq (7), we modify it as w(s: 9: : (p (y/ = 1[5; 9. y: = Ppseudo (y = 1g 9. )) (so, o(s:) (o(-) is goal space mapping) since the value function itself implicitly represents the temporal distance if we use the sparse reward or custom-defined reward similar to the sparse one. In this case, our proposed intrinsic reward outputs for the desired goal and 0 for the explored states, and it works similarly to the sparse one.
We experimented with this modified curriculum learning objective (+Value), and the results are shown in Figure 19 20 shows that there is no significant difference, which supports the superiority of our method in that our method achieves wate-of-the-an results without considering additional temporal distance information.
Dun
Ourt
Out
destane 2 1
chstence
154
steds 0K)
steos [X]
Stets [k]
(a) Complex-Mane Maze
(b) Medium- Maze
(c) Spiral-Maze
Duss
Ous Volee
Ous Vilue
Ous 10k
detence
distanca
C. 2
cistance
ST600 OX?
S7800 (X]
St000 (K]
(d) Ant Locomotion
(e) Sawyer Push
(I) Sawyer Pick&Place
Figure 11: Ablation study in terms of the average distance from the curriculum goals the final goals (Lower is better). Value means that we additionally consider the value function bias in the curriculum learning objective to reflect the temporal distance from the initial state distribution.
1.0
10
28
Qum
02
Qutt Wil
Cuit- Wal
Curc Velle
00
1200
150 230
2.0
200
283
Da
1088 .00
64
280
ude Sleos
D20
OK)
(a) Complex-Maze
(b) Medium-Maze
(c) Spiral-Maze
SUCCOSA 0200
02
00
452
3000
2000
2002
2023
122
2002
2000
1050
120
2002
2029
2000
25.00
1200
27835
52805
(d) Ant Locomotion
(e) Sawyer Push
(f) Sawyer Pick&Place
Figure 12: Ablation study in terms of the episode success rates. Value means that additionally consider the value function bias in the curriculum learning objective to reflect the temporal distance from the initial state distribution.
21

---

Choice of goal candidates in training conditional classifiers. As mentioned in the main script, we also experimented with different choices of the goal candidates when we train the conditional classifiers (Fg (5) The default setting is DG = Dr, and we also experimented with Da = BUpt (g). The results are shown Figure 21 [2 shows that there is no significant difference, which means we can even make the problem setting more strict by conditioning the classifier only with the visited states and the given desired outcome examples.
DD
D
distanee " C
distence
21006 (K)
27030 (K)
57005 (K]
(a) Complex-Maze
(b) Medium- Maze
(c) Spiral-Maze
distanca
distance
e
STR00 OX?
ST8.0 [X]
S2000 (k]
(d) Ant Locomotion
(c) Sawyer Push
(f) Sawyer Plac&Place
Figure 13: Ablation study in terms of the average distance from the curriculum goals to the final goals (Lower is better). There are significant differences between the choice of goal candidates to train the conditional classifiers.
1.0
SU02000
00
200
750
120
1500
2000
06
250
360
750
2008
120
aide
2000
1158 C9
25
380
156
2009
(a) Complex-Maze
(b) Medium-Maze
(c) Spiral-Maze
Dy m
STe00 OK)
578005
52e05
(d) Ant Locomotion
(c) Sawyer Push
(f) Sawyer Pick&Place
Figure 14: Ablation study in terms of the episode success rates. There are no significant differences between the choice of goal candidates to train the conditional classifiers.
D Training & Experiments details
D.1 Training details
Baselines. The baseline curriculum RI. algorithms are trained as follows,
OUTPACE (5:
We follow the default setting in the original implementation from https //ttthhw..co//yyyyEooo////ttteee
22

---

HGG [37
We follow the default setting in the original implementation from https
//github.. com/ Stilwell-GittHanneeeeeeeeee
Goal Seaat---oa--Geeeitiiio Generation
CURROT [8] We follow the default setting in the original implementation from https
//github. com wwwp//ccclln/cccrrrrroooooooo
PLR [6]. VDS 49), ALP-GMM 36 We follow the default setting in implementation
from htttp:://gtiiiiii
www.pscckknaaeaaa currot
D2C and all the baselines are trained by SAC [3] with the sparse reward except for the OUTPACE which uses an intrinsic reward based on Wasserstein distance with a time-step metric.
Training details. We used NVIDIA A5000 GPU and AMD Ryzen Threadripper 3960X for training, and each experiment took about I~~ days for training. We used small noise from a uniform distribution with environmeent-specitt noise scale (Table for augmenting the conditioned goal in Eq (5) Also, we used the mapping o(.) that abstracts the state space into the goal space when we use the diversified conditional classifiers (i.e. ((((s):g) g). For example, xf-) abstracts the proprioceptive states (e.g. zy position of the agent) in navigation tasks, and abstracts the object-centric states (e.g. zyz position of the object) in robotic manipulation tasks.
Table 4: Hyperparameters for D2C
critic hidden dim critic hidden depth
512
discount factor 7/
0.99
batch size
512
critic target
0.01
init temperature C'nit of SAC
0.3
Critic target update frequency actor hidden dim actor hidden depth actor update frequency
replay buffer B size learning rate for f; learning rate for Critic & Actor optimizer
3e6
512
1e-3
le-4
adam
Table 5: Default env -speciicc hyperparameters for D2C
Env name
ji update
fi # of iteration
max episode
heads
freq (step)
per update
horizon
Complex-Maze
2
0.5
2000
16
100
Medium-Maze
2
0.5
2000
16
100
Spiral-Maze
2
0.5
2000
16
100
Ant Locomotion
2
1.0
4500
16
300
Sawyer- Peaee-PeePPuss-
2
1
0.025
3000
16
200
Sawyer-Peg-Picke Pick&Place
2
0.025
3000
16
200
D.2 Environment details
The observation consists of the zy position, angle, velocity, and angular velocity 'point'. The action space consists the velocity and angular velocity of the 'point'. The initial state of the agent is [0, 0] and the desired outcome states are obtained from the default goal points [8, 16), [-8, 166, [16, - 88, 1-1,, 8). The size of the map is 36 x 36.
Complex-Maze:
the Complex-Maze environment except that the desired outcome states are obtained from the default goal points [16, 16), 16, 16), [16, [-16, 16).
Medium-Maze: It is the
Spizal-Maze: The observation space and actions space and initial state of the agent are the same as in the Complex Maze environment. The desired outcome states are obtained from the default goal points [12, 16], |-12, -16]. The size of the map 28 36.
Ant Locomotion:
The observation consists the æy= position, ayz velocity, joint angle, and joint angular velocity of the 'ant'. The action space consists of the torque applied on the rotor of the 'ant". The initial state of the agent [0. 0) and the desired outcome states are obtained from the default goal points [4, 8). [-4, 8). The size of the map is 12 x 20.
23

---

ae
(a) Complex-Marz
(b) Medium-Maze
(c) Spiral-Maze
S
(d) Ant Locomotion
(c) Sawyer Push
(f) Sawyer Pick&Place
Figure 15: Environments used for evaluation: yellow stars indicate the desired outcome examples. (a)-(c) the agent should navigate various maze environments with multi-moda desired outcome distribution. (d) the ant locomotion environment with multi-modal desired outcome distribution. (e) the robot has to push or pick place a peg to multi-modal desired locations while avoiding an obstacle at the center of the table.
Sawyer-Peg-Puse The observation consists of the zyz position of the end-effector; the object, and the gripper's state. The action space consists of the ryz position of the end-effector and gripper open/close control. The initial state of the object is 00.4, 0.8, 0.02) and the desired outcome states are obtained from the default goal points [-0.3, 0.4, 0.02), [-0.3, 0.8, 0.02], [0.4, 0.4, 0.02]. The wall is located at the center of the table. Thus, the robot arm should detour the wall reach the desired goal states. We referred to the metaworld (48) and EARL [40 environments.
is the same as the Sawyer-Peg-Push environment ex- cept that the desired outcome states
obtained from the default goal points -0.3, 0.4, 0.2], [-0.3, 0.8, 0.2), [0.4, 0.4, 0.2). and the wall is located at the center of the table, fully blocking path for pushing. Thus, the robot should pick and move the object over the wall to reach the desired goal states.
24

---

E
Algorithm
Algorithm 2 Overview of D2C algorithm
1: Input: desired outcome examples p* (g). total training episodes N, Env, environment horizon
H. actor и, critic Q, replay buffer
2: for 1ter. do
3: ÿ" < sample K curriculum goals that minimize Eq (6) We refer to HGG 37 for solving the
bipartite matching problem.
4: for i=1,2....KK do
5: Env.reset()
6:
5-
for 1-1 do if achieved then g - random goal (randomly sample a few states near 8, and measure Ppseudo- Then select state with the highest value of Ppscudo.) end if
8:
9:
11:
Ge f (T((s,,)) g)
12:
8:-11 < E.v..tte(aa) end for
13:
14:
+ U {50, 00. 91 81...
15:
end for
16:
for i=0,1....,MM do
17:
Sample minibatch b from B and replace the original reward with intrinsic reward in
Section[444 (We used relabeling technique based on (2).
18:
Train and with b via SAC [[3].
19:
Sample another minibatch b' from Ds ~ {(B, y = 0), (Do, y = 1)) and Dr ~ U.
20:
Train f. with b' via Eq. 5
21:
end for
22:
end for
25

---

More experimental results
F.1 Full results of the main script
We included the full results of the main script in this section. We include the visualization of the proposed curriculum goals in all environments Figure T6 The visualization results of the Sawyer- Peyver-ee--Pee Peg-Pick&Plaae are not included as it shares the same map with the Sewyer-Peg-Putt environment.
Foldel currlcuten
Fiaal curricalun
bultea
curriculsm
curricutor
Fonal
teldel curricutum
Fieal
curicalus
telitel
corriculmm
Fisal
(a) Ours
(b) OUTPACE
(c) HGG
Figure 16: Curriculum goal visualization of the proposed method and baselines in all environments. First row: Complex-Maze, Second row: Medium-Maze. Third row: Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place.
26

---

F.2 Additional ablation study results
Full ablation study results of the main script. We conducted ablation studies described in our main script in all environments. Figure [7 the average distance from the proposed curriculum goals to the desired final goal states along the training steps, and Figure [8 shows the episode success rates along the training steps. As we can see in these figures, we could obtain consistent analysis with the results the main script in most of the environments.
27

---

11.0
8
hoaded
12.3
dssancc
ST603 [K]
stops
stops (K)
12
Dara Gutt Rnteor/
kw0.1
5
hoalmg
12.
ditance
2.0
s389s [k]
staps 04)
staps (8)
band 3 (1efhil))))
Dart Dars wlu Mittuuttonn
A-01
astence
distance
12
PC
1399
1P9
199
193
1783
1939
1701
7539
1982 1987
1939
70t1
S2005
stops
stops (O)
Cart
axterce
distanc
dstance
1930
000
2231
13:00
11222
2020
hoadwd
2
3izonce
27605 (K]
stops
Stopc (8)
(ipttuc)) 1-01
(DDhhtt head boadee
Dara Burs Ail innoereeeen
1w05
60
52605 [K]
stop5 (K)
stons
(a) Auxiliary loss weight À
(b) Number of prediction heads
(c) Intrinsic reward & Curriculum
Figure 17: Ablation study in terms of the distance from the proposed curriculum goals to the desired final goal states (Lower is better). There no results for the ablation study without curriculum proposal since there are no curriculum goals to measure the distance from the desired final goal states. First row: complex-Maze Second row: Medium-Maze. Third row: Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place. The shaded area represents a standard deviation across seeds.
28

---

Dart
08
Am01
Dara w/s Curiouuuu
Paiv alo Mittocoooo
SUCCe00 pare
mm0
250
1300
1250
1900
1958
3000
2.0
153
1068
1999
1950
2500
253
1068 1750 steps (K)
1590
3000
Staps (K]
steps
Daiv aia Marraeoooaa
Succetoo retee
400
250
1330
19K
1996
2002
253
150
1793
1930
1990
75a
28
253
1962 1353
1999
1750
3.0
1-0.5
headed
Dara Cumoouuu
km?
SUCCCHOO cete
1
20002 (K]
stops
stops (R)
1.0
3.0
Darn
head-1 hoaded
Oais wlo Hnlisseeeeed
A-05
Dara w/o Caniculu
SUECCIO cate
22005 [k]
stops (K)
stops (8)
head "(ad--/De
Data
0.0
1101
hoades
Burs w/n kmies/iieee
Data w/o
1209
1900
2500
00
500
1963
2000
3000
3000
1560
2000
2049
3000
SteDs
steps
steps
- biDdfaul)
Bart
Daro alo Balnaidoonad
(226
5Xcceeto, rete
540
1003
1530 2000
3000
55
530
3000
1349
2300
2000
3000
500
3000
1300
02)
2100
2040
3000
(a) Auxiliary loss weight X
(b) Number of prediction heads
(c) IItrinniic reward Curriculum
Figure 18: Ablation study in terms of the episode success rate. First row: Complex-Maze Second row: Medium-Maze Third row: Spiral-Maze. Fourth row: Ant Locomotion. Fifth row: Sawyer Push. Sixth row: Sawyer Pick & Place. The shaded area represents standard deviation across seeds.
29

---

Curriculum learning objective type. We conduct additional experiments to validate whether reflecting the temporal distance in a curriculum learning objective (Eq ( required since there are a few works that estimate the temporal distance from the initial state distribution to propose the curriculum goals in a temporally distant region or explore based on this temporal informa- tion [5 33.. To reflect the temporal distance in the cost function (Eq (7), we modify it as w(s: 9: : (p (y/ = 1[5; 9. y: = Ppseudo (y = 1g 9. )) (so, o(s;) (o(-) is goal space mapping) since the value function itself implicitly represents the temporal distance if we use the sparse reward or custom-defined reward similar to the sparse one. In this case, our proposed intrinsic reward outputs for the desired goal and 0 for the explored states, and it works similarly to the sparse one.
We experimented with this modified curriculum learning objective (+Value), and the results are shown in Figure 19 20 shows that there is no significant difference, which supports the superiority of our method in that our method achieves wate-of-the-an results without considering additional temporal distance information.
Dun
Ourt
Out
destane 2 1
chstence
154
steds 0K)
steos [X]
Stets [k]
(a) Complex-Mane Maze
(b) Medium- Maze
(c) Spiral-Maze
Duss
Ous Volee
Ous Vilue
Ous 10k
detence
distanca
C. 2
cistance
ST600 OX2
S7800 (X]
St000 (K]
(d) Ant Locomotion
(e) Sawyer Push
(I) Sawyer Pick&Place
Figure 19: Ablation study in terms of the average distance from the curriculum goals the final goals (Lower is better). Value means that we additionally consider the value function bias in the curriculum learning objective to reflect the temporal distance from the initial state distribution.
1.0
10
28
Oum
02
Qutt- Win
Cuit- Wal
Curc Velle
00
1200
150 230
2.0
200
283
Da
1088 .00
64
380
ude Sleos
D20
OK)
(a) Complex-Maze
(b) Medium-Maze
(c) Spiral-Maze
SUCCOSA 1200
02
00
452
3000
2000
2002
2023
122
2002
2000
1050
120
2002
2029
2000
25.00
1000
27835
52805
(d) Ant Locomotion
(e) Sawyer Push
(f) Sawyer Pick&Place
Figure 20: Ablation study in terms of the episode success rates. Value means that we additionally consider the value function bias in the curriculum learning objective to reflect the temporal distance from the initial state distribution.
30

---

Choice of goal candidates in training conditional classifiers. As mentioned in the main script, we also experimented with different choices of the goal candidates when we train the conditional classifiers (Fg (5) The default setting is DG = Dr, and we also experimented with Da = BUpt (g). The results are shown Figure 21 [2 shows that there is no significant difference, which means we can even make the problem setting more strict by conditioning the classifier only with the visited states and the given desired outcome examples.
Bup!lg
By Epp
By Syy
dostence
destane
518t0s 022
Si65s (4)
Sleos (4)
(a) Complex-Maze
(b) Medium-Maze
(c) Spiral-Maze
05
distance
12
652 3300
3000 2003
2023
1030
323
2003
2003
20:00
2000
2303
20.00
(d) Ant Locomotion
Sawyer Push
(f) Sawyer Pick&Place
Figure 21: Ablation study in terms of the average distance from the curriculum goals to the final goals (Lower is better). There significant differences between the choice of goal candidates to train the conditional classifiers.
Dy -2
Cit De Ba pry
Ruptig)
Da P
$.02000
saccess
S1806 06)
$1806 (4)
Spaps (K)
(a) Complex-Maze
(b) Medium-Maze
(c) Spiral-Maze
08
0.
SUCCesS 0238
1312
09
<9
21000 (K]
520000 04)
52005
(d) Ant Locomotion
(e) Sawyer Push
(l) Sawyer Pick&Place
Figure 22: Ablation study in terms of the episode success rates. There are significant differences between the choice of goal candidates train the conditional classifiers.
31


Reviews:
[{"review_id": "vq5NivJxQc", "reviewer": "Reviewer_LCMf", "summary": "The paper proposed Diversify for Disagreement & Conquer (D2C), which is an outcome-directed curriculum RL method. D2C performs diversification of the goal-conditional classifiers to identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in a simple and intuitive way. D2C then employs bipartite matching to define a curriculum learning objective that produces a sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region.", "questions": "1. OUTPACE is the latest model in comparison, but it performs (near) the worst in all tasks (Fig.4 & 5). I had a quick check of OUTPACE paper, and it reports completely different results. Please explain how this could happen.\n\n2. What is the complexity of the proposed model?", "limitations": "The paper has a good summary of its limitations and potential societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "D2C performs a classifier diversification process to distinguish the unexplored region from the explored area and desired outcome example.\n\nD2C conquers the unexplored region by proposing the curriculum goal and the shaped intrinsic reward.\n\nD2C enables the agent to automatically progress toward the desired outcome states without prior knowledge of the environment.", "weaknesses": "1. The writing of the paper needs some polishing to improve its readability.\n\n2. The settings of the experiments, e.g. the map types and sizes, are not challenging to show the significance of the proposed model.\n\n3. The complexity analysis is missing. Empirically, it takes 1 - 2 days to train a solution for a 36 * 36 map as described in Appendix. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness:**\n1. The writing of the paper needs some polishing to improve its readability.\n\n→ Thank you for your suggestion. Since there is no way to revise the current manuscript in the rebuttal phase, we will try to rewrite some phrases and fix some grammatical errors as much as possible after the decision. If you specify some examples, it would be really helpful for us to revise our work with better readability.\n\n2. The settings of the experiments, e.g. the map types and sizes, are not challenging to show the significance of the proposed model.\n\n→ Thank you for your comment. We would like to note that we followed the settings of the experiments in prior curriculum RL works. We think the environments used in this work are already challenging enough to show the significance of our model since it outperforms other prior state-of-the-art works in these environments. In other words, only D2C can achieve all the tasks regardless of their domain, geometry, distribution of the desired outcome states, etc. If you propose a specific environment that is appropriate to show the significance of our model, we will try to experiment and evaluate our method. It would be really helpful to improve our work’s contribution.\n\n3. The complexity analysis is missing. Empirically, it takes 1 - 2 days to train a solution for a 36 * 36 map as described in Appendix.\n\n→ Question section\n\n\n**Questions:**\n1. OUTPACE is the latest model in comparison, but it performs (near) the worst in all tasks (Fig.4 & 5). I had a quick check of OUTPACE paper, and it reports completely different results. Please explain how this could happen.\n\n→ Thank you for your comment about the baseline. As described in lines 267-272, OUTPACE uses Wasserstein distance-based temporal distance estimation to propose curriculum goals into the temporally distant region from the initial state distribution. Once the agent starts to explore a temporally far region in a specific direction earlier, the curriculum proposal is prone to be collapsed toward this area (Figure 3). Thus, there exists only one way to the desired outcome state (i.e. uni-modal desired outcome distribution) in the environments used in OUTPACE to prevent such a problem. Also, the curriculum objective in OUTPACE includes meta-learning-based uncertainty quantification to combine with the temporal distance estimation, which results in extensive hyperparameter tuning for numerical stability and balancing the trade-off between uncertainty and temporal distance. In practice, without such tuning, we found that OUTPACE’s loss frequently blows up. On the contrary, our method simply consists of a single module (i.e. classifiers for disagreement) with cross-entropy loss and mutual information computation, which results in a stable training process compared to the meta-learning process and Wasserstein distance estimation in OUTPACE.\n\n2. What is the complexity of the proposed model?\n\n→ Thank you for your comment about the complexity analysis. Since our method is based on simple MLP-based classifiers with multiple heads, the inference time of the classifier is negligible. The classifier training is performed every 2000~4500 steps as described in Table 3 in Appendix A. Therefore, classifier training time is nearly negligible compared to the SAC’s update which is performed at every step. The curriculum optimization itself requires solving a bipartite matching problem as described in section 4.3, and this is addressed via the Minimum Cost Maximum Flow algorithm, which is implemented as a few lines of c code adopted from prior work [1]. It consists of a double for-loop. The first loop is iterations on a few trajectories sampled from the replay buffer, and the second loop is iterations on the desired goals. In practice, this takes around 10 seconds, but curriculum optimization is also performed every thousands of steps (it depends on the maximum episode horizon of each environment), thus, the overall curriculum optimization time is not a significant bottleneck. We believe that these responses adequately address this comment about the complexity. However, if you still have remaining concerns or questions please let us know.\n\n[1] Ren, Zhizhou, et al. \"Exploration via hindsight goal generation.\" *Advances in Neural Information Processing Systems* 32 (2019).\n"}, {"review_id": "jM3UPuEbJD", "reviewer": "Reviewer_pHjM", "summary": "Focusing on goal-conditioned RL, the author propose Diversify for Disagreement & Conquer (D2C) to form outcome-directed exploration by generating a sequence of curriculum goals, given some desired outcome examples. By ensuring multiple classifiers disagree on unseen states, D2C uses bipartite matching to create curriculum goals, which are interpolated between the initial state distribution and arbitrarily distributed desired outcome states for enabling the agent to conquer the unexplored region. The experiments demonstrate that D2C surpasses previous curriculum RL methods in goal-conditioned RL experiments.", "questions": "- Please clarify the interpolation process as referenced in weakness.\n\n- Figure 6 can be better if the labels do not cover the curves. The font sizes in the figures are too small to read.", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I appreciate the idea to use the diversifying functions for disagreement on underspecified data and to quantify the similarity between the visited states and desired outcome states. To use the pseudo probability as the intrinsic reward is also straightforward. The proposed method is simple but effective. \n\n- The paper is well written.\n", "weaknesses": "- It is not clear for me how to interpolate the curriculum distribution from the initial state distribution through minimizing Eq 4.\n- Some related work and its comparison missing. Diversity has been considered in Curriculum RL. For example, Curriculum-guided Hindsight Experience Replay (NeurIPS 2019).\n- It is not open sourced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n1. It is not clear for me how to interpolate the curriculum distribution from the initial state distribution through minimizing Eq 4.\n\n→ Question section.\n\n2. Some related work and its comparison missing. Diversity has been considered in Curriculum RL. For example, Curriculum-guided Hindsight Experience Replay (NeurIPS 2019).\n\n→ Thank you for your suggestion about the related work. We had a quick check of the CHER paper, and there exist some related keywords, so we will add this work in the revised version. But, in terms of methodology, there are several key differences between CHER and our method.\n\nFirst, CHER is about goal relabeling technique like HER during the RL update phase rather than proposing curriculum goal during the rollout phase. That is, CHER is appropriate for an add-on-module on our method rather than a direct comparison since our method is also based on the HER as described in Algorithm 1 in Appendix B.\n\nSecond, as far as we understand CHER, the distance metric in the proximity measurement is based on the Euclidean distance metric, which is not appropriate for the geometry-agnostic curriculum proposal property. Also, diversity in CHER means how representative sampled curriculum candidates are with respect to the states in the replay buffer. However, diversity in D2C induces disagreement between the classifiers for the unlabeled target data, and it is reflected in curriculum optimization cost to encourage the algorithm to propose curriculum goals into the unexplored region. Thus, the same keyword “diversity” is somewhat different between our work and CHER.\n\nDue to the short period of the rebuttal phase, we do not have enough time to experiment with CHER. But, based on the results in the CHER paper, we think that CHER has contributed to performance increases compared to RL+HER rather than making the algorithm work in an environment where RL+HER completely fails. Since most of the tasks used in D2C cannot be solved with RL+HER without additional tools such as curriculum proposal (based on our own experimental results), we expect that CHER will not bring significant improvement compared to RL+HER in the tasks used in D2C.\n\n3. It is not open sourced.\n\n→ The source code is already included in the supplementary material. Also, the code will be open-sourced after the decision.\n\n**Questions:**\n1. Please clarify the interpolation process as referenced in weakness.\n\n→Thank you for your comment. The pseudo probability in Eq (3) represents how similar the queried state is to the explored states or desired outcome states (lines 178-188), and disagreement between classification heads is represented as intermediate values between the 0 (explored states) and 1 (desired outcome states) when optimizing the curriculum loss in Eq (4). \nFor example, before discovering the desired outcome states, the batch data sampled from the replay buffer in Eq (4) include a diverse range of states from frequently visited states to rarely visited ones such as the states in the frontier of the explored region. As described in lines 178-188, $P_{pseudo}(y=1|g^+)$ is nearly 1 and the loss will be large for the frequently visited states as all classifiers correctly classify them as a label 0 (i.e. $P_{pseudo}(y=1|s)=0$), and the loss will be small for the states in the frontier since the classifiers disagree due to the diversification (i.e. $P_{pseudo}(y=1|s)$ is larger than 0). Thus, the curriculum objective encourages selecting the states in the frontier region.\nAfter discovering the desired outcome states, the curriculum objective encourages selecting the states close to the desired outcome state since the corresponding loss will be the smallest among the losses of states in the replay buffer.\n    \nIn the case of multi-modal desired outcome distribution as described in Fig 2 (two desired outcome examples, K=2 in Eq (6)), the conditional classifier enables non-collapsed curriculum proposals even when the agent achieves a specific desired outcome earlier, since we utilize bipartite matching (i.e. no duplicative selection) to find the curriculum goal candidates in the replay buffer.\n    \nFor clarity, we would like to note that cross-entropy in Eq (4) is used as a convex function that decreases as $P_{pseudo}(y=1|s)$ increases, not as an exact entropy minimization to match the probability distribution (because $P_{pseudo}$ in Eq (3) is not mathematically probability, so we coined it with “pseudo”).\n\n2. Figure 6 can be better if the labels do not cover the curves. The font sizes in the figures are too small to read.\n\n→ Thank you for your suggestion. We modified the location of the labels and font size of Figure 6, and we attached the pdf with modified figures of experimental results at the global response. Due to the single-page limit, current figures are a little bit small. After the decision, we will further modify the figures with better visibility and readability on the additional page."}, {"review_id": "ovBGgM4krm", "reviewer": "Reviewer_geTA", "summary": "This paper propose training multiple classifiers on a data set of \"desired\" and \"reached but undesired\" states, motivating these classifiers to be different off distribution, and using this divergence as a metric for exploration.", "questions": "What advantage does this approach have over generic model-disagreement, or random network distillation?", "limitations": "Given the number of existing approaches, and an unclear understanding of their relative limitations, it is unclear when this approach should be preferred over the others in a new domain.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The approach is simple and easy to implement in many domains.", "weaknesses": "Since the approach is very similar to model-disagreement, I would expect a simple comparison to such an approach like exploration via disagreement.  Moreover, given that random-network distillation works so well for exploration, its unclear if the objective of the classifier here is really too important.  It may only be important that the metric has to do with the disagreement between two networks, and it may not matter what they are disagreeing about.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n\n1. Since the approach is very similar to model-disagreement, I would expect a simple comparison to such an approach like exploration via disagreement. Moreover, given that random-network distillation works so well for exploration, its unclear if the objective of the classifier here is really too important. It may only be important that the metric has to do with the disagreement between two networks, and it may not matter what they are disagreeing about.\n\n→ Thank you for your insightful comment. As you rightly pointed out, it is similar to model-disagreement in terms of encouraging the agent to explore uncertain areas. VDS, which utilizes the epistemic uncertainties of the value function ensembles to propose curriculum goal, is included as the baseline in the experiment section for the model-disagreement-based method. Also, Random Network Distillation (RND) encourages exploration by defining intrinsic reward based on the prediction error of the network rather than explicitly proposing the curriculum goal. However, VDS and RND do not have a convergence mechanism to the desired outcome states. That is, they just encourage endless exploration to find the novel states even after discovering the desired outcomes, which is not desirable for goal-reaching settings.\n\nEven though we can imagine the combination between external reward given from the environmental interaction and the RND's intrinsic reward, it still requires hyperparameter tuning for balancing the external (exploitation) and intrinsic reward (exploration). Also, since these methods do not consider multi-modal desired outcome distribution, they are prone to exploitation. That is, the agent does not explore other desired outcome states once it found a specific desired outcome distribution earlier. However, our proposed method keeps exploring when the agent is in the same setting because the curriculum cost for the undiscovered desired outcome state is not minimized yet. Furthermore, our method provides integrated methodology not only for the desired outcome-directed curriculum generation but also for the goal-conditioned intrinsic reward compared to VDS, and RND that require an assumption of access to the external reward.\n\nFor the last concern, we would like to note that it matters what the classifiers are disagreeing. If the classifiers disagree on already explored states, efficient exploration toward the frontier is unavailable since the agent tries to reach the proposed curriculum goals in the already explored regions. Also, if the classifiers disagree on desired outcome states, the proposed curriculum goals will not be converged to the desired outcomes. It not only hinders the acceleration of the curriculum proposal toward the desired outcomes after discovering these but also prevents the agent from repeatedly practicing to reach the desired outcomes. Since we assume a setting where only some desired outcome examples are given before training begins and access to the desired outcome distribution and external reward are unavailable, convergence to the desired outcome states is required. Therefore, we think it matters that the classifiers disagree only on the unexplored states.\n\nWe believe that these responses together adequately address this comment. However, if you still have remaining concerns or questions please let us know.\n\n**Questions:**\n\n1. What advantage does this approach have over generic model-disagreement, or random network distillation?\n\n→ Weakness section\n\n**Limitations:**\n\n1. Given the number of existing approaches, and an unclear understanding of their relative limitations, it is unclear when this approach should be preferred over the others in a new domain.\n\n→ Thank you for your comment. We added more details of the baseline as follows. (Also, there are conceptual comparisons between D2C and baselines in Table 1 and brief explanations are in lines 249-259. A quick review of these would be helpful to clarify the pros and cons of each different approach.)\n\nHGG and CURROT utilize Euclidean distance metric, they are prone to get stuck in an obstacle in an environment such as Maze, while our method D2C does not have such a problem. \n\nPLR and ALP-GMM leverage the regret or learning progress, which implicitly indicates the novelty or difficulty of the proposed curriculum. But these methods do not have a convergence mechanism to the desired outcome distribution, which induces endless exploration rather than achieving the given task. Furthermore, ALP-GMM depends on the Gaussian Mixture Model (GMM) that is susceptible to focusing on infeasible goals where the agent’s capability stagnates in the intermediate level of difficulty.\n\nOUTPACE is similar to our approach, but it requires Wasserstein distance-based temporal distance estimation for curriculum proposal, which can result in collapsed curriculum goals as described in lines 267-272. Also, for uncertainty quantification, it adopts meta-learning (MAML [1]) that requires gradient computation at every optimization iteration, while our method only requires a single neural network inference, leading to much faster curriculum optimization.\n\nOverall, the proposed method, D2C, has several advantages compared to other curriculum RL baselines, and we think it could be preferred over others in general. \n\n[1] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. \"Model-agnostic meta-learning for fast adaptation of deep networks.\" *International conference on machine learning*. PMLR, 2017."}, {"review_id": "kjia78RiS2", "reviewer": "Reviewer_WkJV", "summary": "This paper proposes a new reinforcement learning algorithm called Diversify for Disagreement & Conquer (D2C). \nThe idea of the algorithm is to divide the search space and let conditional classifiers explore these subspaces. \nWith a set of experiments, the authors verify the effectiveness of D2C compared to various other approaches. \nAs new to the field, I can not judge the relevance of the contribution for reinforcement learning.", "questions": "See above", "limitations": " ", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "As a person new to the field of RF, this paper is well written since it introduces necessary concepts at an appropriate depth, and the idea and motivation are intuitive. \nIn addition, the approach is geometric flexible, shows its effectiveness in a range of experiments, and the author provides an in-depth ablation study of relevant hyperparameters.\n", "weaknesses": "Again I need to admit that I am entirely new to this field. At the same time, the other baseline experiments all do explore the search undirected. Is it a fair comparison when your approach can explore many directions situationally?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n\n1. Again I need to admit that I am entirely new to this field. At the same time, the other baseline experiments all do explore the search undirected. Is it a fair comparison when your approach can explore many directions situationally?\n\n→ First of all, it is slightly unclear the meaning of “situationally”, so if we misunderstood your question and the answer is not appropriate, please let us know. \n\nWe would like to note that most of the baselines and our method perform undirected exploration until the agent discovers the desired outcome states. Only HGG and CURROT perform directed exploration from the initial phase. Some baselines keep doing undirected exploration even after the agent discovers the desired outcome states (e.g. VDS, ALP-GMM, PLR), and the other baselines and our method propose converged curriculum goals after discovering the desired outcome states (e.g. HGG, CURROT, OUTPACE). It is summarized in Table 1 (column: Target dist. of curriculum). Therefore, we think it is a fair comparison since all algorithms do an undirected search by their own strategies before discovering the desired outcome states."}, {"review_id": "qw6WC6KoWO", "reviewer": "Reviewer_TCnF", "summary": "This paper introduces a novel curriculum learning algorithm (D2C) in the context of reinforcement learning. Unlike previous approaches, D2C does not rely on prior knowledge of the environment's structure, including the distance measure between states. Instead, it leverages a goal-conditioned binary classifier to differentiate between visited states and desired states. When learning this classifier, D2C additionally incorporates a diversification objective to encourage the classifier to disagree on unvisited states. The paper claims that D2C could automatically generate a diverse set of curriculum goals. These goals, in turn, provide intrinsic rewards that guide the agent towards achieving the desired goal state.\n", "questions": "1. Regarding the diversification: It seems that when optimizing the objective of curriculum distribution, the uncertainty/disagreement between different classification heads are not in the objective. Equation 4 simply minimizes the cross entropy between the averaged probability of the proposed curriculum state being a goal state and the average probability of the true goal states. Where does the uncertainty of the unexplored region play a role in the overall objective of the algorithm?\n2. The problem of over-exploration: Based on the learning objective of D2C, suppose the agent is given a maze environment similar to the one shown in Figure 2. But if the goal distribution only includes the top one, then is it true that this proposed algorithm would still explore both the top and bottom part of the maze? \n", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The technique introduced in this paper that identify the similarities between the explored states and desired outcomes with a diversification objective appears to be novel within the domain of curriculum RL.\n2. Empirical evaluation of the proposed methods in the selected maze and robotics manipulation environment shows the significant improvement of the proposed algorithm.\n", "weaknesses": "1. The D2C algorithm relies on having access to a dataset of unlabeled target data in order to model unvisited states. However, this assumption can be a significant limitation of the algorithm. In the experiments conducted in this paper, the author was able to overcome this limitation by uniformly sampling states from the upper and lower bounds of the state space. Nevertheless, obtaining such a collection of (legal) states within the state space of agents or robots is often a challenging task. For instance, if the agent receives pixel observations or if the legal state space of a robotic arm is a constrained subset of the entire space defined by the cartesian product each dimension, the assumption of sampling from the legal state space to obtain the unlabeled target dataset may not be applicable. In general, this assumption holds true only in specific environments or tasks.\n2. The current version of the proposed algorithm has some fundamental limitations. For example, the proposed algorithm could be prone to over-explore. Also, how the diversification process helps the overall curriculum proposal process is not clear to me. Please refer to my questiona in the next section.\n\nI am happy to raise the score if my questions are addressed by the author. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n\n1. The D2C algorithm relies on ...\n\n→ We appreciate your insightful comment about the extension to the high-dimensional spaces. As you have rightly pointed out, directly applying the bipartite matching problem (section 4.3) in high-dimensional spaces may not work. However, we can consider an encoder-decoder structure with ELBO loss-based representation learning techniques such as VAE to abstract high-dimensional inputs to low-dimensional latent vectors $z$ and measure the Euclidean distance between $z$ of the querying image and goal image similar to [1]. We think these ideas may extend our bidirectional curriculum generation method to high-dimensional spaces and not be restricted to low-dimensional spaces.\n\nAlso, for state inputs in the diversified conditional classifiers, we used the mapping $\\phi(\\cdot)$ that abstracts the state space into the goal space as described in Appendix A (e.g. $\\phi(\\cdot)$ abstracts the global xyz position of the object in robot manipulation task). It enables us to construct simple and legal state space within the lower and upper bound in most cases. Since knowing each state element’s meaning is a commonly utilized assumption in RL (e.g. hindsight relabeling technique), we think this is not a significant limitation.\n\nAlso, as you may concern, there could be infeasible state space within the lower and upper bound. For instance, we can consider the existence of the obstacle at the center of the end-effector’s feasible range, or conflict between the end-effector’s position obtained by applying forward kinematics to the (uniform) randomly sampled joint position and the end-effector’s position (if joint states are included in states). However, the D2C classifiers disagree even in these infeasible states because these are not included both the desired outcome examples and explored states in the replay buffer because the agent cannot reach these states. Since we obtain the curriculum goals from the replay buffer, infeasible states cannot be proposed as curriculum goals, and the agent will keep explore toward feasible, unexplored areas based on the disagreement between classifiers, and discover the desired outcome states. \n\nIf you still have remaining concerns or if we misunderstood your question, please let us know. We would be happy to discuss this.\n\n[1] Nair, Ashvin V., et al. \"Visual reinforcement learning with imagined goals.\" *Advances in neural information processing systems*\n     31 (2018).\n\n2. The current version of the proposed ...\n\n→ Question section\n\n**Questions:**\n\n1. Regarding the diversification ...\n\n→ Thank you for your comment about the diversification. The pseudo probability in Eq (3) represents how similar the queried state is to the explored states or desired outcome states (lines 178-188), and disagreement between classification heads is represented as intermediate values between the 0 (explored states) and 1 (desired outcome states) when optimizing the curriculum loss in Eq (4). \n\nFor example, before discovering the desired outcome states, the batch data sampled from the replay buffer in Eq (4) include a diverse range of states from frequently visited states to rarely visited ones such as the states in the frontier of the explored region. As described in lines 178-188, $P_{pseudo}(y=1|g^+)$ is nearly 1 and the loss will be large for the frequently visited states as all classifiers correctly classify them as a label 0 (i.e. $P_{pseudo}(y=1|s)=0$), and the loss will be small for the states in the frontier since the classifiers disagree due to the diversification (i.e. $P_{pseudo}(y=1|s)$ is larger than 0). Thus, the curriculum objective encourages selecting the states in the frontier region.\n\nAfter discovering the desired outcome states, the curriculum objective encourages selecting the states close to the desired outcome state since the corresponding loss will be the smallest among the losses of states in the replay buffer.\n\nFor clarity, we would like to note that cross-entropy in Eq (4) is used as a convex function that decreases as $P_{pseudo}(y=1|s)$ increases, not as an exact entropy minimization to match the probability distribution (because $P_{pseudo}$ in Eq (3) is not mathematically probability, so we coined it with “pseudo”).\n\nWe believe that these responses together adequately address this comment. However, if you still have remaining concerns or questions please let us know.\n\n2. The problem of over-exploration ...\n\n→ Thank you for your insightful comment. If the goal distribution only includes the top one, then D2C will propose curriculum goals in both directions at the initial phase, since the frontier states of both ways will have the pseudo probability of 0.5 (for example, in the case of two classification heads). However, as the agent discovers the goal distribution at the top, the curriculum goals will be converged to the distribution at the top since states in these distributions will have the pseudo probability of 1, and the agent does not explore toward the other direction anymore. \n\nWe would like to note that we assume a setting where only some desired outcome examples are given before training begins and access to the desired outcome distribution is unavailable. Therefore, the setting where the goal distribution only includes the top one means we want the agent to reach the top one only. Thus, after discovering the top one, no more exploration toward the other direction is a natural consequence. If we want to explore the other direction also, it means there exist desired outcomes in the other direction, so we should be equipped with the desired outcome examples corresponding to the bottom one before training begins. The detailed motivation of this outcome/example-driven method is well summarized in prior works such as [1].\n\n[1] Fu, Justin, et al. \"Variational inverse control with events: A general framework for data-driven reward definition.\" *Advances in neural information processing systems* 31 (2018)."}]

Author rebuttal:
We sincerely thank all the reviewers for reviewing our work and providing constructive feedback. We hope that our response has adequately addressed your comments. If you have any remaining questions (existing or new ones) that we can address in our follow-up response to improve your opinion about our work, please do not hesitate to provide additional feedback in the comments. It would be greatly appreciated if we could have more discussions about our work which would provide valuable insights towards further developing our research into a meaningful contribution in the RL domain. 

Also, there is an ask for figures with large font sizes, so we attached modified figures in the pdf file. 

Meta-review: The paper studies an outcome-directed curriculum RL method, which encourages disagreement on underspecified data in order to improve the explore vs exploit trade-off in large state spaces. This diversity mechanism enables quantifying (and rewarding exploration of) the unexplored region, by ensuring that classifiers disagree on states from out-of-distribution. The method then proposes a curriculum learning objective that produces a sequence of intermediate goals, which enable the agent to explore and conquer unexplored regions.

Reviewer consensus after the discussion phase is that this is an interesting contribution. We hope the detailed feedback can help to clarify the camera-ready version of the paper.
Decision: Accept (poster)