Title: Diversify \& Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
Venue: NeurIPS 2023
Accepted: yes
Keywords: Curriculum learning

Out-of-distribution disagreement

Underspecification

Outcome-directed RL
Abstract: Reinforcement learning (RL) often faces the challenges of uninformed search problems where the agent should explore without access to the domain knowledge such as characteristics of the environment or external rewards. To tackle these challenges, this work proposes a new approach for curriculum RL called $\textbf{D}$iversify for $\textbf{D}$isagreement \& $\textbf{C}$onquer ($\textbf{D2C}$). Unlike previous curriculum learning methods, D2C requires only a few examples of desired outcomes and works in any environment, regardless of its geometry or the distribution of the desired outcome examples. The proposed method performs diversification of the goal-conditional classifiers to identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in a simple and intuitive way. The proposed method then employs bipartite matching to define a curriculum learning objective that produces a sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region. We present experimental results demonstrating that D2C outperforms prior curriculum RL methods in both quantitative and qualitative aspects, even with the arbitrarily distributed desired outcome examples.

Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement

Abstract

Reinforcement learning (RL) often faces the challenges of uninformed search problems where the agent should explore without access to the domain knowledge such as characteristics of the environment or external rewards. To tackle these challenges, this work proposes a new approach for curriculum RL called D iversify for D isagreement & C onquer ( D2C ). Unlike previous curriculum learning methods, D2C requires only a few examples of desired outcomes and works in any environment, regardless of its geometry or the distribution of the desired outcome examples. The proposed method performs diversification of the goal-conditional classifiers to identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in a simple and intuitive way. The proposed method then employs bipartite matching to define a curriculum learning objective that produces a sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region. We present experimental results demonstrating that D2C outperforms prior curriculum RL methods in both quantitative and qualitative aspects, even with the arbitrarily distributed desired outcome examples.

1 Introduction

Reinforcement learning (RL) has great potential for the automated learning of behaviors, but the process of learning individual useful behavior can be time-consuming due to the significant amount of experience required for the agent. Furthermore, in its general usage, RL often involves solving a challenging uninformed search problem, where informative rewards or desired behaviors are rarely observed. While there are some techniques that can alleviate the exploration burden such as rewardshaping [30] or preference-based reward [21, 6], they often require significant domain knowledge or human intervention. This makes it difficult to utilize RL directly for problems that require challenging exploration, especially in many domains of practical significance. Thus, it is becoming increasingly crucial to tackle these challenges from the algorithmic level by developing RL agents that can learn autonomously with minimal supervision.

One potential approach is a curriculum learning algorithm. It involves proposing a carefully designed sequence of curriculum tasks or goals for the agent to accomplish, with each step building upon the previous one to gradually progress the curriculum. By doing so, it enables the agent to automatically explore the environment and improve the capability in a structured way. Previous studies primarily involve a mechanism to adjust the curriculum distribution by maximizing its entropy within the explored region [35], or taking into account the difficulty level [43, 11], or learning progress of the agent [36]. However, efficient desired outcome-directed exploration is not available under these

Unlabeled

Curriculum

Explored

(label0)

states

D

source

targetdata

proposal

Desired

-Label1

Unlabeled

outcome

source data

targetdata

(label 1)

states

MLP

|multiplevalid

unexplored areas

Quantify

layers

classifiers

viadisagreement

Unlabeled

targetdata

source data.

Explored

Unexplored

Desired

state

state

Dtarget

★：Curriculum goals

Label1

Unlabeled

:Agent

source data

targetdata

Statespace

frameworks as these approaches do not have converging curriculum objectives, resulting in a naive search for unseen states.

Providing the agent with examples of successful outcomes can make the RL problem more manageable, as opposed to letting the agent explore without a particular objective and with an undirected reward. Such examples can offer considerable guidance on how to achieve a task if the agent can estimate the similarities between the desired outcome examples and visited states. In the realm of desired outcome-directed RL, previous studies try to maximize the probability of reaching desired outcomes [12, 41, 8]. However, these approaches do not have tools for explicitly quantifying an unexplored region, leading the agent to settle for reaching an initially discovered desired outcome example, rather than continuing to explore further to find another example. Other works try to minimize the distance between the generated curriculum distribution and the desired outcome distribution to propose intermediate task goals [37, 18]. But, these approaches have mainly been confined to problems that do not involve significant exploration challenges because they rely on the assumption that the Euclidean distance metric can represent the geodesic interpolation between distributions. This assumption is not universally applicable to all environmental geometries, making these methods less versatile than desired.

Therefore, it is necessary to develop an algorithm that enables the agent to automatically perform the outcome-directed exploration by generating a sequence of curriculum goals, which can be applied to arbitrary geometry and distribution of the given desired outcome examples. To do so, we propose D iversify for D isagreement & C onquer ( D2C ), which only requires desired outcome examples and does not require prior domain knowledge such as 1) the geometry of the environment 2) or the number of modes or distribution of the desired outcomes, or 3) external reward from the environment. Specifically, D2C involves diversifying the goal-conditioned classifiers to identify the similarities between the visited states and the desired outcome states. This is accomplished by ensuring that the outputs of each classifier disagree on unseen states, which not only allows determining the unexplored frontier region but also provides an arbitrary goal-conditioned intrinsic reward. Based on such conditional classifiers, we propose to employ bipartite matching to define a straightforward and easy-to-comprehend curriculum learning objective, which produces a range of well-adjusted curriculum goals that interpolate between the initial state distribution and arbitrarily distributed desired outcome states for enabling the agent to conquer the unexplored region.

To sum up, our work makes the following key contributions.

We propose a new outcome-directed curriculum RL method that only needs a few arbitrarily distributed desired outcome examples and eliminates the need for external reward.

To the best of our knowledge, D2C is the first algorithm for curriculum RL that allows for automatic progress in any environment without being restricted by its geometry or the desired outcome distribution by proposing diversified conditional classifiers.

In various goal-conditioned RL experiments, our method consistently outperforms the previous curriculum RL methods through precisely calibrated guidance toward the desired outcome states in both quantitative and qualitative aspects.

2 Related Works

Despite various attempts to improve exploration in RL, it remains a difficult issue that has not been fully solved. Some previous works for exploration have suggested methods based on information theoretical approaches [9, 39, 50, 20, 17, 27], maximizing the state visitation distribution's entropy [47, 25, 26], counting the state visitation [3, 31], utilizing similarity or curiosity [33, 45], and quantifying uncertainty through prediction models [4, 34]. Other approaches provide a curriculum to allow the agent to explore the environment through intermediate tasks. Curricula are usually created by adjusting the distribution of goals to cover new and unexplored areas. It is achieved by considering an auxiliary objective such as entropy [35] or disagreement between the model ensembles [49, 15, 28] or difficulty level of the curriculum [11, 43], regret [16], and learning progress [36]. However, these methods only focus on visiting diverse frontier states or do not provide a mechanism to converge toward the desired outcome distribution. In contrast, our method enables more efficient outcome-directed exploration via curriculum proposal with an objective to converge rather than simply exploring various frontier states, only requiring a few desired outcome examples.

Assuming access to desired outcome samples or distribution, some prior methods try to accomplish the desired outcome states by maximizing the probability of reaching these states [12, 41, 8]. However, they lack a mechanism for quantifying an under-explored region and synthesizing the knowledge acquired from the agent's experiences into versatile policies that can accomplish novel test goals. Some algorithms generate curricula as an interpolation between the distribution of desired target tasks and auxiliary tasks [37, 18], but they still rely on the Euclidean distance metric, which is insufficient to handle arbitrary geometric structures. There exists a work that addresses geometry-agnostic curriculum generation using the given desired outcome states, similar to our approach [5]. But, it requires carefully tuned Wasserstein distance estimation that depends on the optimality of the agent. It leads to inconsistent estimation before the convergence, resulting in numerically unstable training, while our method does not have such dependence. Also, it adopts a meta-learning-based technique [10, 23] that requires gradient computation at every optimization iteration, while our method only requires a single neural network inference, leading to much faster curriculum optimization.

A core idea behind our method is utilizing classifiers to learn a diverse set of hypotheses that minimize the loss on source inputs but make differing predictions on target inputs [32, 22]. It is related to ensemble methods [7, 19, 14] that aggregate the multiple functions' predictions, but the proposed method is distinct in terms of directly optimizing on an underspecified target dataset for enhancing diversity. Even though this diversifying strategy for target data is typically considered from the perspective of out-of-distribution robustness in conditions of distribution shift [29, 24, 38] or domain adaptation [44, 46, 42], we demonstrate how it can be applied for classifiers to quantify the similarity between the visited states and desired outcome states, and discuss its links to exploration and curriculum generation in RL.

3 Preliminary

We consider the Markov decision process (MDP) M = ( S , G , A , P , γ ) , where S indicates the state space, G the goal space, A the action space, P ( s ′ | s, a ) the transition dynamics, and γ the discount factor. In our framework, the MDP is not provided a reward function and we consider a setting where only the desired outcome examples { g + k } K k =1 from the desired outcome distribution p + ( g ) are given. Thus, our method utilizes an intrinsic reward r : S × G × A → R . Also, we represent the curriculum distribution obtained by our method as p c ( s ) .

3.1 Acquiring knowledge from underspecified data

For diversification-based curriculum RL, we train a classification model y = f ( x ) in a supervised learning setting where x ∈ X is input, and y ∈ Y is the corresponding label. The model f is trained with a labeled source dataset D S ∼ p S ( x, y ) that includes both the given desired outcome examples ( y = 1 ) and the visited states in the replay buffer B of the RL agent ( y = 0 ). We assume that the desired outcome distribution can be modeled as a mixture of outcome distribution, p + ( x ) = ∑ o ∈ O w o p o ( x ) , where each o ∈ O corresponds to a specific outcome distribution p o ( x ) . The selection of model f from hypothesis class f ∈ F is achieved by minimizing the predictive risk E p S ( x,y ) [ L ( f ( x ) , y )] , where L is a standard classification loss.

Although f demonstrates good generalization on previously unseen data acquired from the source distribution p S ( x, y ) , it is ambiguous to evaluate the model f in distribution shift conditions such as when querying target data obtained from an out-of-distribution (e.g. a state lies in an unexplored region). This is because there could be many potential models f that can minimize the predictive risk. To formalize this intuition, we introduce the concept of an ε -optimal set defined as F ε := { f ∈ F | L p ( f ) ≤ ε } [22], where L p represents the risk associated with a distribution p , and ε ≥ 0 .

The definition of the ε -optimal set implies that the predictions of any two models f 1 , f 2 in the set are almost identical on p S ( x, y ) when ε is small. But, using F ε as it is has a few drawbacks: 1) there is no criterion to prefer any particular hypotheses of F ε over another, and 2) the trained model f might not be suitable for evaluating the target data distribution as it does not have a mechanism to quantitatively discriminate unseen target data from the labeled source data.

To address this point, we utilize an unlabeled target dataset D T ∼ p T ( x ) for comparing the functions within F ε by examining how their predictions differ on D T . Since D T can be viewed as indicating the directions of functional change that are most crucial to the automatic exploration of the RL agent, we set p T ( x ) as the uniform distribution between the lower and upper bound of the state space to include all possible candidate states to visit. Knowing the state space's bounds is a commonly utilized assumption in many curriculum learning works [37, 18, 5] and it does not require being aware of the dynamically feasible areas. Then, our objective is to identify a set of classification models that perform well in p S ( x, y ) and disagree in p T ( x ) to recognize the unseen state from the unexplored region by quantifying the similarity between the observed states in B and desired outcome examples. Such a function will lie within F ε of p S ( x, y ) , and the model leverages D T to identify a diverse set of functions within the near-optimal set.

4 Method

For an automatic exploration toward the desired outcome distribution via calibrated guidance of the curriculum, the proposed D2C suggests curriculum goals via diversified classifiers that disagree in the unexplored region and enables the agent to conquer this area by exploring through the goalconditioned shaped intrinsic reward. It allows the agent to make progress without any prior domain knowledge of the environment such as obstacles or distribution of the desired outcome states and advances the curriculum towards the desired outcome distribution p + ( g ) .

4.1 Diversification for disagreement on underspecified data

For the automatic exploration of the RL agent with the curriculum proposal, quantification of whether a queried state is already explored or not is required. To obtain such a quantification, we diversify a collection of classifier functions by comparing predictions for the target dataset while minimizing the training error as briefly described in Section 3.1. The intuition behind this is that diversifying predictions will produce functions that disagree with data in the ambiguous region [22].

Specifically, we use a multi-headed neural network with N heads to train diverse multiple functions. Each head i produces a prediction for an input x represented as f i ( x ) . To ensure that every head has a low predictive risk on p S ( x, y ) , we minimize the cross-entropy loss for each head using L xent ( f i ) = E x,y ∼D S [ CE ( f i ( x ) , y )] . Ideally, it is desirable for each function to rely on distinctive predictive features of the input to encourage differing predictions. Therefore, we train each pair of heads to generate statistically independent predictions, which implies disagreement in predictions. It could be achieved by minimizing the mutual information between each pair of heads:

where the input data is obtained from target dataset D T . For implementation, we compute empirical estimates of the joint distribution p ( f i ( x ) , f j ( x )) and the product of the marginal distributions p ( f i ( x )) ⊗ p ( f j ( x )) , which can be computed using libraries developed for deep learning [22].

In summary, the overall objective for classifier diversification is represented with a hyperparameter λ :

̸

(2) Non-conditional

(3)Conditional

$1★

gi

gi

$1★

★：Curriculum candidate

S2★

S2★

S2★

：Agent

:Desired outcome

S5

S5

S5

Explored

Unexplored

Desired

state

state

★S3,S4

★S3,S4

★S3,S4

92

92

(1)

p(y = 1|/s)

$1

S2

S3

S5

p(y = 1ls;g+)

$1

S2

S3

S5

gt

0.5

0.4

1

0.9

0

gi

0.5

0.4

0

0

0

9

0.5

0.4

1

0.9

0

￥6

0.5

0.4

1

0.9

0

Select S1,S3

Select S3,S4

4.2 Quantifying unexplored regions by conditional classifiers

From this section, we slightly abuse the notation s instead of x for the RL setting. Considering the diversification process in Section 4.1, we can quantify how much a queried state s is similar to the desired outcome example from p + ( g ) . Specifically, we can define a pseudo probability of the queried point s by averaging predictions of each head:

When the queried state s is in proximity to the data within p S (with the desired outcome example labeled as 1 and states from the replay buffer B labeled as 0), it becomes challenging for the classifiers to give a high probability to labels that substantially differ from the neighboring data, since each classifier f i is trained to minimize the loss for the source data. On the other hand, if the queried state s is significantly different from the data in p S , each prediction head will output different values since the classifiers are trained to diversify the predictions on the target data (uniform distribution on the state space) to make their prediction values disagree as much as possible. For example, in the case of two heads, one predicts 1 and the other predicts 0 for the same queried target data, resulting in the pseudo probability of 0.5. This could be interpreted as a quantification of how much disagreement exists between classifiers, or uncertainty of the queried data, which can be utilized as an unexplored region-aware classification (Figure 1).

Thus, we can consider a curriculum learning objective for interpolating the curriculum distribution from the initial state distribution to p + ( g ) represented by the following cross-entropy loss:

Intuitively, before discovering the desired outcome examples in p + ( g ) , the curriculum goal candidate s ∼ B that minimizes Eq (4) is proposed in the frontier of the explored regions where classifiers f i disagree. And, as the agent explores and discovers the desired outcome examples, the curriculum candidate is updated to converge to p + ( g ) to minimize the discrepancy between the predicted labels of g + and s .

However, it is not applicable for a case when the desired outcome examples are spread over multimodal distribution or arbitrarily because the trained classifiers f i do not distinguish which p o the desired outcome example is obtained from (Figure 2). In other words, f i will predict a value close to

1 for any desired outcome example g + , and the loss in Eq (4) will be close to 0. As the curriculum goal candidates are obtained from the replay buffer B , the curriculum optimization may collapse if the agent achieves one of the desired outcome distributions ( p o ) earlier, which is not desirable for achieving all the desired outcome examples regardless of its distribution.

Since we assume that we do not know which p o the desired outcome example is obtained from, nor the number of modes ( o ) of the desired outcome distributions, we propose to utilize a conditional classifier to address this point while satisfying the assumption. Specifically, we define goal-conditioned classifiers, where each classifier takes input s and is conditioned on g , and these classifiers are trained to minimize the following modified objective of Eq (2):

̸

where ε is small noise (from uniform distribution around zero or standard normal distribution with a small variance) for numerical stability, and D G can be either D T or B ∪ p + ( g ) for training arbitrary goal-conditioned & unexplored region-aware classifiers. We found that there is no significant difference between these choices. Then, the pseudo probability can be represented as p pseudo ( y = 1 | s ; g ) := 1 N ∑ N i =1 f i ( s ; g ) and we can address the arbitrarily distributed desired outcome examples without curriculum collapse (Figure 2). This proposed conditional classifier-based quantification is one of the key differences from the previous similar outcome-directed RL methods [23, 5]. Because the previous works require computing gradients of thousands of data for meta-learning-based network inference, while our method only requires a single feedforward inference without backpropagation which leads to fast computation.

4.3 Curriculum optimization via bipartite matching

As we assume that we have access to desired outcome examples from p + ( g ) instead of their explicit distribution, we can approximate it using the sampled set ˆ p + ( g ) ( | ˆ p + ( g ) | = K ). Then, the problem is formulated by the combinatorial setting that requires finding the curriculum goal candidate set ˆ p c ( s ) that will be assigned to each sample of ˆ p + ( g ) , and it can be solved via bipartite matching. With curriculum goal candidates and desired outcome examples, the curriculum learning objective is represented as follows:

The intuition behind this objective is similar to Eq (4), but it is different in terms of considering conditional quantification, which enables addressing arbitrarily distributed desired outcome examples. Then, we can create a bipartite graph G with edge costs w by considering the sets of nodes V a and V b , which represent achieved states in replay buffer B and ˆ p + ( g ) , respectively. We define the bipartite graph G ( { V a , V b } , E ) with edge weights E ( · , · ) = -w ( · , · ) . To solve this bipartite matching problem, we employ the Minimum Cost Maximum Flow algorithm [1, 37] to find K edges with the minimum cost w . The entire curriculum RL process is shown in Algorithm 2 in Appendix E.

4.4 Conditional classifier-based intrinsic reward

As we have trained conditional classifiers and defined the pseudo probability, we can additionally use this value as a shaped intrinsic reward for enabling the agent to solve the uninformed search problem. As the pseudo probability outputs 0 for the state in the replay buffer B , 1 for the desired outcome state, and a value between 0 and 1 for the state where the classifiers disagree (meaning unexplored region), we can define the goal-conditioned intrinsic reward as r = p pseudo ( y = 1 | s ; g ) . We use this intrinsic reward in all of our experiments. The ablation study for this reward is detailed in Section 5.2.

Conditional quantification

Target dist. of curriculum

Arbitrary desired outcome dist.

Geometry-agnostic

Without external reward

HGG

✗

p

+

(

g

)

✓

✗

✗

CURROT

✗

uniform or

p

+

(

g

)

✓

✗

✗

PLR

✗

✗

✗

✓

✗

VDS

✗

✗

✗

✓

✗

ALP-GMM

✗

✗

✗

✓

✗

OUTPACE

✗

p

+

(

g

)

✗

✓

✓

Ours

✓

p

+

(

g

)

✓

✓

✓

(a) Ours

(b) OUTPACE

(c) HGG

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

5 Experiment

We conduct experiments on 6 environments that have multi-modal desired outcome distribution to validate our proposed method. We use various maze environments (Point Complex-Maze, MediumMaze, Spiral-Maze) to validate our curriculum proposal capability, which is not limited to specific geometries. Additionally, we evaluate our method on more complex dynamics or other domains such as the Ant-Locomotion and Sawyer-Peg Push, Pick&Place with obstacle environments to demonstrate its effectiveness in domains beyond the navigation. (Refer to Appendix D for more details.)

We compare our method with several previous curriculum generation approaches, each of which possesses the following characteristics. OUTPACE [5] prioritizes goals that are considered uncertain and temporally distant from the initial state distribution through meta-learning-based uncertainty quantification and Wasserstein-distance-based temporal distance approximation. HGG [37] aims to reduce the distance between the desired outcome state and curriculum distributions, using a value function bias and the Euclidean distance metric. CURROT [18] interpolates between the desired outcome state and curriculum distribution while considering the agent's current capability using the Wasserstein distance. VDS [49] proposes epistemic uncertainty-based goals by utilizing the value function ensembles. ALP-GMM [36] models absolute learning progress score by a GMM. PLR [16] prioritizes increasingly challenging tasks by ranking task levels based on TD errors. We summarize the conceptual comparison between our method and baselines in Table 1.

5.1 Experimental results

First, to validate the quality of curriculum goal interpolation from the initial state distribution to the desired outcome distribution, we qualitatively and quantitatively evaluate the curriculum progress achieved during the training by optimizing Eq (6). For qualitative evaluation, we visualize the curriculum goals obtained by the proposed method and other baselines (Figure 3). The proposed method shows calibrated guidance toward the desired outcome states, even with the multi-modal distribution. But, HGG shows an inappropriate curriculum proposal due to the Euclidean distance metric, and OUTPACE shows curriculum goals collapsed only toward a single direction because it utilizes the Wasserstein distance for biasing curriculum goals into the temporally distant region. Once the agent explores a temporally far region in a specific direction earlier, the curriculum proposal is collapsed toward this area. In contrast, our method does not have such dependence, which enables

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

20

25

15

20

distance

distance

15

distance

10

10

5

D

250

500

75010001250150017502000

0

250500

75010001250150017502000

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

12

0.6

0.6

10

0.5

0.5

distance

distance

0.4-

distance

0.4

6

0.3

0.3

0.2

0.2

0.1

0.1

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

■

OUTPACE

CURROT

口

VDS

HGG

ALP-GMM

口

PLR

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

1.0

1.0

1.0

0.8

0.8-

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250

steps (k)

1500

1750

2000

0.0

0

250

500

750

steps (k)

10001250

15001750

2000

00

0

250

500

750

steps (k)

10001250

150017502000

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

400

0

500

1000

steps (k)

1500

2000

2500

3000

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

0'0

0

500

1000

steps (k)

1500

2000

2500

3000

our method to always propose curriculum goals in any uncertain direction quantified by disagreement between the diversified classifiers.

We also plot the average distance from the proposed curriculum goals to the desired outcome states for quantitative evaluation (Figure 4). As we evaluated with multi-modal desired outcome distribution, the distance cannot be measured by naively averaging the distance between the randomly chosen desired outcome states and curriculum goals. Thus, we measure the distance by bipartite matching with the l 2 distance metric, which computes the distance between the desired outcome states and their assigned curriculum goals. As shown in Figure 4, only the proposed method consistently shows superior interpolation results from the initial state distribution to desired outcome distribution, while other baselines show some progress only in a simple geometric structure due to the Euclidean distance metric, or get stuck in some local optimum area due to the lack of or insufficient unexplored region quantification. We also plot the performance of the outcome-directed RL in Figure 5. As expected by the average distance measurement in Figure 4, the proposed method is the only one that consistently and quickly achieves the desired outcome states through the guidance of calibrated curriculum goals, indicating the advantage of the proposed diversified conditional classifiers.

(a) Auxiliary loss weight

λ

(b) Number of prediction heads

(c) Intrinsic reward & Curriculum

1.0

入=1(Default)

1.0

head=2(Default)

1.0

Ours

0.8

入=0.5

入 = 0.1

入=2

rate

0.8

head=4

head=6

0.8

Ours w/o IntrinsicReward

0.6

success_

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

750

1000125015001750

2000

0.0

0

250

500

750

10001250

150017502000

0.0

0

250

500

75010001250

1500

1750

2000

steps (k)

steps (k)

steps (k)

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

A = 0.1

head=4

Ours w/o IntrinsicReward

0.8

A= 0.5

0.8

head=6

0.8

rate

入= 2

rate

0.6

0.6

0.6

success_

success_

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

5.2 Ablation study

Number of prediction heads & Auxiliary loss weight λ . To investigate the sensitivity to the hyperparameters of our method, we experimented with the different values of the auxiliary loss' weight λ and with the different number of prediction heads of the conditional classifiers. As shown in Figure 6a, 6b, the results are slightly dependent on the environment's characteristics such as the presence of object interaction or complex geometry. However, the overall performance trend is not significantly affected by the number of prediction heads and λ , except in the extreme case, which supports the superiority of our proposed method. More qualitative/quantitative analyses and other ablation studies are included in Appendix F.

Reward type & Curriculum. We evaluate the effectiveness of the proposed intrinsic reward and curriculum proposal by conducting two ablation experiments. First, we replace the intrinsic reward with a sparse reward ( Ours w/o IntrinsicReward ), which is typically used in goal-conditioned RL problems. Second, we conduct experiments without the curriculum proposal while utilizing the intrinsic reward ( Ours w/o Curriculum ). As shown in Figure 6c, there is performance degradation without using the proposed intrinsic reward in an environment with complex dynamics since an informative reward signal is crucial. In addition, the absence of a curriculum proposal leads to performance degradation in an environment with complex geometry as it requires carefully crafted guidance for exploration. These results highlight the importance of both proposed components, i.e. intrinsic reward and curriculum proposal.

6 Conclusion

We propose D2C that 1) performs a classifier diversification process to distinguish the unexplored region from the explored area and desired outcome example and 2) conquers the unexplored region by proposing the curriculum goal and the shaped intrinsic reward. It enables the agent to automatically progress toward the desired outcome states without prior knowledge of the environment. We demonstrate that our method outperforms the previous methods in terms of sample efficiency and geometry-agnostic, desired-outcome-distribution-agnostic curriculum progress, both quantitatively and qualitatively.

Limitation & Broader impacts. Despite the promising results, there is a limitation in the scalability of the proposed method since we use small noise to augment the conditioned goal for numerical stability, making it difficult to scale to high-dimensional inputs such as images. Thus, addressing this point would be an interesting future research direction to develop a more generally applicable method. Also, our work is subject to the potential negative societal impacts of RL, but we do not expect to encounter any additional negative impacts specific to this work.

References

R K Ahuja, T L Magnanti, and J B Orlin. Network Flows: Theory, Algorithms, and Applications . Prentice Hall, Englewood Cliffs, NJ, 1st edition, 1993.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv:1707.01495 , 2017.

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. arXiv preprint arXiv:2301.11741 , 2023.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Thomas G Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Italy, June 21-23, 2000 Proceedings 1 , pages 1-15. Springer, 2000.

Ben Eysenbach, Sergey Levine, and Russ R Salakhutdinov. Replacing rewards with examples: Examplebased policy search via recursive classification. Advances in Neural Information Processing Systems , 34: 11541-11552, 2021.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , pages 1126-1135. PMLR, 2017.

Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning , pages 1515-1528. PMLR, 2018.

Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: Ageneral framework for data-driven reward definition. Advances in neural information processing systems , 31, 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence , 12(10):993-1001, 1990.

Edward S Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. arXiv preprint arXiv:2303.13002 , 2023.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning , pages 4940-4950. PMLR, 2021.

Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. arXiv preprint arXiv:2106.14305 , 2021.

Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning , pages 11341-11358. PMLR, 2022.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161 , 2022.

Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091 , 2021.

Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data. arXiv preprint arXiv:2202.03418 , 2022.

Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. In International Conference on Machine Learning , pages 6346-6356. PMLR, 2021.

Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning , pages 6781-6792. PMLR, 2021.

Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning , pages 6736-6747. PMLR, 2021.

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems , 34:18459-18473, 2021.

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreographer: Learning and adapting skills in imagination. arXiv preprint arXiv:2211.13350 , 2022.

Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems , 34:24379-24391, 2021.

Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems , 33:20673-20684, 2020.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml , volume 99, pages 278-287. Citeseer, 1999.

Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning , pages 2721-2730. PMLR, 2017.

Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. arXiv preprint arXiv:2202.04414 , 2022.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 2778-2787. PMLR, 2017.

Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning , pages 5062-5071. PMLR, 2019.

Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698 , 2019.

Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning , pages 835-853. PMLR, 2020.

Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems , 32, 2019.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657 , 2019.

Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Benchmarking and formalism. arXiv preprint arXiv:2112.09605 , 2021.

Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854 , 2019.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407 , 2017.

Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. Domain adaptation in computer vision applications , pages 153-171, 2017.

David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11359 , 2018.

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687-10698, 2020.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning , pages 11920-11931. PMLR, 2021.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning , pages 1094-1100. PMLR, 2020.

Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems , 33:7648-7659, 2020.

Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprint arXiv:2103.08107 , 2021.

A Training & Experiments details

A.1 Training details

Baselines. The baseline curriculum RL algorithms are trained as follows,

OUTPACE [5]: We follow the default setting in the original implementation from https: //github.com/jayLEE0301/outpace_official .

HGG [37] : We follow the default setting in the original implementation from https: //github.com/Stilwell-Git/Hindsight-Goal-Generation .

CURROT [18]: We follow the default setting in the original implementation from https: //github.com/psclklnk/currot .

PLR [16], VDS [49], ALP-GMM [36] : We follow the default setting in implementation from https://github.com/psclklnk/currot .

D2C and all the baselines are trained by SAC [13] with the sparse reward except for the OUTPACE which uses an intrinsic reward based on Wasserstein distance with a time-step metric.

Training details. We used NVIDIA A5000 GPU and AMD Ryzen Threadripper 3960X for training, and each experiment took about 1 ∼ 2 days for training. We used small noise from a uniform distribution with an environment-specific noise scale (Table 5) for augmenting the conditioned goal in Eq (5). Also, we used the mapping ϕ ( · ) that abstracts the state space into the goal space when we use the diversified conditional classifiers (i.e. f i ( ϕ ( s ); g ) . For example, ϕ ( · ) abstracts the proprioceptive states (e.g. xy position of the agent) in navigation tasks, and abstracts the object-centric states (e.g. xyz position of the object) in robotic manipulation tasks.

A.2 Environment details

Complex-Maze: The observation consists of the xy position, angle, velocity, and angular velocity of the 'point'. The action space consists of the velocity and angular velocity of the 'point'. The initial state of the agent is [0 , 0] and the desired outcome states are obtained from the default goal points [8 , 16] , [ -8 , -16] , [16 , -8] , [ -16 , 8] . The size of the map is 36 × 36 .

Medium-Maze: It is the same as the Complex-Maze environment except that the desired outcome states are obtained from the default goal points [16 , 16] , [ -16 , -16] , [16 , -16] , [ -16 , 16] .

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

★

以

N

Spiral-Maze: The observation space and actions space and initial state of the agent are the same as in the Complex-Maze environment. The desired outcome states are obtained from the default goal points [12 , 16] , [ -12 , -16] . The size of the map is 28 × 36 .

Ant Locomotion: The observation consists of the xyz position, xyz velocity, joint angle, and joint angular velocity of the 'ant'. The action space consists of the torque applied on the rotor of the 'ant'. The initial state of the agent is [0 , 0] and the desired outcome states are obtained from the default goal points [4 , 8] , [ -4 , -8] . The size of the map is 12 × 20 .

Sawyer-Peg-Push: The observation consists of the xyz position of the end-effector, the object, and the gripper's state. The action space consists of the xyz position of the end-effector and gripper open/close control. The initial state of the object is [0 . 4 , 0 . 8 , 0 . 02] and the desired outcome states are obtained from the default goal points [ -0 . 3 , 0 . 4 , 0 . 02] , [ -0 . 3 , 0 . 8 , 0 . 02] , [0 . 4 , 0 . 4 , 0 . 02] . The wall is located at the center of the table. Thus, the robot arm should detour the wall to reach the desired goal states. We referred to the metaworld [48] and EARL [40] environments.

Sawyer-Peg-Pick&Place: It is the same as the Sawyer-Peg-Push environment except that the desired outcome states are obtained from the default goal points [ -0 . 3 , 0 . 4 , 0 . 2] , [ -0 . 3 , 0 . 8 , 0 . 2] , [0 . 4 , 0 . 4 , 0 . 2] , and the wall is located at the center of the table, fully blocking a path for pushing. Thus, the robot arm should pick and move the object over the wall to reach the desired goal states.

B Algorithm

Algorithm 1 Overview of D2C algorithm

C More experimental results

C.1 Full results of the main script

We included the full results of the main script in this section. We include the visualization of the proposed curriculum goals in all environments in Figure 16. The visualization results of the Sawyer-Peg-Pick&Place are not included as it shares the same map with the Sawyer-Peg-Push environment.

(a) Ours

(b) OUTPACE

(c) HGG

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

C.2 Additional ablation study results

Full ablation study results of the main script. We conducted ablation studies described in our main script in all environments. Figure 17 shows the average distance from the proposed curriculum goals to the desired final goal states along the training steps, and Figure 18 shows the episode success rates along the training steps. Note that there are no results for the ablation study without a curriculum proposal in Figure 17c (unlike Figure 18c) since there are no curriculum goals to measure the distance from the desired final goal states. As we can see in these figures, we could obtain consistent analysis with the results in the main script in most of the environments.

distance o1

(a) Auxiliary loss weight

λ

(b) Number of prediction heads

(c) Intrinsic reward

12

入 = 1(Default)

A = 0.1

12

head=2(Default)

head=4

15.0

Ours

Ours w/o IntrinsicReward

10

A = 0.5

10

head=6

12.5

distance

入=2

distance

7.5

5.0

2.5

0

250

500

75010001250 1500 1750 2000

0

250

500

75010001250150017502000

0.0

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

17.5

入= 1(Default)

12

head=2(Default)

17.5

Ours

15.0

A = 0.1

A = 0.5

10

head=6

head=4

15.0

Ours w/o IntrinsicReward

10.0

12.5 -

A= 2

distance

8

distance

10.0

12.5

7.5

7.5

5.0

5.0

2.5

2

2.5

0.0

0

0.0

0

250

500

75010001250150017502000

steps (k)

250

500

75010001250150017502000

steps (k)

0

250

500

75010001250

steps (k)

150017502000

25

A = 0.1

入 = 1(Default)

25

head=2(Default)

head=4

25

Ours

Ours w/o IntrinsicReward

20

A = 0.5

20

head=6

20

A = 2

distance

15

10

5

0

0

250

500

75010001250150017502000

steps (k)

250

500

75010001250

steps (k)

150017502000

250

500

75010001250150017502000

steps (k)

10

入=0.1

入=2(Default)

10

head=2(Default)

head=4

10

Ours w/oIntrinsicReward

Ours

入= 0.5

head=6

distance

入=1

入=4

distance

distance

2

0

0

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

0.6

入=1(Default)

0.6

head=2(Default)

0.6

Ours

0.5

A = 0.1

入 = 0.5

0.5

head=6

head=4

0.5

Ours w/o IntrinsicReward

入= 2

0.3

0.3

.3

0.2

0.2

021

0.1

0.1

0.1

0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

0.6

入 = 1(Default)

0.6

head=2(Default)

0.6

0.5

A = 0.1

A = 0.5

0.5

head=6

head=4

0.5

Ours w/o IntrinsicReward

入=2

distance

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0.0

0.0

0.0

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

0000

steps (k)

steps (k)

steps (k)

(a) Auxiliary loss weight

λ

(b) Number of prediction heads

(c) Intrinsic reward & Curriculum

1.0

入 =1(Default)

1.0

head=2(Default)

1.0

Ours

入= 0.1

head=4

Ours w/o IntrinsicReward

0.8

A = 0.5

0.8

head=6

0.8

success_rate

入=2

success_rate

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250150017502000

00

250

500

75010001250150017502000

0.0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

A = 0.1

head=4

Ours w/o IntrinsicReward

0.8

A = 0.5

0.8

head=6

0.8

A= 2

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250150017502000

steps (k)

0.0

250

500

75010001250150017502000

steps (k)

0.0

250

500

75010001250

steps (k)

150017502000

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

success_rate

0.8

入= 0.5

入=2

入 = 0.1

success_rate

0.8

head=6

head=4

success_rate

0.8

Ours w/o IntrinsicReward

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

250

500

7501000 125015001750 2000

0.0

250

500

75010001250

150017502000

0.0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

1.0

1.0

1.0

入=2(Default)

head=2(Default)

入=0.1

head=4

Ours w/o IntrinsicReward

0.8

A=0.5

入=1

0.8

head=6

0.8

0.6

A = 4

0.6

0.6

0.4

0.2

0.2

0.2

0.0

00

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

A = 0.1

head=4

Ours w/o IntrinsicReward

0.8

入= 2

A = 0.5

success_rate

0.8

head=6

success_rate

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

0

500

1000

steps (k)

1500

2000

2500

3000

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

0.8

入=0.1

入=0.5

0.8

head=6

head=4

0.8

Ours w/o IntrinsicReward

入=2

success_rate

0.6

0.4

0.4

0.2

0.2

0.2

00

0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

Curriculum learning objective type. We conduct additional experiments to validate whether reflecting the temporal distance in a curriculum learning objective (Eq (7)) is required since there are a few works that estimate the temporal distance from the initial state distribution to propose the curriculum goals in a temporally distant region or explore based on this temporal information [5, 37]. To reflect the temporal distance in the cost function (Eq (7)), we modify it as w ( s i , g + i ) := CE ( p pseudo ( y = 1 | s i ; g + i ); y = p pseudo ( y = 1 | g + i ; g + i )) -V π ( s 0 , ϕ ( s i )) ( ϕ ( · ) is goal space mapping) since the value function itself implicitly represents the temporal distance if we use the sparse reward or custom-defined reward similar to the sparse one. In this case, our proposed intrinsic reward outputs 1 for the desired goal and 0 for the explored states, and it works similarly to the sparse one.

We experimented with this modified curriculum learning objective ( +Value ), and the results are shown in Figure 19, 20. It shows that there is no significant difference, which supports the superiority of our method in that our method achieves state-of-the-art results without considering additional temporal distance information.

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

12

10

Ours

Ours + Value

10

12

Ours + Value

sno

25

Ours + Value

Ours

20

distance

distance

8

distance

15

6

10

2

2

5

0

0

0

0

250

500

75010001250150017502000

0

250500

75010001250150017502000

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

10

Ours + Value

Ours

0.6

0.5

Ours + Value

Ours

0.6

0.5

Ours

Ours + Value

distance

distance

0.4

0.3

0.2

0.2

2

0.1

0.1

10

0.0

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

1.0

1.0

1.0

Ours

Ours

Ours

Ours +Value

Ours + Value

Ours + Value

0.8

0.8

0.8

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250150017502000

steps (k)

0.0

0

250500

75010001250150017502000

steps (k)

0.0

0

250

500

75010001250150017502000

steps (k)

1.0

1.0

1.0

Ours

sino

Ours + Value

Ours + Value

Ours + Value

0.8

0.8

0.8

success_rate

_rate

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

500

1000

1500

2000

2500

3000

0.0

0

500

1000

1500

2000

2500

3000

00

0

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

Choice of goal candidates in training conditional classifiers. As mentioned in the main script, we also experimented with different choices of the goal candidates when we train the conditional classifiers (Eq (5)). The default setting is D G = D T , and we also experimented with D G = B∪ p + ( g ) . The results are shown in Figure 21, 22. It shows that there is no significant difference, which means we can even make the problem setting more strict by conditioning the classifier only with the visited states and the given desired outcome examples.

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

12

Dg = D

12

Dg = D

25

Dg = D

10

Dg = Bu p+(g)

10

Dg = Bu p+(g)

20

Dg = Bup+(g)

distance

8

distance

distance

15

6

10

4

2

5

0

0

0

0

250

500

75010001250150017502000

0

250

500

75010001250150017502000

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

10

DG = D

0.6

DG = D

0.6

DG = D

DG= Bu p+(g)

0.5

Dg = Bup+(g)

0.5

Dg =Bup+(g)

distance

distance

0.4

6

0.3

0.3

0.2

0.2

2

0.1

0.1

10

0.0

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

1.0

1.0

1.0

DG = D

DG = D

Dg = Dr

0.8

Dg = Bu p+(g)

0.8

Dg= Bup+(g)

0.8

Dg = Bup+(g)

success_rate

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250150017502000

0.0

0

250

500

75010001250150017502000

0.0

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

1.0

1.0

1.0

DG = D

DG = D

DG=D

0.8

Dg=Bup+(g)

0.8

DG = Bu p+(g)

0.8

DG=Bu p+(g)

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

D Training & Experiments details

D.1 Training details

Baselines. The baseline curriculum RL algorithms are trained as follows,

OUTPACE [5]: We follow the default setting in the original implementation from https: //github.com/jayLEE0301/outpace_official .

HGG [37] : We follow the default setting in the original implementation from https: //github.com/Stilwell-Git/Hindsight-Goal-Generation .

CURROT [18]: We follow the default setting in the original implementation from https: //github.com/psclklnk/currot .

PLR [16], VDS [49], ALP-GMM [36] : We follow the default setting in implementation from https://github.com/psclklnk/currot .

D2C and all the baselines are trained by SAC [13] with the sparse reward except for the OUTPACE which uses an intrinsic reward based on Wasserstein distance with a time-step metric.

Training details. We used NVIDIA A5000 GPU and AMD Ryzen Threadripper 3960X for training, and each experiment took about 1 ∼ 2 days for training. We used small noise from a uniform distribution with an environment-specific noise scale (Table 5) for augmenting the conditioned goal in Eq (5). Also, we used the mapping ϕ ( · ) that abstracts the state space into the goal space when we use the diversified conditional classifiers (i.e. f i ( ϕ ( s ); g ) . For example, ϕ ( · ) abstracts the proprioceptive states (e.g. xy position of the agent) in navigation tasks, and abstracts the object-centric states (e.g. xyz position of the object) in robotic manipulation tasks.

D.2 Environment details

Complex-Maze: The observation consists of the xy position, angle, velocity, and angular velocity of the 'point'. The action space consists of the velocity and angular velocity of the 'point'. The initial state of the agent is [0 , 0] and the desired outcome states are obtained from the default goal points [8 , 16] , [ -8 , -16] , [16 , -8] , [ -16 , 8] . The size of the map is 36 × 36 .

Medium-Maze: It is the same as the Complex-Maze environment except that the desired outcome states are obtained from the default goal points [16 , 16] , [ -16 , -16] , [16 , -16] , [ -16 , 16] .

Spiral-Maze: The observation space and actions space and initial state of the agent are the same as in the Complex-Maze environment. The desired outcome states are obtained from the default goal points [12 , 16] , [ -12 , -16] . The size of the map is 28 × 36 .

Ant Locomotion: The observation consists of the xyz position, xyz velocity, joint angle, and joint angular velocity of the 'ant'. The action space consists of the torque applied on the rotor of the 'ant'. The initial state of the agent is [0 , 0] and the desired outcome states are obtained from the default goal points [4 , 8] , [ -4 , -8] . The size of the map is 12 × 20 .

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

★

以

N

Sawyer-Peg-Push: The observation consists of the xyz position of the end-effector, the object, and the gripper's state. The action space consists of the xyz position of the end-effector and gripper open/close control. The initial state of the object is [0 . 4 , 0 . 8 , 0 . 02] and the desired outcome states are obtained from the default goal points [ -0 . 3 , 0 . 4 , 0 . 02] , [ -0 . 3 , 0 . 8 , 0 . 02] , [0 . 4 , 0 . 4 , 0 . 02] . The wall is located at the center of the table. Thus, the robot arm should detour the wall to reach the desired goal states. We referred to the metaworld [48] and EARL [40] environments.

Sawyer-Peg-Pick&Place: It is the same as the Sawyer-Peg-Push environment except that the desired outcome states are obtained from the default goal points [ -0 . 3 , 0 . 4 , 0 . 2] , [ -0 . 3 , 0 . 8 , 0 . 2] , [0 . 4 , 0 . 4 , 0 . 2] , and the wall is located at the center of the table, fully blocking a path for pushing. Thus, the robot arm should pick and move the object over the wall to reach the desired goal states.

E Algorithm

Algorithm 2 Overview of D2C algorithm

F More experimental results

F.1 Full results of the main script

We included the full results of the main script in this section. We include the visualization of the proposed curriculum goals in all environments in Figure 16. The visualization results of the Sawyer-Peg-Pick&Place are not included as it shares the same map with the Sawyer-Peg-Push environment.

(a) Ours

(b) OUTPACE

(c) HGG

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

Initial

curriculum

Final

curriculum

F.2 Additional ablation study results

Full ablation study results of the main script. We conducted ablation studies described in our main script in all environments. Figure 17 shows the average distance from the proposed curriculum goals to the desired final goal states along the training steps, and Figure 18 shows the episode success rates along the training steps. As we can see in these figures, we could obtain consistent analysis with the results in the main script in most of the environments.

distance

12

入= 1(Default)

入 = 0.1

12

head=2(Default)

head=4

15.0

Ours

Ours w/o IntrinsicReward

10

入 = 0.5

10

head=6

12.5

distance

入=2

distance

7.5

5.0

2.5

0

250

500

75010001250150017502000

0

250

500

750100012501500 1750 2000

0.0

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

17.5

入= 1(Default)

12

head=2(Default)

17.5

Ours

15.0

A = 0.1

A = 0.5

10

head=6

head=4

15.0

Ours w/o IntrinsicReward

10.0

12.5 -

A= 2

distance

8

10.0

12.5

7.5

7.5

5.0

5.0

2.5

2

2.5

0.0

0

0.0

0

250

500

75010001250150017502000

steps (k)

250

500

75010001250

steps (k)

150017502000

0

250

500

75010001250

steps (k)

150017502000

25

入 =1(Default)

入 = 0.1

head=2(Default)

head=4

25

Ours

Ours w/o IntrinsicReward

20

入 = 0.5

入= 2

distance

20

head=6

20

15

15

10

5

0

0

250

500

75010001250150017502000

steps (k)

250

500

750 1000 1250 1500 1750 2000

steps (k)

250

500

75010001250150017502000

steps (k)

10

入=0.1

入=2(Default)

10

head=2(Default)

head=4

10

Ours w/oIntrinsicReward

Ours

入= 0.5

head=6

distance

入=1

入=4

distance

distance

2

0

0

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

0.6

入 =1(Default)

0.6

head=2(Default)

0.6

Ours

0.5

A = 0.1

入 = 0.5

0.5

head=6

head=4

0.5

Ours w/o IntrinsicReward

入= 2

0.3

0.3

.3

0.2

0.2

021

0.1

0.1

0.1

0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

0.6

入 = 1(Default)

0.6

head=2(Default)

0.6

0.5

A = 0.1

A = 0.5

0.5

head=6

head=4

0.5

Ours w/o IntrinsicReward

入=2

distance

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0.0

0.0

0.0

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

0000

steps (k)

steps (k)

steps (k)

(a) Auxiliary loss weight

λ

(b) Number of prediction heads o1

(c) Intrinsic reward & Curriculum

(a) Auxiliary loss weight

λ

(b) Number of prediction heads

(c) Intrinsic reward & Curriculum

1.0

入 =1(Default)

1.0

head=2(Default)

1.0

Ours

入= 0.1

head=4

Ours w/o IntrinsicReward

0.8

A = 0.5

0.8

head=6

0.8

success_rate

入=2

success_rate

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250150017502000

00

250

500

75010001250150017502000

0.0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

A = 0.1

head=4

Ours w/o IntrinsicReward

0.8

A = 0.5

0.8

head=6

0.8

A= 2

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250150017502000

steps (k)

0.0

250

500

75010001250150017502000

steps (k)

0.0

250

500

75010001250

steps (k)

150017502000

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

success_rate

0.8

入= 0.5

入=2

入 = 0.1

success_rate

0.8

head=6

head=4

success_rate

0.8

Ours w/o IntrinsicReward

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

250

500

7501000 125015001750 2000

0.0

250

500

75010001250

150017502000

0.0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

1.0

1.0

1.0

入=2(Default)

head=2(Default)

入=0.1

head=4

Ours w/o IntrinsicReward

0.8

A=0.5

入=1

0.8

head=6

0.8

0.6

A = 4

0.6

0.6

0.4

0.2

0.2

0.2

0.0

00

0.0

0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

A = 0.1

head=4

Ours w/o IntrinsicReward

0.8

入= 2

A = 0.5

success_rate

0.8

head=6

success_rate

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

0

500

1000

steps (k)

1500

2000

2500

3000

1.0

1.0

1.0

入=1(Default)

head=2(Default)

Ours

0.8

入=0.1

入=0.5

0.8

head=6

head=4

0.8

Ours w/o IntrinsicReward

入=2

success_rate

0.6

0.4

0.4

0.2

0.2

0.2

00

0

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

500

1000

steps (k)

1500

2000

2500

3000

Curriculum learning objective type. We conduct additional experiments to validate whether reflecting the temporal distance in a curriculum learning objective (Eq (7)) is required since there are a few works that estimate the temporal distance from the initial state distribution to propose the curriculum goals in a temporally distant region or explore based on this temporal information [5, 37]. To reflect the temporal distance in the cost function (Eq (7)), we modify it as w ( s i , g + i ) := CE ( p pseudo ( y = 1 | s i ; g + i ); y = p pseudo ( y = 1 | g + i ; g + i )) -V π ( s 0 , ϕ ( s i )) ( ϕ ( · ) is goal space mapping) since the value function itself implicitly represents the temporal distance if we use the sparse reward or custom-defined reward similar to the sparse one. In this case, our proposed intrinsic reward outputs 1 for the desired goal and 0 for the explored states, and it works similarly to the sparse one.

We experimented with this modified curriculum learning objective ( +Value ), and the results are shown in Figure 19, 20. It shows that there is no significant difference, which supports the superiority of our method in that our method achieves state-of-the-art results without considering additional temporal distance information.

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

12

10

Ours

Ours + Value

10

12

Ours + Value

sno

25

Ours + Value

Ours

20

distance

distance

8

distance

15

6

10

2

2

5

0

0

0

0

250

500

75010001250150017502000

0

250500

75010001250150017502000

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

10

Ours + Value

Ours

0.6

0.5

Ours + Value

Ours

0.6

0.5

Ours

Ours + Value

distance

distance

0.4

0.3

0.2

0.2

2

0.1

0.1

10

0.0

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

1.0

1.0

1.0

Ours

Ours

Ours

Ours +Value

Ours + Value

Ours + Value

0.8

0.8

0.8

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250

500

75010001250150017502000

steps (k)

0.0

0

250500

75010001250150017502000

steps (k)

0.0

0

250

500

75010001250150017502000

steps (k)

1.0

1.0

1.0

Ours

sino

Ours + Value

Ours + Value

Ours + Value

0.8

0.8

0.8

success_rate

_rate

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

500

1000

1500

2000

2500

3000

0.0

0

500

1000

1500

2000

2500

3000

00

0

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

Choice of goal candidates in training conditional classifiers. As mentioned in the main script, we also experimented with different choices of the goal candidates when we train the conditional classifiers (Eq (5)). The default setting is D G = D T , and we also experimented with D G = B∪ p + ( g ) . The results are shown in Figure 21, 22. It shows that there is no significant difference, which means we can even make the problem setting more strict by conditioning the classifier only with the visited states and the given desired outcome examples.

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

12

Dg = D

12

Dg = Dr

25

Dg = D

10

DG = Bu p+(g)

10

Dg= Bup+(g)

20

Dg =Bup+(g)

distance

distance

distance

15

6

10

2

2

5

0

0

0

0

250

500

75010001250150017502000

0

250

500

75010001250150017502000

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

10

Dg = Dr

0.6

Dg = D

0.6

Dg = D

Dg = Bup+(g)

0.5

Dg =Bup+(g)

0.5

Dg =Bup+(g)

distance

distance

0.4

6

0.3

4

0.2

0.2

2

0.1

0.1

0

00

0

500

1000

steps (k)

1500

2000

2500

3000

0

500

1000

steps (k)

1500

2000

2500

3000

0

500

1000

steps (k)

1500

2000

2500

3000

(a) Complex-Maze

(b) Medium-Maze

(c) Spiral-Maze

(f) Sawyer Pick&Place

1.0

1.0

1.0

Dg = Dt

DG = D

Dg = Dr

0.8

Dg = Bup+(g)

0.8

Dg= Bup+(g)

0.8

Dg = Bup+(g)

success_rate

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

250500

75010001250150017502000

0.0

0

250

500

75010001250150017502000

0.0

0

250

500

75010001250150017502000

steps (k)

steps (k)

steps (k)

1.0

1.0

1.0

Dg =Dt

DG =D

DG=D

0.8

Dg=Bup+(g)

0.8

Dg =Bup+(g)

0.8

Dg =Bup+(g)

success_rate

rate

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

500

1000

1500

2000

2500

3000

0.0

0

500

1000

1500

2000

2500

3000

0.0

0

500

1000

1500

2000

2500

3000

steps (k)

steps (k)

steps (k)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel curriculum learning algorithm (D2C) in the context of reinforcement learning. Unlike previous approaches, D2C does not rely on prior knowledge of the environment's structure, including the distance measure between states. Instead, it leverages a goal-conditioned binary classifier to differentiate between visited states and desired states. When learning this classifier, D2C additionally incorporates a diversification objective to encourage the classifier to disagree on unvisited states. The paper claims that D2C could automatically generate a diverse set of curriculum goals. These goals, in turn, provide intrinsic rewards that guide the agent towards achieving the desired goal state.\n\nSTRENGTHS:\n1. The technique introduced in this paper that identify the similarities between the explored states and desired outcomes with a diversification objective appears to be novel within the domain of curriculum RL.\n2. Empirical evaluation of the proposed methods in the selected maze and robotics manipulation environment shows the significant improvement of the proposed algorithm.\n\nWEAKNESSES:\n1. The D2C algorithm relies on having access to a dataset of unlabeled target data in order to model unvisited states. However, this assumption can be a significant limitation of the algorithm. In the experiments conducted in this paper, the author was able to overcome this limitation by uniformly sampling states from the upper and lower bounds of the state space. Nevertheless, obtaining such a collection of (legal) states within the state space of agents or robots is often a challenging task. For instance, if the agent receives pixel observations or if the legal state space of a robotic arm is a constrained subset of the entire space defined by the cartesian product each dimension, the assumption of sampling from the legal state space to obtain the unlabeled target dataset may not be applicable. In general, this assumption holds true only in specific environments or tasks.\n2. The current version of the proposed algorithm has some fundamental limitations. For example, the proposed algorithm could be prone to over-explore. Also, how the diversification process helps the overall curriculum proposal process is not clear to me. Please refer to my questiona in the next section.\n\nI am happy to raise the score if my questions are addressed by the author.\n\nQUESTIONS:\n1. Regarding the diversification: It seems that when optimizing the objective of curriculum distribution, the uncertainty/disagreement between different classification heads are not in the objective. Equation 4 simply minimizes the cross entropy between the averaged probability of the proposed curriculum state being a goal state and the average probability of the true goal states. Where does the uncertainty of the unexplored region play a role in the overall objective of the algorithm?\n2. The problem of over-exploration: Based on the learning objective of D2C, suppose the agent is given a maze environment similar to the one shown in Figure 2. But if the goal distribution only includes the top one, then is it true that this proposed algorithm would still explore both the top and bottom part of the maze?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper proposes a new reinforcement learning algorithm called Diversify for Disagreement & Conquer (D2C). 
The idea of the algorithm is to divide the search space and let conditional classifiers explore these subspaces. 
With a set of experiments, the authors verify the effectiveness of D2C compared to various other approaches. 
As new to the field, I can not judge the relevance of the contribution for reinforcement learning.

STRENGTHS:
As a person new to the field of RF, this paper is well written since it introduces necessary concepts at an appropriate depth, and the idea and motivation are intuitive. 
In addition, the approach is geometric flexible, shows its effectiveness in a range of experiments, and the author provides an in-depth ablation study of relevant hyperparameters.

WEAKNESSES:
Again I need to admit that I am entirely new to this field. At the same time, the other baseline experiments all do explore the search undirected. Is it a fair comparison when your approach can explore many directions situationally?

QUESTIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper propose training multiple classifiers on a data set of "desired" and "reached but undesired" states, motivating these classifiers to be different off distribution, and using this divergence as a metric for exploration.

STRENGTHS:
The approach is simple and easy to implement in many domains.

WEAKNESSES:
Since the approach is very similar to model-disagreement, I would expect a simple comparison to such an approach like exploration via disagreement.  Moreover, given that random-network distillation works so well for exploration, its unclear if the objective of the classifier here is really too important.  It may only be important that the metric has to do with the disagreement between two networks, and it may not matter what they are disagreeing about.

QUESTIONS:
What advantage does this approach have over generic model-disagreement, or random network distillation?

LIMITATIONS:
Given the number of existing approaches, and an unclear understanding of their relative limitations, it is unclear when this approach should be preferred over the others in a new domain.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

comments: SUMMARY:
Focusing on goal-conditioned RL, the author propose Diversify for Disagreement & Conquer (D2C) to form outcome-directed exploration by generating a sequence of curriculum goals, given some desired outcome examples. By ensuring multiple classifiers disagree on unseen states, D2C uses bipartite matching to create curriculum goals, which are interpolated between the initial state distribution and arbitrarily distributed desired outcome states for enabling the agent to conquer the unexplored region. The experiments demonstrate that D2C surpasses previous curriculum RL methods in goal-conditioned RL experiments.

STRENGTHS:
- I appreciate the idea to use the diversifying functions for disagreement on underspecified data and to quantify the similarity between the visited states and desired outcome states. To use the pseudo probability as the intrinsic reward is also straightforward. The proposed method is simple but effective. 

- The paper is well written.

WEAKNESSES:
- It is not clear for me how to interpolate the curriculum distribution from the initial state distribution through minimizing Eq 4.
- Some related work and its comparison missing. Diversity has been considered in Curriculum RL. For example, Curriculum-guided Hindsight Experience Replay (NeurIPS 2019).
- It is not open sourced.

QUESTIONS:
- Please clarify the interpolation process as referenced in weakness.

- Figure 6 can be better if the labels do not cover the curves. The font sizes in the figures are too small to read.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed Diversify for Disagreement & Conquer (D2C), which is an outcome-directed curriculum RL method. D2C performs diversification of the goal-conditional classifiers to identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in a simple and intuitive way. D2C then employs bipartite matching to define a curriculum learning objective that produces a sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region.\n\nSTRENGTHS:\nD2C performs a classifier diversification process to distinguish the unexplored region from the explored area and desired outcome example.\n\nD2C conquers the unexplored region by proposing the curriculum goal and the shaped intrinsic reward.\n\nD2C enables the agent to automatically progress toward the desired outcome states without prior knowledge of the environment.\n\nWEAKNESSES:\n1. The writing of the paper needs some polishing to improve its readability.\n\n2. The settings of the experiments, e.g. the map types and sizes, are not challenging to show the significance of the proposed model.\n\n3. The complexity analysis is missing. Empirically, it takes 1 - 2 days to train a solution for a 36 * 36 map as described in Appendix.\n\nQUESTIONS:\n1. OUTPACE is the latest model in comparison, but it performs (near) the worst in all tasks (Fig.4 & 5). I had a quick check of OUTPACE paper, and it reports completely different results. Please explain how this could happen.\n\n2. What is the complexity of the proposed model?\n\nLIMITATIONS:\nThe paper has a good summary of its limitations and potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}