Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation
Authors: Peide Huang
Mengdi Xu
Jiacheng Zhu
Laixi Shi
Fei Fang
Ding Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
curriculum learning
domain adaptation

Abstract: Curriculum Reinforcement Learning (CRL) aims to create a sequence of tasks, starting from easy ones and gradually learning towards difficult tasks. In this work, we focus on the idea of framing CRL as interpolations between a source (auxiliary) and a target task distribution. Although existing studies have shown the great potential of this idea, it remains unclear how to formally quantify and generate the movement between task distributions. Inspired by the insights from gradual domain adaptation in semi-supervised learning, we create a natural curriculum by breaking down the potentially large task distributional shift in CRL into smaller shifts. We propose GRADIENT which formulates CRL as an optimal transport problem with a tailored distance metric between tasks. Specifically, we generate a sequence of task distributions as a geodesic interpolation between the source and target distributions, which are actually the Wasserstein barycenter. Different from many existing methods, our algorithm considers a task-dependent contextual distance metric and is capable of handling nonparametric distributions in both continuous and discrete context settings. In addition, we theoretically show that GRADIENT enables smooth transfer between subsequent stages in the curriculum under certain conditions. We conduct extensive experiments in locomotion and manipulation tasks and show that our proposed GRADIENT achieves higher performance than baselines in terms of learning efficiency and asymptotic performance.

Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation
Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, Ding Zhao Carnegie Mellon University Pittsburgh, PA 15213 {peideh, mengdixu, jzhu4, laixis, feifang, dingzhao Haaddrew cmu cdu
Abstract
Curriculum Reinforcement Learning (CRL) aims to create a sequence of tasks, starting from easy we focus on the idea of framing CRI. as interpolations between a source (auxiliary) and targer task distribution. Although existing studies have shown the great potential of this idea, it remains unclear how to formally quantify and generate the movement between task distributions. Inspired by the insights from gradual domain adaptation in semi-supervised learning, we create a natural curriculum by breaking down the potentially large task distributional shift in CRL into smaller shifts. We propose GRADIENT, which formulates CRL as an optimal transport problem with tailored distance metric between tasks. Specifically, task distributions geodesic interpolation (i.e., Wasserstein barycenter) between the source and target distributions. Different from many existing methods, our algorithm considers a dask-dependent contextual distance metric and is capable handling nonparametric distributions in both continuous and discrete context settings. In addition, thcoretically show that GRADIENT enables smooth transfer between subsequent stages the curriculum under certain conditions. We conduct extensive experiments in locomotion and manipulation tasks and show that our proposed GRADIENT achieves higher performance than baselines in terms learning efficiency and asymptotic performance.
and gradually learning towards difficult tasks. In this work,
generate a sequence of
Introduction
Learning (RL) [1] has demonstrated great potential in solving complex decision- making tasks (2), including but not limited to video games [3] chess [4], and robotic manipulation [5]. Among them, various prior works highlight daunting challenges resulting from sparse rewards. For example, in the maze navigation task, since the agent needs to navigate from the initial position the goal to receive a positive reward, the task requires large amount of randomized exploration. One solution to address this issue is Curriculum Reinforcement Learning (CRL) 7 of which the objective is to create sequence of environments facilitate the learning of difficult tasks.
Reinforcement
Although there are different interpretations of CRL, we focus on the one that views a curriculum as a sequence of task distributions that interpolate between source (auxiliary) task distribution and targer task distribution 8 9 10 11
12 [3] This interpretation allows more general presentations of tasks and wider range of objectives such generalization (uniform distribution over the task collection) or learning to complete difficult tasks (a subset of the task collection). Again we use example. Given fixed maze layout and goal, task is then defined by start position, and the task distribution a categorical distribution over all possible start positions. With a target distribution putting mass over start positions far away from the goal position, natural
the maze navigation
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Source 0.00
Target a= 1.00
0.25
0.50
0.75
SD
Probabbiity bibubbiitt
(a) Linear interpolation
Source 0.00
Target = 1.00
0.25
0.50
0.75
E
20
0.3
0.0
(b) GRADIENT interpolation (ours)
Figure I: Intermediate task distributions generated by (a) linear interpolation and (b) method for the maze navigation task. The green cell represents the goal. The red cells represent the initial positions (the darker the color is, the higher the probability is). The first column shows the source task distribution and the last column shows the target task distribution. In (a), linear interpolations do not cover cells where the source and the target have probability, which hardly benefits the learning. In contrast, (b) GRADIENT interpolations creates a curriculum that gradually morphs from the source to the target, covering tasks of intermediate difficulty and improving the learning efficiency.
curriculum accelerate the learning process putting start positions close to the goal position first, and gradually moving them towards the target distribution.
However, most of the existing methods, that interpret the curriculum shifting distributions, Kullback-Leibler (KL) divergence to measure the distance between distributions. This setting imposes several restrictions. First, due either problem formulations or the computational feasibility, existing methods often require the distribution to be parameterized, e.g., Gaussian 8 9 10 [3]. which limits the usage in practice. Second, most of the existing algorithms using KL divergence implicitly assume an le Euclidean space which ignores the manifold structure when parameterizing RL environments
[14]
In light of the aforementioned issues with the existing CRL method, we propose GRADIENT, an algorithm that creates a sequence of task distributions gradually morphing from the source to the target distribution using optimal transport (OT). GRADIENT approaches CRL from a gradual domain adaptation (GDA) perspective, breaking the potentially large domain shift between the source and the target into smaller shifts to enable efficient and smooth policy transfer. In this work, we first define distance metric between individual tasks. Then can find series of task distributions that interpolate between the easy and the difficult task distribution by computing the Wasserstein barycenter. GRADIENT is able deal with both discrete and continuous environment parameter spaces, and nonparametric distributions (represented either by explicit categorical distributions or implicit empirical distributions of particles). Under some conditions [5]] GRADIENT provably ensures smooth adaptation from one stage to the next.
We summarize main contributions follows:
1. We propose GRADIENT, novel CRL framework based on optimal transport to generate gradually morphing intermediate task distributions. As a result, GRADIENT requires little effort to transfer between subsequent stages and therefore improves the learning efficiency towards difficult tasks.
2. We develop acontextual-distaann barycenters as intermediate task distributions. Our proposed method able to deal with both continuous and discrete context spaces as well nonparametric distributions. We also prove the theoretical bound of policy transfer performance which leads to practical insights.
to measure the task similarity and compute the Wasserstein
3. We demonstrate empirically that GRADIENT has stronger learning efficiency and asymptotic performance in a wide range of locomotion and manipulation tasks when compared with state-of- the-art CRL baselines.

---

2 Related Work
Curriculum reinforcement learning. Curriculum reinforcement learning (CRL) 6 16 the generation of training environments for RL agents. There are several objectives in CRL: improving learning efficiency towards difficult tasks ttime-to-threshodlh maximum return (asymptotic perfor- mance), or transfer policies solve unseen tasks (generalization) From a domain randomization perspective, Active Domain Randomization [5] [7 uses curricula to diversify the physical parameters of the simulator to facilitate the generalization in sim-to-real transfer. From
focuses on
theeethee perspective, adversarial training is also developed to improve the robustness of RL agents in unseen
environments [18] 19 20 21. From an intrinsic motivation perspective, methods have been proposed
to create curricula even in the absence of target task to be accomplished [2 13 23
CRL as an interpolation of distributions. In this work, we focus on another stream of works that interprets CRL as an explicit interpolation between an auxiliary task distribution and a difficult task distribution 88 9 TO I. Self-Paced Reinforcement Learning (SPRL) [8] is proposed to generate intermediate distributions by measuring the task distribution similarity using Kullback-Leibler (KL) divergence. However, as we will show in this paper, KL divergence brings several shortcomings that may impede the usage of the algorithms. First, although the formulation of [8 | I0 11] does
not restrict the distribution class, the algorithm realization requires the explicit computation of KL divergence, which is analytically tractable only for a restricted family of distributions. Second, using KL. divergence implicitly assumes le Euclidean space which ignores the manifold structure when parameterizing RL environments. In this work, we use Wasserstein distance instead of KL divergence to measure the distance between distributions. Unlike KL divergence, Wasserstein distance considers the ground metric information and opens up a wide variety of task distance measures.
CRL using Optimal Transport. Hindsight Goal Generation (HGG) [24] aims to solve the poor exploration problem in Hindsight Experience Replay (HER). HGG computes 2- 2-Wasserstein barycenter approximately to guide the hindsight goals towards the target distribution in an implicit curriculum. Concurrent to our work, CURROT 22 also uses optimal transport to generate intermediate tasks explicitly. CURROT formulates CRL as a constrained optimization problem with 2- Wasserstein distance measure the distance between distributions. The main difference that we propose task dependent contextual distance metrics and directly treat the interpolation as the geodesic from the source to the target distribution.
Gradual domain adaptation in semi-supervised learning Gradual domain adaptation (GDA) 226 27 28
considers the problem of transferring classifier trained in source domain with labeled data, to a target domain with unlabelled data. GDA solves this problem by designing a sequence of learning tasks. The classifier is retrained with the pseudolabels created by the classifier from the last stage in the sequence. Most of the existing literature assumes that there exist intermediate domains. However, there are few works aiming to tackle the problem when intermediate domains, or the index (i.e., stage in the curriculum), are not readily available. A coarse-to-fine framework is proposed to sort and index intermediate domain data [3] Another study proposes to create virtual samples from intermediate distributions by interpolating representations of examples from source and target domains and suggests using the optimal transport map create interpolated data in semi-supervised learning (32) It is demonstrated theoretically in (27] that the optimal path of samples is the geodesic interpolation defined by the optimal transport map. Our work is inspired by the divide and conquer paradigm in GDA and also uses the geodesic as our curriculum plan (although in a different learning paradigm).
29
30 31
32 3
3 Preliminary
3.1 Contextual Markov Decision Process
A contextual Markov decision process (CMDP) extends the standard single-task MDP to multi- task setting. In this work, we consider discounted infinite-horizon CMDPs, represented by tuple M = A. R, P. Po- P. 7). Here, S is the state space, C is the context space, A is the action space, R is the context-dependent transition function, Po C + A(S) is the context-dependent initial state distribution, p E A(C) is the context distribution and y E (0, 1) is the discount factor. Note that goal-conditioned reinforcement learning [2] can be considered as special case of the CMDP.
A C + IR is the context-dependent reward function, P: S x x C + A(S)
3

---

To sample trajectory T : {st, Qe., re re in CMDPs, the context ~ p is randomly generated by the environment the beginning step I, the agent follows a policy я to select an action az ~ (88,, c) and receives reward R(s,, at, c). Then the environment transits to the next state 8t+1 ~ P(- | 8t, at, c), Contextual reinforcement learning naturally extends the original RL objective to include the context distribution p. To find the optimal policy, we need to solve the following optimization problem:
each episode. With the initial state Sp ~ Po( c)2 at each time
max (p) max E
R(st, ar, c~ P3
3.2 Optimal Transport
Wasserstein distance. The Kantorovich problem [4 a classic problem in optimal transport 35
aims to find the optimal coupling l which minimizes the transportation cost between measures μr, M(C). Therefore, the Wasserstein distance defines the distance between probability distributions:
inf / d(c,, Cz) dd(ce, ce), subject to θ = {0 7 μ. 7 v} (E@(U.))
Wa(u,v)
(2)
where C' is the support space, (uu, v) is the set of all couplings between μt and v, d(.,-) CxC + Rz0 is a distance function, - is the projection from C x C onto C, and Te generally denotes the push-
forward measure of P by map T' [36 37 38
This optimization is well-defined and the optimal
always exists under mild conditions 335
To construct curriculum that allows the agent efficiently solve the difficult task distribution, we follow the Wasserstein geodesic (39), the shortest path (27) under the Wasserstein distance between the source and the target distributions. While the original Wasserstein barycenter 440 41 problem focuses on the Frèchet mean of multiple probability measures in a space endowed with the Wasserstein metric, we consider only two distribu- tions μu and μl. The set of barycenters between μ4 and μ1 the geodesic curve given by McCann's
Wasserstein Geodesic and Wasserstein Barycenter.
interpolation 42 43 Thus, the interpolation between two given distributions fi and μr is defined as:
Va = arg min(1 a)Wa(po, v/) + aWa(y's P1),
1/2
where each E [0, 1] specifies one unique interpolation distribution on the geodesic. While the
computational cost of Wasserstein distance objective in Equation (2) could be potential obstacle, we can follow the entropic optimal transport and utilize the celebrated Sinkhorn's 's algorithm [44). Moreover, we can adopt a smoothing bias to solve for scalable debiased Sinkhorn barycenters
4 Curriculum Reinforcement Learning using Optimal Transport
We first formulate the curriculum generation as an interpolation problem between probability distributions in Section 4.1 Then we propose a distance measure between contexts Section 4.2
in order to compute the Wasserstein barycenter. Next, we show our main algorithm, GRADIENT,
in Section 4.3 and an associated theorem that provides practical insights in Section 4.4
4.1 Problem Formulation
Formally, given a target task distribution v(c) E ACC), we aim to automatically generate a curriculum of task distributions, pu(c), (c). (c)...., ,Pk (c) with K stages, that enables the agent to gradually adapt
from the auxiliary task distribution f(c) to the target v(c), i.c., Po(c) + p(c) and px (c) + v(cc.
the context space C is discrete (sometimes called fixed-support in OT 35 with cardinality [C]. the task distribution represented by a categorical distribution, e.g., p(c) = fp(c)).((2)) .... p(c
where p(c;) 2 0, P(G)) = 1. Whereas the context space C is continuous (sometimes called free-support in OT), the task distribution is then approximated by set of particles sampled from the distribution, e.g., p(c) R p(c) = - E'= de,, (c), where 62,, (c) is a Dirac delta at C.p. This highlights the capability of dealing with nonparametric distributions in our formulation and algorithms,
in contrast to existing algorithms [8] 13 3 9 10 M.]
4

---

4.2 Contextual Distance Metries
To define the distance between contexts, we start by defining the distance between states. Bisimulation metries [66 47 48 measure states' behaviorally equivalence by defining recursive relationship between states. Recent literature in RL 49 [4] uses the bisimulation concept to train state encoders that facilitate multi-tasking and generalization. (50) uses this bisimulation concept to enforce the policy to visit state-action pairs close to the support of logged transitions in offline RI..
However, the bisimulation metric is inherently "pessimistic" because considers equivalence under all actions, even including the actions that never lead to positive outcomes for the agent [48]. To address this issue, [48] proposes on-policy n-bisimulation that considers the dynamics induced by the policy π. Similarly, we extend this notion to the CMDP settings and define the contextual r-bisimulation metric:
&e Pates (s (5), |
R 84.0 - R
K5.Ez1 + ?Waa(P s.,e P Pgj.g))
where Rg := E. T(a s)Rss, a, c) and P == E. P(. s, a, c) s). With the definition of contextual r-bismmulation metric, we are ready to propose the r-contextual-ddstance to measure the distance between two contexts:
Definition 4.1 contextualldistanee Given CMDP M = (,, the distance between two contexts d' (G,,0 0j) under the policy n defined as
p.
d" (ci, Cj) (es.
Conceptually, the r-contextual-diitanee approximately measures the performance difference be- tween two contexts e; and çj under и. The algorithm to compute simplified version of this metric under some conditions is detailed in the Appendix C.1 Note that is difficult to compute this metric precisely in general. In practice, we can design and compute some surrogate contextual distance metrics, depending on the specific tasks. There are situations where it is reasonable use l2 as surrogate metric when the contextual distance resembles the Euclidean space, such as in some condiiondt continuous environments (24). addition, although the contextual distance is not a strict metric, we can still use it as a ground metric in the OT computation. With the concepts of contextual distance metric, we now introduce the algorithms to generate intermediate task distribution to enable the agent to gradually transfer from the source to the target task distribution.
4.3 Algorithms
We present our main algorithm in Algorithm I We introduce an interpolation factor Ac to decide the difference between two subsequent task distributions the curriculum. Smaller Aa means a smaller difference between subsequent task distributions. In the algorithm, we treat Ao as constant for simplicity but in effect, it can be scheduled or even adaptive, which we leave future work. Note that the Aa is analogical to the KL divergence constraint in 99 10 8. The main difference is that use Wasserstein distance instead of KL divergence.
At the beginning of each stage in the curriculum, we add Aa to the previous a (starting from 0). Then we pass the a into the function ComputeBayycenter intermediate task distribution. Note that when a = 0 and 1, the generated distribution are the source and target distribution respectively. In the ComputeBarycentee method differs depending on whether the context is discrete or continuous. After generating the intermediate task distribution, we optimize the agent this task distribution until the accumulative return G reaches the threshold G. Then the curriculum enters the next stage and repeats the process until a = 1. In other words, the path of the intermediate task distributions is the OT geodesic on the manifold defined by Wasserstein distance.
(Algorithm 2 to generate the
function, the computation
4.4 Theoretical Analysis
The proposed algorithm benefits from breaking the difficulty of learning in the target domain into multiple small challenges by designing a sequence of K stages that gradually morph towards the target. This motivates us to theoretically answer the following question:
Can achieve smooth transfer to new stage based on the optimal policy of the previous stage?
5

---

Algorithm 1: GRAdual Domain adaptation for curriculum EennntreEment learNing via optimal Transport (GRADIENT)
Algorithm 2: ComputeBarycenter
Input: Source and target μ,, interpolation
constant a, distance metric d.
Input: Source task distribution pa(c), target task distribution v(c), interpolation factor Aa, distance metric d, reward threshold G,
W1 f a, W2 < 1 a, (61, (2), < 1;
if discrete then
A1 t j((c), X2 v(c)
maximum number of stages K.
Cost matrix:
C: d(:,)) : C C + R Rn.XRi
Initialize the agent policy æ for k in 0, 2, .... K do a < min(k Aa, 1); p(c)
else
A = + L Csis x2 = - Ctji
the
Tie
Cost matrix C: Cij = d(csi: Cty);
Computeaarycenter(uvooo d);
K = exp (-C/c) // Sinkhorn param
do
while not converge do
G + Optimize in the task distribution p(c) and return training reward:
d1 + (>1/ (n/Kb)) ) 02 t A2/Kbb);
p + O I (K am "
3m)"
b1 - (p/K K 01 b2 - (p/K 02);
until G> G:
L V @ (p/Kl);
Output: Agent policy π
Output: p
Towards this, motivated by the maze navigation example shown in Figure we focus on the collection of CMDPs which obeys the following assumption:
| throughout this section,
Assumption 4.1 (Consistency and homogeneity) The initial state distribution consistent with the associated context distribution, namely Pu(s| c) = 1(s c). In addition. the transition kernels and rewards of the CMDP are homogeneous with respect to the context. Specifically,
P(s' 88 a, c.) = P(s' (8,a,c)) a(),,,c)) = R(s,a,c,).
Vei,e E C, E,
To continue, for k-th stage associated with the context distribution Pk E A(C), denote the optimal policy in k-th stage as _ = arg max, V (ps). Armed with the above notations and assumptions, we are positioned to introduce our main theorem.
Theorem 4.1 Let m be problem-dependen positive constant. Consider any MDP under the assumption 4.1 and the situations further specified in Appendix For any two subsequent stages k 1 []. the optimal policies Te. Tk41 obey
V (pk+1) - (p S mWa- (pke, Pk+1).
V
(6)
Here, d' is the distance metric defined on the metric space (C,d*) which obeys (U(C) denote the uniform distribution over C)
n" max V (U(C)),
d'(,)==-' (s.).
The proof can be found in Appendix A.4 In addition, the following corollary can be directly verified by Theorem 4.1
Corollary 4.1.1 By constant speed geodesics (7.2 of (57), the stages generated by GRADIENT obey VT (px+1) S mAaWa. (p,1).
v (
indicates that with tailored stages by GRADIENT (sufficiently small Aa), it is easy adapt the learned optimal policy πīद in the current stage to the optimal policy Tb in the next stage defined by pk+1, since the performance gap is small. In particular, after we obtain the optimal policy nī associated with the k-th stage in the curriculum and transfer this policy to the next stage, the performance gap between it and the optimal policy in the (k: + 11t stage can be controlled by the Wasserstein distance between distributions and therefore a fraction of the Wasserstein distance between the source and the target (which can be seen as a constant), By properly controlling the Wasserstein distance using GRADIENT. we can achieve gradual domain adaptation from the source to the target task distribution.
Remark 4.1 Theorem 4.1 combined with Corollary 4.1.1
subsequent context

---

Experiments
Evaluation Metric. To quantify the benefit of CRL, we compare the learning progress an agent evaluated on the larget task distribution ! More concretely, the metric we use is time-to- toetheshld [6]. which shows how fast an agent can learn policy that achieves satisfying return the target task it transfers knowledge. We also compare the asymptotic performance of the agent. For the learner, we use the SAC [52 and PPO 53 implementations provided
24
in the Stabl baabllee eBasel i nee library
[54] For the optimal
tranport computation, we use POT
55
The first Maze environment is to inves- tigate the performance of GRADIENT and the effect of Aa. Maze has a discrete context space and therefore the exact context distance metric is available. The next two environments are to investigate the benefits of GRADI I- ENT when the context space is continuous, and we use l2 distance as surrogate contextual distance. In the Point- Mass environment, the source and the target distribution are both Gaussian distributions, which are in line with the settings of existing algorithms. In the FetchPush environment, we want to investigate the case where the source and the target are not Gaussian distributions.
Environments.
Figure 2: Environment visualizations. For the Maze (upper-left), the context is the initial state. For the PointMass (lower-left), the context is the position and width of the gap on the wall. For the FetchPush, the context is the goal position on a 2D surface.
We benchmark our algorithm with six baselines, including six CRL algorithms:
1. No- Moo-Curriullin um: sampling contexts according to the target task distribution.
2. Ramain-Randomizaa zationiiaiiion uniformly sampling the contexts from the context space.
3. HER [56] implicit CRL method by relabelling trajectories.
4. Linear- L.inear-Itterpolatta
sampling contexts based weighted average between two distributions, i... (1 a)μ + av.
5. ALP-GMM [33 state-of-the-art intrinsically motivated CRL algorithm that maximizes the
absolute learning progress of the agent.
6. Goal-GAN 12] using GAN to generate contexts of intermediate difficulties and automatically
explore the goal space.
7. SPDL & and WB-SPDL [57]: interpreting the curriculum generation variational inference problem to progressively approach the target task distribution. WB-SPDL extends SPDL by using Wasserstein distance instead of KL divergence.
5.1 Can GRADIENT Handle Discrete Contexts and What is the Effect of Aa?
In the Maze task, we fix the layout of the maze and the goal. The state and action spaces are discrete. The action space includes movements in four directions. The context is the initial position, and therefore the context space overlaps with the state space. The observation is the flattened value representation of the maze, including the goal, the current position, and the layout.
We visualize the optimal π r--contextall-distame metric in Figure Ba (computed using the optimal policy and normalized to make the maximum value be 1). The distance metric can be represented by symmetrical matrix. We then generate curricula using GRAD IENT with Aa 0.2, 0.1, 0.05. From Figure 3bb] our proposed method outperforms the baselines significantly in terms of the -threshll evaluated in the target distribution. With smaller Aa, GRAD IENT lears slightly slower; nevertheless, all choices achieve good performances. In this simple environment, Domain- Randomi zation on also improves the learning efficiency compared with No-Curricull (which cannot learn at all). Interestingly, Doma i n-Ra Rn-andomi z ion even outperforms the tinear-Interge at on, which demonstrates the caveat of bad intermediate task distribution.
'Code available under http://gittuucccmm//eeettuaaaeae

---

1.0
GRADIENT,
GRADIENT, de 0.1
10
GRADIENT, 0.05
No Curriculum
20
0.66
Lineer Interpolation
Domain Randomicatton
8
Excluation
80
0.2
8
20000
<DDDE
60COD
80000
100000
Enviromment SAMD
(a) secootexttulddisttatett
(b) Evaluation reward in target
Figure 3: The w-contextull-dittanee and evaluation results in Maze. The shaded area represents the standard deviation. We use a PPO learner and show that GRADIENT converges faster than baselines.
5.2 How Does GRADIENT Perform When the Distributions Are Gaussian?
In PointMass the agent needs navigate pointmass through wall with small gap at an off-center position to reach the goal. The context is 2-dimension vector representing the width and position the gap on the wall. Following the setting in the SPDL paper [8] the target distribution an isotropic Gaussian distribution centered at 22.5, 0.5 with a negligible variance [4 10 3.75 x 10 10-3) (which is effectively a point as shown in Figure Sa The source distribution is an isotropic Gaussian
distribution centered at [0, 4.25] with variance of [2, 1.875).
observe that both GRADIENT and SPDL significantly outperform other baselines terms of the time-to-threshold and achieve the same asymptotic performance. It is reasonable since these methods capable specifying the target distribution to guide curriculum generation. We present the curricula visualization of both methods in Figure Saaand show that the intermediate task distributions gradually move from the source the target distribution. However, the other three baselines lack the flexibility to leverage the target distribution. This experiment highlights the importance of being able specify the target distribution.
From Figure 4a
5.3 How Does GRADIENT Perform When the Distributions Are NOT Gaussian?
In FetchPush (58), the objective is the gripper to push the box a goal position. The observation space a 28-dimension vector, including information about the goal. The context is 2-dimension vector representing the goal position on surface. The target distribution is uniform distribution over the cireunference of half-cirele (Figure 5b). The source distribution is uniform distribution over square region centered at the box position, excluding the region within a certain radius of the object. We use this experiment to highlight the importance of the capability to handle arbitrary distributions rather than only the parametric Gaussian distributions. Since SPDL can only
GRADIENT
No Curiculum
Domain Randomization
ALP-GMM
Goal-GAN
SPDL
HER
SPOL
1.0
80
Episodic Reward
Success Rate
0.6
80
Environment Step
Environment Step
(a) PointMass
FetchPush
Figure 4: Evaluation performance the target task distribution (with rolling average of 10) for PointMass and FetchPush. In PointMass, where both the source and target distributions are defined by a parametric Gaussian, GRAD IENT converges faster than baselines. In FetchPush where the target task distribution NOT be accurately approximated by Gaussian, GRADIENT converges faster and has a better asymptotic performance. We use SAC as the learner for both environments.

---

Target
0.00
0.25
0.50
0.75
1.00
7.5
7.5
7.5
7.5
7.5
Witth
5.0
5.0
5.0
5.0
5.0
2.5
2.5
2.5
2.5
2.5
0.0
0.0
0.0
0.0
0.0
Position
Position
Position
Position
Position
(a) PointMass context visualization
2
0.00
-
0.25
- 0.50
a
0.75
a 1.00
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
1.2
1.2
X
(b) FetchPush context visualization
X
X
Figure 5: Visualizations of context distributions and curricula in PointMass and FetchPush. We present GRADIENTE curricula corresponding to a = 0.00, 0.25, 0.50, 0.75, 1.00 together with SPDL's contexts from representative stages. For PointMass, the contexts of SPDL are taken from environment steps at 10k, 50k, 100k, 200k, and 300k. For FetchPush, the contexts of SPDL are taken from environment steps at 5k, 50k, 100k, 250k, and 500k.
deal with parametric Gaussian distribution, we first fit the target and the source distribution with two Gaussian distributions and feed them into the baseline algorithms.
From Figure 4b] GRADIENT outperforms the baseline methods in terms of the time-to- -hreeshehhld in the target distribution. For ALM and Goal-GAM since these two methods do not consider the target distribution, they achieve lower performance than the target-aware methods such as SPDL and GRADIENT, We further compare these two target-aware methods and find that GRADTENT outperforms the SPDI. in terms of learning efficiency and asymptotic performance. The reason can be explained by visualizing the context distribution in Figure 5b SPDL indeed shows the behavior of moving the context distribution towards the target. However, since the target distribution cannot be fit with Gaussian well, many sampled contexts are in fact far away from the target distribution, which may not improve the learning efficiency in the target domain. Although Domain in Randomi zat on learns slightly lower than the CRL methods, it achieves even higher asymptotic performance than ALP GMMM and Goal-GAM The potential reason could be that it does not bias towards a certain area while it is possible for ALP-GMM and Goal-GAN.
5.4 Beyond Euclidean Distance Metrie: Using Distance Embedding for Continuous Spaces
In practice, it is not always appropriate to use the Euclidean distance as the surrogate contextual distance. Though there are many existing OT algorithms based on Euclidean distance metric, solving for free-support Wasserstein barycenters on non oun-Euelieean space is non-trivial [59 60].
One solution to bypass this issue is to embed contexts to latent space such that the Euclidean interpolation the latent space approximates the interpolation in the original non-Fuclidean space after reconstruction.
For the fixed-support setting, Deep Wasserstein Embedding (DWE) pro- posed in [61] uses Siamese networks [62] to learn mapping function such that the Euclidean distance in the latent space approximates the Wasser- stein distance in the original space. Motivated by this idea, we instead develop an embedding mechanism for free supports, i.e., continuous con- texts. We leverage encoder to embed contexts to latent space in which Euclidean distance mimics the original distance metric. We train the encoder and decoder based on reconstruction loss and distance embed- ding loss. We include the details of distance embedding in Algorithm and modified version of GRAD IENT in Algorithm 4 in Appendix C.2
Figure 6: U-Maze en- vironment. The orange and red balls represent the agent and the goal, respectively. The con- text the goal position.

---

123
13
50-
60
50
50-
23
23
23
-1
-10
-55
23
305 125
-50 -25
23
25
75 18 125
-50 -5
50
75
10.0 120
(a) Stage
(b) Stage 5
Stage
Figure 7: U-Maze interpolation results. The original source and target distributions two Gaussian distributions centered at (0, 0j and 80. The color of the heat map represents the estimate of J (c). The colored circles represent the Wasserstein interpolations. More specifically, the circles with cyan to purple color represent the potential interpolation results from the current source to target with 0.0, 0.1, 0.2, ..... 1.0. The yellow circles highlight the selected barycenter corresponding the current stage.
We show an example of using Algorithm Here we consider classical U-shaped maze with continuous spaces in F We assume that the source and target distributions are two Gaussian distributions at the two ends of the maze. Due to the existence of the obstacle in the middle, it is not appropriate to use the l2 distance as the contextual distance. In this case, we use df (c1, 02) := (ci) (c2) as surrogate metric for the crontextual-distanee where (c) represents the expected episodic reward of context c under the policy π.
The interpolation results are shown in Figure 7 At the first few stages, since the agent does not have a good policy or a good estimate of the episodic reward, the potential geodesic interpolation tries to go through the obstacle in the middle. Fortunately, since is small at first, the selected intermediate barycenters (yellow circles) are relatively close to the source distribution, so GRAD will not generate unreasonable tasks. As the learning progressing, the estimates become better and better and therefore produce better interpolation results.
Conclusion and Limitation
In this work, we propose novel curriculum reinforcement learning framework optimal transport problem. We also develop a distance metric to measure the difference between tasks. We show that our proposed method is able to achieve smooth transfer between subsequent stages in the generated curriculum. Finally, the empirical results highlight the three features of our methods: considering metric information, being aware of the target, and being able to handle nonparametric distributions. For future work. it could be interesting to investigate how to use the intrinsic curiosity driven type of automatically generate the source distribution. In addition, wider choices of the context distance can be explored to provide more insights into the OT OT-based CRL methods. From a theoretical perspective, OT also provides powerful mathematical tools for future researches to investigate rigorously why CRL methods can help reduce the sample complexity.
method as in 2 1 23
There are main limitations of our proposed GRADIENT. First, due the limitation of OT computation complexity, we only examine the settings that are either discrete or low-dimensional. Such experiment design choice is common in the related works and baselines [3] 12 It comes from both the fact of many tasks can be represented by a low-dimensional vector, such as goal positioned or physical parameters of robots. We believe how deal with high-dimensional contexts, such as images, is a promising future direction. Second, our analysis is conducted based on restrictive subset of contextual MDPs (Assumption [.TTehere the context controls only the initial state distribution. It would be beneficial to relax this assumption by imposing assumptions on the specific manner in which the context controls the dynamics or reward.
Acknowledgments
We gratefully acknowledge support from the National Science Foundation under grant CAREER CNS-2047454. Fei Fang was supported in part by NSF grant ISS-2046640 (CAREER).
10

---

References
[1] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[2] Mengdi Xu, Wenhao Ding, Jiacheng Zhu, Zuxin Liu, Baiming Chen, and Ding Zhao. Task- agnostic online reinforcement learning with an infinite mixture of gaussian processes. Advances in Neural Information Processing Systems, 1464229-4400, 2020.
[3] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13/.5522, 2013.
[4] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, Science, 362(6419):1 140-1 144, 2018.
[5] Ilge Akkaya, Marcin Andrychowicz. Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's s cube with robot hand. arXiv preprint 13000111 2019.
[6] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of machine learning research: JMLR, 21(181):1-50, 2020.
[7] Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint wrXiv:2209.00022 2022.
[8] Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. probabilistic interpretation of self-paced learning with applications to reinforcement learning. Journal of Machine Learning Research, 22(182):1-55 2021.
[9] Pascal Klink, Carlo D' Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning, Advances in Neural Information Processing Systems, 33:9216-9227, 2020.
[10] Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforcement learning. In Conference on Robot Learning, pages 513-529. PMLR, 2020.
[11] Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang. Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems. Advances in Neural Information Processing Systems, 34, 2021.
[12] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference machine learning, pages 1515-1528. PMLR, 2018.
[13] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Y Oudeyer. Teacher algorithms for curriculum learning of deep continuously parameterized environments. Conference on Robot Learning, pages 835 353. PMLR, 2020.
[14] Philippe Essrrch, Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisimulation makes analogies in conl-conditioned reinforcement learning. arXiv preprint arXiv:2204.1300 13060. 2022.
[15] Gen Li, Laixi Shi, Yuxin Chen, Yuantao Gu, and Yuejie Chi. Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning. Advances in Neural Information Processing Systems, 34:17762-17776, 2021.
[16] Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Yves Oudeyer. Teachmyagent: a benchmark for automatic curriculum learning in deep rl. International Conference on Machine Learning, pages 9052-9063. PMLR, 2021.
11

---

[17] Bhairav Mehta, Manfred Diaz. Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1176. PMLR, 2020.
[18] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826.
PMLR, 2017.
[19] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in Neural Information Processing Systems, 33:13049-13061 2020.
[20] Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao. Robust reinforcement learning as a stack- elberg game via adaptively-regullrrzed adversarial training. arXiv preprint 2022.
[21] Mengdi Xu, Peide Huang, Fengpei Li, Jiacheng Zhu, Xuewei Qi, Kentaro Oguchi, Zhiyuan Huang, Henry Lam, and Ding Zhao. Accelerated policy evaluation: Learning adversarial environments with adaptive importance sampling. arXiv preprint arXiv:22066 10566, 2021.
[22] Adrien Baranes and Pierre-Y Oudeyer. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In 2010 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1766-1773. IEEE, 2010
[23] Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, and Pierre-Y Oudeyer. Accuracy- based curriculum learning deep reinforcement learning. arXiv preprint arXiv:/806.099 096114 2018.
[24] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32, 2019.
[25] Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum rein- forcement learning via constrained optimal transport. In International Conference on Machine Learning, pages I 11341-1133 1358. PMLR, 2022.
[26] Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self self-training for gradual domain adaptation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5468-5479. PMLR, 13-18 Jul 2020.
[27] Haoxiang Wang. Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. arXiv preprint arXiv:2204. 08200, 2022.
[28] Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evolving visual domains. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 867-874, 2014.
[29] Michael Gadermayr, Dennis Eschweiler, Barbara Mara Klinkhammer, Peter Boor, and Dorit Merhof. Gradual domain adaptation for segmenting whole slide images showing pathological variability. In International Conference on Imuge and Signal Processing, pages 461-469. Springer, 2018
[30] Markus Wulfmeier, Alex Bewley, and Ingmar Posner. Incremental adversarial domain adaptation for continually changing environments. In 2018 IEEF International conference on robotics and automation (ICRA), pages
IEEE, 2018.
[31] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.00 012.0/00077 2020.
[32] Samira Abnar, Rianne van den Berg, Golnaz Ghiasi, Mostafa Dehghani, Nal Kalchbrenner, and Hanie Sedghi. Gradual domain adaptation in the wild: When intermediate distributions are absent. arXiv preprint arXiv:2106 06080, 27X6002000000080, 2021.
12

---

[33] Hong- You Chen and Wei-Lun Chao. Gradual domain adaptation without indexed intermediate domains. Advances in Neural Information Processing Systems, 34, 2021.
[34] Leonid V Kantorovich. On the translocation of masses. Journal of mathematical sciences, 138143223 2006.
[35] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
[36] Jiacheng Zhu, Aritra Guha, Dat Do, Mengdi Xu, XuanLong Nguyen, and Ding Zhao. Functional optimal transport: map estimation and domain adaptation for functional data. arXiv preprint arXiv:2102.03895, 2021.
[37] Yiwen Dong. Jiacheng Zhu, and Hao Young Noh. Re-vibe: Vibration- based indoor person re-identification through cross-structure optimal transport. 1st International Workshop on The Future of Work, Workplaces, and Smart Buildings (FoWSB), 2022.
[38] Jiacheng Zhu, Gregory Darnell, Agni Kumar, Ding Zhao, Bo Li, Xuanlong Nguyen, and Shirley You Ren. Physiomtl: Personalizing physiological patterns using optimal transport multi-task regression. In Conference on Health, Inference, and Learning, pages 354-374.
PMLR, 2022.
[39] Vivien Seguy and Marco Cuturi. Principal geodesic analysis for probability measures under the optimal transport metric. Advances in Neural Information Processing Systems, 28, 2015.
[40] Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis, (3(2):904-922) 2011.
[41] Jiaojiao Fan, Amirhossein Taghvaci, and Yongxin Chen. Scalable computations wasserstein barycenter via input convex neural networks. arXiv preprint arXiv:2007.04462 2020.
[42] Robert J McCann. A convexity principle for interacting gases. Advances in mathematics, 128(1):153-179, 1997.
[43] Jiacheng Zhu, Jielin Qiu. Zhuolin Yang. Douglas Weber, Michael A Rosenberg, Emerson Liu, Bo Li, and Ding Zhao. Geoecg: Data augmentation via wasserstein geodesic perturbation for robust electrocardiogram prediction. arXiv preprint arXiv:2208. 0120000 2022.
[44] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[45] Hicham Janati, Marco Cuturi, and Alexandre Gramfort. Debiased sinkhorn barycenters. In International Conference Machine Learning, pages 4692-4701, PMLR, 2020.
[46] Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal Computing, 4066):1662-1114, 2011. 2011.
[47] Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In UAI, pages 210-219. Citeseer, 2014.
[48] Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10069-10076, 2020.
[49] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[50] Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot. Olivier Pictquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In International Conference Machine Learning, pages 2307-2318. PMLR, 2021.
[51] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: metric spaces and in the space of probability measures. Springer Science & Business Media, 2005.
13

---

[52] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[53] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
[54] Antonin Rafin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselinee Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021.
[55] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021.
[56] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[57] Pascal Klink, Haoyi Yang, Jan Peters, and Joni Pajarinen. Metries matter: closer look on saff-paced reinforcement learning. 2021.
[58] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1rXiv:160.011400 2016.
[59] Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. In International conference on machine learning, pages 685-693. PMIR, 2014.
[60] Lingxiao Li, Aude Genevay, Mikhail Yurochkin, and Justin M Solomon. Continuous regularized wasserstein barycenters. Advances in Neural Information Processing Systems, 33:17755-17765. 2020.
[61] Nicolas Courty, Rémi Flamary, and Mélanie Ducoffe. Learning wasserstein embeddings. arXiv preprint 17Xiv:1710.07457, 2017.
[62] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition SOVPR'05), volume 1, pages 539-546 IEEE, 2005.
[63] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, volume 4, pages 162-169, 2004.
[64] Yuxin Deng and Wenjie Du. The kantorovich metrie in computer science: A brief survey. Electronic Notes in Theoretical Computer Science, 253 82, 2009.
[65] Xinfeng Zhou. A Practical Guide to Quantitative Finance Interviews. Lulu, com, 2008.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "tmV_DoqMuZD", "reviewer": "Reviewer_G5ua", "summary": "This paper considers the task of curriculum reinforcement learning (CRL), and propose an algorithm, GRADIENT, which leverages ideas from gradual domain adaptation (GDA) and optimal transport (OT). Essentially, GRADIENT focuses on the probability distribution $\\rho(c)$ over the context variable $c$, and fine-tunes it from the source distribution to the target distribution gradually by treating it as a Wasserstein barycenter between the source and target (moving gradually from the source to target). The authors performed some brief theoretical analysis and conducted experiments over several benchmarks. The empirical results show GRADIENT has better sample complexity compared with the considered baselines.", "questions": "See Weaknesses above.", "limitations": "See Weaknesses above.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper applies gradual domain adaptation (GDA) to reinforcement learning in a reasonable way. It seems that the task of curriculum reinforcement learning (CRL) involves a directional adaptation (i.e., from a source to a target) of probability distribution in a continuous way , which is indeed what GDA could help with. It is good to see that GDA is becoming useful in more and more application scenarios.\n\n2. The authors provide a theoretical analysis of the proposed algorithm to corroborate it, which is nice. It looks that the derived RL bounds, Eq. (6)(8), look very similar to the bound in Lemma 1 of [1], which are derived in the supervised learning setting for GDA. Overall, I think the theoretical analysis is elegant. (Notice to area chair: I didn't check the proof since I'm not an expert of RL theory)\n\nWeaknesses:\n\n1. Novelty: The application of GDA and Wasserstein barycenter is quite straightforward, as long as one knows both GDA and CRL. So from the perspective of GDA, this paper is not very novel in terms of methodology.\n\n2. [**IMPORTANT**] Missing discussions on a concurrent/prior work: This paper briefly mentioned a concurrent work [2] in Sec. 2 in one sentence. However, [2] actually has lots of similarity with this work, and it appeared online in Oct 2021, which seems more like a prior work rather a concurrent one. [2] also applied optimal transport (OT) and Wasserstein barycenters to CRL, thus in my opinion, it should be discussed in this paper more carefully. Reviewers need to know the difference between this paper and [2] clearly.\n\n2. [**IMPORTANT**] Missing baseline in experiments: as I mentioned above, [2] is prior work, and it has released code. So there is no good reason for the authors not to compare GRADIENT against [2]. \n\n\n[1] Wang et al. Understanding Gradual Domain Adaptation: Improved Analysis, Optimal Path and Beyond. ICML 2022 (https://arxiv.org/abs/2204.08200)\n\n[2] Klink et al. Metrics Matter: A Closer Look on Self-Paced Reinforcement Learning. 2021 (https://openreview.net/forum?id=lKcq2fe-HB)\n\n\n-------------------------------------\n**Post-Rebuttal**\nThe authors' responses addressed most of my concerns above, so I increased my rating.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YL6Inhcks30", "reviewer": "Reviewer_4tAz", "summary": "This work proposes GRADIENT, a method for curriculum learning, by interpolating the initial context distribution and the target context distribution using the Wasserstein distance. This paper provides a theoretical justification for the proposed algorithm that the optimal value function under two adjacent contexts can be bounded by the Wasserstein distance between the contextual distributions of these adjacent contexts. This paper also demonstrates the practicality of the GRADIENT method in several goal-condition tasks.", "questions": "1. It seems that the context dimension of all experiments conducted so far is either discrete or dimensional <=2, is it possible to run the GRADIENT algorithm on some non-goal-conditioned tasks?\n2. It seems that assumption 4.1 assumes the state space is a subset of the context space ($\\mathcal{S}\\subset\\mathcal{C}$), but in the FetchPush environment, the context is a 2-dimensional vector representing the goal while the state is a 28-dimensional vector, perhaps the author can clarify this?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The main body of this paper is very well written and the Wasserstein distance indeed serves good metric for measuring distributions.\n2. The experimental results are very interesting as they clearly reflect the interpolation of the Wasserstein Barycenter.\n3. The experimental results seem promising -- as in all demonstrated tasks, they outperform the prior arts by a large margin.\n\nCons (Updated Aug 3rd):\n1. (Major; clarified) The connection between the theoretical results and the experiments is not carefully addressed. The paper would be much better if the author can address how the main assumption (4.1) is reflected in the experiments. Since all experiments are low-dimensional, so it should not incur the \"curse of dimensionality of the Wasserstein distance\", hence it would be nice to address how the assumption is actually reflected in the selected experiments.\n2. (Major; draft improved) The proof of theorem 4.1 is hard to follow. The readability of the paper would be largely improved if the author can summarize the notations/assumptions in the appendix, as the proof adopt many non-standard RL theory notations.\n3. (Minor; draft improved) There are some minor citation issues in the related work section -- when using a paper to start a sentence, it would be better if the author used \\citet{} instead of \\citep{} (e.g, \"[6] proposes Self-PacedReinforcementLearning(SPRL)\"; \"Concurrent to our work,[19] also propose to replace KL divergence\").\n\nThe reviewer would like to improve the rating if the author can improve the readability of the proof.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VhN0_78AExl", "reviewer": "Reviewer_KSyL", "summary": "this paper propose GRADIENT, a novel curriculum reinforcement learning method based on optimal transport. Therefore, GRADIENT improves the learning efficiency and asymptotic performance in a wide range of tasks compared with state-of-the-art baselines. \n\nFor example, in the maze example, GRADIENT algorithm can better interpolates the environment task distribution from source to target env compared to linear interpolation. ", "questions": "Q: dealing with sparse rewards problem, there are some other useful RL algorithms, such as HER. have the authors conducted some experiments compared to these baselines? and why GRADIENT?\n\nI appreciate the authors add HER as a baseline. Considering the discussion and revision, I increased my score", "limitations": "No", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nOT in CRL is novel\n\nthe proposed method deals with both discrete and continuous context space.\n\nempircally,this paper shows its proposed method's learning efficiency\n\nthis paper provides a theorectical bound for policy transfer.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PrYNxdWoRAZ", "reviewer": "Reviewer_pAPM", "summary": "The paper considers the problem of building curricula of tasks for reinforcement learning in order to learn the target task more easily. The authors propose to approach this problem by finding the interpolations between tasks using the Optimal Transport framework, thus replacing previous methods which heavily relied on f-divergences and the KL divergence in particular. Additionally, the paper introduces the $\\pi$-contextual-distance in order to measure the distance between different contexts, particularly in settings where the context space is discrete. The authors provide theoretical proof that under certain assumptions the performance gap between each pair of neighboring tasks is small. Finally, they test the proposed method empirically and show that it outperforms competitors in settings where the context cannot be well approximated with a Gaussian distribution.", "questions": "The only suggestion I think is reasonable within the scope of the rebuttal period is adding the limitations to the paper. Other than that, please refer to the \"Weaknesses\" and \"various comments\" sections above. ", "limitations": "The paper does not discuss the potential negative societal impact and I don't think such discussion is required.\n\nThere is no discussion on the limitations of this work, which I think would be very useful for the reader. I listed a few possible limitations in the  Weaknesses section above.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed approach is intuitive and follows naturally from the previous line of research on applying KL divergence to find a good sequence of tasks.\n- The authors provide interesting and plausible scenarios when their approach is especially useful. In particular, Figure 5 illustrates this point visually by plotting the context space.\n- The results in settings where the context cannot be effectively modeled as a Gaussian distribution are quite good.\n- The paper is well written, the description of the method and experiments is mostly clear, and the mathematical notation is well defined.\n\nWeaknesses:\n- The empirical evaluation could be extended, as two out of three tasks (Maze and PointMass) are rather toyish. It would be useful to have another more complicated benchmark, such as the Ant locomotion environment that was used e.g. in the SPDL paper.\n- Although the checklist at the end of the paper says otherwise, the limitations are not properly described. For example, I would suggest discussing the following limitations:\n    - The method is only tested in settings where the context space is nice and relatively low-dimensional. \n    - The assumptions for Theorem 4.1 are quite strict (the context cannot change the transition dynamics nor the reward function).\n    - The $\\Delta\\alpha$ hyperparameter seems to be very important, but there is no practical guidance on how to set it.\n- This is subjective, but I'm not entirely convinced about the level of novelty of the proposed approach. The main idea is to take an approach that works well with the Kullback-Leibler divergence and use another way to measure the similarity of distributions, namely the Wasserstein distance. In the same way, the $\\pi$-contextual-distance which the authors list as a contribution seems to be a straightforward extension of the distance introduced in [1]. On the one hand, this seems a bit iterative, but on the other I still think overall the contribution is significant, so treat this as minor criticism.\n\nIn the end, although I have some doubts about novelty and empirical evaluation, I think the contributions in this paper are significant. For the moment I decided to go with \"Weak accept\".\n\nVarious comments:\nFigure 3(b) (and the rest of the figures with performance): Just to make sure, does the x-axis shows the number of steps of training on the target environment or the whole training process (i.e. starting with pre-training on the auxiliary task)?\nLine 111: unnecessary parenthesis. \n\n[1] Castro, Pablo Samuel. “Scalable methods for computing state similarity in deterministic Markov Decision Processes.” AAAI (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept