Metrics Matter: A Closer Look on Self-Paced Reinforcement Learning
Authors: Pascal Klink
Haoyi Yang
Jan Peters
Joni Pajarinen
Venue: ICLR 2022 Submitted (2022)
Keywords: Curriculum Learning
Reinforcement Learning
Self-Paced Learning

Abstract: Curriculum reinforcement learning (CRL) allows to solve complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. However, the generation of such task sequences is largely governed by application assumptions, often preventing a theoretical investigation of existing approaches. Recently, Klink et al. (2021) showed how self-paced learning induces a principled interpolation between task distributions in the context of RL, resulting in high learning performance. So far, this interpolation is unfortunately limited to Gaussian distributions. Here, we show that on one side, this parametric restriction is insufficient in many learning cases but that on the other, the interpolation of self-paced RL (SPRL) can be degenerate when not restricted to this parametric form. We show that the introduction of concepts from optimal transport into SPRL prevents aforementioned issues. Experiments demonstrate that the resulting introduction of metric structure into the curriculum allows for a well-behaving non-parametric version of SPRL that leads to stable learning performance across tasks.

Under review as a conference paper at ICLR 2022
METRICS MATTER: A CLOSER LOOK ON SELF-PACED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Curriculum reinforcement learning (CRL) allows to solve complex tasks by gen- erating a tailored sequence of learning tasks, starting from easy ones and subse- quently increasing their difficulty. However, the generation of such task sequences is largely governed by application assumptions, often preventing a theoretical in- vestigation of existing approaches. Recently, Klink et al. (02T showed how self-paced learning induces a principled interpolation between task distributions in the context of RL., resulting in high learning performance. So far, this interpola- tion is unfortunately limited to Gaussian distributions. Here, we show that on one side, this parametric restriction insufficient in many learning cases but that on the other, the interpolation of self- salf-paced RL (SPRL) can be degenerate when not restricted to this parametric form. We show that the introduction of concepts from optimal transport into SPRL prevents aforementioned issues. Experiments demon- strate that the resulting introduction of metric structure into the curriculum allows for well-behaving non-parametric version of SPRI. that leads to stable learning performance across tasks.
INTRODUCTION
Reinforcement learning (RL) (Sutton & Barto 1998) has celebrated great successes as a framework for autonomous acquisition of desired behavior. With ever-increasing computational power, this framework and the algorithms developed under it have allowed to create learning agents capable of solving non-trivial long-horizon planning (Mnih al.) 2015 [Silver al.] (017) and control tasks [Akkaya et al., 2019). However, these successes have also highlighted the need for certain forms of regularization, such as leagues in the context of boardgames (Silver et al.) 2017). gradual di-
versification of simulated training environments for robotic manipulation (Akkaya et al. 2019) or
tailored training pipeline in the context of humanoid control for soccer (Liu et al.] [021]). These regularizations help to overcome shortcomings of modern RL agents, such poor exploratory be- havior- a problem that is an active topic of research (Bellemare et al. 2016 Ghavamzadeh et al.)
2015 Machado et al. 2020).
One can view aforemenniined regularizations under the umbrella term of curriculum reinforcement learning (Narvekar al. 2020), where the idea is to avoid shortcomings of modern (dcep) RL agents such as aforementioned poor exploration by learning on a tailored sequence of tasks. Such curric- ula materialize in variety of ways and are motivated from many perspectives in the literature (Andrychowicz et al.) 2017 Florensa et al. 2017 Wöhlke et al. 2020). Although the resulting cur- ricula can often be interpreted as a sequence of task distributions, these sequences typically lack formal connection to the reinforcement learning objective of maximizing the expected reward under given target task distribution. In recent line of work, Klink al. (2021) proposed the idea of self-pacce reinforeement learning (SPRL), borrowing from the concept of self- self-paced learning that has been established in the supervised learning literature (Kumar et al. 2010; Tiang et al., [015] Meng et al.] (017). Klink et al. showed connection between a regularized RL objective and sequence of task distributions that trade-off between yielding high expected reward and tasks likely under the target distribution. This interpolant has, however, so far been restricted to Gaussian distri- butions (Klink et al., 2020a b) 2021). While successful in experimental evaluations, this Gaussian assumption clearly imposes limitation the flexibility of the curriculum and disconnects the al- gorithmic implementation from the established theory. This disconnect raises the question whether the observed performance of SPRI. is due the Gaussian approximation.

---

Under review as a conference paper at ICLR 2022
Contribution:
The key insight presented in this paper is that the Gaussian approximation of existing SPRL implementations is indeed important for their empirical performance. We show that
Parametric assumptions in SPRL hinder the learning performance in task spaces with non- Gaussian target distributions.
SPRL can, however, fail to facilitate learning on the target task distributions when leaving these parametric assumptions behind.
Equipping SPRI. with Wasserstein metrics allows for a flexible, particle-based represen- tation of the task distribution that ensures meaningful interpolation in aforementioned failure cases, providing higher performance.
RELATED WORK
The main foeus of this work is self-paced reinforcement learning (SPRL., Klink et al. (2020a b
)02T)) that takes the concept of self- self-paced curriculum learning (Kumar et al. (010) from supervised-
reinforcement learning (RL). Opposed supervised learning, where there ongoing discussion
about the mechanics of curricula and their effect in different situations (Weinshall & Amir 2020; Wu
al. 202I). the mechanies seem be more agreed upon in RL. In RL. curricula improve learning performance of an agent by adapting the training environments to its proficiency, and with that e.g.
bypass poor exploratory behaviour of non-proficient agents. Applications are by now widely spread and different terms have been established. Adaptive Domain Randomization (Akkaya et al. 2019) uses curricula to gradually diversify training parameters of a simulator to facilitate sim-to-real trans- fer. Unsupervised environment discovery (Dennis et al., 2020 Jiang et al. 2202 a) similarly aims to efficiently train an agent which is robust to variations in the environment. Automatic curriculum learning methods (Florensa al., 2017; Sukhbaatar et al. 2018 Florensa et al. 2018 Portelas al., 2019 Zhang et 20220 Racaniere et al.) 20200] Eimer et al. (021). to which SPRL belongs to, par- ticularly focus improving the learning speed and/or performance of agent on set of desired tasks. Curricula are often generated as distributions that maximize a certain surrogate objective, such as learning progress (Baranes & Oudeyer 2010 Portelas et al., 2019), intermediate task difficulty (Florensa et al. 2018), regret (Jiang et al. 2021b) or disagreement between Q-functions (Zhang al. 2020). Curriculum generation can also be interpreted as a two-player game (Sukhbaatar al., 2018 The work by Jiang et al.] (2021a) even hints to a link between surrogate objectives and two- player games. Opposed these interpretations. SPRL has been shown to perform an interpolation between task distributions by Klink et al. (202T), allowing to formally relate the effect of SPRL to the concept of annealing in statistics (Neal, 2001) and homotopic continuation methods in optimization (Allgower& & Georg 2003). We wish to add to this formal understanding of SPRL by investigating the interpolation that it produces more closely.
As this investigation will lead to the problem of optimal transport, we wish point out important literature in this field. Dating back to the work by Monge in the 18th century, optimal transport has been understood an important fundamental concept touching upon many fields in both the- ory and application (Liu et al. 2019 Peyré et al. 2019 Chen et al. (021)). In probability theory, optimal transport translates to the so-called Wasserstein metric (Kantorovich 1942) that compares two distributions under a given metric on the sample space. From computational perspective, entropy entropy-regularien regularized Wasserstein metrics (Cuturi, 2013) have led tangible speed-ups in computa- tions revolving around optimal transport and are hence widely applied (Feydy et al. 2019).
3 PRELIMINARIES
This section serves to introduce the necessary background on (contextual) RL, self-paced RL and optimal transport.
3.1 CONTEXTUAL REINFORCEMENT LEARNING
Contextual reinforcement learning (Hallak et al. 2013) can be seen as a conceptual extension to the (single task) reinforcement learning (RL) problem
8 Σ 1=
max J(T) max E (sa)))). ( (
rrr(s av)
(1)
2

---

Under review as a conference paper at ICLR 2022
which aims maximize the above expected reward objective by finding optimal policy π -.AAAA + for given MDP M (S,A, p, Pu) with initial state distribution po and transition dynamics p. Contextual RL extends this objective to space of MDPs M(c) = (S,A, PesTes Tes Po,e)
equipped with a distribution p.C. + over contextual variables c. E
max ( μ) max p(ee) [J(T,
The policy SxCxA H R is conditioned u(c) encodes the tasks M(c) that the agent is expected to encounter. Objective [ c) in Eq. (2) corresponds the objective J(x) in Eq. [ where, however, the initial state distribution po. the transition dynamics as well as the reward function of M are replaced by their counterparts in M(c). This contextual model of optimal decision making is well-suited for learning in multiple related tasks as is the case in multi-task (Wilson et al. 2007 goal-conditioned (Schaul et al. 2015)
the contextual parameter The distribution
or curriculum RL (Narvekar et al. 2020).
3.2 SELF-PACED REINFORCHMENT LEARNING
Self- Self-paced reinforcement learning (SPRL) has been introduced by Klink et al. (2020a b 2021
curriculum RL algorithm that alters the context distribution (u(c) in the contextual RL objective (2) increase the learning performance of an agent and/or make it less susceptible local optima of
the objective function. SPRL computes a surrogate distribution p C + I under which to train the RL agent, i.e. optimize J(x,)),
This surrogate distribution is found by optimizing the KL divergence to the target distribution μu() subject two constraints (see Klink et al. (202T] Section 8))
min DKL (p(c) [(()) s.t. [TT,P) 2 P
DRL (p(c) ((c)) <.
The distribution p(c) balances between tasks likely under the (target) distribution p(c) and tasks in which the agent currently obtains large rewards. The KL divergence constraint w.r.t. the previous context distribution q(c) prevents large changes in p(c) during subsequent iterations, making the curriculum robust against errors in the estimates of the expected agent performance J(H,,) c).
A particularly interesting aspect of this work is that objective (3) can be interpreted to perform specific interpolation between the distributions ju(e), ((c) and maximum entropy distribution (c) x exp(nJ(z,, c)) encoding high reward tasks. This interpolation is given by
(c) x p(c) 44 q(c) if exp(nJ c)) 1t=
The two parameters a and controlling the interpolation are the Lagrangian multipliers of the two constraints objective (3). So far, Klink et al. (2020a by (201)) restricted the distribution Pa.,(c) (c) to Gaussian distributions pz (c) = [c(c Dj. In this case, optimizing (3) w.r.t. lu and Σ of pa corresponds to performing an 1-projection of the analytic optimal distribution to the Gaussian restriction, i.c minimizing DKL (c)(c) (c))(c)) w.r.t.
In this work, we are interested in investigating the distribution Pa.n outside of this parametric re- striction Pv: i.e. truly employing the distribution instead of its I-projection Gaussian.
3.3 OPTIMAL. TRANSPORT
The problem of optimally transporting density between two distributions has been initially investi- gated by Monge (1781). As of today, generalizations established by Kantorovich (1942) have led to so called Wasserstein distances as metrics between probability distributions defined on metric space M- = Cd with metric d C Rz
inf (ET(pp.p)) Joxe d(c,, (2), d2)), 1/P
Wy(Pi,P2)
D21
T(pı,p2) (0)0)=(x00m C C Pzo / (2) 02) de2, p2(02) (2) p
The distance between p1 and pa results from solving optimization problem that finds so-called plan y. This plan encodes how to equalize Pl and p2 taking into account the cost of moving density between between parts of the space C. This cost is encoded by the metric d. In the following,
3

---

20- Under review as a conference paper at ICLR 2022
Algorithm Self-Paced Reinforcement Learning Implementations
Input: Context dist. po(c), target context dist. plc),, performance threshold 6, distance bound
for to K do
Agent Improvement:
Sample contexts C2 px(c), i E [1,
Train policy и under c, and observe episodic rewards R; = -i=1 re. (s,, ar), i MM]
Estimate [T, from the dataset D = {(c,, Ro)li [1, [L}
Context Distribution Update:
G-SPRL Optimize 3 w.r.t. 1 and Σx+1, where Pk+1(c) = Wcc Ex+1)
NP-SPRL Optimize w.r. a and y using a discrete approximation Pa,n R Pa,n(C)
WB-SPRL Optimize w.r.t. ,3 to obtain pp(c)
end for
will always assume to work with 2-Wasserstein distances under euclidean metric, i.e. p 2 and d(c), C2) |cc C2ll2. Particularly interesting for the investigation conducted in this paper are so called Wasserstein barycenters
(S) that perform a weighted interpolation between a set of distributions Pk by computing a distribu- tion p that minimizes the above sum of Wasserstein distances. As illustrated in appendix [A] these barycenters allow to smoothly interpolate between complex distributions w.r.t. the defined metric d.
PW, (c) arg min, Σ Kkm wy W2 Ps),
Σ =
NON-PARAMETRIC SELF-PACED REINFORCEMENT LEARNING
To highlight the importance of non-parametric variant of SPRL that, nonetheless, respects the metric structure of the context space C, we investigate three versions of SPRL: G-SPRL- the Gaussian version from Klink et al. (202T), NP-SPRL- a faithful implementation of the SPRL objective (3) that computes and uses the distribution (c) 4 by discretizing the context space C, and WB-SPRL- an instantiation of Wasserstein barycenters the SPRL framework. As shown in Algorithm [ all implementations only differ in the computation of the context distributions.
4.1 NP-SPRL
Computing the analytic distribution Pa.n(() (c) and adjusting the parameters a and 17 via the SPRL objective will require approximations in the general case, sampling arbitrary continuous dis- tributions 18 an open research problem (Brooks et al.] 2011] Liu & Wangl 2016 Liu et al. 2019] Wibisono 2018). Using approximate methods resulting from this research would, however, directly interfere with our intent evaluating the hehavior of Po.n(c) as exactly as possible. Consequently. discretize the continuous context spaces in experiments in order to faithfully sample and evaluate expectations w.r.t. (C). This discretization is schematically shown in Figure I for one of the evaluation environments. Although such discretization will clearly scale gracefully to higher dimensions, it allows us to investigate the "analytic" behavior of SPRL, which, as we will show, can be sub-optimal even for diw-dimnnsional context spaces.
With context space C C IRd discretized into N cells, we can represent Pa.n vector Po., E To sample continuous context E C, we can then first sample an index of a cell from Pa., and then sample uniformly within this cell to obtain context Further, the KL divergences in the SPRL objective
(3) are straightforward to evaluate when working with Pa.n- To evaluate the expected per- formance, we simply evaluate the performance measure ( c) at the N cell centers с, to obtain the vector ((r) E N With that we obtain ((T,,)..) = PT., [(T).
4.2 WB-SPRL
To leverage optimal transport in SPRI. style algorithm, realize the interpolation between the current distribution g(c), target distribution p(c) and "value distribution" ps(c) & exp(nJ(xT.)) by Wasserstein barycenters
Pa(c) = arg (111-------)W(p.)) H(c)) + + ((W))))) (6)
4

---

Under review as a conference paper at ICLR 2022
(a) Maze Environment
(b) Context Space C
(c) Function f on C
(d) ʃ discretized on
Figure/ 1: (a) The first environment for evaluation of SPRL is maze task simulated in MuJoCo Todorov et al.] 2012). in which a point-mass needs to move in a maze of circular shape to reach desired target positions. (b) The target position is encoded via the contextual variable с E CCR? The area highlighted in red visualizes the initial position of the point mass and the walls of the environment are shown in black. (c In order faithfully evaluate expectations over functions or compute KL. divergences e.g. in the SPRL objective 3 we discretize the context space C.
The weights 3 of the interpolation are adjusted with the goal of minimizing (2(pa. (4) while ensur- ing a constraint on expected performance and distance to the previous distribution g(c)
min (e((p((),
J (T. 2 Wa(p)(c),,(())) S.
/
difference w.r.t. SPRL is that the parameter 7) of the value distribution p)(c) is not adjusted in optimization problem (7). Instead we adjust 7) before optimizing 7 such that J(T, P.) 2 δн,, where 8н > 5 is another performance threshold. This ensures that with 31 - 1, holds that J(RT, P/3) 2 8. We resort to particle-based representations of the distributions when implementing (6) and (7). This allows us to make use of Monge maps to compute (c) efficiently. More details on the implementation of WB-SPRL are provided in appendix
5 EXPERIMENTS
The experiments in this section serve to show the need for a realization of SPRI. without parametric restrictions on Pa.n but to also highlight that NP-SPRL our due its ignorance w.r.t. the metric on C. Experimental details can be found in appendix D Code provided in the supplementary material and will be made available upon acceptance. To situate our method in the field of curriculum RL and to showcase how our method performs w.r.t. the cur- rent state-of-the art, we further evaluate ACL, GOALGAN, ALP-GMM, VDS and PLR (Graves et al.) 2017 Florensa et al., 2018 Portelas al. 2019 Zhang et al. 2020 Jiang et al.) 2021b) alongside investigated SPRT. variants.
necessarily well-suited for this endeav-
MAZE
We first turn to sparse- maze-like environment depicted in Figure] I in which agent needs to reach a desired goal. Such environments have e.g. been investigated by Florensa et al.] (2018). The contexts unsolvable contexts (goals inside of a wall or the inner circle of the maze). Defining p(c) to be uniform over the context space, the curriculum needs identify and train the agent the subspace of feasible tasks in order to achieve good learning performance. This subspace of feasible tasks highly non-Gaussian, making it an interesting testbed for NP- and WB-SPRL. We discretize the context space C C R² of goal positions into 50 x 50 grid. Figure compares the performance of the different CRL algorithms. We see that both NP- and WB-SPRL perform better than G-SPRL, since the performance constraint Ep(c) [J(z, c)] 2 8 in objective (3) at some point prevents the Gaussian context distribution of G-SPRI. from expanding, as otherwise too many infeasible tasks would be encoded in (c) and hence the performance constraint violated. As shown in Figures and 7 the non-pare -ommmaraectric versions SPRL can flexibly assign probability to feasible contexts, resulting in high learning performance.
C of this environment encode the goal position to be reached and hence contains
5.2 POINT-MASS
We now consider the point- point-mass mass environment investigated by KKlink et al.) 2020a b 202T). As shown in Figure 3al point mass needs to be steered through narrow gate reach goal position
5

---

Note that "R²0" corresponds to a value of exp(-1000), as this ensures that μu(c) absolutely con- tinuous w.r.t Pa.n(c) and (c),), which is required to compute the KL divergence between them. Figure 3 shows that the performance of G- and NP-SPRL depends drastically on the choice of tar- get distribution. Further, we that NP-SPRL does not outperform G-SPRL in this environment, although should be able to match the bi-modal target distributions, which G-SPRL cannot. Finally, that WB-SPRL outperforms both other versions of SPRL and achieves consistent perfor- mance across target distributions. The illustrations in Figures and serve to illustrate the underly- ing problem. As evident in Figure 4 NP-SPRL indeed able match the target distribution μn (c) correctly while G-SPRL ultimately covers only one mode of the target density Y1 (c). As shown in appendix B, NP-SPRL still only learns solve one of the two tasks likely under Mi (c().. Investigating Figure 4b more closely reveals that NP-SPRL. only generates a "proper" curriculum for one of the two target tasks (the left one in the images) by gradually interpolating between easy tasks and the target task, while simply incorporating the second target task into Pa,n without generating an appro- priate curriculum for the agent to learn this task. Figure 5 shows that the curricula generated by G- Under review as a conference paper at ICLR 2022
-GSPRLL Random
GoalGAN -ACL- -PLR
IN--PPLL
WB SPRL
ALP-GMM- VDS
Success Rate
0.8
(b) SPRL context distributions pv over iterations
0.2
100
200
300
400
Epoch
(a) Learning Performance (Maze)
(c) NP-SPRI context distributions pr.y over iterations
Figure 2: Achieved success rate of different curricula and uniform sampling baseline (Ran- dom) over iterations. Mean and standard error are computed from 10 runs. b) Parametric and non- parametric context distributions py (c) and Pa.y(c) for run of G-SPRL and NP-SPRL respectively. The distributions are represented by 2000 samples drawn from them.
on the other side of wall. While Klink al. only considered narrow gate at one specific position as the target task, we will investigate a version in which the gate is located at one of two opposing positions ci [-3 3 0.5] and c2 0.5), making p(c) bi-modal distribution. This again challenges the Gaussian restriction of P& (c) in the G-SPRI. algorithm. We again discretize the context space S I22 into a grid of 50 bins on each axis. We investigate two target distributions with different log-likelihoods, that nonetheless produce very similar samples
1 NN (c,, (c,, 10 41) + W (c2, (c2,10 10 41 nn (c)
p2(c)
2
if E (c1,c2)
20
else.
and NP-SPRL for (2(c) are even more problematic. G-SPRL increases the variance of the Gaussian context distribution to match the constant (negligible) likelihood that assigned to the non-target
C R2 Po
G-SPRL
SPRL
ALP-
NP-SPRI.
Default
GoalGNN
ACL
VDS
Cum. Disc. Ret.
M4 (c)
(2(c)
10g
50
150
200
Epoch
Epoch
(a) Environment
(b) Learning Performance (Point-Mass)
Figure 3: (a) The point mass environment with its two-dimensional context space. The target distri-
butions N4 (c) and (c)(c) encode the two gates with width =0.5 in which the agent (black dot) required to navigate through narrow gate at different positions to reach the goal (red cross). (b) Discounted cumulative return over iterations obtained under different curricula for both distribu- tions. Statistics (mean and standard error) are computed from 10 seeds.

---

Under review as a conference paper at ICLR 2022
KT. Ki.Diverenne
(b) Po (c) for different iterations of NP-SPRL (Point-Mass)
G-SPRL NP-PPL
(a) DRa (p(c) || (x())
(c) (c) for different iterations of G-SPRL (Point-Mass)
Figure 4: (a) Mean KL. divergences between ju and the final context distribution computed by G- and NP-SPRL as well as the minimum and maximum over 10 seeds. (b) and (c) visualize Pa.n of NP-SPRL and (c) of G-SPRL for increasing iterations of the algorithms (left to right).
tasks, while NP-SPRL now interpolates between none of the target tasks. Consequently, neither the agent under G- nor NP-SPRI. learns to solve any of the target tasks. The observed shortcomings of G- and NP-SPRI. are caused by their use of the KI. divergence to measure similarity hetween distribu- tions. Figure shows that the KL divergence generates interpolations that are highly dependent the likelihood function of the distributions, as the KL divergence does not take the metric structure of the sample space C. into account. Indeed Gaussian distributions encode (Euclidean) metric via their log-like ligeiik,,,ood, explaining the acceptable performance of G- and NP-SPRL for μr (c). Looking at Figure (6b and the curricula generated by WB-SPRL in Figure 7 we see that the explicit notion of a (Euclidean) metric in WB-SPRL leads to a gradual change in the tasks encoded by the curriculum for both target distributions. This leads higher learning performance visualized in Figure
5.3 PICK AND PLACE
Next, we consider the pick-and-place task in the OpenAl gym environment suite (Brockman et al,, 2016 in which robot tasked to grasp block on a table and move it to desired position (see Fig- ure Rp. The sparse reward of this environment, only rewarding the robot upon completing the desired task, makes it very challenging exploration problem,
be in Figure 8 in which (default) SAC does not learn this pick and place task. One way to alleviate such challenging exploration prob- lems is learn the task via curriculum of starting states. Such starting states be obtained from an expert demonstration, i.e. a trajectory TEXpERt [0, 1] + S. A curriculum over this trajectory is then formally defined by choosing the context space be the unit interval C [0, 1] C R. The con- textual parameter only influences the initial state distribution Po.c 6x Tx=rexpeer(e (c)- We investigate the NP- and WB-SPRI. algorithms in this setting by recording one execution of hand-crafted con- troller that first moves the end-effector above the block to be grasped, then lowers the end-effector, grasps the block and moves to the target. We slightly randomize the position of the goal as well as the block to enforce that the agent learns robust policy. In a realistic scenario. we only record the expert demonstration at a discrete set of states (t, = 1..... ......NY N) and hence we have a truly discrete CRL setting in this experiment very well suited for NP-SPRL. Furthermore, the finite number of
500
Kl Diveeregee
200
(b) Po.n for different iterations of NP-SPRL (Point-Mass)
100
G-SPRL
(a) DRL (p(c) || (2()))
(c) (c) for different iterations of G-SPRL (Point-Mass)
Figure 5: (a) Mean KL divergences between P2(c) and the final context distribution computed by G- and NP-SPRL as well as the minimum and maximum over 10 seeds. (b) and (c) visualize (c) of NP-SPRL and (c) of G-SPRL for increasing iterations of the algorithms (left to right).
7

---

Under review as a conference paper at ICLR 2022
(a) KL divergence interpolation: arg min, Σt= WeDri. (p Pk)
(b) Wasserstein baryeenter (Equation 5
Figure 6: Interpolations between unimodal (left) and bi-model (right) distributions p1 Pi(c) and P2(c) via a KL divergence-basse based interpolation and Wasserstein barycenters (5). In one case, all distribu- tion are Gaussians or mixture Gaussians (blue). In the other case, the distributions are uniform distributions (orange). The Wasserstein barycenters tion (Feydy et al. 2019). The visualized PDFs are then estimated using a kernel density estimation. This results in small amount of smoothing for the uniform distribtions.
computed using particle-based approxima-
contexts avoids the necessity for function approximators to approximate the expected performance J(n,c) in context sliding window. We again investigate two target context distributions, one being a narrow Gaussian target distribution with mean Dirac-delta with a negligible amount of probability in any context to be absolutely continuous w.r.t any distribution (c). The results in Figure show that WB-SPRL generates curriculum that allows the agent to learn a reliable policy by smoothly moving probability from later steps of the ex- pert demonstration towards earlier ones. This learned policy is more reliable (25% vs. 100% success rate) and faster ( 699 vs. 13 steps for task completion) than the demonstration. Both default learning (always starting from the initial state) and NP-SPRL do not lead to successful policies. In appendix we show that the non-effectiveness of NP-SPRL is again grounded in the degenerate interpolation that puts no probability mass at intermediate time-steps regardless the target distribution.
simply estimate the performance in each (discrete) context via
and negligible variance (μi (c)) and the other being again
TEACHMYAGENT BENCHMARK
A final environment for evaluation the TeachMyAgen benchmark proposed by (Romac et al, 2021]. In their evaluations accompanying the benchmark, Romac et al. reported poor performance of G-SPRL compared to other automatic curriculum RL methods. Consequently, we want to inves- tigate the improvement of WB-SPRL this benchmark verify the observed benefits and identify further directions for improving the SPRL framework. Figure p shows that WB-SPRL significantly improves upon G-SPRL, achieving top performance across the methods in the mostly unfeasible sce- nario. We see that WB-SPRI, particularly lacks behind AL.P-MM in the mostly trivial setting. Figure 9c|shows that WB-SPRI. WB-SPRI..uickly reduces to uniform sampling in this setting (on average after 60 out of 370 iterations), as it can incorporate all tasks while fulfilling the desired performance threshold. This observation explains the similar performance between random sampling and WB-SPRL in the mostly trivial setting and further highlights an important conceptual difference between SPRL and
(b) WB-SPRL (Point Mass μt (c))
(a) WB-SPRL (Maze)
(c) WB-SPRL (Point Mass P2(c))
Figure 7: Visualizations of the empirical distributions Pa generated by WB-SPRL in the maze- (a)
and paint-mass environments for both target distributions μa (c) and pz(c) ((b) (c)).
8

---

Under review as a conference paper at ICLR 2022
Random
GoalGAN
ACL
VDS
Defoult
WR-SPRI.
ALP-GMM
NP-SPRI.
Success Rate
M1 (1(c)
(2(c)
100
p(t)
0.5
0.0
0.0
0.33
0.66
0
50
100
150
200
50
100
150
200
Epoch
Epoch
(a) WB-SPRL Curriculum (Pick and Place)
(b) Learning Performance (Pick and Place)
Figure 8: (a) A WR-SPRI. curriculum for the pick-and-place task. The vertical dotted lines and small images visualize the states corresponding to different time steps of the expert trajectory. The colored bars indicate the context distributions pa(c) for different iterations of WB-SPRL. The corre- spondence between color and algorithm is shown via the colorbar on the right. (b) Success rates over iterations for different learning algorithms in the pick and place task for target distribution (c(c) and (2(c)). Mean and standard error is computed from 10 algorithm runs.
The maximization of learning progress in ALP-GMM leads to a prioritized sampling of the target (uniform distribution), efficiently avoiding tasks the agent has already learned. SPRL does not have a notion of avoiding "too easy" tasks as of now. Introducing such prioritized sampling of p(c) is hence interesting future research direction and may allow to combine the benefits of the SPRL framework with other successful curriculum RL methods.
ALP-GMM:
7 CONCLUSION
We investigated self-paced reinforcement learning (SPRL) outside of its current parametric restric- tions, revealing important shorteomings of the KL divergence for measuring the similarity of (task) distributions curriculum reinforcement learning. We showed that replacing the KL. divergences, originally employed in SPRL, with Wasserstein distances alleviates the observed problems of SPRL. Apart from achieving higher empirical performance in the SPRL framework, our results indicate that introducing a notion metric structure on the task space may an important next step for deriving principled, yet practical, understanding of curriculum RL. Our findings motivate vari- cty of future investigations, such as more elaborate implementations of WB-SPRL that e.g. directly move the individual particles of the context distribution via the gradient of the performance mea- sure [T, c) instead of using the value distribution Ps(c) as a proxy. As highlighted Section 6 importance-sampling of the context distributions are expected to further improve performance. Finally, investigating other metrics than the Euclidean one considered in this paper is expected to allow for principled interpolations in non- oun-Euclidean spaces.
Student-thai forgets
Mostly trivial task space
Mostly unfeasible
task space
0
Covar-GMM
(b) Evolution of pa in mostly unfeasible scenario
ALP-GMM
G-SPRI.
Rugged
Varjety students aoook
RIAC
difficulty wewe
WB-SPRL
(c) Evolution of in the mostly trivial scenario
(a) Results TeachMyAgent benchmark
Figure 9: Performance on the TeachMyAgen benchmark in the no expert knowledge setting. Baseline results are taken from (Romac al., 202T). Please refer to (Romac et al.) (02T) for detailed explanation of the setup. b) c) Visualization of the WB-SPRL curricula the mostly unfeasible and the mostly trivial task space scenario. The r-axis encodes the height of the obstacles that the agents encounters while the y-axis encodes the distance between them.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Iige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 2019.
Eugene L. Allgower and Kurt Georg. Introduction to numerical continuation methods. SIAM, 2003.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Picter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS), 2017.
Adrien Baranes and Pierre-Y Yves Oudeyer. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In International Conference on Intelligent Robots and Systems (IROS), 2010.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Neural Information Processing Systems (NeurIPS), 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (rXXi:16666005444 2016.
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of markov chain monte carlo. CRC press, 2011.
Yongxin Chen, Tryphon T Georgiou, and Michele Pavon. Stochastic control liaisons: Richard sinkhorn meets gaspard monge on schrödinger bridge. SIAM Review (SIREV), 63(2):249-33) 2021.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Neural Informa- fion Processing Systems (NeurIPS), 2013.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Neural Information Processing Systems (NeurIPS), 2020.
Theresa Eimer, André Biedenkapp, Frank Hutter, and Marius Lindauer. Self-paced context evalu- ation for contextual reinforcement learning. In International Conference on Machine Learning (ICML), 2021.
Jean Feydy, Thibault Séjourné, François- Xaviie Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Picter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference Robot Learning (CoRL), 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning (ICML), 2018.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 8(5-6):359-483, 2015.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Auto- mated curriculum learning for neural networks. In International Conference on Machine Learning (ICML), 2017.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint 102.:11522.0259 2015.
Ashley Hill, Antonin Rafiin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alee Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. hht //gitub.. com/hi 11l11110 11-a/ss stablaallll.. basel inea 2018.
10

---

Under review as a conference paper at ICLR 2022
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced cur- riculum learning. In AAAI Conference on Artificial Intelligence (AAAI), 2015.
Minqi Jiang. Michael Dennis, Jack Parker-Holder Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guides adversarial environment design. 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning (ICML), 2021b.
Leonid Kantorovich. On the transfer of masses (in russian). Doklady Akademii Nauk, 37(2):227- 229, 1942.
Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforce- ment learning. In Conference on Robot Learning (CoRL), 2020a.
Pascal Klink, Carlo D' Eramo, Jan R Peters, and Joni Pajarinen. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Neural Information Processing Systems (NeurIPS), 2020b.
Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D' Eraao, Jan Peters, and Joni Pajarinen. A probabilistic interpretation of self-paced learning with applications to reinforcement learning- Journal of Machine Learning Research (JMLR), 22 12):: 2021.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Neural Information Processing Systems (NeurIPS), 2010.
Chang Liu, Jingwei Zhuo, Pengyu Cheng. Ruiyi Zhang, Jun Zhu, and Lawrence Carin. Understand- ing and accelerating particle-based variational inference. In International Conference on Machine Learning (ICML), 2019.
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. In Neural Information Processing Systems (NeurIPS), 2016.
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Eslami, Daniel Hennes, Wojeiech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play 2021.
in simulated humanoid football. arXiv preprint 205.
Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Deyu Meng, Qian Zhao, and Lu Jiang. A theoretical understanding of self-paced learning. Informa- fion Sciences, 414:319-328, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Gaspard Monge. Mémoire sur théorie des déblais et des remblais. De l'Imprimerie Royale, 1781.
Sanmit Narvekar, Bei Peng. Matteo Leonetti, Jivko Sinapov, Matthew Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research (JMLR), 21(181):1-50, 2020.
Radford M Neal. Annealed importance sampling. Statistics and Computing, 11(2): 2001.
Gabriel Peyré, Marco Cuturi, et Computational optimal transport: With applications data science. Foundations and Trends in Machine Learning, 11(5-6))355-077,
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Y Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Conference on Robot Learning (CoRL), 2019.
Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy Lillicrap. Automated curricula through setter-solver interactions. In International Conference Learning Representations (ICLR), 2020.

---

Under review as a conference paper at ICLR 2022
Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre- Yves Oudeyer. Teachmyagent: benchmark for automatic curriculum learning in deep rl. International Conference on Machine Learning (ICML), 2021.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning (ICML), 2015.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3554, 2017.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference Learning Representations (ICLR), 2018.
Richard Sutton and Andrew G Barto. Introduction to Reinforcement Learning. MIT Press, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In International Conference on Intelligent Robots and Systems (IROS). 2012.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jomathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C. J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-22 272, 2020.
Daphna Weinshall and Dan Amir. Theory of curriculum learning, with convex loss functions. Jour- nal of Machine Learning Research (JMLR), 21(222):1-19, 2020.
Andre Wibisono. Sampling as optimization in the space of measures: The langevin dynamics as a composite optimization problem. In Conference on Learning Theory (COL.T), 2018.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In International Conference on Machine Learning (ICML), 2007.
Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A performance-base based start state curriculum frame- work for reinforcement learning. International Conference on Autonomous Agents and Multi- agent Systems (AAMAS), pp. 1503-1511, 2020.
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In International
Conference on Learning Representations (ICLR), 2021.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. In Neural Information Processing Systems (NeurIPS), 2020.
12

---

Under review as a conference paper at ICLR 2022
(a) Gaussian po (c) centered in the middle of the spiral target Pc(
(b) Gaussian (c) located top-left of the spiral target (c().
(c) Sine wave with Gaussian noise po(c).
Figure 10: Wasserstein barycenters between two distributions Po(c) and (c)). The above fig- ures visualize the barycenters (c) arg min, wWP(p.P1) + (1 w)W2(p, po) for 0, 0.2. 0.4, 0.6, 0.8, (from left to right). The target pi(c) is shown in green, while the respec- tive barycenter (c) is shown in blue.
A ILLUSTRATION OF WASSERSTEIN BARYCENTERS
This short appendix serves to showcase the capacity of Wasserstein barycenters to interpolate be- tween complicated target distributions. Figure 10 visualizes different barycenters that interpolate between different source distributions and a target distribution encoding samples contained in a spi- ral. Modelling such a spiral, e.g. via Gaussian mixture models would require many components to accurately model the narrow, curved line to which the samples are constrained. As represent the distributions via set of particles (see appendix C). the barycenter computation does not require knowledge of the log-likelihood function of the target distribution but only requires samples from it, which is beneficial is the specification of the likelihood function is challenging, We see that the barycenters gradually displace the particles to match the target distribution. For the sinusoidal source distribution in Figure 10c] we can see that the idea of minimum action, which is inherent to Wasserstein metrics, creates non-trivial interpolations that are still smooth w.r.t. the defined metric (in this case the euclidean metric on 2
B QUALITATIVE RESULTS
Figure I ssoww trajectories that have been generated by agents trained with different curricula. We see that directly learning on the two target tasks prevents the agent from finding the gates in the wall to pass through. Consequently, the agent minimizes the distance the goal by moving right in front of the wall (but not crashing into it) to accumulate reward over time. We see that random learning indeed generates meaningful behavior. This behavior is, however, not precise enough to pass reliably through the wall. As mentioned in the main paper. G-SPRL and NP-SPRL only learn to pass through one of the gates. We see that in the particular run displayed Figure [1] NP-SPRI. stayed closer to the wall than G-SPRL in those contexts in which it could not pass the wall. This better behavior probably a result of NP-SPRL ultimately sampling both target tasks during training while G-SPRL only focuses on one. Finally, WB-SPRL learns policy that can pass through both gates reliably, showing that the gradual interpolation towards both target tasks allowed to learn both of them. ALP-GMM, PLR and VDS also learn good policies. The generated trajectories are, however, not as precise as the ones learned with WB-SPRL. The reason for this lack of precision is probably that these methods have notion of target distribution and hence cannot focus learning on the target
13

---

Under review as a conference paper at ICLR 2022
(a) Default
(b) Random
(c) G-SPRL
(d) NP-SPRL
WB WB-SPRL
(f) ALP-GMM
(g) GOALGAN
(h) ACL
(i) PLR
(i) VDS
Figure 11: Final trajectories generated by the different investigated curricula in the point mass en- vironment. The color encodes the context: Blue represents gates positioned the left and red gates positioned at the right.
tasks at later iterations. GOALGAN partly creates meaningful behavior. However, this behavior is unreliable and hence leads to low returns due to the agent frequently crashing into the wall. ACL does not learn solve the task. The reason for the poor performance of ACL in this task is probably explained by the fact that ACL models the curriculum generation as a bandit problem, which in the 50 50 grid used in the point mass experiment needs choose between 2500 arms.
Figure 12 shows the distribution (c) generated by NP-SPRI. for different iterations in the pick and place environment. We see that regardless of the target distribution, the interpolation skips all intermediate states of the expert demonstration. This is surprising, given that the interpolation worked at least somewhat desirably in the point mass environment for a Gaussian target distribution. Comparing Figures 12a and 12b we see that the Gaussian target distribution leads to Pa.n (c) putting probability mass on a few more states than only the initial one. However, these states are still so close to the initial state that no learning takes place.
C IMPLEMENTATION DETAILS OF WB-SPRL
The implementation of WB-SPRL relies on the computation of the Wasserstein barycenter (c). To compute this barycenter, we represent the distributions puc). ps(c) and g(c) via a set particles. For distribution p(c). we denote the particle-based representation as ) ΣN cº ~p(c) and & Dirac delta located at cº. This representation allows to compute the so-called Monge maps Tp, p. C + C that encode how to move the particles from the distribution #r to Pa. the case of the Wasserstein-22 2 distances, we can use a linear combination of these Monge maps as reasonable approximation to the exact barycenter (c)
===1 &et, (c) where
pp(c) (BITg-pp + 32 pT {(c)
Pacelt) Fo.a
Pa,af() Fo.n(() (0)
lteration
Iteration
Iteration
Iteration
Iteration 28
Iteration
0.5
0.5
0,5
(a) Target yau (c)
(b) Target pa(C)
Figure 12:
Visualization of the curriculum distribution (c) of NP-SPRL for different iterations the pick and place tasks. (a) shows the curriculum generated for the Gaussian target distribution (c()). (b) shows the curriculum in case of using the Dirac-delta-like distribution (2(c)).
14

---

Under review as a conference paper at ICLR 2022
GENERAL
G-SPRI.
NP-SPRI.
WB-SPRI.
ENV. 8
CKL JcM
VISTE
NOFSET
LB
DKL,x
LLB
DKLE
MAZE
.65
.25
10000
5
.9
POINT-MASS
4
.25 .2
4096
5
[22 .1875]
8000
1000
PNP
.5 .53
3000
TMA
160
-.22
50 EP.
270
Table I: Hyperparameters of the investigated SPRL algorithms in the different learning environ- ments, as described in appendix D Note the differentiation between for G- and NP-SPRI. (<KL.) and WB-SPRI. (6M).
where q is the push-forward of by T, i.e. the application of the weighted Monge maps the particles ạ
= - (c))
The benefit of this approximation is that the maps Tg-pp and Tg-+a only need to be computed once in order to compute (c(c) ) for arbitrary g. This fast computation of pa(c) allows to quickly optimize via objective (7). To make the optimization of Pa(c) even faster, we replace the two Wasserstein distances in by approximations, resulting
N Σ
min MApEROOL
Σ (Tg- - cR 12 s.t Σ J(T
c211 S.
The objective term of above constrained optimization problem a computationally cheap replace- ment for (P(pp. 14). Instead of computing the Wasserstein distance to μr, we simply ask the trans- ported particles of Pa to follow the transport plan Tz=ā as closely as possible. This avoids the (comparatively) expensive computation of W2(p3, (4) for each choice of [3, Finally, also avoid
to compute W2(pp, 4) but rather replace it with a pessimistic substitute NV E ||c, cgll2 S c.
This substitute is guaranteed to be larger than (2 q)² since when using the particle representa- tion, the Wasserstein distance W2(ps, q) corresponds to finding permutation [I, + [1, N] that optimizes
ming V 1 EN ||c - c² (n(m)
Note that the approximations to the Wasserstein distances ( do not alter the transport plan and with that (c). The approximations may simply alter the optimal parameters 3 to (7). However, since the approximations have the same asymptotic behavior, these differences do not hinder good learning performance as can be seen the experimental section. For the computations of the Monge maps, we use the GeomLoss library (Feydy et al.) 2019) which allows to solve entropy regularized optimal transport problems (Cuturi (2013). We then use the SciPy library (Virtanen al., 2020) to adjust the weights B subject to the performance and distance constraints.
D EXPERIMENTAL DETAILS
This section discusses hyperparameters and additional details of the conducted experiments that could not be provided the main text due to space limitations.
D.1 ALGORITHM HYPERPARAMETERS
The two main parameters the three SPRI. algorithms are the performance threshold 5 (and 5µ) as well as the allowed distance between subsequent distributions c (note that we say "distance" as the interpretation different for WB-SPRL than for G- and NP-SPRL). We did not perform an extensive hyperparameter search for these parameters but proceeded as follows: The performance threshold is chosen such that it is around 50% of the maximally achievable reward and the parameter eM is chosen such that around 10 iterations space. We then evaluated larger and a lower value of the parameters and chose the best. For G- and SP-SPRI, initialized ég1. with value of 0.05 used in the initial experiments by Klink et al.
required to move particles across the whole context
15

---

Under review as a conference paper at ICLR 2022
However, realized that larger values slightly improved performance. Additionally, the number of steps per epoch saup (i.e. between an update of the context distribution) and the number of epochs that take place before any context distribution update is allowed Raffset need to be chosen. In our experiments, the sstep parameter is chosen such that at least 20 episodes are completed per context distribution update. This seemed enough for the two- and one-dimensional context spaces consid- ered. For number of completed episodes or to reduce the trust-region parameter, as the regression problem for the expected performance [ c) will probably require more data. The Noliset parameter mostly serves to give the policy π some time to become proficient on easy tasks and the estimator of J J c) to get somewhat accurate. Nonetheless, SPRL can also works without this offset, as can be seen in the TMA benchmark. Hence, Roffser can rather seen as "safery" parameter that interacts with the trust-region parameters M and 'KL as well as Nsspp- For G-SPRL, Klink et al. employed a lower bound on the standard deviation о of the context distribution that needs to be respected until the KL divergence w.r.t. μ(() falls below threshold DEL This avoids a collapse of the context distribution in the case of a very narrow target context distribution p(c). In NP-SPRL,
context spaces, we assume that it is advisable to either increase the
translated this into lower bound the entropy of the context distribution. For WB-SPRL, this lower bound is not required. Only, the additional performance lower bound 8F1 5 for computing the "value distribution" pz(c). Aforementioned parameters are listed in T the different environments.
For ALP-GMM, the relevant hyperparameters are the percentage of random samples drawn from the context space Prand. the number of completed learning episodes between the update of the context distribution Reallout as well as the maximum buffer size of past trajectories to keep Shuffer- Similar to Klink et al. (2021), we chose them by a grid-search over (Prand. Rrollout Shulter) E {0.1, 0.2, 0.33 x (50, 100, 200} x {500, 1000, 2000) for the maze and pick and place environment. For the point-mass environment, we took the values from Klink et al. (2021).
For GOALGAN, we again took the parameters from Klink et al. (022)) for the point-mass environ- ment and tuned the amount of random noise that is added on top of each sample ònoise. the number of policy rollouts between the update of the context distribution Nrlloou as well as the percentage of samples drawn from the success buffer Psuccess for the maze and pick and place task via grid search over (Snoise> Proloout Psnceess E (0.025, 0.05, 0.13 x (50, 100, 200) x {0.1, 0.2.0.3).
Finally, for ACL, the Exp3.S algorithm that ultimately realizes the curriculum requires two hy- perparameters to be chosen: the scale factor for the updates of the arm probabilities and the e parameter of the c-greedy exploration strategy. We combine ACL with the absolute learn- ing progress (ALP) metric also used in ALP-GMM and conduct hyperparameter search over (n,e) E {0.05, 0.1, 0.2} x {0.01, 0.025, 0.05]. In the discrete pick and place (or discretized maze and point-mass) environment, the absolute learning progress in a context с can be estimated by keep- ing track of the last reward obtained in e and then computing the absolute difference between the return obtained from the current policy execution and the stored last reward. Implementing the ACL algorithm by Graves et al. (2017), we had numerical issues due to the normalization of the ALPs via quantiles computed from representative sample. This is probably due the sparse nature of the task. Consequently, we normalized via the maximum and minimum ALP seen over the entire history of tasks.
For PLR, the staleness coefficient p. the score temperature 8 as well as the replay probability need to be chosen. We did a grid-search over Bp. E f0.15, 0.3, 0.45) x {0.15, 0.45) x (0.55, 0.7, 0.85} and chose the best configuration for each environment. For VDS, the parameters for the training of the Q-function ensemble, i.c. the learning rate Ir, the number of epochs Mep and the number of minibatches Thaich, need to be chosen. Just as for PLR, we conducted a grid-search over (Ir, Nep. Tbbuch)) E {10-4, 5x 10-2, 10 10-3} x {3. 5, 10) x {20, 40, 80). We chose the same number of environment steps between training of the Q-function as for the SPRL. algorithms. The parameters of all employed baselines are given in tables and 3
D.2 TASK DESCRIPTIONS
We now detail details of the individual experiments, such as context-, state- and action spaces as well as the employed RL algorithms. As RL agents, we use PPO and SAC implemented in the St Stable Baselines library (Hill et al., 2018). In all tasks, we represent policy- and/or value function networks using neural networks with three hidden layers of 128 neurons and tanh activations.
16

---

Under review as a conference paper at ICLR 2022
ALP-GMM
GOALGAN
ENV.
PRAND
MBOLLOUT
SoUPFER
ÓNOISE
TROLLOUT
PsUCcEsS
MAZE
.2
100
500
.1
100
.3
0.01
0.2
POINT-MASS
-2
100
500
.1
200
.3
0.01
0.2
PNP
-2 50
500
.2
.05
.05
Table 2: Hyperparameters of the investigated baseline algorithms in the different learning environ- ments, as described in appendix D
D.2.1
MAZE
The maze task simulated using MuJoCo (Todorov et al. 2012) by defining a sphere with radius .15 that can move along two prismatic joints along and y-direction. The goal (i.e. the context) can be chosen within = [-3, 15] [-3, 15 C R. The actually reachable space of positions (and with that goals) is subset of [-1, x [-1, 13] due to the "hole" because of the inner walls of the maze. The state of the environment given by the x- and y-position and -velocity of the agent. The reward is sparse, only returning reward of one if the goal is reached and zero otherwise. A goal considered reached if the Euclidean distance between goal and position of the point-mass falls below 0.15.
We use the PPO algorithm for learning in this task, updating the policy every 10.000 enviromment steps for 10 epochs with 20 mini-batches. The parameter À for the generalized advantage estimation is set to 0.995 and the entropy coefficient is set to zero. All other parameters are set to the default of the Stable Baselines implementation. For algorithms that support the specification of an initial context distribution (SPRL variants and GOALGAN)) we initialize the context distribution be uniform over [-1, 1] x [-1,1] for GOALGAN, NP-SPRI. and WB-SPRI. and (0, .06I) for G-SPRL. This initial context distribution samples goals around the initial position of the agent.
D.3 POINT-MASS
The environment setup is the same the investigated by Klink et al. (2020b 2021) with the only difference the target context distributions (c) and (2(c). We use PPO with 4.096 steps per policy update, four update epochs with 8 mini-batches each policy update, A=0.99 and an entropy coefficient of zero. All other parameters are left to the implementation defaults of the Stable Basel ines implementation.
D.4 PICK AND PLACE
The investigated environment is a slightly modified version of the PetchPickAnee ronment from the OpenAI Gym Brockman et al. (2016) environment suite. The dynamics and reward functions are exactly the same as the ones from the original environment. We only modify the initial position of the cube as well as the goal to which the cube needs to be moved. The initial and y-position of the cube the table is sampled uniformly within [1.31, 1.37 [0.72, 0.78) and the goal position is sampled within [1.31, 1.37 x [0.87, 0.93) x [0.59, 0.65). If the environment reset to state in which the cube is in contact with the gripper, its position not randomized as otherwise the cube may c.g. fall out of the gripper due to randomization, which would counteract the idea behind the curriculum.
envi-
PL.
ENV.
P
B
p
I.R
MAZE
3
.3
.7
5x10
10
40
POINT-MASS
33
.3
.7
5x10
PNP
3
.45
.7
5x10-
10
40
Table 3: Hyperparameters of the investigated baseline algorithms in the different learning environ- ments, as described appendix
17

---

Under review as a conference paper at ICLR 2022
The demonstration is generated from simple controller that first aligns the gripper with position 10cm above the cube, then lowers the gripper until the cube is between its two fingers, closes the gripper and then moves the endeffector to the goal position. The control command for the robot endeffector is computed via
u sign(Ax) max (0.2, [Ax)), 4x X - X,
where Xdes is the desired- and the current endeffector position. Lower bounding the control com- mand by 0.2 avoids the movements becoming very small as the endeffector approaches the target position, which would lead unnecessarily long demonstrations. As soon we are within a 0.01 distance to the target position we switch to the next step in the previously outlined sequence.
As an RL agent, we use SAC with steps per policy update, a batch size of 512, a maximum replay buffer size of 100.000 and learning rate 3 3x10- 10-4, All other parameters are left to the implementa- tion defaults of the Stable Basel iines implementation.
As in the maze task, specify initial context distributions for GOALGAN and the investigated SPRL variants. These context distributions assign uniform probability to the last time-steps of the trajec- tory.
18


Reviews:
[{"review_id": "tgBOgMfiN5", "reviewer": "Reviewer_zmCP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found this paper to be well written and I understand the issues with prior work it aims to resolve. This being said, I will caveat my review by saying that I am not familiar enough with optimal transport math to be confident in my ability to review the equations and more technical details.\n\nI enjoyed Figure 6 as it illustrated why working with the wasserstein metric would give a smoother interpolation from unimodal to multi-modal distributions.\n\nThe goal spaces in this paper all seem rather low dimensional and simple. It would be more promising to me if there were experiments with e.g. harder mazes (more invalid regions). What happens when the starting distribution is \"deceiving\" — i.e. assume the particle must first move away from the goal down a corridor, will this notion of KL/wasserstein fail because the only reachable states will be in the opposite direction of the target distribution?\n\nFigure 3: Why do all methods underperform the default at the beginning of training? Is it due to the choice of starting distribution?\n\nIt looks to me like the margin between paragraphs has been editted. Please revert it to what is expected with the ICLR style.\n\nIn the maze task, why does NP-SPRL have non-zero (or closer to zero) mass attributed to the unreachable center part of the maze?\n\nPick and place is solvable without a recorded demonstration with e.g. hindsight experience replay and no curriculum. It's a bit disappointing that the experiments with this environment are with a recorded trajectory.", "summary_of_the_paper": "The authors try to alleviate the need for parametric distributions (namely a Gaussian) from the self paced RL framework (SPRL). They seek to shift current distribution over tasks to the target distribution of tasks under 2 constraints: keeping the current distribution close to the target and also that the expected return of the current distribution remain above some threshold. They compare SPRL using a KL divergence metric between a discretized proposal distribution and the target as well as a Wasserstein distance metric between particle based approximations. They show that in a set of toy tasks with low dimensional goal spaces can outperform existing baselines, e.g. GoalGan, as well as the previous instantiation of SPRL with parametric distributions. ", "summary_of_the_review": "I enjoyed the paper; however, because I'm not familiar with the math I will defer to other reviewers on their assessment on this front. I'm giving this paper a 6 because I felt that in the end it did not prove it's new formulation was better but rather showed it experimentally. For a primarily experimental paper, I felt the experimental domains were rather simplistic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "srsUkHF9xto", "reviewer": "Reviewer_5LxC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n\n1) This paper tackles an important problem, namely that of developing principled automatic curricula for RL. \n\n2) The experiments are well designed and provide useful insights into the learning dynamics and properties of different methods. I particularly liked the analysis showing how the context distributions change throughout training, as well as the interpolation between different distributions, for G-SPRL and NP-SPRL. \n\n3) The paper is clear overall and I found the claims regarding the limitations of NP-SPRL and G-SPRL relative to SPRL to be well supported by the experiments. However, I believe a few of the empirical claims regarding WB-SPRL are not as well motivated (see more details below). \n\n4) Even if the proposed method is a natural extension of SPRL using the Wasserstein metric, this hasn't been done before (as far as I know) and it is well motivated, making it a reasonable contribution at ICLR (in terms of novelty). \n\n### Weaknesses\n\n1) While this paper makes a reasonably convincing case for using WB-SPRL rather than G-SPRL or NP-SPRL, especially when the distribution of contexts isn't Gaussian and the space is high-dimensional, it doesn't convincingly show or explain the benefits of using WB-SPRL over other curriculum-based methods. In particular, ALP-GMM is just as good on two out of the three tasks used for evaluation and almost as good on the remaining one. In addition, the paper doesn't contain an in depth discussion of how WB-SPRL compares with other curriculum-based methods and when one can expect it to work better or why one may prefer to use this method (rather than e.g. ALP-GMM) even if it performs just as well. \n\n2) Most of the evaluation domains used (except for PNP) seem somewhat contrived as if they were designed for NP-SPRL and G-SPRL to fail and WB-SPRL to perform well. Thus, it would be useful to see how these methods perform on more realistic / natural domains e.g. other robotics tasks similar to PNP. \n\n3) Evaluating the methods on other tasks (e.g. other configurations of Maze or even a setting where both the maze layout and the goal location are changing, different configurations of Point Mass as used in Florensa et al. 2019, or other more realistic robotic tasks), and comparing WB-SPRL with other effective curriculum-based methods (e.g. Asymmetric Self-Play by Sukhbaatar et al. 2017 which outperforms GoalGAN on certain tasks or an adaptation of AMIGo by Campero et al. 2020 to continuous control domains), would help better motivate the use of the proposed approach and shed light on when this method can help the most. \n\n4) The paper seems to be missing experiments with ACL-GMM on Maze and Point Mass. Given that ACL-GMM is the best method on PNP, it would be useful to see how it performs on the other domains as well to better understand whether WB-SPRL provides additional gains on those settings. \n\n5) This paper also misses mentions to a lot of the relevant literature on curriculum learning, intrinsic motivation, exploration, or other approaches for solving challenging / sparse reward RL tasks. I found the related work section to be too narrow, brief, and lacking the necessary context to understand the paper's contribution. A discussion of how SPRL and in particular WB-SPRL compares with other approaches is also needed. \n\n### Questions / Requests\n1) Could you provide experiments showing how the performance of NP-SPRL scales with the resolution of the discretization? Additional analysis showing the $p_{\\alpha, \\eta}(c)$ at different iterations would also be valuable. \n2) Do you have any intuition of why ALP-GMM performs just as well as WB-SPRL on Maze? And why does ALP-GMM perform worse on Point Mass?\n3) In Figures 4 and 5, it would be useful to add the insights drawn from these experiments in the caption (for completeness and ease of read).\n4) In Figure 8a, can you show p(t) for the other approaches (e.g. G-SPRL, NP-SPRL)? Can you explain what the colors mean and also add this information in the caption?\n5) Do you have any intuition of why GoalGAN doesn't work as well, especially on Maze and Point Mass, since these domains are very similar to the ones used in the original paper?\n6) When can we expect WB-SPRL to provide a benefit over other curriculum-based approaches such as ALP-GMM or GoalGAN? Or why should one choose to use this method over the alternatives?\n7) Can you include the goal density / context distributions of GoalGAN,ALP-GMM, and WB-SPRL for comparison i.e. include them Figure 2 in order to better understand how the learning dynamics / learned behaviors differ from other SPRL variants?\n", "summary_of_the_paper": "This paper investigates self-paced reinforcement learning (SPRL) which is a type of curriculum-based RL. The authors empirically demonstrate the (different) limitations of a few SPRL variants (one which uses a Gaussian approximation, G-SPRL, and a non-parametric one, NP-SPRL). They propose the use of a Wasserstein metric instead of the KL divergence (WB-SPRL) and show that this leads to higher performance, more desirable behaviors, and more meaningful interpolation between different MDPs / contexts, on three continuous control tasks. The authors also compare their approach with other automatic curriculum algorithms. ", "summary_of_the_review": "Overall, I think this paper provides a good understanding of the different SPRL variants, their limitations, and proposes a new method which provides some empirical benefits (over other SPRL methods) and is theoretically motivated. \n\nMy main concerns are the incomplete related work section and the limited benefits demonstrated by WB-SPRL over other curriculum-based methods. Hence, I believe the paper needs some more work to warrant acceptance at ICLR.  \n\nI'm willing to increase the score if the authors include references to the relevant works from the literature, a discussion of how WB-SPRL compares with other curriculum RL methods (theoretically and empirically), and include more experiments that (more convincingly) demonstrate WB-SPRL's benefits over other approaches (with some intuition of why / in which settings we can expect WB-SPRL to be better). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0cYA8C3VDjU", "reviewer": "Reviewer_w8rz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Correctness\n\nThe paper clearly states the algorithm, intended changes and their purpose as well the experimental details. The change in metric itself makes sense, as demonstrated in the paper, to increase the range of problems to which self-paced reinforcement learning (SPRL) can be applied. The experiments not only show the increased performance using the curriculum update, but also explore how the curriculum itself develops.\n\n### Technical Novelty And Significance\n\nThe paper is incremental by extending SPRL by changing the metric. So, it’s a combination of two existing ideas (which totally makes sense), but the overall concept of the SPRL algorithm stays the same. Although the experiments show a clear performance improvement, the relevance of the results are not fully clear and therefore also not the significance of the proposed SPRL modification; see below for more details.\n\n### Empirical Novelty And Significance\n\nThe authors show extended use cases for the SPRL algorithm and use appropriate baselines, although the experimental scope is still quite limited compared to other curriculum learning papers in RL. Even though target distributions can take arbitrary shapes now, the authors stick to very narrow targets and do not discuss generalization beyond simple bimodal distributions. This limits comparability with other methods and ultimately the significance of the paper’s contributions. \n\n### General comments\n\nSections 5.1 and 5.3 in particular highlight the need for non-parametric target distributions. The PointMass experiment in 5.2, however, was not as convincing as it does not show how well the Wasserstein barycenters (WB-SPRL) approach scales in general. If the target distribution can now be bimodal instead of unimodal, it is still an improvement, but perhaps not as interesting as if 10 or more different gate configurations would be possible. I find this particularly relevant as this is the only experiment in which the naive discretization (NP-SPRL) looks to perform significantly worse than WB-SPRL. Would it be fair to say that for unimodal cases, there is no large performance gap between the two? \nI would rate the overall significance of these findings much higher if WB-SPRL can be expected to perform well beyond the bimodal case. Do you have any intuition on the limitations for context distributions for WB-SPRL?\n\n\nOn a similar line of thoughts, I wonder about the experimental setting. For example, in the experiments on Maze (Section 5.1), the context c was able to encode an infeasible environment. Thinking about real-world applications, I would expect that only feasible contexts and environments are available for training, especially given that we assume a fairly good understanding of the context space (we already know quite a bit about instance difficulties, for example). Furthermore, I believe that the target distribution \\mu should be fairly broad in many cases, since many possible environments have to be covered by a generalizing agent. The target distribution in the paper at hand are all fairly narrow, which might benefit SPRL’s performance compared to other baselines, but also limit the algorithm in practice. Could the authors please comment on the real-world applicability of their SPRL variant in relation to wider distributions of \\mu.\n\nI’d also be interested to know more about SPRL’s hyperparameters. While they are stated in the appendix, there is no clear indication to what degree they affect the performance and if this differs between the variations. How were they chosen in the first place? Was there any hyperparameter optimization involved?\n\nThe related work on curriculum learning in RL could be improved. There have been many publications with good results in the past years beyond SPRL, two additional papers on the topic from 2018 and 2019 do not reflect the state of the art. I recommend that this section should be extended, even though not all work on CL in RL may have exactly the same focus of solving specific hard instances or generating curricula through generating instances as SPRL. Examples include:\n* Automatic Curriculum Learning through Value Disagreement, Zhang et al. https://proceedings.neurips.cc/paper/2020/file/566f0ea4f6c2e947f36795c8f58ba901-Paper.pdf\n* Safe Reinforcement Learning via Curriculum Induction, Turchetta et al.\nhttps://proceedings.neurips.cc/paper/2020/file/8df6a65941e4c9da40a4fb899de65c55-Paper.pdf\n* Prioritized Level Replay, Jiang et al.\nhttp://proceedings.mlr.press/v139/jiang21b/jiang21b.pdf\n* Self-Paced Context Evaluations for Contextual Reinforcement Learning, Eimer et al.\nhttp://proceedings.mlr.press/v139/eimer21a/eimer21a.pdf \n* TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL, Romac et al.\nhttp://proceedings.mlr.press/v139/romac21a.html \n\nCould the authors please also briefly explain in the rebuttal, why there is no empirical comparison against these baselines? Or why a comparison against other baselines made more sense? \n\n### Minor questions and comments:\n* Equation 4: Why can we get rid of \\delta from Equation 3?\n* End of Section 3: Why do you assume 2-Wasserstein distances under an euclidean metric?\n* There’s no reference for contextual RL in the related work section. The oldest reference I’m aware would be Hallak et al. 2015\n* Did you adapt the baseline hyperparameters at all from their original settings? I do not think the appendix states this explicitly. Did you re-implement them or could you use the original code for all of them?\n", "summary_of_the_paper": "The authors extend the Self-Paced Reinforcement Learning, which samples environment instances from a distribution that shifts from an (easy) starting to a (hard) target distribution. The algorithm has previously been shown to enable agents to solve hard environments. Previously, the target distribution was limited to a Gaussian, which the authors now extend using Wasserstein barycenters. This extension allows more elaborate target distributions, e.g. expert demonstrations.", "summary_of_the_review": "The paper proposes a well-motivated extension to the SPRL algorithm and demonstrates its usefulness in interesting experiments. The significance of these results is somewhat unclear, though, as the naive context discretization seems to perform just as well as the proposed metric in every but a very limited case. Additionally, the nature of the work is rather incremental with limited impact and novelty. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper changes the metric in self-paced reinforcement learning to be a Wasserstein distance and shows that this outperforms other metrics in simple toy-like experiments.

Even after discussions with the authors, two major concerns were identified with this submission: First, the proposed modification of the metric appears to be rather incremental with regards to the original paper. Second, the proposed method is only evaluated on relatively simple environments. The approach should be evaluated on more difficult tasks.

Given that there was no strong champion for acceptance among reviewers of this paper and the above mentioned limitations, I recommend rejecting this paper.
Decision: Reject