Year: 2021

Abstract:
Curriculum reinforcement learning (CRL) allows to solve complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. However, the generation of such task sequences is largely governed by application assumptions, often preventing a theoretical investigation of existing approaches. Recently, Klink et al. (2021) showed how self-paced learning induces a principled interpolation between task distributions in the context of RL, resulting in high learning performance. So far, this interpolation is unfortunately limited to Gaussian distributions. Here, we show that on one side, this parametric restriction is insufficient in many learning cases but that on the other, the interpolation of self-paced RL (SPRL) can be degenerate when not restricted to this parametric form. We show that the introduction of concepts from optimal transport into SPRL prevents aforementioned issues. Experiments demonstrate that the resulting introduction of metric structure into the curriculum allows for a well-behaving non-parametric version of SPRL that leads to stable learning performance across tasks.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) (Sutton & Barto, 1998) has celebrated great successes as a framework for autonomous acquisition of desired behavior. With ever-increasing computational power, this framework and the algorithms developed under it have allowed to create learning agents capable of solving non-trivial long-horizon planning (Mnih et al., 2015; Silver et al., 2017) and control tasks (Akkaya et al., 2019). However, these successes have also highlighted the need for certain forms of regularization, such as leagues in the context of boardgames (Silver et al., 2017), a gradual diversification of simulated training environments for robotic manipulation (Akkaya et al., 2019) or a tailored training pipeline in the context of humanoid control for soccer (Liu et al., 2021). These regularizations help to overcome shortcomings of modern RL agents, such as poor exploratory behavior – a problem that is an active topic of research (Bellemare et al., 2016; Ghavamzadeh et al., 2015; Machado et al., 2020). One can view aforementioned regularizations under the umbrella term of curriculum reinforcement learning (Narvekar et al., 2020), where the idea is to avoid shortcomings of modern (deep) RL agents such as aforementioned poor exploration by learning on a tailored sequence of tasks. Such curricula can materialize in a variety of ways and are motivated from many perspectives in the literature (Andrychowicz et al., 2017; Florensa et al., 2017; Wöhlke et al., 2020). Although the resulting curricula can often be interpreted as a sequence of task distributions, these sequences typically lack a formal connection to the reinforcement learning objective of maximizing the expected reward under a given target task distribution. In a recent line of work, Klink et al. (2021) proposed the idea of self-paced reinforcement learning (SPRL), borrowing from the concept of self-paced learning that has been established in the supervised learning literature (Kumar et al., 2010; Jiang et al., 2015; Meng et al., 2017). Klink et al. showed a connection between a regularized RL objective and a sequence of task distributions that trade-off between yielding high expected reward and tasks likely under the target distribution. This interpolant has, however, so far been restricted to Gaussian distributions (Klink et al., 2020a;b; 2021). While successful in experimental evaluations, this Gaussian assumption clearly imposes a limitation on the flexibility of the curriculum and disconnects the algorithmic implementation from the established theory. This disconnect raises the question whether the observed performance of SPRL is due to the Gaussian approximation.
Contribution: The key insight presented in this paper is that the Gaussian approximation of existing SPRL implementations is indeed important for their empirical performance. We show that
■ Parametric assumptions in SPRL hinder the learning performance in task spaces with nonGaussian target distributions. ■ SPRL can, however, fail to facilitate learning on the target task distributions when leaving these parametric assumptions behind. ■ Equipping SPRL with Wasserstein metrics allows for a flexible, particle-based representation of the task distribution that ensures a meaningful interpolation in aforementioned failure cases, providing higher performance.

heading: 2 RELATED WORK
text: The main focus of this work is on self-paced reinforcement learning (SPRL, Klink et al. (2020a;b; 2021)) that takes the concept of self-paced curriculum learning (Kumar et al., 2010) from supervisedto reinforcement learning (RL). Opposed to supervised learning, where there is ongoing discussion about the mechanics of curricula and their effect in different situations (Weinshall & Amir, 2020; Wu et al., 2021), the mechanics seem to be more agreed upon in RL. In RL, curricula improve learning performance of an agent by adapting the training environments to its proficiency, and with that e.g. bypass poor exploratory behaviour of non-proficient agents. Applications are by now widely spread and different terms have been established. Adaptive Domain Randomization (Akkaya et al., 2019) uses curricula to gradually diversify training parameters of a simulator to facilitate sim-to-real transfer. Unsupervised environment discovery (Dennis et al., 2020; Jiang et al., 2021b;a) similarly aims to efficiently train an agent which is robust to variations in the environment. Automatic curriculum learning methods (Florensa et al., 2017; Sukhbaatar et al., 2018; Florensa et al., 2018; Portelas et al., 2019; Zhang et al., 2020; Racaniere et al., 2020; Eimer et al., 2021), to which SPRL belongs to, particularly focus on improving the learning speed and/or performance of an agent on a set of desired tasks. Curricula are often generated as distributions that maximize a certain surrogate objective, such as learning progress (Baranes & Oudeyer, 2010; Portelas et al., 2019), intermediate task difficulty (Florensa et al., 2018), regret (Jiang et al., 2021b) or disagreement between Q-functions (Zhang et al., 2020). Curriculum generation can also be interpreted as a two-player game (Sukhbaatar et al., 2018). The work by Jiang et al. (2021a) even hints to a link between surrogate objectives and twoplayer games. Opposed to these interpretations, SPRL has been shown to perform an interpolation between task distributions by Klink et al. (2021), allowing to formally relate the effect of SPRL to the concept of annealing in statistics (Neal, 2001) and homotopic continuation methods in optimization (Allgower & Georg, 2003). We wish to add to this formal understanding of SPRL by investigating the interpolation that it produces more closely. As this investigation will lead us to the problem of optimal transport, we wish to point out important literature in this field. Dating back to the work by Monge in the 18th century, optimal transport has been understood as an important fundamental concept touching upon many fields in both theory and application (Liu et al., 2019; Peyré et al., 2019; Chen et al., 2021). In probability theory, optimal transport translates to the so-called Wasserstein metric (Kantorovich, 1942) that compares two distributions under a given metric on the sample space. From a computational perspective, entropy-regularized Wasserstein metrics (Cuturi, 2013) have led to tangible speed-ups in computations revolving around optimal transport and are hence widely applied (Feydy et al., 2019).

heading: 3 PRELIMINARIES
text: This section serves to introduce the necessary background on (contextual) RL, self-paced RL and optimal transport.

heading: 3.1 CONTEXTUAL REINFORCEMENT LEARNING
text: Contextual reinforcement learning (Hallak et al., 2015) can be seen as a conceptual extension to the (single task) reinforcement learning (RL) problem
max π J(π) = max π Ep0(s0),p(st+1|st,at),π(at|st) [ ∞∑ t=0 γtr(st,at) ] , (1)
which aims to maximize the above expected reward objective by finding an optimal a policy π : S×A 7→ R for a given MDP M = ⟨S,A, p, r, p0⟩ with initial state distribution p0 and transition dynamics p. Contextual RL extends this objective to a space of MDPs M(c) = ⟨S,A, pc, rc, p0,c⟩ equipped with a distribution µ:C 7→ R over contextual variables c ∈ C
max π J(π, µ) = max π Eµ(c) [J(π, c)] . (2)
The policy π : S×C×A 7→ R is conditioned on the contextual parameter c. The distribution µ(c) encodes the tasks M(c) that the agent is expected to encounter. Objective J(π, c) in Eq. (2) corresponds to the objective J(π) in Eq. (1) where, however, the initial state distribution p0, the transition dynamics p as well as the reward function r of M are replaced by their counterparts in M(c). This contextual model of optimal decision making is well-suited for learning in multiple related tasks as is the case in multi-task (Wilson et al., 2007), goal-conditioned (Schaul et al., 2015) or curriculum RL (Narvekar et al., 2020).

heading: 3.2 SELF-PACED REINFORCEMENT LEARNING
text: Self-paced reinforcement learning (SPRL) has been introduced by Klink et al. (2020a;b; 2021) as a curriculum RL algorithm that alters the context distribution µ(c) in the contextual RL objective (2) to increase the learning performance of an agent and/or make it less susceptible to local optima of the objective function. SPRL computes a surrogate distribution p : C 7→ R under which to train the RL agent, i.e. optimize J(π, p). This surrogate distribution is found by optimizing the KL divergence to the target distribution µ(c) subject to two constraints (see Klink et al. (2021, Section 8))
min p
DKL (p(c) ∥ µ(c)) s.t. J(π, p) ≥ δ DKL (p(c) ∥ q(c)) ≤ ϵ. (3)
The distribution p(c) balances between tasks likely under the (target) distribution µ(c) and tasks in which the agent currently obtains large rewards. The KL divergence constraint w.r.t. the previous context distribution q(c) prevents large changes in p(c) during subsequent iterations, making the curriculum robust against errors in the estimates of the expected agent performance J(π, c). A particularly interesting aspect of this work is that objective (3) can be interpreted to perform a specific interpolation between the distributions µ(c), q(c) and a maximum entropy distribution pJ(c) ∝ exp(ηJ(π, c)) encoding high reward tasks. This interpolation is given by
pα,η(c) ∝ µ(c) 1 1+α q(c) α 1+α exp(ηJ(π, c)) 1 1+α . (4)
The two parameters α and η controlling the interpolation are the Lagrangian multipliers of the two constraints in objective (3). So far, Klink et al. (2020a;b; 2021) restricted the distribution pα,η(c) to a Gaussian distributions pν(c) = N (c|µ,Σ). In this case, optimizing (3) w.r.t. µ and Σ of pν corresponds to performing an I-projection of the analytic optimal distribution (4) to the Gaussian restriction, i.e. minimizing DKL (pν(c) ∥ pα,η(c)) w.r.t. ν. In this work, we are interested in investigating the distribution pα,η outside of this parametric restriction pν , i.e. truly employing the distribution (4) instead of its I-projection to a Gaussian.

heading: 3.3 OPTIMAL TRANSPORT
text: The problem of optimally transporting density between two distributions has been initially investigated by Monge (1781). As of today, generalizations established by Kantorovich (1942) have led to so called Wasserstein distances as metrics between probability distributions defined on a metric space M = (d, C) with metric d : C × C 7→ R≥0
Wp(p1, p2) = (
inf γ∈Γ(p1,p2) ∫ C×C d(c1, c2) p dγ(c1, c2) )1/p , p ≥ 1
Γ(p1, p2) = { γ : C × C 7→ R≥0 ∣∣∣∣p1(c1) = ∫ γ(c1, c2) dc2, p2(c2) = ∫ γ(c1, c2) dc1} The distance between p1 and p2 results from solving an optimization problem that finds a so-called plan γ. This plan encodes how to equalize p1 and p2 taking into account the cost of moving density between between parts of the space C. This cost is encoded by the metric d. In the following, we
Algorithm 1 Self-Paced Reinforcement Learning Implementations Input: Context dist. p0(c), target context dist. µ(c), performance threshold δ, distance bound ϵ for k = 0 to K do
Agent Improvement: Sample contexts ci ∼ pk(c), i ∈ [1,M ] Train policy π under ci and observe episodic rewards Ri = ∑∞ t=1 rci(st,at), i ∈ [1,M ] Estimate J(π, c) from the dataset D = {(ci, Ri)|i ∈ [1,M ]} Context Distribution Update: G-SPRL Optimize (3) w.r.t. µk+1 and Σk+1, where pk+1(c) = N (c|µk+1,Σk+1) NP-SPRL Optimize (3) w.r.t α and η using a discrete approximation p̄α,η ≈ pα,η(c) (4) WB-SPRL Optimize (7) w.r.t. β to obtain pβ(c)
end for
will always assume to work with 2-Wasserstein distances under euclidean metric, i.e. p = 2 and d(c1, c2) = ∥c1 − c2∥2. Particularly interesting for the investigation conducted in this paper are so called Wasserstein barycenters
pW2(c) = argminp ∑K k=1 wkW2(p, pk), ∑K
k=1 wk = 1, (5) that perform a weighted interpolation between a set of distributions pk by computing a distribution p that minimizes the above sum of Wasserstein distances. As illustrated in appendix A, these barycenters allow to smoothly interpolate between complex distributions w.r.t. the defined metric d.

heading: 4 NON-PARAMETRIC SELF-PACED REINFORCEMENT LEARNING
text: To highlight the importance of a non-parametric variant of SPRL that, nonetheless, respects the metric structure of the context space C, we investigate three versions of SPRL: G-SPRL– the Gaussian version from Klink et al. (2021), NP-SPRL– a faithful implementation of the SPRL objective (3) that computes and uses the distribution pα,η(c) (4) by discretizing the context space C, and WB-SPRL– an instantiation of Wasserstein barycenters in the SPRL framework. As shown in Algorithm 1, all implementations only differ in the computation of the context distributions.

heading: 4.1 NP-SPRL
text: Computing the analytic distribution pα,η(c) and adjusting the parameters α and η via the SPRL objective (3) will require approximations in the general case, as sampling arbitrary continuous distributions is an open research problem (Brooks et al., 2011; Liu & Wang, 2016; Liu et al., 2019; Wibisono, 2018). Using approximate methods resulting from this research would, however, directly interfere with our intent of evaluating the behavior of pα,η(c) as exactly as possible. Consequently, we discretize the continuous context spaces in our experiments in order to faithfully sample and evaluate expectations w.r.t. pα,η(c). This discretization is schematically shown in Figure 1 for one of the evaluation environments. Although such a discretization will clearly not scale gracefully to higher dimensions, it allows us to investigate the “analytic” behavior of SPRL, which, as we will show, can be sub-optimal even for low-dimensional context spaces. With a context space C ⊆ Rd discretized into N cells, we can represent pα,η as a vector p̄α,η∈RN≥0. To sample a continuous context c ∈ C, we can then first sample an index of a cell from p̄α,η and then sample uniformly within this cell to obtain a context c. Further, the KL divergences in the SPRL objective (3) are straightforward to evaluate when working with p̄α,η . To evaluate the expected performance, we simply evaluate the performance measure J(π, c) at the N cell centers c̄n to obtain the vector J̄(π) ∈ RN . With that we obtain J(π, pα,η) = p̄Tα,ηJ̄(π).

heading: 4.2 WB-SPRL
text: To leverage optimal transport in an SPRL style algorithm, we realize the interpolation between the current distribution q(c), target distribution µ(c) and “value distribution” pJ(c) ∝ exp(ηJ(π, c)) by Wasserstein barycenters
pβ(c) = argmin p
(1−β1−β2)W(p(c), µ(c)) + β1W(p(c), pJ(c)) + β2W(p(c), q(c)). (6)
The weights β of the interpolation are adjusted with the goal of minimizing W2(pβ, µ) while ensuring a constraint on expected performance and distance to the previous distribution q(c)
min β
W2(pβ(c), µ(c)) s.t J(π, pβ) ≥ δ W2(pβ(c), q(c)) ≤ ϵ. (7)
A difference w.r.t. SPRL is that the parameter η of the value distribution pJ(c) is not adjusted in optimization problem (7). Instead we adjust η before optimizing (7) such that J(π, pJ) ≥ δH , where δH > δ is another performance threshold. This ensures that with β1 → 1, it holds that J(π, pβ) ≥ δ. We resort to particle-based representations of the distributions when implementing (6) and (7). This allows us to make use of Monge maps to compute pβ(c) efficiently. More details on the implementation of WB-SPRL are provided in appendix C.

heading: 5 EXPERIMENTS
text: The experiments in this section serve to show the need for a realization of SPRL without parametric restrictions on pα,η but to also highlight that NP-SPRL is not necessarily well-suited for this endeavour due its ignorance w.r.t. the metric on C. Experimental details can be found in appendix D. Code is provided in the supplementary material and will be made available upon acceptance. To situate our method in the field of curriculum RL and to showcase how our method performs w.r.t. the current state-of-the art, we further evaluate ACL, GOALGAN, ALP-GMM, VDS and PLR (Graves et al., 2017; Florensa et al., 2018; Portelas et al., 2019; Zhang et al., 2020; Jiang et al., 2021b) alongside the investigated SPRL variants.

heading: 5.1 MAZE
text: We first turn to a sparse-reward, maze-like environment depicted in Figure 1, in which an agent needs to reach a desired goal. Such environments have e.g. been investigated by Florensa et al. (2018). The contexts c ∈ C of this environment encode the goal position to be reached and hence contains unsolvable contexts (goals inside of a wall or in the inner circle of the maze). Defining µ(c) to be uniform over the context space, the curriculum needs to identify and train the agent on the subspace of feasible tasks in order to achieve a good learning performance. This subspace of feasible tasks is highly non-Gaussian, making it an interesting testbed for NP- and WB-SPRL. We discretize the context space C ⊆ R2 of goal positions into a 50 × 50 grid. Figure 2 compares the performance of the different CRL algorithms. We see that both NP- and WB-SPRL perform better than G-SPRL, since the performance constraint Ep(c) [J(π, c)] ≥ δ in objective (3) at some point prevents the Gaussian context distribution of G-SPRL from expanding, as otherwise too many infeasible tasks would be encoded in pν(c) and hence the performance constraint violated. As shown in Figures 2 and 7, the non-parametric versions of SPRL can flexibly assign probability to feasible contexts, resulting in high learning performance.

heading: 5.2 POINT-MASS
text: We now consider the point-mass environment investigated by (Klink et al., 2020a;b; 2021). As shown in Figure 3a, a point mass needs to be steered through a narrow gate to reach a goal position
on the other side of a wall. While Klink et al. only considered a narrow gate at one specific position as the target task, we will investigate a version in which the gate is located at one of two opposing positions c1=[−3 0.5] and c2=[3 0.5], making µ(c) a bi-modal distribution. This again challenges the Gaussian restriction of pν(c) in the G-SPRL algorithm. We again discretize the context space C ⊆ R2 into a grid of 50 bins on each axis. We investigate two target distributions with different log-likelihoods, that nonetheless produce very similar samples
µ1(c) = 1
2 N
( c1, 10 −4I ) + 1
2 N
( c2, 10 −4I )
µ2(c) = { 1 2 , if c ∈ {c1, c2} ≈0 , else.
Note that “≈0” corresponds to a value of exp(−1000), as this ensures that µ(c) is absolutely continuous w.r.t. pα,η(c) and pν(c), which is required to compute the KL divergence between them. Figure 3 shows that the performance of G- and NP-SPRL depends drastically on the choice of target distribution. Further, we see that NP-SPRL does not outperform G-SPRL in this environment, although it should be able to match the bi-modal target distributions, which G-SPRL cannot. Finally, we see that WB-SPRL outperforms both other versions of SPRL and achieves a consistent performance across target distributions. The illustrations in Figures 4 and 5 serve to illustrate the underlying problem. As evident in Figure 4, NP-SPRL is indeed able to match the target distribution µ1(c) correctly while G-SPRL ultimately covers only one mode of the target density µ1(c). As shown in appendix B, NP-SPRL still only learns to solve one of the two tasks likely under µ1(c). Investigating Figure 4b more closely reveals that NP-SPRL only generates a “proper” curriculum for one of the two target tasks (the left one in the images) by gradually interpolating between easy tasks and the target task, while simply incorporating the second target task into pα,η without generating an appropriate curriculum for the agent to learn this task. Figure 5 shows that the curricula generated by Gand NP-SPRL for µ2(c) are even more problematic. G-SPRL increases the variance of the Gaussian context distribution to match the constant (negligible) likelihood that is assigned to the non-target
tasks, while NP-SPRL now interpolates between none of the target tasks. Consequently, neither the agent under G- nor NP-SPRL learns to solve any of the target tasks. The observed shortcomings of Gand NP-SPRL are caused by their use of the KL divergence to measure similarity between distributions. Figure 6 shows that the KL divergence generates interpolations that are highly dependent on the likelihood function of the distributions, as the KL divergence does not take the metric structure of the sample space C into account. Indeed Gaussian distributions encode a (Euclidean) metric via their log-likelihood, explaining the acceptable performance of G- and NP-SPRL for µ1(c). Looking at Figure 6b and the curricula generated by WB-SPRL in Figure 7, we see that the explicit notion of a (Euclidean) metric in WB-SPRL leads to a gradual change in the tasks encoded by the curriculum for both target distributions. This leads to higher learning performance as visualized in Figure 3.

heading: 5.3 PICK AND PLACE
text: Next, we consider the pick-and-place task in the OpenAI gym environment suite (Brockman et al., 2016) in which a robot is tasked to grasp a block on a table and move it to a desired position (see Figure 8). The sparse reward of this environment, only rewarding the robot upon completing the desired task, makes it a very challenging exploration problem, as can be seen in Figure 8 in which (default) SAC does not learn this pick and place task. One way to alleviate such challenging exploration problems is to learn the task via a curriculum of starting states. Such starting states can be obtained from an expert demonstration, i.e. a trajectory τ EXPERT : [0, 1] 7→ S. A curriculum over this trajectory is then formally defined by choosing the context space to be the unit interval C = [0, 1] ⊆ R. The contextual parameter c only influences the initial state distribution p0,c = δs=τ EXPERT(c). We investigate the NP- and WB-SPRL algorithms in this setting by recording one execution of a hand-crafted controller that first moves the end-effector above the block to be grasped, then lowers the end-effector, grasps the block and moves to the target. We slightly randomize the position of the goal as well as the block to enforce that the agent learns a robust policy. In a realistic scenario, we only record the expert demonstration at a discrete set of states {ti|i = 1, . . . , N} and hence we have a truly discrete CRL setting in this experiment very well suited for NP-SPRL. Furthermore, the finite number of
contexts avoids the necessity for function approximators to approximate the expected performance J(π, c) in a context c, as we can simply estimate the performance in each (discrete) context via a sliding window. We again investigate two target context distributions, one being a narrow Gaussian target distribution with mean at c = 0 and negligible variance (µ1(c)) and the other being again a Dirac-delta with a negligible amount of probability in any context to be absolutely continuous w.r.t. any distribution (µ2(c)). The results in Figure 8 show that WB-SPRL generates a curriculum that allows the agent to learn a reliable policy by smoothly moving probability from later steps of the expert demonstration towards earlier ones. This learned policy is more reliable (25% vs. 100% success rate) and faster ( 69 vs. 13 steps for task completion) than the demonstration. Both default learning (always starting from the initial state) and NP-SPRL do not lead to successful policies. In appendix B, we show that the non-effectiveness of NP-SPRL is again grounded in the degenerate interpolation that puts no probability mass at intermediate time-steps regardless of the target distribution.

heading: 6 TEACHMYAGENT BENCHMARK
text: A final environment for evaluation is the TeachMyAgent benchmark proposed by (Romac et al., 2021). In their evaluations accompanying the benchmark, Romac et al. reported a poor performance of G-SPRL compared to other automatic curriculum RL methods. Consequently, we want to investigate the improvement of WB-SPRL in this benchmark to verify the observed benefits and identify further directions for improving the SPRL framework. Figure 9 shows that WB-SPRL significantly improves upon G-SPRL, achieving top performance across the methods in the mostly unfeasible scenario. We see that WB-SPRL particularly lacks behind ALP-GMM in the mostly trivial setting. Figure 9c shows that WB-SPRL quickly reduces to uniform sampling in this setting (on average after 60 out of 370 iterations), as it can incorporate all tasks while fulfilling the desired performance threshold. This observation explains the similar performance between random sampling and WB-SPRL in the mostly trivial setting and further highlights an important conceptual difference between SPRL and
ALP-GMM: The maximization of learning progress in ALP-GMM leads to a prioritized sampling of the target (uniform distribution), efficiently avoiding tasks the agent has already learned. SPRL does not have a notion of avoiding “too easy” tasks as of now. Introducing such a prioritized sampling of p(c) is hence an interesting future research direction and may allow to combine the benefits of the SPRL framework with other successful curriculum RL methods.

heading: 7 CONCLUSION
text: We investigated self-paced reinforcement learning (SPRL) outside of its current parametric restrictions, revealing important shortcomings of the KL divergence for measuring the similarity of (task) distributions in curriculum reinforcement learning. We showed that replacing the KL divergences, originally employed in SPRL, with Wasserstein distances alleviates the observed problems of SPRL. Apart from achieving higher empirical performance in the SPRL framework, our results indicate that introducing a notion of metric structure on the task space may be an important next step for deriving a principled, yet practical, understanding of curriculum RL. Our findings motivate a variety of future investigations, such as more elaborate implementations of WB-SPRL that e.g. directly move the individual particles of the context distribution via the gradient of the performance measure ∇cJ(π, c) instead of using the value distribution pJ(c) as a proxy. As highlighted in Section 6, importance-sampling of the context distributions are expected to further improve performance. Finally, investigating other metrics than the Euclidean one considered in this paper is expected to allow for principled interpolations in non-Euclidean spaces.

heading: B QUALITATIVE RESULTS
text: Figure 11 shows trajectories that have been generated by agents trained with different curricula. We see that directly learning on the two target tasks prevents the agent from finding the gates in the wall to pass through. Consequently, the agent minimizes the distance to the goal by moving right in front of the wall (but not crashing into it) to accumulate reward over time. We see that random learning indeed generates meaningful behavior. This behavior is, however, not precise enough to pass reliably through the wall. As mentioned in the main paper, G-SPRL and NP-SPRL only learn to pass through one of the gates. We see that in the particular run displayed in Figure 11, NP-SPRL stayed closer to the wall than G-SPRL in those contexts in which it could not pass the wall. This better behavior is probably a result of NP-SPRL ultimately sampling both target tasks during training while G-SPRL only focuses on one. Finally, WB-SPRL learns a policy that can pass through both gates reliably, showing that the gradual interpolation towards both target tasks allowed to learn both of them. ALP-GMM, PLR and VDS also learn good policies. The generated trajectories are, however, not as precise as the ones learned with WB-SPRL. The reason for this lack of precision is probably that these methods have no notion of target distribution and hence cannot focus learning on the target
tasks at later iterations. GOALGAN partly creates meaningful behavior. However, this behavior is unreliable and hence leads to low returns due to the agent frequently crashing into the wall. ACL does not learn to solve the task. The reason for the poor performance of ACL in this task is probably explained by the fact that ACL models the curriculum generation as a bandit problem, which in the 50× 50 grid used in the point mass experiment needs to choose between 2500 arms. Figure 12 shows the distribution pα,η(c) generated by NP-SPRL for different iterations in the pick and place environment. We see that regardless of the target distribution, the interpolation skips all intermediate states of the expert demonstration. This is surprising, given that the interpolation worked at least somewhat desirably in the point mass environment for a Gaussian target distribution. Comparing Figures 12a and 12b, we see that the Gaussian target distribution leads to pα,η(c) putting probability mass on a few more states than only the initial one. However, these states are still so close to the initial state that no learning takes place.
C IMPLEMENTATION DETAILS OF WB-SPRL
The implementation of WB-SPRL relies on the computation of the Wasserstein barycenter pβ(c). To compute this barycenter, we represent the distributions µ(c), pJ(c) and q(c) via a set of particles. For a distribution p(c), we denote the particle-based representation as p̃(c)= 1N ∑N n=1 δcpn(c) where cpn∼p(c) and δcpn is a Dirac delta located at cpn. This representation allows to compute the so-called Monge maps Tp̃1→p̃k : C 7→ C that encode how to move the particles from the distribution p̃1 to p̃k. In the case of the Wasserstein-2 distances, we can use a linear combination of these Monge maps as a reasonable approximation to the exact barycenter pβ(c)
where T#q̃ is the push-forward of q̃ by T , i.e. the application of the weighted Monge maps to the particles of q̃
T#q̃(c) = 1 N ∑N n=1 δT (cqn)(c).
The benefit of this approximation is that the maps Tq̃→p̃J and Tq̃→µ̃ only need to be computed once in order to compute p̃β(c) for arbitrary β. This fast computation of p̃β(c) allows to quickly optimize β via objective (7). To make the optimization of p̃β(c) even faster, we replace the two Wasserstein distances in (7) by approximations, resulting in
min wµ,wJ N∑ n=1 ∥Tq̃→µ̃(cqn)− c pβ n ∥2 s.t 1 N N∑ n=1 J(π, c pβ n ) ≥ δ 1 N N∑ n=1 ∥cpβn − cqn∥2 ≤ ϵ.
The objective term of above constrained optimization problem is a computationally cheap replacement for W2(pβ, µ). Instead of computing the Wasserstein distance to µ, we simply ask the transported particles of p̃β to follow the transport plan Tq̃→µ̃ as closely as possible. This avoids the (comparatively) expensive computation of W2(pβ, µ) for each choice of β. Finally, we also avoid to compute W2(pβ, q) but rather replace it with a pessimistic substitute 1N ∑N n=1 ∥c pβ n − cqn∥2 ≤ ϵ. This substitute is guaranteed to be larger than W2(pβ, q)2 since when using the particle representation, the Wasserstein distance W2(pβ, q) corresponds to finding a permutation π : [1, N ] 7→ [1, N ] that optimizes
minπ √ 1 N ∑N n=1 ∥c pW n − cqπ(n)∥2.
Note that the approximations to the Wasserstein distances in (7) do not alter the transport plan and with that p̃β(c). The approximations may simply alter the optimal parameters β∗ to (7). However, since the approximations have the same asymptotic behavior, these differences do not hinder good learning performance as can be seen in the experimental section. For the computations of the Monge maps, we use the GeomLoss library (Feydy et al., 2019) which allows to solve entropy regularized optimal transport problems (Cuturi, 2013). We then use the SciPy library (Virtanen et al., 2020) to adjust the weights β subject to the performance and distance constraints.

heading: D EXPERIMENTAL DETAILS
text: This section discusses hyperparameters and additional details of the conducted experiments that could not be provided in the main text due to space limitations.
D.1 ALGORITHM HYPERPARAMETERS
The two main parameters of the three SPRL algorithms are the performance threshold δ (and δH ) as well as the allowed distance between subsequent distributions ϵ (note that we say “distance” as the interpretation is different for WB-SPRL than for G- and NP-SPRL). We did not perform an extensive hyperparameter search for these parameters but proceeded as follows: The performance threshold δ is chosen such that it is around 50% of the maximally achievable reward and the parameter ϵM is chosen such that around 10 iterations are required to move particles across the whole context space. We then evaluated a larger and a lower value of the parameters and chose the best. For Gand NP-SPRL, we initialized ϵKL with value of 0.05 used in the initial experiments by Klink et al.
However, we realized that larger values slightly improved performance. Additionally, the number of steps per epoch nstep (i.e. between an update of the context distribution) and the number of epochs that take place before any context distribution update is allowed noffset need to be chosen. In our experiments, the nstep parameter is chosen such that at least 20 episodes are completed per context distribution update. This seemed enough for the two- and one-dimensional context spaces considered. For higher-dimensional context spaces, we assume that it is advisable to either increase the number of completed episodes or to reduce the trust-region parameter, as the regression problem for the expected performance J(π, c) will probably require more data. The noffset parameter mostly serves to give the policy π some time to become proficient on easy tasks and the estimator of J(π, c) to get somewhat accurate. Nonetheless, SPRL can also works without this offset, as can be seen in the TMA benchmark. Hence, noffset can be rather seen as a “safety” parameter that interacts with the trust-region parameters ϵM and ϵKL as well as nstep. For G-SPRL, Klink et al. employed a lower bound on the standard deviation σlb of the context distribution that needs to be respected until the KL divergence w.r.t. µ(c) falls below a threshold DKL. This avoids a collapse of the context distribution in the case of a very narrow target context distribution µ(c). In NP-SPRL, we translated this into a lower bound on the entropy of the context distribution. For WB-SPRL, this lower bound is not required. Only, the additional performance lower bound δH > δ for computing the “value distribution” pJ(c). Aforementioned parameters are listed in Table 1 for the different environments. For ALP-GMM, the relevant hyperparameters are the percentage of random samples drawn from the context space prand, the number of completed learning episodes between the update of the context distribution nrollout as well as the maximum buffer size of past trajectories to keep sbuffer. Similar to Klink et al. (2021), we chose them by a grid-search over (prand, nrollout, sbuffer) ∈ {0.1, 0.2, 0.3} × {50, 100, 200} × {500, 1000, 2000} for the maze and pick and place environment. For the point-mass environment, we took the values from Klink et al. (2021). For GOALGAN, we again took the parameters from Klink et al. (2021) for the point-mass environment and tuned the amount of random noise that is added on top of each sample δnoise, the number of policy rollouts between the update of the context distribution nrollout as well as the percentage of samples drawn from the success buffer psuccess for the maze and pick and place task via a grid search over (δnoise, nrollout, psuccess) ∈ {0.025, 0.05, 0.1} × {50, 100, 200} × {0.1, 0.2, 0.3}. Finally, for ACL, the Exp3.S algorithm that ultimately realizes the curriculum requires two hyperparameters to be chosen: the scale factor for the updates of the arm probabilities η and the ϵ parameter of the ϵ-greedy exploration strategy. We combine ACL with the absolute learning progress (ALP) metric also used in ALP-GMM and conduct a hyperparameter search over (η, ϵ) ∈ {0.05, 0.1, 0.2} × {0.01, 0.025, 0.05}. In the discrete pick and place (or discretized maze and point-mass) environment, the absolute learning progress in a context c can be estimated by keeping track of the last reward obtained in c and then computing the absolute difference between the return obtained from the current policy execution and the stored last reward. Implementing the ACL algorithm by Graves et al. (2017), we had numerical issues due to the normalization of the ALPs via quantiles computed from a representative sample. This is probably due to the sparse nature of the task. Consequently, we normalized via the maximum and minimum ALP seen over the entire history of tasks. For PLR, the staleness coefficient ρ, the score temperature β as well as the replay probability p need to be chosen. We did a grid-search over (ρ, β, p) ∈ {0.15, 0.3, 0.45} × {0.15, 0.3, 0.45} × {0.55, 0.7, 0.85} and chose the best configuration for each environment. For VDS, the parameters for the training of the Q-function ensemble, i.e. the learning rate lr, the number of epochs nep and the number of minibatches nbatch, need to be chosen. Just as for PLR, we conducted a grid-search over (lr, nep, nbatch) ∈ {10−4, 5×10−4, 10−3} × {3, 5, 10} × {20, 40, 80}. We chose the same number of environment steps between training of the Q-function as for the SPRL algorithms. The parameters of all employed baselines are given in tables 2 and 3.
D.2 TASK DESCRIPTIONS
We now detail details of the individual experiments, such as context-, state- and action spaces as well as the employed RL algorithms. As RL agents, we use PPO and SAC implemented in the Stable Baselines library (Hill et al., 2018). In all tasks, we represent policy- and/or value function networks using neural networks with three hidden layers of 128 neurons and tanh activations.
D.2.1 MAZE
The maze task is simulated using MuJoCo (Todorov et al., 2012) by defining a sphere with radius .15 that can move along two prismatic joints along x- and y-direction. The goal (i.e. the context) can be chosen within C = [−3, 15]× [−3, 15] ⊆ R. The actually reachable space of positions (and with that goals) is a subset of [−1, 13]× [−1, 13] due to the “hole” because of the inner walls of the maze. The state of the environment is given by the x- and y-position and -velocity of the agent. The reward is sparse, only returning a reward of one if the goal is reached and zero otherwise. A goal is considered reached if the Euclidean distance between goal and position of the point-mass falls below 0.15. We use the PPO algorithm for learning in this task, updating the policy every 10.000 environment steps for 10 epochs with 20 mini-batches. The parameter λ for the generalized advantage estimation is set to 0.995 and the entropy coefficient is set to zero. All other parameters are set to the default of the Stable Baselines implementation. For algorithms that support the specification of an initial context distribution (SPRL variants and GOALGAN), we initialize the context distribution to be uniform over [−1, 1]× [−1, 1] for GOALGAN, NP-SPRL and WB-SPRL and N (0, .06I) for G-SPRL. This initial context distribution samples goals around the initial position of the agent.
D.3 POINT-MASS
The environment setup is the same as the one investigated by Klink et al. (2020b; 2021) with the only difference in the target context distributions µ1(c) and µ2(c). We use PPO with 4.096 steps per policy update, four update epochs with 8 mini-batches in each policy update, λ=0.99 and an entropy coefficient of zero. All other parameters are left to the implementation defaults of the Stable Baselines implementation.
D.4 PICK AND PLACE
The investigated environment is a slightly modified version of the FetchPickAndPlace environment from the OpenAI Gym Brockman et al. (2016) environment suite. The dynamics and reward functions are exactly the same as the ones from the original environment. We only modify the initial position of the cube as well as the goal to which the cube needs to be moved. The initial x- and y-position of the cube on the table is sampled uniformly within [1.31, 1.37] × [0.72, 0.78] and the goal position is sampled within [1.31, 1.37]× [0.87, 0.93]× [0.59, 0.65]. If the environment is reset to a state in which the cube is in contact with the gripper, its position is not randomized as otherwise the cube may e.g. fall out of the gripper due to randomization, which would counteract the idea behind the curriculum.
The demonstration is generated from a simple controller that first aligns the gripper with a position 10cm above the cube, then lowers the gripper until the cube is between its two fingers, closes the gripper and then moves the endeffector to the goal position. The control command for the robot endeffector is computed via
u = sign(∆x)max(0.2, |∆x|), ∆x = xdes − x,
where xdes is the desired- and x the current endeffector position. Lower bounding the control command by 0.2 avoids the movements becoming very small as the endeffector approaches the target position, which would lead to unnecessarily long demonstrations. As soon as we are within a 0.01 distance to the target position we switch to the next step in the previously outlined sequence. As an RL agent, we use SAC with 5 steps per policy update, a batch size of 512, a maximum replay buffer size of 100.000 and a learning rate 3×10−4. All other parameters are left to the implementation defaults of the Stable Baselines implementation. As in the maze task, we specify initial context distributions for GOALGAN and the investigated SPRL variants. These context distributions assign uniform probability to the last 4 time-steps of the trajectory.

Review summaries:
The authors extend the Self-Paced Reinforcement Learning, which samples environment instances from a distribution that shifts from an (easy) starting to a (hard) target distribution. The algorithm has previously been shown to enable agents to solve hard environments. Previously, the target distribution was limited to a Gaussian, which the authors now extend using Wasserstein barycenters. This extension allows more elaborate target distributions, e.g. expert demonstrations.

The authors try to alleviate the need for parametric distributions (namely a Gaussian) from the self paced RL framework (SPRL). They seek to shift current distribution over tasks to the target distribution of tasks under 2 constraints: keeping the current distribution close to the target and also that the expected return of the current distribution remain above some threshold. They compare SPRL using a KL divergence metric between a discretized proposal distribution and the target as well as a Wasserstein distance metric between particle based approximations. They show that in a set of toy tasks with low dimensional goal spaces can outperform existing baselines, e.g. GoalGan, as well as the previous instantiation of SPRL with parametric distributions. 

This paper investigates self-paced reinforcement learning (SPRL) which is a type of curriculum-based RL. The authors empirically demonstrate the (different) limitations of a few SPRL variants (one which uses a Gaussian approximation, G-SPRL, and a non-parametric one, NP-SPRL). They propose the use of a Wasserstein metric instead of the KL divergence (WB-SPRL) and show that this leads to higher performance, more desirable behaviors, and more meaningful interpolation between different MDPs / contexts, on three continuous control tasks. The authors also compare their approach with other automatic curriculum algorithms.