Metrics Matter: A Closer Look on Self-Paced Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) has celebrated great successes as a framework for autonomous acquisition of desired behavior . With ever-increasing computational power , this framework and the algorithms developed under it have allowed to create learning agents capable of solving non-trivial long-horizon planning ( Mnih et al. , 2015 ; Silver et al. , 2017 ) and control tasks ( Akkaya et al. , 2019 ) . However , these successes have also highlighted the need for certain forms of regularization , such as leagues in the context of boardgames ( Silver et al. , 2017 ) , a gradual diversification of simulated training environments for robotic manipulation ( Akkaya et al. , 2019 ) or a tailored training pipeline in the context of humanoid control for soccer ( Liu et al. , 2021 ) . These regularizations help to overcome shortcomings of modern RL agents , such as poor exploratory behavior – a problem that is an active topic of research ( Bellemare et al. , 2016 ; Ghavamzadeh et al. , 2015 ; Machado et al. , 2020 ) . One can view aforementioned regularizations under the umbrella term of curriculum reinforcement learning ( Narvekar et al. , 2020 ) , where the idea is to avoid shortcomings of modern ( deep ) RL agents such as aforementioned poor exploration by learning on a tailored sequence of tasks . Such curricula can materialize in a variety of ways and are motivated from many perspectives in the literature ( Andrychowicz et al. , 2017 ; Florensa et al. , 2017 ; Wöhlke et al. , 2020 ) . Although the resulting curricula can often be interpreted as a sequence of task distributions , these sequences typically lack a formal connection to the reinforcement learning objective of maximizing the expected reward under a given target task distribution . In a recent line of work , Klink et al . ( 2021 ) proposed the idea of self-paced reinforcement learning ( SPRL ) , borrowing from the concept of self-paced learning that has been established in the supervised learning literature ( Kumar et al. , 2010 ; Jiang et al. , 2015 ; Meng et al. , 2017 ) . Klink et al . showed a connection between a regularized RL objective and a sequence of task distributions that trade-off between yielding high expected reward and tasks likely under the target distribution . This interpolant has , however , so far been restricted to Gaussian distributions ( Klink et al. , 2020a ; b ; 2021 ) . While successful in experimental evaluations , this Gaussian assumption clearly imposes a limitation on the flexibility of the curriculum and disconnects the algorithmic implementation from the established theory . This disconnect raises the question whether the observed performance of SPRL is due to the Gaussian approximation . Contribution : The key insight presented in this paper is that the Gaussian approximation of existing SPRL implementations is indeed important for their empirical performance . We show that ■ Parametric assumptions in SPRL hinder the learning performance in task spaces with nonGaussian target distributions . ■ SPRL can , however , fail to facilitate learning on the target task distributions when leaving these parametric assumptions behind . ■ Equipping SPRL with Wasserstein metrics allows for a flexible , particle-based representation of the task distribution that ensures a meaningful interpolation in aforementioned failure cases , providing higher performance . 2 RELATED WORK . The main focus of this work is on self-paced reinforcement learning ( SPRL , Klink et al . ( 2020a ; b ; 2021 ) ) that takes the concept of self-paced curriculum learning ( Kumar et al. , 2010 ) from supervisedto reinforcement learning ( RL ) . Opposed to supervised learning , where there is ongoing discussion about the mechanics of curricula and their effect in different situations ( Weinshall & Amir , 2020 ; Wu et al. , 2021 ) , the mechanics seem to be more agreed upon in RL . In RL , curricula improve learning performance of an agent by adapting the training environments to its proficiency , and with that e.g . bypass poor exploratory behaviour of non-proficient agents . Applications are by now widely spread and different terms have been established . Adaptive Domain Randomization ( Akkaya et al. , 2019 ) uses curricula to gradually diversify training parameters of a simulator to facilitate sim-to-real transfer . Unsupervised environment discovery ( Dennis et al. , 2020 ; Jiang et al. , 2021b ; a ) similarly aims to efficiently train an agent which is robust to variations in the environment . Automatic curriculum learning methods ( Florensa et al. , 2017 ; Sukhbaatar et al. , 2018 ; Florensa et al. , 2018 ; Portelas et al. , 2019 ; Zhang et al. , 2020 ; Racaniere et al. , 2020 ; Eimer et al. , 2021 ) , to which SPRL belongs to , particularly focus on improving the learning speed and/or performance of an agent on a set of desired tasks . Curricula are often generated as distributions that maximize a certain surrogate objective , such as learning progress ( Baranes & Oudeyer , 2010 ; Portelas et al. , 2019 ) , intermediate task difficulty ( Florensa et al. , 2018 ) , regret ( Jiang et al. , 2021b ) or disagreement between Q-functions ( Zhang et al. , 2020 ) . Curriculum generation can also be interpreted as a two-player game ( Sukhbaatar et al. , 2018 ) . The work by Jiang et al . ( 2021a ) even hints to a link between surrogate objectives and twoplayer games . Opposed to these interpretations , SPRL has been shown to perform an interpolation between task distributions by Klink et al . ( 2021 ) , allowing to formally relate the effect of SPRL to the concept of annealing in statistics ( Neal , 2001 ) and homotopic continuation methods in optimization ( Allgower & Georg , 2003 ) . We wish to add to this formal understanding of SPRL by investigating the interpolation that it produces more closely . As this investigation will lead us to the problem of optimal transport , we wish to point out important literature in this field . Dating back to the work by Monge in the 18th century , optimal transport has been understood as an important fundamental concept touching upon many fields in both theory and application ( Liu et al. , 2019 ; Peyré et al. , 2019 ; Chen et al. , 2021 ) . In probability theory , optimal transport translates to the so-called Wasserstein metric ( Kantorovich , 1942 ) that compares two distributions under a given metric on the sample space . From a computational perspective , entropy-regularized Wasserstein metrics ( Cuturi , 2013 ) have led to tangible speed-ups in computations revolving around optimal transport and are hence widely applied ( Feydy et al. , 2019 ) . 3 PRELIMINARIES . This section serves to introduce the necessary background on ( contextual ) RL , self-paced RL and optimal transport . 3.1 CONTEXTUAL REINFORCEMENT LEARNING . Contextual reinforcement learning ( Hallak et al. , 2015 ) can be seen as a conceptual extension to the ( single task ) reinforcement learning ( RL ) problem max π J ( π ) = max π Ep0 ( s0 ) , p ( st+1|st , at ) , π ( at|st ) [ ∞∑ t=0 γtr ( st , at ) ] , ( 1 ) which aims to maximize the above expected reward objective by finding an optimal a policy π : S×A 7→ R for a given MDP M = ⟨S , A , p , r , p0⟩ with initial state distribution p0 and transition dynamics p. Contextual RL extends this objective to a space of MDPs M ( c ) = ⟨S , A , pc , rc , p0 , c⟩ equipped with a distribution µ : C 7→ R over contextual variables c ∈ C max π J ( π , µ ) = max π Eµ ( c ) [ J ( π , c ) ] . ( 2 ) The policy π : S×C×A 7→ R is conditioned on the contextual parameter c. The distribution µ ( c ) encodes the tasks M ( c ) that the agent is expected to encounter . Objective J ( π , c ) in Eq . ( 2 ) corresponds to the objective J ( π ) in Eq . ( 1 ) where , however , the initial state distribution p0 , the transition dynamics p as well as the reward function r of M are replaced by their counterparts in M ( c ) . This contextual model of optimal decision making is well-suited for learning in multiple related tasks as is the case in multi-task ( Wilson et al. , 2007 ) , goal-conditioned ( Schaul et al. , 2015 ) or curriculum RL ( Narvekar et al. , 2020 ) . 3.2 SELF-PACED REINFORCEMENT LEARNING . Self-paced reinforcement learning ( SPRL ) has been introduced by Klink et al . ( 2020a ; b ; 2021 ) as a curriculum RL algorithm that alters the context distribution µ ( c ) in the contextual RL objective ( 2 ) to increase the learning performance of an agent and/or make it less susceptible to local optima of the objective function . SPRL computes a surrogate distribution p : C 7→ R under which to train the RL agent , i.e . optimize J ( π , p ) . This surrogate distribution is found by optimizing the KL divergence to the target distribution µ ( c ) subject to two constraints ( see Klink et al . ( 2021 , Section 8 ) ) min p DKL ( p ( c ) ∥ µ ( c ) ) s.t . J ( π , p ) ≥ δ DKL ( p ( c ) ∥ q ( c ) ) ≤ ϵ . ( 3 ) The distribution p ( c ) balances between tasks likely under the ( target ) distribution µ ( c ) and tasks in which the agent currently obtains large rewards . The KL divergence constraint w.r.t . the previous context distribution q ( c ) prevents large changes in p ( c ) during subsequent iterations , making the curriculum robust against errors in the estimates of the expected agent performance J ( π , c ) . A particularly interesting aspect of this work is that objective ( 3 ) can be interpreted to perform a specific interpolation between the distributions µ ( c ) , q ( c ) and a maximum entropy distribution pJ ( c ) ∝ exp ( ηJ ( π , c ) ) encoding high reward tasks . This interpolation is given by pα , η ( c ) ∝ µ ( c ) 1 1+α q ( c ) α 1+α exp ( ηJ ( π , c ) ) 1 1+α . ( 4 ) The two parameters α and η controlling the interpolation are the Lagrangian multipliers of the two constraints in objective ( 3 ) . So far , Klink et al . ( 2020a ; b ; 2021 ) restricted the distribution pα , η ( c ) to a Gaussian distributions pν ( c ) = N ( c|µ , Σ ) . In this case , optimizing ( 3 ) w.r.t . µ and Σ of pν corresponds to performing an I-projection of the analytic optimal distribution ( 4 ) to the Gaussian restriction , i.e . minimizing DKL ( pν ( c ) ∥ pα , η ( c ) ) w.r.t . ν . In this work , we are interested in investigating the distribution pα , η outside of this parametric restriction pν , i.e . truly employing the distribution ( 4 ) instead of its I-projection to a Gaussian .

Summary: This paper investigates self-paced reinforcement learning (SPRL) which is a type of curriculum-based RL. The authors empirically demonstrate the (different) limitations of a few SPRL variants (one which uses a Gaussian approximation, G-SPRL, and a non-parametric one, NP-SPRL). They propose the use of a Wasserstein metric instead of the KL divergence (WB-SPRL) and show that this leads to higher performance, more desirable behaviors, and more meaningful interpolation between different MDPs / contexts, on three continuous control tasks. The authors also compare their approach with other automatic curriculum algorithms.