ES-MAML: Simple Hessian-Free Meta Learning

1 INTRODUCTION . Meta-learning is a paradigm in machine learning that aims to develop models and training algorithms which can quickly adapt to new tasks and data . Our focus in this paper is on meta-learning in reinforcement learning ( RL ) , where data efficiency is of paramount importance because gathering new samples often requires costly simulations or interactions with the real world . A popular technique for RL meta-learning is Model Agnostic Meta Learning ( MAML ) ( Finn et al. , 2017 ; 2018 ) , a model for training an agent which can quickly adapt to new and unknown tasks by performing one ( or a few ) gradient updates in the new environment . We provide a formal description of MAML in Section 2 . MAML has proven to be successful for many applications . However , implementing and running MAML continues to be challenging . One major complication is that the standard version of MAML requires estimating second derivatives of the RL reward function , which is difficult when using backpropagation on stochastic policies ; indeed , the original implementation of MAML ( Finn et al. , 2017 ) did so incorrectly , which spurred the development of unbiased higher-order estimators ( DiCE , ( Foerster et al. , 2018 ) ) and further analysis of the credit assignment mechanism in MAML ( Rothfuss et al. , 2019 ) . Another challenge arises from the high variance inherent in policy gradient methods , which can be ameliorated through control variates such as in T-MAML ( Liu et al. , 2019 ) , through careful adaptive hyperparameter tuning ( Behl et al. , 2019 ; Antoniou et al. , 2019 ) and learning rate annealing ( Loshchilov & Hutter , 2017 ) . To avoid these issues , we propose an alternative approach to MAML based on Evolution Strategies ( ES ) , as opposed to the policy gradient underlying previous MAML algorithms . We provide a detailed discussion of ES in Section 3.1 . ES has several advantages : ∗Equal contribution . †Work performed during Google internship . ‡Work performed during the Google AI Residency Program . http : //g.co/airesidency 1 . Our zero-order formulation of ES-MAML ( Section 3.2 , Algorithm 3 ) does not require estimating any second derivatives . This dodges the many issues caused by estimating second derivatives with backpropagation on stochastic policies ( see Section 2 for details ) . 2 . ES is conceptually much simpler than policy gradients , which also translates to ease of implementation . It does not use backpropagation , so it can be run on CPUs only . 3 . ES is highly flexible with different adaptation operators ( Section 3.3 ) . 4 . ES allows us to use deterministic policies , which can be safer when doing adaptation ( Section 4.3 ) . ES is also capable of learning linear and other compact policies ( Section 4.2 ) . On the point ( 4 ) , a feature of ES algorithms is that exploration takes place in the parameter space . Whereas policy gradient methods are primarily motivated by interactions with the environment through randomized actions , ES is driven by optimization in high-dimensional parameter spaces with an expensive querying model . In the context of MAML , the notions of “ exploration ” and “ task identification ” have thus been shifted to the parameter space instead of the action space . This distinction plays a key role in the stability of the algorithm . One immediate implication is that we can use deterministic policies , unlike policy gradients which is based on stochastic policies . Another difference is that ES uses only the total reward and not the individual state-action pairs within each episode . While this may appear to be a weakness , since less information is being used , we find in practice that it seems to lead to more stable training profiles . This paper is organized as follows . In Section 2 , we give a formal definition of MAML , and discuss related works . In Section 3 , we introduce Evolutionary Strategies and show how ES can be applied to create a new framework for MAML . In Section 4 , we present numerical experiments , highlighting the topics of exploration ( Section 4.1 ) , the utility of compact architectures ( Section 4.2 ) , the stability of deterministic policies ( Section 4.3 ) , and comparisons against existing MAML algorithms in the few-shot regime ( Section 4.4 ) . Additional material can be found in the Appendix . 2 MODEL AGNOSTIC META LEARNING IN RL . We first discuss the original formulation of MAML ( Finn et al. , 2017 ) . Let T be a set of reinforcement learning tasks with common state and action spaces S , A , and P ( T ) a distribution over T . In the standard MAML setting , each task Ti ∈ T has an associated Markov Decision Process ( MDP ) with transition distribution qi ( st+1|st , at ) , an episode length H , and a reward function RTi which maps a trajectory τ = ( s0 , a1 , ... , aH−1 , sH ) to the total reward R ( τ ) . A stochastic policy is a function π : S → P ( A ) which maps states to probability distributions over the action space . A deterministic policy is a function π : S → A . Policies are typically encoded by a neural network with parameters θ , and we often refer to the policy πθ simply by θ . The MAML problem is to find the so-called MAML point ( called also a meta-policy ) , which is a policy θ∗ that can be ‘ adapted ’ quickly to solve an unknown task T ∈ T by taking a ( few ) 1 policy gradient steps with respect to T . The optimization problem to be solved in training ( in its one-shot version ) is thus of the form : max θ J ( θ ) : = ET∼P ( T ) [ Eτ ′∼PT ( τ ′|θ′ ) [ RT ( τ ′ ) ] ] , ( 1 ) where : θ′ = U ( θ , T ) = θ + α∇θEτ∼PT ( τ |θ ) [ RT ( τ ) ] is called the adapted policy for a step size α > 0 and PT ( ·|η ) is a distribution over trajectories given task T ∈ T and conditioned on the policy parameterized by η . Standard MAML approaches are based on the following expression for the gradient of the MAML objective function ( 1 ) to conduct training : ∇θJ ( θ ) = ET∼P ( T ) [ Er′∼PT ( τ ′|θ′ ) [ ∇θ′ logPT ( τ ′|θ′ ) RT ( τ ′ ) ∇θU ( θ , T ) ] ] . ( 2 ) We collectively refer to algorithms based on computing ( 2 ) using policy gradients as PG-MAML . 1We adopt the common convention of defining the adaptation operator with a single gradient step , to simplify notation . It can be extended to multiple steps . Since the adaptation operator U ( θ , T ) contains the policy gradient ∇θEτ∼PT ( τ |θ ) [ R ( τ ) ] , its own gradient∇θU ( θ , T ) is second-order in θ : ∇θU = I+α ∫ PT ( τ |θ ) ∇2θ log πθ ( τ ) RT ( τ ) dτ+α ∫ PT ( τ |θ ) ∇θ log πθ ( τ ) ∇θ log πθ ( τ ) TRT ( τ ) dτ . ( 3 ) Correctly computing the gradient ( 2 ) with the term ( 3 ) using automatic differentiation is known to be tricky . Multiple authors ( Foerster et al. , 2018 ; Rothfuss et al. , 2019 ; Liu et al. , 2019 ) have pointed out that the original implementation of MAML incorrectly estimates the term ( 3 ) , which inadvertently causes the training to lose ‘ pre-adaptation credit assignment ’ . Moreover , even when correctly implemented , the variance when estimating ( 3 ) can be extremely high , which impedes training . To improve on this , extensions to the original MAML include ProMP ( Rothfuss et al. , 2019 ) , which introduces a new low-variance curvature ( LVC ) estimator for the Hessian , and T-MAML ( Liu et al. , 2019 ) , which adds control variates to reduce the variance of the unbiased DiCE estimator ( Foerster et al. , 2018 ) . However , these are not without their drawbacks : the proposed solutions are complicated , the variance of the Hessian estimate remains problematic , and LVC introduces unknown estimator bias . Another issue that arises in PG-MAML is that policies are necessarily stochastic . However , randomized actions can lead to risky exploration behavior when computing the adaptation , especially for robotics applications where the collection of tasks may involve differing system dynamics as opposed to only differing rewards ( Yang et al. , 2019 ) . We explore this further in Section 4.3 . These issues : the difficulty of estimating the Hessian term ( 3 ) , the typically high variance of∇θJ ( θ ) for policy gradient algorithms in general , and the unsuitability of stochastic policies in some domains , lead us to the proposed method ES-MAML in Section 3 . Aside from policy gradients , there have also been biologically-inspired algorithms for MAML , based on concepts such as the Baldwin effect ( Fernando et al. , 2018 ) . However , we note that despite the similar naming , methods such as ‘ Evolvability ES ’ ( Gajewski et al. , 2019 ) bear little resemblance to our proposed ES-MAML . The problem solved by our algorithm is the standard MAML , whereas ( Gajewski et al. , 2019 ) aims to maximize loosely related notions of the diversity of behavioral characteristics . Moreover , ES-MAML and its extensions we consider are all derived notions such as smoothings and approximations , with rigorous mathematical definitions as stated below . 3 ES-MAML ALGORITHMS . Formulating MAML with ES allows us to employ numerous techniques originally developed for enhancing ES , to MAML . We aim to improve both phases of MAML algorithm : the meta-learning training algorithm , and the efficiency of the adaptation operator . 3.1 EVOLUTION STRATEGIES METHODS ( ES ) . Evolution Strategies ( ES ) ( Wierstra et al. , 2008 ; 2014 ) , which recently became popular for RL ( Salimans et al. , 2017 ) , rely on optimizing the smoothing of the blackbox function f : Rd → R , which takes as input parameters θ ∈ Rd of the policy and outputs total discounted ( expected ) reward obtained by an agent applying that policy in the given environment . Instead of optimizing the function f directly , we optimize a smoothed objective . We define the Gaussian smoothing of F as f̃σ ( θ ) = Eg∼N ( 0 , Id ) [ f ( θ + σg ) ] . The gradient of this smoothed objective , sometimes called an ES-gradient , is given as ( see : ( Nesterov & Spokoiny , 2017 ) ) : ∇θf̃σ ( θ ) = 1 σ Eg∼N ( 0 , Id ) [ f ( θ + σg ) g ] . ( 4 ) Note that the gradient can be approximated via Monte Carlo ( MC ) samples : In ES literature the above algorithm is often modified by adding control variates to equation 4 to obtain other unbiased estimators with reduced variance . The forward finite difference ( Forward-FD ) estimator ( Choromanski et al. , 2018 ) is given by subtracting the current policy value f ( θ ) , yielding ∇θf̃σ ( θ ) = 1σEg∼N ( 0 , Id ) [ ( f ( θ + σg ) − f ( θ ) ) g ] . The antithetic estimator ( Nesterov & Spokoiny , 2017 ; Mania et al. , 2018 ) is given by the symmetric difference ∇θf̃σ ( θ ) = 12σEg∼N ( 0 , Id ) [ ( f ( θ + 1 ESGrad ( f , θ , n , σ ) inputs : function f , policy θ , number of perturbations n , precision σ 2 Sample n i.i.d N ( 0 , I ) vectors g1 , . . . , gn ; 3 return 1nσ ∑n i=1 f ( θ + σgi ) gi ; Algorithm 1 : Monte Carlo ES Gradient σg ) − f ( θ − σg ) ) g ] . Notice that the variance of the Forward-FD and antithetic estimators is translation-invariant with respect to f . In practice , the Forward-FD or antithetic estimator is usually preferred over the basic version expressed in equation 4 . In the next sections we will refer to Algorithm 1 for computing the gradient though we emphasize that there are several other recently developed variants of computing ES-gradients as well as applying them for optimization . We describe some of these variants in Section 3.3 and appendix A.3 . A key feature of ES-MAML is that we can directly make use of new enhancements of ES .

Summary: This paper proposes a method, ES-MAML, for optimizing the Model Agnostic Meta Learning (MAML) objective by using Evolution Strategies (ES) gradients instead of policy gradients (PG) as in the previous approaches in the literature. As a result, the use of ES avoids the need of second-order derivative estimation resulted from PG in computing the gradients of the MAML objective; second-order derivatives in MAML are known to be tricky for proper estimation. They also explore ES-MAML with different advanced adaptation operators to improve the ES gradient estimator. They perform empirical study to demonstrate the benefits of ES-MAML as compared with PG-MAML. In particular, they evaluate the comparable algorithms (ES-MAML and variants vs PG-MAML) in terms of exploratory behaviors in sparse-reward environments, adaptation ability, the stability of deterministic policies in unstable environments, and low-K benchmarks. The experimental results are rigorous and promising. They also discuss several potential extensions to ES-MAML in the appendix.