ES-MAML: Simple Hessian-Free Meta Learning
Authors: Xingyou Song
Wenbo Gao
Yuxiang Yang
Krzysztof Choromanski
Aldo Pacchiano
Yunhao Tang
Venue:  (2020)
Keywords: ES
MAML
evolution
strategies
meta
learning
gaussian
perturbation
reinforcement
learning
adaptation

Abstract: We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES). Existing algorithms for MAML are based on policy gradients, and incur significant difficulties when attempting to estimate second derivatives using backpropagation on stochastic policies. We show how ES can be applied to MAML to obtain an algorithm which avoids the problem of estimating second derivatives, and is also conceptually simple and easy to implement. Moreover, ES-MAML can handle new types of nonsmooth adaptation operators, and other techniques for improving performance and estimation of ES methods become applicable. We show empirically that ES-MAML is competitive with existing methods and often yields better adaptation with fewer queries.

Published as a conference paper at ICLR 2020
ES-MAML: SIMPLE HESSIAN-FREE META LEARNING
Xingyou Song", Yuxiang Yang', Krzysztof Choromanski Google Brain (xingyousong, yxyang, kchoro ) coo
Aldo Pacchiano UC Berkeley
@acchianc@berkelege eddu
Wenbo Gao Yunhao Tangl
Columbia University
(wg2279, 2544 Jêcol umb edu
ABSTRACT
We introduce ES-MAMI, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strulegies (ES). Existing algo- rithms for MAML are based on policy gradients, and incur significant difficulties when attempting estimate second derivatives using backpropagation stochas- tic policies. We show how ES can be applied to MAML to obtain an algorithm which avoids the problem of estimating second derivatives, and is also conceptu- ally simple and easy to implement. Moreover, ES-MAML can handle new types of non-smooth adaptation operators, and other techniques for improving perfor- mance and estimation of ES methods become applicable. We show empirically that ES-MAML is competitive with existing methods and often yields better adap- tation with fewer queries.
INTRODUCTION
Meta-learning is a paradigm in machine learning that aims to develop models and training algo- rithms which quickly adapt new tasks and data. Our focus in this paper meta-learning in reinforcement learning (RL), where data efficiency is of paramount importance because gathering new samples often requires costly simulations interactions with the real world. A popular tech- nique for RL meta- meta-learning is Model Agnostic Meta Learning (MAML) (Finn et al. 2017; 2018), model for training an agent which can quickly adapt new and unknown tasks by performing one (or few) gradient updates in the new environment, We provide formal description of MAML in Section
MAML has proven to be successful for many applications. However, implementing and running MAML continues be challenging. One major complication is that the standard version of MAML. requires estimating second derivatives of the RL reward function, which is difficult when using backpropagation on stochastic policies; indeed, the original implementation
MAML (Finn et al, 2017) did so incorrectly, which spurred the development of unbiased higher-order estimators (DiCE, (Forstter et al., (2188) and further analysis of the credit assignment mechanism in MAML (Rothfuss) al. 2019). Another challenge arises from the high variance inherent in policy gradient methods, which can be ameliorated through control variates such as in '-MAML. (Liu et al.) 2019), through careful adaptive hyperparameter tuning (Behl al. 2019 Antoniou et al. 2019) and learning rate annealing (Loshchilov & Hutter 2017).
To avoid these issues, we propose alternative approach to MAML based on Evolution Strategies (ES), as opposed to the policy gradient underlying previous MAML algorithms. We provide a detailed discussion of ES in Section
ES has several advantages:
"Equal contribution.
Work performed during Google internship.
Work performed during the Google AT Residency Program. http::/g.cc/////ee dency

---

Published as a conference paper at ICLR 2020
1. Our zero-order formulation of ES-MAML (Section 3.2 timating any second derivatives. This dodges the many issues caused by estimating second derivatives with backpropagation on stochastic policies (see Section 2 for details).
Algorithm 3 does not require
2. ES is conceptually much simpler than policy gradients, which also translates to ease of implementation. It does not backpropagation, so it can be run CPUs only.
3. ES is highly flexible with different adaptation operators (Section 3.3
4. ES allows us use deterministic policies, which can be safer when doing adaptation (Sec- tion 4.3 ES is also capable of learning linear and other compact policies (Section 4.2).
On the point a feature of ES algorithms is that exploration takes place the parameter space. Whereas policy gradient methods are primarily motivated by interactions with the environment through randomized actions, ES is driven by optimization in high-dimensional parameter spaces with an expensive querying model. the context of MAML, the notions of "exploration" and "task identification" " have thus been shifted to the parameter space instead of the action space. This dis- tinction plays key role in the stability of the algorithm. One immediate implication is that we can use deterministic policies, unlike policy gradients which is based on stochastic policies. Another difference that ES uses only the total reward and not the individual state-action pairs within each episode. While this may appear to be a weakness, since less information is being used, we find in practice that it seems to lead to more stable training profiles.
This paper is organized follows. In Section 2 we give formal definition of MAML, and discuss related works. In Section B] we introduce Evolutionary Strategies and show how ES can be applied to create new framework for MAML. In Section[4] present numerical experiments, highlighting the topics of exploration (Section 4.1 the utility of compact architectures (Section 4.2). the stability
of deterministic policies (Section 4.3) and comparisons against existing MAML algorithms in the few-shot regime (Section 4.4). Additional material can be found in the Appendix.
MODEL AGNOSTIC META LEARNING IN RL
We first discuss the original formulation of MAML (Finn et al. 2017). Let T be a set of rein- forcement learning tasks with common state and action spaces S, A, and P(T) distribution over T. In the standard MAML. setting, each task T; E T has an associated Markov Decision Process (MDP) with transition distribution (8t41 St. at), an episode length H, and a reward function RT, which maps trajectory 7 (so, di,..., -H-1; 8H) to the total reward R(T). A stochastic policy is a function æ : S + ((A) which maps states to probability distributions over the action space. A deterministic policy is a function S + A. Policies typically encoded by neural network with parameters 0, and we often refer to the policy Ta simply by 8.
The MAML problem is to find the so-called MAMI. point (called also meta-policy), which is policy 8* that can be 'adapted' quickly to solve an unknown task T E TT by taking gradient steps with respect to T. The optimization problem to be solved in training (in its one-shot version) is thus of the form:
a (few policy
max J(0) = PT) (r
(RT [Rr
where: 0' U(,T)) 0 Eg [T) is called the adapted policy for step size 0 and Pr-(-ln) is distribution over trajectories given task T T and conditioned the policy parameterized by 77.
Standard MAML approaches are based on the following expression for the gradient of the MAML objective function to conduct training:
VoJ ET-P(T) (mr-p(r11r Ver BF
(2)
We collectively refer to algorithms based on computing 2 using policy gradients as PG-MAML.
We adopt the common convention of defining the adaptation operator with single gradient step, to sim- plify notation. It can he extended multiple steps.
2

---

Published as a conference paper at ICLR 2020
Since the adaptation operator ((0,T) T) contains the policy gradient "(B) its own gradient (OU (0, T) is second-order in 0:
VOU= /
/ Pr log To
(T) log log no (T) R (T
(3)
Correctly computing the gradient (2) with the term (3) using automatic differentiation is known
be tricky. Multiple authors (Foerster et al., 2018 Rothfuss et al. 2019 Liu et al., 2019) have pointed
that the original implementation of MAMI. incorrectly estimates the term 3 which inadver- tently causes the training to lose "pre-adaptation credit assignment'. Moreover, even when correctly implemented. the variance when estimating (3) can be extremely high, which impedes training. To improve on this, extensions to the original MAML include ProMP (Rothfuss et al. 2019), which introduces new low-variance curvature (LVC) estimator for the Hessian, and T-MAML (Liu al.) 2019), which adds control variates to reduce the variance of the unbiased DiCE estimator (Foerster) et al. 2018). However, these are not without their drawbacks: the proposed solutions are com- plicated, the variance of the Hessian estimate remains problematic. and LVC introduces unknown estimator bias.
Another issue that arises in PG-MAML is that policies are necessarily stochastic. However, ran- domized actions can lead to risky exploration behavior when computing the adaptation, especially for roboties applications where the collection of tasks may involve differing system dynamics as opposed to only differing rewards (Yang et al. 2019). We explore this further in Section 4.3
These issues: for policy gradient algorithms in general, and the unsuitability of stochastic policies in some do-
the difficulty of estimating the Hessian term 3 the typically high variance of (
mains, lead to the proposed method ES-MAMI. in Section 3
Aside from policy gradients, there have also been biologically-inspreed algorithms for MAML,, based on concepts such as the Baldwin effect (Fernando al. (0018). However, we note that despite the similar naming, methods such as 'Evolvability ES' (Gajewski et al., 2019) bear little resemblance our proposed ES-MAML. The problem solved by our algorithm is the standard MAML, whereas Gajewski et al.) 2019) aims to maximize loosely related notions of the diversity of behavioral char- acteristics. Moreover, ES-MAMI. and its extensions we consider are all derived notions such as smoothings and approximations, with rigorous mathematical definitions as stated below.
3 ES-MAML ALGORITHMS
Formulating MAML with ES allows us to employ numerous techniques originally developed for enhancing ES, to MAML. We aim to improve both phases of MAML algorithm: the meta-learning training algorithm, and the efficiency of the adaptation operator.
3.1 EVOLUTION STRATEGIES METHODS (ES)
Evolution Strategies (ES) (Wierstra et al. 2008 2008|2014), which recently became popular for RL. (Sal-
imaass al. 2017), rely optimizing the smoothing of the blackbox function Ed + R, which takes as input parameters 0 E IRd of the policy and outputs total discounted (expected) re- ward obtained by an agent applying that policy in the given environment. Instead of optimizing the function f directly, we optimize a smoothed objective. We define the Gaussian smoothing of F as (0) = Eg + ag)]. The gradient of this smoothed objective, sometimes called an ES -SSgeradinnt, given (see: (Nesterov & Spokoiny 2017)):
vof.(0) E E og)g).
Note that the gradient can be approximated via Monte Carlo (MC) samples:
In ES literature the above algorithm is often modified by adding control variates to equation 4 to obtain other unbiased estimators with reduced variance. The forward finte difference (Forward FD)
estimator (Choromanski et al. 20188) is given by subtracting the current policy value f()), yielding
Vafa (0)
AEgg~ [[ (0 + ag) - f(0)g). The antithetic estimator (Nesterov & Spokoiny
2017
Mania et al. 2018 is given by the symmetric difference Vola(0) - (gg-
3

---

Published as conference paper at ICLR 2020
ESGrad (f.0,
inputs: function f. policy 0, number of perturbations n. precision Ø
Sample n i.i.d (0, I) vectors 9
return 120 2i=1 I'O og:)9; )9: Algorithm 1: Monte Carlo ES Gradient
- [g] Notice that the variance of the Forward-FD and antithetic estimators is f(0- canslation-invarie with respect to f. In practice, the Forward-FD antithetic estimator is usually preferred over the basic version expressed equation 4
In the next sections will refer Algorithm 1 for computing the gradient though we emphasize that there are several other recently developed variants of computing ESS-gradients as well as apply- ing them for optimization. We describe some of these variants in Section 3.3 and appendix [A.3 A key feature of ES-MAML is that we can directly make use of new enhancements of ES.
3.2 META-
META-TRAINING TRAINING MAML WITH ES
To formulate MAML the ES framework, we take more abstract viewpoint. For each task T E T, let f() be the (expected) cumulative reward of the policy 0. We treat fr as a blackbox, and make no assumptions on structure (so the task need not even be MDP, and fr may be nonsmooth). The MAMI. problem is then
max (0) = BI-P(T) f (U(O,T)) 0
(5)
As argued in (Liu al., 2019 Rothfuss et 2019) (see also Section 2 major challenge for policy gradient MAML is estimating the Hessian, which is both conceptually subtle and difficult to correctly implement using automatic differentiation. The algorithm we propose obviates the need to calculate any second derivatives, and thus avoids this issue.
Suppose that we can evaluate (or approximate) (0) and (0,T)), but fr and U(, () may be nonsmooth or their gradients may be intractable. We consider the Gaussian smoothing J, of the MAML reward (5), and optimize Ja using ES methods. The gradient (f)( is given by
(J. = ET-P(T) (0.I)
[ (U (U. T))g
(6)
and can be estimated by jointly sampling over (T.g) and evaluating (T + og, T)). This algorithm is specified in Algorithm 2 box, and we refer
(zero-order) ES-MAML.
Data: initial policy to adaptation step size a,
meta step size B, number of queries K
Data: initial policy 0o, meta step size for f = 0,..... do
for t = 0, 1......0 do Sample tasks Ti, Ti...., Tú and iid vectors gl...., gn ~ (O,,
Sample n tasks Ti...., Tn and iid vectors g1..... ...... gn NO,
foreach (T. do
foreach (T,, B:) do
d( ESGRAD( 0, ogi, K,o);
Vi (. (U (0. ogi; T:))
0 0, + ogg ad
end
0: - Pi (0!
0.41 + 0, + on end
8 Σn Di=1 Ugg
end
O.t1 <- 0, + 8 Σn Vigi
Algorithm 2: Zero-Order ES-MAML
(general adaptation operator U()
end
Algorithm Zero-Order ES-MAML with ES- Gradient Adaptation
The standard adaptation operator (-,, T) nonsmooth in our setting, we use the adaptation operator U@0,T) 0 + o ff (0) acting on its smoothing. Expanding the definition of J.. the gradient of the smoothed MAML is then given by
1) is the one-step task gradient. Since fo is permitted to be
(.) 1 ETTPIT
ft 0 (0+0g+ 1 EEn og oh)h] g
(7)
g~\(00.)
4

---

Published as a conference paper at ICLR 2020
This leads to the algorithm that we specify in Algorithm 3 where the adaptation operator U(, T) is itself estimated using the ES gradient in the inner loop.
We can also derive algorithm analogous to PG-MAML by applying first-order method to the MAML reward Er-p(r) aV (0)) directly, without smoothing. The gradient is given by
VJ(0)= = (F
(8)
which corresponds to equation (3) Liu et al. 2019) when expressed in terms of policy gradients. Every term in this expression has a simple Monte Carlo estimator (see Algorithm 4 in the appendix for the MC Hessian estimator). We discuss this algorithm in greater detail in Appendix [A.1] This formulation can be viewed as the "MAML of the smoothing", compared to the "smoothing of the MAML" which is the basis for Algorithm 3 It is the additional smoothing present in equation 6 which eliminates the gradient of (-. (and hence, the Hessian of f²). Just with the Hessian estimation in the original PG-MAML, we find empirically that the MC estimator of the Hessian (Algorithm 4 has high variance, making it often harmful in training. We present some comparisons between A Algorithm 3 and Algorithm 5 with and without the Hessian term, in Appendix A.1.2
Note that when (+, 7') is estimated, such as in Algorithm 3 the resulting estimator for VJ, will in general be biased. This is similar to the estimator bias which occurs in PG-MAML because we do not have access to the true adapted trajectory distribution. We discuss this further in Appendix A.2
3.3 IMPROVING THE ADAPTATION OPERATOR WITH ES
A allows for great flexibility in choosing new adaptation operators. The simplest extension is to modify the ES gradient step: we can draw on general techniques for improving the ES gradient Some other methods are explored below.
estimator, some of which are described in Appendix A.3
3.3. IMPROVED EXPLORATION
Instead of using i.i.d Gaussian vectors to estimate the ES gradient in U') we consider samples constructed according to Determinantal Point Processes (DPP). DPP sampling (Kulesza & Taskar 2012 Wachinger & Golland 2015) is a method of selecting a subset of samples so as to maximize the 'divrrsity' of the subset. has been applied to ES to select perturbations g: so that the gradient estimator has lower variance (Choromanski et al. 2019a). 'he sampling matrix determining DPP sampling can also be data-dependent and use information from the meta-training stage to construct a learned kernel with better properties for the adaptation phase. In the experimental section we show that DPP-ES can help in improving adaptation in MAML.
3.3.2 HILL CLIMBING AND POPULATION SEARCH
Nondifferentiable operators U-, can be also used in Algorithm 2 One particularly interesting example is the local search operator given by U(0, T) = ff
||0' 01l S R). where R > 0 is the search radius. That is, U(0 T) selects the best policy for task T which is in 'neighorrhood' of O. For simplicity, we took the search neighborhood to be the ball B(, R) here, but we may also use more general neighborhoods of O. In general, exactly solving for the maximizer of 1 over (0, Rt) is intractable, but local search can often be well approximated by a hill climbing algorithm. Hill climbing creates a population of candidate policies by perturbing the best observed policy (which is initialized to 8), evaluates the reward fr for each candidate, and then updates the best observed policy. This is repeated for several iterations. A key property of this search method is that the progress is monotonic, so the reward of the returned policy (e, T) will always improve over This does not hold for the stochastic gradient operator, and appears to be beneficial on some difficult problems (see Section 4.1.. It has been claimed that hill climbing and other genetic algorithms (Moriarty et al. 1999) are competitive with gradient- based methods for solving difficult RL tasks (uch et al. 2017] Risi & Stanley 2019). Another stochastic algorithm approximating local search is CMA-E. ES (Hansen et al. 2003 Igel 22003] Krause et al., 2016), which performs more sophisticated search by adapting the covariance matrix of the perturbations.
5

---

Published as a conference paper at ICLR 2020
Figure 1: (a) ES-MAMI. and PG-MAML exploratio behavior. (b) Different exploration methods when K is limited (K = 5 plotted with lighter colors) or large penalties are added on wrong goals.
MANL Exploration
PG MANL Exploration
Four: Corner (K-5, 10)
Four Corner Hard (K=40),
-50
DPP
MC
Mata Totley
-100-
HC
Adspt Polly
130
-200
8 -250
-200-
-550
157
(a)
4 EXPERIMENTS
The performance of MAML algorithms can be evaluated in several ways. One important measure is the performance of the final meta-policy: whether the algorithm can consistently produce meta- policies with better adaptation. In the RL setting, the adaptation of the meta-policy is also a function of the number K of queries used: that is, the number of rollouts used by the adaptation operator U.. The meta-learning goal of data efficiency corresponds to adapting with low K. The speed of the meta-training is also important, and can be measured in several ways: the number of meta- policy updates, wall-clock time, and the number of rollouts used for meta-training. In this section, we present experiments which evaluate various aspects of ES-MAML and PG-MAML in terms of data efficiency (K) and meta- veeaa-taining time. Further details of the environments and hyperparameters are given in Appendix A.7
In the RL setting, the amount of information used drastically decreases if ES methods are applied in comparison the PG setting. To be precise, ES uses only the cumulative reward over an episode, whereas policy gradients use every acaton pair. Intuitively, we may thus expect that ES should have worse sampling complexity because it uses less information for the same number of rollouts. However, it seems that in practice ES often matches or even exceeds policy gradients approaches (Salimans et al., 2017 Mania et al., 2018). Several explanations have been proposed: the PG case, especially with algorithms such PPO, the network must optimize multiple additional surrogate objectives such as entropy bonuses and value functions as well as hyperparameters such as the TD- step number. Furthermore, it has been argued that ES is more robust against delayed rewards, action infrequency, and long time horizons (Salimans et al. 2017). These advantages of ES in traditional we show empirically in this section. ES may lead to additional
RL also transfer MAML, advantages (even if the numbers of rollouts needed in training is comparable with PG ones) in terms of wall-clock time, because does not require backpropagation, and can be parallelized over CPUs.
4.1 EXPLORATION: TARGET ENVIRONMENTS
In this section, we present two experiments environments with very sparse rewards where the meta-policy must exhibit exploratory behavior to determine the correct adaptation.
The four corners benchmark was introduced in (Rothfuss et al. 2019) to demonstrate the weak- nesses of exploration in PG-MAML. An agent on a 2D square receives reward for moving towards selected corner of the square, but only observes rewards once it is sufficiently close to the target corner, making the reward sparse. An effective exploration strategy for this set of tasks is for the meta-policy H* to travel in circular trajectories to observe which corner produces rewards; however, for a single policy to produce this exploration behavior is difficult. In Figure [] we demonstrate the behavior of ES-MAML for adaptation as used in (Rothfuss et al. 2019). the basic version of Algorithm is able to correctly explore and adapt the task by finding the target corner. Morcover, it does not require any modifi- cations to encourage exploration, unlike PG-MAML. We further used K = 10, 5, which caused the performance to drop. For better performance in this low-information environment, we experimented with two different adaptation operators U(: () in Algorithm 2 which are HC (hill climbing) and DPP-ES. The standard ES gradient is denoted MC.
the four corners problem. When K 20. the same number of rollouts

---

Published as a conference paper at ICLR 2020
Furthermore, ES-MAML is not limited to "single goal" exploration. We created more difficult task, six circles, where the agent continuously accrues negative rewards until it reaches six target points "deactivate" them. Solving this task requires the agent to explore in circular trajectories, similar to the trajectory used by PG-MAMI. on the four corners task. We visualize the behavior in Figure Observe that ES-MAML with the HC operator is able develop strategy to explore the target locations.
From Figure I we observed that both oper- ators DPP-ES and HC were able to improve exploration performance. We also created modified task by heavily penalizing incorrect goals, which caused performance to dramati- cally drop for MC and DPP-ES. This is due to the variance from the MC-gradient, which may result in adapted policy that accidentally pro- duces large negative rewards or become stuck in local-optima (i.e. refuse explore due to negative rewards). This is also fixed by the HC adaptation, which enforces non-decreasing wards during adaptation, allowing the ES-MAML to progress.
Figure 2: ES-MAML exploration on six circle task (K = 20). ES 2S-MAML (MC), Circles
ES-MAML (HC), Circles
0-
0
-2
Meta Policy
Adapt. Pollcy
ES-Adapt. Q
4
Additional examples the classic Navigation- task are presented in Appendix A.4 highlighting the differences in exploration behavior between PG-MAML and ES-MAML.
4.2 GOOD ADAPTATION WITH COMPACT ARCHITECTURES
One of the main benefits of ES is due to its ability to train compact linear policies, which can outperform hidden-layer policies. We demonstrate this several benchmark MAML problems in the HalfCheetah and Ant environments in Figure 3 In contrast, (Finn & Levine 2018) observed that PG-MAML empirically and theoretically suggested that training with more deeper layers under SGD increases performance. We demonstrate that on the Forward-Bl Forward-Backward and Goal-Velocy MAML benchmarks, ES-MAML is consistently able to train successful linear policies faster than deep networks. We also show that, for the Forward- -awaar-Backwwrd Ant problem, ES-MAML with the new HC operator is the most performant. Using more compact policies also directly speeds up ES-MAML, since fewer perturbations are needed for gradient estimation.
Figure& 3: The -araad-Bacckwrddd and Goal-Velocin MAML problems. We compare the perfor- mance for Linear (L) policies and policies with one hidden layer (H) for different K.
Fonwer@Backyarrchheeehh
GosiVeelcityCheeeea
ForwardBackwwardant
1000
-50
500
Lowds
800
1100
LHt_x-a0
LDP1C-40
Avg. Rewerd
-150
300
-200
200
-250
200
JUC
20
LX-23
-190-
LFC.6-20 H--CC-20
1000
1500
2000
2500
10--0
3000
400
400
8do
-100
500
1000
1500
ESMAML Iterations
ESMAML herazions
ESMAML Iterations
4.3 DETERMINISTIC POLICIES
We find that deterministic policies often produce more stable behaviors than the stochastic ones that are required for PG, where randomized actions in unstable environments can lead to catastrophic outcomes. In PG, this often mitigated by reducing the entropy bonus, but this has an undesirable side effect of reducing exploration. In contrast, ES-MAML explores in parameter space, which mitigates this issue. To demonstrate this, we use the "Biased-Sensor CartPole" environment from Yang al., 2019). This environment has unstable dynamics and sparse rewards, it requires exploration but is also risky. We see in Figure that ES-MAML able to stably maintain the maximum reward (500).
7

---

Published as a conference paper at ICLR 2020
Figure 4: Stability comparisons of ES and PG the Biased-Sensor CartPole and Swimmer,
Walker2d environments. (L), (H), and (HH) denote linear, one- and two-hidden layer policies.
c.artPolseenooBBias.---
#foreendackward@wiinner K=
[DrwerdBackwerrdWWllee K-2
500
600
400
Avg. Reward
CS MAML
300
200 Aug.
PG MAML
200
LES HE5
0
H45
IFRG
0
50
100
150
-20
-100
1000
18:05
We also include results in TigureAfrron two other environments, Swimmer and Walker2d, for which it is known that surprisingly unstable, and yields better training (Mania et at. 2018). Notice that again find linear policies (L) outperforming policies with one (H) two (HH) hidden layers.
4.4 Low-K K BENCHMARKS
For real-world applications. we may be constrained to use fewer queries K than has typically been demonstrated in previous MAML works. Hence, is of interest compare how ES-MAML com- pares to PG-MAML for adapting with very low K.
One possible concern is that low K might harm ES in particular because it uses only the cumulative rewards; if for example K 5. then the ES adaptation gradient can make use of only 5 values. In comparison, PG-MAML uses H state-action pairs, so for 5, 200, PG-MAML still has 1000 pieces of information available.
However, we find experimentally that the standard ES-MAMI. (Algorithm 3) remains competitive with PG-MAML even in the low-K setting. In Figure we compare FS-MAMI. and PG-MAMI the Brward-Backwwadd and Goal-Vele Velocity tasks across four environments (HalfCheetah, Swimmer, Walker2d, Ant) and two model architectures. While PG-MAML can generally outperform ES- MAML on the Goal- Velocity task, ES-MAML similar or better on the Forward-Backwan task. Moreover, observed that for low K, PG-MAML can be highly unstable (note the wide error bars), with some trajectories failing catastrophically. whereas ES- MAML is relatively stable. This is important consideration in real applications, where the risk of catastrophic failure is undesirable.
Figure 5: Low K comparisons between ES-MAML and PG-MAML.
FB-Cheetah.KK3 FB-Swimmer.K-p
FB-Walke,,K-5
FB-AntK=20
300
220
200
190
200
20-
150
0-
200
2000
200
109
200
424
1000
800
3009
GV-Cheetah,Ke
Gw..Swimmer.K
Gy-Walker,K-5
G-Ant,
0-
652
4.2000
L-30
802
-000
351
650
-200
332
3520
252
550
-300
202
2050
-00
152
-30
102
500
300
/00
1000
z0o ii
do
min
34t3 1035
xin
de
mi
roto 1980 1000
300
603
000
3033
PG-HH
8

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
We have presented a new framework for MAML based on ES algorithms. The ES-MAML approach avoids the problems of Hessian estimation which necessitated complicated alterations in PG-MAML and is straightforward to implement. ES-MAML is flexible in the choice of adaptation operators, and can be augmented with general improvements ES, along with more exotic adaptation op- erators. In particular, ES-MAML can be paired with nonsmooth adaptation operators such as hill climbing, which found empirically yield better exploratory behavior and better performance on sparse-reward environments. FS-MAMM. performs well with linear or compact deterministic policies, which is an advantage when adapting if the state dynamics are possibly unstable.

---

Published as a conference paper at ICLR 2020
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In In- ternational Conference on Learning Representations, 2018.
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Harkirat Singh Behl, Atilim Günes Baydin, and Philip S. Torr. Alpha MAML: adaptive model- agnostic meta-learning. CoRR, hts/1995.077335 2019.
Jose Blanchet, Donald Goldfarb, Garud Iyengar, Fengpei Li, and Chaoxu Zhou. Unbiased simulation for optimizing stochastic function compositions. urXiv:1711.0773 2017.
Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard E. Turner, and Adrian Weller. Structured evolution with compact architectures for scalable policy optimization. In Proceed- ings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässon, Stockholm, Sweden, July 10-15, 2018, pp. 969-977, 2018
Krzysztof Choromanski, Aldo Pacchiano, Jack Parker-Holder, and Yunhao Tang. Struc- tured monte carlo sampling for nonisotropic distributions via determinantal point processes. vrXiv::905.126677 2019a.
Krzysztof Choromanski, Aldo Pacchiano, Jack Parker-Holder, and Yunhao Tang. From complexity to simplicity: Adaptive es- es-active subspaces for blackbox optimization. NeurIPS 2019, 2019b.
Krzysztof Choromanski, Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Deepali Jain, Yuxiang Yang, Atil Iscen, Jasmine Hsu, and Vikas Sindhwani. Provably robust blackbox optimization for reinforcement learning. accepted CoRL 2019, 2019c.
Chrisantha Fernando, Jakub Sygnowski, Simon Osindero, Jane Wang. Tom Schaul, Denis Teplyashin, Pablo Sprechmann, Alexander Pritzel, and Andrei A. Rusu. Meta-learning by the baldwin effect. In Proceedings of the Genetic and Evolutionary Computation Confer- ence Companion, GECCO 2018, Kyoto, Japan, July 15-19, 2018 pp. 109-110, 2018. doi: 145/ 111111111115522555550 3205763
00.1145/220565133255770 URL pttttp::/doi..rree
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1126-1135, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Ad- vances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, 3-8 December 2018, Montréal, Canada., pp. 9537- 9548, 2018
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DiCE: The infinitely differentiable Monte Carlo estimator. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1538, 2018.
Alexander Gajewski, Jeff Clune, Kenneth O. Stanley, and Joel Lehman. Evolvability ES: scalable and direct optimization of evolvability. In Proceedings of the Genetic and Evolutionary Com- putation Conference, GECCO 2019, Prague, Czech Republic, July 13-17, 2019, pp. 107-115, 2019. doi: 03013/33322107733211777 URL https:/ /ttp://...00011111///33332322223
R33 1776
10

---

Published as a conference paper at ICLR 2020
Nikolaus Hansen, Sibylle Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary Computation, 11(1):1-18, 2003.
Mingyi Hong, Zhi-Quan Luo, and Meisam Razaviyayn. Convergence analysis of alternating direc- tion method of multipliers for a family of nonconvex problems. SIAM Journal on Oplimization, 26(1):337-36) 2016.
Christian Igel. Neuroevolution for reinforcement learning using evolution strategies. In 2003 IEEE Congress on Evolutionary Compulation, 2003.
Oswin Krause, Didac Arbones, and Christian Igel. Cma-es with optimal covariance update and storage complexity. Advances in Neural Information Processing Systems, pp. 370-378, 2016.
Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, 5(2-3):123-286, 2012.
Hao Liu, Richard Socher, and Caiming Xiong. Taming MAML: efficient unbiased meta- reinforcement learning. In Proceedings of the 36th International Conference on Machine Learn- ing, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, pp. 4061-4071 2019.
Ilya Loshchilov and Frank Hutter. SGDR:
stochastic gradient descent with warm restarts. In Sth International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings, 2017.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides competitive approach reinforcement learning. Advances Neural Information Processing Systems 31, pp- 1800-1809, 2018.
David Moriarty, Alan Schultz, and John Grefenstette. Evolutionary algorithms for reinforcement learning. Journal of Artificial Intelligence Research, 1 11:241-276, 1999.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):227-566, 2017.
Sebastian Risi and Kenneth Stanley. Deep neuroevolution recurrent and discrete world models. arXiv:1906.0885,, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. httv:/703.38880 2017.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning. arXiv:1712.065677 2017.
Christian Wachinger and Polina Golland. Sampling from determinantal point processes for scalable manifold learning. Information Processing for Medical Imaging, pp. 687-698, 2015.
Mengdi Wang, Ji Liu, and Xingyuan Fang. Accelerating stochastic composition optimization. Jour- nal of Machine Learning Research, 18:1-23, 2017.
Daan Wierstra, Tom Schaul, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation, pp. 3381-3387 2008
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15:949-980, 2014.
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Jie Tan, and Chelsea Finn. Norml: No-reward meta learning. In Proceedings of the 18th International Conference Autonomous Agents and Multi- Agent Systems, AAMAS '1,, Montreal, QC, Canada, May 13-17, 2019, pp. 323-331, 2019.
11

---

Published as a conference paper at ICLR 2020
A.1 FIRST-ORDER MS-MALL
A.1.1 ALGORITHM
Suppose that first apply Gaussian smoothing the task rewards and then form the MAML. problem, so we have J(0) Er~P(T) (U().T)). The function J Js then itself differentiable, and can directly apply first-order methods to it. The classical case where U T) 0 + (V
yields the gradient
J((0) ET-P(T ( (0+ (0) aV
P()
(9)
This is analogous formulas obtained in e-g (Liu al. 2019) for the policy gradient MAML.. We can then approximate this gradient as an input to stochastic first-order methods. An example with standard SGD shown in Algorithm
Data: initial policy 0. adaptation step size a, meta step size (3, number of queries K for = 0, 1.... do
ESHess Of,0,n,o*
Sample n tasks Ti...... Tni
inputs: function f. policy 6, number of perturbations n, precision
foreach T; do
d < ESGRAD( O2 K)
i. vectors gi........
H + ESHESS( K,)
> i= f(O + ag:);
000 0.+a.d,
H
2i=1 f(0 og.)..g.:
d < ESGRAD( 000 K,o);
return
# (H v ---I); I);
Algorithm 4: Monte Carlo ES Hessian
end
0.t1 0. + Σ E H
end
Algorithm 5: First Order ES-MAML
A central problem, as discussed in (Rothfuss et al., 2019 Liu et al. 2019) is the estimation of 2 (?)) However, simple expression exists for this object in the ES setting; it can be shown that
(2 d² (Ba=wio.,| oh)hh") -
(10)
Note that for the vector h, h" the transpose (and unrelated to tasks 2). A basic MC estimator is shown in Algorithm 4 Given an independent estimator for V fr (0 + aV (0)), we can then take the product to obtain an estimator for J.J
A.1.2 EXPERIMENTS WITH FIRST-ORDER ES-MAML
Unlike zero-order ES-MAML (Algorithm 3). the first-order ES-MAML explicitly builds an approx- imation of the Hessian of ft, Given the literature on PG-MAML, we expect that estimating the Hessian V2 (² with Algorithm 4 without any control variates may have high variance. We com- pare two variants of first-order ES-MAMI.
1. The full version (FO-Hessian) specified in Algorithm 5
2. The "first-order approximation' (FO-NoHessian) which ignores the term I+ and approximates the MAML gradient Ey-P(T) ( ( setting ()) in line 5 of Algorithm
This is equivalent to
The results the four corner exploration problem (Section 4.1 and the Borkard-Bacckwrd Ant, using Linear policies, are shown in Figure[Al] On outperformed FO-Hessian, the inclusion of the Hessian term actually slowed convergence. On the four corners task, both FO-Hessian and FO-NoHessian have large error bars, and FO-Hessian slightly outperforms FO-NoHessian.
Ant, FO-NoHessian actually
There is conflicting evidence as to whether the same phenomenon occurs with PG-MAML; (Finn et al. 2017 55.2) found that supervised learning MAMI,, omitting Hessian terms is competitive
12

---

Published as a conference paper at ICLR 2020
Figure A1: Comparisons between the FO-Hessian and FO-NoHessian variants of Algorithm
-150-
Four, Corner (K=10)
300
ForwardBackwardAnt (L.K=40).
FC-Hessien
FD-Hessian
Fo-Nollessian
250
FO-Nottessian
-200
Avg. Reward
Avg. Reward
200
-250
150
100
-300
50
-350-
-400
-50
5000
10000
15000
2000
4000 6000
8000 10000
lterations
Iterations
but slightly worse than the full PG-MAMI, and does not report comparisons with and without the Hessian on RL MAML. (Rothfuss et al.) 2019| Liu et al.] 2019) argue for the importance of the second-order terms in proper credit assignment, but use heavily modified estimators (LVC, control variates; see Section 2 in their experiments, so the performance is not directly comparable to the 'naive' estimator in Algorithm 4 Our interpretation is that Algorithm 4 has high variance, making the Hessian estimates inaccurate, which can slow training relatively "easier' tasks like Forward- Backward walking but possibly increase the exploration on four corners.
HalfCheetah and Ant in Figure A2 In this experiment, the two methods ran on servers with different number of workers measure the score by the total number of rollouts. We found that Fo-Nsan was slightly faster than Algorithm 3 when measured by rollouts on Ant, but FO-NoHessian had notably poor performance when the number of queries was low (K 5) HalfCheetah, and failed to reach similar scores as the others even after running for many more rollouts.
We also compare FO-NoHessian against Algorithm S
available,
Figure A2: Comparisons between FO-NoHessian and Algorithm 3 by rollouts ForwardBackwardAnt
Forwarddaccwardtaaicheetth
FO-HK=40
1000
400
FO-H.K=20
800
20-HK=40
Avg. Reward
300
ZO-H.K=20
600
Avg. Reward
400
200
100
FO-H.K20
FO-HK-5
0
-400-
2D-H.K-20
Z0-1IK55
-600
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
Rollouts
1e7
Rollouts
1e7
A.2 HANDLING ESTIMATOR BIAS
Since the adapted policy U T) generally cannot be evaluated exactly, we cannot easily obtain unbiased estimates of f (U T)). This problem arises for both PG-MAMI. and ES-MAMI.
We consider PG-MAML first as example. In PG-MAML, the adaptation operator is U(O,T)) + a V VolE,~r (r(o) [. In general, we can only obtain an estimate of V [[()) and not its exact value. However, the MAML gradient given by
(0) = (BrTrpTTBBrPPPpp (F(10) Vo log
(11)
which requires exact sampling from the adapted trajectories T' (r( (B) Since this is a nonlinear function of (0, T), we cannot obtain unbiased estimates of ()( by sampling 7 generated by estimate of U U00,7 T)
13

---

Published as a conference paper at ICLR 2020
In the case of ES-MAML, the adaptation operator is ()) = : =8+a () En T; Th for ~ -~(( I), where u(0, h) = + ff ohjh. Clearly, f (u(8, T; h)) is not an unbiased estimator of f (U(O,T))
We may question whether using an unbiased estimator of (O) is likely to improve per- formance. One natural strategy is to reformulate the objective function so as to make the desired estimator unbiased. This happens be the case for the algorithm E-MAML (AI-Shedivat et al. 2018), which treats the adaptation operator as an explicit function of K sampled trajectories and "moves the expectation outside". That is, we now have an adaptation operator U and the objective function becomes
Tk),
Er (
(12)
An unbiased estimator for the E-MAMI. gradient can be obtained by sampling only from T ~ (r((1()) (Al-Shedivat et al. 2018). However, it has been argued that by doing so, E-MAMI does not properly assign credit to the pre-adaptation policy (Rothfuss et al. 2019). Thus, this particular mathematical strategy seems to be disadvantageous for RL.
((E is difficult and while general unbiased estimation methods exist (Blanchet et al. (017), they are often complicated and suffer from high variance. estimator (Rothfuss et al. 2019), which is biased, against the unbiased DiCE estimator (Foerster et al. 2018), for the Hessian term in the MAMI. gradient, and found that the lower variance of LVC produced better performance than DiCE. Alternatively, control variates can be used reduce the variance of the DiCE estimator, which is the approach followed in (Liu et al. 2019).
The problem of finding estimators for dunction-of-expectatiie
the context of MAML, ProMP compares the low variance curvature (LVC)
In the ES framework, the problem can also be formulated to avoid exactly evaluating T-; and hence circumvents the question of estimator bias. We observe an interesting connection between MAML and the stochastic composition problem. Let us define (n T) u(0, T: h) and fg (0) fF + ag). For a given task T,, the MAML reward is given by
F (U = 7 [E,, 7) J
(13)
This is a two-layer nested stochastic composition problem with outer function = Eggg and inner function (-,, T) = E,,n(.TT) An accelerated algorithm (ASC-PG) was developed in (Wang et al. ]017]] for this class of problems. While neither fz nor (., ( T) is smooth, which is assumed in [aan et al. 2017). we can verify that the crucial content of the assumptions hold:
1. En wh (0. 7) T
2. We can define two functions
C (0) - (0) (T) = I+ &
-2 (K))hhh - (O))
such that for any 01, 02,
E ITT (8:) (Bz) JU TV
where JU denotes the Jacobian of (",T), and g, h are independent vectors sampled from (0, I). This follows immediately from equation and equation 10
The ASC-PG algorithm does not immediately extend to the full MAML problem, upon taking an outer expectation over T, the MAML reward J(0) = Eg (IS, (1,,,, (0, ()) is no longer a stochastic composition of the required form. In particular, there are conceptual difficulties when the number tasks in T is infinite. However, it can be used to solve the MAML problem for each task within a consensus framework, such as consensus ADMM (Hong al. 2016).
f
A.3 EXTENSIONS OF ES
In this section, we discuss several general techniques for improving the basic ES gradient estimator (Algorithm []. These can be applied both to the ES gradient of the meta-training (the 'outer loop' of Algorithm 3 and more interestingly, the adaptation operator itself. That is, given ((0 T)
14

---

Published as a conference paper at ICLR 2020
8 (O), replace the estimation of U by ESGRAD line 4 of Algorithm with an improved estimator of V. (f), which even may depend on data collected during the meta-training stage. Many techniques exist for reducing the variance of the estimator such as Quasi Monte Carlo sampling (Choromanski et al. 2018). Aside from variance reduction, there are also methods with special properties.
A.3.1 ACTIVE SUBSPACES
Active Subspaces is a method for finding a low-dimensional subspace where the contribution of the gradient is maximized. Conceptually, the goal is to find and update on-the- fly low-rank subspace L so that the projection V (0) of V (0) into L is maximized and apply Vf instead of (O). This should be done in such way that V (f) does not need to be computed explicitly. Optimizing
liver-dimensional subspaces might be computationally more efficient and can be thought of an example of guided ES methods, where the algorithm is guided how to explore space in the anisotropic way, leveraging its knowledge about function optimization landscape that it gained in the previous steps of optimization. In the context of RL, the active subspace method ASEBO (Choromanski et al., (2019b) was successfully applied to speed up policy training algorithms. This strategy can made data-dependent also in the MAMI. context, by learning an optimal subspace using data from the meta-training stage, and sampling from that subspace in the adaptation step.
A.3.2 REGRESSION- OPTIMIZATION
Regression-Bast Based Optimization (RBO) an alternative method of gradient estimation. From Taylor series expansion we have f(0 + d) f(0) = f d + O(Id By taking multiple finite difference expressions f(0 + d) f(0) for different d, we can recover the gradient by solving a regularized regression problem. The regularization has additional advantage it was shown that the gradient can be recovered even if substantial fraction the rewards f(0 + d) are corrupted (Choromanski et al., 20T9c). Strictly speaking, this is not based on the Gaussian smoothing in ES, but is another method for estimating gradients using only zero-th order evaluations.
A.3.3 EXPERIMENTS
We present preliminary experiment with RBO and ASEBO gradient adaptation in Figure A3 To be precise, the algorithms used are identical to Algorithm 3 except that in line 4, ( ESGRAD is replaced by (f RBO (yielding RBO-MAML) and d ASEBO (yielding ASEBO-MAMI respectively.
Figure A3: RBO-MAML and ASEBO-MAML compared ES-MAML. FB-Swimmer (L,K=40).
50
GoalVelocityCheetah (H,K=5).
LP-Decoding
40
Unbiased
Avg. Reward
30
Avg. Reward
-200
20
-20
0-
-300
ASEBO Adaptation Adaptation
-10-
-350
200
400
600
800
1000
1200
50
100
150 200 250 300 350 400 450 Iterations
Iterations
On the left plot, we test for noise robustness on the paring standard ES-MAML (Algorithm 3) to RBO-MAML. To simulate noisy data, randomly corrupt 25% of the queries fr (0 + ag) used to estimate the adaptation operator ( T) with an enormous additive noise. This is the same type of corruption used in (Choromanski al. 2019c
Swimmer MAML task, com-
15

---

Published as a conference paper at ICLR 2020
Interestingly, RBO does not appear to be more robust against noise than the standard MC estimator, which suggests that the original ES-MAML has some inherent robustness to noise.
On the right plot, compare ASEBO-MAML ES-MAML the Goal- Volo-ielocity HalfCheetah task in the low-K setting. We found that when measured in iterations, ASEBO-MAML outperforms ES-MAML. However, ASEBO requires additional linear algebra operations and thus uses signif- icantly more wall-clock time (not shown in plot) per iteration, so measured by real time, then ES-MAML more effective.
A.4 NAVIGATION-2D EXPLORATION TASK
Navigation-22 (Finn et al. 2017) is classic environment where the agent must explore adapt to the task. The agent is represented by point on a 2D square, and at each time step, receives reward equal to its distance from given target point on the square. Note that unlike the four corners and six circles tasks, the reward for Navigation-2D is dense. We visualize the differing exploration strategies learned by PG-MAMI. and ES-MAML. in Figure A4 Notice that PG-MAML makes many tiny movements in multiple directions to 'triangulate the target location using the differences in reward for different state-action pairs. On the other hand, ES-MAML learns meta-policy such that each perturbation of the meta-policy causes the agent to move in different direction (represented by red paths), so can determine the target location from the total rewards of each path.
Figure A4: Comparing the exploration behavior of PG-MAML and ES-MAML on the Navigation 2D task. We use K 20 queries for each algorithm.
PG-MAML Exploration, 2-D Navigation
ES-MAML Exploration, 2-D Navigation
1.0
0.5
0.5-
-0.5
-0.5
Meta-Policy
Adapt. Policy
Adapt Q
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
10
16

---

Published as a conference paper at ICLR 2020
A.5 PG-MAML RL BENCHMARKS
In Figure A5 we compare MASMA and PG-MAML on the tasks for HalfCheetah, Swimmer, Walker2d, and Ant, using the same values of K that were used in the original experiments of (Finn et al. 2017).
and Goal-Vekk Velocity
Figure A5: Comparisons between ES- ES-MAML and PG-MAML using the queries K from (Finn al, 2017
rB-Cheetah,K-2
FQ Swimmer,
Ti-WWalken.K--
FD-AntK-40
700
360
500
310
300
400
260
400
200
300
100
200
200
190
180
200
200
11:00
2003
3003
Ky-Cheeta
KV55w/mmer.KK22g
GV-Walker, K=2p
G-nntKK44
2200
300
900
-130
251
2000
259
202
150
3000
-10
-30
102
400
900
ES-L
ES-H
FG-H
PG-HH
A.6 REGRESSION AND SUPERVISED LEARNING
MAML has also been applied to supervised learning. We demonstrate ES-MAML on sine regression FFnn al., 2017), where the task is fit a sine curve with unknown amplitude and phase given a set of K pairs (I,, (x;)). The meta-policy must be able to learn that all of tasks have common correctly adapt to unknown sine curve outside of the points 24.
periodic nature, so that
For regression, the loss is the mean-squee eean-squared error (MSE) between the adapted policy Tz( and the true curve f (z). Given data samples {(x,, fx;)) = the empirical loss is ((0) # (f T Note that unlike in reinforcement learning, works, this is by automatic differentiation. Thus, we opt to use Tensorflow to compute the adaptation operator (0, T) in Algorithm This in accordance with the general principle that when gra- dients are available, it is more efficient to use the gradient than to approximate it by zero-order method (Nesterov Spokoiny 2017).
exactly compute V (();; for deep net-
The adaptation step size is = 0.01, which is the same in (Finn al. 2017). For comparison, (Finn et al. 2017) reports that PG-MAML can obtain loss of ~ 0.5 after one adaptation step with / = 5, though it is not specified how many iterations the meta-policy was trained for. ES-MAML approaches the same level of performance, though the number of training iterations required is higher than for the RI. tasks, and surprisingly high for what appears be simpler problem. This is likely again reflection the fact that for problems such regression where the gradients are available, it is more efficient to gradients.
We show several results in Figure A6
As an aside, this leads to a related question of the correct interpretation of the query number in the supervised setting. There is distinction between obtaining data sample (I;, f(;)), and doing a computation (such as a gradient) using that sample. If the main bottleneck collecting the data f(+))( }, then may be satisfied with any algorithm that performs any number of operations on the data, as long as it uses only K samples. On the other hand, in the (on-policy) RL setting, samples cannot typically be 're-used' the same extent, because rollouts sampled with given
17

---

Published as a conference paper at ICLR 2020
Figure A6: The MSE of the adapted policy, for varying number of gradient steps and query number K. Runs are averaged across seeds.
Sine, Varying Grad. Steps (K=20)
Sine, 1 Grad. Step, Varying K
Negative MSE
Negative MSE
-2
-2
-3
-4
-4
Grad. Steps
K=5
Grad. Steps
K=10
-5
3 Grad. Steps
K-20
0
1000
2000
3000
0
1000
2000
3000
Iterations
Iterations
policy me follow unknown distribution ( which reduces their usefulness away from 0. Thus, the corresponding notion to rollouts in the SL setting would be the number of backpropagations (for PG-MAML) or perturbations (for ES-MAML,, but clearly these have different relative costs than doing simulations in RL.
18

---

Published as a conference paper at ICLR 2020
A.7 HYPERPARAMETERS AND SETUPS
A.7.1 ENVIRONMENTS
Unless otherwise explicitly stated, we default to K = 20 and horizon 200 for all RL experiments. We also use the standard reward normalization in (Mania et al. 2018), and use global state nor- malization (i.e. the same mean, standard deviation normalization values for MDP states are shared across workers).
For the Ant environments (Goal-Position Ant, Forward-Backward Ant), there are significant differ- ences in weighting the auxiliary rewards such as control costs, contact costs, and survival rewards across different previous work (c.g. those costs are downweighted in (Finn et al. 2017) whereas the coefficients are vanilla Gym weightings in (Liu et al. 2019)). These auxiliary rewards can lead to local minima, such as the agent staying stationary to collect the survival bonus which may be con- fused with movement progress when presenting training curve. To make sure the agent is explicitly performing the required task, we opted to remove such costs our work and only present the main goal-distance cost and forward-movement reward respectively.
For the other environments, we used default weightings and rewards, since they do not change across previous works.
A.7.2 ES-MAML HYPERPARAMETERS
Let N be the number of possible distinct tasks possible. We sample tasks without replacement, which is important N < 5, as each worker performs adaptations on all possible tasks.
For standard ES-MAML (Algorithm
3
we used the following settings.
Setting
Value
(Total Workers, # Perturbations, # Current Evals)
(300, 150, 150)
(Train Set Size, Task Batch Size, Test Set Size)
(50,5,5) or (N.N.N)
Number of rollouts per parameter
Number of Perturbations per worker
Outer-Loop Precision Parameter
0.1
Adaptation Precision Parameter
0.1
Outer-Loop Step Size
0.01
Adaptation Step Size (a:)
0.05
Hidden Layer Width
32
ES Estimation Type
Forward-FD
Reward Normalization
True
State Normalization
True
For ES-MAML and PG-MAML, we took 3 seeded runs, using the default TRPO hyperparameters found in (Liu et al. 2019).
19


Reviews:
[{"review_id": "rygqNADaKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method, ES-MAML, for optimizing the Model Agnostic Meta Learning (MAML) objective by using Evolution Strategies (ES) gradients instead of policy gradients (PG) as in the previous approaches in the literature. As a result, the use of ES avoids the need of second-order derivative estimation resulted from PG in computing the gradients of the MAML objective; second-order derivatives in MAML are known to be tricky for proper estimation. They also explore ES-MAML with different advanced adaptation operators to improve the ES gradient estimator. They perform empirical study to demonstrate the benefits of ES-MAML as compared with PG-MAML. In particular, they evaluate the comparable algorithms (ES-MAML and variants vs PG-MAML) in terms of exploratory behaviors in sparse-reward environments, adaptation ability, the stability of deterministic policies in unstable environments, and low-K benchmarks. The experimental results are rigorous and promising. They also discuss several potential extensions to ES-MAML in the appendix. \n\nRegarding the theoretical and algorithmic contributions, this paper combines existing techniques from ES and gradient estimators to make ES gradients work for MAML. Thus, I feel that the paper does not provide significantly new results on these dimensions. For example, the substitution of policy gradient for evolution strategies in Eq. (6) is straightforward, and there is no theoretical justification for the choice of algorithmic designs made in the paper.  However, given that the paper attempts to address an important problem (stably optimizing the MAML objective) with interesting perspective (using ES), that the proposed methods are well developed and extended, and that rigorous experiments to evaluate the proposed methods are provided, this paper could be an interesting contribution to the conference where it can encourage different perspective beyond the gradient policy view for MAML problems. \n\nQuestions and comments. \n\n1. On page 3, with reference to the text “These issues: the difficulty of estimating the Hessian term (3), the typically high variance of ∇θJ(θ) for policy gradient algorithms in general, and the unsuitability of stochastic policies in some domains, lead us to the proposed method ES-MAML in Section 3.” I agree that the use of ES gradients avoids the need of second-order derivative estimation; however I am not very sure if we could say that ES-MAML here can address the high variance issue of PG given that ES can also suffer from high variance and that there is a rich literature in reducing variance of PG. \n\n2. Could you clarify which version of PG-MAML was used as the baseline in your experiments? Is this the “vanilla” version from Eq. (2) without any variance reduction techniques (e.g., Rothfuss et al. (2019), Liu et al. (2019)) or did you include one of the variance reduction techniques to the baseline PG-MAML? \n\n3. In section 4.2, with reference to the text “one of the main benefits of ES is due to its ability to train compact linear policies, which can outperform hidden-layer policies”, could you clarify what did this text mean? Did you mean that compact linear policies are better than hidden-layer policies for MAML, or does it mean that ES is not good at training hidden-layer policies, so it can train linear policies better than hidden-layer policies? \n\nMinor comments.  \n1. Page 2, R has not been introduced. \n2. Page 3, Section 3.1: does F mean f?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ryekKugcOr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes ES for the task of Model agnostic meta learning. Instead of the gradient-approximation which requires computing a hessian matrix, MC samples from a search distribution are used to estimate a search direction. The approach is validated on a number of experiments.\n\nUnfortunatly, I am unable to accept this paper for a number of reasons. Mainly that the ES used is inferior and the constant step-size used can have a major effect on the experimental outcome. \n\nAlmost all proper ES literature with real working ES algorithms are missing and ESGrad is more than 20 years behind SOTA in the field. Since ES are central to the paper, an algorithm that would not even be considered a baseline at any conference in that field is difficult to accept.\nThe reason for this is that nowadays all ES use dynamic sample-variances based on progress measures, e.g. Cumulative step-size adaptation and Two-Point-Adaptation as the SOTA. Without this, it can be very difficult to find reasonable solutions.\n\t\nMost important missing references from the ES-field in this context:\n\t\n1. and most importantly The original ES-based RL paper:\nHeidrich-Meisner, Verena, and Christian Igel. \"Neuroevolution strategies for episodic reinforcement learning.\" Journal of Algorithms 64.4 (2009): 152-168.\n\t\n2. CMA-ES and NES\nHansen, N., Müller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary computation, 11(1), 1-18.\nKrause, O., Arbonès, D. R., & Igel, C. (2016). CMA-ES with optimal covariance update and storage complexity. In Advances in Neural Information Processing Systems (pp. 370-378).\nWierstra, D., Schaul, T., Peters, J., & Schmidhuber, J. (2008, June). Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence) (pp. 3381-3387). IEEE.\n\n3. Review of SOTA in large-scale ES:\nVarelas, K., Auger, A., Brockhoff, D., Hansen, N., ElHara, O. A., Semet, Y., ... & Barbaresco, F. (2018, September). A comparative study of large-scale variants of CMA-ES. In International Conference on Parallel Problem Solving from Nature (pp. 3-15). Springer, Cham.\n\n4. Recent developments for noisy functions (also references other relevant algorithms with noise-handling)\nKrause, O. (2019, July). Large-scale noise-resilient evolution-strategies. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 682-690). ACM.\n\nSection 3.2\n-Why should in (7) the same sigma be used as in (6)? Sigma, alpha etc should be learnable parameters learned by the outer ES.\n-3.3.2: you are writing below (1) that rollouts come from a distribution, i.e. are stochastic. How would you implement a hill-climber in the stochastic setting? e.g. consider the case when the rewards are heavy-tailed.\n- using a hill-climber goes completely against the SOTA in ES which showed repeatedly over the last 20 years that hill-climbing is inferior, especially in larger dimension search-spaces (>100).\n\nExperiments:\n- I am not an expert of MAML, but i would not consider this as different tasks, just as different environments for the same task. i.e. a circular running strategy should be optimal for all environments. but when considering different tasks, we would consider different policies to be optimal.\n- The experiments use the same hyper parameters for all variants. However, i am not sure this is a fair comparison. E.g. HC has way more spread over the search-space than the other two methods for a given sigma, with following sample steps allowing for fixing the \"too large\" or \"too small\" spread.\nSince the graph of the objective function is flat in a large area of the search space, the additional exploration through stocasticity alone might explain the results of Figure 1. In this case, the result would be pretty artificial, because real ES would adapt their step-size.\n- Similar holds for the number of samples used by the outer ES (n, but named differently in th appendix?). The gradient-based approaches might require a lot more initial points with a smaller K , especially on the flat surfaces of the objectives.\n- In Figure 3, middle image, why does the green curve appear to have decreasing performance after iteration 200?\n- Figure 3/ 4.2 why do the three settings have different values for number of iterations and K? Why does L-DPP only appear in the third task?\n-Section 4.3 and Figure 4: why is there no L-PG and HH-ES? the only curve which is is available for both algorithms has the same performance.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Ske4fI5znS", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Note: I was asked to write a last-minute review for this paper since the overall rating of the other reviews are not consistent. Therefore, the review is rather brief and I will comment also on concerns raised by the other reviewers.\n\nThe paper introduces a new MAML algorithm based on evolutionary strategies (ES) for reinforcement learning tasks. Compared to prior MAML algorithms requiring an estimation of the Hessian, ES-MAML demonstrated to be more stable and efficient. Overall, the paper is well motivated, well written and uses a sound mathematical formulation of the solution approach. Furthermore, the results are convincing and show quite some promise.\n\nConcerning the remarks from Reviewer #3, I believe that it is totally fair to use here a simple ES algorithm that still shows reasonable performance. Of course, we would expect that other ES algorithms might perform better, but this is clearly not the point of the paper. Furthermore, also other papers [1,2] showed that very simple ES algorithm can perform very well on weight optimization of policies. \n(Remark: since there is no page limit for refs, I would recommend to cite [1,2] in the paper)\n\nI share some concerns from Reviewer #4 regarding the hyperparameters. By now, it is well known that hyperparameter tuning can improve the performance of RL algorithm quite a bit and is sometimes even the main factor for superior performance. The authors wrote in their reply to Reviewer #4: “In fact, we did not perform much tuning,”. I would like to reply: In fact, this is not a very useful answer.  If there was hyperparameter tuning involved, the amount has to be quantified (in the appendix) and the same amount should be applied to all approaches being compared in the paper.\n\nFurthermore, I missed a discussion about the limitations of the approach. For example, I would expect that the approach will fail if the networks get too large (and thus  the parameter space is too large (>1Mio Parameters?)) and the task is fairly complicated such that the parameter space is not too redundant. I think there is a reason why people tried to use ES for optimizing DNNs for decades, but failed, and now nearly everyone uses GD variants. So, the authors should be more explicit about potential failure cases and limitations.\n\nSmall remark: I haven’t found a description of the architectures used in Section 4.4. Since the paper should be self-contained, I would recommend to briefly make this explicit in the appendix.\n\n[1] Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter: Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari. IJCAI 2018: 1419-1426\n[2] Lior Fuks, Noor Awad, Frank Hutter, Marius Lindauer:\nAn Evolution Strategy with Progressive Episode Lengths for Playing Games. IJCAI 2019: 1234-1240", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bygg6qV1sB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new method for model agnostic meta learning (MAML) based on evolution strategies (ES) rather than policy gradients (PG). The proposed method has clear advantages over prior work: it is conceptually much simpler, simpler to implement and is a zero-order method (while PG-MAML requires 2nd order derivatives and differentiation through the update steps).  Also, the method natively allows to incorporate methods from evolution strategies, e.g., to improve exploration. Empirical results are convincing: ES-MAML consistently outperforms PG-MAML (or is at least not worse) on various tasks. Also, ES-MAML seems to be much more robust compared to PG-MAML, which is known to be brittle. The paper is well motivated and well written. The mathematical formalism is precise.\n\n\nComment/questions:\n\n- PG-MAML is known to be very sensitive w.r.t. hyperparameters, is this also the case for ES-MAML? How were good hyperparameters found for ES-MAML?\n- While this work focuses on RL, it would be interesint to see if ES-MAML is also advantages over vanilla MAML for common few-shot learning image classification problems.\n- What’s the efficiency of ES-MAML compared to PG-MAML in terms of wall-clock time?\n- (minor:) multiple times in the paper, \\citep{} and \\citept{} are used incorrectly.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an evolution strategy for solving the MAML problem. Following up on some other evolutionary methods as alternatives for RL algorithms, this ES-MAML algorithm appears to be quite stable and efficient. The idea makes sense, and the experiments appear strong.

The scores of the reviews showed a lot of variance: 1,6,8. Therefore, I asked a 4th reviewer for a tie-breaking review, and he/she gave another 8. The rejecting reviewer mostly took objection to the fact that learning rates / step sizes were not tuned consistently, which can easily change the relative ranking of different ES algorithms. Here, I agree with the authors' rebuttal: the fact that even a simple ES algorithm performs well is very promising, and further tuning would only strengthen that result. Nevertheless, it would be useful to assess the algorithm's sensitivity w.r.t. its learning rate / step size.

In summary, I agree with the tie breaking review and recommend acceptance as a poster.
Decision: Accept (Poster)