Improving Generalization in Meta Reinforcement Learning using Learned Objectives
Authors: Louis Kirsch
Sjoerd van Steenkiste
Juergen Schmidhuber
Venue:  (2020)
Keywords: meta reinforcement learning
meta learning
reinforcement learning

Abstract: Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

Published as a conference paper at ICLR 2020
IMPROVING GENERALIZATION IN META REINFORCE- MENT LEARNING USING LEARNED OBJECTIVES
Louis Kirsch, Sjoerd van Steenkiste, Jürgen Schmidhuber The Swiss Lab IDSIA, USI, SUPSI (1 (louis, s, soorrd, juergen) dsia a.sia.c.
ABSTRACT
Biological evolution has distilled the experiences many learners into the gen- eral learning algorithms of humans. Our novel meta reinforcement learning algo- rithm MetaGenRL is inspired by this process. MetaGemRL distills the experiences of many complex agents to meta-learn low-complexity neural objective func- tion that decides how future individuals will learn. Unlike recent meta-RL. algo- generalize
rithms, MetaGenRL from those used for meta-training. In some cases, it even outperforms human- engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.
environments that entirely different
INTRODUCTION
The process of evolution has equipped humans with incredibly general learning algorithms. They enable us solve a wide range of problems, even in the absence of large number of related prior experiences. The algorithms that give rise to these capabilities are the result of distilling the collec- tive experiences of many learners throughout the course natural evolution. By essentially learning from learning experiences in this way, the resulting knowledge can be compactly encoded in the ge- netic code of an individual to give rise to the general learning capabilities that we observe today.
In contrast, Reinforcement Learning (RL) in artificial agents rarely proceeds in this way. The learn- ing rules that are used to train agents are the result of years of human engincering and design, (e.g. Williams (1992); Wierstra et al. (2008); Mnih et al. (2013); Lillicrap et al. (2016); Schulman et al. (2015a)). Correspondingly. artificial agents are inherently limited by the ability of the designer to incorporate the right inductive biases in order to learn from previous experiences.
Several works have proposed an alternative framework based on meta reinforcement learn- ing (Schmidhuber, 1994; Wang al., 2016; Duan et al., 2016; Finn al., 2017: Houthooft al., 2018; Clune, 2019). Meta-RI. distinguishes between learning to act in the environment (the rein- forcement learning problem) and learning to learn (the meta-learning problem). Hence, learning itself is now learning problem, which in principle allows one to leverage prior learning experi- ences to meta-learn general learning rules that surpass human-engineered alternatives. However, while prior work found that learning rules could be meta-learned that generalize to slightly different environments or goals (Finn et al., 2017; Plappert et al., 2018; Houthooft al., 2018), generalization to entirely different environments remains open problem.
In this paper we present MetaGenRI. novel meta reinforcement learning algorithm that meta- learns learning rules that generalize to entirely different environments. MetaGenRI. is inspired by the process of natural evolution as it distills the experiences of many agents into the parameters of objective function that decides how future individuals will learn. Similar to Evolved Policy Gradients (EPG; Houthooft et al. (2018)), it meta-leams low complexity neural objective functions that can be used train complex agents with many parameters. However, unlike EPG, it is able to meta-learn using second-order gradients, which offers several advantages as we will demonstrate.
We evaluate MetaGenRL variety of continuous control tasks and compare RI.² (Wang al., 2016; Duan et al., 2016) and EPG in addition to several human engineered learning algorithms.
Cooee available http://t///ooooooo rach. cwm htt////dd//metttenrr

---

Published as a conference paper at ICLR 2020
Compared to RL2 we find that MetaGenRL does not overfit and is able to train randomly initialized agents using meta-learned learning rules on entirely different environments. Compared to EPG we find that MetaGenRL is more sample efficient, and outperforms significantly under fixed budget of environment interactions. The results of an ablation study and additional analysis provide further insight into the benefits of our approach.
PRELIMINARIES
Notation We consider the standard MDP Reinforcement Learning setting defined by tuple
(S, A, Po. r,", ", T) consisting states S, actions A, the transition probability distribution P S x A x
RR, initial state distribution fo
R4, the reward function r S A
[- RRmas Rmarl. a discount factor y and the episode length T. The objective for the probabilistic
policy T
A R parameterized by di is maximize the expected discounted return:
T-1
E
Σ rre,, where 8o Po(so), at To(as(ss), 8141 P(8t41/8,,0)) 0t), r(st, (c),
with r (sp, 40, ro, $1, ST 1,
Human Engineered Gradient Estimators popular gradient-based approach to maximiz- ing Equation 1 is REINFORCE (Williams, 1992). It directly differentiates Equation I with respect to ø using the likelihood ratio trick to derive gradient estimates of the form:
T-1
nT)L = IV log ne (arls)))
t'=t
Although this basic estimator is rarely used in practice, it has become building block for an entire class of policy-gradies algorithms of this form. For example, popular extension from Schulman et al. (201 5b) combines REINFORCE with Generalized Advantage Estimate (GAE) to yield the following policy gradient estimator:
V LLAE,L Ty. V) [. Σ log (a)ls)) A(T, V,t)
(3)
where A(T, V,t) is the GAE and V S R is value function estimate. Several recent other extensions include TRPO (Schulman et al., 2015a), which discourages bad policy updates using trust regions and iterative off-policy updates, or PPO (Schulman et al., 2017), which offers similar benefits using only first order approximations.
Parametrized Objective Functions In this work we note that many of these human engineered policy gradient estimators can be viewed as specific implementations of general objective function that is differentiated with respect to the policy parameters:
E, [L(T No. V)]-
(4)
Hence, it becomes natural to consider generic parametrization of L that, for various choices of parameters a, recovers some of these estimators. this paper, we will consider neural objective functions where L,, is implemented by neural network. Our goal is then to optimize the parameters a of this neural network in order to give rise to a new learning algorithm that best maximizes Equation on entire class of (different) environments.
META-LFARNING NEURAL OBJECTIVES
In this work we propose MetaGenRL, novel meta reinforcement learning algorithm that meta- learns neural objective functions of the form La(T, kg. V). MetaGenRL makes use of value functions and second-order gradients, which makes it more sample efficient compared prior work (Duan et al., 2016; Wang et al., 2016; Houthooft et al., 2018). More so, as we will demonstrate, MetaGenRI. meta-learns objective functions that generalize vastly different environments.
2

---

Published as a conference paper at ICLR 2020
Environment
Replay Butter
Meta Learning
e
B
+ V, _ls,s with & $ Vyl
Critic
A noural objective function that implements algorithm
Store
1,, Ago V)
Learning e Volu
Palicy
Interact
Figure 1: A schematic of MetaGenRL. On the left population of agents (i E 1,...,N,, each member consist of critic h8 and a policy :T() that interact with a particular environment e(i) and store collected data in a corresponding replay buffer B(O) On the right meta-learned neural objective function La, that shared across the population. Learning (dotted arrows) proceeds as follows: Each policy is updated by differentiating Las while the critic is updated using the usual TD-error (not shown). La is meta-learned by computing second-order gradients that can be obtained by differentiating through the critic.
.N,, where
Our key insight is that differentiable critic Qo S A R can be used to measure the effect of locally changing the objective function parameters a based on the quality of the corresponding policy gradients. This enables population of agents use and improve single parameterized objective function La through interacting with a set of (potentially different) environments. During evaluation (meta-test time), the meta-learned objective function can then be used train randomly initialized RL agent in a new environment.
3.1 FROM DDPG TO GRADIENT-BASED META-LEARNING OF NEURAL OBJECTIVES
We will formally introduce MetaGenRL as an extension of the DDPG actor-critic framework (Silver et al., 2014; Lillicrap et al., 2016). In DDPG, parameterized critic of the form Qo S x A + R transforms the non-differentiable RL reward maximization problem into a myopic value maximiza- tion problem for any 8t S. This is done by alternating between optimization of the critic Qo and the (here deterministic) policy To. The critic is trained to minimize the TD-error by following:
V
Σ
(Qo(se, ae) - y,)², where y = ra + y Q0(s141, (8(+1))));
(5)
(e,
and the dependence of y/ on the parameter vector 0 is ignored. The policy T is improved to increase the expected return from arbitrary states by following the gradient V E. Qo (s)))) Both gradients can be computed entirely off-policy by sampling trajectories from replay buffer.
MetaGenRL builds this idea of differentiating the critic Qo with respect to the policy parameters. incorporates a parameterized objective function La that is used to improve the policy (i.e. by following the gradient L,,,). which adds one extra level of indirection: The critic Qa improves La, while L., improves the policy TT. By first differentiating with respect to the objective function parameters a, and then with respect to the policy parameters á, the critic can be used to measure the effect of updating T using La on the estimated return?:
V Va (st,, Tor (s:)), where o' = ó (TLa(T,z()),
(6)
This constitutes a type of second order gradient V. V that can be used to meta-train I., to provide better updates to the policy parameters in the future. In practice we will use batching to optimize Equation 6 over multiple trajectories T.
Similarly the policy-gradient estimators from Section the objective function (T, x()), V) receives as inputs an episode trajectory 7 ($0:2-11, 40:2-11 1o: -1) the value function estimates
In case of probabilistic policy Te (aee (s:) the following becomes expectation under and reparam- eterizable form is required (Williams, 1988; Kingma & Welling, 2014; Rezende et al., 2014). Here we focus learning deterministic target policies.
3

---

Published as a conference paper at ICLR 2020
Algorithm 1 MetaGenRL: Meta-Training
Require: p(e) a distribution of environments
P = {(e ~ p(e), 61, 01, B1 <- 2)....}
>> Randomly initialize population of agents
Randomly initialize objective function La
while La has not converged do
for e, Φ, 8, B E do if extend replay buffer B then Extend using T in e
> For each agent i in parallel
Sample trajectories from B Update critic Qo using TD-error Update policy by following VoLo
Compute objective function gradient A for agent i according to Equation 6
Sum gradients Σ, A to update La
V, and an auxiliary input (()) (previously za) that can be differentiated with respect to the policy parameters. The latter critical to be able to differentiate with respect é and the simplest case consists of the action predicted by the policy. While Equation is used for meta- meta-learning Las the objective function La itself is used for policy learning by following VoLa (T, x (6), ; V). See Figure alternate between
for overview. MetaGenRL consists of two phases: During meta-training, critic updates, objective function updates, and policy updates to meta-learn an objective function L. described in Algorithm I. During meta-testing in Algorithm 2, we take the learned objective function La and keep fixed while training a randomly initialized policy in a new environment to assess its performance.
We note that the inputs to La are sampled from a replay buffer rather than solely using on-policy data. If La were to represent a REINFORCE-lype objective then would mean that differentiat- ing La yields biased policy gradient estimates. In our experiments we will find that the gradients from La work much better in comparison biased off-policy REINFORCE algorithm, and to an importance-samplee unbiased REINFORCE algorithm, while also improving over the popular on-policy REINFORCE and PPO algorithms.
3.2 PARAMETRIZING THE OBJECTIVE FUNCTION
We will implement La using an LSTM (Gers al., 2000; Hochreiter & Schmidhuber, 1997) that iterates over 7 in reverse order and depends on the current policy action (s;)) (see Figure 2). At every time-step L,, receives the reward rt, taken action a,, predicted action by the current policy (8()), the time I, and value function estimates V,, V S At each step the LSTM outputs the objec- tive value 4, all of which with respect to Φ. In order to accommodate varying action dimensionalities across different environ- ments, both (s:) and a are first convolved and then averaged to obtain an action embedding that does not depend on the action dimensionality. Additional details, including suggestions for more expressive alternatives are available Appendix
summed yield single scalar output value that can be differentiated
By presenting the trajectory in reverse order to the LSTM (and La correspondingly). it is able to assign credit to an action a, based on its future impact on the reward, similar to policy gradient estimators. More so, as general function approximator using these inputs, the LSTM is in prin- ciple able to learn different variance and bias reduction techniques, akin to advantage estimates, generalized advantage estimates, importance weights Due these properties, we expect the class of objective functions that is supported to somewhat relate to REINFORCE (Williams, 1992) estimator that generalized advantage estimation (Schulman et al., 2015b).
'The value estimates derived from the Q-function, ie.. V, = Qa(se, nh (st)). and treated constant input. Hence, the gradient V. T.a can not flow backwards through Qo. which ensures that Tr, can naively learn to implement a DDPG-like objective function.
Wee note that in practice it is is difficult assess whether the meta-learned object function incorporates bias variance reduction techniques, especially because MetaGenRI is unlikely recover known techniques.
4

---

Published as a conference paper at ICLR 2020
Algorithm 2 MetaGenRL: Meta- Testing
Require: A test environment e, and
LSTM
objective function La
Randomly initialize nœ. Vo. B while n has not converged do if extend replay buffer B then Extend B using T in e
conv
Fp V 1
Sample trajectories from Update V6 using TD-error Update policy by following Vola
ap. TT((s)
Figure 2: An overview of La (T, x(0), V).
3.3 GENERALITY AND EFFICIENCY OF METAGENRL
MetaGenRL offers a general framework for meta-leaming objective functions that can represent a wide range of learning algorithms. In particular, it is only required that both Ts and La can be differentiated w.r.t. to the policy parameters ń. In the present work, we use this flexibility to meta-optimization, increase sample efficiency through off-policy second- order gradients, and to improve the generalization capabilities of meta-learned objective functions.
leverage population-base
Population-Bass Based A general objective function should be applicable wide range of environ- ments and agent parameters. To this extent MetaGenRL is able to leverage the collective experience of multiple agents to perform meta-learning by using single objective function L,, shared among a population of agents that each act in their own (potentially different) environment. Each agent locally computes Equation 6 over a batch of trajectories, and the resulting gradients combined to update La. Thus, the relevant learning experience of each individual agent is compressed into the objective function that is available the entire population at any given time.
Sample Efficiency An alternative to learning neural objective functions using population of agents through evolution as in EPG (Houthooft et al., 2018). However, we expect meta-learning in MetaGenRI. to be much more sample efficient. This is due to
using second-order gradients off-policy training of the objective function La and its subsequent off-policy use to improve the policy. Indeed, unlike in evolution there is no need to train multiple randomly initialized agents in their entirety in order to evaluate the objective function, thus speeding up credit assignment. Rather, any point in time, any information that is deemed useful for future environment interactions directly be incorporated into the objective function. Finally, using the formulation in Equation one can measure the effects of improving the policy using L., for multiple steps by increasing the corresponding number of gradient steps before applying Qo. which we will explore in Section 5.2.3.
Mittaiienerliizttionn The focus of this work to learn general learning rules that during test- time can be applied to vastly different environments. A strict separation between the policy and the learning rule, the functional form of the latter, and training across many environments all contribute to this. Regarding the former, a clear separation between the policy and the learning rule as in MetaGenRL is expected to be advantageous for two reasons. Firstly, allows us to specify the number of parameters of the learning rule independent of the policy and critic parameters. For example, our implementation to 384K parameters for the policy and critic. Hence, we are able to only use short description length for the learning rule. A second advantage that is gained is that the meta-learner is unable to directly change the policy and must, therefore, learn to make use of the objective function. This makes it difficult for the meta-learner to overfit to the training environments.
L., uses only 15K parameters for the objective function compared
4 RELATED WORK
Among the earliest pursuits in meta-learning are meta-hierarchies of genetic algorithms (Schmidhu- ber, 1987) and learning update rules in supervised learning (Bengio et al., 1990). While the former introduced a general framework of entire meta-hied neeta-hiearrches. it relied on discrete non-differentiable programs. The latter introduced local update rules that included free parameters, which could be
5

---

Published as a conference paper at ICLR 2020
learned using gradients in supervised setting. Schmidhuber (1993) introduced differentiable self-referential RNN that could address and modify its own weights, albeit difficult to learn.
Hochreiter et al. (2001) introduced differentiable meta-learning using RNNs to scale to larger prob- lem instances. By giving an RNN access to its prediction error, it could implement its own meta- learning algorithm, where the weights are the meta-learned parameters, and the hidden states the subject of learning. This was later extended to the RL setting (Wang et al., 2016; Duan et al., 2016; Santoro al., 2016; Mishra et al., 2018) (here refered to as RL²). As we show empirically in our pa- per, meta-learning with RL 2 does not generalize well. It lacks a clear separation between policy and objective function, which makes it easy to overfit on training environments. This is exacerbated by the imbalance of (nn²) meta-learner parameters to learn (n) activations, unlike in MetaGenRL.
Many other recent meeaallerning algorithms learn a policy parameter initialization that is later fine- tuned using a fixed reinforcement learning algorithm (Finn et al., 2017; Schulman et al., 2017; Grant et al., 2018; Yoon et al., 2018). Different from MetaGenRL, these approaches use second order gradients Albeit in principle general (Finn & Levine, 2018), the mixing of policy and learning algorithm leads to complicated way of expressing general update rules. Similar to RL", adaptation to related tasks is possible, but generalization difficult (Houthooft et al., 2018).
the same policy parameter vector instead of using separate objective function.
Objective functions have been learned prior to MetaGenRL. Houthooft et al. (2018) evolve ob- jective function that is later used to train an agent. Unlike MetaGenRL, this approach is extremely costly in terms of the number of environment interactions required to evaluate and update the ob- jective function. Most recently, Bechtle et al. (2019) introduced learned loss functions for rein- forcement learning that also make use of second-order gradients, but use a policy gradient estimator instead of a Q-function. Similar to other work, their focus is only on narrow task distributions. Learned objective functions have also been used for learning unsupervised representations (Metz et al., 2019), DDPG-like meta gett-ggaadeets for hyperparameter search (Xu et al., 2018), and learning from human demonstrations (Yu al., 2018). Concurrent to our work, Alet et al. (2020) uses tech- niques from architecture search to search for viable artificial curiosity objectives that are composed of primitive objective functions.
Li & Malik (2016; 2017) and Andrychowicz et al. (2016) conduct meta-learning by learning op- timizers that update parameters á by modulating the gradient of some fixed objective function L: L)) where ar is learned. They differ from MetaGenRL. in that they only modulate the
AΦ = fa AΦ= gradient of a fixed objective function L instead of learning L itself.
Another connection exists to meta-learned intrinsic reward functions (Schmidhuber, 1991a; Dayan & Hinton, 1993; Wiering & Schmidhuber, 1996; Singh et al., 2004; Nickum et al., 2011; Zheng et al., 2018; Jaderberg et al., 2019). Choosing La = V Σ (T) where Fy is a meta-learned reward and Vo is gradient estimator (such as value based or policy gradient based estimator) reveals that meta-learning objective functions includes meta-learning the gradient estimatior V itself as long as is expressible by a gradient Vo on an objective La. In contrast, for intrinsic reward functions, the gradient estimator V is normally fixed.
Finally, we note that positive transfer between different tasks (reward functions) as well envi- ronments (e.g. different Atari games) has been shown previously in the context of transfer learn- ing (Kistler et al., 1997; Parisotto et al., 2015; Rusu et al., 2016; 2019; Nichol et al., 2018) and meta-critic learning across tasks (Sung et al.. 2017). In contrast to this work, the approaches that have shown be successful in this domain rely entirely human -unan-engineered learning algorithms.
EXPERIMENTS
We investigate the learning and generalization capabilities of MetaGenRL on several continuous control benchmarks including HalfCheetah (Cheetah) and Hopper from MuJoCo (Todorov et al., 2012), and CiturraandeCContinoon environments differ significantly in terms of the properties of the underlying system that is to be controlled, and in terms of the dynamics that have to be learned complete the environment. Hence, by training meta- RLL algorithms on one environment and testing on other environments they provide reasonable measure of aut-of-distribution generalization.
(Lunar) from OpenAl gym (Brockman et al., 2016). These

---

Published as a conference paper at ICLR 2020
Table 1: Mean return across multiple seeds (MetaGenRL: 6 meta-train meta-test seeds, RL 6 meta-train X 2 meta-test seeds, EPG: 3 meta-train x 2 meta-test seeds) obtained by training randomly initialized agents during meta-test time on previously seen environments (cyan) and on unseen environments (brown). Boldface highlights best meta-learned algorithm. Mean returns (6 seeds) of several human ennineeee algorithms are also listed.
Training Testing
Cheetah
Hopper
Lunar
Cheetah Hopper
MetaGenRI.
2185
2439
18
EPG
-571
-540
RL
5180
289
-479
Lunar & Cheetah
MetaGenRI.
2552
23363
2.58
EPG
-701
8
-707
RL
2218
283
Lunar & Hopper & Walker Ant Cheetah Lunar Walker Ant
MetaGenRI. (40 agents)
3106
2869
201
3331
2452
Cheetah & Hopper & Walker & Ant
2541
2345
-148
PPO
1455
1894
187
DDPG / TD3
8315
2718
288
off-policy REINFORCE (GAE)
-88
1804
168
on-policy REINFORCE (GAE)
38
565
120
In experiments, will mainly compare EPG and RL,2 to evaluate the efficacy of ap- proach. We will also compare to several fixed model-free RL algorithms to measure how well the algorithms meta-learned by MetaGenRL compare these handcrafted alternatives. Unless other- wise mentioned, we will meta-train MetaGenRL. using 20 agents that are distributed equally over the indicated training environments.. Meta-learning uses elipped double-Q learning, delayed policy & objective updates, and target policy smoothing from TD3 (Fujimoto et al., 2018). We will allow for 600K environment interactions per agent during meta-training and then meta-test the objective function for 1M interactions. Further details are available in Appendix B.
5.1 COMPARISON TO PRIOR WORK
Evaluating on previously seen environments We meta-train MetaGenRL on Lunar and compare its ability train randomly initialized agent at test-time (i.e. using the learned objective function and keeping it fixed) to DDPG, PPO, and on- and off-policy REINFORCE (both using GAE) across multiple seeds. Figure 3a shows that MetaGenRL markedly outperforms both the REINFORCE baselines and PPO. Compared to DDPG, which finds the optimal policy, MetaGenRL perforrs only slightly worse on average although the presence of outliers increases its variance. particular, we find that some meta-test agents get 'stuck' for some time before reaching the optimal policy (see Section A.2 for additional analysis). Indeed, when evaluating only the best meta-learned objective function that was obtained during meta-training (MetaGenRL (best objective func) Figure 3a) we are able to observe a strong reduction in variance and even better performance.
We also report results (Figure 3a) when meta-training MetaGenRL both Lunar and Cheetah, and compare to EPG and RL that were meta-trained on these same environments". For MetaGenRL we were able obtain similar performance meta-training on only Lunar in this case. In contrast, for EPG it can be observed that even one billion environment interactions is insufficient to find a good objective function (in Figure 3a quickly dropping below -300). Finally, we find that RI,2 reaches the optimal policy after 100 million meta-training iterations, and that its performance is unaffected by additional steps during testing on Lunar. We note that RL² does not separate the policy and the learning rule and indeed in similar 'within distribution' evaluation, RL² was found successful (Wang et al., 2016; Duan et al., 2016).
$ An ablation study in Section A.3 revealed that large numher of agents is indeed required. 'I order to ensure good baseline we allowed for maximum of 100M environment interactions for RL and 1 B for EPG, which is more than eight/ eighty times the amount used by MetaGenRI. Regarding EPG, this did require reduce the total number of sceds meta-train 2 meta-test seeds.

---

Published as a conference paper at ICLR 2020
Twiling on Lunst
Teading on Hopper
2000
Neat
Misabant
Heracanti
-120
C07G
eff REMFFORE 1ith (AE)
ce pot REIKFORCT SHF)
ITO
500
0.0M
0.2M
0:4M
0,53
0.0M
1 0.09 0.2M
GM
AM
2.20 3.20
(a) Previously seen Lunar environment.
(b) Unseen Hopper environment.
Figure Comparing the test-time training behavior of the meta-learned objective functions by MetaGenRL other (meta) reinforcement learning algorithms. We train randomly initialized agents on (a) environments that were encountered during training, and (b) on significantly different environ- ments that were unseen. Training environments are denoted by in the legend. All runs are shown with mean and standard deviation computed over multiple random seeds (MetaGenRL: meta-trait 2 meta-ter meta-testt seeds, RL?: 6 meta-train 2 meta-test seeds, EPG: 3 meta-train 2 meta-test seeds, and 6 seeds for all others).
Table provides a similar comparison for two other environments. Here we find that in general MetaGenRL able to outperform the REINFORCE baselines and PPO, and in most (except for Cheetah) performs similar to DDPG?. We also find that MetaGenRL consistently outperforms EPG, and often 2 For an analysis of meta-training on more than two environments we refer to Appendix A.
Generalization to vastly different environments We evaluate the same objective functions learned by MetaGenRL, EPG and the recurrent dynamics by RL on Hopper, which is signifi- cantly different compared to the meta-training environments. Figure 3b shows that the learned objective function by MetaGenRL continues to outperform both PPO and our implementations of REINFORCE, while the best performing configuration is even able to outperform DDPG.
When comparing to related meta-RL. approaches, we find that MetaGenRL is significantly better in this case. The performance of EPG remains poor, which was expected given what observed on previously seen environments. On the other hand, we now find that the RL baseline fails com- pletely (resulting in flat low-reward evaluation), suggesting that the learned learning rule that was previously found to be successful in fact entirely overfitted to the environments that were seen during meta-training. We were able to observe similar results when using different train and test environment splits as reported in Table 1, and in Appendix A.
5.2 ANALYSIS
5.2.1
PROGRESSION OF OBJECTIVE FUNCTIONS
Previously we focused on test-time training randomly initialized agents using an objective function that was meta-trained for a total of 600K steps (corresponding to total of 12M environment inter- actions across the entire population). We will now investigate the quality of the objective functions during meta-training.
Figure displays the result evaluating an objective function on Hopper at different intervals dur- ing meta- training Cheetah and Lunar. Initially (28K steps) it can be seen that due to lack of meta-traine meta-training there is only a marginal improvement in the return obtained during test time. How- ever, after only meta-training for 86K steps we find (perhaps surprisingly) that the meta-trained
We emphasize that the neural objective function under consideration is unable to implement DDPG and only uses constant value estimate (i.c. V.V= = by using gradient stopping) during testing.
8

---

Published as a conference paper at ICLR 2020
2340
6.7M
0.00
7.0
3030
1930
75c
175x
200c
bnhormenkittee teteraaae the population
Figure 4: Meta-training with 20 agents on Cheetah and Lunar. We test the objective function at five stages of meta-training by using to train three randomly initialized agents on Hopper.
Tritng Derta
102
100
0.50
100
(a) Mcta-training on Lumar & Cheetah
(b) Meta-testing on Cheetah
Figure 5: We meta-train MetaGenRL using several alternative parametrizations of La on a) Lunar and Cheetah, and b) present results of testing Cheetah. During meta-training representative ex- ample single agent population shown with shaded regions denoting standard deviation across the population. Meta-test results are reported per usual across 6 meta-train x 2 meta-test seeds.
objective function is already able to make consistent progress in optimizing randomly initialized agent during test-time. On the other hand, we observe large variances at test-time during this phase of meta-training. Throughout the remaining stages of meta-training then observe increase in convergence speed, more stable updates, and lower variance across seeds.
5.2.2 ABLATION STUDY
We conduct an ablation study of the neural objective function that was described in Section 3.2. In particular, we assess the dependence of L. on the value estimates V1,V++1 and on the time compo- nent that could to some extent learned. Other ablations, including limiting access to the action chosen or to the received reward, are expected to be disastrous for generalization to any other envi- ronment (or reward function) and therefore not explored.
Dependence on t We use a parameterized objective function of the form L, (ar, Tt. Ve, (s:)) e 0,-.., as in Figure 2 except that it does not receive information about the time-step f each step. Although information about the current time-step required in order to learn (for example) generalized advantage estimate (Schulman et al., 2015b), the LSTM could in principle learn such time tracking on it own, and we expect only minor effects on meta-training and during meta-testing. Indeed in Figure 5b can be seen that the neural objective function performs well without access to f, although it converges slower on Cheetah during meta-training (Figure 5a).
Dependence on V We use parameterized objective function of the form L., (a,, Tr, (s:)lt O,..., T 1) as in Figure 2 except that does not receive any information about the value estimates time-step There exist reinforcement learning algorithms that work without value function esti- mates (eg. Williams (1992); Schmidhuber & Zhao (1998)), although in the absence of an alternative baseline these often have a large variance. Similar results observed for this ablation in Figure 5a

---

Published as a conference paper at ICLR 2020
HationnL uing prodes m1 HARGGNNLL grodlct
300
3000
589
200
Figure 6: We meta-train MetaGenRL on the LunarLander and HalfCheetah environments using one, three, and five inner gradient steps on Φ. Meta-test results are reported across 3 meta-train x meta-test seeds.
during meta- weta-training where possibly large variance appears to affect meta-training. Correspond- ingly during test-time (Figure 5b) we do not find any meaningful training progress to take place. In contrast, we find that we can remove the dependence on one of the value function estimates, i.e. remove Vi+1 but keep V,, which during some runs even increases performance.
5.2.3 MULTIPLE GRADIENT STEPS
We analyze the effect of making multiple gradient updates to the policy using La before applying the critic to compute second-order gradients with respect to the objective function parameters as in Equation 6. While in previous experiments we have only considered applying single update, multiple gradient updates might better capture long term effects of the objective function. At the same time, moving further away from the current policy parameters could reduce the overall quality of the second-order gradients. Indeed, in Figure it be observed that using gradient steps already slightly increases the variance during test-time training on Hopper and Cheetah after meta- training Luderlander and Cheetah. Similarly, find that further increasing the number of gradient steps to 5 harms performance.
CONCLUSION
We have presented MetaGenRL, a novel off-policy gradient-based meta reinforcement learning al- gorithm that leverages population of DDPG-like agents meta-learn general objective functions. Unlike related methods the meta-learned objective functions do not only generalize in narrow task distributions but show similar performance REINFORCE and PPO. We have argued that this generality is due to MetaGenRL's explicit sepa- ration of the policy and learning rule, the functional form of the latter, and training across multiple agents and environments. Furthermore, the use of second order gradients increases MetaGenRL'ss sample efficiency by several orders of magnitude compared to EPG (Houthooft et al., 2018).
entirely different tasks while markedly outperforming
In future work, we aim to further improve the learning capabilities of the meta-leamed objective functions, including better leveraging knowledge from prior experiences. Indeed, our current implementation, the objective function is unable to observe the environment or the hidden state of (recurrent) policy. These extensions are especially interesting as they may allow more compli- cated curiosity-based (Schmidhuber, 1991b; 1990; Houthooft et al., 2016; Pathak et al., 2017) or model-based (Schmidhuber, 1990; Weber et al., 2017: Ha & Schmidhuber, 2018) algorithms to be leared. To this extent, it will be important develop introspection methods that analyze the learned objective function and to scale MetaGenRL to make use of many more environments and agents.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
We thank Paulo Rauber, Imanol Schlag, and the anonymous reviewers for their feedback. This work was supported by the ERC Advanced Grant (no: 742870) and computational resources by the Swiss National Supercomputing Centre (CSCS, project: s978). We also thank NVIDIA Corporation for donating DGX-1 part of the Pioneers of Al Research Award and to IBM for donating a Minsky machine.
REFERENCES
Ferran Alet, Martin F Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling, Meta-learning curiosity algorithms. In International Conference on Learning Representations, 2020.
Marcin Andrychowicz, Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 3988- 3996, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer Normalization. arXiv preprint arXiv:1607.064450 2016.
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta-learning via learned loss. arXiv preprint arXiv:1906.00334, 2019.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Université de Montréal, 1990.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv:1606.01544 2016.
Jeff Clune. AI-GAs: Al-generating algorithms,, an alternate paradigm for producing general artificial intelligence, arXiv preprint 1rXxv:1905,10988 2019.
P Dayan and Hinton. Feudal Reinforcement Learning. In D Lippman, J Moody, and Touretzky (eds.). Advances in Neural Information Processing Systems (NIPS) 5, pp. 271-278. Morgan Kaufmann, 1993.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL"2: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv preprint a6 1.02779, 2016.
Chelsea Finn and Sergey Levine. Meta-Learning and Universality: ent Descent can Approximate any Learning Algorithm. In International Conference on Learning Representations, 2018.
Deep Representations and Gradi-
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126 1126-1135, 2017.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Proceedings of Machine Learning Research, 80: 10:1587- 2018.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10):2451-247) 2000.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting Gradient- Based Meta-Learning as Hierarchical Bayes. In International Conference on Learning Represen- tations, 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2450-2462. 2018.

---

Published as a conference paper at ICLR 2020
S Hochreiter and Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1335-178) 1997.
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, 2001.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational Information Maximizing Exploration. In Advances in Neural Information Processing Systems, pp. 1109-1111 117, 2016.
Rein Houthooft, Richard Y. Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Picter Abbeel. Evolved Policy Gradients. In Advances in Neural Information Processing Systems, pp. 5400-04099, 2018.
Max Jaderberg, Wojeiech M Czamecki, lain Dunning, Luke Marris, Guy Lever, Antonio Gar- cia Castañeda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science (New York, N.Y.), 364(6443):55--865 2019.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
Diederik Kingma and Max Welling. auto-Encoding Variational Bayes. In International Confer- ence on Learning Representations, 12 2014.
Werner M Kistler, Wulfram Gerstner, and Leo Hemmen. Reduction of the Goadgkin-Huxler equations to a single-variable threshold model. Neural Computation, 95) 1997.
Ke Li and Jitendra Malik. Learning to Optimize. arXiv preprint 1rXiv:1606.0188, 2016.
Ke Li and Jitendra Malik, Learning to Optimize Neural Nets. arXiv preprint 1ao::170300444 2017.
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox. Ken Goldberg, Joseph E Gonzalez, Michael I Jordan, and lon Stoica. Rllib: Abstractions for distributed reinforeement learning. In International Conference Machine Learning, pp. (063-3062, 2018.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning Unsuper- vised Learning Rules. In International Conference on Learning Representations, 2019.
Nikhil Mishra, Mostafa Rohaninejad, and Xi UC Chen Pieter Abbeel Berkeley. A Simple Neural Attentive Meta-Learner. In International Conference Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with Deep Reinforeement Learning. arXiv preprint 13Xiv:1312.5661 12.55602, 2013.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman Openai. Gotta Learn Fast: A New Benchmark for Generalization in RL. arXiv preprint 18Xiv:1804.03370, 2018.
Scott Nickum, Lee Spector, and Andrew Barto. Evolution of reward functions for reinforcement learning. In Proceedings of the 13th annual conference companion on Genetic and evolutionary computation, pp. 177-178, 2011.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning. arXiv preprint 151 1tt::/1111..333 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Auriosity-driven exploration by self-supervised prediction. In 34th International Conference on Machine Learning. ICML 20/7, volume 6, pp. 4261-42777 2017. ISBN 9781510855144 doi: 10. P.100///.....00...00
12

---

Published as a conference paper at ICLR 2020
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, and others. Multi-goal rein- forcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In International Conference on Machine Learning, pp. 1278-1286, 2014. ISBN 9781634393973. doi: 10. 00105770004-3661221137733
Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy Dis- tillation. In International Conference on Learning Representations, 2016.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. In International Conference on Learning Representations, 7 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- Learning with Memory-Augmented Neural Networks. In International conference machine learning, pp. 1842-1850, 2016.
J Schmidhuber. Learning to Generate Sub-Goals for Action Sequences. Kohonen, K Mäkisara, O Simula, and J Kangas (eds.), Artificial Neural Networks, pp. 967-972. Elsevier Science Pub- lishers B. V., North-Hollane 1991a.
J Schmidhuber. On learning how to learn learning strategies. Technical Report FKI-198-94, Fakultät für Informatik, Technische Universität München, 1994.
J Schmidhuber and Zhao. Direct policy search and uncertain policy evaluation. Technical Report IDSIA-50-98, IDSIA, Lugano, Switzerland, 1998.
Jürgen Schmidhuber: Evolutionary principles in self-referential learning. Diploma thesis, Institut für Informatik, Technische Universität München, 1987.
Jürgen Schmidhuber. Making the world differentiable: On Using Fully Recurrent SuperSuue Neural Networks for Dynamic Reinforcement Learning and Planning in Non-Stationary Environ- ments. Technical Report FKI-126-90 (revised), Institut für Informatik, Technische Universität München, 11 1990.
Jürgen Schmidhuber. A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers. A Meyer and W Wilson (eds.), Proc. of the International Conference Simulation of Adaptive Behavior: From Animals to Animats, pp. 222 22-227.220 MIT Press/Bradford Books, 1991b.
Jürgen Schmidhuber. A self-referential weight matrix. In Proceedings of the International Confer- ence on Artificial Neural Networks, Amsterdam, pp. 446-451. Springer, 1993.
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust Region Policy Optimization. In International conference on machine learning, pp. 1889-1897, 2015a. doi: 10.1063/1.4997337
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv preprint arXiv: 02438, 2015b.
John Schulman. Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv: 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris. Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In 31st International Conference on Machine Learning, ICMI. 2014, volume 1, pp. 605-619, 2014. ISBN 9781634393973.
13

---

Published as a conference paper at ICLR 2020
Satinder Singh, Satinder Singh, A.G. Barto, A.G. Barto, Nuttapong Chentanez, and Nuttapong Chentanez. Intrinsically motivated reinforcement learning. 18th Annual Conference on Neu- ral Information Processing Systems (NIPS), 17:1281-1288, 2004. ISSN 1943-0604. doi:
1A110/9AAMD.2210220500000
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn: Meta-critic networks for sample efficient learning. arXiv preprint arXivv 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Jane Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint
2016.
Théophane Weber, Sébastien Racanière, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomênech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, David Silver, and Daan Wierstra. IonginationAAummente Agents for Deep Reinforcement Learning. In Advances in neural information processing sys- lems, pp. 5690-5701, 2017.
M Wiering and J Schmidhuber. HQ-Learning: Discovering Markovian Subgoals for Non-Markovian Reinforcement Learning. Technical Report IDSIA-95-96 IDSIA, 1996.
Daan Wierstra, Tom Schaul, Jan Peters, and Jürgen Schmidhuber. Natural Evolution Strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp.
2008.
R J Williams. On the Use of Backpropagation in Associative Reinforcement Learning. In IEEE International Conference on Neural Networks, San Diego, volume 2, pp. 263-270, 1988.
Ronald J Williams. Simple Statistical Fradient-Follownng Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, 8:229-256, 1992.
Zhongwen Xu, Hado Van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in Neural Information Processing Systems, volume 2018-Decem, pp. 2407, 2018.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian Model-Agnostic Meta-Learning. In Advances in Neural Information Processing Sys- tems, pp. 7332-7342, 2018.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. Interna- tional Conference on Learning Representations, Workshop Track, 2018.
Zeyu Zheng. Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances Neural Information Processing Systems, pp. 4644 4654, 2018.
14

---

Published as a conference paper at ICLR 2020
A ADDITIONAL RESULTS
A.1 ALL TRAINING AND TEST REGIMES
In the main text, we have shown several combinations of meta-training, and testing environments. We will now show results for all combinations, including the respective human engineered baselines.
Testing on Hopper
Teaking Hopper
2000
Hoppee
GettaGenii fonit etjective find
(0/4)
nERFORCE be GAR: chosllyy RaKFoncE with GACI
01 colcy WHOXGS owt Gake
2903
25:00
ompalay MEHTOnnE
Voan rotmr
2000
Cheetan Hopond
Chestah hoppor)
2000
ERG Oheetan Lonor)
1000
1000
5:0
Environment rtaractbora
(a) Within distribution generalization.
(b) Out of distribution generalization.
Figure 7: Comparing the test-time training behavior of the meta-learned objective functions by MetaGenRL to other (meta) reinforcement learning algorithms on Hopper. We consider within distribution testing (a), and out of distribution testing (b) by varying the meta-training environments (denoted by i) for the meta-RL approaches. All runs are shown with mean and standard deviation computed over multiple random seeds (MetaGenRL: meta-train x 2 meta-test seeds, RL?: meta- train X 2 meta-test seeds, EPG: 3 meta-traij 2 meta-test seeds, and seeds for all others).
Hopper On Hopper (Figure 7) we find that MetaGenRL works well, both in terms of generaliza- tion to previously seen environments, and to unseen environments. The PPO, REINFORCE, RI.2. and EPG baselines are outperformed significantly. Regarding RI.2 we observe that it is only able to obtain reward when Hopper was included during meta-training, although performance is gener- ally poor. Regarding EPG, observe some learning progress during meta-testing on Hopper after meta-training on Cheetah and Hopper (Figure 7a), although it drops back down quickly as test-time training proceeds. In contrast, when meta-testing Hopper after meta-trainin Cheetah and Lunar (Figure 7b) no test-time training progress is observed at all.
Cheetah Similar results are observed in Figure 8 for Cheetah, where MetaGenRL outperforms PPO and REINFORCE significantly. On the other hand, it can be seen that DDPG notably out- performs MetaGenRL on this environment. It will be interesting to further study these differences in the future to improve the expressibility of our approach. Regarding RL. and EPG only within distribution generalization results are available due to Cheetah having larger observations and or action spaces compared to Hopper and Lunar. We observe that RL performs similar to carlier findings on Hopper but significantly improves in terms of within-distribution generalization (likely consistently observed for other splits). EPG shows initially more
due to greater overfitting, promise on within distribution generalization (Figure 8a), but ends up like before.
Lunar On Lunar (Figure 9) we find that MetaGenRL is only marginally better compared to the REINFORCE and PPO basclines in terms of within distribution generalization and worse in terms of out of distribution generalization. Analyzing this result reveals that although many of the runs train rather well, some get stuck during the early stages of training without or only delayed recovering. These outliers lead to a seemingly very large variance for MetaGenRL in Figure 9b. We will provide more detailed analysis of this result in Section A.2. If we focus on the best performing objective function then we observe competitive performance to DDPG (Figure 9a). Nonetheless, we notice that the objective function trained Hopper generalizes worse to Lunar, despite our earlier result that objective functions trained on Lunar do in fact generalize well to Hopper. MetaGenRL still able to outperform both RI?2 and EPG in terms of out of distribution generalization. We do note
15

---

Published as a conference paper at ICLR 2020
Testing
theetah
Teeting on Cheelah
Vetalerkl (heenn Lunn
Mrtaainkl sospers tura)
8003
NetaCerfiL Deesat Nopset
5000
MetaGmmi juran
VataCerfi seed lax)
MateGenti, staedtes
SENNPORGE wwth GAZI 7P3
AZMMORnE GAE)
PPO
Croetah Hepper) 6/ in Cheesen & Hipoed
Nea
2003
2000
0.0M
0.2M
2:0M
6:55 10M 0.09
0.2M
GM
2.04 3.78
(a) Within distribution generalization.
(b) Out of distribution generalization.
Figure Comparing the test-time training behavior of the meta-learned objective functions by MetaGenRL to other (meta) reinforcement learning algorithms Cheetah. We consider within distribution testing (a), and out of distribution testing (b) by varying the meta-training environments (denoted by i) for the meta-RL approaches. All runs shown with mean and standard deviation computed over multiple random seeds (MetaGenRL:: meta-train x 2 meta-test seeds, RL?: meta- train 2 meta-test seeds, EPG: meta-train 2 meta-test seeds, and seeds for all others).
Texting Lunar
Testing on Luner
200
Micacarti dt Sheddal Uone??
Moan
Herscanti Lanar)
EDRG TOO
NEWFORCE (wth
off RE be GAE:
RERKFORCE Ea GAD)
pel RERFORNEE GACI
-150
ITO
-400
Dradinn Lanatl
(1 Lumar)
Creatah Haspoa
0.04
6.2
0:1M
6.CM
0.N
1CM
6.MM
6.3M
6:11
6.5M
3.30
Drvisoment bowneeeioo
mewmant Intrrcttoos
Within distribution generalization.
(b) Out of distribution generalization.
Figure 9: Comparing the test-time training behavior of the meta-learned objective functions by MetaGenRI. to other (meta) reinforcement learning algorithms Lunar. We consider within dis- tribution testing (a), and out of distribution testing (b) by varying the meta-training environments (denoted by f) for the meta-RL approaches. All runs are shown with mean and standard devia- tion computed over multiple random seeds (MetaGenRL: 6 meta-train x 2 meta-test seeds, RL 2: 6 meta-train meta-test seeds, EPG: 3 meta-train meta-test seeds, and 6 seeds for all others).
that EPG is able to meta-learn objective functions that are able to improve to some extent during test time.
Comparing final scores An overview of the final scores that were obtained for MetaGenRL in comparison to the human engineered baselines is shown in Table 2. It can be seen that MetaGenRL outperforms PPO and cetplccyy REINFORCE in most configurations while DDPG with TD3 tricks remains stronger on two of the three environments. Note that DDPG is currently not among the representable algorithms by MetaGieRII.
A.2 STABILITY OF LEARNED OBJECTIVE FUNCTIONS
In the results presented in Figure 9 on Lunar observed seemingly large variance for MetaGenRL that was due to outliers. Indeed, when analyzing the individual runs meta-trained on Lunar and tested Lunar found that that of the converged local optimum early during
16

---

Published as a conference paper at ICLR 2020
Table Agent mean return across multiple seeds (MetaGenRL: 6 meta-train 2 meta-test seeds, and 6 seeds for all others) for meta-test training on previously seen environments (cyan) and on unseen (different) environments (brown) compared to human engineered baselines.
Training (below) / Test (right)
Cheetah
Hopper
Lunar
MetaGenRI. (20 agents)
Cheetah & Hopper
2185
2433
18
Cheetah & Lunar
2551
2363
258
Hopper Lunar
4160
2966
146
Hopper
3646
2937
-62
Lunar
4366
2717
244
MetaGenRL (40 agents)
Lunar & Hopper & Walker & Ant
3106
2869
201
Cheetah & Lunar & Walker & Ant
3331
2452
Cheetah & Hopper & Walker Ant
2541
2345
-148
PPO DDPG / TD3
1455
1894
187
8315
2718
288
off-policy REINFORCE (GAE) on-policy REINFORCE (GAE)
-88
1804
168
120
Teiding dfee
after ENNC
Teading ofte C5
Todkin
2.2m
1.20
Meto Training Agents LuBar
400K Erctorment agant maining 333 Jiiiin
9035
500K
1990C
Meta-training with 20 agents LunarLander. We meta-test the objective function at different stages in training on the same environment.
Figure 10:
training and was unable to recover from this afterwards. On the other hand, we also observed that be 'stuck' for long time to then make very fast learning progress. It suggests that the objective function may sometimes experience difficulties in providing meaningful updates to the policy parameters during the early stages of training.
We have further analyzed this issue by evaluating one of the objective functions at several intervals throughout meta-training in Figure 10. From the meta-training curve (bottom) can be seen that meta-training in Lunar converges very early. This means that from then on, updates to the objective function will be based on mostly converged policies. As the test-time plots show, these additional updates appear to negatively affect test-time performance. We hypothesize that the objective func- tion essentially 'forgets' about the early stages of training randomly initialized agent, by only incorporating information about good performing agents. A possible solution to this problem would be keep older policies in the meta-training agent population use early stopping.
Finally, if we exclude four random seeds (of 12), we indeed find significant reduction in the variance (and increase in the mean) of the results observed for MetaGenRL (see Figure 11).
A.3 ABLATION OF AGENT POPULATION SIZE AND UNIQUE ENVIRONMENTS
In our experiments we have used population of 20 agents during meta-training to ensure diversity in the conditions under which the objective function needs optimize. The size of this population a crucial parameter for stable meta-optimization. Indeed, Figure 12 it can seen that meta- training becomes increasingly unstable the number of agents in the population decreases.
Using a similar argument, one would expect to gain from increasing the number of distinct envi- ronments (or agents) during meta-training. In order to verify this, have evaluated two additional
17

---

Published as a conference paper at ICLR 2020
Twiling on Lunst
Teaking on Lunet
Metieenk [- Hirpee
Netieen Conrtnn tura?
Ir Createn Nospet)
VelaCarMi111 Luna)
200
100
-150
Hetstennl: (iRggper
HotsGendi (1Creetthh Luna)
Chustahi fosppa
Latan))
0.00
0.2M
0:40
0,5M
0LCM
1.MM
0.00
0:2M
5.6%
0.09
Figure 11: The left plot shows all 12 random seeds on the meta-test environment Lunar while the right has the 4 worst random seeds removed. The variance is now reduced significantly.
Subset agents meto tralting Lunar
Testing on Happer
2500
VeleGenti, ChCheeah Jaman
VetaGonAL C1WA-300
VetuGenti, CUUAA2201
25:00
-500
-50
-2000
Chevish
Cheerah Lutal hgeeet
Cheethh Lumar- aggett
0.24
0.19
20
2.MM
SAM
0.5
0.391
2.04 0.29
0AAM
5.50
0.80
1.M
3002
Envernmrr Socto
Figure 12: Stable meta-training requires a large population size of at least 20 agents. Meta- training performance is shown for a single run with the mean and standard deviation across the agent population.
Figure 13: Meta-training on Cheetah, Lunar, Walker, and Aur with 20 or 40 agents, meta- testing on the out-of-distribution Hopper envi- ronment. We compare to previous MetaGenRI. configurations.
settings: Meta-training on Cheetah Lunar & Walker Ant with 20 and 40 agents respectively. Figure 13 shows the result of meta-testing on Hopper for these experiments (also see the final re- sults reported for 40 agents in Table 2). Unexpectedly,
find that increasing the number of distinct environments does not yield a significant improvement and, in fact, sometimes even decrease per- formance. One possibility is that this is due the simple form of the objective function under consideration, which has no access to the environment observations to efficiently distinguish be- tween them. Another possibility is that MetaGenRLL's hyperparameters require additional tuning in order to be compatible with these setups.
18

---

Published as a conference paper at ICLR 2020
B EXPERIMENT DETAILS
In the following we describe all experimental details regarding the architectures used, meta- meta-training, hyperparameteers, and baselines. The code to reproduce experiments is available at http: code/metagenrl..
//louiskirsch.coo ccmp ottoolssk..ssscccccc/o////eeeeee
B.1 NEURAL OBJECTIVE FUNCTION ARCHITECTURE
Neural Architecture In this work we LST' implement the objective function (Figure 2). The LSTM runs backwards in time over the state, action, and reward tuples that were encountered during the trajectory т under consideration. At each step the LSTM receives as input the reward re. value estimates of the current and previous state Ve, V4+1, the current timestep & and finally the action that was taken at the current timestep a, in addition the action as determined by the current policy T T (sy). The actions are first processed by one dimensional convolutional layers striding over the action dimension followed by reduction to the mean. This allows for different action sizes between environments. Let A E IIID be the action from the replay buffer, A RIID be the action predicted by the policy, and W E <R2xN a learnable matrix corresponding to N outgoing units, then the actions are transformed by
1
1 D
where (a. b) is a concatenation of a and b along the first axis. This corresponds to a convolution with kernel size 1 and stride 1. Further transformations with non-linearities be added after applying W. if necessary. We found it helpful (but not strictly necessary) to use ReLU activations for half of the units and square activations for the other half.
At each time-step the LSTM outputs a scalar value l, (bounded between -7 and v using a scaled tanh activation), which are summed obtain the value of the neural objective function. Differentiating this value with respect to the policy parameters é then yields gradients that can be used to improve Ta. We only allow gradients to flow backwards through n (s;) to ø. This implementation closely related to the functional form of REINFORCE (Williams, 1992) estimator using the generalized advantage estimation (Schulman et al., 2015b).
All feed-forward networks (critic and policy) use ReLU activations and layer normalization (Ba et al., 2016). The LSTM uses tanh activations for cell and hidden state transformations, sigmoid activations for the gates. The input time / is normalized between 0 at the beginning of the episode and 1 at the final transition. Any other typer-parametern can be seen in Table 3.
Extensibility The expressability of the objective function can be further increased through several means. One possibility is to add the entire sequence of state observations 01:7 to its inputs, or by introducing a bi-directional LSTM. Secondly, additional information about the policy (such as the hidden state a recurrent policy) can be provided to L. Although not explored this work, this would in principle allow one to learn objective that encourages certain representations emerge, e.g. a predictive representation about future observations, akin to a world model (Schmidhuber, 1990; Ha & Schmidhuber, 2018; Weber al., 2017). In turn, these could create pressure to adapt the policy's 's actions to explore unknown dynamics in the environment (Schmidhuber, 1991b; 1990; Houthooft al., 2016; Pathak et al., 2017).
B.2 META-TRAININ
Annealing with DDPG At the beginning of meta-training (learning L.). the objective function randomly initialized and thus does not make sensible updates to the policies. This can lead to irreversibly breaking the policies early during training. Our current implementation circumvents this issue by linearly annealing Lo the first 10k timesteps (~ 2% of all timesteps) with DDPG VoQu(SE, (s.)). Preliminary experiments suggested that an exponential learning rate schedule on the gradient of VoLa for the first 10k steps can replace the annealing with DDPG. The learning rate anneals exponentially between learning rate of zero and 1e-3. However, in some rare cases this may still lead to unsuccessful training runs, and thus we have omitted this approach from the present work.
19

---

Published as a conference paper at ICLR 2020
Standard training During training, the critic is updated twice many times the policy and objective function, similar to TD3 (Fujimoto et al., 2018). One gradient update with data sampled from the replay buffer is applied for every timestep collected from the environment. The gradient with respect to d Equation 6 is combined with @ using fixed learning rate in the standard way. all other parameter updates use Adam (Kingma & Ba, 2015) with the default parameters. Any other hyper-parameters can be seen in Table and Table 4.
Using additional gradient steps In our experiments (Section 5.2.3) we analyzed the effect of applying multiple gradient updates to the policy using la before applying the critic to compute second-order gradients with respect to the objective function parameters. For two updates, this gives
V Va Qolo, Fet (sx)) with o¹ = o' vo
and ø' = Φ VoLL(T2.(()))1))
and can be extended to more than two correspondingly. Additionally, we use disjoint mini batches of data 72, When updating the policy using VoLo we continue to use only single gradient step.
B.3 BASELINES
RL The implementation for RL mimics the paper by Duan al. (Duan al., 2016). How- ever, we were unable to achieve good results with TRPO (Schulman et al., 2015a) on the MuJoCo environments and thus used PPO (Schulman et al., 2017) instead. The PPO hyperparameters and implementation are taken from rllib (Liang et al., 2018). Our implementation uses an LSTM with 64 units and does not reset the state of the LSTM for two episodes in sequence. Resetting after additional episodes were given did not improve training results. Different action and observation dimensionalittis across environments were handled by using an environment wrapper that pads both with zeros appropriately.
EPG We use the official EPG code base ht : //ggt::///htat/b// com/ openai /EPG from the orig- inal paper (Houthooft et al., 2018). The hyperparameters are taken from the paper, V 64 noise vectors, an update frequency of M = 64, and 128 updates for every inner loop, resulting in an inner loop length 8196 steps. During meta-test training, we run with the same update frequency for total of million steps.
PPO & On-Policy REINFORCE with GAE We use the tuned implementations from https: html which include
//spinminguu www.nggeppppaeaa.....o openai. com/en /e//////llaae den/tttes////eeeeeeeeeeeeee GAE (Schulman et al., 2015b) baseline.
Off-Policy Reinforce with GAE The implementation is equivalent to MetaGenRL except that the objective function is fixed to be the REINFORCE estimator with a GAE (Schulman et al., 2015b) baseline. Thus, experience is sampled from replay buffer. We have also experimented with importance weighted unbiased estimator but this resulted in poor performance.
DDPG Our implementation is based on https:/ /spinningnpppp..//////// openai. com/er. en. latest/ Eatest/spinningug haapch/behhh.. html and uses the same TD3 tricks (Fujimoto et al., 2018) and hyperparameters (where applicable) that MetaGenRL uses.
20

---

Published as a conference paper at ICLR 2020
Table 4: Training hyperparameters
Parameter
Value
Table 3: Architecture hyperparameters
Truncated episode length
20
Global norm gradient clipping
1.0
Parameter
Value
Critic learning rate Å
le-3
Critic number of layers
3
Policy learning rate X2
le-3
Critic number of units
350
Second order learning rate X
Ic-3
Policy number of layers Policy number of units
3
Obj. func. learning rate X
le-3
350
Critic noise Critic noise clip
0.2
Objective function LSTM units Objective function action conv layers
32
0.5
Target network update speed
0.005
Objective function action conv filters Error bound
32
Discount factor
0.99
1000
Batch size
100
Random exploration timesteps
10000
Policy gaussian noise std
0.1
Timesteps per agent
1M
21


Reviews:
[{"review_id": "ryxWqYyctr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a meta reinforcement learning algorithm called MetaGenRL, which meta-learns learning rules to generalize to different environments. The paper poses an important observation where learning rules in reinforcement learning to train the agents are results of human engineering and design, instead, the paper demonstrates how to use second-order gradients to learn learning rules to train agents. Learning learning rules in general has been proposed and this paper is another attempt to further generalize what could be learned in the learning rules. The idea is verified on three Mujoco domains, where the neural objective function is learned from one / two domains, then deployed to a new unseen domain. The experiments show that the learned neural objective can generalize to new environments which are different from the meta-training environments. \n\nOverall, the paper is a novel paper and with clear motivation, I like the paper a lot! Hope that the authors could address the following concerns and make the paper even better:\n\n1. The current experiment setup is a great proof-of-concept, however it seems a bit limited to support the claims in the paper. The meta-training has only at most two environments and the generalization of the neural objective function is only performed at one environment. It would be great if the authors could show more results with more meta-training environments (say, 10 meta-training environments) and more meta-testing environments (the current setup is only with one);\n\n2. The paper states a hypothesis that LSTM as a general function approximator, it is in principle able to learn variance and bias reduction techniques. However, in practice, due to learning dynamics and many other factors, it's not necessary true, i.e., how many samples are required for an LSTM to learn such technique is unclear. At the same time, at Page 8, Section \"Dependence on V\" actually acts as an example of LSTM couldn't figure out an effective variance-reduction method during the short meta-training time. The authors may want to put more words around the learnability of variance-bias trade-off techniques.\n\nNotation issues which could be further improved:\n1.  Page 2, \"Notation\" section and all of the following time indexing. Note that in Equation (1), r(s_1, a_t) has discount gamma^1, which is not true, I'd recommend the authors to follow the time indexing starting from 0, so that the Equation (1) is correct. (Alternatively, the authors could change from gamma^t into gamma^{t-1});\n2. Section \"Human Engineered Gradient Estimators\" is missing the formal introduction of the notation \\tau;\n3. Overall, the authors seem to use \\Phi and \\theta interchangeably, it's better to use a unified notation across the paper;\n4. In the paper, the authors choose \\alpha to represent the neural net for learning the objective function, to make it clearer for the readers, the authors could consider to change \\alpha into \\eta, because \\alpha is often considered as learning rate notation;\n5. I'd suggest the authors to rewrite the paragraph in Page 3 \"MetaGrenRL builds on this idea of ....,  using L_\\alpha on the estimated return\". This describes a key step in the algorithm while at the moment it's not very clear to the readers what's going on there;\n6. Section 3.1 is missing a step to go from Q into V;\n7. The authors could consider to describe the details of the algorithms in a more general actor-critic form, instead of starting from DDPG formulation. It would make the methods more general applicable (for example, extension to discrete action space).  \n\n ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HkxPiyXRKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to meta learn the objective function of a policy gradient algorithm using second order gradients of the objective function w.r.t the state-action value Q. \n\nThis is an interesting approach, however, I think the experimental evidence is not sufficiently convincing. \n\n- In particular, I think the most important baseline that is compared against is not RL2, but DDPG: RL2 is not designed to generalize but to learn quickly on new tasks from the training-task distribution. Because the proposed algorithm does not depend on the observed states, it generalizes much better, but is also much slower than RL2. On the other hand, it shares a lot of design choices with DDPG: Using TD3 and Double Q-learning, as well as using as objective function the Q-values. \nLooking at Figures 2, it is not clear that the proposed algorithm is substantially better than DDPG. \n- Cheetah, Hopper and Lunar Lander are very simple environments. Evaluation on (slightly) larger scale environments would show that the algorithm can scale. \n- The authors claim that the algorithm allows sharing of exploration strategies, which I don't believe can be the case based on it's current design.\n- Lastly, I have a question about Fig 5a vs. Fig 2b: Shouldn't the performance of MetaGenRL be the same in both? It appears to perfom much better in Figure 2b.\n\nMinor remark/question (didn't influence score):\nThe authors claim in the very first paragraph (and in the 4th paragraph) that inductive biases in humans are learned by natural evoluation through \"distilling the collective learning experiences of many learners\" by \"learning from learning experiences\". I'm not familiar with the relevant literature, but this seems like a strong statement which I believe should be supported by a citation. \n\nEdit because I can't make my response visible to authors anymore:\nThank you for your response to my review and apologies for my delayed answer.\n\nAfter reading your responses I agree that PPO is a fairer comparison than DDPG and that you are outperforming PPO is promising.\nI further agree that it is relevant to show that RL2 overfits (although I personally don't find that very surprising - see below). \n\nHowever, I still don't think that RL2 is a relevant baseline for this approach. There's a fundamental trade-off between the speed of adaptation and the amount of overfitting. \nIf I want to adapt very quickly (like RL2 does), I need to leverage as much task-information as possible, thereby overfitting to the task-distribution. \nOn the other hand, MetaGenRL is slow, it has training speed comparable with gradient-based approaches (by generalizes better by construction because it e.g. doesn't receive states as inputs). \nConsequently, because MetaGenRL doesn't offer any speed improvements over gradient-based approaches, it should be compared to them, and not RL2.\n\nTaken both the positive results vs. PPO and the negative results vs. DDPG, together with the fact that there's no learning speed advantage for MetaGenRL, I would see it as an interesting, and promising, research direction, but so far without proof that it can advance state of the art, as it looses to RL2 in terms of speed and DDPG in terms of final performance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgkRUO6Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a novel meta reinforcement learning algorithm capable of meta-generalizing to unseen tasks. They make use of a learned objective function used in combination with DDPG style update. Results are presented on different combinations of meta-training and meta-testing on lunar, half cheetah, and hopper environments with a focus on meta-generalization to vastly different environments.\nMotivation:\nThe work is well motivated and is tackling an important problem. There are a number of design decisions presented and only some are validated experimentally. Given the complexity of many existing meta-rl methods this seems fine but could obviously be improved upon either with more empirical work or with some guiding theory.\nExperiments:\nOverall the experiments are not convincing to me. Given that this is the majority of your paper is empirically based this is my main criticism. More detailed comments follow.\nFigure 2a concerns me that just meta-training on lunar performs worse than ddpg (what your algorithm is based on). This suggests that this added complexity is not aiding in capacity and or hurting training. Can you comment on this? This result also casts doubt onto figure 2b, which, in isolation seems like an extremely promising example of meta-generalization. This makes me fear there is something indirect and not interesting occurring (e.g. the learned loss modifies with the DDPG algorithm which happens to increase noise in generated samples which improve performance only on some environments and hurts in others for example.)\nTable 1 should include hand designed algorithms imo. Given how weak EPG is (as you stated for number of frames) and how RL^2 will never generalize across these different tasks it's hard to get a sense of the numbers. Your appendix does include a figure like this which shows ddpg performs quite well. Additionally, I don't understand why meta-training on lunar and transferring to hopper does better than meta-training on hopper (table 1, middle column). Can you comment on this?\nWhile figure 3 is cool, I would appreciate if it put the meta-test performance on the same graph as meta-train performance. From eyeballing the curves it looks like it decreases at 100k iterations then finally increases again at 200k. This is strange. This also seems fraught from an empirical comparison point of view. How do you select when to test these algorithms? Ideally you would have a meta-validation set of tasks then only meta-test on the selected task but I see no mention of this.\nKey details such as meta-training are also not discussed in depth nor ablated. From the details and curriculum scheme presented in the appendix this seems like quite a feat. Further study of these factors could be useful.\nHyperparameters of your baseline do not appear to be tuned (taken from appendix) where as for your method has a number of choices. How are you tuning these choices? Once again a meta-validation set would be the principled thing to tune against.\nFinally, the experimental setup presented here is quite complicated. There are a ton of factors at play -- exploration, meta-generalization, meta-training, inner-training, instability of ddpg, so on. These all complicate the resulting picture. Having some simplified / more controlled setup to demonstrate these pieces would be greatly appreciated.\nOther Suggestions:\nSection 3 generalization: I think you mean meta-generalization.\nPlease include what error bars are for all plots.\n \nRating:\nI am borderline leaning towards reject on this paper. I enjoyed reading this work and found the ideas interesting but the empirical comparisons are confusing and not convincing. I hope the authors continue to work to improve this! \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a meta-RL algorithm that learns an objective function whose gradients can be used to efficiently train a learner on entirely new tasks from those seen during meta-training. Building off-policy gradient-based meta-RL methods is challenging, and had not been previously demonstrated. Further, the demonstrated generalization capabilities are a substantial improvement in capabilities over prior meta-learning methods. There are a couple related works that are quite relevant (and somewhat similar in methodology) and overlooked -- see [1,2]. Further, we strongly encourage the authors to run the method on multiple meta-training environments and to report results with more seeds, as promised. The contributions are significant and should be seen by the ICLR community. Hence, I recommend an oral presentation.

[1] Yu et al. One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
[2] Sung et al. Meta-critic networks
Decision: Accept (Spotlight)