Year: 2020

Abstract:
Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms humanengineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

heading: 1 INTRODUCTION
text: The process of evolution has equipped humans with incredibly general learning algorithms. They enable us to solve a wide range of problems, even in the absence of a large number of related prior experiences. The algorithms that give rise to these capabilities are the result of distilling the collective experiences of many learners throughout the course of natural evolution. By essentially learning from learning experiences in this way, the resulting knowledge can be compactly encoded in the genetic code of an individual to give rise to the general learning capabilities that we observe today.
In contrast, Reinforcement Learning (RL) in artificial agents rarely proceeds in this way. The learning rules that are used to train agents are the result of years of human engineering and design, (e.g. Williams (1992); Wierstra et al. (2008); Mnih et al. (2013); Lillicrap et al. (2016); Schulman et al. (2015a)). Correspondingly, artificial agents are inherently limited by the ability of the designer to incorporate the right inductive biases in order to learn from previous experiences.
Several works have proposed an alternative framework based on meta reinforcement learning (Schmidhuber, 1994; Wang et al., 2016; Duan et al., 2016; Finn et al., 2017; Houthooft et al., 2018; Clune, 2019). Meta-RL distinguishes between learning to act in the environment (the reinforcement learning problem) and learning to learn (the meta-learning problem). Hence, learning itself is now a learning problem, which in principle allows one to leverage prior learning experiences to meta-learn general learning rules that surpass human-engineered alternatives. However, while prior work found that learning rules could be meta-learned that generalize to slightly different environments or goals (Finn et al., 2017; Plappert et al., 2018; Houthooft et al., 2018), generalization to entirely different environments remains an open problem.
In this paper we present MetaGenRL1, a novel meta reinforcement learning algorithm that metalearns learning rules that generalize to entirely different environments. MetaGenRL is inspired by the process of natural evolution as it distills the experiences of many agents into the parameters of an objective function that decides how future individuals will learn. Similar to Evolved Policy Gradients (EPG; Houthooft et al. (2018)), it meta-learns low complexity neural objective functions that can be used to train complex agents with many parameters. However, unlike EPG, it is able to meta-learn using second-order gradients, which offers several advantages as we will demonstrate.
We evaluate MetaGenRL on a variety of continuous control tasks and compare to RL2 (Wang et al., 2016; Duan et al., 2016) and EPG in addition to several human engineered learning algorithms.
1Code is available at http://louiskirsch.com/code/metagenrl
Compared to RL2 we find that MetaGenRL does not overfit and is able to train randomly initialized agents using meta-learned learning rules on entirely different environments. Compared to EPG we find that MetaGenRL is more sample efficient, and outperforms significantly under a fixed budget of environment interactions. The results of an ablation study and additional analysis provide further insight into the benefits of our approach.

heading: 2 PRELIMINARIES
text: Notation We consider the standard MDP Reinforcement Learning setting defined by a tuple e = (S,A, P, ρ0, r, γ, T ) consisting of states S, actions A, the transition probability distribution P : S × A × S → R+, an initial state distribution ρ0 : S → R+, the reward function r : S × A → [−Rmax, Rmax], a discount factor γ, and the episode length T . The objective for the probabilistic policy πφ : S ×A→ R+ parameterized by φ is to maximize the expected discounted return:
Eτ [ T−1∑ t=0 γtrt], where s0 ∼ ρ0(s0), at ∼ πφ(at|st), st+1 ∼ P (st+1|st, at), rt = r(st, at), (1)
with τ = (s0, a0, r0, s1, ..., sT−1, aT−1, rT−1).
Human Engineered Gradient Estimators A popular gradient-based approach to maximizing Equation 1 is REINFORCE (Williams, 1992). It directly differentiates Equation 1 with respect to φ using the likelihood ratio trick to derive gradient estimates of the form:
∇φEτ [LREINF (τ, πφ)] := Eτ [∇φ T−1∑ t=0 log πφ(at|st) · T−1∑ t′=t γt ′−trt′)]. (2)
Although this basic estimator is rarely used in practice, it has become a building block for an entire class of policy-gradient algorithms of this form. For example, a popular extension from Schulman et al. (2015b) combines REINFORCE with a Generalized Advantage Estimate (GAE) to yield the following policy gradient estimator:
∇φEτ [LGAE(τ, πφ, V )] := Eτ [∇φ T−1∑ t=0 log πφ(at|st) ·A(τ, V, t)]. (3)
where A(τ, V, t) is the GAE and V : S → R is a value function estimate. Several recent other extensions include TRPO (Schulman et al., 2015a), which discourages bad policy updates using trust regions and iterative off-policy updates, or PPO (Schulman et al., 2017), which offers similar benefits using only first order approximations.
Parametrized Objective Functions In this work we note that many of these human engineered policy gradient estimators can be viewed as specific implementations of a general objective function L that is differentiated with respect to the policy parameters:
∇φEτ [L(τ, πφ, V )]. (4)
Hence, it becomes natural to consider a generic parametrization of L that, for various choices of parameters α, recovers some of these estimators. In this paper, we will consider neural objective functions where Lα is implemented by a neural network. Our goal is then to optimize the parameters α of this neural network in order to give rise to a new learning algorithm that best maximizes Equation 1 on an entire class of (different) environments.

heading: 3 META-LEARNING NEURAL OBJECTIVES
text: In this work we propose MetaGenRL, a novel meta reinforcement learning algorithm that metalearns neural objective functions of the form Lα(τ, πφ, V ). MetaGenRL makes use of value functions and second-order gradients, which makes it more sample efficient compared to prior work (Duan et al., 2016; Wang et al., 2016; Houthooft et al., 2018). More so, as we will demonstrate, MetaGenRL meta-learns objective functions that generalize to vastly different environments.
Our key insight is that a differentiable critic Qθ : S × A → R can be used to measure the effect of locally changing the objective function parameters α based on the quality of the corresponding policy gradients. This enables a population of agents to use and improve a single parameterized objective function Lα through interacting with a set of (potentially different) environments. During evaluation (meta-test time), the meta-learned objective function can then be used to train a randomly initialized RL agent in a new environment.

heading: 3.1 FROM DDPG TO GRADIENT-BASED META-LEARNING OF NEURAL OBJECTIVES
text: We will formally introduce MetaGenRL as an extension of the DDPG actor-critic framework (Silver et al., 2014; Lillicrap et al., 2016). In DDPG, a parameterized critic of the form Qθ : S × A → R transforms the non-differentiable RL reward maximization problem into a myopic value maximization problem for any st ∈ S. This is done by alternating between optimization of the critic Qθ and the (here deterministic) policy πφ. The critic is trained to minimize the TD-error by following:
∇θ ∑
(st,at,rt,st+1)
(Qθ(st, at)− yt)2, where yt = rt + γ ·Qθ(st+1, πφ(st+1)), (5)
and the dependence of yt on the parameter vector θ is ignored. The policy πφ is improved to increase the expected return from arbitrary states by following the gradient∇φ ∑ st Qθ(st, πφ(st)). Both gradients can be computed entirely off-policy by sampling trajectories from a replay buffer.
MetaGenRL builds on this idea of differentiating the critic Qθ with respect to the policy parameters. It incorporates a parameterized objective function Lα that is used to improve the policy (i.e. by following the gradient ∇φLα), which adds one extra level of indirection: The critic Qθ improves Lα, while Lα improves the policy πφ. By first differentiating with respect to the objective function parameters α, and then with respect to the policy parameters φ, the critic can be used to measure the effect of updating πφ using Lα on the estimated return2: ∇αQθ(st, πφ′(st)), where φ′ = φ−∇φLα(τ, x(φ), V ). (6) This constitutes a type of second order gradient∇α∇φ that can be used to meta-train Lα to provide better updates to the policy parameters in the future. In practice we will use batching to optimize Equation 6 over multiple trajectories τ .
Similarly to the policy-gradient estimators from Section 2, the objective function Lα(τ, x(φ), V ) receives as inputs an episode trajectory τ = (s0:T−1, a0:T−1, r0:T−1), the value function estimates
2In case of a probabilistic policy πφ(at|st) the following becomes an expectation under πφ and a reparameterizable form is required (Williams, 1988; Kingma & Welling, 2014; Rezende et al., 2014). Here we focus on learning deterministic target policies.
Algorithm 1 MetaGenRL: Meta-Training Require: p(e) a distribution of environments P ⇐ {(e1 ∼ p(e), φ1, θ1, B1 ← ∅), . . .} . Randomly initialize population of agents Randomly initialize objective function Lα while Lα has not converged do
for e, φ, θ, B ∈ P do . For each agent i in parallel if extend replay buffer B then
Extend B using πφ in e Sample trajectories from B Update critic Qθ using TD-error Update policy by following ∇φLα Compute objective function gradient ∆i for agent i according to Equation 6
Sum gradients ∑ i ∆i to update Lα
V , and an auxiliary input x(φ) (previously πφ) that can be differentiated with respect to the policy parameters. The latter is critical to be able to differentiate with respect to φ and in the simplest case it consists of the action as predicted by the policy. While Equation 6 is used for meta-learning Lα, the objective functionLα itself is used for policy learning by following∇φLα(τ, x(φ), V ). See Figure 1 for an overview. MetaGenRL consists of two phases: During meta-training, we alternate between critic updates, objective function updates, and policy updates to meta-learn an objective function Lα as described in Algorithm 1. During meta-testing in Algorithm 2, we take the learned objective function Lα and keep it fixed while training a randomly initialized policy in a new environment to assess its performance.
We note that the inputs to Lα are sampled from a replay buffer rather than solely using on-policy data. If Lα were to represent a REINFORCE-type objective then it would mean that differentiating Lα yields biased policy gradient estimates. In our experiments we will find that the gradients from Lα work much better in comparison to a biased off-policy REINFORCE algorithm, and to an importance-sampled unbiased REINFORCE algorithm, while also improving over the popular on-policy REINFORCE and PPO algorithms.

heading: 3.2 PARAMETRIZING THE OBJECTIVE FUNCTION
text: We will implement Lα using an LSTM (Gers et al., 2000; Hochreiter & Schmidhuber, 1997) that iterates over τ in reverse order and depends on the current policy action πφ(st) (see Figure 2). At every time-step Lα receives the reward rt, taken action at, predicted action by the current policy πφ(st), the time t, and value function estimates Vt, Vt+13. At each step the LSTM outputs the objective value lt, all of which are summed to yield a single scalar output value that can be differentiated with respect to φ. In order to accommodate varying action dimensionalities across different environments, both πφ(st) and at are first convolved and then averaged to obtain an action embedding that does not depend on the action dimensionality. Additional details, including suggestions for more expressive alternatives are available in Appendix B.
By presenting the trajectory in reverse order to the LSTM (and Lα correspondingly), it is able to assign credit to an action at based on its future impact on the reward, similar to policy gradient estimators. More so, as a general function approximator using these inputs, the LSTM is in principle able to learn different variance and bias reduction techniques, akin to advantage estimates, generalized advantage estimates, or importance weights4. Due to these properties, we expect the class of objective functions that is supported to somewhat relate to a REINFORCE (Williams, 1992) estimator that uses generalized advantage estimation (Schulman et al., 2015b).
3The value estimates are derived from the Q-function, i.e. Vt = Qθ(st, πφ(st)), and are treated as a constant input. Hence, the gradient∇φLα can not flow backwards through Qθ , which ensures that Lα can not naively learn to implement a DDPG-like objective function.
4We note that in practice it is is difficult to assess whether the meta-learned object function incorporates bias / variance reduction techniques, especially because MetaGenRL is unlikely to recover known techniques.
Algorithm 2 MetaGenRL: Meta-Testing Require: A test environment e, and an
LSTM
!lt
!at, πϕ(st)
conv !rt, Vt, Vt+1, t

heading: 3.3 GENERALITY AND EFFICIENCY OF METAGENRL
text: MetaGenRL offers a general framework for meta-learning objective functions that can represent a wide range of learning algorithms. In particular, it is only required that both πφ and Lα can be differentiated w.r.t. to the policy parameters φ. In the present work, we use this flexibility to leverage population-based meta-optimization, increase sample efficiency through off-policy secondorder gradients, and to improve the generalization capabilities of meta-learned objective functions.
Population-Based A general objective function should be applicable to a wide range of environments and agent parameters. To this extent MetaGenRL is able to leverage the collective experience of multiple agents to perform meta-learning by using a single objective function Lα shared among a population of agents that each act in their own (potentially different) environment. Each agent locally computes Equation 6 over a batch of trajectories, and the resulting gradients are combined to update Lα. Thus, the relevant learning experience of each individual agent is compressed into the objective function that is available to the entire population at any given time.
Sample Efficiency An alternative to learning neural objective functions using a population of agents is through evolution as in EPG (Houthooft et al., 2018). However, we expect meta-learning using second-order gradients as in MetaGenRL to be much more sample efficient. This is due to off-policy training of the objective function Lα and its subsequent off-policy use to improve the policy. Indeed, unlike in evolution there is no need to train multiple randomly initialized agents in their entirety in order to evaluate the objective function, thus speeding up credit assignment. Rather, at any point in time, any information that is deemed useful for future environment interactions can directly be incorporated into the objective function. Finally, using the formulation in Equation 6 one can measure the effects of improving the policy using Lα for multiple steps by increasing the corresponding number of gradient steps before applyingQθ, which we will explore in Section 5.2.3.
Meta-Generalization The focus of this work is to learn general learning rules that during testtime can be applied to vastly different environments. A strict separation between the policy and the learning rule, the functional form of the latter, and training across many environments all contribute to this. Regarding the former, a clear separation between the policy and the learning rule as in MetaGenRL is expected to be advantageous for two reasons. Firstly, it allows us to specify the number of parameters of the learning rule independent of the policy and critic parameters. For example, our implementation of Lα uses only 15K parameters for the objective function compared to 384K parameters for the policy and critic. Hence, we are able to only use a short description length for the learning rule. A second advantage that is gained is that the meta-learner is unable to directly change the policy and must, therefore, learn to make use of the objective function. This makes it difficult for the meta-learner to overfit to the training environments.

heading: 4 RELATED WORK
text: Among the earliest pursuits in meta-learning are meta-hierarchies of genetic algorithms (Schmidhuber, 1987) and learning update rules in supervised learning (Bengio et al., 1990). While the former introduced a general framework of entire meta-hierarchies, it relied on discrete non-differentiable programs. The latter introduced local update rules that included free parameters, which could be
learned using gradients in a supervised setting. Schmidhuber (1993) introduced a differentiable self-referential RNN that could address and modify its own weights, albeit difficult to learn.
Hochreiter et al. (2001) introduced differentiable meta-learning using RNNs to scale to larger problem instances. By giving an RNN access to its prediction error, it could implement its own metalearning algorithm, where the weights are the meta-learned parameters, and the hidden states the subject of learning. This was later extended to the RL setting (Wang et al., 2016; Duan et al., 2016; Santoro et al., 2016; Mishra et al., 2018) (here refered to as RL2). As we show empirically in our paper, meta-learning with RL2 does not generalize well. It lacks a clear separation between policy and objective function, which makes it easy to overfit on training environments. This is exacerbated by the imbalance of O(n2) meta-learned parameters to learn O(n) activations, unlike in MetaGenRL.
Many other recent meta-learning algorithms learn a policy parameter initialization that is later finetuned using a fixed reinforcement learning algorithm (Finn et al., 2017; Schulman et al., 2017; Grant et al., 2018; Yoon et al., 2018). Different from MetaGenRL, these approaches use second order gradients on the same policy parameter vector instead of using a separate objective function. Albeit in principle general (Finn & Levine, 2018), the mixing of policy and learning algorithm leads to a complicated way of expressing general update rules. Similar to RL2, adaptation to related tasks is possible, but generalization is difficult (Houthooft et al., 2018).
Objective functions have been learned prior to MetaGenRL. Houthooft et al. (2018) evolve an objective function that is later used to train an agent. Unlike MetaGenRL, this approach is extremely costly in terms of the number of environment interactions required to evaluate and update the objective function. Most recently, Bechtle et al. (2019) introduced learned loss functions for reinforcement learning that also make use of second-order gradients, but use a policy gradient estimator instead of a Q-function. Similar to other work, their focus is only on narrow task distributions. Learned objective functions have also been used for learning unsupervised representations (Metz et al., 2019), DDPG-like meta-gradients for hyperparameter search (Xu et al., 2018), and learning from human demonstrations (Yu et al., 2018). Concurrent to our work, Alet et al. (2020) uses techniques from architecture search to search for viable artificial curiosity objectives that are composed of primitive objective functions.
Li & Malik (2016; 2017) and Andrychowicz et al. (2016) conduct meta-learning by learning optimizers that update parameters φ by modulating the gradient of some fixed objective function L: ∆φ = fα(∇φL) where α is learned. They differ from MetaGenRL in that they only modulate the gradient of a fixed objective function L instead of learning L itself.
Another connection exists to meta-learned intrinsic reward functions (Schmidhuber, 1991a; Dayan & Hinton, 1993; Wiering & Schmidhuber, 1996; Singh et al., 2004; Niekum et al., 2011; Zheng et al., 2018; Jaderberg et al., 2019). Choosing∇φLα = ∇̃φ ∑T t=1 r̄t(τ), where r̄t is a meta-learned reward and ∇̃θ is a gradient estimator (such as a value based or policy gradient based estimator) reveals that meta-learning objective functions includes meta-learning the gradient estimatior ∇̃ itself as long as it is expressible by a gradient ∇θ on an objective Lα. In contrast, for intrinsic reward functions, the gradient estimator ∇̃ is normally fixed. Finally, we note that positive transfer between different tasks (reward functions) as well as environments (e.g. different Atari games) has been shown previously in the context of transfer learning (Kistler et al., 1997; Parisotto et al., 2015; Rusu et al., 2016; 2019; Nichol et al., 2018) and meta-critic learning across tasks (Sung et al., 2017). In contrast to this work, the approaches that have shown to be successful in this domain rely entirely on human-engineered learning algorithms.

heading: 5 EXPERIMENTS
text: We investigate the learning and generalization capabilities of MetaGenRL on several continuous control benchmarks including HalfCheetah (Cheetah) and Hopper from MuJoCo (Todorov et al., 2012), and LunarLanderContinuous (Lunar) from OpenAI gym (Brockman et al., 2016). These environments differ significantly in terms of the properties of the underlying system that is to be controlled, and in terms of the dynamics that have to be learned to complete the environment. Hence, by training meta-RL algorithms on one environment and testing on other environments they provide a reasonable measure of out-of-distribution generalization.
In our experiments, we will mainly compare to EPG and to RL2 to evaluate the efficacy of our approach. We will also compare to several fixed model-free RL algorithms to measure how well the algorithms meta-learned by MetaGenRL compare to these handcrafted alternatives. Unless otherwise mentioned, we will meta-train MetaGenRL using 20 agents that are distributed equally over the indicated training environments5. Meta-learning uses clipped double-Q learning, delayed policy & objective updates, and target policy smoothing from TD3 (Fujimoto et al., 2018). We will allow for 600K environment interactions per agent during meta-training and then meta-test the objective function for 1M interactions. Further details are available in Appendix B.

heading: 5.1 COMPARISON TO PRIOR WORK
text: Evaluating on previously seen environments We meta-train MetaGenRL on Lunar and compare its ability to train a randomly initialized agent at test-time (i.e. using the learned objective function and keeping it fixed) to DDPG, PPO, and on- and off-policy REINFORCE (both using GAE) across multiple seeds. Figure 3a shows that MetaGenRL markedly outperforms both the REINFORCE baselines and PPO. Compared to DDPG, which finds the optimal policy, MetaGenRL performs only slightly worse on average although the presence of outliers increases its variance. In particular, we find that some meta-test agents get ‘stuck’ for some time before reaching the optimal policy (see Section A.2 for additional analysis). Indeed, when evaluating only the best meta-learned objective function that was obtained during meta-training (MetaGenRL (best objective func) in Figure 3a) we are able to observe a strong reduction in variance and even better performance.
We also report results (Figure 3a) when meta-training MetaGenRL on both Lunar and Cheetah, and compare to EPG and RL2 that were meta-trained on these same environments6. For MetaGenRL we were able to obtain similar performance to meta-training on only Lunar in this case. In contrast, for EPG it can be observed that even one billion environment interactions is insufficient to find a good objective function (in Figure 3a quickly dropping below -300). Finally, we find that RL2 reaches the optimal policy after 100 million meta-training iterations, and that its performance is unaffected by additional steps during testing on Lunar. We note that RL2 does not separate the policy and the learning rule and indeed in a similar ‘within distribution’ evaluation, RL2 was found successful (Wang et al., 2016; Duan et al., 2016).
5An ablation study in Section A.3 revealed that a large number of agents is indeed required. 6In order to ensure a good baseline we allowed for a maximum of 100M environment interactions for RL2 and 1B for EPG, which is more than eight / eighty times the amount used by MetaGenRL. Regarding EPG, this did require us to reduce the total number of seeds to 3 meta-train × 2 meta-test seeds.
Table 1 provides a similar comparison for two other environments. Here we find that in general MetaGenRL is able to outperform the REINFORCE baselines and PPO, and in most cases (except for Cheetah) performs similar to DDPG7. We also find that MetaGenRL consistently outperforms EPG, and often RL2. For an analysis of meta-training on more than two environments we refer to Appendix A.
Generalization to vastly different environments We evaluate the same objective functions learned by MetaGenRL, EPG and the recurrent dynamics by RL2 on Hopper, which is significantly different compared to the meta-training environments. Figure 3b shows that the learned objective function by MetaGenRL continues to outperform both PPO and our implementations of REINFORCE, while the best performing configuration is even able to outperform DDPG.
When comparing to related meta-RL approaches, we find that MetaGenRL is significantly better in this case. The performance of EPG remains poor, which was expected given what was observed on previously seen environments. On the other hand, we now find that the RL2 baseline fails completely (resulting in a flat low-reward evaluation), suggesting that the learned learning rule that was previously found to be successful is in fact entirely overfitted to the environments that were seen during meta-training. We were able to observe similar results when using different train and test environment splits as reported in Table 1, and in Appendix A.

heading: 5.2 ANALYSIS

heading: 5.2.1 META-TRAINING PROGRESSION OF OBJECTIVE FUNCTIONS
text: Previously we focused on test-time training randomly initialized agents using an objective function that was meta-trained for a total of 600K steps (corresponding to a total of 12M environment interactions across the entire population). We will now investigate the quality of the objective functions during meta-training.
Figure 4 displays the result of evaluating an objective function on Hopper at different intervals during meta-training on Cheetah and Lunar. Initially (28K steps) it can be seen that due to lack of meta-training there is only a marginal improvement in the return obtained during test time. However, after only meta-training for 86K steps we find (perhaps surprisingly) that the meta-trained
7We emphasize that the neural objective function under consideration is unable to implement DDPG and only uses a constant value estimate (i.e. ∇φV = 0 by using gradient stopping) during meta testing.
objective function is already able to make consistent progress in optimizing a randomly initialized agent during test-time. On the other hand, we observe large variances at test-time during this phase of meta-training. Throughout the remaining stages of meta-training we then observe an increase in convergence speed, more stable updates, and a lower variance across seeds.

heading: 5.2.2 ABLATION STUDY
text: We conduct an ablation study of the neural objective function that was described in Section 3.2. In particular, we assess the dependence of Lα on the value estimates Vt,Vt+1 and on the time component that could to some extent be learned. Other ablations, including limiting access to the action chosen or to the received reward, are expected to be disastrous for generalization to any other environment (or reward function) and therefore not explored.
Dependence on t We use a parameterized objective function of the form Lα(at, rt, Vt, πφ(st)|t ∈ 0, ..., T − 1) as in Figure 2 except that it does not receive information about the time-step t at each step. Although information about the current time-step is required in order to learn (for example) a generalized advantage estimate (Schulman et al., 2015b), the LSTM could in principle learn such time tracking on it own, and we expect only minor effects on meta-training and during meta-testing. Indeed in Figure 5b it can be seen that the neural objective function performs well without access to t, although it converges slower on Cheetah during meta-training (Figure 5a).
Dependence on V We use a parameterized objective function of the form Lα(at, rt, t, πφ(st)|t ∈ 0, ..., T − 1) as in Figure 2 except that it does not receive any information about the value estimates at time-step t. There exist reinforcement learning algorithms that work without value function estimates (eg. Williams (1992); Schmidhuber & Zhao (1998)), although in the absence of an alternative baseline these often have a large variance. Similar results are observed for this ablation in Figure 5a
during meta-training where a possibly large variance appears to affect meta-training. Correspondingly during test-time (Figure 5b) we do not find any meaningful training progress to take place. In contrast, we find that we can remove the dependence on one of the value function estimates, i.e. remove Vt+1 but keep Vt, which during some runs even increases performance.

heading: 5.2.3 MULTIPLE GRADIENT STEPS
text: We analyze the effect of making multiple gradient updates to the policy using Lα before applying the critic to compute second-order gradients with respect to the objective function parameters as in Equation 6. While in previous experiments we have only considered applying a single update, multiple gradient updates might better capture long term effects of the objective function. At the same time, moving further away from the current policy parameters could reduce the overall quality of the second-order gradients. Indeed, in Figure 6 it can be observed that using 3 gradient steps already slightly increases the variance during test-time training on Hopper and Cheetah after metatraining on LunarLander and Cheetah. Similarly, we find that further increasing the number of gradient steps to 5 harms performance.

heading: 6 CONCLUSION
text: We have presented MetaGenRL, a novel off-policy gradient-based meta reinforcement learning algorithm that leverages a population of DDPG-like agents to meta-learn general objective functions. Unlike related methods the meta-learned objective functions do not only generalize in narrow task distributions but show similar performance on entirely different tasks while markedly outperforming REINFORCE and PPO. We have argued that this generality is due to MetaGenRL’s explicit separation of the policy and learning rule, the functional form of the latter, and training across multiple agents and environments. Furthermore, the use of second order gradients increases MetaGenRL’s sample efficiency by several orders of magnitude compared to EPG (Houthooft et al., 2018).
In future work, we aim to further improve the learning capabilities of the meta-learned objective functions, including better leveraging knowledge from prior experiences. Indeed, in our current implementation, the objective function is unable to observe the environment or the hidden state of the (recurrent) policy. These extensions are especially interesting as they may allow more complicated curiosity-based (Schmidhuber, 1991b; 1990; Houthooft et al., 2016; Pathak et al., 2017) or model-based (Schmidhuber, 1990; Weber et al., 2017; Ha & Schmidhuber, 2018) algorithms to be learned. To this extent, it will be important to develop introspection methods that analyze the learned objective function and to scale MetaGenRL to make use of many more environments and agents.

heading: ACKNOWLEDGEMENTS
text: We thank Paulo Rauber, Imanol Schlag, and the anonymous reviewers for their feedback. This work was supported by the ERC Advanced Grant (no: 742870) and computational resources by the Swiss National Supercomputing Centre (CSCS, project: s978). We also thank NVIDIA Corporation for donating a DGX-1 as part of the Pioneers of AI Research Award and to IBM for donating a Minsky machine.

heading: A ADDITIONAL RESULTS
text: A.1 ALL TRAINING AND TEST REGIMES
In the main text, we have shown several combinations of meta-training, and testing environments. We will now show results for all combinations, including the respective human engineered baselines.
Hopper On Hopper (Figure 7) we find that MetaGenRL works well, both in terms of generalization to previously seen environments, and to unseen environments. The PPO, REINFORCE, RL2, and EPG baselines are outperformed significantly. Regarding RL2 we observe that it is only able to obtain reward when Hopper was included during meta-training, although its performance is generally poor. Regarding EPG, we observe some learning progress during meta-testing on Hopper after meta-training on Cheetah and Hopper (Figure 7a), although it drops back down quickly as test-time training proceeds. In contrast, when meta-testing on Hopper after meta-training on Cheetah and Lunar (Figure 7b) no test-time training progress is observed at all.
Cheetah Similar results are observed in Figure 8 for Cheetah, where MetaGenRL outperforms PPO and REINFORCE significantly. On the other hand, it can be seen that DDPG notably outperforms MetaGenRL on this environment. It will be interesting to further study these differences in the future to improve the expressibility of our approach. Regarding RL2 and EPG only within distribution generalization results are available due to Cheetah having larger observations and / or action spaces compared to Hopper and Lunar. We observe that RL2 performs similar to our earlier findings on Hopper but significantly improves in terms of within-distribution generalization (likely due to greater overfitting, as was consistently observed for other splits). EPG shows initially more promise on within distribution generalization (Figure 8a), but ends up like before.
Lunar On Lunar (Figure 9) we find that MetaGenRL is only marginally better compared to the REINFORCE and PPO baselines in terms of within distribution generalization and worse in terms of out of distribution generalization. Analyzing this result reveals that although many of the runs train rather well, some get stuck during the early stages of training without or only delayed recovering. These outliers lead to a seemingly very large variance for MetaGenRL in Figure 9b. We will provide a more detailed analysis of this result in Section A.2. If we focus on the best performing objective function then we observe competitive performance to DDPG (Figure 9a). Nonetheless, we notice that the objective function trained on Hopper generalizes worse to Lunar, despite our earlier result that objective functions trained on Lunar do in fact generalize well to Hopper. MetaGenRL is still able to outperform both RL2 and EPG in terms of out of distribution generalization. We do note
that EPG is able to meta-learn objective functions that are able to improve to some extent during test time.
Comparing final scores An overview of the final scores that were obtained for MetaGenRL in comparison to the human engineered baselines is shown in Table 2. It can be seen that MetaGenRL outperforms PPO and off-/on-policy REINFORCE in most configurations while DDPG with TD3 tricks remains stronger on two of the three environments. Note that DDPG is currently not among the representable algorithms by MetaGenRL.
A.2 STABILITY OF LEARNED OBJECTIVE FUNCTIONS
In the results presented in Figure 9 on Lunar we observed a seemingly large variance for MetaGenRL that was due to outliers. Indeed, when analyzing the individual runs meta-trained on Lunar and tested on Lunar we found that that one of the runs converged to a local optimum early on during
training and was unable to recover from this afterwards. On the other hand, we also observed that runs can be ‘stuck’ for a long time to then make very fast learning progress. It suggests that the objective function may sometimes experience difficulties in providing meaningful updates to the policy parameters during the early stages of training.
We have further analyzed this issue by evaluating one of the objective functions at several intervals throughout meta-training in Figure 10. From the meta-training curve (bottom) it can be seen that meta-training in Lunar converges very early. This means that from then on, updates to the objective function will be based on mostly converged policies. As the test-time plots show, these additional updates appear to negatively affect test-time performance. We hypothesize that the objective function essentially ‘forgets’ about the early stages of training a randomly initialized agent, by only incorporating information about good performing agents. A possible solution to this problem would be to keep older policies in the meta-training agent population or use early stopping.
Finally, if we exclude four random seeds (of 12), we indeed find a significant reduction in the variance (and increase in the mean) of the results observed for MetaGenRL (see Figure 11).
A.3 ABLATION OF AGENT POPULATION SIZE AND UNIQUE ENVIRONMENTS
In our experiments we have used a population of 20 agents during meta-training to ensure diversity in the conditions under which the objective function needs to optimize. The size of this population is a crucial parameter for a stable meta-optimization. Indeed, in Figure 12 it can be seen that metatraining becomes increasingly unstable as the number of agents in the population decreases.
Using a similar argument, one would expect to gain from increasing the number of distinct environments (or agents) during meta-training. In order to verify this, we have evaluated two additional
settings: Meta-training on Cheetah & Lunar & Walker & Ant with 20 and 40 agents respectively. Figure 13 shows the result of meta-testing on Hopper for these experiments (also see the final results reported for 40 agents in Table 2). Unexpectedly, we find that increasing the number of distinct environments does not yield a significant improvement and, in fact, sometimes even decrease performance. One possibility is that this is due to the simple form of the objective function under consideration, which has no access to the environment observations to efficiently distinguish between them. Another possibility is that MetaGenRL’s hyperparameters require additional tuning in order to be compatible with these setups.

heading: B EXPERIMENT DETAILS
text: In the following we describe all experimental details regarding the architectures used, meta-training, hyperparameters, and baselines. The code to reproduce our experiments is available at http: //louiskirsch.com/code/metagenrl.
B.1 NEURAL OBJECTIVE FUNCTION ARCHITECTURE
Neural Architecture In this work we use an LSTM to implement the objective function (Figure 2). The LSTM runs backwards in time over the state, action, and reward tuples that were encountered during the trajectory τ under consideration. At each step t the LSTM receives as input the reward rt, value estimates of the current and previous state Vt, Vt+1, the current timestep t and finally the action that was taken at the current timestep at in addition to the action as determined by the current policy πφ(st). The actions are first processed by one dimensional convolutional layers striding over the action dimension followed by a reduction to the mean. This allows for different action sizes between environments. Let A(B) ∈ R1×D be the action from the replay buffer, A(π) ∈ R1×D be the action predicted by the policy, and W ∈ R2×N a learnable matrix corresponding to N outgoing units, then the actions are transformed by
1
D D∑ i=1 ([A(B), A(π)]TW )i, (7)
where [a, b] is a concatenation of a and b along the first axis. This corresponds to a convolution with kernel size 1 and stride 1. Further transformations with non-linearities can be added after applying W , if necessary. We found it helpful (but not strictly necessary) to use ReLU activations for half of the units and square activations for the other half.
At each time-step the LSTM outputs a scalar value lt (bounded between−η and η using a scaled tanh activation), which are summed to obtain the value of the neural objective function. Differentiating this value with respect to the policy parameters φ then yields gradients that can be used to improve πφ. We only allow gradients to flow backwards through πφ(st) to φ. This implementation is closely related to the functional form of a REINFORCE (Williams, 1992) estimator using the generalized advantage estimation (Schulman et al., 2015b).
All feed-forward networks (critic and policy) use ReLU activations and layer normalization (Ba et al., 2016). The LSTM uses tanh activations for cell and hidden state transformations, sigmoid activations for the gates. The input time t is normalized between 0 at the beginning of the episode and 1 at the final transition. Any other hyper-parameters can be seen in Table 3.
Extensibility The expressability of the objective function can be further increased through several means. One possibility is to add the entire sequence of state observations o1:T to its inputs, or by introducing a bi-directional LSTM. Secondly, additional information about the policy (such as the hidden state of a recurrent policy) can be provided to L. Although not explored in this work, this would in principle allow one to learn an objective that encourages certain representations to emerge, e.g. a predictive representation about future observations, akin to a world model (Schmidhuber, 1990; Ha & Schmidhuber, 2018; Weber et al., 2017). In turn, these could create pressure to adapt the policy’s actions to explore unknown dynamics in the environment (Schmidhuber, 1991b; 1990; Houthooft et al., 2016; Pathak et al., 2017).
B.2 META-TRAINING
Annealing with DDPG At the beginning of meta-training (learning Lα), the objective function is randomly initialized and thus does not make sensible updates to the policies. This can lead to irreversibly breaking the policies early during training. Our current implementation circumvents this issue by linearly annealing ∇φLα the first 10k timesteps (∼ 2% of all timesteps) with DDPG ∇φQθ(st, πφ(st)). Preliminary experiments suggested that an exponential learning rate schedule on the gradient of ∇φLα for the first 10k steps can replace the annealing with DDPG. The learning rate anneals exponentially between a learning rate of zero and 1e-3. However, in some rare cases this may still lead to unsuccessful training runs, and thus we have omitted this approach from the present work.
Standard training During training, the critic is updated twice as many times as the policy and objective function, similar to TD3 (Fujimoto et al., 2018). One gradient update with data sampled from the replay buffer is applied for every timestep collected from the environment. The gradient with respect to φ in Equation 6 is combined with φ using a fixed learning rate in the standard way, all other parameter updates use Adam (Kingma & Ba, 2015) with the default parameters. Any other hyper-parameters can be seen in Table 3 and Table 4.
Using additional gradient steps In our experiments (Section 5.2.3) we analyzed the effect of applying multiple gradient updates to the policy using Lα before applying the critic to compute second-order gradients with respect to the objective function parameters. For two updates, this gives
∇αQθ(st, πφ†(st)) with φ† = φ′ −∇φ′Lα(τ1, x(φ′), V ) and φ′ = φ−∇φLα(τ2, x(φ), V )
(8)
and can be extended to more than two correspondingly. Additionally, we use disjoint mini batches of data τ : τ1, τ2. When updating the policy using ∇φLα we continue to use only a single gradient step.
B.3 BASELINES
RL2 The implementation for RL2 mimics the paper by Duan et al. (Duan et al., 2016). However, we were unable to achieve good results with TRPO (Schulman et al., 2015a) on the MuJoCo environments and thus used PPO (Schulman et al., 2017) instead. The PPO hyperparameters and implementation are taken from rllib (Liang et al., 2018). Our implementation uses an LSTM with 64 units and does not reset the state of the LSTM for two episodes in sequence. Resetting after additional episodes were given did not improve training results. Different action and observation dimensionalities across environments were handled by using an environment wrapper that pads both with zeros appropriately.
EPG We use the official EPG code base https://github.com/openai/EPG from the original paper (Houthooft et al., 2018). The hyperparameters are taken from the paper, V = 64 noise vectors, an update frequency of M = 64, and 128 updates for every inner loop, resulting in an inner loop length of 8196 steps. During meta-test training, we run with the same update frequency for a total of 1 million steps.
PPO & On-Policy REINFORCE with GAE We use the tuned implementations from https: //spinningup.openai.com/en/latest/spinningup/bench.html which include a GAE (Schulman et al., 2015b) baseline.
Off-Policy Reinforce with GAE The implementation is equivalent to MetaGenRL except that the objective function is fixed to be the REINFORCE estimator with a GAE (Schulman et al., 2015b) baseline. Thus, experience is sampled from a replay buffer. We have also experimented with an importance weighted unbiased estimator but this resulted in poor performance.
DDPG Our implementation is based on https://spinningup.openai.com/en/ latest/spinningup/bench.html and uses the same TD3 tricks (Fujimoto et al., 2018) and hyperparameters (where applicable) that MetaGenRL uses.
Table 4: Training hyperparameters
Parameter Value

Review summaries:
This paper presents a novel meta reinforcement learning algorithm capable of meta-generalizing to unseen tasks. They make use of a learned objective function used in combination with DDPG style update. Results are presented on different combinations of meta-training and meta-testing on lunar, half cheetah, and hopper environments with a focus on meta-generalization to vastly different environments.

The paper proposes a meta reinforcement learning algorithm called MetaGenRL, which meta-learns learning rules to generalize to different environments. The paper poses an important observation where learning rules in reinforcement learning to train the agents are results of human engineering and design, instead, the paper demonstrates how to use second-order gradients to learn learning rules to train agents. Learning learning rules in general has been proposed and this paper is another attempt to further generalize what could be learned in the learning rules. The idea is verified on three Mujoco domains, where the neural objective function is learned from one / two domains, then deployed to a new unseen domain. The experiments show that the learned neural objective can generalize to new environments which are different from the meta-training environments.