Improving Generalization in Meta Reinforcement Learning using Learned Objectives

1 INTRODUCTION . The process of evolution has equipped humans with incredibly general learning algorithms . They enable us to solve a wide range of problems , even in the absence of a large number of related prior experiences . The algorithms that give rise to these capabilities are the result of distilling the collective experiences of many learners throughout the course of natural evolution . By essentially learning from learning experiences in this way , the resulting knowledge can be compactly encoded in the genetic code of an individual to give rise to the general learning capabilities that we observe today . In contrast , Reinforcement Learning ( RL ) in artificial agents rarely proceeds in this way . The learning rules that are used to train agents are the result of years of human engineering and design , ( e.g . Williams ( 1992 ) ; Wierstra et al . ( 2008 ) ; Mnih et al . ( 2013 ) ; Lillicrap et al . ( 2016 ) ; Schulman et al . ( 2015a ) ) . Correspondingly , artificial agents are inherently limited by the ability of the designer to incorporate the right inductive biases in order to learn from previous experiences . Several works have proposed an alternative framework based on meta reinforcement learning ( Schmidhuber , 1994 ; Wang et al. , 2016 ; Duan et al. , 2016 ; Finn et al. , 2017 ; Houthooft et al. , 2018 ; Clune , 2019 ) . Meta-RL distinguishes between learning to act in the environment ( the reinforcement learning problem ) and learning to learn ( the meta-learning problem ) . Hence , learning itself is now a learning problem , which in principle allows one to leverage prior learning experiences to meta-learn general learning rules that surpass human-engineered alternatives . However , while prior work found that learning rules could be meta-learned that generalize to slightly different environments or goals ( Finn et al. , 2017 ; Plappert et al. , 2018 ; Houthooft et al. , 2018 ) , generalization to entirely different environments remains an open problem . In this paper we present MetaGenRL1 , a novel meta reinforcement learning algorithm that metalearns learning rules that generalize to entirely different environments . MetaGenRL is inspired by the process of natural evolution as it distills the experiences of many agents into the parameters of an objective function that decides how future individuals will learn . Similar to Evolved Policy Gradients ( EPG ; Houthooft et al . ( 2018 ) ) , it meta-learns low complexity neural objective functions that can be used to train complex agents with many parameters . However , unlike EPG , it is able to meta-learn using second-order gradients , which offers several advantages as we will demonstrate . We evaluate MetaGenRL on a variety of continuous control tasks and compare to RL2 ( Wang et al. , 2016 ; Duan et al. , 2016 ) and EPG in addition to several human engineered learning algorithms . 1Code is available at http : //louiskirsch.com/code/metagenrl Compared to RL2 we find that MetaGenRL does not overfit and is able to train randomly initialized agents using meta-learned learning rules on entirely different environments . Compared to EPG we find that MetaGenRL is more sample efficient , and outperforms significantly under a fixed budget of environment interactions . The results of an ablation study and additional analysis provide further insight into the benefits of our approach . 2 PRELIMINARIES . Notation We consider the standard MDP Reinforcement Learning setting defined by a tuple e = ( S , A , P , ρ0 , r , γ , T ) consisting of states S , actions A , the transition probability distribution P : S × A × S → R+ , an initial state distribution ρ0 : S → R+ , the reward function r : S × A → [ −Rmax , Rmax ] , a discount factor γ , and the episode length T . The objective for the probabilistic policy πφ : S ×A→ R+ parameterized by φ is to maximize the expected discounted return : Eτ [ T−1∑ t=0 γtrt ] , where s0 ∼ ρ0 ( s0 ) , at ∼ πφ ( at|st ) , st+1 ∼ P ( st+1|st , at ) , rt = r ( st , at ) , ( 1 ) with τ = ( s0 , a0 , r0 , s1 , ... , sT−1 , aT−1 , rT−1 ) . Human Engineered Gradient Estimators A popular gradient-based approach to maximizing Equation 1 is REINFORCE ( Williams , 1992 ) . It directly differentiates Equation 1 with respect to φ using the likelihood ratio trick to derive gradient estimates of the form : ∇φEτ [ LREINF ( τ , πφ ) ] : = Eτ [ ∇φ T−1∑ t=0 log πφ ( at|st ) · T−1∑ t′=t γt ′−trt′ ) ] . ( 2 ) Although this basic estimator is rarely used in practice , it has become a building block for an entire class of policy-gradient algorithms of this form . For example , a popular extension from Schulman et al . ( 2015b ) combines REINFORCE with a Generalized Advantage Estimate ( GAE ) to yield the following policy gradient estimator : ∇φEτ [ LGAE ( τ , πφ , V ) ] : = Eτ [ ∇φ T−1∑ t=0 log πφ ( at|st ) ·A ( τ , V , t ) ] . ( 3 ) where A ( τ , V , t ) is the GAE and V : S → R is a value function estimate . Several recent other extensions include TRPO ( Schulman et al. , 2015a ) , which discourages bad policy updates using trust regions and iterative off-policy updates , or PPO ( Schulman et al. , 2017 ) , which offers similar benefits using only first order approximations . Parametrized Objective Functions In this work we note that many of these human engineered policy gradient estimators can be viewed as specific implementations of a general objective function L that is differentiated with respect to the policy parameters : ∇φEτ [ L ( τ , πφ , V ) ] . ( 4 ) Hence , it becomes natural to consider a generic parametrization of L that , for various choices of parameters α , recovers some of these estimators . In this paper , we will consider neural objective functions where Lα is implemented by a neural network . Our goal is then to optimize the parameters α of this neural network in order to give rise to a new learning algorithm that best maximizes Equation 1 on an entire class of ( different ) environments . 3 META-LEARNING NEURAL OBJECTIVES . In this work we propose MetaGenRL , a novel meta reinforcement learning algorithm that metalearns neural objective functions of the form Lα ( τ , πφ , V ) . MetaGenRL makes use of value functions and second-order gradients , which makes it more sample efficient compared to prior work ( Duan et al. , 2016 ; Wang et al. , 2016 ; Houthooft et al. , 2018 ) . More so , as we will demonstrate , MetaGenRL meta-learns objective functions that generalize to vastly different environments . Our key insight is that a differentiable critic Qθ : S × A → R can be used to measure the effect of locally changing the objective function parameters α based on the quality of the corresponding policy gradients . This enables a population of agents to use and improve a single parameterized objective function Lα through interacting with a set of ( potentially different ) environments . During evaluation ( meta-test time ) , the meta-learned objective function can then be used to train a randomly initialized RL agent in a new environment . 3.1 FROM DDPG TO GRADIENT-BASED META-LEARNING OF NEURAL OBJECTIVES . We will formally introduce MetaGenRL as an extension of the DDPG actor-critic framework ( Silver et al. , 2014 ; Lillicrap et al. , 2016 ) . In DDPG , a parameterized critic of the form Qθ : S × A → R transforms the non-differentiable RL reward maximization problem into a myopic value maximization problem for any st ∈ S. This is done by alternating between optimization of the critic Qθ and the ( here deterministic ) policy πφ . The critic is trained to minimize the TD-error by following : ∇θ ∑ ( st , at , rt , st+1 ) ( Qθ ( st , at ) − yt ) 2 , where yt = rt + γ ·Qθ ( st+1 , πφ ( st+1 ) ) , ( 5 ) and the dependence of yt on the parameter vector θ is ignored . The policy πφ is improved to increase the expected return from arbitrary states by following the gradient∇φ ∑ st Qθ ( st , πφ ( st ) ) . Both gradients can be computed entirely off-policy by sampling trajectories from a replay buffer . MetaGenRL builds on this idea of differentiating the critic Qθ with respect to the policy parameters . It incorporates a parameterized objective function Lα that is used to improve the policy ( i.e . by following the gradient ∇φLα ) , which adds one extra level of indirection : The critic Qθ improves Lα , while Lα improves the policy πφ . By first differentiating with respect to the objective function parameters α , and then with respect to the policy parameters φ , the critic can be used to measure the effect of updating πφ using Lα on the estimated return2 : ∇αQθ ( st , πφ′ ( st ) ) , where φ′ = φ−∇φLα ( τ , x ( φ ) , V ) . ( 6 ) This constitutes a type of second order gradient∇α∇φ that can be used to meta-train Lα to provide better updates to the policy parameters in the future . In practice we will use batching to optimize Equation 6 over multiple trajectories τ . Similarly to the policy-gradient estimators from Section 2 , the objective function Lα ( τ , x ( φ ) , V ) receives as inputs an episode trajectory τ = ( s0 : T−1 , a0 : T−1 , r0 : T−1 ) , the value function estimates 2In case of a probabilistic policy πφ ( at|st ) the following becomes an expectation under πφ and a reparameterizable form is required ( Williams , 1988 ; Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . Here we focus on learning deterministic target policies . Algorithm 1 MetaGenRL : Meta-Training Require : p ( e ) a distribution of environments P ⇐ { ( e1 ∼ p ( e ) , φ1 , θ1 , B1 ← ∅ ) , . . . } . Randomly initialize population of agents Randomly initialize objective function Lα while Lα has not converged do for e , φ , θ , B ∈ P do . For each agent i in parallel if extend replay buffer B then Extend B using πφ in e Sample trajectories from B Update critic Qθ using TD-error Update policy by following ∇φLα Compute objective function gradient ∆i for agent i according to Equation 6 Sum gradients ∑ i ∆i to update Lα V , and an auxiliary input x ( φ ) ( previously πφ ) that can be differentiated with respect to the policy parameters . The latter is critical to be able to differentiate with respect to φ and in the simplest case it consists of the action as predicted by the policy . While Equation 6 is used for meta-learning Lα , the objective functionLα itself is used for policy learning by following∇φLα ( τ , x ( φ ) , V ) . See Figure 1 for an overview . MetaGenRL consists of two phases : During meta-training , we alternate between critic updates , objective function updates , and policy updates to meta-learn an objective function Lα as described in Algorithm 1 . During meta-testing in Algorithm 2 , we take the learned objective function Lα and keep it fixed while training a randomly initialized policy in a new environment to assess its performance . We note that the inputs to Lα are sampled from a replay buffer rather than solely using on-policy data . If Lα were to represent a REINFORCE-type objective then it would mean that differentiating Lα yields biased policy gradient estimates . In our experiments we will find that the gradients from Lα work much better in comparison to a biased off-policy REINFORCE algorithm , and to an importance-sampled unbiased REINFORCE algorithm , while also improving over the popular on-policy REINFORCE and PPO algorithms .

Summary: The paper proposes a meta reinforcement learning algorithm called MetaGenRL, which meta-learns learning rules to generalize to different environments. The paper poses an important observation where learning rules in reinforcement learning to train the agents are results of human engineering and design, instead, the paper demonstrates how to use second-order gradients to learn learning rules to train agents. Learning learning rules in general has been proposed and this paper is another attempt to further generalize what could be learned in the learning rules. The idea is verified on three Mujoco domains, where the neural objective function is learned from one / two domains, then deployed to a new unseen domain. The experiments show that the learned neural objective can generalize to new environments which are different from the meta-training environments.