ProMP: Proximal Meta-Policy Search

1 INTRODUCTION . A remarkable trait of human intelligence is the ability to adapt to new situations in the face of limited experience . In contrast , our most successful artificial agents struggle in such scenarios . While achieving impressive results , they suffer from high sample complexity in learning even a single task , fail to generalize to new situations , and require large amounts of additional data to successfully adapt to new environments . Meta-learning addresses these shortcomings by learning how to learn . Its objective is to learn an algorithm that allows the artificial agent to succeed in an unseen task when only limited experience is available , aiming to achieve the same fast adaptation that humans possess ( Schmidhuber , 1987 ; Thrun & Pratt , 1998 ) . Despite recent progress , deep reinforcement learning ( RL ) still relies heavily on hand-crafted features and reward functions as well as engineered problem specific inductive bias . Meta-RL aims to forego such reliance by acquiring inductive bias in a data-driven manner . Recent work proves this approach to be promising , demonstrating that Meta-RL allows agents to obtain a diverse set of skills , attain better exploration strategies , and learn faster through meta-learned dynamics models or synthetic returns ( Duan et al. , 2016 ; Xu et al. , 2018 ; Gupta et al. , 2018b ; Saemundsson et al. , 2018 ) . Meta-RL is a multi-stage process in which the agent , after a few sampled environment interactions , adapts its behavior to the given task . Despite its wide utilization , little work has been done to promote theoretical understanding of this process , leaving Meta-RL grounded on unstable foundations . Although the behavior prior to the adaptation step is instrumental for task identification , the interplay between pre-adaptation sampling and posterior performance of the policy remains poorly understood . In fact , prior work in gradient-based Meta-RL has either entirely neglected credit assignment to the pre-update distribution ( Finn et al. , 2017 ) or implemented such credit assignment in a naive way ( Al-Shedivat et al. , 2018 ; Stadie et al. , 2018 ) . To our knowledge , we provide the first formal in-depth analysis of credit assignment w.r.t . preadaptation sampling distribution in Meta-RL . Based on our findings , we develop a novel Meta-RL algorithm . First , we analyze two distinct methods for assigning credit to pre-adaptation behavior . ∗authors contributed equally to this work We show that the recent formulation introduced by Al-Shedivat et al . ( 2018 ) and Stadie et al . ( 2018 ) leads to poor credit assignment , while the MAML formulation ( Finn et al. , 2017 ) potentially yields superior meta-policy updates . Second , based on insights from our formal analysis , we highlight both the importance and difficulty of proper meta-policy gradient estimates . In light of this , we propose the low variance curvature ( LVC ) surrogate objective which yields gradient estimates with a favorable bias-variance trade-off . Finally , building upon the LVC estimator we develop Proximal MetaPolicy Search ( ProMP ) , an efficient and stable meta-learning algorithm for RL . In our experiments , we show that ProMP consistently outperforms previous Meta-RL algorithms in sample-efficiency , wall-clock time , and asymptotic performance . 2 RELATED WORK . Meta-Learning concerns the question of “ learning to learn ” , aiming to acquire inductive bias in a data driven manner , so that the learning process in face of unseen data or new problem settings is accelerated ( Schmidhuber , 1987 ; Schmidhuber et al. , 1997 ; Thrun & Pratt , 1998 ) . This can be achieved in various ways . One category of methods attempts to learn the “ learning program ” of an universal Turing machine in form of a recurrent / memory-augmented model that ingests datasets and either outputs the parameters of the trained model ( Hochreiter et al. , 2001 ; Andrychowicz et al. , 2016 ; Chen et al. , 2017 ; Ravi & Larochelle , 2017 ) or directly outputs predictions for given test inputs ( Duan et al. , 2016 ; Santoro et al. , 2016 ; Mishra et al. , 2018 ) . Though very flexible and capable of learning very efficient adaptations , such methods lack performance guarantees and are difficult to train on long sequences that arise in Meta-RL . Another set of methods embeds the structure of a classical learning algorithm in the meta-learning procedure , and optimizes the parameters of the embedded learner during the meta-training ( Hüsken & Goerick , 2000 ; Finn et al. , 2017 ; Nichol et al. , 2018 ; Miconi et al. , 2018 ) . A particular instance of the latter that has proven to be particularly successful in the context of RL is gradient-based metalearning ( Finn et al. , 2017 ; Al-Shedivat et al. , 2018 ; Stadie et al. , 2018 ) . Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task . A desirable property of this approach is that even if fast adaptation fails , the agent just falls back on vanilla policy-gradients . However , as we show , previous gradient-based Meta-RL methods either neglect or perform poor credit assignment w.r.t . the pre-update sampling distribution . A diverse set of methods building on Meta-RL , has recently been introduced . This includes : learning exploration strategies ( Gupta et al. , 2018b ) , synthetic rewards ( Sung et al. , 2017 ; Xu et al. , 2018 ) , unsupervised policy acquisition ( Gupta et al. , 2018a ) , model-based RL ( Clavera et al. , 2018 ; Saemundsson et al. , 2018 ) , learning in competitive environments ( Al-Shedivat et al. , 2018 ) and meta-learning modular policies ( Frans et al. , 2018 ; Alet et al. , 2018 ) . Many of the mentioned approaches build on previous gradient-based meta-learning methods that insufficiently account for the pre-update distribution . ProMP overcomes these deficiencies , providing the necessary framework for novel applications of Meta-RL in unsolved problems . 3 BACKGROUND . Reinforcement Learning . A discrete-time finite Markov decision process ( MDP ) , T , is defined by the tuple ( S , A , p , p0 , r , H ) . Here , S is the set of states , A the action space , p ( st+1|st , at ) the transition distribution , p0 represents the initial state distribution , r : S × A → R is a reward function , and H the time horizon . We omit the discount factor γ in the following elaborations for notational brevity . However , it is straightforward to include it by substituting the reward by r ( st , at ) : = γ tr ( st , at ) . We define the return R ( τ ) as the sum of rewards along a trajectory τ : = ( s0 , a0 , ... , sH−1 , aH−1 , sH ) . The goal of reinforcement learning is to find a policy π ( a|s ) that maximizes the expected return Eτ∼PT ( τ |π ) [ R ( τ ) ] . Meta-Reinforcement Learning goes one step further , aiming to learn a learning algorithm which is able to quickly learn the optimal policy for a task T drawn from a distribution of tasks ρ ( T ) . Each task T corresponds to a different MDP . Typically , it is assumed that the distribution of tasks share the action and state space , but may differ in their reward function or their dynamics . Gradient-based meta-learning aims to solve this problem by learning the parameters θ of a policy πθ such that performing a single or few steps of vanilla policy gradient ( VPG ) with the given task leads to the optimal policy for that task . This meta-learning formulation , also known under the name of MAML , was first introduced by Finn et al . ( 2017 ) . We refer to it as formulation I which can be expressed as maximizing the objective JI ( θ ) = ET ∼ρ ( T ) [ Eτ ′∼PT ( τ ′|θ′ ) [ R ( τ ′ ) ] ] with θ′ : = U ( θ , T ) = θ + α∇θEτ∼PT ( τ |θ ) [ R ( τ ) ] In that U denotes the update function which depends on the task T , and performs one VPG step towards maximizing the performance of the policy in T . For national brevity and conciseness we assume a single policy gradient adaptation step . Nonetheless , all presented concepts can easily be extended to multiple adaptation steps . Later work proposes a slightly different notion of gradient-based Meta-RL , also known as E-MAML , that attempts to circumvent issues with the meta-gradient estimation in MAML ( Al-Shedivat et al. , 2018 ; Stadie et al. , 2018 ) : JII ( θ ) = ET ∼ρ ( T ) [ Eτ1 : N∼PT ( τ1 : N |θ ) τ ′∼PT ( τ ′|θ′ ) [ R ( τ ′ ) ] ] with θ′ : = U ( θ , τ 1 : N ) = θ+α∇θ N∑ n=1 [ R ( τ ( n ) ) ] Formulation II views U as a deterministic function that depends on N sampled trajectories from a specific task . In contrast to formulation I , the expectation over pre-update trajectories τ is applied outside of the update function . Throughout this paper we refer to πθ as pre-update policy , and πθ′ as post-update policy . 4 SAMPLING DISTRIBUTION CREDIT ASSIGNMENT . This section analyzes the two gradient-based Meta-RL formulations introduced in Section 3 . Figure 1 illustrates the stochastic computation graphs ( Schulman et al. , 2015b ) of both formulations . The red arrows depict how credit assignment w.r.t the pre-update sampling distribution PT ( τ |θ ) is propagated . Formulation I ( left ) propagates the credit assignment through the update step , thereby exploiting the full problem structure . In contrast , formulation II ( right ) neglects the inherent structure , directly assigning credit from post-update return R′ to the pre-update policy πθ which leads to noisier , less effective credit assignment . Both formulations optimize for the same objective , and are equivalent at the 0th order . However , because of the difference in their formulation and stochastic computation graph , their gradients and the resulting optimization step differs . In the following , we shed light on how and where formulation II loses signal by analyzing the gradients of both formulations , which can be written as ( see Appendix A for more details and derivations ) ∇θJ ( θ ) = ET ∼ρ ( T ) [ E τ∼PT ( τ |θ ) τ ′∼PT ( τ ′|θ′ ) [ ∇θJpost ( τ , τ ′ ) +∇θJpre ( τ , τ ′ ) ] ] ( 1 ) The first term ∇θJpost ( τ , τ ′ ) is equal in both formulations , but the second term , ∇θJpre ( τ , τ ′ ) , differs between them . In particular , they correspond to ∇θJpost ( τ , τ ′ ) = ∇θ′ log πθ ( τ ′ ) R ( τ ′ ) ︸ ︷︷ ︸ ∇θ′Jouter ( I + αR ( τ ) ∇2θ log πθ′ ( τ ) ) ) ︸ ︷︷ ︸ transformation from θ′ to θ ( 2 ) ∇θJIIpre ( τ , τ ′ ) = α∇θ log πθ ( τ ) R ( τ ′ ) ( 3 ) ∇θJIpre ( τ , τ ′ ) = α∇θ log πθ ( τ ) ( ( ∇θ log πθ ( τ ) R ( τ ) ) > ︸ ︷︷ ︸ ∇θJ inner ( ∇θ′ log πθ′ ( τ ′ ) R ( τ ′ ) ) ︸ ︷︷ ︸ ∇θ′Jouter ) ( 4 ) ∇θJpost ( τ , τ ′ ) simply corresponds to a policy gradient step on the post-update policy πθ′ w.r.t θ′ , followed by a linear transformation from post- to pre-update parameters . It corresponds to increasing the likelihood of the trajectories τ ′ that led to higher returns . However , this term does not optimize for the pre-update sampling distribution , i.e. , which trajectories τ led to better adaptation steps . The credit assignment w.r.t . the pre-updated sampling distribution is carried out by the second term . In formulation II , ∇θJIIpre can be viewed as standard reinforcement learning on πθ with R ( τ ′ ) as reward signal , treating the update function U as part of the unknown dynamics of the system . This shifts the pre-update sampling distribution to better adaptation steps . Formulation I takes the causal dependence of PT ( τ ′|θ′ ) on PT ( τ |θ ) into account . It does so by maximizing the inner product of pre-update and post-update policy gradients ( see Eq . 4 ) . This steers the pre-update policy towards 1 ) larger post-updates returns 2 ) larger adaptation steps α∇θJ inner , 3 ) better alignment of pre- and post-update policy gradients ( Li et al. , 2017 ; Nichol et al. , 2018 ) . When combined , these effects directly optimize for adaptation . As a result , we expect the first meta-policy gradient formulation , JI , to yield superior learning properties .

Summary: In this paper, the author proposed an efficient surrogate loss for estimating  Hessian in the setting of Meta-reinforcement learning (Finn.et al, 2017), which significantly reduce the variance while introducing small bias. The author verified their proposed method with other meta-learning algorithms on the Mujoco benchmarks. The author also compared with unbiased higher order gradient estimation method-DiCE in terms of gradient variance and average return.