ProMP: Proximal Meta-Policy Search

1 INTRODUCTION . A remarkable trait of human intelligence is the ability to adapt to new situations in the face of limited experience . In contrast , our most successful artificial agents struggle in such scenarios . While achieving impressive results , they suffer from high sample complexity in learning even a single task , fail to generalize to new situations , and require large amounts of additional data to successfully adapt to new environments . Meta-learning addresses these shortcomings by learning how to learn . Its objective is to learn an algorithm that allows the artificial agent to succeed in an unseen task when only limited experience is available , aiming to achieve the same fast adaptation that humans possess ( Schmidhuber , 1987 ; Thrun & Pratt , 1998 ) . Despite recent progress , deep reinforcement learning ( RL ) still relies heavily on hand-crafted features and reward functions as well as engineered problem specific inductive bias . Meta-RL aims to forego such reliance by acquiring inductive bias in a data-driven manner . Recent work proves this approach to be promising , demonstrating that Meta-RL allows agents to obtain a diverse set of skills , attain better exploration strategies , and learn faster through meta-learned dynamics models or synthetic returns ( Duan et al. , 2016 ; Xu et al. , 2018 ; Gupta et al. , 2018b ; Saemundsson et al. , 2018 ) . Meta-RL is a multi-stage process in which the agent , after a few sampled environment interactions , adapts its behavior to the given task . Despite its wide utilization , little work has been done to promote theoretical understanding of this process , leaving Meta-RL grounded on unstable foundations . Although the behavior prior to the adaptation step is instrumental for task identification , the interplay between pre-adaptation sampling and posterior performance of the policy remains poorly understood . In fact , prior work in gradient-based Meta-RL has either entirely neglected credit assignment to the pre-update distribution ( Finn et al. , 2017 ) or implemented such credit assignment in a naive way ( Al-Shedivat et al. , 2018 ; Stadie et al. , 2018 ) . To our knowledge , we provide the first formal in-depth analysis of credit assignment w.r.t . preadaptation sampling distribution in Meta-RL . Based on our findings , we develop a novel Meta-RL algorithm . First , we analyze two distinct methods for assigning credit to pre-adaptation behavior . ∗authors contributed equally to this work We show that the recent formulation introduced by Al-Shedivat et al . ( 2018 ) and Stadie et al . ( 2018 ) leads to poor credit assignment , while the MAML formulation ( Finn et al. , 2017 ) potentially yields superior meta-policy updates . Second , based on insights from our formal analysis , we highlight both the importance and difficulty of proper meta-policy gradient estimates . In light of this , we propose the low variance curvature ( LVC ) surrogate objective which yields gradient estimates with a favorable bias-variance trade-off . Finally , building upon the LVC estimator we develop Proximal MetaPolicy Search ( ProMP ) , an efficient and stable meta-learning algorithm for RL . In our experiments , we show that ProMP consistently outperforms previous Meta-RL algorithms in sample-efficiency , wall-clock time , and asymptotic performance . 2 RELATED WORK . Meta-Learning concerns the question of “ learning to learn ” , aiming to acquire inductive bias in a data driven manner , so that the learning process in face of unseen data or new problem settings is accelerated ( Schmidhuber , 1987 ; Schmidhuber et al. , 1997 ; Thrun & Pratt , 1998 ) . This can be achieved in various ways . One category of methods attempts to learn the “ learning program ” of an universal Turing machine in form of a recurrent / memory-augmented model that ingests datasets and either outputs the parameters of the trained model ( Hochreiter et al. , 2001 ; Andrychowicz et al. , 2016 ; Chen et al. , 2017 ; Ravi & Larochelle , 2017 ) or directly outputs predictions for given test inputs ( Duan et al. , 2016 ; Santoro et al. , 2016 ; Mishra et al. , 2018 ) . Though very flexible and capable of learning very efficient adaptations , such methods lack performance guarantees and are difficult to train on long sequences that arise in Meta-RL . Another set of methods embeds the structure of a classical learning algorithm in the meta-learning procedure , and optimizes the parameters of the embedded learner during the meta-training ( Hüsken & Goerick , 2000 ; Finn et al. , 2017 ; Nichol et al. , 2018 ; Miconi et al. , 2018 ) . A particular instance of the latter that has proven to be particularly successful in the context of RL is gradient-based metalearning ( Finn et al. , 2017 ; Al-Shedivat et al. , 2018 ; Stadie et al. , 2018 ) . Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task . A desirable property of this approach is that even if fast adaptation fails , the agent just falls back on vanilla policy-gradients . However , as we show , previous gradient-based Meta-RL methods either neglect or perform poor credit assignment w.r.t . the pre-update sampling distribution . A diverse set of methods building on Meta-RL , has recently been introduced . This includes : learning exploration strategies ( Gupta et al. , 2018b ) , synthetic rewards ( Sung et al. , 2017 ; Xu et al. , 2018 ) , unsupervised policy acquisition ( Gupta et al. , 2018a ) , model-based RL ( Clavera et al. , 2018 ; Saemundsson et al. , 2018 ) , learning in competitive environments ( Al-Shedivat et al. , 2018 ) and meta-learning modular policies ( Frans et al. , 2018 ; Alet et al. , 2018 ) . Many of the mentioned approaches build on previous gradient-based meta-learning methods that insufficiently account for the pre-update distribution . ProMP overcomes these deficiencies , providing the necessary framework for novel applications of Meta-RL in unsolved problems . 3 BACKGROUND . Reinforcement Learning . A discrete-time finite Markov decision process ( MDP ) , T , is defined by the tuple ( S , A , p , p0 , r , H ) . Here , S is the set of states , A the action space , p ( st+1|st , at ) the transition distribution , p0 represents the initial state distribution , r : S × A → R is a reward function , and H the time horizon . We omit the discount factor γ in the following elaborations for notational brevity . However , it is straightforward to include it by substituting the reward by r ( st , at ) : = γ tr ( st , at ) . We define the return R ( τ ) as the sum of rewards along a trajectory τ : = ( s0 , a0 , ... , sH−1 , aH−1 , sH ) . The goal of reinforcement learning is to find a policy π ( a|s ) that maximizes the expected return Eτ∼PT ( τ |π ) [ R ( τ ) ] . Meta-Reinforcement Learning goes one step further , aiming to learn a learning algorithm which is able to quickly learn the optimal policy for a task T drawn from a distribution of tasks ρ ( T ) . Each task T corresponds to a different MDP . Typically , it is assumed that the distribution of tasks share the action and state space , but may differ in their reward function or their dynamics . Gradient-based meta-learning aims to solve this problem by learning the parameters θ of a policy πθ such that performing a single or few steps of vanilla policy gradient ( VPG ) with the given task leads to the optimal policy for that task . This meta-learning formulation , also known under the name of MAML , was first introduced by Finn et al . ( 2017 ) . We refer to it as formulation I which can be expressed as maximizing the objective JI ( θ ) = ET ∼ρ ( T ) [ Eτ ′∼PT ( τ ′|θ′ ) [ R ( τ ′ ) ] ] with θ′ : = U ( θ , T ) = θ + α∇θEτ∼PT ( τ |θ ) [ R ( τ ) ] In that U denotes the update function which depends on the task T , and performs one VPG step towards maximizing the performance of the policy in T . For national brevity and conciseness we assume a single policy gradient adaptation step . Nonetheless , all presented concepts can easily be extended to multiple adaptation steps . Later work proposes a slightly different notion of gradient-based Meta-RL , also known as E-MAML , that attempts to circumvent issues with the meta-gradient estimation in MAML ( Al-Shedivat et al. , 2018 ; Stadie et al. , 2018 ) : JII ( θ ) = ET ∼ρ ( T ) [ Eτ1 : N∼PT ( τ1 : N |θ ) τ ′∼PT ( τ ′|θ′ ) [ R ( τ ′ ) ] ] with θ′ : = U ( θ , τ 1 : N ) = θ+α∇θ N∑ n=1 [ R ( τ ( n ) ) ] Formulation II views U as a deterministic function that depends on N sampled trajectories from a specific task . In contrast to formulation I , the expectation over pre-update trajectories τ is applied outside of the update function . Throughout this paper we refer to πθ as pre-update policy , and πθ′ as post-update policy . 4 SAMPLING DISTRIBUTION CREDIT ASSIGNMENT . This section analyzes the two gradient-based Meta-RL formulations introduced in Section 3 . Figure 1 illustrates the stochastic computation graphs ( Schulman et al. , 2015b ) of both formulations . The red arrows depict how credit assignment w.r.t the pre-update sampling distribution PT ( τ |θ ) is propagated . Formulation I ( left ) propagates the credit assignment through the update step , thereby exploiting the full problem structure . In contrast , formulation II ( right ) neglects the inherent structure , directly assigning credit from post-update return R′ to the pre-update policy πθ which leads to noisier , less effective credit assignment . Both formulations optimize for the same objective , and are equivalent at the 0th order . However , because of the difference in their formulation and stochastic computation graph , their gradients and the resulting optimization step differs . In the following , we shed light on how and where formulation II loses signal by analyzing the gradients of both formulations , which can be written as ( see Appendix A for more details and derivations ) ∇θJ ( θ ) = ET ∼ρ ( T ) [ E τ∼PT ( τ |θ ) τ ′∼PT ( τ ′|θ′ ) [ ∇θJpost ( τ , τ ′ ) +∇θJpre ( τ , τ ′ ) ] ] ( 1 ) The first term ∇θJpost ( τ , τ ′ ) is equal in both formulations , but the second term , ∇θJpre ( τ , τ ′ ) , differs between them . In particular , they correspond to ∇θJpost ( τ , τ ′ ) = ∇θ′ log πθ ( τ ′ ) R ( τ ′ ) ︸ ︷︷ ︸ ∇θ′Jouter ( I + αR ( τ ) ∇2θ log πθ′ ( τ ) ) ) ︸ ︷︷ ︸ transformation from θ′ to θ ( 2 ) ∇θJIIpre ( τ , τ ′ ) = α∇θ log πθ ( τ ) R ( τ ′ ) ( 3 ) ∇θJIpre ( τ , τ ′ ) = α∇θ log πθ ( τ ) ( ( ∇θ log πθ ( τ ) R ( τ ) ) > ︸ ︷︷ ︸ ∇θJ inner ( ∇θ′ log πθ′ ( τ ′ ) R ( τ ′ ) ) ︸ ︷︷ ︸ ∇θ′Jouter ) ( 4 ) ∇θJpost ( τ , τ ′ ) simply corresponds to a policy gradient step on the post-update policy πθ′ w.r.t θ′ , followed by a linear transformation from post- to pre-update parameters . It corresponds to increasing the likelihood of the trajectories τ ′ that led to higher returns . However , this term does not optimize for the pre-update sampling distribution , i.e. , which trajectories τ led to better adaptation steps . The credit assignment w.r.t . the pre-updated sampling distribution is carried out by the second term . In formulation II , ∇θJIIpre can be viewed as standard reinforcement learning on πθ with R ( τ ′ ) as reward signal , treating the update function U as part of the unknown dynamics of the system . This shifts the pre-update sampling distribution to better adaptation steps . Formulation I takes the causal dependence of PT ( τ ′|θ′ ) on PT ( τ |θ ) into account . It does so by maximizing the inner product of pre-update and post-update policy gradients ( see Eq . 4 ) . This steers the pre-update policy towards 1 ) larger post-updates returns 2 ) larger adaptation steps α∇θJ inner , 3 ) better alignment of pre- and post-update policy gradients ( Li et al. , 2017 ; Nichol et al. , 2018 ) . When combined , these effects directly optimize for adaptation . As a result , we expect the first meta-policy gradient formulation , JI , to yield superior learning properties .

Summary: In this paper, the authors investigate the gradient calculation in the original MAML (Finn et al. 2017) and E-MAML (Al-Shedivat et al. 2018). By comparing the differences in the gradients of these two algorithms, the authors demonstrate the advantages of the original MAML in taking the casual dependence into account. To obtain the correct estimation of the gradient through auto-differentiation, the authors exploit the DiCE formulation. Considering the variance in the DiCE objective formulation, the authors finally propose an objective which leads to low-variance but biased gradient. The authors verify the proposed methods in meta-RL tasks and achieves comparable performances to MAML and E-MAML.