Meta-Model-Based Meta-Policy Optimization

Model-based reinforcement learning ( MBRL ) has been applied to meta-learning settings and has demonstrated its high sample efficiency . However , in previous MBRL for meta-learning settings , policies are optimized via rollouts that fully rely on a predictive model of an environment . Thus , its performance in a real environment tends to degrade when the predictive model is inaccurate . In this paper , we prove that performance degradation can be suppressed by using branched meta-rollouts . On the basis of this theoretical analysis , we propose Meta-Modelbased Meta-Policy Optimization ( M3PO ) , in which the branched meta-rollouts are used for policy optimization . We demonstrate that M3PO outperforms existing meta reinforcement learning methods in continuous-control benchmarks . 1 INTRODUCTION . Reinforcement learning ( RL ) methods have achieved remarkable success in many decision-making tasks , such as playing video games or controlling robots ( e.g. , Gu et al . ( 2017 ) ; Mnih et al . ( 2015 ) ) . In conventional RL methods , when multiple tasks are to be solved , a policy is independently learned for individual tasks . In general , each learning requires millions of training samples from the environment . This independent learning with a large number of samples prevents conventional RL methods from being applied to practical multi-task problems ( e.g. , robotic manipulation problems involving grasping or moving different types of objects ( Yu et al. , 2019 ) ) . Meta-learning methods ( Schmidhuber et al. , 1996 ; Thrun & Pratt , 1998 ) have recently gained much attention as a promising solution to this problem ( Finn et al. , 2017 ) . They learn a structure shared in the tasks by using a large number of samples collected across the parts of the tasks . Once learned , these methods can adapt quickly to new ( or the rest of the ) tasks with a small number of samples given . Meta-RL methods have previously been introduced into both model-free and model-based settings . For model-free settings , there are two main types of approaches proposed so far , recurrent-based policy adaptation ( Duan et al. , 2017 ; Mishra et al. , 2018 ; Rakelly et al. , 2019 ; Wang et al. , 2016 ) and gradient-based policy adaptation ( Al-Shedivat et al. , 2018 ; Finn & Levine , 2018 ; Finn et al. , 2017 ; Gupta et al. , 2018 ; Rothfuss et al. , 2019 ; Stadie et al. , 2018 ) . In these approaches , policies adapt to a new task by leveraging the history of past trajectories . Following previous work ( Clavera et al. , 2018 ) , we refer to these adaptive policies as meta-policies in our paper . In these modelfree meta-RL methods , in addition to learning control policies , the learning of policy adaptation is also required ( Mendonca et al. , 2019 ) . Thus , these methods require more training samples than conventional RL methods . For model-based settings , there have been relatively few approaches proposed so far . Sæmundsson et al . ( 2018 ) and Perez et al . ( 2020 ) use a predictive model ( i.e. , a transition model ) conditioned by a latent variable for model predictive control . Nagabandi et al . ( 2019a ; b ) introduced both recurrentbased and gradient-based meta-learning methods into model-based RL . In these approaches , the predictive models adapt to a new task by leveraging the history of past trajectories . In analogy to the meta-policy , we refer to these adaptive predictive models as meta-models in our paper . Generally , these model-based meta-RL approaches are more sample efficient than the model-free approaches . However , in these approaches , the meta-policy ( or the course of actions ) is optimized via rollouts relying fully on the meta-model . Thus , its performance in a real environment tends to degrade when the meta-model is inaccurate . In this paper , we address this performance degradation problem in model-based meta-RL . After reviewing related work ( Section 2 ) and preliminaries ( Section 3 ) , we present our work by first formulating model-based meta-RL ( Section 4 ) . Model-based ( and model-free ) meta-RL settings have typically been formulated as special cases of solving partially observable Markov decision processes ( POMDPs ) ( e.g. , Duan et al . ( 2017 ) ; Killian et al . ( 2017 ) ; Perez et al . ( 2020 ) ) . In these special cases , specific assumptions , such as intra-episode task invariance , are additionally introduced . However , there are model-based meta-RL settings where such assumptions do not hold ( e.g. , Nagabandi et al . ( 2019a ; b ) ) . To include these settings into our scope , we formulate model-based meta-RL settings as solving POMDPs without introducing such additional assumptions . Then , we conduct theoretical analysis on its performance guarantee ( Section 5 ) . We first analyse the performance guarantee in full meta-model-based rollouts , which most of the previous model-based metaRL methods hold . We then introduce the notion of branched meta-rollouts . Branched meta-rollouts are Dyna-style rollouts ( Sutton , 1991 ) in which we can adjust the reliance on the meta-model and real environment data . We show that the performance degradation due to the meta-model error in the branched meta-rollouts is smaller than that in the full meta-model-based rollouts . On the basis of this theoretical analysis , we propose a practical model-based meta-RL method called Meta-Model-based Meta-Policy Optimization ( M3PO ) where the meta-model is used in the branched rollout manner ( Section 6 ) . Finally , we experimentally demonstrate that M3PO outperforms existing methods in continuous-control benchmarks ( Section 7 ) . We make the following contributions in both theoretical and empirical frontiers . Theoretical frontier : 1 . Our work is the first attempt to provide a theoretical relation between learning the metamodel and the real environment performance . In the aforementioned model-based meta-RL literature , it has not been clear how learning the meta-model relates to real environment performance . Our theoretical analysis provides relations between them ( Theorems 1 , 2 and 3 ) . This result theoretically justifies meta-training a good transition model to improve overall performance in the real environment . 2 . Our analysis also reveals that the use of branched meta-rollouts can suppress performance degradation due to meta-model errors . 3 . We refine previous fundamental theories proposed by Janner et al . ( 2019 ) to consider important premises more properly ( Theorems 4 and 5 ) . This modification is important to strictly guarantee the performance especially when the model-rollout length is long . Empirical frontier : We propose and show the effectiveness of M3PO . Notably , we show that M3PO achieves better sample efficiency than existing meta-RL methods in complex tasks , such as controlling humanoids . 2 RELATED WORK . In this section , we review related work on POMDPs and theoretical analysis in model-based RL . Partially observable Markov decision processes 1 : In our paper , we formulate model-based metaRL as solving POMDPs , and provide its performance guarantee under the branched meta-rollout scheme . POMDPs are a long-studied problem ( e.g. , ( Ghavamzadeh et al. , 2015 ; Sun , 2019 ; Sun et al. , 2019 ) ) , and many works have discussed a performance guarantee of RL methods to solve POMDPs . However , the performance guarantee of the RL methods based on branched meta-rollouts has not been discussed in the literature . On the other hand , a number of researchers ( Igl et al. , 2018 ; Lee et al. , 2019 ; Zintgraf et al. , 2020 ) have proposed model-free RL methods to solve a POMDP without prior knowledge of the accurate model . However , they do not provide theoretical analyses of performance . In this work , by contrast , we propose a model-based meta-RL method and provide theoretical analyses on its performance guarantee . Theoretical analysis on the performance of model-based RL : Several theoretical analyses on the performance of model-based RL have been provided in previous work ( Feinberg et al. , 2018 ; Henaff , 2019 ; Janner et al. , 2019 ; Luo et al. , 2018 ; Rajeswaran et al. , 2020 ) . In these theoretical analyses , standard Markov decision processes ( MDPs ) are assumed , and the meta-learning ( or POMDP ) setting is not discussed . In contrast , our work provides a theoretical analysis on the meta-learning ( and POMDP ) setting , by substantially extending the work of Janner et al . ( 2019 ) . Specifically , Janner et al . ( 2019 ) analysed the performance guarantee of branched rollouts on MDPs , and introduced branched rollouts into a model-based RL algorithm . We extend their analysis and algorithm to a meta-learning ( POMDP ) case . In addition , we modify their theorems so that important premises 1We include works on Bayes-adaptive MDPs ( Ghavamzadeh et al. , 2015 ; Zintgraf et al. , 2020 ) because they are a special case of POMDPs . ( e.g. , the effect of multiple-model rollout factors ) are more properly considered . See A.1 in the appendix for a more detailed discussion of our contribution . 3 PRELIMINARIES . Meta reinforcement learning : We assume online adaptation situations ( Nagabandi et al. , 2019a ; b ) where the agent can leverage a few samples to adapt to a new task . Here , a task specifies the transition probability and the reward function . Information about task identity can not be observed by the agent , and the task may change at any step in an episode . A meta-RL process is composed of meta-training and meta-testing . In meta-training , a policy and a predictive model that are prepared for efficient adaptation are learned with a meta-training task set . In meta-testing , on the basis of the meta-training result , the policy and the predictive model adapt to a new task . For the adaptation , the trajectory observed from the beginning of the episode to the current time step is leveraged . As we noted earlier , we call an adaptive policy and a predictive model of this sort a meta-policy and a meta-model , respectively . Partially observable Markov decision processes : We formalize our problem with a POMDP , which is defined as a tuple 〈O , S , A , pob , r , γ , pst〉 . Here , O is a set of observations , S is a set of hidden states , A is a set of actions , pob : = O × S × A → [ 0 , 1 ] is the observation probability , pst : = S × S × A → [ 0 , 1 ] is the state transition probability , r : S × A → R is a reward function and γ ∈ [ 0 , 1 ) is a discount factor . At time step t , these functions are used as pst ( st|st−1 , at−1 ) , pob ( ot|st , at−1 ) 2 and rt = r ( st , at ) . The agent can not directly observe the hidden state , but receives the observation instead . The agent selects an action on the basis of a policy π : = p ( at+1|ht ) . Here , ht is a history ( the past trajectories ) defined as ht : = { a0 , o0 , ... , at , ot } . We denote the set of the histories by H. Given the definition of the history , the history transition probability can be defined as p ( ht+1|at+1 , ht ) : = p ( ot+1|ht ) . Here , p ( ot+1|ht ) : = ∑ st+1 ∑ st p ( st|ht ) p ( st+1|st , at ) p ( ot+1|st+1 , at ) , where p ( st|ht ) is the belief about the hidden state . The goal of RL in the POMDP is to find the optimal policy π∗ that maximizes the expected return R : = ∑∞ t=0 γ trt ( i.e. , π∗ = arg max π Ea∼π , h∼p [ R ] ) .

Summary: The paper concerns model-based meta-RL. It exploits the fact that meta-RL can be formulated as POMDP in which the task indicator is part of the (unobserved) hidden state. Thus, the paper effectively analyzes and proposes model-based algorithms for POMDPs. The paper bounds the gap between the expected reward of a policy in the actual POMDP and the estimated model and then theoretically shows that this gap can be reduced when using dyna-style / branched rollouts instead of full rollouts under the learned model. Motivated by this finding, the paper proposes a Dyna-like algorithm for POMDPs. In the experimental evaluation, the paper compares its proposed method, M3PO, to two recent meta-RL approaches in a range of meta-RL environments for continuous control.