Year: 2020

Abstract:
Model-based reinforcement learning (MBRL) has been applied to meta-learning settings and has demonstrated its high sample efficiency. However, in previous MBRL for meta-learning settings, policies are optimized via rollouts that fully rely on a predictive model of an environment. Thus, its performance in a real environment tends to degrade when the predictive model is inaccurate. In this paper, we prove that performance degradation can be suppressed by using branched meta-rollouts. On the basis of this theoretical analysis, we propose Meta-Modelbased Meta-Policy Optimization (M3PO), in which the branched meta-rollouts are used for policy optimization. We demonstrate that M3PO outperforms existing meta reinforcement learning methods in continuous-control benchmarks.

text: Model-based reinforcement learning (MBRL) has been applied to meta-learning settings and has demonstrated its high sample efficiency. However, in previous MBRL for meta-learning settings, policies are optimized via rollouts that fully rely on a predictive model of an environment. Thus, its performance in a real environment tends to degrade when the predictive model is inaccurate. In this paper, we prove that performance degradation can be suppressed by using branched meta-rollouts. On the basis of this theoretical analysis, we propose Meta-Modelbased Meta-Policy Optimization (M3PO), in which the branched meta-rollouts are used for policy optimization. We demonstrate that M3PO outperforms existing meta reinforcement learning methods in continuous-control benchmarks.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) methods have achieved remarkable success in many decision-making tasks, such as playing video games or controlling robots (e.g., Gu et al. (2017); Mnih et al. (2015)). In conventional RL methods, when multiple tasks are to be solved, a policy is independently learned for individual tasks. In general, each learning requires millions of training samples from the environment. This independent learning with a large number of samples prevents conventional RL methods from being applied to practical multi-task problems (e.g., robotic manipulation problems involving grasping or moving different types of objects (Yu et al., 2019)). Meta-learning methods (Schmidhuber et al., 1996; Thrun & Pratt, 1998) have recently gained much attention as a promising solution to this problem (Finn et al., 2017). They learn a structure shared in the tasks by using a large number of samples collected across the parts of the tasks. Once learned, these methods can adapt quickly to new (or the rest of the) tasks with a small number of samples given.
Meta-RL methods have previously been introduced into both model-free and model-based settings. For model-free settings, there are two main types of approaches proposed so far, recurrent-based policy adaptation (Duan et al., 2017; Mishra et al., 2018; Rakelly et al., 2019; Wang et al., 2016) and gradient-based policy adaptation (Al-Shedivat et al., 2018; Finn & Levine, 2018; Finn et al., 2017; Gupta et al., 2018; Rothfuss et al., 2019; Stadie et al., 2018). In these approaches, policies adapt to a new task by leveraging the history of past trajectories. Following previous work (Clavera et al., 2018), we refer to these adaptive policies as meta-policies in our paper. In these modelfree meta-RL methods, in addition to learning control policies, the learning of policy adaptation is also required (Mendonca et al., 2019). Thus, these methods require more training samples than conventional RL methods.
For model-based settings, there have been relatively few approaches proposed so far. Sæmundsson et al. (2018) and Perez et al. (2020) use a predictive model (i.e., a transition model) conditioned by a latent variable for model predictive control. Nagabandi et al. (2019a;b) introduced both recurrentbased and gradient-based meta-learning methods into model-based RL. In these approaches, the predictive models adapt to a new task by leveraging the history of past trajectories. In analogy to the meta-policy, we refer to these adaptive predictive models as meta-models in our paper. Generally, these model-based meta-RL approaches are more sample efficient than the model-free approaches. However, in these approaches, the meta-policy (or the course of actions) is optimized via rollouts relying fully on the meta-model. Thus, its performance in a real environment tends to degrade when the meta-model is inaccurate. In this paper, we address this performance degradation problem in model-based meta-RL.
After reviewing related work (Section 2) and preliminaries (Section 3), we present our work by first formulating model-based meta-RL (Section 4). Model-based (and model-free) meta-RL settings have typically been formulated as special cases of solving partially observable Markov decision processes (POMDPs) (e.g.,Duan et al. (2017); Killian et al. (2017); Perez et al. (2020)). In these special cases, specific assumptions, such as intra-episode task invariance, are additionally introduced. However, there are model-based meta-RL settings where such assumptions do not hold (e.g., Nagabandi et al. (2019a;b)). To include these settings into our scope, we formulate model-based meta-RL settings as solving POMDPs without introducing such additional assumptions. Then, we conduct theoretical analysis on its performance guarantee (Section 5). We first analyse the performance guarantee in full meta-model-based rollouts, which most of the previous model-based metaRL methods hold. We then introduce the notion of branched meta-rollouts. Branched meta-rollouts are Dyna-style rollouts (Sutton, 1991) in which we can adjust the reliance on the meta-model and real environment data. We show that the performance degradation due to the meta-model error in the branched meta-rollouts is smaller than that in the full meta-model-based rollouts. On the basis of this theoretical analysis, we propose a practical model-based meta-RL method called Meta-Model-based Meta-Policy Optimization (M3PO) where the meta-model is used in the branched rollout manner (Section 6). Finally, we experimentally demonstrate that M3PO outperforms existing methods in continuous-control benchmarks (Section 7).
We make the following contributions in both theoretical and empirical frontiers. Theoretical frontier: 1. Our work is the first attempt to provide a theoretical relation between learning the metamodel and the real environment performance. In the aforementioned model-based meta-RL literature, it has not been clear how learning the meta-model relates to real environment performance. Our theoretical analysis provides relations between them (Theorems 1, 2 and 3). This result theoretically justifies meta-training a good transition model to improve overall performance in the real environment. 2. Our analysis also reveals that the use of branched meta-rollouts can suppress performance degradation due to meta-model errors. 3. We refine previous fundamental theories proposed by Janner et al. (2019) to consider important premises more properly (Theorems 4 and 5). This modification is important to strictly guarantee the performance especially when the model-rollout length is long. Empirical frontier: We propose and show the effectiveness of M3PO. Notably, we show that M3PO achieves better sample efficiency than existing meta-RL methods in complex tasks, such as controlling humanoids.

heading: 2 RELATED WORK
text: In this section, we review related work on POMDPs and theoretical analysis in model-based RL.
Partially observable Markov decision processes 1: In our paper, we formulate model-based metaRL as solving POMDPs, and provide its performance guarantee under the branched meta-rollout scheme. POMDPs are a long-studied problem (e.g., (Ghavamzadeh et al., 2015; Sun, 2019; Sun et al., 2019)), and many works have discussed a performance guarantee of RL methods to solve POMDPs. However, the performance guarantee of the RL methods based on branched meta-rollouts has not been discussed in the literature. On the other hand, a number of researchers (Igl et al., 2018; Lee et al., 2019; Zintgraf et al., 2020) have proposed model-free RL methods to solve a POMDP without prior knowledge of the accurate model. However, they do not provide theoretical analyses of performance. In this work, by contrast, we propose a model-based meta-RL method and provide theoretical analyses on its performance guarantee.
Theoretical analysis on the performance of model-based RL: Several theoretical analyses on the performance of model-based RL have been provided in previous work (Feinberg et al., 2018; Henaff, 2019; Janner et al., 2019; Luo et al., 2018; Rajeswaran et al., 2020). In these theoretical analyses, standard Markov decision processes (MDPs) are assumed, and the meta-learning (or POMDP) setting is not discussed. In contrast, our work provides a theoretical analysis on the meta-learning (and POMDP) setting, by substantially extending the work of Janner et al. (2019). Specifically, Janner et al. (2019) analysed the performance guarantee of branched rollouts on MDPs, and introduced branched rollouts into a model-based RL algorithm. We extend their analysis and algorithm to a meta-learning (POMDP) case. In addition, we modify their theorems so that important premises
1We include works on Bayes-adaptive MDPs (Ghavamzadeh et al., 2015; Zintgraf et al., 2020) because they are a special case of POMDPs.
(e.g., the effect of multiple-model rollout factors) are more properly considered. See A.1 in the appendix for a more detailed discussion of our contribution.

heading: 3 PRELIMINARIES
text: Meta reinforcement learning: We assume online adaptation situations (Nagabandi et al., 2019a;b) where the agent can leverage a few samples to adapt to a new task. Here, a task specifies the transition probability and the reward function. Information about task identity cannot be observed by the agent, and the task may change at any step in an episode. A meta-RL process is composed of meta-training and meta-testing. In meta-training, a policy and a predictive model that are prepared for efficient adaptation are learned with a meta-training task set. In meta-testing, on the basis of the meta-training result, the policy and the predictive model adapt to a new task. For the adaptation, the trajectory observed from the beginning of the episode to the current time step is leveraged. As we noted earlier, we call an adaptive policy and a predictive model of this sort a meta-policy and a meta-model, respectively.
Partially observable Markov decision processes: We formalize our problem with a POMDP, which is defined as a tuple 〈O,S,A, pob, r, γ, pst〉. Here, O is a set of observations, S is a set of hidden states, A is a set of actions, pob := O × S × A → [0, 1] is the observation probability, pst := S × S × A → [0, 1] is the state transition probability, r : S × A → R is a reward function and γ ∈ [0, 1) is a discount factor. At time step t, these functions are used as pst(st|st−1, at−1), pob(ot|st, at−1) 2 and rt = r(st, at). The agent cannot directly observe the hidden state, but receives the observation instead. The agent selects an action on the basis of a policy π := p(at+1|ht). Here, ht is a history (the past trajectories) defined as ht := {a0, o0, ..., at, ot}. We denote the set of the histories by H. Given the definition of the history, the history transition probability can be defined as p(ht+1|at+1, ht) := p(ot+1|ht). Here, p(ot+1|ht) := ∑ st+1 ∑ st p(st|ht)p(st+1|st, at)p(ot+1|st+1, at), where p(st|ht) is the belief about the hidden state. The goal of RL in the POMDP is to find the optimal policy π∗ that maximizes the expected return R := ∑∞ t=0 γ
trt (i.e., π∗ = arg max π Ea∼π,h∼p [R]).

heading: 4 FORMULATING MODEL-BASED META-RL
text: In this section, we formulate model-based meta-RL as solving a POMDP by using a parameterized meta-policy and a meta-model. The outline of our formulation is shown in Figure 4 in the appendix.
In our formulation, the task is included in the hidden state: S := T × S ′. Here T is the set of task τ and S ′ is the set of the other hidden state factors s′. With this definition, the state transition probability, observation probability and reward function can be defined respectively as follows: p(st+1|st, at) = p(τt+1, s′t+1|τt, s′t, at), p(ot+1|st+1, at) = p(ot+1|τt+1, s′t+1, at) and r(st, at) = r(τt, s ′ t, at). In addition, as with Finn & Levine (2018); Finn et al. (2017); Rakelly et al. (2019), we assume that the task set T and the initial task distribution p(τ0) do not change in meta-training and meta-testing. Owing to this assumption, in our analysis and algorithm, metatraining and meta-testing can be seen as an identical one. Note that the task can be changed during an episode. Namely, the value of τt+1 is not necessarily equal to that of τt.
We define a meta-policy and meta-model as πφ(at+1|ht) and pθ(rt, ot+1|ht), respectively. Here, φ and θ are learnable parameters for them. rt and ot+1 are assumed to be conditionally independent given ht, i.e., pθ(rt, ot+1|ht) = pθ(rt|ht) · pθ(ot+1|ht). As with p(ht+1|at+1, ht), the meta-model for the history can be defined as pθ(ht+1|at+1, ht) := pθ(ot+1|ht). We use the parameterized meta-model and meta-policy as shown in Algorithm 1. This algorithm is composed of 1) data collection in the real environment and 2) optimization of the meta-policy and meta-model. In 1), the data is collected from the real environment with the meta-policy and stored into a dataset D. In 2), the meta-policy and meta-model are optimized to maximize Ea∼πφ,r,h∼pθ [R] − C( m(θ), π(φ)). Here, Ea∼πφ,r,h∼pθ [R] is a meta-model return (the return of
2For simplicity, we use these probabilities by abbreviating the subscripts “st” and “ob.”
Algorithm 1 Abstract Meta Model-based Meta-Policy Optimization (Abstract M3PO) 1: Initialize meta-policy πφ, meta-model pθ and dataset D. 2: for N epochs do 3: Collect trajectories from environment in accordance with πφ: D = D ∪ {(ht, ot+1, rt)}. 4: Optimize πφ and pθ: (φ, θ) ← arg max
(φ,θ)
Ea∼πφ,r,h∼pθ [R] − C( m(θ), π(φ)). Here, D is
used to evaluate m(θ) and π(φ) and generate an initial history h0. 5: end for
the meta-policy on the meta-model) 3. C( m(θ), π(φ)) is a discrepancy depending on the two error quantities m and π . Their detailed definitions are introduced in the next section.
Our POMDP-based formulation covers a wide range of meta-RL settings including Bayes-adaptive MDPs (Zintgraf et al., 2020), hidden parameter-MDPs (Killian et al., 2017; Perez et al., 2020), parameterized MDPs (Duan et al., 2017; Finn & Levine, 2018), and the settings considered by Nagabandi et al. (2019a;b) and Jabri et al. (2019). These settings can be primarily recovered in our formulation by introducing both or either of the following assumptions (Asm1 and Asm2) 4. Asm1: the task does not change during an episode. Asm2: observation o is identical to the hidden state s′: o = s′. The Bayes-adaptive MDPs, hidden-parameter MDPs and parameterized MDPs can be recovered by introducing both Asm1 and Asm2. The setting of Nagabandi et al. (2019a;b) can be recovered by introducing Asm2, and the setting of Jabri et al. (2019) can be recovered by introducing Asm1. As a detailed example, we recover the parameterized MDPs in Appendix A.5.

heading: 5 PERFORMANCE GUARANTEES OF MODEL-BASED META-RL
text: In this section, we analyse the performance guarantee of model-based meta-RL with an inaccurate meta-model. In Section 5.1, we provide the performance guarantee in a full meta-model-based rollout. In Section 5.2, we introduce the notion of a branched meta-rollout and analyse its performance guarantee. We show that the meta-model error is less harmful in the branched meta-rollout than that in the full meta-model-rollout.

heading: 5.1 PERFORMANCE GUARANTEE IN A FULL META-MODEL-BASED ROLLOUT CASE
text: Our goal is to outline a theoretical framework in which we can provide performance guarantees for Algorithm 1. To show the guarantees, we construct a lower bound taking the following form:
Eπφ,p [R] ≥ Eπφ,pθ [R]− C( m(θ), π(φ)). (1)
Here, Eπφ,p [R] denotes a true return (i.e., the return of the meta-policy in the real environment). The discrepancy between these returns, C, can be expressed as the function of two error quantities: the generalization error of the meta-policy and the distribution shift due to the updated meta-policy. For our analysis, we define the bounds of the generalization error m and the distribution shift π as follows: Definition 1. m(θ) := maxt Ea∼πD,h∼p [DTV (p(ht+1|at+1, ht)||pθ(ht+1|at+1, ht))]. Here, DTV is a total variation distance and πD is the data-collection policy whose actions contained in D follow 5. Definition 2. π(φ) := maxht DTV (πD(at+1|ht)||πφ(at+1|ht)).
We also assume that the expected reward is bounded by a constant rmax. Definition 3. rmax > maxt ∣∣∑ st p(st|ht)r(st, at) ∣∣. Now we present our bound, which is an extension of the theorem proposed in Janner et al. (2019).
3For simplicity, we use the abbreviated style Eπφ,pθ [R]. 4For simplicity, we omit implementation-level assumptions (e.g., “meta-policy or meta-model are implemented on the basis of gradient-based MAML” in Finn & Levine (2018); Nagabandi et al. (2019a)). 5As with Janner et al. (2019), to simplify our analysis, we assume that the meta-model can accurately estimate reward. We discuss the case in which the reward prediction of a meta-model is inaccurate in A.8.
Theorem 1 (The POMDP extension of Theorem 4.1. in Janner et al. (2019)). Let m = maxt Ea∼πD,h∼p [DTV (p(ht+1|at+1, ht)||pθ(ht+1|at+1, ht))] and π = maxht DTV (πD(at+1|ht)||πφ(at+1|ht)). Then, the true returns are bounded from below by meta-model returns of the meta-policy and discrepancy:
Eπφ,p [R] ≥ Eπφ,pθ [R]− rmax [ 2γ ( m + 2 π)
(1− γ)2 + 4 π (1− γ) ] ︸ ︷︷ ︸
C( m(θ), π(φ))
(2)
This theorem implies that the discrepancy of the returns under full meta-model-based rollout scales linearly with both m and π . If we can reduce the discrepancy C, the two returns are closer to each other. As a result, the performance degradation is more significantly suppressed. In the next section, we discuss a new meta-model usage to reduce the discrepancy induced by the meta-model error m.

heading: 5.2 PERFORMANCE GUARANTEE IN THE BRANCHED META-ROLLOUT CASE
text: The analysis of Theorem 1 relies on running full rollouts through the meta-model, causing metamodel errors to compound. This is reflected in the bound by a factor scaling quadratically with the effective horizon, 1/(1 − γ). In such cases, we can improve the algorithm by choosing to rely less on the meta-model and instead more on real environment data.
To allow for adjustment between meta-model-based and model-free rollouts, we introduce the notion of a branched meta-rollout. The branched meta-rollout is a kind of Dyna-style rollout (Sutton, 1991), in which the meta-model-based rollout is run as being branched from real environment data. More concretely, the rollout is run in the following two processes. 1) We begin a rollout from a history under the data-collection meta-policy’s history distribution pπD (ht), and 2) we then run k steps in accordance with πφ under the learned meta-model pθ.
Under such a scheme, the true return can be bounded from below: Theorem 2. Under the k steps branched meta-rollouts, using the bound of a meta-model error under πD, m = maxt Ea∼πD,h∼p,t [DTV (p(h′|h, a)||pθ(h′|h, a))], the bound of the meta-policy shift π = maxht DTV (πD||πφ), and the return on the meta-model E(a,h)∼Dmodel [R] where Dmodel is the set of samples collected through branched rollouts, the following inequation holds,
(3)Eπφ,p [R] ≥ E(a,h)∼Dmodel [R]− rmax { 1 + γ2 (1− γ)2 2 π + γ − kγk + (k − 1)γk+1 (1− γ)2 ( π + m)
+ γk − γ γ − 1 ( π + m) + γk 1− γ (k + 1)( π + m)
} .
The discrepancy factors relying on m in Theorem 2 can be smaller than those relying on m in Theorem 1 6. This indicates that the performance degradation due to the meta-model error can be more suppressed than that in the full meta-model-based rollout 7.

heading: 6 META-MODEL-BASED META-POLICY OPTIMIZATION WITH DEEP RL
text: In the previous section, we show that the use of branched meta-rollouts can suppress performance degradation. In this section, on the basis of this result, we modify Algorithm 1 so that the metapolicy and meta-model are optimized with E(a,h)∼Dmodel [R] − C( m(θ), π(φ)), instead of with Eπφ,pθ [R]− C( m(θ), π(φ)). More specifically, we propose the following modifications to Algorithm 1:
Meta-policy optimization: The meta-policy is optimized with the branched meta-rollouts return E(a,h)∼Dmodel [R] 8. For the optimization, we adapt PEARL (Rakelly et al., 2019) be-
6See Corollary 1 in the appendix. 7In A.6, we also prove that the discrepancy can be further reduced by introducing an additional assumption. 8To stabilize the learning, we omit C( m(θ), π(φ)) from the optimization objective for the meta-policy. The transition of π(θ) during learning with this limited optimization objective is shown in Figure 12 in the appendix. The result indicates that π(θ) tends to decrease as the training data size (epoch) grows.
Algorithm 2 Meta-Model-based Meta-Policy Optimization with Deep RL (M3PO) 1: Initialize meta-policy πφ, meta-model pθ, environment datasetDenv, meta-model datasetDmodel.
2: for N epochs do 3: Train meta-model pθ with Denv: θ ← arg max
θ EDenv [pθ(rt, ot+1|ht)]
4: for E steps do 5: Take actions according to πφ; add the trajectory to Denv 6: for M model rollouts do 7: Sample ht uniformly from Denv 8: Perform k-step meta-model rollouts starting from ht using meta-policy πφ; add fictitious trajectories to Dmodel 9: end for
10: for G gradient updates do 11: Update policy parameters with Dmodel: φ← φ−∇φJDmodel(φ) 12: end for 13: end for 14: end for
cause it achieved a good learning performance in meta-learning settings. We use the fictitious trajectories generated from the branched meta-rollouts to optimize the meta-policy. Formally, πφ is optimized by using the gradient of optimization objective JDmodel(φ) := E(a,h)∼Dmodel [ DKL ( πφ||exp ( Qπφ − Vπφ ))] . Here, DKL is the Kullback-Leibler divergence,
Qπφ(at+1, ht) := E(r,h)∼Dmodel [R|at+1, ht] and Vπφ(ht) := ∑ at+1
Qπφ(at+1, ht)πφ(at+1|ht). As in PEARL, in πφ, the latent context is estimated by using past trajectories. The estimated context is then used to augment the policy input. Formally, the meta-policy is implemented as πφ(at+1|ht) = ∑ z πφ(at+1|ot, z)pφ(z|a0, o0, ..., at, ot). z is the latent context and pφ(z|a0, o0, ..., at, ot) is a context encoder. πφ(at+1|ot, z) is the conditioned policy. Similarly, in Qπφ and Vπφ , the estimated latent context is also used to augment their input. Meta-model optimization: The meta-model is optimized to minimize the discrepancy (i.e., minimize m) 9. For the meta-model, to consider both aleatoric and epistemic uncertainties, we use a bootstrap ensemble of B dynamics models {p1θ, ..., pBθ }. Here, piθ is the i-th conditional Gaussian distribution with diagonal covariance: piθ(rt, ot+1|ht) = N ( rt+1, ot+1|µiθ (ht) , σiθ (ht) ) . µiθ and σiθ are the mean and standard deviation, respectively. In our implementation, we use the recurrentbased architecture inspired by Duan et al. (2017); Nagabandi et al. (2019b); Rakelly et al. (2019); at each evaluation of the model, {a1, o1, ..., at−1, ot−1} in ht is fed to a recurrent neural network (RNN), and then its hidden unit output and {at, ot} in ht are fed to the feed-forward neural network that outputs the mean and standard deviation of the Gaussian distribution. We use the gated recurrent unit (GRU) (Cho et al., 2014) for the RNN. The recurrent layer in the GRU is composed of five sigmoid units. In addition, the feed-forward neural network is composed of an input layer, two hidden layers, and an output layer. The input and hidden layers are composed of 400 swish units (Ramachandran et al., 2017). To minimize m, we learn the meta-model via maximum likelihood estimation with Denv. As in Lakshminarayanan et al. (2017), the ensemble {p1θ, ..., pBθ } is learned on the shuffle of the real trajectories inDenv. We apply a re-parameterization trick to evaluate the distributions, and the meta-model parameters θ are optimized with a gradient-based optimizer. For the gradient-based optimizer we use Adam (Kingma & Ba, 2015). To avoid overfitting, we use weight decay and early termination (Bishop, 2006).
The resulting algorithm is shown in Algorithm 2. The modifications “Meta-model optimization” and “Meta-policy optimization” in the above paragraph are reflected in line 3 and lines 4–13, respectively. In line 4, k-step branched meta-rollouts are run. The appropriately set k contributes to decreasing the discrepancy, and suppresses performance degradation. Thus, we treat k as a hyperparameter, and set it to different values in different environments so that the discrepancy decreases. For the experiments described in the next section, we tune the hyperparameters for this algorithm by a grid search. The search result for the hyperparameter values is described in Table 1 in the appendix.
9The transition of m over training epochs is shown in Figure 11 in the appendix, and it indicates that the model error tends to decrease as the number of epochs increases.
For our experiments, we implemented our algorithm by extending the codebase of Model-based Policy Optimization 10. We made two main extensions: (1) introduce a latent context to the policy as in PEARL and (2) replace a predictive model with the meta-model based on the RNN.

heading: 7 EXPERIMENTS
text: In this section, we report our experiments 11 aiming to answer the following questions: Q.1: Can our method (M3PO) outperform existing meta-RL methods? Q.2: How do meta-model-rollout lengths k affect the actual performance?
In our experiments, we compare our method (M3PO) with two baseline methods: PEARL (Rakelly et al., 2019) and Learning to adapt (L2A) (Nagabandi et al., 2019a). More detailed information is described in A.10 in the appendix. We conduct a comparative evaluation of the methods on a variety of simulated robot environments using the MuJoCo physics engine (Todorov et al., 2012). We prepare the environments proposed in the meta-RL (Finn & Levine, 2018; Nagabandi et al., 2019a; Rakelly et al., 2019; Rothfuss et al., 2019) and robust-RL (Hiraoka et al., 2019; Rajeswaran et al., 2017) literature: Halfcheetah-fwd-bwd, Halfcheetah-pier, Ant-fwd-bwd, Ant-crippledleg, Walker2D-randomparams and Humanoid-direc. In the environments, the agent is required to adapt to a fluidly changing task that the agent cannot directly observe. Detailed information about each environment is described in A.11 in the appendix.
Regarding Q1, our experimental results indicate that M3PO outperforms existing meta-RL methods. In Figure 1, the learning curves of M3PO and existing meta-RL methods (L2A and PEARL) on meta-training phases are shown. These learning curves indicate that the sample efficiency of M3PO is better than those of L2A and PEARL 12. The performance (return) of L2A remains poor and does not improve even when the training data increases. PEARL can improve meta-policy performance via training in all environments. However, the degree of improvement of PEARL is smaller than that of M3PO. In a number of the environments (e.g., Halfcheetah-pier), the relative performance of M3PO against PEARL becomes asymptotically worse. This indicates that, as with Nagabandi et al. (2018), dynamic switching from M3PO to PEARL or other model-free approaches needs to be considered to further improve overall performance.
Regarding Q2, we conducted an evaluation of M3PO by varying its model-rollout length k. The evaluation results (Figure 2) indicate that the performance tends to degrade when the model-rollout length is long. We can see significant performance degradation especially in Ant-fwd-bwd and Humanoid-direc. In Ant-fwd-bwd, the performance at k = 100 is significantly worse than that at k = 10. In Humanoid-direc, the performance at k = 5 is significantly worse than that at k = 1. As we have seen, the performance degradation in Humanoid-direc is more sensitive to the model-rollout length than that in Ant-fwd-bwd. One reason for this is that the meta-model error in Humanoid-direc is larger than that in Ant-fwd-bwd (Figure 11 in the appendix).
An example of meta-policies learned by M3PO with 200k samples in Humanoid-direc is shown in Figure 3, and it indicates the learned policy successfully adapts to different tasks. Additional examples of meta-policy learned by PEARL and M3PO are shown in the video at the following link: https://drive.google.com/file/d/1DRA-pmIWnHGNv5G_ gFrml8YzKCtMcGnu/view?usp=sharing

heading: 8 CONCLUSION
text: In this paper, we analysed the performance guarantee (and performance degradation) of MBRL in a meta-learning setting. We first formulated model-based reinforcement learning in a meta-learning setting as solving a POMDP. We then conducted theoretical analyses on the performance guarantee in both the full model-based rollout and the branched meta-rollout. We showed that the performance degradation due to the meta-model error in the branched meta-rollout is smaller than that in the full
10https://github.com/JannerM/mbpo 11The source code to replicate the experiments will be open to the public. 12Note that, in an early stage of the training phase, there are many test episodes in which unseen tasks appear. Therefore, the improvement of M3PO over L2A and PEARL at the early stage of learning indicates its high adaptation capability for unseen situations.
meta-model-based rollouts. Based on the theoretical result, we introduced branched meta-rollouts to policy optimization and proposed M3PO. Our experimental results show that it achieves better sample efficiency than PEARL and L2A.

heading: A APPENDICES

heading: A.1 HOW DOES OUR WORK DIFFER FROM JANNER ET AL. (2019)?
text: Although our work is grounded primarily on the basis of Janner et al. (2019), we provide non-trivial contributions in both theoretical and practical frontiers: (1) We provide theorems about the relation between true returns and returns on inaccurate predictive models (model returns) on a “meta-learning (POMDPs)” setting (Section 5). In their work, they provide theorems about the relation between the true returns and the model returns in the branched rollout in MDPs. In contrast, we provide theorems about the relation between the true returns and of the model returns in the branched rollout in POMDPs. In addition, in the derivation of theorems (Theorems 4.2 and 4.3) in their work, a number of important premises are not properly taken into consideration (the detailed discussion on it is described in the second paragraph in A.7). We provide new theorems, in which the premises are more properly reflected, for both MDPs and POMDPs (A.3, A.6, and A.7). (2) We extend the modelbased policy optimization (MBPO) proposed by Janner et al. into the meta-learning (POMDPs) setting (Section 6). MBPO is for the MDP settings and does not support POMDP settings, while our method (M3PO) supports POMDP settings. Furthermore, we empirically demonstrate the usefulness of the meta-model usage in the branched rollout manner in the POMDP settings (Section 7).
A.2 OUTLINE OF OUR MODEL-BASED META-RL FORMULATION
14

heading: A.3 PROOFS OF THEOREMS IN THE MAIN CONTENT
text: Before starting the derivation of the main theorems, we introduce a lemma useful for bridging POMDPs and MDPs. Lemma 1 (Silver & Veness (2010)). Given a POMDP 〈O,S,A, pob, r, γ, pst〉, consider the derived MDP with histories as states, 〈H,A, γ, r̄, phi〉, where ∀t. phi := p(ht+1|at+1, ht) =∑ st∈S ∑ st+1∈S p(st|ht)p(st+1|st, at)p(ot+1|st+1, at) and r̄(ht, at) := ∑ st∈S p(st|ht)r(st, at). Then, the value function V̄π(ht) of the derived MDP is equal to the value function Vπ(ht) of the POMDP.
Proof. The statement can be derived by backward induction on the value functions. See the proof of Lemma 1 in Silver & Veness (2010) for details.

heading: PROOF OF THEOREM 1:
text: Proof. By Lemma 1, our problem in POMDPs can be mapped into the problem in MDPs, and then Theorem 4.1 in Janner et al. (2019) can be applied to the problem.
Similarly, the proof of Theorem 2 is derived by mapping our problem into that in MDPs by Lemma 1 and leveraging theoretical results on MDPs.

heading: PROOF OF THEOREM 2:
text: Proof. By Lemma 1, our problem in POMDPs can be mapped into the problem in MDPs, and then Theorem 4 in A.7 can be applied to the problem.

heading: A.4 THE DISCREPANCY RELYING ON THE META-MODEL ERROR IN THEOREM 1 AND THAT IN
text: THEOREM 2
Corollary 1. The discrepancy factors relying on m in Theorem 1, CTh1,m, are equal to or larger than those relying on m at k = 1 in Theorem 2, CTh2,m.
Proof. By Theorem 1 and 2,
(4)CTh1,m = rmax 2γ m
(1− γ)2 .
(5)CTh2,m = rmax γ
1− γ 2 m.
Given that γ ∈ [0, 1), rmax > 0 and m ≥ 0,
(6)CTh1,m − CTh2,m = rmax 4γ m − 2γ2 m
(1− γ)2 ≥ 0.

heading: A.5 CONNECTION TO A TYPICAL META-RL SETTING
text: In Section 4, we formulate model-based meta-RL to solve a POMDP. However, this formulation may make it difficult for certain readers to comprehend the connection to a typical meta-RL setting. Although in the normative meta-RL setting (the parameterized MDPs (Finn et al., 2017)), the objective to be optimized is given as the return expected with respect to a task distribution, such an objective does not appear in the formulation in Section 4. In this section, we show that such an objective can be derived by specializing the formulation in Section 4 under a number of assumptions
(Corollary 2). Then, we explain why we did not adopt such a specialization and maintained a more abstract formulation in Section 4.
First, letting a task set and a task distribution be denoted by T and p(τ) where τ ∈ T , respectively, we introduce the following assumptions: Assumption 1. S := O × T . Assumption 2. p(st+1|st, at) := p(ot+1|ot, τt, at) · 1(τt+1 = τt). Assumption 3. For t > 0, p(st|ht) := p(τt|ht) · 1(τt = τ0). Assumption 4. p(τ0|h0) := p(τ0).
Here, 1(·) is the indicator function that returns one if the argument is true, and zero otherwise. With these assumptions, the following corollary holds: Corollary 2. Given a POMDP 〈O,S,A, pob, r, γ, pst〉 and a task set T , consider the parameterized MDP with histories as states, 〈H,A, γ, r̄, p̄ob〉, where ∀t. p̄ob := p(ot+1|ot, τ0, at) and r̄ := r(ot, τ0, at). Under Assumptions 2∼5, the expected return on the parameterized MDP Ea∼π,h∼p,τ∼p(τ) [ ∑∞ t γ tr̄t] := ∑ τ∈T p(τ)Ea∼π,h∼p [ ∑∞ t γ
tr̄t|τ ] is equal to the expected return on the POMDP Ea∼π,h∼p [R].
Proof. By applying Lemma 1, the value function in a POMDP 〈O,S,A, pob, r, γ, pst〉 can be mapped to the value function V̄π(ht) in the derived MDP, which is 〈H,A, γ, r̄, phi〉, where ∀t. phi := p(ht+1|at+1, ht) = ∑ st∈S ∑ st+1∈S p(st|ht)p(st+1|st, at)p(ot+1|st+1, at) and r̄(ht, at) :=∑
st∈S p(st|ht)r(st, at). By applying the assumptions, this value function can be transformed to a different representation that explicitly contains τ and its distribution: For t > 0,
(7)
V̄π(ht) = ∑ at+1 π(at+1|ht) ∑ st∈S p(st|ht)r(st, at)
+ γ ∑
ot+1∈O ∑ st∈S ∑ st+1∈S p(st|ht)p(st+1|st, at)p(ot+1|st+1, at)V̄π(ht+1)  = ∑ at+1 π(at+1|ht) r(ot, τ0, at)︸ ︷︷ ︸ r̄t +γ ∑ ot+1∈O p(ot+1|ot, τ0, at)︸ ︷︷ ︸ p̄ob V̄π(ht+1)
 . Likewise, for t = 0,
(8)
V̄π(h0) = ∑ a1 π(a1|h0) {∑ τ0 p(τ0)r(o0, τ0, a0) + γ ∑ o1∈O ∑ τ0 p(τ0)p(o1|o0, τ0, a0)V̄π(h1) }
= ∑ τ0 p(τ0) ∑ a1 π(a1|h0)
{ r(o0, τ0, a0) + γ
∑ o1∈O p(o1|o0, τ0, a0)V̄π(h1) } ︸ ︷︷ ︸
V̄π(h0)
.
Therefore,
(9)
Ea ∼π,h∼p [R] = ∑ h0 p(h0)V̄π(h0)
= ∑ τ0 p(τ0) ∑ h0 p(h0)V̄π(h0)
= Ea∼π,h∼p,τ∼p(τ) [ ∞∑ t γtr̄t ]
vertical axis represents k ∈ [1, 50] and each horizontal axis represents m, m′ ∈ [0, 1]. In all figures, for evaluating the discrepancy values, we set the other variables as rmax = 1, π = 1 − m for (a), and π = 1− m′ for (b). A key insight with the figures is following two points. First, discrepancy values in Theorem 2 (a) tend to increase as the k value increases, whereas those in Theorem 3 (b) are not. Thus, there could be an optimal k value that larger than one. Second, the contour colours in (b) are paler than those in (a). This implies, given m = m′ , Theorem 3 provides a tighter bound of the performance.
By Corollary 2, our formulation in Section 4 can be specialized into a problem where the objective to be optimized is given as the return expected with respect to a task distribution. We can derive the meta-model returns with discrepancies for bounding the true return (i.e., Ea∼πφ,h∼pθ,τ∼p(τ) [ ∑∞ t γ
tr̄t]−C( m, π)) by using Corollary 2 instead of Lemma 1 in the proofs of Theorem 1 and replacing pθ(ot|o0, τ0, at) with pθ(ot|ht−1). The main reason that we do not adopt such a specialization in Section 4 is to avoid restrictions induced by the assumptions (Assumption 2∼5). For example, Assumption 2 states that hidden states are composed by observations and a task. Since observations can be observed from an agent by its definition, the information in the hidden states that cannot be observed from an agent is the task. However, in many meta-RL settings (e.g., application domains (Jabri et al., 2019) where video images are used as the observation), there should be other information that cannot be observed from the agent. In addition, Assumptions 3 and 4 state that the task is invariant in each episode. However, in many meta-RL settings (e.g., Nagabandi et al. (2019a;b)), the task can change to a different one in each episode. To avoid such restrictions, we decided not to specialize the formulation by introducing the assumptions.

heading: A.6 ADDITIONAL ANALYSIS OF THE PERFORMANCE DEGRADATION IN THE BRANCHED META-ROLLOUT
text: Figure 5a shows that the discrepancy value in Theorem 2 tends to monotonically increase as the value of k increases, regardless of the values of γ and m. This means that the optimal value of k is always 1. However, intuitively, we may expect that there is an optimal value for k that is higher than 1 when the meta-model error is small. As will be shown in this section, this intuition is correct. One of the main reasons for the mismatch of the intuition and the tendency of the discrepancy in Theorem 2 is that, for its analysis, the meta-model error on the data collection meta-policy πD (i.e., m) is used instead of that on the current meta-policy πφ. This ignorance of the meta-model error on the current meta-policy induces the pessimistic estimation of the discrepancy in the analysis (see the evaluation of “term B” and “term C” in the proof of Theorem 4 via the proof of Theorem 2 in the appendix for more details).
To estimate the discrepancy more tightly, as in Janner et al. (2019), we introduce the assumption that the meta-model error on the current meta-policy can be approximated by π and m: Assumption 5. An approximated meta-model error on a current policy m′ : m′( π) ≈ m + π
d m′ d π , where d m′d π is the local change of m′ with respect to π .
To see the tendency of this approximated meta-model error, we plot the empirical value of d m′/d π , varying the size of training samples for the meta-model in Figure 10 in A.12. The figure shows that, as the training sample size increases, the value tends to gradually approach zero. This means that training meta-models with more samples provide better generalization on the nearby distributions.
Equipped with the approximated meta-model’s error on the distribution of the current meta-policy πφ, we arrive at the following bound: Theorem 3. Let m′ ≥ maxt Ea∼πφ,h∼p [DTV (p(h′|h, a)||pθ(h′|h, a))],
(10)Eπφ,p [R] ≥ E(a,h)∼Dmodel [R]− rmax { 1 + γ2 (1− γ)2 2 π + γ − kγk + (k − 1)γk+1 (1− γ)2 ( m′ − π)
+ γk − γ γ − 1 ( m′ − π) + γk 1− γ (k + 1)( m′ − π)
} .
Proof. By Lemma 1, our problem in POMDPs can be mapped into the problem in MDPs, and then Theorem 5 in A.7 can be applied to the problem.
Given that m = m′ , it is obvious that the discrepancy in Theorem 3 is equal to or smaller than that in Theorem 2. In the discrepancy in Theorem 3, all terms except the first term become negative when m′ < π . This implies that the optimal k that minimizes the discrepancy can take on the value higher than 1 when the meta-model error is relatively small. The empirical trend of the discrepancy value (Figure 5b) supports it; when m′ is lower than 0.5 (i.e., m′ < π), the discrepancy values decrease as the value of k grows regardless of the value of γ. This result motivates us to set k to the value higher than 1 in accordance with the meta-model error for reducing the discrepancy.
A.7 THE DERIVATION OF THE RELATION OF THE RETURNS IN k-STEP BRANCHED ROLLOUTS (k ≥ 1) IN MARKOV DECISION PROCESSES
In this section, we discuss the relation of the true returns and the model returns under the branched rollout in an MDP, which is defined by a tuple 〈S,A, r, γ, pst〉. Here, S is the set of states, A is the set of actions, pst := p(s′|s, a) is the state transition probability for any s′, s ∈ S and a ∈ A, r is the reward function and γ ∈ [0, 1) is the discount factor. At time step t, the former two functions are used as p(st|st−1, at), rt = r(st, at). The agent selects an action on the basis of a policy π := p(at+1|st). We denote the data collection policy by πD and the state visitation probability under πD and p(s′|s, a) by pπD (st). We also denote the predictive model for the next state by pθ(s
′|s, a). In addition, we define the upper bounds of the reward scale as rmax > maxs,a|r(s, a)|. Note that, in this section, to discuss the MDP case, we are overriding the definition of the variables and functions that were defined for the POMDP case in the main body. In addition, for simplicity, we use the abbreviated style Eπ,p [R] for the true return Ea∼π,s∼p [R := ∑∞ t=0 γ trt].
Although the theoretical analysis on the relation of the returns in the MDP case is provided by Janner et al. (2019), in their analysis, a number of important premises are not properly taken into consideration. First, although they use the replay buffers for the branched rollout (i.e. datasets Denv and Dmodel in Algorithm 2 in Janner et al. (2019)), they do not take the use of the replay buffers into account in the their theoretical analysis. Furthermore, they calculate state-action visitation probabilities based solely on a single model-based rollout factor. In the branched rollout, stateaction visitation probabilities (except for the one at t = 0) should be affected by multiple past model-based rollouts. For example, a state-action visitation probability at t (s.t. t > k) is affected by the model-based rollout branched from real trajectories at t− k and ones from t− k+ 1 to t− 1 (in total, k model-based rollouts). However, in their analysis (the proof of Lemma B.4 in Janner et al. (2019)), they calculate state-action visitation probabilities based solely on the model-based rollout. For example, in their analysis, it is assumed that a state-action visitation probability at t (s.t. t > k) is affected only by the model-based rollout branched from real trajectories at t − k. These oversights of important premises in their analysis induce a large mismatch between those for their theorems and those made for the actual implementation of the branched rollout (i.e., Algorithm 2 in
Janner et al. (2019)) 13. Therefore, we decided to newly derive the theorems on the branched rollout, reflecting these premises more appropriately.
The outline of our branched rollout is shown in Figure 6. Here, we assume that the trajectories collected from the real environment are stored in a dataset Denv. The trajectories stored in Denv can be seen as trajectories following the true dynamics p(s′|s, a) and data collection policy (i.e., a mixture of previous policies used for data collection) πD. At each branched rollout, the trajectories in Denv are uniformly sampled 14, and then starting from the sampled trajectories, k-step modelbased rollouts in accordance with π under pθ is run. The fictitious trajectories generated by the branched rollout is stored in a model dataset Dmodel 15. This process more appropriately reflects the actual implementation of the branched rollout (i.e., lines 5–8 in Algorithm 2) in Janner et al. (2019) 16. The performance of π is evaluated as the expected return under the state-action visitation probability in Dmodel.
13The mismatch becomes larger especially when the model-rollout length k is large because state-action visitation probabilities are affected by these rollouts more significantly with large k.
14Thus, the initial state probability for the rollout starting from the trajectories follows pπD (s) 15Here, when the trajectories are stored in Dmodel, the states in the trajectories are augmented with time step information to deal with the state transition depending on the time step. 16Note that the extension of this process to the POMDP case is compatible with the implementation of the branched meta-rollout in our algorithm (lines 4–13 in Algorithm 2).
Formally, we define the return in the branched rollout E(a,s)∼Dmodel [R] as:
E(a,s)∼Dmodel [R] := ∑ s0,a0 pπD (s0, a0)r(s0, a0) + k−1∑ t=1 ∑ st,at γtpbrt<k(st, at)r(st, at)
+ ∞∑ t=k ∑ st,at γtpbrt≥k(st, at)r(st, at) (11)
pbrt<k(st, at) := 1
t t−1∑ i=0 pbrt<k,i(st, at) (12)
pbrt≥k(st, at) := 1
k k−1∑ i=0 pbrt≥k,i(st, at) (13)
pbrt<k,i(st, at) := ∑
si,...,st−1 ∑ ai,...,at−1 pπD (si)Π t−1 j=ipθ(sj+1|sj , aj+1)π(aj+1|sj) (14)
pbrt≥k,i(st, at) := ∑
st−k+i,...,st−1 ∑ at−k+i,...,at−1
pπD (st−k+i)Π t−1 j=t−k+ipθ(sj+1|sj , aj+1)π(aj+1|sj) (15)
Here, pbrt<k,i(st, at) and p br t≥k,i(st, at) are the state-action visitation probabilities that the i-th yellow trajectories (node) from the bottom at each timestep t in Figure 6 follow. In later discussions, for simplicity, we use the abbreviated style EDmodel [R] for E(a,s)∼Dmodel [R].
Before starting the derivation of theorems, we introduce a useful lemma.
Lemma 2. Assume that the rollout process in which the policy and dynamics can be switched to other ones at time step tsw. Letting two probabilities be p1 and p2, for 1 ≤ t′ ≤ tsw, we assume that the dynamics distributions are bounded as m,pre ≥ maxt′ Es∼p1 [DTV (p1(st′ |st′−1, at′)||p2(st′ |st′−1, at′))]. In addition, for tsw < t
′ ≤ t, we assume that the dynamics distributions are bounded as m,post ≥ maxt′ Es∼p1 [DTV (p1(st′ |st′−1, at′)||p2(st′ |st′−1, at′))]. Likewise, the policy divergence is bounded by π,pre and π,post. Then, the following inequation holds
(16) ∑ st,at |p1(st, at)− p2(st, at)| ≤ 2(t− tsw)( m,post + π,post) + 2tsw( m,pre + π,pre)
Proof. The proof is done in a similar manner to those of Lemma B.1 and B.2 in (Janner et al., 2019).
∑ st,at |p1(st, at)− p2(st, at)|
= ∑ st,at |p1(at)p1(st|at)− p2(at)p2(st|at)|
= ∑ st,at |p1(at)p1(st|at)− p1(at)p2(st|at) + (p1(at)− p2(at))p2(st|at)|
≤ ∑ st,at p1(at) |p1(st|at)− p2(st|at)|+ ∑ at |p1(at)− p2(at)|
≤ ∑ st,at p1(at) |p1(st|at)− p2(st|at)|+ ∑ at,st−1 |p1(at, st−1)− p2(at, st−1)|
= ∑ st,at p1(at) |p1(st|at)− p2(st|at)|
+ ∑
at,st−1
|p1(st−1)p1(at|st−1)− p1(st−1)p2(at|st−1) + (p1(st−1)− p2(st−1))p2(at|st−1)|
≤ ∑ st,at p1(at) |p1(st|at)− p2(st|at)|+ ∑ at,st−1 p1(st−1) |p1(at|st−1)− p2(at|st−1)|
+ ∑ st−1 |p1(st−1)− p2(st−1)|
≤ ∑ st,at p1(at) |p1(st|at)− p2(st|at)|+ ∑ at,st−1 p1(st−1) |p1(at|st−1)− p2(at|st−1)|
+ ∑
st−1,at−1
|p1(st−1, at−1)− p2(st−1, st−1)|
≤ 2 m,post + 2 π,post + ∑
st−1,at−1
|p1(st−1, at−1)− p2(st−1, st−1)|
≤ 2(t− tsw)( m,post + π,post) + ∑
stsw ,atsw
|p1(stsw , atsw)− p2(stsw , stsw)|
≤ 2(t− tsw)( m,post + π,post) + 2tsw( m,pre + π,pre)
(17)
Now, we start the derivation of our bounds.
Theorem 4. Under the k-step branched rollouts, using the bound of a model error under πD, m = maxt Ea∼πD,s∼p,t [DTV (p(s′|s, a)||pθ(s′|s, a))] and the bound of the policy shift π = maxsDTV (π||πD), the following inequation holds,
(18)Eπ,p [R] ≥ EDmodel [R]− rmax { 1 + γ2 (1− γ)2 2 π + γ − kγk + (k − 1)γk+1 (1− γ)2 ( π + m)
+ γk − γ γ − 1 ( π + m) + γk 1− γ (k + 1)( π + m)
} .

heading: Proof.
text: ∣∣Eπ,p [R]− EDmodel [R]∣∣ = ∣∣∣∣∣∣∣
∑ s0,a0
{pπ(s0, a0)− pπD (s0, a0)} r(s0, a0) + ∑k−1 t=1 ∑ st,at γt { pπ(st, at)− pbrt<k(st, at) } r(st, at)
+ ∑∞ t=k ∑ st,at γt { pπ(st, at)− pbrt≥k(st, at) } r(st, at) ∣∣∣∣∣∣∣ ≤  ∑ s0,a0 |pπ(s0, a0)− pπD (s0, a0)| |r(s0, a0)| + ∑k−1 t=1 γ t ∑ st,at ∣∣pπ(st, at)− pbrt<k(st, at)∣∣ |r(st, at)| + ∑∞ t=k γ t ∑ st,at ∣∣∣pπ(st, at)− pbrt≥k(st, at)∣∣∣ |r(st, at)| 
≤ 
rmax ∑ s0,a0
|pπ(s0, a0)− pπD (s0, a0)|︸ ︷︷ ︸ term A
+rmax ∑k−1 t=1 γ t ∑ st,at ∣∣pπ(st, at)− pbrt<k(st, at)∣∣︸ ︷︷ ︸ term B
+rmax ∑∞ t=k γ t ∑ st,at ∣∣pπ(st, at)− pbrt≥k(st, at)∣∣︸ ︷︷ ︸ term C  (19)
For term A, we can bound the value in similar manner to the derivation of Lemma 2.∑ s0,a0 |pπ(s0, a0)− pπD (s0, a0)| = ∑ s0,a0 |pπ(a0)p(s0)− pπD (a0)p(s0)|
= ∑ s0,a0 |pπ(a0)p(s0)− pπ(a0)p(s0) + (pπ(a0)− pπD (a0)) p(s0)|
≤ ∑ s0,a0
pπ(a0) |p(s0)− p(s0)|︸ ︷︷ ︸ =0
+ ∑ a0
|pπ(a0)− pπD (a0)|︸ ︷︷ ︸ ≤2 π
≤ 2 π (20)
For term B, we can apply Lemma 2 to bound the value, but it requires the bounded model error under the current policy π. Thus, we need to decompose the distance into two by adding and subtracting pπD : ∑
st,at |pπ(st, at)− pbr,t<k(st, at)| = ∑ st,at ∣∣∣∣ pπ(st, at)− pπD (st, at)+pπD (st, at)− pbrt<k(st, at) ∣∣∣∣
≤ ∑ st,at
|pπ(st, at)− pπD (st, at)|︸ ︷︷ ︸ ≤2t π
+ ∑ st,at ∣∣pπD (st, at)− pbrt<k(st, at)∣∣ (21) ∑ st,at ∣∣pπD (st, at)− pbrt<k(st, at)∣∣ = ∑ st,at
∣∣∣ 1t ∑t−1i=0 pπD (st, at)− 1t ∑t−1i=0 pbrt<k,i(st, at)∣∣∣ ≤ 1
t t−1∑ i=0 ∑ st,at ∣∣pπD (st, at)− pbrt<k,i(st, at)∣∣ (A)
≤ 1 t t−1∑ i=0 {2 (t− i) · ( π + m)}
= 1
t
{ t2( π + m) + t( π + m) } (22)
For (A), we apply Lemma 2 with setting m,post = m and π,post = π for the rollout following π and pθ(s′|s), and m,pre = 0 and π,pre = 0 for the rollout following πD and p(s′|s), respectively. To recap term B, the following equation holds:
∑ st,at |pπ(st, at)− pbr,t<k(st, at)| ≤ 2t π + t( π + m) + ( π + m) (23)
For term C, we can derive the bound in a similar manner to the term B case:
∑ st,at |pπ(st, at)− pbr,t≥k(st, at)| = ∑ st,at ∣∣∣∣ pπ(st, at)− pπD (st, at)+pπD (st, at)− pbrt≥k(st, at) ∣∣∣∣
≤ ∑ st,at
|pπ(st, at)− pπD (st, at)|︸ ︷︷ ︸ ≤2t π
+ ∑ st,at ∣∣pπD (st, at)− pbrt≥k(st, at)∣∣ (24)
∑ st,at ∣∣pπD (st, at)− pbrt≥k(st, at)∣∣ = ∑ st,at ∣∣∣ 1k∑k−1i=0 pπD (st, at)− 1k∑k−1i=0 pbrt≥k,i(st, at)∣∣∣ ≤ 1
k k−1∑ i=0 ∑ st,at ∣∣pπD (st, at)− pbrt≥k,i(st, at)∣∣ ≤ 1
k k−1∑ i=0 {2 (k − i) · ( π + m)}
= 1
k
{ k2( π + m) + k( π + m) } (25)
To recap term C, the following equation holds:
∑ st,at ∣∣pπ(st, at)− pbrt≥k(st, at)∣∣ ≤ 2t π + k( π + m) + ( π + m) (26)
By substituting Eqs. 20, 23, and 26, into Eq. 19, we obtain the result:∣∣Eπ,p [R]− EDmodel [R]∣∣ ≤  rmax2 π +rmax ∑k−1 t=1 γ
t {2t π + t( π + m) + ( π + m)} +rmax ∑∞ t=k γ t {2t π + k( π + m) + ( π + m)}  = rmax { 2 π + 1−kγ(k−1)+(k−1)γk (1−γ)2 γ (3 π + m) + γk−γ γ−1 ( π + m)
+ ∑∞ t=k γ t {2t π + k( π + m) + ( π + m)}
}
= rmax  2 π + 1−kγ(k−1)+(k−1)γk (1−γ)2 γ (3 π + m) + γk−γ γ−1 ( π + m) + ∑∞ t=1 γ
t {2t π + k( π + m) + ( π + m)} − ∑k−1 t=1 γ t {2t π + k( π + m) + ( π + m)}  = rmax  2 π + 1−kγ(k−1)+(k−1)γk (1−γ)2 γ (3 π + m) + γk−γ γ−1 ( π + m) + 2(1−γ)2 γ π + γ
1−γ {k( π + m) + ( π + m)} − ∑k−1 t=1 γ t {2t π + k( π + m) + ( π + m)}

= rmax  2 π + 1−kγ(k−1)+(k−1)γk (1−γ)2 γ (3 π + m) + γk−γ γ−1 ( π + m) + 2(1−γ)2 γ π + γ 1−γ {k( π + m) + ( π + m)} − 1−kγ (k−1)+(k−1)γk (1−γ)2 2γ π
−γ k−γ γ−1 {k( π + m) + ( π + m)}  = rmax  1+γ2 (1−γ)2 2 π + γ−kγk+(k−1)γk+1 (1−γ)2 ( π + m)
+γ k−γ γ−1 ( π + m) +
( γ
1−γ − γk−γ γ−1 ) (k + 1)( π + m)  = rmax { 1+γ2 (1−γ)2 2 π + γ−kγk+(k−1)γk+1 (1−γ)2 ( π + m)
+γ k−γ γ−1 ( π + m) + γk 1−γ (k + 1)( π + m)
} (27)
Theorem 5. Let m′ ≥ maxtEa∼π,s∼p [DTV (p(s′|s, a)||pθ(s′|s, a))],
(28)Eπ,p [R] ≥ EDmodel [R]− rmax { 1 + γ2 (1− γ)2 2 π + 1− kγ(k−1) + (k − 1)γk (1− γ)2 γ( m′ − π)
+ γk − γ γ − 1 ( m′ − π) + γk 1− γ (k + 1)( m′ − π)
} .
Proof. The derivation of this theorem is basically the same as that in the Theorem 4 case except for the way of evaluation of the bound of terms B and C.
For term B, we can apply Lemma 2 to bound the value:∑ st,at ∣∣pπ(st, at)− pbrt<k(st, at)∣∣ = ∑ st,at ∣∣∣ 1t ∑t−1i=0 pπD (st, at)− 1t ∑t−1i=0 pbrt<k,i(st, at)∣∣∣ ≤ 1
t t−1∑ i=0 ∑ st,at ∣∣pπ(st, at)− pbrt<k,i(st, at)∣∣ (D)
≤ 1 t t−1∑ i=0 {(t− i)2 m′ + i2 π}
= 1
t
{ 2t2 m′ − (t− 1)t m′ + (t− 1)t π } = 1
t
{ 2t2 m′ − t2 m′ + t m′ + t2 π − t π } = 1
t
{ t2( m′ + π) + t( m′ − π) } = t( m′ + π) + ( m′ − π) (29)
For (D), we apply Lemma 2 with setting m,post = m′ and π,post = 0 for the rollout following π and pθ(s′|s), and m,pre = 0 and π,pre = π for the rollout following πD and p(s′|s). For term C, we can derive the bound in a similar manner to the case of term B:∑ st,at ∣∣pπ(st, at)− pbrt≥k(st, at)∣∣ = ∑ st,at
∣∣∣ 1k∑k−1i=0 pπD (st, at)− 1k∑k−1i=0 pbrt≥k,i(st, at)∣∣∣ ≤ 1
k k−1∑ i=0 ∑ st,at ∣∣pπ(st, at)− pbrt≥k,i(st, at)∣∣ ≤ 1
k k−1∑ i=0 {(k − i)2 m′ + (t− k + i)2 π}
= 1
k
{ 2k2 m′ − (k − 1)k m′ + 2tk π − 2k2 π + (k − 1)k π } = 1
k
{ 2kt π + 2( m′ − π)k2 − (k − 1)k( m′ − π) } = 1
k
{ 2kt π + ( m′ − π)k2 + k( m′ − π) } (30)
By substituting Eqs. 20, 29, and 30, into Eq. 19, we obtain the result:∣∣Eπ,p [R]− EDmodel [R]∣∣ ≤ rmax{ 2 π +∑k−1t=1 γt {t( m′ + π) + ( m′ − π)}+∑∞t=k γt { 1k {2kt π + ( m′ − π)k2 + k( m′ − π)}} }
= rmax
{ 2 π + 1−kγ(k−1)+(k−1)γk (1−γ)2 γ( m′ + π) + γk−γ γ−1 ( m′ − π)
+ ∑∞ t=k γ t 1 k { 2kt π + ( m′ − π)k2 + k( m′ − π)
} }
= rmax  2 π + 1−kγ(k−1)+(k−1)γk (1−γ)2 γ( m′ + π) + γk−γ γ−1 ( m′ − π) + ∑∞ t=1 γ t 1 k { 2kt π + ( m′ − π)k2 + k( m′ − π) } − ∑k−1 t=1 γ t 1 k { 2kt π + ( m′ − π)k2 + k( m′ − π) } 
= rmax  2 π + 1−kγ(k−1)+(k−1)γk (1−γ)2 γ( m′ + π) + γk−γ γ−1 ( m′ − π) + 1(1−γ)2 2γ π + γ 1−γ {( m′ − π)k + ( m′ − π)} − 1−kγ (k−1)+(k−1)γk (1−γ)2 2γ π
−γ k−γ γ−1 {( m′ − π)k + ( m′ − π)}  = rmax  1+γ2 (1−γ)2 2 π + γ−kγk+(k−1)γk+1 (1−γ)2 ( m′ − π) +γ
k−γ γ−1 ( m′ − π) +
( γ
1−γ − γk−γ γ−1 ) (k + 1)( m′ − π)  = rmax { 1+γ2 (1−γ)2 2 π + γ−kγk+(k−1)γk+1
(1−γ)2 ( m′ − π) +γ
k−γ γ−1 ( m′ − π) + γk 1−γ (k + 1)( m′ − π)
} (31)

heading: A.8 THE RELATION OF RETURNS IN THE CASE WHERE A REWARD PREDICTION IS INACCURATE
text: In Section 5, we discuss the relation between the true returns and the returns estimated on the metamodel under the assumption that the reward prediction error is zero. The theoretical result under this assumption is still useful because there are many cases where the true reward function is given and the reward prediction is not required. However, a number of readers still may want to know what the relation of the returns is under the assumption that the reward prediction is inaccurate. In this section, we provide the relation of the returns under inaccurate reward prediction in the MDP case 17.
17Here, we do not discuss the theorems in the POMDP case because those in the MDP case can be easily extended into the POMDP case by utilizing Lemma 1.
We start our discussion by defining the bound of the reward prediction error r: Definition 4. r := maxt E(at,st)∼Dmodel [|r(st, at)− rθ(st, at)|], where rθ(st, at) := Ert∼pθ [rt|at, st].
We also define the return on the branched rollout with inaccurate reward prediction.
EDmodel [R̂] := ∑ s0,a0 pπD (s0, a0)rθ(s0, a0) + k−1∑ t=1 ∑ st,at γtpbrt<k(st, at)rθ(st, at)
+ ∞∑ t=k ∑ st,at γtpbrt≥k(st, at)rθ(st, at) (32)
Now, we provide the relation between the returns under inaccurate reward prediction. Theorem 6 (Extension of Theorem 4 into the case where reward prediction is inaccurate). Under the k steps branched rollouts, using the bound of a model error under πD, m = maxt Ea∼πD,s∼p,t [DTV (p(s′|s, a)||pθ(s′|s, a))], the bound of the policy shift π = maxsDTV (π||πD) and the bound of the reward prediction error r = maxt E(at,st)∼Dmodel [|r(st, at)− rθ(st, at)|], the following inequation holds,
(33)Eπ,p [R] ≥ EDmodel [R̂]− rmax { 1 + γ2 (1− γ)2 2 π + γ − kγk + (k − 1)γk+1 (1− γ)2 ( π + m)
+ γk − γ γ − 1 ( π + m) + γk 1− γ (k + 1)( π + m)
} − γ
1− γ r.
Proof.
(34) ∣∣∣Eπ,p[R]− EDmodel [R̂]∣∣∣ = ∣∣∣Eπ,p[R]− EDmodel [R] + EDmodel [R]− EDmodel [R̂]∣∣∣
≤ ∣∣Eπ,p[R]− EDmodel [R]∣∣+ ∣∣∣EDmodel [R]− EDmodel [R̂]∣∣∣
(35)
∣∣∣EDmodel [R]− EDmodel [R̂]∣∣∣ = ∣∣∣∣∣∣
∑ s0,a0
pπD (s0, a0) {r(s0, a0)− rθ(s0, a0)} + ∑k−1 t=1 ∑ st,at
γtpbrt<k(st, at) {r(st, at)− rθ(st, at)} + ∑∞ t=k ∑ st,at γtpbrt≥k(st, at) {r(st, at)− rθ(st, at)} ∣∣∣∣∣∣ ≤ ∑ s0,a0 pπD (s0, a0) |r(s0, a0)− rθ(s0, a0)|
+ k−1∑ t=1 ∑ st,at γtpbrt<k(st, at) |r(st, at)− rθ(st, at)|
+ ∞∑ t=k ∑ st,at γtpbrt≥k(st, at) |r(st, at)− rθ(st, at)|
≤ ∞∑ t=0 γt r = γ
1− γ r
By substituting Eqs. 27 and 35 into Eq. 34, we obtain the result:
(36) ∣∣∣Eπ,p[R]− EDmodel [R̂]∣∣∣ ≤ rmax
{ 1+γ2
(1−γ)2 2 π + γ−kγk+(k−1)γk+1 (1−γ)2 ( π + m)
+γ k−γ γ−1 ( π + m) + γk 1−γ (k + 1)( π + m)
} + γ
1− γ r
Theorem 7 (Extension of Theorem 5 into the case where reward prediction is inaccurate). Let m′ ≥ maxtEa∼π,s∼p [DTV (p(s′|s, a)||pθ(s′|s, a))] and r = maxt E(at,st)∼Dmodel [|r(st, at)− rθ(st, at)|],
(37)Eπ,p [R] ≥ EDmodel [R̂]− rmax { 1 + γ2 (1− γ)2 2 π + γ − kγk + (k − 1)γk+1 (1− γ)2 ( m′ − π)
+ γk − γ γ − 1 ( m′ − π) + γk 1− γ (k + 1)( m′ − π)
} − γ
1− γ r.
Proof. Similar to the derivation of Theorem 6, we obtain the result by substituting Eq. 31 and 35 into Eq. 34.

heading: A.9 PEARL IN SECTIONS 6 AND 7
text: The PEARL algorithm used in Sections 6 and 7 refers to “PEARL with RNN-traj” in (Rakelly et al., 2019). The comparison result with other types of PEARL (i.e., vanilla PEARL and PEARL with RNN-tran) in (Rakelly et al., 2019) and M3PO are shown in Figure 7. The figure indicates that M3PO achieves better sample efficiency than them.

heading: A.10 BASELINE METHODS FOR OUR EXPERIMENT
text: PEARL: The model-free meta-RL method proposed in Rakelly et al. (2019). This is an offpolicy method and implemented by extending Soft Actor-Critic (Haarnoja et al., 2018). By leveraging experience replay, this method shows high sample efficiency. We reimplemented the PEARL algorithm on TensorFlow, referring to the original implementation on PyTorch (https: //github.com/katerakelly/oyster). Learning to adapt (L2A): The model-based meta-RL proposed in Nagabandi et al. (2019a). In this method, the meta-model is implemented with MAML (Finn et al., 2017) and the optimal action is found by the model predictive path integral control (Williams et al., 2015) on the full meta–model based rollouts. We adapt the following implementation of L2A to our experiment: https://github.com/iclavera/learning_to_adapt

heading: A.11 ENVIRONMENTS FOR OUR EXPERIMENT
text: For our experiment in Section 7, we prepare simulated robot environments using the MuJoCo physics engine (Todorov et al., 2012) (Figure 8): Halfcheetah-fwd-bwd: In this environment, meta-policies are used to control the half-cheetah, which is a planar biped robot with eight rigid links, including two legs and a torso, along with six actuated joints. Here, the half-cheetah’s moving direction is randomly selected from “forward” and “backward” around every 15 seconds (in simulation time). If the half-cheetah moves in the correct direction, a positive reward is fed to the half-cheetah in accordance with the magnitude of movement, otherwise, a negative reward is fed. Halfcheetah-pier: In this environment, the half-cheetah runs over a series of blocks that are floating on water. Each block moves up and down when stepped on, and the changes in the dynamics are rapidly changing due to each block having different damping and friction properties. These properties are randomly determined at the beginning of each episode. Ant-fwd-bwd: Same as Halfcheetah-fwd-bwd except that the meta-policies are used for controlling the ant, which is a quadruped robot with nine rigid links, including four legs and a torso, along with eight actuated joints. Ant-crippled-leg: In this environment, we randomly sample a leg on the ant to cripple. The crippling of the leg causes unexpected and drastic changes to the underlying dynamics. One of the four legs is randomly crippled every 15 seconds. Walker2D-randomparams: In this environment, the meta-policies are used to control the walker, which is a planar biped robot consisting of seven links, including two legs and a torso, along with six actuated joints. The walker’s torso mass and ground friction is randomly determined every 15 seconds. Humanoid-direc: In this environment, the meta-policies are used to control the humanoid, which is a biped robot with 13 rigid links, including two legs, two arms and a torso, along with 17 actuated joints. In this task, the humanoid moving direction is randomly selected from two different directions around every 15 seconds. If the humanoid moves in the correct direction, a positive reward is fed to the humanoid in accordance with the magnitude of its movement, otherwise, a negative reward is fed.

heading: A.12 COMPLEMENTARY EXPERIMENTAL RESULTS
text: d π
) and the horizontal axis represents
the training sample size (x1000). The red-dotted line is the linear interpolation of the blue dots, which shows the trend of the local change decreasing as the training sample size grows.

heading: A.13 COMPLEMENTARY ANALYSIS
text: In addition to Q1 and Q2 in the main content, we also conducted a complementary analysis to answer the following question. Q.3: Does the use of a meta-model in M3PO contribute to the improvement of the meta-policy?
In an analysis in this section, we compare M3PO with the following method. Model-based MetaPolicy Optimization (M2PO): This method is a variant of M3PO, in which a non-adaptive predictive model is used instead of the meta-model. The predictive model architecture is the same as that in the MBPO algorithm (Janner et al., 2019) (i.e., the ensemble of Gaussian distributions based on four-layer feed-forward neural networks).
Regarding Q3, our experimental result indicates that the use of a meta-model contributed to the performance improvement in a number of the environments. In Figure 15 in the appendix, we can clearly see the improvement of M3PO against M2PO in Halfcheetah-fwd-bwd. In addition, in the Ant environments, although the M3PO’s performance is seemingly the same as that of M2PO, the qualitative performance is quite different; the M3PO can produce a meta-policy for walking in the correct direction, while M2PO failed to do so (M2PO produces the meta-policy “always standing” with a very small amount of control signal). For Humanoid-direc, in contrast, M2PO tends to achieve better sample efficiency than M3PO. We hypothesize that the primary reason for this is that during the plateau at the early stage of training in Humanoid-direc, the predictive model used in M2PO generates fictitious trajectories that make meta-policy optimization more stable. To verify this hypothesis, we compare TD-errors (Q-function errors) during training, which is an indicator of the stability of meta-policy optimization, in M3PO and M2PO. The evaluation result (Figure 16 in the appendix) shows that during the performance plateau (10–60 epoch), the TD-error in M2PO was actually lower than that in M3PO; this result supports our hypothesis. In this paper, we did not focus on the study of meta-model usage to generate the trajectories that make meta-policy optimization stable, but this experimental result indicates that such a study is important for further improving M3PO.

heading: A.14 HYPERPARAMETER SETTING
text: Table 1: Hyperparameter settings for M3PO results shown in Figure 1. x → y over epohs a → b denotes a thresholded linear function, i.e., at epoch e, f(e) = min(max(x+ e−a
b−a · (y − x), x), y).
H al
fc he
et ah
-f w
dbw
d
H al
fc he
et ah
-p ie
r
A nt
-f w
dbw
d
A nt
-c ri
pp le
dle
g
W al
ke r2
D -r
an do
m pa
ra m
s
H um
an oi
ddi
re c
N epoch 200 E environment step per epoch 1000 M meta-model rollouts per environment step 1e3 5e2 1e3 5e2 B ensemble size 3 G meta-policy update per environment step 40 20 k meta-model horizon 1 1 1→ 25 1 1
over epoch 20→ 100
A.15 NOTATIONS
Table 3: Ek[k = 1] settings for results shown in Figure 17. a → b denotes a thresholded linear function, i.e., at epoch e, f(e) = min(max(1− e−a
b−a , 1), 0).
H al
fc he
et ah
-p ie
r
W al
ke r2
D -r
an do
m pa
ra m
s
H um
an oi
ddi
re c
Ek[k = 1] expected meta-model horizon 80→ 130 50→ 100 150→ 250

heading: A.16 COMPLEMENTARY ANALYSIS 2
text: In Figures 1 and 13, we can see that, in a number of the environments (Halcheetah-pier, Walker2Drandomparams and Humanoid-direc), the long-term performance of M3PO is worse than that of PEARL. This indicates that a gradual transition from M3PO to PEARL (or other model-free approaches) needs to be considered to further improve overall performance. In this section, we propose to introduce such a gradual transition approach to M3PO and evaluate it on the environments where the long-term performance of M3PO worse than that of PEARL.
For the gradual transition, we introduce the notion of the “expected” model-rollout length Ek[k] for k ∈ {0, 1} to M3PO. In this notion, Ek[k = 1] means the probability of one-step meta-model rollout. Namely, with the probability of Ek[k = 1], the fictitious trajectory generated by one-step meta-model rollout is used for the policy update (in line 11 in Algorithm 2), and, with the probability of 1 - Ek[k = 1], the real trajectory in Denv are used for the policy update. In our implementation, we replace Dmodel in line 11 in Algorithm 2 with the mixed dataset Dmix. Here, Dmix is defined as
Dmix = Ek[k = 1] · Dmodel + (1− Ek[k = 1]) · Denv. (38)
We linearly reduce the value of Ek[k = 1] from 1 to 0 depending on training epoch. With this value reduction, the M3PO is gradually transitioned to PEARL (i.e., the model-free approach).
We evaluate M3PO with the gradual transition (M3PO-h) in three environments (Halfcheetahpier, Walker2D-randomparams and Humanoid-direc) where the long-term performance of M3PO is worse than that of PEARL in Figures 1 and 13. The hyperparameter setting (except for setting schedule for the value of Ek[k = 1]) for the experiment is the same as that for Figures 1 and 13 (i.e., the one shown in Table 1). Regarding to setting schedule for the value of Ek[k = 1], we reduce the value of Ek[k = 1] in accordance with Table 3.
Evaluation results are shown in Figure 17. Due to the limitation of our computational resource, we could not continue the training of M3PO-h until convergence, and the resulting training epoch is much shorter than those of PEARL. Nevertheless, we can see that some trials of M3PO-h achieve same or better scores with the long-term performances of PEARL in all environments (e.g., M3POh-1 and M3PO-h-2 achieve the same or better performance than the best scores of PEARL-2 and PEARL-3 in Halfcheetah-pier).

heading: A.17 HIGH-LEVEL SUMMARY

Review summaries:
The paper concerns model-based meta-RL. It exploits the fact that meta-RL can be formulated as POMDP in which the task indicator is part of the (unobserved) hidden state. Thus, the paper effectively analyzes and proposes model-based algorithms for POMDPs. The paper bounds the gap between the expected reward of a policy in the actual POMDP and the estimated model and then theoretically shows that this gap can be reduced when using dyna-style / branched rollouts instead of full rollouts under the learned model. Motivated by this finding, the paper proposes a Dyna-like algorithm for POMDPs. In the experimental evaluation, the paper compares its proposed method, M3PO, to two recent meta-RL approaches in a range of meta-RL environments for continuous control.

This paper focus on model-based RL on a POMDP setting (they call it "meta RL"), where the policy and model need to infer the current hidden state according to history. It provides a theoretical relation between true environment returns and the returns from learned models in a POMDP setting. And it also provides a practical algorithm called M3PO and shows this algorithm is more sample efficient than some meta-RL baselines in some continuous control tasks.