Meta-Model-Based Meta-Policy Optimization
Authors: Takuya Hiraoka
Takahisa Imagawa
Voot Tangkaratt
Takayuki Osa
Takashi Onishi
Yoshimasa Tsuruoka
Venue:  (2021)

Abstract: Model-based reinforcement learning (MBRL) has been applied to meta-learning settings and has demonstrated its high sample efficiency. 
However, in previous MBRL for meta-learning settings, policies are optimized via rollouts that fully rely on a predictive model of an environment. 
Thus, its performance in a real environment tends to degrade when the predictive model is inaccurate. 
In this paper, we prove that performance degradation can be suppressed by using branched meta-rollouts. 
On the basis of this theoretical analysis, we propose Meta-Model-based Meta-Policy Optimization (M3PO), in which the branched meta-rollouts are used for policy optimization. 
We demonstrate that M3PO outperforms existing meta reinforcement learning methods in continuous-control benchmarks. 

Under review as a conference paper at ICLR 2021
META-MODEL--ASHD META-POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based reinforcement learning (MBRL) has been applied to meta-learning settings and has demonstrated its high sample efficiency. However, in previous MBRL for meta-learning settings, policies optimized via rollouts that fully rely on a predictive model of an environment. Thus, its performance in a real environment tends to degrade when the predictive model is inaccurate. In this pa- per, we prove that performance degradation can be suppressed by using branched meta-rollouts. On the basis of this theoretical analysis, we propose Meta-Model- based Meta-Policy Optimization (M3PO), in which the branched meta-rollouts are used for policy optimization. We demonstrate that M3PO outperforms exist- ing meta reinforcement learning methods in continuous-control benchmarks.
INTRODUCTION
Reinforcement learning (RL) methods have achieved remarkable success in many decision-making tasks, such playing video games controlling robots (e.g.. Gu et al. (2017); Mnih et al. (2015). In conventional RL methods, when multiple tasks are to be solved, a policy is independently learned for individual tasks. In general, each learning requires millions of training samples from the environ- ment. This independent learning with a large number of samples prevents conventional RL methods from being applied to practical multi-task problems (e.g.. robotic manipulation problems involving grasping or moving different types of objects Yu et al.) 2019)). Meta-learning methods (Schmidhu- ber et al. 1996 Thrun & Pratt 1998) have recently gained much attention as promising solution to this problem (Finn et 2017). They learn structure shared in the tasks by using large number of samples collected across the parts of the tasks. Once learned, these methods can adapt quickly to new (or the rest of the) tasks with a small number of samples given.
Meta-RL methods have previously been introduced into both model-free and model- settings. For model-free settings, there are two main types of approaches proposed far, recurrent- based policy adaptation (Duan et al.) 2017 Mishra et al.) 2018 Rakelly et al. 2019 Wang et al. 2016) and gradient-based based policy adaptation (Al- (Al-Shedivat al. 2018 Finn & Levine 2018 Finn et al. 2017 Gupta et al.) 2018 Rothfuss et al.) 2019 Stadie et al. 2018). In these approaches, policies adapt
task by leveraging the history of past trajectories. Following previous work (Clavera et al., 2018). we refer to these adaptive policies as meta-policies in our paper. In these model- free meta-RL methods, in addition to learning control policies, the learning of policy adaptation is also required (Mendonca et al. 2019). Thus, these methods require more training samples than conventional RL methods.
For model-based settings, there have been relatively few approaches proposed so far. Sæmundsson al. (2018) and [Perez et al. (2020) use a predictive model (i.e., transition model) conditioned by a latent variable for model predictive control. Nagabandi et al. (2019a b) introduced both recurrent- based and gradient-base baased meta-learning methods into model- based RL. these approaches, the predictive models adapt to new task by leveraging the history of past trajectories. In analogy to the meta-policy, we refer to these adaptive predictive models as meta-models in our paper. Generally, these model-based meta-RL approaches are more sample efficient than the model-free approaches. However, in these approaches, the meta meta-policy (or the course of actions) is optimized via rollouts relying fully the meta-model. Thus, its performance in real environment tends degrade when the meta-mode is inaccurate. this paper, we address this performance degradation problem in model-based meta-RI..

---

Under review as a conference paper at ICLR 2021
After reviewing related work (Section 2 and preliminaries (Section present our work by first formulating model- based meta-RL (Section 4. Model-based (and model-free) meta-RL settings have typically been formulated as special cases of solving partially observable Markov decision processes (POMDPs) (e.g. Duan et al. 2017); Killian et al. (2017): Perez et al. (2020)). In these special cases, specific assumptions, such as intra-episode task invariance, duced. However, there are model-based meta-RL settings where such assumptions do not hold (c.g.. Nagabandi al. (2019a b)). To include these settings into our scope, we formulate model-based meta-RL. settings as solving POMDPs without introducing such additional assumptions. Then, we conduct theoretical analysis on its performance guarantee (Section [5]. We first analyse the perfor- mance guarantee in full meta-model-based rollouts, which most of the previous model-bass meta- RL methods hold. We then introduce the notion of branched meta-rollouts. Branched meta-rollouts are Dyna-style rollouts (Suttonn 1991) in which we can adjust the reliance on the meta-model and real environment data. We show that the performance degradation due to the meta-model error in the branched meta-rollouts is smaller than that in the full meta- meta--oodll-basee rollouts. On the basis of this theoretical analysis, we propose a practical model based meta-RL. method called Meta-Model-basse Meta-Policy Optimization (M3PO) where the meta-model is used in the branched rollout manner (Section (). Finally, we experimentally demonstrate that M3PO outperforms existing methods in continuous-control benchmarks (Section 7
additionally intro-
We make the following contributions in both theoretical and empirical frontiers. Theoretical fron- tier: 1. Our work is the first attempt to provide theoretical relation between learning the meta- model and the real environment performance. In the aforementioned model-based meta-RL litera- ture, it has not been clear how learning the meta-model relates to real environment performance. Our theoretical analysis provides relations between them (Theorems and 3). This result theoretically justifies -rainini good transition model to improve overall performance in the real environ- ment. 2. Our analysis also reveals that the use of branched meta-rollouts can suppress performance degradation due to meta-model errors. 3. We refine previous fundamental theories proposed by Janner et al.) (2019) to consider important premises more properly (Theorems + [) This modi- fication is important to strictly guarantee the performance especially when the model-rollout length is long. Empirical frontier: We propose and show the effectiveness of M3PO. Notably, we show that M3PO achieves better sample efficiency than existing meta-RI. methods in complex tasks, such as controlling humanoids.
RELATED WORK
In this section, we review related work on POMDPs and theoretical analysis in model-based RL.
Partially observable Markoy decision processes T In our paper, we formulate model-bbaed meta- RL as solving POMDPs, and provide its performance guarantee under the branched meta-rollout scheme. POMDPs are a long-studied problem (e.g., (Ghavamzadeh et al.] 2015 Sun 2019 Sun (et al. ). and many works have discussed a performance guarantee of RL methods to solve POMDPs. However, the performance guarantee of the RL methods based on branched meta-rollouts has not been discussed in the literature. On the other hand, a number of researchers (Igl et al. 2018 Lee al. 2019: Zintgraf et al., 2020) have proposed model-free RL methods to solve POMDP without prior knowledge of the accurate model. However, they do not provide theoretical analyses of performance. In this work, by contrast, we propose model-based meta-RI. method and provide theoretical analyses on its performance guarantee.
Theoretical analysis on the performance of model-based RL: Several theoretical analyses on the performance of model-basee RL have been provided in previous work (Feinberg et al. [2018] Henaff
2019 Janner et al. 2019 Luo et al., 2018 Rajeswaran et al. 2020). In these theoretical analyses, standard Markov decision processes (MDPs) are assumed, and the meta-learning (or POMDP) set- ting is not discussed. In contrast, our work provides theoretical analysis on the meta-learnig (and POMDP) setting, by substantially extending the work of Janner et al. (2019). Specifically, Janner et al. (2019) analysed the performance guarantee of branched rollouts on MDPs, and introduced branched rollouts into a model-based RL algorithm. We extend their analysis and algorithm to meta-lcarning (POMDP) case. In addition, we modify their theorems so that important premises
We include works on Bayes-adaptive MDPs Gihavamzadeh al. are special case of POMDPs.
2015 Zintgraf et al. 2020) because they
2

---

Under review as a conference paper at ICLR 2021
(e.g., the effect of multiple-model rollout factors) more properly considered. See A.1 in the appendix for a more detailed discussion of our contribution.
3 PRELIMINARIES
Meta reinforcement learning: We assume online adaptation situations (Nagabandi et al. 2019a bp where the agent can leverage a few samples to adapt to a new task. Here, a task specifies the transition probability and the reward function. Information about task identity cannot be observed by the agent, and the task may change at any step in an episode. A meta-RL process is composed of meta-training and meta-testing. In veeta-training, policy and predictive model that are prepared for efficient adaptation are learned with a meta-training task set. In meta-testing, on the basis of the meta-training result, the policy and the predictive model adapt to new task. For the adaptation, the trajectory observed from the beginning of the episode to the current time step is leveraged. As we noted earlier, we call an adaptive policy and a predictive model of this sort a meta-policy and a meta-model, respectively.
Partially observable Markov decision processes: We formalize our problem with a POMDP. which is defined as a tuple (O,S, A. Pob, Pa). Here, O is a set of observations, S is a set of hidden states, A is a set of actions, Pob := O x S x A [0, 1] is the observation prob- ability, Pa == S x S x A + [0. 1] is the state transition probability,
: S x A + R a reward function and Y E [0. 1) a discount factor. At time step t, these functions are used as (a((s at-1), Pob (or188, ax-1) and Tr r(s,, az). The agent cannot directly observe the hidden state, but receives the observation instead. The agent selects an action on the basis of policy :- p(a++ilhe). Here, hu is a history (the past trajectories) defined as h4 := {ag. Ou.... ae, 0r We denote the set of the histories by H. Given the definition of the history, the history transition probability can be defined as p(hc+1la4+1 h2) == p(oc=1lhe). Here, p(or+il/e)) == -Es+.1 E., p((e (( (()))((+ 1 |8,, at )p(o)+ 1 az), where p( 8 ((e)/h)) is the belief about the hidden state. The goal of RL in the POMDP is to find the optimal policy n² that maximizes the expected return R := 2 7e (i.e., nt" arg max Eg hwp [R]).
4 FORMULATING MODEL-BASED META-RL
In this section, we formulate model-based meta-RL as solving POMDP by using parameterized meta-policy and a meta-model. The outline of our formulation is shown in Figure in the appendix.
In our formulation, the task is included in the hidden state: S == T x S' Here T is the set of task t and S' is the set of the other hidden state factors s'. With this definition, the state transition probability, observation probability and reward function can be defined respectively as follows: p($t_1|s;,a)) at) = p( (T 4 Fe. s/, a.). p(o 0,) p 1 (T-11. a,) and r(st, a.) =r(Te, a.). In addition, with Finn & Levine 2018); Finn et al. (2017); RRakelyy jet al. (2019). we assume that the task set T and the initial task distribution p(Tn) do not change in meta-training and meta-testing. Owing to this assumption, in our analysis and algorithm, meta-
training and meta-testing can be seen as an identical one. Note that the task can be changed during an episode. Namely, the value of Tc+1 is not necessarily equal to that of Te.
We define a meta-policy and meta-model as (a(44 (he) and pa (Te, 112)), respectively. Here, ø and 0 are learnable parameters for them. rt and 0z+11 are assumed be conditionally independent given h, i.e., pe(rt. 0441 h2) = Po (r(rehh.) (o(00-11hh) As with p(he41l0441. he), the mota-model for the history can be defined as (he++ 1 h4) == po(or11lhe)
We use the parameterized meta-model and meta-policy as shown in Algorithm|! I This algorithm composed of 1) data collection in the real environment and 2) optimization of the meta neta-policy and meta-model. In 1), the data is collected from the real environment with the meta meta-policy and stored into dataset DD In 2). the meta-policy and meta-model are optimized to maximize Ea- [R] C( (0), ex (o)). Here, Ee- [R] is meta-model return (the return of
FFor simplicity, we use these probabilities by abbreviating the subscripts "st" and "oh."
3

---

Under review as a conference paper at ICLR 2021
Algorithm Abstract Meta Model-based Meta-Policy Optimization (Abstract M3PO)
1: Initialize meta-pott meta-policy Rd meta-model py and dataset D.
2: for N epochs do
3: Collect trajectories from environment in accordance with ne: D = DU ((he, ot)),
4: Optimize T and py: (Ø, () <- arg max Baw.o, [R]
(6)). Here, D is
(o.0)
used to evaluate fm (0) and é, (d) and generate initial history ho.
5: end for
the meta-policy on the meta-model) quantities Cm and c,- Their detailed definitions introduced in the next section.
p Clem ex (6)) is a discrepancy depending on the two error
Our POMDP-based formulation covers wide range of meta-RL settings including Bayes-adaptivg MDPs Zintgraf et al., 2020). hidden parameter-MDPs (Killian et al. 2017;; Perez al.) 2020),
parameterized MDPs (Duan et al.; 2017; Finn & Levine 2018), and the settings considered by Nagabandi al. (2019a and Jabri al. 2019). These settings be primarily recovered in
formulation by introducing both or either of the following assumptions (Asml and Asm2) [ Asm1: the task does not change during an episode. Asm2: observation o is identical the hidden state s': o = 8'. The Bayes-adaptive MDPs, hidden-parameter MDPs and parameterized MDPs can be recovered by introducing both Asml and Asm2. The setting of Nagabandi et al. (2019a b) can be recovered by introducing Asm2, and the setting Jajri et al. (2019) can be recovered by introducing Asm1. As a detailed example, we recover the parameterized MDPs in Appendix A.5
PERFORMANCE GUARANTEES OF MODEL-BAR -BSSE META-RL
In this section, we analyse the performance guarantee of model-based meta-RL with an inaccurate meta-model. In Section 5.1 we provide the performance guarantee in full veta-modellbasee roll-
out. In Section 5.2] we introduce the notion of branched meta-rollout and analyse its performance guarantee. We show that the meta-man model error is less harmful in the branched meta-rollout than that the full meta-model-rollout.
5.1 PERFORMANCE GUARANTEE IN A FULL META-MOOEL--BSED ROLLOUT CASE
Our goal is to outline theoretical framework in which can provide performance guarantees for Algorithm To show the guarantees, we construct a lower bound taking the following form:
Era.p
R Ex,.po
[ C((m (o)).
(1)
Here, Ex,.p [ denotes true return (i.c., the return of the meta-policy in the real environment). The discrepancy between these returns, C, be expressed as the function of two error quantities: the generalization error of the meta-policy and the distribution shift due to the updated meta-policy. For our analysis, define the bounds of the generalization error em and the distribution shift ex as follows:
Definition 1. Em (0) = max; Eawap hoop [DVv (p(he+1lat+= he) (loo 1 he))]. Here, Dry is total variation distance and TD is the data-collection policy whose actions contained in D follow? []
Definition 2. (x()) == max,, Drv (FD(att)lh))lTT (he))
We also assume that the expected reward is bounded by constant Tmaa-
Definition 3. Tmaz max, Σ p(st he)r(s, a-)]
Now we present our bound, which an extension of the theorem proposed in Janner et al. (2019).
For simplicity, we use the abbreviated style Eng.rs [Rt].
'For simplicity, omit implementaaiin-leved assumptions (c.g.. "meta-policy meta-model imple- mented on the basis gradient-based MAMI." in |inn & Levine (2018); Nagabbadi et al. (2019a)).
As with Janner et al. (2019). to simplify our analysis, we assume that the meta-model can accurately estimate reward. We discuss the in which the reward prediction of meta-model is inaccurate in A.8
4

---

Under review as a conference paper at ICLR 2021
Theorem (The POMDP extension of Theorem 4.1. in Janner et al. 2019 max; Bewxp.hop Drv (p(h441 (as41, ht)llpo (le+ aa (,4+1,1)) and 6x maxt. Drv (TD(ae+1) (11)) ( ( (.)).. Then, the true returns are bounded from below by meta-model returns of the meta-policy and discrepancy:
Let Cm =
27 Cmn 2Ex)
4e,
E Emg. [R] 2 Po
[R]
mmax
(1 7)
(1 (1-7)
((em(0).er(()
This theorem implies that the discrepancy the returns under full meta-model-based rollout scales linearly with both 6n and C.- If we reduce the discrepancy C, the two returns closer each other. As a result, the performance degradation is more significantly suppressed. the next section, discuss new meta-model usage reduce the discrepancy induced the meta-model error 6m-
5.2 PERFORMANCE GUARANTEE IN THE BRANCHED META-ROLLOUT CASE
The analysis of Theorem I relies on running full rollouts through the meta-model, causing meta- model errors to compound. This is reflected in the bound by factor scaling quadratically with the effective horizon, 1/(1 - 7). In such cases, we can improve the algorithm by choosing to rely less on the meta-model and instead more real environment data.
To allow for adjustment between meta-model- bbased and model-free rollouts, we introduce the notion of a branched meta-rollout. The branched meta-rollout is kind of Dyna-style rollout (Sutton 1991], in which the meta-model-based rollout is run as being branched from real environment data. More concretely, the rollout run in the following two processes. history under the data-collection meta-policy's history distribution Pro (ha), and 2) we then run k steps in accordance with & under the learned meta-model pa-
We begin a rollout from
Under such scheme, the true return can be bounded from below:
Theorem 2. Under the k steps branched meta-rollouts, using the bound of meta-model error under WD. fm = max, (Bawan.h-p.. (rr (p(h')tt (h, a) (llpppo' a))], the bound of the meta-policy shift ex maxi, Drv (TD) [rp||Te To), and the return on the mela-model Eiy,, model [R] where Dmodet the set of samples collected through branched rollouts, the following inequation holds,
1 1+
22x + kot + (K -1)1))
E E pt [R] 2 (aa,,i)) [R]
(Ex + em)
max
{ (1 7)2
(1-
(3)
k
7*
(er cm) + 1 1- -(k (1+1)(Er + tmm)
+ 7
The discrepancy factors relying on en Theorem can be smaller than those relying on 6m in Theorem This indicates that the performance degradation due to the meta-model error can be more suppressed than that in the full merta-mode--bee rollout
META-MODEL-BASSE
META-POLICE OPTIMIZATION WITH DEEP RL
In the previous section, we show that the use of branched meta-rollouts can suppress performance degradation. In this section, on the basis of this result, we modify Algorithm so that the meta- policy and meta-model are optimized with B(aah)>-Dmott [R] C' (0), c (Ø)), instead of with
Emg.po [R] (()m(0)) e0), (Ø)).
More specifically, we propose the following modifications to Algorithm [
Meta-policy optimization:
The meta-policy is optimized with the branched meta-rollouts re- [R] S For the optimization, we adapt PEARL (Rakelly et al. 2019) be-
turn (((dh))PPoodell
"See Corollary in the appendix.
A.6 we also prove that the discrepancy be further reduced by introducing an additional assumption. 'To stabilize the learning.
omit C(em (0), ex (o)) from the optimization objective for the meta- meta-policy. The transition of (0) during learning with this limited optimization objective is shown in Figure 12 in the appendix. The result indicates that (d) tends decrease the training data size (epoch) grows.

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Meta-Modellbased Meta-Policy Optimization with Deep RL (M3PO)
1: Initialize meta meta-policy Td, meta-model pe, environment dataset Denv, meta-model dataset D model-
2: for N epochs do
3: Train meta-model pe with Dewv: 0 t arg max Denn [po(Te, 01+1
4: for E steps do
5:
Take actions according to тe; add the trajectory to Denv
6: for M model rollouts do
7:
Sample h1 uniformly from Deny Perform k-step meta-model rollouts starting from by using meta-policy Tei add fictitious
8:
trajectories to moodel end for
9:
10: for G gradient updates do
11:
Update policy parameters with mosdet: Φ s- Φ Vo model (Ø) end for
12:
13: end for
14: end for
cause it achieved good learning performance in meta-leariing settings. We use the fic- titious trajectories generated from the branched meta-rollouts to optimize the meta-policy. Formally, Th. optimized by using the gradient of optimization objective JD (Ø) ((a,)))-Poodd [Del (Tollexp (Q - V=n))]. Here, DEL is the Kullback-Leibler divergence, (a141, h4) == B(r.))-Pmoeel [Rla+11. hu] and (he) == Earei (at+1 ( As in PEARL, in тæ. the latent context is estimated by using past trajectories. The esti- mated context then used to augment the policy input. Formally, the meta-policy imple- mented TT(at+1|he) po(z|ag, oo..., (r,0r) 0r) is a context encoder. To (at++ Or, 2) is the conditioned policy. Similarly, in Qr, and V=,, the estimated latent context is also used to augment their input.
Eg (e(4t+1lor, 2)po(z/a0, oo...., ar, at, or). z is the latent context and
Meta-model optimization: The meta-model is optimized to minimize the discrepancy (i.e., min- imize cm 7 For the meta- meta-model, to consider both aleatoric and epistemic uncertainties, use bootstrap ensemble of B dynamics models [ub,..., p }. Here, ph is the i-th conditional Gaussian distribution with diagonal covariance: p& (re, 1 = . (re+1, 0c+1 1116 (h+), , (h+))- ng and o are the mean and standard deviation, respectively. In our implementation, we use the recurrent- based architecture inspired by Duan al. (2017); Nagabandi et al.) (2019b) Rakelly al. (2019); at each evaluation of the model, {o1, 01 .... ar-1, 0 in h is fed to recurrent neural network (RNN), and then its hidden unit output and (ar, 0z } in h2 are fed to the ferd-forward neural net- work that outputs the mean and standard deviation of the Gaussian distribution. We use the gated recurrent unit (GRU) (Cho et al. 2014) for the RNN. The recurrent layer in the GRU is composed of five sigmoid units. In addition, the feed-forward neural network is composed of an input layer, two hidden layers, and an output layer. The input and hidden layers are composed of 400 swish units (Ramachandran et al. 2017). To minimize c... we learn the metal via maximum like- lihood estimation with Denv- As in Lakshminarayanai et al. (2017). 7), the ensemble (pg.... p is learned on the shuffle of the real trajectories in Denv We apply a re-parameterization trick to evaluate the distributions, and the meta-model parameters 0 are optimized with a gradient-based optimizer. For the gradient-based optimizer we use Adam (Kingma Ba 2015). To avoid overfitting, we use weight decay and early termination (Bishop) 2006).
The resulting algorithm is shown in Algorithm 2 The modifications "Meta-model optimization" and "Meta-policy optimization" in the above paragraph are reflected in line 3 and lines 4-13, re- spectively. In line 4, k-step branched meta-rollouts are run. The uppropriately set k contributes to decreasing the discrepancy, and suppresses performance degradation. Thus, we treat ic as hyperpa- rameter, and set it to different values in different environments so that the discrepancy decreases. For the experiments described in the next section, we tune the hyperparameters for this algorithm by grid search. The search result for the hyperparameter values is described in Table [ in the appendix.
9hhe transition of em over training epochs is shown in Figure 11 in the appendix, and it indicates that the model error tends decrease the number of epochs increases.

---

Under review as a conference paper at ICLR 2021
For our experiments, implemented algorithm by extending the codebase of Model-based Policy Optimization T0 We made two main extensions: (1) introduce a latent context to the policy as in PEARL and (2) replace predictive model with the meta-model based on the RNN.
7 EXPERIMENTS
In this section, report our experiments 11 aiming to answer the following questions: Q.1: Can our method (M3PO) outperform existing meta-RL methods? Q.2: How do mota--model-rollot lengths A affect the actual performance?
In our experiments, we compare our method (M3PO) with two baseline methods: PEARL (Rakelly et al. 2019) and Learning to adapt (L2A) (Nagabandi al. 2019a). More detailed information is described in A0 in the appendix. We conduct a comparative evaluation of the methods on a variety of simulated robot environments using the MuJoCo physics engine (Todorov et al. 2012)
We prepare the environments proposed the meta-RI. (Finn & Levine, 2018; Nagabandi et al, 2019a Rakelly et al. 2019, Rothfuss al., 2019) and robust-RL (Hiraoka al., 2019; Rajeswaran iet al., 2017) literature: Hallcheetah-fwd-bwe
Halfcheetah-piees Ant-fwd-bwd, Ant-crippled- leg. Walker22-randomparams and Humanoid-diree In the environments, the agent required to adapt to fluidly changing task that the agent cannot directly observe. Detailed information about each environment is described in in the appendix.
Regarding Q1, our experimental results indicate that M3PO outperforms existing meta-RL. methods. In Figure [ the learning curves of M3PO and existing meta-RI. methods (I.2A and PEARL. on meta-training phases are shown. These learning curves indicate that the sample efficiency of M3PO better than those of L2A and PEARL The performance (return) of L2A remains poor and does not improve even when the training data increases. PEARL can improve meta- meta-policy performance via training in all environments. However, the degree of improvement that of M3PO. In number of the environments (e.g.. Halfchectah-piee -ierr), the relative performance of M3PO against PEARL becomes asymptotically worse. This indicates that, as with Nagabandi al. (2018), dynamic switching from M3PO to PEARL other model-free approaches needs be considered to further improve overall performance.
PEARL is smaller than
Regarding Q2, we conducted an evaluation of M3PO by varying its model-rollout length k. The evaluation results (Figure 2 indicate that the performance tends to degrade when the model-rollout length is long. We can see significant performance degradation especially in Ant-fwd-bwd and Hluannoid-dire In Ant-fwd-bwd. the performance at k = 100 is significantly worse than that at k 10. In Humanoid-ding direc,, the performance at k = 5 is significantly worse than that at k As we have seen, the performance degradation in Humanoid-diin length than that in Ant-fwd-bwd. One reason for this that the meta-model error Humanoid-direc is larger than that in Ant-fwd-bre (Figure I1 in the appendix).
is more sensitive to the model-rollout
An example of meta-policies learned by M3PO with 200k samples in Humanoid-dires is shown in Figure 3 and it indicates the learned policy successfully adapts to different tasks. Additional examples of meta-policy learned by PEARL and M3PO are shown in the video com/ www.ffll/d//// www.i//wwSSD-nnnniiiooi wttp://aee..aaaiiaaaa////aaaaaaa
at the following link:
gF
8 CONCLUSION
In this paper, we analysed the performance guarantee (and performance degradation) of MBRL in meta-learning setting. We first formulated model-based reinforcement learning in a meta-learning setting as solving a POMDP. We then conducted theoretical analyses on the performance guarantee in both the full model-based rollout and the branched meta-rollout. We showed that the performance degradation due to the meta-model error in the branched meta-rollout is smaller than that in the full
"9ttpps: / / thub. com Janne /mbpp
1he source code replicate the experiments will be open the public.
Note that, in an early stage of the training phase, there are many test episodes in which unseen tasks appear. Therefore, the improvement of M3PO over 1.2A and PEARL at the early stage of learning indicates its high adaptation capability for unseen situations.

---

Under review as a conference paper at ICLR 2021
Halfcheetah -owd
Halfcheetah-pler
WailerZDDranddmparrmms
Average return
3000
4000
8000
1000
2000
4000
50
Training samples
148
1000
2000
12A
ARL
MSPO
PEARL long
M3PD long
-1000
2000
Humanoid-direc
Ant-twd-bwd
Ant-crippled-leg
10000
8000
1500
1000
8000
1030
4000
2000
-1000
500
50
09 148 107
2000
30 99 340 397
Figure
Learning curves. In each figure, the vertical axis represents returns, and the horizontal axis represents numbers of training samples (x (000)). The meta-policy and meta-model are fixed and evaluated in terms of their average return on 50 episodes at every 5000 training samples for 1.2A and 1000 training samples for the other methods. In each episode, the task is initialized and changed randomly. Each method is evaluated six trials, and average returns on the 50 episodes further averaged over the trials. The averaged returns and their standard deviations are plotted in the figures. We also plot the best performances of PEARL and M3PO trained for a longer-term as PEARL-Long and M3PO-long. Learning curves of PEARL and M3PO in the longer-term training shown in Figure 13
Halfcheetah-pier
Walke2D-randomparaa
Halfcheetah-fwdd
2000
7000
2000
5000
1000
3000
Average
148
0
1000
Training samples
1000
50
90
148
1100
148
k-10
1-100
1-5
Ant- fwdd bwd
Ant-crippled-leg
Humanoid direc
2000
1500
5000
3000
500
1000
-1000
2200
50
99
148
197
11000
101
2010
10000
ko1-925
1001 K-10
100
www
Ks1-925
105
1-10
10000 --- 10
Figure 2: Learning curves of M3PO. In each figure, the vertical axis represents returns, and the horizontal axis represents numbers of training samples (x1000). The meta-policy and meta-model fixed and evaluated their expected returns on 50 episodes per 1000 training samples. We run experiments by varying the rollout length & of M3PO. In these experiments, the values of hyperparamters except k are the same as those described in Table [ Each case is evaluated in six trials, and the average return the 50 episodes further averaged over the trials. The averaged expected returns and their standard deviations are plotted in the figures. Dashed lines represent M3PO's learning curves Figure l:
means i values lincarly changes from x to y during leaning.
meta-model--e rollouts. Based on the theoretical result, we introduced branched meta-rollouts to policy optimization and proposed M3PO. Our experimental results show that it achieves better sample efficiency than PEARL. and I.2A.
8

---

Under review as a conference paper at ICLR 2021
Figure 3: Example of meta meta-policy learned by M3PO with 200k samples in Humanoid- The humanoid
is highlighted in accordance with task (red: move to left, and blue: move to right). The figures are time-lapse snapshots, and the first figure the left is the snapshot the beginning time. Figures show the policy successfully adapting to the task change.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, lya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. Proc. ICLR, 2018.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Leamming phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. EMNLP, 2014.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Proc. CoRL, pp. 617-629, 2018.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Picter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning. In Proc. ICLR, 2017.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph Gonzalez, and Sergey Levine. Model-based value expansion for efficient model-free reinforcement learning. In Proc. ICML, 2018.
Chelsea Finn and Sergey Levine, Meta-learning and universality: Deep representations and gradient descent approximate any learning algorithm. In Proc. ICLR, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proc. ICML, pp. 1 1126-1135, 2017.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends()) Machine Learning, 8(5-6):359 483, 2015.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proc. ICRA, pp. 3389-3396,
IEEE, 2017.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Proc. NeurIPS, pp. 5302-5311, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proc. ICML, pp. 1856- 1865, 2018
Mikael Henaff. Explicit explore-exploi algorithms in continuous state spaces. In Proc. NeurlPS, 2019.
Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori. Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. In Proc. NeurIPS, 2019.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for POMDPs. In Proc. ICMI., pp. 17-212,,, 2018.
Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Unsu- pervised curricula for visual meta-reinforcemmnt learning. In Proc. NeurlPS, pp. 10519-10531, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine, When to trust your model: Model- based policy optimization. In Proc. NeurlPS, 2019.
Taylor W Killian, Samuel Daulton, George Konidaris, and Finale Doshi-Velez. Robust and efficient transfer learning with hidden parameter Markov decision processes. In Proc. NIPS, pp. 6250- 6261, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
10

---

Under review as a conference paper at ICLR 2021
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proc. NIPS, pp. 6402-6413, 2017.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv: 1099900099533 2019.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In Proc. ICLR, 2018.
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. Proc. NeurIPS, pp. 9653-9664, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. A simple neural attentive meta- learner. In Proc. ICLR, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):599-333222 2015.
A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In Proc. ICRA, pp. 7559-7566, 2018.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments via meta-reinforcement learning. Proc. ICLR, 2019a.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based RL. In Proc. ICLR, 2019b.
Christian F Perez, Felipe Petroski Such, and Theofanis Karaletsos. Generalized hidden parameter MDPs transferable model-based RL in a handful of trials. In Proc. AAAI, 2020.
Aravind Rajeswaran, Sarvjeet Ghotra, Sergey Levine, and Balaraman Ravindran. EPOpt: Learning
Robust Neural Network Policies Using Model Ensembles. In Proc. ICLR, 2017.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning, 2020.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Proc. ICML, pp. 5331-5340, 2019.
Prajit Ramachandran, Barret Zoph, and Quoc Le. Searching for activation functions. preprint arXiv: 2017.
arXiv
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In Proc. ICLR, 2019.
Steindór Sæmundsson, Katja Hofmann, and Mare Peter Deisenroth. Meta reinforcement learning with latent variable Gaussian processes. arXiv preprint 1rXiv:1803.0733 2018.
Juergen Schmidhuber, Jieyu Zhao, and MA Wiering. Simple principles of metalearning. Technical report IDSIA, 69:1-23, 1996.
David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In Proc. NIPS, pp. 2164 2172, 2010.
Bradly C Stadie, Ge Yang. Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu. Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforreemet learning. arXiv preprint arX 2018.
Wen Sun. Towards Generalization and Efficiency in Reinforcement Learning. PhD thesis, Carnegie
Mellon University, 2019.
11

---

Under review as a conference paper at ICLR 2021
Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-basee RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Proc. COLT, pp. 2898 29998, 2019.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart
Bulletin, 2(4)) 1991.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In science & business media, pp. 3-17. Springer, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proc. IROS, pp. 5026-5033. IEEE, 2012.
Jane Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint 1r 1.05663, 2016.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint 1eXiv:1509.000 555000911199 149, 2015.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proc. CoRL, 2019.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: A very good method for Bayes-adaptive deep RL via meta- learning. Proc. ICLR, 2020.
12

---

Under review as a conference paper at ICLR 2021
A APPENDICES
A.1 HOW DOES OUR WORK DIFFER FROM JANNER ET AL.. (019)
Although our work is grounded primarily on the basis Janner et al. 2211), we provide non-trivial contributions in both theoretical and practical frontiers: (1) We provide theorems about the relation between true returns and returns on inaccurate predictive models (model returns) on "meta-learning (POMDPs)" setting (Section 5) In their work, they provide theorems about the relation between true returns and the model returns in branched rollout MDPs. In contrast, we provide theorems about the relation between the true returns and of the model returns in the branched rollout in POMDPs. addition, in the derivation of theorems (Theorems 4.2 and 4.3) in their work, number important premises is described in the second paragraph in A.7). We provide new theorems, in which the premises are more properly reflected, for both MDPs and POMDPs (A.3) A.6 A.7). (2) We extend the model- based policy optimization (MBPO) proposed by Janner et al. into the meta-learning (POMDPs) setting (Section( MBPO for the MDP settings and does not support POMDP settings, while our method (M3PO) supports POMDP settings. Furthermore, we empirically demonstrate the usefulness of the meta-model usage in the branched rollout manner in the POMDP settings (Section 7
properly taken into consideration (the detailed discussion
13

---

A.2
OUTLINE OF OUR MODEL-BASED META-RL FORMULATION
Under review as conference paper at [CLR 2021
Transfer
Meta Training
Meta Testing
Agent
Agent
Meta-policy
a
Meta-policy
h
Th (T
TT((++1lh+))
Trajectories DB
Trajectorles DB
Meta-model
0,r
Meta-model Po(oe-ilh).Po(raall
Pa(Owrllh..Pa(r.ll
Environment (for meta-training)
Environment (for meta-testing)
Observation probability
Transition probability p(T+i, P(11:11''+11 ,. s', 4,)
Observatlon probablllty p(0r.1/Te+1: S.a.: s'.+,,
T, 5'
Transition probability p(ter S s., a,)
Reward function S',
Reward function 5(,,,
Initial task probability p(tp)
Figure 4: The outline of Model-based meta-RL. formulation. Here, J a task, s' is hidden state factors, o is an observation, h a history, a is an action and r is a reward.

---

Under review as a conference paper at ICLR 2021
A.3 PROOFS THEOREMS THE MAIN CONTENT
Before starting the derivation of the main theorems, we introduce a lemma useful for bridging POMDPs and MDPs.
Lemma 1 (iiver & Veness 2010 Given a POMDP (O, S,A, Pob, Pa), consider the de- rived MDP with histories as states, (H, Y,F, P.F.).). where Vt. Phi == p(h a141, hi) = Enes Emtles (Ss)hh))((s++) a))((oc+1ss441 az) and F(he, a1) : Enes plall)))sss Then, the value function Va (h+) of the derived MDP is equal to the value function Va (hi) of the POMDP.
Proof. The statement can be derived by backward induction on the value functions. See the proof of Lemma in Silver& & Veness (2010) for details.
PROOF OF THEOREM D
Proof. By Lemma 1 problem in POMDPs can be mapped into the problem in MDPs, and then
Theorem 4.1 in Janner et al. (2019) can be applied to the problem.
Similarly, the proof Theorem is derived by mapping our problem into that MDPs by Lemma and leveraging theoretical results on MDPs.
PROOF OF THEOREM
Proof. By Lemmal I our problem in POMDPs can be mapped into the problem in MDPs, and then Theorem can be applied the problem.
A.4 THE DISCREPANCY RELYING ON THE META-MODEL ERROR IN THEOREM 1 AND THAT IN THEOREM
Corollary 1. The discrepancy factors relying on 6m in Theorem Crhi are equal to or larger than those relying on cn at k = 1 in Theorem Cm22.
Proof. By Theorem
and
2
OThlm = Tmax (1 -1-93 7)2
(4)
CT22. = Tmax 7 -
22m
(5)
Given that 7 [0, 1), Tmax and cm 0,
4-ym 272
CTh.. CTh2m
-maxx
(1 7)2
0.
A.5 CONNECTION TO A TYPICAL META-RL SETTING
In Section 4 we formulate model-based meta-RL to solve POMDP. However, this formulation may make it difficult for certain readers to comprehend the connection to typical meta-RL set- ting. Although in the normative meta-RL setting (the parameterized MDPs (Finn et al. 2017)). the objective be optimized is given the return expected with respect task distribution. such an objective does not appear in the formulation in Section In this section, we show that such an objective can be derived by specializing the formulation in Section under number of assumptions
15

---

Under review as a conference paper at ICLR 2021
(Corollary 2)
Then, explain why we did not adopt such specialization and maintained more abstract formulation Section
First, letting task set and task distribution be denoted by T and p(r) where T respectively, we introduce the following assumptions:
Assumption 1. S == O x T.
Assumption 2. p(81+1|81, ax) p(o ar) 1(T+11 = Tt) Assumption 3. For > p(s,(h,) = p(T.(h,) 1(T, = To).
Assumption 4. p(ro(ho) = p(T)
Here, 1(-) is the indicator function that returns one if the argument is true, and zero otherwise.
With these assumptions, the following corollary holds:
Corollary 2. Given a POMDP (O,S.A, A. Pob: T; Y: P:P)) and a task set T, consider the parame- terized MDP with histories states, (H, n,F, Pob), where Vt. Pob p(o To, 0z) and F = r(Ot, To,a))a))))))) as). Under Assumptions 2-5, the expected return on the parameterized MDP B.-x.h-p.rrprr) DE 7'rz] [+:]==TTTT ETE P(T P(T))an,,, EE Felelr) is equal to the expected return on the POMDP howp [R]
Proof. By applying Lemma the value function in a POMDP (0,S, Pob: v; 7: Pa) can be mapped the value function (h+) in the derived MDP, which is (H, A, 7, Pui). where Vt. phi p(hi+1 (tt1,h)) he) = Enes Emyles p(5 (ppp))p((ss)))) (p ( 4x) and i(he at) == Enes ((s))h))r(ss, ac).
By applying the assumptions, this value function can be transformed to a different representation that explicitly contains and its distribution:
For > 0,
V=(h+) = Σ T(a++1)he)
Σ
p(sslhiarsss
S1ES
Σ Σ
Σ
ar) Vx) (be+1))
a 111 81 8.E8 8111ES
Σ T(u)+1l4))
r(ur, To, at) +7 Σ
plo To, ae) Va (hett))
01I1E0
Pob
Likewise, for 0,
V=(ho) = > hoal)ho) Ex(ror(on To) +7 Σ Σ
P(To)p(orlon To, ao) Vz)
41
01E0 To
(8)
= Σ p(7)) Σ x(ailho) r((0 r(00, ao) + Σ
plo1100. To, Va 00
V+(ho)
Therefore,
Eg hopp [R] =
Σ p(ho) P(ho)) Vz(ho)
Σ p(7)) Σ C((ho) To
ho
= Earn Hupp...
7'7
16

---

Under review as a conference paper at ICLR 2021
Gamma=0.85 85
Gamma=0.0
Gamma=0.95
Gamma-0.85 85
Cammp=0.0
Cammaa00.05 0.95
230
2000
2000
1.90
330
220
20
23
20
10
10
250
10
200
50
65
1.0
0.0
0.5
0.0
0.5 1.0
0.0
0.0
0.5
0.0
05
essionn
eyolloraar
(a) Discrepancy in Theorem
(b) Discrepancy in Theorem
Figure 5: Discrepancy between BTg.. [R] and R[a..)))PPPaa model
[R] in Theorem 2 and that Theorem Each
vertical axis represents k E [1, 50] and each horizontal axis represents Cmuy Cm² E [0, 1]. In all Tigures, for evaluating the discrepancy values, we set the other variables as r'ax 1, En and 6 1 - emv for (b). A key insight with the figures is following two points. First, discrepancy values in Theorem (a) tend to increase as the value increases, whereas those in Theorem 3 (h) are not. Thus, there could be an optimal la value that larger than one. Second, the contour colours in (h) are paler than those in (a). This implies, given em Eml, Theorem provides tighter bound of the performance.
c. for (a).
By Corollary 2 our formulation in Section can be specialized into a problem where the objective to be optimized is given as the return expected with respect to a task distribution. We can derive the meta-model returns with discrepancies for bounding the true return (i.e., Ba- r-p(() IE: x y'frl Clem, )x by using Corollary 2annllary[instaad instead of Lemma I in the proofs of Theorem D and replacing (o oo. To, dz) with (o 114-1).
The main reason that we do not adopt such a specialization in Section is avoid restrictions induced by the assumptions (Assumption 2~5)). For example, Assumption 2 states that hidden states are composed by observations and a task. Since observations can be observed from an agent by its definition, the information in the hidden states that cannot be observed from an agent is the task, However, in many meta-RI. settings (e.g., application domains (Jabri et al.) 2019) where video images are used as the observation), there should be other information that cannot be observed from the agent. In addition, Assumptions and state that the task is invariant in each episode. However, in many meta-RL settings (e.g.. g..aaaaanddi et al. (219a b)), the task can change to different one in each episode. To avoid such restrictions, we decided not to specialize the formulation by introducing the assumptions.
A.6 ADDITIONAL ANALYSIS OF THE PERFORMANCE DEGRADATION IN THE BRANCHED RETA-ROLLOUT
Figure 5a shows that the discrepancy value in Theorem 2 tends to monotonically increase as the value of l: increases, regardless of the values of r and Gr-- This means that the optimal value of ks is always 1. However, intuitively, we may expect that there is an optimal value for k that is higher than when the meta-mode error is small. As will be shown in this section, this intuition is correct. One of the main reasons for the mismatch of the intuition and the tendency of the discrepancy in Theorem 2 is that, for its analysis, the meta-model error on the data collection meta-policy Ta (i.e., em. is used instead of that on the current meta-pont meta-policy Tg. This ignorance of the meta-model error on the current meta-policy induces the pessimistic estimation of the discrepancy in the analysis (see the evaluation of "term B" and "term C"' in the proof of Theorem via the proof of Theorem 2 in the appendix for more details).
To estimate the discrepancy more tightly, as in Janner et al. (2019) we introduce the assumption that the meta-model error on the current meta-policy can be approximated by ex and em: Assumption 5. An approximated meta- meta-model error on a current policy Cmri Cm (mx) a Cm +
dem' where dem'. is the local change of enp with respect to ex- des
Gx de,"
17

---

Under review as a conference paper at ICLR 2021
To see the tendency of this approximated meta-model error, we plot the empirical value of dem' /de varying the size of training samples for the meta meta-model in Figure [ A.12 The figure shows that, as the training sample size increases, the value tends to gradually approach zero. This means that training meta-models with more samples provide hetter generalization on the nearby distributions.
Equipped with the approximated meta-model's error on the distribution of the current meta-policy Th. we arrive at the following bound:
Theorem 3. Let 6m² 2 max, Eaw hhpp Wrv (p(h))1,, a)llpo (k'), a])]
72
22x + 7 k7t + (k- 1)7
Ex [R] 2 B(a.A)) [R]]
T Tmax
(1 71-
(1 - 71-
(cm - Ex) (10)
+ 2h ?
7 (Em - 6x) + (k + 1) - 60)
2
1 l-7
Proof. By Lemmal I our problem in POMDPs can be mapped into the problem in MDPs, and then Theorem A.7 can be applied to the problem.
Given that cm Em², it is obvious that the discrepancy in Theorem 3 is equal to or smaller than that in Theorem 2 In the discrepancy in Theorem Bl all terms except the first term become negative when 6y < Gy. This implies that the optimal & that minimizes the discrepancy can take on the value higher than 1 when the meta-model error is relatively small. The empirical trend of the discrepancy value (Figure 5b) supports it; when ém is lower than 0.5 (i.e., 6m/ < )x), the discrepancy values decrease the value of l: grows regardless of the value of y. This result motivates to set k the value higher than 1 in accordance with the meta-model error for reducing the discrepancy.
A.7 THE DERIVATION OF THE RELATION OF THE RETURNS IN k-STEP BRANCHED ROLLOUTS (k: 2 1) IN MARKOY DECISION PROCESSES
In this section, we discuss the relation of the true returns and the model returns under the branched rollout in an MDP, which is defined by a tuple (8, AA, r, Y, Ps)). Here, the set of states, A the set of actions, Ps. := p(s'ls. a) is the state transition probability for any s' E S and a E A. is the reward function and 7 [0, 1) is the discount factor. At time step f, the former two functions are used as p(se(st-1, 0:), Ft = r(st, oz). The agent selects an action on the basis of policy := p(a++1] (ss). We denote the data collection policy by TD and the state visitation probability under TD and p(s'|s, a) by Pxp (st). We also denote the predictive model for the next state by pa(''|s, a). In addition, we define the upper bounds of the reward scale as T'mx >> maxx,a |r(s, a)}- Note that, in this section, to discuss the MDP case, we overriding the definition of the variables and functions that were defined for the POMDP case in the main body. In addition, for simplicity, use the abbreviated style E. [R] for the true return Bawn,sng
R = Ei
Although the theoretical analysis on the relation of the returns in the MDP case is provided by Janner et al. (2019), in their analysis, number of important premises are not properly taken into consideration. First, although they use the replay buffers for the branched rollout (i.e. datasets Dem and Dmuntel in Algorithm in Janner al.] (2019)) they do not take the use of the replay buffers into account in the their theoretical analysis. Furthermore, they calculate state-action visitation probabilities based solely on single model-based rollout factor. In the branched rollout, state- action visitation probabilities (except for the one at t 0) should be affected by multiple past model- rollouts. For example, state-action visitation probability at (s.t. & > k) is affected by the model-based rollout branched from real trajectories at f - k and ones from t - A: + 1 to (in total, & model-based rollouts). However, in their analysis (the proof of Lemma B.4 in Janner et al. (2019). they calculate state-action visitation probabilities based solely on the model-based rollout. For example, in their analysis, it is assumed that a state-action visitation probability at (s.t. t > k) is affected only by the model-based rollout branched from real trajectories at t k. These oversights of important premises in their analysis induce a large mismatch between those for their theorems and those made for the actual implementation of the branched rollout (i.e., Algorithm 2 in
18

---

Under review as a conference paper at ICLR 2021
PT (Sc ap)
D
modke
3
4
8:0.1
Uniform sampling
Storing
pbr EER (5p
Figure 6: An example of branched rollouts with l: = 3. Here, the blue dots represent trajectories contained in environment dataset Dew and the yellow dots represent fictitious trajectories generated in accor- dance with current policy under predictive model pa.
Janner et al. 2019
13
Therefore, we decided to newly derive the theorems the branched rollout,
reflecting these premises more appropriately.
The outline of our branched rollout is shown in Figure 6 Here, we assume that the trajectories collected from the real environment are stored in dataset Deny The trajectories stored in Don trajectories following the true dynamics p(s'ls, a) and data collection policy (i.c., mixture of previous policies used for data collection) Tp. At each branched rollout, the trajectories in Denv uniformly sampled [4] and then starting from the sampled trajectories, k-step model- based rollouts in accordance with æ under Po is run. The fictitious trajectories generated by the This process more appropriately reflects
be
branched rollout is stored model dataset Dmodel the actual implementation of the branched rollout (i.e., lines 5-8 in Algorithm 2) in Janner et al.) (2019) 16 The performance of is evaluated the expected return under the state- action visitation probability in model.
Thee mismatch becomes larger especially when the model-rollout length & is large because state-action visitation probabilities are affected by these rollouts more significantly with large "Thus, the initial state prohability for the rollout starting from the trajectories follows Pap (s) Here, when the trajectories information to deal with the state transition depending on time step.
stored in Dandel: the states in the trajectories are augmented with time step
Note that the extension of this process the POMDP case is compatible with the implementation of the branched meta-rollout in algorithm (lines 4-13 in Algorithm 2
19

---

Under review as a conference paper at ICLR 2021
Formally, define the return in the branched rollout ((a,a)-mmoste [R] as:
k-1
[((,,)) [
Σ
Pop (80, (n)r ao) + Σ E
I' (
50.00
Σ
>' (ee
(11)
[-1
(1,
br
(81, az)
(12)
t
(8),
Pizki (8t, at)
(13)
k
(st,,
be
Σ
Σ
Pen
(14)
p (8t, 0z)
Σ
Σ
0f 2
Pxp (st -ل))) po po ((+1))(()+115))))
(15)
Here, (St, az) and p (8t, trajectories (node) from the bottom at each timestep & in Figure 6 follow. In later discussions, for simplicity, we use the abbreviated style REppodeel [R] for Ex(a,)> Poodel R
de) are the state-action visitation probabilities that the i-th yellow
Before starting the derivation of theorems, introduce a useful lemma.
Lemma 2. Assume that the rollout process in which the policy and dynamics can be switched to other ones at time step taw. Letting two probabilities be Pi and p2. for < t' I taw, we assume that the dynamics distributions are bounded as Cm.pre tww t'
In addition, for we assume that the dynamics distributions are bounded as Cm Z maxe Earm Drr (p1 (8t s1-1, ar) (alll/2(se (1 Likewise, the policy divergence is bounded by pre and post. Then, the following inequation holds
maxe Espei [Dry (pr (. Mlp22lspss 11
Σ (s)) p2()))))) S 2(t two (em + Empua) 20 xxpor)) (16) 8e:0e
20

---

Under review as a conference paper at ICLR 2021
Proof.
The proof is done in similar manner to those of Lemma B.1 and B.2 in Janner al. 2019)
Σ R2
pr (st, a4) - P2(se, ar)l
Σ
- pp(()aaaa)))
- (a Σ pelcaalllslaaa--aaaaaaplllllle + (pi(as) p2(ax))la(sslla))
Σ p(())) (sr(ae))) [p((saaa))))
Σ (a,) -p2(ar)
Σ (i() pi((c(ae) -2((s (p-(0) - p2(0 8t-1)
Re
Re,6e-1
E St.Rt
+ Σ hnls--Dalaos-------o-hbbaaalllll + (pi(s(-1) pe(sL-)))pp(asss-:))
Gr.8t-1
Σ (1( p((l)))))))))pp((aaa)))))) Σ
+ Σ -P2ss-))
St-1
Σ (i())) (sclar) p2(selat)1+ +
Σ
p2 (arl
01184-1
(21(5--1,0-11) - P2($(-1180-1))
< 20 + 2.
(81-11 p2 12 81-1))
11-1
< 2(1 (mw) (Cm.poot + est (sr.post) +
Σ
Ip1 (Sess: arow) - (8 Stsww) )
Rigg
< 2(t tww) (Gmppost + Cr.posa) + 2t (em,poc + Ex.,ue)
(17)
Now, we start the derivation of our bounds.
Theorem 4. Under the k-step branched rollouts, using the bound of model error under wp. Em max, Ea-wo, [Dry (p(s')s, (s, (5,))ll (sppps''sss a))l and the bound of the policy shift en max, Drv (I (T]] the following inequation holds,
+ 22
7 ky
+ (k: 1)x+1
E Ex,pp [ 2 D.Daadd [R] Tmaxx
26 +
(En + +m) (18)
(1 7
(1 1]-
+ *
?
x
(k: + +1(((z + tm))
(Gx + 6m)
+
-
1 I-
21

---

Under review as a conference paper at ICLR 2021
Proof.
Em (Pr(so, ao) - -xx (s0. ao)) r(su. ao) Eana of {pa (st, at) p (st ar) r(st: az)
B (R]
Ep madel
R
t= Emnor 96th Pr (r) 01) - th (5 ar) r(s,, ai)
Exo.ae Pr 00) Pxp (80, an) [ an)l
+ 21=1 7dt Lisiae Pr (St 0z) - pi (8t, az) (r(se, az)l
Eizk? o Esna Pr (st as) P ai) (r(ss, at)l
F mmax
(su an) - Pan (su, an)l
$0.40
term A
max
Eis
(8t,
P (8e,, ac)l
(19)
+mmax Ltck 7d
br |p (z)
Pizk (8ts az)l
ReiRe
term
For term A, we can bound the value in similar manner to the derivation of Lemma 2
Σ 50.40
1Pr (50; uo) Prp (so, ao)l
=
Σ
Pr (p(()))((0)) - Pro (ao))((()))
#o
Σ
Pr (po()))p ((()) Pr (roo)))(5)) (p=(ap) pup (an)) p(so)]
40.00
Pa (ao) (p(s0) p(so) Σ
1px (ao) Pwo (ao))
60.00
26
(20)
For term B, we can apply Lemma to bound the value, but requires the bounded model error under the current policy Thus, we need decompose the distance into by adding and subtracting Psp:
Px (Se, az) - Prp (St, az)
Σ (pr az) (8t,) $2,00
Σ
+Pop (81, (t) - p p at)
Pa (sr, az) Pro (1),
Sc.0c
Sate,
hr phee Pro (St, ax)
(21)
Σ
Σ Se.at
E Pro (st,aa)= - El-D
Pro (St: 01) br (Sc,, ar)
(4)
<
Σ (2(1-i) -) (er +mm)
f (f(er + em) + t((c + mm)
(22)
22

---

Under review as a conference paper at ICLR 2021
For (A), we apply Lemma 2 with setting Cm.post Cm and cx.post cx for the rollout following
and po (s'ls)) and 6m.pre = and Ca for the rollout following ap and p(s"(s), respectively. To recap term B, the following equation holds:
(prr(sr, at) - ( S
2te, t(er + em) + (ex + Em)
(23)
Pe
For term C, we can derive the bound in similar manner to the term B case:
Pa (st, at) - PTp (st, 0z)
Σ ( at) Po,,
Σ
tPT ($,, a1) (1)
- P (s), ac)
br
Pr (5t, at) Pro (3 (+),
te.de
<2tex
PPn (st, a1)
br (12 ar)
(24)
Σ
Pao (8t, - -
br (86,
= Σ
Di PED (st,
Et-o br (s,
St
1
Pno (sts ax) pbr (s,,
tc.0c
k-
Σ (2 i) (Ex + em))
1=0
= [k2(ez + (m) + k((e + 6m)
(25)
To recap term C, the following equation holds:
Σ (s a4) - -00 -122 ar)
<
2te, + k(ez + em) + (ex + em)
(26)
94.04
23

---

Under review as a conference paper at ICLR 2021
By substituting Eqs.
20
23 and
26 into Eq.
19 we obtain the result:
m'mx
Ex.p
[R]
ED model
R
+ 2 7²t {2te t(cr + cm) + (ez + cm]}
55 -
+Tmax 21= 7' {21ez + k(tr + fm) + (ex + {m)
2xx + 4(0
(3ez + fm) + - (er + cm)
T'ax
(1-2)²
+ + 7 (2tc, + k(c, + (m) + (cr + mm)
2 + 1-6y(1-1)+ (1---))+1111 11 ? (367 + (m) -- ((z + (m)
!1-2)2
7-1
T
maax
+ + 7' 12lez + k(er + +m) + (tr + {m)
-- 71 {2tr, + k(c, + (m) + (c + mm)
1- 1-ky(k-1) +(k -1)= ? (36, (m) - (cx + 6m)
26
(1-1)²
Tmax
+ [1 2) 7e + - (k(er + +m) + (ez + +m))
2
Σ Ei=1 72 {2te; + k(cz + 6m) + (cx + Gm)
2z + 1 11- 112² ? (3rr + (m)
"ー (cx + (m)
(1-
7-1
+ (1-933 TC + |- [k(CT + +m)
7
+ (Ex + cm)}
max
1 1- + (1- 117 27Exx
(1-71=
بー (k(c Cm) (cx + mm)
[1-7)² 2c + -- + (cx + cm)
1+7²
(1-7)²
Tmax
-ー 7-1 (Ez + cm) + ( - (k+ + 1) +
1+2 (1 22x + I I +(6-))))) (ex + cm)
Tmax
(1-7)3
(27)
(Cr + Cm) + 1- (ks 1) + cm)
Theorem 5. Let 6m
6m 2 max, Frans DTr (1)00/(s)))l0(((
1 1+2²
1 1 10- + (k - 1)md 7(mm' - (28)
E Ex.p
R 2 ED model [R]
Tmax
(11 7)2 2cn +
(1 71-
+ 7h 71 (Em Ex) 7
I- (k (k+ + 1)((m - -mn }
7ht
6x) +
Proof. The derivation of this theorem is basically the same as that in the Theorem case except for the way of evaluation of the bound of terms and C.
For term B, we can apply Lemma
2 to bound the value:
Σ (st, az) pbc (st, ax)
Eiz Por (Sc, at) Phe
Σ (pr( - -SE ar)
t 1=0 Seiae
(D)
1 1-1
Σ {(t 1)26ml + 12cr}}
1
(2t em - (t - 1)tem' + (t - L)tex }
{
1
7 e - Peem + bem + Pen - lex }
1
7 {E(em + Exx + bl(mm - Ex)
(e6m + (x) + (em Ex)
(29)
24

---

Under review as a conference paper at ICLR 2021
For (D), we apply Lemma 2 with setting cm. Em² and Cr.post 0 for the rollout following and po(s"1s) and Cm.pre = and Ex.pre = 6x for the rollout following To and p(s
For term C. we can derive the bound in similar manner to the case of term B:
Σ ( a1) br (())
Σ B1,0e
b Ei=o Pao (80, ut) Ei phe (
Pr (r ar) - hr -1, at)
1=
91:41
S k
{(k: 1))mml + (t 1)267}
1
{ 22k Em (k - 1)kem + 2thex - 2222Er + (K - 1)ken} }
k
1
{ (2ktex + 2(Emm - en - (k- 1)k(Cm - =xm}
k
1
{2ktex + (em' - )k2 + +hemmm - Ex)
(30)
k
By substituting Eqs.
20
29, and
30
into Eq.
19 we obtain the result:
2e + I4-I 72t {M(Em + +x) + (emm - cx))
E Fxp
[R]
E mmodel
[
I' mmax
+ Di=k 7t iF (2ktex + (em' - -xx)h² + k(em' 1 (xm) ex))]
2e + 1-kg (1-7)²
7(Em + +() + 2 ( - (7)
Tmax
Dt=k 702 {2kte, + (cm - cx)h²² + 'k(mmm - -xx)
2c + 1- 11-(0-11-(-1 7(6m + (7) + 7-1
"" - (7)
(1-))²
T'mx
+ Σ 000 7/
x [2ktex + (Em - En)h² + K(Emm - -G7) }
(2ktez (Em C, k²² k(Cm' (r)
-1=1
22x + 1- 1-b)1--1)-(k (1t-1) 7(m + Ex) + 2 2=
(Em - Ex)
1 [1-
7-1
+ (I-7)z 27 + T {(mm (7)hk + (Fml - =x7)
Tmax
1-k (4-1) + 27Ex
(1-7)²
メー 7-1
[(Em3 -x)K (Em Ex)}
(1-7)7 2cg + (1-7)2
14,²
(Em' Ex)
= l'mxx
7 7-1 (Cm - -c7) + -1- - - (k+ 1)(cm - -G))
2c, + I- - 6+1
(Em En) + T-m (k + 1) - Ex }
(cm - Ex)
=
Tmax
(1-2)²
(31)
A.8 THE RELATION OF RETURNS IN THE CASE WHERE A REWARD PREDICTION INACCURATE
In Section 5 we discuss the relation between the true returns and the returns estimated on the meta- model under the assumption that the reward prediction error is zero The theoretical result under this assumption is still useful because there are many cases where the true reward function is given and the reward prediction is not required. However, number of readers still may want to know what the relation of the returns is under the assumption that the reward prediction is inaccurate. In this section, we provide the relation of the returns under inaccurate reward prediction in the MDP case
Here, we do not discuss the theorems in the POMDP case hecause those in the MDP case can be easily extended into the POMDP ease hy utilizing Lemma T
25

---

Under review as a conference paper at ICLR 2021
We start our discussion by defining the bound of the reward prediction enror c,:
Definition 4. Cr = max; ((a.,e))~PPooll model [Ir(st, a+) ro(ss, az)]], where ro(st, at) Erc-po Iiv ae, 8e]-
=
We also define the return the branched rollout with inaccurate reward prediction.
A-1
B model
[
=
Σ
Pro (sv, (o) an)
Σ
E
(s, arrooarrrrs...... at)
$0,00
Se.
Σ t= 82,0r
7 p (i2 ar) a))ro(ss,
(32)
Now, we provide the relation between the returns under inaccurate reward prediction.
Theorem 6 (Extension of Theorem into the case where reward prediction inac- curate). Under the steps branched rollouts, using the bound of model error un- the bound of the policy
der TD. cm max, Eaway, Drv (p(s' ( shift En max, DTv (π][TT) and the bound of the reward prediction error e, max, p(a,:a))-Poole [|((() 0e) - (r( a.)ll. the following inequation holds,
+ 72
J -- ky + (A: 1)y
E.. R 2 Ep
modell [R]
26
(ez (m) (33)
mmax
(1 - 7)
(1 (1-
**
(cz Cm)
(k +)((( + +mm)
1-7 cr.
1
--9
Proof.
E m.
R
ED
model
[ = B...pl R
Ex model
[R]
+
E model
[
- E
model
[
(34)
x E..pp R
Ex Exmodte model
R|
+
Ex
model
[
ED
model
[
Expoas Puo (80, ao) (r(so, no) re(s0: an)}
ED mmmaell
[
Ey model
[
si, "p fr(ss,, at) - r5,, ar}
1=k En.o. I'p az) (r(8t, 0z) rs az)
Σ Prp (s0, an) |r ao) ro(so.
Sp 9p.
Σ 7 T'D (Se, az) [r(s, ar) - ro(se, ar)
(35)
t= 8e ae
8
Yrh (s) (r(s,, ar) - re(s),
t= 9600
Er
t=
By substituting Eqs. 27 and 35 into Eq.
34
we obtain the result:
2e + 7 14 (Ex + Em)
Er.pp [R] Ep. momeeel [R]
< Tmax
(1 (1-7)0
+
?
(36)
(Cr + cm) + =(c (k + 1)(c7 + cm)
1 1-n
26

---

Under review as a conference paper at ICLR 2021
Theorem 7 (Extension of Theorem
5 into the case where reward prediction is
inaccurate). Let Cm/ 2 max, Sawn, [Dr (p(s"|s, ()) and er
max, B(ais,)-D modd [(( ar) ro a.)l,,
mmax { 1 9
(1 21- 2ex + ? -- ky + (k: 1)ght1
Ex Ex.p [ 2
ED
model [R]
(Em - -xx) (37)
(1 7)2
x
7
1 1- (k: 1)((mm (aan--r))) cx)
7
(Gm' 6x)
1- 7 cr-
Proof. Similar to the derivation of Theorem 6 obtain the result by substituting Eq. and 35
into Eq. 34
A.9 PEARL IN SECTIONS G AND
The PEARL algorithm used in Sections and refers to "PEARL with RNN-traj" in (Rakelly al. The comparison result with other types of PEARL (i.c., vanilla PEARL and PEARL with
2019).
RNN-tran) in (Rakelly al. 2019) and M3PO are shown in Figure M3PO achieves better sample efficiency than them.
The figure indicates that
Halfcheetah bowd
Halfcheetah-pier
Walker2D-randomparams
2500
2000
6000
1500
500
1000
-500
50
99
145
197
2000
-TEANL
TEARL-trmn
M3PO
-2000
Ant-fwd-bwee
Ant-crippled-leg
500
0
-500
11000
50
99
145
197
50 140 197
Figure The learning curve of vanilla PEARL (vPEARL), PEARL with RNN-traj (PEARL-tran), M3PO, and its variants in which the vPEARL and PEARL- tran are used for meta-policies. In each figure, the vertical axis represents expected returns and the horizontal axis represents the number of training samples (x1000). The meta-policy and meta-model (and predictive model) expected returns were evaluated 50 episodes every 1000 training samples for the other methods. In each episode, the task was initialized and changed randomly. Each method was evaluated at least three trials, and the expected return on the 50 episodes was further averaged over the trials. The averaged expected returns and their standard deviations are plotted in the figures.
fixed and their
A.10 BASELINE METHODS FOR OUR EXPERIMENT
The model-free meta-RL method proposed in Rakelly et al. (011)). This is an off- policy method and implemented by extending Soft Actor-Critic (Haarnoja et al,, 2018). By leveraging experience replay, this method shows high sample efficiency. We reimplemented the PEARL algorithm on TensorFlow, referring to the original implementation on PyTorch (htpps: //github. com www.kkatteeelllyae wow./.ateeraklll//yysteerrr
PEARL:
Learning to adapt (L2A)) The model-based meta-RL proposed in Nagabandi et al. (001aa). In this method, the meta-me model is implemented with MAML (Finn et al., (017) 7) and the optimal action is found by the model predictive path integral control (Williams et al. 2015) the full meta-mode based rollouts. We adapt the following implementation of L2A to our experiment: avera na__t_adapt_p
https: // tww.. www..//wwwwwwa..aaoaoa/a.aa
27

---

Under review as a conference paper at ICLR 2021
A.11 ENVIRONMENTS FOR OUR EXPERIMENT
For our experiment in Section 7, we prepare simulated robot environments using the MuJoCo physics engine (Todorov et al. 2012) (Figure
Halfcheetab--wd--bved. In this environment, meta-policies are used to control the half-cheetah, which is a planar biped robot with eight rigid links, including two legs and a torso, along with six actuated joints. Here, the half-cheetah's 's moving direction is randomly selected from "forward" and "backward" around every 15 seconds (in simulation time). the half-cheetah moves the correct direction, positive reward is fed to the half-cheetah in accordance with the magnitude of move- ment, otherwise, a negative reward is fed.
Halfcheetah-pier: In this environment, the half-cheetah runs over series of blocks that are floating on water. Each block moves up and down when stepped on, and the changes in the dynamics are rapidly changing due to each block having different damping and friction properties. These proper- ties are randomly determined at the beginning of each episode.
Ant-fwd-bwd: Same Halfcheetah-fwd-bee except that the meta-policies used for controlling the ant, which a quadruped robot with nine rigid links, including four legs and a torso, along with eight actuated joints.
Ant-crippled-leg In this environment, we randomly sample leg on the ant to cripple. The crip- pling of the leg causes unexpected and drastic changes to the underlying dynamics. One of the four legs is randomly crippled every 15 seconds. Walker2D-randompan randompaaams::
this environment, the meta-policies are used to control the walker, which is a planar biped robot consisting of seven links, including two legs and torso, along with six actuated joints. The walker's torso mass and ground friction randomly determined every 15 seconds.
Humanoid-diree In this environment, the meta-policies are used to control the humanoid, which biped robot with 13 rigid links, including legs, two arms and torso, along with 17 actuated joints. In this task, the humanoid moving direction is randomly selected from two different direc- tions around every 15 seconds. If the humanoid moves in the correct direction, positive reward is fed to the humanoid in accordance with the magnitude of its movement, otherwise, a negative reward is fed.
Left: Halfcheetah--fwd-bwod Center: Halfcheetah-pier, Right: Walker2DDranddopaarrss
Left: Ant-fwd-bwd, Center: Ant-crippied-leg Right: Humanoid-direc
Figure 8: Environments for experiment
28

---

Under review as a conference paper at ICLR 2021
A.12
COMPLEMENTARY EXPERIMENTAL RESULTS
1600
Gamma=0.85
Gamma=0.9
1400
Gamma=0.95
1200
Discrepancy
1000
800
600
400
200
0
0.0
0.2
0.4 epsilon_m
0.6
0.8
10
Figure 9: Discrepancy between Eug.p [R] and Eag-po [R] in Theorem
The vertical and horizontal axes
represent the discrepancy value and cm E [0, 1). respectively. -
We set the other variables as ea
Em and Faax
1.
Halfcheetah fwdd bowdd-dd
Halfcheetah pier
Walker2D randomparamo
deml (d6e
20
Sample size
0
30
40
50
10
20
40
50
Ant-fwdd bwd
Ant-crippled leg
Humanoid diree
10
1.5
20
40
60
80 100
20 40 60
80 100
100
150
200
Figure 10: The local change in 6m with respect to é, versus training sample size. In each figure, the vertical axis represents the local change of the meta-model error ((m")) and the horizontal axis represents the training sample size (x1000). The red-dotted line is the linear interpolation of the blue dots, which shows the trend of the local change decreasing as training sample size grows.
29

---

Under review as a conference paper at ICLR 2021
He Helfcheetahh-wo--bed
Halfcheetah-pier
Walker2D randomparams
10
28
10
10
37
10
19
28
46
Training samples
-10
Trink2 minl--3
Tril- Trars
Tri--1 --1---
71002
Try-3
T1a-- -T1a15
Trial -T---Tikk Tril
Trl
Tiaa -Tia15
Ant fwd
crippled log
Humanoid direc
37
-8
Traf-1
T-ah-5
Trial
T1015
Trll
Figure Transition of model errors on training. In each figure, vertical represents empirical values fm and the horizontal axis represents the number of training samples (x1000). ran five trials with different random seeds. The result of the æ-th trial is denoted by Trial-z. We used the negative of log-likelihood of the meta-model on validation samples the approximation of e The figures show that the model error tends decrease epochs elapse.
Halfcheetah pierr
Walker-D-randdoparans
Halfcheetah- fwd- bwd
200
500
200
150
200
100
50
100
10
19
Training samples
37
16
Triel-
Tria-2
inttfwud-bwd
Ant-cripled-leg
Humanoid-dires
300
300
200
200
100
100
Figure 12: Transition of meta-policy divergence on training. In each figure, the vertical axis represents empir- ical values of c, and the horizontal axis represents the number of training samples (x1000). We ran three trials with different random seeds. The result of the z-th trial is denoted Trial-z: For cx. we used empirical Kullback-Leible divergence ms and mp. Here, ap has the same policy network architecture with To and is learned by maximizing the likelihood actions in (D for Al Algorithm 2.
30

---

Under review as a conference paper at ICLR 2021
Halfcheetah-fwwe bwd
Halfcheetah-pier
Waller2D-randomparamm
Average return
2000
6000
10
19
37
46
4000
8000
Training samples (x50k)
2000
2000
PEARL-1 PEARL
3
M3P0-1--a-- --P..-----------. 3
-2000
10
19
46
Ant-fwd-bwwl
Ant-crippled-leg
Humanoid-diiine
3000
1500
10000
500
4000
2000
-1000
10 30
10
Figure 13: Learning curve of PEARL and M3PO in a long-term training. In each figure, the vertical axis represents expected returns and the horizontal axis represents the number of training samples (x50000). The meta-policy and meta-model were fixed and their expected returns were evaluated 50 episodes at every 50,000 training samples. Each method was evaluated three trials, and the result the x-th trial denoted by method-z. Note that scale of the horizontal axis is larger than that in Figure I by 50 times (i.e., 4 in this figure equal to 200 in Figure
Halfcheetah-fwd-bwwr
Halfcheetah-pier
Wailer2D-randomparrms
2000
6000
1000
4000
2000
-1000
50
99
1.08
197
50
148
Trial-1
Trial-2
M3PO
Triak-1
Triak2
M3PO
Trial-1 Trisk-
M3PO
Ant-fwd-bbwd
Ant-crippled-leg
Humanoid-direc
1500
1000
4000
500
500
2000
50
-500
-500
Trial-1
Trinl-2
MIPD
Trial11
Trinl-2
MIPO
Trial-1 Trisk-2
MIPO
Figure 14: Comparison of GHP-MDP (Algorithm in Perez al. (2020)) and M3PO. The figures show learn- ing curves GHP-M HDP M3PO. In each ligure, the vertical axis represents expected returns and the horizontal axis represents the number of training samples (x1000). GHP-MDP was eval- uated in two trials, and each trial was run for three days in real-times. Due to the limitation of computational resources,
could not this experiment many days and trials other exper-
iments. The expected returns GHP-MDP each trial (denoted by "Trial-1" 1" and "Trial-2") plotted in the figure. The results M3PO is referred to those Figure [[] From the comparison result, we can see that M3PO achieves better sample efficiency than GHP-MDP.
31

---

Under review as a conference paper at ICLR 2021
Helfcheetat-fww--bee
Halfcheeta piee
Wallee2D-randdoparems
3000
2000
8000
2000
1000
1000
Average
2000
Training samples
50
1401
187
50
90
148
107
M2PO
M3PO
M2PO
MBPO
M2PO
M3PO
Humanold-direc
Ant-
Ant-crippled-leg
10000
3000
2000
1500
-1000
50
148
197
50
00
148
197
-2000
-5000
M2PO
MIPO
M2PO MSPO
MMPO -MPP
Figure 15: Learning of M3PO and M2PO. In each figure, the vertical axis represents expected returns and the horizontal axis represents number of training samples (x (x1000). The meta-policy and meta- model (and predictive model) were fixed and their expected returns were evaluated on 50 episodes at every 1000 training samples for the other methods. In each episode. the task was initialized and changed randomly. Each method evaluated in least five trials, and the expected return the episodes was further averaged over the trials. The averaged expected returns their standard deviations are plotted in figures.
Humanoid -iree
100
Average TD-error
80
60
40
20
17
25
33
41
49
57
65
73
81
89
97
105
113
121
129
137
145
153
161
169
177
185
193
201
Training samples
M2PO
M3PO
Figure 16: Transition of TD-errors (Q-function error) training. In each figure. the vertical axis represents empirical values 6m the horizontal axis represents the number training samples (x1000). We ran ten trials with different random seeds and plotted average of their results. The error means one standard deviation.
32

---

Under review as a conference paper at ICLR 2021
A.13 COMPLEMENTARY ANALYSIS
In addition to QI and Q2 in the main content, we also conducted a complementary analysis to answer the following question. Q.3: Does the use of meta-model in M3PO contribute the improvement of the meta-policy?
In analysis in this section, we compare M3PO with the following method. Model-based Meta- Policy Optimization (M2PO): This method is variant of M3PO, in which a non-adaptive predic- tive model is used instead of the meta-model. The predictive model architecture is the same that in the MBPO algorithm (Janner et al. 2019) (i.e., the ensemble of Gaussian distributions based on four-layer ferd-forward neural networks).
Regarding Q3, our experimental result indicates that the use of meta-model contributed to the performance improvement in a number of the environments. Figure 15 the appendix, we can clearly see the improvement of M3PO against M2PO in Halfcheetah-fwd-boe Ant environments, although the M3PO's performance seemingly the same as that of M2PO, the qualitative performance is quite different; the M3PO can produce meta-policy for walking in the correct direction, while M2PO failed to do so (M2PO produces the meta- meta-policy "always standing" with very small amount of control signal). For Humanoid- achieve better sample efficiency than M3PO. We hypothesize that the primary reason for this that during the plateau the early stage of training in Humanoid-direc, the predictive model used in M2PO generates fictitious trajectories that make meta-policy optimization more stable. To verify this hypothesis, we compare TD-errors (Q-function errors) during training, which an indicator of the stability of meta- peeccy optimization, in M3PO and M2PO. The evaluation result (Figure [6] in the appendix) shows that during the performance plateau (10-66 60 epoch), the TD-error in M2PO was actually lower than that in M3PO; this result supports our hypothesis. In this paper, we did not focus on the study of meta-model usage generate the trajectories that make meta-policy optimization stable, but this experimental result indicates that such study is important for further improving M3PO.
In addition, in the
in contrast, M2PO tends to
A.14 HYPERPARAMETER SETTING
Table 1: Hyperparameter settings for M3PO results shown in Figure - y epohs b denotes =), x).
thresholded linear function, i... at epoch f(e) min(maxxe + e-e
Walker2D-ran randomparams
Halfchectah- fiaill--ww-bwdddd howd
Halfcheetah--ie pier
Ant-crippled-leg
Humanoid-dires
Ant-fwd-bwo
N
epoch
200
E
environment step per epoch
1000
M
meta-model rollouts per environment step
1e3
5e2
1e3
5e2
ensemble size
3
G
meta-policy update per environment step
40
20
meta-model horizon
- 25
over epoch
20 - 100
A.15 NOTATIONS
33

---

Under review as a conference paper at ICLR 2021
Table 2:
List mathematical notation in the main body, For simplicity, we do not include notation introduced in the appendix.
Notation
Meaning
Time step
Ot O
Observation
Set of observations Hidden state
St, s', S
8, S'
Set of hidden states
Action
Set of actions
Poh; p(Or) St, az-1)
Observation probability function
pa, p(8r) 8 a1-1)
State transition probability function
rv. r(s(, ac)
Reward function
Discount rate
n. no. p(a (44)
(Meta-) pooliyy
h, hr, h
History (past trajectories)
H
Set of history
Learnable parameters of meta-policy
Learnable parameters of meta-model
p(s h4)
Belief state
Discounted return
E Ba hoppo
[R],
E TT R
Meta-model return under full meta-model rollout
Discrepancy between the meta-mode) return and true return
Cm
Bounds of generalization error
CT
Bounds of distribution shift
D, Denv
Dataset (trajectories collected from the real environment)
DTv
Total variation distance
Imaz
Bound of the reward expected on the basis of belief
D model
Set of samples collected through branched rollouts Meta-model horizon (meta-modes rollout length)
TD
Data collection policy
Bxg.p [R]
True return (the return of the veta-policy in the real environment)
Em
Bound of meta-model error current policy
((a, [R]
Meta-model return branched meta-rollouts
Latent context
p un. 00...., at, 0t)
History enoder
DKL
Kullback-Leeibe divergence
JD Dmodel
Optimization objective for meta-policy optimization in M3PO
Qx
Action value function
V
State value function
B
Ensemble size
Index
N
Gaussian distribution
Pg
Mean of Gaussian distribution
og
Standard deviation of Gaussian distribution
T
Task
Task set
34

---

Under review as a conference paper at ICLR 2021
Table 3: Ba settings for results shown in Figure 17 -> b denotes thresholded linear function, i... epoch e, f(e min(max(1 e- . 0).
Walker2D randomparams
Halfchectah- seiie pier
Humanoid-diree
Eg 1]
expected meta- modal horizon
80
130
50
100
150 - 250
A.16 COMPLEMENTARY ANALYSIS 2
In Figures | and 13 we can see that, number of the environments (Halcheetah-piee Walker2D- randomparams and Humanoid-direc), the long-term performance of M3PO is worse than that of PEARL. This indicates that gradual transition from M3PO to PEARL (or other model-free ap- proaches) needs to be considered to further improve overall performance. In this section, we propose introduce such a gradual transition approach to M3PO and evaluate it on the environments where the long-term performance of M3PO worse than that of PEARL.
For the gradual transition, we introduce the notion of the "expected" model-rollout length E& for A: E {0, 1} to M3PO. In this notion, Eg[k 1] means the probability of one-step meta-model rollout. Namely, with the probability of Ex jk 1]. the fictitious trajectory generated by one-step meta-model rollout is used for the policy update (in line 11 in Algorithm (2. and, with the probability of 1 - -- [k: = 1], the real trajectory in Deny are used for the policy update. In our implementation, replace Deddel in line 11 in Algorithm ( with the mixed dataset D Here, D is defined as
D Ex jk
Dmndel (1 Ex [k 1]) Den.
(38)
We linearly reduce the value of E,[k 1] from 1 to 0 depending training epoch. With this value reduction, the M3PO gradually transitioned to PEARL (i.c., the model-free approach).
We evaluate M3PO with the gradual transition (M3PO-h) in three environments (Halfcheetah- pier, Walker2D- -aakkrerD--adddmparass and Humanoid-ding direc) where the long-term performance of M3PO worse than that of PEARL in Figures and [3] The hyperparameter setting (except for setting schedule for the value of E, [k = 1)) for the experiment is the same as that for Figures and 13 (i.e., the shown in Table D. Regarding setting schedule for the value of Ex 1]. reduce the value of R. [k = 1] in accordance with Table
Evaluation results shown in Figure 17 Due to the limitation of computational resource, could not continue the training of M3PO-h until convergence, and the resulting training epoch is much shorter than those of PEARL. Nevertheless, same or better scores with the long-term performances of PEARL in all environments (e.g., M3PO h-1 and M3PO-h-2 achieve the same or better performance than the best scores of PEARL-2 and PEARL-3 in Halfcheetah-pier).
can see that some trials of M3PO-h achieve
35

---

Under review as a conference paper at ICLR 2021
Halfcheetah-pier
8000
Average return
6000
4000
2000
0
-2000
10
19
28
37
46
Training samples (x50)
PEARL-1
PEARL-2
PEARL-3
M3PO-1
--- ----M3P0-2
M3PO-3
MMSPOhh1-=-M3P0-h22 3 M3PO-h-3
Waller22-randomparams
8000
6000
4000
2000
10
19
28
37
46
55
64
73
82
91
Humanoid-direc
12000
10000
8000
6000
4000
2000
0
1
10
19 28
37 46 55 64 73 82 91
Figure 17: Learning curve PEARL, M3PO and M3PO-h a long-term training. each figure, the vertical axis represents expected returns and the horizontal axis represents the numher of training samples (x50000). The meta-policy and meta-model were fixed and their expected returns were evaluated 50 episodes at every 50,000 training samples. Each method was evaluated three trials, and the result the z-th trial denoted by method-z. Note that scale of the horizontal axis larger than that in Figure I by 50 times (I.c., 4 in this figure equal to 200 in Figure
36

---

Under review as a conference paper at ICLR 2021
A.17
HIGH-LEVEL SUMMARY
Markov decision process
Partially observablc Markov decision process
Extend (Sec. 5.1)
(as generalized meta meeaareiiforremmet learning settings)
Performance guarantee of full-rroee-basedd rollouts Theorem 4.1 Janner et al., (2019)
Performance guarantee full-model-Dased rollouts
Theorem 1
Performance guarantee of branched rollouts Theorems 4.3
al., (2019)
Compare (Sec. and A.6)
Refine (Sec. A.7)
Extend
Refined performance guarantee of branched rollouts Theorems
Performance guarantee of branched rollouts Theorems
Derive (Sec. B)
Generalize
Mela-reinforcement learning settings proposed by precursors
Seec.
Practical algorithm of branched rollouts
Algorithm 2
Figure 18: High-lovel summary of what we do in this paper based on previous works. Our core contribution
is proposing model-based meta-reinforcement learning (RL) algorithm (i.e., Algorithm 2 with performance guarantee. To achieve this, first generalize various existing model-based meta-RL settings as solving a partially observable Markov decision process (POMDP) (Section
We then We extend
conduct theoretical analysis by extending the theorems proposed in Janner et al. (2019). (1.1)) their theorems (Theorems 4.1, 4.2 and 4.3) into our POMDP setting (Sections [5.1 and 5.2). As part of extension, refine Theorem 4.2 and 4.3 in order strictly guarantee the performance of hranched rollouts (Section A.7 We also compare these theorems order motivate using branched rollouts for model-based meta-RI. (Sections (52 a.2]6n).6))) We finally propose a practical algorithm based on the result of our theoretical anyways (Section
37


Reviews:
[{"review_id": "R9qcFIlPDS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper summary\n\nThis paper focus on model-based RL on a POMDP setting (they call it \"meta RL\"), where the policy and model need to infer the current hidden state according to history. It provides a theoretical relation between true environment returns and the returns from learned models in a POMDP setting. And it also provides a practical algorithm called M3PO and shows this algorithm is more sample efficient than some meta-RL baselines in some continuous control tasks.\n\n### Pros\n\n- Extend an existing work ( Janner et al. (2019) ) to a POMDP setting\n- Refine some theorems in Janner et al. (2019), by taking more important premises into consideration\n\n### Cons\n\n- The theory part of this paper is quite similar to Janner et al. (2019). It seems they just apply similar theorems to the POMDP setting. For example, theorem 1 in this paper is an immediate result when combining Theorem 4.1 in Janner et al. (2019) and Lemma 1 in Silver & Veness (2010). And the idea of using branched rollout to control model errors is not first proposed by this paper either.\n- The proposed algorithm (M3PO) is also very similar to algorithm 2 in Janner et al. (2019), and they just use this algorithm in POMDPs. From an implementation perspective, there are only two differences: 1. using GRU as network architectures, 2. use PEARL in policy optimization. So I feel the novelty in the proposed algorithm is limited.\n- In the experiments, it seems M3PO (their algorithm) only has significant advantages over PEARL on Halfcheetah-fwd-bwd and Ant-fwd-bwd. In the other environments, the performance gap seems not very clear. Also, in Fig 1, they only show the horizon of 0.2 M steps, which is a relatively short training time for these difficult tasks. And if we compare M3PO-long and PEARL-long, it seems M3PO-long is overall worse than PEARL-long.\n\n### Questions\n\n- Why MBPO ( Janner et al. (2019) ) does not support POMDP? Just because it does not use RNN?\n\n### Minor points:\n\n- The figures are pretty hard to read, especially fig 2.\n\n### References\n\n- Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Proc. NeurIPS, 2019.\n- David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In Proc. NIPS, pp. 2164– 2172, 2010.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QPFhxk3hokp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n\n## General Summary\n\nThe contributions of this paper are two-fold. For one, the authors propose a meta model-based RL algorithm working with a meta-policy and a meta-model. In addition, the authors provide a lower bound of the meta-policy performance.\nThe algorithm is compared against PEARL and L2A, there is a further comparison against M2PO in the appendix.\n\n## Writeup\n\nI find the writeup quite dense and difficult to follow. For one, the idea behind meta-learning is generalization across different tasks/environments. While this was introduced well in sections 3 and 4, the connection is missing in the experiment section and algorithm section. The notation can be better introduced.\n\n## Pros\n\nThe theoretical analysis is the greatest asset of this paper, it is useful to have a bound on the actual performance of the policy that depends on the length of the model rollout, model error and policy distribution shift. Also, it is a valid extension of MBPO by Jenner et al. (2019). \n\n## Cons\n\nThe policy distribution shift and model error based bounds have been addressed in the model-based RL literature. What separates this paper is the extension to POMDPs and meta-learning formalism. The writeup is dense, although the idea behind is simple (branched model rollouts in POMDPs).\n\n## Comments\n\nMaybe it's because I don't work on meta-learning, but I think that you are really missing a clear description of the algorithm. Where are the task switches happening, how does the initial task distribution come into play? This should be clear from algorithm 2 in my opinion.\n\nIn Algorithm 2. you could introduce a loop over tasks maybe? I realize that in the environments that you used, task switches happen  during the episode, but I wouldn't rely on that in the algorithm description because it's a bit confusing, looking at the the algorithm itself, I don't see a meta-learning algorithm but rather a model-based RL algorithm for POMDPs. \n\nThe result that performance degrades with increasing model rollout horizon is not surprising and is the consequence of error compounding. This is a well-known result coming from standard model-based RL and MPC literature.\n\nBranched rollouts (although meta-branched as the authors name them) are also also not a novel addition, as the authors note in the paper. The improvement was introduced in earlier literature (Dyna, MBPO).\n\nCouple of questions for the experiment section:\n\n* Are the results for L2A correct? They seem to be too bad to be true based on the results reported in the original paper.\n* In Fig. 1, by number of training samples it is meant number of time steps in the environment? \n* The variance of the performance is quite high for the ant fwd-bwd, much higher that the other methods. Any explanation for that?\n* In Fig. 1, the legend should appear for the whole figure and not just for the upper-left graph.\n* The performance of PEARL with long-run is better in 3/6 experiments and seems comparable in the others, does this mean that M3PO doesn't use new data efficiently? Also, I would be interested in knowing how the variance looks like for those longer runs.\n* In Fig. 2, it is not really surprising that the performance degrades so much with the rollout length. What is surprising to me is that oftentimes the performance is best for k=1, indicating that there is not much use of the model except for 1 time step, this is in contrast to Janner et al. (2019) in the non-meta setting.\n* I would be interested in seeing how the method generalizes across  OOD unseen tasks (not seen on train), such that the training and test distributions are different and then plot training vs test performance.\n\nIt's good that the authors make a clear comparison between this work and Jenner et al. (2019) because there is a lot of overlap and as I understand, code was reused. Nevertheless, I am concerned about the amount of novelty, \nthis is an extension to POMDPs with a correction to the theory. \n\nIs the meta-learning formalism even needed for this kind of algorithm and evaluation? \n\nText-intricacies:\n* check that the references are correct, p. 10 you have a reference of the form:  MBPO. [link]\n* sec. 3 par. 2 polity->policy\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OglSMOcOSnG", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### **Summary and Contributions of Paper** \nThis paper modifies the work \"When to Trust Your Model: Model-Based Policy Optimization\" (Janner, 2019) to handle the meta-learning case. The definitions, paper structure, and algorithm flows are all very similar to (Janner, 2019). The main difference is that the model architecture is an RNN (and hence conceptually, a new hidden state vector h_{t} is introduced) to handle adaptation for different given tasks, whereas (Janner, 2019) used a probabilistic feed-forward network as the model.\n\nExperiments are conducted on standard RL meta-learning benchmarks (e.g. ForwardBackward Halfcheetah), to compare sample efficiencies between different model-based techniques.\n\n### **Strengths**\n- Contributes to the literature in model-based techniques in the meta-learning setting, which is generally lacking. \n- Experimentation seems to show the desired result - i.e. the paper's method is more sample efficient than both PEARL and L2A.\n- Paper shows that a small modification of (Janner, 2019) using an RNN can solve meta-learning problems in RL.\n\n### **Weaknesses**\n- A bit _too_ similar to (Janner, 2019). Novelty-wise, it seems like a small (but effective) modification of the work in (Janner, 2019) from model-based optimization in the single MDP setting to the meta-learning setting, by including a recurrent/RNN model. The main contribution seems to be this modification, but all other techniques of the paper (e.g. defining the correct terms such as a _branched rollout_, generating the concepts such as C(eps, eps'), etc.) are exactly the same in (Janner 2019). I think the paper overemphasizes the portions that have already been written in (Janner, 2019).\n\n- A bit hard to read at times, due to the denseness of the notation. This is especially encountered in page 6, where there are large chunks of text + math without breathing room. I suggest that the authors provide a high level summary in the main body, especially because most terms have already been defined in (Janner, 2019).\n\n### **Clarity Comments**\n- Since the paper is very similar to (Janner, 2019), it may be useful to highlight the main differences (e.g. different coloring of changed lines in the Algorithmic boxes), rather than redefining everything already mentioned - it was difficult to parse which sections were novel/changed and which were from (Janner, 2019). \n\nOverall, the main benefit seems to be this RNN modification and relevant meta-learning experiments (which is why I give a marginal accept), but I do think that the authors can re-write the paper in a much simpler fashion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4UIsjvTZfEt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary ===\n\nThe paper concerns model-based meta-RL. It exploits the fact that meta-RL can be formulated as POMDP in which the task indicator is part of the (unobserved) hidden state. Thus, the paper effectively analyzes and proposes model-based algorithms for POMDPs. The paper bounds the gap between the expected reward of a policy in the actual POMDP and the estimated model and then theoretically shows that this gap can be reduced when using dyna-style / branched rollouts instead of full rollouts under the learned model. Motivated by this finding, the paper proposes a Dyna-like algorithm for POMDPs. In the experimental evaluation, the paper compares its proposed method, M3PO, to two recent meta-RL approaches in a range of meta-RL environments for continuous control.\n\n=== Merits of the paper ===\n\nOverall, the paper contributes a new algorithm for model-based meta-RL which is neatly motivated by the paper’s theoretical analysis. Both the theoretical analysis and the proposed algorithm are sound. The experimental evaluations demonstrate that the algorithm performs similar or better than previous model-based meta-RL algorithms and thus could be a relevant contribution to the field of meta-RL.\n\n=== Strenghts ===\n- The experiments include relevant meta-learning environments and algorithms to compare to.\n- The proposed algorithm is sound and motivated by theory.\n- I have done a very simple simulation study, being able to confirm that the gap in Theorem 2 is indeed (much) smaller than the one Theorem 1.\n\n=== Weaknesses / Concerns ===\n- Both Theorem 1 and 2 seem to be a straightforward combination of the results in [1] and the fact that POMDPs can be cast as MDPs with history-states. Thus the theoretical contribution/novelty is quite limited.\n- As expected, from my simulation study it seems that bounds are vacuous and their usefulness beyond motivating branched rollouts is questionable.\n- The only difference to original dyna-like approaches is the fact that a recurrent model with internal/hidden state is used – thus the algorithmic contribution is small as well.\n- Overall, the clarity of the paper could probably be improved a lot – Many paragraphs and sentences are hard read and lack vital explanations. For examples, see below.\n\n=== Overall Assessment ===\n\nIn my opinion, the paper is a borderline case. Currently, I see the paper slightly below the acceptance threshold. Both the theoretical and the algorithmic contribution of the paper are small. Overall, the paper is a straightforward extension of [1] to POMDPs with meta-RL experiments. Yet, due to a lack of clarity, it is hard to read. Nonetheless, the proposed algorithm seems to be practically relevant for meta-RL - thus I am happy to hear other opinions and open to being convinced to increase my score.\n\n=== Questions & tips for improvement ===\n\nAppendix A.1 briefly discusses the differences to [1], and claims to derive similar Theorems under more appropriate assumptions / premises in Appendix A.7. How do the theorems in A.7. differ from the ones in Section 5? If the theorems build on more appropriate assumptions, why not use them in the main part of the paper?\n\nAdding a simple simulation study that shows how much smaller the gap/discrepancy is, when branched instead of full rollouts are used, would require little effort and strengthen the claim that branched rollouts are preferable.\n\nI found section 5.1. to be very confusing – It is stated that the gap can be “expressed as the function of two error quantities of the meta-model: generalization error due to sampling and distribution shift due to the updated meta-policy”. Indeed Def. 2 is probably the generalization error due to sampling. However, Def. 1 is the expected TV distance of the estimated model from the true model. Then, how should this be the “distribution shift due to the updated meta-policy”?\n\n=== Minor comments ===\n\n- Section 4, 3rd paragraph: It should probably be \"$r_{t}$ and $o_{t+1}$ are assumed to be conditionally independent\n- Section 5.1.: It would be good to have a proper definition if $\\pi_{\\mathcal{D}}$, i.e. the data-collection policy\n- Theorem 1: The definition of $\\epsilon_m$ is duplicate whereas the $\\epsilon_\\pi$ is not included\n- Section 7, 4th paragraph: Should be “In Humanoid-direct, the performance”\n\n\n[1] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. NeurIPS 2019\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a meta-learning for Model-based RL that introduces branched rollouts to improve sample efficiency of the learned model.

While the paper addresses an important topic of sample efficiency in RL, and provides theoretical analysis, the reviewers raised concerns with the novelty and clarity. The extension to POMDP setting is certainly important technological contribution, albeit a straightforward. To be suitable for publication the work needs to make stronger case for the significance of the method.  
Decision: Reject