Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
Authors: Russell Mendonca
Xinyang Geng
Chelsea Finn
Sergey Levine
Venue:  (2021)
Keywords: Meta-Reinforcement Learning
Meta Learning
Reinforcement Learning

Abstract: Reinforcement learning algorithms can acquire policies for complex tasks autonomously. However, the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning methods have enabled agents to leverage prior experience to adapt quickly to new tasks, their performance depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data for on-policy meta-training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time. Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, more easily than policies and value functions. These dynamics models can then be used to continue training policies and value functions for out-of-distribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task.

Under review as a conference paper at ICLR 2021
META-REINFORCEMENT LEARNING ROBUST TO DISTRIBUTIONAL SHIFT VIA MODEL IDENTIFICATION AND EXPERIENCE RELABELING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning algorithms can acquire policies for complex tasks au- tonomously. However, the number of samples required to learn diverse set of skills can be prohibitively large. While meta-reinforcement learning methods have enabled agents to leverage prior experience to adapt quickly to new tasks, their performance depends crucially on how close the new task is to the previously experienced tasks. Current approaches can do so at the expense of requiring extremely large amounts of data for on-policy meta-training. In this work, we present model identification and experience rela- beling (MIER), meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks test time. Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off- off-policy data, more easily than policies and value functions. These dynamics models can then be used to continue training policies and value functions for out-of-distributin tasks without using renforeiiforremmen learning at all, by generating synthetic experience for the new task.
either not able to extrapolate well,
INTRODUCTION
Recent advances in reinforcement learning (RL) have enabled agents to autonomously acquire policies for complex tasks, particularly when combined with high-capacity representations such as neural networks (Lillicrap et al. 2015 Schuiman al., 2015 Mnih et al. 2015 Levine et al., 2016). However, the number of samples required to learn these tasks is often very large. Meta-reinforcement learning (meta-RL) algorithms can alleviate this problem by leveraging experience from previously seen related tasks (Duan et al. 2016; Wang et al. 2016 Finn et al., 2017a), but the performance of these methods on new tasks depends crucially on how close these tasks are to the meta-training task distribution. meta-training, but lose much of their benefit when adapting to tasks that are too far away from the meta-training set. This places a significant burden on the task distributions that sufficiently cover the kinds of tasks that may be encountered at test time.
agents can adapt quickly to tasks that are similar to those seen during
carefully construct meta-training
Many meta-RL methods either utilize variant of model-agnostic meta-learning (MAML) and adapt to new tasks with gradient descent (Finn et al.) 2017a] Rothfuss et al.) 2018; Nagabandi al. 2018), or use encoder-based formulation that adapt by encoding experience with recurrent models (Duan et al.) 2016 Wang et al.) 2016), attention mechanisms (Mishra et al., 2017) or variational inference (Rakelly et al. 2019). The encoder-based methods struggle when adapting to -ut-of-distribution tasks, because the adaptation procedure is entirely learned and carries no guarantees for duttrf-distribution inputs (as with any learned model). Methods that utilize gradient- based adaptation have the potential of handling -uittf-diittrbution tasks more effectively, since gradient descent corresponds to a well-defined and consistent learning process that has a guarantee of improvement regardless of the task (Finn & Levine 2018). However, in the RL setting, these methods (Finn et af.) 201 Rothfuss et al.] 2018) utilize on-policy policy gradient methods for meta-traie veta-training, which require very large number meta- teeta-triining samples (Rakelly et al. 2019).
In this paper, we aim to develop a meta-R RL algorithm that can both adapt effectively to out-of- distribution tasks and be meta-trained efficiently via off-policy value-based algorithms. One approach

---

Under review as a conference paper at ICLR 2021
Model Identification
Experience Relabeling
Model Meta-Training
Polley Training
Continued Policy Training
Triniin Task Data
Policy (m)
Model (5)
Policy (4)
Inner adaptation (Auoe)
(ala,, dr)
Policy conditioned
Model (F)
updated
Synthetic Experience
Figure 1: Overview of our approach. The model context variable (Ø) is adapted using gradient descent, and the adapted context variable (áT) is fed to the policy alongside state so the policy can trained with standard RI. (Model Identification). The adapted model is used to relabel the data from other tasks by predicting next state and reward, generating synthetic experience to continue improving the policy (Experience Relaheling).
might be to directly develop value-based off-policy meta-RL method that uses gradient-base -based meta-learning. However, this very difficult, since the fixed point iteration used value-based RL algorithms does not correspond to gradient descent, and to knowledge no prior method has successfully adapted MAML to the off-policy value-based setting. We further discuss this difficulty in Appendix A Instead, we propose to leverage simple insight: dynamics and reward models can be adapted consistently, using gradient based update rules with off off-policy data, even if policies and value functions cannot. These models can then be used train policies for cuitt--distribution tasks without using meta-RL at all, by generating synthetic experience for the new tasks.
Based this observation, we propose model identification and experience relabeling (MIER), meta-RL algorithm that makes use of two independent novel concepts: model identification and experience relabeling. Model identification refers to the process of identifying a particular task from a distribution of tasks, which requires determining its transition dynamics and reward function. We use gradient- supervised meta-learning method to learn a dynamics and reward model and a (latent) model context variable such that the model quickly adapts to new tasks after a few steps of gradient descent on the context variable. The context variable must contain sufficient information about the task to accurately predict dynamics and rewards. The policy can then be conditioned on this context (Schaul et al. 2015 Kaelbling 1993) and therefore does not need to be meta-traine or adapted. Hence it can be learned with any standard RL algorithm, avoiding the complexity of meta-reinforcemenn learning. We illustrate the model identification process Figure (left).
When adapting
tasks meta-test time, the adapted context variable may itself be out of distribution, and the context-conditionel policy might perform poorly. However, since MIER adapts the model with gradient descent, can continue improve the model using more gradient steps. To continue improving the policy, we leverage all data collected from other tasks during meta-training, by using the learned model relabel the next state and reward on every previously seen transition, obtaining synthetic data to continue training the policy. We call this process, shown in the right part of Figure experience relabeling. This enables MIER to adapt to tasks outside of the meta- veta-training distribution, outperforming prior renforeeifor learning methods in this setting.
PRELIMINARIES
Formally, the reinforcement learning problem is defined by a Markov decision process (MDP). We adopt the standard definition of MDP, T (S,A,p P. nl; 7), where S is the state space, is the action space, p(s'|s |s. a) is the unknown transition probability of reaching state s' at the next time step when an agent takes action a at state s, (s)s) the initial state distribution, r(s, a) the reward function, and (0, 1) is the discount factor. An agent acts according to some policy I (als)) and the learning objective is to maximize the expected return, E,, 7'rss,,
We further define the meta-reinforee reinforeenr learning problem. Meta-training uses distribution over MDPs, (T),, from which tasks are sampled. Given a specific task T, the agent is allowed to collect small amount of data ((7) adopt and adapt the policy to obtain गт. The objective of meta-training is to maximize the expected return of the adapted policy ETTxp(T) s.. 1E+ 7/r((1 a.)).
2

---

Under review as a conference paper at ICLR 2021
MIER also makes use of learned dynamics and reward model, similar to model-based reinforcement learning. In model-based reinforcement learning, a model p(s'. r/s, a) that predicts the reward and next state from current state and action is trained using supervised learning. The model may then be used to generate data to train a policy, using an objective similar to the RI objective above: arg max; Fx,, a EE, 7r(s,, a-)]- Note that the expectation is now taken with respect to the policy and learned model, rather than the policy and the true MDP transition function p(s' [s, a).
In order to apply model-based RL methods in meta-RL, we need solve a supervised meta-learning problem. We briefly introduce the setup of supervised meta-learning and the model agnostic meta- learning approach, which an important foundation of our work. In supervised meta-learning we also have a distribution of tasks p(T) similar to the meta-RL setup, except that the task T is now pair of input and output random variables Yr). Given a small dataset D(T) from specific task T, the objective is to build a model that performs well on the evaluation data D(T), sampled from the same task. If we denote our model as f(X; f), the adaptation process as A(O, D and our loss function C. the objective can be written as: adapt)
edayt sampled
min E (T) [< (f F.A
A (B, (0, addae .xx7]
Model agnostic letanlea Finn et al. (0177)) is an approach to solve the supervised meta-learning problem. Specifically, the model f[X; is represented as neural network, and the adaptation process is represented as few steps of gradient descent. For simplicity of notation, we only write out one step of gradient descent:
Anssa (0, (O.. edapt) = - a00-aVoBxa aVo...Y ( (x:)(
adopt
The training process of MAML can be summarized as optimizing the loss of the model after few steps of gradient descent on data from the new task. Note that because Ansa is the standard gradient descent operator, our model is guaranteed to improve under suitable smoothness conditions regardless of the task distribution p(T), though adaptation to in-distribution tasks is likely to be substantially more efficient.
META TRAINING WITH MODEL IDENTIFICATION
As discussed in Section ! MIER is built on top of two concepts, which we call model identification and experience relabeling. We first discuss how we can reformulate the meta-RI. problem into model identification problem, where we train a fast-adapting model to rapidly identify the transition dynamics and reward function for new task. We parameterize the model with latent context variable, which meta-trained to encapsulate all of the task-specific information acquired during adaptation. We then train universal policy that, conditioned on this context variable, can solve all of the meta-training tasks. Training this policy is a standard RL problem instead of a meta-RL problem, so any off- the- pfllcy algorithms can be used. The resulting method can immediately be used to adapt to new in-distribution tasks, simply by adapting the model's context via gradient descent, and conditioning the universal policy this context. We illustrate the model identification part of our algorithm in the left part of Figure I and provide pseudo-code for our meta-training procedure in Algorithm
In a meta-RL problem, where tasks are sampled from a distribution of MDPs, the only varying factors are the dynamics p(s'ls, a) and the reward function r. Therefore, a sufficient condition for identifying the task is to learn the transition dynamics and the reward function, and this is exactly what model-based RL methods do. Hence, we can naturally formulate the meta-task identification problem as model- RL. problem and solve it with supervised meta-learning methods.
Specifically, we choose the MAML method for its simplicity and consistency. Unlike the standard supervised MAML formulation, we condition our model on a latent context vector, and we only change the context vector when adapting to new tasks. This formulation is previously known as bias transformation for MAML (Finn et al. 2017b: Finn & Levine 2018 Zintgral et al., 2018). 8). Since all task-specific information is thus encapsulated in the context vector, conditioning the policy on this context should provide it with sufficient information to solve the task. This architecture is illustrated in the left part of Figure]] [ We denote the model as p(s', r|s, a; 8, Φ), where 0 is the neural network parameters and & is the latent context vector that is passed in input to the network.
3

---

Under review as a conference paper at ICLR 2021
One step of gradient adaptation can be written as follows:
ΦT = Auasa (0, o, D(7)
adapt/ = aV& (s.sss ry (T) log (s', r/s. j. a: A, Ø)].
(1)
adapt
We use the log likelihood as our objective for the probabilistic model. We then evaluate the model using the adapted context vector ΦT, and minimize its loss on the evaluation dataset to learn the model. Specifically,, we minimize the model meta-loss function Jp(0, Φ, D(T) D(T)) to obtain the optimal parameter and context vector initialization 6:
arg min Jp (0.0) Do D(tal) arg min E 0,0
0,0 (s.a.s' (s.a'')) r) |- log (s', r/s, a: 0 ΦT)]
The main difference between our method and previously proposed meta-RL methods that also use context variables (Rakelly et al., 2019; Duan et al.) (2166) is that the context. Adaptation will be much faster for tasks that are in-distribution, since the meta-training process explicitly optimizes for this objective, but the model will still adapt duitr--distriuution tasks given enough samples and gradient steps, since the adaptation process corresponds to a well-defined and convergent learning process. However, for out-of-distribttion tasks, the adapted context could be out-of -uttof--distibution for the policy. We address this problem in Section
gradient descent to adapt
Given the latent context variable from the adapted model Φ7. the meta-RL problem can be effectively reduced to a standard RI. problem, as the task specific information has been encoded in the context variable. We can therefore apply any standard model-free RL algorithm to obtain a policy, as long as we condition the policy on the latent context variable.
Algorithm 1 Model Identification Meta- Training
Input: task distribution p(T), training steps N, learning rate a Output: policy parameter s model parameter 0, model context
Randomly initialize th, 0, Φ
Initialize multitask replay buffer R(TT) + while ø. db not converged do
In our implementation, we utilize the soft actor- critic (SAC) algorithm (Haarnoja et al. 2018). though any efficient model-free RL. method could be used. We briefly introduce the pol- icy optimization process for a general actor- critic method. Let us parameterize our pol- icy Tu by parameter vector vb Actor-critic methods maintain an estimate of the Q val- ues for the current policy, Q**(s, a, ④T) Es,a, .n. |E, (s,,,, ar)[so = 8, ag a, T]. via Bellman backups, and improve the policy by maximizing the expected Q values under the policy, averaged over the dataset D) The policy loss can be written as:
Sample task T~ p(T)
Collect ((T)
adupt using F. and á
Compute &J Ansaa (0, Φ, D(T) adapt)
Collect D(T) using v and OJ
emal
R(T) + R(T) adaps (D(T) eval
for i= to do
Sample task T
Sample (DT)
adape eval R(T)
Update 0 < 0 aa 6, D(T)
adapt D(T)
Update + 1 V (1 D(T)
evaly ØT)
end
end
J= D, (T) -Ea-D (Q (s, dT)]
Note that we condition our value function Q** (s, ΦT) and policy Ty (als, ΦT) on the adapted task specific context vector ΦT. so that the policy and value functions are aware of which task is being performed (Schaul et al. [2015) Kaelbling] 1993). Aside from incorporating the context Φ7, the actual RL algorithm is unchanged, and the main modification is the concurrent meta-training of the model to produce ΦT.
4 ADAPTING TO OUT-OF-DISTRIBUTION TASKS VIA EXPERIENCE RELABELING
At meta-test time, when our method must adapt to a new unseen task T, it first samples small batch of data and obtain the latent context Φ by running the gradient descent adaptation process on the context variable, using the model identification process introduced in the previous section. While model identification method is already complete meta-RL RL algorithm, it has guarantees of consistency. That is, it might not be able to adapt to out-of-distribiue tasks, even with large amounts of data: although the gradient descent adaptation process for the model is consistent and will continue
4

---

Under review as a conference paper at ICLR 2021
to improve, the context variable Φ7 produced by this adaptation may still be -ut-o--distribution for the policy when adapting to an duttof-distribution task. However, with an improved model, we can continue to train the policy with standard off poolyy RL, by generating synthetic data using the model. In practice we adapt the model for as many gradient steps as necessary, and then use this model to generate synthetic transitions using states from all previously seen meta-training tasks, with new successor states and rewards. We call this process experience relabeling. Since the model ís adapted via gradient descent, it guaranteed to eventually converge to a local optimum for any new task, even a task that is outside the meta- veta-training distribution. We illustrate the experience relabeling process in the right part of Figure and provide pseudo-code in Algorithm 2
When using data generated from a learned model to train a policy, the model's 's predicted trajectory often diverges from the real dynam- ies after large number of time steps, due to accumulated error (Janner et al.) 2019). We can mitigate this issue in the meta- RL case by leveraging all of the data from other tasks that was available during meta- training. Although new task is previously unseen, the other train- ing tasks share the same state space and action space, and so we can leverage the large set of diverse transitions collected from these tasks. Using the adapted model and policy, we can relabel these transitions, denoted (s, a, r), by sampling new actions with our adapted pol- icy, and by sampling next states and rewards from the adapted model. The relabeling pro- cess can be written
Algorithm 2 Experience Relabeling Adaptation
Input: test task T, multitask replay buffer RTT), Adaptation steps for context N fast, Training steps for policy Ng. Training steps for model N,.
Output: policy parameter &
Collect D'2 for = to Nasst do
adupt from I using ae and s
Update Φा according to Eq. I end while va not converged do for i = 1 to N₂ do
Sample T ~ R and D(T) ~ R(T) Di + ReRlabbbl(( 6,)
Train policy & + - - a Va DDT) ,T)
end
end
Rclabel (D, = ass, r)]s D;a~ x(ass 0T), ~ p(s'sr(ss, a; 0, ΦT)}.
We use these relabeled transitions to continue training the policy. The whole adaptation process is summarized in Algorithm 2 Since the learned model is only used to predict one time step into the future, our approach does not suffer from compounding model errors. We also note that our experience relabeling method is general tool for adapting to out-of-distribution tasks, and could be used independently of our model identification algorithm. For example, we could apply standard, non-context-based dynamics and reward model to generate synthetic experience to finetune a policy obtained from any source, including other meta-RL RL methods.
Similar to our method, the MQL algorithm (Fakoor et al., 2020) also reuses data collected during meta-training time to continue improve the policy during adaptation. However, the way it reuses data different. Given a new adaptation dataset Dudaps and replay buffer R containing data from other
tasks, MQL estimates density ratio g(s.a.s'.r) {'(',,,''r)) where q and q' are the corresponding probability density on Dadapt and R. MQL then re-weight the transitions in the replay buffer using this density ratio to compute the loss for the policy and value function. This implicitly assumes that the data distributions of different tasks share the same support, i.e. q(s, a, s', r) >> 0 for transitions sampled from other tasks in the replay buffer. This assumption may not be true in many practical domains. We will show empirically that indeed when this assumption is violated, MQL not able to adapt effectively. Our method avoids this problem by using an adapted model to relabel the data. By generating a synthetic next state and reward, the relabeled transition would be useful for the new task, even if the original transition is not. The only assumption for our method is that different tasks share the same state and action space, which is true for most meta-RL RL domains.
5 RELATED WORK
Meta- reinffrceeet learning algorithms extend the framework of meta-learning (Schmidhuber 1987;
Thrun & Pratt 1998 Naik & Mammone 1992 Bengio et al. 1991) to the reinforcement learning setting. Model-free encoder- based methods encode the transitions seen during adaptation into a latent context variable, and the policy is conditioned on this context to adapt it to the new task.
5

---

Under review as a conference paper at ICLR 2021
The context encoding process is often done via recurrent network (Duan et al. 2016; Wang et al. 2016; Fakoor et al., 2020; Stadie et al., 2018), an attention mechanism (Mishra et al. 2017). 7) or via amortized variational inference (Rakelly et al. 2019; Humplik et al.) 2019). While inference is efficient for handling in-distribution tasks (Fig.
2p. it is not effective for adaptation to out-of- tasks through the use of consistent gradient descent learner for the model, followed by a consistent (non-meta-trained)
distribution tasks (Fig. 4). On the other hand, MIER can handle
off- off-policy reinforcement learning method.
Model-free gradient-based meta-RL. RL methods (Finn et al. 2017a Rothfuss et al.,
2018 Rusu et al.
2018: Liu et al. 2019; Gupta et al., 2018; Sung et al. 2017; Houthooft et al.; 2018 implement
gradient descent as the adaptation process. However, they are based on on on-policy RL algorithms. and
thus require a large number of samples for training and adaptation (Fig. 22. There are also works that combine gradient- based and encoder-based methods (L.nn et al. 2019). However, such methods still suffer from the same sample efficiency problem as other gradient based methods, because of the use of on-policy policy gradients. Our method avoids this problem by combining gradient-based supervised meta-learning algorithm with an off-policy RL algorithm to achieve sample efficiency comparable that of off- policy encoder- mesder-based methods.
There has been some work that uses off-po policy gradients for sample efficient meta-training (Mendonca al., 2019), but this still requires quite few trajectories for policy gradient based adaptation at test time. MIER avoids this by reusing the experiences collected during training to enable fast adaptation with minimal amount of additional data.
Model based meta-RL RL methods meta-train a model rather than a policy (Nagabandi et al., 2018
Sæmundsson et al. 2018). At test time, when the model is adapted to a particular task, standard planning techniques such as model predictive control (Williams et al. 2015 Camacho & Alba 2013) are often applied select actions. Unfortunately, pure model-based mL methods typically attain lower returns than their model-free counter-part particularly for sang-hoiioon tasks. Our method can attain comparatively higher final returns because we only use one-step predictions from our model to provide synthetic data for model-free RL method (Fig. 4.. This resembles methods that combine
model learning with model-free RL in single-tasks settings (Sutton] 1991 Janner et al. 2019)
EXPERIMENTAL EVALUATION
We aim to answer the following questions in our experiments: (1) Can MIER meta-trait efficiently on standard meta-RL benchmarks, with meta-training sample efficiency that is competitive with state-of- the-art methods? (2) How does MIER compare to prior meta-learning approaches for extrapolation to meta-test tasks with dut-of-distribution (a) reward functions and (b) dynamics? ? (3) How important is experience relabeling in leveraging the model to train effective policies for out-of-distributio tasks?
To answer these questions, we first compare the meta-training sample efficiency of MIER existing methods on several standard meta-RL benchmarks. We then test MIER on a set of outto--distribution meta-test tasks analyze extrapolation performance. We also compare against a version of our method without experience relabeling, in order to study the importance of this component for adaptation. All experiments are run with OpenAI gym (Brockman et al. 2017) and use the mujoco simulator (Todorov et al.) 2012). Additional implementation and experiment details including hyperparameters are included Appendix C
6.1
META-TRAINE SAMPLE EFFICIENCY ON META-RI. BENCHMARKS
We first evaluate MIER on standard meta-Ro Re benchmarks, which were used in prior work (Finn et al. 2017a| Rakelly et al.) 2019 Fakoor et al. 2020) and show the result in Figure 2 We compare to PEARL (Rakelly et al. 2019), which uses an off-policg encoder-based method, but without consistent adaptation, meta Qelemnning (MQL) (Fakoor et al. 2020). which also uses an encoder, MAML (Finn) cet al., 2017a) and PRoMP (Rothfuss et al.) 2018), which use MAML-based adaptation with on-policy policy gradients, and RL2 (Duan et al. 2016). which uses an on-policy algorithm with encoder. We plot the meta-test performance after adaptation (on in-distribution tasks) against the number of meta- meta-trainin training samples, averaged across random seeds. On these standard tasks, we run variant of our full method which we call MIER-wR (MIER without experience relabeling). Note that our implementation generates two hundred thousand exploration samples from the initial policy during

---

Under review as a conference paper at ICLR 2021
HoBiChhetah
Art-Twe--Bas
Bute Fland- Parama
Noum
13K
10
26
24
35 40
10
20
05
24
OR
10
Time (148)
MIER-WR (Ours)
PEARL
MQL
MAML
RL2
ProMP
Figure 2: Performance on standard meta- RL benchmarks. Return is evaluated over the course of the meta- training process on meta-test tasks that are in-distribution.
meta-training prior beginning meta-training, following the same convention PEARL (Rakelly et al. 2019). whereas MQL does not generate these samples, which explains the difference in terms of where training begins (most visible in Half-Cheetah-Vell resulting in constant offset of the learning curve along the x-axis. Our method achieves performance comparable to or better than the best prior methods, indicating that our model identification method provides a viable meta-learning strategy that compares favorably to state-of-the of-the-art methods. However, the primary focus of our paper is on adaptation to out-of-distribution tasks, which we analyze next.
6.2 ADAPTATION TO OUT-OF-DISTRIBUTTION TASKS
Next, we evaluate how well MIER can adapt to out-of-distribuiig both on tasks with varying reward functions and tasks with varying dynamics. We compare the performance of our full method (MIER), and MIER without experience relabeling (MIER-wR), to prior meta-learning methods for adaptation to out-of-distributio tasks. All algorithms are meta-trained with the same number samples (2.5M for Ant Negated Joints, and 1.5M for all other domains) before evaluation. For performance of algorithms as a function of data used for meta- veta-training, see Figure 6 in Appendix
Extrapolation over reward fuucction: To evaluate extrapolation to out-of-distributiin rewards, we first test on the velocity extrapolation environments of HalfCheetah introduced by Fakoor et al. (2020). Half-Cheetah- -heetahh-ee--Meeiueee Vel-Medium meta-trains on tasks where the cheetah is required to run at target speeds ranging from 0 to 2.5 m/s, while Half-Cheetah-Hard meta-trains on speeds from 0 to 1.5 m/s, as depicted in Figure 3(a) In both settings, the meta-test set has target speeds sampled from 2.5 to 3 m/s. In Figure 4 we see that our method matches MQL on the easier environment, and outperforms all prior methods including MQL the Half- Selff-Cheetah- Vel-Hard setting, where the meta-test tasks are further outside the distribution of meta-training tasks. We see that experience relabeling improves performance in both settings.
Vel-Medium
We also evaluate reward function extrapolation on the Ant tasks, where meta-training task directions sampled from 3 quarters of the circle, and the meta-test set contains the remaining quadrant, as illustrated in Figure 3(b We see in Figure that our method outperforms PEARL and MAML by a large margin in this setting, while MQL attains better performance. We provide a more fine-grained analysis of udaptation performance on different tasks in the test set in Figure 6 We see that while the performance of all methods degrades as validation tasks get farther away from the training distribution, MIER and MIER-wR perform consistently better than MAMI. and PEARL.. Note, however, that the reward extrapolation represents in some sense the easiest setting for extrapolate, since the dynamics are preserved across all tasks, and our method's performance would likely improve with better models. In the next paragraph, we study the more challenging setting, where extrapolation requires adapting to different dynamics.
dynamics, we con- structed variants of the HalfCheetah and Ant environments where we negate the control of randomly selected groups of joints as shown in Figures a(d)and 3(d) During meta-training, we never negate the last joint so we can construct out-of -it-of-distribution tasks by negating this last joint together with a randomly chosen subset of the others. For HalfCheetah, we negate 3 joints at a time from among the first 5 during meta-training, and always negate the 6th joint (together with a random subset of 2 of the other 5) for testing, such that there are 10 meta- meta-training tasks and 10 -ntt----istriuutionn evaluation tasks. For Ant, we negate joints from among the first 7 during trainira,n and always negate the 8th (together with a random subset of 3 of the other 7) for evaluation, resulting in 35 meta- tetaitrainnng tasks and 35 evaluation tasks, out of which we randomly select 15.
Extrapolation over dynamics: To study adaptation to
7

---

Under review as a conference paper at ICLR 2021
weto
Regited
Contrul
Dircctin
Gontrol Dincction
Monement Dinction
(a) Cheetah Velocity
(b) Ant Direction
(c) Cheetah Negated Joints
(d) Ant Negated Joints
Figure 3: Illustration of out-of distriibbittonn adaptation tasks: (a) Chectah- Velocity Medium (target velocity
training set in blue, test set in red) and Cheetah- Cheetah-Velocity Hard (target velocity training set in green, test set in red), (h) Ant Direction (target direction training tasks green, test tasks in red), (c) Cheetah Negated Joints and (d) Ant Negated Joints. Training and test sets are indicated in figure for (a) and (b). the negated joint envin duvironments, the control is negated to a set of randomly selected joints, and the movement direction when control is applied is depicted for both negated and normal joints.
Feff-Cheelah-VVe Vel- Veiialuuu Medium
Hatf hatf-Cheetah-Vel-Haadd
Ant. Dir
Avvenge Detum
200
1200
400
Average
MGR
MiR-vRtt
MGL
PEARL
MER
NIERAR
PEARL MAML
Cheotsh- Beadtah-Nagaadd-uoo
Ant NegsteedJoonts
Averago Reoum
M82 MEE MDL
PEARL OBAL NAVL
MER MIRRWR
MCL
PEARL
MARE
MIER (Ours)
MIER-wR (Ours)
PEARL
MQL
MAML
GrBAL
Figure 4: Performance on out-of-distribution tasks. All algorithms are meta-trained with the same amount of data, and then evaluated on out-of- tasks. Cheetah- Cheetah-Velocity and Ant-Direction environments have varying reward functions, while Cheetah- Beeeaah-Negated-Jointt and Ant-Negated-Joints have different dynamics.
In addition to MQL, PEARL and MAML, we compare against GrBAL Nagabandi et al. 2018). model based meta-RL. RL method. We could not evaluate GrBAL on the reward extrapolation tasks, since it requires the analytic reward function to be known, but we can compare to this method under varying dynamics. From Figure 4 we see that performance on Cheetah- Segettah-NegatedJJoin adaptation (MIER-wR) is substantially better than prior methods, and there is further improvement by using the model for relabeling (MIER). On the more challenging Ant-Negated-JJinn Joints environment, MIER-wR wR shows similar performance to MQL and PEARL, and leveraging the model for relabeling again leads to substantially better performance for MIER. We note that in this harder dynamics extrapolation setting, the full MIER method attains significant improvement hypothesize is based the shared support assumption described in Section 4 is violated in these two environments.
with just context
MQL, which
7 CONCLUSION
In this paper, we introduce a consistent and sample efficient meta-RL algorithm by reformulating the meta-RL. problem as model identification, and then described a method for adaptation out- of-distribution tasks based on experience relabeling. Our algorithm can adapt consistently to out-of-distribution tasks by adapting the model first, relabeling data from the meta-training tasks with this model, and then fine-tuning that data using standard off-policy RL method. While model identification and experience relabeling can be used independently, with the former providing for a simple meta-RL framework and the latter providing for adaptation to out-of-distribution tasks, show that combining these components leads to good results across range of challenging meta-R RL problems that require extrapolating distr---ittiiittonnn tasks at meta-test time.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. In IJCNN-9I-Seatle International Joint Conference on Neural Networks, volume ii, pp. 969 vol.2-, 1991.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 160.001400. 2017.
E. F. Camacho and B. Alba. Model predictive control. In Learning to learn. Springer Science and Business Media, 2013.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL3: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learnnng In International Conference on Learning Representations, 2020. URL [RL//attpp://opereviee.
hett/forum21d=500D33CCC
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradi- ent descent can approximate any learning algorithm. In International Conference on Learning Representations, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1135. JMLR. org, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. arXiv preprint arXiv: 1709. 700.0090090 2017b.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Picter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 10XXvv88011012200 2018.
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint 1802. 04822, 2018.
Jan Humplik, Alexandre Galashov, Leonard Hasenelever, Pedro Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint 1xXiv:1955.0647 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. arXiv preprint arXiv: 190660822555 2019.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, 1993.
Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta reinforcement learning with task embedding and shared policy. arXiv preprint arXiv: 1905.0679 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):133-1333, 2016.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2015.
Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta-reinffrreement learning. In International Conference Machine Learning, pp. 4061-4071, 2019.
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta- veea--poicy search. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 9656-9667. Curran Associates, Inc., 2019. URL http: http:://aapprss.ee nips cc/paper/ 91 67600-uiio ded met sea www.cyyseeersppppoo
ded

---

Under review as a conference paper at ICLR 2021
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv:1707.0314, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-53) 2015.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reinforcement learning. arXiv preprint 1..Kvv1803311133 2018.
Devang K Naik and RJ Mammone. Meta-neural networks that learn by learning. In (Proceedings ]992] IJCN International Joint Conference on Neural Networks, volume 1, pp. 437-442. IEEE, 1992.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off- off-pooicy meta-reinforcemmet learning via probabilistic context variables. arXiv preprint arXiv: 1903.08254, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810006784, 2018.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 2018.
Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint
2018.
Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 1312-1320 JMLR.org, 2015.
Jürgen Schmidhuber, Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2015.
Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01 181, 2018
Flood Sung, Li Zhang. Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn: Meta-critic networks for sample efficient learning. arXiv preprint 1rXXv::7006.9529, 2017.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 264):160-163, 1991.
O. Tange. Gnu parallel - the command-line power tool. :login: The USENIX Magazine, 36(1):42-47, Feb 2011. URL http:://///ee
.nu. http://www.........//// llel
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pp. 3-17. Springer, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. ArXiv, abs/161 61 60166110000 1.05763, 2016.
10

---

Under review as a conference paper at ICLR 2021
Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control using covariance variable importance sampling. CoRR, abs/1509.01149, 2015. URL
wttp://..rrgabs///5.0.... htttpp//aaxuuu.orrr//////0...... 01149
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Caml: Fast context adaptation via meta-learning arXiv preprint arXiv: 1810.0364,, 2018.

---

Under review as a conference paper at ICLR 2021
Appendices
A THE DIFFICULTY OF COMBINING GRADIENT-BASED META-LEARNING WITH VALUE-BASED RL METHODS
One straightforward idea of building a sample efficient off- off-pp meta- RL algorithm that adapts well to out-of out-of-distribution task is to simply combine MAML with an off off-cp actor-critic RL algorithm. However, this seemingly simple idea very difficult practice, mainly because of the difference between Bellman backup iteration used in actor-critic methods and gradient descent. Consider the Bellman backup of Q function Q" for policy n.
Q" (s a) r(s, a) + + (s)"ssp))).))).... (s.)). "x) (s',a'))
which backs up the next state Q value to the current state value. One iteration of Bellman backup can only propagate value information backward in time for one timestep. Therefore, given a trajectory with horizon T, even if can perform the backup operation exactly every iteration, at least T iterations of Bellman backup is required for the Q function to converge. Therefore, it cannot be used as the inner loop objective for MAML, where only few steps of gradient descent is allowed. In practice I' is usually 200 for MuJoCo based meta-RL domains, and applying MAML with 200 steps of inner loop certainly intractable. we only perform K steps of Bellman backup for the inner loop, where K is small number, we would obtain Q function that is greedy in K steps, which gives us very limited performance. In fact, we realized this limitation only after implementing this method, where we were never able to get it to work in even the easiest domain.
B SAMPLE EFFICIENCY AND ANALYSIS FOR EXTRAPOLATION TASKS
Half-Cheetah-Ve--MMedium
Hall-Cheotah-Vo-HHard
Ant-Dir
250
Average relur
-200
-100
200
150
Average
-400
-400
-600
-660
-50
0.0
0.2
0.4
Time steps (1e6)
0.6
0.00
0.26
0.00
Time steps (1e6)
0.75
1.00
125
1.50
1/76
2.00
steps (1e6)
08
Cheetah-Negated-JJint
Ant-Negatedd-oints
1000
Average return
Average return
180
400
100
0.5
08
1.2
1.4
Time steps (1e6)
steps (166)
MIER (Ours)
MIER-wR (Ours)
PEARL
MQL
MAML
Figure 5: Extrapolation performance OOD tasks. In all experiements,
method exceeds matches
the performance previous state-of of the art methods. We also observe that experience relabeling crucial to getting good performance on out-of ofe-offrribbuiio tasks.
12

---

Under review as a conference paper at ICLR 2021
Half-Cheetah--el
Ant-Dirr
Average Retum
Avsrago Rotum
300
200
-100
MIER (Ours)
-100
-00
MER-RR (Ours)
PEARL
175 20 223 2.7 30 32h 15 375 40
Target Velecity (m/6)
Dredtoo
MAML
Figure 6: Performance evaluated on validation tasks of varying difficulty. For Cheetah Velocity, the training distribution consists of target speeds from 0 to 1.5 m/s, and so tasks hecome harder left to right along the axis. Ant Direction consists of training tasks ranging from 0 1.5 radians, the hardest tasks are in the middle.
C IMPLEMENTATION DETAILS
Please see the released codebase for code to meta-train models and extrapolate to of--istriibuutin tasks. We also include code for the simulation environments included in the paper.
C.I DATASETS
All experiments are run with OpenAl gym (Brockman et al. 2017). use the mujoco simulator (Todorov al. 2012) and are run with 3 seeds (We meta-train models, and run extrapolation for each). The metric used to evaluate performance is the average return (sum of rewards) over a test rollout. The horizon for all environments 200. For the meta-RL benchmarks (Fig. 2b. performance on test tasks is plotted versus number of samples meta-trained on. The out-of-distribution plots (Fig. 4 and 6 report performance
algorithms meta-trained with the same number of samples (2.5M for Ant Negated Joints, and 1 1.5M for all other domains). For the standard meta-RI. benchmark tasks, we use the settings from PEARL (Rakelly et al., 2019) for number of meta-train tasks, meta-test tasks test task. For the oittt--distiittioon experiments, the values
and data points for adaptation used for datasets are listed in Table [ The description of the meta-train and meta-test task sets for -utto--disttibution tasks is included in Section 6.2
C.2 EXTRAPOLATION EXPERIMENT DETAILS
For the settings with varying reward functions, the state dynamics does not differ across tasks, and so we only meta-train reward prediction model. We only relabel rewards and preserve the (state, action, next state) information from cross task data while relabelling experience in this setting, For domains with varying dynamics, we meta-learn both reward and state models.
When continually adapting the model to out of distribution tasks, we first take number of gradient steps (N) that only affect the context, followed by another number of gradient steps (M) that affect all model parameters. We also note that if the model adaptation process overfits to the adaptation data, using generated synthetic data will lead to worse performance for the policy. To avoid this, we only use 80% of the adaptation data to learn the model, and use the rest for validation. The model is used to produce synthetic data for a task only if the total model loss on the validation set below a threshold (set to -3).
Table I: Settings for out-of-distribution environments
Environment
Meta-train tasks
Meta-test tasks
Data points for adaptation
N
M
Cheetah-vel veluuhhuul
100
30
200
10
100
Cheetah-vel hel-hard
100
30
200
10
100
Ant-direction
100
400
20
Cheetah theettahnnegated-jonnt
10
10
400
10
0
Ant-negated-jone
400
10
Walker-rand-parame
40
20
400
10
100
13

---

Under review as a conference paper at ICLR 2021
C.3 HYPER-PARAMETER.
For the MIER experiments hyper-parameter are kept mostly fixed across all experiments, with the model-related hyperparameters set default values used in the Model Based Policy Optimization codebase (Janner et al. 2019), and the policy-related hyperparameters set to default settings in PEARL (Rakelly et al., 2019), and their values are detailed Table 2 We also ran sweeps on some prper-parameter, detailed in Table 3
For the baselines, used publicly released logs for the benchmark results, and code released by the authors for the out-of-distributio tasks. Jyyee-parameten were set to the default values in the codebases. We also swept on number of policy optimization steps and context vector dimension for PEARL, similar to the sweep in Table 3
Table 2: Default Hyper-parameters
(a) Model-related
(b) Policy-related
Hyperparameter
Value
Hyperparameter
Value
Model arch
200-200-200,
Critic arch
300-300-300
200
Policy arch Discount factor
300-300-300
Meta batch size
0.99
Inner adaptation steps
Learning rate
3e-4
Inner learning rate Number of cross tasks for rela-
0.01
Target update interval
1
20
Target update rate
0.005
belling
Sac reward scale
1
Batch-size for cross task sam-
lc5
Soft temperature Policy training batch-size
1.0
pling
256
Dataset train-val ratio for model
0.8
Ratio of real synthetic data for continued training
0.05
adaptation
Number policy optimization
steps per synthetic batch genera-
tion
Table 3: Hyper-parameter sweeps
Hyper-parameter
Value
Selected Values
Number of policy optimization steps per meta-training iteration
1000, 2000, 4000
1000
Context vector dimension
5, 10
Gradient norm clipping
10, 100
10
All experiments used GNU parallel (Tange 2011) for parallelization, and were run GCP instances with NVIDIA Tesla K80 GPUS.
14


Reviews:
[{"review_id": "_5N5zNGwmzv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary: this paper proposes a novel meta-RL method that works as a 2-stage pipeline (with the 2nd stage being optional). The first stage does model identification by having a network doing meta-SL on a learned model that predicts transitions and rewards. This model uses a MAML-like algorithm to adapt a context variable that is passed to a policy network. This policy network does regular RL taking the context variable (describing the task) and the states as inputs. This approach alone would work well in regular meta-RL, but we aim at robustness under distribution shift, for which we need the second part: experience relabeling. There we use the adapted model from the first part to relabel the data coming from other tasks in meta-training and continue training the policy on that synthetic data.\n- Pros:\n    - 1. The method is nice, simple and well-founded.\n    - 2. We're turning a meta-RL problem into a meta-SL + RL problem; that's a nice decomposition.\n    - 3. Generalizing outside of the meta-distribution is useful and making the approach consistent is a good requirement \n    - 4. The paper is well-written and clear and the experiments are well-executed.\n- Cons:\n    - 1. I think the experience relabeling shouldn't work well whenever the state distribution changes substantially and thus we haven't seen the states at meta-training time. For instance, in the ant domain where we go to a corner that has never been explored, there shouldn't be much states to relabel in that portion of the state-space (the experimental results in that particular experiment are also worse than MQL). That defeats the extrapolation purpose of the relabeling and should be discussed further. This issue needs to be discussed further. W.r.t. this point, at the end of section 4 it is claimed that MQL will perform worse when the data support is different because MQL assumes it's the same support, which feels similar to what I'm saying; however, then I don't understand why MQL performs better in Ant-Dir (the example domain I gave).\n    - 2. The paper mentions the proposed approach being composed of \"two independent novel concepts\", but each has been studied before: model identification is common in controls and experience relabeling has the same feeling as Hindsight Experience Replay ` https://arxiv.org/abs/1707.01495 ` or the off-policy corrections of this work ` https://arxiv.org/pdf/1805.08296.pdf .`\n    - 3. The experimental results are underwhelming: in the in-distribution the results are roughly on par with PEARL and MQL; which is to be expected since that's not the main contribution of the paper. However, the out-of-distribution tasks results are quite underwhelming, given that this is the only algorithm with out-of-distribution in mind: the proposed approach is only significantly better than MQL in 2/5 and statistically worse in 1 and also only better in 1/5 than its ablation MIER-wR.\n- Clarity: high\n- Significance: medium\n- Originality: medium-low\n- Questions:\n    - Negated joints means that putting 'a' in that joint is the same as putting -a if the joint was not negated?\n    - My understanding is that we are _not_ backproping the policy training back to $\\phi_T$ and then back to the model inner loop training that produced $\\phi_T$. Is that correct? If we don't, wouldn't it make more sense to do so?\n- Note:\n    - The explanation of appendix A was useful, thanks for including it.\n- Reasoning behind decision: this paper proposes combining two relatively well-established ideas in a novel and potentially-useful way. If the extrapolation experiments were more convincing I would lean towards acceptance; but, as this is currently not the case, I am hesitant to endorse it.\n\n=============\nUPDATE POST DISCUSSION\n\nThank you for your responses.\n- I am more convinced about the novelty of the proposed methods.\n- I am still unconvinced about relabeling being the best we can do for out-of-distribution state distributions. Building learned methods that generalize more broadly or many-inner-step meta-learners could help quite a bit.\n- **I got concerned that a question by AnonReviewer2 \"Why for in-distribution experiments, Experience Relabeling was removed?\" went unanswered.** Having evaluated MIER-wR, adding MIER to the plots during rebuttal should have been very easy. Moreover, in practice, we shouldn't be able to leverage the fact that we know a task is in-distribution or out-of-distribution and choose between MIER and MIER-wR accordingly (which would also be a bit ugly). Therefore, in my opinion, putting MIER-wR as a representative of the paper instead of MIER is not valid.\n\nSince most of my concerns pre-discussion are still there and we cannot assess whether MIER does well on in-distribution (and MIER-wR is not better than MQL for OOD tasks) I, unfortunately, have to decrease my score from 5 to 4.\n\n- This prompted me to take a closer look at the actual numbers of the plots and I got more concerned. I didn't take it into account in my rating update because I should have brought this up before the discussion: \n   1. I feel something is off with PEARL vs MIER-wR when looking at in-distribution vs OOD in half-cheetah vel. In particular, we see PEARL does considerably better than MIER-wR on in-distribution (Fig 2), but its performance starts already much lower than MIER-wR in Fig 6 left. \n   2. Another concern is whether the numbers for OOD are \"decent\" in the sense of creating meaningful policies. Half-cheetah-vel-hard has a reward of ~-325, which is lower than MQL with 0 data for in-distribution half-cheetah-vel. This may be because the evaluated tasks are different, but raises concerns that maybe all baselines and MIER are all doing poorly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DMorMguf8wn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new meta-RL algorithm containing two main components: 1. Model Identification 2. Experience Relabeling. Model Identification models the next state and reward function condition on the previous state, action, and context.  Experience Relabeling module uses the model learned in the previous step to generate synthetic transitions to improve policy learning. Results on in-distribution tasks are comparable with the previous method and it shows good performance on the out-of-distribution task.\n\n1. Paper claims on page 4 second paragraph that the main difference between context variable in this paper and previous works is that gradient descent is used to adapt the context. That is not entirely true as previous work like MQL does use gradient descent to adapt the context (MQL should be cited there too). Am I missing something here?\n\n2. On page 3, the second paragraph from the bottom, you claim that \"a sufficient condition for identifying the task is to learn the transition dynamics and the reward function, and this is exactly what model-based RL methods do.\" If that is true, why did you use context? I don't think this method will work without context (an ablation study would have been nice with/without context). \n\n3. I am not sure if I fully follow your argument about gradient-based meta-learning with value-based in Appendix A or at least it is a very weak argument about the relation between Q function convergence, bellman backup iteration used in actor-critic, and gradient descent.  Can you clarify a bit?\n\n4. Can you explain how Experience Relabeling helps with out-of-distribution tasks? As far as I understood from your paper, model(\\hat(p)) is trained using **current** tasks distribution and this model will be then used to generate synthetic data. Then how come it can help with out-of-distribution tasks when it only saw in-distribution tasks? \n\n5. Why for in-distribution experiments, Experience Relabeling was removed? experience relabeling should be as useful as for in-distribution tasks as for out-distribution ones.\n\n6. Do you **only** update context parameters in Eq. 1? not \\theta? if that is true, figure 1 is very confusing as it implies the model( \\theta) and context will be updated together. \n\n7. How context is constructed? is it a recurrent network? or? \n  \n8. Is it really required to use MAML regression to learn the model? wouldn't be enough to just use simple regression to learn the model?\n\n9. This paper writing needs improvement especially sections 3 and 4 as they don't have a smooth flow. It made me read these two sections multiple times to understand this MIER.  And introduction needs improvement too.\n\n10. Utilizing other benchmarks like meta-world [https://meta-world.github.io/] will be more useful than the current benchmarks to evaluate this method per MQL findings.\n\n11. The paper would have been stronger, if the authors had provided ablation studies in which they closely study different component of their method (w/wo context, simple regression to build model, show who close or far generated data from actual out-of-distribution tasks, etc.)\n\n12. Results in this paper are neither that convincing nor that strong. This method only shows good performance on Cheetah-Negated-Joints and Ant-Negated-Joints on out-distribution.  In addition, I don't understand why this method can't have better performance on in-distribution tasks.  Model + context should be more expressive than just using context ( e.g. MQL vs. this method.)\n\n\n------- Update after rebuttal ---------\n\nThanks for your response. Even though your responses clarified some of my comments, I still don't understand how Experience Relabeling can help with OOD and why your method doesn't good enough with in-dist data. As as result, I stay with my current evaluation and score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5WEifGhq8s", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel meta-RL algorithm that can both meta-train and adapt to new tasks in an off-policy way. Multiple ideas are incorporated into the algorithm. Meta-learning parameters for both the context variable and the policy then updating them through gradient descent endows the policy with strong learning ability, while the dynamics model allows the reuse of training data by relabeling them. The authors conduct experiments in two types of task settings, which are in-distribution tasks and out-of-distribution tasks. Experiments show that MIER outperforms some existing methods on out-of-distribution tasks in terms of sample efficiency, while achieving comparable results on in-distribution tasks. The work is original, and has a certain contribution to the meta-RL setting.\n\n## Pros:\n1. Adaptation ability by modeling the dynamics. Leveraging the strong ability of gradient based models and consistency of the dynamics model, MIER can adapt to both in-distribution and out-of-distribution tasks. \n2.Sample efficiency by reusing training data. Most gradient based methods require a large number of samples to calculate the loss function for gradient descent. Although a similar off-policy idea has been adopted in MQL, the authors use the training data in another way, which is by relabeling them. This technique weakens the assumption in MQL, thus is suitable for both in-distribution and OOD tasks.\n3. Clarity of the paper. The authors clearly describe the motivation of the proposed algorithm, which is to achieve sample efficiency on OOD tasks. The authors extend the meta-RL problem setting to in-distribution and out-distribution tasks with varying reward functions and dynamics, and conduct experiments on these settings with multiple existing methods. The experimental results are sufficient to support most of the claims.  \n\n## Cons:\n1. Assumption of simple MDP. The authors assume that a few steps of exploration are enough to capture the key information of tasks and guide the dynamics model, which might be insufficient in more complex changes in the MDP.\n2. Complexity of the algorithm. In the experiments, the authors treat the information whether the task is in-distribution or out-of-distribution as known, which is unrealistic in most cases. The full adaptation process, which requires two stages of policy gradients and one stage of data relabeling, is much more complex than that of most existing methods.\n3. Limitation of the dynamics model on OOD tasks. The adaptation to OOD tasks is based on the assumption that the dynamics model gives true or near-true transitions, which limits MIER’s ability. In the experiments, the authors conduct experiments that do not change some properties of MDP, more specifically, the state transition remains the same given the desired action. For some other meta-tasks where the fundamental of MDP changes, e.g. Walker-Rand-Params, the dynamics model may output undesired state-action pairs on the unseen tasks since it was not trained on these MDPs.\n4. Uncertainty of MIER. Fig. 5 shows a strong uncertainty of the training curves and the training process does not seem to have converged for all experiments but Half-Cheetah Velocity. Since the paper focuses on adaptation to OOD tasks, the authors should discuss this phenomenon in the main body of the paper.\n5. Necessity of data relabeling. Fig. 6 raises a question for the necessity of data relabeling in terms of the final performance, since both MIER and MEIR-wR converge to similar performance in most OOD cases.\n\n## Questions:\n1. Assume the information whether the task is in-distribution or out-of-distribution is unknown, so that data relabeling is performed anyway, will MIER still achieve a good performance on the in-distribution tasks?\n\nAlthough the paper is well-organized and mostly well-written, I would like to point out some typos:\n1. On page 5, in Algorithm 2: N_m is not mentioned in the algorithm.\n2. In section “Experimental Evaluation” on page 6 & In C.1, Appendix C: “mujoco” -> “MuJoCo” .\n3. In section “Citation”: The authors should pay more attention to the rules of references.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4aqG9SHGxBa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents MIER, a novel meta-RL method that is designed specifically to be able to deal with out-of-distribution test tasks. The idea is to learn two separate modules: a dynamics/reward model that consists of model parameters $\\theta$ and context parameters $\\phi$. This model is trained using a MAML-like gradient-based supervised meta-learning algorithm, and updates the context vector $\\phi$ using gradient descent per task (and the initialisations of $\\phi$, $\\theta$ are meta-trained), together with a policy that is conditioned on the \"context vector\" $\\phi$, which is task-specific after the gradient update. In order to adapt to out-of-distribution tasks at test time, the dynamics/reward model can (1) be trained with the new data, updating both $\\phi$ and $\\theta$, and (2) be used to train the policy using data obtained from he new model.\n\nImpression:\nThe problem setting of out-of-distribution adaptation in metaRL is very interesting and relevant to the community. I think the idea of separating the model and the policy makes a lot of sense, since the model can be used to generate synthetic data on a new task for training the policy. I'm a bit underwhelmed by the empirical evaluation, and on the one hand I have a lot of open questions about what MIER is capable of doing, and on the other hand I do think that MIER *can* potentially do quite interesting things but the experiments in the paper don't really show this. I find the \"extrapolation over dynamics\" experiments most interesting. Here the MIER method really shines compared to other approaches. I think the paper could be made stronger by investigating the properties of MIER on a wider set of interesting experiments, and really analysing what it does and why it works well for certain problems (see concrete questions/suggestions below). I also feel like the adaptation to out-of-distribution tasks is a bit heuristic (see question below) and it's unclear to me how this would work on other / real world problems. Overall, I think this method is of interest to the community, but I also think the impact could be much higher with more discussion and empirical analysis. Hence this is a borderline paper for me at this moment.\n\nQuestions:\n1. In practice, you would have to determine whether or not to update $\\theta$ in addition to $\\phi$, and for how many gradient steps. \n a. How do you do that, and how can you prevent under- and over-fitting? \n b. Since $\\theta$ isn't trained to adapt quickly to new tasks (unlike $\\phi$), could it happen that with little data this actually is a really bad initialisation and I won't be able to appropriately adapt? So even though MAML is consistent, this doesn't really help me if I don't have a lot of data.\n c. If you're looking at an out-of-distribution task, do you update both $\\phi$ and $\\theta$? Do you update both at the same time, or first $\\phi$ and when that's not enough you update $\\theta$? I guess you'd want to make sure that *most* of the new info goes into $\\phi$ since that will make training the policy with the learned model easier. Is that true?\n2. The pre-update policy is responsible for data collection. I expect this to be somewhat random, which works well in settings where even random rollouts contain enough information about the task. But what if that's not the case? I can see this becoming an issue in settings with sparse rewards. Do you have a sense for the exploratory behaviour that the pre-adaptation policy exhibits when collecting $\\mathcal{D}_{adapt}$? In some follow-up papers to MAML (ProMP https://arxiv.org/abs/1810.06784, E-MAML https://arxiv.org/pdf/1803.01118.pdf) it has been shown that the pre-adaptation behaviour in standard MAML can be quite poor. In the environments you evaluated MIER on, the rewards are so dense that even on out-of-distribution tasks, a somewhat random pre-adaptation policy can gather data that is useful to update the model. How would you handle cases where this is no longer the case (e.g., sparse reward settings or environments that require better spatial exploration to learn about the task)?\n3. In the abstract you write \"Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data for on-policy meta-training\". Can you explain this a bit more? Why is an on-policy more likely able to extrapolate well compared to an off-policy method? Do you have examples of methods where this can be seen empirically? \n4. What is the end-performance of RL2, MAML and ProMP when trained for longer? I agree that sample efficiency of MIER is a bonus, but I think we would get a better sense of how the model works and compared to baselines if we let those baselines converge.\n5. I disagree with some statements about model consistency: \n - Paragraph after Eq 2: \"the model will still adapt to out-of-distribution tasks given enough samples and gradient steps, since the adaptation process corresponds to a well-defined and convergent learning process.\"\n - First paragraph of section 4: \"although the gradient descent adaptation process is consistent and will continue to improve (...)\".\n - The thing I disagree about is that updating $\\phi$ only is not a consistent learning algorithm, because $\\phi$ is only an input vector, and might not be expressive enough such that a completely new task can eventually be learned. So to my understanding, this argument only holds for $\\theta$ (+$\\phi$). Could the authors comment on this?\n\nSuggestions:\n- It would be nice to add numbers to all equations, not just the first one because this makes it easier to talk about them. In my review I refer to equation numbers by just counting them.\n- Sometimes I'm a little bit confused what you mean when you write \"model\". It's often the dynamics/reward model, but it's easily confused with the policy (e.g. last paragraph in introduction), and it's also not always clear if you refer to $\\theta$ or $\\phi$ (e.g., second sentence after Eq 2;  2nd+3rd sentence on page 5; and when you talk about model consistency). The readability of the paper can be improved by making this more explicit by saying \"the dynamics model\" or \"the model parameters $\\theta$\" where appropriate.\n- Zintgraf et al (2019) show that you don't need to meta-learn the initialisation of $\\phi$, but you can always set it to to $0$ before the inner-loop update, which saves memory and is easier to implement. I can imagine that this helps with learning the policy because the values of $\\phi$ might fluctuate less during meta-training. The policy that collects $\\mathcal{D}_{adapt}$ then also *always* gets $\\phi=0$ as input, which again could stabilise training. It might also be a way forward to look at ways to learn a good exploration policy (because now it's easier for the policy to *know* that it hasn't gathered any information yet). Might be worth a try!\n- I found it weird that you compare to GrBAL on one of the environments, since it's much worse anyway. If you add this method, then it would help to explain why you compare to it, and what you can take away from this comparison. (And if there's no take-away, consider moving it to the appendix because it just distracts otherwise.)\n- The experiments section has little discussion about why MIER works the way it does, in comparison to other methods. Especially on the \"extrapolation over dynamics\" section, where MIER really shines, a more in-depth discussion or even analysis of what is going on would really strengthen the paper, and help the read understand the method. E.g., for the \"shared support assumption hypothesis\" that you mention compared to MQL - can you design an algorithm where you explicitly test this?\n- It would also be interesting to add an analysis / plot on how good the dynamics model is doing when it is updated on a new task, and how much data it needs to learn this. \n\n--------------------------------------------------------------------------------------------------\nUPDATE\n\nI read the other reviews and the author's responses. Thanks for replying to my questions, that made some things clearer!\n\nSome final comments: \n- \"We performed a hyper-parameter search to find the best settings of the steps to take on $\\phi$ and $\\theta$ for each of the environments. To ensure that we don’t over-fit the model when adapting for extrapolation, we use 80% of the available test-data to train the model, and use the rest for validation.\" - If I understand this correctly you tune the number of gradient steps on $\\phi$/$\\theta$ on part of the out-of-distribution adaptation data, and evaluate on the rest. That feels somewhat restrictive to me and might not always be possible - in which case your results are a bit optimistic. In the response to R1 you wrote that \"the knowledge of in-distribution or out-of-distribution is not necessary to apply our method\", but in this case the question is how to determine how to decide how often to adapt $\\phi$, then $\\theta$, before training the policy. At the very least, this should be discussed in more detail in the main part of the paper, given that this is central to your method. \n- Thanks for pointing me to the appendix of the paper that has the end-performance of your baselines. However I do think these should be included in your paper in order for the reader to get the full picture. There might be differences in implementation (as an example, some papers use different horizon lengths in MuJoCo) so it's not always possible to compare numbers across papers, and your paper should be self-sufficient. \n- Not meta-learning the initialisation of $\\phi$ (but set it to zero before every inner-loop update) seems like an important detail to me - this means in Eq (2) you don't actually take the gradient w.r.t. $\\phi$. It might also be an important to stabilise policy learning (because $\\phi$ doesn't change too much over the course of meta-training). Even if it is an implementation detail, it should at least be mentioned in the appendix so that somebody who wants to re-implement your method can reproduce results!\n- Using something like [2] for good pre-adaptation exploration is a good idea, I agree! (And I agree it's out of scope, but might be worth mentioning in the paper.)\n\nI think overall the idea is promising but the paper falls short in terms of how the method is evaluated. I feel like too many things are buried in the appendix, and it remains unclear to me if MIER can, under realistic circumstances, really adapt to OoD tasks. I agree with R2 that Meta-World might be a suitable benchmark to evaluate MIER on, since  the training and tests tasks are distinct and there's a clear evaluation protocol for ML10 and ML45 (you have to adapt within 10 episodes) which would make it easier to compare to existing methods. \n\nGiven the above I stand by my initial rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was evaluated by 4 knowledgeable reviewers and got mixed scores. While most reviewers appreciated the new intuitive approach to meta RL. there were severe concerns about algorithmic choices and the evaluations that led to a poor score from some reviewers. These concerns are summarized below:
- The motivation of experience relabeling for out of distribution samples is not clear (R2)
- It is unclear why experience relabeling does not work for in distribution samples (R2, R4)
- The reported performance is not a fair comparison as it is typically not known when a task is in-distribution or out-distribution, so we would either have to take always experience relabeling or never (or learn when do use which algorithm)
- The paper falls short in terms of evaluations (R3, R4), in particular it remains unclear to me if MIER can, under realistic circumstances. It is suggested to use more established benchmarks such as Meta-World to evaluate the performance of MIER. 

For the given reasons, I recommend that the authors do these corrections and  go through another round of reviews at another conference.

Decision: Reject