Consistent Meta-Reinforcement Learning via Model Identification and Experience Relabeling
Authors: Russell Mendonca
Xinyang Geng
Chelsea Finn
Sergey Levine
Venue:  (2020)
Keywords: Meta-Reinforcement Learning
Reinforcement Learning
Off-Policy
Model Based

Abstract: Reinforcement learning algorithms can acquire policies for complex tasks automatically, however the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning has enabled agents to leverage prior experience to adapt quickly to new tasks, the performance of these methods depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data due to on-policy training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, even if policies and value functions cannot. These dynamics models can then be used to continue training policies for out-of-distribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task. 

Under review as a conference paper at ICLR 2020
CONSISTENT META-REIINFORCEMEN LEARNING VIA MODEL IDENTIFICATION AND EXPERIENCE RELABEL- ING
Anonymous authors
Paper under double-blind review
ABSTRACT
Reinforcement learning algorithms can acquire policies for complex tasks auto- matically, however the number of samples required to learn diverse set of skills can be prohibitively large. While meta-reinforceement learning has enabled agents to leverage prior experience to adapt quickly to new tasks, the performance of these methods depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, can do so at the expense of requiring extremely large amounts of data due to on-policy training. In this work, we present model identification and experience relabeling (MIER), meta-reinforcemenn learning algorithm that is both efficient and extrapolates well when faced with oul-of-distribution tasks test time based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, even if policies and value functions cannot. These dynamics models can then be used to continue training policies for out-of- distribution tasks without using meta-reinforceemnnt learning at all, by generating synthetic experience for the new task.
INTRODUCTION
Meta-reinforcemmet learning algorithms can enable acquisition of new tasks from just a small number of samples by leveraging experience from previous related tasks (Duan et al., 2016 Wang et al. 2016; Finn et al. 2017). However, the performance of these methods on new tasks depends crucially on how close the tasks are to the meta-training task distribution. Meta-learned models can adapt quickly to tasks that are similar to those seen during training, but can lose much of their benefit when adapting to tasks that are too far away from the meta-training distribution. This places significant burden on the user to carefully construct meta-training task distributions that sufficiently cover the kinds of tasks that may be encountered test time.
Many meta-RL RL methods either utilize a variant of model-agnostic meta-learning (MAML) (Finn et al., 2017 Rothfuss al. 2018: Nagabandi et al., 2018),
inference- formulation with recur- rent Duan et al. 2016]] Wang et al. (016)) attentional (Mishra et al.] 2017). or variational (Rakelly al. 2019) methods. The latter class of methods generally fails to handle -iitt--ddittibttionn tasks, be- cause the model cannot adequately deal with fuittf-distribution inputs corresponding to the new task. Most of the former class of methods, where gradient-based adaptation corresponds to a well-defined and consistent learning process, require on-policy samples, resulting in high sample complexity for meta- tota-training in order produce efficient adaptation. In this paper, we make use of a simple insight to develop model identification and experience relabeling (MIER), meta RI. algorithm that is both efficient and which extrapolates effectively when faced with out-of-distribution tasks at meta-test time: we recognize that dynamics and reward models can be adapted consistently, using MAML-style update rules with off- off-policy data, even if policies and value functions cannot. These models can then be used to train at all, by generating synthetic experience for the new task.
policies for out-of-distrine diit----iitiittiion tasks without using meta-reinforcement learning
To be able to quickly learn dynamics models, we reformulate the meta-reinforceement learning problem as one of MDP identification. We use this approach since identifying a task involves determining its transition dynamics and reward function, which is exactly the information that model of the

---

Under review as a conference paper at ICLR 2020
task needs to represent. Specifically, we use supervised meta-learning method that optimizes for dynamics model initialization such that conditioned on a MDP context descriptor, prediction error on validation batch of data sampled from the task is minimized, after updating only the context descriptor with a few gradient steps. Effective model training requires the validation batch to contain data corresponding to optimal behavior for the tasks, which we obtain by training universal policy conditioned on the context descriptor. Note that since our formulation ensures that the context descriptor contains sufficient information about the task, the policy does not need be meta-trained or adapted, and can hence be learned with simple and efficient off- ooflpo RL algorithms, without needing to handle the complexity of meta-reinforcement learning.
At test time, given out-of-distributie tasks, the adapted context descriptor would indeed be out of distribution and thus our context-conditionel model and policy might not perform well for the test task. However, since our method uses gradient descent which is consistent learning method for adaptation, we can continue to improve our model using more gradient updates. We then leverage all of the experience collected from other tasks during meta-training, by using the learned model to relabel the next state and reward information, thus obtaining synthetic data to continue training the policy.
Our main contribution is an off-policy meta-RL algorithm that is sample efficient, stable and which extrapolates well to out-of-distribution tasks. By formulating the meta-adaptation as MDP ideniifica- tion, we are able to transform the meta-RL. problem into a supervised meta-learning problem and thus benefit from the stability and consistency of supervised learning methods. The consistency of our model also enables us to continue improving our policy without collecting extra data by relabeling data collected from other tasks, thus allowing us efficiently adapt out-of-distribution tasks.
2 RELATED WORK
Meta-reiiforremment learning algorithms extend the framework of meta-learning (Schmidhuber 1987;
Thrun & Pratt 1998 Naik & Mammone 1992) to the reinforcement learning setting. Current meta- reinforcement learning algorithms be roughly categorized into 3 large categories, model-free context based methods, model-free gradient based methods and model based methods.
Model-free, context based methods such Duan al. (0166); Wang et al. (0016); Mishra al. (2017); ; Rakelly et al. (2019); ; Humplik et al. (2019) often encode the experience during adaptation into a latent context variable, and the policy is conditioned on the task specific context to adapt to a given task. The context encoding process is often done via recurrent network (Duan et al. 2016 Wang et al. 2016), via an attention mechanism (Mishra et al.. 2017) or via amortized probabilistic inference (Rakelly et al.] 2019; Humplik et al. 2019). While many of these methods can efficiently summarize experiences during adaptation, is often hard for them to encode any universal learning algorithm such as gradient descent. This is because given a fixed distribution of training tasks, the most efficient way of adapting to a particular task is to directly infer it from the task distribution, rather than applying any universal learning algorithm. While efficient in handling in distribution tasks, the lack of universal learning algorithm in these methods implics that they would be able to adapt out of distribution tasks. Our method, the other hand, is able to handle out of distribution tasks through continual adaptation with gradient descent.
Model-free gradient based meta-RL RL methods, such as Finn al.) (207)); Rothfuss al. (2018);
Zintgraf et al. (2018); Rusu et al. (2018); Liu et al. (2019), implement gradient descent as the adapta- tion process. The parameters are often optimized such that the model achieves good performance after only a few steps of gradient descent. Since gradient descent is a universal learning algorithm, with a large capacity model, these methods are consistent in the way that they would guarantee to improve even on out-of diitr---iittibttion tasks. However, most of these methods are based on on -o-ooiicy RL algorithm, which means they are rather sample inefficient at training time and requires more data to be collected when adapting to new tasks. It is worth noting that there are also works that combine gradient based and context based methods such as Lan et al. (2019). However such methods still suffer from the same sample efficiency problem other gradient based methods because of the use of on- policy gradient descent. Our method mitigate this problem by applying an off- policy RL algorithm top of a gradient based meta- learning algorithm to achieve better sample efficiency at training time. At test time, our method reuses the experiences collected during training to enable fast adaptation without collecting lot more data.
2

---

Under review as a conference paper at ICLR 2020
Model based meta-RL methods such as Nagabandi et al. (2018) ; Sæmundsson al. (2018). a rapidly adapting model instead of a policy is learned. Since learning a fast adapting model is a supervised meta-learning problem, broader range of meta-learning algorithms can be applied. If universal learning algorithm is used to represent the adaptation process, the model learning can then be made consistent such as in Nagabandi et al. (2018). At time, when the model is adapted to particular task, standard planning techniques such as model predictive control is often applied to select actions. While possible be consistent, the model based meta-RL methods often suffer in long horizon tasks due to the accumulated error of model while planning long time ahead into the future. Our method does not suffer from this problem since we only use one step prediction of our model train model-free policy, and only execute the policy during test time. This approach is similar to methods presented in Sutton (999]; Janner et al. (2019).
PRELIMINARIES
3.1 REINFORCEMENT LEARNING
Formally, a reinforcement learning problem is defined by a Markov decision process (MDP). We adopt the standard definition of MDP T (S,A, P, Mo- Tr 7:), where S is the state space, A is the action space, p(s' [s, a) is the unknown transition probability of landing on next state s' at the next time step when an agent takes action a state s, (o(s) the initial state distribution, r(s, a) the reward function, and (0, 1) is the discount factor. An agent acts according to some policy i (aas)) and the objective of learning is to maximize the expected rewards E,, EE, //,/Aee (r((s)
3.2 MODEL BASED REINFORCEMENT LEARNING
In model based reinforcement learning, a model p(s', rrss, a) that predicts the reward and next state from current state and action is trained using standard supervised learning approaches. The model then used to generate data to train policy. Specifically, given model, we perform the following optimization to obtain the policy: arg max; [g pel r(s,, az)]. Note here the expectation is taken with respect to the model distribution of states instead of the true environment distribution.
3.3 META-REINFORCEMEER LEARNING
In meta-reinforrcement learning, we represent various of similar tasks with a distribution of MDPs (TT), where each specific task is sample drawn from the distribution. Given specific task T, the agent allowed to collect little amount of data Delept adapt the policy to form सт according to the data. The objective of this problem is to maximize the expected rewards of the adapted policy ET (T) /E,7/(ss, 7/r(st, a:)].
3.4 SUPERVISED META- AND MODEL AGNOSTIC META LEARNING
We briefly introduce the supervised meta-learning problem and the model agnostic meta-learning approach, which is important foundation of our work. In supervised meta-learning, we also have distribution of tasks p(T) similar to the meta-RL setup, except that the task T is now a pair of input and output random variables (X,, Yr). Given small dataset DTT adapt sampled from a specific task T. the objective to built a model that performs well on the evaluation data (TT) eval sampled from the same task. If we denote our model as f(X; 0), the adaptation process as E(0, adapt) and our loss function as L. the objective can be written as:
min E ET (C((( (C : ((, D )), Y7]] J.
Model agnostic meta-learning (Fin et al. 2017) is approach to solve the supervised meta-learning problem. Specifically, the model f[X; 0) is represented as neural network, and the adaptation process is represented few steps of gradient descent. For simplicity of notation, we only write the one step of gradient descent:
Exson (8, D(T) D(TTpp)) = 8 a.
X.Y [C(f(X;6).Y) YY
adapt
3

---

Under review as a conference paper at ICLR 2020
Note that because Exama is the standard gradient descent, it does not contain any trainable parameters and hence the trainable parameters are all contained in the network weights 8. Therefore, the training process of MAML can be summarized optimizing the loss of the model after few steps of gradient descent on data from the new task.
4 CONSISTENT META-OPTIMIZATION BY RELABELING EXPERIENCE
In this section, we lay the meta- training and test time adaptation process of MIER. We first re- formulate the meta-RL problem into a meta-model identification problem, where we train fast adapting model to rapidly idee- tify transition dynamics and re- ward function for a given new task. We parameterize the model with a latent context descriptor to contain all task specific in- formation aquired during adap- tation. We then train a universal policy that condition on context descriptor to solve all tasks sam- pled from the training distribu- tion. At test time, given a po- tentially out of distribution task, we continual to improve our pol- model to rela- bel data from previously expe- rienced tasks. The consistency of our model adaptation process and policy improvement ensures that the policy will continue to improve even the task out-of- distribution. We provide the de- tailed explain of each part of our method in the following sections, and we lay out the pseudo code for meta-training time in Algorithm and test time in Algorithm
Algorithm 1 -eta-Training of MIER
INPUT: task distribution p(T), model training steps Nmodet. policy training steps Npoliey, learning rate a
OUTPUT: policy parameter 4 model parameter 8, model context ø
Randomly initialize policy parameter 1/, model parameter 0, model con- text Φ
Initialize multitask replay buffer R(T) < 0
while 0, d. & not converged do
Sample task T ~ p(T) Collect data batch D'7) addppt before adaptation using अe and d
Compute adapted context ØJ = Euvan. (0, o, Dadage)) Collect data batch D after adaptation using н and Ør Add data replay buffer RRT) R(T) U D(T) for i = 1 to Nmodet do Sample task in replay buffer T ~ RR
adapt ((D
Sample two batches of data adape: D D(Td R(T)
Train model (0, Φ) (0, x) Vo a. 0, D(T) end for i 1 Nppaticy do
adept: D coal)
icy by using
Sample task in replay buffer T ~ R
Sample two batches of data ()T)
eval ~ R(T)
adapt
Compute adapted context Φा = EMaMa (0, 6, ((T) aslapt)
Train policy vo + v (V,
end
end
4.1 MODEL IDENTIFICATION WITH LATENT CONTEXT
In mL problem, tasks are drawn from distribution of MDPs, and agent adapts to each of the tasks by observing some data for the tasks. The adaptation process is usually performed as either an implicit or explicit inference of some task representations. That is, the agent adapts to task by identifying it from other tasks in the same distribution. In distribution of MDPs, the only varying factors are the dynamics p and the reward function r. Therefore, sufficient condition for identifying the task is to learn the transition dynamics and the reward function. and this is exactly what model based RL methods do. Hence, we can naturally formulate the meta-task identification problem as model based RI. and solve it with supervised meta-learning methods.
Specifically, we choose the MAML method for its simplicity and consistency. Unlike the standard supervised MAML formulation, we condition our model on a latent context vector, and we only change the context vector when adapting to new tasks. With this formulation, we restrict all the task specific information learned during adaptation to the context vector, and thus allowing the policy to condition on it. Let us denote our model as p(s', r/s, a: 6, Φ). where 9 is the neural network parameters and ó is the latent context vector that is passed in input to the network. One step of gradient adaptation can be written as follows:
ΦJ Exaoa (8, o. (T) )) = Φ a (s.,s' (s.,"')) D(T) |- log p(s' r/s, a; 8, Dodopt))
ndopt
4

---

Under review as a conference paper at ICLR 2020
We use the log likelihood as objective for the probabilistic model. Then evaluate the adapted context vector ΦT and minimize its loss on the evaluation dataset to learn the model. Specifically, we minimize the model meta-loss function Ja(O, o, D Padopt: D to obtain the optimal parameter 0
and context vector initialization d).
arg min Jp(O, 0, DeTepe D(T) 8,0
odept: eval arg min EE 0.0 (a.a,s' D() aval
[- log pis' r/s, a; 0, ΦT)
The main difference between our method and previously proposed context based meta- RL methods (Rakelly et al. 2019 Duan et al., 2016) that we use gradient descent adapt the context. Therefore, given model with large enough capacity, the adaptation process is consistent, in the sense that it will continue to improve regardless of which task it is presented with. In-distribution tasks will of course have much faster adaptation, since the meta-training process explicitly optimizes for this objective, but the model for out-of-distribution tasks will still adapt to the task given enough samples and gradient steps, since the adaptation process corresponds well-defined and convergent learning process.
4.2 POLICY OPTIMIZATION WITH LATENT CONTEXT
Given the latent context variable from the adapted model, the meta-RL problem can be effectively reduced to a standard RI. problem, as the task specific information has been all encoded in the context variable. Therefore we can apply any model-free RI algorithm to obtain a policy, long condition the policy on the latent MDP descriptor context.
Specifically, we choose the soft actor critic (SAC) method (Haarnoja al. 2018) for its sam- ple efficiency and stability. Let us parameterize our policy Ty. by a parameter v. The soft actor critic method maintains estimate of the values for the current policy * a, ΦT)
Bn. E,, y'r(s;, as)[sp = au = a. 7] and V** (s, ΦT) = aerr Ee ("r, as) =
8, 'T through Bellman backup, and improves the policy by minimizing the KI. divergence between the policy and the exponential advantage Jr (vi, D) ED[DKI ( exp(Q=* V** }}] over some dataset D. Note that we condition our value functions QTV, VTh and policy ग all on the adapted task specific context vector Φा so that the policy and value functions can access full information about the task.
4.3 CONTINUAL ADAPTATION AND EXTRAPOLATION THROUGH DATA RELABELING
At meta-test time, when the model is given an unseen task T. it will first sample a small batch of data and obtain the la- tent context Φt by running the gradient descent adaptation pro- cess on the context variable. If the new task is in-distribution, we can directly feed Φt to the policy. However, in many cases, we might encounter tasks that are out of distribution. If the task T is out of the meta-training distri- bution, then the obtained latent context oy will also be out of distribution for the policy. Even though our model is adapted with a consistent method, the improve- ment guarantee is only for the model - the latent context Φr that might be obtained for out- of-distribution task might not im- mediately produce effective policy. However, with improved model, can generate new
Algorithm 2 Test Time Adaptation of MIER
INPUT: test task T, multitask replay buffer R(T). model training steps Nondet, policy training steps Npotieg, policy parameter sb. model parameter 0, model context é, learning rate
OUTPUT: policy parameter vb model parameter 0, model context é
Collect data batch (T)) from task 7 using my and Φ
Compute adapted context Φ4 Exaaa (0, d. D Palapt) while 6, & not converged do for 1 to Nmodee do
Train model +- a. ( |- logg'(ss 0, end for i= to Nynticy do Sample task in replay buffer T ~ R
Sample data batch D(T) R(T)
Relabel data DCF) + (elabbl(D(T)) 00007)
Train policy to + 1 aV v. ((T)
end end
5

---

Under review as a conference paper at ICLR 2020
data for the task without sampling from the environment to continue training the policy, providing sample-efficient (but computationally more costly) adaptation method for out-of dit-of--istrbbution tasks.
When using data generated from learned model to train policy, common caveat is that the model's predicted trajectory often diverges from the trajectory under the real dynamics in long horizon tasks, due to accumulated error. In the special case of meta-RL,, can be avoided by the multi-task setup in meta-RL. Although we have never seen this particular new task, have seen many other tasks that share the same state space and action space, and have collected a lot of experience from these tasks. Using the adapted model and policy, we can relabel the transitions (s, a, r) from any other tasks by sampling a new actions with our adapted policy, and sample next states and rewards with the adapted model. The relabeling process can be written as Relabel(D, 0, ΦT) = {(s, a, s', (s) a) D; (s',r) p(s', r/s, a: 0, ΦT)). We use these relabeled transitions to continue training the policy. By doing so, we effectively transferred experiences from other tasks into this new task, and thus avoided the divergence problem of sampling long horizon trajectories as we only doing one step prediction. This relabeling scheme is similar to the Dyna algorithm introduced by Sutton (199T). By relabeling experiences from tasks seen before, we can efficiently reuse the large amount of data collected from other tasks to improve policy this new task.
argue that this issue
5 EXPERIMENTAL EVALUATION
We aim to answer the following questions in our experiments: (1) Can MIER meta-train efficiently on standard meta-RL benchmarks without requiring large amounts of data? (2) How does MIER compare to prior meta-learning approaches for extrapolation to meta-test tasks with cut-of-distribution (a) reward functions and (b) dynamics?
(3) How important cross task relabeling leveraging the model to learn effective policy efficiently?
To answer these questions, we first compare the sample efficiency of MIER to existing methods on several standard mL. benchmark environments. We then test MIER on a set of environment with out distribution meta-test tasks to analyze extrapolation performance. We further run an ablation of method without cross task labelling, and show that this isn't able to adapt the policy as effectively as our method.
5.1 SAMPLE EFFICIENCY ON META-RE RL BENCHMARKS
Figure I: HalfCheetah and Humanoid robot simulated in MuJoCo physics engine. Our experiments use these two domains.
We evaluate MIER on two standard meta-RL benchmark environments based on the MuJoCo physics simulator (Todorov et al. 2012), shown in Figure These environments were introduced by Finn al. (2017). and later used by the state-of-the-ar context based meta-RI. method Rakelll al. (2019). They include half cheetah robot running at different velocities (HalfCheetah-Wal Vel) and humanoid robot running in different directions (Humanoid-De
For all environments, we follow the exact setting in Rakelly et al. (019)). For comparison, we include performance of the PEARL algorithm (Rakelly et al. 2019) and MAML algorithm (Finn et al. (017) as our baselines, and we average the performance of all algorithms across 3 random seeds.
We plot the test time performance after adaptation versus number of steps in the environment in Figure 2 We that method is able achieve performance comparable to the state of the art

---

Under review as a conference paper at ICLR 2020
method in meta-RL in terms of sample efficiency (PEARL). This is due to the ability of our method to perform supervised meta- veta-learning for the model, and off ooolyyy reinforeement learning for the policy, which results in stable and efficient optimization procedure during meta- training. Note that here, off-policy refers to the meta-training process. However, as we discuss in Section 4.3, our method can also perform ooolpy adaptation via cross task relabeling. We evaluate this in the next section.
Chectah-VSS Vel Standard Banchmank
Humanord Standladd Benchmark
-50
1002
NIER (O.rr)
MAVE
PEARL
900
-150
1AML
200
Rooe
-250
-300
300000 400200 500000
750000 B00000
200000 400000 500000
805003
Figure 2: Performance on standard meta-RL benchmarks for left: Half Cheetah velocity right: Humnaood direction. The return is evaluated on (in-distribution) meta-test tasks at different steps during the meta-trant meta-training process, following the protocol from Rakelly et al. (2019), Note that both our method and PEARL are substantially more efficient than the on-policy MAML algorithm, and our method performs comparably to the off-policy PEARL algorithm.
5.2 EXTRAPOLATION TO OUT-OF-DISTRIIUTION TASKS
We evaluate the extrapolation capability of MIER separately environments with varying reward functions, and those with different dynamics. In order to ensure fair comparisons, we train all algorithms convergence before testing on new tasks. Hence, MAML trained for 80 millions steps. while PEARL and MIER only require about a million steps. Unless stated otherwise, we adapt with MIER using cross task relabeling in all of these experiments.
5.2.1
ADAPTATION TO DITTTRRUDDTTIIIIIIO REWARDS
For testing extrapolation ability to tasks with different rewards, we take the HalfCheetah- Vel environ- ment from the standard benchmarks and restrict the meta-training task distribution. This is required since, in the original environments, the meta-training distribution covers almost the entire feasible task space, leaving little room for extrapolation. Instead, if we limit the training distribution, we can easily find out-of-distribution tasks test the extrapolation capability of our method.
The training tasks consist of target velocities at which the cheetah needs to run sampled uniformly from to 1. We then evaluate the ability of the meta-learned polices to adapt in order to run at new speeds chosen at 0.5 intervals from 0.5 to 2.5, using only 2 trajectories sampled from the environment. From Figure 3 we see that MIER attains much higher return than other methods for target velocities that are substantially outside of the meta-training distribu- tion. Eventually, performance degrades for all methods, since the state space distribution differs too much from the meta-training tasks, making cross task relabeling inef- fective. Nonetheless, this experiment shows that MIER's ss capacity to adapt to outts--dist tasks substantially exceeds that of prior meta-RL methods.
HalfChestah Tarqet Velocity Extrapolation Perfor mance
-1.0
Avsrage Return
-300
-400
MAML
NIER (Oum)
PFARI
0.50
0.75
2.00 Target Velacity
175
Figure 3: Target velocity (reward function) extrapolation for HalfCheetah environment.
5.2.2 ADAPTATION TO OUT-OF- DISTRIBUTION DYNAMICS
To study adaptation to uuttof-distribution dynamics, we take the HalfCheetah environment, and randomly negate the control of 3 joints out of the total joints of the robot. During meta-training, only select joints from the first joints to negate, and during meta-test time, we select joints from the first 5 and always negate the 6th joint. This ensures that the agent never sees the negation of the
7

---

Under review as a conference paper at ICLR 2020
6th joint during meta-training time, and hence needs to extrapolate well to get good return on the test tasks. In total, we have 10 training tasks and 10 test tasks.
We compare the return of all methods after adapting on 2 new trajectories of the validation task. From the plot of average performance across test tasks, in Figure 4 we see that MIER achieves superior performance compared to existing methods. We able to extrapolate efficiently since able to leverage off- off-policy data collected during meta- meta-training. This turn enabled by our ability learn model quickly using minimal data, which the consistent meta-training objective.
HolfCheetah Dynomics Extrapolation Performance
Relabelling Analysis
1000
1000
Average Katurn
800
NIER (Ou's) tack relaballing
400
400
200
T
PEARL NAML
MIER (Ours)
Number of relabelling steps
Figure 4: Dynamics extrapolation HalfCheetah with negated joints. Left: Return after adaptation for each method. Right: Ablation analysis on the effect relabeling. The results show that MIER with cross task relabeling substantially outperforms prior methods when adapting to this task with out-of-distribunt dynamics. The ablation analysis shows that the cross-task relabeling is essential to attain this level of performance.
To further highlight the importance of relabeling data from other tasks during adaptation, we run an ablation where the policy is learned only using the sampled data. Figure 4 shows performance of the policy plotted against the number steps where we relabel exsiting data and continue to train the policy. We see that without the benefit of off-policy data, the policy performance deteriorates rapidly due to overfitting to the small batch of data sampled from the test task.
6 CONCLUSION
In this paper, introduce consistent and sample efficient meta-RL algorithm by reformulating the meta-RL. problem as model identification. Our algorithm can adapt to new tasks by determining the parameters of the model, which predicts the reward and future transitions. This allows us to perform meta-learning via supervised learning of the model, which is more stable and sample efficient. More importantly, this provides with consistent adaptation procedure, where adaptation is performed via gradient descent. This means that, even for oul-of-distribution meta-test tasks, our method eventually learns the right model. This model then allows us relabel past experience to reuse it for the new task, running off- off-policy reinforcement learning to acquire the policy for the new task. Experiment results show that our method achieves superior performance compared to existing methods, especially adaptation of the model followed by relabeling and off-policy reinforcement learning substantially outperforms prior methods that adapt to the new task directly.
-iitr--distriuutionn tasks that requires extrapolation, where consistent
REFERENCES
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RI:: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.0277,, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, pp. 1126-1135. JMLR. org, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor, arXiv preprint arXiv:1801.01220 2018.
8

---

Under review as a conference paper at ICLR 2020
Jan Humplik, Alexandre Galashov, Leonard Hasenelever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint 1905 06424,, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. arXiv preprint arXiv: 190060825553 2019.
Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta reinforcement learning with task embedding and shared policy. arXiv preprint arXiv: / 190.006577 2019.
Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta-reinforcement learning. In International Conference Machine Learning, pp. 4061-4071, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXi::1707.03144 2017.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt dynamic, real-world environments through meta-reinforcement learning. arXiv prepriní arXiv:1803: 1347, 2018.
Devang K Naik and RJ Mammone. Meta-neural networks that learn by learning. In /Proceedings 1992] IJCN International Joint Conference on Neural Networks, volume 1, pp. 437-442. IEEE, 1992.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinforcement learning via probabilistic context variables. arXiv preprint arXiv: 1rXiv:I903.080 1903.0824, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.0677 2018.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05566) 2018.
Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint 1xXiv:8803.0753) 2018.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 264):160-163, 1991.
Sebastian Thrun and Lorien Pratt. Learning learn: Introduction and overview. In Learning learn, pp. 3-17. Springer, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Jane X.
Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel 7. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew M Botvinick. Learning to reinforcement learn. ArXiv, abs/1611.000 abs/1611.05763. 2016.
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Caml: Fast context adaptation via meta-learning, arXiv preprint 180.008,,,,3642,, 2018.
9

---

Under review as a conference paper at ICLR 2020
Appendices
A IMPLEMENTATION DETAILS
The implementation of the algorithm largely builds on the Model Based Policy Optimization code (Janner et al. (2019)), and the enviromments are standard meta-RL benchmarks, with slight modi- fications for the extrapolation tasks described in the experiment section. The context vector is variable that is appended to the states and actions when sent to the model, and appended to the states when passed to the policy. The ppper-prrameters major change is that environments with changing dynamics have models predicting both next state and reward, while the environments with only changing reward have prediction just for reward.
kept mostly fixed across all experiments. One
Some key hyperparameters (common across experiments):
1. Number of fast adapt steps for model meta-training.
2. Fast adapt learning rate for the model 0.1
3. Batch size for model adaptation 256
4. Reward-dynamics model architecture: 200-200-200-200
5. Dimension of context vector 5
6. Regression weight on post-update contexts 0.01
7. Number of tasks sampled per training epoch
8. Number of data points sampled from a task per training epoch 1000
9. Number of SAC training steps per training epoch 1000
10. Learning rate for Q functions in SAC: : 0.0003
Q network hidden layers : 256-256
10


Reviews:
[{"review_id": "S1lwHLn2tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe authors propose an algorithm for meta-rl which reduces the problem to one of model identification. The main idea is to meta-train a fast-adapting model of the environment and a shared policy, both conditioned on task-specific context variables. At meta-testing, only the model is adapted using environment data, while the policy simply requires simulated experience. Finally, the authors show experimentally that this procedure better generalizes to out-of-distribution tasks than similar methods.\n\nMajor comments\n--------------\nMaking meta-rl algorithm generalize better outside of the meta-training distribution is a relevant open problem, and this work proposes nice ideas towards its solution. The paper is well-organized and easy to read. The idea of reducing meta-rl to a task identification problem is not completely novel since some recent works have been proposed in this direction (see later). Anyway, the proposed approach is interesting and seems (at least from the proposed experiments) effective. My main concerns follow.\n\n1. Though they attempt to address all relevant questions about the proposed approach, I found the experiments quite weak. Only two Mujoco domains are used for the standard meta-rl experiment, and only one of them (HalfCheetah) is used to test the out-of-distribution capabilities. Regarding the first experiment, MIER always performs comparably or worse than PEARL. What is the intuition behind this result? Does it suggest that MIER is paying additional sample complexity in \"in-distribution\" tasks in order to be more robust to out-of-distribution ones? On the other hand, the generalization experiments seem much more promising, but I would like to see more (at least the humanoid robot as well) to confirm that this result is not only a specific case of this domain. Furthermore, from Figure 3 it seems that MIER improves over PEARL even on in-distribution tasks, while it performed significantly worse in Figure 2. Why does this happen?\n\n2. Related to the previous point, I did not find any description of the parameters adopted in all experiments (learning rates, batch sizes, etc.). I do not believe I would be able to reproduce the results at the present time.\n\n3. The proposed method is somewhat related to other recent works [1,2]. In particular, [2] presents similar ideas, where the authors meta-learn a fast-adapting model (actually, a task encoder) and a shared universal policy conditioned on the task representation. The main focus is still to improve the generalization to out-of-distribution tasks. Can the authors better discuss the relations to these works?\n\nMinor comments\n--------------\n- In the introduction: \"Effective model training requires the validation batch to contain data corresponding to optimal behavior for the tasks...\". Why? In principle we could train a good model of the environment by running a sufficiently-explorative policy.\n- In the related works: \"Our method does not suffer from this problem since we use our model to train a model-free policy\". It is not clear why (though it becomes later) since simulating long trajectories from a learned model could lead to the usual divergence issues.\n- In Sec. 3.2: r in \\hat{p} should not be bold. Also, \"f\" in the subscript of the expectation was not defined (is it \\hat{p}?).\n- In Sec 3.4: there is a minimization over \\phi which however does not appear in the objective.\n- In the optimization problem at page 5: \\phi_{\\Tau} should probably be \\phi.\n- In Fig. 2, why is MIER run for less steps than the other algorithms?\n\n[1] Humplik, J., Galashov, A., Hasenclever, L., Ortega, P. A., Teh, Y. W., & Heess, N. (2019). Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424.\n[2] Lan, L., Li, Z., Guan, X., & Wang, P. (2019). Meta Reinforcement Learning with Task Embedding and Shared Policy. IJCAI 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlNvGfaFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###  Summary\n1. The paper proposes an algorithm capable of off-policy meta-training (Similar to PEARL) as well as off-policy policy adaptation (By relabelling previous data using the adapted model and reward function). \n\n2. The basic idea is to meta-learn a model that can adapt to different MDPs using a small amount of data. Moreover, the adaptation is done by only changing the latent context vector (Similar to CAVIA or CAML). The remaining parameters of the model (theta) are fixed after meta-training. \n\n3. The paper also proposes learning a universal policy that, when given the context vector of a task, can maximize the reward for that task. This means that for with-in distribution meta-testing tasks, the policy can be used as it is (by giving it the right context vector which can be computed by adapting the model). For out-of-distribution tasks, however, it is important to update this policy. \n\n4. To update the policy, the paper proposes combining previously stored data (for example data used in meta-training) with the adapted model to do off-policy learning (Using SAC).  \n\n### Decision with reasons \n\nI vote for rejecting the paper in its current form for the following reasons:\n\n1- The paper assumes that it is possible to learn models for out-of-distribution tasks with a few samples that are accurate on all the previously stored data. This is fundamentally incorrect. If the MDP changes in a significant way, it is not reasonable to expect that we can adapt a model from a few samples. Moreover, even if we can adapt the model using a lot of new experience, it is not reasonable to expect that we can use this model to accurately label all previous data. The authors do acknowledge this when describing results in Figure 3, however they seem to underplay this limitation. \n\n2- Turning the meta-RL problem into a supervised learning problem has already been explored. For instance, Nagabandi et al. (2018)[1] showed that it is possible to quickly adapt models to changes using meta-learning. They, however, used decision time planning for the control policy (By random shooting method). This paper, on the other hand, uses Dyna style planning with an off-policy learning algorithm on previously stored data. The only difference is the choice of off-the-shelf planning algorithm which is not a significant contribution (There are some other small differences, such as learning a context vector and not model initialization, learning a universal policy etc, however, I don't see how they are essential for the proposed approach; maybe the authors can clarify why those choices are essential) \n\n3- The paper assumes a context vector alone is sufficient to capture changes in MDPs (It keeps the rest of the model fixed at adaptation). This might be reasonable if the context vector is sufficiently large, but the paper does not even mention the size of the context vector. It also skips other important details. For example, it does not mention any details about hyper-parameter selection, how the context-vector used in the model, etc. It's hard to judge the importance of the experimental results because of this. \n\n### Questions \n\n1- \"Effective model training requires the validation batch to contain data corresponding to optimal behavior for the tasks, which we obtain by training a universal policy conditioned on the context descriptor\"\n\nIt's not clear to me why the validation batch must contain data corresponding to the optimal behavior. \n\n2- Is the proposed framework really consistent? At adaptation, only the context vector is being updated whereas model parameters (theta) are fixed. Why is a context vector alone sufficient to adapt the model to drastic changes in the MDP? \n\n\n[1] https://arxiv.org/abs/1812.07671\n\n### UPDATE\n\nThe authors gave a detailed response to the reviews and answered some of my main concerns. However, I'm still not convinced that the paper, in its current form, can be accepted. My issues are: \n\nThe paper combines some existing ideas in a new way but falls short of justifying the choices it made. The proposed contribution is that it is consistent (meta-learning methods that learn a network initialization are also consistent), can do off-line meta-training (So can PEARL) and can use old meta-training data at meta-test time (This is novel to this paper). However,  the proposed methodology also has some downfalls. For example: \n\nIt does not allow continual adaptation. This is an important limitation of existing consistent meta-learning methods and this paper does not address it. Nagabandi et al 18 [1], on the other hand, propose a similar solution that is also capable of continual adaptation. \n\nMOST IMPORTANTLY, the empirical evaluation in the paper is very unsatisfactory. Even though the authors have included hyper-parameters in the appendix in the updated version of the paper, they still do no specify how these parameters were selected. Were the parameters selected to maximize the performance of their method and then copied for the baselines? This would not be a fair comparison. \n\nGiven the above-mentioned issues, I don't think the paper in its current form can be accepted and I'm maintaining my initial score. I think the authors should do a more thorough empirical investigation and tune the baselines and their method separately (using comparable compute budget). They should also report results on multiple environments using the same parameters (i.e. tune hyper-parameters on one or a few environments and reports results on some other environments as commonly done in Atari) \n\n[1] Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgRKlV0OH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel approach to reformulate the meta-RL problem as model identification with a gradient descent based algorithm. Such innovation not only allowed us to perform meta-learning via supervised learning of the model, which is more stable and sample efficient, but also allowed us to leverage off-policy RL to learn the policy instead of meta RL.\n\nPros:\n1. Paper clarity. Although the submission had a few typos (I am not a native English speaker, but I'd encourage the authors to polish the writing of the paper), it's a very well-written paper overall. The flow and logic of this paper was clean, and the authors stroke a good balance between being focused about the core contribution of the paper, and reviewing related work and introducing sufficient preliminaries. As a result, I think this paper was accessible to both domain experts and the broader ICLR community. \n\n2. Novelty. This paper proposed a novel approach to reformulate the meta-RL problem as model identification with a gradient descent based algorithm. To the best of my knowledge, this was the first paper broke the meta-RL problem into a simpler meta supervised learning problem and an off-policy RL learning problem. Although each component of the proposed solution was not new, e.g., \"relabel\" was used in Dyna, MAML was first introduced in 2017, the combination of each component to address the meta-RL problem seemed to the novel to me. And I think the idea could be interesting to the ICLR community. \n \n\nCons:\nIt's weak accept rather than accept from me because of how the empirical evaluation were conducted in the paper, and I think the experiments conducted in the paper were a little bit weak (common for most ICLR submissions). Examples:\n\n1. Number of gradient steps is an important tuning parameter for MAML, it would be interesting to discuss number of gradient steps within the context of MIER.\n2. It might be useful to conduct some qualitative results to understand the model learned with MIER against the baselines, e.g., how well MIER adapt to the out-of-distribution tasks with simulated data points (examples o such qualitative studies could be found, say, in Finn et al., 2017).\n3. Given the fact that one major contribution of this paper was reformulating the meta-RL problem as model identification, it would be useful to conduct some quantitative study to help the readers understand the effectiveness of learning the environment model p(s’, r|s,a) compared to ground-truth, and how the quality of the learned environment model made an impact on the overall performance of the model.\n4. Some implementation details of MIER were missing, I don’t feel confident about how reproducible this research would be. For example, the specification of both environment and policy models were not discussed in the paper.\n5. In general, it would be useful to conduct more experiment results on more diverse data sets, say, in a supplement material.\n\n\nA few questions to the authors:\n1. Section 3.2: I assume the expectation should be taken w.r.t. p’ rather than f?\n2. In Algorithm 1 & 2, how was the adapted context \\phi_T used to update policy \\psi? Was it as input to the model parametrized by \\psi? It might be useful to make it clearer.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an algorithm for meta-rl which reduces the problem to one of model identification. The main idea is to meta-train a fast-adapting model of the environment and a shared policy, both conditioned on task-specific context variables. At meta-testing, only the model is adapted using environment data, while the policy simply requires simulated experience. Finally, the authors show experimentally that this procedure better generalizes to out-of-distribution tasks than similar methods.

The reviewers agree that the paper has a few significant shortcomings. It's unclear how hyper-parameters are selected in the experimental section; the algorithm does not allow for continual adaptation; all policy learning is done through data relabelled by the model. 

Overall, the problem the paper addresses is very important, but we do not deem the paper publishable in its current form.
Decision: Reject