CoMPS: Continual Meta Policy Search
Authors: Glen Berseth
Zhiwei Zhang
Grace Zhang
Chelsea Finn
Sergey Levine
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement Learning

Abstract: We develop a new continual meta-learning method to address challenges in sequential multi-task learning. In this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinforcement learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly. We introduce a new method, continual meta-policy search (CoMPS), that removes this limitation by meta-training in an incremental fashion, over each task in a sequence, without revisiting prior tasks. CoMPS continuously repeats two subroutines: learning a new task using RL and using the experience from RL to perform completely offline meta-learning to prepare for subsequent task learning. We find that CoMPS outperforms prior continual learning and off-policy meta-reinforcement methods on several sequences of challenging continuous control tasks.

Under review as a conference paper at ICLR 2022
COMPS: CONTINUAL META POLICY SEARCH
Anonymous authors
Paper under double-blind review
ABSTRACT
We develop continual meta-learning method to address challenges in quential multi-task learning. this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinnfrreemet learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly. We introduce new method, con- tinual meta-policy search (CoMPS), that removes this limitation by meta-training in an incremental fashion, over each task in a sequence, without revisiting prior tasks. CoMPS continuously repeats two subroutines: learning new task using RL and using the experience from RL to perform completely offline meta-learning to prepare for subsequent task learning. We find that CoMPS outperforms prior continual learning and off-policy meta-reinforcement methods on several sequences challenging continuous control tasks.
INTRODUCTION
Meta-reinforcemeet learning algorithms aim to address the sample complexity challenge of conven- tional reinforcement learning (RL) methods by learning to learn - utilizing the experience of solving prior tasks in order to solve new tasks more quickly. Such methods can be exceptionally powerful. learning to solve tasks that are structurally similar to the meta- meta-training training tasks with just few dozen tri- als (Finn al., 2017; Duan et al., 2016; Wang et al., 2016; Zintgraf et al., 2020). However, prior work on rentorreinor learning is generally concerned with asymptotic meta-learning performance, or how well the meta- teeiner policy can adapt to single new task at the end of long meta-traiminn period. The meta-training process itself requires iteratively attempting each meta-training task in a "round-robin" fashion. While this is reasonable in supervised settings, reinforcement learning revisiting and repeatedly interacting with previously seen tasks in the real world may be costly or even impossible. For example, when learning to tidy in different homes - effective generalization requires visiting many homes and interacting with many items, but needing revisit every prior home and item on each iteration of meta-training would be impractical. Instead, we would want the robot use each new experience in each new home incrementally augment its skillset that can acquire new cleaning skills in new homes more quickly, as shown in Figure using examples from Meta World (Yu et al., 2020b). In this paper, we study the continual meta-reinforcement learning setting, where tasks are experienced one at a time, without the option to collect additional data on previous tasks. The objective to decrease the time it takes to learn each successive task, as well as achieve high asymptotic performance.
This paper proposes a novel meta-learning algorithm for tackling the continual multi-task learning problem in a reinforcement learning setting. The key desiderata for such method are the following. First, an effective continual meta-learning algorithm should adapt quickly to new tasks that resemble previously seen tasks, and at the same time adapt (even if slowly) to completely novel tasks. This adaptation is crucial in the early stages of meta- veeta-trainnn when the number of tasks seen so far is small, and every new task appears new and different. Second, effective continual meta- weta-learning algorithm should be able to use all previously seen tasks to improve its ability to adapt to future tasks, integrating information even from tasks much earlier in training, and are therefore far off-po To address

---

Under review as a conference paper at ICLR 2022
Training/ Interaction
Training/ Interacton
Training /
Do
D
D2
Esseraction
Figure I: the continual meta-RL setting, the agent interacts with a single task at a time and, once finished with task, never interacts with it again. To complete the full sequence of tasks efficiently, the agent must reuse experience from preceding tasks to more quickly adapt to subsequent tasks.
the first requirement, we base approach on model-agnostic meta-learning (MAML) (Finn et al., 2017). MAML adapts to new tasks via gradient descent, while the meta-training process optimizes the initialization for this gradient descent process to enable the fastest possible adaptation. Because the adaptation process in MAML corresponds to a well-defined learning algorithm, even new out-of- distribution tasks are learned (albeit slowly), while tasks that resemble those seen previously will be learned much more quickly (Finn & Levine, 2018). To address the second requirement, and make it possible to incorporate data from older tasks without revisiting them, devise off-policy method where the adaptation process corresponds to on-policy policy gradient. This meta-training uses behavioral cloning on successful episodes experienced by the agent from older tasks. Although the "inner loop" policy gradient adaptation process is on-policy, and the agent adapts to each new task with on on-policy experience, the meta-training process, which is similar to distillation of previously collected experience, is off-policy. Essentially, the agent meta-traing the model such that a few steps of policy gradient result policy that mimics the most successful episodes on each previously seen task. This can effectively enable approach to incorporate experience from much older policies and tasks into the meta-training process. Although our algorithm is intended to operate in settings where new tasks before, it still relies on storing experience from prior tasks, similar to rehearsal- based methods (Isele & Cosgun, 2018; Riemer al., 2018; Rolnick al., 2019; Atkinson al., 2021), but instead uses this information to accelerate sequential task acquisition.
revealed sequentially, one time, in the home cleaning robot example
Our primary contribution is meta-reinforcemeen learning algorithm that supports the sequential multi-task learning setting, where the agent cannot revisit previous tasks to collect data. To evaluate our approach, we modify collection of commonly used meta-RL benchmarks into continual multi- task problems, with tasks presented one at time. Our method outperforms other methods, achieving higher average reward with fewer samples on average over each of the tasks the sequence. In addition, we evaluate each method's ability to generalize over a collection of held-out tasks during training. We find that CoMPS achieves higher meta-test time performance on held-out tasks. Our experiments show that as the agent experiences more tasks, learning time on new tasks decreases, indicating that meta-reinforcement learning performance increases asymptotically with the number of tasks. To our knowledge, our work is the first to formulate and address the continual retf-reeinooremment learning problem, which we propose is realistic formulation of meta-RL for real-world settings, where prior tasks cannot be revisited after they have been solved. While several prior methods can be used in this setting, we show our experiments that they perform significantly worse than CoMPS.
RELATED WORK
Meta-learning, or learning-to-lert (Schmidhuber, 1987; Bengio et al., 1990; Thrun & Pratt, 2012), concerned with the problem of learning prior, given set of tasks, that enables more efficient learning in the future. We focus on meta-learning for reinforcement learning (Schmidhuber, 1987;
Duan al., 2016; Wang al.. 2016; Finn et al., 2017; Mishra et al., 2017). There are many ways to represent the meta-learned model, including black box models (Duan et al., 2016; Wang et al., 2016; Mishra al., 2017; Stadie et al., 2018; Rakelly al., 2019; Zintgraf al., 2020; Sæmundsson et al.,
2

---

Under review as a conference paper at ICLR 2022
2018; Fakoor et al., 2019; Sun et al., 2019; Dorfman & Tamar, 2020), applying gradient descent from initial parameters (Finn et al., 2017; Gupta et al., 2018b; Rothfuss et al., 2018; 8; Clavera et al., 2018; Mendonca et al., 2019; Zintgraf et al., 2019; Mitchell al., 2020), and training a critic to provide policy gradients (Sung et al., 2017; Houthooft et al., 2018; Yu et al., 2018; Chebotar et al., 2019). Recent work has made progress toward using supervised meta-learning in online setting (Ren et al., 2018; Finn et al., 2019; Khodak et al., 2019; Jerfel et al., 2019; Grant et al., 2019; Zhuang et al., 2019; Antoniou al., 2020; Wallingford et al., 2020; Yu et al., 2020a; Yao et al., 2020). Adaptation without task boundaries or inside of an RL episode has also become a new area of investigation for meta-learning (Nagabandi al., 2019; Javed & White, 2019; Harrison al., 2019; Al-Shedivat et al., 2018; He et al., 2019; Harrison et al., 2019). Our work focuses on a distinct problem setting, where for meta-training the RI. tasks are experienced sequentially, and the goal is to learn each RI. task more quickly by leveraging the experience from the prior tasks, without the need to revisit prior tasks. The general on- on-policy meta-RL setting with infinite task sets also corresponds to the continual learning problem well, however, the finite task setting is more representative of the sorts of situations that RL agents are likely to encounter in the real world that benefit from generalization.
Continual or online learning studies the streaming data setting, where experience is used for training as soon as it is received (Thrun & Pratt, 2012; Hazan et al., 2016; Chen & Liu, 2016; Parisi et al., 2019; Khetarpal et al., 2020). Both terms describe the process of learning tasks in sequence while avoiding the problem of forgetling (French, 1999; Rusu et al., 2015; Parisotto et al., 2016; Rusu et al., 2016; Chen et al., 2015; Tessler et al., 2016; Aljundi al., 2019). Recent beeannle based methods have made inroads on the continual meta-learning but they are limited to non-RL problems (K J & N Balasubramanian, 2020; Gupta et al., 2020). Following Rolnick et al. (2019), we do not explicitly aim to address the problem of forgetting, and instead retain data from prior tasks in a replay buffer to use for meta-RL RL training a model that can adapt quickly to new tasks. Other recent methods support continual learning without ground truth task boundaries (Schmidhuber, 2011; Srivastava et al., 2012; Nguyen et al., 2017), but have not yet been demonstrated to perform well in reinforcement learning settings. Even with replay buffers, the data from previous tasks is still challenging to reuse since was collected by a different policy.
Off-policy RL methods are known to achieve good sample efficiency by reusing prior data. Therefore, several recently proposed sample-efficient meta-RL algorithms have been formulated as off-policy methods (Rakelly et al., 2019; Fakoor et al., 2019; Nagabandi et al., 2019; Sæemundsson et al., 2018). 8). In principle, these methods could be extended to the continual meta-learning setting. However, in practice, their ability to utilize data from past tasks collected under older policy is limited, and we find, via our experiments, that they tend to perform poorly in the continual rentrreeifor learning setting, possibly due their limited ability to extrapolate to the new out of distribution tasks.
To meta-train without revisiting prior tasks, our method uses a type of self-imitation. Although this resembles imitation learning, it does not use external demonstrations: meta-self-imitation learning uses high reward experience the agent itself collected in prior tasks. There are number of non-meta- learning based methods that used behavioral cloning, distillation, and self- self-imitation for re-integrating previous experience (Rusu et al., 2015; Parisotto et al., 2016: Levine et al., 2016; Teh et al., 2017 7; Ghosh et al., 2018; Berseth et al., 2018). Our work uses meta-self- self-imitaiionn learning as the outer objective. As such not only trains a model to imitate a previous policy but also trains this policy to adapt given little data quickly. The actual meta-learning procedure in CoMPS resembles GMPS, but with several important changes that empirically lead to significant improvement in the continual meta-learning setting: (1) CoMPS uses a PPO-based inner policy gradient formulation for meta-rl training and RL-based task learning, where GMPS uses vanilla policy gradient for meta-rl -rl training and off-policy SAC (Haarnoja et al., 2018) to train expert policies; by using SAC, a non-policy gradient-based RL algorithm, GMPS can not train expert policies on new tasks using the meta-rl- trained policy parameters to accelerate learning. (2) CoMPS meta- vecta-traiss using only off off-policy data. While these components use prior techniques (Espeholt et al., 2018), their integration in CoMPS and their application to continual meta-reinforcement learning is novel.
3 PRELIMINARIES
Reinforcement learning. RL problems are generally formalized as a Markov decision process (MDP), defined by the tuple MDP (S, A, P, R. P.7, T), with the state space E S, the action
3

---

Under review as a conference paper at ICLR 2022
space E A. a transition probability function P(s'|s, a)) a reward function R(s, a), an initial state distribution p(su), a discount factor 7 E [0, 1] and a time horizon T. The agent's 's actions are defined by policy T[a 8) parametrized by 0. The objective of the agent is to learn an optimal policy: (1)a))
0" := argmax J(O). where ](0) = discounted return.
( R(st, ae)] is the expected
Meta-reinforceemett learning. Meta-learning aims to leverage a set of meta-training tasks to enable fast adaptation on a different set of meta-test tasks not seen during training. MAML (Finn et al., 2017) accomplishes this by meta-training a set of initial parameters 0 over the training tasks to efficiently adapt to a new task. In metta-eiinorremment learning, each RL task T; is a different MDP, with its own task objective Jr. defined as before. The state S and action space A are the same for these MDPs, however their transitions, rewards, and initial states can differ. This meta-training process itself is sample inefficient (even though meta-test time adaptation is fast), requiring on-policy trajectories to estimate the inner policy gradient and many more trajectories for the outer objective. To reduce the cost of needing additional trajectories for the outer objective, Mendonca et al. (2019) propose meta-training with the expected discounted return as the inner task loss and supervised imitation as the outer loss:
min
Σ Σ
ET,
(O, = D.)=
Σ
log π (a 0:). (I)
0
Ti T.
The outer behavioral cloning loss LBC does not require collecting more data from the environment, but on-policy data from the environment is needed for computing the inner update on the policy parameters Φ; 0 + a a (f). This meta-RL method is more sample efficient when we have near-optimal data for the outer behavioral cloning loss LBc, but it cannot be trivially extended to a continual setting, the inner objective requires data to be repeatedly collected from each meta-training task. The following section will outline the continual multi-task learning problem and describe how we can extend GMPS to such setting, removing the need to revisit prior tasks and carefully include a process for the agent to generate its own ncar-optimal data.
CONTINUAL META POLICY SEARCH
In the continual multi-tast reinforcement learning set- ting, which we study in this paper, an agent proceeds through many tasks Ti, one time. The agent's goal is to quickly solve each new task using a learning rule L, achieving high reward as efficiently as possible. In order to accomplish this, the agent can use experience from solving tasks To: to learn how to adapt quickly to each new task T/+1, but cannot revisit past tasks to collect additional data once it moves on to the next task. This differs from the standard reinforcement learning setup, in which there is a single task To, and the standard meta-RI. setting, where all tasks can be revisited as many times as needed during meta-training but is closer to and RL approximation of online meta learning (Finn et al., 2019). After each round of training on a new task, the agent produces new set of policy parameters fg to serve initialization for the next task. The learning rule L needs to serve two purposes: (1) solve the current task; (2) prepare the model parameters for efficiently solving future tasks. This process is depicted in Figure 2.
To
T1 L
T2
0
L
0
02
0g
Figure 2: The continual multi-task rein- forcement learning problem for sequence of three tasks. The agent applies learning algorithm L on task T; and forwards policy parameters 8.+1 after each task.
CoMPS overview. CoMPS addresses the continual multi-task RL problem one task at a time through sequence of tasks T, The I. process for CoMPS consists of two main parts, a rein- forcement learning RL. process to learn the new task involving potentially hundreds of training steps and a meta-RL M process, which uses the off-policy experience from previous tasks to meta-train the initial parameters for RL. We illustrate the flow of these processes for CoMPS in Figure The combination of RL and M creates a solution to the continuous multi-task RL problem that can perform non-trivial learning via meta-learning across tasks to accelerate learning. The RL step consists of using an on-policy RL algorithm to optimize a policy on T, (described next), which benefits from the previous rounds of meta-training and is therefore very fast. This RL training process produces a dataset of trajectories D, = (To. ..., T where {(so, ro,ro), ro)................ ... (ST, aT, r.)}. ). From this dataset, we set aside the experience that achieved the highest reward on the task as D + maX E(aree)er R5 0z) called the skilled experience.
4

---

Under review as a conference paper at ICLR 2022
The details of the RL step, the algorithm used, and the imple- mentation is given in Section 4.1. The meta-training in M uses the experience collected during RL in two separate data sets. The first dataset corresponds to the skilled experience Di.-- which is used in the outer meta-imitation learning objective. The second dataset consists of all the experience seen so far, Do: which is used for estimating the inner policy gradient based on Equation 1. Thus, instead of naively finetuning param- eters on each new task, as in the case of standard continual RL methods, the M step in CoMPS produces meta-trained param- eters that are optimized such that all prior tasks can be learned as quickly as possible starting from these parameters. When there are enough such tasks, these meta-trained parameters can provide forward transfer and generalize to enable fast adapta- tion to new tasks, enabling them to be acquired more efficiently than with naive finetuning. However, to accomplish this, the M step must train from off-policy data from prior tasks. In Section 4.2, we describe how we implement this procedure.
T:
Ti+1
RL
RL
D
D
0
M
D
M
D Po:++1
Figure 3: CoMPS is split in two process: a reinforcement learning step RL and a meta-learning step M. The meta-lcarning step M uses the data gathered thus far, denoted Di. to meta-train 0z. RL is initial- ized from these parameters 0, for the next task.
4.1 TASK ADAPTATION VIA POLICY GRADIENT
To solve each task in the RL step, we use the popular policy gradient algorithm PPO (Schulman et al., 2017). PPO uses stochastic policy gradients to determine how to update the policy parameters 0 compared to a recent version of the parameters that generated the current data 0', using a distribution =(a|s.0)
ration (0) *(a)e.0r)- A first-order constraint, in the form of a gradient clipping term, is used
to limit on-policy distribution shift of ra while optimizing the policy parameters using (0) (lmin(r;,0)) A (llp(r, c)),((r(0), - e, + e) )), The advantage Å.g. = rr+1 + 7V (8t+1) V=,, (8x) is a score function that measures the improvement an action has over the expected policy performance Va (st). CoMPS increases the sample efficiency of PPO in the RL step via initialization from the network parameters 0, from the M step. which performs off off-policy meta-RL, that describe next. During the RL step, for each episode we collect 20 trajectories and perform 16 training updates with batch size of 256. Additional details, including the learning parameters and network design, can be found in Appendix C.
4.2 OUTER LOOP META-LEARNING
The M process of CoMPS meta-trains set of parameters 6; using meta-self-ae self-imitation from
Algorithm 1 CoMPS Meta-Learning
the skilled experience. The outer self-imitation learning objective in Eq. uses the skilled ex- perience Di., to train the agent to be capable of (re)lcarning these skilled behaviors from one or few policy gradient steps using previously logged off-policy experience, sampled randomly from Du.. In contrast to methods that are con- cerned with forgetting, the parameters produced
1: require: 0, skilled Dú., and off-policy Dosi for t O...N do
2:
3:
for j + 0... i do
4:
"D sample m trajectories from D,
5:
oj + 0 + (() (via imp. weights, sub- section 4.3)
6: Sample data Dyal ~ DD
7: Update 0 + 0 BV LBC Djuu)
by this meta-RL training can quickly learn new behaviors that are similar to the high-value poli-
8: end for
9: end for
cies from previous tasks and, if enough prior
tasks have been seen, likely generalize to quickly learn new tasks as well. The use of gradient-based meta-learning is particularly important here: as observed in prior work (Finn & Levine, 2018), gradient-based meta-learning methods are more effective at generalizing to new tasks under mild distributional shift as compared to contextual methods, making them well-suited for continual meta- learning with non-stationary task sequences, where new tasks can deviate from the distribution of tasks seen previously. However, in the continual setting, where prior tasks cannot be revisited, a significant challenge in this procedure is that the meta-RL optimization needs to estimate the policy gradient J(f in its inner loop for each previously seen task, without collecting additional data from the task (which it is not allowed to revisit). To address this challenge, we will utilize an
5

---

Under review as a conference paper at ICLR 2022
Off Off-Policy Meta-Self-lmitation
Reinforcement Learning
update policy O3 02 + V Johs (f)
meta-selflmmitaioon
skilled experience
I0in Σ B.. Typp(T)
D 4 (0:4)
RL
M
off-policy experience
Drr,
collect task experience
off-policy is update @. = 1=8+aVo +VVo
D lo:
D
Dn M
Figure 4: Outline of CoMPS. The left side corresponds the M block for Figure 3 and the right the RL block. On the right (RL), for each round the policy we, is initalized using the previously trained meta-policy parameters. After iterations of RI. training on task i, the experience for task i is collected into the off-policy buffer Do: and skilled experience is stored in another buffer Dū:-- This experience is given M that uses the off- off-policy experience for the inner expected reward updates. The outer step behavior clones from the skilled experience the RL. agent generated itself previously.
importance-sampe update that we describe in Section 4.3, using samples from the full dataset of off-policy experience for that task, Do:i, to estimate inner loop policy gradient. In effect, this procedure trains the model to learn policies that are close to the near-optimal trajectories in Da: by taking (off- coff-policy) policy gradient steps the sub-optimal trajectories in Do.s. The complete meta-training process is summarized in Algorithm I, and corresponds to a reinforcement learning inner update and meta-imitation learning outer update, though imitation uses the agent's own experience without requiring any demonstrations. In our implementation, the skilled data for task i consists of the 20 sccorigg trajectories. We also found that only 5% of experience from prior tasks needed to be stored for good performance. Further details on the networks and hyperparameters used for Algorithm are available in Appendix D.
4.3 OFF-POLICY INNER GRADIENT ESTIMATION FOR META- LEARNING
In this section, we describe the particular form of the inner-loop policy gradient estimator used in Algorithm 1. Although many prior works have studied importance-samplet policy gradient updates, and GMPS (Mendonca et al., 2019) uses an importance-sampling update based on PPO (Schulman al.. 2017), we found this simple importance sampled approach to be insufficient handle the highly off-policy data in Dos This is because the data for the earlier tasks may have been collected by substantially different policies than the data from the latest tasks. To enable our method to handle such highly off-po data, we utilize both an aeroortace-saaplll policy gradient estimator and an inportance-samplee value estimate for the baseline the policy gradients. The former is estimated via clipped importance weights, while the latter uses an estimator based on V-trace (Espeholt et al., 2018) to compute value estimates, which are then used as the baseline. For state Sm given a trajectory (st, dzs T' fu we define the n-step V-trace value targets for ((sm) = Ei 7'T,, as:
m+n=1
L-1 [I Pc(re + (V V(s)))
Um = V(sm) +
Σ t-m m
The values p; = min(p, and C. = min(e, (0)) are truncated importance weights, where P and ē are hyperparameters, and re (0) T(as (oalss,0)/Tl ((elss, 8'), where B' denotes the parameter vector of the policies that sampled the trajectory in the dataset. The value function parameters w are trained minimize the 12 loss (mm - (em))) The V-trace value estimate is then used estimate the advantage values for policy gradient, which are given by Åm I' + Un++1 (sm)s) and the gradient is then given by [gJ(() x 1 E, p: V log no (a.(s;) analogously to PPO and other iaportance-sampler policy gradient algorithms. We this estimator for the inner loop update in Algorithm 1 line 5. We show in our ablation experiments that this approach is needed to enable successful meta-training using the exhaustive off- off-policy experience collected by CoMPS.

---

Under review as a conference paper at ICLR 2022
4.4 COMPS SUMMARY
An outline of the entire CoMPS algorithm, including how data is accumulated as more tasks are solved, is shown in Figure 4. The RL process keeps track of the set of trajectories that achieve the highest sum of rewards D The RL. step returns the separate skilled experience Di and all experience collected during RL training in Dj. The M step uses this experience to perform meta- RL via training a model to learn how to reproduce the best policies achieved from previous tasks. Significantly, this meta-RL. training process can accelerate the RL process even in fully offline settings, allowing the agent to train a meta-RL. -L model without collecting additional experience.
5 EXPERIMENTS
Our experiments aim to analyze the performance of CoMPS on both stationary and non-stationary task sequences in the continual meta-learning setting, where each task is observed once and never revisited again during the learning process. To this end, we construct a number of sequential meta-learning problems out of previously proposed (non-sequential) meta-reinforcemeet learning benchmarks. We separate our evaluation into experiments with stationary task distributions, where each task in the sequence is sampled identically, and non-statioaary task distributions, where the tasks either become harder over time or else are selected to be maximally dissimilar for prior tasks (see discussion below). We deseribe the task domains and the methods in our comparisons below, with further details provided in Appendix A.
Tasks. An illustration of the tasks in our evaluation is provided in Figure 5, along with a visualization of the tasks, and includes the following task families. In the Ant Goal environment, a quadrupedal robot must reach different goal locations, arranged
semicircle in front of the robot. The non- stationary distribution selects the locations that are furthest from the previously chosen location each time, while the stationary one selects them at random. The Ant Direction tasks use the same quadrupedal robot that must now run in a particular direction. The non-stationary and stationary distributions are constructed as above. In Half Cheetah the goal is to control the half-cheetah to at different velocities, either forward or backward. The direction is chosen randomly, but in the non-stationary distribution, the desired velocity magnitude increases over time. Last, we utilize the suite of robotic manipulation tasks from Meta World Yu al. (2020b), which we arrange into sequence. The non- non-stationary sequence orders the tasks in increasing difficulty, as measured by how long regular PPO can solve the tasks individually.
Prior methods. We compare CoMPS both to prior meta-learning methods and to prior methods for continual learning. Since learning without revisiting prior tasks requires an off-policy algorithm, we include PEARL (Rakelly et al., 2019) as a meta-learning baseline, which utilizes the off-policy SAC (Haarnoja et al., 2018) algorithm and can in principle learn without revisiting prior tasks. While several prior methods use policy gradients with meta-RL (Rothfuss et al., 2018: Al-Shedivat et al., 2018), these methods require on-policy data, making them unsuited for this continual meta-learning problem setting. For continual learning, we include PPO transfer learning baseline (denoted PPO+TL), which trains sequentially on the tasks, as well as the more sophisticated Progress & Compress (PNC) method (Schwarz et al., 2018), which further guards against forgetting of prior tasks. Although we don't 't evaluate backward transfer, we still include PNC as a representative example of prior continual learning methods. We also compare to the on-policy MAML method (Finn & Levine, 2018) in online mode (MAML+TL), where each iteration only meta-trains on the latest task. The on-policy MAML method is not designed for this setting, and therefore we expect perform poorly, but we include it for completeness.
Meta-learning over stationary task distributions. In our first set of experiments, we compare the methods on stationary task sequences. The results are presented in Figure 6. The plots show the average number of episodes needed to reach success threshold on each of 40 tasks, lower is better. In this evaluation protocol, once the fraction of successful trajectories exceeds a threshold, the algorithm moves on to the next task, and the goal is to solve all of the tasks as fast as possible. For additional details on how success is computed and the thresholds used see Appendix A. In these experiments, we can see that CoMPS solves all of the tasks the fastest, and in fact solves tasks faster as more tasks experienced, indicating the benefits of meta-learning This indicates the benefits of continual meta-learning where each task enables the method to solve new tasks even more quickly. Without the option collect on-policy data from each task MAML performs very poorly across all
7

---

Under review as a conference paper at ICLR 2022
-30
-20
1.0
22
3.0
(a) Ant Goal
(b) Ant Direction
(c) Half cheetah
(d) MetaWorld
Figure 5: In Ant Goal the non-stationary task distribution selects the next goal location that is furthest
from all previous locations: e.g.. To:s ((5, 0), (-5, 0), (0, .)....) In Ant Direction the tasks start at 0 along the x-axis and rotate cow 70 The Half Cheetah tasks start with low target velocity and alternate between larger +- velocities: e.g., Tow (0.5, -0.5, 1.0,...). In Meta World, set of goals is randomly choosen across a collection of environments including hammar and close-drawer.
Figure 6: These figures show the average number of episodes needed to solve each new task after completing tasks (fewer is better). Results are computed over 6 sequences of 20 tasks, averaged over 6 random seeds with the standard error shown.
environments. Reaching the success threshold on the more challenging Meta World tasks is generally difficult, however, CoMPS still performs the best. In Appendix E we include additional results that show CoMPS receives higher reward on average for these experiments and in subsubsection E. we show the learning behavior of CoMPS is statistically different than other algorithms. The forgetting analysis in Appendix G shows CoMPS exhibits strong backwards transfer as well. The next paragraph provides fine-grained average reward analysis over non-stationary task distributions.
Meta-lcarning over non-stationary task distributions. In our second set of experiments, we evaluate all of the methods on non-stationary task sequences. The results of these comparisons are presented in Figure 7, which shows complete learning curves for each method over the task sequence (left), as well a plot of the average performance on each task (right). The plot on the right is obtained by averaging within each task, and provides clearer visualization of aggregate performance. The plots show that CoMPS attains the best performance across most task families, and the gains particularly large on the higher-dimensional Ant Direction and Ant Goal tasks. Note that the decrease performance on the Half-Cheetah task due to the increasing difficulty of tasks later in the sequence, since the target velocities increase more tasks are observed (see Figure 5 for details), but CoMPS still attains higher rewards than other methods. On Ant Goal and Ant Direction, the average performance of CoMPS increases as tasks are seen, indicating that the meta-learning procedure accelerates the acquisition of new tasks. PEARL generally performs poorly across all tasks: although SAC is an off-policy algorithm, it well known that such methods do not perform well when they are not allowed to gather any additional online data (as, for example, in the case of offline RL) (Kumar al., 2019). PPO+TL and PNC provide strong baselines, but do benefit from meta-learning as CoMPS does, and therefore their performance does not improve as much with more tasks. On the Meta World tasks, the best performing methods are CoMPS, PPO+TL and GPP.+PPO. In the plots the left of Figure 7, we can see that CoMPS generally learns each task faster (as we expect due to meta-learning), but PPO+TL attains somewhat better final performance on most tasks. However, we can see in the Appendix, Figure 11 that CoMPS reaches these rewards in fewer interations compared to PPO+TL and GMPS+PPO.
Ablation study. We perform an ablation analysis to compare CoMPS to GMPS+PPO that does not use E-trace-based off-policy importance sampling. The results in Figure 6 show that GMPS+PPO can not make good use of the off-policy experience and, consequence, performs worse than CoMPS, especially on Ant Goal and Ant Direction. Additional ablation analysis showing the improvements from the importance sampling and training components of CoMPS are provided in Appendix H.
8

---

Under review as a conference paper at ICLR 2022
-200
Ant Gool
PARLA
COMPS
GNPSS
MAYLIT
PEARL
INC
PPO-TL
5000
10000
15400
20:00 Epinades
25000
30000
35490
4000
Ant Dir
121
MANL11
PEARL
PNC
#PO-TL
5000
10000
15000
20000 Episdde
25000
30000
35690
4000
Chemaa
Helf' Cheetah
WANL+T
PEARL
#PONTL
2500
5000
7500
10000 Ep chosodea
12500
15000
17500
2000
Metaworld
CoSpPs GRO
Epiaodes 12500 15500
Figure 7: On the left are "lifelong" plots of rewards received for every episode of training over 40 tasks. The points are subsampled by averaging points together to make one point in the plot. Results are averaged over 6 seeds, Cheetah and Meta World get 500 episodes and Ant-Goal and Ant-Direction get 1000 per task where each episode collects 5000 samples. The right plots show the average return across the episodes each of the 40 individual tasks. CoMPS achieves higher average returns and improves its performance as more tasks are solved.
DISCUSSION
In this work, we proposed CoMPS, a new method for continual meta-reinforcemenn learning. Unlike standard meta-RL methods, CoMPS learns tasks one at a time, without the need to revisit prior tasks. Our experimental evaluation shows that CoMPS can acquire long task sequences more efficiently than prior methods, and can master each task more quickly. Crucially, the more tasks CoMPS has experienced, the faster can acquire new tasks. At the core of CoMPS is a hybrid meta-RL. approach that uses off-policy importance-samplee sampled inner loop policy gradient updated combined with simple supervised outer loop objective based on imitating the best data from prior tasks produced by CoMPS itself. This provides for simple and stable approach that be readily applied policy
wide range of tasks. CoMPS does have several limitations. Like all gradient methods, the variance of the importance weights can become large, necessitating clipping and other tricks. We found that including a V-trace off-policy value estimator for the baseline helps mitigate this, providing better performance even for highly off off-policy prior task data, but better gradient estimators could likely lead to better performance in the future. Additionally, CoMPS still requires prior data to stored and does not provide for any mechanism to handle forgetting. While this is reasonable in some settings, interesting direction for future work could be to develop a fully online method that does not require this. Since CoMPS does not require revisiting prior tasks, it can be practical choice for real-worle meta-reinforcemment learning, and particularly exciting direction for future work is to apply CoMPS to realistic lifelong learning scenarios for real-world applications, in domains such robotics.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
Current applications of robots in the real world require enormous amounts of supervision and instruction from scarce and skilled people. Our work can broaden the use of robotics by reducing steadily the time to learn new tasks, making them smarter as they operate. However, the method we have developed a step in the direction of enabling robots to help provide essential services. Still, there is some risk in that skilled robots can be repurposed for other undesired tasks.
8 REPRODUCIBILITY STATEMENT
In effort to make our work reproduceable we have provided extensive details for the method in the appendix including how our hyperparameter analysis was performed. We have also provided the code used for the experiments. This code includes instructions how to reproduce the experiments.
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference on Learning Representations, 2018. 3.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11254-11263, 2019. 3
Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey. Defining bench- marks for continual few-shot learning. arXiv preprint arXiv:2004. /967,, 2020.
Craig Atkinson, Brendan McCane, Lech Szymanski. and Anthony Robins. Pseudo-rchearsal: Achiev- ing deep reinforcement learning without catastrophic forgetting. Neurocomputing, 428:291-307, March 2021. 2
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Université de Montréal, Département d' d'informatique de recherche .... 1990.
Glen Berseth, Cheng Xie, Paul Cernek, and Michiel Van de Panne. Progressive reinforcement learning with distillation for multi-skilled motion control. In International Conference Learning Representations, 2018. 3
Yevgen Chebotar, Artem Molchanov, Sarah Bechtle, Ludovic Righetti, Franziska Meier, and Gauray Sukhatme. Meta-learning via learned loss. arXiv preprint arXiv: (e0:::906605555 05374, 2019. 3
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv: 1,06411, 2015.
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 1045):1-155, 2016.
Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt: Meta-learning for model-based control. arXiv preprint arXiv: 1347, 3, 2018. 3
Ron Dorfman and Aviv Tamar. Offline meta reinforcement learning. arXiv preprint arXh::2008.02598.
2020. 3
Yan Duan. John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 171.799611.02779, 2016.
2
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL. with importance weighted actor-learner architectures. In Jennifer Dy and
10

---

Under review as a conference paper at ICLR 2022
Andreas Krause (eds.), Proceedings the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 1407-1416, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. 3,
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-learning. 2019.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. 2018. 2, 7,
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1 1112--1122 1226-135, 135. JMLR. org, 2017. 3,
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine Online meta-learning. In International Conference on Machine Learning. pp. 1920-1930, 2019, 3, 4, 22
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 (4):128-135. 135, 1999. 3
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. conquer reinforcement learning. 2018.
Divide-and-
Erin Grant, Ghassen Jerfel, Katherine Heller, and Thomas Griffiths. Modulating transfer between
tasks in gradient-based based meta-learning, 2019. 3
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised Meta-
Learning for reinforcement learning. June 2018a. 16
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018b. 3
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 11588-11598. Curran As- sociates, Inc., 2020. URL t wwwp:r/rwweeiiinneeeneeeeeeeeeee neurips caeascc///o httpppp..cpppppppppppee wwtppppppppppp//eeeeeee $S5gaaaceiccc222aaaeeeee66222222
3
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. 80:1861-1870, 10-15 Jul 2018. 7.7
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. arXiv preprint arXiv:1912.088600 2019. 3
Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends in Opti- mization, 2(3-4):157-325, 2016. 3
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint 1oXiv:9006005000 2019. 3
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.
Deep reinforcement learning that matters. 2017. 20
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Advances in Neural Information Processing Systems, pp. 5400-5409, 2018.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. AAAI, 32(1),
April 2018. 2
Khurram Javed and Martha White. Meta-learning representations for continual learning. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d Alche-Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems 32, pp. 1820-1830 Curran Associates, Inc., 2019. 3
11

---

Under review as a conference paper at ICLR 2022
Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta- meta-learning and continual learning with online mixtures of tasks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 9122-9133. Curran Associates, Inc., 2019. 3
Joseph K J and Vineeth N Balasubramanian. Meta-consolidation for continual learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14374-14386, Curran Asso- neurips oc cc/papprr2222/ELLL. ciates, Inc., 2020. URL https::/ppocceeeeiiiiiieee Paper pof. pdf. 3
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. December 2020. 3
Mikhail Khodak, Maria- Piria-Flooriaa F Balcan, and Ameet S Talwalkar. Adaptive gradient- based meta- learning methods. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Baee -uc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 5917-5928. Curran Associates, Inc., 2019. 3
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. 32, 2019. 8
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39)11-00, 2016. 3
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel. Sergey Levine, and Chelsea Finn. Guided meta-policy search. arXiv preprint arXiv: 00956, 2019. 3, 4, 6, 15
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv: 2017. 2
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. arXiv preprint arXiv:2008.0666 2020.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning Con- tinual adaptation for model-based RL. In International Conference Learning Representations, 2019. 3
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E TuTrner Variational continual learning. arXiv preprint arXiv:1710.10633 2017. 3
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 11:54 71, 2019. ISSN
0893-6080.
Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In International Conference on Learning Representations, 2016. 3
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinforceement learning via probabilistic context variables. arXiv preprint arXiv: I 1903.08254, 2019. 2, 3, 7, 15, 18
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations, 2018.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 350-360.
Curran Associates, Inc., 2019. 2,3
12

---

Under review as a conference paper at ICLR 2022
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784, 2018. 3,7
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukeuoglu, and Raia Hadsell. Policy distillation. arXiv preprini arXiv: 1.002/2/.....5... 2015.
Andrei Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv, 2016. 3
Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint 18X3v:8803.075330 2018. 2,
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987. 2
Jürgen Schmidhuber. POWERPLAY: training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. CoRR, abs/l 12.5309, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: IXX.::707.00647,, 2017. 6
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka
Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp- 4528 4537, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. 7, 1818
Rupesh Kumar Srivastava, Bas R. Steunebrink, and Jürgen Schmidhuber. First experiments with powerplay. CoRR, abs/1210083855 2012. 3
Bradly C Stadie, Ge Yang. Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01116 2018. 2
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardic. Dream: A challenge data set and models for dialogue based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231, 2019. 3
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn: Meta-critic networks for sample efficient learning. arXiv preprint 1706::1706.0955555 2017. 3
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint orXiv:1707.04117 2017. 3
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. A Deep Hierar- chical Approach to Lifelong Learning in Minecraft. arXiv, pp. 1-6, 2016.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012. 2,
Matthew Wallingford, Aditya Kusupati, Keivan Alizadeh- Vahid, Aaron Walsman, Aniruddha Kemb- havi, and Ali Farhadi. Are overfitting experimental setups in recogmition?, 2020. 3
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos. Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. CoRR, abs/1611 00763,773 2016. 1, 2
Huaxiu Yao, Yingbo Zhou. Mehrdad Mahdavi, Zhenhui Li, Richard Socher, and Caiming Xiong. Online structured meta-learning. arXiv preprint arXiv:2010.11555, 2020.
Tianhe Yu, Chelsea Finn, Annie Xie. Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. 2018.
13

---

Under review as a conference paper at ICLR 2022
Tianhe Yu, Xinyang Geng, Chelsea Finn, and Sergey Levine. Variable-shot adaptation for online meta-learning. arXiv preprint arXiv:2012.07769, 2020a.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
In Conference on Robot Learning, pp. 1094-1100. PMLR, 2020b. 1, 7, 16, 17
Zhenxun Zhuang, Yunlong Wang, Kezi Yu, and Songtao Lu. Online meta-learning on non-convex setting. arXiv preprint arXiv: 10/96, 2019.
Luisa Zintgraf. Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference Machine Learning, pp. 7693-7702,
2019. 3
Luisa Zintgraf, Kyriacos Shiarlis. Maximilian Igl. Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep via meta- meta-learning. In International Conference on Learning Representations, 2020. 1, 2
14

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS
Here provide additional information on the task sequences and reward functions for each envi- ronment. The non-stationary task sequences were chosen to ensure neighboring tasks are different from each other. This was chosen so that dissimilar to previous tasks such that the agent will have to draw on old experience to perform well on each new task. For example, in the Ant Goal task, each successive task goal is 70 degrees more cow compared to the last goal, ensuring a significant amount of adaptation is needed to solve the next task.
focus evaluating forward transfer to tasks that are
Ant Goal.
Goal description: With the agent starting at the origin (x = 0, = 0), goal locations are sampled 3 units away from the origin uniformly on circle.
2. Non-stationary task sequence: We fixed twenty tasks, [-171%, 9 -662, 18 -153", 27",
- -144", 36 -135, 45" 1266. 54
1177. 63%. - 1088. 72", -99" 81%. -90%, 90" -18 99 -72", 108 -63 117"
-54", 126", -45,, 135", -36", 144".
-27", 153".
-18% 162 -9., 1 0" 180"]. where each number reperesents the rotation of the goal's
position, where for 0 the goal position of (1: = 3, y = 0) and 90 represents goal position of
(x: = 0, y = 3). The stationary task distribution randomly shuffels this sequence.
3. Maximum Trajectory length I We set 7 7Ta 200, same as GMPS Mendonca et al. (2019).
4. Reward function: We modified the reward function in PEARL Rakelly et al. (2019), reducing the control cost portion of the reward by an order of magnitude. This is accomplished by multiplying the control value by 0.1.
Ant Direction:
1. Goal description: With the agent starting at the origin, goal directions, as vectors (z, y). are sampled uniformly.
2. Non-stational task sequence:
We fixed forty tasks, [-171,, 9o, 162 18 -53
27" 1444, 36 -135, 45" 1266, 54" 11170 63 108 72" -99", 81 -90",
90º. -81%, 99 -72", 108 -33 117" 126 -54 -55, -36", 144 -27
153", -18% 162", -9%, 171", 09, 180%, where each number reperesents the degree of the goal's angle, e.g. 0 represents a goal that aligned with the positive x-axis and 90 represents positive y-direction. For stationary distribution experiments the sequence is shuffled randomly.
3. Maximum Trajectory length TTmae: We set m = 200, same as GMPS
4. Reward function: We modified the reward functions from the version of the environment used in PEARL by adding a scaling factor of 0.1 to the control cost.
Half Cheetah:
1. Goal description: The target velocities can be in the range [-3, 3).
2. Non-stationary task sequence: We fixed forty tasks described by there goal velocity along
the x-axis [-2.66, 0.14, -2.52, 0.28, -2.38, 0.42. -2244, 0.56, -2.1, 0.7, -1.6, 0.84, -1.82, 0.98, -1.68, 1.12, -1,54, 1.26. -1.4, 1.4, -1.26, 1.54, -1.12, 1.68, -0.98, 1.82, -0.84, 1.96. -0.7, 2, -0.56, 2.24, -0.42, 2.38, -0.28, 2.52, 14,14, 2.66, 0.0, 2.8, J. For the stationary task distribution the above sequence randomly shuffled.
3. Maximum Trajectory length Tmaaz We set m 200, same as GMPS Mendonca et al. (2019).
4. Reward function: There were no changes the reward function from the environment used in PEARL Rakelly et al. (2019).
15

---

Under review as a conference paper at ICLR 2022
Figure 8: These figures show tasks use in the Meta World environment.
Meta World:
1. Goal description: We chose three environments from Metaworld: Bish-Wall-V2 V2, Button- Press-Wall-V2. and Hammer V-2. These environments were chosen based on their dissimilar- ity and difficulty measured by the level of success regular PPO could achieve on these tasks. For each task, we specify both the environment (Push-Wall-Yy (P), Button-
(B), or Hammer V-2 (H)) and the goal index of that environment. The goal index controls additional variation in the tasks. For example, in the Hammer v-2 environment, shown in Figure 8 different goal tasks change the location of the hammer at the start of the episode.
2. Non-stationary task sequence: We fixed tasks, [P-0, B-1, H-2, P-3, B-4, H-5, P-6, B-7, H-8, P-9, B- 10, H-11, P- 12, B- 13, H- 14, P- IP, B- IB, H-17, P-18, B-19, H-20, P-21, B-22, H-23, P-24, B-25, H-26, P-27, B-28, H-2H, P-30, B-31, H-32, P-33, B-34, H-35, P-36, B-37, H-38, J. where the letter the left of "-' represents the environment ('H' for Hammer-V2. 'B' for Button-Press V2, and 'P" for Push-wall-V2) V2), while the number to the right of ' specifies the goal index used in the environment. For the stationary task distribution, the above sequence is randomly shuffled depending on the random seed.
3. Maximum Trajectory length Tmaz, We set m 150, same as the original Metaworld codebase Yu et al. (2020b).
4. Reward function: We used the same functions for the three environments as the original Metaworld codebase.
Success Thresholds For the experimental evaluation, we are interested in measuring how quickly an agent can solve tasks. Previous leeanne environments (Yu et al., 2020b) have used measure of success to determine the agent's progress on its tasks. These success measures are more robust to tasks that can have largely different reward scales. Each environment includes success indicator that is 1 when the agent is within distance of the desired task goal and 0 otherwise. In this training version, the agent moves on the next task when it has been told it has succeeded on the current one. This setting makes particular sense in the continual multi-task setting, where the goal is to do well on all tasks with minimal interaction. CoMPS uses these success thresholds as a means of forcing the agent to sufficiently solve tasks before moving on to new tasks. This helps increase the quality of the collected data. The maximum amount of time given for each task can be increased to ensure the agent has enough time to solve the task. Although, even if the prior task was not solved the data from that task is likely to be unique and add diversity to the meta training dataset which will help with learning more generalizable parameters (Gupta et al., 2018a).
1. Ant Goal: The agent is successful if the agent's position (az, ay) and the goal's position (gw,9y) satisfy (la,- Ie lay gyl < 0.5). We calculate the number of successful samples from a batch of trajectories and average the success across all samples. We found that agents trained through PPO training could reach success rate of slightly over 0.3. We therefore set the success rate threshold to 0.3.
2. Ant Direction: The agent is successful its unit direction vector v, and the unit goal direction vector iú satisfy |/u, - -Upl < 0.4 while at the same time, the agent's speed must be greater than 0.5 m/sec. We calculate the number of successful samples from a batch of trajectories and average the success across all samples. Through qualitative inspection of the agent, we found that expert behavior occurs at 50% success. Therefore we use success rate threshold of 0.5.
16

---

Under review as a conference paper at ICLR 2022
3. Half-Chhetall Velocity: The half-cheetah agent is successful if the difference between its velocity Uo and the goal velocity 1g is less than 30% of the goal speed ([ve - rigl < 0.3 e.y). We calculate the number of successful samples from batch of trajectories and average the success across all samples (this is the same as Ant Goal and Ant Direction). We set the success rate threshold to 0.45. The success threshold of 0.45 was chosen by trial and error according to the ability of the learning agent. We found that the highest success PPO could attain, terms of average success, across these tasks was 0.5 and at 0.45, the agent showcases expert skill on the task.
4. Metaworld: We use the built-in success function from the Metaworld codebase Yu et al., 2020b). In the metaworld environments, we care about whether the agent has successfully performed the task during the episode. For this reason, the success value is computed differently than the other environments. We instead average the ratio of trajectories that contained at least one success. We found that agents trained with PPO can reach success rate of over 0.2 across most tasks while showeasing expert skills in the three environments. We set the success rate threshold to 0.2. While the success thresholds that were used may appear low in terms of reward, we do find that their qualitative performance is good. This can be seen the paper website, where visualizations of the policy results show that policies that pass the threshold qualitatively perform the correct hehaviors. In most environments (Ant goal, Ant Dir, and Cheetah), it is impossible to be 100% successful because the threshold is averaged over every timestep in a trajectory. With this, we can interpret the success threshold as meaning how fast the agent has reached the goal and maintained that goal. This means that for example, the success threshold of 0.3 for Ant Goal may be the best any algorithm can do given the max episode length. A threshold of 0.3 does not mean that the method fails 70% of the time, but that it spends 30% of the total time in the "success state" on average. This is why we used (single task) PPO, that is used in all methods in the paper, to determine the threshold - this amounts to saying that our threshold for success is the performance obtained by a good RL algorithm on only that task with plentiful data, is a reasonable "upper bound" for few-shot method. Measuring success this way works well for these environments where, for example, Cheetah needs to maintain a desired velocity for as long as possible. However, for MetaWorld used the same method from the Meta World paper that averages the success over all trajectories where success is 1 for trajectory if any step was a success. We do note that according the meta world paper the best any flat RL algorithm could do at solving all tasks was achieving ~ 30% success. We were able achieve a similar performance with PPO the findings in MetaWorke of 25%. This and the lower performance of the meta-RL methods speak to the difficulty of the tasks. However, we do find that CoMPS achieves good qualitative results that can be seen on the webpage.
Compute resources For all our experiments, we used different cloud compute resources. Across these resources, we trained using virtual machines with 8 CPUs and 16 GiB of memory.
A.1 ENVIRONMENT DETAILS
We changed the control costs in the PEARL environments because we initially found that, in our continual setting, a naive PPO baseline would attain very poor results because would focus too much on these costs, resulting in agents that stood still and did not perform the task. After the reduction in control costs, naive PPO demonstrated substantial improvement in solving each task across the PEARL environments. This change also makes the PEARL environment similar to the Meta World environments, which do not use any control torque costs. However, for completeness, we reran all our experiments in the original environments from the PEARL paper (without any changes to the reward functions). We find that generally, the trends are similar, with the exception that PPO+TL and CoMPS perform similarly on the Ant- Dir in the -tationny task distribution case.
B COMPARISON TRAINNING DETAILS
Here provide training details for the algorithms compare.
17

---

Under review as a conference paper at ICLR 2022
This algorithm uses version of PPO that many methods share in this work. This agent trains on each task using only the stochastic gradient objective outlined for PPO. Unlike CoMPS, between tasks PPO+TL does not perform additional meta-learning and instead transfers the network parameters learned this far directly to the next task.
PPO+TL:
The comparison to PEARL uses the provided codebase released with the paper (Rakelly et al., 2019). We updated the PEARL code to change the tasks the PEARL agent trains over. Instead of the original ~ 130 tasks used for training, according to the task distributions outlined in the paper. The agent collects 5000 simulation steps between evaluations.
PEARL:
refine the agent train on one task time
The PNC algorithm consists of two phases similar to CoMPS. One phase is normal RL which we train using the same parameters and version of PPO that we used for both CoMPS and PPO+TL. The second phase is distilation phase that uses elastic weight consolidation to train different head of the network to imitate the behavior of the current policy on that task while reducing the forgetfulness via constraint of the change in the output distribution. This distillation phase needs on- on-policy data; therefore, we devote the last %5 of training to this distillation phase in our experiments. PNC also uses a different network structure than all other methods in the analysis. For PNC the network consists of a policy and a knowledge base with the additional output head for imitaaionn/distilaatinn. Both networks have hidden layers of size [128, 128) and according (Schwarz et al., 2018) the layers in the knowledge base are connected to the layers of the policy.
PNC:
C RL TRIANING DETAILS
At the end of RL training, we split the data into two portions. These are the Dj and Dj. collections. The number of trajectories of D depends 7 for each task, but we fix the batch size to be 5000 samples. We save a portion of the data the RL agent generated to reduce memory costs. We keep a portion of the min of N episodes Mathematically, we sample episodes of data where i = min(Ms N 0.05). Then we randomly sample 100 trajectories from the k episodes of data. When M is large, this processing avoids keeping around too much data that may lead to memory issues on the computer.
RL or the M episodes needed to reach the success threshold.
D COMPS DETAILS
For the RL part of CoMPS, GMPS+PPO. PPO+TL, and PNC we explored the best hyperparameters by evaluating PPO on each task in paraliel starting from a randomly initialized policy. We searched for the best values for the learning rate, stochastic policy variance, kl clipping_Jens (for PPO), and network shape that would receive the highest reward over the individual tasks described in the paper. For the meta-step (M) that CoMPS and GMPS+ PPO share we tuned those algorithms using a similar process. We searched for the best parameter values of learning-ratt -rate, the number of training updates per batch of data, alpha, batch size, and the number of total iterations or after each new task. We selected the parameters that achieved the highest stable rewards during training. We used the same values for both CoMPS and GMPS+PPO. For PNC we also performed hyperparameter tuning over the learning rate, the number of consolidation/compee performed hyperparameter tuning over the learning rate, batch size and the number of gradient steps. We believe that this procedure provides the most favorable possible hyperparameters for the prior methods and baselines.
steps, and EWC weight. For PEARL
We used grid search to find the best parameter settings. The search would include 3-4 different parameters settings for each hyperparameter. For example, here is the json code that lists the grid of parameters searched for PPO.
"ppo lepo.nnaani "; [0.01, 5e-3, 0.001, 0.0001].
"hidden-sizes". [[128. 128]. [128. 64], [64, 64]].
"init ": [0.3, 0.5, 0.7],
18

---

Under review as a conference paper at ICLR 2022
After this search, used these same parameters for all methods that used PPO and only searched method-specifff hyperparameters. For example, here is the list of hyperparameters evaluated for CoMPS:
"adam "adammsteps"" ": [50. 100, 250, 500]. "meta_itr": [20, 50, 100],
"metaastep_size" [0.005, 0.001. 0.0005].
The same process is used across all methods and baselines. The only difference is the additional hyperparameters that may be available to different algorithms. We will include these details in the final version the paper. The score function that is used to rate the search parameters is the average reward over all tasks and environments. We endeavor to use the same budget for each method by allowing for an equal number of randomly seeded experiments. An average of 36 combinations of parameters to test each with 6 random seeds leads to 216 simulations per method. These simulations were run parallel on cloud computing technology (SS/CC///uu////uumm We will include these details the number of simulations used and the hardware it paper.
in the final version of the
We attempted to select parameters based only single task, however, those parameters did not generalize well to all tasks and environments. We did find that this simple grid search over hyperparameters generally provided better performance for all baselines compared to their default parameters, and therefore in the paper we used the best settings for all baselines to ensure a fair comparison.
Using the above analysis we found a learning rate of 0.005 for each CoMPS meta-learning step, with total of 25 training steps. For all experiments, we use two-layered hidden network of sizes [128, 64). We use single inner gradient step for all tasks with inner learning rates shown below (Algorithm line 5). For the outer step of CoMPS, we perform imitation learning on Dj for steps on Algorithm line 7.
Environment
Inner Learning Rate
Ant Goal
0.005
Ant Direction
0..055
Half- Half-Cheetah Velocity
0.005
Metaworld
0.0025
Table 1: CoMPS Hyperparameters
E ADDITIONAL. RESULTS
Additional training results across environments.
E. STATIONARY TASK DISTRIBUTION
In Section we performed experiments to evaluate the performance of CoMPS compared other methods in terms of learning speed. This performance is measured as the number of learning episodes needed for an agent to successfully solve the task. These results can be found in Figure Here we include additional results and metrics for evaluation in addition to these results to get a broader view of the findings. Figure 9 we show the average return received for the same experiments as in Figure 6. In these results that use uniformly random task distributions, we can see that CoMPS achieves higher average reward while completing the tasks. The CoMPS also improves its ability to achieve high rewards quickly more tasks are solved. This performance improvement can also be seen in Figure 12(top) that shows the return averaged over tasks, where we average the learning graphs together for each task.
19

---

Under review as a conference paper at ICLR 2022
Mard
Figure 9: These figures show the average return for each new task. On average CoMPS achieve the highest average return while training over an entire task at time. Results are computed over sequences of 40 tasks, averaged over random seeds.
Figure 10: These figures show the average return achieved during the RL. phase over all tasks. This shows average return the agent achieves if any episode is picked random from any task. Still, CoMPS performs better than other methods considering this analysis. Results are averaged over 6 sequences of 40 tasks, and 6 random seeds.
E.1.1 STATISTICAL SIGNIFICANCE
We perform two-sided independent t-test the learning data from the stationary task distribution experiments using a bootstrap of 30, 000 over 6 randomly seeded trials. This test provides information on whether the baseline results appear be from the same distribution shown in Table 2. This analysis shows that the distribution average number of episodes needed to complete task for all algorithms is not the same CoMPS. This analysis is suggested in Henderson et al. (2017) to provide futher data on the statistical significance of the results.
CoMPS. Our findings are
Alg
Ant Goal
Cheetah
Ant Direction
Meta World
PEARL
1.85E-11
0.0002371241398
7.36F-09
2.26F-13
PPO+TL
4.54E-11
2.66E-05
0.01990761802
0.006441451648
MAML+TL
1.12E-II
4.17E-18
1.87E-08
0.007026979433
PNC
0.01833026502
0.002410406845
6.35E-09
1.26E-10
Table p-values from a two-sided independent t-test (CoMPS, Alg) performed over the stationary task distribution experimental data.
E.2 NON-STATIONARY TASK DISTRIBUTION
In Figure 7(left), show the average reward of each method for every episode of learning 40 tasks from a non-stationary distribution. On the right of Figure 7 we show the average reward achieved over the same 40 tasks. To provide more aggregate data, we also compute the average reward achieved over all tasks. We provide this information in Figure 10. In this data, we can see that CoMPS obtains median average return that higher than all other methods. PEARL, particular, exhibits a high variance in rewards across tasks, which can also be seen in Figure 7left). GMPS+PPO, which does not include the additional off-policy importance weighting corrections used in CoMPS, achieves some of the lowest returns across all methods. This further illustrates the importance of the off-policy corrections in CoMPS, We also look at performance per episode averaged over tasks in Figure 12(top). The reason these curves are smooth with low variance is because of the approximately 100 samples being averaged over x number of tasks = 240. CoMPS accumulates the largest area under the curve compared all other methods in the stationary and non-stationary case. These results also show that CoMPS achieves performance improvments in early episodes across tasks compared to all other methods.
20

---

Under review as a conference paper at ICLR 2022
Figure 11: Similar Figure 6, these figures show the average number of learning iterations it takes for each method to reach success on the non-stationary tasks. In this case we can see that CoMPS solves most tasks faster than other methods.
AETLO
FOR
Figure 12:
These figures show the average return achieved for each episode in the RL phase over all tasks. The top row is for the stationary task distribution and the bottom for the non-stationary task distribution. These graphs show that CoMPS is achieves the highest average return over all other methods when performance is averaged over tasks. Results averaged over 6 sequences of 40 tasks, and 6 random seeds.
F.2.
STATISTICAL SIGNIFICANCE
We perform two-sided independent t-test on the learning data from the non-stationary task distri- bution experiments using bootstrap of 30, 000 over randomly seeded trials. This test provides information on whether the baseline results appear to be from the same distribution as CoMPS. Our findings are shown in Table 3. This analysis shows that the distribution of average reward for most algorithms is not the same as CoMPS. In the Meta MetaWorld and Ant Direction environments PPO+TL very similar to CoMPS and somewhat similar to CoMPS respectively.
Alg
Ant Goal
Cheetah
Ant Direction
Meta World
PPO+TL
0.0003071245518
7.32B-08
0.2102584056
0.9973906496
PNC
0.00002119488748
3.93H-08
0.07235237317
1.49H-05
MAML+TL.
0.0001044653709
7.91E-08
0.01082454019
8.97H-06
PEARL
0,0001308992933
1.39B-06
0.02516455006
9.79E-06
Table 3: p-values from two-sided independent t-test (CoMPS, Alg) performed over the non-stationary task distribution experimental data.
E.3 VIDEO RESULTS
To videos of the learned policies, see https:/siiisseoogg.comm//vvvvves
F FURTHER USE CASES
We believe that there many settings in robotics and other domains where revisiting previous tasks is an issue. For example, imagine a robot at recycling plant that has to disassemble electronics: each item of electronics might be different, and once it has been disassembled, it cannot be disassembled
21

---

Under review as a conference paper at ICLR 2022
CARE
CARK
11.0
These figures show the backwards transfer for CoMPS averaged over prior tasks. The y-axis if the normalized reward where 1 means the method retained the original performance on the task.
Figure 13:
again. So there is a sequence of tasks, prior tasks cannot be revisited, and it is desirable to accelerate all future tasks. Similarly, imagine a house cleaning service performed by a robot that puts away objects once the robot has cleaned one house, it goes to different one. Revisiting houses in round-robin fashion would be impractical and unnecessary because the houses will already be clean.
G BACKWARD TRANSFER
In Figure 13 we show the backwards transfer of CoMPS. We plot the normalized reward co....taa de where dp.....k is the average reward over the first iteration of learning for the pre- Bo..... "tg...... vious k tasks (representing the lowest reward the agent recieved), bo.....: is the average reward on the last iteration (representing the highest reward the agent recieved). Finally, co..... is the average reward the CoMPS agent recieves when evaluated over the prior le tasks. When the normalized reward there is no loss in performance on the tasks, when the value is 0 the method has forgetten how to perform the task. We can see from these plots that CoMPS exhibits strong backwards transfer on the Ant Direction, Ant Goal, and Cheetah environments. For Ant Goal and Ant Direction backwards transfer appears to stabilize, indicating that learning new tasks may not affect performance tasks. This observation aligns with the properties of meta-learning and MAML such that adding more tasks generally increases adaptation performance. The reducing performance related to the increasing difficulty of tasks. CoMPS demonstrates backwards transfer on Metaworld initially; however, this reduces as more tasks are seen, again indicating the difficulty of discovering generalizable structure across the diverse Metaworld tasks.
old
Cheetah is likely
H COMPS ABLATION
Here, we perform additional analysis on the components of CoMPS and how they affect the perfor- mance of the overall method. GMPS performs very poorly across all tasks. The addition of PPO (GMPS+PPO) increases performance on Cheetah and Metaworld but the relative performance is still poor. significant improvement is given with the introduction of off-policy importance sampling (GMPS + PPO + IS). As described in the paper, the importance sampling allows CoMPS to compute unbiased gradient information from a large collection of off-policy data. We also investigate storing additional skilled data at the end of learning each new task (GMPS with IS + x2 the skilled data). The addition of more skilled data does not generally improve the performance of CoMPS. Following pattern similar to FTML Finn et al. (2019), CoMPS initializes meta-RL training after each task from the policy trained on the previous task, across environments this improves learning speed (CoMPS/ours (GMPS + PPO + IS FTML)). In particular, the addition of FTML-based training decreases the variation in performance. It results in a smoother reduction in the time used by CoMPS solve new tasks.
22

---

Under review as a conference paper at ICLR 2022
Ant Goal
Ant
1000
(GMPS CMPS PPO- iS x2 the skilled data
1000
Average tpisade Count
Averogge Eplods Count
900
CMPS #PO
800
800
GMPS
700
600
CoMPsroour (GMPS PPO FTMU
6MPS PPO +-1 x2 the skilled data
6MP5
400
GMPS
20
25
30
25
40
15
30
25
40
Cheetah
Metaworld
500
1000
CoMPS/uurs (GMPS + PPO is FTM
Rvengge Epacde Count
Average Episoll Count Count
GMPS P90 skilldd
800
CouPsrours (GMPS PPO 15 FTML)
GMP5 PRO
GMPS PPO skilled
GMPS
CMPS PPO
200
GMPS
200
40
20
30
Task
Number
Task Number
Figure 14: These figures show performance of variations of CoMPS. As remove features from CoMPS we can see the performace reduce considerably.
23


Reviews:
[{"review_id": "zy3neGHkGr6", "reviewer": "Reviewer_FZ5X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Merits\n\n1.\tThis paper studies the problem of continual Meta-RL, which is important and deserves more attention from the community.\n\n2.\tThe general idea of the proposed CoMPS method is sensible and conceptually simple for addressing the challenges of continual Meta-RL. CoMPS meta-learns on previously seen tasks to acquire a good parameter initialization that fast adapts to new tasks. These adapted policies in turn collect data for meta-learning. The main challenge is that previous tasks cannot be revisited during continual learning, and the policy has to be trained on limited offline data. This challenge is addressed by utilizing importance-sampling policy gradient and off-policy value estimator in the inner loop and behavior cloning in the outer loop. The whole framework is reasonable, and, in some sense, CoMPS can be viewed as an approximate implementation of the FTML algorithm [1].\n\n[1] Finn, Chelsea, et al. \"Online meta-learning.\" International Conference on Machine Learning. PMLR, 2019.\n\n3.\tExperimental results look impressive and show that the proposed method achieves superior performance in several sequences of continuous control tasks.\n\n4.\tThis paper is generally well-written and structured.\n\n## Limitations and Concerns\n\n1.\tOne major concern is that the technical contribution of the proposed method is very limited. CoMPS is a simple adaptation from the previous method GMPS by using an off-policy variant of PPO in the inner-loop.\n\n2.\tAnother concern is that CoMPS may not handle well for some badly-learned tasks, as its outer loop objective uses behavior cloning. If previous tasks are not learned well, and the skilled experience dataset $D^*$ contains bad trajectories, the meta-learning procedure may learn an initialization that quickly adapts to imitate the bad trajectories, which harms learning new tasks.\n\n3.\tBackward transfer: although the authors claim that they do not evaluate backward transfer or forgetting, it is one of the core challenges in the field of continual learning. It is interesting to see how CoMPS performs on previous tasks, which may contribute to a deeper understanding of CoMPS.\n\n4.\tRelated works: most related works have been adequately discussed. However, several works incorporating meta-learning into continual supervised learning are missing [2,3]. A discussion over these works should be added.\n[2] Joseph, K. J., and Vineeth N. Balasubramanian. \"Meta-consolidation for continual learning.\" arXiv preprint arXiv:2010.00352 (2020).\n[3] Gupta, Gunshi, Karmesh Yadav, and Liam Paull. \"La-maml: Look-ahead meta-learning for continual learning.\" arXiv preprint arXiv:2007.13904 (2020).\n\n5.\tExperimental details: environment descriptions of Ant Direction and MetaWorld in the caption of Figure 5 do not seem to correspond to Appendix A, which is confusing.\n\n6.\tOther minor things: at the start of each meta-learning procedure (Algorithm 1), how is $\\theta$ initialized? Do you use the $\\theta$ achieved at the end of the last meta-learning procedure, or do you use the parameters updated after the RL procedure?\n", "summary_of_the_paper": "This paper proposes a method that incorporates fast adaptation into continual learning. The main challenge is that previous tasks cannot be revisited during continual learning, and the policy has to be trained on limited offline data. The authors address this challenge by utilizing importance-sampling policy gradient and value estimators in the inner loop and behavior cloning in the outer loop, which enables meta-training with fully offline data. Experimental results show that their method achieves superior forward transfer compared to prior methods.", "summary_of_the_review": "Overall, this paper is an interesting step towards solving the problem of continual Meta-RL. Its presentation is clear, and the proposed general framework is reasonable. One major concern is its novelty, which resembles the previous approach GMPS. Although several key limitations have been identified in the discussions of this paper, addressing some of them is important and can significantly strengthen this paper, particularly requiring the data storage of previous tasks and no mechanism for handling forgetting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gyMX8p1YB_m", "reviewer": "Reviewer_1xJQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main novelty of this work is the proposed continual meta RL setup and the design of the experiments. Regarding the algorithm, I find it to be reasonable application of existing ideas to this setup. \nTo give a few examples: \n\nThe ablation study indicates that the method performs better by using an existing algorithm for doing off-policy learning. While this is nice to know, it is not very surprising, and we can learn very little from looking at figure 6 about that. A more carful investigation of this question will include an experiment that is centered on the off policy questions in this setup, will report some more statistics (e.g., the IS weigths as the tasks change) and will compare to more baselines. \n\nThe baselines that were chosen are complete methods, that were not specifically designed for this new setup and are not high performing in it for that reason. A much more interesting comparison would be a carful ablation analysis on top of COMPS showing which components make it superior and when. For example, I don't learn a lot from the comparison to PEARL as a whole, and I would appreciate a comparison that uses COMPS with SAC as the RL agent. I am interested to know what is needed in this setup and why, I am less interested why papers that were designed for different setups are being outperformed (while it is good to know that this is the case). \n\nTo summarize, I feel that the paper will have to make stronger algorithmic contributions and more carful ablation study for it to get accepted to ICLR. In this form, I would recommend rejecting it. ", "summary_of_the_paper": "The paper propose a continual meta-learning setup where the agent is faced with a sequence of tasks and has to adapt quickly to new tasks based on the experience it gathered in previous tasks. In addition, at future iterations the agent can only interact with the previous tasks by doing off-policy learning on the data it collected during its interaction with these tasks and cannot further interact with these previous tasks. The authors then propose an algorithm that combines RL on the current task with offline meta RL on the previous tasks. The authors then construct setups that fit this problem setup, evaluate their method on it and compare to previous baselines. ", "summary_of_the_review": "Interesting meta RL setup and nice experiment construction. At the same time, the algorithmic novelty is questionable and the ablation study is not convincing enough. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RUGDN8rbMOa", "reviewer": "Reviewer_8AHL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main observation is that the paper reveals possible novelty and innovations very late and unclearly. The abstract explains the problem that the algorithm attempts to solve, but reveals no insight about how this happens, and thus does not provide any concrete claim. This issue continues in the introduction where the contribution part focuses on what problem is being solved, but not how. What are the main scientific and technological innovation that are introduced with respect to existing approaches? Eventually, section 4 (CoMPS overview) enters in the description of the proposed algorithm, but even the following sections 4.1 and 4.2 do not clearly state what specific novel aspects are introduced with respect to existing methods, and which are re-used or adapted, which makes it very difficult to appreciate the novel contributions and originality of the work. In section 4.3, it is mentioned that the proposed method uses \"both an improtance-sampled policy gradient estimator and an importance-sampled value estimate for the baseline in the policy gradients” and it claims that the following ablation study demonstrate it’s importance. Is this one of the major contribution of the paper? \n\nSimilarly, the idea that the novel algorithm uses an alternation of off-policy (for the inner loop) and imitation learning (for the outer loop) transpires without sufficient clarity and related claims. An interesting hypothesis, e.g. is mentioned in section 4.2 : \" In contrast to methods that are concerned with forgetting, the parameters produced by this meta-RL training can quickly learn new behaviors that are similar to the high-value policies from previous tasks and, if enough prior tasks have been seen, likely generalize to quickly learn new tasks as well. However, this is not sufficiently followed up in the experimental analysis. How does this mechanism affect performance in the various tasks? What can we learn from the fact that CoMPS performs better than the baselines on the first three tasks, but it's comparable to PPO+TL in the arguably more challenging MetaWorld tasks? \n", "summary_of_the_paper": "This paper proposes a meta-RL algorithm that is focused in particular on the learning of sequential tasks without revisiting previously seen tasks. This approach, as stated in the paper, could be particularly beneficial in real-world applications and robotics where it might be unfeasible to revisit previously seen tasks. The paper builds on a successful meta-RL algorithm (MAML) and appears to leverage on the use of off-policy methods to replay previous data from previous policy without re-engaging with older tasks. \n", "summary_of_the_review": "In summary, the paper appears to have solid basis in the literature and tackles an important problem in meta-RL, and it appears to propose an interesting combination of known approaches to solve the stated problem of sequential-task meta-RL without task repetition. The experimental results are encouraging. However, it fails to make clear claims or clearly explain the key novel aspects of the algorithm, how and why they are sufficiently novel. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8E4g0Rt0ZM", "reviewer": "Reviewer_cHnK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n- this paper considers a more realistic and difficult scenario, continual Meta-RL learning, and proposed an extended meta-rl framework that adapts this setting;\n- the experiments are sufficient and the performance improvment is significant;\n- the writing is good and key details are clearly described.\n\nweakness:\n- This paper can be viewed as a combination of several existing techiniques. In particularly, this paper claims that it addressed a type of continual learning problem but did not consider one of the main characteristic of contual learning - i.e., the forgetting issue. Hence a reasonable baseline to use the past experience is the offline RL learning instead of consider it as a meta learning problem. Overall，it looks to me that the whole method is a kind of  MAML + off-policy RL + online RL. \n\n- though the paper claims that this is the first to  formulate and address the continual meta-learning problem, this problem setting has already been discussed before, some work can be referred to:\n[1] Online Meta Learning. Chelsea Finn, et.al.\n[2] Continuous adaptation via meta-learning in nonstationary and competitive environments. Maruan Al-Shedivat, et.al.\n\nsome questions and typos:\n1. the inner optimization for each sequential task is actually separate, so why not use a recurrent neural networks structure to avoid the forgetting by utilizing the historical information?\n2. the subscripts of both summation and quadrature symbols in Eq.2 have problems.\n3. the caption of Figure.7  is wrongly written as '... the average average return across the episodes ...'.", "summary_of_the_paper": "This paper considers  the challenges in sequential multi-task learning, in which the agent will visit all the tasks in a sequence, and will not revisit the previously learned task. This paper proposed a continual meta-RL framework, named Contunual Meta-Policy Search (CoMPS), so as to extend the traditional meta RL framework, GMPS(Guided Meta Policy Search) to a continual Meta-RL setting. Different with GMPS, CoMPS firstly adopts self-imitation learning as the outer optimization instead of a true imitation learning and secondly adopts an importance sampling-based policy gradient for off-policy inner optimization, which is suitable for the continual Meta-RL. The empirical results show that CoMPS can outperform the prior continual learning or meta learning methods on both stationary and non-stationary task distributions.", "summary_of_the_review": "The issues considered and the solutions proposed in this paper sound reasonable, and the proposed method does achieve the better performance. But some details of the whole pipeline need more justification, and the optimization between sequential tasks seems to be independent. I'm also not sure that this is able to address the issue about catastrophic forgetting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0BbK56ozmht", "reviewer": "Reviewer_S7vL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. The paper presents a novel, potentially useful setting/objective for the meta-RL/transfer learning communities to focus on, which in some sense bridges objectives pursued by the meta-RL and transfer learning communities.\n\n2. The paper also presents a novel approach to tackle the corresponding problem which would act as a strong baseline for methods tackling the said problem.\n\n3. The paper is also really well written with clear diagrams and algorithm figures which make it very easy to read. \n\n4. I also liked some of the ablations and the choice of baselines as they provide a nice peek at what some of the critical components of the model were that made it click. \n\n5. The paper also does a nice job of tuning the baseline hyperparameters to make sure the results are competitive. I would suggest maybe moving some of those details from Appendix D to the main paper as it might serve as a nice reference for how to perform a fair and competitive evaluation of the baselines!\n\n**Weaknesses**\n\n1. I would like to see more ablations. It was unclear to me from reading the methods section as to why certain design decisions were taken : for example, \n    * Why use an on-policy method for online adaptation instead of an off-policy method here as well given you are storing the experiences anyway? \n    * Why the specific choice of off-policy method over so many others?\n    * What happens when you replace BC with another offline/off policy RL method? Etc.\n \n    In essence I want to understand how much each of the design decisions contribute to the overall performance and which ones were more important than others.\n\n2. I would also like some analysis on why the authors think PPO+TL was able to perform as well as CoMPS (or slightly better) on the meta-world non-stationary tasks. It does seem like a surprising result to me and I think some analysis or discussion there would have been nice! Some hypotheses I have here, for example, are that, maybe the specific non-stationarity ends up providing a kind of curriculum to PPO -TL? Maybe the off-policy updates have too much variance? Maybe the trajectories used for BC update aren't good enough in these types of problems? etc..\n\n3. Related to the point above, I would also like some discussion on the broad class of scenarios where the approach presented in the paper could provide a genuine boost over approaches like PPO+TL, and scenarios where it’s uncertain and why the uncertainty exists (perhaps due to some empirical or numerical issues with the algorithm presented?)\n\n**Other comments**\n\n1. It is also not entirely clear to me if the scenario or motivations presented in the paper for real world applications of the continual RL setting are reasonable? For example, for the home cleaning robot example, it’s hard to imagine that you would ever actually deploy a robot in the real home to collect experiences and learn using RL directly in the sequential setup. You would probably either have some sort of sim2real setup or offline experiences collected in some other (safer?) way used to train the model. I don’t hold this as a major weakness of the work, but I think it would be nice to provide motivations which are more grounded. For example, the disassembling example given at the end of the appendix seems more reasonable to me and might be useful to think along those lines instead!\n\n2. I would also be curious to know how effective the meta-learning objective would be for learning to explore in the new environment when trying to learn the new task (something like “Rothfuss, Jonas, et al. \"Promp: Proximal meta-policy search.\" arXiv preprint arXiv:1810.06784 (2018).”).\n\n3. It might also be a good idea to slightly change how the confidence intervals are computed using suggestions from https://www.google.com/url?q=https://ojs.aaai.org/index.php/AAAI/article/view/11694&sa=D&source=docs&ust=1635372130978000&usg=AOvVaw1BsO05shZBrpwhKwk76dSN (say bootstrap etc)\n\n4. typos/grammatical errors\n    * at the same time still adapt to ->  at the same time adapt to\n    * rehersal-based methods ->  rehearsal-based methods \n    * and the implementation are given ->  and the implementation is given\n    * utilize both an improtance-sampled -> utilize both an importance-sampled\n    * furthest from previously chosen location -> furthest from the previously chosen location \n    * we provide futher analysis -> we provide further analysis\n    * behavior of CoMPS is statistical different -> behavior of CoMPS is statistically different\n    * due to the increasing difficult of tasks -> due to the increasing difficulty of tasks\n    * accelerates acquisition of new tasks -> accelerates the acquisition of new tasks", "summary_of_the_paper": "The paper presents a new method for continual meta-reinforcement learning called CoMPS. The objective is to quickly achieve a high reward over any sequence of tasks even in non-stationary task distributions. Unlike previous methods in meta-RL, CoMPS uses a hybrid meta-RL approach, where experiences from past tasks are used to learn a fast adaptation procedure (using a meta-RL objective with off-policy inner loop updates and BC outer loop updates). The model is then adapted to any new task using a standard on-policy RL algorithm like PPO. The authors show that this approach outperforms generic meta-learning and transfer learning approaches across various environments for the specific settings considered in the paper.", "summary_of_the_review": "I like the paper overall, it is clearly written and I think it tries to address an interesting problem. I would however, have liked to see some more ablations and analysis of the approaches and maybe some better motivations too.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a novel continual meta-reinforcement learning algorithm that focuses on learning sequential tasks without revisiting previous tasks. The setting is compelling, and the method is well-developed with good empirical results. The initial version of the paper included a variety of issues, especially lack of clarity in some aspects and the contributions, that were remedied through discussion with the reviewers and subsequent revisions. The discussion among the reviewers seems to have settled on leaning toward a weak accept overall, with one low score that should be dismissed claiming lack of novelty (which isn't correct - the paper certainly is sufficiently novel).  There do remain some concerns by two reviewers that although "the paper has enough meat to be accepted, ... [it] needs more careful and well thought out ablations and analysis to be truly valuable." Although the authors have revised the paper to address this issue of a precise analysis, adding material into the appendices with some changes to the main text, they are encouraged to make certain that these aspects are integrated and clear throughout.
Decision: Accept (Poster)