Evaluations of the Gap between Supervised and Reinforcement Lifelong Learning on Robotic Manipulation Tasks
Authors: Fan Yang
Chao Yang
Huaping Liu
Fuchun Sun
Venue: CoRL2021 Poster (2021)
Keywords: Lifelong Learning
Reinforcement Learning
Manipulation

Abstract: Overcoming catastrophic forgetting is of great importance for deep learning and robotics. Recent lifelong learning research has great advances in supervised learning. However, little work focuses on reinforcement learning(RL). We focus on evaluating the performances of state-of-the-art lifelong learning algorithms on robotic reinforcement learning tasks. We mainly focus on the properties of overcoming catastrophic forgetting for these algorithms. We summarize the pros and cons for each category of lifelong learning algorithms when applied in RL scenarios. We propose a framework to modify supervised lifelong learning algorithms to be compatible with RL. We also develop a manipulation benchmark task set for our evaluations.

Evaluations of the Gap between Supervised and Reinforcement Lifelong Learning on Robotic Manipulation Tasks
Fan Yang, Chao Yang, Huaping Liu" Fuchun Sun
Department of Computer Science and Technology, Tsinghua University, China Beijing National Research Center for Information Science and Technology, China "Corresponding author: Huaping Liu (hpliu @wwe@@tisnnn...com
Abstract: Overcoming catastrophic forgetting is of great importance for deep learning and robotics. Recent lifelong learning research has great advances supervised learning. However, little work focuses on reinforcement learning(RL). We focus on evaluating the performances state-of of-tthe-rtttttt lifelong learning algo- rithms robotic reinforcement learning tasks. We mainly focus on the properties overcoming catastrophic forgetting for these algorithms. We summarize the pros and cons for each category of lifelong learning algorithms when applied in RI. scenarios. We propose a framework to modify supervised lifelong learning algorithms to be compatible with RL. We also develop manipulation benchmark task set for our evaluations.
Keywords: Lifelong Learning, Reinforcement Learning, Manipulation
Introduction
Humans and animals continuously learn different skills and knowledge in their lifetime. It is natural for them master new skill without forgetting those learned before in most cases. However, neural networks suffer a lot from memorizing previously learned tasks. This phenomenon is called catastrophic forgetting [1, 2, 3]. Recent research has focused on this problem and proposed various solutions called lifelong learning or continual learning.
On the other hand, lifelong learning of great importance for robotics. It provides possible solutions for developmental robotics [4]. Humans learn different skills as they grow up rather than completely rely on innate skills. It would also be difficult pre-program all the required skills for advanced robots when setting up in the factory. Future robots will have to continuously master different tasks humans do in their lifetime. The ability of lifelong learning will make robots fit better in various applications.
In this work, we focus the lifelong learning problem for robotic manipulation tasks in the rein- forcement learning(RL) domain, especially forgetting properties. Most of the recent lifelong learn- ing research focuses on supervised learning. However, lifelong learning for RL is much more chal- lenging than supervised learning. Not only because the training of RL itself is challenging and unstable[5], but in most RL tasks, policies are sensitive forgetting. Forgetting could be consid- ered as an attack in the action space of the agent. Recent work has demonstrated the vulnerability of action space[6). Moreover, in RL tasks, action at step will influence
RL agents to attack states and actions followed up, which could aggravate the influence of forgetting. For example, in grasping tasks, slight misplacement of fingertip is likely to break the balance of a grasp and lead to its failure. Though recent lifelong learning research has promising performances on supervised learning, little evidence has proven that they could directly be applied robotic RL tasks. Research on how the state-of-the-as lifelong learning algorithms perform on robotic reinforcement learning tasks is urgently needed in the research community, which the major motivation of our work.
In this paper, we establish a framework to study the possibilities to extend the existing supervised lifelong learning method to the reinforcement scenarios, demonstrating their effectiveness on
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

carefully-designed multi-task manipulation scenario. Our contributions can be summarized into three-folds':
1. We evaluate Soft-Actor- ( with the state-of-the-ee lifelong learning algorithms from four different categories using our benchmark. Forgetting performances and robustness to task orders are evaluated in experiments. We further analyze the possibility of applying different lifelong learning algorithms into RI. based on our experiments.
2. We propose manipulation benchmark for lifelong learning with ten different manipulation tasks. Policies trained on our benchmark could quickly converge.
3. We propose a framework which could modify traditional lifelong learning algorithms from supervised learning into reinforcement learning.
2 Related Works
2.1 Reinforcement Learning with Multiple Tasks
In addition to lifelong learning, meta-learning and multi-task learning both include multiple tasks in their problem formulations. They are similar domains but share plenty of differences. It is im- portant to notice that in traditional lifelong learning formulations, tasks assigned to the agent sequentially, which means that the agent will not be able to have access to the training data and environments after finishing training this task. Furthermore, recent researches in lifelong learn- ing focus on overcoming catastrophic forgetting and memorizing all the tasks learned. Though the abilities of transfer learning may also be discussed, they not the questions in this paper.
In the following sub-sections, we will discuss meta-learning and multi-task learning. Especially on the differences and similarities between these domains and lifelong learning.
2.1.1 Meta-L.arrning
In meta-learning, research focuses on achieving fast adaptation based on agents' previous learn- ing procedures. Specifically, [8] introduces an additional gradient descent procedure to update the network parameters for novel task. [9] applies meta-learning to quickly adapt new dynamic model in environment, which leads faster generalization in environment. [10, 11] introduces latent variable to represent the dynamic environment, and could be used to help fast adaptation of robotic tasks.
2.1.2 Multi- Mask- Learning
Multi-task learning is defined as solving multiple tasks at the same time, Multi-task learning includes lifelong learning but is not limited to it. Recent work multi-task learning focuses exploiting the similarities of tasks to accelerate the learning how auxiliary tasks help improve the performance and learning speed of current tasks. Distral[14] introduces a condensed shared policy for faster learning of new task. IMPALA[1S] and its updated version PopArt- Maattt-MPAAA[III uses multiple agents to explore and train in different tasks at the same time. Note that in multi-task learning, tasks are not required to be learned sequentially, which is different from lifelong learning.
novel task. Specifically, [12, 13] focuses on
In summary, lifelong learning, meta-learning, and multi-task learning are mutually similar in the RL domain. However, they focus different performances. In lifelong learning, tasks are trained sequentially and the performance memorizing policies for previous tasks highlighted.
2.2 Lifelong Learning in Reinforcement Learning
In terms of overcoming catastrophic forgetting, most lifelong learning algorithms only focus on solving supervised learning tasks, while work on RI. has received little attention. There are some algorithms trying to solve RL tasks, but they are applied in image-based video games instead of robotic tasks, e.g.. [17] can only be used in video games due its generative image-based methods. [1] and [18] evaluates their methods only on Atari video games instead of robotic tasks. Some
Details of the project can be
website: http:///tdoagggiitcccmmmmmmmmema

---

work on lifelong learning is applied in the robotics domain, but they choose preliminary tasks and they only provide little insight on how the oftt-oofthhe-r lifelong learning algorithms perform in robotic tasks. For example, [19] is implemented to real robots, but their method can be only used in locomotion tasks with limited control space.
[20]evaluated lifelong learning on reinforcement tasks for robotics. It is the most related work in reinforcement lifelong learning. They modify the method mentioned in [15] make it compatible with lifelong settings. Though having good performances, their method is only evaluated on up to three tasks in sequence. [21] presents an algorithm which demonstrates exceeding performance in lifelong learning and multi-task learning. However, it focuses on how previous learning experi- ences would help boost the learning of new related task, while discusses little about catastrophic forgetting, which is the major concern in lifelong learning.
In conclusion, the lifelong learning problem, especially catastrophic forgetting, of reinforcement learning in robotics is poorly studied. An overall investigation about how the state-of-the-ar lifelong learning methods perform on robotic reinforcement learning tasks needed, which the motivation of our work to modify traditional lifelong learning algorithms and compare their performance in RL.
3 Preliminaries
3.1 Markoy Decision Process
We formulate the problem Markov Decision Process (MDP). We use s E Rdd to denote the state of the agent at time step t in task k. The action vector of Rdd sampled from the policy To (at), where 0 denotes the policy parameters. p(st41l 1s a²) denotes the probability of transforming from the state sf to si-1 by executing the action af. The agent will receives a reward rsst, at s(t,) each step. Our goal is to maximizing the sum of the discounted reward for all the tasks Em, >>
k= Ei ( at, st+1)
3.2 Actor-Critic Framework
Croo--Crttccbbseed algorithms compose a majority of RL algorithms. [7, 22, 23] all incorporate the framework. In traditional supervised learning problems, the loss for each trial can be computed by comparing the outputs with the ground-truth labels. However, in most reinforcement learning cases, such labels do not exist. Instead, in the Actor-Critic framework, a Critic is trained simultaneously with an Actor to evaluate the action the Actor chooses. The Critic loss is usually defined by the mean square Bellman error.
L(Φ) El((a(s), at) - (r2 711 dx) max a(
att
where Q denotes the Critic network with parameters Φ) dt denotes whether the state s is terminal state in k-th task. The Actor loss is defined by: L(Ø) = - Qs(st (s))))
4 Reinforcement Lifelong Learning Framework
In this section, we propose the framework to modify traditional supervised lifelong learning algo- rithms to reinforcement learning. We purposefully design this framework to be as simple as possible. Our main contribution lies in the evaluation of lifelong learning algorithms. We want to minimize the influence of the additional network structure the evaluation results.
Traditional lifelong learning algorithms cannot be directly applied to RL because of the following two reasons: (a) the loss in supervised learning is clearly defined, which is the error between pre- dictions and labels. However, it is not straightforward in RL. critic network is needed for the evaluation of the policy network. (b) It would be hard to indicate which task the robot is performing, while supervised learning, e.g., in image classification,, the network usually does not need to know the task ID since the images themselves have sufficient information to indicate which task they are within. In order to solve these two problems, we modify the traditional lifelong learning algorithms in the following ways.
3

---

4.1 Actor-Critic Framework in the Lifelong Learning Algorithms
We adopt the Actor-Critic algorithm into our lifelong learning framework have a parametric eval- uation of the actor policy. we replace the loss (usually MSE or cross entropy) in traditional lifelong learning algorithms with critic loss. In addition, since real-world applications, crities are usu- ally discarded and only the actor (policy network) is used, we only consider the continual learning properties of the actor while we use traditional deep RI. algorithms to learn task-specific critics. However, it is also straightforward to expand the lifelong learning algorithms for the critics.
4.2 Task-Specific Sub-Network
The observation spaces in RL tasks could be identical. In such cases, the policy network will have to distinguish different tasks by task IDs. It would be challenging if the task ID is directly input to the network for learning. Instead, we add task-specific layers into the policy network if lifelong learning algorithms themselves do not include such task-specific parts. Formally, given & tasks, the parameters of the policy network Ty can be decomposed into common parts and specific parts: 0 = [0c} UK {0)}. For task i, the action of can be given by: of To(at) To: (To, (st))- Different sub-networks are chosen for different tasks according to task IDs. In our experiments, the task-specific sub-networks are only a single-layer affine transformation. If net-nes with deeper structures are implemented, the shared parts could be simply an identity transformation while the policy totally relies on the sub-network, which is basically training a policy network for each task.
Details shown in the appendix.
5 Lifelong Manipulation Benchmarks
In this section, we introduce the lifelong manipulation benchmarks. Our benchmarks adapt from ROBEL D' D'Claw[24], a mujoco-based robotic manipulation platform. We modify the tasks and get ten different manipulation tasks. The benchmarks consist of a robotic hand with three independent fingers. Each finger has three degrees of freedom. In our task, the robotic hand needs to turn the valve below 180 degrees. The observation space of the benchmarks consists of joint angles, joint velocities, and the valve angle. The action space of the robot consists of robot joint angles. The reward function is designed based on robot configurations and the valve position to ensure safe manipulation. Each task only differs in the valve shape. Though each task seems to be similar, a policy trained on single task cannot achieve high rewards on the other tasks. All the tasks are designed to have dense rewards for a faster learning procedure. Note that the absolute reward value itself does not have much meaning. A higher performance. Details of the reward function can be seen in [24].
lower reward only indicates better worse
Though Meta- [ provides multi-task benchmark and it is likely to be applied in lifelong learning, Their tasks are far more challenging than ours. Tasks in Meta- World could be difficult in lifelong learning settings. Our benchmarks are used for different levels of evaluations.
One of the important challenges in reinforcement lifelong learning lies in the training time. RL policies are notably hard to train. This problem will be aggravated in the lifelong learning scenarios since each task should be trained one after another by definition. In our benchmark, each task is stable and easy to train. In our experiments, each policy can converge after 80,000 steps of updates using SAC. The details of the benchmark could be seen in Fig. I.
Experiments
6.1 Evaluation Metries
We will mainly focus on the forgetting of lifelong learning. We evaluate each algorithm using the average performance. is is slightly modified from the average accuracy, a standard lifelong learning evaluation metric used [26][27] and [28].:
Average Performance: Let Pi, be the cumulative reward of the task j after the model finishes learning task i. Then the average performance P is defined as: P p= * Σド PK,j, where K denotes the total number of tasks. When evaluating, since use the cumulative reward to evaluate

---

Lifelong Manipulation Benchmarks
Task #1. Square
Task n2, SückA
Task StickB
Task 14. StickC
Task #5. StickD
Task Hexagram
Task #7. Triangle
Task RB. Pentagon
Task #9, Hexagon
Task #10.
Figure I: 10 manipulation tasks for lifelong manipulation benchmarks
policies, the time steps for each episode are set to constant for fairness, no matter whether the task has already converged.
6.2
Taxonomy of Lifelong Learning
Recent lifelong learning algorithms can be classified into the following four categories:
6.2.1
Regularization-Bit Methods
Regularization-basse methods add an additional regularization term in the loss function to ensure the learned network in the new task will not forget important information for the previous tasks. The most intuitive method would add an MSE loss to minimize the parameter changes between the current task and previous tasks. Instead, [1] proposed Elastic Weight Consolidation(EWCC) incorporates the Fisher information matrix to compute the importance of network parameters and tries to keep the important ones unchanged. The most important advantage of regularization methods is that the model size will not change when the number of tasks increases.
6.2.2 Memory-Based Methods
Memory- based methods would use tiny replay memory to store several samples from previously learned tasks and replay when learning tasks. The key idea of memory-based methods is trying to transform lifelong learning problems into classic deep learning tasks. Episodic Replay(FR)[22) combines samples from previous tasks with current tasks when computing the loss for the current task. [29] is an updated version of [28], which adds weight to learn the importance of the current task and previous task.
6.2.3 Gradient-Coontraing Methods
Cradient-constrail methods modify the gradient to ensure that the update for the current task would have minimal influence on previous tasks. Gradient Episodic Memory(GEM)J26] projects the gradi- ent into spaces to ensure the update for the current task will not decrease the performance of previous tasks. [27] proposed A- A-GEM and optimize the computation efficiency GEM. Orthogonal Gradi- ent Descent (GD)) believes moving orthogonally to the gradients of previous tasks would have smallest influence on them. Therefore, it projects the gradients for the current tasks into the or- thogonal space of previous tasks. Continual Learning in Orthogonal Subspaces(CI.OSS)) framework to ensure that the gradients are orthogonal by nature. It combines gradient-constraint methods and memory- based methods for better performance.
designs

---

6.2.4 Aapansion-Based Methods
Expansion-based methods add a new sub-module when learning new task. Thus, the size of the model increases with the learning procedure. Additive Parameter Decomposition(AAP)]]3 includes a shared parameter space with task-specific sub-networks. Learning a new task would add new sub-network and the algorithm would also constrain the changes in the shared parameters.
In order to have an overall comparison of lifelong learning algorithms on RL, we choose one or two algorithms from the categories mentioned above. Oracle means that each task is trained with an individual policy. Oracle does not consider the lifelong learning problem. It is used to indicate the best possible performance each algorithm can achieve. Random means using a random agent in the benchmark tasks. is used to indicate the worst possible performance for each algorithm. SAC[7] means using the original Soft-Actor-Critee algorithm trained continuously from one task to another. However, for the fairness of comparison, we also modify the actor of SAC into shared parts and task parts. It designed to demonstrate the performance of the state-of-the -frte--rrr RL algorithms without lifelong learning. EWC[1] and L2 are representatives of regularization-bar -based methods. L2 only introduces MSE loss for the network differences between the current actor parameters and previous task parameters, as is mentioned in Sec. 6.2.1. L2 is the most intuitive lifelong learning algorithm. AGEM[27] and GEM[26] and CLOS[31] are chosen representa- tives for gradient-constrall chosen for expansion-basee" methods. In addition, APD and CLOS are the state-of-the-ard ones to demonstrate the performance of the frontier research on lifelong learning.
methods. [R[28] is chosen for memory-based methods. APD[32] is
6.3 Results
We first show the performances each task after finishing learning the last task. Each algorithm is trained and updated with nearly the same steps for a fair comparison. The environments for the previous tasks are not allowed to be re-accessed after finishing the training. All the algorithms trained with the same task order. The results are shown in Fig. 2. The average performance is shown in Table I.
In Fig. 2, SAC only achieves a high reward on the last task. indicates that traditional RL cannot solve the lifelong tasks designed in our benchmark tasks. Previous tasks still have a slightly higher performance than a random agent, which suggests each task is not totally independent from others. L. achieves worse results than SAC. It indicates that improper constraints of lifelong learning, e.g., regularization, are not only detrimental to memorizing previous tasks but will impede the learning of the current task. In comparison, EWC achieves better results. However, it has much worse forgetting performance compared to its applications in supervised learning. It could result from the sensitiveness of manipulation tasks mentioned in Section 1. GEM has better performances than AGEM. Compared to AGEM, GEM samples more data from previous tasks when updating. It could lead to a more conservative update, which is important in the sensitive RL environment. Though simple and intuitive, ER achieves a good performance. It is worth noting that in RL, we do not need to design additional replay memories since RL algorithms will need to use replay memories them- selves. CLOS includes nemory-based methods into gradient-based methods by replaying previous tasks when training. It achieves better performances than gradient- based ones, which suggests that memory-based methods could help improve the performance of other lifelong algorithms. Though APD achieves a good result, expansion-based methods will add a sub-module when learning new task in practice. The size of the added sub-module could be as large as training new policy from scratch, which degrades the effect of lifelong learning.
Table 1: The evaluation performance of all the lifelong learning algorithm for manipulation bench-
marks.
Algorithm
Oracle
Random
SAC
1.2
EWC
AGEM
GEM
ER
CLOS
APD
Mean
1709.44
-810,432
-35,57
-18887
249.73
286.44
549.52
867.21
1543.90
1379.65
Std.
131.97
69.87
64.01
98.11
824.50
736.11
929.91
586.73
239.49
171.25
To better demonstrate the forgetting details of each algorithm, we provide figures of cumulative rewards during the training procedures. The results are shown in Fig. 3. The x-axis denotes training steps. The y-axis denotes cumulative rewards. Each column shows performances of a policy for all

---

(al SAC
(b/12
(c) FWc
AGEM
(c) GEM
(1) ER
(8) APD
(h) CLOS
Figure 2: Performance of lifelong learning methods with the RL framework mentioned in Sec. 4. The x-axis denotes task IDs. The y-axis denotes cumulative rewards. Each reinforcement lifelong learning algorithm consists of a lifelong learning algorithm and SAC as the RL algorithm. The figures show the performances of the policy each task after finishing learning the last one. Oracle and random agent is included in each figure for comparison.
the tasks, from before training a certain task to after training the last task. It is used to demonstrate the influence of training other tasks the current task.
In Fig. 3, the performance for each task in SAC only peaks when training it, which means it quickly forgets the policy for previous tasks. The rewards in ER, APD, CLOS will keep at high level even after training a certain task. CLOS combines several methods together and has the best performance among the algorithms mentioned above. However, the training of CLOS is less stable than APD and ER. Specifically, the performance will vary a lot during the training procedure. One has to save several policies and search for the best one when using CLOS.
D012
Te CDV
Figure 3: Performance changes of lifelong learning methods along with the RL training procedure. Different lifelong learning algorithms trained with the same task order. Each column demon- strates the cumulative reward of the policy from before training on a task to finishing training the last task.
A reliable lifelong learning algorithm would not be influenced by the task orders. We train each algorithm in three different task orders and evaluate the policies on all the ten tasks after finishing training the last task. Results shown in Fig. 4.
Task order experiments Fig. demonstrate that the algorithms with satisfying performance in the previous experiments are usually also robust order changes. APD demonstrates the best robustness
7

---

to task orders, since expanding the network makes it similar to training an independent policy for each task.
[a] SAC
L2
(a] EWC
[6] AGEM
8138.
(a)
(a)
APD
[a] CLos
Figure 4: Performances of lifelong learning algorithms trained with different task orders. Perfor- mances are evaluated after training the last task. Columns with the same color indicate experiments within the same task order.
7 Discussion
In this section, we conclude several concise wake-away conclusions of our evaluations.
Firstly, RL tasks would intensify the weakness of lifelong learning algorithms. EWC, GEM, and AGEM, which are notably effective in supervised lifelong learning, have a poor performance in RL. Even some could have a good result on supervised learning, the performance could be dramatically degraded in RL. Specifically, lifelong learning in robotic manipulation tasks far sensitive than that in supervised learning. In RL settings, the gaps between different lifelong learning methods are greater than that in supervised learning.
Secondly, expansion-basee based methods and memory-basee methods have good performance on lifelong RL methods. Future work could consider incorporating these two methods part of its algorithm for a better performance. Regularization based methods and gradient-baut based methods still suffer in RL tasks. But on the other hand, the model size is limited in these methods, which has great potential in real robot applications.
Thirdly, existing lifelong learning methods requires memorizing lots of data, which makes it impractical robotic applications. They require memorizing samples or models from previous tasks. However, for applications on real robots, it will not be possible for the robot to remember too many samples or will not allow a substantial increase in the model size. Existing lifelong learning methods still do not have perfect solution to it.
We also point out that more comprehensive evaluation metries in reinforcement lifelong learn- ing is needed. Unlike supervised learning, even if the cumulative reward of task could decrease greatly, it does not really mean the policy forgets greatly about the task. Since RL tasks could be highly sensitive and the policy might not be robust, slight forgetting could lead to complete failure of the task.
8 Conclusion
In this paper, we focus on the problem of how the state-of-the-an lifelong learning algorithms per- form on RL manipulation tasks. We design framework to modify traditional lifelong learning algo- rithms used in supervised learning into the ones in RL. We also develop manipulation benchmark for evaluation. We test seven lifelong learning algorithms with three baselines on our benchmark. According to our experiments, we point out the sensitiveness challenge of lifelong RI. tasks com- pared to lifelong supervised learning tasks. We also summarize the potentials of applying different kinds of lifelong learning tasks in RI.

---

Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 62122 62120106005, and the Joint Fund of Science & Technology Department of Liaoning Province and State Key Laboratory of Robotics, China (2020-KT-22-06..
References
[1] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, Quan, T. Ramalho, A.
et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
[2] M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, Elsevier, 1989.
[3] J. L. McClelland, B. L. McNaughton, and R. C. O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
[4] M. Lungarella, G. Metta, R. Pfeifer, and G. Sandini. Developmental robotics: a survey. Con- nection science, 15(4):151-190, 2003.
[5] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In Thirty-Secont AAAI Conference on Artificial Intelligence, 2018.
[6] X. Y. Lee, Ghadai, K. L. Tan, C. Hegde, and S. Sarkar. Spatiotemporally constrained action space attacks on deep reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 4577-4584, 2020.
[7] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.59055, 2018.
[8] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1133 135. PMLR, 2017.
[9] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint 1803 2018
[10] R. Raileanu, M. Goldstein, A. Szlam, and Fergus. Fast adaptation to new environments via policy-dynamics value functions. In International Conference on Machine Learning, pages 7920-7931. PMLR, 2020.
[11] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In International conference on ma- chine learning, pages 5331-5344 PMLR, 2019.
[12) Z. D. Guo, B. A. Pires, B. Piot, J.-B. Grill, F. Altché, R. Munos, and M. G. Azar. Boot- strap latent-predictive representations for multitask reinforcement learning. In International Conference on Machine Learning, pages 3875-3886. PMLR, 2020.
[13] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEÉE international conference on robotics and automation (ICRA), pages 3758-3765. IEEE, 2018.
[14] Y. W. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pas- canu. Distral: Robust multitask reinforcement learning. arXiv preprint 1007:1777.041111 2017.
[15] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih T. Ward, Y. Doron, V. Firoiu, Harley, Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML, pages 1407-1416. PMLR, 2018.

---

[16] M. Hessel, H. Soyer, L. Espeholt, W. Czamecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforeement learning with popart. Proceedings of the AAAI Conference on Artificial 2019.
Intelligence, volume 33, pages
[17] H. Caselles-Dupré, M. Garcia-Ortic and D. Filliat. S-trigger: Continual state representation learning via triggeri generative replay. arXiv preprint arXiv: 09434, 2019.
[18] J. Schwarz, W. Czarnecki, J. Luketina, A. R. Hadsell. Progress & compress: A scalable framework for continual learning. In Interna- tional Conference on Machine Learning, pages 4528-4537. PMLR, 2018.
Y. W. Teh, R. Pascanu, and
[19] R. Traoré, H. Caselles-Duprt T.
Lesort, T. Sun, N. Fiaz-Rodriguez, and D. Filliat. Contin- ual reinforcement learning deployed in real-life using policy distillation and sim2real transfer. arXiv preprint arXiv: 1906::400444422 2019.
[20] D. Rolnick, A. Ahuja, J. Schwarz, P. Lillicrap, and G. Wayne. Experience replay for contin- ual learning. arXiv preprint arXiv: /./ /682, 2018.
[21] H. B. Ammar, E. Eaton, P. Ruvolo, and M. Taylor. Online multi-task learning for policy gradient methods. In ICML, pages 2206-121. PMLR, 2014.
[22] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[23] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[24] M. Ahn, H. Zhu, K. Hartikainen, H. Ponte, A. Gupta, S. Levine, and V. Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In CORL, pages 1300-1313 1313. PMLR, 2020.
[25] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CORL, pages 1094-1100. PMLR, 2020.
[26] D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. In Proceed- ings of the 31st International Conference on Neural Information Processing Systems, pages 6470-6479, 2017.
[27] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018.
[28] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajamthan, P. K. Dokania, P. H. Torr, and M. Ran- zato. On tiny episodic memories continual learning. arXiv preprint arXiv: 1902. 10486, 2019.
[29] Y. Guo, M. Liu, T. Yang, and T. Rosing. Improved schemes for episodic memory-based lifelong learning. Advances in Neural Information Processing Systems, 33, 2020.
[30] M. Farajtabar, N. Azizan, A. Mott, and A. Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762- 3773. PMLR, 2020.
[31] A. Chaudhry, N. Khan, P. Dokania, and P. Torr. Continual learning in low-rank orthogonal subspaces. Advances Neural Information Processing Systems, 33, 2020.
[32] J. Yoon, S. Kim, E. Yang, and S. J. Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Eighth International Conference Learning Represen- tations, ICLR 2020. ICLR, 2020.
10


Reviews:
[{"review_id": "t7aKsJ7Gl3P", "reviewer": "Reviewer_FpJK", "summary": "This work provides a framework for evaluating various life long learning techniques in a robotic reinforcement learning context. Several approaches (regularization, memory, gradient, and expansion based) are compared. Each approach must learn a set of 10 knob turning tasks in sequence, and performance on all 10 is recorded afterwards.   The memory and expansion based approaches outperform the regularization and gradient based approaches at the cost of requiring replay buffer storage of past episodes or a growing policy size.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe evaluation setup from this paper is well motivated and implemented.  It captures a temporal nature of tasks changing over time, different from a multi-task setup where multiple tasks are learned concurrently.  \n\nWeaknesses:\n- This work doesn't propose a novel method, only a benchmark and comparison of existing methods. \n\nPerhaps another interesting dimension in addition to the forgetting of previous tasks would be number of rollouts required to learn a new task to some threshold success rate. For example, one would hope a capable life long learning agent with more past experiences would be able to learn a new task more quickly than an agent with less past experiences to leverage.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The framework for a life long learning benchmark is a great idea.  That being said, the benchmark articulated in this work leaves a good bit to be desired. It would be interesting if other dimensions beyond catastrophic forgetting where to be discussed.  For example, looking at how number of rollouts needed to learn a new task changes overtime.  \n\nIt is also a bit unclear to me how well a set of 10 very similar tasks captures the requirements needed for a life long learning system. \n\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "\n- Table 1: Could you elaborate on what a negative score means? Is the knob spinning in the opposite direction?", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "_v4LSlTbS6f", "reviewer": "Reviewer_x2T2", "summary": "The main contribution of this paper are summarized as follows:\nThe manuscript proposes a novel framework which could modify traditional lifelong learning algorithms in supervised learning into reinforcement learning.\nThe manuscript proposes a manipulation benchmark for lifelong learning with ten different manipulation tasks. \nThe manuscript evaluates Soft-Actor-Critic with the state-of-the-art lifelong learning algorithms from four different categories using proposed benchmark, and evaluates forgetting performances and robustness to task orders in experiments. In the end, the manuscript analyzes the potential of different lifelong learning algorithms applied in RL based on experiments.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main advantages of this manuscript are two aspects. First, this manuscript proposes a novel reinforcement lifelong learning framework in order to make reinforcement learning algorithm owns the ability of overcoming catastrophic forgetting for previously learned tasks, and owns the ability of speeding up the convergence rate for new task. Second, this manuscript proposes a manipulation benchmark for lifelong learning with ten different manipulation tasks. \nHowever, there are some technical problems left at current version, some of which are given as follows.\n1) The manuscript states that \"Though recent lifelong learning research has promising performances on supervised learning, little evidence has proven that they could directly be applied in robotic RL tasks.\" in introduction. However, lifelong reinforcement learning has been studied for several years. For example, \"Online Multi-Task Learning for Policy Gradient Methods\". It would be better to pay more attention in investigation.\n2) The novel reinforcement lifelong learning framework in this manuscript is similar with overcoming catastrophic forgetting algorithm framework in \"Learning without Forgetting\". It would be better to provide more details about difference between novel reinforcement lifelong learning framework and the framework in paper \"Learning without Forgetting\".\n3) It would be better to provide the comparative experiment with more details for novel reinforcement lifelong learning framework and state-of-the-art baselines. Current tables and figures in manuscript could support the advantage of manipulation benchmark for lifelong learning rather than the advantage of performance for proposed novel reinforcement lifelong learning framework.\n4) It would be better to unify proper noun. For example, the manuscript states that \" Reinforcement Lifelong Learning \" in title, and states that \" Lifelong Reinforcement Learning \" in content.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The related investigation was not enough. And current tables and figures in manuscript could not support the advantage of performance for proposed novel reinforcement lifelong learning framework.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1) The novel reinforcement lifelong learning framework in this manuscript is similar with overcoming catastrophic forgetting algorithm framework in \"Learning without Forgetting\". It would be better to provide more details about difference between novel reinforcement lifelong learning framework and the framework in paper \"Learning without Forgetting\".\n2) It would be better to provide the comparative experiment with more details for novel reinforcement lifelong learning framework and state-of-the-art baselines. Current tables and figures in manuscript could support the advantage of manipulation benchmark for lifelong learning rather than the advantage of performance for proposed novel reinforcement lifelong learning framework.\n3) It would be better to unify proper noun. For example, the manuscript states that \" Reinforcement Lifelong Learning \" in title, and states that \" Lifelong Reinforcement Learning \" in content.\n4) The manuscript states that \"Though recent lifelong learning research has promising performances on supervised learning, little evidence has proven that they could directly be applied in robotic RL tasks.\" in introduction. However, lifelong reinforcement learning has been studied for several years. For example, \"Online Multi-Task Learning for Policy Gradient Methods\". It would be better to pay more attention in investigation."}}, {"review_id": "TQQ8Y1T3dhW", "reviewer": "Reviewer_R49v", "summary": "This paper evaluates 7 lifelong learning methods from supervised learning for reinforcement learning. It uses a credible, minimal lifelong learning framework based on SAC and a bespoke manipulation benchmark, based on ROBEL D'Claw. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n* Presents important experiments which establish a baseline for lifelong RL research\n* High coverage of different LL methods and techniques\n* Considers important LL questions, such as task ordering\n* Simple paper which seeks to do one thing well\n\nWeaknesses\n---\n* Editing and writing is sometimes weak; many sentences are awkward, grammatically incorrect, or not very idiomatic; missing labels on plots; awkward use of whitespace (use the ~ for spacing references, i.e. `Fig.~\\ref{fig:my_figure}`!)\n* I would like to see some more concise take-aways in the discussion section. As it is, it is quite dense.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think evaluative and empirical work is very neglected in the field, and I would love to see more papers like this in the future. \n\nI support accepting this paper because it contains important data and analysis for authors and reviewers working in lifelong reinforcement learning. The authors are correct that there are precious few resources with any information on the performance of popular LL methods on reinforcement learning, and especially few which do so using robotics challenges. This paper, in its best form (see my comments below and above) would be an asset for those studying LLRL for robotics.\n\nI rated this paper weak accept because I believe it is held back by a few presentation issues (see my comments above). I think this paper could use a lot of attention in the discussion section, to provide readers with a few very clear take-aways from the analysis (and perhaps a summary table, if appropriate). Papers like this have the most impact as a quick reference for future researchers. Additionally, a general once-over on presentation, typesetting, grammar, and style would do wonders for the digestability of this work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "See comments above.\n\nDetailed comments:\nLine 95: I don't think it's ever possible for a method (which works well) to be \"too simple and straightforward\". The best method is no method at all! Please reconsider your phrasing here.\n\nFigure 3: Please provide axis labels!\n\nTable 1: Please have a clear heading for this table, and consistent use of rows/columns."}}]

Author rebuttal:

Meta-review: **Updated meta-review after rebuttal**
After careful consideration of the reviews/rebuttals and paper revisions, I recommend accept.

This paper is focussed on an empirical evaluation of continual learning approaches applied to RL (continual learning approaches are typically proposed for the supervised learning setting, and not for the RL setting, so we have little insight on how these approaches would fare in the RL setup.) I agree with the reviewer that states that not proposing a novel method is not a good reason for reject (which is the major concern of the other 2 reviewers). The manuscript initially did have some issues on properly portraying the scope of the presented evaluation, but that was fixed.

**Initial meta-review**
**Summary**:
This paper benchmarks lifelong learning algorithms in the reinforcement learning on manipulation tasks. Current SOTA continual learning algorithms are adapted to the reinforcement learning setting, and then evaluated on a sequence of tasks. 

**Strengths**:
Lifelong learning is an important problem of robotics, and this manuscript provides important insights on how current lifelong learning algorithms that were mostly developed for supervised learning problems perform in RL settings.

A good coverage and evaluation of existing methods

**Weaknesses**:

The reviewers point out that the metrics evaluated in this work are limited to measuring forgetting. However, lifelong learning is also about understanding how quickly a task can be (re-)learned given previously learned representations. The authors should consider adding such metrics to their evaluation.

Clarity and crisp take aways are lacking in a few places. The authors should carefully discuss and analyze the results, and improve the clarity of the manuscript

There is prior work on lifelong reinforcement learning (for simulated robotic tasks), and the authors should discuss such work. Given that this benchmark is also done in simulation, the authors should maybe not put too much emphasis on “little prior work for robotic lifelong RL exist” , but put more emphasis on “benchmarking of lifelong RL algorithms has received little attention”


Decision: Accept (Poster)