Replay-enhanced Continual Reinforcement Learning
Authors: Tiantian Zhang
Kevin Zehua Shen
Zichuan Lin
Bo Yuan
Xueqian Wang
Xiu Li
Deheng Ye
Venue: Accepted by TMLR ()

Abstract: Replaying past experiences has proven to be a highly effective approach for averting catastrophic forgetting in supervised continual learning. However, some crucial factors are still largely ignored, making it vulnerable to serious failure, when used as a solution to forgetting in continual reinforcement learning, even in the context of perfect memory where all data of previous tasks are accessible in the current task. On the one hand, since most reinforcement learning algorithms are not invariant to the reward scale, the previously well-learned tasks (with high rewards) may appear to be more salient to the current learning process than the current task (with small initial rewards). This causes the agent to concentrate on those salient tasks at the expense of generality on the current task. On the other hand, offline learning on replayed tasks while learning a new task may induce a distributional shift between the dataset and the learned policy on old tasks, resulting in forgetting. In this paper, we introduce RECALL, a replay-enhanced method that greatly improves the plasticity of existing replay-based methods on new tasks while effectively avoiding the recurrence of catastrophic forgetting in continual reinforcement learning. RECALL leverages adaptive normalization on approximate targets and policy distillation on old tasks to enhance generality and stability, respectively. Extensive experiments on the Continual World benchmark show that RECALL performs significantly better than purely perfect memory replay, and achieves comparable or better overall performance against state-of-the-art continual learning methods.

Published in Transactions on Machine Learning Research (11/2023)
Replay-enhanced Continual Reinforcement Learning
Tiantian Zhang" Tsinghua University
zI.19@mailss wall90mmall...iinnaaa....m edu.ccm
Kevin Z. Shen" The University of British Columbia
kecins@0@@studend.uue ubc.c.
Zichuan Lin Tencent
aichuamlkn@@@eencenl..comm
Bo Yuan Tsinghua University
boguan@ieee.cog
Xucqian Wang
wwng.uag0sz..singgwaa... edu.ccm
Tsinghua University
Xiu Li Tsinghua University
W.niuussz.ssinghwaa.e edw.c
Deheng Ye Tencent
dericyee@lenceenlccom
Reviewed on OpenReview:
https: openreview. net/ forum? id= fMUUku
Abstract
Replaying past experiences has proven be a highly effective approach for averting catas- trophic forgetting in supervised continual learning. However, some crucial factors still largely ignored, making it vulnerable to serious failure, when used solution to forgetting in continual reinforcement learning, even in the context of perfect memory where all data of previous tasks are accessible in the current task. On the one hand, since most reinforcement learning algorithms are not invariant to the reward scale, the previously well-learned tasks (with high rewards) may appear to be more salient to the current learning process than the current task (with small initial rewards). This causes the agent concentrate on those salient tasks at the expense of generality on the current task. On the other hand, offline learning replayed tasks while learning new task may induce distributional shift between the dataset and the learned policy on old tasks, resulting in forgetting. In this paper, we introduce RECALL, a replay-enhan eppaay-enhanced method that greatly improves the plasticity of existing replay-based methods on new tasks while effectively avoiding the recurrence of catastrophic forgetting in continual reinforcement learning. RECALL leverages adaptive normalization on approximate targets and policy distilaaton on old tasks to enhance generality and stability, respectively. Extensive experiments on the Continual World benchmark show that RECALL performs significantly better than purely perfect memory replay, and achieves comparable or better overall performance against slale-of-the-art continual learning methods.
1 Introduction
Continual learning, emerging machine learning paradigm, examines multiple learning tasks in sequence, where the data distribution and learning objective change through time and is considered an important.
"This work done when Tiantian Zhang and Kevin Shen worked interns Tencent. i *Corresponding authors.

---

Published in Transactions on Machine Learning Research (11/2023)
Second Disk
Second Tisk
Second Tisk
7
0.96
0.95 0.51
0.96 0.57 0:95 0.92
0.57
0.58
0.91
0.93
0.0
0.58
0.66
0.05
0.05
0.79
0.0
0.91 3.0
0.02
0.010.37
0.03 8.0 0.05 0.52 0.01
0.05
0.04
1 0.74 0.73
0.79
0.62
0.57
0.70
0.64
0.74
0.65
0.77
0.0
0.52
0.21
0.72
0.0
0.53
0.41
0.0
0.92
0.99
2 0.00
0.04
0.01
0.14
0:13
0.0 0.05 0.01
0.07
0.03
0.8
0.8
0.8
3.0 0.93 0.09
0.99
0.93
0.78
0.94
3.0
0.98
0.0
0.09
0.28
3.0
0.0
0.00
0.53
0.0
0.04
0.006
000-0.12-00000
0.03
-0.2
0.02
02 0
0.6
066
0.6
p.77 0.64 0.52
0.56
0.7 0:85
0:05
0.64
0.76
0.75
0.0
0.0
0.02
0.0
0.77
0.29
0.0
0.0
0.51
0.0
0.05 0.06-0.02
0.32
0.02
0.01
0.0
0.03
0.02
0.03
0.93 6.02 0.87
0.02 0.94
0.04
5.88
0.03
0.07
0.32
0.07
00
0.0
0.96
0.77
0.0
0.96
00.0
0.01 0.04
05
- 3.05- 0.0 0.02
7 0.79 0.62 0.68 0.84 0.74 0:54
0.78 0.91 0.84 0.66
9.3
9:34
0.68
0:39 0.26
9.55
0.16
0.89
3.0
> 0.09 0.1 0.07
0.030.00
9.06 0 0.000
-0.2
-02
-02
0.75
0:35
0.11
0.020.19
2.0
0.99
1.0
2.0 1.0 0.39 0:93 0.00
1.0
0.0
0.2
0.08
0.29
5.0 0.58
0.57
0.0
0.35
2.0
0.0
0.0
0.0
0.0 .00 0.01 0.04 0.0
0.0
0.0
(a) First. task performance
(b) Second task performance
(c) First task forgetting
The evaluation matrices in terms of success rate with Perfect Memory on pairwise sequential tasks from Continual World. The numbers 0 ~ 9 indicate identifications of ten different tasks, and the mapping between them and the proper task names are shown in Figure 7 in Appendix [ For example, if the identifications of two tasks is 5 and 0 respectively, it means that the learning is conducted on task sequence M = STANDLE-PRRSSS-SIDEEE performance in (a) and (b) and a reversed version to show the level of forgetting in (c), where darker red indicates worse results. The average values of (a), (b), and (c) are 0.87,0.4 and 0.02, respectively. It is clear that naive experience replay with perfect memory can guarantee the stability to a large extent on RL tasks, Nevertheless, it still exhibits certain degree of forgetting on some tasks. Even worse, it suffers from severe plasticity restriction on the learning of new tasks.
Figure 1:
HAMMER-V1], We use the same colorbars to visualize the
step toward artificial general intelligence (Parisi et al., 2019 De Lange et al. 2021 Wang et al., 2023p. An effective continual learning system must emphasize two potentially conflicting optimization goals. First, when learned scenario is encountered again, the agent is expected to immediately demonstrate good performance, idcally as good as before. Second, when a new scenario arises, the agent should conduct quick learning and gain new skills without being limited by the maintenance of previously acquired skills. These conflicting objectives - adapting to new tasks while maintaining the knowledge of old ones, correspond to the challenge known as the plasticity-stability dilemma in artificial and biological neural systems (Mirzadeh et al., 2020).
Catastrophic forgetting is the quintessential failure mode of continual learning in which the acquisition of new knowledge gradually overwrites old knowledge, resulting in desirable plasticity but limited stability. Inspired by the memory consolidation mechanism of hippocampus replay inside biological systems, replaying previous data is considered a simple yet effective way to mitigate catastrophic forgetting (Rebuffi et al. 2017 [sele & Cosgun 2018 Rolnick et al.) 2019 Korycki & Krawezyk/ 2021), and has been widely adopted in supervised
continual learning (Rebuffi et al. 2017 Iscle & Cosgun 2018 Korycki & Krawezyk 2021).
Different from supervised learning with naturally well scaled loss functions (e-g., cross entropy) and stationary training distribution, reinforcement learning (RL) is a goal-oriented online sequential meeiiion-mmking and learning process (Sutton & Barto 2018). It involves iteratively interacting with the environment and collecting experiences, typically with the most recently learned policy, and then using these experiences to improve the policy to maximize the reward function. In this process, the distribution of collected experiences is inherently non-stationarr, due to the constantly updated policy. Recently, a technique named CLEAR demonstrates the effectiveness of experience replay for reducing catastrophic forgetting in contimual RL (Rolnick et al. 2019 However, other related works (Wolezyk et al., 2021 2022) show that replay-based continual RL. methods suffer from rather poor performance on the newly proposed Continual World benchmark. Based on this, we conducted systematic experimental study on related tasks. As shown in Figure [ the naive migration of raslay-based methods to continual RL may not perform well on learning sequential tasks by single learning system with limited representation capacities, even in the context of perfect memory where all experiences are kept in the buller. More specifically, inspired by the stated balancing issue of multiple tasks competing for limited resources of a single learning system in multi-task deep reinforcement learning (Hessel et al.) 2019), the saliency of a task for the agent increases with the magnitude and density of the rewards observed in that task, which may differ dramatically across tasks or at various learning phases within the same task. This factor is likely to encourage the agent to focus tasks that have been learned well in the past instead
2

---

Published in Transactions on Machine Learning Research (11/2023)
of the current task that presents small and sparse initial rewards (suppressing plasticity). Additionally, using past experiences from old tasks as offline data via RL loss to prevent forgetting is a typical offline RL paradigm due to the absence of further interaction. It may cause standard off-policy RL methods to fail due to overestimation of values induced by the distributional shift between the dataset and the Icarned policy, degrading the well learned performance on previous tasks and resulting in forgetting (disrupting stability).
In this paper, we address the aforementioned two issues to provide an effective replay-enhanced method for continual RL settings. We propose Replay-Enhanne ContinuAL rL (RECALL), an improved version of the naive replay for continual RL, which incorporates the adaptive normalization mechanism approximate targets used in value function learning and the policy distillation technique for offline policy preservation. The main contributions of this work are summarized as below:
Scale invariant replay-enhanced continual RL. We investigate the issue of limited plasticity for subsequent tasks in replay-basod continual RL settings, and introduce adaptive normalization on the largets to balance the contribution of each task to the agent's updates, alleviating this limitation.
Policy distillation for offline policy preservation. We apply the distillation technique to the policies for old tasks to prevent forgetting caused by offline training, further enhancing stability.
Empirical validation on Continual World. Extensive experiments on suite of realistic robotic manipulation tasks are conducted to validate the overall superiority of our method over baselines in terms of average performance, forgetting, and forward transfer.
2
Related Work
Catastrophic forgetting has long been recognized as a key issue in neural networks, particularly in situations where soquential tasks are learned continuously (Ring 1997 French 1999). Recently, a variety of approaches have been investigated to combat catastrophic forgetting in continual learning. According to how the knowledge of previous tasks is retained and leveraged, they can be classified into three major categories: parameter isolation methods, regularization- baasdd methods, and replay methods.
Parameter isolation methods This family of works separately optimizes an isolated parameter subspace dedicated to each task throughout the network, where the architectural resources can be fixed (Fernando ct al. 2017 Mallya & Lazebnik, 2018) incrementally expanded (such as the network capacity (Rusu et al., 2016b or policy library (Wang et al. 2019 2022)). These strategies avoid catastrophic forgetting by protecting ing all weights for the previous tasks from being perturbed by new information but knowledge transfer and generalization between tasks might be restricted, with unnecessary redundancy in the network structure.
Regularization-basel methods Regularization-basel approaches protect learned knowledge from forget- ting by imposing an extra regularization term on the learning objective, penalizing large updates on important weights (Kirkpatrick et al., 2017; Kessler et al.] 2020) or policies (Rusu al.) 2016aa Traoré et al. 2019 Zhang et al. 2022 2023) for previous tasks. This family of works requires careful design of regularization terms and fine-tuning of their associated coefficients. It is easy to implement and tends to perform well on small sets of tasks, but still faces performance trade-offs on new and old tasks as their number increases.
Replay methods Experience replay is a basic and powerful strategy for reinforcing the significance of experiences from past tasks during continual learning. The core idca of replay methods is to store samples of past tasks (Isele & Cosgun| 2018: Rolnick et al., 2019 Riemer et al. 2019: Korycki & Krawczyk 2021)
Atkinson et al. 2021) to maintain knowledge about the past in the network. These previous task samples are replayed while learning new tasks in the form of either being reused as model inputs for rehearsal (Shin et al., 2017 Isele & Cosgun 2018
generate pseudo-samples from generative model (Shin et al.) 2017
Rolnick et al. 2019 Korycki & Krawczyk] 2021 Atkinson et al.] 2021) or constraining the optimization of
new tasks (Rolnick et al. 2019 Riemer et al. 2019), yielding decent results against catastrophic forgetting.
While storing past experiences in replay methods can be memory-intensive it is an attractive strategy when memory is sufficient. due to its simplicity and excellent performance in redueing forgetting. A theoretical
3

---

Published in Transactions on Machine Learning Research (11/2023)
analysis (Knoblanch et al. 2020) has demonstrated the necessity of perfect memory to resolve the NP-hard problem of optimal continual learning. It. also shows that replaying or reconstructing observations from previously observed tasks is likely to be more effective in developing reliable continual learning algorithmss in comparison with regularization-bant baassa approaches. Mcanwhile, some studics (Rebuffi et al. 2017 Isele & Cosgun 2018 Rolnick et al. 2019) show that it sufficient preserve a small quantity selective experiences using sampling tactics such as reservoir sampling when memory is severely constrained.
Most existing replay-based studies concentrate classification tasks, whereas only few works look into deep RL. CLEAR (Rolnick al. 2019 provides preliminary evidence on the value of replay within the deep RL framework, but it has only been empirically validated on tasks with comparable reward scales, without any consideration of how the scale of rewards across sequential lasks may affect the learning process. Recent works (Wolzzyk et al. 2021 2022) on a benchmark suite for continual RL, called Continual World, indicate that even with perfect memory, common replay-basce methods might still suffer from significant failures on certain robotic tasks. By contrast, our proposed RECALL soeks to offer an effective remody to tackle
this challenge, inspired by the power of knowledge distillation (Rusu et. al. 2016a
Zhang et al.
2022 and
adaptive normalization for target scale invariant updates (van Hasselt et al.
2016 Hessel et al.
2019
3 Preliminaries
Reinforcement Learning RL is commonly studied fol- lowing the MDP framework, which is defined as a tuple M = (S, A, A,P, R, 7), where S is the set of states; A is the set of actions; P. S Axx + 1] is the transition probability function; R S x A S + R is the reward function, and E [0, is the discount factor. At each time step E N, the agent moves from 8t to 8t+1 with probability p(5441|s;,a) after lakes action (t,, and receives instant reward rz- The goal of RL to find an optimal policy from experimental trials and relatively simple feedbacks received, enabling the agent to actively interact with the environment to obtain maximum cumulative reward.
1.0
Current Task Success
0.8-
0.6-
First Task
Second Task
0.2
0.0
0.0
0.5
1.0
1/5
2.0
Steps (x10*)
Figure 2: An example of the learning curve of Perfect Memory showing poor plasticity.
Soft Actor Critic Similar to (Woczyyk et al., 2022), we use the soft actor-critic (SAC) (Haarnoja et al., 2018a by as the underlying RL algorithm in this paper. It is an off-policy algorithm with experience replay, based on the maximum entropy principle, which is especially beneficial for replay-based continual learning. Formally, let To (ar(()) denote the policy network with parameters e and Qu(:t, uz) denote the Q-value function with parameters 0. Then, the Q-function can be trained to minimize the soft Bellman residual
(o) )ao 1 (Q)(s))a)) a.) (r(st, az) + (1
(1)
where = - (sr 0z) a log (T((a is the soft state value function and a is the temperature parameter that determines the relative importance of the entropy term versus the reward. The policy can be updated by minimizing
L (x( = EyyD Eu,~ (a (og(5)())1s))) - Qo(s,, at)]]-
Notably, under the replay-based continual RL setting, replay buffer D here stores both new experiences Dnew collected from the current task and replayed experiences Data from the historical ones, ie.,, D = D Dnew U Data-
Perfect Memory Replay in Continual World To examine the replay method in the context of continual RL, we systematically conduct preliminary experiments on 100 sequential tasks created through permuting two of the ten different realistic robotic manipulation tasks (sce appendix A from the latest Continual World (Wolczyk al., 2021) benchmark, where each task lasts for 1M steps in its corresponding environment. We assume multi-head network setting, and keep all the experiences in the replay buffer to allow for
4

---

Published in Transactions on Machine Learning Research (11/2023)
generous replay, dubbed Perfect Memory in (Woleyyk et al. 2021 After ending the training on both tasks, we evaluate the final performance (success rate) on the first and second tasks as well as the forgetting of the first task. The results are shown in Figure | from which we can observe the following findings:
Decent stability, According to Figure [(a]
(0.87 average succoss rate), the agent does perform well on the majority of first tasks after training is complete, which demonstrates that replaying experiences of past tasks, in supervised learning, can also effectively ensure the stability of continual learning algorithms in RI. scenarios.
Limited plasticity. Unfortunately, as shown in Figure 1(b) (0.44 average success rate), the agent shows no (31 out of 100 tasks have a success rate of zero) or weak (25 out of 100 tasks have a success rate of less than 0.5) success on a considerable percentage of the second tasks, indicating that the plasticity is severely restricted. Figure 2 illustrates an example of the training curve in terms of success rate for the current task (the one being trained) that suffers such plasticity limitation, in which the agent does not get any effective learning the second task. More details about the losses and Q value curves for the corresponding current task are provided in Appendix C.1 (see Figure Additionally, we conducted four experiments in Appendix [.1 that used actor and critic networks that were either shared or not shared among tasks. The results (See Table 5 demonstrate that the limited plasticity suffered by Perfect Memory is primarily due to the shared critic network. Inspired by the studics in (van Hassolt et al.) 2016 Hessel et al., 2019). we mainly focus on addressing the adverse effects on value function optimization caused by significant difference in the initially observed reward scale of the subsequent task relative to that of well learned old tasks.
Mild forgetting. While the agent performs well on most of the first tasks, Figure 1(c) (0.02 average
forgetting) shows that there are still a few tasks that exhibit some degree of forgetting (the success rate decreased by over 0.1 in 13 out of 100 tasks) and the maximum forgetting reached 0.32. The
learning curves for the performance of these 13 tasks are provided in Appendix C.2 (see Figure
9p. We also verified that the forgetting shown here is essentially a result of the offline learning for
replayed lasks in Appendix C.2 by conducting further experiments of Perfect Memory using offline and online replay modes, respectively (See Table 6
4 The RECALL Method
RECALL employs multi-head neural network training for both actor and critic, with each head being responsible for a specific task, which is widely used in continual learning. We define a task sequence M = [M, M2,..., Mx] of N tasks, where M₁ i E NV is specific MDP that symbolizes the zth task encountered during learning. When the zth task emerges, the aim of RECALL is to update parameters θ = 0 of policy To and value function Qo to achieve maximum return on all encountered tasks [M,, M2..., M.]: O arg maxe Σ;= JM, (O), where JM, is the expected return on task Mj.
In RECALL, propose to utilize adaptive normalization on targets to balance the contribution of each task to the agent's updates to ensure the plasticity for new tasks, together with the distillation technique to the policies for old tasks to prevent forgetting caused by offline training. The core components of the training framework are shown in Figure 3.
The PopArt method is a technique specifically used address the value function scaling problem in deep RL (van Hasselt al., 2016). It scales the value function to ensure that its output is within a suitable range, which helps to improve training stability and efficiency. In multi-task RL, agents typically switch between different tasks, hence the need to train different value functions for each task. Since each task may have different reward signals, the output range of cach task's value function may also differ. This leads to the value function scaling problem (Hessel et al.) 2019). Similarly, in replay-based continual RL, value functions on both current and past task experiences need to be learned while learning a new task. Since each task also generally have different reward scales, we can view this process as a multi-task learning on current and historical tasks. Therefore, applying the PopArt method to address the value function scaling problem in replay-based continual RT is reasonable and necessary.
5

---

Published in Transactions on Machine Learning Research (11/2023)
FC layers last layer
(w,, b2)
Q norm
Normalized Target
(W2, b2)
Qa porm
norm
Q,
Quarn
a+)
0
(Wi, b.)
Qa morm
unnormalized
normalized
Q0.corm
(w,, Bn)
Qa
(a) Normalized Q network
(b) Loss function normalized Q network
Figure 3: The core components of the RECALL scheme. For each input (st, de), the normalized Q network ultimately outputs only the normalized Q value of the head associated with the task to which it belongs.
To this end, we employ PopArt normalization, developed to derive scale invariant algorithm for value-based RL, to facilitate learning new tasks. Concretely, we consider optimizing normalized value function Qe = [Q) L Re 2 with N output heads, one for each task in the task sequence. In the following content, for each input (st, at). we default to using the normalized Q value of the head corresponding to the lask to which it belongs and updating the related parameters. Based on this, we omit the subscript i for clarity. Given the targets denoted as Qø.r. we conduct an affine transformation on it to get normalized targets as Qg.- = a (Qa.r p), where a and μ are scale and shift parameters. Notably, in the normalized Q network, each head has its own (a, 14) learned from the data of the associated task. Under this setting, the loss of Qo. can be expressed as:
LQ (0) E( D UDold Q0 uz) - Q, (st, a.)
(3)
where
Q0, r(ss, at) +. BBeesiop [Bat (oQo nnrm (8141, a++11 + p a log To (ctts))
(4)
and oQu, n.omm (st, at) + is the unnormalized function of the target normalized Q network Q Accordingly, the loss function of the policy network is rewritten as:
mo (Ø) = BBe, Bai (a log(TT((;()))) ne (st, a:)]].
(5)
Here, the loss functions La (0) and f.r.norm (Ø) are applied on experiences from both old and new tasks. In general, our experiments use a 50-50 experience mixture of novel and replayed tasks, as recommended in (Rolnick et al. 2019). For each sample, only the head associated to the task that belongs to in the value and policy networks are updated. In addition, after each SAC update, RECALL is required to incrementally update the scale and shift parameters to achieve adaptively targets rescaling:
Ne = = + Be(Qo.- mt-1) and of = 2 where
vt-1 + Bc(Q 1)-1),,
(6)
where ii estimates the second moment of the targets, and 3, [0, 1] is the step size. Then, the last layer weights bw of the corresponding head in the normalized Q network also need to be updated accordingly to preserve the outputs of the unnormalized function precisely after the scale and shift change:
w' = w'=o-low b' = o -0b -
(7)
In addition, in order to prevent forgetting caused by oflline training, we employ the policy distillation technique on the replayed tasks to preventing the distributional shift between the past experiences and the Icarned policics of old tasks while learning a new task. Specifically, the data collection depends only on the policy used for interaction rather than the value function, and the agent generally achieves good policy on the corresponding task at the end of each task's learning period. Therefore, only need to add additional

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm Heelay-Enhannee ContinuAL rL (RECALL)
Input: task sequence M = M2, MN], policy Ts. value function Qo, replay buffer Dold D Dnew
Parameter: regularization coefficient for policy distillation À
Output: approximate optimal policy and value function T Q7
1: Train SAC with PopArt normalization
lask M1:
2: Interact with environment of task M1 and store transitions in Dnew
3: Sample mini-batches from Dnew and minimize Q (0), nornor (0).
4: for task M,, i 2,..,N N do
Gather actor outputs Totd(-|st) for each state st Dnew and populate Dold Drew 0
Interact with environment of task M, through the best-return exploration the beginning of each task and store transitions in Dnew
8: Train SAC on task M,, with the following modified update rule:
9: Sample (st, at, Ft, 86+1] ~ Dotd Unew and compute Lq (0), L morm (0)
10: Sample [se, Tota( (-(+]] |] Dold and compute Creveea (Ø)
11:
Minimize LQuamm (0) + (o.)rm(()) + A (0).
12: end for
13: return 3 Qg-
regularization term to the loss for policy (actor) network optimization to penalize the KL divergence between the historical and current policy distributions on the old tasks when training the policy network. Formally, this corresponds to adding the distillation loss function:
Cravena (0) = Ber [KL(TT(-(s)) TT)d(-1s:]]]-
(8)
Note that Craasmu (Ø) is only applied replayed experiences of old tasks, and sold the historical policy obtained after ending the training on the associated replayed task. In implementations, before each new task training starts, we compute Tota(-ls;) for all experiences of the previous task through the latest learned policy and store them along with the corresponding experiences in Dota for subsequent use.
The Complete Scheme Finally, we combine Equations 3 5 and g to form joint optimization scheme. Namely, we solve the continual RL problem based on the experience replay method with the following optimization objective:
min Lauar (0) + Lr, (0) + A (Ø) 8,0
where the hyperparameter À is the policy distillation regularization coefficient to control the deviation degree between the historical and current policy distributions of old tasks. The complete procedure of RECALL is described in Algorithm [ As an additional note, at the beginning of cach new task, initialize the weights of its associated output head in both actor and critic to the already learned head that obtained the best return on that task to facilitate exploration and adaptation. This is referred to as best-return exploration in (Wolczyk et al. 2022) and has been shown experimentally to be a non-negligible SAC component for promoting forward transfer.
5 Experimental Evaluation
We conduct comprehensive experiments suite of realistic robotic manipulation tasks from the Continual World benchmark (De Lange et al. 2021 sceking to answer the overarching questions:
QI: Can RECALL eliminate plasticity limitation while increasing stability?
Q2: Does RECALL achieve better continual reinforcement learning compared with state-of the-art methods?

---

Published in Transactions on Machine Learning Research (11/2023)
Q3: How do the adaptive normalization and policy distillation mechanisms affect the continual RL performance, respectively?
Q4: How is RECALL's scalability regarding longer task sequences?
5.1 Experimental Settings
Datasets We perform our experiments on the new Continual World benchmark (De Lange et al. 2021 designed as a testbed for evaluating RT, agents with respect to challenges incurred by the continual learning paradigm. It consists of ten realistic robotic manipulation tasks. The structure of the observation and action spaces remains the same between lasks, allowing for multi-lask learning with a single learning system. For all tasks, the robot must either manipulate one object with a variable goal position, or two objects with fixed goal position. The observation space is represented as a 12-dimensional vector containing the coordinates of the robot's gripper and relevant objects. The action space is 4-dimensional vector describing the gripper movement. Reward functions are shaped to make each task solvable and the binary success metric is used to indicate whether the desired goal has been successfully accomplished. The tasks are arranged in sequences and the training on cach task lasts for 1M steps. Continual World provides eight triplct sequences of three tasks to allow rapid experimenting, while a longer sequence contains 10 different tasks arranged in a fixed order (called CW10), and CW20 consists of CW10 repeated twice. Soe Appendix A for more details.
We evaluate our method in comparison to five standard baselines: (1) Fine-tuning is the vanilla continual learning baseline where the model is trained on sequential tasks without any concern of preventing forgetting or facilitating forward transfer.
Baselines
(2) EWC (Kirkpatrick et al., (017)) is classic regularization- based method that uses the Fisher information matrix to approximate the importance of each weight and apply quadratic regularization to network weights to reduce forgetting. (3) PackNet (Mallya & Lazebnik 2018) strictly prevents the performance from deteriorating on the previous tasks by iteratively pruning, retraining, and freezing parts of the network after each task. It is a parameter isolation method, showing good performance on Continual World (Wolczyk et al. 2021p. (4) ClonEr (Wolczyk et al. 2022) is another regularization--ene based method combining behavioral cloning and best-return exploration, which demonstrates the best average performance and forward transfer on Cont World. (5) Perfect Memory (Wolezyk el al.j 2022) is simple replay method primarily investigated in this paper which keeps all data from past tasks in the SAC's buffer to avoid forgetting. We abbreviate it to PM in our experimental results for simplicity.
Implementations We use an implementation of the underlying RL algorithm SAC (Haarnoja et al., 2018a b Zhou et al.) 2022) based on (Wolezyk et al.] 2021); in which the maximum entropy coellicient a is tuned automatically according to the adjustment rule provided in (Haarnoja et al. 2018b). We follow exactly the same experimental setup (including network structure and hyperparameters) from (Wolczyk et al. 2022) for all baselines and the common settings for RECALL, ensuring fair comparison. The actor and critic are implemented as two separate MLP networks, each with 4 hidden layers of 256 units and assuming the multi-head setting. The difference is that we keep the actor's single-layer head structure consistent with that in (Wolezyk et al. 22022) while designing the critic's output head with 3 hidden layers to avoid introducing too much bias in new tasks during the value function approximation process. The model was trained on each task for 1M steps, and performance was evaluated by testing the current policy on all tasks every 20k steps. The SAC exploration phase takes 10k steps. By default, employ the best-return exploration in RECALL that reuses old policy head to facilitate exploration when the new task begins, as used by ClonEx, as well as inherit the corresponding critic head for faster adaptation. For each task sequence, we search method-specific rogularization coofficient À for policy distillation of RECALL in {0.01, 0.1, 1, 10, 100), and the final selected value is 10. Replay bufler size is set to be consistent with that Perfect Memory and batch size 128. All experiments were conducted with 5 different seeds and we also provide 90% confidence intervals through bootstrapping. More details can be found in Appendix B
Metrics Following the convention in Wolczyk et al. 2021 , we use average performance, forgetting, and forward transfer across all tasks as the primary metries for evaluation. Specifically, assume (t) E [0, 1] as the success rate of task i at time I, and that each of the N tasks is trained for A steps, so that (1) the average
8

---

Published in Transactions on Machine Learning Research (11/2023)
Second Tank
Second Tack
Second Tank
1
7
0:94
0.92
0.91
0.90 93 0.92
0:91 0.93 0.93 0.91
0:97
9.76 9.94
1.0
0.68 0.92
0.8
0.96
1.0
0.05 0.060.06 0:01 0.05 0.00 0.060 0.400.0
0.04
1 0.GR 0.08
0.70
5.0
0.82
3.65 0.77
0.73
0.60
0.91
9.
1 0.07
077-0.09-00000
08 03-0
130
0.03
0.
0.56
0:36
0.36
0.02
0.07
0.000.00000
0.99 1.0 0.59 0.99
0.96
0.94
0:98
1.0
1.0 0.97
00.79
0:69
0:97
1.0
0.78
06
0:85
0.8
0.96
3.0
0.02
0.02
0.0
0.020.03
0.00
0.04
0.09
0.01
0.0
0.01
-0.6
-06
0.6
Froe Tue
p:75 0.75 0.87
0.00
0.73 0.06
0.88
0.52
9.93
0.08
5.92
00.34
0:03
-3380 0-12--00000
Frnt
4 0.8 0.54 0.01 0.94
088 0.92 5.88
0.0 0.80
00.93
0.75 0.53
2.0
0.77 0.07
0.929.38
0.06
0.93
0.04 0.03 0.02
0.040.0
0.
0 0.000.00000000 0.0.0.02..01
7 0.8 0.67 0.02 0.77
0.91
0.43
0.66
0.02 0.83
0..00.00--0 01
5.06 -01
0.00
-22
9 0.99 0.99 1. 0.99 0.99 0.99 3.0 1.0 9.59
1.0
0.78 0.4
9.52
1.0 0.49 0.58 0.79 0.96
3.0
0.01 0.01 0.0 0.02 0.01 0.0 0.0 0.0 0.01
0.0
9.0
0.0
0.0
(a) First task performance
(b) Second task performance
(c) First task forgetting
The evaluation matrix for RECALL pairwise sequential tasks from Continual World. The average values of (a), (b), and (c) are 0.91, 0.85 and -0.02, respectively. It is clear that RECALL considerably enhances the adaptability of the model for new tasks, and also performs well on eliminating mild forgetting, in comparison with Perfect Memory shown in Figure
Figure 4:
performance at time l is P(()= N Σ Pe((); (2) the forgelling metric is measured by the average difference between the performance after training on each task versus the performance at the end of training on all tasks, denoted as F = & Ei p.(i- A) P:NNN A); (3) the forward transfer for all task is FT = N Σi=1 FT, where FT, is the forward transfer of task i, defined as a normalized area between its training curve AUC, and the reference training curve AUC; from training from scratch, i.e., FT; AUC;-AUCH AUC; 1/ A 1)- pi pi())lt,
I-AUC,
AUC;- 2 (t(d,, and (t) [0, 1] is the reference performance.
5.2 Plasticity and Stability
Our first experiment was designed to demonstrate the effi- cacy of RECALL on facilitating plasticity on new tasks as well as reducing forgetting from offline learning (Q1). We
1.0
Current Task Success
0.8-
apply RECALL to 100 pairs of sequential tasks used in the precoding preliminary experiments and summarize the results
Fiss
Second Task
0.4
in Figure g Our method effectively promotes the learning on second tasks, while eliminating mild forgetting caused by offline learning (see Perfect Memory in Figure for ref- erence). More preciscly, RECALL reduces the number of second lasks with success rate less than 0.5 to from 56 for Perfect Memory, whilst achieving the dropoff in success rate of less than 0.1 on all 100 first tasks (87 for Perfect Memory). Accordingly, an example of the current task's training curve of RECALL is provided in Figure 5 When the task switches, the agent can quickly adapt to the new environment, showing significantly better plasticity for new tasks than Perfect Memory.
0.2
0.0
o's
1.0 Steps (x10*)
a/s
2.0
Figure 5: An example of the learning curve of RECALL showing good plasticity.
The fact that RECALL can achieve plasticity and stability simultaneously appears to go against the conventional wisdom about the plasticity-stability trade-off, which maintains that the plasticity of artificial and biological neural systems is improved at the expense of stability, whereas too much stability will in turn impede the efficient learning of new knowledge. We argue that the aforementioned perception is fundamentally based on the premise that the capacity of the neural system is fully and well exploited. That is, additional factors affect the model's performance except for the plasticity-stability dilemma. However, the issue of limited plasticity discussed in this study is caused by the magnitude of rewards rather than excessive attention to stability. Likewise, the mild forgetting that we alleviate is not brought on by too much focus on plasticity, but rather by the offline learning for historical tasks. As a result, it is feasible to address these two parallel issues at the same time to encourage the dual enhancement of plasticity and stability, which is also supported by the results presented in Figure
9

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: Average performance, forgetting, and forward transfer of all the methods on the triplet sequences (CW3). Here and in related tables, the 90% confidence intervals are provided through bootstrapping. The best possible performance (confidence intervals are considered) for each task is marked in boldface.
Average Performance
CW3 Fine-tuning
EWC
PackNet
ClonEx
PM
RECALL
1
0.30 [0.29, 0.31]
0.71 [0.56, 0.85]
0.70 [0.60, 0.82]
0.84 [0.77, 0.91]
0.52 [0.40, 0.59]
0.90 [0.88, 0.92]
2
0.31
[0:29, 0.33]
0.58 [0.54, 0.62]
0.86 [0.83, 0.89)
0.90 [0.79, 0.96]
0.74 [0.65, 0.83]
0.92 [0.80, 0.94
3
0.24 [0.22, 0.26]
0.42 [0.30, 0.51]
0.61 [0.53, 0.69)
0.73 [0.64, 0.81]
0.26 [0.15, 0.36)
0.91 [0.90, 0.93
4
0.33 [0.32, 0.33]
0.75 [0.62, 0.89]
0.53 [0.43, 0.62]
0.88 [0.83, 0.933
0.28 [0.24, 0.32]
0.87 [0.84, 0.89
5
0.33 [0.33, 0.33]
0.54 [0.45, 0.62]
0.74 [0.65, 0.84]
0.89 [0.79, 0.97]
0.35 [0.28, 0.466
0.92 [0.90, 0.94
6
0.27 [0.24, 0.29]
0.82 [0.74, 0.89)
0.40 [0.32, 0.49)
0.74 [0.69, 0.80]
0.33 [0.25, 0.45]
0.91 [0.89, 0.93
7
0.33
[0.33, 0.33]
0.80 [0.68, 0.92]
0.91 [0.86, 0.95]
0.90 [0.81, 0.99]
0.90 [0.78, 0.97]
0.95 [0.93, 0.96
8
0.33
[0.33, 0.33]
0.41 [0.34, 0.52]
0.81 [0.67, 0.93]
0.81 (0.62, 0.93]
0.50 [o.10, 0.61]
0.95 [0.93, 0.97]
mcan
0.31
0.63
0.70
0.84
0.49
0.92
Forgetting
CW3
Fine-tuning
EWC
PackNet
ClonEx
PM
RECALL
1
0.59 [0.58, 0.61]
0.10 [-0.02, 0.22]
0.02 Jo.00, 0.04] 0.02 J-0.01, 0.07]
0.02 [0.01, 0.04]
-0.01 [-0.03, 0.02]
2
0.53 [0.51, 0.55]
0.26 [0.23, 0.29]
-0.05 [-0.09, 0.00]
-0.04 1-0.08, -0.01]
0.04 [-0.04, 0.14]
0.00 [-0.02, 0.02]
3
0.61 [0.59, 0.63]
0.24 [0.21, 0.27]
-0.06 [-0.13, 0.01]
0.01
[-0.06, 0.09]
-0.02 (-0.07, 0.02]
-0.04 [-0.06, -0.01)
4
0.53 [0.51, 0.54]
0..03 [-0.10, 0.07]
-0.06 (-0.09, -0.02]
-0.03 [-0.09, 0.04]
0.02 [-0.01, 0.05)
0.01 [-0.01, 0.03]
5
0.55 [0.49, 0.59)
0.03 [-0.01, 0.07)
0.01 [-0.05, 0.07]
-0.01 [-0.05, 0.03)
0.01 [o.00, 0.02]
-0.03 [-0.05, -0.02]
6
0.58 [0.56, 0.60]
0.04 [-0.03, 0.12)
0.02 [o.01, 0.03]
0.05 [-0.02, 0.13]
-0.01 (-0.03, 0.01]
-0.03 [-0.05, 0.00]
7
0.55 [0.51, 0.58]
0.13 [0.00, 0.27]
0.02 [-0.01, 0.05]
0.04
[-0.04, 0.12]
-0.01
[-0.03, 0.00]
-0.01 [-0.03, 0.01]
8
0.57 [0.53, 0.61]
0.16 [0.04, 0.26]
0.02 [-0.04, 0.08)
0.05 [-0.01, 0.12] 0.00 [-0.02, 0.02)
0.06 [-0.09, -0.03
mean
0.56
0.12
-0.01
0.01
0.01
-0.02
Forward Transfer
CW3
Fine-tuning
EWC
PackNet
ClonEx
PM
RECALL
1
0.14 Jo.05, 0.21]
0.10 [-0.24, 0.02]
-0.23 [-0.47, 0.03]
0.32 [0.24, 0.40]
0.96
|-1.32, -D.62]
0.35 [0.29, 0.41]
2
0.22 [0.11, 0.32]
0.03 [-0.13, 0.19]
-0.10 [-0.22, 0.01]
0.32 (0.03, 0.52]
-0.09
[-0.25, 0.09]
0.47 [0.44, 0.50
3
0.33 [0.29, 0.38]
0.03 [-0.19, 0.16]
0.02 [-0.19, 0.17]
0.31 [0.16, 0.42]
-0.57
[-0.68, -0.17]
0.49 [0.45, 0.53]
4
0.40 [0.36, 0.44]
0.26 [o.21, 0.29]
-0.13 [-0.34, 0.08]
0.41 [0.23, 0.53]
-0.29
[+0.37, -D.21]
0.45 (0.41, 0.48
5
0.51 [0.42, 0.59]
0.12 [-0.08, 0.31]
0.23 [0.15, 0.31]
0.48 [0.32, 0.62]
-0.19
[-0.33, -0.07]
0.52 (0.46, 0.57]
6
0.31 [0.18, 0.42)
0.19 [-0.02, 0.35)
-0.15 [-0.48, 0.08)
0.41 [0.26, 0.53]
-0.70
[-1.02, -0.42]
0.55 (0.52, 0.58
7
0.32
[0.27, 0.38)
0.28
[0.20, 0.36]
0.08
[-0.01, 0.17]
0.56 [0.50, 0.63]
-0.10
(-0.43, 0.14]
0.55 (0.48, 0.62]
8
0.47
[0.39, 0.54]
0.09 [-0.42, 0.16]
-0.01 [-0.50, 0.38)
0.30 [-0.81, 0.17]
00.44
1.0.78, -0.14]
0.52 (0.47, 0.55]
mean
0.34
0.09
-0.03
0.39
-0.42
0.49
5.3 Performance Evaluation
Here systematically perform quantitative evaluation of RECALL against the five standard baseline methods (Finc-tuning, EWC, PackNet, ClonEx, and PM) (Q2). We apply them to eight triplcts (referred to as CW3) and their twice repeated version (referred to as CW6) for fast experimenting and summarized the results in Table and Table 2 The networks used in all task sequences are exactly the same. From the results, we find that RECALL obtained slightly better overall performance than ClonEx, the statc-of-tho-a method, and significantly better performance than the other four baselines, across all three metrics of average performance, forgetting, and forward transfer.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: Average performance, forgetting, and forward transfer of all the methods on CW6.
Average Performance
CW6
Fine-tuning
EWC
PackNet
ClonEx
PM
RECALL
1
0.10 [0.06, u.14]
0.71 [0.57, 0.84]
0.79 [0.71, 0.87]
0.87 [0.81, 0.92]
0.47 [0.44, 0.30]
0.95 [0.94, 0.95
2
0.16 [0.16, 0.17]
0.59 [0.41, 0.74]
0.80 [0.71, 0.86]
0.90 [0.85, 0.95]
0.50 [0.15, 0.35]
0.98 [0.97, 0.99
3
0.11 [0.09, 0.12]
0.61 [0.57, 0.05)
0.50 [0.42, 0.59]
0.81 [0.75, 0.85]
0.14 [0.13, 0.14]
0.87 (0.86, 0.88]
4
0.17 [0.17, 0.17]
0.56 [0.53; 0.58)
0.86 [0.82, 0.89)
0.85 [0.81, 0.88]
0.17 [0.13, 0.21]
0.89 [0.86, 0.90
5
0.17 [0.17, 0.17]
0.42 [0.33, 0.52]
0.75 [0.61, 0.87]
0.91 [0.86, 0.96]
0.32 [0.29, 0.37]
0.97 (0.97, 0.98)
6
0.13
[0.13, 0.14]
0.75 [0.65; 0.86]
0.64 [0.57, 0.71]
0.74 [0.70, 0.79]
0.28 [0.17. 17, 0.39)
0.95 [0.94, 0.97]
7
0.17
[0.17. 0.17]
0.96 [0.95, 0.96]
0.87 [0.79, 0.93]
0.96 [0.94, 0.98]
0.85 [0.75, 0.95]
0.97 [0.95, 0.99
8
0.17
[0.17, 0.18)
0.51 [0.43, 0.59)
0.82 [0.67, 0.96]
0.97 [0.96, 0.98)
0.64 [0.61, 0.65]
0.95 [0.92, 0.97
mean
0.15
0.64
0.75
0.88
0.42
0.94
Forgetting
CW6
Fine-tuning
EWC
PackNet
ClonEx
PM
RECALL
1
0.71 [0.67, 0.75)
0.07 [0.00, 0.13)
0.00 |-0.2, 0.02]
0.02 [-0.01, 0.05]
0.01 [-0.02, 0.04]
-0.05
[-0.06, -0.04]
2
0.73 [0.71, 0.75]
0.20 [0.11, 0.29]
0.00 1-0.22, 0.01]
0.02 [-0.02, 0.06]
0.05 [0.00, 0.11]
-0.05
[-0.06, -0.03
3
0.70 [0.68, 0.72]
0.02 [-0.05, 0.08]
-0.03 [-0.07, -0.01]
0.04 [0.01, 0.08]
0.00 [-0.01, 0.01]
-0.05
[-0.07, -0.04]
4
0.59 [0.54, 0.64]
0.05 j0.03, 0.06]
0..04 |-0.07, 0.00]
0,02 [-0.01, 0.06]
-
-0.02 [-0.05, 0.01]
-0.04 [-0.06, -0.02)
5
0.74 [0.70, 0.77]
0.01 [-0.01, 0.02]
-0.04
[-0.08, -0.01]
0.03 [-0.02, 0.08)
-0.01 [-0.04, 0.02]
-0.04 [-0.05, -0.04)
6
0.68 [0.62, 0.74]
0.01 [-0.08, 0.06]
0.01 |-0.03, 0.00]
0.08 [0.02, 0.15]
-0.02 [-0.05, 0.01]
-0.03 [-0.03, -0.02)
7
0.75 (0.73, 0.78]
0.06 [-0.09, -0.03]
0.03
|-0.08, 0.02]
-0.01
(-0.03, 0.01]
0.07 [0.00, 0.15]
-0.01 [-0.03, 0.00]
8
0.71
[0.64, 0.76]
0.05 [-0.01, 0.11] 0.03
|-0.06, 0.00] -0.01
[-0.03, 0.01]
0.01 [o.00, 0.02]
-0.05 (-0.06, -0.03]
mean
0.70
0.04
-0.02
0.02
0.01
-0.04
Forward Transfer
CW6
Fine-tuning
EWC
PackNet
ClonEx
PM
RECALL
1
0.00 110.11, 0.11]
0.02 [-0.18, 0.14]
0.09 1-0.19, 0.00]
0.34
[0.22, 0.45]
0.92
[-1.04, -0.82]
0.36 [0.32, 0.40
2
0.26 (0.19, 0.33]
0.02 [-0.27, 0.17]
0.09
[-0.20, 0.02]
0.51
[0.45, 0.57]
-0.5
[-0.90, -0.42]
0.55 [0.50, 0.59
3
0.24 [o.18, 0.29]
0.11 [-0.26, 0.03]
0.08 1-0.19, 0.03]
0.46
[o.43, 0.49]
-0.66
[-0.74, -D.60]
0.48 [0.45, 0.50
4
0.28 Jo.25, 0.31)
0.13 j0.05, 0.20]
0.41 (0.36, 0.45]
0.54
[0.51, 0.57]
-0.49
[-0.57, -0.42]
0.53 [0.50, 0.56
5
0.42 [0.30, 0.53)
0.12 [-0.28, 0.05]
0.18 (0.02, 0.34]
0.67 [0.61, 0.72]
-0.25
|.0.34, -D.18)
0.67 [0.64, 0.70
6
0.39 [0.32, 0.45]
0.18 [-0.03, 0.34]
0.02 (-0.15, 0.17]
0.33 [o.15, 0.51]
-0.73 |-1.07, -D.42]
0.65 [0.59, 0.70
7
0.45 [0.41, 0.48]
0.25 (0.12, 0.37]
0.27 [-0.59, 0.06]
0.55 [0.48, 0.62]
-0.05
(-0.30, 0.19]
0.64 [0.59, 0.68
8
0.43 [0.35, 0.50]
0..0 (-0.42, -0.02]
0.05 [-0.38, 0.42]
0.68 [0.62, 0.73]
0.11 [o.02, 0.18]
0.61 [0.56, 0.66]
mean
0.31
0.01
0.02
0.51
-0.46
0.56
It is worth noting that the fundamental reason that RECALL outperforms ClonEx is that they use completely different mechanisms to alleviate catastrophic forgetting. To be specific, ClonEx is a regularization-bass approach that reduces forgetting by adding regularization term to constrain updates of network weights. In general, if the network capacity is adequate, the optimal outcome that can be attained by this mechanism is to entirely preserve the performance on previous tasks and achieve zero forgetting. It rarely obtains positive backward transfer unless the solution space of subsequent tasks includes that of historical tasks. According to the experimental results, it generally exhibits some level of forgetting on most task sequences due to the requirement to ensure plasticity on following tasks, which is particularly apparent in long task sequences.
By contrast, to avoid catastrophic forgetting, RECALL maintains the training on past tasks by replaying experiences while learning new tasks. If the agent does not reach optimal performance at the end of the
11

---

Published in Transactions on Machine Learning Research (11/2023)
1,0
10
1,0
Average Performance
None
Average Performance
None
Average Parformace
None
0.8
TN
2.8
6.6
TN
PD
PD
PD
0.6
TN-PD
9.6
TN+PD
0.6
TN+
0.4
0.4
0.4
0.2
0.2
0.0
0.0
1.0
2.0
2.0
4.0
5.0
6.0
0.0
1.0
2.0
3.0
5.0
6.0
0.0
1.0
2.0
3.0
4.0
5.0
Stops (x10%)
Steps (x10*)
(x110))
(a) 1st task sequence
(b) 2nd task sequence
(c) 3rd task sequence
1.0
Average Performance
None
Average Perfomance Comannee
None
suerage Perfermance
None
0.8
TN
TN
PD
PD
TN-PD
TN+PD
0.4
0.0
1.0
2.0
3.0
4.0
5.0
0.0
1.0
2.0
6.0
0.0
1.0
2.0
3.0
5.0
Steps (x10%)
Steps (x10*)
Steps (x109)
(d) 4th task sequence
(e) 5th task sequence
(f) 6th task sequence
1.c
1.0
Average Performanc
0.8
TN
6.8
TN
PD
0.6
TN--PD
0.6
TN-FD
0.4
0.4
0.2
0.2
0.0
0.0
0.0
1.0
3.0
4.0
5.0
Steps (x10f)
Steps (x10f)
(8) 7th task sequence
(h) 8th task sequence
Figure 6: Average (over tasks) success rate per iteration of the four variants in CW6 task sequences.
respective training period of the old tasks, such experience replay (further training) is likely to make the agent perform better these tasks rather than just preventing catastrophic forgetting. Consequently, RECALL can allow for positive backward transfer (i.c., produce negative forgetting values and improve average performance). Furthermore, the combination of experiences from new and replayed tasks for joint training can aid the model in finding a better common solution space, improving the final performance on all tasks, and also facilitating faster learning of new tasks to achieve more positive forward transfer relative to regularization-ban based methods.
5.4 Ablation Study
In this section, we consider the individual effects of target normalization (TN) and policy distillation (PD) mechanisms (Q3). To this end, we conduct experiments by manipulating a single variable at a time for in-depth analysis. For each new task, the following four variants of the proposed method are applied for continual learning in the new environment: (1) None: Neither target normalization nor policy distillation mechanism is used, i.e., degenerating to the naive experience replay with best-return exploration. (2) TN: Only apply target normalization mechanism. (3) PD: Only apply policy distillation mechanism. (4) TN+PD: Both target normalization and policy distillation mechanisms are used, i.e., representing RECALL. The learning curves in terms of average performance per iteration of these four variants on CW6 task sequences are shown in Figure
First, the variants None and TN (or PD and TN+PD) are compared to identify how the target normalization mechanism affects continual learning performance. In all eight task sequences, the targets are normalized for each task throughout the whole training process, and the performance in terms of average success rate
12

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: Results of all the methods on CW10 and CW20 sequences. forgetting, and forward transfer (F. Transfer) are shown in columns.
Average performance (Ave. Perf.),
CW10
CW20
Method Ave. Perf.
Forgetting
F. Transfer
Ave. Perf.
Forgetting
F. Transfer
Fine-tuning 0.10 [0.10, 0.10]
0.74 [0.72, 0.76]
0.29 [0.25, 0.32]
0.05 [o.05, 0.05]
0.73 [0.69, 0.76]
0.20 [0.14, 0.26]
EWC 0.61 [0.50, 0.63]
0.06 [0.05, 0.08]
0.03 [-0.04, 0.09]
0.61 [0.55, 0.68)
0.02 [-0.01, 0.06]
-0.13 [-0.21, -0.04]
PackNet
0.87 [0.83, 0.91]
0. 04 [-0.06, -0.02]
0.29 [0.22, 0.35]
0.79 [0.75, 0.82]
-0.01 [-0.03, 0.00]
0.16 [0.08, 0.22]
ClonEx 0.85 [0.80, 0.90]
0.00 [-0.02, 0.03]
0.39 [0.36, 0.43]
0.82 [0.79, 0.86)
0.05 [0.04, 0.06]
0.39 [0.35, 0.42]
PM 0.26 [0.23, 0.28] 0.02 [-0.01, 0.06]
-
-1.23
[-1.37, -1.10
0.09 [0.03, 0.15
0.10 [0.04, 0.16]
-
11.36 [-1.44, -1:29]
RECALL 0.89 [0.56, 0.91] 0.03 [-0.04, -0.03] 0.44
[0.35, 0.43] 0.90 [0.87, 0.92] 0.04 [-0.05, -0.03]
0.42 [0.39, 0.44
is maintained or slightly improved for the first task (the first 1M steps) and dramatically enhanced for the subsequent tasks. It verifies that training a scale invariant value function (a normalized Q network) in replay-based continual RL leads to better adaptation for subsequent tasks, as stated in Section E
Next, the PD and TN variants are compared with None and TN+PD to verify the effectiveness of the policy distillation mechanism. Compared with the naive experience replay, distilling the policy from previous tasks when learning on the new task can achieve a certain degree of performance improvement sequences. Conversely, it shows an obvious performance degradation on other sequences, potentially because excessive policy distillation inhibits the learning of new tasks. By contrast, applying the policy distillation to the TN variant can consistently improve the performance of all task sequences. This observation confirms the assumption in Section 4 that applying a proper distillation of the policies from old tasks can help mitigate the mild forgetting caused by offline learning.
some task
Finally, all four variants are compared. It can be observed that the target normalization mechanism can improve learning performance better than policy distillation, and combining the two mechanisms together (TN+PD) leads to the best performance on those various task sequences.
5.5 Scalability
To evaluate the scalability of RECALL as well as how it compares with other baseline methods, we test them against longer task sequences CW10 and CW20 (Q4). As shown in Table 3 RECALL outperforms or matches the overall performance of all other baselines on CW10, and is consistently superior to them on CW20 except for the forward transfer which is comparable to ClonEx. One possible reason that PackNet can also perform well in reducing forgetting is that its total training time for each task is longer since after the initial network training, it undergoes iterative pruning, freezing and retraining parts of the network. Nonetheless, this advantage is rather minor and RECALL still surpasses Pack average performance on CW20 and is significantly better than it on all task sequences investigated in our experiments in terms of forward transfer. In addition, the performance gap becomes more evident on the longer task sequence CW20 where RECALL outperforms PackNet in all aspects along with the other methods. A possible factor is that PackNet struggles against the increasing complexity of managing shared and task-specific parameters as the number of tasks becomes large. More generally, we find that while all other methods face a considerable drop in performance as the task sequence length doubled from CW10 to CW20, RECALL experiences the opposite with improvements across all three aspects, albeit marginally. These results highlight the desirable scalability of RECALL, and its robustness in handling lengthy task sequences, rendering it promising solution for continual RL in complicated scenarios.
6 Conclusions
In this work, we present a systematic investigation of replay-based continual RL. Due to the potentially significant difference in scale of rewards across tasks contained in the same sequence, we observe that there exists serious limitation in the plasticity for subsequent tasks, which hinders the learning of new tasks and
13

---

Published in Transactions on Machine Learning Research (11/2023)
further limits the continual RL agent's final performance. To address this, propose RECALL to optimize scale invariant normalized value function by introducing adaptive normalizat ion mechanism on targets, that all new tasks will have a similar impact on the learning dynamics to that of the previously well-learned tasks, thus allowing the efficient learning of subsequent tasks. In addition, the policy distillation mechanism for old lasks is used lo further alleviate lorgetting caused by offline learning on the replayed lasks. Extensive experiments on a suite of realistic robotic manipulation task sequences show that RECALL significantly outperforms or matches state-of-the-art baselines in terms of average performance, forgetting, and forward transfer. Meanwhile, it demonstrates superior scalability on longer task sequences.
Note that the two mechanisms contained in RECALL can serve a plug-and-play component that can be effortlessly integrated into existing replay-based algorithms to improve their performance on continual RL tasks. We believe that this work constitutes the first step towards understanding the difference between experience replay in supervised continual learning and continual RL. Furthermore, provides a promising prospect for the adoption and extension of replay- based continual learning techniques in the RL context.
Acknowledgments
This work was supported by the STI 2030-Major Projects under Grant 20212D0201404 and Tencent Rhino- Bird Resoarch Elite Program.
References
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting. Newrocompuling, 428:291 307, 2021.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions Pattern Anolysis and Machine Intelligence, 44(7):3366 1--33666355,,, 2021.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. Preprint Xrx: (00.178018844 2017.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4): 128-135, 1999.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the International Conference on Machine Learning, pp. 1861-1870, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprinl ar Xiv: 182.. 05905, 2018b.
Mattco Hessel, Hubert Soyer, Lasse Espcholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1196,,803, 2019.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J Roberts. UNCLEAR: A straightforward method for continual reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kicran Milan, John Quan, Tiago Ramalho, Agnicszka Grabska- Grabskk-BBrrinskaa, et al. Overcoming catastrophic forgetting in neural networks. In Proceedings of the National Academy of Sciences, volume 114, pp. 3521-33 3521-3526, 2017.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is np-hard. In Proceedings of the International Conference on Machine Learning, pp. 5-27533753355 5337, 2020.
Lukasz Korycki and Bartosz Krawezyk. Class-increment incremental experience replay for continual learning under concept drift. In Proceedings of the Conference on Computer Vision and Pattern Recognation, pp. -355,, 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single net by iterative pruning. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. In Advances Neural Information Processing Systems,, volume 33, pp. 7308 7320, 2020.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Seivestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning, In Proceedings of the Conference on Computer Vision and Pottern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In Proceedings of the International Conference on Learning Representations, pp. 1-31, 2019.
Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. Experience replay for continual learning. In Proceedings of the Conference on Neural Information Processing Systems, 2019.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcchre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukeuoghu, and Raia Hadsell. Policy distillation. In Proceedings of the International Conference on Learning Representations, 2016a.
Andrei A RUsu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. Preprint arXiu: 2016b.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Newral Information Processing Systems, volume 30, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: on introduction. MIT Press, 2018
René Traoré, Hugo Caselles-Dupré, Timothée Lesort, Te Sun, Guanghang Cai, Natalia Rodd-Rorr,,,es, and David Filliat. Discorl: Continual reinforcement learning via policy distillation. In Advances in Newral Information Processing Systems Workshop, 2019.
Hado P van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. Learning values across many orders of magnitude. In Advances in Neural Information Processing Systems, volume 29, 2016.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. Preprint orXiv:2302.00454 2023.
Zhi Wang, Han-Xiong Li, and Chunlin Chen. Incremental reinforcement learning in continuous spaces via policy relaxation and importance weighting, IEEE Transactions on Newral Networks and Learning Systems, 1166)11880 2019.
Zhi Wang, Chunlin Chen, and Daoyi Dong. A dirichlet process mixture of robust task models for scalable lifelong reinforcement learning. IEFF. Transactions on Cybernetics, 2022.
15

---

The details of the Published in Transactions on Machine Learning Research (11/2023)
Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciúskii, and Piotr Miloš. Continual world: A robotic benchmark for continual reinforcement learning. In Advances in Newral Information Processing Systems, volume 34, pp. 28496 28510, 2021.
Maciej Wolezyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciníski, and Piotr Miloć. Disentangling transfer in continual reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pp- 6304-6317, 2022.
Tiantian Zhang, Xueqian Wang, Bin Liang, and Bo Yuan. Catastrophic interference in reinforcement learning: A solution based context division and knowledge distillation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Tiantian Zhang, Zichuan Lin, Yuxing Wang, Deheng Ye, Qiang Fu, Wei Yang, Xueqian Wang, Bin Liang, Bo Yuan, and Xiu Li. Dynamics-adaptive continual reinforcement learning via progressive contextualization. IEEE Transactions Neural Networks and Learning Systems, 2023.
Haibin Zhou, Zichuan Lin, Junyou Li, Qiang Fu, Wei Yang, and Deheng Ye. Revisiting discrete soft actor-critic. arViv preprint arXiv:2200.1081, 2022.
A Continual World Benchmark
We briefly present the ten robotic tasks from the Continual World benchmark in Figure 7 lask sequences CW3 used in this paper are as follows:
PUSH-V1 - WINDOW-CLOSE-VE HAMMER-V1
HAMMER V1 > HENDDWW-CLOS.... V1 > FAUCET-CLOSE-VI
3. STICK-PULL-V1 PUSH-BACK-V1 PUSH-WALL-V1
PUSH-WALL-V1 SHELF-PLACE-VA PUSH-BACK-VA
FAUCET-CLOSE-VA + SHELF-PLACE-V V1 PUSH-BACK-VI
6. STICK-PULL-VA > SLN-UNNLUUGSSDE---- STICK- PUILK-PULL-VI
WINDOW-CLOSE-V/ HANDLE-PRESS-
PEG-UNPLUG-SIDE-WV
8. FAUCET-CLOSE-VE + SHELF-PLACE- V1 FEG-UNPLUG-SIDE-I
CW6 is CW3 repeated twice. The CW10 sequence is:
HAMMER-V1 > PUSH-WALL-VI HANDLE-PRESS-SIDE- PEG-UNPLUG-SIDE-VV
FAUCET-CLOSE-VI + PUSH-BACK-V1 STICK-PULL-VI
PUSH-V1 7 SHELF-PLACE-V1 WINDOW-CLOSE--V
CW20 is CW10 repeated twice.
B Implementation Details
We use the same hyperparameters as (Wotcyyk et al. 2022) for the underlying SAC algorithm. Table 4 lists the numerical settings of some core parameters the experimental evaluation. For the method-specific hyperparameters involved in the continual learning baselines compared in our experiments, also inherit the final values obtained after tuning in (Wolezyk et al. 2022):
EWC: selected regularization coefficient for actor is 10, 000 and that for critic is 0.
16

---

Published in Transactions on Machine Learning Research (11/2023)
0 HAMMER-VI
1 WALL-VI
BAUCET-CLOSE-VE
PUSH-BACK-V1
4
5 SOINDLE-PRESS SIDE-VI
6 PUSH-VI
7
WINDOW-CLLOSE:
9 PEG-UNPLUG- SIDE-VI
VI
Figure 7: Ten robotic tasks adopted by Continual World benchmark.
Table 4: Core hyperparameters used for the underlying SAC algorithum
Parameter
Value
optimizer learning rate
Adam
1 x 10-3
batch size
128
discount factor (7)
0.99
nonlinearity
ReLU
target smoothing coefficient (r)
0.005
larget update interval
target output std (at)
0.089
replay bulfer size
10
PackNet: the number of retraining steps is set to 100, 000, and global gradient norm clipping is
2 x 10-5
ClonEx: selected regularization coefficient for actor is 100 and that for critic is 0. Episodic memory per task is set to 10, 000, and global gradient norm clipping is 0.1.
Perfect Memory: selected batch size is 512, and replay buller size is N x 10 where is the number of tasks the task sequence to learned.
C Additional Experimental Results
C.1 Plasticity Analysis of Perfect Memory
Figure shows the learning curves in terms of actor and critic losses and average predicted action value on the current task, corresponding to the task sequence described in Figure 2). It can be observed that within a new task (i.e., the second task), the loss functions of actor and critic
few time steps after the learning together with the output of the value function rapidly converge to a stable state that remains around 0, further indicating that the model does not learn any useful information about the new task.
We further investigate the correlation between the above failures and network sharing by experimenting with the actor and critic networks in the following four different combinations:
1) Shared actor and shaved critic: All tasks adopt the same shared policy network and value network for policy learning.
2) Unshared actor and shared critic: All tasks adopt the same shared value network but separate actor network for each task for policy learning.
17

---

Published in Transactions on Machine Learning Research (11/2023)
365
Average Action Value (Q)
0.0-
2.0-
Actor Loss (2c)
1.2
1.8-
Rist Task
Second Task
Loss
12-
First
Second Task
0.8
Rist Task
Second
-0.8-
0.8-
0.4
-1.--
0.4
0.0-
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
0.0
1.6
1.5
2.0
Steps (x109)
Steps (x109)
Steps (x10%)
(a) Loss Value Policy Network
(b) Loss Value of Q Network
(c) Average Predicted Action Value
Figure & Learning curves of Perfect Memory with poor plasticity on the current task in terms of actor and critic losses and average predicted action value (corresponding to the task sequence described in Figure 2
Table 5: Comparison of the final performance (success rate) each task obtained by Perfect Memory under different actor and critic networks settings (taking the paired task sequence described in Figure 2 as illustration). Here, "shared" means that two tasks share the corresponding network for policy learning, while "unshared" indicates that separate network is initialized for each task.
Network Setting
First Task
Second Task
shared actor, shared critic unshared actor, shared critic
0.96 j0.94, 0.97]
0.00 [0.00, 0.00]
0.96 (0.95, 0.97
0.00 [0.00, 0.00)
sharod actor, unshared critic
0.95 (0.93, 0.98
0.93 [0.90, 0.95]
unshared actor, unshared critic
0.96 (0.95, 0.98
0.94 [0.92, 0.97]
3) Shared actor and unshared critic: All tasks adopt the same shared policy network but separate critic network for each task for policy learning.
4) Unshared actor and unshared critic: Building separate policy network and a separate value network for each task for policy learning. It be regarded as an upper bound for continuous RL.
Similarly, taking the task sequence described in Figure 2 as example, the final performance each task is summarized in Table 5 It can be seen that the limited plasticity of the model on the new task is largely caused by the shared critic network.
C.2 Forgetting Analysis of Perfect Memory
To further investigate the forgetting performance of Perfect Memory, we visualize the learning curves of the first task the 13 pairs of task sequences with more than 0.1 forgetting. As seen from Figure g all the first lasks achieve relatively good performance after ending learning in their own environment. Nevertheless, as the second task learned, the performance of the first tasks declines significantly after maintaining for a period of time. That is, in the continual RL setting, forgetting may still occur for experience replay based methods. The magnitude of it is relatively small but negliggg,,,,,, there is still need to address it further.
We use the following two replay modes to conduct additional experiments of Perfect Memory on 13 task sequences with obvious forgetting in Figure ! to verify the influence of offline learning on this issuc.
Offline learning: The replay mode actually implemented in Perfect Memory. Replaying exporiences collected from past tasks while learning the new task, rather than interacting with the old environments again that is not allowed under continual learning paradigm.
Online learning: Gathers experiences for replayed tasks by interacting with the old environments again while learning the new task. Although this pattern goes against the principle that continual
18

---

Published in Transactions on Machine Learning Research (11/2023)
1.0
First Task Success
0.8
0.6
[1,3]
(3,6)
[8,0]
0.4
(1.4)
(4.3)
(8,2)
12.21
(7.1)
(8.4)
0.2
12.31
(7.9)
(8.6)
(2.8)
0.0
0.0
1.0
1.5
2.0
Steps (x10f)
Figure 9: Success rate of the first task for Perfect Memory the 13 pairs of task sequences with more than 0.1 forgetting. Each curve corresponds to specific pairwise task sequence. The gray vertical dashed line represents the boundary of task switching.
Table 6: Performance and forgetting of Perfect Memory on the first task of 13 pairwise task sequences under different replay modes.
Offline Learning
Online Learning
Task Sequence
Performance
Forgetting
Performance
Forgetting
(1,3)
0.62 [0.60, 0.64]
0.14 [0.09, 0.19)
0.90 [0.90, 0.91]
-0.10 [-0.11, -0.08)
(1,4)
0.57 [0.54, 0.59]
0.13 [0.10, 0.15]
0.76 j0.73, 0.80
0.00 (-0.02, 0.01]
(2,2)
0.84 [0.79, 0.88]
0.11 [0.09, 0.12]
0.96 [0.95, 0.97]
0.00 [-0.01, 0.00]
(2,3)
0.86 [0.84, 0.89]
0.10 [0.08, 0.13]
0.95 [0.95, 0.97]
-0.04 [-0.06,
(2,8)
0.85 [0.80, 0.90]
0.11 [0.09, 0.12]
0.98 [0.97, 0.98
0.01 [-0.03, 0.03]
(3,6)
0.78 [0.72, 0.85]
0.11 [0.08, 0.12]
0.98 [0.96, 0.98]
-0.03 [-0.05, -0.01]
(4.3)
0.58 [o.55, 0.63]
0.32 [0.23, 0.39]
0.90 [0.88, 0.93
-..15 J-0.19, -0.12]
(7,1)
0.82 [0.79, 0.84]
0.10 [0.07, 0.12]
0.87 [0.85, 0.88)
-0.02 [-0.05, 0.01]
(7.9)
0.66 [0.63, 0.68]
0.24 [0.21, 0.28]
0.90 [0.88, 0.98]
-0.04 [-0.05, -0.02]
(8.0)
0.85 [0.81, 0.87]
0.11 [0.07, 0.16]
0.98 [0.96, 0.98
-0.08 [-0.12, -0.05]
(8,2) 0.75 [0.72, 0.79]
0.19 [0.15, 0.23]
0.94 [0.93, 0.96
-0.07 [-0.09, -0.04]
(8,4) 0.80 [0.80, 0.81]
0.12 [0.09, 0.16]
0..0 [0.87, 0.92]
0.02 [o.00, 0.04]
(8,6)
0.77 (0.75, 0.78)
0.15 [0.11, 0.18)
0.91 [0.85, 0.93]
-0.03 10.05,-0.01)
RL agents can only access the current environment any time period, is more compatible with the online learning required by the majority of RL algorithms.
We calculate the final performance and forgetting for the first task in all task sequences at the end of the whole training, and the results are summarized in Table p It can be seen that Perfect Memory can prevent forgetting well the sequence of RL tasks after modifying the replay mode from offline to online. Therefore, can conclude that the forgetting of Perfect Memory discussed above is primarily caused by the offline learning of the replayed lasks.
C.3 Parameter Analysis
We vary the coefficient for policy distillation in RECALL and measure its impact the three evaluation metrics: average performance, forgetting, and forward transfer. We run experiments on the first task sequence of CW6. The results are presented in Table indicating that appropriate policy distillation of the actor can significantly improve performance.
It worth noting that we exclusively conduct policy distillation for actor while not for critic in RECALL. This is because, regardless of critic, the agent interacts with the environment collect data just by carrying out the policy from actor, and the primary goal of our policy distillation mechanism is to reduce distributional
19

---

Published in Transactions on Machine Learning Research (11/2023)
Table Average performance, forgetting, and forward transfer metrics on the first task sequence of CW6 for RECALL, for different values of the policy distillation regularization coefficient A.
Regularization Coefficient
Ave. Perf.
Forgetting
F. Transfer
0.01
0.84 [0.82, 0.86]
0.02 [0.01, 0.04
0.37 [0.34, 0.39)
0.1
0.84 [0.82, 0.87]
0.01 [0.00, 0.02]
0.39 [0.38, 0.40)
0.87 [0.86, 0.88)
-0.01 [-0.01, 0.00]
0.34 [0.33, 0.36)
10
0.95 [0.94, 0.95]
-0.05 1-0.06, -0.04]
0.36 [0.32, 0.40)
100
0.88 [0.86, 0.89)
0.05 [-0.06, -0.03]
0.18 [0.16, 0.20)
shift between the dataset and the learned policies caused by offline learning old tasks, that is, the deviation between the replayed experience distribution of old tasks and their action policies.
C.4 Performance Curves results
We also provide the average performance curves in CW6 task soquences for RECALL and basclincs, as shown in Figure
10
Average Perforannce
Ane- Ane-tuning
FWC
Perfomance
FWC
Rne-tu nin-tuning
Average Performanes
Rnn tuning
Pockdict
Packlett
Pacidict
ClonEx
ClanEx
RM
PM
RECALL
Average
RECALL
6.0
0.0
1.0
2.0
3.0
5.0
6.0
0.0
1.0
2.0
5.0
6.0
0.0
1.0
2.0
3.0
5.0
Steps (x10%)
Steps (x10*)
Steps (x10%)
(a) 1st task sequence
(b) 2nd task sequence
(c) 3rd task sequence
Average Performance
0.8
Finettuning
EWC
Perfomanna
0.8
EWC
Fine-tuning
Average Performance
0.6
EWC
Rne-tuning
PackNet
0.6-
ClonEx
0.6
ClonEx
0.6-
ClonEx
PM
0.4
RECALL
Average
0.4
RECALL
0.4
RECALL
0.2
0.2
6.2
0.0
3.0
0.0
4.0
Steps (x10%)
Steps
Steps (x10f)
(d) 4th task sequence
(c) 5th task sequence
(f) 6th task sequence
1.0
1.0
Fine tuning
Average Porforaacce
Ane tuning
0.8
0.8
PackNet
Packieet
9.6
ClonEx
ClonEx
RM
PM
0.4
6.0
1.0
2.0
3.0
4.0
5.0
6.0
6.0
1.0
2.0
3.0
4.0
5.0
6.
Steps (x10
Steps (x10%)
(g) task sequence
(h) 8th task sequence
Figure 10: Average (over tasks) success rate for all methods in CW6 task sequences.
20

---

Published in Transactions on Machine Learning Research (11/2023)
Table 8: Summary of the additional computation steps required by all methods in addition to the original SAC training computation. s_" indicates that the corresponding method does not involve that step.
Method
Regularization Term
Additional Training
Fine-tuning
EWC
weight consolidation
PackNet
retraining parts of the network after pruning
ClonEx
behavioural cloning
Perfect Memory
RECALL
policy distillation
PopArt parameters updates
Table Statistics of the data storage the training process of all methods, where is the number of tasks in the task sequence to be learned. 4_" indicates that the corresponding method does not involve that term.
Method
Deww size
Dold size
Fine-tuning
EWC
PackNet
x 106
ClonEx
N x 10a
Perfect Memory
(N - 1) X 106
RECALL
(N 1) x 10c
D Computational Efficiency and Data storage
To help readers further understand the learning process of RECALL and all basclines, summarized the major additional computation steps involved in all methods in addition to the original SAC training computation as well as the comparison of replay buffer sizes required by each method. The results are shown in Tables and g respectively.
Fine-Luning and Perfect Memory are the two simplest methods, and both require exactly the same amount of computation as the original SAC training. The key difference between them is that Perfect Memory needs store data of historical tasks and replay them while learning new tasks, while FT only stores and learns current task experiences at any time. EWC and ClonEx are regularization-bast involves Fisher information matrix calculation for estimating the importance of neural network weights, and ClonEx needs to store historical data for behavioral cloning. In contrast, PackNet, a parameter isolation method, gets rid of the above constraints of computing regularization terms and storing historical experiences, but it introduces additional retraining of parts of the network at task change.
methods, wherein EWC
Our proposed method, RECALL, is a method that combines experience replay and regularization constraints. Compared Perfect Memory, although introduces additional computation related to policy distillation and target normalization, the resulting performance improvement is quite significant. Compared to regularization- based methods such as EWC and ClonEx, RECALL introduces additional paramcters update about the target normalization. Nevertheless, the nunber of parameters in this part is relatively small compared to the terms of historical data storage,
massive weights in the actor and critic networks common in all methods. as discussed in our related work in the main text, there have been a lot of studies that show that preserving small amount of selective experiences or reconstructing observations through generative model as an alternative is also sufficient.
21


Reviews:
[{"review_id": "RBgLn3KLd9", "reviewer": "Reviewer_FYXr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe primary strength of the paper is performance of the method. The results consistently show that the proposed method has strong performance on the Continual World benchmark. \n\n## Weaknesses\n\nThere are several limitations that should be addressed. First of all, we note that the method is a relatively simple combination of two established techniques (and the impact of policy distillation is fairly small). While the paper still has value, the methodological contribution of the paper is fairly weak. Other points:\n\n- The authors do not meaningfully engage with the interaction between Popart/multi-task learning and SAC; in particular, different systems often require different target entropies or alpha coefficients for good performance. Meanwhile, the effective weight of the entropy in the loss function varies as both alpha and sigma vary. Perhaps the updates for alpha and sigma could be coupled to improve efficiency? In general, the investigation of this element of the methodology is non-existent and may potentially improve performance and strengthen the contribution of the paper. \n- Have the authors considered adding the distillation loss to the SAC loss in place of the entropy term, corresponding to a history-based prior as opposed to a uniform prior? This is common in, for example, modern RLHF. \n- The experiments are limited in the set of environments investigated. In particular, the authors only evaluation manipulation tasks, which may moderate the impact of varying entropy weights as discussed above. It would substantially strengthen the paper to include other evaluation settings. \n- There are two potential limitations induced by the proposed changes. First, the Popart normalization may harm efficiency for small amounts of per-task data. Second, the distillation substantially increases training and data storage cost compared to (especially regularization-based) baselines. Ideally, metrics should be presented that report performance as a function of data and compute used. Currently, per-task data is fixed in experiments and the only discussion of compute requirements is the limited discussion in Section 5.5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of continual RL, in which RL tasks are sequentially presented. The method presented (\"RECALL\") proposes two changes: Popart-style per-task normalization and policy distillation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The below changes are listed in order of decreasing importance (ie first item most important).\n\n- Address the interaction of SAC with Popart\n- Add further experimental settings beyond robot manipulation. \n- Add metrics that take into account compute and data storage expenses.", "broader_impact_concerns": "No major concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HdouOJq0Bu", "reviewer": "Reviewer_6uKu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "At a first glance, the method itself is not particularly novel as it combines two methods that address orthogonal issues. However, it effectively addresses issues raised in prior work and achieves good results with simple methods. The evaluation is generally extensive enough to support the most important claims. As a result, the paper should be of interest to the lifelong RL community.\n\nStrengths:\n+ The paper is well written and easy to read.\n+ The proposed method is effective and tested on a previously established benchmark while also being simple and straightforward to apply.\n\nWeaknesses:\n+ One key strength of replay based approaches to continual learning is that they work well even when the task boundaries are unknown or + the dynamics are transitioning in a smooth way. The same is not true for pop-art, reducing the applicability of this method. The authors  claim that this method can be a drop-in replacement for replay based approaches but this claim should be moderated.\n+ The authors show that in the case of this benchmark, the perceived lack of plasticity is actually due to issues of scaling. Advantages go up as the agent observes more reward and the scale of values increases and only drop later in training as the agent converges (the authors imply that they are highest for solved tasks but while this is possible, it should not be the case when the capacity of the network is sufficient); however, this is a phenomenon that is closely linked to RL algorithms while plasticity issues have been observed in supervised settings as well. In that case, the magnitude of the loss would be highest when the task is new and a similar normalization procedure would likely not work. The work could do more to identify when plasticity issues are due to scaling issues and when they are due to other reasons.\n+ I’m not convinced that the claim of the effect of off-policyness is backed up by evidence in this work. The distillation loss can also be seen as a regularizer, similar to other regularizers in the continual learning literature and may help with plasticity more generally rather than simply addressing off-policyness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a method for lifelong reinforcement learning which combines a large replay buffer with pop-art normalization. The main contribution of this paper is to show that plasticity issues that have been pointed out to hold back replay methods in prior work can be explained by loss normalization issues and addressed with corresponding methods that have previously been proposed in the multi-task reinforcement learning literature. The authors evaluate the method on a recently proposed benchmark of continual robotics tasks and compare against similar methods as have been tested on that benchmark before. The authors furthermore evaluate an additional regularization term, constraining the KL divergence to prior policies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "+ Please add additional discussion regarding the limitations of the multi-head approach as well as how this work relates to plasticity issues observed in supervised learning.\n+ If possible, please add value/loss/gradient magnitude or similar curves to illustrate the problem with perfect memory.\n+ Is there supporting evidence that can be added that the issues ascribed to off-policyness are indeed due to that reason?\n+ Could you clarify how much of the perceived plasticity issues with the baseline approach lies in the critic compared to the policy? The suggested solution suggests this is mainly an issue of the critic but showing this would strengthen the claim. Reinitializing the corresponding networks between tasks may show this.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d1TpLNexyb", "reviewer": "Reviewer_qBJi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is well written and the method is well tested empirically in continual world. The method is also, mostly, well discussed and presented with only a few lingering questions (see the weaknesses). While I have many more weaknesses than strengths, they are mostly effort the authors need to take to clarify some aspects of their work or better motivate their work in clearer terms.\n\n\nW1. While the main motivation of the paper is rooted in continual learning, many limitations to the method are apparent in the case of continual (lifelong) RL as defined by (Khetrapal et al., 2022). Instead, the focus of this work seems to more aptly follow from multi-task reinforcement learning. This distinction should be made clear, especially because of a lack of discussion around making this method work for continual (lifelong) RL. \n\nW2. For the distill loss, it is unclear where $\\pi_{old}$ comes from. Is this data stored along with the experience, or is it re-computed in some way? This should be clarified.\n\nW3. While this method is introduced in isolation, it seems like several innovations from ClonEx are employed (as discussed in section 5.1 in the implementations details). While I appreciate the details, I feel as though these should be included in your method specification and possibly as part of the ablation study (if that is in your computational budget). With this weakness, I find myself uncertain what algorithms is being run labeled *None* in the ablation study. Is it closer to ClonEx, or something else?\n\nW4. Do all the baselines sample from all the data generated throughout the agent's lifetime? If not, this should be clarified. In any case, you might want to highlight that this is a limitation of all the methods you propose, as it is a significant limitation of this (and any \"perfect memory\" approach) approach for C(LL)RL. \n\nW5. You do not include how hyperparameters were selected, or the empirical analysis of the regularization coefficient for policy distillation in the main paper. While these results are in the appendix, I think they should be discussed in the main paper. At the very least there should be a statement that you are using the hyperparameters from (Wolczyk et al., 2022)\n\nW6. You should use (Khetrapal et al., 2022) to relate your work to the reset of CRL. There are lots of definitions pervasive, and any new work should better understand which specific problem they are solving, not letting the motivation of the general C(LL)RL encroach into different, more specific problems (at least not unknowingly).\n\n**Other misc Questions:**\n- Q1. In figure 1(b), why do you think there are poor results across the diagonal? Is this to do with the fact we are learning a new policy head and the initialization may not suite the current representation of the network?\n\nEdits:\n- I think the caption for Figure 1 is slightly confusing because of the use of the proper task titles (\"representing the learning of task sequence M=[PUSH-V1, HAMMER-v1]\").", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The primary contribution of this paper lays at what they call RECALL, which is a set of modifications to using replay as a means to overcome the plasticity/catastrophic forgetting issues faced in continual reinforcement learning. The modifications consist of two parts: 1) The use of PopArt to enforce scale invariance for the set of value-based RL policies learned in their network, and 2) a policy distillation loss function to mitigate forgetting caused by the off-policy nature of replaying old tasks. They set out to verify their modifications improve the performance of replay based and SOTA based approaches for continual reinforcement learning. The empirical section consists of a series of results in Continual World, starting with a controlled experiment only looking at two sequential tasks at a time and then moving towards more difficult variants of CW6, CW10, and CW20. RECALL does admirably, outpacing both perfect replay and the set of SOTA algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I think the manuscript is clean and written well. I believe the above weaknesses should be easily rectified through some editing and re-motivating the work from the perspective of multi-tasking reinforcement learning (at least clarifying this as such) (**W1**), and clarifying how their approach uses some modification from ClonEx (**W3**) and how your approach may have more data than others (**W4**). While 1,3,4 are most important in my eyes, the other weaknesses should also be addressed to the best of the ability of the authors.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JfoZM38HQa", "reviewer": "Reviewer_ZCap", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n## Paper is clearly written and easy to follow\n\nI really appreciate how easy it was to identify the contributions and see what specific problems they were designed to overcome. \n\n## Experiments were well thought out\n\nQualitative results to motivate the problem, followed by a broad comparison to alternative methods, and proper ablations -- well done!\n\n# Weaknesses\n\n## Is forgetting still an issue for experience replay based methods?\n\nYou claim backwards transfer / catastrophic forgetting is a key problem you want to address, and then demonstrate that it is a problem in Figure 1. But this figures shows an average degradation in the single digits percentage points whereas forward transfer failures are seen to be massive (literally 1 to 0 in one case). It would've been fine to focus the paper around preserving the already adequate backwards transfer of experience replay while improving its forward transfer. Alternatively, you should demonstrate that these very small drops are behaviorally significant or find a problem setting where they are more pronounced.\n\n## Results interpreted as being stronger than the given analysis warrants\n\nYou are doing better than most by providing a reasonable measure of spread (90% bootstrapped CI) in your results figures. But then you proceed to highlight the 'best' result in cases where the CIs significantly overlap. I understand the tendency -- I imagine many of these results might become significant with extra seeds (which might be prohibitive to run given the breadth of your baselines). I'd recommend some of the methods found in [1] that would allow you to better express significance in cases where you run many related experiments (e.g. CW6 tasks 1-8). Barring these new analyses, I'd recommend simply removing the offending highlights and hedging your language somewhat.\n\n## The motivation for your method relies on mechanisms that are never shown empirically\n\nYou claim that forward transfer is poor because of the change in reward magnitudes that occur when going between tasks and backwards transfer is poor because of off-policy updates, but neither are shown empirically. This should be straightforward to address: just show that reward magnitudes change across task boundaries and (ideally) that the magnitude of the change correlates with a breakdown in forward transfer (and do the analogous thing for the backwards transfer/off policy case).\n\nA small related point -- shouldn't something like the TD-error magnitude matter more than the raw reward magnitude? And I would imagine this would actually be larger for the new task, since (at convergence) these errors would go to zero.\n\n\n[1]: Deep Reinforcement Learning at the Edge of the Statistical Precipice ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce RECALL, a new method for continual reinforcement learning which combines experience replay with policy distillation to combat forgetting and target normalization to encourage forward transfer. This method is evaluated on the Continual World benchmark against a suite of benchmarks, and often out performs all of them. The two components of RECALL (policy distillation and target normalization) are then ablated, shows that both contributions are relevant for improving performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "These directly correspond to my 3 weaknesses, plus one observation.\n\n1) Show that the backwards transfer your baseline (i.e. perfect memory) is actually insufficient (e.g. maybe the drop is small in aggregate, but corresponds to a small number of dramatic failures, or maybe less replay improves forward transfer at the cost of backwards transfer) or reframe the paper around preserving backwards transfer while addressing the clear failure of forwards transfer.\n\n2) Either add additional analyses to back up the strong claims of your method's superiority, or remove the offending highlights and hedge your claims.\n\n3) Provide evidence that changes in reward magnitude correlate with changes in forward transfer and that changes in off-policyness (e.g. importance sampling ratios) correlate with changes in backwards transfer.\n\n4) (Optional) Your policy distillation loss + RL loss is quite similar to CQL for offline RL. Since older tasks are functionally offline RL problems in your setup, this provides a slightly different motivation. If you rerun with CQL and it still works, I'd recommend going with that (which also obviates the need to provide the evidence I asked for in requested change 3).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the reviewers agreed that this paper was a combination of pre-existing methods, they also all agreed that the results were solid and the experimentation was thorough.
The reviewers also all appreciated the authors' revisions of the paper based on their feedback.
All reviewers were ultimately supportive of accepting this work, and I agree with them.
Decision: Accept (poster)