Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline
Authors: Hongjoon Ahn
Jinu Hyeon
Youngmin Oh
Bosun Hwang
Taesup Moon
Venue: ICLR 2025 Poster (2025)
Keywords: continual reinforcement learning
negative transfer
plasticity loss
Primary area: reinforcement learning
TL;DR: We carry out extensive analysis on the negative transfer in continual reinforcement learning, and also propose a simple baseline, Reset & Distill (R&D) method.

Abstract: We argue that the negative transfer problem occurring when the new task to learn arrives is an important problem that needs not be overlooked when developing effective Continual Reinforcement Learning (CRL) algorithms. Through comprehensive experimental validation, we demonstrate that such issue frequently exists in CRL and cannot be effectively addressed by several recent work on either mitigating plasticity loss of RL agents or enhancing the positive transfer in CRL scenario. To that end, we develop Reset & Distill (R&D), a simple yet highly effective baseline method, to overcome the negative transfer problem in CRL. R&D combines a strategy of resetting the agent's online actor and critic networks to learn a new task and an offline learning step for distilling the knowledge from the online actor and previous expert's action probabilities. We carried out extensive experiments on long sequence of Meta World tasks and show that our simple baseline method consistently outperforms recent approaches, achieving significantly higher success rates across a range of tasks. Our findings highlight the importance of considering negative transfer in CRL and emphasize the need for robust strategies like R&D to mitigate its detrimental effects.

Published as a conference paper at ICLR 2025
PREVALENCE OF NEGATIVE TRANSFER IN CONTINUAL REINFORCEMENT LEARNING: ANALYSES AND A SIM- PLE BASELINE
Youngmin Oh', Bosun Hwang', and Tacsup Moon #t I Department of Electrical and Computer Engineering (ECE), Scoul National University,
Hongjoon Ahn'; Jinu Jinu
2 Interdisciplinary Program in Artificial Intelligence (IPAI), Scoul National University,
3 Samsung Advanced Institute of Technology.
4 Department of ECE/ IPAI/ ASRI / INMC, Seoul National University
(hong0805, kanasimy (youngminO. .aa00.oh, oh, bosun.. hwang) ( samsung com, tsmoon@snu, ac
) @enu. ac kr,
ABSTRACT
We argue that the negative transfer problem occurring when the new task learn arrives is an important problem that needs not be overlooked when developing effective Continual Reinforcement Learning (CRL.) algorithms. Through com- prehensive experimental validation, we demonstrate that such issue frequently exists in CRL and cannot be effectively addressed by several recent work on either mitigating plasticity loss of RL agents or enhancing the positive transfer in CRL scenario. To that end, we develop Reset & Distill (R&D), a simple yet highly effective baseline method, to overcome the negative transfer problem in CRL. R&D combines a strategy of resetting the agent's online actor and eritic networks to learn new task and an offline learning step for distilling the knowledge from the online actor and previous expert's 's action probabilities. We carried out extensive experi- ments long sequence of Meta World tasks and show that simple bascline method consistently outperforms recent approaches, achieving significantly higher success rates across a range of tasks. Our findings highlight the importance of considering negative transfer in CRL and emphasize the need for robust strategies like R&D mitigate its detrimental effects. The code implementation available at pattps://qiee com/ http:////oooooooo000o//oeeeeeee
INTRODUCTION
Following the impressive recent success of reinforcement learning (RL) (Mnih al. 2013 Silver ctt al., 20166] Mnih et al., 2015 OpenAl et al. 2020) in various applications, a plethora of research has been done in improving the learning efficiency of RL algorithms. One important avenue of the extension is the Continual Reinforcement Learning (CRL), in which an agent aims to continuously learn and improve its decision-making policy over sequentially arriving tasks without forgetting previously learned tasks. The motivation for such extension is clear since it is not practical to either re-train an agent learn multiple tasks seen so far or train dedicated agent for each task whenever a new task to learn arrives. The need for CRL is particularly pressing when the sequentially arriving tasks to learn are similar to each other as in robot action learning (Kober al. 2013).
In general, one of the main challenges of continual learning (CL) is to effectively transfer the learned knowledge to a new task (i.e., improve plasticity) while avoiding catastrophic forgetting of previously learned knowledge (i.e., improve stability). So far, most of the CRL methods (Mendez et al. 2020
2022 Rolnick et al., (2019] Wolczyk et al. 2022) also focus on addressing such a challenge, largely inspired by the methods developed in the supervised learning counterparts; e.g., improving the
stability by regularizing the deviation
the important parameters (Kirkpatrick et al., 2017] Zenke
ct al., 2017 Ahn et al. 2019; Jung et al. 2020), storing the subset of dataset previous tasks (Chaudhry ct al. 2019a bj Lopez- Paz & Ranzato 2017) or isolating the important parameters (Mallya
& Lazebnik 2018 8 Mallya et al.) 2018 Hung al. 2019 Yoon et al. 2018). Furthermore, several works mainly focused on improving the plasticity of the network by transferring the knowledge from
"Equal contribution.
Corresponning author

---

Published as a conference paper at ICLR 2025
previous tasks (Rusu et al. 2016 Schwarz et al. 2018) selectively re-using the important parts for learning new tasks (Mendez et al. 2022 2020 Mendez & Eaton (021).
Due to the aforementioned trade-off. it is generally understood that the plasticity degradation occurs in continual learning mainly due to the emphasis on stability. However, several recent work pointed out that, particularly in RL., the plasticity of a learner can decrease even when learning single
task (Nikishin et al. [022] [ewandowski et al. 2023 Kumar et al.] 202T Lyle et al.] [022] 2023]
Sokar et al.. 2023) Berariu et 2021)). in which the stability is not considered at all. Those works identified that the occurrence of such plasticity loss may be largely due to using non-stationary targets while learning the value function. These findings give some clues for understanding the plasticity degradation phenomenon in CRL, which occurs quite often not only when learning each task but also when task transition happens, but not the full explanation.
Namely, in CRL, even when the simple fine-tuning is employed for sequentially learning tasks, it not hard to observe that a learner already suffers from learning new task as we show in our experiments in later sections. We may attempt to explain this plasticity degradation of fine-tuning, which does not consider stability whatsoever, through the lens of the plasticity loss mentioned above; i.e., since the non-stationarity of the learning objectives (or the reward functions) arises when task transition happens, the plasticity loss occurs and hampers the learning ability. However, as we observe from our careful empirical analyses, above explanation is not fully satisfactory since such plasticity degradation turns out to be dependent on what specific task a learner has learned previously. That is, show that the dissimilarity between the learned tasks also becomes critical factor for the plasticity degradation (of fine-tuning) in CRL, which we identify as the negative transfer problem that has been also considered in conventional transfer learning literature (Zhang et al. 2022 Taylor & Stone 2009; Chen et al. 2019).
To that end, we mainly focus on and try to address the negative transfer problem in CRL. In Section 3 we first carry out a simple three-task experiment that exhibits a severe negative transfer for fine-tuning,
We show that simple adoption of the various remedy for the plasticity loss in RL agents proposed in
recent works (Nikishin et al. 2022 [e22|Leeanddoo Lewandowski et al.] 2023 Kumar et al. 2021 Lyle et al.) 2022
2023, Sokar et al.] | Berariu et al. [202]) cannot successfully mitigate the negative transfer in our setting. Moreover, we also demonstrate that when such negative transfer phenomenon prevails, methods that promote positive transfer (beyond fine-tuning) can also result in detrimental results. Subsequently, via more extensive experiments using the Meta World YYu et al. 2020). DeepMind Control Suite (Tassa et al.] (018)) and Atari- 100k (Mnih et al. [013] Kaiser et al. 2020) environments, identify that various levels of negative transfer exist depending on the task sequences and RL algorithms. From these findings, in Section 4 we propose a simple method, dubbed as Reset & Distill (R&D), that is tailored for CRL and prevents both the negative transfer (via resetting the online learner) and forgetting (via distillation from offline learner). Finally, in Section 5| we present experimental results on longer task sequences and show R&D significantly outperforms recent CRL baselines as well as methods that simply plug-in the recent plasticity loss mitigation schemes to the CRL baselines. The quantitative metric comparisons show the gain of R&D indeed comes from addressing both negative transfer and forgetting. We stress that our result underscores addressing negative transfer phenomenon is indispensable in CRL since our simple baseline method can already surpass the methods that try to promote positive transfers between tasks.
2 BACKGROUND
2.1 PRELIMINARIES
Notations. In CRL, an agent needs to sequentially learn multiple tasks without forgetting the past tasks. We denote the task sequence by task descriptor E (1,.... 7). in which T' is the total number of tasks. At each task t, the agent interacts with the environment according to a Markov Decision Process (MDP) A,, Pr,rr), Ar,p,, ry), where S, and A, are the set of all possible states and actions for task r. Given St+1, St E S, and a; E Ar, Pr ($t+1 |st, ax) is the probability of transitioning to s;+11 given a state s and action at. r=(st, a:) is the reward function that produces scalar value for each transition (8t, dz). The objective of an RL agent is to obtain a policy ((aa) (8t) that can maximize the sum of expected cumulative rewards for each task T.
RL setting. In this paper, we mainly focus on the actor-critic method which combines both value- based and policy-based methods. This method includes networks: an actor that learns a policy
2

---

Published as a conference paper at ICLR 2025
and critic that learns the value function; the critic evaluates the policy by estimating the value of each state-action pair, while the actor improves the policy by maximizing the expected reward. Given task and s E S,, 0 E Ar, we denote the actor parameterized by 0 [a,]s,;; 0-), and the critic parameterized by Φ as Q(a+, 81; Φ,). For the algorithms that only use the state information in the critic, denote the critic network as ((st; Φ,) In our study, adopt SAC (Haarnoja al. 2018) and PPO (Schulman et al. (017) as representative actor-critic methods due to their stability and efficiency.
2.2 LOSS OF PLASTICITY IN RL
Here, outline recent studies that pointed out the plasticity loss of RL algorithms from several different viewpoints. Igl et al. (202T) found an evidence that using the non-stationary target when learning the value function, unlike the stationary target of supervised learning, can permanently impact the latent representations and adversely affect the generalization performance. From a similar perspective. Kumar et al. (00TT) and Lyle al. (2022) figured out that the non-stationarity of the target may diminish the rank of the feature embedding matrix obtained by the value network. They hypothesize that this phenomenon ultimately results in the capacity loss of the value function and hinders the function from learning new tasks. To address this issue, [yye et al. (2022) proposed a regularization method, InFeR, to preserve the rank of the feature embedding matrix. Nikishin et al.] (2022) considered another viewpoint and demonstrated that RI. methods that tend to highly overfit to the initial data in the replay buffer can suffer from primacy bias that leads to the plasticity degradation for the incoming samples. Furthermore. SSokar et al. (2023) argued that the large number of dormant neurons in the value network, which could be caused by using the non-stationary targets for learning, maybe another reason for the plasticity loss. To address this issue, they proposed ReDo that selectively resets the dormant neurons to enlarge the capacity of the network.
While above proposals certainly made some progress, they still remained to be partial explanation for the plasticity loss. Namely, Lyle et al. (202)) showed that as opposed to the analyses in Kumar cet al.] (202I) and Lyle et al. (2022), the high correlation between the rank of the feature embedding matrix and the plasticity loss only appears when the underlying reward function is either easy or hard to learn. For example, they showed that the environment produces the sparse rewards, there is low correlation between the feature rank and the plasticity loss. Subsequently, they also showed that the large number of dormant neurons affected the plasticity loss only when the underlying network architecture happens to be multi-layer perceptron. Lyle et al. (2023) proposed a new insight that the root cause the plasticity loss is the loss of curvature in the loss function. [ewandowski et al. (2023) also stressed that the optimization landscape has diminishing curvature and proposed Wasserstein regularization that regularizes the distribution of parameters if it is far from the distribution of the randomly initialized parameters. [ee et al. (2023) divided the plasticity into two aspects. One is the input plasticity which implies the adaptability of the model to the input data, and the other is the label plasticity which implies the ability of the model to adapt to evolving input-output relationship. Lee et al. (2023) show that combining all the methods (e.g. layer normalization, sharpness aware minimization (SAM) (Foret et al.. 2021), D) and reset (Nikishin et al. (022) that improve the input and label plasticity can enhance the overall plasticity. To the end, Naumann et al. (2024) broadly analyzed various regularization techniques for improving the plasticity, and figure out that resetting the network surpasses other schemes. In Nikishin et al. (2022), [ee et al. (2023) and Nauman et al. (2024), all of them show the effectiveness of the resetting the network while learning a single task. However, since resetting the network can cause complete forgetting of past tasks in CRL setting, naively applying the resetting schemes in CRL would be counterintuitive.
Alternativey,, Dohare et al. (222)) considered the degradation of the plasticity of stochastic gradient descent in both continuous supervised and reinforcement learning. Furthermore, Abbas et al. (2023) provided empirical results showing that as learner repeatedly learns task sequence multiple times, the performance of each task degrades. They proposed that when the loss of plasticity occurs, the weight change of the value function network consistently shrinks the gradient descent proceeds. To address this issue, they adopted Concatenated ReLU (CReLUU to prevent the gradient collapse. Despite the difference, the authors have referred to this phenomenon as plasticity loss as well.
2.3 NEGATIVE TRANSFER IN TRANSFER LEARNING
The negative transfer problem has been identified as one of the important issues to consider in transfer learning (Taylor& & Stone, 2009). Namely, in Wang et al. (2019), Cao et al. (2018), Ge et al.] (2014) and Rosenstein al. 2005), they observed that when the source and target domains not
3

---

Published as a conference paper at ICLR 2025
sufficiently similar, the transfer learned model on the target task may perform even worse than the model that learns the target task from scratch (i.e., negative transfer occurs). In CRL, one may argue that such phenomenon is just another version of plasticity loss mentioned in the previous subsection since the task transition causes the non-stationarity of targets for learning an agent. However, as show in the next section, our simple experimental results demonstrate that merely applying the methods in Lyle et al. (2022), Abbba et al. (2023) and Lewandowski et al. (2023) that aim to address the plasticity loss issue in RL do not readily resolve the negative transfer problem in CRL.
(a) push-wall
3 THE NEGATIVE TRANSFER IN CRL
3.1 A MOTIVATING EXPERIMENT
0.75
0.50
In this section, we carry out simple experiment on a popular Meta World (Yu et al. 2020) environment, which consists of various robotic manipulation tasks, to showcase the negative transfer problem in CRI.
0.25
SAC
Success Rate
145
(b) window-close
1.00
Existence of negative transfer.
Firstly, Figure aa) and
0.75
(b) show the success rates for learning the push-wall and window wonddwwwwloo 2018) and PPO (Schulman et al., 2017) algorithms for 3 million (M) steps from scratch, respectively. Note that both algorithms achieve success rates close to 1, showing both tasks are quite easy to learn from scratch. Now, Figure aa) shows the results for continuously learn ing sweep- into, push-wall 11 and windoww close tasks with simple fine-tuning (red) for 9M steps (3M steps for each task). Namely, SAC and PPO
0.50
close Los tasks with SAC (Haarnoja et al.
SAC
190
2.0
Environment steps
let
Figure 1: The success rates of SAC and PPO on (a) push-wa and (b) window close tasks.
are simply fine-tuned to mush-wall 1 after learning sweep-into and window-closs after push-wall. In this context, tne-ttning refers adjusting all parameters of the network without imposing any freezing or regularization constraints, hence, it does not put any emphasis on the stability to combat catastrophic forgetting. In the results, we clearly observe that both SAC and PPO completely fail to learn push-wall after learning sweep-ine into even when the fully plas- tic fine-tuning is employed. Hence, we note such failure cannot be attributed to the well-known dilemma in continual learning.
Mitigating plasticity loss cannot fully address negative transfer. For an alternative explanation, can check whether such a failure can be identified by the indicators of the plasticity loss developed by the studies presented in Section 2.2 Figure b) shows the number of dormant neurons (Sokar eet al., 2003), rank of the feature embeddings (Lyle et al.) 2022 Kumar al., (021), weight deviation (Abbas et al. 2023), and Hessian sRank (Lewandowski et al. 2023) of the fine-tuned model's actor and critic across the three tasks. When focusing on the push wall task, we observe mixed results; namely, while some indicators (i.e., high dormant neurons and low feature rank) indeed point to the plasticity loss of the model, the others (i.e., high Hessian sRank and high weight deviations) are contradicting. Furthermore, in Figure a)). we also plot the results of the methods - i.e., ReDO, InFeR, CReLU, and Wasserstein Regularization that aim to mitigate the plasticity loss of the model from the perspective of each respective indicator with the same color code in Figure (b).. Still, the success rates of all methods on push- wiichhwwlll. 11 remain significantly lower than the one in Figure [
Based on these results, we note that the dramatic performance degradation of the fine-tuning model on the push-wal task cannot be well explained by the previous work on identifying and mitigating the plasticity loss of an RL agent. Furthermore, if the plasticity loss were truly occurring at the task transitions, the high success rate of the third, wi window-cclo los task cannot be well understood, argue that the learnability of an RL task may depend on the preceding task, and the negative transfer from the preceding task, which cannot be solely captured by previous research, one of the main obstacles to overcome in CRL.
eitheu"] Therefore,
Promoting positive transfer cannot address negative transfer. There are several works that aim to promote positive transfers between tasks in CRL (beyond simple fine-tuning) (Schwarz al. 2018 Mendez et al. 2020; Mendez & Eaton 2021 Mendez et al. 2022). One may expect those
While the results vary depending on the learning algorithms, it is still clear that the third task has much higher success rates than the second task.
4

---

Published as a conference paper at ICLR 2025
SAC
SAC
SAC Feature Rank
SAC Weigt Change My!
SAC
sweep-into push-wall
window-close
Dormant Neuron Ratio
Hess Rank
fele
Succent
Actor
Acti Citci
Adtar Cr
Acci C/ttic
Sweap inco
pish wa Lall noow close
Cittic
stops
PPO
PPO Dormart nt Neuron Ratio
PPO Feature Rank
PPO Weight Change Mgl
PPO Hoss an Rank
Sweep into wall windoww close
Succes rate
Bell
Acto
Actor CrtCi
Adtor
Acc
Sweep-inn
Pioww close
Cmc
C/tc
Environment stops
(a)
Figure 2: Results on continual fine-tuning SAC (top) and PPO (bottom) on 3 tasks. (a) Success rates with various methods. (b) Various indicators of the plasticity loss of the models across the three tasks.
methods to resolve the negative transfer issue by transferring useful knowledge from the previous task while learning a new task. However, from simple ablation study on Progress & Compress (P&C) (Schwarz et al. 2018). well-known baseline of CRL that promotes positive transfer, we observe that such methods also show detrimental performance when there is a negative transfer phenomenon.
More specifically, Figure 3 shows the results of SAC on the same three tasks as in Figure when combined with several variations of P&C. Namely, P&C employs an adaptor to promote positive transfer from previous task. and evaluated the schemes with (w/) and without (w/o) the adaptor. Moreover, as the original P&C paper (Schwarz et al.) 2018) has also pointed out, the knowledge learned in the active column and the adaptor may hinder the learning of new incoming task; thus, in addition to the original 'without (w/o) reset' mode, we also tested with (w/) reset, which randomly initializes the network parameters of both active column and adapter when learning new task begins. Hence, the 'w/ adaptor & w/o reset' and 'w/ adaptor & w/reset' modes are the variations of P&C originally proposed in Schwarz al. (2018).
rasst
ni inteptor
MEG aclanier
inteptorr
From the figure, we first observe that the "w/o reset' mode, regardless of whether use the adaptor or not, still fails to learn the second task, push wwall. This result shows that the mechanism for promoting positive transfer is not helpful at all, if not harmful, for resolving the severe neg- ative transfer. Secondly, the 'w/reset & w/adaptor' mode slightly increases the performance on push push-wall 1, but it still achieves much lower performance than learning from scratch. This suggests that the mechanism for promoting positive transfer may in fact hurt the CRL performance when the degree of the negative transfer is severe. In such a case, addressing the negative transfer may have a higher priority than promoting positive transfer.
Seccess (current task)
0.8
0.4
Swees- Into)
pushhwall
Window close
steps
Figure 3: Results of the 3-task experi- ment with P&C variants, utilizing SAC.
In summary, we show that methods for both mitigating plasticiy loss and promoting positive transfer cannot successfully address the severe negative transfer in our three-task example.
3.2 IDENTIPYING VARIOUS LEVELS OF NEGATIVE TRANSFERS
Motivated by the previous subsection, we carried out more extensive experiments using the Meta World (Yu al. 2020), DeepMind Control Suite (Tasaa et al. 2018), and the Atari- 100k (Mnih et al.) 2013 Kaiser et 2020) environment to check the various patterns of negative transfer in CRL.
Meta World. We carefully selected 24 tasks that can be successfully learned from scratch, i.e., that can achieve success rates close to 1, within 3M steps. We then categorized them into 8 groups by grouping the tasks that share the same first word in their task names. The 8 task groups were [Button, Door, F Faucet, Handle, P Plate, Push, Sweep, Window), ), and for more details on the specifie tasks in each group, please refer the Appendix B Note that the groups were constructed
5

---

Published as a conference paper at ICLR 2025
(a) Meta World, SAC
Plate-1
Push
Sweep-(Q
()+Platee
()+Pushi
(-SSweep
Success Difference (ryecond Fscratch
0.0
0.0
0.4
-0.6
-0.6
Tack group
Button
Doerr
Rawcet
Handle
Plate
Push
Sweep
Window
(b) MetaWorld,
PPO
0.2
Plato-el )
Push-UCi
Sweep-())
0.2
(>-Plate
(1-PPush
J--Swoep
0.0
0.0
-02
-0.-
0.4
-0.6
0.6
-0.8
-0.8
Figure 4: Negative transfer patterns for the two-task fine-tuning in Meta World with (a) SAC and (b) PPO, when tasks from Plate, Push and Sweep groups are learned as the first (left) or the second (right) task.
simply based the names of the tasks, hence, the tasks that are in the same group can also be largely dissimilar - the main reason for the grouping is to save computational cost for our experiments.
After the task grouping, we carried out substantial two-task CRL experiments with fine-tuning as shown in Figure 4 Namely, we first picked three groups, Plate, Push, and Sweep, and verified the patterns of the negative transfer the second tasks depending (i) when they come the first task, (ii) when they come as the second task, and (iii) when the applied RL algorithm varies. More specifically, the left two figures in Figure 4 are for the results when tasks from P Plate, Push, or Sweep task group come as the first task and show the learnability degradation in the 8 task groups that come the second task (i.e., we did not carry out the exhaustive pairwise two-task experiments, but averaged the results of the following randomized experiments. Namely, we randomly sampled tasks from the first and second task groups and sequentially learned those tasks with 3M steps each with fine-tuning, for 10 different random seeds. When the first and second task groups are identical, we sampled two different tasks from the group and carried out the two-task learning. Then, we computed the average of the differences r Tsccond, the success rate of the second task learned by fine-tuning after learning the first task, and Tseratch the success rate of learning the second task from scratch, for each second task group. The average was done over the number of episodes and random seeds, and the more negative difference implies the severer negative transfer. The right two figures for the reverse case, i.e., when tasks from elate, Push, or Sweep task group come as the second task, the average success rate differences
(i)). In order
the computation for the experiments,
depicted depending the first task group (i.e., and lower figures are for the two popular RL algorithms, SAC and PPO (i.e., case (iii)). Overall, we did 10 (random seeds) 39 (two-task pairs) x 2 (algorithms) 780 two-task experiments.
(ii)) Finally, the upper
From the figures, we can first observe that PPO tends to suffer from the negative transfer more severely than SAC in general. Furthermore, it apparent that the negative transfer pattern differs depending on the specific task sequence. Namely, for the P Plate task group with SAC, the negative transfer rarely occurs regardless of the task group being the first or second tasks. However, for the Push group with SAC, we observe that while the tasks in the group do not cause too much negative transfer on the second tasks when they are learned first, they tend to suffer from negative transfer when learned after other tasks. Finally, for the Sweep group with SAC, it is evident that the tasks in the group both cause negative transfer on the second tasks and suffer from the negative transfer from the first tasks.
DM Control. In this experiment, we selected tasks, which are (ball-in-cup-catao) cartpole-balance cartpole-swineeeeee finger-turn-easy fi fish-upright, point- mas mass-easy, zeacher-easyl Using those tasks, we trained SAC and PPO, and similar to the
"The experiments for the overlapping pairs in (i) are repeated, but the results shown.

---

Published as a conference paper at ICLR 2025
DeepMind Control, SAC
bali catch cortpole-balance
cartpole swingup
fish uprighhthht
reacheneasy
fnger-turn-eayy
point easy
ball- poll-n-cup-canch
Inngrturreeary
his-upright
1- ffngertum-cad
Return Difference (Guscee Genwee)
156.0
1500
ballin-cup.catch
100.0
1050
$5.0
530
5.0
436.0
200.0
2000
Figure 5: Negative transfer patterns in DeepMind Control environment with SAC.
Atari-100k, Rainbow
amider
baxing
Normalized Score Difference
bank heist-4())
battic-zone--il
brea ko
heiskkhheist
( (-)-batttc zonc
[
())-breako
0.6
0.2
-0.2
0.2
-0.4
0.4
-066
Figure 6: Negative transfer patterns Atari- environment with Rainbow.
case in the Meta World experiment, also selected representative tasks, (ball-in-cup-cated) finger-t Singer-turneeal foyy.iiso fish-upright, Here, we did not make task groups. Figure [erure/]sshowe shows the results of the SAC. To measure the degree of negative transfer, report the return difference. In the figure, the negative transfer also occurs quite frequently, and especially for fish wassssuuriie igt task, this task always suffer from the negative transfer regardless of the first task. This phenomenon is similar to push task group in the Meta World. For more details, please refer to Appendix
Atari. Different from Meta World and DM Control, Atari environment produces visual observation. which is much more complex than previous experiments. In this experiment, we selected 8 tasks, which are (alien, assault, bank-heist boxing, dar, asterix, battle-zone, breakout| }. In this experiment, we trained Rainbow (Hessel et al. 2018) on two task pairs, and we selected representative tasks, (bank-hett heist, battle zone, breakout] J. Figure shows results. We report the difference of the normalized score which is normalized by the score obtained from scratch. the figure, we can clearly observe that the negative transfer problem also occurs between Atari games. Especially for the batt setttleeene,,,, when comes the second task, it suffer from severe negative transfer in most cases. For more details, please refer to Appendix [
A SIMPLE BASELINE FOR ADDRESSING THE NEGATIVE TRANSFER IN CRL
Motivation. In the previous section, it highlighted that in cases where negative transfer is severe, the previous knowledge from earlier tasks can become an obstacle to learning. To mitigate the effects of negative transfer, it is essential consider two key factors; first, all prior knowledge from previous tasks should be crased during the learning of the current task, and second, the catastrophic forgetting must be prevented enable sequential learning across multiple tasks. A straightforward approach to eliminating previous knowledge is to randomly re- re-initialize all network parameters. However, this insufficient on its own, as inevitably leads to forgetting within the network, making difficult to meet both requirements. To overcome this challenge, we propose the use of two actor networks: the online and offline learners. This dual-network framework with periodic reset aims to balance the need to discard outdated knowledge learn the current task while maintaining the ability to leam sequentially without forgetting. Our method can be considered analogous to the 'w/reset, w/o adaptor' mode in the experiment shown in Figure 3. However, while P&C applies dual network mechanism in
7

---

Published as a conference paper at ICLR 2025
conjunction with an adaptor to facilitate positive transfer, our primary motivation lies in reducing transfer itself. A more detailed discussion on the comparison with P&C is provided in Appendix [F]
Reset and Distill (R&D) To describe method in details, we denote the parameters of online actor and critic network as online and @ Ponline respectively, and the parameters of offline actor as O We periodically reset online and Ponline after finishing learning each task. For the notational convenience, we denote the parameters of the online actor right after learning task т, but before resetting them, as Or, Note these parameters are not stored or utilized directly in the subsequent tasks.
Based on the notation, consider when T = 1, i.e., the learning the first task. Clearly, the online actor and critic can learn the task with existing RL algorithms like SAC or PPO. Once the learning is done, we can then generate a replay buffer D, by utilizing the expert actor with parameter 0 Next, we train the offline actor using the state-action pairs in D, by distilling the knowledge from 0- to the offline actor with O ofline. Then, we store Mr, a small subset of 'Dr, in the expert buffer M. After completing the training for the initial task, we reser all of the parameters oullne and Lonline before initiating learning for the next task. The whole process is iteratively applied to subsequent tasks T 2, .... T. During the distillation process after the first task, the buffer for the current task, D,, and the buffer containing information from all previously encountered tasks, M, are used together to prevent forgetting. Hence, the loss function for the offline actor for task T becomes
&R&D.r (eeDD:-(0ftit (ffflin))
Σ
KL (L(1(1151 0;) [+|s; fotline) +
Σ (B
(L((T(1se Oomine (-1s:
(BBo,
(b)
in which BD, and B mini- sampled from D, and M, respectively, n, è T--ss;; 0;) and Ty A TT[1s,; 0g), and & < refers to the tasks preceding the training of the current task т. Note that the term (a) corresponds to the knowledge distillation from the online actor to the offline actor, and the term (b) corresponds to the behavior cloning prevent the forgetting. The final outcome of this method is the offline actor, fomine. which has sequentially learned all tasks. Note our method has two distinct training phases: the first to reset parameters for the online learner, and the second to distill knowledge to the offline learner. Consequently, we dub our algorithm as Reset and Distill (R&D), and comprehensive summary of the Algorithm given Appendix A
Potential concerns about R&D. First, unlike the online learner, one may argue that the offline actor could still suffer from negative transfer. However, we expect the degree of negative transfer to be lower for the offline learner, for which the learning occurs with fixed target labels, than the online learner, which uses the network's outputs as the varying target labels during RL bootstrapping. Second, due to the periodic reset of the online learner, it could be argued that R&D never takes the positive transfer into account. While it is a valid point, as elaborated in Section 3.T we have observed that the CRL methods that promote positive transfer would often suffer from significant performance degradation due to severe negative transfer rather than gaining benefits of positive transfer. Hence, the primary design goal of R&D was to focus solely on mitigating negative transfer. While positive transfer remains a critical issue in CRL, and R&D cannot be seen as a permanent solution for CRL, if R&D demonstrates superior performance compared to other algorithms, it would indicate that addressing negative transfer should take priority before considering positive transfer.
5
EXPERIMENTAL EVALUATION
5.1 TWO-TASK FINE-TUNING EXPERIMENTS WITH VARIOUS METHODS
We applied R&D to SAC and PPO on two consecutive tasks to evaluate its effectiveness in mitigating negative transfer, as detailed in Section 3 We then compared these results with fine-tuning, CReLU (Abbas et al. 2023) and InFeR (Lyle et al. 2022) to assess their relative performance. Figure 7 provides the results. In many cases, we can observe that CReL. and InFeR still suffer from the
'For an off pollcy algorithm like SAC, we may reuse the replay buffer to train the online actor and critic. However, such reuse may lead to degradation in the performance of the offline policy, due to the discrepancy hetween state-action pairs and the expert. The experimental results on this matter are in Appendix H
8

---

Published as a conference paper at ICLR 2025
(a) SAC
(b) PPO
Paathvll
01
Pate-l
Puch-
Sweep-13
0.1
[1-PPate
Urernen (acooa)
311
Averagg
02
Rinetuning
trFell
CRUL
05-
4.5
Figure 7: Negative transfer patterns using (a) SAC and (b) PPO with various methods when tasks from P Plate, Push and Sweep groups are learned as the first or the second task. For each method, the difference of success rates is averaged over all randomly sampled first or second tasks.
negative transfer in PPO, and R&D effectively resolved the negative transfer. In SAC, the overall performance of R&D for tackling the negative transfer is much better than InFeR and CReLU. The main difference between R&D and the other methods in terms of the learning procedure is the usage of bootstrapping. Different from the RL methods which use bootstrapping for learning the value functions, since R&D learns distillation from the online actor in term (a)), the degree the negative transfer is much less than the fine-tuning variations.
task only through the supervised learning (i.e., the knowledge
Finetuning
P&C
ClonEx(BC) CReLU
R&D(Ours)
SAC
PPO
EWC
ClonEX(BC)
ClonEx(BC) InFeR
Easy
Hard
Random
1.0
Average Success Rate
80-
0.5
35
0.1
50
35
16
0.
22
15
23
244
0.0
0.3
0.5
0.5
3.2
3.5
2.5
2.2
244
3.0
13
35
20
12
25
2.4
Environment Steps
Figure 8: The average success rates of different methods for three types of sequences.
5.2 EVALUATION ON LONG SEQUENCE OF TASKS
We also evaluated R&D on long task sequences which consist of multiple environments from Meta World, and compare the results with several state-of-the-art baselines. We used total of 3 task sequences: Easy, Hard, and Random. For Easy and Hard sequences, the degree of the negative transfer is extremely low and high, respectively. For the Random sequence, we randomly selected and shuffled 8 tasks. For the details on each sequence, please refer to the Appendix C
The three baselines used are the following: EWC (Kirkpatrick et 2017). P&C (Shhwarz et al. Furthermore, we also
2018), and ClonEx (Wolczyk et al. 2022). along with naïve tine-tuning' compare our method ClonEx with InFeR (Lyle et 2022) and CReLU (Abaa al.) 2023) to check whether those methods can tackle both negative transfer and catastrophic forgetting. Note that ClonEx leverages the best-reward exploration technique originally designed only for SAC, leading us to choose Behavioral Cloning (BC) as the method for PPO implementation.
Figure g shows the results. Since the offline actor of R&D learns new tasks in an offline way, we instead put markers on the results of R&D and connected them with lines to notice the difference between the baselines. All results are averaged over 10 random seeds. In this figure, we can observe that when the negative transfer rarely occurs ("Easy'), ), the performances of R&D and ClonEx, ClonEx with InFeR, and ClonEx with CReLU are similar. However, when the negative transfer frequently occurs ('Hard" 'Random"), R&D outperforms all the baselines.

---

Published as a conference paper at ICLR 2025
Table 1: The results on negative transfer and forgetting with various schemes.
Measure
Negative transfer (T)
Forgetting ())
Soquence
Easy
Hard
Kandom
Easy
Hard
Random
SAC/PPO
Fine-tuning
-0.0967 -0,0967-0.379 -0.379
11500/-0:624
-0.193 7 -0.425
0.9007 0,361
3300470.33
0.71770.336
EWC
-0.071 / /-0.536
-0.457 / -0.676
-0.260
-0.375
0.852 / 00.39
0.512 / 0.281
0.671 /0.430
P&C
00.071
00,57
-0.207
0,871 /-
0.472/
0.702
ClonEx
-0,057 -0.422
-0.513 / -0.608
-0.276/ / -0.438
0.015 0.027
0.005 / 0.043
0.040 1440//.011 00001
ClonEx CReLU
-0.196/ -D.325
00.558 / -0.610
-0.213 -0.275
0.039 0.029
0.067 / 0.003
0.012/ -0.014
ClonEx InFeR
-0.117 -0.075
-0.503 /-0.462
-0,232 --02266
0.031 0.043
0.001 /-0.014
0.038 000000
R&D
-0,002/ 0.025
-0.041 / 0.025
-0,014 0.013
0.000 0.050
0.008 / 0.029
0.045 0.002
5.3 ANALYSES ON NEGATIVE TRANSFER AND FORGETTING
To quantitatively analyze how negative transfer and forgetting actually occurs in our experiments, we measured the forgetting and transfer of methods: R&D, Fine-tuning, EWC, P&C, ClonEx(BC), ClonEx with CReLU, and ClonEx with InFeR. Let us denote the success rate of the task when the actor immediately finished learning task as R. and the success rate after training task from scratch as H2 ia,ingle Then the transfer after learning task т, denoted as T,, and the forgetting of task i after learning task 7, denoted as Fr,i, are defined follows, respectively:
= - RSigle and Fr.i te(1....T-1)
max Ra, - R
After learning all T tasks, for the transfer and the forgetting, we report the average of T, and Fr,, for all task E {I, ] and {I...., T),, respectively. In this measure, for the transfer, if this has negative value, indicates the negative transfer occurs. Note that the higher values of transfer and the lower values of forgetting are better in our setting. prreeets the results on the transfer and forgetting, while [ provides the results with standard deviations for reference. this table, all CRL baselines, except for R&D, display vulnerability to negative transfer. Across all methods, negative transfer tends to be more prominent the 'Hard' sequence compared to the 'Easy' sequence, whereas it appears to be at moderate level for the 'Random' sequence. It is worth mentioning that, as discussed in Section 3 PPO exhibits a higher propensity for negative transfer compared to SAC.
In terms of forgetting, it appears that CRL methods, excluding ClonEx and R&D, also experience catastrophic forgetting. Given that SAC typically exhibits greater forgetting than PPO, one might infer that PPO is more suitable choice for CRL. But this is not the case, as negative transfer rate of PPO is higher than that of SAC, resulting in smaller number of trainable tasks in the sequence for PPO. Therefore, it is inappropriate to directly compare the forgetting of SAC and PPO.
In our previous findings, we observed that while the average success rate of ClonEx surpasses that of other CRL baselines, still falls short of the average success rate achieved by R&D. However, the results indicate that ClonEx exhibits forgetting comparable to R&D. Hence, we can deduce that the performance degradation of ClonEx is attributed to negative transfer rather than forgetting.
CONCLUSION
In this paper, we demonstrate the pervasiveness of negative transfer in the CRL setting. Specifically, we show that recent studies addressing plasticity loss do not effectively mitigate this issue, as evidenced by comprehensive and extensive experiments conducted in the Meta World environment. To effectively address negative transfer in CRL, we propose R&D, simple yet highly effective method. Experimentally, we illustrate that R&D, utilizing both resetting and distillation, not only addresses negative transfer but also effectively mitigates the catastrophic forgetting problem.
"Note that because of the severe negative transfer when training P&C with PPO, the loss diverges to infinity, hence, unable train P&C with PPO. Therefore, only report the results of P&C with SAC.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Research Foundation of Korea (NRF) grant [RS-2021- NRQ59237] and by Institute of Information communications Technology Planning Evaluation (IITP) grants [RS-2021-112] 1343, 2S-2021-11222066 RS-2022-112201 13, RS-2022-112209590 funded by the Korean government (MSIT). was also supported by AOARD Grant No. FA2236-23-1-40779 and Samsung Electronics Co., Ltd [Grant No. 10220808-01793-01]
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents (CoLLAs), pp. 620-636, 2023.
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty- based continual learning with adaptive regularization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4394 4404, 2019.
Tudor Berariu, Wojeiech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106. 000000 2021.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Partial transfer learning with selective adversarial networks. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2724-2732, 2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference Learning Representations (ICLR), 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H.S. Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902. 190...992....... 10486, 2019b.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2 06325, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimization for efficiently improving generalization. In International Conference on Learning Representations (ICIR), 2021.
Liang Ge, Jing Gao, Hung Ngo, Kang Li, and Aidong Zhang, On handling negative transfer and imbalanced distributions in multiple source transfer learning. Statistical Analysis and Data Mining: The ASA Data Science Journal, 7(4):254-277 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning (ICML), pp. 1861-1870, 2018
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint
2015.
11

---

Published as a conference paper at ICLR 2025
Ching- Yi Hung. Cheng-Hao Tu, Cheng-En Wu, Chien- Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems (NeurIPS), pp- 3647-3658, 2020.
Łukasz Kaiser, Mohammad Babaeizadeh, Piotr Mitos, Blažej Osiński, Roy Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Conference Learning Representations (ICLR), 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnka--Ba Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 1 114(13):3521-33
2017.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 52(11):1238-122 2013.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-paraaetterzzatonn
inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Dachoon Gwak, Joonkee Kim, Jaegul Choo, Se- Young Yun, and Chulhee Yun. PLASTIC: Improving input and label plasticity for sample efficient reinforcement learning. In Advances Neural Information Processing Systems (NeurIPS), pp.
62270-62295, 2023.
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado. Directions of curvature as an explanation for loss plasticity. arXiv preprint arXiv:2312.0024: 2023.
David Lopez-Paz and Mare Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6467-6476, 2017.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. In International Conference Machine Learning (ICML), pp. 23190-23211, 2023.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7765-7773. 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning mask weights. In European Conference on Computer Vision (ECCV), pp. 67-82, 2018.
Jorge A. Mendez and Eric Eaton. Lifelong learning of compositional structures. In International Conference on Learning Representations (ICLR), 2021.
Jorge A. Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. In Advances in Neural Information Processing Systems (NeurIPS), pp. 14398-1 14409, 2020.
Jorge A. Mendez, Harm van Seijen, and Eric Eaton. Modular lifelong reinforcement learning via neural composition. In International Conference Learning Representations (ICLR), 2022.
12

---

Published as a conference paper at ICLR 2025
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, (18((78))::5595555 2015.
Michal Nauman, Michal Bortkiewicz, Mateusz Ostaszewski, Piotr Miloš, Tomasz Trzciński, and Marek Cygan, Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint 4rXiv:2403.00514 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning (ICML), pp. 16828-16847 2022.
OpenAI, Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in in-hand manipulation. The International Journal of Robotics Research, 39(1):3-00, 2020.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Michael T. Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G. Dietterich. To transfer not to transfer. In NeurIPS 2005 Workshop on Transfer Learning, volume 898, 2005.
Andrei Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1r0iv:1606.044 04671, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka
Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference Machine Learning (ICML), pp. 4528-4537, 2018
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In International Conference on Machine Learning (ICML), pp. 2217-2225, 2016.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning (ICML), pp. 32145-32168, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. arXiv preprini
2018.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey, Journal of Machine Learning Research, 10(7):1633-1685, 2009.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11293-11 1332, 2019.
13

---

Published as a conference paper at ICLR 2025
Maciej Wolczyk, Michal Zajac. Razvan Pascanu, Lukasz Kuciński, and Piotr Mitoš. Disentangling transfer in continual reinforcement learning. In Advances Neural Information Processing Systems (NeurIPS), pp. 6304-6317, 2022.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations (ICLR), 2018.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. hera-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), pp. 1094-1100, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), pp. 3987-3995, 2017.
Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. JEEE/CAA Journal of Automatica Sinica, 10(2):305-32,, 2022.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A ALGORITHM
Algorithm Reset and Distill (R&D)
Input:
Number of epochs E; Total number of tasks T Initialize: Network parameters ooliinss Ponline and Outtine; Expert buffer M 0 for r = 1.... -do do
if then
Randomly re-initialize Online and & unline
end if
Learn task 7 using f and Ponline Generate replay buffer D
for e 1, E do
Sample BD, ~ Dr and BM ~ M Compute &R&D (O using BD, and BM Update O Outiine with VERKD(O (fomine) end for
Store small subset M, of DD into M end for
B DETAILS ON 8 TASK GROUPS
Zo To-Suppee
Drowadd sappoo
Sowooo--oooo yoow
200
P
24 Win Kott
Thsc name
Figure 9: Success rates after training 50 task in Meta- World for 3M steps. SAC was used for training. Results from two different random seeds are distinguished by different colors. The bar plot represents the success rate, and the line marker represents the area under the curve (AUC) of the success rate curve obtained during training.
15

---

Published as a conference paper at ICLR 2025
Prior examining negative transfer in CRL, we identified tasks that could be learned within 3M steps among the 50 robotic manipulation tasks included in Meta- World Yu et al. (2020).
Figure 9 illustrates the success rates when training the 50 tasks using the SAC algorithm Haarnoja et al. (2018) for 3M steps. In this figure, tasks with lower area under the curve (AUC) values can be interpreted as requiring a relatively larger number of steps for training. This implies that some tasks may not be learned within 3M steps in certain cases. Therefore, identify negative transfer in specific tasks, it is necessary to prioritize tasks that can be fully learned within 3M steps, i.e., tasks with high success rates and AUC values. Following this criterion, we selected 24 tasks:
Eade
Figure 10: Visualization button-press (left) and button-press-topdooxy (right).
As indicated by their names, the tasks can be classified based similarity. For example, as seen in Figure [0] both borton-press and but -orttoon---------- toodown involve the robot pressing a button, with the only difference being the direction of the button. By grouping similar tasks together, the 24 selected tasks can be classified into a total of groups
Button:
pitoo--pres-toooownr wall, outton-press. button-press-wall
Door: [door-close, door-lock, door-open, door-unlock]
Faucet: (faucet-open, faucet-close)
Handle: Shandle-presss-iiee candle-press, handle- handle-pull-sidee handle-pull.
Plate: (plate-slide-back-sidde plate-slide-hack,
plate-slide-side, plate-slide)
Push: (push, push-wall)
Sweep: (sweep-into, sweep)
Window: (window-close, window-open)
C DETAILS ON THE LONG SEQUENCE EXPERIMENTS
We evaluated R&D on long task sequences which consist of multiple environments from Meta-World. and compare the results with several state-of-the-art CL baselines. For the experiment, used total of 3 task sequences. Firstly, we identified task pairs that exhibit negative transfer when fine-tuning two tasks consecutively. With this information, it is possible compare the potential difficulties between the task sequences we want to learn. For example, consider different task sequences like A-BBCCD and -H where each alphabet represents one task. If observed negative transfer occurring in consecutive task pairs (A, B), (C, D) and (F, G) within the sequences, the first sequence contains two pairs likely to exhibit negative transfer, while the second has only one such pair. Therefore can expect the first sequence to be more challenging than the second.
We utilized this method to create two task sequences, each with a length 8: 'Hard' and 'Easy', The 'Hard' sequence comprises 6 task pairs where negative transfer occurs in the 2-task setting, while the 'Easy" sequence generated by connecting only those task pairs where negative transfer does not occur. To further validate the results in arbitrary sequence, 24 tasks employed in the preceding section and conducted training by shuffling them based on each random seed. Henceforth, will refer these arbitrary sequences the 'Random' sequence.
randomly chose 8 out of the
The sequences constructed using as above are as follows. Task name marked bold indicates that negative transfer may occur when it is learned continuously followed by the previous task.
Easy (faucet-open - door-close + button- press-topdown-wall + handle-pull + window-close plate-slide-hack-sidd handle-press - door-lock)
Hard [faucet-open + push + sweep - button-press-topdown - window-open -> sweep-into - button-press-wall push-wall)
Random (door-unlock, faucet-open, handle-press-sice handle-pull-side, slate-slidd-baac--sdd,,,, plate- slide-side, shelf-place, window-close)
16

---

Published as a conference paper at ICLR 2025
D DETAILS ON THE BASELINES
In this section, we explain the details of the baselines in our experiments.
EWC (Kirkpatrick et al., 2017): Elastic Weight Consolidation (EWC) is a regularization- based continual learning method in which the Fisher information matrix captures the impor- tance of each weight and gives high regularization strength on important weights.
P&C (Schwarz et al. 2018): Progress and Compress (P&C) is a regularization-ban method that adopts EWC policy consolidation scheme and progressive network promote forward transfer.
ClonEx (Wolczyk et al. 2022): ClonEx is a simple yet effective method which performs behavior cloning previous tasks' policy increase the stability, and do exploration actively on an arriving task to increase the plasticity. As in our results, ClonEx is effective preventing the catastrophic forgetting in continual reinforcement learning.
CReLU (Abbas et al. (023): Concatenated ReLU (CReLU) is originally designed for object recognition Shang et al. (2016), but the authors of (Abbas et al. (203)) find that the use of CReLU can effectively address plasticity loss in continual reinforcement learning. CReLU(()) (x), (-xx) is a concatenation of the ReLU outputs of the original input and its negation. Since CReLU can propagate the gradient regardless of the sign of the input, the authors expected that the use of CReLU can promote the weight change which can mitigate the loss of plasticity.
InFeR (Lyle al., 2022): (Lyle al. 2022) show that training the RL agent with the non-stationary target diminishes the numerical feature rank which indicates the target- fitting capacity, and result the plasticity loss causes. To tackle this problem, (Lyle al. 2022) introduce a regularization scheme, InFeR, which regularizes the features of the penultimate layer with the features of randomly initialized network which contains high target-fittingg capacity with high feature rank.
Compared to the baselines, the additional memory and computational budget of R&D is marginal. Namely, both EWC (Kirkpatrick al., (017)) and P&C (Schwarz al. 2018 also store the networks that learned previous and current tasks, while ClonEx (Wolczyk et al., 2022) stores state samples for computing the behavioral cloning loss. Usually, storing the samples takes much larger memory budget than storing the network parameters. For R&D, it stores both two models and state samples in the buffer. In case of the memory budget, compared to ClonEx, the additional component in R&D is the network for offline actor, and in our experiment, the number of parameters for the networks we used is small. Therefore, R&D does not require a large amount of memory budget compared to the ClonEx. In case of the computational budget, training the online actor for all methods takes hours. For R&D, extracting the rollouts takes 15 minutes and the training time for offline actor takes 25 minutes. Therefore, the additional computational budget for R&D is also small compared the baselines.
17

---

Published as a conference paper at ICLR 2025
WFD POURS
w/rdast
KSD
E RESULTS OF R&D ON 3-TASK EXPERIMENTS
adaggoor
with adhotoo
SAC PGC
2.0
In this section, we include not only the results of R&D but also the results of the online actor of R&D which corresponds 'w/reset & w/o adaptor'. Figure IT shows the results. In the figure, we can observe that the online actor does suffer from the negative transfer, and eventually, the R&D can also effec- tively resolve the negative transfer. Through the above results, we want to stress that resetting the whole agent and discarding the previously learned knowledge effective on tackling the negative transfer.
0.8
0.4
Environment stops
106
Figure 11: Results of the 3-task experiment with P&C variants and R&D, utilizing SAC
F A MORE DETAILED DISCUSSION OF THE DIFFERENCES BETWEEN P&C AND R&D
F.1 THE ADAPTORS IN P&C
In P&C (Schwarz al. 2018), there exists module called adaptor, which serves as additional mechanism designed to enable the active column to utilize past information stored in the knowledge base Specifically, the adaptor combines the activations of the active column and the knowledge base in layer-wise manner, as expressed by the following equation:
h4 = a (W + a; @ Ugo (VGh + c:) + b.)
Here, W, and b, represent the weight and bias of the active column, while U; V,, and e; denote the weight and bias of the adaptor. To compute the activation h; of the active column, the activation AKB, from the knowledge base is processed through the adaptor. Additionally, the original paper mentions that re-initializing the network parameters of the active column and adaptor allows for the successful learning of a more diverse range of tasks.
F.2 FUNDAMENTAL DIFFERENCES IN DESIGN AND MOTIVATION BETWEEN P&C AND R&D
While the online/offline actors mechanism in R&D is indeed similar to P&C's implementation of the active column and knowledge base, P&C and R&D differ not only in architecture but also in their fundamentally distinct underlying motivations
A key feature of P&C is the utilization of models trained on previous tasks through lateral connections. To achieve this, adaptor modules, described above, are extensively employed facilitate these connections. This is not merely structural detail; the adaptors serve essential components for leveraging knowledge acquired from previous tasks to effectively learn the current task, thereby enabling progress. In this regard, the knowledge base in P&C is not only a continual learner but also a supporting mechanism that enhances the efficient learning of the active column.
In contrast, R&D does not utilize adaptors or other lateral connections. This indicates that there no transfer from the offline actor to the online actor, as such transfer can sometimes negatively impact the learning of the current task. We have demonstrated this experimentally in Figure [I] As result, unlike P&C's knowledge base, the offline actor in R&D functions solely as a continual learner, receiving knowledge distillation from the online actor.
18

---

Published as a conference paper at ICLR 2025
G EFFECT OF THE SIZE OF M, AND D,
In this section, we additionally investigate the impact of the size of the expert and the replay buffer on the per- formance of R&D. To examine the effect, we conducted experiments by varying the size of the replay buffer used in the distillation phase from 10k to 1M, and the size of 10k. We used 'Hard' se-
SAC
10-
Average success rate
the expert buffer from quence, which can be considered as the most challenging sequence in the previous experiments as it showed the highest negative transfer among the 3 sequences, and mea-
[M4] -5
[
23
PPO
sured the average success rate of each task after all tasks
were learned. Figure 12 illustrates the results. Note that [D=] and LM indicate the size of replay and expert buffer respectively. Both SAC and PPO algorithms show that as the replay buffer size increases, the average success rate also increase. This is because if the size of the play buffer is too small, the total number of samples used for training the model decreases, leading to insufficient learning. When we varied the size of the expert buffer, we did not observe any noticeable differences. Based on this result, we can reduce the expert buffer size to achieve better memory efficiency.
36-
LMf -
[AP]
[M - 000
Figure 12: The average success rates of R&D with SAC and PPO various |D,| and [.].
H EFFICIENCY OF BUFFER GENERATION IN R&D
potential concern with applying R&D the reliance on expert rollout for distillation, which may increase training costs. While additional rollouts do contribute to the over- all computational requirements, their impact is minimal compared to the training time of the online learner.
SAC R&D
1.0
Replay buffer from the online leamar
Aperage Succors Rute
Expert rolbuut
0.8
0.6
In experiments, training SAC single task for 3 million steps required approximately 8 hours, whereas generating the buffer through expert rollouts took only about 15 minutes, accounting for roughly 3% of the total training time. This indicates that the buffer construction process is relatively efficient and does not significantly hinder overall training.
Number eevvronmenn
Figure 13:
Comparison of performance on Hard sequence between using the re- play buffer from the online learner (blue) and generating a new buffer through ex- pert rollouts (red) in SAC + R&D.
Furthermore, off- off-policy RL algorithms such as SAC inher- ently utilizes a replay buffer during online learner train- ing, which can be directly repurposed for the distillation process R&D. While this approach may lead to slight performance degradation compared to constructing a separate buffer through expert rollouts, the difference remains marginal. As shown Figure [3] the results indicate that reusing the replay buffer provides viable alternative with comparable effectiveness, potentially reducing the need for additional expert rollouts.
19

---

easily find the negative Published as a conference paper at ICLR 2025
I EXPERIMENTS ON DEEPMIND CONTROL SUITE
(TASSA ET AL., 2018 8
(a) SAC, DeepMind Control
ball-in-cup-cacch
hungenturm-essy
fish-upright)
150.0
150.0
ball-i--cp--cacch
fingerluur-essyy
uprigirigttt
100.0
100.3
0.0
0.0
647
-50.0
Return Difference (Gaacond - Gpoucel
-156.0
-150.9
-200.0
bellHn-up--crt cathh
Secipil--o carppll-oolTppp cwippup Imgen-tur t-eady
falh-ppight
reacherewayy
caripole
pont-mass easy
(b) PPO, DeepMind Control
ball-in-cup-catch
fingen-urr-eessy
Fish-uright
400.0
400.0
ball-in-cup-catchh
(inger-uum-esyy
fish-upright
25.3.M
-20.00
-200.0
-400.0
-400.3
Figure 14: Two-task fine- mne-tuning results for (a) SAC and (b) PPO with standard deviation. The values with + sign refer to the standard deviation.
To show the existence of the negative transfer in other domain, also carry out experiments DeepMind Control Suite Tassa et al. (2018). First, we select 7 tasks (ball- 1all-in-cupp cat--cupppcctttt cartpole swingup, finger-turn-eassy peint-mass-eane reacher-easyl Those tasks are carefully selected which can be suc- cessfully learned from scratch within 1M steps. Different from the experiment in Section 3. we do not make groups on those tasks. After selecting the tasks, we also carry out the two-task CRL experiments 18 pairs like in Figure with 5 different random seeds. Figure [4] shows the results. In this case, similar to the case in Meta World experiment, the negative transfer in PPO is much severe than SAC. When the three tasks are first tasks, the negative transfer occurs more frequently, Only ball 1011-10 -in-innnn cup catch is getting worse when it lies the second task. In terms of SAC, there are some cases where the negative transfer occurs rarely or severely. For example, for the fish-upris ght task, the phenomenon is quite opposite when it lies the first task (rarely occurs) or the second task (frequently occurs). For the other tasks, we can also find the negative transfer quite often. Therefore, also in the DeepMind Control Suite environment, transfer problem.
cartpole-balane
fi fishuppright
20

---

Published as a conference paper at ICLR 2025
1.1
RESULTS OF R&D ON DEEPMIND CONTROL SUITE (ASSA ET AL. 2018
SAC R&D, DeepMind Control
cup-ihhcu--atteh
Enngr-tur--aayy
rish-piight
ball-in-cup
fingentum upriaht
100.0
100.0
50.0
50.0
11.39
50.0
30.0
100.0
100.0
Return Difference (Gacona Gronteal
Second task
Rist task
sal it ih-cup-coth
Tigeriurn-easy fsh-uprihtt
point: masi reacheneasy
ballin-cup satch cottieebibroc rattore-wiinup
Trier-tum-see Fishupright
poincmass eady
Tpole balance
reacher easy
(b) PPO R&D, DeepMind Control
bo boll-ih-cup
finger-uun-eaap
upright
bali-In-cus cup catch
fingentum-eapy easy
fish upright
X160
200.0
2200.
400.0
400.0
400.0
.400.0
Figure 15: Two-task R&D results for SAC and (b) PPO with standard deviation.
To further investigate the effectiveness of R&D for tackling the negative transfer in other domain, we carry experiments on DeepMind Control Suite Tassa et al. (2018). The overall experiment same as in Section [ In Figure 15 (a), in most scenarios, our observations indicate that the R&D framework effectively alleviates negative transfer when compared to fine-tuning. However, an exception was noted when R&D was utilized in conjunction with SAC, leading
unexpected performance decline in specific task (cartpole-swingup) To investigate this phenomenon further, we conducted additional experiments. In these experiments, knowledge from a policy trained on the cartpole-swingup task to a randomly initialized agent. The results revealed return difference of -209.1 + 149.1, aligning closely with the performance observed when the offline actor was pre-trained on other tasks without resetting. These findings imply that the inability to learn the eartpole-swingup task is attributed to factors unrelated to negative transfer during the distillation process. If negative transfer were the underlying cause, applying knowledge distillation to randomly initialized network would not have led performance degradation relative to training the agent from scratch.
employed knowledge distillation by transferring
21

---

Published as a conference paper at ICLR 2025
J EXPERIMENTS ON ATARI GAMES (MNIH ET AL. 2013
Atari-100k. Rainbow
ellen
amiktar
asssult
asterix
bank-heist
battle-zone
boxinng
breskoutt
Nomallzeed Score Difference
0.6
bank-heist--(
battle-zone-- )
breakout-
(3-bank-heis)
7-battil-zong
C-breeaout
0.6
0.4
0.4
0.2
0.0
63
6622
22
222
0.4
Figure 16: Two-task fine-tuning results for Rainbow with standard deviation. The values with + sign refer to the standard deviation.
To show the existence of the negative transfer in visual domain, carry experiments on Atari games (Mnih et al. 2013). First, we select 8 games falien, assault, bank-heiss boxing, ami asterix, battle-zone breakout), }. In this experiment, we trained Rain- bow (Hessel et al. 2018 on two task pairs, and we selected 3 representative tasks, (bank-her st, battle-zone, breakout).. Same as the experiment on DeepMind Control Suite, we did not make group this experiment. We carry out two-task CRL experiment on 21 task pairs with 5 differ- ent random seeds. Figure shows the result. First, this figure, we can observe that the negative transfer frequently occurs across various task pairs. For example, when the tasks bank bank-heiss and breakout lie on the first task, the most of the second tasks perform poorly. Furthermore, case of the negative transfer pattern is opposite when battle-22 zone lies the first task or the second task. For the former case, the degree of the negative transfer small. However, for the latter case, most of the proceeding tasks suffer from the negative transfer severely.
K
ANALYSIS ON THE ORDER OF THE KL DIVERGENCE IN R&D
Forward BC Forward KD
Forward BC/ Reverse KD
Reverse BC Forward KD lours)
Reverse BC Reverse KD
Easy
Hard
Random
Average Success Rate
PPO
0.4
0.4
03
Environment Steps
Figure 17: Results of experiments changing the direction of KL divergence for the R&D loss across three 8-task sequences (Easy, Hard, Random).
One may observe a difference in the orders of KL divergence between (a) and (b). This discrepancy arises from the fact that the order employed in (a) adheres to the traditional form of knowledge distillation Hinton et al. (2015), while the order in (b) follows the convention used in Wolczyk et al. (2022). In this section, performed experiments by changing the direction of KI. divergence for the R&D loss. These experiments were applied to the three 8-task sequences mentioned in the main text. Figure 17 shows the results. In the figure. the results indicate that the order of KL divergence did significantly impact the performance.
22

---

Published as a conference paper at ICLR 2025
L THE RESULTS WITH ERROR BARS
In this section, we report the results of Table Figure and Figure 7 with error bars, which corresponds to Table Figure 19 and Figure respectively.
Pnnturing Inf
(b)
Success Rata Differance
02
[
Swees-13
(Fesners fecocel
+00
-00
-200
-2
-33
0.6
-..6
Figure 18: Two-task CRL experiments on various methods. Note that for the methods with CReLU, the results of 'From scratch' are obtained by training vanilla RL methods with CReLU.
(a) SAC
0.2
Piste-(1
Push-uii
Sweep-())
0.2
QHPlate
GJ-PPsh
(--Sweep
$400
Success Rate Difference (rsecond [screech)
EPO
0.4
0.4
-0.8
-1.0
Second
Rist
Bultion
Buthon
(b) PPO
Plate-1
Push-xy)
Sweep-())
Platte
(J)Pushh
Swepp
0.0
2V
Row
-0.2
-0.2
10.40
-0.8
-0.8
-10
-1.0
Figure 19: Negative transfer patterns for the two-task fine-tuning with (a) SAC and (b) PPO, when tasks from Plate, Push, Sweep groups are learned as the first (left) or the second (right) task. The values with d sign refer the standard deviation.
23

---

Published as a conference paper at ICLR 2025
Table 2: The transfer and forgetting results with standard deviation. Note that the numbers after +
represent the standard deviation.
Measure
Transfer (T)
Forgetting (U)
Sequence
Easy
Hard
Random
Easy
Hard
Random
SAC
Fine tuniny
000955
0.0929
40,5002
1 0.1256
-0.1925
0.132
0,8999 1 0,0912
0.5040 0.1333
0.7766
0.11I1
EWC
-0.7088
0.0813
<0.4367
1: 0.0995
-0.2598
0.1229
0.8517 1: 0.1129
0.5123 1: 0.00999
0.6714
0.1337
P&C
-0.7088
0..1134
40.5065
4: 0.1439
-0.2077 +: 0.1517
0,8714 + 0.1
0.4723 4 0.1338
0.7023
0.1335
0.1574
0.1322
0.0437
1.00632
ClonEx CRelll
-0.1958
0.199
00,5500
+ 0.1166
-0.2132
+. 0.1947
0.0389 4: 0.0857
0.0671 +- 0.0997
0.0117
0.0291
ClonIxx-IIIeR
-0.1122
0.1030
0.5003
1: 0.1654
-0.2322
0.1655
0.0311
1: 0.0626
0.0006 1 0.0666
0.0377
0.1075
R&D
-0.0200
0.0232
40,0112
+: 0.0566
-0.0140
+ 0.0603
0.0000
4: 0.0000
0.0083 1: 0.0359
0.0454
0.0701
PPO
Vine-tuning
0.3788
0.1866
0.6238
0.1439
0.4250
0.2318
0.3614 1 0.1114
0.3514 0.111
0.3357 1 0.1367
EWC
-0.5363
0.2493
40.6763 +: 0.1365
-0.3750
0.1250
0.3186
+ 0.1250
0.2814 1 0.1591
0,4300 + 0.0043
P&C
ClonEs
0,1785 -0.4250
-0.6075 + 0.1576
-0.4375 + 0,2183
0.0271 4: 0.0621
0.0429 1: 0.0655
0.0143 1 0.0429
ClonEx CRRLU
-0.6100 0.1814
-0.2750
0.1458
0.0286 0.0571
0.0029 0.0086
-0.0143 0.0769
CloaEx-InFeRR
-000500
0.1696
+0,4625 1: 0,2440
-0.2875
0.3115
0.0429
1: 0.0655
-0.0143 0.0429
0.0000 1: 0.0000
R&D
0,0250 0.0500
0,0250 0.0500
0.0125 + 0.0375
0.0500 + 0.0906
0.0286 + 0.0571
0.0286 + 0.0571
M DETAILS ON EXPERIMENT SETTINGS
In the all experiments, we used Adam optimizer and the code implementations for all experiments based on Garage proposed in Yu et al. (2020). For the machines, we used 16 A5000 GPUs for all experiments.
M.1 HYPERPARAMETERS FOR THE EXPERIMENTAL RESULTS
The hyperparameters for SAC and PPO are described in Table 3 and Table respectively, For the hyperparameters on the CRL methods, the details are described as follows:
EWC, P&C: The regularization coefficient was set to 1000
BC: The regularization coefficient was set to 1, and the expert buffer size [Mal was set to 10k for task
R&D: The regularization coefficient was set to 1, and the expert buffer size [Myl was to 10k for task k. Furthermore, the replay buffer size [D] was set to 10
Table 3: Model hyperparameters for SAC
Value
Value
Description
(Meta World)
(DeepMind Control)
General Hyperparameters
Maximum episode length
500
1000
Environment steps per task
3M
1M
Evaluation steps
100k
100k
Gradient updates per environment step
1
0.25
Discount factor
0.99
0.99
Slgorithm-Specifie Hyperparameters
Hidden sizes
(256, 256)
(1024, 1024)
Activation function
ReLU
ReLU
Policy learning rate
3 x 10-1
1 x 10-1
Q-function learning rate
3 x 10
1 X 10
Replay buffer size
105
106
Mini batch size Policy min. std policy max. std Soft target interpolation Entropy coefficient(a))
64
1024
e -0
E -0 6²
2²
5 x 10
5 x 10-3
automat tuuninng
0.2
24

---

Published as a conference paper at ICLR 2025
Table 4: Model hyperparameters for PPO
Value
Value
Description
(Meta World)
(DeepMind Control)
General Hyperparameters
Maximum episode length
500
1000
Environment steps per task
3M
1M
Mini batch size
64
1024
Evaluation steps
100k
100k
Gradient updates per environment step
1
1
Discount factor
0.99
0.99
Algorithm-Specifee Hyperparameters
Batch size
15000
10000
Hidden sizes
(128, 128)
(1024, 1024)
Policy activation function
ReLU
ReLU
Value activation function
tanh
tanh
Policy learning rate
5 10
3 10
Value learning rate Policy min. std Policy max. std Likelihood ratio clip range Advantage estimation
5 x 10-4
3 x 10-4
0.5
0.5
1.5
1.5
0,2
0.2
0.95
0.95
Entropy method
no_entropy
mi_entropp
Normalize value input / output
True
True
N SOCIETAL IMPACTS
The R&D method effectively addresses the negative transfer problem, significantly enhancing the performance and adaptability of AI systems. This improvement allows Al to learn new tasks more effectively without detrimental effects from previous experiences, leading to more robust applications. Industries reliant on AI for automation and optimization can benefit from increased efficiency and cost savings, AI systems reduce downtime and the need for retraining. Additionally, advancements in robotics (c.g.. healthcare robots, autonomous vehicles, and industrial robots) can lead to safer and more reliable robots, enhancing their integration into everyday and high-stakes environments.
O LIMITATIONS
Though, in our work, we only consider the effect of the negative transfer, considering the positive transfer also important point in CRL. Our method, R&D, can effectively resolve the negative transfer, but does not have the ability the positive transfer by utilizing the useful information the previous tasks. Furthermore, our experiments are mainly focused on Meta World environment, and did carry out experiments much larger scale such Atari Deepmind Lab.
25


Reviews:
[{"review_id": "cHDUg3fsqj", "reviewer": "Reviewer_U6a3", "summary": "This work considers negative transfer problem in continual RL, where steam of RL tasks may severely hurt the policy's performance. The authors proposed to a method named Reset & Distill (R&D) for mitigating such issue. R&D stores  a copy of single-task policy as offline policy for each task in the task stream and also a subset of data of that task; and it then optimize the KL divergence between online policy and all offline policies on the stored data of prior tasks, and the KL divergence between online policy and offline policy of current task.", "questions": "- As I am not particularly familiar with the literature, I wonder do the baselines store copies of policies and data. If some of them have the same setting, storing policies and data, what are the corresponding budget, compared to R&D.\n\n- Is the sequential evaluation of Meta World a standard benchmark in the CRL literature or it is customized in this work? (If it is customized, is there a commonly used CRL evaluation benchmark in the literature?)\n\n\n---\n\npost-rebuttal: 5 -> 6.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Method is simple and plausible\n\n- Empirical evaluation appears strong", "weaknesses": "- I find the paper is a bit difficult to follow as a lot of concepts are described quite verbally such as negative/positive transfer and also the baselines -- although I am not familiar with the literature of continual RL, there might be room for clarity improvement for general RL audience with continual learning background\n\n- Despite I like the visualization of Figure 2, and I understand the R&D is introduced after that, it might be more visually clear how R&D perform if the authors could also include R&D in Figure 2.\n\n---\n\nWhile I think continual RL could be an interesting area and the method proposed could be plausible to mitigate the negative transfer issue, the overall writing is somewhat verbal, hence slightly difficult to follow especially for people working in general RL. Besides, I am not sure if this evaluation is a standard evaluation protocol in the CRL community. Therefore, I find this paper is boarderline to me although with low confidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FMVt9NFzoc", "reviewer": "Reviewer_PBSr", "summary": "Authors demonstrate the existence of negative transfer in CRL setup and show that this issue can not be mitigated with previous CRL approaches. Authors also propose a solution to the problem which demonstrates strong reduction of the negative transfer effect", "questions": "I don't have any major concerns about this work. The only interesting thing for me would be to see the R&D performance on DM Control and Atari domains.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Authors reveal the existence of the negative transfer effect which might strongly affect the further research in CRL field. Further studies might take this aspect into consideration when designing novel approaches or applying existing for tasks of their interest.\n2. The presence of the negative transfer is demonstrated across 3 different domains  (Meta World, DM and Atari) which provide significant evidence that the negative transfer might be a frequent effect.\n3. Using a wide range of techniques it is shown that promoting positive transfer cannot address negative transfer.\n4. The possible solution (R&D) for mitigating the negative transfer is presented. It it compared against multiple CRL approaches and shown to be effective.\n5. Ablation study on hyperparamters effect for R&D is conducted.", "weaknesses": "1. While R&D reduces negative transfer effect, sometimes it suffers from forgetting more than other approaches.\n2. When applied with SAC, R&D requires a vast amount of expert rollouts to keep the good performace. \n3. R&D performance is shown only on Meta World tasks if I understand correctly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WWCLCV5ans", "reviewer": "Reviewer_MLHD", "summary": "Author addresses the challenges of negative transfer in Continual Reinforcement Learning (CRL). Negative transfer occurs when learning new tasks negatively impacts performance, which is problematic for CRL systems that aim to handle multiple tasks sequentially without forgetting. The authors identify that existing CRL methods often focus on mitigating plasticity loss or promoting positive transfer, but these methods fail to effectively counteract negative transfer.\nThey propose a new baseline method called Reset & Distill (R&D) to address negative transfering problem. R&D works by resetting the agent’s online networks for each new task while preserving learned knowledge through distillation into an offline model. and this dual-actor setup, which allows an online network for current learning and an offline network for cumulative knowledge,enableing R&D to handle sequential tasks more effectively by preventing both negative transfer and forgetting. Experiments in their work demonstrate R&D outperforms other models on task success rates, emphasizing the need to prioritize countering negative transfer in CRL​, which seems promising", "questions": "① Balancing Negative and Positive Transfer: The R&D method seems to heavily focus on preventing negative transfer by resetting the online learner's parameters. While this is a novel approach, I wonder about the potential for R&D to leverage positive transfer. Is there a way to modify R&D to harness the knowledge from previous tasks without incurring negative transfer? Your thoughts on this balance would be insightful.\n② Theoretical Underpinnings: The paper does a great job showcasing the empirical success of R&D, but I must say, I'm left wanting more in terms of theoretical analysis. What are the theoretical reasons behind R&D's effectiveness in reducing negative transfer? How does the periodic reset and distillation process interact with the learning dynamics at a deeper level? A bit more depth here could really help solidify the contribution of your work.\n③ Computational Cost: The dual-network architecture of R&D, with periodic resets and distillation, seems promising. However, I'm concerned about the computational overhead this might introduce, especially for resource-constrained settings. Could you discuss the computational efficiency of R&D and any potential strategies to mitigate these costs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Firstly, it focuses on an important yet overlooked issue,the occurrence of negative transfer in Continual Reinforcement Learning (CRL). This problem has a significant impact in practical applications, but prior research has largely concentrated on preventing forgetting or promoting positive transfer, often neglecting negative transfer. Authorr introduces the \"Reset & Distill\" (R&D) method specifically to tackle this issue, demonstrating a strong sense of innovation and hitting the problem \"right on target.\" The experimental design is also solid, verifying the prevalence of the negative transfer issue across different environments, such as Meta World, DeepMind Control Suite, and Atari, which adds credibility to the results. Additionally, the paper is written in a clear manner, particularly in explaining the design and implementation of the R&D method, making it easy for readers to understand the authors' approach. Overall, this paper brings fresh perspectives and practical solutions to the CRL field", "weaknesses": "While the paper does a good job of demonstrating the effectiveness of the R&D method through extensive experiments, it somewhat lacks a deep theoretical analysis explaining why this approach is effective at mitigating negative transfer. What are the fundamental mechanisms that allow R&D to sidestep the pitfalls of negative transfer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C9LUsgIO8Y", "reviewer": "Reviewer_xWNm", "summary": "The paper proposes Reset and Distill (R&D), a method inspired by policy distillation [1] and neural network growth [2], to address catastrophic forgetting in knowledge transfer. R&D operates by reusing expert samples from previous tasks to transfer knowledge to subsequent ones while resetting the online learner network’s knowledge. The authors demonstrate, through comprehensive experiments, that the sequence of tasks [3] significantly affects performance when learning multiple tasks. \n\n*[1] Hinton, Geoffrey. \"Distilling the Knowledge in a Neural Network.\" arXiv preprint arXiv:1503.02531 (2015).*\n\n*[2] Rusu, Andrei A., et al. \"Progressive neural networks.\" arXiv preprint arXiv:1606.04671 (2016).*\n\n*[3] S. Narvekar, J. Sinapov, and P. Stone, “Autonomous task sequencing for customized curriculum design in reinforcement learning,” in (IJCAI), The 2017 International Joint Conference on Artificial Intelligence, 2017.*", "questions": "- [Q1] If task sequencing is feasible, could curriculum learning enhance R&D performance by optimizing task order?\n\n- [Q2] It’s unclear why mitigating negative transfer does not necessarily imply facilitating positive transfer, and vice versa.\n\n- [Q3] Could you provide more details about why Proximal Policy Optimization (PPO) appears more susceptible to negative transfer than Soft Actor-Critic (SAC)? Given the comparison between these algorithms, understanding the underlying reasons could offer valuable guidance for developing more robust approaches to continual learning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* *Comprehensive literature review*: The authors provide a thorough and well-structured review of recent research related to plasticity loss in non-stationary environments and negative transfer in transfer learning. This positions the paper well in the context of existing research.\n\n* *Robust experiments*: The empirical validation is solid, with experiments conducted on recognized benchmarks in the RL community. These experiments effectively test the assumption that the arrival order of tasks impacts learning performance, and R&D outperforms previous methods in mitigating catastrophic forgetting, providing valuable insights.", "weaknesses": "* *Similarity with P&C*: In Figure 3, the concept of an \"adaptor\" within the Progress & Compress (P&C) framework is not well explained. While the term \"reset\" may be somewhat intuitive, it is not clearly detailed in either the main text or the Appendix. Given that the novelty of R&D is acknowledged to be marginal compared to P&C, a more in-depth discussion of how these concepts differ is necessary to strengthen the paper's contribution.\n\n* *Sequence of tasks focus*: The paper emphasizes that the arrival order of tasks have impact in negative transfer, yet the proposed method does not directly address this issue. Instead, R&D relies on known techniques (distillation and resetting) to mitigate catastrophic forgetting.\n\n\n* *Contradictory hypothesis*: The authors propose that *all prior knowledge from previous tasks should be erased* when learning a new task. However, this approach seems contradictory, as retaining certain previous knowledge can, in some cases, enhance learning, especially when tasks share high similarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)