Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
Authors: Hongyao Tang
Johan Obando-Ceron
Pablo Samuel Castro
Aaron Courville
Glen Berseth
Venue: ICML 2025 poster (2025)
Keywords: Plasticity
Continual Learning
Reinforcement Learning
Generalization
Primary area: reinforcement_learning->deep_rl
TL;DR: Reducing churn prevents the rank decrease of NTK, thus mitigates plasticity loss and improves continual RL.

Abstract: Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability induced by the data in each training batch. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.

Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
Hongyao Tang" 2 Johan Obando-Ceron 12
Pablo Samuel Castro
12 Aaron Courville 12 Glen Berseth 2.2
Abstract
challenging, and there have been number of recently pro- posed remedies, such as resetting dormant neurons (Sokar et al., 2023), regularization (Kumar et al., 2023b), and vari- ants of backpropagation (Dohare et al., 2024). It has also been argued that rank decrease of the empirical Neural Tan- gent Kernel (NTK) (Achiam et al., 2019) is consistent indicator of plasticity loss (Lylee et al., 2024; Lewandowski et al., 2023). However, the true causal factors and underly- ing mechanism remain largely unknown.
Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is cru- cial for continual learning. this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability for out-of- data induced by mini-batch training. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tan- gent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, introduce Continual Chum Approximated Re- duction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environ- ments on OpenAl Gym Control, ProcGen, Deep- Mind Control Suite, and MinAtar benchmarks.
In this work, we study the loss of plasticity from the an- gle of churn (Schaul et al., 2022; Tang & Berseth, 2024): network output variability for out-of-batch data induced by mini-batch training. While has been shown that churn is endemic to most deep RL networks, affecting both gener- alization (Bengio et al., 2020) and interference (Liu et al., 2023), we demonstrate that plasticity loss is highly corre- lated with increased churn. We use the NTK matrix as a formal tool to establish the connection between churn and plasticity loss, based on which we analyze the learning dy- namics of continual learning across a sequence of tasks. We demonstrate that under the continual changes in the data distribution and objective function, the agent gradually loses the rank information of its NTK matrix, leading to highly correlated gradients and eventually the exacerbation of churn. Consequently, this destroys the stability and con- vergence of the learning dynamics, which is related to the pathological learning behavior of plasticity loss.
1. Introduction
Reinforcement learning (RI.), when coupled with non-linear function approximators, suffers from optimization chal- lenges due to the non-stationarity of the data and the learn- ing objectives (Sutton & Barto, 1988; van Hasselt et al., 2018). This difficulty is amplified when there is a sequence of changing tasks, as in continual RL (Abel et al., 2023).
Driven by the connection between plasticity and churn, we extend the idea of churn reduction originally proposed for dingle-MDP RL (Tang & Berseth, 2024) to continual RL. We introduce Continual Churn Approximated Reduction (C- CHAIN). C-CHAIN continually minimizes the churn for the data out of the training batch alongside the regular training of continual RL. Moreover, we formally demonstrate that C-CHAIN has two-fold efficacy in mitigating plasticity loss with a gradient decorrelation effect by suppressing the off-diagonal entries of the NTK matrix, and a step-size adj adjustment effect by taking the gradient projection of the data out of the training batch.
One of the causes for this is what is known as loss of plas- ticity (Berariu et al., 2021; Lyle et al., 2022; Dohare et al., 2024), whereby an agent gradually loses its ability to adapt to new data or objective function. It is hypothesized that this is due to the agent's 's network tendency to overfit early experience, hampering its ability to learn on later experi- ence (Nikishin et al., 2022a). Addressing this pathology is
'Mila- Québec AI Institute UUniversité de Montréal. Cor- respondence to: Hongyao Tang stang.hongyao maangya@@mll.qquee quebec, Glen Berseth <glen.berseth @ quile..uebec>.
evaluate our approach under extreme non-stationarity, we conduct the experiments in various continual RL envi- ronments built OpenAl Gym Control (Brockman et al.,
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
2016), ProcGen (Cobbe al., 2020), DeepMind Control Suite (Tassa et al., 2018), and MinAtar (Young & Tian, 2019) benchmarks, with total of 24 continual RL envi- ronments. The results show that reducing churn effectively improves the agent's ' performance in continual RI, and out- performs related methods in most environments.
typically occurs when the network settles into sharp local minima or becomes overly specialized to the initial stages of training, resulting in reduced learning capacity, slower convergence, or poor generalization to new data (Nikishin et al., 2022b; 2024).
The underlying mechanisms responsible for the loss of plas- ticity during training remain an area of active research (Ma et al., 2023a; Lyle al., 2024). Researchers have pro- posed various strategies to mitigate the loss of plasticity; approaches include ensuring active unit engagement (Sokar et al., 2023), mitigating gradient starvation (Gogianu et al., 2021; Dohare et al., 2023), employing smaller batch sizes (Ceron et al., 2023), minimizing deviations from initial pa- rameter values (Leeandowwsi et al., 2023; Kumar et al., 2023a), resetting optimizer (Asadi et al., 2023; Ellis et al..
Our main contributions can be summarized as:
We demonstrate the connection between plasticity loss and increased churn, and show the pathological learn- ing dynamics this connection induces.
We unbox the efficacy of reducing churn in continual RL by identifying a gradient decorrelation effect and a step-size adjustment effect.
2024), and regularizing weight orthogonality (Chung et al.,
We propose C-CHAIN' and demonstrate it effectively mitigates the loss of plasticity and outperforms prior methods in a range of continual RL settings.
2024). Periodically resetting network parameters has been effective (Ash & Adams, 2020; Frati et al., 2024), especially effective in data-efficient reinforcement learning (Schwarzer et al., 2023; Nauman et al., 2024). Ceron et al. (2024a;b); Liu et al. (2025) showed that dynamic sparse training leads to improved performance and enhanced network plasticity.
2. Related Work
Nonstationarity in Reinforcement Learning Non- stationarity naturally occurs in various contexts, with con- tinual RL being among the most prominent examples. Non- stationarity can manifest as gradual changes. This may happen when an RL agent continuously improves its policy over time (Zhang et al., 2024) or when the data generation process slowly evolves (Ellis et al., 2024). These different forms of non-stationarity can coexist. For example, in con- tinual reinforcement learning (Xie et al., 2021), both the reward function and the transition dynamics may change over time, requiring learning systems to adapt dynamically.
3. Preliminaries
3.1. Reinforcement Learning
Consider a Markov Decision Process (MDP) defined by a tu- ple (S, A,P, R, 7, Po; T), with the state set S, the action set A. the transition function P : S x A + P(S), the reward function R S x A + R. the discounted factor y E [o, 1), the initial state distribution po and the horizon T. The agent interacts with the MDP by performing actions with its policy ae ~ (sc) that defines the mapping from states to action distributions. The objective of an RL agent is to optimize its policy to maximize the expected discounted cumula- tive reward (TT) = Ex [ rrr,). where so ~ Po (so), St+1 ~ P (8t+1 8t; az) and T, = R(s,, The state- action value function q" defines the expected cumulative discounted reward for all 8, a E S x A and the policy n, i.e., (s,as = E5 Σ Ei 7'r, r 50 s, ag = a).
Due to the use of a non-linear function approximator under non-stationarity, the model weights and optimization may not lead to optimal policy. There are two main chal- lenges in optimizing under distribution shift, catastrophic forgetting (also called backward transfer) (Rolnick et al., 2019; Chaudhry et al., 2019; Nath et al., 2023) or the loss of plasticity (called forward transfer) (Berariu et al., 2021; Lyle et al., 2022; Dohare et al., 2024).
Continual Learning and the Loss of Plasticity Contin- ual learning focuses on training models to learn sequential tasks. It broadly divided into continual supervised learn- ing, which aims retain knowledge across tasks in static datasets (Wang et al., 2024), and continual RL, where agents adapt to dynamic environments while retaining prior experi- ence (Khetarpal et al., 2022; Kumar et al., 2023a; Dohare et al., 2024; Elsayed & Mahmood, 2024). The loss of plas- ticity refers to a phenomenon in neural network training where the model gradually loses its ability to adapt or sig- nificantly update its parameters (Dohare et al., 2024). This
In deep RL, policy and value functions are approximated with deep neural networks, conventionally denoted by Qo and Js with network parameters and d. The parameter- ized policy F can be updated by taking the gradient of the objective, i.c., d' < Φ + a (V..(TT)) with a step size a (Silver et al., 2014; Mnih et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018).
3.2. Continual Learning and Plasticity
Beyond the standard MDP setting which describes sta- tionary decision-making task, we consider a continual learning scenario where the agent learns to solve tasks
httpps: h/tp:::://ttt //g thub. com /bl wecoontaa/C-CFAAA
2

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
T= (Ti, T2 -2, The that arrive in sequence. In this work, each task is an MDP with a different reward or dynam- ics function and has a budget of N interactions for the agent. Aside from this, we do not use any assumption on the task sequence, e.g., the similarity between tasks. The non- stationarity manifests in the continual switch of learning tasks. We focus on plasticity in this work, and the objective of a continual RL agent in this context is to maximize the average performance over the course of continual learning on the task sequence.
To study the relationship between the two concepts, we use the empirical NTK (Achiam et al., 2019) matrix for- mal tool, as its rank has been shown to be good indicator of plasticity loss (Lyle et al., 2024). For a parameterized network function fo(x) E R, we use g(x) to denote the gradient of fo with respect to its parameters at the data x, i.e., g(x) = Volo(x) E IEd where [0] = d is the dimension- ality. The Neural Tangent Kernel matrix No is the matrix of gradient dot products between all data points (Achiam et al., 2019 Lyle et al., 2024):
Formally, for any point within the total budget, we have the
N) [V)) Vofo(az) (o.
for Ca, 2y
policy T; where E {1....,k] k), and the performance J(T,)) corresponding to it. Note that (TT,) is the performance evaluated on the task TTa) i.e., the current task at time step
(2)
Go Go
denotes the ma- trix form of the gradients for all datapoints. This is also referred to as the outer-produs approximation of the Hes- sian matrix (Lewandowski et al., 2023). In practice, the empirical NTK matrix is often used by sampling a batch of datapoints for the convenience of computation and vi- sualization. Lyle et al. (2024) show that networks tend to exhibit low rank of the empirical NTK matrix No when they lose plasticity. This is also observed when the full gradient is approximated by the penultimate layer network representation (Kumar et al., 2022; Ma et al., 2023b).
where Go = 9(2)) (( (zz))....
For convenience, we define the average performance regarding task T as J(T)) Thus, the average performance of the agent throughout the learning on the sequence T is:
N -i=(j- - (T;)).
k J(T)
AN
= k j=1
L ((T)
= KN
1
Σ J(T.). 1=1
(1)
Similar to the concept of Area Under Curve (AUC), the average performance metric aggregates the learning perfor- mance in terms of efficiency, stability and convergence. We use average performance to evaluate and compare different methods, and in practice, we take intervals within (1.... }} for the estimation of J(T).. In the context of continual learning, the loss of plasticity means that the agent grad- ually exhibits worse learning performance on later tasks. Empirically, it can often be identified when we observe J(TT) - J(T)) 2 J(T,)) ((T)) denotes the performance of an agent of good plastic- ity on the task T, e.g., newly initialized network for the current task.
Now consider the parameter changes from fo to for and let AS = 0' 0. For data T E Beer (the reference data) not included in training batch Buain used for the parameter update Ao, the churn of fo at E with respect to Ao is (Tang & Berseth, 2024):
J(T)) for 1 S S j S k, where
C5(2,0, 0, Do) = for -
(3) fo(T)
(3)
=Vofo(E) Do + 1(1
The parameter update Ag made by mini-batch training can have different forms, which depends on the context. Let- ting L()) be a loss function of fe. we can express the parameter update in general form via the chain rule: A V (,, x)). where x E Buain is the data sampled for the mini-batch training. We use vanilla stochastic gradient descent and use for the learning rate here for legibility. By plugging the general form back, we have the NTK expression of the churn:
4. Mitigating the Loss of Plasticity by Reducing Churn
In this section, we formally study the loss of plasticity in continual RL from the lens of churn. First, we establish a connection between plasticity loss and churn by taking the
NTK matrix as a framework (Section 4.1), based on which we analyze the continual learning dynamics (Section 4.2). Further, introduce Continual Churn Approximated Re- duction (C-CHAIN), and formally dissect the efficacy of churn reduction on plasticity (Section 4.3).
(2. ~ ( Velo(( L(,,x))
= Lr No (I. x) v L(),
(4) Further, we can obtain the churn in vector form, which is equivalent to the approximate change of the function value for all datapoints:
4.1. NTK as the Bridge between Plasticity and Churn
The loss of plasticity is a phenomenon that stems from the pathological learning behavior of a deep network, while churn is an innate feature of the network where the network output for datapoints not included in the current training batch is implicitly and potentially uncontrollably changed.
GoSGL -nnNo.GL..
(5)
where Gy is the gradient matrix of V L() and S is a diag- onal matrix of the same size as No with {0. b values
3

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Equation 7 characterizes the iterative evolution of the error regarding the task Ti, which is mainly determined by the factor 1 - NyNe, St No, is full-rank when it has positive di- agonal values (i.e., non-zero gradients fo (z,,) & 0) and has zero off-diagonal entries, i.e., (o,)(z;) (z)) = 0 for * j. In this case, resembles the tabular function ap- proximation where generalization and interference among different datapoints do not exist. The learning process sta- ble when a proper 1) is selected. However, in most practical cases, the off-diagonal entries of No, are non-zero and No. is not full-rank, making the learning dynamics intricate.
Training
Data
Reference
Data
Figure 1. An illustration of the matrix So where the datapoints are arranged with separation between the training data and the remaining ones (i.c., the reference data).
on its diagonal that corresponds to the sampling results of Brain The vector form in Equation 5 indicates that the NTK matrix No plays an important role in determining the churn, independent of the loss function (or objective function) and the sampling strategy (or data distribution). Intuitively, No determines how the explicit mini-batch training shapes the entire function landscape by generalization or interference between different datapoints. Figure shows an illustration of the matrix No.S. For the symmetric NTK matrix No, mul- tiplying the sampling matrix S; zero-masks out the columns corresponding to the reference data, denoted by the area. The remaining entries consist of the training updates to the sampled training data and the changes caused by churn to the reference data
Next, we continue to analyze the learning dynamics by taking into consideration the task change in T = (Ti, ...; T Different from a stationary learning sce- nario, both the data distribution (related St) and the objec- tive function (related to fr, E,) change throughout learning. In the early stages of learning. the parameters of the agent's network are close to the initialization, thus having no plas- ticity issue. Meanwhile, the empirical objective function estimated with finite online data (related to S,, & ttends to lie in low-dimensional space. With stochastic gradient de- scent, the network fits the empirical objective function with an implicit preference for simple functions (Arpit al., 2017; Nakkiran et al., 2019; Damian et al., 202 in low- dimensional parameter subspace (Schneider et al., 2024; Tang et al.. 2024). In this process, the agent prefers the parameters that have high correlations between different datapoints (Kumar et al., 2022) in order to generalize the updates, and thus loses the rank information in No.
As the empirical NTK matrix plays important roles in both plasticity loss and chur, it serves as a natural bridge be- tween the two. We further our analysis below.
When the task changes from Tr T++1, the agent continues to fit the new objective function from e (0) with distri- bution shift of sampling data S However, the function landscape regarding the new data distribution was implicitly shaped via churn in prior training. Compared to the pa- rameters in the early stage, the gradients between different datapoints correlate more and the rank of Ng becomes lower. This echoes the findings in prior works (Lyle et al., 2024; Kumar et al., 2021; Shah et al., 2020). As the task changes along the sequence T, the rank decrease of the NTK matrix and the exacerbation of churn operate in vicious cycle in the learning dynamics (Equation 7). Therefore, it leads to even less stable learning and worse approximation results, which is the pathological learning behavior identi- fied as the loss of plasticity.
4.2. Exacerbation of Churn Induced by NTK Collapse
Starting from a random parameter initialization, the network of a learning agent digests data from the sequential tasks T (Ti, 72,... .... Th } and updates it parameters to solve each task at hand the best. Notice that each task is an MDP and the data on the task is collected online by the agent within a budget of interaction steps.
To formally characterize the learning process, let (0) fo ff be the error vector of the network fo regarding the optimal objective function f corresponding to task Ti with E {I...., k.. We consider the commonly-used loss function (,(), x)) = 2[f(( - fe(=)]2. The dynamics of the error & for iterative updates O₂ + 0c+1 within the same
task T/ can be derived as:
E((0 (,,) (fr -fo44) (fi fes)
Naturally, a method that prevents the rank decrease of No.S, and the exacerbation of churn should mitigate the loss of plasticity. Therefore, we dissect the efficacy of reducing churn on continual learning in the next subsection.
=so. soe41
(6)
-C,((B,, 0:41 0z).
By plugging the vector form of churn in Equation 5 and G - - -oo)) = (O.), we further have:
4.3. Effects of Continual Churn Reduction
10(++)) R (,()) - N,,
We extend the study on churn reduction and explore its potential in mitigating plasticity loss in continual RL. We
= (T - -.)
4

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Algorithm 1 Deep Continual RL with Continual Churn Approximated Reduction (C-CHAIN).
for L as a common example in deep RL for demonstra- tion, where Lo( = Ez ( ((r ((++>=-(z)) - Qo(x)) 9
1: Task sequence = (7a, T2, ... J. interaction budget per task N, continual RL algorithm &
Note that we use x for a and Q is the target Q-function detached from backpropagation. In this case, we have Ve (0, x) = Qa(()) (z) + + + (a²)) and thus:
2: Initialize agent's $ parameter & and buffer D
3: for task i = 1,2,3...., k do
4: for step = 1, 1,2.3, .N do
5: Interact with the task by performing agent's policy
- (g T g)g = - -proj 1
(12)
6: Update the huffer D according to A.
7: Perform regular parameter update Ae- A(0, Bumin) with the training batch Busin ~ D
Equation 12 shows that the second efficacy of reducing chur can be interpreted the projection of 9 on g with the square norm of y as a scaling factor. This term could either dampen or accelerate the regular gradient yV,, و L which depends on the sign of the kernel between the reference data and the training data, and also the ,
8: Regularize agent's parameter to reduce churn for the reference batch Byg ~ (Bret O Buain = 0)
9: end for
10: end for
The two terms work together when reducing churn during the continual learning process. The ① term decorrelates the gradients between the reference data and the training data by suppressing the off-diagonal entries in the NTK matrix Ng. This combats the rank decrease and mitigates the loss of plasticity under the changes of the data distribution and the objective function as discussed in Section 4.2. Besides, the @ term regulates the regular training gradient based on additional gradient information of the reference data. More regularization occurs when the gradients correlates more.
introduce Continual Churn Approximated Reduction (C-
CHAIN). As depicted by the pseudo-code in Algorithm 1, the agent interacts with the tasks that arrive a sequence. Alongside the regular training of the continual RL algorithm, C-CHAIN continually minimizes the churn for the data out of the current training batch (i.e., the reference batch Bra)..
Formally, the churn reduction loss function of C-CHAIN is defined with the reference data F E Bret as follows:
(0) 2 (C, 2o)
To summarize, reducing churn improves the stability of the continual learning dynamics in Equation 7 and mitigates the loss of plasticity. In the next section, we conduct the empirical study for our formal findings discussed above.
Next, we look into the gradient of the churn reduction loss to shed light on how it influences the continual learning dynamics discussed in the previous subsection:
-, VoC, 0, Ao) x -n
(9)
5. Experiments
In our experiments. we first evaluate C-CHAIN in com- parison with recent related methods, to find out whether it improves continual RL (Section 5.1). Then, we conduct the empirical analysis to examine whether reducing chur pre- vents the rank decrease and how the two effects contribute differently (Section 5.2). Finally, we extend the evaluation to more continual learning settings (Section 5.3).
For clarity we use = Vala (x). [x. Vo)e(T), thus the kernel No( x) = g g. and ,,L, = 20) We
then have the derivative in the expectation below:
Velg ,4,,) Vo
[ VVo
g)
+9
(10)
The ① term is the partial derivative of the gradients of fe- With Hg be the Hessian matrix of fo. we have:
5.1. Performance Evaluation
To evaluate the performance of continual RL, we adopt OpenAl Gym Control (Brockman et al., 2016) and Proc- Gen (Cobbe et al., 2020) and follow the setups in TRAC (Muppidi et al., 2024).
① = Ho (z)g + Ho(x))) v,
(11)
Equation 11 indicates the first efficacy of reducing churn. It takes the gradient of the kernel No(ā, x)) regarding each pair of the reference data v and the training data x, with W (as well as Cy(a, 0, Ag) playing the role of a weighting factor. As result, the off-diagonal entries in the NTK matrix No (corresponding to the area of Figure 1) are suppressed to zero for minimization.
Setups For Gym Control, we use four environ- ments: CartPole-vl, Acrobot-vl, LunarLander-v2 and MountainCar-vo For each environment, task sequence T is built by chaining k instances of the environment with a unique Gaussian observation noise ~ (O, o²) sampled once for each. The number k is 10 for all the environments, except that I is 5 for MountainCar-vo For ProcGen, we use all sixteen environments in the suit, while only four
The @ term depends on the specific form of Ly regarding which the regular training (i.e., Ag) is performed. Here we use the Temporal Difference (TD) learning of Q-network
5

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Table 1. Performance comparison on continual Gym Control. The reported scores are the average performance (Eq. 1) in terms of episode return with mean and standard error over six seeds. The top-2 values (excluding Oracle) are marked by bold and underline.
Calibration Baselines
Related Methods
Owrs
Env
Oracle
Vanilla
TRAC'
Weight Clipping
L2 Init
C-CHAIN
Acmmbott
1255736 1 8,303
-372.725 1 3,268
1166.584 1 8.922
118.821 +: 13.440
-134,038
14,292
119.211 1 6.929
C-CarPPle
299.925 = 5,986
61.766 6.927
217.376 = 26.307
154.306 + 16.047
266.114 + 5.345
160.396 + 14.643
C-LumarLannder
1.118 + 2.002
-499.396 + 19,571
-58.059 # 474
-58.575 + 2.896
-6.666
5.903
-16.699 8, 3,669
C-MountainCar
-322.608 + 9.384
-346.604 4.635
-399.989 + 0.010
267.900 + 6.883
-369.395 27.914
-455.746 + 14.691
Agg. Score
.147.302
-1156.958
.007.256
.290.99
.2439855
2221.22
Table 2. Performance comparison continual ProcGen. The reported scores the average performance (Eq. 1) in terms of episode return with mean and standard error over six seeds. The top-2 values (excluding Oracle) on each row are marked by bold and underline.
Calibration Baselines
Related Methods
Ours
Enw
Oracle
Vanilla
TRAC
Weight Clip
L22 Init
LayerNorn
ReDo
AdamRel
C-CHAIN
Strpilor
13.142 + 0,262
8.028 0.667
18. 18.106 + 0.716
8,200 0.729
11,383 + 0.207
14.085+0.652
9.824+0.221
11 11,863-0,910
19.717 = 0.699
Fruitbot
0.105
L,49 0.139
0.855 0.327
L457 0.090
16657 :
0.92111.254
1.586110.227
1487100.19
1.689 0.188
Chaser
2,9334 1 0.016
2.714 = 0.028
3,096 0.021
28812 0.039
3.067 1: 0.023
2.960.10.034
2.894
2.72010.053
3.432 0.033
Dodgeball
5.017 + 0.415
3,636 + 0.488
6.757 0.617
3.066 0.600
4.859 + 0.459
5.895+00777
3,799+0.324 324
3,780+0.643
7.690 + 0.840
Bigfish
4.206 0.343
2,363 0.632
5,783 0.729
2,561
0.334
3:752 0.267
4,406++0.434
3.010+0.199
0.746+0.343
7.866 0.319
Caveflyer
5,889 1 0,285
5.605 0.282
6.760
0.400
5.157 0.362
5.922 1: 0.307
6.103:10.228
5.595
5,232:10499
7.653 0.587
Climber
5,404 + 0.659
4.557 + 0.692
2.587
0.500
5.003 + 0.740
5.415 + 0.662
5,349+0.688
5.372+0.634
5,207++0.677
5.594 + 0.579
Ninja
2,169 0.311
1.679 0.708
1,278
0.556
1,314
0.601
8.219 1 0.364
2.760+0.572
2.184+00434
1.297++0.597
3. 3.146 0.436
Coinzun
4,953 1: 0.458
4.711 0.676
3,943
1.414
4,985 0.757
4.857 1: 0.362
5.459.1 0063
4.68910.377
4.377:10.503
6.529 1.093
Miner
5,147 1: 0.235
4,250 1: 0.314
4,288 0.807
5,182 1: 0.634
6.153 4: 0.228
5.49111.5442
4.853.10.397
5.500:10.645
9.924 :: 0.636
Jumper
2,175 0.330
1.095 0:211
1,753
0.456
1,536
0.474
2.598 0.299
2,666++04771
1.200+0.2744
2.225+005009
2.83Ki 0.371
Heist
2,426 1: 0.295
2.404 4: 0.290
2.424
0.294
2.506
0.295
2.929 4: 0.343
2.402:10.292
2.691
2.66811.3722
3,950 0.681
Leuper
0,350 4: 0,302
0.347 2: 0.302
0.304 0.334
0.474 4: 0.305
0.557 1:: 0.287
0.509 +: 0.443
0.467
0.511:0.314
0,668 .:: 0.608
Maze
5,528 0,525
5.324 0.621
4,916
0.774
5.314
0.888
5.701 0.999
5.084++0.550
4.747+00616
5.299+0.867
5.9440 0.422
Plunder
7,113 0.413
4
0.430
0.776 9.719
5.290
0.389
6.695 +: 0.383
7.262:+0366
5:910=0.331
5.296++0229
10.340 :: 0.287
Bosstight
2.735 4: 0,369
L9966 1: 0.667
4.722
1.027
2.235 1: 0.630
3.791 4: 0.436
3.503-10.320
2.290
1.54110.506
4.920 +:: 0.45
Age. Score
70.789
55,049
77.289
57.092
72,961
75.154
61.180
61.443
101.792
Sampling unique observation noise for each task the sequence T
baseline. The oracle baseline learns each task from ran- dom initialization, thus free of the influence of task change. Additionally, we consider six related methods including TRAC (Muppidi et al., 2024), Weight Clipping (Elsayed et al., 2024b) and 1.2 Init (also called Regenerative Regular- ization) (Kumar al., 2023b), Recycles Dormant Neurons (ReDo) (Sokar et al., 2023), Layer Normalization (Lyle et al., 2024), Adam with Relative Timesteps (AdamRel) (EI- lis et al., 2024) as our baselines. Note that TRAC outper- forms other related methods like Concatenated ReLU (Ab- bas et al., 2023), EWC (Schwarz et al., 2018), Modulating Masks (Nath et al., 2023) in the same settings adopted in our experiments, Therefore, we do not include them here. Besides, we found that L2 regularization performed very similarly (slightly worse) to L2 Init in experiments, thus we use I.2 Init for representation as they are very similar when the network parameterization is close to zero.
me
Sampling new procedurally generated level for each task in the sequence T
Figure 2.
The continual learning settings for Gym Control and ProcGen. The task switches for continual CartPole (above) and Starpilot (below) are illustrated for demonstration.
(i.e., Starpilot, Fruitbot, Chaser, Dodgeball) were adopted in (Muppidi al., 2024). For each environment, the task sequence consists of 5 instances procedurally generated by sampling a unique game level. The budget is 2M steps per task. Figure 2 illustrates the continual RL setups.
Implementation We use the official implementation of TRAC as the code base, based on which we implement all the baselines above as well as C-CHAIN (i.e., Algorithm I with the vanilla PPO agent as A) to ensure that they only differ in the corresponding algorithm features. One thing to note is that C-CHAIN PPO, both Bunin and B are sampled from the online interaction data collected by the policy in the current iteration. Therefore, CHAIN PPO does not use
Baselines To calibrate the performance of continual RL, we make use of a standard Proximal Policy Optimization (PPO) (Schulman et al., 2017) agent as the vanilla baseline. The agent is initialized once before the continual learning starts. On the other side, we use a PPO agent that gets fully ee-initialized every time the task is switched to be the oracle
6

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Acrobot-vt
starpliot
-109
Epicodo Raturn
--199
Episode Return
PO [Gracle]
PP0 (Vanlla)
TRAC
Vetget Clippin
CartPole v1
dodigbbal
+
(Oracte)
Episoole Return
Episode Return
PPO (Vanilia)
TRAC
C-CHAIN
Weight Clipping
Init
230
66c
3200
Lumantaa denie2
Eploode Return
Episode Return
MountainCaewwt Io600 pno
miner
-109
Epicodo Raturn
-159
Episode Return
2.20
Treatee Ibat:
Figure 3. Learning curves of different methods in four contin- ual Gym Control environments. The curves and the shades are mcans and standard errors over six seeds.
Figure 4. Learning curves of different methods in four (of six- teen) continual ProcGen environments. The curves and the shades are means and standard errors over six seeds.
a cross-task buffer and does not need to be aware of task switches. For the hyperparameters of the PPO baseline agent, we use the default values and keep them consistent across all the methods. For the hyperparameters specific to each method, we search around the recommendation values in the original papers and report the best. More experiment details are provided in Appendix A.
learning within each task, while C-CHAIN also mitigates it effectively. This also indicates that naive network resetting is not adequate to address plasticity loss in continual RL.
Among the methods in comparison, TRAC a competitive baseline in most environments except for the failure in envi- ronments like MountainCar and Coinrun. Weight Clipping works well in Gym Control but performs almost on par with Vanilla in ProcGen. Similarly, L2 Init performs the best in Lunarl aander and CartPole while it falls below C-CHAIN by a clear margin. A possible explanation is that both Weight Clipping and L2 Init mitigate plasticity loss by constraining the feasible parameter space near the initialization. There- fore, for problems where the agent needs complex solution that is not easy to be found near the initialization, Weight Clipping and L2 Init can limit the learning in this sense. Our method C-CHAIN achieves the best aggregation scores in both continual Gym Control and ProcGen, demonstrating its superiority in improving continual RL.
Quantitative Comparison Table 1,2 summarizes the quantitative evaluation for different methods on continual Gym Control and ProcGen, respectively. Figure 3 and Fig- ure show the learning curves. For clarity, we plot the learning curves for three related baseline methods for Proc- Gen. As task switch happens frequently, the learning curves of all methods show a phasic pattern. For the calibration baselines, i.c., Vanilla and Oracle, Vanilla quickly degrades and collapses (especially in Gym Control) after learning the first task in the sequence, while Oracle learns each task as it re-initializes when task switch happens. Over- all, C-CHAIN and the other baseline methods effectively prevent the collapse of Vanilla and improve the learning performance greatly. One may note that TRAC, 1.2 Init and C-CHAIN outperform Oracle in ProcGen. This is be- cause, aside from task switch, -on-stationarity also exists in the learning process of single-MDP RL. Oracle still suffers from plasticity loss due to the change of sampling and policy
Moreover, for continual MountainCar that needs more ex- ploration and is bit more difficult than the other three Gym control tasks, we notice that TRAC almost totally fail on it. The low early-stage performance of C-CHAIN might also be due to the exploration feature of MountainCar, as reducing churn could also slow down the generalization of exploration behavior learned by the policy. Therefore, C-
7

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Median
KQM
Mean
Optimaitty Gap
C-CHAI AdamRel
UN
12 Iniz
Weight Clipping
PPO (Vanilaa-
PPO (Oraclel
0.75
1.00
1.25
1.50
0.75
1.00
1.25
1.50
0.6
0.9
1.2
1.5
1.8 0.0
0.1
0.2
0.3
0.4
Figure Performance comparison with Reliable metries (Agarwal et al., 2021). For each method in the comparison, the results are aggregated sixteen continual ProcGen environments with six random seeds for each.
Approx Rank of (Vanilial: Approx Rank C-CHAIN:
Appron. Rank
C-CHAIN
000
Timestop (2e4)
of Of--ilgg
Figure 7. Visualization of the empirical NTK matrix (100 by 100) for Vanilla (left) and C-CHAIN (right) at 8M timesteps (10M in total) in continual ProcGen Starpilot.
Timestep (2e4)
400
Figure Analysis on NTK matrix terms of approximate rank (above) and the sum of absolute off didgonal values (below) in continual ProcGen Starpilot.
matrix (corresponding to Section 4.2) and (2) the ablation of the two effects of reducing churn (i.e., ①, in Section 4.3).
On the Evolvement of the Empirical NTK Matrix We empirically compute and record the NTK matrix of the agent's policy network at every iteration for the analysis. The details are provided in Appendix the approximate rank (Kumar et al., 2021) of the empirical NTK matrix and the summation of absolute off-diagonal values. For the vanilla PPO agent, the curve of approximate rank exhibits large instability, overall decreasing to a low rank throughout learning. This corresponds to the sharp increase of the scale of off-diagonal values. As expected, C-CHAIN suppresses the off-diagonal values of the NTK matrix and maintains stable and high rank. This explains the performance of C-CHAIN in Figure 4 and Table 1,2, and empirically confirms our formal analysis in Section 4.
CHAIN learns slowly but improves steadily, while vanilla PPO learns quicker and collapses. Continual ProcGen envi- romments like Leaper, Jumper have sparse or even episodic rewards. To gain more understanding for these environ- Max, Mean,
Figure shows
ments, we looked into the average scores Min curves and observed that C-CHAIN improves the av- erage Max scores over PPO Vanilla and the average Mean scores (either by improving Max by reducing failure numbers); while it does not fully avoid the (near-)zerd azero Min scores due to the limited exploration ability of PPO base.
Reliable Metrics To obtain conclusive evaluation, provide the aggregation evaluation by using Reliable met- rics (Agarwal et al., 2021). The evaluation uses Mean, Me- dian, Inter-Quantil Mean (IQM) and Optimality Gap, with 95% confidence intervals. It aggregates over 9 methods, seeds for each of 16 ProcGen tasks as reported in Table 2, i.c., 864 runs in total.
Moreover, as shown by the visualization in Figure 7, we can observe that the vanilla PPO agent exhibits a highly correlated NTK with off-diagonal cells large absolute values (i.e., blue or red). The existence of high correlation leads to low rank, i.c., the sign of plasticity loss, which also echoes the empirical findings in (Lyle al., 2024). In contrast, C-CHAIN shows less correlated NTK and thus maintains the plasticity of the agent.
The results shown in Figure 5. We observe that C-CHAIN performs the best regarding all four metrics and outperforms the second-ber best with no overlap of Confidence Intervals for Median, IQM, and minor one for Mean.
On the Ablation of the Two Effects of C-CHAIN We empirically dissect the gradient (denoted by zot of C- CHAIN's churn reduction loss function (Equation 8) into a projective component (i.e., pro and pPO for the gra- dient the PPO base agent) and an orthogonal component (i.e., g - -rooj We use the projective component to
5.2. Empirical Analysis
To better understanding how C-CHAIN improves contin- ual RL and examine whether the empirical efficacy of C- CHAIN matches our formal analysis. We conduct empirical analysis on (1) the rank evolvement of the empirical NTK
8

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
51t plot
Continual MinAtar To evaluate the generality across different base agents, we implement C-CHAIN Dou- bleDQN (van Hasselt et al., 2016) and evaluate it in con- tinual MinAtar (Young & Tian, 2019) setting built by us. Specifically, we chain three tasks: SpaceInvaders, Asterix, Seaquest. The results in Table 4 show that C-CHAIN also improves the continual learning performance upon Dou- bleDQN in sequence of totally different tasks. Details are provided in Appendix A.3.
Eppeode Return
chasor
Raturn
(Gracel)
C-CHAIN
CCHAIN (Orth Orlyl
Table 4. Performance comparison in continual MinAtar. The reported scores are means and standard errors over twelve seeds.
CCHAIN (Proj Only)
10:00
Figure& 8. Ablation study for the projective and orthogonal gra- dient components of C-CHAIN in ProcGen Starpilot and Chaser.
Eav
Vanilla
C-CHAIN
C-MinAtar
22,044 + 0,733 29.513 + 0.682
approximate the @ term and use the orthogonal component for the 4 term. By applying either component only, we have two variants of C-CHAIN, denoted by Proj Only and Orth Only. We then evaluate the contribution of the two components by comparing with C-CHAIN and Vanilla.
Continual Supervised Learning (SL) on MNIST We extend our empirical evaluation of C-CHAIN to continual supervised learning setting. We follow the settings in L2 Init (Kumar et al., 2023b) and adopt RandomLabbel-MNIS. and Permuted-MNIST as our testbed. The results and imple- mentation details can be found in Table 8 and Appendix A.4. The results show that C-CHAIN improves the vanilla agent but does not perform on par with L2 Init and Weight Clip- ping. The efficacy of C-CHAIN is relatively limited the two continual SL settings, in contrast its superiority in the continual RL experiments. We hypothesize this is because RL suffers more from churn accumulation due to the chain effect (Tang & Berseth, 2024), which acts as prominent cause of plasticity loss in continual RL, thus C-CHAIN can address it effectively. Besides, the two MNIST settings are simple, where a good solution can be found near the initial- ization as preferred by L.2 Init and Weight Clipping. A more thorough study on this point is expected in the future with more continual supervised learning settings.
As in Figure 8, we can observe that both the orthogonal and the projective components are beneficial the vanilla baseline. The two components contribute collectively and add up to the effectiveness of C-CHAIN. Morcover, the orthogonal component contributes more than the projective component. This indicates that suppressing the off-diagonal entries of the NTK (i.e., the @ term) is critical to C-CHAIN. This naturally delineates the difference between the efficacy of C-CHAIN and the related methods that alters RL gradient with projective information in different ways (Chaudhry et al., 2019; Yu et al., 2020).
5.3. C-CHAIN in More Continual Learning Settings
Continual Med Control Suite (DMC) In addition to the discrete-action settings above, we evaluate C-CHAIN in continual continuous control. We build three contin- ual RL environments based on DMC (Tassa et al., 2018): Continual Walker (Stand- Run), Continual Quadruped Walk-Run- Run-Walk), Continual Dog (Stand-Wall Walk-Run-Troof each of which continual RL scenario that chains the corresponding individual tasks in a sequence.
6. Conclusion
In this paper. we study plasticity loss from the lens of churn. We established a formal connection between the two with NTK and demonstrated the interplay between the rank de- crease of NTK and the exacerbation of churn in learning dynamics. We show that our method C-CHAIN effectively improves continual RL results across a wide range of tasks.
Table 3. Performance comparison in continual DMC. The re- ported scores are means and standard errors over twelve seeds.
Limitation and Future work Beyond theoretical ad- vancements, applying churn reduction to real-world contin- ual learning settings, as robotics or large language models (LLMs), could address plasticity loss and catastrophic for- getting in adaptive learning settings (Wu al., 2024; Zhai et al., 2024). Investigating its role in reinforcement learn- ing from human feedback (RLHF) could improve stability in policy updates, ensuring LLMs retain learned behaviors while adapting to evolving human preferences. Besides, Equation 11 reveals an interesting direction to realizing churn reduction in a more direct and effective way by esti- mating Hessian (Elsayed et al., 2024a).
Env
Oracle
Vanilla
C-CHAIN
C-Walker
395.971 + 8.116
305,199 18.519
472.828 + 17.865
C-Quddruped
234.529 19.193
250.153 26.385
314.510 = 44.321
C-Dog
137.080 1. 3.133
129.744 1. 5.914
174.098 1 96
The results are shown in Table 3. We can observe that C-CHAIN PPO significantly outperforms PPO in all three settings, showing the effectiveness for continual control. C-CHAIN also outperforms PPO Oracle because it does not reset for each task and thus can transfer the learned skill from previous tasks to future ones (c.g.. Dog-Stand to Dog- Walk). Details are provided in Appendix A.2.

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Acknowledgements
ity of neural networks, arXiv preprint, 1rX.i:2106.00042. 2021.
We want to acknowledge funding support from NSERC, FQRNT. Google, CIFAR AI and compute support from Digital Research Alliance of Canada, Mila IDT, and NVidia.
Brockman, G., Cheung, V., Pettersson, L.. Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint, arXiv:1606.09 01 540, 2016.
We thank the anonymous reviewers for their valuable help in improving our manuscript. We would also like to thank the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing tools that enabled this work, including NumPy (Harris et al., 2020), Matplotlib (Hunter, 2007), Jupyter (Kluyver et 2016) and Pandas (McKinney, 2013).
Ceron, J. S. O., Bellemare, M. G., and Castro, P. S. Small batch deep reinforcement learning. In NeurIPS, 2023.
Ceron, J. S. O., Courville, A., and Castro, P. S. In value- based deep reinforcement learning, a pruned network is a good network. In ICML. PMLR, 2024a.
Ceron, J. S. O., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J. N., Dziugaite, G. K., Precup, D., and Cas- tro, P. S. Mixtures of experts unlock parameter scaling for deep rl. In International Conference on Machine Learning, pp. 38520-38540. PMLR, 2024b.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with A-GEM. In ICLR, 2019.
References
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference Lifelong Learning Agents, volume 232, pp. 620-636, 2023.
Chung. W., Cherif, L., Precup, D., and Meger, D. Parseval regularization for continual reinforcement learning. In NeurIPS, 2024.
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning. In /CML, volume 119, pp. 2048-20000 2056, 2020.
Abel, D., Barreto, A., Roy, B. V., Precup. D., van Hasselt, H. P., and Singh, S. A definition of continual reinforce- ment learning. In NeurlPS, 2023.
Damian, A., Ma, T., and Lee, D. Label noise SGD provably prefers flat global minimizers. In NeurlPS, pp. 27449-27461 2021.
Achiam, J., Knight, E., and Abbeel, Towards charac- terizing divergence in deep q-learning. arXiv preprint, 1tXiv:1903.08894 2019.
Dohare, S., Hernandez Garciia J. F., Rahman, P., Sutton, R. S., and Mahmood, A. R. Maintaining plasticity deep continual learning. arXiv preprint, 1rX1v.2306.33812. 2023.
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. NeurlPS, 2021.
Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A. C., Bengio, Y., and Lacoste-Julien, S. closer look at memorization deep networks. ICML, volume 70,
Dohare, S., (eemandez-Garcian J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026):768-777 2024.
pp. 233-242, 2017.
Ellis, B., Jackson, M. T., Lupu, A., Goldie, A. D., Fel- lows, M., Whiteson, S., and Foerster, J. Adam on local time: Addressing nonstationarity in rl with relative adam timesteps. arXiv preprint, arXiv:2412. 2,1 13, 2024.
Asadi, K., Fakoor, R,, and Sabach, S. Resetting the opti- mizer in deep RL: an empirical study. In NeurlPS, 2023.
Ash, J. and Adams, R. P. On warm-starting neural network training, NeurlPS, 33:3884 2020.
Elsayed, M. and Mahmood, A. R. Addressing loss of plas- ticity and catastrophic forgetting in continual learning. arXiv preprint, 1.XXv:2404.0077 2024.
Bengio, E., Pineau, J., and Precup, Interference and generalization in temporal difference learning. In ICML, 2020.
Elsayed, M., Farrahi, H., Dangel, F., and Mahmood, A. R. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. ICML,, 2024a.
Berariu, T., Czarnecki, W., De. S., Bornschein, J., Smith, S. L.., Pascanu, R., and Clopath, C. study on the plastic-
10

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Elsayed, M., Lan, Q. Lyle, C., and Mahmood, A. R. Weight clipping for deep continual and reinforcement learning. RLJ, 5:2198-2217, 2024b.
Liu, J., Ceron, J. S. O., Courville, A., and Pan, L. Neu- roplastic expansion in deep reinforcement learning. In The Thirteenth International Conference Learning Representationss 2025. URL https: httttppss//peeeeviers net / forum? ? d=
Frati, I.., Traft, N., Clune, J., and Cheney, N. Reset it and forget it: Relearning last-layer weights improves continual and transfer learning. In ECA/ 2024, pp. 2998- 3005. 2024.
Liu, V., Wang, H., Tao, R. Y., Javed, K. White, A., and White, M. Measuring and mitigating interference in rein- forcement learning. In ICML, 2023.
Gogianu, F., Berariu, T., Rosca, M. C., Clopath, C., Busoniu, L., and Pascanu, R. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In ICML, pp. 3734-3744, 2021.
Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss reinforcement learning. In ICLR, 2022.
Haarnoja, T. Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In JCML, 2018.
Lyle, C., Zheng, Z., Khetarpal, K., Hasselt, H. V., Pas- canu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint, arXiv:4402.18762 2024.
Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., al. Array programming with 2020.
Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. arXiv preprint, arXiv:2310.074 2023a.
numpy. Nature, (85((885))3557--88
Hunter, J. D. Matplotlib: A 2d graphics environment. Com- puling in science & engineering, 9(03):90-95 2007.
Ma, Y., Tang. H., Li, D., and Meng, Z. Reining generaliza- tion in offline reinforcement learning via representation distinction. In NeurlPS, 2023b.
Khetarpal, K., Riemer, M., Rish, I., and Precup, D. To- wards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
McKinney, W. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. O'Reilly Media, edition, February 2013. ISBN 9789351 100065. URL http: http://wwwwwaa...//////a/.. amazon com/ exec//bioosssooooooooooeooooeeee redirect Trakaeeoo-ta AS ASTN/144931977
Kluyver, T., BaaannKKelly,, B., Pérez, F., Granger, B., Bus- sonnier, M., Frederic, J., Kelley, K., Hamrick, J., Grout, J.. Corlay, S., Ivanov, P., Avila, D., Abdalla, S., Will- ing, C., and Jupyter Development Team. Jupyter Note- books-a publishing format for reproducible computa- tional workflows. In IOS Press, pp. 87-90. 2016. doi: 177733/978-1-614999-111
Mnih. V., Badia, A. P., Mirza, M.. Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In ICML, volume 48, pp. 1928-1937, 2016.
Kumar, A., Agarwal, R., Ghosh, D., and Levine, Im- plicit paraattrrraat inhibits data-efficient deep reinforcement learning. In ICLR, 2021.
Muppidi, A., Zhang, Z., and Yang, H. Fast trac: A parameter- free optimizer for lifelong reinforcement leam- ing. NeurlPS, 2024.
Kumar, A., Agarwal, R., Ma, T., Courville, A., Tucker, G., and Levine, S. DR3: value-based deep reinforcement learning requires explicit regularization. In ICLR, 2022.
Nakkiran, P., Kaplun, G., Kalimeris, D., Yang, T., Edelman, I.., Zhang, F., and Barak, B. SGD on neural networks learns functions of increasing complexity. In NeurIPS. pp. 3491-3501, 2019.
Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., and Van Roy, B. Continual learning as computation- ally constrained reinforcement learning. arXiv preprint, arXiv: 04345, 2023a.
Nath, S. Peridis, C., Ben-Iwhiwhu, E., Liu, X., Dora, S., Liu, C., Kolouri, S., and Soltoggio, A. Sharing lifelong reinforcement learning knowledge via modulating masks. In Conference Lifelong Learning Agents, volume 232, pp. 936-960, 2023.
Kumar, S., Marklund, H., and Roy, B. V. Maintaining plas- ticity in continual learning via regenerative regularization. arXiv preprint, arXiv:2308. 1958, 2023b.
Lewandowski, A., Tanaka, H., Schuurmans, D., and Machado, M. C. Directions of curvature as an expla- nation for loss of plasticity. volume arXiv:2312.00246 2023.
Nauman, M., Bortkiewicz, M., Miloš, P., Trzcinski, T., Os- taszewski, M., and Cygan, M. Overestimation, overfitting. and plasticity in actor-critic; the bitter lesson reinforce- ment learning. In ICML., 2024.

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P., and Courville, A. C. The primacy bias in deep reinforcement learning. In ICML, Proceedings of Machine Learning Research, 2022a.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. IEEE Transactions Neural Networks, 16: 285-286, 1988.
Tang, H. and Berseth, G. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. In NeurIPS, 2024.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In ICML, pp. 16828-16847, 2022b.
Tang, H Zhang, M., Chen, C., and Hao, J. The ladder in chaos: Improving policy learning by harnessing the parameter evolving path in low-dimensional space. In NeurIPS, 2024.
Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection. NeurIPS, 2024.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., Lillicrap, P., and Riedmiller, M. A. Deep- mind control suite. arXiv preprint, 1XXv:1801.00690 2018.
Oliphant, T. E. Python for scientific computing. Computing in Science & Engineering, 9(3):10-20, 2007. doi: 10. 1109/MCSE.2007.55
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., and Wayne, G. Experience replay for continual learning. In NeurIPS, pp. 348-358, 2019.
van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double q-learning. In AAAI, 2016.
Schaul, T. Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn. arXiv preprint, arXiv:2206.00730, 2022.
van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad. arXiv preprint, arXiv:1812.02648, 2018.
Schneider, J., Schumacher, P., Guist, S., Chen, L., Häufle, D. F. B., Schölkopf, B., and Büchler, D. Identifying policy gradient subspaces. arXiv preprint, arXiv:2401.066 1.66044 2024.
Van Rossum, G. and Drake Jr, F. L. Python reference ual. Centrum voor Wiskunde Informatica Amsterdam, 1995.
Wang. L.. Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint, arXiv: (el.1777.06666 063,06637770 2017.
Schwarz, J. Czarnecki, W., Luketina, J., Grabska- Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. Progress & compress: A scalable framework for con- tinual learning. In ICML, volume 80, pp. 4544-4544, 2018.
Wu, T., Luo, L., Li, Y.-F., Pan, S., Vu, T.-T., and Haffari, G. Continual learning for large language models: A survey. arXiv preprint, arXiv:2402.01364 2024.
Xie, A., Harrison, J., and Finn, C. Deep reinforcement learning amidst lifelong non-stationarity. In ICML, 2021.
Schwarzer, M., Ceron, J. S. O.,, Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In ICML, pp. 30365-30380. 2023.
Yang, Y., Zhang, G., Xu, Z., and Katabi, D. Harnessing structures for value-based planning and reinforcement learning. In ICLR, 2020.
Young. K. and Tian, T. Minatar: An atari Mtari-inspired testbed for more efficient reinforcement learning experiments. arXiv preprini, trXiv:1903.0341 2019.
Shah, H., Tamuly, K., Raghunathan, A.. Jain, P., and Netra- palli, P. The pitfalls of simplicity bias neural networks. In NeurIPS, 2020.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, Gradient surgery for multi-task learning. In NeurlPS, 2020.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. A. Deterministic policy gradient algorithms. In ICML,, 2014.
Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, Y. J., and Ma, Y. Investigating the catastrophic forgetting in multi- modal large language model fine-tuning. In Conference on Parsimony and Learning, pp. 202-227. PMLR, 2024.
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon in deep reinforcement learning. In ICML. pp. 32145-32168, 2023.
12

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Zhang, W., Li, Y., Yang, B., and Lu, Z. Tackling non- stationarity in reinforcement learning via causal-origin representation. In ICML, volume 235, pp. 59264-59288, 2024.
13

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
A. Experimental Details
A.1. Continual RL
Environment Setups We follow the setups in (Muppidi et al., 2024). For Gym Control, we use four environments: CartPole-vI, Acrobot-v1, LunarLander--2 and MountainCar-vO. For each environment, task sequence T is built by chaining instances of the environment with a unique Gaussian observation noise e; ~ NN(,, o²) sampled once for each. The number & is 10 and the interaction budget N is O. 1.6 for all the environments, except that k is and N is 0.32M for MountainCar-vO. For CartPole-vI, Acrobot-vl 1 and LunarLander-v2, we use 2.0 provided in (Muppidi et al., 2024). For MountainCar-vO, it was not originally included in (Muppidi et al., 2024) and we found Ø = 2.0 is too large for this environment to be learnable. Therefore, we use o 0.02 for MountainCar-vO.
For ProcGen, we use all sixteen environments in the suite, while only four (i.e., Starpilot, Fruitbot, Chaser, Dodgeball) were adopted in (Muppidi et al., 2024). For each environment, the task sequence consists of 5 instances procedurally generated by sampling a unique level of the environment. The budget is 2M steps per task.
Algorithm Implementation We use the official implementation of TRAC (Muppidi et al., 2024) our code base. We implement all other methods to ensure that they only differ in the corresponding algorithm features. For the hyperparameters of the PPO baseline agent, we use the default values recommended in the code base and keep them consistent across all the methods. For the hyperparameters specific to each method, we search around the recommendation values in the original papers and report the best.
For C-CHAIN, we follow the practice in (Tang & Berseth, 2024) and we only reduce churn for the policy network of the PPO agent. The coefficient A of the policy churn reduction regularization is determined by an auto-adjustment mechanism, which keeps consistent relative scale (denoted by (3) between the churn reduction regularization terms and the original DRL objectives. Specifically, by maintaining the running means of the absolute PPO loss |xx and the policy churn reduction term LLpcl. the churn reduction regularization coefficient Aa is computed dynamically as Ay. = 3 [Lvel For the size of Biets we did the experiments with different batch sizes on continual Gym control tasks. Our findings were that using 2x. 4x or 8x batch size for Be (compared to the regular training batch size) sometimes improved the learning performance, but not consistently. To some degree, increasing the batch size of Ber acts similarly to increasing the regularization coefficient, as both of them reduce more churn. Thus, we did not search for the best batch size for Bud and set it equal to the training batch size alleviate the hyperparameter choice burden.
The hyperparameters are provided in Table 5. The codebase will be released upon publication of this work.
Table 5, Hyperparameters of different methods used in continual Gym Control and ProcGen environments. We use 1-1 to denote the 'not applicable" situation.
Agent
Hyperparam
Gym Control
ProeGen
Learning Rate
le
le Adam
Optimizer
Adam
Discount Factor (7)
0.99
0.99
GAE Parameter (A)
0.95
0.95
Vanilla PPO (Schulman et al., 2017)
Training Interval
800 steps
1000 steps
Mini-batch Size
32
125
Update Epoch
5
Clipping Range Parameter (e)
0.2
0.2
Entropy Loss Coef
0
0.01
TRAC (Muppidi et al., 2024)
(Hyperparam Free)
Weight Clipping (Elsayed al., 2024b)
Clipping Range
Best in (0.1, 1, 10)
Best in (0.1, 0.5, 1, 10)
1.2 Init (Kumar et al., 2023h)
Regularization Coeff
Best in (0.1, 1, 10, 100, 1000)
Best in (0.1, 1, 10, 100, 1000)
C-CHAIN (Ours)
Target Relative Loss f for Auto A,
Best (1000, 1e4, 1e5)
Best in (1000, le4, 1e5)
Compute Resource For the continual Gym Control and ProcGen experiments, we allocate single V100 GPU, 16 CPUs and 32GB memory for 4 jobs, typically running for around hours for Gym Control and 20 hours for ProcGen to
/https://gi thub. com wwwp//////aaaaai//oooocootcoooct onal Rabot is TRAC
14

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
complete.
A.2. Continual Deep.Mind Control Suite
We use the public implementations of PPO and DMC task setups in CleanRI² as our codebase. The actor and critic networks are two-layer MLPs with 256 units for each layer. We made no change the network structure, recommended hyperparameter choices, etc. The hyperparameters are in Table 6.
Upon the code base, we build three continual RL settings in with DMC tasks:
Continual Walker: run Walker-stand, Walker-walk, Walker-run, sequentially.
Continual Quadruped: run Quadruped-walk, Quadruped-run, Quadruped-walk, sequentially (repeat because only two Quadruped tasks are available in DMC).
Continual Dog: run Dog-stand, Dog-walk, Dog-run, Dog-trot, sequentially.
We run IM steps for each task.
Table 6. Hyperparameters of PPO and C-CHAIN used in continual DMC environments, The values of conventional hyperparameters are taken from the recommended values CLeanRL.
PPO Hyperparameters
Learning Rate Training Interval Discount Factor (?)
3c
2048 steps
0.99
GAE Parameter (A)
0.95
Num. of Minibatches
32
Update Epoch
10
Clipping Range Parameter (e)
0.2
0.5 for Continual Walker
Target Relative Loss |3 for Auto Xx
0.05 for Continual Quadruped and Continual Dog
Similarly, we compare PPO Oracle, PPO Vanilla, and C-CHAIN in the three settings. The results are summarized in Table and Figure shows the learning curves. We can observe that C-CHAIN PPO significantly outperforms PPO all three settings, showing the effectiveness for continual control. C-CHAIN also outperforms PPO Oracle because it does reset for each task and thus transfer the learned skill from previous tasks to future ones (e.g., MognStand to hog-Walk).
A.3. Continual MinAtar
We use the official code of MinAtar paper* codebase. For DoubleDQN, modify the DQN implementation provided in the official MinAtar code with change the network structure, recommended hyperparameter choices, etc. We use DoubleDQN the base agent, and apply C-CHAIN to the training of the value network, with modification to the hyperparameters of DoubleDQN. The byperparameters are summarized in Table 7.
To build the continual MinAtar setting, we chain three tasks: SpaceInvaders, Asterix, Seaquest, by padding the observation space to be [10, 10, 10). We run 1.5M steps for each task, i.e., 4.5M in total.
We compare DoubleDQN (i.c., Vanilla) and C-CHAIN DoubleDON with 12 random seeds. The results are shown in Table and Figure 10 shows the learning curves. The results show that C-CHAIN also improves the continual learning performance upon DoubleDQN in a sequence of totally different tasks. This indicates that C-CHAIN is not only beneficial to PPO in continual RL but has the potential to be general and easy-to-use remedy to different off- off-th RI. algorithms in combating plasticity loss in continual RL scenarios.
Shttps: thub. com cww.c/mmc////// comm/wxxyjj//clean *htpps: // thub. com com/keejjyungg/Mina.. MAttar
15

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
PPO (Vanilla)
PPO (Oracle)
Episode Return
C-CHAIN
Timestep (1e4)
(a) Continual Walker
PPO (Vanilla)
500
(Oracle) C-CHAIN
500-
Timestep (1e4)
(b) Continual Quadruped
PPO (Vanilla)
400
PPO (Oracle)
Episode Return
C-CHAIN
300
200
250
Timestep (1e4)
Continual Dog
Figure Learning curves PPO and C-CHAIN PPO in continual DMC. curves and shades are means standard error over twelve seeds.
16

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Table Hyperparameters of DoubleDQN and C-CHAIN used in continual MinAtar environments. The values of conventional hyperpa- rameters are taken from the recommended values in (Young & Tian, 2019).
DOu Hyperparameters
Learning Rate
3e 4
Training Interval
step
Discount Factor (?)
0.99
Hard Replacement Interval
1000 steps
Replay Buffer Size
0.5M
Batch Size
32
Initial e
1.0
End e Decay Steps
0.5M
Initial Random Steps
10k
Target Relative Loss 3 for Auto Xy
0.01
DDQN (Vanilla)
Episode Return
80-
C-CHAIN ($=0.05)
C-CNAIN ($=0.01)
80-
Timestep (5e4)
Figure 10. Learning curves for DoubleDQN and C-CHAIN in continual MinAtar. The curves and the shades are means and standard enrors twelve seeds.
A.4. Continual Supervised Learning
We follow the settings in L2 Init (Kumar et al., 2023b) and adopt RandomLabel-MNIST and Permuted-MNIST as our testbed. For HundomLabel-MNNSS each task in the task sequence is classification learning task with 1200 images randomly sampled from the MNIST dataset and random labels assigned to each individual image. The task sequence consists of 50 tasks and we use train the agent for a total of 400 epochs per task. For Permuted-MNIST, each task is a classification a Permuted MNIST task is
learning task with 10000 images sampled from MNIST. Different from characterized by applying a fixed randomly sampled permutation to the input pixels of all the images. The task sequence consists of 500 tasks and the agent is trained for one epoch per task.
We use the official implementation of L2 Init as the code base and implement C-CHAIN and Weight Clipping with simple modifications. We make no modifications to common hyperparameters and use the recommended values in the code base.
The results are summarized in Table 8. We can observe that C-CHAIN improves the vanilla agent but does not perform on par with L2 Init and Weight Clipping- This shows that the efficacy of C-CHAIN is relatively limited in the two continual supervised learning enviromments, especially in contrast to its superiority in the continual RL experiments. We hypothesize this is because RL suffers more from churn accumulation and thus more severe plasticity loss due to the chain effect of churn (Tang& & Berseth, 2024). Besides, Permuted -NIIST and LandomLabel-MNISS are simple, where the agent can find good solution the initialization. We expect evaluate C-CHAIN in more complex continual supervised learning environments in the future.
B. Empirical NTK Analysis Details
During the learning process, we collect the empirical NTK matrices in each training iteration of the PPO agent. Specifically, for the interaction experience collected within each iteration, we divide into m mini-batches randomly (m = 100 for our NTK analysis in ProcGen). Then, for each mini-batch, we compute the gradient of the policy network's parameters 0 regarding the PPO policy optimization objective, denoted by 9, with E 21,2...., m). The empirical NTK matrix No
Shttps://gg thub. com wa..ummr9876/1/0 9876/1.2_66111
17

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
Table Performance comparison in continual supervised learning environments. The reported scores are the average performance (Equation 1) in terms of accuracy with mean and standard error over three seeds.
Algorithm
RandomLLabe-MMIIST
Permuted-MNIST
Vanilla C-CHAIN
0.1501 0.0030
0.6430 1 0.0029
0.2482 i 0.0141
0.6797 1 0.0042
L2 Init
0.8607 0.0021
0.8039 0.0027
Weight Clipping
0,4304 + 0.0052
0.8281 + 0.0001
then computed by following the definition in Equation 2: No(i, Di dj for i, (1, m). With the empirical NTK matrix, statistics for the off-diagonal values (i.c., Figure 6, below and Figure 11, middle) and the diagonal values (i.e., Figure 11, below) can be computed conveniently.
For the computation of approximate rank, we follow the prior convention in (Yang et al., 2020; Kumar et al., 2021). For a matrix N, let {or (N)} be the singular values of it arranged in decreasing order, i.c., 01 2 .... ad 2 0. the approximate rank regarding threshold 5 is defined as sranks = min : Σ (: 2 (1 - 5) Σ (NV)). In this paper, we use 5 = 0.01. In other words, this means the approximate rank is the first & singular values that hold more than 99% information of all singular values. Accordingly, we use this computation for the results of approximate rank in Figure (above) and Figure 1I (above).
Apart from our analysis on empirical NTK shown in Figure 6, we provide additional NTK analysis by including TRAC in comparison and also showing the summation of the diagonal values of empirical NTK matrix. The results shown in Figure 11. C-CHAIN not only suppresses the off-diagonal values to decorrelate gradients, but also suppresses the diagonal values to prevent the increased scale of gradient, which is also deemed to be pathology of neural network. Moreover, we found TRAC lies between C-CHAIN and Vanilla. This indicates that the NTK statistics used our work also explain the efficacy of TRAC.
18

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
80-
Approx. Rank
70-
50
40
PPO (Vanilla)
30
TRAC
20
C-CHAIN
10
200
Timestep (2e4)
Sum of Abs. diag (1e4) Sum of Abs. Off-diag (1e5)
5-
100
200
300
400
500
Timestep (2e4)
100
200
400
500
Timestep (2e4)
Figure Analysis on NTK matrix terms approximate rank (above), the sum of absolute off -d-iggnnnal values (middle), and sum of absolute diagonal values(below) in continual ProcGen Starpilot.
19

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
C. Complete Learning Curves
The learning curves for the missing twelve contiunal ProcGen environments are provided in Figure 12, Figure 13 and Figure 14.
fruitbot
Episode Return
200
400
600
800
1000
Timestep (2o4)
chaser
PPO (Oracle)
Episode Return
PPO (Vanilla)
TRAC
C-CHAIN
Weight Clipping
L2 Init
Timosstop (204) bigfish
17.5
Episode Return
15.0
12.5
10.0
7.5
5.0
2.5
0.0
200
400
600
1000
(2e4)
caveflyer
Episode Return
10
200
400
600
800
1000
Timestep (2e4)
Figure Learning curves of different methods in four (of sixteen) continual ProcGen environments: fruitbot, chaser, bigfish, caveflyer. The curves and the shades are means and standard errors over six seeds.
20

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
climber
10.
Episode Return
uw
Timeston ninja
5.
PPO (Oracle)
Episode Return
PPO (Vanilla)
TRAC
C-CHAIN
Weight Clipping
2.
L2 Init
200
800
1000
Timestep (2e4)
jumper
Episode Return
6
3
2
sw mmm
um
200
400
600
800
1000
Timestep (2e4) heist
Episode Return
8.
200
400
600
800
1000
Timestop (2e4)
Figure 1.3. Learning curves of different methods four (of sixteen) continual PracGen environments: climber, ninja, jumper, heist. The curves and shades are means and standard errors over seeds.
21

---

Mitigating Plasticity Loss in Continual RL by Reducing Churn
leaper
Episode Return
2
Timesteo (2e4) maze
Episode Return
PPO (Oracle)
PPO (Vanilla)
w
TRAC
C-CHAIN
Weight Clipping
L2 Init
200
400
600
800
1000
Timestep (2x4) plunder
Episode Return
20
25-
ww sarn
New
mm mnn
10-
Woot
200
400
600
800
1000
Timestep (2ed) bossfight
Episode Return
10
8
6
200
Timestep (2e4)
Figure 14. Learning curves of different methods in four (of sixteen) continual ProeGen environments: leaper, maze, plunder, hossfight. The curves and the shades are means and standard errors over six seeds.
22


Reviews:
[{"review_id": "UVDGpA6MBO", "reviewer": "Reviewer_mKWp", "summary": "The manuscript investigates the loss of plasticity in continual reinforcement learning (CRL) from the perspective of churn. The authors establish a connection between plasticity loss and churn through the Neural Tangent Kernel (NTK) framework, demonstrating that churn exacerbation correlates with the rank decrease of the NTK matrix. To address this, the authors propose the Continual Churn Approximated Reduction (C-CHAIN) method, which reduces churn during training, mitigating plasticity loss. Empirical results on OpenAI Gym Control and ProcGen benchmarks show that C-CHAIN outperforms baseline methods in most environments. The manuscript also includes theoretical analyses, experimental evaluations, and discussions on the broader implications of churn reduction in continual learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s valuable comments and the recognition of our method and experiments.  Our response aims to address these aspects in detail.\n\n> On the experimental setups, task difference, and additional CRL setups\n\n**[Clarification on environment choice and task difference]** \n\nWe **followed the experimental setting in TRAC paper** (Muppidi et al., NeurIPS 2024) and took it as a major baseline. For a more comprehensive experimental comparison, we **added MountainCar (where TRAC fails almost totally)** to the continual Gym control suite, and we **extended the 4 ProcGen tasks used in TRAC to all 16 tasks** provided by ProcGen suite.\n\nThese tasks are representative CRL scenarios with **semantically related tasks**, and they are challenging to the vanilla PPO agent in the sense that apparent degradation occurs as the learning goes on (for most of them).\n\n**[Additional experimental setups in DMC]**\n\nAs suggested by the reviewer, instead of chaining totally different Atari games (Abbas et al., 2023), we **additionally established two continual RL settings in DeepMind Control (DMC) suite**:\n- **Continual Walker**: chain Walker-stand, Walker-walk, Walker-run\n- **Continual Quadruped**: chain Quadruped-walk, Quadruped-run, Quadruped-walk (repeat because only two Quadruped tasks are available in DMC).\n\nSimilarly, we compare PPO Oracle, PPO Vanilla, and C-CHAIN in the three settings. We run 1M steps for each task, i.e., 3M in total for each setting. The results are averaged over 12 random seeds for each configuration, as shown below (the corresponding learning curves will be added to our revision).\n\n| Method/Task   | Walker/Stand-Walk-Run | Quadruped/Walk-Run-Walk |\n|--------------|--------------|--------------|\n| PPO (Oracle) | 395.971 $\\pm$ 8.116 | 234.529 $\\pm$ 19.193 | \n| PPO (Vanilla) | 305.199 $\\pm$ 18.519 | 250.153 $\\pm$ 26.385 |\n| C-CHAIN | 472.828 $\\pm$ 17.865 | 314.510 $\\pm$ 44.321 |\n\nWe can observe that **C-CHAIN improves PPO in continual DMC**. Actually, from the learning curves (will be added in the revision), we found PPO learns more slowly with a decreased slope in the second and the third task, and C-CHAIN learns faster and achieves higher scores. \n\n> On the choice of baseline methods\n\n**[Existing evidence in prior work]** \n\nFor the choice of other baseline methods, as we mentioned in Line 315-325, TRAC outperforms Concatenated ReLU (Abbas et al., 2023), EWC (Schwarz et al., 2018), Modulating Masks (Nath et al., 2023) in their paper, as well as LayerNorm (Lyle et al., 2024) in Figure 15 of TRAC paper (i.e., LayerNorm Adam). Besides, more related methods (e.g., ReDo) are compared with L2 Init (i.e., Regen Reg) in ProcGen CoinRun in Figure 5 of (Juliani and Ash, NeurIPS 2024). The results show that L2 Init performs comparably with ReDo and LayerNorm.\n\n**[Additional results for three more baselines and reliable aggregate evaluation]** Please refer to the response to Reviewer cU5H, due to the space limitation.\n\n> On a more precise mathematical formulation of the connection between churn, NTK rank, and plasticity loss\n\nIn this paper, we are more focused on using formal expressions to describe, analyze and dig out intuitive insights. To give a thorough and rigorous theory, we need (at least) definitions and assumptions from three aspects:\n- **[RL-oriented definition of plasticity]** To our knowledge, the existing formal definition of plasticity (i.e., Target-ﬁtting capacity, Definition 1 in Clare et al., 2022) is based on a general family of objective functions, which is too vague and broad for RL. This is why a direct plasticity loss metric was not used in our paper, and instead we used the rank loss of NTK since it is the best empirical practice in the literature (Clare Lyle et al., 2024). One possible RL-oriented definition could be developed upon the Value Improvement Path (Definition 1, Dabney et al., 2021).\n- **[Proper assumptions on deep RL]** A Rigorous theoretical analysis on deep RL learning process is challenging as we need concrete assumptions on network structure, optimization, objective function. Theoretical analysis methods and results under practical assumptions are still lacking in deep RL.\n- **[Proper assumptions on continual learning setting]** In the literature of continual RL, task sequence and switch scheme are usually manually designed for experimentation, little formal definition and assumption of task distribution and task switch have been established so far.\n\nDue to the lack of theoretical foundation above, providing a thorough and rigorous theory is out of the scope of this paper. But as suggested, we will update the discussion in Section 4 to provide more formal support and additional discussions for the connections between churn, NTK and plasticity.\n\n> On the results for the tasks like Leaper\n\nPlease refer to the response to Reviewer 65iv.\n\n> **Other remaining questions**\n\n**Due to the space limitation, we will provide them during the discussion stage.**", "claims_and_evidence": "The manuscript provides empirical evidence supporting its claims, particularly in demonstrating the efficacy of C-CHAIN in mitigating plasticity loss and improving performance in CRL tasks. However, the connection between plasticity loss, churn, and the NTK matrix rank is not established with sufficient mathematical rigor. For example, while Section 4.2 discusses the interplay between these factors in CRL, the arguments lack precise mathematical formulations. Additionally, the manuscript does not clearly delineate whether the theoretical insights are specific to reinforcement learning or applicable to other continual learning paradigms.", "methods_and_evaluation_criteria": "The proposed C-CHAIN method is well-motivated and aligns with the problem of mitigating plasticity loss in CRL. However, the experiments focus on environments with relatively small task differences, limiting the generalizability of the findings. A broader evaluation across task sequences with greater variability would strengthen the conclusions", "theoretical_claims": "The manuscript provides theoretical insights into the relationship between churn, NTK rank, and plasticity loss. However, the theoretical claims are not fully substantiated. For instance, the discussion in Section 4.2 is somewhat vague, and additional mathematical clarity—such as explicit equations or proofs—would enhance the credibility of the theoretical framework.", "experimental_designs_or_analyses": "The experimental design is generally sound, with a focus on evaluating C-CHAIN on 20 environments in Gym Control and ProcGen benchmarks. The results demonstrate the method's effectiveness in most environments, particularly those with dynamic task difficulties. However, the experiments do not include task sequences composed of diverse environments, which would better illustrate the method's ability to handle significant plasticity loss. The authors should also clarify the rationale behind the choice of baseline methods.", "relation_to_broader_scientific_literature": "The manuscript is well-positioned within the broader literature on continual reinforcement learning and plasticity loss. It builds upon prior work on nonstationarity, catastrophic forgetting, and the loss of plasticity in neural networks. The discussion of related work is thorough.", "essential_references_not_discussed": "The manuscript adequately cites most key references in the field.", "other_strengths_and_weaknesses": "Strengths:\n\nThe manuscript explores an important problem in CRL, providing both theoretical insights and practical solutions.\nThe proposed method is novel and demonstrates strong empirical performance in most tested environments.\nThe experimental evaluation is extensive, covering a wide range of environments and including analysis studies.\n\nWeaknesses:\n\nThe theoretical analysis lacks mathematical rigor, particularly in establishing the connection between churn, NTK rank, and plasticity loss of CRL.\nThe experimental results are limited to environments with relatively small task differences, reducing the generalizability of the findings.\nCertain figures, such as Figure 1, lack clear explanations, which detracts from the clarity of the manuscript.", "other_comments_or_suggestions": "Improve the clarity of Figure 1 by providing a detailed explanation of its purpose and implications.\n\nIntroduce a direct metric for quantifying plasticity loss to better evaluate the effectiveness of C-CHAIN.\n\nExpand the experimental evaluation to include task sequences with greater variability and diversity.\n\nClarify whether the theoretical insights are specific to CRL or applicable to other continual learning paradigms.", "questions_for_authors": "Can the authors provide a more precise mathematical formulation of the connection between churn, NTK rank, and plasticity loss? How does this connection specifically apply to reinforcement learning?\n\nAre the theoretical insights on churn reduction in CRL applicable to supervised continual learning? If so, can the authors provide evidence or discussion to support this claim?\n\nHow does C-CHAIN address the trade-off between plasticity and stability in continual learning? Does it mitigate catastrophic forgetting while improving plasticity?\n\nCan the authors evaluate C-CHAIN on task sequences composed of diverse environments, similar to those used in \"Loss of Plasticity in Continual Deep Reinforcement Learning\" (2023)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vFsK2PrL6b", "reviewer": "Reviewer_wByN", "summary": "This paper studies the loss of plasticity in the continual reinforcement learning problem. The authors present a method that is based on reducing the churn to help prevent the collapse of the NTK rank. Through a series of experiments, the paper shows the effectiveness of the proposed method against other baselines.\n\\\n\\\n\\\n**Update after rebuttal:** I thank the authors for their effort to improve their empirical evaluation. The new results with more runs and also with more baselines increase my confidence in the paper's conclusion. Thus, I raised my score to reflect that change.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s valuable comments and the recognition of the importance of the topic studied in our paper. \n\n> On the reviewer’s comments including “the empirical evaluation doesn't show conclusive results”, “the improvement is not consistent”, and “little insight is provided to explain it”\n\n**[Additional reliable aggregate evaluation]** \n\nTo obtain a conclusive evaluation, we provide the aggregation evaluation by using Reliable Metrics (Agarwal et al., NeurIPS 2021). The evaluation uses Mean, Median, Inter-Quantile Mean (IQM) and Optimality Gap, with **95% confidence intervals and 50000 bootstrap replications** (recommended by Reliable Metrics official github implementation). It aggregates over **9 methods, 6 seeds for each of 16 ProcGen tasks**, i.e., **864 runs** with 10M steps for each. \n\nThe results are shown in the table below (the corresponding plot will be added to the revision). We can observe that **C-CHAIN performs the best regarding all four metrics** and **outperforms the second-best with no overlap of Confidence Intervals for Median, IQM, and a minor one for Mean**.\n\nDue to the space limitation, **please refer to the concrete markdown table we will upload in the common response or the discussion response after the rebuttal deadline**.\n\n**[On the consistency of improvement]** \n\nAs summarized by Table 1, our method **C-CHAIN (PPO) consistently outperforms vanilla PPO in all 20 envs**. And **C-CHAIN performs the best in 15 out of 16 tasks** (and performs the second-best in the remaining one) on continual ProcGen benchmark. In this sense, our method achieved overall consistent improvement.\n\n**[On the insights and explanations]**\n\nFor continual CartPole and continual LunarLander, where L2 Init outperforms C-CHAIN, we provided the possible explanation in Line 328-365. Our insight is, for simple tasks like CartPole and LunarLander where the agent can find a good policy near parameter initialization, L2 Init performs the best; in contrast, it limits the policy learning as in continual ProcGen where L2 init is outperformed by C-CHAIN in most tasks.\n\n\n> On the experimental design\n\nWe **followed the experimental settings used in TRAC** (as also mentioned by the reviewer’s comment “The authors selected standard benchmarking tasks accepted by the community to study loss of plasticity”), and we further **extended the ProcGen tasks from the four ones used in TRAC to all 16 tasks, as well as continual MountainCar where TRAC collapses**.\n\n**[On the task diversity and the varying difficulties]** \n\nSince the tasks are different in game logic, reward density, visual complexity, and etc., they provide a range of environments of diversity and varying difficulties, which is also a commonly adopted principle of designing a benchmark. Therefore, we think it is natural for Oracle PPO or vanilla PPO to have different levels of performance across different tasks.\n\n\n> On the math derivation\n\n**[How did both $\\nabla_{\\theta} f_{\\theta}(\\bar x)$ and $\\nabla_{\\theta} f_{\\theta}(x)$ become $G_{\\theta}$?]**\n\nWe refer the reviewer to the definition of the NTK matrix (i.e., the first line for entry definition and the second line for matrix definition in vector form) in Equation 2.\n\nAs in the second line of Equation 4, $\\nabla_{\\theta} f_{\\theta}(\\bar x)^{\\top} \\nabla_{\\theta} f_{\\theta}(x)$ becomes $N_{\\theta}(\\bar x, x)$ by the definition in Equation 2. Then, Equation 5 is the vector form of Equation 4, correspondingly, the entry definition $N_{\\theta}(\\bar x, x)$ becomes $N_{\\theta} = G_{\\theta}^{\\top}G_{\\theta}$ as in Equation 2.\n\n\n**[The transition from Equation 4 and Equation 5, about the diagonal $S$ matrix]**\n\nAs mentioned above, Equation 5 is the vector form of Equation 4. Therefore, the sampling $x \\sim B_{train}$ is replaced by the $S$ matrix as below in Equation 5. More specifically, $S$ has the same size as $N_{\\theta}$, i.e., $|X|$ by $|X|$. As mentioned in Line 155-157, $S$ is a diagonal matrix, and its diagonal is ${0,1}$-binary with 1 for the sampled data and 0 for the non-sampled.\n\nWe appreciate the reviewer for pointing this out. We will make this clearer as suggested by the reviewer.\n\n> On the evaluation of C-CHAIN in continual supervised learning\n\nWe provided two explanations in Section 5.3. **A significant difference** to note is that **RL suffers from the chain of churn**, formally characterized in (Tang and Berseth, NeurIPS 2024), which stems from the iterative policy improvement and policy evaluation nature (i.e., the Generalized Policy Iteration paradigm), **while SL does not** as it learns from static labels. \n\nThis means that the exacerbation of churn during the process of NTK rank loss on both the policy network side and the value network side can further (negatively) affect each other due to the chain effect. Since C-CHAIN is proposed from the perspective of churn, this explains to some extent why C-CHAIN is not that effective for CSL.", "claims_and_evidence": "- The reason why C-CHAIN performs poorly under the simpler setting of continual supervised learning is not convincing. \n- The experimental design, where the tasks are not on the same level of complexity (shown by the inconsistent performance of Oracle PPO), makes empirical evaluation hard. \n- The authors compare their methods against other baselines and show their effectiveness. However, the improvement is not consistent across environments and domains, and little insight is provided to explain it.\n- Most learning curves have overlapping confidence intervals, which compromises the statistical significance of the results.", "methods_and_evaluation_criteria": "The authors selected standard benchmarking tasks accepted by the community to study loss of plasticity. In addition, they used empirical NTK, which gained popularity when investigating plasticity.", "theoretical_claims": "I haven’t closely checked the theoretical claims or the math.", "experimental_designs_or_analyses": "The empirical evaluation doesn't show conclusive results since, in many experiments except for a few, there are overlapping confidence intervals. I suggest the authors would increase the number of independent runs to convince the reader of the validity of their approach.", "relation_to_broader_scientific_literature": "The paper studies continual RL, which is a very important problem, and the results are relevant to a large number of researchers, especially since it’s building on the concept of churn and its relationship to plasticity, which has been studied before. The empirical evaluation itself doesn’t provide conclusive results, but the analysis given might be helpful for future research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "There are missing steps between Equation 4 and Equation 5, namely about the diagonal $S$ matrix. Also, how did both $\\nabla_\\theta f_\\theta(\\bar{x})$ and $\\nabla_\\theta f_\\theta(x)$ become $G_\\theta$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atpcoUlasc", "reviewer": "Reviewer_cU5H", "summary": "The paper investigates the loss of plasticity issue in the continual deep reinforcement learning setting from the lens of churn (\"undesirable generalization\", empirical excess out-of-training-batch variation).\nThey claim that \n1) loss of plasticity and high churn are connected: the decrease of the NTK matrix rank indicates high churn and had previously been shown to correlate with plasticity loss.\n2) addressing churn has a positive effect on the decorrelation of the NTK, and serves as an implicit adaptive step size\n3) their proposed C-CHAIN effectively addresses churn in the continual RL setting, shows the benefits of addressing churn on the NTK empirically, and validates the connection between churn and plasticity.\n\n\n## update after rebuttal\nI have read the other reviews and comments from the authors, and I maintain the score of my review.\nThe authors have agreed to discuss the two additional papers I have cited in the references section, to make the presentation modifications about the figure illustrating the connection between NTK and churn, and added new baselines and experiments.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s valuable comments and recognition. Our response aims to add more discussion and clarification on the points mentioned in the inspiring comments. \n\nIn addition, we provide additional experimental results for AdamRel [1], ReDo and LayerNorm, along with Reliable Metrics (Agarwal et al., NeurIPS 2021). Besides, we provide two additional continual RL experimental setups in DMC. \n\n> On the data distribution regarding the expectation in Eq. 4 and $S_x$\n\nWe appreciate the reviewer for pointing this out. Yes, $S_x$ denotes the stochastic variable of sampling $B_{train}$, which should be defined along with a sampling distribution. For illustrative purposes, using a uniform distribution is sufficient. And we expect to take the concrete form of the sampling distribution (intuitively, it should depend on factors like on-policy learning/off-policy learning, exploration policy, experience replay strategy, etc.) into the analysis in the future. We will clarify this point as suggested.\n\n\n> On the mentioned related [1], [2], [3]\n\n**[Additional Discussions]** \n\nFor some discussions on [2] and [3], the Proximal Feature Optimization (PFO) regularization proposed in [2] is closely related to DR3 (Kumar et al., ICLR 2022) and CHAIN (Tang and Berseth, NeurIPS 2024). This is because that the feature difference can be viewed as the gradient difference when the network is viewed as a linear approximation i.e., $\\pi(s) = \\phi(s) W$ as in DR3; in another view, regularizing feature difference should share some overlapped effects with regularizing network-output difference as in CHAIN.\n\nAnd for [3], we found the experimental results for ProcGen CoinRun by the Figure 5 in their paper provide a useful reference of the learning performance of other related methods not included in our submission version. For example, ReDo and LayerNorm perform comparably with Regen reg (i.e., the L2 Init baseline method adopted in our paper).\n\n**[Additional Experimental Results]** \n\nWe added **AdamRel [1], ReDo and LayerNorm** to our experimental comparison in 16 ProcGen tasks. The aggregate comparison is shown below. We can find ReDo and LayerNorm perform comparably with L2 Init, which is basically consistent with the evidence in existing papers.\n\n| Method   | PPO Oracle | PPO Vanilla    | TRAC      | **C-CHAIN**  | LayerNorm | ReDo| AdamRel | Weight Clipping  | L2 Init | \n|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|\n| ProcGen Agg. | 70.789       | 55.049       | 77.289         | **101.792**      |  75.164 | 61.180 | 61.443  | 57.092       | 72.961    |\n\n\nMoreover, we provide the aggregation evaluation by using **Reliable Metrics** (Agarwal et al., NeurIPS 2021). The evaluation uses Mean, Median, Inter-Quantile Mean (IQM) and Optimality Gap, with **95% confidence intervals and 50000 bootstrap replications** (recommended by Reliable Metrics official implementation). It aggregates over **9 methods, 6 seeds for each of 16 ProcGen tasks**, i.e., **864 runs** with 10M steps for each. \n\nThe results are shown in the table below (the corresponding plot will be added to the revision). We can observe that **C-CHAIN performs the best regarding all four metrics** and **outperforms the second-best with no overlap of confidence intervals for Median, IQM, and a minor one for Mean**.\n\n| Method | Median | IQM | Mean | Optimality Gap (lower is better) |\n|--------------|--------------|--------------|--------------|--------------|\n| PPO (Oracle) | 1.000 (0.960, 1.036) |  0.969 (0.933, 1.010) |  1.000 (0.908, 1.130) |  0.128 (0.098, 0.155) | \n| PPO (Vanilla) | 0.835 (0.728, 0.925) |  0.781 (0.722, 0.841) |  0.811 (0.704, 0.949) |  0.300 (0.252, 0.348) | \n| TRAC | 0.977 (0.894, 1.130) |  0.999 (0.887, 1.103) |  1.017 (0.891, 1.166) |  0.255 (0.205, 0.305) | \n| **C-CHAIN** | 1.452 (1.287, 1.522) |  1.388 (1.298, 1.472) |  1.461 (1.291, 1.719) |  0.086 (0.058, 0.110) | \n| Weight Clipping | 0.889 (0.759, 0.948) |  0.822 (0.752, 0.889) |  0.859 (0.744, 0.996) |  0.275 (0.223, 0.328) | \n| L2 Init | 1.029 (0.981, 1.103) |  1.035 (0.987, 1.092) |  1.116 (1.016, 1.237) |  0.098 (0.067, 0.130) | \n| LN | 1.057 (0.996, 1.121) |  1.018 (0.952, 1.086) |  1.131 (0.976, 1.311) |  0.146 (0.107, 0.187) | \n| ReDO | 0.945 (0.815, 0.974) |  0.862 (0.821, 0.904) |  0.912 (0.810, 1.039) |  0.209 (0.169, 0.249) | \n| AdamRel | 0.905 (0.804, 0.957) |  0.860 (0.792, 0.925) |  0.899 (0.781, 1.037) |  0.250 (0.199, 0.301) | \n\n> On other advice \n\n**[The position of Figure 1]**\nWe are glad to know that Figure 1 helps to understand the NTK equations. We will move Figure 1 close to Equation 2 and 5 with additional explanations to improve the smoothness of the introduction of the NTK equations.\n\n**[The broader connection to optimizing neural networks]** \nWe appreciate the reviewer’s inspiring comments. We will discuss these broader connections in our revision.", "claims_and_evidence": "All claims are backed by enough theoretical or empirical justification.\n\n1) Connection between churn and plasticity through the NTK: \nThe connection between churn and the NTK is clearly presented and backed by a theoretical derivation. It is also shown empirically in the experiments.\nThe connection between the NTK and plasticity heavility relies on the results from Lyle et al. (2024), which has not been accepted to a peer-reviewed venue yet, which can be a limitation. However, the paper provides evidence of low-rank NTK matrices in the presence of plasticity loss, so the connection does hold independently of the motivation from Lyle et al. (2024).\n\n2) The effects of reducing churn on the NTK and the optimization:\nThis is well justified theoretically in section 4.\n\n3) The proposed C-CHAIN algorithm:\nThe algorithm is competitive with recently proposed methods and empirically shows the benefits of using churn on the NTK.", "methods_and_evaluation_criteria": "Overall, the choice of environments and baselines is good.\n\nThe ProcGen environments have been used in several previous papers on plasticity and are a valid benchmark for this paper.\nTo my knowledge, the Gym environments have not been used in work on plasticity before, but the non-stationary version introduced by the paper seems to provide a good testbed to validate their claims.\n\nThe use of mean performance is a valid evaluation metric for this work.\n\nThe baselines considered give a good understanding of the performance of the proposed C-CHAIN.", "theoretical_claims": "I checked all the derivations in the main paper.\n\nin eq 5, there does not seem to be a term corresponding to the data distribution (as there was an expectation in eq 4). I think it can easily be plugged into $S_x$. The authors can clarify this or say that for illustrative purposes, they assume a uniform distribution.", "experimental_designs_or_analyses": "There is enough statistical significance, and the hyperparameter optimization seems reasonable.\n\nThe analysis of the results does not make claims beyond what the results show, and the provided analysis backs up the claims made.\n\nShowing the limitations of C-CHAIN in the supervised learning setting is appreciated.", "relation_to_broader_scientific_literature": "Plasticity is an important problem in continual learning. This work proposes a solution to address it by addressing the dynamics of the NTK matrix,\nThe work discussed other work addressing plasticity loss with other methods; however, addressing the NTK explicitly can have a broader connection to optimizing neural networks, and these broader connections are not discussed.", "essential_references_not_discussed": "The essential literature has been discussed.\n\nNevertheless, the paper would benefit the community more by referencing/discussing the following works to give a broader presentation of\nthe methods addressing plasticity loss in PPO, with which the main results of the paper have been presented.\nI believe [1] has been discussed in the paper, although [2] and [3], which came earlier, have not.\n\n1. Ellis, Benjamin, et al. \"Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps.\"\n2. Moalla, Skander, et al. \"No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO.\"\n3. Juliani, Arthur, and Jordan Ash. \"A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning.\"\n\nAll in Advances in Neural Information Processing Systems 37 (2024): 113884-113910.", "other_strengths_and_weaknesses": "Everything is mentioned in the previous sections.", "other_comments_or_suggestions": "Figure 1 was critical for me to understand the equations using the NTK to relate to churn, as one is a symmetric matrix, and the other contrasts two different batches. I was confused by eq 5 until I saw the figure. I think it would make the paper much easier to grasp if the figure were introduced with eq 5.", "questions_for_authors": "No additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ME2O40aWvF", "reviewer": "Reviewer_65iv", "summary": "A recent line of research has highlighted a problem where standard deep-learning methods gradually lose plasticity (i.e., the ability to learn new things) in continual learning settings (Lyle et al., 2022; Dohare et al., 2024). This paper examines plasticity loss in deep continual reinforcement learning (RL) through the lens of churn—network output variability for out-of-batch data caused by mini-batch training (Schaul et al., 2022; Tang & Berseth, 2024). The authors identify correlations between plasticity loss and increased churn by studying the rank collapse of the Neural Tangent Kernel (NTK) matrix $N_\\theta$, which is defined as the matrix of gradient dot products between all data points (Lyle et al., 2024): $N_\\theta(i,j) = \\nabla_\\theta f_\\theta(x_i)^⊤ \\nabla_\\theta f_\\theta(x_j)$ for $x_i, x_j$. They empirically demonstrate that a rank collapse in the NTK matrix is a sympton of disrupted learning dynamics and hence leads to poor performance. To address this, they propose Continual Churn Approximated Reduction (C-CHAIN), a method to mitigate plasticity loss in settings where tasks switch every $N$ steps. They validate their approach with empirical results on four OpenAI Gym environments and 16 ProcGen tasks.\n\n\n**References**\n1. Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. In ICLR, 2022.\n2. Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026):768–774, 2024.\n3. Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn. arXiv preprint, arXiv:2206.00730, 2022.\n4. Tang, H. and Berseth, G. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. In NeurIPS, 2024.\n5. Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., & Dabney, W. (2024). Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> On other possible baseline methods, e.g., L2 regularization, LayerNorm (Lyle et al., 2024) continual backprop (Dohare et al., 2024) and ReDo (Sokar et al. 2023)\n\nIn addition to existing empirical evidence for LayerNorm, L2 Regularization and ReDo, we additionally implemented and ran LayerNorm, ReDo and AdamRel in our continual ProcGen experiments.\n\n**[Existing Evidence]** The comparison between LayerNorm and TRAC (i.e., our major baseline and codebase) in continual Gym control tasks can be found in Appendix D, Figure 15 of TRAC paper (Muppidi et al., NeurIPS 2024). Their results show TRAC outperforms LayerNorm (i.e., LayerNorm Adam) in their Figure 15. For L2 regularization, Lyle et al. (2024) mentioned “We found that batch normalization and L2 regularization interfere with learning” in their Section 4.2, RL evaluation. In our work, we did L2 regularization and we found it performed very similarly (slightly worse) to L2 Init (i.e., Regenerative Regularization).\n\nIn another paper (Juliani and Ash, NeurIPS 2024), their Figure 5, ReDo, LayerNorm and other related methods are compared with L2 Init (i.e., Regen Reg in the figure) and L2 norm in ProcGen CoinRun. The results show that L2 Init performs comparably with ReDo and LayerNorm and slightly outperforms L2 norm (which is consistent with our findings as mentioned above). \n\nFor continual backpropagation, it is similar to ReDo which uses a different recycling metric, and in Appendix C.4, Figure 25 of the ReDo paper, they achieved similar results. Therefore, we only consider ReDo here.\n\n\n**[Additional Experiments]** Please refer to the response to Reviewer cU5H and Reviewer mKWp.\n\n> Questions on the buffer \n\n**[On buffer flush and awareness of task switch]** \n\nIn our method, i.e., C-CHAIN PPO, both $B_{ref}$ and $B_{train}$ are sampled from the online interaction data collected by the policy in the current iteration (e.g., every 2048 interactions). Thus, it does not matter for C-CHAIN PPO whether the buffer is flushed after each task switch and the agent does not need to be aware of task switches.\n\n**[The impact of the size of $B_{ref}$]** \n\nWe did the experiments for different batch sizes for $B_{ref}$ on continual Gym control tasks. Our findings were that using a 2x, 4x or 8x batch size for $B_{ref}$ sometimes improved the learning performance, but not consistently. To some degree, increasing the batch size of $B_{ref}$ acted similarly to increasing the regularization coefficient, as both of them reduce more churn. Thus, we did not search for the best batch size for $B_{ref}$ to alleviate the hyperparameter choice burden. \n\n> Questions on concrete experimental results in continual RL\n\n**[Explanation for the slow decreasing of the approximate rank for C-CHAIN in Fig. 4]**\n\nFor C-CHAIN, the approximate rank decreases very slowly, from around 85 to 80 over 10M steps (in contrast, the approximate rank decreases is 75 to 30 for vanilla PPO). We think it is reasonable as the agent continually consumes the plasticity to learn new tasks and our method is not perfect.\n\n**[The performance in MountainCar]** \n\nWe think MountainCar is a bit more difficult than the other three Gym control tasks, as it needs more exploration and TRAC almost totally failed in it (note that MountainCar is not included in TRAC paper). One possible reason for C-CHAIN’s low early-stage performance might also be due to the exploration feature of MountainCar. Reducing churn could also be viewed to prevent the generalization of exploration behavior learned by the policy in some states to similar states. Therefore, C-CHAIN learns slowly but improves steadily, while vanilla PPO learns quicker and collapses.\n\n**[The performance on Fruitbot, Jumper, and Plunder]**\n\nBy checking the task details in the official document of ProcGen, tasks like Leaper, Jumper are stuff-collection tasks with sparse rewards, and especially Leaper is an episodic-reward environment. \n\nTo gain more understanding, we provided the average scores of the **max/mean/min** curves for Leaper, Fruitbot, Plunder, Jumper below. We can observe that C-CHAIN improves the average Max scores over PPO Vanilla and the average Mean scores (by improving Max or by reducing failures when Max is similar); while it does not fully address the (near-)zero performance due to the limited exploration ability of the PPO base agent.\n\n| Method/Task   | Leaper | Fruitbot      | Plunder   |   \n|--------------|--------------|--------------|--------------|\n| PPO (Oracle) | 2.020 /  0.350 / 0.000 | 6.398 / 1.613 / -0.079 | 11.858 / 7.113 / 2.731 |\n| PPO (Vanilla) | 2.006 / 0.347 / 0.000 | 5.882 / 1.549 / -0.068 | 8.304 / 4.801 / 1.944 | \n| C-CHAIN | 4.004 / 0.668 / 0.000 | 6.153 / 1.689 / -0.043 | 16.916 / 10.340 / 4.949 |\n| TRAC | 2.002 / 0.334 / 0.000 | 3.624 / 0.855 / -0.034 | 17.104 / 9.719 / 2.724 |\n\n> **Other remaining questions**\n\n**Due to the space limitation, we will provide them during the discussion stage.**", "claims_and_evidence": "Here, I present some claims from the paper verbatim and explain why I agree or disagree with them, while also questioning their soundness.\n\n> We demonstrate the connection between plasticity loss and increased churn, and show the pathological learning dynamics this connection induces.\n\nYes, they demonstrate this connection for a specific algorithm—PPO—in a particular continual learning setting where tasks switch every few million steps. They do this by showcasing rank collapse in the neural tangent kernel matrix  with prolonged training over a sequence of tasks.\n\n> We unbox the efficacy of reducing churn in continual RL by identifying a gradient decorrelation effect and a step-size adjustment effect.\n\nHonestly, I am unsure what \"unboxing the efficacy of reducing churn\" means in this context. I would appreciate it if the authors could clarify this in plain language. In addition, I’d like further clarification on the step-size adjustment effect. How does it differ from momentum-based optimizers like Adam or RMSprop, which also implicitly affect the step size?\n\n> We propose C-CHAIN and demonstrate it effectively mitigates the loss of plasticity and outperforms prior methods in a range of continual RL settings.\n\nThis is supported by their experiments. However, I have concerns about the experiments and analyses, which I will discuss later.\n\n> We demonstrate that under the continual changes in the data distribution and objective function, the agent gradually loses the rank information of its NTK matrix, leading to highly correlated gradients and eventually the exacerbation of churn. \n\nThe authors seem to be making an important point here, but it is not entirely clear to me. What exactly do they mean by highly correlated gradients? How does their method prevent this from occurring? Is it by leveraging the gradient information of the reference batch $B_{ref}$​ in their churn regularizer loss?", "methods_and_evaluation_criteria": "The proposed method C-CHAIN is well-suited for the continual learning setting in consideration, especially one where tasks switch every N timesteps. I'm not particularly convinced by some of the design choices, but I want to assure the authors that I'd keep an open mind and re-evaluate my reviews based on their rebuttal response. \n\n1. On the usage of domain randomization to Gym classic control environments:\n\n> For Gym Control, we use four environments: CartPole-v1, Acrobot-v1, LunarLander-v2 and MountainCar-v0. For each environment, a task sequence $T$ is built by chaining $k$ instances of the environment with a unique Gaussian observation noise $\\epsilon_i \\sim \\mathcal{N}(0, \\sigma^2)$ sampled once for each.\n\nIsn't this domain randomization (Tobin et al. 2017), but with the parameters changed every few million timesteps? Is this only to induce non-stationarity in the environment? Adding noise arbitrarily to the observations makes it seem a little contrived. Have you considered more natural tasks such as varying factors like gravity or surface slipperiness or even the terrain in MuJoCo? For inspiration, you can refer to robotics works such as Kumar et al. (2021). \n\n\n2. In several environments, including Fruitbot, Jumper, and Plunder, the standard errors overlap significantly with the second-best reported method. Is this due to the use of only 6 seeds? Are the authors confident that the method would prove superior with more seeds?\n\n3. Patterson et al. (2024) suggest the following: *\"In general, we advocate that you do not report standard errors. They are like a low-confidence confidence interval, and it is more sensible to decide on the confidence interval you want to report.\"* Could the authors clarify why they chose to report standard errors, especially given that only 6 seeds are used? Would it be possible to use a different, more appropriate metric, such as a bootstrap confidence interval?\n\n**References**\n1. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017, September). Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 23-30). IEEE.\n2. Kumar, A., Fu, Z., Pathak, D., & Malik, J. (2021). RMA: Rapid Motor Adaptation for Legged Robots. Robotics: Science and Systems XVII.\n3. Patterson, A., Neumann, S., White, M., & White, A. (2024). Empirical design in reinforcement learning. Journal of Machine Learning Research, 25(318), 1-63.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. Question on  C-CHAIN for Continual Supervised Learning\n> Besides, Permuted-MNIST and RandomLabel-MNIST are simple, where the agent can find a good solution near the initialization.\n\n- This is a strange argument—it's as if the authors are saying, \"Our method is too strong, which is why it failed...\"\n- Continual Mountain Car seems even simpler than Permuted-MNIST to me. Could the authors propose an alternative plausible explanation for why their method may not be performing well in this case?\n\n2. In Fig. 3, for Mountain Car, why does C-CHAIN (in blue) exhibit a flat curve in the first phase? What changes in the later stages that allow for successful learning?\n\n3. It's unclear to me why some methods were not included in the experimental comparison, especially when they are cited as references. For example, Dohare et al. (2024) proposed continual backprop, and Lyle et al. (2024) suggest that using layer normalization and L2 regularization is effective for maintaining plasticity. Why were these methods not considered? This is a significant concern for me. I would be willing to re-evaluate my score if the authors provide additional empirical comparisons with these two methods.\n\n\n\n**References**\n1. Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026):768–774, 2024.\n2. Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., & Dabney, W. (2024). Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762.", "relation_to_broader_scientific_literature": "The paper examines the loss of plasticity in deep neural networks through the lens of churn. While loss of plasticity has been demonstrated in several works, including Lyle et al. (2022), Sokar et al. (2023), and Dohare et al. (2024), its cause is not well understood. The authors make a novel contribution by studying this phenomenon from the perspective of churn and provide interesting insights.", "essential_references_not_discussed": "The authors overlook some relevant references on step-size adaptation. I recommend considering the following citations.\n\n*Step-size adaptation*\n- Sutton, R. S. (1992, July). Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI (Vol. 92, pp. 171-176).\n- Dabney, W., & Barto, A. (2012). Adaptive step-size for online temporal difference learning. AAAI Conference on Artificial Intelligence (pp. 872-878).\n- Martens, J., & Grosse, R. (2015). Optimizing neural networks with kronecker-factored approximate curvature. International Conference on Machine Learning (pp. 2408-2417).\n- Elsayed, M., Vasan, G., & Mahmood, A. R. (2024). Streaming Deep Reinforcement Learning Finally Works. arXiv preprint arXiv:2410.14606.", "other_strengths_and_weaknesses": "**Strengths**  \n- The introduction was well-written, making it easy to understand the problem, the solution, and the authors' contributions.  \n- The visualizations and analysis using the NTK are interesting and insightful.  \n\n**Weaknesses**  \n- I’ve listed the weaknesses in the previous sections. I’m hopeful the authors will address at least some of these in the rebuttal.  \n- The figures could be clearer. For example, in Fig. 4, consider placing the legend outside the figure, as its current position occludes large portions of the figure.  \n- C-CHAIN is tested only with PPO. It’s unclear whether it is necessary for other methods.", "other_comments_or_suggestions": "> Reinforcement learning (RL), when coupled with non-linear function approximators, suffers from optimization challenges due to the non-stationarity of the data and the learning objectives, i.e. the deadly triad\n\n- Why mention the deadly triad here? What is off-policy about your learning problem? I believe the introductory line doesn’t need to reference the deadly triad. Perhaps the authors mention it implicitly to acknowledge PPO’s clip surrogate loss?\n\n\nI like the motivation and approach of the paper and believe it could be much stronger in its current form. If the authors provide additional experimental evidence and address my concerns, I would be happy to increase my score.", "questions_for_authors": "1. Why aren’t continual backprop (Dohare et al. 2024) and ReDo (Sokar et al. 2023) included as baselines?\n2. Is this method specific to PPO? Could you include a small experiment with SAC or another algorithm to demonstrate its generality? I believe a small experiment like this would significantly strengthen the case for the method’s generality.\n3. How does it perform in continuous action spaces? It seems all experiments here use discrete action spaces\n4. We sample $B_{ref}$​ and $B_{train}$​ from the buffer $D$. Does the size of $B_{ref}$​​ have on impact on C-CHAIN’s performance?\n5. PPO is known to suffer from policy collapse (Dohare et al. 2024). Does this also occur in off-policy algorithms like SAC or TD3? Would using a large replay buffer help in this setting? Intuitively, it seems like it would mitigate catastrophic forgetting rather than plasticity loss.\n6. Should the buffer be flushed after each task switch? In other words, does the agent need to be aware of task switches?\n7. Out of curiosity, have the authors considered the streaming RL setting (Elsayed, Vasan & Mahmood 2024)? Is it possible to extend this method to the streaming RL setting?\n8. Is layer norm + l2 regularization effective in maintaining plasticity (Lyle et al. 2024)?  \n9. In Fig. 4, the approximate rank appears to be slowly decreasing for C-CHAIN over time. Is this the case? If so, is there an explanation for this behavior?\n\n\n**References**\n1. Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026):768–774, 2024.\n2. Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In ICML, pp. 32145–32168, 2023.\n3. Elsayed, M., Vasan, G., & Mahmood, A. R. (2024). Streaming Deep Reinforcement Learning Finally Works. arXiv preprint arXiv:2410.14606.\n4. Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., & Dabney, W. (2024). Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper studies the loss of plasticity problem in deep continual reinforcement learning (RL). By establishing a connection between plasticity loss and churn through the Neural Tangent Kernel (NTK) framework, it offers insightful observations regarding the relationship among churn, NTK, and plasticity loss. Additionally, extensive experiments strongly support the claims presented in the paper. Although some reviewers have concerns about the mathematical rigor of the analysis, we believe that the strengths of the paper outweigh its weaknesses, and it makes solid contributions to the research community.
Decision: Accept (poster)