A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning
Authors: Arthur Juliani
Jordan T. Ash
Venue: NeurIPS 2024 spotlight (2024)
Keywords: plasticity loss
reinforcement learning
regularization
continual learning
optimization
Primary area: reinforcement_learning

Abstract: Continual learning with deep neural networks presents challenges distinct from both the fixed-dataset and convex continual learning regimes. One such challenge is plasticity loss, wherein a neural network trained in an online fashion displays a degraded ability to fit new tasks. This problem has been extensively studied in both supervised learning and off-policy reinforcement learning (RL), where a number of remedies have been proposed. Still, plasticity loss has received less attention in the on-policy deep RL setting. Here we perform an extensive set of experiments examining plasticity loss and a variety of mitigation methods in on-policy deep RL. We demonstrate that plasticity loss is pervasive under domain shift in this regime, and that a number of methods developed to resolve it in other settings fail, sometimes even performing worse than applying no intervention at all. In contrast, we find that a class of ``regenerative'' methods are able to consistently mitigate plasticity loss in a variety of contexts, including in gridworld tasks and more challenging environments like Montezuma's Revenge and ProcGen.

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning
Arthur Juliani Jordan T. Ash Microsoft Research NYC (ajuliani, ash. (ordaa)@miccooooffo com
Abstract
Continual learning with deep neural networks presents challenges distinct from both the fixed-dataset and convex continual learning regimes. One such challenge is plasticity loss, wherein neural network trained in online fashion displays a degraded ability to fit new tasks. This problem has been extensively studied in both supervised learning and off-policy reinforcement learning (RL), where number of remedies have been proposed. Still, plasticity loss has received less attention in the on-policy deep RL setting. Here we perform an extensive set of experiments examining plasticity loss and a variety of mitigation methods in on-policy deep RL. We demonstrate that plasticity loss is pervasive under domain shift in this regime, and that number of methods developed to resolve it in other settings fail, sometimes even performing worse than applying no intervention all. In contrast, we find that class of "regenerative" methods are able to consistently mitigate plasticity loss in variety contexts, including in gridworld tasks and more challenging environments like Montezuma's Revenge and ProeGen.
1 Introduction
In many important machine learning domains, such continual learning and online reinforcement learning (RL), training data are not wholly available simultancously. Rather, in these settings, data ar- rives sequentially over long period of time. Irrespective of the available quantity of training data, would typically like to have the highest performing model possible at any given point. Unfortunately, these sequential learning scenarios Specifically, if a model is naively updated to convergence each time new data arrives, and the amount of training data effectively increases, the resulting model generalizes worse than it would under more standard, non-sequential constraints. This phenomenon has been introduced by Ash and Adams (2020) as the "warm-start problem." where it was found that randomly initialized models generalize well but are expensive to fit, and warm-started models generalize poorly but converge far more quickly,
known present optimization issues for neural networks.
An adjacent problem to warm-starting is that of plasticity loss (Lyle et al., 2023), which presents itself both in the supervised learning and reinforcement learning settings. Like in the warm-start problem, plasticity loss describes a degradation in a model's ability to fit new data as training progresses, ultimately harming both sample efficiency and asymptotic performance. If an agent achieves worse performance when fitting first a source distribution and then a distinct target distribution than it would if instead trained on the target distribution in isolation, the degradation is attributed to plasticity loss.
Whereas the warm-start problem is restricted to test performance, plasticity loss is generally measured degradation in performance on training data, often under some form of distribution shift. Despite the typical focus on training data, an ideal solution to plasticity loss would be able to prevent performance degradation both for data seen and unseen when fitting the policy. Importantly, the plasticity loss phenomenon is distinct from overfitting, as resolving plasticity loss should improve both training performance and generalization. In contrast, overfitting describes the opposite, where improving training performance has deleterious effects generalization.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This work studies the plasticity loss phenomenon in detail for the on on-policy reinforcement learning setting. We introduce three distinct kinds of distribution shift to facilitate our analysis as well as variety of environments and tasks. While several interventions have already been proposed, we demonstrate that some of the methods which proved successful for addressing plasticity loss in the supervised learning or off-policy reinforcement learning setting fail in the on-policy setting or under our proposed forms of distribution shift. We further highlight several criteria that we believe are necessary to remedy the issue. Based on these insights, we describe several techniques that resolve plasticity loss on the environments we consider.
In summary, the contributions of this paper are:
We extend studies of plasticity loss and the warm-start problem the on-policy regime, finding that they present a persistent issue across a variety of model architectures and environmental distribution shift conditions.
We provide an in-depth analysis of the correlates of these pathologies, studying several types of environmental settings, model architectures, and previously proposed approaches for mitigation in settings related to on-policy reinforcement learning. Importantly, we include generalization trends in our consideration of these phenomena.
We characterize properties of methods that seem necessary for interventions to be successful at both addressing plasticity loss and ensuring generalization performance is maintained, and provide recommendations along these lines.
2 Preliminaries
An RL algorithm is described as "on policy" if it is trained using data collected from its own policy. While generally more sample inefficient than off-policy alternatives, where the data collection mechanism deviates from the policy being fit, on on-policy methods have become more commonly used in practical RL problems because of their simplicity and stability (Andrychowicz et al., 2020).
Proximal Policy Optimization (PPO) is a ubiquitous on-policy deep RL algorithm (Schulman et al., 2017). As a trust-region method, PPO constrains the extent by which weight update can modify the current policy, quelling instability issues caused by high-variance policy gradients. Despite this robust- ness, on on-poiicy algorithms still update on data corresponding only the agent's current experience. which may be sub-optimal for capturing necessary information around solving the underlying MDP. There some evidence that plasticity loss can be present on-policy RI. learning problems (Dohare et al., 2023b), demonstrating the effect in a somewhat ad hoc manner, we systematically use multiple environments and distribution shift conditions to more thoroughly characterize plasticity loss and how to resolve it-in the on-policy setting.
2.1 Simulating environmental distribution shift
This paper primarily studies three distinct forms of distribution shift that we can reasonably expect plasticity-prest prreeriig intervention to mitigate. Each experiment is organized into r distinct rounds. In each round we supply & environments from some distribution to the agent, fit a policy for that set of environments, and measure performance at the end of the round. To induce distribution shift, between each round we modify the data according to one of the three strategies described below. The environmental modifications we consider are:
Permute. Input pixels are randomly shuffled from round to round. Each of the l environ- ments in a given round are shuffled in the same way, but distinct from the shuffling in any other round. This method was utilized in Dohare et al. (2023b). We expect that an optimal learning algorithm will be able to achieve equivalent per- formance on the initial and subsequent rounds.
Figure 1: Examples of gridworld environment tasks. The agent (black triangle) begins each episode in the center of the environment. Blue jewels provide +/ reward, red jewels provide -/ reward, and dark grey walls prevent movement. Objects are placed randomly.
Window. We supply the agent with l new en- vironments for the same task. Here previously seen environments are no longer accessible, so the agent can only optimize with respect to the current k. This approach was considered by
2

---

Abbas et al. (2023) and Lyle et al. (2023). We expect that an optimal learning algorithm will be able to achieve equivalent or better performance on subsequent rounds as compared to the initial round.
Expand. Like in the window modification, & new environments are supplied to the agent. Here, however, they are appended to the total amount of training data, such that the number of available environments in the final round of training is k x n. This is most similar to the warm-start environment studied primarily in Ash and Adams (2020), and revisited in Igl et al. (2020). In this setting we expect training performance to decrease each round, even for an optimal learning algorithm, training data grows and becomes more difficult to fit. Correspondingly, we expect the generalization performance to increase the training distribution expands better capture the true distribution.
available
In the case of the latter two modifications, data at each new round are drawn from the exact same distribution as environments from previous rounds. As we demonstrate in Section 4. 1, these still induce significant performance depredation in warm-started models. This observation is surprising we often think of a model's 's initialization as a sort of "prior" over learned functions, and here we demonstrate that even an initialization obtained by training on data distributed identically to the current round still hinders performance in on-policy learning.
We use simple gridworld task our main sandbox for studying plasticity loss. The environment is drawn from the NeuroNav library (Juliani et al., 2022), and the goal of the agent is to collect rewarding blue jewels while avoiding punishing red jewels in fixed time window (100 time-steps per episode). Each sample of the environment from the distribution of possible tasks changes both the location of the jewels and the walls of the maze. Figure shows environment modification examples.
3 Existing Approaches
A number of methods have been proposed to address plasticity loss in deep neural networks, though none were designed with on-policy RL explicitly in mind. Instead, these methods have been primarily validated in the context of either supervised learning or off- off-policy RL. Each of these techniques can be divided into one of two groups: Interventions which are performed intermittently during training, and those which are applied continuously, either as part of the model architecture or as part of the model update process. For additional implementation details see Appendix Section A.
3.1 Intermittent Interventions
We consider an intervention to be intermittent if it is applied only at specific points during training. Periodically, the interventions in this category are applied, and training proceeds normally otherwise. Most of these are applied each time there is a training distribution change, implying that there must be awareness of when this occurs; in practice this information may be unavailable and difficult detect.
Resetting final layer. Proposed by Nikishin et al., this method involves periodically replacing the weights the final layer in the network with newly initialized values. was demonstrated that this alleviates plasticity loss in certain off ooflpy RL settings.
Shrink=Perturb. This technique periodically scales the magnitude of all weights in the network by a factor and then adds a small amount of noise (Ash and Adams, 2020). It was demonstrated that this improves performance in batched continual learning settings. In our implementation the weight of these two factors are entangled such that they sum to one.
Plasticity Injection. Plasticity Injection replaces the final layer of a network with a new function that sum of the final layer's output and the output of newly initialized layer subtracted by itself (Nikishin et al., 2023). The gradient is then blocked in both the original layer and the subtracted new layer. was shown that this method increases performance of off off-policy RL agents in the ALE.
ReDo. This technique resets individual neurons within the network based on a dormancy criteria at fixed intervals (Sokar et al., 2023), leading to improvement in performance in the context of off oollco RL agents trained on the ALE. Following Sokar et al., we reset the model parameters more frequently than we apply environmental distribution shifts.
3.2 Continuous Interventions
We characterize a method as continuous if it is applied at every step of optimization. Continuous interventions are desirable because they do not require an awareness of when a distribution shift has occurred, which can be challenging in practical scenarios where we want to mitigate plasticity loss.
3

---

Warm start
Reset all
(A) Pormutt
(B) Permate
(C) Window
(D) Expand
Rowwurd [twin)
Ruwand (taiio)
Roward (tram) m)
Roward
-0.25
-0.5
20
40
60
80
100
120
140
160
180 200
0 6
1 2 425456788
6.1 0123456788 6766
Epech
Gnn
1000s)
Round
Round
Round
(C) Window
(D) Expand
Roward (tes)
Rewoad (twet)
Sowar (trst)
0.00
0.5
9.5
0.0
-0.50
0.0
80 120
6784
Epoch 1000s)
Round
Round
Round
Figure 2: Performance in the gridworld environment under each of the three distribution shift conditions. The degradation between rounds is evidence of plasticity loss. (A): Epoch-level training performance for permute modification. Dotted vertical lines indicate the end of each round, before a new environmental distribution shift is applied. (B); Round-level training performance for the Permute modification. Data points correspond normalized mean reward in final 50 episodes of round. Shaded regions correspond to standard error. (C. D): Round-level training performance for the Window and Expand conditions. Top row: Training performance. Bottom row: Test performance.
L2 Norm. Discussed by Lyle al., this method involves regularizing the network using the L2 norm. It was demonstrated that in certain continual learning settings this reduces plasticity loss.
LayerNorm. LayerNorm (Ba al., 2016) is ubiquitous deep learning regularization technique, and was shown useful for mitigating plasticity loss by Lyle et al.. The authors demonstrated a performance improvement the ALE when using off-policy RI..
CReLU activations. Proposed by Abbas et al. and studied further by Lee et al., this method involves replacing ReLU activations with the CReLU activation function. This ensures that the gradient is non-zero for all units a given layer, which appears to mitigate plasticity loss when periodically switching between tasks in the ALE with off- off-policy methods.
Regenerative Regularization. Regenerative regularization uses an L2 penalty to encourage model weights to be near their initial values (Kumar et al., 2023). It has been shown to be effective in number continual learning settings.
4 Experiments
Below we present results of a suite of experiments conducted using 2D gridworld environment (Ju- liani et al., 2022), the procedurally generated CoinRun environment (Cobbe et al., 2019), and the Atari game Montezuma's 's Revenge (Bellemare et al., 2013). All models are trained using PPO (Schulman al., 2017). For the gridworld experiments we use an MI.P encoder, and for the CoinRun and Montezuma's 's Revenge experiments we instead use convolutional encoder. We set the number of environment instances (k) to 100 for all experiments conducted with the gridworld and CoinRun environments. Additional training details can be found in Appendix Section B.
4.1 Plasticity loss is apparent in on-policy RL
We find evidence of plasticity loss in the on-policy RI setting in the presence of multiple kinds of dis- tribution shift, as presented Figure For round-level plots like these, reward values are normalized such that the mean reward at the end of the first round is set to zero. As such, positive and negative values correspond respectively to increases and decreases in performance relative to the first round. Decreasing normalized reward indicates decay in relative performance, implying plasticity loss.
For both the permute and window shift conditions, Figure 2 demonstrates clear degradation in mean episodic reward for the warm-started model as compared to a model which has all of its weights reset between each round. While there some positive transfer for the warm-start method in the expand shift condition, we find generalization performance suffers in this condition as compared to reset-all. We provide statistical tests comparing the performance of these methods to the warm-start and random initialization baselines in Section D of the appendix.

---

aisst
abrink
sara
Iulettin Rebo
Royen rog
Shrmik partarb
0.005.p
0.734
= p.246.p
0.008
R? 0.000.p p 0.997
R² 0.000. 0.913
=0=02210p p 0.016
(tran)
8.0-
0.0
Roward Orain)
9.0
Roware orann
0.9
-0.5
-1.0-
-100
-1.0-
-1.0
-1.5-
-15
-1.3
0.205
0,015
0.722
0.06
3.28
0622
0.065
0.001
0217
-0.777
0.405
1.75
0004
2.039
0.59
Joliy entropy
Weggh magnitado
Weight dillevnnee
Gradiork neem
Dead unis
R 0.147.
0.049
R
0.041.p 0.312
R 0.032. 191
R 0.011.p
0,600
R 22=0 0.272. 0.005
Dovonn (taq)
2.0
Dowoun thee
10
Soaund Dee
1.0
Donward (eec))
8.5
0.
0.5
0.3
8.0-
0.0
0.-
0.0
0.3
-0.5
0.015
0.022
1.764
0.000
Wright
Grudiest
Weight magnitude
Grudient norm
Dead units
Reward (train)
0.06
0.50
0.6
0.25
Woight mageiitud
Desd units
0.06
Rewant (train)
0.00
1.0
Crailent
0.02
-0.25
0.2
0.5
-0500
0.00
0.0
Round
Round
Round
Round
Correlation plots of normalized mean reward in the gridworld environment compared against identified metrics. Each point is averaged over five replicates, and shows the final values produced after a ten-round experiment. Values for each measurement are normalized by its baseline level initialization. Measurements that significantly correlate (p < 0.05) with normalized reward are bolded. First row: Training distribution performance. Second row: Test distribution performance. Bottom: Values of the three predictive metrics during the course of training for each intervention and window change condition as compared to training performance. Shaded regions correspond to stan- dard error. Final values of plots like these correspond single point in the correlation plots above.
Figure 3: Top:
4.2 Predictors of plasticity loss and generalization
Identifying the exact statistical underpinnings of plasticity loss is an active area of research in continual and reinforeement learning. Here identify several factors which are significantly correlated with plasticity loss as well as a degradation in generalization performance. Figure presents correlation plots of normalized reward with various metrics of interest. Here normalize rewards (i.e. subtract the initial reward value from the final reward value) to compensate for the fact that many of these algorithms have regularization effect that improves performance average but does not necessarily improve performance trends-in it is possible to improve generalization, for example, without remedying plasticity loss.
Note that the LayerNorm and CReLU methods effects on the weight magnitude and dead unit counts, respectively. For graphs of the underlying metrics over time during training. see Supplemental Figure 11. We find that weight magnitude and number of dead units are both significantly correlated with the normalized reward Ip < 0.05). "Dead units" refer to ReLU activations that never produce non-negative outputs.
included in this analysis, due to their direct
We find that neither the gradient magnitude nor the magnitude of weight difference between updates has significant, linear relationship with either plasticity or generalization. This is in contrast to the findings of Abbas al. and Nikishin et al., each of which examined plasticity loss in off-policy rather on-policy settings. The former found that weight difference was correlated with loss of plasticity while the latter found that gradient norms were predictive.
We also fit a generalized linear model (GLM)) using five metrics as independent variables and normalized reward as the dependent variable. Here find that both gradient and weight magnitudes
5

---

GridWorld
Expand
shrink-perturth Regen reg (122 LN-
ReDo Reset-
LayerNorm (LN)
Resat-final
Injection
CReLU
Sbrink+perturs
L2 norm-
Reset-all
Warm-start
-2
-0.8
-0.4
0.0
0.4
-0.8
-0.4
0.0
0.4
Normalized Reward (Train)
Permute
Window
Expand
Soft shrink-tpetturt L.N
Regen reg (122
ReDo Reset- LayerNorm (LN)
Reset-final
Injection
CReLU
Shrink+pertutt
L2 norm-
Reset-all
Warn-start
-0.8
-0.4
0.0
-0.4
0.4
0.8
0.4
1.2
Normalized Reward (Test)
Figure 4: Performance of intervention methods compared to warm-start and reset-all baselines on the Gridworld environment. Final round mean reward normalized by the performance end of the first round, and interval bars denote standard error. Top: Train performance. Bottom: Test performance.
are significantly predictive of train performance Ip < 0.01], while dead unit count no longer is [p > 0.05]. This suggests that the predictive power of the dead unit count can be accounted for by the weight magnitude. An additional GLM was to the test data, and we find that only the dead unit count remains predictive (p < 0.05).
Although these analyses do not establish causality, the robust significance of weight magnitude predicting plasticity loss suggests that plasticity loss may be a function of how much trainable parameters deviate from their initialization distribution. As such, successful mitigation strategies would induce a sub-lincar increase in the weight norm of the network parameters over the course of learning. This has the downstream effect of minimizing the number of dead units in the network. Below we compare the mitigation strategies, finding that those which address this underlying cause directly indeed perform best.
4.3 Novel architectural methods do not fully address plasticity loss
Recently number of methods have been introduced to specifically address plasticity loss using changes to the network architecture. These methods were previously validated in the off-policy setting only. Here we consider the behavior of two of these methods it applies to the on-policy setting: CReLU Abbas et al. (2023) and Plasticity injection Nikishin et al. (2023). We find that both methods fail to address plasticity loss in the permute shift condition. The CReLU method is able to address plasticity loss in the window and expand settings only, while the plasticity inject method underperforms even the warm-start baseline in all three contexts (Figure 4).
4.4 Regularization methods address plasticity loss
We next consider the class of methods which regularize the weights of the network, either continu- ously or intermittently. These methods include final layer reset (Nikishin et al., 2022), shrink+perturb (Ash and Adams, 2020), continuous "soft" variant shrink+perturb (Dohare et al., 2023b), L2 regularization, ReDo (Sokar et al., 2023), and regenerative regularization (Kumar et al., 2023). We find final layer resel fails to address plasticity loss in any of the three conditions. In contrast, other five methods mitigate both plasticity loss and the warm-start problem to some extent in all three shift conditions (Figure 4). Of these methods, ReDo performs the worst as it unable to benefit from the positive transfer which is possible in the cases of the "window" and "expand" conditions,

---

CoinRun
Permute
Windaw
Expand
Soft sbrink +perturb LN-
Regen (12) LN-
LayerNorm (LN)
Reset-final
Injection
CRelU
Soft shrink perturb
S Regen reg (1.2)
1.2 norm
Reset-all-
Warm-start-
-6
-3
-4
2
-8
-4
Normalized Reward (Train)
Permute
Window
Expand
Soft shrink+ perturb LN
Regen reg (L2) LN
ReDo Reset
LayerNNorm (LN)
Reset-final
Injection
CReLU
Soft Murink+-erture
Shrink+ perturb
Regen reg (L2)
L2 norm
Reset-all
Warm start.
Normalized Reward (Test)
Figure 5: Performance of intervention methods compared to warm-start and reset-all baselines on the CoinRun environment. Final round mean reward is normalized by the performance at end the first round, and interval bars denote standard error. Top: Train performance. Bottom: Test performance.
as methods that retain information from previous rounds effectively have access to more training data in these conditions. ReDo underperforms even the warm-start baseline in the "expand" condition.
4.5 LayerNorm addresses training plasticity loss but not generalization
We also consider the effect of LayerNorm plasticity loss. was previously demonstrated in the off-policy setting that LayerNorm was able to mitigate some effects plasticity loss (Lyle et al., 2023). Given that LayerNorm tends to improve learning across number of contexts (Ba al., 2016), it is important to attempt to separate the effects of LayerNorm on performance generally from its effect on plasticity specifically. We find that LayerNorm resolves plasticity loss terms of training performance, but is inconsistent in its effect on generalization performance. For example, the LayerNorm model significantly underperforms even the warm-star start baseline on the test data of the permute shift condition. We find however that combining soft shrink+pertur or regenerative regularization with LayerNorm results in overcoming both training plasticity loss and deleterious generalization trends (Figure 4).
4.6 Plasticity loss in CoinRun
We also use the CoinRun environment from the ProcGen suite of tasks to evaluate the of candidate interventions Cobbe et al. (2019, 2020). ProeGen is built using procedural generation Cobbe et al. (2020), making it possible to study the same three distribution-shift conditions which were considered in the gridworld experiments. Here we present performance results for each of the intervention methods on the CoinRun environment.
Overall, we find results largely consistent with those of the gridworld task. There significant plasticity loss present in all three tasks as measured by the gap between the warm-start and reset-all conditions. We also find that reset-final, CReLU, and plasticity injection all unable to fully resolve the loss of plasticity. In contrast, the three most effective methods in the gridworld task (LayerNorm, regenerative regularization, and soft shrink+ are all also effective CoinRun well.
7

---

See Figure 5 for plots of the normalized mean rewards over the course of training for all meth- ods. For results on two additional ProcGen en- vironments, see Appendix D. For round-level performance plots of all methods, see Section D in the appendix. We find that relative efficacy of the intervention methods is consistent across ProcGen environments.
Montezuma's Revenge
6000
5000
Reward
4000
3000
2000
Baseline
Regenerative (L2)
4.7 Plasticity loss in Montezuma's 's Revenge
1000
Soft shrink + perturb
Montezuma's Revenge is an Atari game that is often used to benchmark the quality of explo- ration procedures in RL Bellemare al. (2013); Salimans and Chen (2018). The environment consists of many separate rooms, some only ac- cessible through a locked door. The agent must avoid obstacles, find and use keys, and traverse several rooms before obtaining non-trivial reward. Because of this reward sparsity, exploration is needed to incentivize the agent to advance beyond the periphery of its experience.
Timestop
1e8
Figure 6: A comparison between RND agents trained with and without a plasticity-loss mitigat- ing intervention. Experiments are done over twenty replicates and shaded regions show standard error.
Montezuma's Revenge is designed such that the agent only sees the room it currently occupies. Accordingly, the distribution over input states expands as the agent learns to explore the environment. This introduction of significantly different states induces level of distribution shift which is unique among ALE environments. Within our framework, the distribution shift of Montezuma's 's Revenge is most similar the "expand" condition studied here.
In this experiment we consider two versions of a policy trained using the Random Network Distillation (RND) exploration bonus (Burda al., 2018). All architectures and hyperparameters are inherited from the original RND paper. Figure 6 compares a typical RND policy to ones that are trained in conjunction with the soft shrink+perturb or regenerative regularization interventions. The plasticity loss mitigating policy achieves higher reward than a corresponding agent fitted in standard fashion. Due to computation constraints, we did not evaluate the other intervention methods.
5 Discussion
Plasticity loss has been identified and rigorously studied in the continual learning and off-policy reinforcement learning settings. We find that is also an issue in the on-policy setting under variety of different conditions. Methods introduced in the off off- oof-policy setting such as plasticity injection and CReLU appear to be not as consistently effective in the on-policy setting. We hypothesize that this degradation may be due to the violation of the iid assumption which supervised learning and off-policy reinforcement learning rely on. Further, these methods require architectural modifications of the policy and value networks, and are thus less general than regularization-ben alternatives.
In contrast, the class of regularization methods, which include L2 regularization. ReDo, shrink+perturb, and regenerative regularization all are able to significantly mitigate plasticity loss in our experiments. Of these methods, soft shrink+perturb displayed the best generalization per- formance, and can be easily combined with layer normalization. LayerNorm, now ubiquitous method in supervised learning, was previously introduced to address plasticity loss in the off-policy setting Lyle et al. (2023), and we find that it is also effective in the on-policy setting. Due its general regularizing effects, LayerNorm also increases baseline performance in the absence of distributional shift Ba et al. (2016), suggesting that a generally useful method for on-policy reinforcement learning methods such as PPO.
consistent feature of the class of regularization methods discussed here that they all normalize the network parameters towards their initial distribution. This has the effect of decreasing the weight magnitude and reducing the number of dead or saturated activation units, as seen Figure 3. has recently been proposed that both these metrics may be proxy for the underlying curvature of the optimization landscape (Lewandowski et al., 2023). Studying this connection more deeply would potentially be fruitful avenue for future research.
8

---

6 Related Work
In the context of generalization in continual learning settings, a related issue to plasticity loss is the warm-start problem (Ash and Adams, 2020). Recent work has shown that the benefits of methods such as shrink+perturb to generalization were dependent on the presence of noise in the labels used for training (Zaidi et al., 2023). In the absence of noisy labels, other regularization methods were found to be competitive with methods that utilize re-initialization such as shrink+perturb.
Another perspective on the problem of loss of plasticity has focused on the content of early learning in network. Achille et al. (2018) demonstrate that neural networks have critical early periods of sensitivity to training data which are analogous to those in animals (Hensch, 2004). In the biological context this sensitivity is tied directly to neuronal plasticity. This connection to critical period learning was also drawn in work on off- off-policy RI, in which "primacy bias" was proposed to explain the loss of plasticity seen in a number of task domains (Nikishin et al., 2022). Utilizing this insight, the periodic soft- soft-resetting of the neural network has enabled significant increases in the sample efficiency of off-policy RL methods (D' Oro et al., 2022; Schwarzer et al., 2023).
In the deep RL domain a similar problem called ray interference has been identified (Schaul et al., 2019). Ray interference arises from the conflicting gradient signals related to unique sub-tasks required to solve a single more complex task in online deep reinforcement learning. It is possible that for complex tasks such CoinRun Montezuma's 's Revenge there unique sub-tasks which are learned and whose gradients may interfere with one another. Exploring the distinction between ray interference and plasticity loss in the RL setting is promising future direction.
In the off-policy RL setting the problem of plasticity loss has also been studied under the name of "capacity loss" (Lyle et al., 2022). While plasticity loss refers to a property of the network-task interaction, capacity loss is a more generic term refer to the properties of a network that are task invariant. This work advocates for a solution similar to that of Kumar et al. (2023), and can thus be considered in the class of methods which regularize the network towards initialization distribution.
Within the domain of on-policy RL there is a related phenomena to plasticity loss called "policy collapse." which refers to degradation of performance as training progresses, even on a fixed dataset (Dohare et al., 2023c). Although these are distinct phenomena, there is likely overlap in the success of mitigation methods. A novel optimizer strategy "Non-stationary Adam" was proposed to address policy collapse, and although we did not study it, it may provide benefits on the tasks examined here. In terms of direct studies of plasticity loss in on-policy RL, both Dohare et al. (2023a) and Igl et al. (2020) show plasticity loss in the on-policy setting, and propose a method to address
As part of their analysis, Dohare et al. (2023b) propose a plastictty-nhhaccin method called "Contin- ual Backprop." which is another form of regularization towards the initialization distribution, however is one which is selective on a per-neuron basis. This method is similar to ReDo (Sokar et al., 2023), which we analyze here. It is also related to DrM (Xu et al., 2023), which builds on the DrQ algorithm.
It is finally worth mentioning that notions of plasticity loss have been studied in the neuroscience community as well. These range from classic work demonstrating the inability of animal models to fully accommodate new sensory input after developing cognitively without it (Wiesel and Hubel, 1963), to more recent work characterizing plasticity loss in humans as a component of various psychopathologies (Peled, 2005; Carhart- -arrris et al., 2023; Juliani et al., 2024).
7 Conclusion
In this work we studied the problem of plasticity loss in the context of on-policy deep RL. We find that similar to continual learning and off off-policy RL, plasticity loss is an issue across number of different environments and forms of distributional shift. We find that some methods previously proposed to resolve the issue in other problem settings, such as CReLU and plasticity injection do not transfer to on-policy RL. The methods which best resolve plasticity loss are those which act as continual regularizers opposed to intermittent interventions. Within this class of methods we find that a soft variant of +prink+pertuub combined with LayerNorm performs the best across our evaluated settings, providing simple and general method for addressing plasticity loss.

---

References
Abbas, 7., Zhao, R., Modayil, J., White, A., and Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07577.
Achille, A., Rovere, M., and Soatto, S. (2018). Critical learning periods in deep networks. In International Conference Learning Representations.
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L.,, Geist, M., Pietquin, O., Michalski, M., et al. (2020). What matters in on-policy reinforcement learning? large-scale empirical study. arXiv preprint arXiv:2006.05990
Ash, J. and Adams. R. P. (2020). On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3894,
Ba, L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013). The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:/8100 12894.
Kuchart-Harris R., Chandaria, S., Erritzoe, D., Gazzaley, A., Gim, M., Kettner, H., Mediano, P., Nutt, D., Rosas. F., Roseman, L., al. (2023). Canalization and plasticity in psychopathology. Neuropharmucology. 226: 109398.
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. (2020). Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048-2056. PMLR.
Cobbe, K., Klimov, O., Hesse, C., Kim, I., and Schulman, J. (2019). Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1289. PMLR.
Dohare, S., Hernandez- Garcia. J., Rahman, P., Sutton, R., and Mahmood, A. R. (2023a). Loss of plasticity deep continual learning.
Dohare, S., Srrmandez-Garrinn J. F., Rahman, P., Sutton, R. S., and Mahmood, A. R. (2023b).
Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306. 13812.
Dohare, S., Lan, Q. and Mahmood, A. R. (2023c). Overcoming policy collapse in deep reinforcement learning. Sixteenth European Workshop on Reinforcement Learning.
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L.., Bellemare, M. G., and Courville, A. (2022). Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022.
Hensch, T. K. (2004). Critical period regulation. Annu. Rev. Neurosci,, 27:549-579.
Igl, M., Farquhar, G., Luketina, J., Boehmer, W., and Whiteson, S. (2020). Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.05820
Juliani, A., Barnett, S., Davis, B., Sereno, M., and Momennejad, I. (2022). Neuro-nav: a library for neurally-plausiee reinforcement learning. arXiv preprint arXiv:2206.03312. 03112.
Juliani, A., Safron, A., and Kanai, R. (2024). Deep canals: A deep learning approach to refining the canalization theory of psychopathology. Neuroscience of Consciousness.
Kumar, S., Marklund, H., and Van Roy, B. (2023). Maintaining plasticity via regenerative regulariza- tion. arXiv preprint arXiv:2308. 11958. 11958.
Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y,, and Yun, C. (2024). Plastic: Improving input and label plasticity for sample efficient reinforcement learning. Advances in Neural Information P Processing Systems, 36.
10

---

Lewandowski, A., Tanaka, H., Schuurmans, D., and Machado, M. C. (2023). Curvature explains loss of plasticity, arXiv preprint arXiv:2112.00246.
Lyle, C., Rowland, M., and Dabney, W. (2022). Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.09560,
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. (2023). Understanding plasticity neural networks. arXiv preprint arXiv:230.00488.
Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. (2023). Deep reinforcement learning with plasticity injection. arXiv preprint arXiv:2305.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. (2022). The primacy bias in deep reinforcement learning. In International conference on machine learning, pages 16828-16847. PMLR.
Peled, A. (2005), Plasticity imbalance mental disorders the neuroscience of psychiatry: implications for diagnosis and research. Medical hypotheses, 65(5):947-955
Salimans, T. and Chen, R. (2018). Learning montezuma's 's revenge from single demonstration. arXiv preprint 111v::8812.03881
Schaul, T., Borsa, D., Modayil, J., and Pascanu, R. (2019). Ray interference: source of plateaus in deep reinforcement learning. arXiv preprint arXiv:1904.11455
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777
Schwarzer. M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. (2023). Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference Machine Learning, pages 30365-30388 PMLR.
Sokar, G., Agarwal, R., Castro, S., and Evei, U. (2023). The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145-32168, PMLR.
Wiesel, T. and Hubel, D. H. (1963). Effects of visual deprivation on morphology and physiology of cells in the cat's 's lateral geniculate body. Journal of neurophysiology, 26(6):978-99)
Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., et al. (2023). Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.1966
Zaidi, S., Berariu, T., Kim, H., Bornschein, J., Clopath, C.. Teh, Y. W., and Pascanu, R. (2023). When does re- in-iiialaiizaoonn work? In Proceedings on, pages 12-26. PMLR.

---

A Implementation Details
Below are the specific implementation details for each intervention. Wherever possible we followed the methods provided by the papers which introduced each method. Relevant hyperparameters used are listed in Table
Shrink+perturbb When the intervention is applied all learnable parameters in the network are iterated through and scaled by a. All parameters are then additively combined with newly sampled initialization parameters which are scaled by 3. For a set of parameters this corresponds to Tnew O'Eeurrentt + 3dinitt where Vinst is sampled from the parameter initialization distribution. For all experiments = 1-
Soft shrink+perturb. The shrink+perturb procedure applied after each step of gradient descent instead of only at specific intervals. Specifically, for a set of parameters this corresponds to Tnew QWaarrent + 3 where Tinst is sampled from the parameter initialization distribution. For all experiments a
B.
CReLU. All ReLU activations in the network replaced with CReLU. The equation for CReLU is: concal ( (x),, (Uxr)). We use the method where input parameter sizes are halved compared to the baseline that output parameters remain consistent for each layer. This results in network with half the number of parameters for the effected layers.
Regenerative Regularization (L2). An additional loss term is added to the PPO update in each step of gradient descent. This term corresponds to L2 norm of the difference between the current parameters of the network and the network parameters initialization. This loss is scaled by a.
Plasticity injection. We follow the method described by Nikishin al. for the initial intervention, only performing plasticity injection on the final layers of the network (i.e. value and policy heads). To ensure computational efficiency, during subsequent interventions we combine the previous parameter values into single set of weights and biases (i.e. Wold Wald Wnema. Wnewb). This ensures that at any given time the plasticity-injected layer is represented by Voldd + Tnewa. 8g (gewbb))
LayerNorm. We apply the LayerNorm Ba et al. (2016) function before the ReLU activation at each layer.
Reset-final. At each intervention point we replace the weights of the final layers (i.e. value and policy heads) of the network with freshly initialized values.
L2 Norm. We apply an additional loss term the PPO update which corresponds to the L2 norm of the weights of the network. This loss is scaled by parameter a.
ReDo. At fixed epoch interval the procedure from Sokar al. is performed all layers of the neural network. A mini-batch of training data is used to calculate the dormancy rate for each unit, and units which are below re-initialized.
12

---

B Environment and Model Details
Additional hyperparameters used for Gridworld and CoinRun models are provided in Table 2. All experiments involve rumning five repetitions with unique random seeds each. All experiments are conducted using either a single P100 or V100 GPU on cloud machine. We used preliminary exploratory training
to determine the necessary number of epochs ensure convergence within the initial single round of training. In these longer training runs we did not find evidence of policy collapse Dohare al. (2023c), and thus believe that performance degradation in subsequent rounds can be attributed to plasticity loss. Code which can be used to reproduce our results available at http:///wwwaaaai/deeee---tto
Gridworld. The gridworld environment consisted of an 1xx one-hot encoded observation space. Each observation was flattened into single vector and provided input to the model. In the permute environment variation, observations were split into 1xlx4 patches and randomized according to fixed map within each round. The PPO model consisted of an MLP with two hidden layers, ReLU activations, and a "dual-head" architecture, as described in Schulman et al. (2017). The action space consisted of four possible actions, corresponding to movement in the cardinal directions. All episodes lasted 100 time-steps. Each round of training consists of 20,000 iterations of inference and model updates. Given rounds, the total number of iterations is 200,000.
CoinRun. The CoinRun environment consisted of 64x64x3 observation space. This was provided to the PPO model tensor, and processed by a convolutional encoder with four layers and ReLU activations. Each convolutional layer utilized a kernel size four and stride of two. In the permute environment variation, images were broken into 8x8x3 patches and the position of patches was randomized according to a fixed map within each round. A "dual-head" architecture was also utilized. The action space consisted of the standard nine possible actions provided by the ProcGen environment Cobbe et al. (2020). Episode length varied based on agent behavior. Each round of training consists of 5,000 iterations of inference and model updates. Given ten rounds, the total number of iterations is 50,000.
Montezuma's Revenge. Like in CoinRun, Montezuma's revenge states are processed into obser- vations of size 64x64x3 and passed through convolutional encoder. We inherit hyperparameters from Burda et al. (2018), which were previously optimized for final performance. These include 4 randomly masked update epochs per observation, an entropy coefficient 0.001, TE = 0.999 and y =0.99, We use a learning rate of 0.0001 and 128 simultaneous agents simultaneously. All model updates are performed via the Adam optimizer. We use a shrinkage coefficient 10 and perturbation coefficient of 10
13

---

C Hyperparameters
Method
Gridworld
CoinRun
L2-Norm (a)
le-3
le-3
Regen-Reg (a)
le-4
le-2
Soft-S+P (3)
le-6
le-6
S+P (5)
0.5
0.5
ReDo period
10 epochs
10 epochs
ReDo (r)
0.025
0.025
Table 1: Optimal hyperparameter values used for different environments and interventions. Chosen
values are the result of performing sweep over possible values using permute environment shift condition.
Parameter
Value
Number of hidden units
256
Learning rate
5e-4
Default activation
ReLU
Minibatch size
64
Buffer size
1024
Discount (7)
0.99
GAE parameter (A)
0.95
Clip parameter (e)
0.2
Entropy parameter (3)
0.02
Number of update epochs
Table 2: Default values used in PPO algorithm for all gridworld and CoinRun experiments.
14

---

D Extended Results
LaverNorm
Reseball
Shrisk+ +pectarb
Injection
12 pores
Soit pertuke-
Roset-Siull
GridWorld
(A) Permute
(A) Window
(A) Expand
0.5
Heward (tram)
Rowaed Oraind
0.0
0.0
-0.5
-34
2
5
0
8
5
Round
Bound
Round
Permute
(B) Window
Expand
0.25
1.5
Beward (bea)
0.00
0.5
(bost)
1.0
-0.25
-0.50
0.0
0.0
-0.3
Figure 7: Performance of intervention methods compared to baseline (warm-start) on Gridworld task. Shaded region corresponds normalized mean episodic reward. Tow row: Training distribution performance, Bottom row: Test distribution performance.
Warm start
Regen reg (L2)
CReLU
LayerNorm (1N)
Regen rag (122 LN
Reset
Staink pertuuu
Injection
RaDo Reset
shrink
noem
Reset-fina
CoinRun
(A) Permate
(A) Window
(A) Expand
2.5
Roward (traac)
-2.5
-6-
7..5
-7.5
-10.0
10
0
3
8
0
2
3
4
5
7
9
2
6
9
Round
Round
Hound
Permute
Window
Expand
0
Hewcard (bett)
Rewardd (text)
0.0
-25
5.0
8
-7.5
Round
Figure 8: Performance of intervention methods compared to baseline (warm-start) on CoinRun task.
Shaded region corresponds to normalized mean episodic reward. Top row: Training distribution performance. Bottom row: Test distribution performance.
15

---

Bagen (122)
Reset-tnal
Reseball
Shrink- perturb
Injection
Layaroorm (1.N)
Regen reg (12) 1M
shrink-perturs
Jumper
Window
(A) Expand
Roward (tratel
Reward strate)
N
Reward strated
-4
Round
(D) Permute
(B) Window
(B) Expand
Beward (test)
Nerwant (teett
Rewant Deent
56
Figure 9: Performance of intervention methods compared to baseline (warm-start) on Jumper task.
Top row: Training distribution performance. Bottom row: Test distribution performance.
Regen reg
LaverNorm
Regen rog Suft shrinkkpertuc
Reset-all
Starink partud
Injection
RaDo Reset
norm
Soft shrinic sportur
Rewttffinl
FruitBot
Permute
Window
Expand
Reward (traic)
Reward (trate)
Reward (trans)
-10
(B) Permute
(B) Window
(B) Expand
Roward ttet0
Koward (hest)
Roward thest
o 10
0.
Figure 10: Performance of intervention methods compared to baseline (warm-stare start) on Fruitbot task.
Top row: Training distribution performance. Bottom row: Test distribution performance.
16

---

Warm-ttart
shrink-gerturr
12 zorm
Injection
Reset all
Regen reg (L2)
Reset rital
ReDe Reset
Shiink perturh
Pollcy entropy (Permute)
Pollcy entropy (Window)
Policy entropy (Expand)
0.25
Palky ectropp
0.00
Polty estropp
-0.1
0.0
-0.50
0.2
-0.2
Weight megnitude (Permute)
Weight magnitude (Window)
Weight magnitude (Expand)
Weight megnituuk
Weight mognittsh
0.6
0.4
0.4
0.4
Wiight
0.2
0.2
5
Round
Bound
Round
Weight difference (Permute)
Weight ht difference (Window)
Weight difference (Expand)
Wountt aiterenze
Wought differenee
0.0002
0.00000
0.0000
0.0002
0.00025
-0.0002
Weight
0.0000
-0.00050
-0.0004
-0.0002
Gradient norm (Permute)
Gradient porm (Window)
Gradient norm (Expand)
Gradbwe norm
1.5
Greedinn
5
6
4
5
2
8
5
Round
Rouzd
Round
units (Permute)
(Window)
(Expand)
0.04
0.02
0.00
Doad
Doad
0.02-
0.04
Round
Figure 11: Effect of learning under various conditions five diagnostic metrics of interest.
17

---

Dep. Variable:
No. Observations:
27
Model:
GLM
Df Residuals:
21
Model Family:
Gaussian
Dr Model:
5
Link Function:
Identity
Scale:
0.13264
Method:
IRLS
Log-Likelihood:
-7.6473
Deviance:
2.7855
Pearson chi2:
2.79
No. Iterations:
3
Pseudo R-squ. (CS):
0.5624
coef
std
P>
[0.025
0.975]
const
-0.0920
0.118
-0.783
0.434
-0.322
0.138
entropy
1,6302
0.940
1.734
0.083
-0.213
3,473
weight_maa mag
-1.8262
0.637
-2.868
0.004
-3.074
-0.578
weight_din diff
-35.4734
133.365
-0.266
0.790
-296.865
225.918
grad norm
0.5742
0.184
3.116
0.002
0.213
0.935
dead units
0.5074
5,905
0.086
0.932
-11.066
12.081
Table Results of GLM using various diagnostic metries to predict reward during training.
Dep. Variable:
test_ reward
No. Observations:
27
Model:
GLM
Df Residuals:
21
Model Family:
Gaussian
Df Model:
Link Function:
Identity
Scale:
0.19798
Method:
IRLS
Log-Likelihood:
-13.054
Deviance:
4.1576
Pearson chi2:
4.16
No. Iterations:
3
Pseudo R-squ. (CS):
0.5215
coef
std err
P> |z|
[0.025
0.975]
const
0.3249
0.144
2.263
0.024
0.043
0.606
entropy
-1.4060
1.149
-1.224
0.221
-3.657
0.845
weight_mas mag
1.1259
0.778
1.447
0.148
-0.399
2.651
weight ddff diff 327.8923
162.934
2.012
0.044
8.547
647.238
grad_norm -0.0568
0,225
-0,252
0.801
-0.498
0.384
_ead_units -21.6282
7.214
-2.998
0.003
-35.768
-7.488
Table Results GLM using various diagnostic metrics to predict reward on the test distribution.
Gridworld (Permute Train)
Method
t-value
p-value
Warm-start
((7) = 2.524
0.04
CReLU
(77)
0.0
Injection
((8) 6.21
0.0
Reset-final
((7) = 22.444
0.044
1.2 norm
((7) -0.08
0.939
Shrink+perturb
(8) -0.087
0.933
Soft shrink+perturb
((8) = 0.472
0.65
Regen reg (L2)
((8) 0.134
0.897
LayerNorm (L.N)
((8) 0.787
0)4554
Soft phrink+perturb LN
((8) = -0.212
0.837
Regen reg (L2) LN
(88 -1.368
0.208
Table 5: Results of statistical tests comparing different methods to baseline of reset-all in gridworld (permute) condition. T-values below zero correspond to train distribution performance better than baseline. T-values above zero correspond to performance worse than baseline. P-values above 0.05 correspond to methods which produce mean normalized rewards not significantly different from the baseline.
18

---

Gridworld (Window Train)
Method
t-value
p-value
Warm-star
((8) = 0.759
0.469
CReLU
((7) = -1563
0. 0.162
Injection
((8) 1.741
0.12
Reset-final
((8) = -0.017
0.987
L2 norm
(() = -1.291
0.233
Shrink+perturt
(88 -1.611
0.211
Soft shrink+perturb
((7) = -0.222
0.831
Regen reg (1.2)
(8) = =1.068
0.317
LayerNorm (LN)
((7) -2.107
0.073
Soft shrink+perturb. + LN
((8) =1.5449
0.16
Regen reg (L2) + LN
t(8) -3.106
0.015
Table 6: Results of statistical tests comparing different methods to baseline of reset-all in gridworld (window) condition. T-values below zero correspond to train distribution performance better than baseline. T-values above zero correspond to performance worse than baseline. P-values above 0.05 correspond to methods which produce mean normalized rewards not significantly different from the baseline.
Gridworld (Expand Train)
Method
t-value
p-value
Warm-start
((8) = -0.926
0.382
CReLU
1(8) -2.247
0.055
Injection
((8) -0.715
0.495
Reset-final
((8) -1.301
0.229
L2 norm
((8) -2.252
0.054
Shrink+perturb
(88 = -2.476
0.038
Soft shrink+perturb
t(7) = -1.788
0.117
Regen reg (L2)
((8) = -2.899
0.021
LayerNorm (LN)
((8) = -4.098
0.003
Soft shrink+perturb LN
-3.683
0.006
Regen reg (L2) + LN
((8) = -5.44
0.001
Table Results of statistical tests comparing different methods to baseline of reset-all in gridworld (expand) condition. T-values below zero correspond to train distribution performance better than baseline. T-values above zero correspond to performance worse than baseline. P-values above 0.05 correspond to methods which produce mean normalized rewards significantly different from the baseline.
Gridworld (Permute Test)
Method
t-value
p-value
Warm-start
t(7) 3.0
0.02
CReLU
((7) 6.975
0.0
Injection
1(8) 3.595
0.007
Reset-final
((7) 2.445
0.044
1.2 norm
(7)=3.00)
0.02
Shrink+perturrh
((8) 0.907
0.391
Soft shrink+perturb
((8) = 2.177
0.061
Regen reg (L2)
((8) = 0.007
0.995
LayerNorm (LN)
((8) 6.959
0.0
Soft shrink+perturt LN
((8) 0.235
0.82
Regen reg (L2) + LN
t(8)= 0.804
0.445
Table Results of statistical tests comparing different methods to baseline of reset-all in gridworld (permute) condition. T-values below zero correspond to test distribution performance better than baseline. T-values above zero correspond to performance worse than baseline. P-values above 0.05 correspond to methods which produce mean normalized rewards not significantly different from the baseline.
19

---

Gridworld (Window Test)
Method
t-value
p-value
Warm-start
((8) = 0.882
0.404
CReLU
((7) 0.239
0.818
Injection
((8) = 1.444
0.187
Reset-final
((8) 0.804
0.445
1.2 norm
((8 0.216
0.834
Shrink+perturb
(88) = -0.781
0.458
Soft shrink+perturb
t(7) -0.046
0.965
Regen reg (L2)
( -0.187
0.856
LayerNorm (LN)
t(7) = -0.813
0).443
Soft shrink++prruurt + LN
((8) -2.349
0.047
Regen reg (L2) + LN
((8) = -1.714
0.125
Table 9: Results of statistical tests comparing different methods to baseline of reset-all gridworld (window) condition. T-values below baseline. T-values above zero correspond to performance worse than baseline. P-values above 0.05 correspond to methods which produce mean normalized rewards not significantly different from the baseline.
correspond to distribution performance better than
Gridworld (Expand Test)
Method
t-value
p-value
Warm-start
1(8) = 1.227
0.255
CReLU
((8) 0.891
0.399
Injection
((8) = 1.815
0.107
Reset-finll
((8) 1.457
0.183
L2 norm
t(8) -0.512
0.623
Shrink+perturb
((8) = -1.021
0.337
Soft shrink+perturb
((7) -0.227
0.614
Regen reg (L2)
(8) 0.333
0.748
LayerNorm (LN)
((8) = 0.438
0.673
Soft shrink+perturb LN
-1.793
0.111
Regen reg (L2) + LN
((8) = -0.579
0.579
Table 10: Results of statistical tests comparing different methods to baseline of reset-all in gridworld (expand) condition. T-values below baseline. T-values above zero correspond to performance worse than baseline. P-values above 0.05 correspond to methods which produce mean normalized rewards not significantly different from the baseline.
correspond test distribution performance better than
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract accurately reflects all the findings presented the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed throughout results and discussion sections.
Guidelines:
The answer NA means that paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" " section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
21

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The appendix sections contain the necessary hyperparameters and implementa- tion details reproduce our work. Furthermore, we provide code to reproduce the results and figures.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
22

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide as supplemental file the code necessary to reproduce all results and figures excluding the results presented in section 4.7 (Montezuma's Revenge).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. Sttaddeee/oooeSSmmiiiiiiioo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //iips. wttp:cccpppppccceeeeee.ec.ccccee httpl////guiees/deeuuubiiissee //t////ooeeSesseeeeeeeeee ssionpolicy)) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:[Yee Yes]
Justification:
The results and appendix provides all necessary experimental details fully understand and interpret our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
23

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use error hars for all figures. Furthermore, we provide statistical tests where relevant in the main body of the text, along with more extended statistical results in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the compute resources used in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ httppp/..s..ppblllechccccseeeeee
Question:
Answer: [Yes]
Justification:
Our research fully conforms to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include an impact statement at the end of the main body of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There are no risks associated with our paper or code.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly attribute all relevant software and datasets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification; We provide code to reproduce our results, and this code contains documentation necessary for rumning it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
26

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There were no human subjects involved in our research.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There were no human subjects involved in our research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "gV9qLtAqkT", "reviewer": "Reviewer_UXgL", "summary": "This paper studies the problem of plasticity loss in on-policy deep RL. The study is quite wide as it covers many environments, types of non-stationarities, and solution methods. The first main result is the demonstration of plasticity loss in various settings. The second main result is the analysis of existing methods in the problems. The results show that regularization methods effectively mitigate plasticity loss in this setting.", "questions": "What values of $\\beta$ were used for Adam? Dohare et al. found that equal $\\beta$s in PPO (particularly when used with ReLUs) significantly improve the performance when evaluated over a long time horizon, as may be the case in your experiments.\n\nDohare et al. Overcoming policy collapse in Deep RL. EWRL 2023.", "limitations": "The authors adequately discuss the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is the first study that extensively studies plasticity loss in on-policy RL. Although some prior work has shown plasticity loss in on-policy RL, this study is much more extensive. The paper is generally well-written. The paper claims to be an extensive study of plasticity loss in on-policy learning, and it supports that claim by evaluating a wide range of methods in a wide range of environments. There are some minor weaknesses, but the overall paper is good and would be a good contribution to the plasticity loss literature. I recommend accepting the paper.", "weaknesses": "The statistical significance of the results is unclear. How many runs were performed for all experiments? 5? And what do the shaded regions represent in the figures? All the figure captions should contain details about the number of runs and shaded regions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate you taking the time to read and review our paper.\n\nWe used five replicates per experiment unless indicated otherwise, and the shaded regions of the graphs correspond to standard error. We will revise all figure captions to make both of these points explicit. \n\nWe used a learning rate of 5e-4 for all experiments, as displayed in Appendix C. We further use  the default PyTorch values for Adam's $\\beta_1$ and $\\beta_2$, and we will update the section to reflect this detail.\n\nWe agree that it is possible that using the “Non-stationary Adam” method from Dohare et al. would improve performance on the tasks considered here. That said, we do not find evidence of policy degradation within-rounds (see for example Figure 2 upper-left in contrast to Figure 1 of Dohare et al.). Note that we chose the number of training epochs to reflect the asymptotic behavior of the policy—longer-running experiments, which were used to select this parameter, did not show evidence of policy collapse when the task/environment distribution is held fixed. Regardless, we will include a discussion “Non-stationary Adam” in our related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZfPBOeE4Vf", "reviewer": "Reviewer_gfpF", "summary": "This work studies the loss of plasticity phenomenon in the on-policy continual deep RL setting, where previous work has focused on studying and identifying mitigation strategies for the off-policy RL or supervised learning settings. They conduct experiments over a variety of settings (gridworld, CoinRun, and Montezuma’s revenge) over different variants of environment distribution shift, demonstrating that loss of plasticity still occurs in the on-policy regime. They perform a further analysis of the correlations in both train and test performance with various quantities studied in previous works, across several mitigation strategies previously presented. Based on this, they provide several hypotheses for what properties are needed for successful intervention at addressing plasticity loss as well as ensuring good generalization performance.", "questions": "- What is the reason for adding LN to soft shrink and perturb, versus original shrink and perturb? It seems that shrink and perturb seems to be doing better than the soft variant for most of your results, and I would guess that the procedure being applied after each step of gradient descent instead of longer intervals could hurt performance.\n- Do you have any insights why gradient magnitude/weight change magnitude did not have a significant correlation with plasticity or generalization in your setting, in contrast to previous work?\n- Your Montezuma's Revenge experiments only analyze two intervention methods. How should I interpret the reward numbers in Figure 6? Do you have similar quantities (weight magnitude, gradient norm, dead units, etc.) for these methods?", "limitations": "Authors acknowledge their correlation plots do not attribute causality. Experiments are comprehensive in testing intervention methods except Montezuma's Revenge, which the authors address.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Paper is overall well-written and particularly introduces the loss of plasticity phenomenon/warm start problem well.  Empirical investigations of loss of plasticity for continual RL are important for the community.\n- Experiments are comprehensive and presented clearly, including domains of varied complexity. The authors took care in implementing several intervention methods that have arose previously in the literature, and report correlation results comprehensively.\n- There are new insights about previously successful intervention methods not working in their setting, such as concatenated ReLUs and plasticity injection. From their correlation results, they provide connections between their most successful methods (regularization-based) to the greatest predictor of plasticity loss (weight magnitude, and surprisingly not gradient magnitude or magnitude of weight change).\n- There are interesting next directions exploring mechanisms of plasticity loss and using these metrics as an indicator for what is occurring in the optimization landscape for continual reinforcement learning.", "weaknesses": "- The graphs are difficult to read, given the number of interventions and the colors chosen. It’s difficult for me to see, eg. how much combining soft shrink + perturb with LN improves upon only doing one or the other.\n- In Appendix D, it might be useful to highlight the significantly improved intervention methods for each environment and shift condition, and some written insight about the table results.\n\nMinor typos:\n- Line 24: arrive -> arrives\n- Line 142: withing -> within", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our paper. \n\nBoth you and another reviewer have brought up the readability of our graphs. We are working to improve this for the camera-ready version. Reviewer NjbE suggested utilizing the “rliable” library to generate plots. We took their suggestion and generated a potential new version of Figure 4 using the library, which you can see [here](https://ibb.co/HxJcVmh). Please let us know if it improves readability, and if so we will replace both Figure 4 and 5 with this new layout and move the current figures to the appendix.\n\nThank you for pointing out the typographical errors. We have addressed them in our current draft of the manuscript.\n\nThe issue with the shrink-perturb method is that it is performed only occasionally, rather than at every step of SGD. This is fine in the original context under which it was introduced, where the problem setting enjoys a clear indicator of when it should be applied. In non-contrived RL settings, however, the environment might not provide this information—an implementation of shrink-perturb would either require us to either set the intervention cadance as a hyperparameter or to use additional machinery to detect appropriate points for application. We refer to methods that are applied at every step of SGD (like soft shrink perturb) as continuous, and we argue that this property is desirable in realistic RL settings.\n\nIt is worth clarifying our results concerning predictive measures for plasticity loss. We find that weight magnitude and dead unit count, when considered independently, are the two significantly predictive metrics of plasticity loss. Although we find that by itself gradient magnitude is not predictive, it becomes predictive as part of a Generalized Linear Model (GLM) which uses all the measures as predictors. This suggests that gradient magnitude is able to account for additional variance in plasticity loss which is not accounted for by weight magnitude, suggesting that it may contribute to some aspect of plasticity loss. In contrast, dead unit count is no longer predictive in the GLM due to its high correlation with weight magnitude, which is the more fundamental variable (because dead unit count is a function of weight magnitude). \n\nWe believe our results deviate from previous findings largely because of the breadth of settings we considered. Rather than studying only a single task, type of distribution shift, and model architecture, which might provide only narrow insight into the plasticity loss phenomenon, we explore several. Of course, our study also deviates from prior work in that we also consider only on-policy agents, and the underlying dynamics of plasticity loss in the two settings may differ.\n\nWe considered Montezuma's Revenge to be a more \"natural\" RL task, presented to show a use for what we have learned from environments studied earlier in the paper, which were chosen specifically to probe the plasticity loss pathology. As such, we unfortunately did not log the same diagnostic information for Montazuma's Revenge as we did for earlier tasks. Fitting these agents takes more time than is afforded in the NeurIPS discussion period, but we will collect this information for the camera ready copy.\n\nThe reward on the y-axis of Figure 6 is the agent's score in the game, reflecting how many objects it has collected, new rooms it has entered, and enemies it has vanquished. Our code for this is only a small modification to a popular PyTorch RND implementation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TFxaknNVHu", "reviewer": "Reviewer_NjbE", "summary": "The article introduces an extensive empirical analysis of plasticity loss in on-policy reinforcement learning (RL), focusing on Proximal Policy Optimization (PPO). The main findings include that plasticity loss is also present in on-policy RL and that “regenerative” methods that regularly grow network parameters work well in this setting. \nThe empirical evaluation consists of ProcGen and ALE benchmarks. Additional environmental distribution shifts were introduced to study the plasticity loss root cause thoroughly. \n8 previously introduced methods to counteract plasticity loss in supervised and off-policy RL are examined, namely L2 Norm, LayerNorm, CReLU activations, Regenerative Regularization, Resetting final layer, Shrink+Perturb, Plasticity Injection, and ReDo. The work presents and analyzes metrics, such as weight magnitude, number of dead neurons, and gradient norm, previously related to the loss of plasticity.", "questions": "Regarding deeper insights:\n\n1.    In off-policy methods, it has been shown that the critic network mainly loses plasticity. Can the authors comment on their on-policy experiments through this lens? Specifically, is resetting only the actor's or only the critic's head more or less beneficial? What role does the common backbone play in this problem? If you separate the actor and critic networks, which can overall impact performance [1], will the conclusions remain similar?\n\n2.    What happens if we increase the number of epochs? If we combine this with the examined methods, how will it impact the final performance? Does increasing the number of epochs hurt performance due to more and more outdated samples, or is it primarily due to plasticity loss?\n\n[1] Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., ... & Bachem, O. (2020). What matters in on-policy reinforcement learning? A large-scale empirical study. arXiv preprint arXiv:2006.05990.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The strong part of the work is the extensive evaluation setup of on-policy RL. The claims are well supported by experiments.\nThe paper also interestingly indicates that the problem of warm-start might be partially mitigated by “regenerative” methods that target weight magnitude growth. This has the downstream effect of minimizing the number of dead units in the network previously connected to the plasticity loss.", "weaknesses": "While the experimental setup is comprehensive, the findings could benefit from deeper analysis. The article does not present new insights into plasticity loss nor propose novel methods to mitigate it, primarily noting that the problem also exists in on-policy RL. A more thorough exploration of the interactions between methods and their impact on plasticity would significantly enhance the work. In particular, it would be beneficial to understand the differences in plasticity loss dynamics between off-policy and on-policy methods.\n\nAdditionally, some of the figures are overly cluttered, making them difficult to interpret quickly. To improve clarity, consider summarizing the results from Figures 4 and 5 into a single scalar per method and moving the detailed figures to the appendix. I also suggest using the rliable [1] library for more effective results aggregation.\n\n\n[1] https://github.com/google-research/rliable", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and provide helpful feedback. \n\nOur intention with this work was to focus on the on-policy setting, given that the majority of the work in the area of plasticity loss has focused exclusively on the off-policy setting. Our decision not to include off-policy experiments was based on the fact that we believed there was ample evidence characterizing the problem and various solutions already existing in the literature. We agree that a direct comparison utilizing a single codebase between the two may have presented additional value, but we chose to instead allocate resources towards extensive e experiments specifically in the on-policy setting.\n\nWe agree that the information in Figures 4 and 5 could be presented in a more interpretable way. Our goal is to allow readers to see the entire trends for each method, but given the number of methods and task conditions, this presents a challenge to represent succinctly. Thank you for pointing out the rliable library. We have used it to generate a potential replacement for Figure 4, which you can find [here](https://ibb.co/HxJcVmh). If it seems like an improvement, we will also replace Figure 5 using the same layout, and move the current figures to the appendix.\n\nIn our experiments which involved resetting the “final layer”, we reset both the critic and policy heads. Preliminary experiments (not reported in the paper) suggested that resetting both was superior to resetting either individually. We believe that this differs from the off-policy setting due to the critical role of the policy in the collection of samples. Whereas in off-policy learning there is a stable buffer of data to draw on, in the on-policy setting the data distribution used for training is much more sensitive to the policy. As such, resetting the policy head increases the overall policy entropy, thus ensuring that the data distribution used for training can remain diverse enough to prevent policy collapse. That said, the majority of methods considered here act on the entire network, rather than just the final layers of the network. As such, we believe that our results are largely robust to this choice. Relatedly, because we were considering discrete action policies, we utilized a shared encoder network, as this is the standard set-up in the PPO algorithm.\n\nThe total number of epochs was chosen such that convergence would always be reached within a single round of the experiment. As such, running experiments longer did not result in measurably different behavior than what has been discussed in this paper, and we did not find policy collapse or degradation within a single round. We believe that this suggests that the effects we observe in our experiments are indeed the result of plasticity loss. We will state this clearly in the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To all reviewers:

We thank the reviewers for their time and insights. Two reviewers have suggested clarity improvements to Figures 4 and 5. We have updated Figure 4 using the rliable library suggested by Reviewer NjbE (https://ibb.co/HxJcVmh). If reviewers agree that it is an improvement, we will also update Figure 5 and similar Appendix figures using the same layout.

We respond to individual concerns below.

Meta-review: This paper performs an extensive empirical analysis of plasticity loss in on-policy reinforcement learning methods. All reviewers praised the provided analysis and the contributions of the paper. Everyone agrees the paper should be accepted and I am recommending the paper's acceptance.
Decision: Accept (spotlight)