Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
Authors: Huaicheng Zhou
Zifeng Zhuang
Donglin Wang
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Plasticity
Primary area: deep_learning

Abstract: The integration of Deep Neural Networks in Reinforcement Learning (RL) systems has led to remarkable progress in solving complex tasks but also introduced challenges like primacy bias and dead neurons. Primacy bias skews learning towards early experiences, while dead neurons diminish the network's capacity to acquire new knowledge. Traditional reset mechanisms aimed at addressing these issues often involve maintaining large replay buffers to train new networks or selectively resetting subsets of neurons. However, these approaches either incur prohibitive computational costs or reset network parameters without ensuring stability through recovery mechanisms, ultimately impairing learning efficiency. In this work, we introduce the novel concept of neuron regeneration, which combines reset mechanisms with knowledge recovery techniques. We also propose a new framework called Sustainable Backup Propagation(SBP) that effectively maintains plasticity in neural networks through this neuron regeneration process. The SBP framework achieves whole network neuron regeneration through two key procedures: cycle reset and inner distillation. Cycle reset involves a scheduled renewal of neurons, while inner distillation functions as a knowledge recovery mechanism at the neuron level. To validate our framework, we integrate SBP with Proximal Policy Optimization (PPO) and propose a novel distillation function for inner distillation. This integration results in Plastic PPO (P3O), a new algorithm that enables efficient cyclic regeneration of all neurons in the actor network. Extensive experiments demonstrate the approach effectively maintains policy plasticity and improves sample efficiency in reinforcement learning.

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
Huaicheng Zhou'
Zifeng Zhuang'
Donglin Wang'
Abstract
1. Introduction
The integration of Deep Neural Networks in Re- inforcement Learning (RL) systems has led to remarkable progress in solving complex tasks but also introduced challenges like primacy bias and dead neurons. Primacy bias skews learning towards early experiences, while dead neurons diminish the network's s capacity to acquire new knowledge. Traditional reset mechanisms aimed at addressing these issues often involve maintain- ing large replay buffers to train new networks or selectively resetting subsets of neurons. However, these approaches either incur prohibitive compu- tational costs reset network parameters without ensuring stability through recovery mechanisms, ultimately impairing learning efficiency. In this work, we introduce the novel concept of neuron regeneration, which combines reset mechanisms with knowledge recovery techniques. We also pro- pose a new framework called Sustainable Backup Propagation(SIPP) that effectively maintains plas- ticity in neural networks through this neuron re- generation process. The SBP framework achieves whole network neuron regeneration through two key procedures: cycle reset and inner distillation. Cycle reset involves a scheduled renewal of neu- rons, while inner distillation functions as a knowl- edge recovery mechanism at the neuron level. To validate our framework, we integrate SBP with Proximal Policy Optimization (PPO) and propose a novel distillation function for inner distillation. This integration results in Plastic PPO (P30), a new algorithm that enables efficient cyclic regen- eration of all neurons in the actor network. Ex- tensive experiments demonstrate the approach ef- fectively maintains policy plasticity and improves sample efficiency in reinforcement learning.
Deep reinforcement learning has advanced significantly through the integration of deep neural networks, resulting in notable achievements across various domains (Singh et al., 2022; Arulkumaran et al., 2017; Yu et al., 2021). Despite these advancements, a critical issue that has emerged is the loss of plasticity, as detailed in (Lyle et al., 2023; Abbas et al., 2023). This refers to the diminishing ability of a network learn and adapt over time. As network neurons become saturated, they "become full", losing the capacity to incorporate new information effectively. This reduction in plasticity primarily affects the neurons in the network, leading to decreased effectiveness and eventually causing neurons to become dead (Lu et al., 2019; Shin & Karni- adakis, 2020) or dormant (Sokar et al., 2023b). Addition- ally, the problem of overfitting in deep learning, known as primacy bias (Nikishin et al., 2022), further causes this loss of plasticity. Consequently, there is urgent imperative to develop mechanisms for the repair or revitalization of neu- rons affected by primacy bias or those that have lapsed into dormancy, with the objective of reawakening their "hunger" for novel information.
Reset mechanisms have been proven to be effective mea- sures for addressing the loss of plasticity in neural networks. However, existing reset approaches have demonstrated vari- ous limitations. Early studies (Nikishin et al., 2022: D'Oro et al., 2022; Kim et al., 2024) proposed resetting either the final layer or all neurons to revitalize learning capabili- ties, but these methods often led to a performance-resource trade-off, requiring additional training to recover lost per- formance. More targeted approaches, such as CBP (Dohare et al., 2021) and ReDo (Sokar al., 2023b), focused on selectively resetting non-contributing neurons. While this strategy reduced information loss, it only partially restored plasticity and achieved limited performance improvements. A fundamental challenge across these methods that neu- rons reaching critical importance become either irresetable or damaging when reset, creating a dilemma that renders the reset strategy ineffective and potentially limiting the network's 's overall potential. This limitation underscores the need for more sophisticated approach that can enhance network plasticity while maintaining critical knowledge.
'School of Engineering, Westlake University, Hangzhou, China. Correspondence to: Donglin Wang <wang- donglin @ wwwslake..du..cn..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
epoch=20
epoch=40
epoch=80
epoch=100
Averag Return
4000
Avenae AvenggeRRe Return
4000
600
evggg
2000
2000
-400
11
10 12
9 1011
Environment Steps (Million)
Environment Steps (Million)
Enviromment Steps (Million)
(a) Humanoid
(b) Hopper
(c) Walker2d
Figure 1. Performance of PPO across varying numbers of training epochs of actor network per batch. Increasing training epochs impedes performance improvement.
Inspired by regenerative processes in human cells (Carlson, 2011), we propose the concept of neuron regeneration, biomimetic approach that aims to recover plasticity while preserving crucial knowledge within the network. Simi- lar to how biological cells maintain their vitality through regeneration, our neuron regeneration mechanism enables the renewal of neural network components while retaining essential learned information. This process enables neu- ral networks to maintain long-term learning capabilities without performance degradation. To implement neuron regeneration, we introduce the Sustainable Backup Prop- agation (SBP) approach, which integrates reset and distil- lation mechanisms into traditional backpropagationn(Hcchh Nielsen, 1992). Drawing inspiration from natural cellular regeneration cycles(Sender & Milo, 2021), SBP employs cyclic reset strategy mitigate neuron plasticity degrada- tion and primacy bias. The key innovation lies in our inner distillation process, which facilitates knowledge transfer from reset neurons to others, ensuring effective regeneration without performance loss. This approach not only maintains the network's 's capacity to absorb new information but also perpetuates its learning process, maximizing its potential for continuous growth.
were conducted across diverse environments, including Mu- JoCo (Todorov al., 2012), DeepMind Control Suite (Tassa et al., 2018), and a specially designed MuloCo variant called Cycle Friction that tests adaptation to changing dynamics. Results demonstrate that P30 consistently outperforms stan- dard PPO, achieving both higher average returns and more stable learning curves, validating the effectiveness of our neuron regeneration mechanism in maintaining policy plas- ticity while enhancing sample efficiency. Our contributions in this work can be summarized as follows:
Neuron Regeneration: We introduce the concept of neuron regeneration, a biomimetic approach inspired by cellular regeneration processes. This novel mecha- nism maintains neural network plasticity while preserv- ing learned knowledge, enabling continuous learning without performance degradation.
Sustainable Backup Propagation (SBP): We propose SBP, systematic framework that implements neu- ron regeneration through cyclic reset strategies and in- ner distillation mechanisms. By effectively addressing dead neurons and primacy bias, SBP ensures sustain- able plasticity throughout the network's lifecycle.
Plastic PPO (P3O): We introduce P3O, an enhanced version of PPO that integrates SBP and novel weighted Double KL divergence (a-DKL) loss func- tion. P3O overcomes the primacy bias problem in stan- dard PPO, maintaining plasticity and improved sample efficiency across various reinforcement learning tasks.
As shown in Figure I, Proximal Policy Optimization (PPO) (Schulman et al., 2017) exhibits significant primacy bias, where early training experiences disproportionately influ- ence the final policy, leading to performance degradation as training progresses. This degradation stems from the loss of neuron plasticity, limiting the network's ability to adapt to new experiences. To address this limitation, we integrated our SBP approach with PPO by incorporating neuron regeneration into the policy network. We devel- oped novel a- a-weighted Double KL divergence (a-DKL) loss function that dynamically balances knowledge preserva- tion and update flexibility. This loss function employs two KL terms with an weighting mechanism to effectively fil- ter out harmful information while retaining valuable policy knowledge during the regeneration process. The resulting Plastic PPO (P30) algorithm implements neuron regenera- tion within the actor network to maintain learning efficiency and achieve sustainable plasticity. Extensive experiments
2. Preliminaries and Related Work
2.1. On-policy Reinforcement Learning
Reinforcement learning is formalized as Markov decision process (MDP) (Puterman, 2014). An MDP consists of a tuple (S, A, R, P, 7), where S denotes the set of states, A the set of actions, R: A x S S A + P(S) a possibly stochastic transition probability function, 7 E [0. 1) the discount factor. In reinforcement learning. the goal is to seek an optimal policy x² S + P(A) which maximizes the expected accumulated returns with discounted.
a reward function, P
2

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
In on-policy reinforcement learning, the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) is utilized to update policies during interaction with the en- vironment. The core objective function of PPO, denoted as (0), includes a clipping operation that sets gradi- ents to zero when the probability ratio re r0()) falls outside [1-e, [e]] This prevents the policy from learning from ad- vantages that would push it further outside the trust region, thereby enforcing trust region constraints in a computation- ally efficient manner. This operation ensures that the policy does not deviate excessively from the previous policy. The objective function is expressed as:
et al., 2023b) selectively reinitialize neurons deemed less useful based certain metrics, minimizing the impact on overall performance. This approach highlights the delicate balance between recovering plasticity and maintaining net- work efficiency. However, there remains a need for more sophisticated methods that can effectively regenerate neu- rons while preserving learned knowledge, which is the focus of our proposed approach.
Maintaining neural plasticity represents fundamental chal- lenge in reinforcement learning, as agents must persistently adapt to dynamic environments. Recent studies(D Oro et al., 2022; Schwarzer et al., 2023; Lee et al., 2024a; Ma et al., 2023; Nauman et al., 2024) have demonstrated that enhane- ing neural plasticity can lead to significant improvements in sample efficiency. Furthermore, both offline RL (Zhuang et al., 2023; 2024) and lifelong RL (Ahn et al., 2024) criti- cally depend on maintained plasticity, though they present divergent challenges: static data exploitation in the former versus continual adaptation in the latter.
Leip (0) = Er [min((: Au,
+e)l
(1)
[s(a))ss) Here, ry r0) Tawy [a)] represents the probability ratio
between new and old policies, and Ag is the estimated ad- vantage function. The advantage function measures the value of the current policy relative to a baseline policy, cal- culated from trajectory data acquired during interactions with the environment. In on-policy training environments with dynamic data, where both input data and target values nonstationary, network plasticity may be compromised, leading to suboptimal performance.
2.3. Policy Distillation in Reinforcement Learning
Policy distillation transfers knowledge from an RL agent to a smaller network for improved efficiency (Rusu et al., 2015). Prior research (Igl et al., 2020; Lyle et al., 2022) has highlighted its utility in addressing potential general- ization loss in deep reinforcement learning agents due to nonstationarity and overfitting. This suggests that policy distillation serves two main functions: transferring knowl- edge and enhancing generalization, aligning well with the goals of plasticity recovery such as preserving knowledge and mitigating primacy bias. A critical aspect of effectively implementing policy distillation is quantifying the quality of knowledge transfer, which necessitates appropriate diver- gence measures.
2.2. Plasticity in Reinforcement Learning
Plasticity, the ability of neural networks to adapt to new information, gradually declines over time, posing critical challenge. This issue has prompted extensive research(Lyle et al., 2023; Abbas et al., 2023; Nikishin et al., 2022; Nau- man et al., 2024; Dohare et al., 2024; Juliani & Ash, 2024; Lewandowski et al.) into methods for preserving neural network plasticity, with two main categories of approaches emerging to address this problem. The first involves vari- ous training techniques such as regularization, adjustments in activation functions, weight decay, and normalization strategies (Kumar et al., 2023; Delfosse et al., 2021; Lee et al., 2024a; Lyle et al., 2024; Elsayed et al., 2024). These methods delay plasticity loss by reducing overfitting and preventing large parameters.
A study by (Martins et al., 2021) discusses two types of Kullback-Leiller (KL) divergence measures: Forward KL (FKL) and Reverse KL (RKL). The Forward KL divergence, DRi weights the state space according to the teacher's pol- icy, prioritizing learning in states where the teacher's policy is more probable. Conversely, the Reverse KL divergence, DiL weights according to the student's policy, promot- ing exploration and robustness but risking neglect of some Fracher-faoree behaviors. Their expressions are:
The second category of approaches involves resetting the network (Nikishin et al., 2022; Schwarzer et al., 2023; Nik- ishin et al., 2024; Lee et al., 2024b) to recover the plasticity. This reset-based methodology addresses both neuron dor- mancy and primacy bias. Typically implemented by reinitial- izing weights of specific layers or the entire network, resets have been shown to effectively scale replay ratios, con- tributing to performance improvements (Kim et al., 2024; Xu et al., 2023). However, while resets can revive learn- ing capabilities, they might lead to temporary performance degradation and require additional training to restore pre- viously learned information. To mitigate these drawbacks, methods like CBP (Dohare et al., 2021) and ReDo (Sokar
DRi(TI || (2))= = Σ
M (s)))oog log
T
(2)
T2(s))
SES
(2())
DRi |l (1) = Σ (s) log
(s)
SES
To tackle the performance degradation challenge in plastic- ity recovery, we devise a weighted integration scheme that leverages the complementary characteristics of forward and reverse KI. divergence.
3

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
3. Neuron Regeneration
4. Methodology
Maximizing neural network capacity utilization during train- ing is challenged by diminishing plasticity, a consumable resource that depletes as networks learn. This depletion stems from suboptimal configurations such as inappropriate activation functions( Abbas et al., 2023), poor data quality (Lee et al., 2024a), and backpropagation limitations (Dohare et al., 2024), resulting in biased learning and suboptimal performance.
To maximize the utilization of neural network capabili- ties, we propose the Sustainable Backup Propagation (SBP) method. SBP achieves sustainable plasticity through a novel neuron regeneration mechanism, which combines two key components: Cycle Reset and Inner Distillation.
4.1. Sustainable Backup Propagation
Our neuron regeneration mechanism integrates two com- plementary components: Reset and Distillation. Reset re- juvenates inactive neurons by resetting their parameters to initial states, effectively discarding outdated knowledge. Distillation preserves valuable information by transferring knowledge from the pre-reset to post-reset network. To- gether, these processes enable neurons to regain plasticity while maintaining essential learned features. By ensuring ef- fective regeneration of individual neurons, we can maintain network-wide plasticity.
While neuron reset techniques offer promising solutions for plasticity restoration (D'Oro al., 2022), they introduce critical challenge: the risk of performance degradation without proper implementation (Nauman et al., 2024). Di- rect parameter reinitialization, while effective for restoring plasticity, can destabilize network performance.
To address this trade-off between plasticity restoration and performance preservation, we propose neuron regeneration as dedicated mechanism, defined as:
To implement network- neuron regeneration, we in- troduce the Cycle Reset mechanism, governed by two key parameters: Reset frequency and Reset rate p. Every F training steps, p% of neurons in each layer undergo reset and regeneration following cyclical order. Due to this sequential reset pattern, neurons with the longest survival time- those that have undergone more update steps and po- tentially experienced reduced plasticity- naturally become the targets of each reset cycle. This systematic process, il- lustrated in Figure 2, continuously refreshes neurons across all layers throughout training.
Definition 3.1: Neuron Regeneration
Given a neural network with parameters 0 {0), .2,, 6n), where each 9, represents the param- eters of the i-th neuron, let S S {1,2, .., n} be any subset of neuron indices. The neuron regeneration
operation is defined as:
0' N) =00 (8.,02....,6,, 6,
where:
Oplastic i S
8:
Inner Distillation completes the neuron regeneration process initiated by Reset. As shown in Figure 2, before resetting, the current policy is copied to a temporary policy Tem- Selected neurons then undergo reset, clearing outdated in- formation. Subsequently, through temporarily freezing reset neurons, Inner Distillation enables knowledge transfer from T&m to non-reset neurons in тe. with the flexibility to han- dle any number of reset neurons (even single one). This neuron-specific distillation process complements the Re- set operation, jointly achieving neuron regeneration that yields both enhanced plasticity and maintained or improved performance compared to the original network.
gnew
if S
Oplastic represents the reset plastic state and Orew represents the potentially updated state of non- regenerated neurons. A neuron regeneration op- eration is considered effective if it maintains:
1. Plasticity Recovery: P(O')= > P(O)
2. Performance Guarantee: Per(O') 2 Per(O)
where P denotes a measure of the network's 's plastic- ity and Per represents the network's 's performance.
The proposed neuron regeneration mechanism should com- prise two phases: a reset phase that initializes a subset of neurons to Optastic to enhance plasticity, followed by a re- covery phase that adjusts the remaining parameters grew to compensate for potential performance degradation. This dual-phase design aims to increase network plasticity P(0") while maintaining performance Per(O'). The adjustment of non-regenerated neurons preserves model stability, en- abling sustainable plasticity for long-term learning at the neuron level and addressing traditional limitations in neural network training methods.
The integration of Cycle Reset and Inner Distillation creates an effective neuron regeneration framework. While Cycle Reset maintains network-widk plasticity through systematic neuron regeneration, Inner Distillation ensures the preserva- tion of learned capabilities during this process. Through this balanced regeneration process, SBP enables neural networks to maintain suitable plasticity throughout training and ulti- mately maximize their learning capability utilization. The complete mechanism is formalized as the SBP algorithm, detailed in Algorithm 1.
4

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
reset
discili
nomral
integrate
orspty enepyy
reses
frzeze
Reser Sonee
empty neuron
fruzen connection emppty connection
froren neuron
Figure 2. Left: Cycle Reset. Right: Inner Distillation. The right figure illustrates our method that combines neuron reset and knowledge distillation, where reset neurons restore plasticity while transferring knowledge unreset neurons, thereby enhancing network plasticity while maintaining performance.
4.2. Double KI. Divergence
from adopting potentially harmful or irrelevant infor- mation from the policy Tem-
Our analysis of Figure I revealed that the PPO algorithm faces challenges with plasticity loss. To address this issue, we propose integrating the SBP framework into PPO, aiming to enhance its capabilities. The Inner Distillation process presents a complex scenario where neurons may contain both valuable knowledge and irrelevant information. To maximize generalization and plasticity, must carefully control this distillation process. In the context of reinforce- ment learning, we leverage the KL divergence as our distilla- tion function, building upon previous research(Martiin et al., 2021) that demonstrated the effectiveness both Forward KI. (FKI.) and Reverse (RKL) different aspects of knowledge transfer. Specifically, FKL has shown efficiency in transferring knowledge from teacher policy student policy, while RKL is effective in preventing the infiltration of potentially harmful knowledge into the student model. To capitalize on the strengths of both approaches, we introduce a combined method. We also incorporate a parameter a to adapt the loss function to various distillation scenarios. This results in our proposed a- weighted Double KL divergence (a-DKL), expressed as:
a E [0, is a tuning parameter that balances the con- tributions of the Forward and Reverse KL divergences the overall loss function.
The a-DKL approach offers flexible and robust method for knowledge distillation, allowing us to balance the trans- fer of useful information with the prevention of harmful knowledge infiltration. By adjusting the a parameter, we can fine-tune the distillation process to suit different learn- ing scenarios and optimize the trade-off between knowledge preservation and plasticity restoration. Leveraging a- a-DKL, we propose Plastic PPO (P3O), enhanced version of the PPO algorithm that integrates SBP and employs a-DKL as its distillation loss function. This integration allows P3O to maintain sustainable plasticity throughout the learning process, potentially overcoming the limitations observed in standard PPO implementations. The details of P3O are presented in Algorithm 2.
5. Experiments
5.1. Experimental Setup
[())= = min (Ttem / (Tr) + (1 aa) Dii.(To (TTor TTem)
Environment & Task To evaluate our algorithm's perfor- mance, we employed a diverse set of tasks. These include standard benchmarks from MuloCo (Todorov et al., 2012) and the state-based versions of DeepMind Control Suite (DMC) (Tassa et al., 2018). Additionally, we introduce the Cycle Friction Control task, innovative variant of the Mu- JoCo environment inspired by the slip MuJoCo task (Dohare et al., 2024). Figure 8 shows a task with cyclically chang- ing friction coefficient. It starts at 4, decreases by every million steps to 1, then increases back to 4. This discrete evolution significantly increases environmental complexity, challenging the algorithm.
(4)
Ty represents the primary policy, Trem denotes a tem- porary policy typically used before a reset, and m& signifies the policy after the reset of certain neurons.
(Tte || ) denotes the Forward KL divergence, which measures how well the policy тer approximates the policy Trem- This term is crucial for the effective transfer of essential knowledge from the Frem to the
Tgr.
Dii (mT || ) represents the Reverse KL diver- gence, which acts as a regularizer to prevent the mi
5

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
Table Performance comparison across MuJoCo environments (mean with 95% CT), with results averaged across 25 different random seeds, Percentages show improvement over baseline PPO. H-Stand: HumanoidStandup. Half: HalfCheetah.
PPO
PPO+CBP
PPO+ReDo
PPO+Cycle
P3O
Hopper
3719 [3668,3770]
3692 [3653,3732]
3730 [3672.3788]
3532 [3258,3807]
3765 [3699,3830] (1 %))
H-Stand
135k [129k,140k
137k [133k,141k]
141k [138k,145k
132k [122k,143k]
144k [136k,153k] (7%)
Walker
6270 [5974.6566]
5284 [4991,5576]
6240 [5891,6590]
4993 [4761,5225]
6788 [6380,6987] (8%)
Ant
3,354 [2839,3869]
5480 [5377,5583]
3579 [3318,3840]
2210 [1820,2600]
5522
[5421,5622] (64%) (64%)
Half
3626 4802)
5944 [5687,6201]
4936 (366,6209))
4585 [4025,5146)
6865 [6181,7548] (89 %)
Humanoid
1163 [809,1518]
1966 [1196,2736]
1413 [973,1854]
1066 [886.1247]
6237 [5901,6573] (436%)
PPO
PPO-CBP
PPO+ReDo
PPO+Cycle
P3O
Average Retur
Avezage Retum
Average Retum
3000
5000
100000
2500
1000
50000
0
2
9
101110
45545
6 7
101112
3
101112
Environment Steps (Million)
Eavironment Steps (Millien)
Enviroment Steps (Million)
(a) Hopper
(b) HumanoidStanduup
(c) Walker
Average Return
5000
Avcrogoo Retum
Average Return
5000
5000
25000
2500
1 2 3 6 8 10
0
2 4
9
10
0
5
1011
Environment Steps (Million)
Buvironment Steps (Million)
Environment Stops (Million)
(d) Ant
(c) HalfCheetah
(f) Humanoid
Figure 3. Performance of Various Algorithms in MuloCo Environments
Baseline Throughout the entire experiment, we employed PPO as the base algorithm. the reset experiment, we examined the impact of various reset strategies. CBP (Do- hare al., 2021) involves selecting neurons based on utility function that considers both weight and activation values. ReDo (Sokar et al., 2023b) selects neurons based on a score derived from their activation values. Cycle involves selecting neurons in a specific order.
suggests that we have indeed achieved sustainable plasticity, continuously providing the neural network with the capacity for learning and adaptation. Moreover, our ablation study using standalone cycle reset without distillation demon- strates that knowledge recovery plays crucial role in this process.
As illustrated in Figure 4, we analyzed the average LI norm of network neuron weights during training. Our analysis reveals that while the original PPO algorithm tends to ac- cumulate larger weights, algorithms incorporating resets maintain weights in lower range. This observation aligns with recent research (Dohare et al., 2024) suggesting that excessively large weights indicate reduced neural plasticity. Our detailed analysis in the appendix (Table 4, Figure 16, and Figure 22) demonstrates a clear correlation between reset strategies and weight magnitude: both higher reset frequency and larger reset percentage contribute to smaller overall weights. P3O achieves balanced weight distribu- tion through its moderate reset frequeney and full neuron reset strategy, positioning it between CBP's frequent partial resets and ReDo's limited reset approach. These findings confirm that strategic neuron resetting effectively restores plasticity by maintaining weights within an optimal range.
5.2. Experimental Results
MuJoCo Our experimental results demonstrate the effec- tiveness of our P3O algorithm across several key dimensions. Table shows that P3O consistently outperforms other algo- rithms across six MuJoCo environments, with particularly remarkable improvements in the Humanoid environment, where the performance boost reaches 436%. This substantial increase in maximum rewards indicates that our framework effectively enables neuron regeneration, enhancing learn- ing efficiency and more fully utilizing the neural network's 's capacity. The learning curves in Figure further support this, demonstrating that P3O achieves the highest learning efficiency in most environments and maintains an upward trend for extended periods. This sustained improvement
6

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
PPO+CBP
PPO+ReDo
PPO+Cyele
Weight Norm
Weight Norm
Weight Norm
0.100
0.10
0.05
0.050
0.05
2
10
5
10
0
5
8
10
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(h) HumanoidStandupp
(c) Walker
Weight Norm
Weight Norm
Weight Norm
0.10
0.
0.100
30.075
0.05
0.05
0.050
2
10
0
9
10
Environment Steps (Million)
Environment Steps (Million)
Puvironment Steps (Million)
(d) Ant
(e)
HalfCheetah
(f) Humanoid
Figure Actor Network Weight Norm (Lower norm tend to correlate with higher plasticity)
Sc-4
6e-4
Grdd Norm
Grdd Noraa
6c-4
Grad Stadmmammm Norm
4e-4
$0-4
2e-4
10
Steps Million)
Steps (Million)
Steps (Million)
Hopper
(b) HumanoidStanduu
(c) Walker
Rc.4
Grad Norm
Grad Nora
6e-4
Grad Norm
6c-4
6c-4
4e-4
40-4
4c-4
6
8 10
2
3
4
5
5
8
9
10
0
2
5
6
7
8
9
10
Environment Steps (Million)
Enviroement Steps (Million)
Eavironment Steps (Million)
(d) Ant
(c) HalfCheetah
(f) Humanoid
Figure Actor Network Gradient Norm (Higher norm values tend correlate with higher plasticity)
Beyond the benefits weight regulation, Figure 5 reveals a crucial finding: P3O maintains consistently higher gradi- ent norms than other tested algorithms. This significant because learning occurs through gradients, and within certain range, higher gradients indicate better learning effi- ciency and plasticity. While the phenomenon of vanishing gradients often signals a loss of plasticity, our algorithm effectively reduces weight values while maintaining robust gradients, resulting in optimal performance. Other algo- rithms show less effective gradient maintenance, suggesting the importance of knowledge retention in sustaining gradi- ents. Further analysis in Figure 17 and Figure 23 reveals that both reset percentage and frequency significantly influence gradient maintenance.
stability trade-off. By maintaining sustainable plasticity through systematic neuron regeneration, P3O achieves both smaller weight magnitudes and higher gradient norms, lead- ing to enhanced sample efficiency and learning stability in reinforcement learning tasks. This demonstrates that neural renewal, when properly balanced with knowledge preserva- tion, is key to achieving long-term adaptability.
Ad Avvataages and Costs of Inner Distillation Our distilla- tion process utilizes an online replay buffer and operates in epochs. As shown in Table 5, the statistical results demon- strate the computational overhead of distillation. Our ab- lation studies about distillation are shown in Figure 3 and Figure 24. The statistics show that while inner distilla- tion requires more training epochs, it achieves better per- formance. The results in Figure and Figure 24 reveal that naive approaches to maintaining network plasticity are insufficient: cyclic reset without recovery operations leads performance degradation, and simply increasing recov-
The superior performance P3O stems from its balanced approach: appropriate reset frequency combined with ef- fective knowledge retention. This enables efficient neu- regeneration while effectively addressing the plasticity-

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
ery training epochs fails maintain stable performance improvement. In contrast, inner distillation approach effectively facilitates neural regeneration while preserving crucial knowledge, demonstrating both necessity and effi- ciency in knowledge retention.
PPO
PPO+CBP
PPO-ReDo
P30
1000
Avarag Rotern
100-
Average Rearrn
10
10 Steps (Millio)
As shown in Figure 25, our distillation-based approach achieves consistent performance improvements with in- creased epochs, while enhancing sample efficiency and effectively recovering historical knowledge. This stands in contrast to conventional approaches, where increasing training epochs often leads to performance degradation due to primacy bias (Nikishin et al., 2022), as observed in PPO implementations (Figure 1). Our approach, despite requir- ing additional distillation epochs, demonstrates superior performance over baseline approaches like CBP and ReDo, confirming its effectiveness in sustaining plasticity and en- suring stable learning.
Eaviromment Stepa (Million)
(a) Dog Walk
(b) Quadruped Walk
Average Room
Avangg Renumn
500
10
Steps (Million)
Envirnment Stops (Milkon)
(c) Quadruped Run
(d) Hopper Hop
Figure 7. Performance of Various Algorithms in DMC Environ- ments
inherent limitations of algorithms, often restricts the ac- quisition of valuable knowledge from data. This insight highlights the pivotal role of neural network plasticity in advancing learning capabilities.
PPO
PPO-CBP
PPO-ReDo
P30
Average Retum
2000
6. Conclusion
In this work, we introduced the Sustainable Backup Propa- gation (SBP) framework, which addresses the fundamental challenge of maintaining neural network plasticity through systematic neuron regeneration. The core objective is to maximize the utilization of neural networks by providing sustainable plasticity while preserving their long-term learn- ing capacity.
101112131415
Environment Steps (Million)
Ant-CF
4000
Avcinge
Our implementation, the Plastic PPO (P30) algorithm, demonstrates the effectiveness of this approach across diverse set tasks in deep reinforcement learning. Through its key components a cycle reset mechanism that con- trols weight magnitudes and maintains gradient, combined with an inner distillation strategy that preserves valuable knowledge during neuron regeneration P30 achieves both smaller weight magnitudes and higher gradient norms. These empirical results validate our hypothesis that proper neuron regeneration can create more adaptive Al systems capable of sustained learning over extended periods, effec- tively addressing the plasticity-stability trade-off in deep reinforcement learning by enabling networks to maintain high learning capacity while preserving acquired knowledge ("stay hungry, keep learning").
101112
10 Staps (Milion)
Sleps
(b) Walker-CF
(c) Hopper-CF
Figure 6. Performance of Various Algorithms in Cycle Friction (CF) Environments
DMC & Cycle Friction Figures 7 and 6 reveal that in these complex benchmarks, while PPO struggles and other algorithms show only marginal improvements, our SBP approach achieves substantial progress. This superior per- formance in challenging environments demonstrates our algorithm's 's effectiveness and indicates that increased envi- ronmental complexity demands higher neural plasticity. Our method's success underscores the importance of effective neuron regeneration in complex tasks. These findings not only validate our approach but also highlight the need for further research into maximizing neural plasticity, especially in intricate learning environments.
While our results demonstrate significant progress, several areas require further investigation, including the optimiza- tion of reset scheduling strategies and the theoretical analy- sis of neuron regeneration mechanisms. Nevertheless, our work provides new insights into maintaining network plas- ticity and establishes a foundation for future research in sustainable learning systems.
Additionally, the experimental outcomes observed across Humanoid (Figure 3), Hopper Hop (Figure 7), and Cycle Friction Ant (Figure 6) environments demonstrate that the constrained plasticity of neural networks, rather than the
8

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
Acknowledgments
learning. arXiv preprini arXiv:2303.07500) 2023.
This work was supported by the National Science and Technology Innovation 2030 Major Project (Grant No. 20222D020880),, and NSFC General Program (Grant No. 62176215)
Ahn, H., Hyeon, J., Oh, Y., Hwang, B., and Moon, T. Re- set & distill: A recipe for overcoming negative trans- fer in continual reinforcement learning, arXiv preprint arXiv:2403.05000 2024.
Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
Impact Statement
Our research, "Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning." introduces novel approaches to address fundamental challenges in deep reinforcement learning, specifically focusing on sustainable plasticity and long-term learning capabilities.
Berariu, T., Czarnecki, W., De, S., Bornschein, J., Smith, S., Pascanu, R., and Clopath, C. A study on the plasticity of neural networks. arXiv preprint vrXiv22106.00442 2021.
The primary technical contribution, Sustainable Backup Propagation (SBP), represents a paradigm shift in how we approach neural network plasticity. By implementing a systematic neuron reset strategy combined with Inner Distil- lation, we demonstrate that networks can maintain learning capacity while mitigating performance degradation. This framework challenges conventional wisdom about conser- vative reset strategies and provides practical solution for maintaining network plasticity throughout the learning pro- cess.
Brockman, G., Cheung, V., Pettersson, I.., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym.
arXiv preprint arXiv: 2016.
Carlson, B. M. Principles of regenerative biology. Elsevier, 2011.
Delfosse, Q.. Schramowski, P., Mundt, M. Molina, A., and
Kersting, K. Adaptive rational activations to boost deep reinforcement learning, arXiv preprint 1xXi:22102.09407,, 2021.
To validate our approach. we introduced the Cycle Fric- tion Control task in the MuJoCo environment, which simu- lates real-world scenarios where environmental conditions change cyclically. This novel evaluation framework pro- vides insights into the adaptability of reinforcement learning algorithms in dynamic environments, making our findings particularly relevant for real-world applications.
Dohare, S., Mahmood, A., and Sutton, R. Continual back- prop: Stochastic gradient descent with persistent random- ness. arXiv: Leamning.aXXiv: Learning, Aug 2021.
Dohare, S., Hernandez-Garciaa J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026):768-774 2024.
The broader implications of this work extend beyond techni- cal achievements. By enabling more sustainable and adap- tive learning systems, research contributes to the devel- opment of more resilient and flexible Al systems. However, we acknowledge that enhanced plasticity in AI systems may raise important considerations regarding system stability and reliability. We encourage the research community to carefully consider these trade-offs when implementing simi- lar approaches in practical applications.
D'Oro, P., Schwarzer, M., Nikishin, E.,, Bacon, P.-I.., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
Elsayed, M., Lan, Q., Lyle, C., and Mahmood, A. R. Weight clipping for deep continual and reinforcement learning. arXiv preprint S.XXx::24770170000 2024.
We believe this work represents significant step toward more capable and adaptable reinforcement learning systems, while recognizing the importance of balancing performance improvements with system stability and reliability. As the field continues to advance, we hope our findings will inspire further research into sustainable learning mechanisms and their practical applications.
Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Pro- ceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
References
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado,
M. C. Loss of plasticity in continual deep reinforcement

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
Hecht-Nielsen,, R. Theory of the backpropagation neural network. In Neural networks for perception, pp. 65-93. Elsevier, 1992.
Ma, G., Li, L.,, Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data. Modules and Training Stages, 2023.
Igl, M., Farquhar, G., Luketina, J., Bochmer, W., and Whiteson, Transient stationattn and generali- sation in deep reinforcement learning. arXiv preprint arXiv:2006. 055266 2020.
Martins, J. P., Almeida, I., Souza, R., and Lins, S. Policy distillation for real-time inference in fronthaul congestion control. JEEE Access, 9:154471-154483 2021.
Juliani, A. and Ash, J. A study of plasticity loss in on-policy deep reinforcement learning. Advances in Neural Infor- mation Processing Systems, 37:1 13884-113910, 2024.
Montavon, G., Orr, G., and Müller, K.-R. Neural networks: tricks of the trade, volume 7700. springer, 2012.
Nauman, M., Bortkiewicz, M., Ostaszewski, M., Miloš, P., Trzciński, T., and Cygan, M. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning. arXiv preprint arXiv:2403.005 2024.
Kim, W., Shin, Y., Park, J., and Sung. Y. Sample-effciient and safe deep reinforcement learning via reset deep en- semble agents. Advances in Neural Information Process- ing Systems, 36, 2024.
Nikishin, E., Schwarzer, M., D' D'Oro, P.. Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learn- ing, pp. 16828-16847. PMLR, 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint (4Xx:::412.6980) 2014.
Kumar, S.. Marklund, H., and Van Roy, B. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308. 11958, 2023.
Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36, 2024.
Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., and Yun, C. Plastic: Improving input and la- bel plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024a.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Lee, H., Cho, H., Kim, H., Kim, D., Min, D., Choo, J., and Lyle, C. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. arXiv preprint arXiv:2406.022 025966 2024b.
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Lewandowski, A. Tanaka, H., Schuurmans, D., and Machado, M. C. Directions of curvature as an expla- nation for loss of plasticity, 2024. URL https:/arxiv. org/abs/2312.00246.
Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv: htl1:///11.0695 2015.
Lu, L.,, Shin, Y., Su, Y., and Karniadakis, G. E. Dying relu and initialization: Theory and numerical examples. arXiv preprint 19X3vv9903306733 2019.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.
arXiv preprint 1 2017.
Lyle, C., Rowland, M., Dabney, W., Kwiatkowska, M., and Gal, Y. Learning dynamics and generalization in reinforcement learning. arXiv preprint arXiv:2206.02 2022.
Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. International Conference on Machine Learning, pp. 30365-30388 PMLR, 2023.
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.014580 2023.
Sender, R. and Milo, R. The distribution of cellular turnover in the human body. Nature medicine, 27(1):45-44, 2021.
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pas- canu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402. 18762, 2024.
Shin, Y. and Kariiadakis, G. E. Trainability of relu networks and data detandeptnd initialization. Journal of Machine Learning for Modeling and Computing. 1(1), 2020.
10

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
Singh, B., Kumar, R., and Singh, V. P. Reinforcement learning in robotic applications: comprehensive survey. Artificial Intelligence Review, 55(2)945-999, 2022.
Sokar, G., Agarwal, R., Castro, P., and Evci, The dormant neuron phenomenon in deep reinforcement learning. Feb 2023a.
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon deep reinforcement learning. arXiv preprint arXiv:2302.12900 2023b.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., al. Deepmind control suite. arXiv preprint arXiv:1801.0066 2018.
Todorov, E.. Erez, T., and Tassa, Y. Mujoco: A physics en- gine for model-based control. 2012 IEEE/RS Interna- tional Conference on Intelligent Robots and Systems, Oct 2012. doi: 10.1 dx doi. org/ 10. 1ww1911001rrss 2012 6386109.
URL http://
Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., et al. Drm: Master- ing visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.
2023.
Yu, C., Liu, J., Nemati, S., and Yin, G. Reinforcement learn- ing in healthcare: survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Zhang, C., Vinyals, O., Munos, R., and Bengio, S. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.0689) 068933 2018.
Zhuang, Z., Lei, K., Liu, J., Wang, D., and Guo, Y. Be- havior proximal policy optimization. arXiv preprint 13X1, 2023.
Zhuang, Z., Peng, D., Liu, J., Zhang, Z., and Wang, D. Reinformer: Max-retum sequence modeling for offline rl. arXiv preprint arXi::2405.087400 2024.

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
A. Appendix.
A.1. HyperParameter
Python 3.8
Pytorch 2.0.1 (Paszke et al., 2019)
Gym 0.23.1 (Brockman et al., 2016)
MuloCo 2.3.7 (Todorov et al., 2012)
mujoco-py 2.1.2.14
Figure 8. Cycle Friction.
Our experiment based on PPO and incorporates SBP, CBP, and ReDo variants. We use the hyperparameters described in Table for all algorithms. It's important note that we've slightly modified the ReDo mechanism: instead of using threshold-based selection, each reset is based evaluating the scores of the bottom one percent of neurons. To ensure statistical significance and reproducibility, all experiments were conducted with 5 different random seeds, and the results presented are the mean values with corresponding standard deviations across these runs in figures. Figure illustrates the pattern of friction changes of the cycle friction environments.
Table 2. Algorithm Parameters
Category
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba, 2014)
Learning Rate (Actor & Critic)
3e-4
Online Replay Buffer Size
8192
PPO
Mini-batch Size
256
Discount Factor
0.99
Training Step
1.5e7
Epochs per Update
10
Clip Range
0.2
Clip Grad Norm
0.5
Actor & Critic Hidden Size
256
Architecture
Actor & Critic Hidden Layers
Actor & Critic Activation Function
Tanh
Reset Rate
0.01
Reset Frequency
50000 Environment Step
SBP
Neuron Utility Type
Neuron Lifetime
DKL a
0.4
Distillation Loss Bound
0.01
Reset Rate
0.01
Reset Frequency
10000 Gradient Step Contribution
CBP
Neuron Utility Type
Reset Rate
0.01
ReDo
Reset Frequency
50000 Environment Step ReDo Score
Neuron Utility Type
12

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
A.2. Concept
To systematically analyze plasticity, we introduce key terminology in Table 3. The emergence of primacy bias can be partially attributed to overfitting in neural networks. Dormant neuron identification requires careful hyperparameter selection, particularly:
For ReLU activation: neurons producing zero outputs are considered dead
For tanh/sigmoid: neurons with Koundary-proxime outputs (== +1 for tanh, or 1 for sigmoid) deemed saturated
The activation threshold for determining dormancy must be environment-specifie,, as directly impacts plasticity measure- ments.
Table 3. Key Terminology Definitions
Term
Definition
Plasticity
The ability of neural networks to learn from experiences(Beratiu et al., 2021).
Plasticity Loss
The diminished capacity of neurons to acquire new knowledge. (I.yle et al., 2023; Abbas et al., 2023)
Overfitting
The excessive fitting of model the training data(Zhang al., 2018).
Primacy Bias
The tendency to overfit to earlier training data, resulting in poor learning outcomes on later sampled data(Niiinhin et al., 2022).
Dormant
Neurons with low activation values in ReL.USokaar al., 2023a).
Dead (Saturated)
In ReLU activations, dead neurons occur when the output is zero for all inputs(Shin & Karniadakis, 2020; Lu et al., 2019). In sigmoid tanh functions, neurons are considered saturated when the output approaches extreme values(Glorot & Bengio, 2010; Montavon et al., 2012).
13

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
A.3. Algorithm
In this section, we present two novel algorithms designed achieve sustainable plasticity in neural networks: Sustainable Backup Propagation (SBP) and Plastic Proximal Policy Optimization (P3O). These algorithms address the challenge of maintaining neural network plasticity over extended periods.
A.3.1. SUSTAINABLE BACKUP PROPAGATION
Sustainable Backup Propagation (SBP) is general framework designed to maintain the plasticity of neural networks over extended periods.By By incorporating a neuron regeneration mechanism into standard backpropagation, it backs up sustainable plasticity for long-term network updates, hence enabling "sustainable" backpropagation.
The key components of SBP are:
Cycle Reset: Periodically reinitializes a portion of neurons to prevent overspecialization.
Inner Distillation: Utilizes temporary model as teacher network to recover essential knowledge after neuron reset.
This approach allows neural networks to maintain plasticity indefinitely, continually adapting to new information. The SBP algorithm implements the cycle reset and inner distillation mechanisms. In every F step, the reset rate y determines the proportion of neurons reset in each cycle, while the distillation process ensures that essential knowledge retained after each reset. The distillation process continues until the distillation loss becomes less than 7, indicating the selected neurons have successfully regenerated. The detailed algorithm is presented in Algorithm 1.
Algorithm 1 Sustainable Backup Propagation (SBP)
Neural Network fo. Temporary Model Smpp Reset Rate 7 Training Steps T, Reset Frequency F, Reset Index p Distillation Threshold т. Distillation Loss d None. for = 1 to T do
Update Neural Network fe with standard backpropagation. if't mod F then
Copy the weights of Neural Network fo to Temporary Model fump.
for each layer L of the Network do
Let I neurons of layer L.; Reinitialize input weights neuron i layer L: i [p-1: (p (p+g)-1 -1.
Freeze all the reset neurons in Neural Network fo- while d > T or None do Update Neural Network fo using Temporary Model fump as a teacher network, focusing on reducing the distillation loss according to a distillation loss.
Recalculate Distillation Loss
Unfreeze all the reset neurons in Neural Network fe. if p + < 1 then p = P
else
p == 0;
A.3.2. PLASTIC PROXIMAL POLICY OPTIMIZATION
Plastic Proximal Policy Optimization (P3O) is concrete implementation of the SBP framework within the context of reinforcement learning, specifically tailored for the Proximal Policy Optimization (PPO) algorithm. A specialized distillation function, DKL (Equation 4), is designed for PPO, ensuring effective knowledge transfer in policy space. The choice of PPO as our baseline implementation is particularly motivated by its inherent characteristics. As an on-policy algorithm maintaining only a small replay buffer, PPO faces significant plasticity demands due to the constant data distribution shifts inherent reinforcement learning exploration. These characteristics make PPO one of the most challenging yet representative cases for testing plasticity mechanisms reinforcement learning. We posit that if SBP can effectively enhance PPO's plasticity, should generalize well to other reinforcement learning algorithms.
14

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
Algorithm 2 Plastic PPO(P3O)
Policy no. Temporary Policy Tmpp Reset Rate 7, Training Steps T, Reset Frequency F, Reset Index p = Distillation Threshold т, Distillation Loss d None. for t T do
Update Policy ae with regular policy gradient.
if't mod F then
Copy the weights of Policy To to Temporary Policy Tap
for each layer L of the Network do Let [ neurons of layer L. Reinitialize input weights of neuron i in layer L: p-t: :: (p + 7) .g.
Freeze all the reset neurons in Policy Te.
while > T or None do
Update Policy Ty using Temporary Policy Tamp as teacher network based on Equation 4.
Update Distillation Loss d.
Unfreeze all the reset neurons in Policy me.
p + < then L P = +
else
p= - 0
P3O demonstrates how the general SBP framework can be applied specific machine learning paradigms, enabling sustainable plasticity in reinforcement learning policies. By incorporating neuron regeneration and knowledge distillation, these algorithms offer promising approach to overcoming the limitations of traditional neural network training methods, particularly in scenarios requiring long-term learning and adaptation to changing environments. The detailed algorithm is presented in Algorithm 2.
A.3.3. COMPARISON OF RESET STRATEGIES
In our study, we made deliberate choices reset counting methods to align with each algorithm's is characteristics while maintaining comparability. For CBP, we retained its original approach of using gradient steps for reset counting, preserving its algorithmic features. In contrast, for P3O, opted to environment interaction steps the basis for reset counting. This decision was motivated by our focus understanding how changes in input data affect neural plasticity. For consistency and to facilitate better comparison, we applied this same counting method based on environment interaction steps to ReDo as well. This approach allowed us to maintain the unique aspects of each algorithm while ensuring a meaningful comparative analysis across different reset strategies. Table 4 presents comparative analysis of neuron reset statistics for the CBP, P3O, and ReDo algorithms throughout their respective training processes. The data represents the average across six Mujoco environments. Our findings reveal distinct patterns in reset frequency and scope among these algorithms:
CBP exhibits the highest reset frequency, followed by P3O, with ReDo having the least frequent resets.
In terms of reset scope, both CBP and P30 can reset all neurons, while ReDo has limitations in this aspect.
These reset patterns align with the weight norm distributions observed in Figure 4. The data suggests an inverse relationship between reset frequency and weight magnitude: more frequent resets correspond to smaller neuronal weights. This observation unveils a simple yet significant principle: the more frequent and comprehensive the resets, the smaller the neuron weights tend be. However, excessively frequent resets can lead to performance instability. This trade-off suggests the importance of carefully calibrating both the reset frequency and recovery mechanisms to achieve optimal balance between plasticity and stability.
Table 4. Reset Statistics Comparison (768 Neurons)
Total Resets
Average Resets
Reset Proportion (%)
CBP
44208.6
57.6
100.0
ReDo
1800.0
2.3
22.3
SBP
2304.0
3.0
100.0
15

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
PPO
PPO+CBP
PPO-ReDo
PPO+Cycle
P30
Average Return
4000
Average Return
150000
Averago Retum
2000
100000
5000
50000
1
10
2
10
Enviromment Steps (Million)
Enviromment Steps (Million)
Environment Steps (Million)
Hopper
(h) HumanoidStandup
(c) Walker
Average Return
Average Return
10000
Average Return
5000
5000
5000
1
8 10
2
5
8
10
3
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(e) HalfCheetah
(f) Humanoid
Figure 9. Performance Comparison of Different Algorithms with ReLU Activation MuloCo Environments
A.4. Ablation
1. ABLATION STUDY ACTIVATION
To comprehensively evaluate our algorithm's capability in maintaining plasticity, we conducted additional experiments across different activation functions, recognizing that activation is also factor that affects the plasticity of neural networks. We maintained the same parameter settings as shown in Table 2 across all six environments. As shown in Figure 9 and Figure 3, our method demonstrates consistent performance improvements with both ReLU and Tanh activation functions, with the magnitude of improvement remaining comparable between these conditions. While bascline methods show strong performance with ReLU activation in some environments, such as HalfCheetah and Ant, they exhibit significant performance degradation when using Tanh activation. contrast, our approach maintains stable performance across both activation functions. These results demonstrate that our algorithm exhibits superior robustness compared to existing methods, it maintains consistent effectiveness regardless of the choice of activation function. This indicates that its plasticity plastiity-mmiinainnngg maintaining capabilities are not limited to specific activation functions but rather represent a more general and robust solution.
Plasticity Exploration
To better investigate the PPO algorithm, we chose the Tanh activation function based on previous research experience. Inspired by the neuron activation analysis ReDo, we adopted a straightforward approach by using the absolute value of neuron activations our scoring metric, leveraging the fact that Tanh activation values are bounded in [-1,1]. This score effectively reflects individual neuron activity levels, and network activation. However, during our investigation of neural plasticity using dormancy rate as a metric, we made an unexpected discovery: networks using Tanh activation exhibited clear correlation between activation function outputs and weight magnitudes. As shown Figure 14 and 4, our analysis confirms a clear correlation between the overall magnitude of activation values and network weights under Tanh activation. This correlation suggests that ReDo's 's dormancy calculation method may not be directly applicable to networks using Tanh activation functions.
the mean score across all neurons to quantify overall
Dormant Ratio To address this concern, we conducted additional experiments with ReLU activation functions, with results shown in Figure 9, and calculated the dormant ratio using threshold of 0.1 (Figure 10). The dormancy ratio curves closely align with performance variations lower dormancy rates correlate with higher performance. Our method consistently maintains lower dormancy rates across most environments, following trends similar to those observed in weight and gradient norms (Figure 11 and Figure 12). However, the activation norm shows a more nuanced relationship. Comparing Figure 14 and Figure 13 reveals that both extremely high and low activation norms can be problematic. Our method consistently maintains stable activation norms across environments approximately 0.7 for Tanh and 0.3 for ReL.U activations. This
16

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
PPO+CBP
PPO+ReDo
PPO+Cyele
Dormant Ratio
Dormant Ratio
Dormant Ratio
0.50
0.5
0.25
0.1
0.0
0.00
10
0
3
6
9
10
0
2
6
8
9
10
Enviromment Steps (Million)
Environment Steps (Million)
En Envirumment Steps (Million)
(a) Hopper
(b) HumanoidStandup
(c) Walker
0.2
Dormant Raiio
Dormant Ratio
0.10
0.50
0.1
0.25
0.05
0.0
0.00
0.00
3
7
10
2
3
5
7
9
10
0
2
7
Environment Steps (Million)
Environment Steps (Million)
Environment
Steps (Million)
(d) Ant
(c) HalfCheetah
(f) Humanoid
Figure 10. Dormaney Ratio of Actor Networks with ReLU Activation (Threshold = 0.1) across MuJoCo Environments
Weight Norm
0.10
Weight Norm
0.100
Weight Norm Norm
0.10
0.075
0.05
0.050
0.05
10
10
0
2
Environment Steps (Million)
Enviromment Steps (Million)
Environment Sleps (Million)
(a) Hopper
(b) HumanoidStanduu
(c) Walker
Weight Norm
Weight Norm Norm
0.100
0.10
0.10
Weight
0.075
0.050
0
2
4
6
8
10
3
5
7
9
10
2
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(e)
HalfCheetah
(I) Humanoid
Figure 1I. Actor Network Weight Norm with ReLU Activation (Lower norm tend correlate with higher plasticity)
suggests that maintaining activation values within specific, activation function-dependent ranges might be crucial for plasticity, though this hypothesis requires further investigation. Therefore, weight and gradient norms serve as effective indicators of neural network plasticity and demonstrate the effectiveness of algorithm. While dormancy ratio can also reflect plasticity levels, its applicability may be limited by the choice of activation functions. The relationship between plasticity and different activation functions warrants further investigation.
17

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
PPO
PPO+CBP
PPO+ReDo
PPO+Cycle
P30
6e-4
6e-4
Grad Norm
Grad Norm
Grad Norm
50-4
4c-4
4c-4
2c-4
30-4
8
10
3
6
10
3
6
Enviromment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStanduu
Walker
6c-4
Grad Norm
Grad GradNonm Norm
Grad Norm
6c-4
Se-4
3e-4
4c-4
3
6
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(e) HalfCheetah
Humanoid
Figure 12. Actor Network Gradient Norm with ReLU Activation (Higher tend correlate with higher plasticity)
PPO
PPO+CBP
ReO+ReDo
PPO+Cycle
P3O
Activation Nom
Activation Norm
Activation Norm
0.4
0.50
0.2
10
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStandup
Walker
Activation Nom
Activation Norm
Activation Norm
0.8
0.3
0.5
0.6
0.2
0.1
0.0
10
2
5
2
8
Environment Steps (Million)
Environment Sleps (Million)
Environment Steps (Million)
(d) Ant
(c) HalfCheetah
Humanoid
Figure 13. Actor Network Activation Norm with ReLU in MuloCo Environments
18

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
Activation Norm
Activation Norm
Activation Norm
0.75
0.75
0.8
0.50
0.25
0.25
2
3
4
6
8
10
2
5
8
10
2
4
6
7
8
10
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStanduup
(c) Walker
0.8
Activation Norn
Activation Norm
0.75
Activation Norm
0.8
0.6
0.50
0.6
0.25
2
5
8
10
2
4
6
8
10
0
2
3
5
8
10
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(e) HalfCheetah
(f) Humanoid
Figure Actor Network Activation Norm with Tanh in MuloCo Environments
19

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
20%
p= 50%
80%
Avaraee Retum
Average Retums
5000
Average Retu
Avgrage Retur
5000
5000
5000
2500
2500
4
7
8
10110
3 6
9 1011
2
5 8 10
2
8 1011 12
Enviromment Sleps (Milion))
Envirommnn Steps (Million)
Enviromment Steps (Million)
Enviromment Steps (Million)
(a) Walker
(b) Ant
(c) Half
(d) Humanoid
Figure 15. Performance of Various Network Reset Percentages
Weight Narm
Weight Norm
Weight Norm
Weight Nomm
0.100
0.10
0.10
0.10
0.075
0.05
0,500
0
2
10
2 3
9
10
3
8
Environment Steps (Million)
Environment Steps (Miliin))
Enviroement Steps (Million)
Enviroemecnt Stege (Million)
(a) Walker
(b) Ant
(c) Half
(d) Humanoid
Figure 16. Actor Network Weight Norm Across Various Reset Percentages
6o-4
Grad Narm
Grod Norm
6e-4
Grad Norm
Grod Norm
60-4
No-4
4e-4
Sc-4
4e-4
Enviroument Steps (Million)
Environment Steps (Million)
Environment Stops (Million)
Eaviromment Steps (Million)
Walker
(b)
(c) Half
(d) Humanoid
Figure 17. Actor Network Grad Norm Across Various Reset Percentages
A.4.2. ABLATION OF CYCLE RESET
We present a systematic investigation of cycle reset's impact on neural plasticity through carefully designed ablation studies. Our experimental framework examines three fundamental parameters (reset percentage, rate, and frequency) and evaluates their effects through three complementary metrics: performance outcomes, weight norm distributions, and gradient norm patterns.
Ablation of Reset Percentage
In this section, we investigate how different reset percentages affect training performance to validate the necessity of cycle reset. Our experiments restrict neuron reset to the initial x% (x 20, 50, 80) of the network. Throughout training, we use fixed per-reset rate of 0.01 with reset intervals of 50,000 steps, and cycle reset is applied only within the first x% of neurons. Figure 15, 16, and 17 show the experimental results for performance, weight variations, and gradient variations, respectively.
Our experiments reveal a clear correlation between reset percentage and network characteristics:
In most environments, higher reset percentage lead better performance
Higher reset percentage result in smaller weight magnitudes across the network
Networks with higher reset percentage maintain larger gradient norms throughout training
Our analysis of reset percentages reveals important insights into neural plasticity and neuron regeneration. Higher reset percentages consistently enhance network plasticity, as evidenced by weight magnitudes and gradient characteristics. Notably, while the Walker environment demonstrates increased plasticity with complete network reset, its performance trends differ from other environments. This divergence indicates that enhanced plasticity alone does not guarantce improved performance successful regeneration requires appropriate recovery mechanisms maintain learning stability. Our current
20

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
rate- 0.05
rate 0.1
rate 0.5
rate
Average Retum
4000
Average Retum
200000
Average Retur
5000
2000
1000000
2
2
8 10:112
3
101112
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStandup
(c) Walker
Avernge Return
5000
Average Retum
Averago Return
5000
5000
1011
10
101112 Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(e) HalfCheetah
(f) Humanoid
Figure 18. Performance Across Various Per- Reset Rates
0.08
Weight Norm
Weight Norn
Weight Norm
0.08
0.06
0.06
10
2
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
Hopper
(b) HumanoidStanduup
(c) Walker
0.125
0.08
Weight Norm
Weight Norm
0.100
0.08
Weight
0.075
0.06
0.06
5
7
9
2
4
6
Enviromment Steps (Miliin))
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(c) HalfChectah
(f) Humanoid
Figure 19. Actor Network Weight Norm Across Various Per-Reset Rates
parameter settings, while effective in other environments, require adjustment for the Walker scenario. These findings emphasize that effective neuron regeneration relies carefully balancing network resets with recovery methods to achieve both optimal plasticity utilization and knowledge preservation.
Ablation of Per-reset Rate
To investigate the impact reset rates on P3O, we conducted experiments with four different per-reset rate while maintaining reset frequency of 50,000 environment steps. The results, illustrated in Figure 18 19 and 20, demonstrate several key findings:
In most environments, reset rates 0.05 and 0.1 achieved optimal performance.
Resetting the entire network often outperformed resetting 50% of neurons.
Weight magnitudes and gradient magnitudes reveals similar patterns across different per-reset rates.
This suggests that moderate reset rates can effectively achieve complete neural regeneration. Higher reset rates do not
21

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
Grad Norm
8c4
Norm Grad
6e-4
Grad Norm
8c-4
Sc4
Grad
60-4
4c-4
4e-4
3c-4
3
5
6
7
8
9
10
0
2
3
5
6
7
9
10
0
2
6
7
8
9
10
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStanduu
(c) Walker
Grad Norm
Grad Norm
8e-4
Grad Norm
8c4
6c-4
5e-4
5e-4
4e-4
3
5
6
7
8
9
10
0
2
3
5
6
7
9
10
0
2
4
6
7
8
9
10
Environment Steps (Million)
Enviromment Steps (Million)
Environment Steps (Million)
(d) Ant
(c) HalfCheetah
(t) Humanoid
Figure 20, Actor Network Grad Norm Across Various Per-Reset Rates
necessarily lead to increased plasticity but instead create additional challenges for knowledge recovery, particularly within PPO's limited replay buffer setting. The superior performance of rate 1.0 compared rate 0.5 is particularly evident in environments like Hopper, potentially indicating the influence of primacy bias on learning. These varying responses across environments suggest the need for more robust knowledge recovery mechanisms to handle different regeneration scenarios. Given that different reset rates yield similar plasticity improvements, implementing lower reset rates appears optimal as it minimizes knowledge recovery complexity while maintaining effectiveness.
Ablation of Reset Frequency
To investigate the impact of reset frequency, conducted ablation studies while maintaining fixed reset of 0.01. Figure 21, and presents analysis across different reset intervals (20000, 40000, 80000, and 100000 steps). Our experiments reveal several key findings:
Higher reset frequencies consistently lead smaller weight magnitudes and larger gradient norms, indicating enhanced plasticity maintenance
Higher reset frequencies generally correlate with better performance.
Reset frequency demonstrates a significant impact on neural plasticity restoration, with higher frequencies leading to enhanced plasticity maintenance. However, this relationship presents a trade-off: while more frequent resets maintain higher plasticity levels, excessively high reset frequencies can overwhelm the knowledge recovery process. This limitation potentially prevents the full conversion of increased plasticity into performance improvements. The results suggest the importance of finding an optimal reset frequency that balances plasticity enhancement with effective knowledge preservation.
Through our comprehensive ablation studies cycle reset, systematically investigated three key components: reset percentage reset rate and reset frequency. The experimental results reveal several crucial insights:
Reset Percentage: Cycling through the entire network, rather than restricting resets to subset of neurons, proves more effective in maintaining overall network plasticity.
Reset Rate: Lower reset rates generally achieve optimal performance while maintaining stability.
Reset Frequency: Higher reset frequencies correlate with enhanced plasticity, as evidenced by smaller weight magnitudes and larger gradient norms.
Based on these findings, we recommend a configuration with relatively small per-reset rates, higher reset frequencies, and complete network reset coverage. These results establish the effectiveness our proposed cycle reset strategy while indicating directions for future research, particularly in developing more precise and adaptive reset schedules based on task characteristics and learning dynamics.
22

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
fecq-20000
freq 40000
freq 80000
freq=100000 100000
Average Retum
4000
Average Retum
150000
Averago Retum
5000
2000
100000
2
1011
8 101112
6 101112
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
Hopper
(b) HumanoidStandup
(c) Walker
Average Return
Average Return
5000
5000
Average
2500
2
10 12
5 7
8
9 10
3
6 101112
Environment Steps (Million)
Envirumment Steps (Million)
Environment Steps (Million)
(d) Ant
(c) HalfChectah
(f) Humanoid
Figure 21, Performance Across Various Reset Frequencies
froq=20000
freq=40000
freq=80000
froq=100000
0.10
0.100
Weight Norm
Weight Norm
0.100
0.08
Weight
0.075
0.075
0.06
0.050
0.050
1
8
10
5
7
9
10
7
9
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStandup
(c) Walker
0.10
Weight Norm
Weight Norm
0.100
0.100
Weight
0.08
0.075
0.075
0.06
0.050
0.050
6
8
10
2
4
6
10
0
2
4
7
8
9
10
Environment Steps (Million)
Environment Steps (Million)
Environment
Steps (Million)
(d) Ant
(c) HalfChectah
(f) Humanoid
Figure Actor Network Weight Norm Across Various Reset Frequencies
23

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
freq=20000
freq=40000
freq=80000
Teq=100000
Gradd Norm
6e-
Grad Norm
6e-4
Grad Grad Norm
6c-4
4e-
4e-4
4e-4
2c-4
3
10
7
2
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStandup
(c) Walker
Grad Norm
6e-4
Grad GradNorm Norm
60-4
Grad Norm
6c-4
4e-4
4e-1
4c-4
2
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(c) HalfCheetah
(f) Humanoid
Figure 23. Actor Network Grad Norm Across Various Reset Frequencies
24

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
PPO
PPO+Cycle
PPO+Cycle+RT
P30
Average Return
4000
Average Retum
Average Return
150000
5000
2000
100000
50000
2
10
12
4
5 7
8 101112
3
6 7 10 12
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStandup
(c) Walker
10000
Average Return
5000
Average Return
Avenago Return
5000
5000
2500
0
0
2
3
9 10 12
0
4 5
6
1011
3
4
5
6
7
8
9 101111
Environment Stops (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(c) HalfChectah
(f) Humanoid
Figure Performance Comparison with Recovery Training (RT)
A.4.3, ABLATION OF INNER DISTILLATION
Through extensive ablation studies on cycle reset, we find that knowledge recovery settings play a crucial role in effective neuron regeneration. A robust knowledge preservation mechanism enables flexible neuron reset at arbitrary frequencies and proportions, facilitating effective regeneration of any selected neurons. Our investigation encompasses three key aspects: comparative study between recovery training and knowledge distillation for knowledge preservation, an assessment of distillation robustness under different reset protocols, and systematic exploration of the distillation hyperparameter to optimize preservation performance.
Inner Distillation VS Recovery Training
To rigorously evaluate the recovery feature our distillation mechanism, we conducted comparative experiments between inner distillation and simple extra training after cycle reset. We implemented a recovery training protocol where the network underwent additional training epochs post-reset until the reward difference between the recovered model and the pre-reset model fell below threshold of 100, ensuring performance stability before proceeding the next training cycle.
Our experimental results, shown in Table 5 and Figure 24, demonstrate that while recovery training achieves moderate reward levels with lower computational cost compared to cycle reset alone, it fails to generate substantial performance improvements. This indicates that the ability to recover previous performance does not guarantee continued learning progress. contrast, our distillation mechanism exhibits superior capabilities in both knowledge preservation and learning advancement. Notably, illustrated in Figure 25, the additional epochs required for distillation translate directly into performance improvements, indicating meaningful training rather than computational overhead. These findings suggest that distillation not only effectively preserves and restores knowledge but also enhances sample efficiency by leveraging the network's plastic potential. The synergistic combination of reset and distillation thus emerges as a powerful approach for long-term learning improvement.
Table Extra Training Epochs across Environments under random sccds(PPO baseline: 18,310 epochs)
Hopper
Humanoid Stand
Walker
Ant
HalfCheetah
Humanoid
Distillation Epochs
597.66 809.75
638.19
1536.80
7698.40
2548.40
6525.25
Recovery Epochs
2731.60
524.00
700.20
253.20
485.25
25

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
1.0
700
Improvement of P30(%)
669
Distillation Epoch Ratio
600
0.8
Improvement P30%))
Distillation Epoch Ralio
500
0.6
400
0.42
300
0.4
200
0.2
0.14
100
0.08
62
0.03
0.03
3
Hopper
Humamoid Stand
Walker
Ant
HollChatwah
Humanoid
0.0
Figure 25, Cross-Enan Environment Performance Gains Distillation Epochs(under 5 random sceds)
Random
Random-INDS
CBP+INDS
ReDo+INDS
Average Return
4000
Average Return
150000
Averago Return
5000
2000
1000000
50000
0
2
3
8
9
1011
12
4 5
8 101112
3
4
5
6
7
9
10 101111 11
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(a) Hopper
(b) HumanoidStandup
(c) Walker
Average Return
5000
Average Return
Average Return
5000
1000
2500
500
0
2
4
8
9 10
12
0
6 8
9 10111110 12
2
6
7
8
9 10 12
Environment Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
(d) Ant
(c) HalfCheetah
(f) Humanoid
Figure 26. Performance of Inner Distillation(INISS) Integration with Different Algorithms across MuloCo Environments
Combining Inner Distillation with Baselines
Experiments combining inner distillation with various baselines were conducted to further investigate the efficacy of distillation-basee knowledge preservation. As shown in Figure 26, distillation showed limited effectiveness when applied distinct reset strategies: Random, which randomly resets 0.01 of neurons, and ReDo. However, when integrated with CBP, distillation showed promising results, particularly in the Ant environment where notable performance gains were observed, despite some learning instability. It should be noted that these experiments utilized default parameters, which may have constrained the full potential of our algorithm's integration with CBP. These findings suggest two key implications: first, distillation can effectively complement existing reset strategies as knowledge preservation mechanism in neuron regeneration paradigma gms; second, there remains significant potential for optimization through parameter tuning. Further research focusing on parameter optimization could enhance the robustness and effectiveness of combined distillation and neuronal reset approaches.
26

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
Table 6. Training Time Comparison (hours) across MuJaCo Environments
Hopper
Humanoid Stand
Walker
Ant
HalfCheetah
Humanoid
PPO
5.59
7.62
5.59
6.10
5.08
7.13
P3O
5.62
7.66
5.68
6.53
5.22
7.49
Distillation
0.03
0.04
0.09
0.43
0.14
0.36
In our experiments, we utilized machine equipped with an NVIDIA V100 (32GB) GPU to measure the update time for the PPO, which averaged approximately 0.30 seconds per update epoch. For the distillation phases, we observed an average of 0.20 seconds per epoch, as these phases only require updating the actor network without the need to update the critic. This timing remains consistent across different environments. The differences in training times across environments primarily stem from variations in sampling times. However, since the distillation phases relied on PPO's own replay buffer, they did require additional sampling. The training time of PPO consists of sampling time and update time, while P3O additionally requires distillation time. Ultimately, our results provide strong evidence that distillation does not significantly impact overall training efficiency, as demonstrated in Table 6. This suggests that the benefits gained from distillation in terms of performance do not come at a substantial cost to training time.
Ablation of Distillation a
Parameter a modulates the knowledge transfer process during distillation, which critically influences the neural regeneration outcomes. Our experiments, illustrated in Figure 27, suggest that different alpha values can affect learning efficiency. Our experiments demonstrate the effectiveness of Inner Distillation, with optimal alpha value of 0.3 consistently yielding superior performance. This finding reveals that limiting forward-por prooadd-ropagated information is beneficial and suggests the presence of primacy bias in current learning frameworks. The improved performance with a lower alpha value indicates that conventional learning paradigms may retain excessive redundant knowledge, highlighting the importance of effective neuron regeneration balancing knowledge preservation and acquisition.
Our ablation studies validate Inner Distillation through three aspects: recovery training comparisons demonstrate its efficiency in performance restoration, tests across reset strategies confirm its generalizability. and analysis of parameter reveals effective knowledge transfer control. These results establish Inner Distillation efficient and versatile approach for neural regeneration.
0.3
0.5
0.8
Avenge Roam
4000
Average Rotmm
150000
Average Retum
5000
100000
50000
6 924899999
1011
2
6
8
9 101112
8 10 Enviromment Steps (Million)
Envirooment Steps (Million)
Envirunment Steps (Million)
(a) Hopper
(b) HumanoidStandup
(c) Walker
Average Retu
Average Retum
5000
5000
5000
Averne
2500
10 Environment Steps (Million)
Enviroement Stops (Millien)
Environment Steps (Million)
(d) Ant
(e) HalfCheetah
Humanoid
Figure 27. P3O Performance Sensitivity to in MuJoCo Environments
27

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforeement Learning
A.5. SAC with SBP
As demonstration of our framework's generality in reinforcement learning, extended our evaluation to Soft Actor-Critic (HaCyHaaamoja et al., 2018), testing its performance across four Mujoco environments (Figure 28). We compared the ReDo, CBP, and Cycle reset approaches using the hyperparameter configurations detailed in Table 7. Additionally, incorporated periodic resets of the last hidden layer and periodic resets of the entire network, which were studied in the context of off-policy methods.
Our results demonstrate that SBP consistently improves SAC's performance. The analysis of dormant ratio (Figure 29), gradient norm (Figure 32), weight norm (Figure 31), and activation norm (Figure 30) reveals trends similar to those observed in PPO: lower dormancy rates, larger gradients, smaller weights, and activation values maintained within stable ranges. Notably, despite slightly higher dormancy rate in HalfCheetah, we observed larger activation norms, suggesting that similar performance was achieved with fewer active neurons, indicating more efficient utilization of neural plasticity. However, we believe these improvements represent only fraction of SBP's potential benefit to SAC, particularly considering that our current implementation, which randomly samples just 1% (8,192 1919 samples) from the replay buffer for inner distillation, already achieves significant plasticity enhancement and notable performance gains. The current approach, while effective, leaves substantial room for exploring more sophisticated sampling strategies to better utilize the rich information available in off-policy settings. To summarize, we believe that achieving better performance on SAC requires attention the following two points:
Distillation Buffer: Constructing appropriate distillation buffer is crucial. While the PPO online buffer can be directly used for distillation, SAC requires a suitable method to create a distillation buffer from its large offline buffer. An unsuitable buffer may exacerbate primacy bias.
Our experimental results indicate that various reset strategies for the actor yield similar performance, suggesting that addressing the critic may be necessary to mitigate the loss of plasticity SAC, an issue also discussed in previous work (Nikishin et al., 2022).
Impact the Critic:
SAC faces challenges with larger replay buffer, while PPO operates with a smaller one, leading to different exploration directions. Additionally, the critic seems to play a more significant role in the plasticity of SAC.
Table 7. Hyperparameter Configuration of SAC with SBP
SAC Parameters
SBP Parameters
Activation Function
ReLU
Reset Frequency
10,000 steps
Hidden Size
256
Reset Percentage
0.01
Training Step
2e6
Alpha Value
0.8
Replay Buffer Size
IM
Distillation Buffer Size
8,192
28

---

Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning
SAC
SAC+RcDo SAC+Cycle
SAC+SBI
SAC Complete Reset
SAC+CBP
SAC Last Layer Reset
Average Retum
Average Ratims
Average Renm
Avgrage Retum
5000
10000
5000
2500
0.0
0.5
10
1.5
2,0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
Envvirommnnt Steps (Million)
Envaronment
Steps (Million)
Enviromment
Steps (Million)
Environment Steps (Million)
(a) Half
(b) Hopper
(c) Walker
(d) Ant
Figure 28. SAC Performance Across MuJoCo Environments
Dennant Ratio
Domman Ratio
0.3
Domnan Ratio
0.3
0.3
0.2
Dommant
0.2
0.2
0.1
0.1
0.0
0.5
1.0
2.0
0.0
0.3
1.0
2.0
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.01
1.5
2.0
Environment Steps (Million)
Environment Steps (Milion)
Environment Steps (Million)
Environment Steps (Millin)
(a) Half
(b) Hopper
(c) Walker
(d) Ant
Figure 29. Dormant Ratio of SAC Actor Network in MuJoCo Environments (Threshold 0.1)
Activation Norm
1.0
Activattion Norm
Activation Norm
Antivation Norm
0.4
0.2
0.2
0.2
0.0
01.5
1.0
2.0
11.5
1.0
2,0
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.0
1.5
2.0
Envirommntt
(Million)
Envirenment Steps (Million)
Environment Steps (Million)
Enviromment
(Million)
(a) Half
(b) Hopper
(c) Walker
(d) Ant
Figure Activation Norm SAC Actor Network in MuloCo
0.3
Weightt Norm Norm
Weight Norm
Weightt Norm Norm
WeightNomm Norm
0.2
0.2
0.2
0.2
0.1
0.1
0.3
0.5
1.0
2.0
0.0
0.5
1.0
2.0
0.0
1.0
1.5
2.0
0.5
1.0
1.5
200
Steps (Million)
Enviinement Sleps (Million)
Steps (Million)
Steps (Million)
(a) Half
(b) Hopper
(c) Walker
(d) Ant
Figure 31, Weight Norm SAC Actor Network in MuloCo
2e-2
Gradd Noaro
Se-2
Grnd Norn
Crad Nert
Ie-2
5e-3
Grad
1e-2
0.0
1.0
0.0
0.5
1.5
2.0
1.0
L.3
2.0
0.0
0.5
1.0
Ensiroement Steps (Million)
Environment Steps (Million)
Environment Steps (Million)
Havironment Steps (Million)
(a) Half
(b) Hopper
(c) Walker
(d) Ant
Figure 32, Gradient Norm of SAC Actor Network in MuJoCo
29


Reviews:
[{"review_id": "m3xGap7i5B", "reviewer": "Reviewer_wzCH", "summary": "This paper addresses the loss of plasticity in deep reinforcement learning (DRL) models, which is a phenomenon where neural networks become less adaptable over time as they learn different task distributions. The authors explain that phenomena like primacy bias (overweighting early experiences) and dead neurons (units that cease to activate) are mainly responsible. The authors proposed a neuron regeneration mechanism inspired by cellular regeneration. The algorithm is called Sustainable Backup Propagation (SBP) that involves a periodic resetting of subsets of neurons and a distillation-based mechanism to  transfer knowledge from pre-reset neurons and preserve learning. The additional mechanisms are added on top of PPO, which is renamed as Plastic PPO (P3O). To manage a trade-off between preserving knowledge and adapting to new data, an additional mechanism named α-weighted Double KL Divergence (α-DKL) is introduced.\n\nExperimental evidence  include  MuJoCo, DeepMind Control Suite, and a new Cycle Friction task). Baselines are two methods, CBP and ReDo, specifically designed to reinitialize neurons and mitigate the loss of learning ability of the network.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The paper claims that degrading learning is common in RL settings. This is reported in the literature and verified in the experiments showed in Fig 1.\n\nThe paper claims that the proposed approach ameliorates the issue of degraded learning. The experimental evidence seems to support this claim. \n\nThe claim that this algorithm implements sustainable plasticity is weak. The experimental settings do not involve sufficient distribution shifts to establish the algorithm's robustness.", "methods_and_evaluation_criteria": "In general, the methods and evaluation criteria are sound. However, the evaluation is on single-task only. Despite discussing lifelong/plasticity challenges, no sequential task benchmarks are used (e.g., Meta-World, continual Atari). No experiments on task transfer, catastrophic forgetting, or generalization across task changes. Thus, I believe the method is evaluated in a narrow context (single environment + cyclical changes) that limits the strength of the claims on plasticity in broader continual or real-world settings.", "theoretical_claims": "The paper is largely empirical with limited or no theoretical claims.", "experimental_designs_or_analyses": "Overall, the paper presents a reasonable set of choices in the design of the experiments and the analysis. However, the evaluation over-emphasizes reset-based methods. I appreciate that this is a new emerging area to address degradation of learning, but I find it limiting that no other more general lifelong learning strategies are considered.", "relation_to_broader_scientific_literature": "There is a solid link with the recent literature on plasticity loss. The ideas in this paper clearly stem from recent advances in this area.\n\nThe overall objective to maintain plasticity while reducing forgetting, however, is related to the broader field of lifelong learning which is not particularly expanded upon.", "essential_references_not_discussed": "The literature and approaches to lifelong learning are not discussed. While I understand this is a choice rather than an omission, I believe the paper would be stronger if the relationship between the proposed approach and established methods in lifelong learning was discussed.", "other_strengths_and_weaknesses": "Strengths: \n- The paper addresses a known problem in neural network training, learning degradation when cycling through different distributions. This is an important limitation in continual learning systems, particularly applied to RL. \n- The proposed method offers an interesting integration between reset approaches and continual learning with the inner distillation mechanism. \n- The results seem favourable when compared with existing reset methods\n\nWeaknesses: \n- One main weakness in my opinion is the setup is far from a challenging continual learning scenario with multiple tasks and distribution changes. The paper is addressing a limited and constrained problem in which one parameter (friction coefficient) is responsible for the nonstationarity of the environment that can take 4 different states. Given the limited source of nonstationarity, I suspect that a  context-based meta learner, e.g. CAVIA or PERL, would perform well. I appreciate that those require task boundaries, but SBP goes around it with a periodic reset frequency, which is one additional hyperparameter. \nIn short: \n- The approach is evaluated only in single-task, mostly stationary settings, with nonstationarity limited to a single engineered environment (Cycle Friction).\n- Despite broader claims on plasticity and sustainable learning, there is no multi-task, continual learning, or transfer learning setup.", "other_comments_or_suggestions": "- Abstract, typo: However, These approaches \n\n- Inconsistent spacing before a reference in the text, please check as sometimes comes with a space, sometimes without a space.\n\n- The sentence \"Additionally, the experimental outcomes observed across\nHumanoid (Figure 3), Hopper Hop (Figure 6), and Cycle\nFriction Ant (Figure 7) environments demonstrate that\" \ndoes not seem to match the figures: and-CF is in figure 6 not 7.", "questions_for_authors": "- Five seeds are the bare minimum and I read in the appendix that the shades in the graphs are STD. could you perform a confidence internal analysis and show the confidence interval instead of the STD? \n\n- The shades in Fig 6C seem too narrow to be originating from 5 different seeds, see 6A and 6B in comparison. Can you double check this detail?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "04N0XkwMFJ", "reviewer": "Reviewer_XcpR", "summary": "This paper proposes a new Sustainable Backup Propagation(SBP) framework to maintain plasticity in Deep RL. SBP combines knowledge distillation with cyclical resetting of neurons. Results show that when SBP is combined with PPO, it results in much better performance and stability than PPO.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The claims made in this paper are not supported by clear evidence. This is an empirical paper. However, the empirical analysis is not statistically rigorous, and the results are not statistically significant. The authors study PPO and conduct five runs for all their experiments. This is not sufficient; PPO is known to be extremely noisy (Henderson et al., 2019). Experiments with PPO should have at least 30 runs. It is also not mentioned anywhere what is the shaded region in the plots. Is it the standard error, bootstrapped confidence interval or something else? The paper should report the 95% bootstrapped confidence interval for RL experiments. I suggest the authors read the paper by Patterson et al. (2024) on conducting proper empirical analysis in RL.\n\nI am willing to increase the score if the authors conduct 30 runs and show that the conclusions stay the same.\n\nHenderson et al., Deep Reinforcement Learning that Matters, 2019.\n\nPatterson et al., Empirical Design in Reinforcement Learning, 2024.", "methods_and_evaluation_criteria": "The environments used in this study are appropriate. \nIt is unclear if other baselines (ReDo and CBP) were tuned appropriately. The paper does not contain any information on how the specific hyper-parameters for the baselines were chosen.", "theoretical_claims": "The paper does not contain theoretical claims", "experimental_designs_or_analyses": "I checked the experimental designs and analysis. See above for the detailed issues with the analysis.", "relation_to_broader_scientific_literature": "Selective reinitialization is a common strategy to deal with loss of plasticity. This paper proposes a new method, SBP, to maintain plasticity using selective reinitialization. The results claim that SBP can outperform existing selective reinitialization methods like CBP and ReDo.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The writing in some places is too strong and incorrect. For example, line 24 states that \"these approaches ... fail to effectively reset the entire network, resulting in underutilization ...\". This is written as a matter of fact. However, no evidence is provided for this claim. Similarly, line 45 states, \" ... approaches, such as CBP (Dohare et al., 2021) ... focused on selectively resetting non-contributing neurons. While this strategy reduced information loss, it only partially restored plasticity ...\" This is incorrect. Dohare et al. (2021) showed that CBP maintains plasticity in all cases tested. It is possible that CBP fails to maintain plasticity in some cases, but that needs to be shown before it can be claimed that CBP only partially restores plasticity. The caption of Figure 4 states,\" ... Lower norm indicates higher plasticity\". That is incorrect; lower norms have been found to correlate with higher plasticity, but lower norms do not necessarily mean higher plasticity. The caption of Figure 5 has the same issue.", "other_comments_or_suggestions": "Please report the 95% bootstrapped confidence interval in all tables. Currently, Table 1 does not contain any confidence interval.", "questions_for_authors": "What is the wall clock time for SBP compared to ReDo and CBP? It seems like SBP is signficaintly more computationally expensive than ReDo and CBP due to distillation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iNDnxly9mY", "reviewer": "Reviewer_UGKa", "summary": "The paper presents a new way of increasing plasticity in neural networks used in reinforcement learning. The main idea is to reset some of the neurons, while using a distillation strategy that maintains the \"knowledge\" of the reset neurons by the rest of the network. The method is considered in the context of the Proximal Policy Opitimization (PPO) and evaluated on a few RL environments.\n\n## update after rebuttal\nI am updating my initial score", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' feedback and insights. Below are our detailed responses to the questions and comments raised.\n\n## 1. Reply to Questions 1\nFigure 1 demonstrates that primacy bias also exists in PPO. More training epochs lead to higher data fitting, but fitting the early data can hinder growth in later stages. This phenomenon was previously observed in SAC, and we believe it similarly affects PPO, which is the purpose of this figure.\n\n## 2. Reply to Questions 2\nDetails about the off-policy (SAC + SBP) experiments are included in Appendix A.4. The results show performance improvements, and the plasticity metrics also yielded favorable outcomes, indicating the generalizability of our algorithm.\n\n## 3. Reply to Impact of Runtime and Computational Cost\nSince SBP operates as an independent plug-in outside the standard PPO process, it adds only one extra training step and utilizes PPO's existing replay buffer, incurring no additional sampling time. Consequently, the computational cost of P3O over PPO is primarily due to the overhead of inner distillation. The distillation epochs shown in Table 4 of Appendix A3.3 represent this additional computational cost. Given that the replay buffer and batch size remain consistent, we believe these epochs can be effectively converted into GPU hours, which is why we use epochs as the basis for our computational cost analysis.\n\nWe will include a discussion and analysis of this aspect in the revised manuscript to provide clearer insights into the computational implications of our method.\n\n## 4. Reply to Horizon of PPO \nFor Ant, as shown in Figure 6C, at 15M steps, P30 is still increasing. In preliminary exploratory experiments, we ran up to 50M steps and found that P30 could reach 4000. However, due to the time required for these experiments, we did not conduct more to include them in the paper.\n\n## 5. Reply to Typo\nWe have corrected the formatting and typographical errors and thoroughly reviewed the paper to ensure clarity and precision.", "claims_and_evidence": "1. It is not clear to me how widely applicable the methods are, what is their impact on off-policy methods?\n2. Is there any impact of the method on the runtime of the baseline algorithm?\n\n## update after rebuttal\nBoth points were addressed by the authors. I still think that the off-policy aspect requires more analysis, but I am happy about the presented direction in the supplementary material and authors responses.", "methods_and_evaluation_criteria": "In general yes, but I have some doubts:\n* what is the computational cost of the presented method compared to vanilla PPO? can it be ignored in the evaluation?\n* would the same conclusions be made if the horizon of PPO training was longer?", "theoretical_claims": "The presented results are mostly experimental. I did not find any issues related to formalizing the ideas of the paper.", "experimental_designs_or_analyses": "I find the experimental designs valid.", "relation_to_broader_scientific_literature": "The paper studies methods of improving plasticity in reinforcement learning. Losing plasticity is one of the well known problems of reinforcement learning, I find the study well motivated.", "essential_references_not_discussed": "No suggestions.", "other_strengths_and_weaknesses": "Strenghts:\n* clear idea, well presented,\n* research is well motivated paper, plasticity is a very important aspect in RL\n\nWeaknesses:\n* why only PPO is considered, while other methods such as SAC (which is frequently used together with resets) is not evaluated?\n* is the presented method slowing the baseline algorithm (PPO)? this is not evaluated in the paper", "other_comments_or_suggestions": "* Definitions in section 2.1 seem not to be correct, e.g.:\n    * the domain of the reward function is X x S, what is X?\n    * P : S x A -> P, shoudn't it go to P(S)?\n* The choice of using P vs \\mathcal(P) in Definition 3.1 is suboptimal in my view - better to use something more descriptive (e.g., use subscript hinting at the meaning).\n* section 4.2, you use \\pi_temp and \\pi_tem\n* formatting in Table 1 is inconsitent (sometimes commas are used to separate thousands, sometimes not)", "questions_for_authors": "* What exactly is presented in Fig 1? I don't fully understand those plots.\n* Did you try to use your technique together with SAC or other off-policy algorithms?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BO1tSlxgR9", "reviewer": "Reviewer_2nT8", "summary": "The paper tackles the problem of plasticity in on-policy reinforcement learning. Combining techniques of weight reseting and distillation, the authors propose a technique that the authors call \"Sustainable Backup Propagation\" (SBP). In SBP, some percentage of neurons are reinitialized every $n$ step. To mitigate the negative effects of neuron reinitialization, SBP maintains a copy of the pre-reset policy network which is distilled into the post-reset policy network using a weighted KL objective. The authors performed experiments on a few tasks from the OpenAI gym and DeepMind control suite.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' feedback and insightful comments. Below are our detailed responses to the questions and concerns raised.\n\n## 1. Reply to Questions 1\nWe believe the proposed method enhances learning efficiency in massively parallel setups. SBP acts as a flexible plug-in that integrates an independent module into the standard training process. Upon reaching the reset cycle, it transitions to inner distillation and then returns to normal PPO training, without affecting PPO's deployment.\n\n## 2. Reply to Questions 2\nDetails about the off-policy (SAC + SBP) experiments are included in Appendix A.4. The results show performance improvements, and the plasticity metrics also yielded favorable outcomes, indicating the generalizability of our algorithm.\n\n## 3. Reply to Weaknesses 1\nMassively parallel setups may indeed face challenges related to plasticity. However, our work focuses on effectively maintaining plasticity to enable the network to learn stably over extended periods. We aim to address issues such as primacy bias, dormant neurons, and dead neurons while ensuring the stability of the neural network. This will enhance the algorithm's performance and unlock its potential. There is still much to explore in this area, and we believe that investigations into massively parallel setups may be better suited for future work.\n\n## 4. Reply to Weaknesses 2\nPrevious work related to reset mechanisms (Redo and CBP) has demonstrated that resetting can effectively alleviate the issue of plasticity loss. Our approach builds upon this reset foundation by introducing a recovery mechanism that ensures network stability. This not only addresses plasticity loss but also enhances the efficient utilization of plasticity, leading to improved sample efficiency and performance gains.\n\nPrimacy bias refers to the fitting of early data, which hinders the ability to learn new knowledge later on. This concept aligns closely with plasticity, and prior work has categorized primacy bias as a form of plasticity loss. From both a conceptual standpoint and a consensus within the research community, primacy bias is recognized as a type of plasticity loss.\n\n## 5. Reply to Weaknesses 3\nThis can be addressed by referring to the response to review Zet6, point 5, regarding Figure 1.\n\n## 6. Reply to Benchmarks\nThis can be addressed by referring to the response to review wzCH, point 3.\n\n## 7. Reply to Baselines\nSince we primarily focus on reset-based methods, we emphasize parameters related to resets, such as reset rate and reset frequency, as well as effective recovery strategies. Discussions on regularization methods are included in the related work section, as we consider them distinct types of approaches. While adding these baselines could further clarify our method's position in the broader field, we believe our current selection of baselines is appropriate for the questions we aim to explore.\n\n## 8. Reply to Ablations\nRegarding the importance of the α parameter during distillation and the associated costs, our analysis of the ablation experiments is detailed in Appendix A3.3. We found that the maximum computational cost could exceed standard PPO by approximately 40%.\n\nAs for the idea of resetting every gradient step, our conclusion is that more frequent resets result in smaller weights and maintain higher gradient levels, but do not guarantee stable performance improvements. This poses significant challenges for inner distillation, as each step must have accurately optimal parameters to ensure network stability and consistent performance improvement. This remains an area requiring extensive exploration.\n\n## 9. Reply to Plasticity\nWe believe that plasticity loss begins as soon as training commences, with plasticity being gradually consumed throughout the training process. Primacy bias emerges early on, while issues like dormant and dead neurons increase over time, all of which impact the ability to learn new information. This underscores the concept of plasticity. Additionally, Figure 1 illustrates the presence of primacy bias, and we contend that similar loss of plasticity occurs in PPO.\n\n## 10. Reply to Some Claims\nRecycling only considers restoring neurons to a plastic state, without addressing the stability of the neural network. We believe that regeneration can encompass both aspects.\n\n\"SBP ensures sustainable plasticity throughout the network’s lifecycle,\" and we will revise this to \"SBP can sustainably provide plasticity and largely maintain network stability throughout the network’s lifecycle.\"\n\nRegarding the α-weighted Double KL divergence (α-DKL), our ablation experiments in Appendix A3.3 demonstrate that this weighting has a significant impact on the effectiveness of distillation. Therefore, we propose this loss function to be valuable and meaningful.", "claims_and_evidence": "> We introduce the concept of neuron regeneration, a biomimetic approach inspired by cellular regeneration processes...\n\nA similar concept was introduced in previous work (\"recycling dormant neurons\") [1]. \n\n[1] Sokar, Ghada, et al. \"The dormant neuron phenomenon in deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.\n\n> We propose SBP, a systematic framework that implements neuron regeneration through cyclic reset strategies and inner distillation mechanisms...\n\nThis method seems novel and interesting. Combining resets and model distillation appears like a natural thing to do. \n\n> By effectively addressing dead neurons and primacy bias, SBP ensures sustainable plasticity throughout the network’s lifecycle.\n\nI think this claim should be slightly toned down. Figure 10 shows that the proposed method does not reduce the amount of dormant neurons as compared to the baselines. In general, I find the performance improvements convincing, but I feel that the authors provide a very limited analysis of why this might be (weight and gradient norm were shown to mildly correlate with plasticity in prior works).\n\n> PPO that integrates SBP and a novel α-weighted Double KL divergence (α-DKL) loss function.\n\nAgain, calling weighted Jeffreys divergence novel is a bit of a stretch. Furthermore, whereas the authors provide some rationale for why this design choice might be important, there are no experiments ablating the importance of $\\alpha$.\n\n> Primacy bias/plasticity loss\n\nI do not think this is established that primacy bias is the same as plasticity loss. I think the authors should provide more evidence that the primacy bias is at play (which was shown for SAC-based algorithms) or just stick to plasticity which is a more established umbrella term for learning problems.", "methods_and_evaluation_criteria": "In my opinion, the evaluation of the paper could be slightly improved. Whereas I appreciate the experiments on OpenAI gym and DMC, I think the authors should consider running experiments on environments where policy-based approaches like PPO perform relatively well (e.g. Isaac gym / Procgen). \n\nFurthermore, I find it slightly surprising that the authors do not evaluate any off-policy methods - after all, approaches like ReDo or full-parameter resets were originally proposed for DQN and SAC-based algorithms. I think performing off-policy experiments for the proposed method on DMC/Gym would greatly enhance the presentation of the paper, as well as show the generalizability of the proposed method.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "1. Unconvincing choice of benchmarks - authors evaluate their method on OpenAI gym and DMC, benchmarks where off-policy methods are known to perform much better than PPO. Why not consider GPU-based simulators where PPO is dominant?\n\n2. Missing baselines - recent work has shown that PPO+L2init and PPO+layer normalization are strong baselines when it comes to maintaining plasticity in on-policy algorithms \n\n3. Limited ablations - whereas authors show that their proposed approach performs well, it is hard to attribute this performance improvement to design choices. For example, how important is $\\alpha$ during distillation? Figure 26 suggests that the more resets the better, what would happen if we reset every gradient step? How costly is the distillation?\n\n4. Are we sure this is plasticity at play? - the authors report results of 10mln environment steps training. Given the reported hyperparameters, this results in 800k policy updates during training. In contrast, in previous works, a single reset is performed every 2.5 min gradient steps. If authors claim that plasticity issues can occur in less than 800k gradient steps there should be some experiments to show it.", "relation_to_broader_scientific_literature": "The paper is highly related to previous works on the problem of plasticity (i.e. the decreasing capacity to adapt to new data as the learning progresses). The authors motivate their method by problems with existing solutions to plasticity such as the information churn stemming from full-parameter resets and propose a new method that is supposed to partly address these issues. What I find a bit confusing is that whereas most of the work in plasticity was done for off-policy algorithms, the authors focus on on-policy setup. While there is nothing wrong with that, it makes me wonder how general the insights proposed in this paper are.", "essential_references_not_discussed": "The authors should certainly cite [3].\n\n[3] Juliani, Arthur, and Jordan Ash. \"A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning.\" Advances in Neural Information Processing Systems 37 (2024): 113884-113910.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Plasticity studies for on-policy are important and potentially impactful\n2. The proposed method seems to perform better than the evaluated baselines\n\nWeaknesses:\n\n1. Limited experimental section (only on-policy on a relatively small amount of environment steps). Does the method transfer to off-policy? How does the method pair with massively parallel simulations where plasticity might be more of an issue? These questions are unexplored.\n2. Slightly confusing narrative of the paper: is inspiration by biological processes truly that important in this context? Is the paper tackling primacy bias plasticity or both or are they the same?\n3. (nitpick) Figure 1 should be more related to the paper: the poor performance when increasing the number of epochs is not necessarily related to plasticity (e.g. might be that Q-values stop reflecting the policy). If it is indeed plasticity on the figure, does the proposed method allow for using more epochs?", "other_comments_or_suggestions": "sometimes there is no space before the citation", "questions_for_authors": "1. would the proposed method allow for more efficient learning in massively parallel setup?\n2. would the method work for off-policy approaches like SAC?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZGo5NP4aCv", "reviewer": "Reviewer_4PTe", "summary": "This paper introduces Sustainable Backup Propagation (SBP), a framework designed to maintain neural network plasticity while preserving learned knowledge. SBP employs neuron regeneration through cycle reset and inner distillation and is integrated into Proximal Policy Optimization (PPO), leading to the development of Plastic PPO (P3O). Experiments in MuJoCo and DeepMind Control Suite show that P3O improves policy plasticity and sample efficiency compared to baseline methods. \n\nKey contributions include:\n- Neuron regeneration for sustained plasticity.\n- SBP framework combining cycle reset and inner distillation.\n- P3O as an enhanced PPO with SBP integration.\n- Empirical validation demonstrating improved learning efficiency.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' insightful comments and suggestions. Below are our detailed responses to the questions raised.\n\n## 1. Reply to Questions 1\nDetails about the off-policy (SAC + SBP) experiments are included in Appendix A.4. The results demonstrate performance improvements, and the plasticity metrics indicate favorable outcomes, highlighting the generalizability of our algorithm.\n\n## 2. Reply to Computation Cost\nSince SBP operates as an independent plug-in outside the standard PPO process, it adds only one extra training step and utilizes PPO's existing replay buffer, incurring no additional sampling time. Consequently, the computational cost of P3O over PPO is primarily due to the overhead of inner distillation. The distillation epochs shown in Table 4 of Appendix A3.3 represent this additional computational cost. Given that the replay buffer and batch size remain consistent, we believe these epochs can be effectively converted into GPU hours, which is why we use epochs as the basis for our computational cost analysis.\n\nWe will include a discussion and analysis of this aspect in the revised manuscript to provide clearer insights into the computational implications of our method.\n\n## 3. Reply to References\nWe will incorporate HnT into the related work section for further discussion.", "claims_and_evidence": "1. Neuron regeneration enhances plasticity: Supported by experiments showing controlled weight norms and higher gradient norms.\n2. P3O Outperforms PPO and Other Baselines: Results in MuJoCo and DeepMind Control Suite demonstrate clear gains.\n3. α-DKL Aids Knowledge Retention: Ablation studies confirm its role in balancing learning stability and flexibility​\n4. Computational Overhead is Considered but Lacks Clarity: The paper compares distillation vs. recovery epochs (Table 4) and shows distillation improves performance despite extra training (Figure 27). However, direct runtime comparisons with PPO are missing​\n\nThe claims are well-supported, but clearer computational cost analysis would improve transparency.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally appropriate for the problem at hand:\n\n- Evaluation Benchmarks: The paper evaluates P3O in multiple RL environments (MuJoCo, DeepMind Control Suite, Cycle Friction) covering a range of tasks, ensuring a robust assessment​\n- Baseline Comparisons: The comparison with PPO, CBP, and ReDo is well-structured, but additional baselines such as HnT[1] or recent plasticity-focused algorithms could further strengthen the analysis​\n\n[1] Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks., ICML 2024", "theoretical_claims": "No theoretical claims were made.", "experimental_designs_or_analyses": "The experimental design is generally sound and aligns with standard reinforcement learning research methodologies:\n\n- Multiple Environments: The inclusion of both standard and custom environments (Cycle Friction) strengthens the validity of the claims\n- Ablation Studies: The paper conducts extensive ablations on reset frequency, reset percentage, α-DKL tuning, and alternative recovery methods, providing a comprehensive understanding of SBP’s impact\n\nHowever, the paper does not thoroughly discuss the computational cost of SBP compared to PPO and its impact on training time.", "relation_to_broader_scientific_literature": "The paper builds upon prior work in reinforcement learning plasticity, reset mechanisms, and policy distillation.", "essential_references_not_discussed": "The paper does not cite \"Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks\", a recent work on maintaining plasticity in reinforcement learning. Including this reference would help position SBP within the latest research on plasticity preservation.\n\n[1] Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks., ICML 2024", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Recent works on plasticity preservation often evaluate methods in off-policy settings. Have you tested SBP in an off-policy reinforcement learning framework, such as SAC or DQN? If not, do you anticipate any challenges in adapting it to off-policy algorithms? A comparison in off-policy settings would help assess the generalizability of SBP beyond on-policy methods like PPO.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81g6GwCfWO", "reviewer": "Reviewer_amiC", "summary": "The authors introduce the concept of neuron regeneration and propose a framework named Sustainable Backup Propagation(SBP) that maintains plasticity in neural networks through this neuron regeneration process. The SBP framework achieves network neuron regeneration through two key procedures: cycle reset and inner distillation.   The authors integrate SBP with Proximal Policy Optimization (PPO) and propose a distillation function for inner distillation. Experiments demonstrate the approach maintains policy plasticity and improves sample efficiency in reinforcement learning tasks.\n\n## update after rebuttal\nI want to update the score after reading the authors' responses to other reviews. There are weaknesses in the experiments and the literature has not been fully discussed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The claims are clear and sound.", "methods_and_evaluation_criteria": "The methods and evaluation are solid.", "theoretical_claims": "There are no theoretical results. The paper could be improved with theoretical analysis.", "experimental_designs_or_analyses": "Experiments are solid.", "relation_to_broader_scientific_literature": "The method could potentially improve reinforcement learning.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper was well-written and had a good structure, as well as extensive experiments.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tf2tvWOkgN", "reviewer": "Reviewer_Zet6", "summary": "This paper proposes Plastic PPO (P3O) to address the plasticity loss in online RL. The key idea of P3O is the combination of cyclic neuron reset and inner distillation for policy network, which better balances the plasticity recovery and knowledge retention. The proposed methods are evaluated in MuJoCo and four DMC tasks, along with ablation study, hyperparameter analysis and other plasticity analysis in the appendix.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The scores in Table 1 do not have error bars. The results based on five seeds are not very convincing for PPO, as in Figure 3 the shaded areas overlaps a lot. Statistical significance is not mentioned.", "methods_and_evaluation_criteria": "The ideas of the proposed method make sense to me. The evaluation lacks sufficient random seeds and convincing statistics.\n\nThe motivating analysis in Figure 1 does not fully make sense to me. Since the failure of larger epoch numbers could also stem from increasing off-policyness of PPO training, the authors do not rule out this possibility in this paper.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experiments for P3O include diverse aspects like performance comparison, hyperparam analysis, and ablation.\n\nAn issue is that the authors motivate this work with previous works on primacy bias, which are mainly studied for SAC with high UTD. However, in this paper, the proposed methods and the study of plasticity are for PPO, making it disconnected from previous literature on experimental study. For example, ReDO is originally proposed and evaluated for DQN and Atari. Reset is originally proposed for SAC. I noticed the experiments in Appendix A.4, but the proposed method does not show clear effectiveness upon SAC.\n\nThe motivating analysis in Figure 1 shows that PPO struggles leveraging more epochs due to plasticity loss, however, the direct empirical evidence for how P3O addresses this issue seems missing.\n\nThe direct comparison in terms of wall-clock training time or GPU hour time between baseline methods and the proposed method is missing, which is significant to practical use.", "relation_to_broader_scientific_literature": "The proposed method is related to the topics beyond plasticity loss, including continual RL, RL under non-stationarity.", "essential_references_not_discussed": "An important related work that also proposed a reset-and-distill method is not included in this paper:\n\n- Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learning. arXiv 2403.05066\n\nThere are some other related papers on plasticity loss not included:\n\n- Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps. arXiv 2412.17113\n- Weight Clipping for Deep Continual and Reinforcement Learning. arXiv 2407.01704\n- Directions of Curvature as an Explanation for Loss of Plasticity. arXiv 2312.00246", "other_strengths_and_weaknesses": "- The name of the proposed method, Sustainable Backup Propagation (SBP), from my perspective, has nothing to do with how the method works, i.e., Cycle Reset and Inner Distillation.  I suggest the authors pick another name that reflects the proposed method directly. I felt a bit confused with the current method name SBP after reading Section 4.1.\n- The paper writing needs substantial polish. I found many redundant and repeated expressions in Section 4. The content in Section 3 and Section 4 can be re-organized to be more connected and balanced.", "other_comments_or_suggestions": "- Equation 2 and 3 are a bit problematic in formal expression. I get the point here that $\\pi_1, \\pi_2$ are teacher and student respectively, but as they are placed in different orders, the formulas in Equation 2 and 3 are just normal KL.\n- Line 23, “However, These”.\n- In the first paragraph, spaces are missing between text and citations.\n- Line 107, missing space for “P(” and a right “)” is missing too.\n- Line 110, “learning, The goal”.\n- Line 206, dual use of notation $P$, which denotes the transition probability above.", "questions_for_authors": "1. The distillation loss threshold $\\tau$ is mentioned in Algorithm 1 and 2, but I did not find the detailed discussion nor the hyperparameter analysis experiment about it. Did I miss them?\n2. How were the reset rate and the reset frequency selected? How will different choices influence performance?\n3. In Line 185, the authors mentioned “we introduce the Cycle Reset mechanism, governed by two key parameters: Reset frequency F and Reset rate p”, but I found three factors in the appendix A.3.2, Reset frequency F, and Reset percentage, per-reset rate. After a quick check, I did find “per-reset rate” in the main text and Algorithm 1 and 2.\n4. Although Figure 28 and Table 4 show the computation overhead of the proposed method, the direct comparison in terms of wall-clock training time or GPU hour time between baseline methods and the proposed method is missing.\n5. As Figure 1 shows that PPO struggles leveraging more epochs due to plasticity loss, do P3O address this issue with direct experimental evidence?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is clearly some disagreement among reviewers. Reviewer Zet6 gave a detailed explanation for his negative decision and engaged with authors. However, I believe the details of Figure 1, which is mostly there to introduce the general problem, may be of less relevance than actual reported results in Figures 3 and following. Also, the lesser impact on SAC does not seem like a fatal flaw to me. 

Therefore, overall I am leaning towards acceptance, under the condition that the relevant paper suggested by reviewer Zet6 [1] must be cited. I believe the method's novelty and results, such as they are, more than balance the (valid) observations of reviewer Zet6.

[1] "Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learning"
Decision: Accept (poster)