Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning

Authors: Huaicheng Zhou, Zifeng Zhuang, Donglin Wang
Primary area: reinforcement learning
Keywords: Deep reinforcement learning, Plasticity

Abstract:
The integration of Deep Neural Networks (DNNs) in Reinforcement Learning (RL) systems has led to remarkable progress in solving complex tasks but also introduced challenges like primacy bias and dead neurons. Primacy bias skews learning towards early experiences, while dead neurons diminish the network's capacity to acquire new knowledge. Traditional reset mechanisms aimed at addressing these issues often involve maintaining large replay buffers to train new networks or selectively resetting subsets of neurons. However, These approaches either incur substantial computational costs or fail to effectively reset the entire network, resulting in underutilization of network plasticity and reduced learning efficiency. In this work, we introduce the novel concept of neuron regeneration, which combines reset mechanisms with knowledge recovery techniques. We also propose a new framework called Sustainable Backup Propagation (SBP) that effectively maintains plasticity in neural networks through this neuron regeneration process. The SBP framework achieves whole network neuron regeneration through two key procedures: cycle reset and inner distillation. Cycle reset involves a scheduled renewal of neurons, while inner distillation functions as a knowledge recovery mechanism at the neuron level. To validate our framework, we integrate SBP with Proximal Policy Optimization (PPO) and propose a novel distillation function for inner distillation. This integration results in Plastic PPO (P3O), a new algorithm that enables efficient cyclic regeneration of all neurons in the actor network. This approach facilitates neuron regeneration while maintaining policy plasticity and sample efficiency. Extensive experiments demonstrate that, with proper neuron regeneration methods, the SBP framework can effectively maintain plasticity and improve sample efficiency in reinforcement learning tasks.

Decision: reject
Reviewer scores: 3, 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730306556869

thread_type:
official_review

<thread_object>
creator: reviewer_cxrk

<content>
summary:
This paper proposes a straightforward and reasonable method, to improve the performance of reset, which is one of the most reliable methods to mitigate plasticity loss, in Gym-mujoco and DMC.

strengths:
This paper is well-written and has clear insight.

The method is introduced in a reasonable and theatrical way.

The experimental scenarios are diverse and convincing.

weaknesses:
1. Regarding the experiments with DMC, I couldn't find how you built the module for encoding image input. Could you describe it in detail?

2. Using the original PPO is rarly in DMC testing. Because it is difficult to learn effective policies, it cannot provide algorithm effectiveness support. Examples are Dog Walk and Hopper Hop in the paper. The learning curve shows that none of the methods learned an effective strategy (the highest score is below 0.1). This only adds to my doubts about P3O's performance.

3. The full paper does not say how many seeds were used for each experiment. In addition, the large difference in the variance shading of each curve in Figure 5 calls into question the fairness of its experimental design (0 variance for PPO+CBP).

4. This paper lacks the comparison of the methods on plasticity evaluation metrics, such as covariance metric [1].

5.  Most of the results on Mujoco in Fig 14 show that the proposed method performs significantly worse than other methods on mainstream plasticity evaluation metrics. The reasons given are confusing and reinforce my suspicions about the mismatch between P3O and motivation. I would advise authors to replace the middle layer activation function with ReLU and re-record the metric and learning curve (generally it doesn't influence much). It's very limiting if your approach works on tanh activation functions.

[1]  Ceron, Johan Samir Obando, Aaron Courville, and Pablo Samuel Castro. "In value-based deep reinforcement learning, a pruned network is a good network." Forty-first International Conference on Machine Learning.

questions:
Please see the above suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730364001310

thread_type:
official_review

<thread_object>
creator: reviewer_ru1a

<content>
summary:
This paper introduces the concept of "neuron regeneration" - a mechanism designed to maintain neural network plasticity while preserving stability in deep reinforcement learning systems. The core contribution is the Sustainable Back Propagation (SBP) framework, which implements neuron regeneration through two complementary mechanisms: cycle reset and inner distillation. The cycle reset component periodically reinitializes selected neurons, while inner distillation ensures the preservation of essential knowledge during the reset process.
The authors integrate SBP with Proximal Policy Optimization (PPO) to create Plastic PPO (P3O), demonstrating the practical application of their framework. The experimental results show that P3O achieves superior performance in both standard single-task environments and non-stationary scenarios, suggesting effective maintenance of plasticity without compromising stability. The empirical evaluation spans multiple environments, including MuJoCo benchmarks and custom non-stationary tasks, providing evidence for the framework's effectiveness in addressing the plasticity-stability trade-off in deep reinforcement learning.

strengths:
1. The key innovation lies in the inner distillation mechanism, which effectively mitigates the potential negative impacts of reset. This approach represents a promising solution to preserve critical knowledge during the reset process.

2. The paper presents a novel neuron-level reset approach that periodically refreshes different neurons on a cyclical basis, rather than specifically identifying neurons for reset (as in ReDo). While this represents an interesting direction that warrants further investigation (see weaknesses and questions), it introduces a fresh perspective on maintaining network plasticity.

3. The experimental validation is comprehensive, encompassing both standard control tasks and non-stationary scenarios. This breadth of evaluation, particularly the inclusion of environments with changing dynamics, strengthens the empirical support for the proposed method.

4. The use of weight norm and gradient norm as analytical metrics provides valuable insights into network characteristics. These measurements offer a quantitative lens for understanding how the proposed method affects network behavior and learning dynamics.

weaknesses:
1. The necessity of introducing "neuron regeneration" as a new concept is questionable, as it heavily overlaps with the established plasticity-stability trade-off. The proposed approach essentially describes enhanced plasticity without stability degradation, which could be framed within existing theoretical frameworks.

2. Several terminological choices appear disconnected from their underlying methods. The title "Stay Hungry, Keep Learning" shows little relation to cycle reset and inner distillation. The framework name "Sustainable Back Propagation (SBP)" is potentially misleading as it doesn't directly involve modifications to backpropagation. Moreover, the inconsistent reference to SBP as "Sustainable Brain Plasticity" in the abstract raises concerns about terminology precision.

3. While neuron-level cyclic reset is a key component, the paper lacks sufficient justification for its superiority over targeted reset approaches. Although ReDo comparison is included, a more comprehensive analysis is needed, particularly with aligned reset frequency and magnitude between methods. This is crucial given the central role of the cyclic reset strategy in the paper's contribution.

4. Despite the paper's ambitious claim of achieving "Sustainable Plasticity", the choice of metrics (weight norm and gradient norm) as indirect indicators is less convincing than currently preferred measures in the community, such as fraction of active units or dormant ratio. These more direct plasticity metrics would provide stronger support for the paper's claims.

5. A significant limitation is the framework's validation being restricted to PPO. To demonstrate the effectiveness and generality of SBP, testing on DQN-based methods and other off-policy actor-critic algorithms is essential. This broader validation would substantially strengthen the paper's contribution to the field.

questions:
I believe each weakness identified above requires careful consideration and substantive response from the authors. A clear path to acceptance would involve addressing these concerns with concrete solutions and additional empirical evidence where necessary. I am open to further discussion during the rebuttal period and would be willing to revise my assessment positively if the authors can effectively address these issues.

For acceptance, I would particularly emphasize the need for:
- Better justification of the neuron regeneration concept
- More comprehensive comparison with targeted reset approaches
- Additional plasticity metrics
- Broader algorithmic validation

I look forward to the authors' response and am prepared to engage in constructive dialogue during the rebuttal period. A thorough addressing of these concerns would warrant a more favorable evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730404004466

thread_type:
official_review

<thread_object>
creator: reviewer_ekta

<content>
summary:
A neural network’s ability to learn diminishes over time due to primacy bias and dead neurons. In order to maintain network plasticity, the authors propose a method of neuron regeneration that combines reset mechanisms and knowledge recovery. The approach, Sustainable Back Propagation, resets random neurons and trains the remaining neurons with knowledge/policy distillation from the pre-reset network. This approach is implemented with PPO as P3O and is tested against multiple RL environments.

strengths:
This paper provides good framing and a clear picture of the issues in neuron plasticity. It provides good theoretical backing with neuron regeneration.

Descriptions are clear and concise, visuals are also clear and aid in understanding.

The approach improves on baseline PPO performance in all tested environments, beats the other methods in most experiments, and greatly surpasses the other methods in several experiments.

weaknesses:
While you do an ablation of alpha values (for forward vs backward KL), you don’t test alpha=0.5, which represents a common way to turn KL divergence into a symmetrical distance function. You should test alpha=0.5, because if it is optimal, you wouldn’t need to explain the alpha-DKL as much, as symmetrical KL distance is well-established.

While the improved plasticity is clear with the sustained upward trend of the approach’s reward curves, it is unclear how to quantify plasticity. This may be out-of-scope for this paper, but a measure of plasticity could greatly strengthen the claims made. These could include analyzing the slope of the reward curve, the magnitude of the gradient, or perhaps there is an information theoretic angle to plasticity. You analyzed the magnitude of weights, but it is unclear if lower magnitude weights captures meaningful plasticity.

This is a minor gripe, but the explanation of PPO clipping in 2.1. is incorrect. The clip function does not simply limit the magnitude of policy updates. Instead, the clipping essentially sets the gradient to 0 and is used to prevent the model from learning a policy update that would take it further outside rt(theta) +/- epsilon. Using a similar explanation would be more accurate to the mechanics of PPO.

Minor formatting error: the PPO reference paper Schulman et al. 2017 appears twice in the Reference section.

questions:
How can plasticity be calculated? Are lower magnitude weights always more plastic than higher magnitude weights?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611543933

thread_type:
official_review

<thread_object>
creator: reviewer_tisv

<content>
summary:
This paper targets plasticity loss in reinforcement learning. Specifically, the widely-used periodic resetting technique was used, which is re-named as "Cycle Reset" in this paper. The authors then adopted a knowledge distillation method to distill knowledge from policy before resetting to alleviate performance degradation from resetting. The authors applied the proposed method to PPO and conducted experiments in the MuJoCo and DMC benchmarks. Overall, the technical contribution of this paper is minor and the experimental setting is not convincing, which will be detailed below.

strengths:
1. The authors introduce knowledge distillation into the resetting process of RL to alleviate performance drop after each resetting step, which shows good results in their experiments.

weaknesses:
1. The technical contribution is minor. 

1.1. First of all, I see no essential difference between the introduced "Cycle Reset mechanism" and the existing widely-used resetting mechanism [1]. It is inappropriate for the authors to claim this as their own contribution. 

1.2. Secondly, the authors used an established knowledge distillation (KD) method in their distillation module, leaving the contribution of their method to be introducing KD into the resetting procedure. 

2. The major research challenge this paper targets is the performance degradation after resetting. However, it has been well shown in previous works [1][2] that the reset model can quickly recover the lost ability after a few training steps, which makes the "performance degradation" not a significant problem from my point of view. On the other hand, the introduced KD module in this paper also involves additional cost in the overall RL procedure, which is not compared in the paper.

3. The experiment setting which only uses PPO is not convincing. 

3.1. First of all, in [2], it has been revealed that the critic suffers much more than the actor w.r.t. plasticity loss, indicating that value-based or hybrid methods like DQN and SAC are more suited for plasticity loss study than policy-gradient methods like PPO. In Figure 4, PPO+Cycle also brings no benefit compared with PPO, showing that the proposed Cycle Reset does not help plasticity loss in this setting.

3.2. PPO is known for the difficulty in training, requiring potentially careful hyperparameter tuning. The results provided by the authors seem sub-optimal which may be due to this reason. Additionally, the results reported are highly likely to be unfair for previous methods due to a different experiment setting. In comparison, in [1], the performance of SAC / SAC+ReDo for HalfCheetah (MuJoco) can achieve a return over 10000, which is much higher than the reported on in Figure 4. And in [2], their proposed adaptive replay ratio can achieve a return of ~800 in Quadruped Run (DMC) which is also much higher than the reported result in Figure 9. Therefore, I highly recommend the authors to implement their methods on RL backbones used by previous works such as DQN, SAC, or DrQ. 

[1] Sokar, Ghada, et al. "The dormant neuron phenomenon in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[2] Ma, Guozheng, et al. "Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages." The Twelfth International Conference on Learning Representations. 2024.

questions:
1. Could the authors provide comparison on the extra cost introduced by (1) continually training the model after reset to recover the performance and (2) using knowledge distillation?

2. Could the authors provide experiment results on other RL methods other than PPO, such as DQN or SAC for a fair comparison with previous works on plasticity loss in RL?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733309703443

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Review Response

comment:
## Motivation
We appreciate the reviewers' thorough examination of our research foundations. Our study addresses fundamental plasticity challenges in neural networks, particularly focusing on maximizing neuron utilization. Through extensive studies, we demonstrated that complete neuron reset, while seemingly aggressive, is necessary for maximizing network potential. This insight led us to develop a balanced approach using distillation as a recovery mechanism.

## Experiment
Our choice of PPO as the primary testing ground was deliberate and strategic. PPO presents unique challenges due to its small replay buffer and sensitivity to data distribution shifts, making it an ideal candidate for validating our approach. We reasoned that if our methods could succeed in this challenging environment, they would likely generalize well to other scenarios. This hypothesis was later confirmed through our expanded experiments.

Following reviewer suggestions, we substantially expanded our experimental framework:
- Extended our studies (Appendix A4.1) by implementing ReLU-based PPO alongside initial tanh-based versions
- Broadened algorithm coverage by incorporating SAC implementations (Appendix A5.1)
- Performed comprehensive ablation studies examining cycle reset proportions (Appendix A4.4) and inner distillation effects (Appendix A4.5)

Our expanded experimental results strongly validate our approach through three main findings:
- Cycle reset experiments demonstrate that resetting all neurons effectively maintains small weight norms while preserving large gradient flow (Figures 19-20)
- Inner distillation experiments show superior knowledge preservation compared to recovery training, suggesting improved sample efficiency with increased distillation (Figure 21)
- Implementation tests across P3O and SAC+SBP confirm our method's flexibility, demonstrating effectiveness in both on-policy and off-policy scenarios (Figure 23)

These experimental results validate both the necessity and effectiveness of each algorithm component while demonstrating the comprehensive capabilities of our approach.

## Enhanced Plasticity Metrics
Our expanded evaluation metrics demonstrate the robustness of our approach:
- Weight norm analysis (Figures 5, 16, 26)
- Gradient norm measurements (Figures 6, 13, 27)
- Dormant ratio tracking (Figures 12, 24)
- Covariance analysis (Figures 28-30)

The results show consistently desirable characteristics across different settings: small weight norms, low dormant ratio, and large gradient values. The covariance heatmap analysis further validates P3O's superior capability in maintaining both enhanced plasticity and effective neuron utilization.

## Concept
We have carefully refined our algorithm naming to Sustainable Backup Propagation(SBP) and refined the neuron regeneration concept. Neuron regeneration provides a more intuitive and specific description compared to the abstract plasticity-stability trade-off term. The biological analogy of regeneration serves as an intuitive framework for understanding complex network intervention. This precise conceptualization enhances clarity in the field while facilitating meaningful discussions about neuron plasticity maintenance.

## Conclusion
Our comprehensive experiments demonstrate that SBP achieves effective plasticity maintenance through principled neuron regeneration, providing both valuable insights into neural plasticity and practical solutions. This approach shows consistent improvements across different algorithms, validating its broad applicability. These findings not only address the reviewers' concerns but also open up new possibilities for advancing neural network training methodologies.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Reviewers shared various positive comments about the submission, complimenting the positive framing of the approach, the good presentation of the work, and its comprehensive experiments, as well as commenting that this is an interesting direction that warrants further investigation. Through the rebuttal, the authors also improved some clarity issues while also introducing new plasticity metrics that reviewers argued for. 

While I would have liked to see stronger participation of the reviewers in the discussion (which may have led to a score increase) and could personally envision this submission (with modifications) being published at ICLR, I'm afraid taking all comments together, the average rating (including some discounting of non-responsive reviewers) does not lean strongly enough towards acceptance for this to be justifiable at this stage. Nevertheless, I would like to strongly encourage the authors to re-submit a modified version and feel optimistic about its chances of acceptance in the future.