Title: NBSP: A Neuron-Level Framework for Balancing Stability and Plasticity in Deep Reinforcement Learning
Venue: NeurIPS 2025
Accepted: no
Keywords: deep reinforcement learning

stability-plasticity dilemma

skill neuron
Abstract: In contrast to the human ability to continuously acquire knowledge, agents struggle with the stability-plasticity dilemma in deep reinforcement learning (DRL), which refers to the trade-off between retaining existing skills (stability) and learning new knowledge (plasticity).
Current methods focus on balancing these two aspects at the network level, lacking sufficient differentiation and fine-grained control of individual neurons.
To overcome this limitation, we propose Neuron-level Balance between Stability and Plasticity (NBSP) method, by taking inspiration from the observation that specific neurons are strongly relevant to task-relevant skills. 
Specifically, NBSP first (1) defines and identifies RL skill neurons that are crucial for knowledge retention through a goal-oriented method, and then (2) introduces a framework by employing adaptive gradient masking and experience replay techniques targeting these neurons to preserve the encoded existing skills while enabling adaptation to new tasks.
Numerous experimental results on the Meta-World and Atari benchmarks demonstrate that NBSP significantly outperforms existing approaches in balancing stability and plasticity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a fundamental problem in deep reinforcement learning, namely the balance between stability and plasticity. The authors propose a methodology to identify critical neurons, whose updates (at least partially) are controlled through a masking mechanism.  Once these RL skills neurons are identified, a replay buffer further enhances the stability of the overall method. The method is extensively evaluated and benchmarked against several recent baselines. The results are very competitive, convincingly showcasing the effectiveness of the proposed methodology.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts\n- Excellent presentation and quality of the overall paper\n- Well-thought-out methodology\n- Strong evaluation, including multiple baselines and complete ablations\n- Strong numerical results\n\nWeaknesses\n- Some hyperparameters are chosen empirically\n- A study on task order is missing (see Questions)\n- Some parts of the Appendix (related work, limitations) should be moved to the main text\n\nQUESTIONS:\n- In Sec. 3.3, the fact that $\\alpha$ is set to 0.2 appears to be a rather empirical choice. How was this value selected, and what's the impact on the performance? I would have expected to see a sensitivity study on this parameter (and I would appreciate it if the authors could show this with some additional experiments). Also, to avoid any confusion, the ablation on the proportion of RL skill neurons (where I guess the sweet spot value 0.2 refers to the top m% indicated at p. 5, l. 187) should make it clear that it is *not* about the $\\alpha$ parameter used in Eq. 7, but rather on the top-m value. The fact that the same value of 0.2 is used for both parameters may be confusing.\n- From the description of the method, it's not clear what happens if the same neuron is identified as an RL skill neuron for multiple tasks. Is this a possibility? If so, how is this handled in the method (particularly wrt Eq.s 4, 6, and 7)?\n- When describing the experimental setup in Sec. 4, the authors explain that repeating each task twice within a cycling sequence avoids the influence of task order. As far as I can see, the task order is, however, fixed in the experiment. Could the authors elaborate further as to why repeating tasks twice avoids the influence of task order?\n- Connected to the previous question, it would be interesting to see a set of experiments where the task order is indeed *randomly shuffled*, to see if there is indeed any task order effect. The fixed sequence detailed in App. C.2 suggests that the order may play a role (e.g., drawer-open and drawer-close, which essentially require the same actions but in reverse order, come one after the other; same for window-open and window-close). Would it be possible to see such experiments to rule out any task order effect?\n- While results clearly show that the proposed method outperforms the baselines, the fact that only 3 seeds are used (which I understand may be due to the high computational costs of DRL experiments) may call for more cautious statements (e.g., in Sec. 4.1 \"significantly outperforms \", \"substantially higher than other baselines\", etc.). In the absence of statistical evidence, these statements may be read as overly bold and perhaps should be rephrased.\n- Several important aspects that normally appear in the main text (incl. the related work) are moved to the Appendix. While I understand the rationale behind that, perhaps a better balance between the different sections (e.g., considering the ablations) should be sought to have a self-contained main text.\n- Other works have proposed neuron-level plastic NNs models, see e.g. the node encoding in https://doi.org/10.1016/S0893-6080(00)00032-0 and the more recent NcHL approach from https://arxiv.org/abs/2403.12076. Could the authors elaborate in the related work section how does the present submission relate to these works?\n- To allow reproducibilty, the codebase of this work should be made publicly available.\n\nMinor\n- L. 292, missing space between \"Table 2.\" and \"From\"\n\nLIMITATIONS:\nThe authors discuss the limitations in Appendix E. I would suggest moving this part to the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces NBSP, a neuron-level continual deep-RL framework tackling both stability and plasticity. It consists of three components: 1) activation-based selection of \"skill neurons\", 2) gradient masking based on neuron scores, and 3) a memory replay mechanism. NBSP reduces forgetting by adaptively masking the gradients of neurons that contribute to high performance, combined with periodically replaying experiences from previous tasks. By performing regular updates on neurons that do not contribute to task performance, NBSP maintains plasticity on new tasks. The authors combine their method with SAC and evaluate on cyclical versions of MetaWorld and Atari. They provide ablation studies examining the importance of different components in NBSP.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I find the paper to be well-motivated and the proposed method to be sensible. However, I do have severe questions about evaluation, particularly the selected benchmark methods. Without including more methods that are a) specifically targeting the stability-plasticity trade-off and b) shown to work well in continual RL, I cannot recommend acceptance. If the authors were to include these baselines and show favorable performance of NBSP, I would raise my score.\n\n## Strengths\n\n- The work is well-motivated. It makes sense to look at neuron-level activations in the context of continual RL.\n- The ablation studies are well-designed and provide meaningful insights.\n- The authors evaluate on continuous as well as discrete control tasks.\n- Code is provided, ensuring reproducibility of results.\n\n## Weaknesses\n\nMAJOR:\n\n- The baselines are weak, making the evaluation misleading. Except for ANCL and CoTASP, the authors only benchmark against methods that target either plasticity or stability, but not both. ANCL and CoTASP do not use experience replay, which gives NBSP a big advantage in the off-policy setting. Several baselines that address stability and plasticity are missing: Shrink & Perturb [1], UPGD [2], and NE [3].\n- All three of the methods that I mentioned are not even mentioned in the related work, despite being either very well established (Shrink & Perturb), clearly solving the same stability-plasticity trade-off (UPGD), or being methodologically very similar (NE).\n- The algorithm introduces a lot of hyperparamters which seem to be hard to tune, particularly for larger agents: $m$ for the top-m activation selection, $\\alpha$ for mask generation, the size of the experience buffer $|D_{pre}|$, how often to replay experience from previous tasks $k$. \n- The proposed framework is applied only to a single RL algorithm (SAC).\n\nMINOR:\n\n- I find the writing in the paper to be not very clear, and some details are omitted. For example, how transitions for $D_{pre}$ are selected.\n- Some of the claims seem a bit exaggerated given the limited evaluation, e.g., \"NBSP exhibits excellent generalization in balance stability and plasticity across different benchmarks\".\n- Limitations are hidden in the Appendix.\n- Not all of the NBSP hyperparameters seem to be specified in Table 6 of the appendix.\n- No hyperparameter sensitivity analysis.\n\n[1] Ash, Jordan, and Ryan P. Adams. \"On warm-starting neural network training.\" (2020).\n\n[2] Elsayed, Mohamed, and A. Rupam Mahmood. \"Addressing loss of plasticity and catastrophic forgetting in continual learning.\" (2024).\n\n[3] Liu, Jiashun, et al. \"Neuroplastic Expansion in Deep Reinforcement Learning.\" (2024).\n\nQUESTIONS:\n- How are the samples for experience replay selected? I couldn't find this specified in the paper.\n- Are skill neurons concentrated more in the earlier or in the later layers of the network?\n- Would the algorithm also work for on-policy algorithms such as PPO?\n- How much slower compared to regular SAC is NBSP, given that it has to compute and maintain scores with each forward pass?\n- How would the author set the hyperparameter $m$ related to network size? E.g., select 20% of parameters as $m$?\n- How sensitive is NBSP for its main hyperparameters $m$, $\\alpha$, and $k$?\n\nLIMITATIONS:\nLimitations are included but only in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a strategy for balancing stability and plasticity in reinforcement learning. To support continous adaptation to new tasks, the authors propose to identify neurons that have an activation pattern which is correlated to task success. Applying gradient masking to these neurons supports preserving already learned skills. Meanwhile, the other neurons can freely adapt to new tasks. To further reinforce stability, experience replay, i.e. periodically revisiting past experience prevents excessive drift from previous knowledge.\n\nSTRENGTHS AND WEAKNESSES:\nMotivated by the observation that the activations of some neurons are strongly correlated with task performance, the paper introduces the notion of RL skill neurons. To retain the performance of skill neurons, their updates are limited with a gradient masking technique.\n\n\nS1: The paper tackles an interesting challenge.\nS2: The paper is well-written in general.\nS3: Experimental results show improvements over baselines.\nS4: The code is publicly available.\n \nW1: Experiments are conducted on 2 benchmarks only. Moreover, these benchmarks are rather simple (Atari, Meta-World)\nW2: The discussion of the most related works is very short.\nW3: The method is somewhat ad hoc and the innovativeness is limited.\nW4: The superparameter alpha is just set to 0.2 without explaination (Line 204) \n\nAd W1: There are a lot of more complex benchmarks for RL, e.g., MuJoCo and DeepMind. It is not clear if the method works there.\nAd W2: There are some RL approaches, that also focus on the neuron level. These are only extremely briefly discussed in Lines 109 - 113. Experiments lack comparison to some of those most relevant methods.\nAd W3: Gradient masking and the identification of task-specific neurons are not novel ideas but already known from other works. The same holds for periodic replays during training. The specific combination for reinforcement learning is new but also quite straightforward and not well motivated. There are no theoretical guarantees when this method works and only limited experiments.\nAd W4: There is no explaination, theoretical foundation or guideline on how to set this parameter. It is only experimentally validated.\n\nQUESTIONS:\nHow does your method perform on more complex benchmarks?\nAre there any theoretical explainations why it works? \nCould you explain why you exactly combined those building blocks that you used?\n\nLIMITATIONS:\nA short discussion of limitations in the main paper would be good. Currently only in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how individual neuron in the artificial neural networks can improve the stability and plasticity trade-offs in the continual reinforcement learning setting. The criteria on which neurons are relevant for specific task is based on the comprehensive score defined by the authors, which take into account of the activations of the neuron. In order to preserve the activity of the neuron across the different task to limit interference and forgetting, the model uses an approach known as adaptive gradient masking, which mask the gradients of the skill neurons in order to restrict changes in their activity.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength \n1. Overall, the paper is well-written and easy to read. The authors focused on a less studied area of the \n2. Figure 1 was helpful to see how the magnitude of the activation correlates to the successful or failure outcome of the task.\n3. The results in the Metaworld and Atari benchmark looks promising. It showed that the proposed algorithm was able to outperform several competitive baseline models such as Continual backprop and plasticity injection.\n4. The proposed model does not require resetting parameters in the neural network. This approach is similar to EWC, which focuses on the synaptic connection (weights) between the neurons, rather than the activity of the neuron itself.\n5. Ablation studies have been performed the importance of the use of both gradient masking and experience replay \n\n# Weakness\n1. Requires the sweeping the hyper-parameter to determine the proportion of the skill neurons\n2. Seems to require the use of an experience replay buffer that contains information from the previous tasks\n\nQUESTIONS:\n1. According to Figure 1, it seems that higher activation contributes more to success of the task. However, can the inverse also be true, such that high activations lead to unsuccessful outcome of the task? How robust or consistent is this effect?\n\n2. How critical is it to use the experience replay buffer with information encountered from previous tasks?\n\n3. Line 163: What do the authors mean by average step? Is this corresponding to the number of steps per task?\n\n4. Does the experience replay store information on previously encountered tasks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}