Neuroplastic Expansion in Deep Reinforcement Learning
Authors: Jiashun Liu
Johan Samir Obando Ceron
Aaron Courville
Ling Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Loss of Plasticity
Primacy Bias
Deep Reinforcement Learning
Continual RL
Primary area: reinforcement learning
TL;DR: We introduce Neuroplastic Expansion, a novel method that mitigates plasticity loss in deep RL, which outperforms strong baselines in maintaining adaptability and enhancing performance across various standard and continual RL tasks.

Abstract: The loss of plasticity in learning agents, analogous to the solidification of neural pathways in biological brains, significantly impedes learning and adaptation in reinforcement learning due to its non-stationary nature. To address this fundamental challenge, we propose a novel approach, *Neuroplastic Expansion* (NE), inspired by cortical expansion in cognitive science. NE maintains learnability and adaptability throughout the entire training process by dynamically growing the network from a smaller initial size to its full dimension. Our method is designed with three key components: (1) elastic neuron generation based on potential gradients, (2) dormant neuron pruning to optimize network expressivity, and (3) neuron consolidation via experience review to strike a balance in the plasticity-stability dilemma. Extensive experiments demonstrate that NE effectively mitigates plasticity loss and outperforms state-of-the-art methods across various tasks in MuJoCo and DeepMind Control Suite environments. NE enables more adaptive learning in complex, dynamic environments, which represents a crucial step towards transitioning deep reinforcement learning from static, one-time training paradigms to more flexible, continually adapting models.

Published as a conference paper at ICLR 2025
NEUROPLASTIC EXPANSION IN DEEP REINFORCE- MENT LEARNING
Jiashun Liu HKUST
Johan Obando-Ceron
Aaron Courville
Ling Pan" HKUST
Mila Québec Al Institute Université de Montréal
Mila Québec AI Institute Université de Montréal
ABSTRACT
The loss of plasticity in learning agents, analogous to the solidification of neu- ral pathways in biological brains, significantly impedes learning and adaptation in reinforcement learning due to its non-stationary nature. To address this fun- damental challenge, propose novel approach, Neuroplastic Expansion (NE), inspired by cortical expansion in cognitive science. NE maintains learnability and adaptability throughout the entire training process by dynamically growing the network from a smaller initial size to its full dimension. Our method is designed with three key components: (/) elastic topology generation based on potential gradients, (2) dormant neuron pruning to optimize network expressivity, and (3) neuron consolidation via experience review strike balance the plasticity- stability dilemma. Extensive experiments demonstrate that NE effectively miti- gates plasticity loss and outperforms state-of-the-ard methods across various tasks in MuloCo and DeepMind Control Suite environments. NE enables more adap- tive learning in complex, dynamic environments, which represents a crucial step towards transitioning deep reinforcement learning from static, one-time training paradigms to more flexible, continually adapting models. We make our code publicly available.
INTRODUCTION
In the realm neuroscience, it has been observed that biological agents often experience a di- minishing ability to adapt over time, analogous the gradual solidification of neural pathways in the brain (Livingston, 1966). This phenomenon, typically known as the loss of plasticity (Mateos- Aparicio & Rodríguez-Moreno 2019), significantly affects an agent's capacity to learn continually, especially when agents learn by trial and error in deep reinforcement learning (deep RL) due to the non-stationarity nature. The declining adaptability throughout the learning process can severely hin- der the agent's 's ability to effectively learn and respond to complex or non-stationary scenarios (Ab- bas al., 2023). This limitation presents a fundamental obstacle achieving sustained learning and adaptability in artificial agents, which echoes the flastiiitysstabiliy dilemma (Abraham & Robins, 2005) observed in biological neural networks.
There have been several recent studies highlighting a significant loss of plasticity in deep RL (Ku- mar et al., 2021; Lyle al., 2022), which substantially restricts the agent's ability to learn from subsequent experiences (Lyle et al., 2023; Ma et al., 2024). The identification of primacy bias (Nik- ishin et al., 2022) further illustrates how agents may become overfitted to early experiences, which inhibits learning from subsequent new data. The consequences of plasticity loss further impede deep RL in continual learning scenarios, where the agent struggles to sequentially learn across a series of different tasks (Dohare et al., 2024).
Research on addressing plasticity loss in deep RL is still in its early stages, with recent approaches including parameter resetting (Nikishin al., 2022: Sokar al., 2023) (or its advancement with ran- dom head copies (Nikishin et al., 2024)), and several implementation-leee techniques like normal- ization, activation functions, weight clipping, and batch size adjustments (Obando Ceron al., 2023; Nauman et al., 2024; Elsayed et al., 2024). However, reset-based methods often lead to performance
"Corresponding author, email: lingpan Siggaan@ust.ht uk

---

Published as a conference paper at ICLR 2025
instability and training inefficiency due the need for period re-training, while implementation- level modifications lack generalizability and do not directly target the plasticity issue. The field currently lacks unified methodology that can effectively address plasticity loss while maintaining training stability and efficiency across varying environments.
Humans adapt to environmental changes and novel experiences through cortical cortex expansion in cognitive science (Welker, 1990; Hill et al., 2010). This process involves the gradual activation of additional neurons and the formation of new connections facilitate the ability to learn continually. Drawing inspiration from this biological mechanism, we propose a novel perspective Neuroplastic Expansion, which can help maintain plasticity in deep RL. The key insight is that an agent starting learning with smaller network and dynamically growing to larger size (ultimately reaching the original static network dimension) can effectively tackle plasticity loss by maintaining a high level of elastic neurons throughout training (Figure 1). Elastic neurons means neurons that have plasticity. In this paper, the words elastic and plastic are interchangeably. We first provide empirical evidence in ($3.1) validating its potential for mitigating plasticity loss and improving final performance in certain cases, even with a naive incremental expansion. We provide a detailed description of plasticity loss and pruning for RL in Appendix A.
Building upon this insight, we systemat- ically introduce Neuroplastic Expansion (NE), a simple yet effective mechanism to maximize the benefits of incremental growth training ($3.2). NE adds high- quality elastic candidates based on poten- tial gradients (Evci et al., 2020). How- ever, this will lead to increased compu- tation costs for always enlarging the net- work, and cannot fully leverage its expres- sivity as elastic neurons, which are acti- vated and can be updated to fit the new data, may turn dormant during the course of training. As visualized in Appendix F.6, dormant neurons can diminish representa- tional capacity and lead to suboptimal be-
Elastic Neural Generation
Dormant Pruning
Enyirohimnt
havior.
Figure 1: High-level illustration of Neuroplastic Ex- pansion RL. The network regenerates elastic neurons based on gradient potential, recycles dormant neurons, and undergoes progressive topology growth to mitigate plasticity loss. The agent consolidates neurons through experience review, preserving prior helpful knowledge and ensuring policy stability.
To address this challenge, NE introduces reactivation process for dormant neu- rons, pruning them and potentially reintro- ducing them as candidates in subsequent growth stages; thereby better utilizing the network expressivity and enhancing the policy's 's sustainable learning ability.
Unlike previous reset-based approaches (such as resetting the last layers, introducing copies of heads or reinitializing parts of the network) that risk forgetting and require periodic re-training, NE main- tains learning continuity in a smoother way. To further mitigate potential instability from continu- ous topology adjustments, we introduce an effective Experience Review technique that consolidates neurons by adaptively reviewing prior knowledge during the later training stages, ensuring policy stability and balancing the plasstiiyy dilemma. Extensive experiments on long-term train- ing, continual adaptation, and vision RL tasks demonstrate the effectiveness of our method across various scenarios, showcasing its ability to maintain plasticity while ensuring policy stability.
The main contributions are summarized as follows:
We introduce a novel mechanism, Neuroplastic Expansion (NE), to mitigate the loss of plasticity in deep RI.
We develop effective activated neuron generation and dormant neuron pruning mechanism for better network capacity utilization, and a neuron consolidation technique for preventing forgetting helpful reusable knowledge.
2

---

Published as a conference paper at ICLR 2025
We conduct extensive experiments in standard RL tasks, which demonstrate the effectiveness of NE across various tasks including MuJoCo (Todorov et al., 2012) and DeepMind Control Suite (DMC) (Tassa et al., 2018a) tasks that outperform previous strong baselines by large margin, and can be effectively adapted to continual learning scenarios.
BACKGROUND
Deep Reinforcement Learning The reinforcement learning problem can be typically formulated by Markov decision process (MDP) represented as a tuple (S, P, R. 7), with S denoting the state space, A the action space, P the transition dynamics:
x A x S + [0, 1], R the reward function: S x A + R. and y E [0, 1) the discount factor. The agent interacts with the unknown environment with its policy n, which optimal policy that maximizes the expected discounted long-ter term reward. The state-action value of 8 and a under policy и is defined as Q" (s, a) = E. (2t= RR( ax) | = <p a).
a mapping from states to actions, and aims to learn an
In actor-critic methods, the actor T and the critic Qo are represented using neural networks as function approximators with parameters & and 0 (Fujimoto et al., 2018; Haarnoja et al., 2018). The critic network is updated by minimizing the temporal difference loss, i.e. (Q Ep (Q)os, Q7 (s, a)) where Q7 (s.a) denotes the bootstrapping target R(s, a) Qg ms('s) computed using target network parameterized by and 0 based on data sampled from a replay buffer D. The actor network is typically updated to maximize the Q-function approximation according to VoJ( ED
VaQo(s, a)la=z)(s) (s)
Activated Neuron Ratio Activated neurons are easily updated by new data. Therefore, the pro- portion of activated neurons in the network, i.e. activated neuron ratio is often considered positively correlated with plasticity Ma et al. (2024). It is defined as the proportion of neurons whose output exceeds a certain threshold in the neural network, effectively counting all neurons except dormant ones (Xu et al., 2024). Formally, a neuron in layer is considered activated when output h2x))) >> 0
(in contrast to dormant neurons (Sokar et al., 2023)):
f(4)) Eietzera (1,(2)) >
(1)
N
where Td denotes the input distribution. The relationship between the Activated Neuron Ratio curve and plasticity is as follows: The primary cause behind an agent's plasticity loss is the rapid dor- mancy of numerous neurons training progresses, which diminishes the model's representational capacity (Sokar et al., 2023; Qin et al.). Consequently, delaying the reduction in activated neurons commonly considered as a way to mitigate plasticity loss, i.e, slowing the downward trend of the curve. The Activated Neuron Ratio is widely used for plasticity visualization (Ma et al., 2024) due to its intuitive interpretability.
NEUROPLASTER EXPANSION RL
In this section, begin by discussing the insights of Neuroplastic Expansion RL empirically, and analyze its effect on mitigating the loss of plasticity in Section 3.1. Then, we systematically present the details of our method in Section 3.2.
3.1 ILLUSTRATION OF INSIGHTS OF DYNAMIC GROWING RI.
There have been several recent studies investigating loss of plasticity in supervised learning (Lyle et al., 2023; Lewandowski et al., 2024), which corresponds the phenomenon where neural net- works gradually lose the ability to adapt and learn from new experiences (Lin et al., 2022). This is further exacerbated in RL (Sokar et al., 2023; Nikishin et al., 2024), due to the non-stationary nature and the tendency to overfit prior knowledge, resulting in suboptimal policies. A predominant ap- proach to tackle plasticity loss is based on resetting (Nikishin et al., 2022), where the agent resets the last few layers of its neural network periodically throughout training. However, although effective, this kind of approach typically forgets helpful and reusable knowledge which is critical for learning, and thus the agent experiences drop in performance once resetting is applied.
3

---

Published as a conference paper at ICLR 2025
A major cause for agents to adapt to novel circumstances is the growth of the human cortex (Hill et al., 2010). Motivated by this insight, we propose a novel perspective that RL agents can maintain high plasticity through dynamic neural expansion mechanisms, sustaining the policy's continual adaptation ability based on new experiences. Initially, data collected by a random policy may be of lower quality and therefore require less network expressivity to fit (Burda et al., 2019; Zhelo al,, 2018). As the agent improves, it tends to require more expressive network for fitting the more complex value estimates and policies. Maintaining neuronal vitality and regeneration during training can significantly mitigate plasticity loss, as it allows for ongoing adaptation to new information.
Motivated by this insight, we propose a novel angle, Neuroplastic Expansion, where the agent starts with a smaller net- work that gradually evolves into larger one throughout the learning process. We first present naive implementation of this idea and then systematically discuss methodology in Section 3.2.
CArE
1.0x
1.8k
2,8H 3 0x
3,54
30
120 25 300 750
Concretely, we adopt a -mall-to-large neural expansion approach, where the ini- tial network capacity is 20% of the size of typical full TD3 (Fujimoto al., 2018 8) network (i.e., uniformly selected 20% of neurons from each layer of the full TD3 architecture). To validate the potential of this novel perspective, we first consider the simplest approach for growing the network by uni- formly adding i new connections the current critic topology Ā C 0 and the current actor topol- ogy & for each layer 1 E N, which then participate in gradient propagation according to Igrow Random, g (0,, k) U Random ăgă, (di, k).
Figure 2: Comparison of normalized performance (left) and the ratio of active neurons (right) for vanilla TD3 (Fujimoto et al., 2018) and its variants: with naive random growth network and NE method, across varying network capacities.
We build this idea upon TD3 and conduct fair comparisons with vanilla TD3 (i.e., with static actor and critic networks) using different network capacities. We compare the performance our dynam- ically growing network against naive topology growth strategy and static networks in standard TD3 (Fujimoto et al., 2018) using the HalfCheetah environment. To ensure fairness in comparison, the final network size of NE is to be identical to that of the other methods. Therefore, the comparison with the vanilla TD3 algorithm is fair given the same network capacity at the end of training.
From the left side of Figure 2, observe that even a naive incremental network growth strategy leads to noticeable performance improvements, particularly as network capacity increases. The right side of Figure demonstrates the ratio of active neurons (Sokar et al., 2023) for each algorithm during training. This metric evaluates whether RL agents maintain their expressive capacity, which positively correlated with plasticity retention. As shown, topology growth effectively reduces neuron deactivation, thereby preserving the agent's 's ability to learn policies, mitigating plasticity loss, and alleviating primacy bias.
3.2 METHOD: NEUROPLASTIC EXPANSION
Hill et al. (2010) demonstrated that the cerebral cortex expands in response to environmental stim- uli, learning, and novel experiences. This dynamic growth and reorganization enhance cognitive and decision-making abilities, fostering adaptability to changing situations. Motivated by our prelimi- nary results and inspired by this biological mechanism, we present Neuroplastic Expansion (NE), training framework for RL agents that employs dynamically expanding network to retain learning capacity and adaptability throughout the entire learning process.
Elastic Topology Generation While incremental random topology growth has contributed to maintaining plasticity, we aim to elucidate the underlying mechanisms that drive its effectiveness. To this end, we explore the dormant neuron theory (Sokar et al., 2023). a prevalent explanation for plasticity loss. This theory posits that neurons within network become inactive during train- ing, typically when their activation values approach zero (Lu et al., 2019). When this occurs, these neurons lose their ability to leam, reducing the network's overall capacity to efficiently process
4

---

Published as a conference paper at ICLR 2025
new situations. Their outputs gradually diminish, ultimately falling below the activation function threshold (Lu et al., 2019). Mcanwhile, prior research has shown that parameters that lack prompt exposure to -ighhgradient stimuli more prone to dormancy. Drawing these findings, we derive the following observation:
Naive random topology growth sporadically introduces new connections that can generate significant gradients for neurons nearing dormancy, thereby supplying them with strong backpropagation signals and allowing them to continue learning.
If the above conjecture holds in
deep RL, then selecting topol- ogy candidates based gra- dient magnitude may mitigate plasticity loss more effectively than randomly expanding net- work connections. To this end, we follow the sparse network training framework in Tan et al. (2022). Before each topology expansion, we sample a batch of transitions from buffer D to compute the gradient magnitude across all model parameters, for- malized as V Then, we select & connections that yield the highest gradient magnitudes to aug- ment the topology: groow ArgToopkigg (IV)) I.). The topology growing process G3. adaptively starts every AT step. We try several growing schedules to achieve warm topology growth to prevent training instability caused by sigmificant network modifications. We chose cosine annealing through practice (see Appendix F.5 for experiments): at timestep the k is decayed in a cosine annealing manner: (1 + cos( fr )) where a denotes the discount factor and Tend is the shut down step. Finally, we restrict the weights of the newly included neurons (which are randomly initialized) to a specific range for training stabilization (Nauman et al., 2024) via weight clipping (Elsayed et al., 2024). Following standard practice (Liu et al., 2022), we start applying weight clipping on both the input and output weights of neuron once it is incorporated into the topology.
Normaltzed Purfermance (N)
2.0
wio Frunc
0.8
ratte
0.50
Grow Prune
Rondom grow
0.45
0.4
0.40
0.2
Grow Prume
Grow wi Frune
0.35
Random Grow
0.30
Figure 3: Performance comparison and plasticity evaluation of random growth, our proposed growth method, and pruning mechanisms. All experiments were conducted with seven inde- pendent seeds.
The results in Figure 3 indicate that substituting random growth with gradient- based selection of new connections enhances agent performance and mitigates dormant neuron issues. Additionally, we compare two commonly used topology initialization strategies: uniform and Erdos-Rrenyi (Evei et al., 2020), and we find that initializing the network using the Frdos-Rreny type is slightly more effective for RL. This is because Sedoss-Rrenys enables the number of connections in a sparse layer to scale with the sum of the input and output channels, leading to more stable computations in smaller networks. Consequently, the agent can achieve more efficient learning in the early stages of training.
Dormant Neuron Pruning After replacing the naive growth strategy with our moudient-guides growth process Gg, new challenge emerges, i.e., dormant neurons experience significant decline in representational potency while continuing to occupy network capacity. These inactive neurons may be inherent from initialization or introduced into the network topology through the growth schedule. This situation can lead to a topology that saturates quickly, resulting in disproportionately low computational power. Moreover, Lu et al. (2019); Liu et al. (2019) demonstrated that parameters exclusively forward- linked to dormant neurons are omitted from the gradient calculation process. As a result, they become ineffective, as they fail receive meaningful guidance signals. We hypothesize that pruning and resetting dormant neurons following topology expansion can address this issue.
This process aims free up space for new connections, allowing dormant neurons to be reactivated as new candidates. According to Ceron et al. (2024a;b); Sokar et al. (2025), sparsifying the value- based agents' network can often enhance performance, which empirically supports our real-time pruning approach. Consequently, we introduce a synchronous prune-and-reset mechanism into the topology growth schedule Gg- Here, we adopt the calculation method f(.) shown in Eq. and set to be O, which means only considering fully dormant neurons: Iprune (in) = 0). To facilitate the topology's 's gradual growth and avoid over- wver-pruning, we employ the Truncate process (that operates the pruning set T! prune) which drops excess elements randomly from the set when
5

---

Published as a conference paper at ICLR 2025
its size exceeds predefined upper bound. The new truncated set is then used for pruning (Refer to Appendix D.2 for the detailed pseudo-code of our Truncate Process). In practice, we define the pruning upper bound in each layer as w x grgooll where [ represents the number of elements in the current grow set. The parameter w E [0. 1) acts "discount" factor, moderating the pruning relative to the total network growth in layer I. Thus, the two final parameters controlling topology schedule as GR,-- The complete growth-pruning
evolution are i and We denote our schedule for each layer in actor & and critic 0 networks is defined as:
1. I grow ArgTopk gg (()) U ArgT opk, .0 (V)22)
Gke
{
2. I prune = {Index ())(())) = 0} u U (Index(0)) 0}:
We then use Ty Tyrour Uprune to generate (0, 13. which are dot-multiplied with the network parameters to achieve the evolution of the actor topology ő and critic topology Ö. The results in Figure demonstrate that introducing the pruning mechanism after each growth round further preserves the plasticity of the RL network and enhances overall performance. NE is inspired by plasticity injection (PI) (Nikishin et al., 2024). While both NE and PI maintain plasticity by extending networks, PI focuses one-time large-scale network growth, whereas our method achieves continuous real-time adaptation.
Notably, our sparse network training framework for maintaining stable learning is summarized as follows: (i) Inspired by Evci et al. (2020), we use Erdos-Rrenyi initialization to ensure that the num- ber of connections in a sparse layer scales with the sum of the output and input channels, thereby enhancing initial stability. (ii) We employ cosine annealing to guide gradual network growth, re- ducing noise caused by network changes. (iii) Following sparse training practices in RI. (Tan al., 2022), we keep the first and last layers dense to ensure the stability of the encoding and decoding.
Neuron Consolidation via Experi- ence Review As discussed above, the proposed elastic topology gener- ation and dormant neuron pruning- based dynamic growth approach ef- fectively mitigate plasticity loss. This method maintains a higher pro- portion of active neurons, leading to improved performance. How- ever, closer examination of the re- sults, i.e. Tab. 8, reveals that NE demonstrates high standard devi- ation across seeds in the later stage of training, suggesting potential risk of performance oscillation near convergence. We hypothe- size that frequent network topology changes may introduce slight probabilistic errors in the network structure. Consequently, as the activated neuron ratio decreases, some agents may risk forgetting previously learned skills from the early training stages (e.g., standing in the HalfCheetah task). This phenomenon leads to suboptimal behavior post-convergence and increases variance between sceds.
3420
Ralwen)
5tep
Figure 4: Instability of NE in the later stage without neuron consolidation via experience review.
We examined the behaviors on the HalfCheetah task and found that there are multiple runs in which the agent no longer could stand after a fall (Figure 4). This observation strengthens our confidence in the above conjecture. To address this, we seek a topology -agnosyy and simple mechanism to further enhance the stability of NE by mitigating catastrophic forgetting. Meanwhile, prior studies (Rolnick et al., 2019; Zhou al., 2020) show that experience replay (ER) helps mitigate forgetting and sup- ports network plasticity in both RI and supervised learning. Inspired by these findings, we aim to develop ER mechanism sensitive to both the training stage and the activated neuron ratio.
In practice, we define the absolute slope of the activated neuron ratio curve as: A(f = Eico (0))))] the threshold e. Specifically, e measures the slope of changes in the number of activated neurons (within the critic network) during the recent steps, i..., [150, 450]. A larger positive value of e corresponds to greater reduction in dormant neurons, indicating active plasticity improvements in our dynamically growing network. In contrast, small (close to zero) suggests

---

Published as a conference paper at ICLR 2025
5000
Ours
400
Passs
LN
3000
Reddo
200
200
2000
2000
2010
step
Step
(a) Hopper
(b) Walker2d
(c) HalfCheetah
8000
6000
4:00
2000
200
300
step
ktep
(d) Ant
(c) Humanoid
Activated neuron ratio
Figure 6: Performance comparisons OpenAI MuJoCo environments (Todorov et al., 2012): (a) Hopper, (b) Walker2D, (c) HalfCheetah, (d) Ant, (e) Humanoid. (f) Results of the activated neuron ratio. NE outperforms the baselines in four tasks and maintains a high activated neuron ratio.
bottleneck in plasticity gains, meaning limited capacity for further topology expansion and marking the later stage of policy learning (see Figure 5).
During training, we observe that generally decreases over time, since elastic neuron generation and dormant neuron pruning significantly impact the number of dormant neurons in the early training stage. However, due to limited network capacity, they cannot indefinitely increase the proportion of active neurons. To counteract this effect, we encourage the agent to review earlier experiences from the buffer when dormant neurons are more prevalent. At each training step, random number m E [0, 1] selected, if m > e, the sampling is conducted from the initial quarter of the buffer; otherwise, the conventional sampling method is employed. We provide pseudocode outlining the full algorithmic process in Appendix D.I.
EXPERIMENTS
Activated Neuron Ratio
5.45
In this section, we conduct comprehensive experiments to evaluate whether NE can alleviate the loss of plastic- ity. investigate the following key questions: (i (54.1)) In long-term training setting, NE effectively combat early data overfitting, enhance performance, and attenu- ate the decline in activation neuron ratio? (ii ($4.2)) In continuous adaptation task, can NE enable agents to ef- ficiently adapt to new tasks while slowing the dormancy rate of activated neurons, without being constrained by prior learning limitations?
2:40
05
1.0
1.5
25
3.5
Dynamic Threshold
0.5
stops
146
(ili ($4.3)) What are the effects of NE on the policy and value networks, and what the importance of its different components? (iv (84.4)) Can our method be applied to other deep RI. methods with more complex image inputs?
Figure 5: How changes with plastic- ity measurement. In practice, we set lower bound for
4.1 LONG-TERM TRAINING TASKS
Experimental Setup We conduct series of experiments based the standard continuous trol tasks from OpenAl Gym (Brockman, 2016) simulated by MuJoCo (Todorov et al., 2012) with long-term training setting, i.e. 3M steps 6M. We compare NE. against strong baselines includ-

---

Published as a conference paper at ICLR 2025
ing Reset (Nikishin et al., 2022), ReDo (Sokar al., 2023), Layer Normalization (LN) (Lyle al., 2024b), and Plasticity Injection (PI) (Nikishin et al., 2024). To ensure a fair comparison and mit- official implementations (Sokar et al., 2023;
igate implementation bias, base all baselines Nikishin et al., 2024; 2022), with the same architecture (See Appendix C.1). All baselines are im- plemented using the TD3 (Fujimoto et al., 2018) algorithm (where results based on other backbone deep RL methods are discussed in Section 4.4). To ensure a fair comparison and reprodueibility of the expected performance of all baselines, we follow the recommended hyperparameter setting from Elsayed et al. (2024) (see Appendix C) for all baselines. The clipping parameter is set to 3. Each algorithm is trained with random seeds, and we report the mean and standard deviation. A detailed description of hyperparameters and experimental setup is in Appendix C.2. For all tasks, NE initializes agents 25% of total capacity and grows asymptotically. The growth termination scale is aligned with baselines. The score denotes undiscounted episodic return.
Results As shown in Figure 6, the Reset method suffers from periodic performance drops due to resetting a large portion of layer parameters. This leads to the loss of reusable knowledge, requir- ing relearning, which impedes overall progress. LN and ReDo demonstrate more stable performance than Reset, but result in lower proportion of active neurons. PI maintains the highest level of neuron activation, but performs well only the relatively simpler Hopper environment. NE achieves signif- icant and consistent improvements in learning efficiency and final performance, with a larger margin in more complex environments. Furthermore, NE effectively mitigates plasticity loss, maintaining higher active neuron ratios throughout training compared to the most competitive LN method. Ad- ditionally, NE achieves a superior trade-off between performance and neuron utilization. It performs more stably and outperforms PI, which, despite achieving the highest neuron activation, exhibits suboptimal performance in most tasks.
4.2 CONTINUAL ADAPTATION
In this section, we investigate the continual ability to learn and adapt to changing environments, which is essential capability of deep RI. agents (Willi* al., 2024; Elsayed & Mahmood, 2024).
We follow the experimental paradigm of Abbas et al. (2023) and evaluate our method variant of MuJoCo tasks that introduce non-stationarity due to changing environments over time. Specifically, the agent is trained to master a sequence of 4 environments (HalfCheetah - Humanoid Ant HoHopr))), starting with HalfCheetah for 1000 episodes of training. followed by the next task for the same number of episodes. cycle is completed when training on Hopper is finished. The agent repeats this sequence three times, forming long-term training schedule. Each task maintains an output head (I layer MLP), which is trained in conjunction with the backbone network. Our method is applied all layers except the output head. We compare our approach against the vanilla TD3 agent and the most competitive baseline, Reset.
Experimental Setup
The performance of each method in single en- vironment, when repeatedly learning across sequence of environments as described above, summarized in Fig- ure 8. We aim to evaluate (i) the efficiency of the policy in quickly learning new task after mastering the previous one within single cycle; (ii) the agent's ability to retain the benefits of initial learning for the same task across multiple cycles. As shown, Resetting, which involves ini- tializing the parameters after each task, is currently con- sidered the most effective approach.
Results
0.5
Nettrate eaveee th1
0.4
0.2
0.3
0.1
steps
Activated neuron ratio, recorded for three agents with different settings during cyclic training.
Figure 7:
According to results in Figure 8, NE effectively handles each task without being negatively influenced by prior training. significantly outperforms the original TD3, reinforcing that standard RL algorithms suffer from catastrophic forgetting in continual learning setting. We attribute this success to NE's s ability to introduce real-time topology adjustments, which facilitate storing new knowledge and ensuring continuous learning capabilities.
8

---

Published as a conference paper at ICLR 2025
Notably, in certain environments, such Ant, NE-TD3 retains prior knowledge and seamlessly continues learning. This could be attributed to the experience review module and the preservation of old neurons within the topology, which store critical information essential for successful continual learning. When analyzing column-wise comparisons, NE performs comparably to Reset in most tasks, yet significantly outperforms Reset in the challenging Humanoid task. The comparison of active neurons for each baseline is shown in Figure 7, which illustrates that even during extended training, NE effectively mitigates neuron dormancy while maintaining efficient learning. In fact, even outperforms Reset in the early stages of training, despite Reset reinitializing all parameters.
The spike phenomenon in Figure 7 is related to research examining the relationship between input distribution and plasticity. Lu et al. (2019) observed that neurons failing to activate due to the activation function can be reactivated when the input distribution to the network changes. similar phenomenon has been reported in Ma et al. (2024), where data augmentation was found to alleviate the decline in activated neurons. In the continuous adaptation task we tested, a sudden change in the environment may have altered the input data distribution, leading temporary increase in the activated neuron rate. However, after a short training period, the neurons returned to a dormant state.
(a) 1tt task:
HalfCheetah
(b) 2nd task: Humanoid
(c) 3rd task:Ant
(d) 4th task:Hopper
Figure 8: Learning curves of three methods (NE, Reset, Vanilla) continuous adaptation tasks.
4.3 ABLATION STUDY
In this section, we conduct an additional ablation study to investigate the effectiveness of neuron con- solidation in Neuroplastic Expansion, while the importance of clastic neural generation and dormant neuron pruning has already been demonstrated in Figure 3. Additionally, we examine the experience review (ER) module and assess the impact of our method the policy and value networks.
H2 maak otf
12 0ECC
(a) Validation
(b) Performance
(c) Critic plasticity
(d) Actor plasticity
Figure 9: Ablation study: (a) presents the validity verification of ER, while (b) to (d) illustrate the
impact of NE on the actor and critic from the perspectives of plasticity and performance.
As illustrated in Figure 9a, the results on HalfCheetah show ER improves the training stability, par- ticularly in the later stage, Additionally, we select 4 tasks with image input and sparse reward setting further verify the effectiveness of ER. The results in Appendix F.4 indicate that when assisted with ER, our training framework consistently stabilizes the convergent policy at a high level across differ- ent seeds (i.e., lower standard deviation). In contrast, the absence of this critical component makes the model more vulnerable to policy collapse in later stages (albeit to lesser extend than Reset). This demonstrates the importance of reviewing historical knowledge in maintaining policy stability and preventing the loss of carly valuable information, discussed in Section 3.2.
Figure 9b illustrates a comparative analysis our method and its variants, isolating the effects NE on the actor and critic networks separately. We observe that NE plays particularly crucial role in the critic network. Removing NE from the critic results in significant performance degradation and substantial plasticity loss compared to the full version of NE (Figure 9c, 9d). In contrast, limiting expansion to only the critic leads to a less pronounced drop in performance. This asymmetry in impact may be attributed to the greater challenge faced by the critic, it must provide accurate value

---

Published as a conference paper at ICLR 2025
estimates for various state-action pairs (Meng al., 2021) under non-stationary data distribution. In addition, the critic provides guidance signals for actor optimization, which requires a high level of adaptability (Fujimoto et al., 2024).
4.4 VERSATILITY
In this section, we demonstrate the generality of our proposed method by applying it to another popular deep RI. algorithm and evaluating its performance on more complex tasks with image inputs.
Experimental Setup We evaluate four image-based motion control tasks from the DeepMind Control Suite (DMC) (Tassa et al., 2018b): Reacher Hard, Reacher Easy, Walker Walk, and Cart- pole Swingup Sparse, demonstrate the generalization of NE across deep RL algorithms and its robustness across diverse tasks. For the backbone algorithm, we use DrQ (Yarats et al., 2022), a well-known variant of SAC (Haarnoja al., 2018) specifically designed for processing image in- puts. We compare our approach against the same baselines used in Section 4.1. The weight clipping parameter K is
Tado
Figure 10:
The first row shows the performance of five methods. The second row corresponds to the with seven independent seeds.
percentage of activated neurons. All experiments
The first row of Figure 10 shows that NE can be effectively combined with DrQ achieve stable performance on image-input tasks, outperforming all baselines and demonstrating its ver- satility across different algorithms. Additionally, the second row of Figure 10 (activated neuron ratio) indicates that although NE does not reach the peak activation levels of Reset, it delays neuron dormancy more effectively than other warm mitigation strategies, ensuring stable performance.
Results
5 CONCLUSION
Inspired by the cortical growth in biological agents that triggers neuronal topology expansion to enhance plasticity and adaptability to new situations, we introduce a novel perspective for deep RL agents to mitigate the loss of plasticity through dynamic growing network. Building on this insight, we present comprehensive topology growth framework, Neuroplastic Expansion (NE), which addresses both growth and pruning mechanisms maximize the performance benefits of topology expansion. NE adds high-quality elastic neurons and connections based on the gradient signals improve the network's learning ability; prunes dormant neurons in a timely manner for better utilizing network capacity, and returns them to the candidate set to maintain the agent's s plasticity, making dormant neurons reusable. NE outperforms previous methods that aimed at alleviating plasticity loss on diverse tasks and demonstrates stable, outstanding performance on plasticity metrics. Future directions include exploring adaptive architectures that dynamically adjust their capacity based on the agent's performance or task complexity, as well further reducing computational overhead. Overall, despite its acknowledged limitations, NE provides practical insights into enhancing the plasticity of deep RL agents. We hope findings pave the way for future research, potentially leading to more sample-efficiett and adaptable deep RL algorithms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work is supported by the National Natural Science Foundation of China 62406266. We would also like to thank the Python community (Van Rossum Drake Jr, 1995; Oliphant, 2007) for devel- oping tools that enabled this work, including NumPy (Harris et al., 2020) and Matplotlib (Hunter, 2007).
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp. 620- 636. PMLR, 2023.
Wickliffe C Abraham and Anthony Robins. Memory retention-th the synaptic stability versus plastic- ity dilemma. Trends neurosciences, 28(2):73-78, 2005.
Jordan T. Ash and Ryan P. Adams. On warm-starting neural network training. In Proceedings of the 34th International Conference on Newral Information Processing Systems, NIPS '20, Red Hook, NY. USA, 2020. Curran Associates Inc. ISBN 9781713829525
Tudor Berariu, Wojeiech M. Czarnecki, Soham De, Jörg Bornschein, Samuel L. Smith, Razvan Pas- canu, and Claudia Clopath. A study on the plasticity neural networks. ArXir, abs/2106.00042, 2021. URL https: api semant
Corppu ID: 2602
Brockman. Openai gym. arXiv preprint arXiv:1606.015400 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https:
/ //oeerreview.. net.
forum? www.//orum? d=H1 d=H1133nR55mm
Johan Samir Obando Ceron, Aaron Courville, and Pablo Samuel Castro. In value-based deep inforcement learning, a pruned network is a good network. In Forty-first International Con- ference on Machine Learning, 2024a. URL https:/ seo9V9QRZp.
https://openreval http////ppnreeiieeeeeeeee net fonttffoummid
Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nico- laus Foerster, Gintare Karolina Dziugaite. Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL. In Forty-first International Conference on Machine Learning. 2024b. URL https:://// https://ppenzeview.
hwtp///ppww..e.eeeeeeeeee new. umrommmm fow.mm/ddxX9/hhhFxxo.....
Shibhansh Dohare, Fernando Hernandez-Gane Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Mohamed Elsayed and A Rupam Mahmood. Addressing loss of plasticity and catastrophic forget- ting in continual learning. arXiv preprint axXiv:2404.0078/, 2024.
Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. Reinforcement Learning Conference, 2024.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pp. 2943-2953 PMLR, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, Wii-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: atate-action representation learning for deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2025
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp. 7766-7792. PMLR. 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, 585(7825):357--36) 2020.
Jason Hill, Terrie Inder, Jeffrey Neil, Donna Dierker, John Harwell, and David Van Essen. Simi- lar patterns of cortical expansion during human development and evolution. Proceedings of the National Academy of Sciences, 107(29):11135-11144 2010.
John D Hunter. Matplotlib: 2d graphics environment. Computing in science & engineering, (03):90-95,, 2007.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint 4126:91412.6988, 2014.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterriztion inhibits data-efficient deep reinforcement learning. In International Conference on Learning Rep-
resentations, 2021. URL https: /wtp///peee......e//////. net www.ffrrmmiiiooooo www.ommdddo.oiieeeeeeeoeeeeeeeee fXU.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308, 11958, 2023.
Hojoon Lec, Hanseul Cho, Hyunseung Kim, Dachoon Gwak, Joonkee Kim, Jacgul Choo, Se- Young Yun, and Chulhce Yun. Plastic: Improving input and label plasticity for sample effi- cient reinforcement learning. In Neural Information Processing Systems, 2023. URL https: //api. www.eeetintnnolteeeeeeeo
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In International Conference Machine Learning, pp. 26416-26438. PMLR, 2024a.
Joo Hyung Lee, Wonpyo Park, Nicole Elyse Mitchell, Jonathan Pilault, Johan Samir Obando Ceron, Han-Byul Kim, Namhoon Lee, Elias Frantar, Yun Long, Amir Yazdanbakhsh, et al. Jaxpruner: A concise library for sparsity research. In Conference Parsimony and Learning, pp. 515-528. PMLR, 2024b.
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado. Directions of curvature as an explanation for loss of plasticity, 2024. URL https: htapps://arxive htt///axvvvvrrreeeeeee.e..... abs/
TP Lillicrap. Continuous control with deep reinforcement learning. arXiv:1509.02971, 2015.
arXiv preprint
Guoliang Lin, Hanlu Chu, and Hanjiang Lai. Towards better slasticity-stability trade-off in incre- mental learning: A simple linear connector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 89-98, 2022.
Hsuch-Th Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-13, 2022.
Yang Liu, Jianpeng Zhang, Chao Gao, Jinghua Qu, and Lixin Ji. Natural-ogaritm--rettted acti- vation function in convolutional neural networks. 20/9 IEEE 5th International Conference on Computer and Communications (ICCC), pp. 2000-2008. IEEE, 2019.
Robert B Livingston. Brain mechanisms in conditioning and learning. Technical report, 1966.
12

---

Published as a conference paper at ICLR 2025
Dor Livne and Kobi Cohen. Pops:
Policy pruning and shrinking for deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing, 14(4):789-800) 2020.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv:1903: 06733, 2019.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in re- inforcement learning. International Conference on Learning Representations, 2022. URL
.net wwt.torumm forum?
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp. 23190-23211. PMLR, 2023.
Clare Lyle. Zeyu Zheng, Khimya Khetarpal, James Martens, Hado P van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning, Advances in Neural Information Processing Systems, 37:106440-106444 2024a.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18862. 2024b.
Guozheng Ma, Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang, Yixin Chen, Li Shen, Xueqian Wang, and Dacheng Tao. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. In The Twelfth International Conference on Learning Representations, 2024. URL http:tt//ppeeeeeeieiieeee//// www.:neeooommmmiddaeee.....
Pedro Aptios-Aparicio and Antonio Rodríguez-Moreno. The impact of studying brain plasticity. Frontiers cellular neuroscience, 13:66, 2019.
Lingheng Meng, Rob Gorbet, and Dana Kulić. The effect multi-step methods on overestimation in deep reinforcement learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 347-353. IEFE, 2021.
Michal Nauman, Michal Bortkiewicz, Piotr Miloš, Tomasz Trzcinski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfiuting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 37342-37364, PMLR, 21-27 Jul 2024. URL https: https://proceedinngg http///poceeeengeeeee//// mlr. / press/ v235 http///amaaaaaaaaaammmm html.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, Pp. 16828-16847. PMLR, 17-23 Jul 2022. URL https: http:s//ppccceeingg/.. mlr. p.rsss/ wwtp:n/iw.iiiiiei..eee
html.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Johan Obando Ceron, Marc Bellemare, and Pablo Samuel Castro. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36:26003-26024, 2023.
Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3):
10-20, 2007. doi: 10. 10110//S.22200.555
Haoyuan Qin, Chennan Ma, Mian Deng, Zhengzhu Liu, Songzhu Mei, Xinwang Liu, Cheng Wang, and Siqi Shen. The dormant neuron phenomenon in multi-agent reinforcement learning value factorization. In The eightt Annual Conference Neural Information Processing Systems.
13

---

Published as a conference paper at ICLR 2025
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efñi PMLR, 2023.
ciency. In International Conference on Machine Learning, pp.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th Inter- national Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 32145-32168, PMLR, 23-29 Jul 2023. URL httpp:://ppoceeeeeeeeeee
mmlr.
press/v202 wttp//aaaaraaaaa.. html.
Ghada Sokar, Johan Samir Obando Ceron, Aaron Courville, Hugo Larochelle, and Pablo Samuel Castro. Don't flatten, tokenize! unlocking the key to softmoe's efficacy in deep RL. In The Thirteenth International Conference on Learning Representations, 2025. URL https: // //openraview... www.:eee//ooooe wwtt:/foummmiiie ftt.oommmddooooooaaocoooo
Ghada AZN Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning (poster). In Sparsity in Neural Networks: Advancing Understanding and Practice 2021, 2021.
Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. Rix2: Training a sparse deep reinforcement learning model from scratch. arXiv preprini Stti::2005/50444 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki. Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 18Xi 1800000000 2018a.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap. and Martin Ried- miller. Deepmind control suite, 2018b. URL https::/ /ttp:///aaivviv/iii www.wiwww..w...............ooowe abs 110111111111 00690.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Wally Welker. Why Does Cerebral Cortex Fissure and Fold?, pp. 3-136. Springer US, Boston, MA, URL https://doi.
1990. ISBN 978-1-4615-3824-0. doi: 0001007/978-1-6615-8844444 org/10. 00011000//97 11tp://78----11--888--- 3824--0---
Timon Willi*, Johan Samir Obando Ceron*, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, and Pablo Samuel Castro. Mixture of experts mixture of RL settings. In Reinforcement Learn- ing Conference, 2024. URL https:tt///. hppp///eereeeeeeeeeeeeeeeee
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie 7e, Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning Representations, 2024. URL https: / //openzeview.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https: /ttp//ppppe.e...eeeeeeeee net wow.mmii=rrae.oo.....oooooorreee Satpimmiiid_____ea________e__
Oleksii Zhelo, Jingwei Zhang, Lei Tai, Ming Liu, and Wolfram Burgard. Curiosity-driven explo- ration for mapless navigation with deep reinforcement learning. arXiv preprint arXiv: 2018.
Fan Zhou, Chengtai Cao, Ting Zhong, Kumpeng Zhang, Goce Trajcevski, and Ji Geng. Continual graph learning. ArXiv, 200//000.00908, 2020. URL https:/ /ppi. htppa/pp.aaaaat.ee www.stetincooll................. org/CorpusTD: 461 2001.
14

---

Published as a conference paper at ICLR 2025
A
RELATED WORK
Plasticity loss in RL Recent studies have found that agents guided by the non-stationary objective characteristic of RL suffer catastrophic degradation (Lyle et al., 2023) and ultimately overfit early experiences, which limits their continuous learning ability. This phenomenon is known as plasticity loss. Behaviorally, the loss of plasticity during training can manifest as, for instance, an inability to obtain an effective gradient to guide policy adaptation in fine-tuning (Dohare et al., 2024), limiting potential transferability and making it difficult to adapt to new sampling and unseen data (refer to as primacy bias (Nikishin et al., 2022)). While the research on this topic is still in its early stages, several techniques have been demonstrated to mitigate the loss of plasticity in deep RL. Resetting global networks or dormant neurons (Nikishin et al., 2022; Sokar et al., 2023) and modifying ac- tivation functions (Abbas al., 2023) are observed to mitigate plasticity loss. Lyle et al. (2024b) empirically finds that network normalization, particularly layer regularization (LN), can maintain plasticity. Nikishin et al. (2024) introduced copies of random heads for injecting plasticity im- prove continual learning ability. PLASTIC (Lee et al., 2023) divides plasticity into input plasticity and label plasticity, addressing them separately. NaP (Lyle et al., 2024a) alleviates the over-fitting issue from the perspective of the association between regularization and learning rate. In pixel-based deep RL, data augmentation (Ma et al., 2024) and batch size reduction (Obando Ceron et al., 2023) have been analyzed for their effectiveness in reducing plasticity loss. Recent studies have found that mitigating gradient starvation (Dohare et al., 2024) and constraining deviations from initial weights (Kumar et al., 2023; Lewandowski et al., 2024) can also be effective in mitigating plasticity loss. In supervised learning, Lee et al. (2024a) proposed to use a combination of fast and slow update networks balance between fast, fleeting adaptation and slow, steady generalization in supervised learning. Ash & Adams (2020y; Berariu et al. (2021) found that both warm initialize and pre-train can better adapt to increasing amounts of data.
Pruning in RL Sokar al. (2021) showed that training the deep RL policy with changing topology is difficult due to training instability. Since then, this challenging topic has been well- studied. Policy Pruning and Shrinking (PoPs) (Livne & Cohen, 2020) obtain sparse deep RL agent with iterative parameter pruning. Graesser et al. (2022); Tan et al. (2022); Lee et al. (2024b) attempt to train a sparse neural network from scratch without pre-training dense teacher. Existing methods mainly focus on obtaining a smaller topology to improve training efficiency. However, our paper explores whether growing the topology from small to large can mitigate the plasticity loss in deep RL agents rather than training with the maximum number of parameters. In deep RL. the model size commonly utilized typically small and may encounter policy collapse after scaling up (Schwarzer et al., 2023; Ceron et al., 2024a). Thus, sparse training techniques that have proven beneficial for significantly reducing training costs in fully supervised domains may not yield similar advantages in deep RL in terms of computation cost reduction. Instead, we provide new perspective and focus on making deep RL agents better use of the growing network size to further improve their learning ability.
B RELATED PRELIMINARIES
TD3 In our paper, we use TD3 as practice backbone for most experiments.Thus here we in- troduce the training process of TD3 in detail. TD3 is a deterministic Actor-Critic algorithm. Dif- ferent from the traditional policy gradient method DDPG, TD3 utilizes two heterogeneous critic networks.i.e., Q0 to mitigate the over-optimize problem in Q learning.
The training process of TD3 follows the Temporal difference learning (TD):
LQ(0,) = E.,a,o [(2 - Qo, [) for Vi {1,2).
(3)
Where 3/ - r r 7 min Q0, (gs')), T denotes the target network parameters. The actor is 1=1.2
updated according to the Deterministic Policy Gradient (Fujimoto et al., 2018) At each training step t, the agent first randomly samples batch of state transitions from the buffer and calculates the loss gradient as described above. It is worth noting that the proposed method in our paper is a plug-in and does not affect the training process of the reinforcement learning algorithm itself.
15

---

Published as a conference paper at ICLR 2025
DrQ DrQ is an algorithm based on maximum entropy reinforcement learning. Its training method are the same as SAC (Haarnoja et al., 2018). The soft value function is trained to minimize the squared residual error:
(4) where is sampled according to the current policy instead of the replay buffer. The soft Q- function parameters can be trained to minimize the soft Bellman residual: (0( = E ~ DI (Q. a) a))).)) Then the policy network can be learned by directly minimizing the expected KL divergence
(r(6) (V)(() Qo(s,a) log (alss))
exp Q(5) (( = BBa)==-B(((D( 7(x)
(5)
C EXPERIMENTAL DETAILS
C.I NETWORK STRUCTURE
TD3 The normal size of the TD3 network we used is the official architecture, and the detailed setting is shown in Tab 1. Besides, the network we used for Scale-up experiments incrementally adds the depth both the actor and critic. For the 1.8x agent, we add one MLP as a hidden layer in two critic networks in front of the output head and one MLP layer the embedding layer at the front of the actor. And the hidden dim is also increased to 256. And so on, we follow the above approach to grow the model by equal amount each time.
Layer
Actor Network
Critic Network
Fully Connected
(state dim, 256) ReLU
(stale dim. 256) ReLUU
Activation
Fully Connected
(256, 128)
(256, 128)
Activation
ReLU
ReLU
Fully Connected
(128, action dim) and (128, 1)
Activation
Tanh
None
Table 1: Network Structures for TD3
DrQ We use DrQ to conduct experiments on robot control tasks within DeepMind Control using image input as the observation. All experiments are based on previously superior DrQ algorithms and maintain the architecture from the official setting unchanged.
C.2 IMPLEMENTATION DETAILS
TD3 Our codes implemented with Python 3.8 and Torch 1.12.1. All experiments were NVIDIA GeForce GTX 3090 GPUs. Each single training trial ranges from hours to 21 hours, depending the algorithms and environments, e.g. DrQ spends more time than TD3 to handle the image input and DMC needs more time than OpenAl mujoco for rendering. Our TD3 is imple- mented with reference to github- com/ fuj (erjjimTTD TD3 (TD3 source-code). The hyper-parameters for TD3 are presented Table 2. Notably, for all OpenAl mujoco experiments, we use the raw state and reward from the environment and no normalization or scaling is used. An exploration noise sampled from (0,, 0.1) (Lillicrap. 2015) is added to all baseline methods when selecting an action. The discounted factor 0.95, 0.99 and we use Adam Optimizer (Kingma, 2014) for all algorithms. Tabb2 shows the common hyperparameters of TD3 used in all our experiments. Following (Elsayed et al., 2024), We start applying weight clipping on both input and output weights of a neuron once it becomes part of the topology. To ensure a fair comparison and reproducibility of the expected performance of all baselines, we employ the recommended hyperparameter setting in Elsayed et al. (2024) for all baselines, with the clipping parameter input tasks. In our method, dormant neurons are randomly reinitialized after pruning, preparing them to rejoin the network topology. When they re-selected, weight clipping is applied to the
for MuloCo tasks and к: 2 for image-
16

---

Published as a conference paper at ICLR 2025
corresponding neurons. Weight clipping is well-established, straightforward, and generic imple- mentation technique (that operates independently of core algorithm design and specific algorithm choices, rather than new algorithmic contribution here), which is a standard practice that ap- ply consistently across all implemented methods. Weight clipping is activated after the network initialization and operates throughout the training process. Additionally, malization after the hidden layers of both the critic and actor networks to establish the LN-TD3 baseline for comparison. Notably, other methods do not incorporate normalization mechanisms to ensure fair comparison. We reproduce Plasticity Injection Following the guidelines from the of- ficial paper (Nikishin et al., 2024). We utilize the default settings of the official ReDo codebase for our comparative analysis https:/ /g/thub..b..... com/ /oon/goolleeeeo e/ /doamii ne/ rteeeeee/e/eeeeeeeeeeeee peeemmmater//// dopami i ne/1 /1 labs/ redo. For Reset-TD3, after a lot of empirical debugging, we found that pa- rameter reset every 90, 000 step is good setting, and fixed it as a hyperparameter. We suggest using seed 5 to reproduce the learning curve in Figure 5.
introduce layer nor-
Hyperparamttr
NE-TD>3
Reset-TD3
ReDo-TD3
LN-TD3
PL-TD3
Actor Learning Rate
le-
1e-4
le-
le-1
le-d
1e
Critic Learning Rate
le
1e-3
le
3e"
3e
1r-3
Represcutation Model Learning Rate
None
None
None
le
5e
5e 3
Discount Factor
0.99
0.99
0.99
0.99
0.99
0.99
Batch Size
128
128
128
128
128
128
Buffer Size
1e6
1e6
1e6
1e6
1e6
1e6
Table A comparison of common hyperparameter choices of algorithms. We use 'None" to denote the 'not applicable' situation.
DrQ We use DrQ as the backbone to verify methods DeepMind Control tasks. And All the methods use the same setting. The details can be seen in Tab.3. Notably, For Reset-DrQ, we only reset the last three MLP layers which is suggested by their paper.
Hyperparameter
NE-DiQ
Reset-DeQ
ReDo-DiQ
LN-DiQ
PL-DeQ
Replay buffer capacity
1e6
le6
le6
1e6
1e6
1e6
Action repeat Seed frames Exploration steps
2
4000
4000
4000
4000
4000
4000
2000
2000
2000
2000
2000
2000
step returns
3
Mini-batch size Discount 7
256
256
256
256
256
256
0.99
0.99
0.99
0.99
0.99
0.99
Optimizer
Adam
Adam
Adam
Adam
Adum
Adum
Learinng rate
le 4
le 4
le 4
lc 4
le 4
le
4
Agent update frequency
2
2
Critic Q-function soft-update rate
0.01
0.01
0.01
0.01
0.01
0.01
Features dim. Hidden dim.
50
50
50
50
50
50
1024
1024
1024
1024
1024
1024
Table 3: default set of hyper-parameter used in our DrQ based methods.
NE For Humanoid and Ant tasks, we set grow interval AT = 25000, grow number k = 0.01 * rest capacity, Prune upper bond 0.4, ending step is the max training step, the threshold of ER is 0.35 and the decay weight a = 0.02(which is used in all the tasks). For other OpenAl Mujoco tasks, we set grow interval AT 20000, grow number k 0.15 rest capacity, Prune upper bond w = 0.2, ending step is the max training step, the threshold of ER is 0.25. Notably. as for the continuous adaptation tasks, we set AT 50000 to ensure the topology can continue growing during the long-term training, and other parameters are same as Humanoid. When we combine our method with DrQ, We use single set of hyperparameters to apply on all four tasks. AT' is set as 20000 to ensure the network can grow quickly to handle the complex input. The grow number k is 0.02 * rest capacity for critic and 0.01 rest capacity for actor. The Prune upper bond w 0.23 and the threshold of ER is 0.18. We set ER step C' to 450 for Mujoco tasks and (150, 250} is work on DMC. Additional optimizer resets were eliminated for all methods to ensure the faimess of the experiment.
17

---

Published as a conference paper at ICLR 2025
D PSEUDOCODE CODE
D.1 PSEUDO-CODE FOR NE
Algorithm Neuroplastic Expansion TD3
Te: All parameters in actor. Q0 All parameters in critics, My: Sparse mask in layer 1. Initial sparse rate Sp. Set parameter x. and the clip bounds for all layers: {6f - Ma.
Vnf
{of = Vig Mi for weight clipping operation, where yni denotes the number of the neurons in
layer L.
Initialize the sparse networks for stable starting
keep ( ('E() dense; 0, +- Erdos- (edos-)enyy(Sp)
Neuroplastic Expention (every AT)
# Calculate growing number at t step to achieve warm growing < cosine annealing(t, Tend)
for each la E TT. lo E (1.) do
# Select top k weights from candidates
Igrow = ArgToph (1V) LTD) U ArgTopk ag (V[
# Collect the weights related to selected dormant neurons
Ih = [Index (andex(@))L() (#)) = 0} [Index (0)) 0}
prune
Start Iruncate process
Algorithm 2
Get indexes from Horow, pprnee
Generate topology mask map Mi Mis
Update new topology
0r <- Oz O Mig Φi + Φz O Mi,
# Weight clipping for the newly added neurons
pg grou
+ Weight Clipping Ort min -K.S max = ks{)
T Weight Clipping On min = -KSS max = ks?))
end for
Train the RL policy
4- π. (+) (with Gaussian noise)
Observe r and new state s'
Fill D with (s, a, 5')
# Experience review
if random(O, 1) (f then
sample a batch from bottom D
else
sample a batch from total D end if
Update Q8 T based on TD3
18

---

Published as a conference paper at ICLR 2025
D.2 PSEUDO-CODE FOR TRUNCATE PROCESS
Algorithm 2 Truncate Process
1: Input:
2: # Enter the necessary inputs
3: min value: min pruning threshold (set to be 0)
4: T Tprune: the pruning set
5: Ti grow' the growth set # for determining max pruning amount
6: w: a preset ratio in [0, 1) for determining max pruning amount
7: Output: Truncated pruning set
8:
9: max value = w x ggrowl max pruning amount. |- represents the number of the elements in
the set
10: if [T max value then
Randomly remove excess elements from the pruning set T prune
12: end if
13: Return I1 prune
E DETAILED EXPLANATION OF THE GROWTH MECHANISM
We follow the classical sparse network exploration method in (Evei et al., 2020). We all the weights to calculate the gradient backpropagation because we want to find a weight subgraph com- posed of multiple candidates add the current network, that is, the newly added weight subsets jointly obtain high gradient magnitude under the influence of each other, and calculating the weights one by one will ignore the interaction between them and other weights in the subset, resulting in serious errors and high-cost computation. To this end, we compute the full weight to cheaply select the valid subgraphs. However, this evaluation does introduce errors that add low-quality weights, and the reason why this growth framework is effective is that real-time pruning removes the error caused by growth. the future, we will further optimize the growth mechanism to improve accuracy of its evaluation.
F ADDITIONAL EXPERIMENTS
F.1 ANALYSIS OF GROW & PRUNE RATE
We conducted ablation experiments on dormant neuron pruning across two hard tasks in Tab.4. The findings demonstrate the significant and generalized benefits dormant neuron pruning for our framework.
Method
Cartpole Swingup Sparse
Walker Walk
w/ prune
425.09 + 45.28 382. 14.28 + 29.86
491.28 + 40.62
w/o prune
367.51.24 + 43.06
Table Performance (Average of 10 runs).
F.2 ANALYSIS OF GROW RRUNE RATE
In table.5, we set experiments for growth rate and pruning rate analyses. Our method shows good performance on both image-based and state-based tasks with a very slow growth schedule, i.c. 0.001 ~ 0.009, but it is not sensitive pruning rate, i.c., keeping it between 10% and 40%
Task
0.0005
0,00010
0.00050
0,00100
0.00150
state-input (Ant)
7529.1 132.36 36
7749 25 127.7
7714.28
147.56
7308. 17 1 261.95
7287.39 39 = 381.02
image-inpot (DMC Reacher Hard)
416.
00.37
42.31
56.42
475.46
34.11
431.74 52.09
419.38 37.24
Table 5: The evaluations of growth rate on vector-input and image-input tasks, Average of runs.
19

---

Published as a conference paper at ICLR 2025
F.3 ANALYSIS OF STARTING SPARSITY RATE
We empirically find that starting from around 0.25 is a general and effective setting for growth. For the cycle adaptation task we find that starting from 0.15 is a good choice. the future, we will explore how to find the appropriate sparsity adaptively according to the task.
We uniformly sample different starting sparsity rates and test the performance. In table.6, we find that the effect is similar when the sparsity ratio is between [0.7, 0.8]. too sparse network cannot learn efficiently in the early stage of training, and too dense network will limit the gain brought by network growth. Therefore, we set it to 0.75 uniformly.
Tusk
0.85
0.75
0.7
0.65
state-input (Ant)
0962.52 297 .1
7694.37 + 115.28
7749.25 + 127.73
7681.1 492.13
108.07
7255.34
212.86 432.67
imuge input (DMC Reacher Hand)
425.95
27. 62
480.73 1. 35
475.46 1 34.11
22.53
29.17
Table 6: The evaluations of initial sparsity vector-input and image-input tasks. Average of 3 runs.
F.4 VERIFY THE EFFICIENCY OF EXPERIENCE REPLAY MODULE
The experience replay module related to reducing forgetting in the later training stage and en- hancing stability-plastiiii ability. The experiment depicted in Fig.7a illustrates that frequent alter- ations to the network topology can induce sudden shifts in network computations, which may prune neurons holding valuable information, leading to instability in subsequent training phases. The tra- jectory visualization (Figure 12) demonstrates that the agent may exhibit suboptimal behavior by forgetting the initial skill (standing up). Consequently, we introduce the experience replay mech- anism during the later stages of training to prompt the network to revisit earlier data, mitigating forgetting and fostering stability in the latter training phases.
To further authenticate the efficacy of ER across a spectrum of tasks, we focus on four challenging image input tasks: Reacher Hard and Walker Walk in DMC, along with two sparse reward manipula- tor control tasks: Hammer (sparse) and Sweep Into (sparse). The results in Table.7 shows enhanced performance and notably reduced variance following the integration of the experience replay mech- anism. This outcome underscores that experience replay fosters training stability.
Method
Reacher Hard
Walker Walk
Hammer sparse (success rate)
Sweep Into sparse (success rate)
w/ NE w/o ER
475. 46 34.11
491.28
40.62
0.54 1.133
0.61 0.09
419.25 = 65.73
427.31
73.81
0.47 0.21
0.52 + 0.16
Table 7: The evaluations of initial sparsity vector-input and image-input tasks. Average of runs.
In addition, set study of experience review (ER) various tasks in Tab,8-10, and the results show that without ER, although all the agents with different seeds can learn efficiently in the early stage, the performance will drop at the later training stage and become unstable in some runs(high std). We also find that this phenomenon is more pronounced on complex sparse reward tasks as well as image input scenarios. Therefore we suggest employing ER in complex tasks to assist our method for stable training.
Method
at 1/2 training stage
at 3/4 training stage
end of training
NE w/ ER
6635. 12 + 245.28 6673.54 + 216.41
7258.18 + 151.66 6896.92 = 264.23
7749. 25 + 127.73 773 7235.14 + 493.83
NE w/o ER
Table 8: Performance on vector-input task (Ant) (Average of 10 seeds).
F.5 ANALYSIS OF GROW MODE
The choice of the control mode for topology growth is empirical, and we have tested several modes in the experimental phase, ie uniform schedule, warm decay, and cosine annealing. We show the comparison in Table 11, the results show that although all modes are efficient, cosine annealing performs the best. We guess that this is because it includes cyclicality based on warm decay, which more robust in practice.
20

---

Published as a conference paper at ICLR 2025
Method
at 1/2 training stage
at 3/4 training stage
end of training
NE w/ ER
408. 22 + 74.14 14 413.75 + 79.22
462.09 + 48.74 439.02 + 86.15
475.46 + 34.
NE w/o ER
419.25 + 65.73 73
Table 9; Performance image-input task (Reacher Hard) (Average of 10 seeds).
Method
at 1/2 training stage
at 3/4 training stage
end of training
NE w/ ER
396.69 696.69 + 83.27 397.15 = 79.22
416.39 + 55.26 421.02 = 61.08
424.76 + 47 .68 402.88 + 71.23
NE w/o ER
Table 10: Performance on sparse reward task (Cartpole Swingup (Aerse) of 10 seeds).
Mode
Cartpole Swingup Sparse
Cosine annealing
425.09 + 45.28
Uniform
389.66 + 41.53
warm decay
392.24 + 31.95
Table 11: The evaluations of growth mode on vector-input and image-input tasks. Average of runs.
F.6 VISUALIZATION OF THE NETWORKS WITH Low ACTIVATED NEURON RATIO
When the activated neurons ratio low, means the networks contain lots of dormant (dead) neurons that retain useless (negative) weights. In this section, we add two visualizations to demonstrate the detrimental effects of accumulating lots of dormant neurons in a network: it diminishes the network's ' overall representational capacity during training and leads to suboptimal policy.
We contrasted the backpropagation gradient heatmaps standard network with those of network containing numerous dormant neurons (Figure 11), where darker colors indicate higher gradients and faster weight optimization. Both networks use the same data for input. The findings indicate that when network has substantial number of dormant neurons, the gradient guidance is notably diminished, thereby constraining the learning efficacy. This limitation stems from dormant neurons being unable trigger the activation function, rendering them unseen the computational graph and impeding the backflow gradients.
We compare the behaviors of the regular policy and the policy featuring numerous dormant neurons in Figure 12, it becomes evident that the latter tends to exhibit suboptimal behavior due to its reduced learning capacity.
F.7 ABLATION STUDY OF REPLAY RATIO
We test the effect of different Replay Ratio (RR) on all four DMC tasks. The results in Figure 13 show that with the increase of RR, the performance of baseline and NE increased slightly (the effect of our method was more obvious). We concluded that this was because changing RR proved to be an effective way to alleviate plasticity loss in VRL (Ma et al., 2024).
21

---

Published as a conference paper at ICLR 2025
%
14
12
10
8
6
-4
-2
Unit
Unit
Layer-1
Layer-2
Layer-3
Layer-1
Layer-2
Layer-3
Layer
Layer
Figure 11: Heat map. Darker colors indicate higher gradients and faster weight optimization. Both networks use the same data for input.
Dormant neuron: Although moving, fail to leam the desired running skills
Activated neuron: Achieve smooth running
Behavior after Policy training
Figure 12: Comparison of behaviors after training.
F.8 DETAIL RESULTS DIFFERENT NETWORK SIZE
Figure 14 shows the results for uniform fineggrained size sampling.
22

---

Published as a conference paper at ICLR 2025
100
Normalized Performance(
95
90
85
DrQ (baseline)
80
EN-DrQ
0.5
1
2
4
Figure 13: Analysis different Replay Ratios values.
97.5
95.0
Normalized Performance(%)
92.5
90.0
87.5
85.0
82.5
Static Random Grow
80.0
Ours
77.5
1.0x
1.5x
2.0x
2.5x
3.0x
3.5x
Figure 14: Performance comparison for different final model sizes.
23


Reviews:
[{"review_id": "ewwnHMbTZZ", "reviewer": "Reviewer_LFqg", "summary": "This paper introduces a novel idea, Neuroplastic Expansion (NE), to address the problem of plasticity loss in reinforcement learning (RL). The paper is well-written and presents the concept clearly. However, there are some concerns, particularly regarding its contribution relative to existing work. If these concerns can be resolved, I would consider improving the rating from 5 to 6.", "questions": "1. Previous work, such as Plasticity Injection[1], has already proposed dynamically growing an agent's network, please provide a detailed comparison.\n\n2.Dynamically expanding the size of a neural network could potentially lead to policy instability. For instance, the policy before and after expansion might be inconsistent. However, the results reported in Figure 6 appear very stable. Please provide specific analyses and ablation studies demonstrating how NE maintains policy stability during network expansion. \n\n3.It would be helpful if the authors could provide experiments or analyses to explain the impact of dead neurons during training. Do these neurons store explored knowledge that contributes positively to the learning process, or do they have a negative effect on training? Please provide some analyses and visualizations to illustrate their impact on the learning process.\n\n[1]Deep Reinforcement Learning with Plasticity Injection.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The paper is well-written.\n\n2.The concept of Neuroplastic Expansion (NE) is well-motivated.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y29IZSarTZ", "reviewer": "Reviewer_p17i", "summary": "The paper introduces a new approach to maintaining plasticity for deep reinforcement learning methods based on intuitions about cortical cortex expansion in cognitive science. The approach includes three components: 1) neuron regeneration, 2) dormant neuron pruning, and experience review. While neuron regeneration and dormant neuron pruning parts help maintain plasticity, the experience review reduces instability due to high plasticity. The authors test the effectiveness of their approach and its components in various RL environments and compare it against other baselines.", "questions": "1. Why is it called “Elastic” Neuron Generation? What is elastic exactly here?\n2. I’m not sure how experience review reduces learning instability. I’m not convinced by the revisiting-is-useful argument. Without experience review, sampling is iid, so temporally old samples are still revisited. Why is revisiting old samples with higher probability useful, particularly when the dormant neuron ratio is high?\n3. To get the gradients to decide which weight to regenerate, you need to use the fully expanded network and backpropagate everything, then find the top k weights that do not exist in the actual network. Is that correct? If so, then this metric is inaccurate because it adds all weights that take part in backpropagation. The accurate way is to add one weight at a time and backpropagate gradients each time. This is, of course, very expensive since you need to have $N$ additional forward and backward passes. In contrast, the process you described makes some approximations that are not clearly presented. \n4. Are both pruning and regeneration neuron/unit-based?\n5. In section 5.2, the environments have different action spaces; how did you handle that?\n6. The authors stated that resetting was deemed the most effective approach. But no references are given (line 467).\n7. In Figure 8, why are there spikes in the activated neurons ratio?\n8. What is meant by removing the difference in fitting ability in lines 217-218?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present a novel approach that improves plasticity for deep reinforcement learning methods. The approach seems effective and achieves better performance than many existing methods, such as layer normalization, ReDo, and plasticity injection in many environments. The authors provided an extensive experimental study of their method in different environments (MuJoCO Gym and DMC) and with different learning algorithms (DrQ and TD3).", "weaknesses": "- The paper significantly lacks mathematical rigor. Here are some examples that are representative of these inaccuracies, although they don’t constitute an exhaustive list:\n\n  - It should be $\\breve{\\theta} \\subset \\theta$ not $\\breve{\\theta} \\in \\theta$. Or more precisely, $\\breve{\\theta}_l \\subset \\theta_l, \\forall l \\in \\\\{1,...,N\\\\}$, where $N$ is the number of layers in the network.\n\n  - In line 213, $\\mathbb{I_{grow}}$ is not defined well. It should be a list, but you assign it with two random quantities added together so it looks like a vector or a scaler instead of a set. Additionally, how is the random function defined? The random function should output a set, which you then need to union with the other set, $\\mathbb{I_{grow}}= RandomSet1 \\cup RandomSet2 $ not $\\mathbb{I_{grow}}= RandomSet1 + RandomSet2$. A complete, rigorous mathematical description is expected.\n  - It should be $ArgTopK$ not $TopK$ in equation 2 and line 256\n  - what does this mean to write $\\mathbb{I_{prune}}= f(\\theta_i) \\leq 0$? The left side should be a list, and the right-hand side should be an inequality. It should be something like $\\mathbb{I_{prune}} = \\\\{\\texttt{index}(\\theta_i) | f(\\theta_i) \\leq 0 \\\\}$.\n  - what does it mean to have a dormant ratio of negative in line 301? The ratio possible values are in $[0,1]$.\n- The paper presentation and writing are not clear. \n  - The authors claim NE maintains a high level of elastic neurons (see line 60), but no definition of what elastic neuron is given. Is elasticity here something different from plasticity? How do we measure either of them?\n  - The term plasticity is loosely used to represent activated neuron ratio (e.g., Figure 6f). A clear definition of what plasticity means should be presented. If plasticity is the activated neuron ratio, then the paper's approach does not actually address the loss of plasticity as claimed since in all figures where the activated neuron ratio is presented, we see a decrease in their percentage with the paper's approach, similar to other methods. \n  - The algorithm is not complete. For example, the cosine annealing scheduler is missing, and the experience review part is not clearly shown. Additionally, Algorithm 1 works on the weight level, but the description from the text talks about neuron-level regeneration and pruning. The algorithm needs to reflect that.\n  - Since the authors depend on the sparse network training framework as part of their approach. They should fully explain what the sparse network training framework is in writing and in the algorithm.\n  - The process of experience review is not clear. The fluctuation rate of dormant neurons $\\nabla f$ is a function of each unit, but the authors talk about some aggregate quantity. Is that new quantity a summation of all units in the network, $\\nabla f = \\sum_i f_i$? Why isn’t this part of the algorithm?\n  - Since the authors chose to rely on the activated neuron ratio (1-dormant neuron ratio), equation 1 needs to reflect that, currently, the definition is mentioned in-text, whereas it should be highlighted in equation 1 instead of dormant neuron ratio, which the authors do not really use.\n- Some claims in the paper are not supported by evidence.\n  - The paper overclaims what their approach can address. The authors mention that their approach mitigates loss of plasticity primacy bias, reduces catastrophic forgetting, and strikes a stability-plasticity balance. Most of these claims are not supported by evidence. Using those terms loosely without being precise about what is being studied in an experiment makes the paper hard to navigate.\n  - For example: “topology growth can effectively alleviate neuron deactivation and thus maintain the ability of policy learning to mitigate the loss of plasticity and alleviate the primacy bias.”--- It's unclear how the experiment shows loss of plasticity or primacy bias mitigation. The authors should instead only claim that their approach reduces the dormant neuron ratio and not claim anything about loss of plasticity or primacy bias.\n  - The current ablation is not sufficient. Ideally, the authors should remove each component of the system: 1) neuron regeneration, 2) experience review, and 3) dormant neuron pruning. The authors did 1 and 2 but not 3. We need to know what happens if we remove dormant neuron pruning.\n- Issues in empirical evaluation:\n  - Many figures do not have labels on the axes, so it is hard to know (even after careful investigation) what is being varied. For example, the x-axis in Figure 5 has no label, and I don’t know what 0 to 3 means here. Other examples include but are not limited to Figure 2 (missing x-axis label), Figure 4 (what is the score in the y-axis), and Figure 6 (missing y-axis label).\n  - The results are not statistically significant. A very low number of independent runs (7 runs) are used, and they have overlapping error bars in most figures. More independent runs are needed, especially since the error bars are overlapping. I suggest the authors run each algorithm for 30 independent runs in all of their experiments.\n  - In section 5.2, a fixed number of episodes is used in each task, whereas a fixed number of steps should be used to have consistent amount of experience in each task. \n\n**Minor issues:**\n\n- The author defines the gradient as $L_t$. Then the sentence after that says it’s $\\nabla L_t$. \n- The name of the approach is not very representative of what the algorithm does. It’s called neuroplastic expansion, emphasizing the expansion part. A better name, such as neuroplastic regeneration and pruning, can be more representative and accurate.\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\nOverall, I believe this paper could serve as a good algorithmic contribution to the community if the authors addressed my concerns based on this feedback. So, I’m willing to increase my score given that 1) the authors tuned down the claims and made them modest such that they accurately reflect what is being studied by their experiments, 2) the authors fixed all mathematical inaccuracies and provided a completed algorithm, 3) the terminologies are used carefully precisely instead of loosely, and 4) the empirical work is improved through more independent runs and improved figures.\n\\\n\\\n\\\n\\\n**Update:** The authors worked hard to address my concerns. It was a fruitful back-and-forth discussion that improved the paper's quality immensely. Since my concerns have been addressed, I'm happy to recommend acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DEQIwT8wYe", "reviewer": "Reviewer_df5D", "summary": "This paper presents Neuroplastic Expansion (NE), a generally applicable training scheme designed to mitigate plasticity loss in RL. NE comprises of three components:\n\n1. Adding neuron connections based on potential gradients (elastic neuron generation)\n2. Pruning neuron connections based on dormant ratio (dormant neuron pruning)\n3. A training batch sampling scheme that focuses on early samples depending on dormant ratio fluctuation (neuron consolidation)\n\nCompared to prior methods such as Reset, ReDo and Plasticity Injection, NE showed superior performance in state-based Mujoco tasks (with TD3) and several pixel-based DMC tasks (with DrQ). NE was also able to maintain plasticity while sequentially training through multiple environments in a cyclic manner. Its plasticity — measured by dormant ratio — is well preserved in the majority of the experiments, proving its effectiveness in maintaining trainability and preventing loss of plasticity.", "questions": "1. In Figure 5, at which point does the model reach its maximum capacity (i.e., no more room for growing unless pruned)?\n2. In Figure 6, why does Plasticity Injection fail even before injecting in Walker2d and HalfCheetah? Before injection, shouldn’t they be equivalent to vanilla TD3?\n3. In the 'Dormant Neuron Pruning' section, the expression $Clip(a,b,c)$ is confusing to read without any definition.\n4. The dynamic threshold defined in paragraph in 'Neuron Consolidation' as a whole is too confusing to read. I don't think $\\nabla f(\\theta)$ is the right definition, since it's not a derivative w.r.t. the dormant ratio, but rather an average change rate of the dormant neuron count. Another thing I want to make sure is that  $\\nabla f(\\theta)$ is used as the dynamic threshold $\\epsilon$, right? It's not clearly stated in the text.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The core idea of NE seems very promising in terms of lifelong learning: add new capacity to learn new information, remove useless/dead neurons, and prevent catastrophic forgetting. Connection to biology is also a big plus.\n2. The necessity for each component was well explained (Figure 2,3,4). I found it especially interesting to see a proof of catastrophic forgetting in a Mujoco task.", "weaknesses": "1. The writing is sometimes not detailed enough and causes confusion (see questions and weaknesses below).\n2. Some crucial design choices are not well justified and/or validated.\n    1. Neuron consolidation is proposed to prevent catastrophic forgetting, which often occurs late stage (as shown in Figure 4). However, the dynamic threshold they use to control the strength of consolidation plateaus to its lowest value (strongest consolidation) even before halfway through the training process (Figure 5). This discrepancy raises question on whether this complexity is really necessary, especially since a simple time-dependent scheduling scheme could also fit the justification of ‘not forgetting early state-action distribution’.\n    2. The amount of pruned dormant neurons are forced to be less than amount of added connections in order to guarantee that the network is increased in size. This also looks like an unnecessary detail since we can achieve the same by using ReDo and then growing a small number of neurons. I think there should be an explanation on why this design choice is essential.\n    3. Although less critical, some components of elastic neuron generation also needs more careful consideration, such as the cosine annealing schedule (especially since RigL [1] was not primarily designed for continual learning).\n3. The experimental setup needs more refinement.\n    - For the main experiments, although it’s convincing that NE surpasses prior works in Mujoco and DMC tasks, it would be nice to see whether NE is also effective in more challenging environments.\n    - The recently proposed NaP [2] is an extremely competitive method for continual learning, and I think it’s a crucial baseline in the main experiment.\n    - The cycling Mujoco experiment (Figure 9) is plotted on ‘episodes’ (and is the only one). This is problematic, since different lengths of episodes would result in different number of update steps and thus varying degree of plasticity loss.\n    - It would have been nice to see whether NE can synergize with other methods such as CReLU or LayerNorm.\n\nOverall, I think this paper needs more improvements before it can be published. However, I am also ready to change my scores if some of the above concerns are refuted/addressed.\n\n[1] Rigging the Lottery: Making All Tickets Winners., Evci et al., ICML 2020.\n\n[2] Normalization and effective learning rates in reinforcement learning., Lyle et al., arXiv.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EJLhW1leL4", "reviewer": "Reviewer_46hD", "summary": "This paper addresses the critical issue of plasticity loss in deep reinforcement learning (deep RL), where agents' adaptability decreases over time, hindering continuous learning in dynamic environments. Inspired by biological neural networks, the authors propose Neuroplastic Expansion (NE), a novel mechanism that dynamically enlarges the neural network by adding elastic neurons based on gradient potential. NE maintains high plasticity by regenerating and recycling dormant neurons, effectively mitigating the plasticity-stability dilemma inherent in deep RL.", "questions": "* How is the network reinitialized based on the growth criterion? Is it initialized with random weights similar to the initial stage?\n* If reinitialization involves random weights, how does this approach effectively reduce dormant neurons, especially considering that large feature values after extended training might lead to immediate pruning of reinitialized neurons.\n* Is the Experience Review technique effective across all experimental setups, or is its efficacy primarily validated only in specific environments like HalfCheetah? \n* PThe performance curves indicate that the dynamic actor maintains a stable plasticity rate similar to the static one. Why does the dynamic actor perform better despite this similarity in plasticity rates?\n* For a more comprehensive evaluation, should the authors include additional baselines such as the base TD3 and TD3 with only network growth (without pruning) to isolate the effects of different components of NE.\n* What constitutes a valid starting sparsity rate for NE?\n* What are the optimal rates for growth and pruning, and how do these rates influence overall performance? An analysis of hyperparameter sensitivity would provide deeper insights into NE's robustness and adaptability.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* **Novelty:** The paper introduces a novel idea inspired by human brain mechanisms, specifically cortical expansion, to address plasticity loss in deep RL. This biologically motivated approach offers a novel perspective for significant advancements in continual learning for artificial agents.\n* **Good architectural design:** Neuroplastic Expansion (NE) is meticulously designed to balance network growth with resource efficiency. By adding elastic neurons based on gradient potential and recycling dormant neurons, NE maintains network expressivity and adaptability without causing uncontrolled growth in network size.", "weaknesses": "* **Missing Relevant Work:**\n    * Plasticity-Related Studies: The paper overlooks several relevant studies on neural network plasticity ([1]-[5]).\n    * Cortical Expansion Citations: The discussion on cortical cortex expansion cites works that describe patterns of cortical expansion. I think the authors miss foundational studies that first identified evidence of cortical expansion.\n* **Experimental Setup**:\n    * The evaluation of MuJoCo environments is limited to the TD3 algorithm, which is considered outdated. Assessing NE using more recent and robust algorithms such as TD7, TD-MPC2, or BRO would enhance the relevance and robustness of the findings.\n    * Other than Mujoco, I think it is beneficial to test in the state-based DMC, maybe by trying to compare with reset-based methods under identical experimental configurations as primacy bias paper. \n\n[1] On warm-starting neural network training., Ash et al, 2020. \n\n[2] A study on the plasticity of neural networks., Berariu et al, 2021. \n\n[3] PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning., Lee et al, 2023. \n\n[4] Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks., Lee et al, 2024. \n\n[5] Normalization and effective learning rates in reinforcement learning., Lyle et al, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)