Neuroplastic Expansion in Deep Reinforcement Learning

## Abstract

Abstract The loss of plasticity in learning agents, analogous to the solidification of neural pathways in biological brains, significantly impedes learning and adaptation in reinforcement learning due to its non-stationary nature. To address this fundamental challenge, we propose a novel approach, Neuroplastic Expansion (NE), inspired by cortical expansion in cognitive science. NE maintains learnability and adaptability throughout the entire training process by dynamically growing the network from a smaller initial size to its full dimension. Our method is designed with three key components: ( 1 ) elastic neuron generation based on potential gradients, ( 2 ) dormant neuron pruning to optimize network expressivity, and ( 3 ) neuron consolidation via experience review to strike a balance in the plasticity-stability dilemma.
Extensive experiments demonstrate that NE effectively mitigates plasticity loss and outperforms state-of-the-art methods across various tasks in MuJoCo and DeepMind Control Suite environments. NE enables more adaptive learning in complex, dynamic environments, which represents a crucial step towards transitioning deep reinforcement learning from static, one-time training paradigms to more flexible, continually adapting models.

## 1 Introduction

In the realm of neuroscience, it has been observed that biological agents often experience a diminishing ability to adapt over time, analogous to the gradual solidification of neural pathways in the brain (Livingston, ). This phenomenon, typically known as the loss of plasticity (Mateos-Aparicio and Rodríguez-Moreno, ), significantly affects an agent’s capacity to learn continually, especially when agents learn by trial and error in deep reinforcement learning (deep RL) due to the non-stationarity nature.
The declining adaptability throughout the learning process can severely hinder the agent’s ability to effectively learn and respond to complex or non-stationary scenarios (Abbas et al., ).
This limitation presents a fundamental obstacle to achieving sustained learning and adaptability in artificial agents, which echoes the plasticity-stability dilemma (Abraham and Robins, ) observed in biological neural networks.

There have been several recent studies highlighting a significant loss of plasticity in deep RL (Kumar et al., , Lyle et al., ), which substantially restricts the agent’s ability to learn from subsequent experiences (Lyle et al., , Ma et al., ). The identification of primacy bias (Nikishin et al., ) further illustrates how agents may become overfitted to early experiences, which inhibits learning from subsequent new data.
The consequences of plasticity loss further impede deep RL in continual learning scenarios, where the agent struggles to sequentially learn across a series of different tasks (Dohare et al., ).

Research on addressing plasticity loss in deep RL is still in its early stages, with recent approaches including parameter resetting (Nikishin et al., , Sokar et al., ) (or its advancement with random head copies (Nikishin et al., )), and several implementation-level techniques like normalization, activation functions, weight clipping, and batch size adjustments (Obando Ceron et al., , Nauman et al., , Elsayed et al., ).
However, reset-based methods often lead to performance instability and training inefficiency due to the need for period re-training, while implementation-level modifications lack generalizability and do not directly target the plasticity issue.
The field currently lacks a unified methodology that can effectively address plasticity loss while maintaining training stability and efficiency across varying environments.

Humans adapt to environmental changes and novel experiences through cortical cortex expansion in cognitive science (Hill et al., ).
This process involves the gradual activation of additional neurons and the formation of new connections to facilitate the ability to learn continually.
Drawing inspiration from this biological mechanism, we propose a novel perspective – Neuroplastic Expansion, which can help maintain plasticity in deep RL.
The key insight is that an agent starting learning with a smaller network and dynamically growing to a larger size (ultimately reaching the original static network dimension) can
effectively tackle plasticity loss by maintaining a high level of elastic neurons throughout training (Figure 1).
We first provide empirical evidence in (§4.1) validating its potential for mitigating plasticity loss and improving final performance in certain cases, even with a naive incremental expansion.

Figure: Figure 1: High-level illustration of Neuroplastic Expansion RL. The network regenerates elastic neurons based on gradient potential, recycles dormant neurons, and undergoes progressive topology growth to mitigate plasticity loss. The agent consolidates neurons through experience review, preserving prior helpful knowledge and ensuring policy stability.
Refer to caption: extracted/5916941/figs/framework.png

Building upon this insight, we systematically introduce Neuroplastic Expansion (NE), a simple yet effective mechanism to maximize the benefits of incremental growth training (§4.2). (NE) adds high-quality elastic candidates based on potential gradients (Evci et al., ).
However, this will lead to increased computation cost for always enlarging the network, and cannot fully leverage its expressivity as neurons may turn dormant during the course of training.
To address this challenge, (NE) introduces a
reactivation process for dormant neurons, pruning them and potentially reintroducing them as candidates in subsequent growth stages, thereby better utilizing the network expressivity and enhancing the policy’s sustainable learning ability.
Unlike previous reset-based approaches (either rest last layers or introducing copies of heads or reinitializing some parts in the network) that risk forgetting and require periodic re-training, NE maintains learning continuity in a smoother way.
To further mitigate potential instability from continuous topology adjustments, we introduce an effective Experience Review technique for consolidating neurons by adaptively reviewing prior knowledge during the later training stages to uphold policy stability, striking a balance in the stability-plasticity dilemma.
Extensive experiments across standard and continual RL tasks demonstrate its effectiveness in various scenarios, showcasing its ability to maintain plasticity while ensuring policy stability.

The main contributions are summarized as follows:

- •
We introduce a novel mechanism, Neuroplastic Expansion (NE), to mitigate the plasticity loss in deep RL.
- •
We develop effective elastic neuron generation and dormant neuron pruning mechanism for better network capacity utilization, and a neuron consolidation technique for preventing forgetting helpful reusable knowledge.
- •
We conduct extensive experiments in standard RL tasks, which demonstrate the effectiveness of NE across various tasks including MuJoCo (Todorov et al., ) and DeepMind Control Suite (DMC) (Tassa et al., )
tasks that outperform previous strong baselines by a large margin, and can be effectively adapted to continual learning scenarios.

## 2 Background

#### Deep Reinforcement Learning

The reinforcement learning problem can be typically formulated by a Markov decision process (MDP) represented as a tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$, with $\mathcal{S}$ denoting the state space, $\mathcal{A}$ the action space, $\mathcal{P}$ the transition dynamics: $\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,1]$, $\mathcal{R}$ the reward function: $\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, and $\gamma\in[0,1)$ the discount factor.
The agent interacts with the unknown environment with its policy $\pi$, which is a mapping from states to actions, and aims to learn an optimal policy that maximizes the expected discounted long-term reward.
The state-action value of $s$ and $a$ under policy $\pi$ is defined as $Q^{\pi}(s,a)=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(s_{t},a
_{t})|s_{0}=s,a_{0}=a]$.
In actor-critic methods, the actor $\pi_{\phi}$ and the critic $Q_{\theta}$ are represented using neural networks as function approximators with parameters $\phi$ and $\theta$ (Fujimoto et al., , Haarnoja et al., ).
The critic network is updated by minimizing the temporal difference loss, i.e., $\mathcal{L}_{Q}(\theta)=\mathbb{E}_{D}\big{[}(Q_{\theta}(s,a)-Q^{\mathcal{T}}_
{\theta}(s,a))^{2}\big{]}$, where $Q^{\mathcal{T}}(s,a)$ denotes the bootstrapping target $\mathcal{R}(s,a)+\gamma Q_{\bar{\theta}}(s^{\prime},\pi_{\bar{\phi}}(s^{\prime
}))$ computed using target network parameterized by $\bar{\phi}$ and $\bar{\theta}$  based on data sampled from a replay buffer $D$.
The actor network $\phi$ is typically updated to maximize the Q-function approximation according to $\nabla_{\phi}J(\phi)=\mathbb{E}_{D}\left[\left.\nabla_{a}Q_{\theta}(s,a)\right
|_{a=\pi_{\phi}(s)}\nabla_{\phi}\pi_{\phi}(s)\right]$.

#### Dormant Neuron & Activated Neuron

The concepts of dormant and activated neurons are crucial in assessing network plasticity.
Dormant neurons are defined as those whose output falls below a certain threshold $\tau$ in the neural network (Xu et al., ), providing a quantitative measure of network plasticity. Formally, a neuron $i$ in layer $l$ is considered $\tau$-dormant when $f(l_{i})\leq\tau$ (Sokar et al., ), where

$$ $f(l_{i})=\frac{\mathbb{E}_{x\in Id}|h^{l}_{i}(x)|}{\frac{1}{H^{l}}\sum_{k\in h }\mathbb{E}_{x\in Id}|h^{l}_{k}(x)|},$ (1) $$

with $Id$ denoting the input distribution. The activated neuron ratio, defined as the complement of the dormant neuron ratio (i.e., $1-$ dormant neuron ratio), serves as an alternative metric for plasticity.
Activated neuron ratio is widely used in plasticity visualization (Ma et al., ) due to its intuitive interpretation (higher values correspond to better performance).

## 3 Related Work

#### Plasticity loss in RL

Recent studies have found that agents guided by the non-stationary objective characteristic of RL suffer catastrophic degradation (Lyle et al., ) and ultimately reduce the ability to perform continual learning. This phenomenon is known as plasticity loss. Behaviorally, the loss of plasticity during training can manifest as, for instance, an inability to obtain an effective gradient to guide policy adaptation in fine-tuning (Abbas et al., ), limiting potential transfer performance.
While the research on this topic is still in its primary stages, there are several techniques that have been demonstrated to mitigate the loss of plasticity in deep RL. Lyle et al. () empirically find that network normalization particularly layer regularization (LN), is very effective in mitigating plasticity loss. Recently, Nikishin et al. () proposed a plug-in module that can be used to improve continual learning ability and as a metric of plasticity evaluation, and introduced copies of random heads for injecting plasticity. In pixel-based deep RL, data augmentation (Ma et al., ) and batch size reduction (Obando Ceron et al., ) have been analyzed in reducing plasticity loss.

As highlighted by (Abbas et al., ), plasticity loss diminishes a network’s ability to effectively minimize training loss on new data (trainability) and reduces its capacity to generalize to unseen data (generalizability). In the former case, Nikishin et al. () showed that deep RL agents overfit early experiences, which limits their ability to learn with new experiences later on. Consequently, the agent performs suboptimally in long-term learning. This effect is called the primacy bias.
Recent studies have found that techniques focusing on mitigating primacy bias such as parameter resetting (Schwarzer et al., , Kim et al., ), mitigating gradient starvation (Dohare et al., ), and constraining deviations from initial weights (Kumar et al., , Lewandowski et al., ) can also be effective in mitigating plasticity loss.
Notably, the resetting strategy has been shown to work at scale, with applications in LLM fine-tuning (Noukhovitch et al., ). The reason why the two mentioned domains can share a solution is likely that there is a common incentive, that is, the ability of network parameters to learn cannot be maintained by only following the traditional training processes. Thus, this paper proposes a new general framework from the perspective of training mechanisms to simultaneously alleviate the primacy bias and plasticity loss issues in deep RL with a better trade-off between stability and plasticity.

#### Pruning in RL

Sokar et al. () showed that training the deep RL policy with a changing topology is difficult due to training instability. Since then, this challenging topic has been well-studied. Policy Pruning and Shrinking (PoPs) (Livne and Cohen, ) obtain a sparse deep RL agent with iterative parameter pruning. Graesser et al. (), Tan et al. () attempt to train a sparse neural network from scratch without pre-training a dense teacher. Existing methods mainly focus on obtaining a small topology to improve training efficiency. However, our paper explores whether growing the topology from small to large can mitigate the plasticity loss in RL agents rather than training with the maximum number of parameters. In deep RL, the model size commonly utilized is typically small and may encounter policy collapse after scaling up (Schwarzer et al., , Ceron et al., ). Thus, sparse training techniques that have proven beneficial for significantly reducing training costs in fully supervised domains may not yield similar advantages in deep RL in terms of computation cost reduction. Instead, we provide a new perspective and focus on making deep RL agents better use of the growing network size to further improve their learning ability.

## 4 Neuroplastic Expansion RL

In this section, we begin by discussing the insights of Neuroplastic Expansion RL empirically, and analyze its effect on mitigating loss of plasticity in Section 4.1. Then, we systematically present details of our method in Section 4.2.

### 4.1 Illustration of Insights of Dynamic Growing RL

There have been several recent studies investigating loss of plasticity in supervised learning (Lyle et al., , Lewandowski et al., ), which corresponds to the phenomenon where neural networks gradually lose the ability to adapt and learn from new experiences (Lin et al., ). This is further exacerbated in RL (Sokar et al., , Nikishin et al., ), due to the non-stationary nature and the tendency to overfit prior knowledge, resulting in suboptimal policies.
A predominant approach to tackle plasticity loss is based on resetting (Nikishin et al., ), where the agent resets the last few layers of its neural network periodically throughout training. However, although effective, this kind of approach typically forgets helpful and reusable knowledge which is critical for learning, and thus the agent experiences a drop in performance once resetting is applied.

One of the major causes for agents to adapt to novel circumstances is based on the growth of the human cortex (Hill et al., ). Motivated by this insight, we propose a novel perspective that RL agents can maintain high plasticity through dynamic neural expansion mechanisms, sustaining the policy’s continual adaptation ability from new experiences. Initially, data collected by a random policy may be of lower quality and therefore require less network expressivity (Burda et al., , Zhelo et al., ). As the agent improves, it tends to need a more expressive network for fitting complex value estimates and policies. Maintaining neuronal vitality and regeneration during training can significantly mitigate plasticity loss, as it allows for ongoing adaptation to new information.

Figure: Figure 2: Comparison of normalized performance (left) and the ratio of active neurons (right) for vanilla TD3 (Fujimoto et al., ) and its variant with a naive random growth network and our Neuroplastic Expansion method, across varying network capacities. Experiments are conducted with 7 independent seeds.
Refer to caption: x1.png

Motivated by this insight, we propose a novel angle considering Neuroplastic Expansion, where the agent starts with a smaller network that evolves to a larger one throughout the learning process.
We first demonstrate a naive implementation of this idea and will systematically discuss our methodology in Section 4.2.
Concretely, we consider a small-to-large neural expansion approach, where the capacity of initial networks is $20\%$ of the size of a typical full TD3 (Fujimoto et al., ) network (i.e., uniformly selected $20\%$ of neurons from each layer in the full TD3). To validate the potential of this novel perspective, we first consider the simplest manner for growing the network by uniformly adding
$k$ new connections in the current critic topology $\breve{\theta}\in\theta$ and current actor topology $\breve{\phi}\in\phi$ for each layer $l\in N$ and participate in gradient propagation according to
$\mathbb{I}_{grow}^{l}=\text{Random}_{i\notin\breve{\theta}_{l}}(\theta_{l},k)+
\text{Random}_{i\notin\breve{\phi}_{l}}(\phi_{l},k)$.

We build this idea upon TD3 and conduct fair comparisons with vanilla TD3 (i.e., with static actor and critic networks) with different capacities.
We compared the performance of the dynamic growing network with a naive topology growth strategy, with the typical ones using static networks based on TD3 (Fujimoto et al., ) in the HalfCheetah environment. To remove the difference in fitting ability caused by model capacity and capture the impact of gradual topology growth, the number of termination parameters of the topology growth is set to be consistent with the traditional static parameter methods. Therefore, the comparison with the vanilla TD3 algorithm is fair given the same network capacity at the end of training.

From the left part in Figure 2, we observe that even a naive incremental network growth can bring relatively noticeable gains to the agent considering larger network capacity. The right part in Figure 2 demonstrates the ratio of active neurons (Sokar et al., ) for each algorithm during training. This metric is used to evaluate RL agents regarding whether they maintain expressive ability.
As shown, topology growth can effectively alleviate neuron deactivation and thus maintain the ability of policy learning to mitigate the loss of plasticity and alleviate the primacy bias.

### 4.2 Method: Neuroplastic Expansion

Hill et al. () demonstrated that the cerebral cortex undergoes expansion in response to environmental stimuli, learning, and novel experiences. This dynamic growth and reorganization enhances cognitive and decision-making abilities, fostering adaptability to changing situations. Motivated by our preliminary results and inspired by this biological mechanism, we present Neuroplastic Expansion (NE), which trains RL agents with a dynamic expansion network and seeks to maintain the ability to retain learning capacity and adaptability throughout the entire learning cycle.

#### Elastic Neuron Generation

While incremental random topology growth has contributed to maintaining plasticity, we aim to elucidate the underlying mechanisms that drive its effectiveness. We pursue this goal by delving into the dormant neuron theory (Sokar et al., ), a prevalent explanation for plasticity loss currently. This theory posits that neurons within a network become inactive during training, generally measured by their activations going to zero (Lu et al., ). This hinders said neurons’ ability to learn, reducing the network’s overall ability to efficiently process new situations and the outputs gradually diminish, ultimately dropping below the activation function threshold (Lu et al., ). Meanwhile, another line of work revealed that parameters lacking prompt exposure to high-gradient stimuli are prone to slipping into dormancy. Drawing from the aforementioned findings, we derive the following observation:

The naive random topology growth sporadically introduces new connections that can generate significant gradients for neurons nearing dormancy, thereby supplying them with potent signals during backpropagation and continue learning.

If the above conjecture aligns with the underlying principles in deep RL, choosing topology candidates based on the gradient magnitude could alleviate plasticity loss more effectively than randomly expanding the network connections. To this end, we follow the sparse network training framework in Tan et al. (). Prior to each topology expansion, a batch of transitions is sampled from buffer $D$ to compute the magnitude of gradient $L_{t}$ through the whole model parameters, formalized as $|\nabla_{\theta}L|$. Then $k$ connections capable of yielding the highest magnitude are chosen to augment the topology: $\mathbb{I}_{grow}=Topk_{i\notin\theta^{l}}(|\nabla_{\theta}^{l}L|)$. The topology growing process $G_{k}$ adaptively starts every $\Delta T$ step. And at timestep $t$ the $k$ is decayed in a cosine annealing manner: $\frac{\alpha}{2}(1+\cos(\frac{t\pi}{T_{end}}))$, where $\alpha$ denotes the discount factor and $T_{end}$ is the shut down step.

Figure: Figure 3: Performance comparison and plasticity evaluation of random growth, our proposed growth method, and pruning mechanisms (seven independent seeds).
Refer to caption: x2.png

The results depicted in Figure 3 indicate that substituting random growth with the selection of new connections based on potentially provided gradients can enhance the agent’s performance and mitigate the dormant neuron issue. Furthermore, we compare two commonly used topology initial strategies: uniform and Erdos-Rrenyi (Evci et al., ), and we find that initializing the network in Erdos-Rrenyi type is slightly more effective for RL. Erdos-Rrenyi enables the number of connections in a sparse layer to scale with the sum of the number of output and input channels, which in turn ensures more stable computations in small networks. Furthermore, the agent can also perform more efficient learning in the early stage.

Figure: Algorithm 1 Neuroplastic Expansion TD3

#### Dormant Neuron Pruning

After replacing the naive growth with our gradient guidance growing process $G_{k}$, a new challenge emerges, i.e., dormant neurons exhibit a stark decline in their representational potency while tenaciously occupying network capacity. Such inactive neurons could be inherent from the initialization or introduced into the network topology through the growth schedule. This situation may lead to a topology that saturates quickly with disproportionately small computational power. Moreover, Lu et al. (), Liu et al. () demonstrated that parameters solely forward-linked to dormant neurons are omitted from the gradient calculation process, rendering them ineffective as they fail to receive meaningful guidance signals. We hypothesize that pruning and resetting of dormant neurons following the topology growth could address this issue.

This process aims to free up space for new connections, reactivating dormant neurons as new candidates. According to (Ceron et al., , ), sparsifying the value-based agents’ network can often enhance the performance, which offers assurance for our real-time pruning empirically. Consequently, we introduce a synchronous dormant neuron prune-and-reset mechanism into the topology growth schedule $G_{k}$.

Here, we adopt the calculation method $f(\cdot)$ showed in Eq.1 and set $\tau$ to be 0 0, which means only considering fully dormant neurons:$\mathbb{I}{prune}=f(\theta_{i})\leq 0$. To facilitate the topology’s gradual growth, we set the pruning upper bound $\omega$ for each iteration and the prune range for each layer $l$ is $Clip(0,\mathbb{I}_{prune}^{l},\omega\times\mathbb{I}_{grow}^{l})$.
Therefore, the final two parameters that control topology change are $k$ and $\omega$ so we will use $G_{k,\omega}$ to refer to our whole grow-prune schedule. The complete growth-pruning schedule for each layer $l$ in actor $\phi$ and critic $\theta$ can be rewritten as

$$ $G_{k,\omega}\left\{\begin{array}[]{l}1.\;\mathbb{I}_{grow}^{l}=Topk_{i\notin \breve{\phi}^{l}}(|\nabla_{\phi}^{l}L_{t}^{\phi}|)+Topk_{i\notin\breve{\theta} ^{l}}(|\nabla_{\theta}^{l}L_{t}^{\theta}|)\\ 2.\;\mathbb{I}_{prune}^{l}=f(\breve{\phi}^{l}_{i}\setminus\mathbb{I}_{grow}) \leq 0+f(\breve{\theta}^{l}_{i}\setminus\mathbb{I}_{grow})\leq 0;\end{array}\right.$ (2) $$

We then use $\mathbb{I}_{grow}^{l},\mathbb{I}_{prune}^{l}$ to generate $\{0,1\}$ masks and dot multiply the network parameters to achieve the evolution of the actor topology $\breve{\phi}$ and critic topology $\breve{\theta}$. The results in Figure 3 show that with the addition of the pruning mechanism after each growth round, the plasticity of the RL network is further preserved and the performance is further improved.

#### Neuron Consolidation via Experience Review

As discussed above, the proposed elastic neuron generation and dormant neuron pruning-based dynamic growing approach can greatly mitigate the plasticity loss. This approach smoothly maintains a higher level of active neurons, leading to improved performance.
However, a more careful investigation of the results with different seeds reveals that there is a potential risk of performance oscillation during the later learning stages, as RL is sensitive to random seeds (Eimer et al., ). This is summarized in the left part of Figure 5.

This behavior suggests that the frequent topology alternations may inadvertently cause the agent’s network to forget previously acquired knowledge.
Right part of Figure 5 illustrates the behavior of the agent during the dynamic topology evolution phase. It is worth noting that the policy necessitates the sequential use of three techniques: back straightening, kneeling, and bouncing off the feet. However, a more frequent altering of the topology could result in the forgetting of knowledge acquired initially, which could result in stability issue.

To address this challenge for achieving a better tradeoff, we propose an effective technique to consolidate the agent’s network based on experience review.
We consider the fluctuation rate of dormant neurons $\nabla f(\theta)$ to
determine whether it is necessary to review earlier experiences for alleviating catastrophic forgetting. Specifically, $\nabla f(\theta)$ measures the slope of changes of the number of dormant neurons (within the critic network) during the recent $C$ steps, i.e., $[150,450]$.
A larger positive value of $\nabla f(\theta)$ corresponds to a relatively greater reduction of dormant neurons from our proposed dynamic growing network, while a small value of $\nabla f(\theta)$ close to zero indicates a bottleneck in improved plasticity gains, which also indicates a limited space for further topology expansion and marks the later stage of policy learning (Figure 5).
We generally observe that $\nabla f(\theta)$ decreases during the course of training, as elastic neuron generation and dormant neuron pruning will lead to a larger change in the number of dormant neurons in the early stage (and cannot always enlarge the proportion of active neurons due to limited network capacity). We encourage the agent to review earlier experiences when there are more dormant neurons. At each training step, a random number $m\in[0,1]$ is selected, if $m$ exceeds the dynamic threshold $\epsilon$, the sampling is conducted from the initial quarter of the buffer; otherwise, the conventional sampling method is employed.

## 5 Experiments

Figure: Figure 4: Instability of Neuroplastic Expansion without neuron consolidation via experience review during the later stage.
Refer to caption: x3.png

In this section, we conduct comprehensive experiments to understand our proposed Neuroplastic Expansion NE method. We investigate the following key questions:
(i) How does NE improve the performance of the backbone deep RL method and compare against previous baselines? (ii) Can NE effectively mitigate loss of plasticity?
(iii) What are the effects of NE on the policy and value network and the importance of different components?
(iv) Can our method be applied to other deep RL methods with more complex image inputs?
(v) Can it adapt in continual RL scenarios?

### 5.1 Performance Comparison

#### Experimental Setup

We first conduct a series of experiments based on the standard continuous control tasks on OpenAI Gym (Brockman, ) simulated by MuJoCo (Todorov et al., ).
We compare NE against strong baselines including Reset (Nikishin et al., ),
ReDo (Sokar et al., ),
Layer Normalization (LN) (Lyle et al., ),
and Plasticity Injection (PI) (Nikishin et al., ).
To mitigate implementation bias and ensure a fair comparison, all baselines are implemented based on official implementations (Sokar et al., , Nikishin et al., , ), with the same architecture (See Appendix B.1).
We build all baselines based on the popular TD3 (Fujimoto et al., ) algorithm (where results based on other backbone deep RL methods are discussed in Section 5.3).
For each algorithm we run 7 seeds and report the mean and standard deviation.
A detailed description of the hyperparameters and setup can be found in Appendix B.2. The growth termination scale for NE is set to match that of the baselines. For all tasks, NE initializes agents from $25\%$ of total capacity and grows asymptotically.

#### Results

As shown in Figure 6, Reset inevitably suffers from periodic sharp performance drops after each reset for reviving dormant neurons to a large extent, as it can lead to the loss of reusable knowledge that necessitates relearning and impedes the overall progress.
LN and ReDo demonstrate more stable performance compared to Reset, but result in a lower proportion of active neurons, while PI maintains the highest level of neuron activation, but only performs in the relatively easier Hopper environment.
Our method achieves significant and consistent performance improvements compared to these strong recent baselines in terms of learning efficiency and final performance, with a larger margin in more complex environments.
In addition, it effectively improves the loss of plasticity as it maintains a higher level of active neuron ratio throughout training compared to the most competitive LN method.
It also better trade-off between performance and neuron utilization, as it performs stably and achieves better performance than PI, which has the highest neuron activation but suboptimal performance in most tasks.
It is also worth noting that our proposed method is orthogonal to existing techniques, where additional experiments for building it upon these established baselines can be found in Appendix C allowing for further improvements.

Figure: (a) Hopper
Refer to caption: x5.png

#### Ablation Study.

In this section, we conduct an additional ablation study to investigate the effectiveness of neuron consolidation in Neuroplastic Expansion, while the importance of elastic neural generation and dormant neural pruning has been demonstrated in Figure 3. We also examine the experience review (ER) module and the impact of our method on the policy and value networks.

Figure: (a) Validation of ER
Refer to caption: x11.png

As illustrated in Figure 7a, the experience review technique enhances performance stability across different seeds. In contrast, the absence of this critical component leaves the model vulnerable to potential policy collapse in subsequent stages (albeit slighter compared to Reset), demonstrating the critical role of reviewing historical knowledge in maintaining policy stability and preventing the loss of early valuable information as discussed in Section 4.2.

Figure 7b illustrates the comparison result of our method and its variants, isolating the effects of NE on the actor and critic networks separately.

We observe that NE is particularly crucial for the critic network. Removing this from the critic leads to significant performance degradation and significant loss of plasticity compared to the full version of NE (Figure 7c,7d); whereas limiting the expansion to only the critic results in a less pronounced drop in performance.
This asymmetry in impact may be attributed to the more challenging task faced by the critic, as it needs to provide accurate value estimates for different state-action pairs (Meng et al., ) considering a non-stationary data distribution. In addition, the critic also provides guidance signals to actor optimization, which requires high level of adaptability (Fujimoto et al., ).

### 5.2 Continual adaptation

In this section, we investigate the continual ability to learn and adapt to changing environments, which is a key ability of deep RL agents (Abbas et al., , Elsayed and Mahmood, ).

#### Experimental Setup

We follow the experimental paradigm of Abbas et al. (), and evaluate our proposed method on a variant of MuJoCo tasks that involve non-stationarity due to changing environments over time. Specifically, the agent is trained to master a sequence of $4$ environments (HalfCheetah $\to$ Humanoid $\to$ Ant $\to$Hopper), starting with HalfCheetah for $1000$ episodes of training, followed by the next task for the same number of episodes. A cycle is completed when training on Hopper is finished. The agent revisits this sequence of environments three times, constituting a long-term training schedule.
We compare our approach with the vanilla TD3 agent and the most competitive baseline, Reset, in the continual learning scenario (Abbas et al., ).

Figure: Figure 8: Plasticity measures, i.e., activated neuron ratio, id recorded for three agents with different settings during long term cyclic training.
Refer to caption: x15.png

#### Results

The performance of each method for a single environment when repeatedly learning on a sequence of environments as described above is summarized in Figure 9.
We aim to evaluate (i) the efficiency of the policy in swiftly learning the subsequent task after mastering the previous one within a single cycle, (ii) the agent’s ability to retain the benefits of initial learning for the same task across multiple cycles. As shown, Resetting, which involves initializing the parameters after each task, is currently deemed the most effective approach.

According to the row in Figure 9, NE adeptly handles each task without being influenced by previous training. It significantly outperforms the original TD3, emphasizing that standard RL algorithms do suffer from catastrophic forgetting in the continual learning setting. We attribute this success to NE’s capacity to introduce real-time topology adjustments for storing new knowledge, thereby ensuring continuous learning capabilities. Remarkably, in certain environments, such as Ant, NE-TD3 can retain previous knowledge and seamlessly continue learning. This could be attributed to the experience review module and the preservation of old neurons in the topology, which store pertinent information crucial for successful continual learning. When considering the column-wise comparisons, our method performs comparably to reset in the majority of tasks, yet significantly outshines reset in the challenging Humanoid environment.
The comparison in terms of active neurons for each baseline is shown in Figure 8, which illustrates that even in the long training schedule, NE can effectively alleviate neuron dormancy while efficiently learning, and it even performs better than resetting all parameters during the early training period.

Figure: (a) First task: HalfCheetah
Refer to caption: x16.png

### 5.3 Versatility

In this section, we demonstrate the generality of our proposed method by applying it to another popular deep RL method and evaluating its performance on more complex tasks with image input.

#### Experimental Setup

We evaluate four image-based motion control tasks from the DeepMind Control Suite (DMC) (Tassa et al., ): Reacher hard, Reacher easy, Walker walk and Cartpole Swingup Sparse, to demostrate the generalization of NE for deep RL algorithms and its robustness across diverse tasks. We consider DrQ (Yarats et al., ), a well-known variant of SAC (Haarnoja et al., ) specifically designed for processing image inputs, as the backbone algorithm. We use the same baselines from Section 5.1 for comparison.

Figure: Figure 10: First row shows the performance of five methods. Second row corresponds to the percentage of activated neurons. All experiments run with seven independent seeds
Refer to caption: x20.png

#### Results

First row of Figure 10 shows that NE can be effectively combined with DrQ to achieve stable performance on image input tasks, outperforming other baselines in all four scenarios and demonstrating its versatility across different algorithms. In addition, the second row of Figure 10 (activated neuron ratio) indicates that, although EN cannot reach the peak value set by Reset, it can better delay the dormancy of neurons compared to other warm mitigation startegies and ensure stable performance.

## 6 Conclusion

Inspired by the cortical growth in biological agents that triggers neuronal topology expansion to enhance plasticity and adaptability to new situations, we present a novel perspective for deep RL agents to mitigate the loss of plasticity with a dynamic growing network. Building on this insight, we introduce a comprehensive topology growth framework, Neuroplastic Expansion (NE), which addresses both growth and pruning perspectives to maximize the performance benefits of topology expansion.
NE adds high-quality elastic neurons and connections based on the gradient to improve the network’s learning ability; prunes dormant neurons in a timely manner for better utilizing network capacity, and returns them to the candidate set to maintain the agent’s plasticity, making dormant neurons reusable. NE outperforms previous methods which aimed at alleviating plasticity loss on diverse tasks and demonstrates stable, outstanding performance on plasticity metrics.
Interesting future directions include the exploration of adaptive architectures that dynamically adjust their capacity based on the agent’s performance or task complexity, and further minimizing computational overhead.
Overall, despite the acknowledged limitations, our research provides practical insights into enhancing the plasticity of RL agents. We hope our findings pave the way for diverse future explorations, potentially leading to more sample- and parameter-efficient, as well as adaptable, RL algorithms.

## Appendix A Related Preliminaries

#### TD3

In our paper, we use TD3 as a practice backbone for most experiments.Thus, here we introduce the training process of TD3 in detail. TD3 is a deterministic Actor-Critic algorithm. Different from the traditional policy gradient method DDPG, TD3 utilizes two heterogeneous critic networks,i.e., $Q_{\theta_{1,2}}$, to mitigate the over-optimize problem in Q learning.

The training process of TD3 follows the Temporal difference learning (TD):

$$ $L_{Q}(\theta_{i})=\mathbb{E}_{s,a,s^{\prime}}\big{[}\big{(}y-Q_{\theta_{i}}(s, a)\big{)}^{2}\big{]}\text{ for }\forall i\in\{1,2\}.$ (3) $$

Where $y=r+\gamma\min\limits_{i=1,2}{Q_{\bar{\theta}_{i}}}(s^{\prime},\pi_{\bar{\phi}
}(s^{\prime}))$, $\bar{\phi}$ denotes the target network parameters. The actor $\pi_{\phi}$ is updated according to the Deterministic Policy Gradient (Fujimoto et al., ):

$$ $\nabla_{\phi}J(\phi)=\mathbb{E}_{s}\big{[}\nabla_{\pi_{\phi}(s)}Q_{\theta_{1}} (s,\pi_{\phi}(s))\nabla_{\phi}\pi_{\phi}(s)\big{]}.$ (4) $$

At each training step $t$, the agent first randomly samples a batch of state transitions from the buffer and calculates the loss gradient as described above. It is worth noting that the proposed method in our paper is a plug-in and does not affect the training process of the reinforcement learning algorithm itself. For this reason, the training details of the reinforcement learning algorithm are abbreviated in the main text.

#### DrQ

DrQ is an algorithm based on maximum entropy reinforcement learning. Its backbone network and training method are the same as SAC (Haarnoja et al., ). The soft value function is trained to
minimize the squared residual error:

$$ $J_{V}(\phi)=\mathbb{E}_{(s,a)\sim D}{\frac{1}{2}(V_{\phi}(s)-Q_{\theta}(s,a)+ \log\pi_{\psi}(a|s))}$ (5) $$

where $a$ are sampled according to the current policy, instead of the replay buffer. The soft Q-function parameters can be trained to minimize the soft Bellman residual:
$J_{Q}(\theta)=\mathbb{E}_{(s,a)\sim D}[\frac{1}{2}(Q_{\theta}(s,a)-\hat{Q}(s,a
))^{2}]$. Then the policy network can be learned by minimizing the expected KL divergence

$$ $J_{\pi}(\psi)=\mathbb{E}_{s\sim D}\big{[}D_{KL}\big{(}\pi_{\phi}(\cdot|s)|| \frac{\exp{Q(s)}}{Z(s)}\big{)}\big{]}$ (6) $$

## Appendix B Experimental Details

### B.1 Network Structure

#### TD3

The normal size of the TD3 network we used is the official architecture, and the detailed setting is shown in Tab 1. Besides, the network we used for Scale-up experiments incrementally adds the depth of both the actor and critic. For The 1.8x agent, we add one MLP as a hidden layer in two critic networks in front of the output head and one MLP layer as the embedding layer at the front of the actor. And the hidden dim is also increased to 256 (double). And so on, we follow the above approach to grow the model by an equal amount each time.

**Table 1: Network Structures for TD3**
| Layer | Actor Network | Critic Network |
| --- | --- | --- |
| Fully Connected | (state dim, 256) | (state dim, 256) |
| Activation | ReLU | ReLU |
| Fully Connected | (256, 128) | (256, 128) |
| Activation | ReLU | ReLU |
| Fully Connected | (128, action dim) | (128, 1) |
| Activation | Tanh | None |

#### DrQ

We use DrQ to conduct experiments on robot control tasks within DeepMind Control using image input as the observation. All experiments are based on previously superior DrQ algorithms and maintain the architecture from the official setting unchanged.

### B.2 Hyperparameter

#### TD3

Our codes are implemented with Python 3.8 and Torch 1.12.1. All experiments were run on NVIDIA GeForce GTX 3090 GPUs. Each single training trial ranges from 6 hours to 21 hours, depending on the algorithms and environments, e.g. DrQ spends more time than TD3 to handle the image input and DMC needs more time than OpenAI mujoco for rendering. Our TD3 is implemented with reference to [github.com/sfujim/TD3](github.com/sfujim/TD3) (TD3 source-code). The hyper-parameters for TD3 are presented in Table 2. Notably, for all OpenAI mujoco experiments, we use the raw state and reward from the environment and no normalization
or scaling is used. An exploration noise sampled from $N(0,0.1)$ (Lillicrap, ) is added to all baseline methods when selecting an action. The discounted factor is $0.99$ and we use Adam Optimizer (Kingma, ) for all algorithms. Tab.2 shows the common hyperparameters of TD3 used in all our
experiments. And for NE, following the related work (Elsayed et al., ), we use weight Clip on dormant neurons which are already reset and pruned from the current topology. Notably, We add layer normalization after the hidden layer of both critic and actor to construct the comparison baseline LN-TD3 To construct the comparison baseline LN-TD3, but in all other methods, no normalization mechanism is introduced to ensure the fairness of comparison. We reproduce plasticity Injection according to their official paper (Nikishin et al., ). And use the official code of ReDo in our comparison under the default setting [https://github.com/google/dopamine/tree/master/dopamine/labs/redo](https://github.com/google/dopamine/tree/master/dopamine/labs/redo). For Reset-TD3, after a lot of empirical debugging, we found that parameter reset every $90,000$ step is a generally better setting, and fixed it as a hyperparameter. We use seeds 0 to 7 for all the experiments. And if you want to reproduce the HalfCheeetah learning curve, we suggest you use seeds 0 5. For reset-TD3, reset is used every $90,000$ steps

**Table 2: A comparison of common hyperparameter choices of algorithms. We use None to denote the ’not applicable’ situation.**
| Hyperparameter | NE-TD3 | Reset-TD3 | ReDo-TD3 | LN-TD3 | PI-TD3 |  |
| --- | --- | --- | --- | --- | --- | --- |
| Actor Learning Rate | $1e^{-4}$ | $1e^{-4}$ | $1e^{-4}$ | $1e^{-4}$ | $1e^{-4}$ | $1e^{-4}$ |
| Critic Learning Rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $3e^{-4}$ | $3e^{-4}$ | $1e^{-3}$ |
| Representation Model Learning Rate | None | None | None | None | None | None |
| Discount Factor | $0.99$ | $0.99$ | $0.99$ | $0.99$ | $0.99$ | $0.99$ |
| Batch Size | $128$ | $128$ | $128$ | $128$ | $128$ | $128$ |
| Buffer Size | $1e6$ | $1e6$ | $1e6$ | $1e6$ | $1e6$ | $1e6$ |

#### DrQ

We use DrQ as the backbone to verify our methods on DeepMind Control tasks. And All the methods use the same setting. The details can be seen in Tab.3. Notably, For Reset-DrQ, we only reset the last three MLP layers which is suggested by their paper.

**Table 3: A default set of hyper-parameters used in our DrQ based methods.**
| Hyperparameter | NE-DrQ | Reset-DrQ | ReDo-DrQ | LN-DrQ | PI-DrQ |  |
| --- | --- | --- | --- | --- | --- | --- |
| Replay buffer capacity | $1e6$ | $1e6$ | $1e6$ | $1e6$ | $1e6$ | $1e6$ |
| Action repeat | $2$ | $2$ | $2$ | $2$ | $2$ | $2$ |
| Seed frames | 4000 | 4000 | 4000 | 4000 | 4000 | 4000 |
| Exploration steps | 2000 | 2000 | 2000 | 2000 | 2000 | 2000 |
| n-step returns | 3 | 3 | 3 | 3 | 3 | 3 |
| Mini-batch size | 256 | 256 | 256 | 256 | 256 | 256 |
| Discount $\gamma$ | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 |
| Optimizer | Adam | Adam | Adam | Adam | Adam | Adam |
| Learning rate | $1e-4$ | $1e-4$ | $1e-4$ | $1e-4$ | $1e-4$ | $1e-4$ |
| Agent update frequency | 2 | 2 | 2 | 2 | 2 | 2 |
| Critic Q-function soft-update rate | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 |
| Features dim. | 50 | 50 | 50 | 50 | 50 | 50 |
| Hidden dim. | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 |

#### EN

For Humanoid and Ant tasks, we set grow interval $\Delta T=25000$, grow number $k=0.01*\text{rest capacity}$, Prune upper bond $\omega=0.4$, ending step is the max training step, the threshold of ER is $0.35$ and the decay weight $\alpha=0.02$(which is used in all the tasks). For other OpenAI Mujoco tasks, we set grow interval $\Delta T=20000$, grow number $k=0.15*\text{rest capacity}$, Prune upper bond $\omega=0.2$, ending step is the max training step, the threshold of ER is $0.25$. Notably, as for the continuous adaptation tasks, we set $\Delta T=50000$ to ensure the topology can continue growing during the long-term training, and other parameters are same as Humanoid. When we combine our method with DrQ, We use single set of hyperparameters to apply on all four tasks. $\Delta T$ is set as $20000$ to ensure the network can grow quickly to handle the complex input. The grow number $k$ is $0.02*\text{rest capacity}$ for critic and $0.01*\text{rest capacity}$ for actor. The Prune upper bond $\omega=0.23$ and the threshold of ER is $0.18$. We set ER step $C$ to $450$ for Mujoco tasks and ${150,250}$ is work on DMC.

## Appendix C Additional experiments

Figure: Figure 11: Quadrature gain of our method combined with the baselines, with each method executed using seven independent seeds.
Refer to caption: x21.png

We use our method in combination with other mainstream plasticity loss mitigation methods on two mujoco tasks to further verify whether our method has the potential ability to improve the performance of other techniques. Specifically, we combine NE with layer normalization, ReDo, and reset to verify whether topology growth can improve existing techniques. As shown in Figure 11, results illustrate that NE has obvious orthogonal gains with well-performing baselines. In particular, resetting and ReDO empirically perform much better when combined with our method. 7 runs for each method, and we use the NE score as the normalized performance.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Previous work, such as Plasticity Injection[1], has already proposed dynamically growing an agent's network, please provide a detailed comparison.

2.Dynamically expanding the size of a neural network could potentially lead to policy instability. For instance, the policy before and after expansion might be inconsistent. However, the results reported in Figure 6 appear very stable. Please provide specific analyses and ablation studies demonstrating how NE maintains policy stability during network expansion. 

3.It would be helpful if the authors could provide experiments or analyses to explain the impact of dead neurons during training. Do these neurons store explored knowledge that contributes positively to the learning process, or do they have a negative effect on training? Please provide some analyses and visualizations to illustrate their impact on the learning process.

[1]Deep Reinforcement Learning with Plasticity Injection.
soundness: 2
strengths: 1.The paper is well-written.

2.The concept of Neuroplastic Expansion (NE) is well-motivated.
summary: This paper introduces a novel idea, Neuroplastic Expansion (NE), to address the problem of plasticity loss in reinforcement learning (RL). The paper is well-written and presents the concept clearly. However, there are some concerns, particularly regarding its contribution relative to existing work. If these concerns can be resolved, I would consider improving the rating from 5 to 6.
weaknesses: See questions.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. Why is it called “Elastic” Neuron Generation? What is elastic exactly here?
2. I’m not sure how experience review reduces learning instability. I’m not convinced by the revisiting-is-useful argument. Without experience review, sampling is iid, so temporally old samples are still revisited. Why is revisiting old samples with higher probability useful, particularly when the dormant neuron ratio is high?
3. To get the gradients to decide which weight to regenerate, you need to use the fully expanded network and backpropagate everything, then find the top k weights that do not exist in the actual network. Is that correct? If so, then this metric is inaccurate because it adds all weights that take part in backpropagation. The accurate way is to add one weight at a time and backpropagate gradients each time. This is, of course, very expensive since you need to have $N$ additional forward and backward passes. In contrast, the process you described makes some approximations that are not clearly presented. 
4. Are both pruning and regeneration neuron/unit-based?
5. In section 5.2, the environments have different action spaces; how did you handle that?
6. The authors stated that resetting was deemed the most effective approach. But no references are given (line 467).
7. In Figure 8, why are there spikes in the activated neurons ratio?
8. What is meant by removing the difference in fitting ability in lines 217-218?
soundness: 3
strengths: The authors present a novel approach that improves plasticity for deep reinforcement learning methods. The approach seems effective and achieves better performance than many existing methods, such as layer normalization, ReDo, and plasticity injection in many environments. The authors provided an extensive experimental study of their method in different environments (MuJoCO Gym and DMC) and with different learning algorithms (DrQ and TD3).
summary: The paper introduces a new approach to maintaining plasticity for deep reinforcement learning methods based on intuitions about cortical cortex expansion in cognitive science. The approach includes three components: 1) neuron regeneration, 2) dormant neuron pruning, and experience review. While neuron regeneration and dormant neuron pruning parts help maintain plasticity, the experience review reduces instability due to high plasticity. The authors test the effectiveness of their approach and its components in various RL environments and compare it against other baselines.
weaknesses: - The paper significantly lacks mathematical rigor. Here are some examples that are representative of these inaccuracies, although they don’t constitute an exhaustive list:

  - It should be $\breve{\theta} \subset \theta$ not $\breve{\theta} \in \theta$. Or more precisely, $\breve{\theta}_l \subset \theta_l, \forall l \in \\{1,...,N\\}$, where $N$ is the number of layers in the network.

  - In line 213, $\mathbb{I_{grow}}$ is not defined well. It should be a list, but you assign it with two random quantities added together so it looks like a vector or a scaler instead of a set. Additionally, how is the random function defined? The random function should output a set, which you then need to union with the other set, $\mathbb{I_{grow}}= RandomSet1 \cup RandomSet2 $ not $\mathbb{I_{grow}}= RandomSet1 + RandomSet2$. A complete, rigorous mathematical description is expected.
  - It should be $ArgTopK$ not $TopK$ in equation 2 and line 256
  - what does this mean to write $\mathbb{I_{prune}}= f(\theta_i) \leq 0$? The left side should be a list, and the right-hand side should be an inequality. It should be something like $\mathbb{I_{prune}} = \\{\texttt{index}(\theta_i) | f(\theta_i) \leq 0 \\}$.
  - what does it mean to have a dormant ratio of negative in line 301? The ratio possible values are in $[0,1]$.
- The paper presentation and writing are not clear. 
  - The authors claim NE maintains a high level of elastic neurons (see line 60), but no definition of what elastic neuron is given. Is elasticity here something different from plasticity? How do we measure either of them?
  - The term plasticity is loosely used to represent activated neuron ratio (e.g., Figure 6f). A clear definition of what plasticity means should be presented. If plasticity is the activated neuron ratio, then the paper's approach does not actually address the loss of plasticity as claimed since in all figures where the activated neuron ratio is presented, we see a decrease in their percentage with the paper's approach, similar to other methods. 
  - The algorithm is not complete. For example, the cosine annealing scheduler is missing, and the experience review part is not clearly shown. Additionally, Algorithm 1 works on the weight level, but the description from the text talks about neuron-level regeneration and pruning. The algorithm needs to reflect that.
  - Since the authors depend on the sparse network training framework as part of their approach. They should fully explain what the sparse network training framework is in writing and in the algorithm.
  - The process of experience review is not clear. The fluctuation rate of dormant neurons $\nabla f$ is a function of each unit, but the authors talk about some aggregate quantity. Is that new quantity a summation of all units in the network, $\nabla f = \sum_i f_i$? Why isn’t this part of the algorithm?
  - Since the authors chose to rely on the activated neuron ratio (1-dormant neuron ratio), equation 1 needs to reflect that, currently, the definition is mentioned in-text, whereas it should be highlighted in equation 1 instead of dormant neuron ratio, which the authors do not really use.
- Some claims in the paper are not supported by evidence.
  - The paper overclaims what their approach can address. The authors mention that their approach mitigates loss of plasticity primacy bias, reduces catastrophic forgetting, and strikes a stability-plasticity balance. Most of these claims are not supported by evidence. Using those terms loosely without being precise about what is being studied in an experiment makes the paper hard to navigate.
  - For example: “topology growth can effectively alleviate neuron deactivation and thus maintain the ability of policy learning to mitigate the loss of plasticity and alleviate the primacy bias.”--- It's unclear how the experiment shows loss of plasticity or primacy bias mitigation. The authors should instead only claim that their approach reduces the dormant neuron ratio and not claim anything about loss of plasticity or primacy bias.
  - The current ablation is not sufficient. Ideally, the authors should remove each component of the system: 1) neuron regeneration, 2) experience review, and 3) dormant neuron pruning. The authors did 1 and 2 but not 3. We need to know what happens if we remove dormant neuron pruning.
- Issues in empirical evaluation:
  - Many figures do not have labels on the axes, so it is hard to know (even after careful investigation) what is being varied. For example, the x-axis in Figure 5 has no label, and I don’t know what 0 to 3 means here. Other examples include but are not limited to Figure 2 (missing x-axis label), Figure 4 (what is the score in the y-axis), and Figure 6 (missing y-axis label).
  - The results are not statistically significant. A very low number of independent runs (7 runs) are used, and they have overlapping error bars in most figures. More independent runs are needed, especially since the error bars are overlapping. I suggest the authors run each algorithm for 30 independent runs in all of their experiments.
  - In section 5.2, a fixed number of episodes is used in each task, whereas a fixed number of steps should be used to have consistent amount of experience in each task. 

**Minor issues:**

- The author defines the gradient as $L_t$. Then the sentence after that says it’s $\nabla L_t$. 
- The name of the approach is not very representative of what the algorithm does. It’s called neuroplastic expansion, emphasizing the expansion part. A better name, such as neuroplastic regeneration and pruning, can be more representative and accurate.


&nbsp;
&nbsp;
&nbsp;

Overall, I believe this paper could serve as a good algorithmic contribution to the community if the authors addressed my concerns based on this feedback. So, I’m willing to increase my score given that 1) the authors tuned down the claims and made them modest such that they accurately reflect what is being studied by their experiments, 2) the authors fixed all mathematical inaccuracies and provided a completed algorithm, 3) the terminologies are used carefully precisely instead of loosely, and 4) the empirical work is improved through more independent runs and improved figures.
\
\
\
\
**Update:** The authors worked hard to address my concerns. It was a fruitful back-and-forth discussion that improved the paper's quality immensely. Since my concerns have been addressed, I'm happy to recommend acceptance.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. In Figure 5, at which point does the model reach its maximum capacity (i.e., no more room for growing unless pruned)?
2. In Figure 6, why does Plasticity Injection fail even before injecting in Walker2d and HalfCheetah? Before injection, shouldn’t they be equivalent to vanilla TD3?
3. In the 'Dormant Neuron Pruning' section, the expression $Clip(a,b,c)$ is confusing to read without any definition.
4. The dynamic threshold defined in paragraph in 'Neuron Consolidation' as a whole is too confusing to read. I don't think $\nabla f(\theta)$ is the right definition, since it's not a derivative w.r.t. the dormant ratio, but rather an average change rate of the dormant neuron count. Another thing I want to make sure is that  $\nabla f(\theta)$ is used as the dynamic threshold $\epsilon$, right? It's not clearly stated in the text.
soundness: 2
strengths: 1. The core idea of NE seems very promising in terms of lifelong learning: add new capacity to learn new information, remove useless/dead neurons, and prevent catastrophic forgetting. Connection to biology is also a big plus.
2. The necessity for each component was well explained (Figure 2,3,4). I found it especially interesting to see a proof of catastrophic forgetting in a Mujoco task.
summary: This paper presents Neuroplastic Expansion (NE), a generally applicable training scheme designed to mitigate plasticity loss in RL. NE comprises of three components:

1. Adding neuron connections based on potential gradients (elastic neuron generation)
2. Pruning neuron connections based on dormant ratio (dormant neuron pruning)
3. A training batch sampling scheme that focuses on early samples depending on dormant ratio fluctuation (neuron consolidation)

Compared to prior methods such as Reset, ReDo and Plasticity Injection, NE showed superior performance in state-based Mujoco tasks (with TD3) and several pixel-based DMC tasks (with DrQ). NE was also able to maintain plasticity while sequentially training through multiple environments in a cyclic manner. Its plasticity — measured by dormant ratio — is well preserved in the majority of the experiments, proving its effectiveness in maintaining trainability and preventing loss of plasticity.
weaknesses: 1. The writing is sometimes not detailed enough and causes confusion (see questions and weaknesses below).
2. Some crucial design choices are not well justified and/or validated.
    1. Neuron consolidation is proposed to prevent catastrophic forgetting, which often occurs late stage (as shown in Figure 4). However, the dynamic threshold they use to control the strength of consolidation plateaus to its lowest value (strongest consolidation) even before halfway through the training process (Figure 5). This discrepancy raises question on whether this complexity is really necessary, especially since a simple time-dependent scheduling scheme could also fit the justification of ‘not forgetting early state-action distribution’.
    2. The amount of pruned dormant neurons are forced to be less than amount of added connections in order to guarantee that the network is increased in size. This also looks like an unnecessary detail since we can achieve the same by using ReDo and then growing a small number of neurons. I think there should be an explanation on why this design choice is essential.
    3. Although less critical, some components of elastic neuron generation also needs more careful consideration, such as the cosine annealing schedule (especially since RigL [1] was not primarily designed for continual learning).
3. The experimental setup needs more refinement.
    - For the main experiments, although it’s convincing that NE surpasses prior works in Mujoco and DMC tasks, it would be nice to see whether NE is also effective in more challenging environments.
    - The recently proposed NaP [2] is an extremely competitive method for continual learning, and I think it’s a crucial baseline in the main experiment.
    - The cycling Mujoco experiment (Figure 9) is plotted on ‘episodes’ (and is the only one). This is problematic, since different lengths of episodes would result in different number of update steps and thus varying degree of plasticity loss.
    - It would have been nice to see whether NE can synergize with other methods such as CReLU or LayerNorm.

Overall, I think this paper needs more improvements before it can be published. However, I am also ready to change my scores if some of the above concerns are refuted/addressed.

[1] Rigging the Lottery: Making All Tickets Winners., Evci et al., ICML 2020.

[2] Normalization and effective learning rates in reinforcement learning., Lyle et al., arXiv.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: * How is the network reinitialized based on the growth criterion? Is it initialized with random weights similar to the initial stage?
* If reinitialization involves random weights, how does this approach effectively reduce dormant neurons, especially considering that large feature values after extended training might lead to immediate pruning of reinitialized neurons.
* Is the Experience Review technique effective across all experimental setups, or is its efficacy primarily validated only in specific environments like HalfCheetah? 
* PThe performance curves indicate that the dynamic actor maintains a stable plasticity rate similar to the static one. Why does the dynamic actor perform better despite this similarity in plasticity rates?
* For a more comprehensive evaluation, should the authors include additional baselines such as the base TD3 and TD3 with only network growth (without pruning) to isolate the effects of different components of NE.
* What constitutes a valid starting sparsity rate for NE?
* What are the optimal rates for growth and pruning, and how do these rates influence overall performance? An analysis of hyperparameter sensitivity would provide deeper insights into NE's robustness and adaptability.
soundness: 3
strengths: * **Novelty:** The paper introduces a novel idea inspired by human brain mechanisms, specifically cortical expansion, to address plasticity loss in deep RL. This biologically motivated approach offers a novel perspective for significant advancements in continual learning for artificial agents.
* **Good architectural design:** Neuroplastic Expansion (NE) is meticulously designed to balance network growth with resource efficiency. By adding elastic neurons based on gradient potential and recycling dormant neurons, NE maintains network expressivity and adaptability without causing uncontrolled growth in network size.
summary: This paper addresses the critical issue of plasticity loss in deep reinforcement learning (deep RL), where agents' adaptability decreases over time, hindering continuous learning in dynamic environments. Inspired by biological neural networks, the authors propose Neuroplastic Expansion (NE), a novel mechanism that dynamically enlarges the neural network by adding elastic neurons based on gradient potential. NE maintains high plasticity by regenerating and recycling dormant neurons, effectively mitigating the plasticity-stability dilemma inherent in deep RL.
weaknesses: * **Missing Relevant Work:**
    * Plasticity-Related Studies: The paper overlooks several relevant studies on neural network plasticity ([1]-[5]).
    * Cortical Expansion Citations: The discussion on cortical cortex expansion cites works that describe patterns of cortical expansion. I think the authors miss foundational studies that first identified evidence of cortical expansion.
* **Experimental Setup**:
    * The evaluation of MuJoCo environments is limited to the TD3 algorithm, which is considered outdated. Assessing NE using more recent and robust algorithms such as TD7, TD-MPC2, or BRO would enhance the relevance and robustness of the findings.
    * Other than Mujoco, I think it is beneficial to test in the state-based DMC, maybe by trying to compare with reset-based methods under identical experimental configurations as primacy bias paper. 

[1] On warm-starting neural network training., Ash et al, 2020. 

[2] A study on the plasticity of neural networks., Berariu et al, 2021. 

[3] PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning., Lee et al, 2023. 

[4] Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks., Lee et al, 2024. 

[5] Normalization and effective learning rates in reinforcement learning., Lyle et al, 2024.

Meta-review: This paper addresses the stability-plasticity dilemma inspired by dormant neuron pruning and expansion of connection topology inspired by cortical expansion in cognitive science. The approach is intuitive, but the actual details of expansion seem quite complicated. An additional technique used by this paper is called experience review for neuron consolidation, where basically experience replay from the initial quarter of the buffer is conducted if plasticity gain is bottlenecked. Other times, the experience replay goes on normally.

Careful analysis is conducted to establish that all three together can strike a superior balance between stability and plasticity, providing highly performant learning in Mujoco Gym and DM Control Suite environments.

The writing remains unclear, as the reviewers mentioned, making it difficult to understand what the algorithm exactly does (I appreciate the pseudocode) and what the experiments exactly are. 

I recommend acceptance with caution and strongly advise the authors to undertake a comprehensive revision of the paper before final submission to address the clarity issues.

Decision: Accept (Poster)