Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
Authors: Jiashun Liu
Zihao Wu
Johan Obando-Ceron
Pablo Samuel Castro
Aaron Courville
Ling Pan
Venue: NeurIPS 2025 poster ()
Keywords: Deep Reinforcement Learning
Dormant Neurons
Neuron Resetting
Primary area: reinforcement_learning
TL;DR: Measuring neuronal activity via activations is ineffective in complex agents, as these values do not reflect true learning capacity. We introduce GraMa, which offers robust quantification and resetting guidance across various network architectures.

Abstract: Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss, which impairs their ability to adapt to new data and learn continually. A common method to quantify and address this issue is the $\tau$-dormant neuron ratio, which uses activation statistics to measure the expressive ability of neurons. While effective for simple MLP-based agents, this approach loses statistical power in more complex architectures. To address this, we argue that in advanced RL agents, maintaining a neuron's **learning capacity**, its ability to adapt via gradient updates, is more critical than preserving its expressive ability. Based on this insight, we shift the statistical objective from activations to gradients, and introduce **GraMa** (**Gra**dient **Ma**gnitude Neural Activity Metric), a lightweight, architecture-agnostic metric for quantifying neuron-level learning capacity. We show that **GraMa** effectively reveals persistent neuron inactivity across diverse architectures, including residual networks, diffusion models, and agents with varied activation functions. Moreover, **re**setting neurons guided by **GraMa** (**ReGraMa**) consistently improves learning performance across multiple deep RL algorithms and benchmarks, such as MuJoCo and the DeepMind Control Suite. **We make our code available.**

Measure gradients, not activations! Enhancing
neuronal activity in deep reinforcement learning
Jiashun Liu!* Zihao Wuln
Johan Obando-Ceron Pablo Samuel Castro
Aaron Courville
Ling Pan'
Hong Kong University of Science and Technology
2 Mila Québec Al Institute Université de Montréal
Abstract
Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss, which impairs their ability to adapt new data and learn continually. A common method to quantify and address this issue the r-dormant neuron ratio, which uses activation statistics to measure the expressive ability of neurons. While effective for simple MLP-based agents, this approach loses statistical power in more complex architectures. To address this, argue that in advanced RL agents, maintaining a neuron's 's learning capacity, its ability to adapt via gradient updates, is more critical than preserving its expressive ability. Based on this insight, we shift the statistical objective from activations to gradients, and introduce GraMa (Gradient Magnitude Neural Activity Metric), lightweight, architecture-agnee metric for quantifying neuron-level learning capacity. We show that GraMa effec- tively reveals persistent neuron inactivity across diverse architectures, including residual networks, diffusion models, and agents with varied activation functions. Morcover, resetting neurons guided by GraMa (ReGraMa) consistently improves learning performance across multiple deep RL algorithms and benchmarks, such as MuJoCo and the DeepMind Control Suite. We make our code available".
Introduction
Deep reinforcement learning (Deep RL) has achieved remarkable success across variety of domains, including robotics [Liu et al., 202 I], foundation model fine-tuning [Shao et al., 2024, Liu et al., 2025c, Yu al., 2025, Li al., 2025, Lu et al., 2025, Liu et al., 2025b], and game playing [Berner et al., 2019, Schwarzer et al., 2023]. These advancements have been driven by the expressive power and adaptive learning ability of neural networks which effectively approximate and optimize value functions and/or policies [Sokar et al., 2023]. However, recent studies have uncovered critical and often underexplored challenge: as training progresses, subsets of neurons in these networks often experience a progressive loss of activity and become dormant [Sokar et al., 2023, Ma et al., 2024, Qin al., 2024]. This phenomenon reduces the learning capacity of the network, comprising its ability to adapt to non-stationary data distributions [Nikishin et al., 2022], which in turn hinders their ability to acquire new knowledge and adapt to evolving environments [Abbas al., 2023]. Despite its importance, quantifying and mitigating neuronal activity remains challenging due to its complex underlying mechanisms [Lyle al., 2023, Obando Ceron et al., 2023, Nauman et al., 2024a, Lyle et al., 2024].
To address this problem, a primary principle has been to restore a network's 's learning ability by reactivating or resetting inactive neurons. These approaches span multiple granularities: model-level
"Equal contribution. Correspondence to: (1 (ljshasdream, jobando87 30ggmma110mm0mm0ommo CCode: https: //t/ thub. ctm hoom///orrsslliu/e wtwpsssssssuusddddooeeee wwarrr-baaae bww.bbaseeeeeee pllssticit-mmeteee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and layer-level resets that reinitialize specific layers, while straightforward, often lead to catastrophic forgetting [Nikishin et al., 2023]. On the other hand, neuron- derreddd resets (e.g.. ReDo [Sokar et al., 2023]) target specific underperforming neurons [Xu et al., 2023] and provide a finer-grained approach by selectively reinitializing a subset of neurons identified as dormant, which mitigates forgetting and maintains computational efficiency.
The effectiveness of neuron-dependent resets hinges critically on having reliable criterion to identify which neurons require initialization. Existing methods primarily rely on activation- based metries, such as the т dormant neuron metric [Sokar et al., 2023], which measures neuronal inactivity based on activation values (i.e., the output of activation functions). It has demonstrated utility in standard architectures, and has been used to guide targeted neuron resets to restore activity in simple settings such as serial MLPs with ReLU activations [Agarap, 2018], providing simple means of maintaining learning capability without relying on auxiliary networks [Nikishin et al., 2023] or models [Lee et al., 2024]. By identifying neurons with weak or no activation and selectively reinitializing them, the activation- based dormancy metrics become a widely adopted tool for restoring learning capabilities and preventing performance plateau in standard deep RL settings [Farias and Jozefiak, 2025, Liu et al., 2025a, Juliani and Ash, 2024].
However, deep RL architectures have rapidly evolved beyond these simple architectures, inte- grating advanced components such as residual connections [Nauman et al., 2024b, Lee al., 2025a], mixture of experts (MoEs) [Obando- Ceron et al., 2024, Sokar et al., 2024, Willi et a..], and diffusion-based models [Ren et al., 2025, Wang et al., 2024] to improve scala- bility and performance for tackling complex tasks involving continuous control and high- dimensional observations. Additionally, they also imperceptibly alter the statistical properties of neuron activations, creating significant mis- match with traditional activity metrics. Through
Low laarning Capasiy neauron mnto
Complex Architecture Based Policy
E x 0
ReDo
Vanila
ReGraMa (Ours)
Complex Architecture Based Policy
Nomalted score
RaDo Vanila
RetiraMa (Gurs)
our extensive analysis, we find that resetting based on 7-dormant neuron ratio (ReDo) re- duces effectiveness in these newer architectures.
Staps millins)
Figure 1: Dormant neuron metric struggles in ad- vanced vision RL BRO-net agent [Nauman et al., 2024b]. Neuron resetting based the dormant neuron index (ReDo) cannot restore the learning capacity of the agent, which limits its effectiveness. The curves record normalized score across 3 image-input tasks (15 runs per method), i.e., Dog Stand, Dog Walk, Dog Run.
As shown in Fig. 1, ReDo struggles to identify inactive neurons that have lost their learning capacity the more advanced BRO-net archi- tecture, which features residual paths and layer normalization, ultimately failing to achieve the intended enhancement. This mismatch arises because activation- based metrics focus solely on a neuron's 's current output, evaluating its expressive capacity (how strongly it activates), while neglecting its learning capacity (how effectively it can adapt to new data distributions). Consequently, it becomes particularly problematic as deep RL. architectures evolve beyond simple feedforward networks to incorporate advanced structural elements, diverse activation functions, and sophisticated normalization techniques. In these modern architectures, a neuron's 's activation magnitude often fails to accurately reflect its true learning potential.
To address this mismatch, we propose a fundamental shift in perspective: from evaluating neurons based on their outputs to evaluating their learning potential by leveraging gradient magnitude. While activation values only capture what a neuron currently expresses, gradients tend to measure a neuron's 's capacity in response the given situation that directly drives parameter updates. This makes it general and natural proxy for neuronal health across architectural variations. Building upon this insight, we introduce GraMa (Gradient Magnitude based Neuronal Activity Metric), a robust and lightweight framework for quantifying neuronal activity via the gradient magnitudes. GraMa maintains validity across diverse architectural patterns, making it well-suited for modern deep RL agents.
In addition, GraMa imposes negligible computational and memory overhead by utilizing information already present in the optimization pipeline, which is a critical consideration for iisource-intensive RL training. Leveraging GraMa's 's efficiency, we develop a targeted neuron reset mechanism, (ReGraMa),
2

---

that selectively reinitializes inactive neurons that have lost their learning capacity during training. This mechanism demonstrates robust efficiency across various architectures, including the SAC variant [Nadimpalli et al., 2025], the residual BRO-net [Nauman et al., 2024b], and the diffusion-barn policy DACER [Wang et al., 2024].
Our contributions are summarized as follows:
We show that the widely-adopted activation-basel neuronal health measurements lose sta- tistical power in complex architectures and provide qualitative analysis of the underlying causes.
We reframe neuronal health evaluation through Grama, a gradient- based metric that quan- tifies learning potential independently of architectural complexity, and demostrate that neuronal learning capacity degradation affects even state-of-the-ar network developments.
We develop (ReGr aMa), an efficient neuron resetting mechanism guided by GraMa, which effectively restores neuronal activity across wide range of network architectures.
We conduct extensive experiments on MuloCo [Brockman et al., 2016), DeepMind Control Suite [Tassa al., 2018], showing that GraMa- GraMa-guided resetting improves performance and learning stability across diverse architectures.
2
Background
reinforcement learning (RL) problem is typically formalized a Markov Decision Process (MDP), defined by the tuple (S, A, P, R, 7). where S is the state space, A the action space, P the transition probability function S A X + [0, 1]. R the reward function S A R. and E 10. 1) the discount factor. The state-action value function under policy π is given by: En E> Y ((ss,, 0z) 190 s, ag a). The objective is to find a policy that maximizes the ex- pected return (= a) for each state In actor-critic-basee
deep RL, the Q-function is approximated by a neural network Qo with parameters 6. During training, the agent interacts with the environment and stores trajectories in replay buffer D. bini-batches sampled from D are used to update Qo by minimizing the temporal difference (o)):L(()) = (s.,))))) [[Qa(s, a) Q (s, a))?] where the target is given by Q (s, a) = R(s, a) Q (s))')) and 0, I denote the parameters of target critic and actor networks, respectively.
Neuronal activity measurement based on activation value. Recent studies have identified that the dynamic and non-stationary nature of RL objectives can cause neurons to permanently lose their activity [Lu et al., 2018], impairing the network's ability to fit new data and thereby limiting learning progress [Nikishin et al., 2022, Ceron et al., 2024]. This phenomenon, referred to as the dormant neuron phenomenon, has been quantitatively characterized using the т r-dormant neuron ratio [Sokar et al., 2023], and serves et al., 2024, Liu et al., 2025a]. Specifically, a neuron in layer l is considered r-dormant if its normalized activation (see Eq. falls below threshold where H is the number of neurons in layer f, and D denotes the data distribution. (x) represents the activation value neuron i given input x.
metric for assessing neuron-level plasticity [Xu et al., 2023, Qin
Egan [((x))
S
HJ Eken EAND (x)
3 Related Work
Dynamic objectives may cause irreversible damage to the neuronal activity of networks during training [Lyle al., 2023, Nauman et al., 2024a, Castanyer al., 2025], which also significant in the field of multi-agent RL [Qin et al., 2024] and visual deep RL [Ma et al., 2024]. This issue may lead the phenomenon where deep learning agents progressively lose their ability fit new data [Abbas et al., 2023], and limit their capacity for continual learning [Elsayed and Mahmood, 2024]. In the field of Deep RL, there are some factors that have been found to have direct correlation with the activity of neurons [Juliani and Ash, 2024, Mayor et al., 2025, Castanyer et al., 2025]. Among these, activation functions play partieularly important role, recent findings indicate that replacing ReLU activation can help preserve neuron-level learning capacity [Abbas et al., 2023].
3

---

Fuslon
ReDo
ReLU
Leaky ReLU
tracive neuton
Normalization
Activeted neuron
ReDo
Inexpressive
Still expressive
Fusion obscures the perception of each branch
r fallure causes excessive reset
Modify output leads to misjudgment
(a) The multi-branch structure
(b) Non-ReLU activation functions
(c) Normalization
Key techniques involved in advanced models may reduce the efficiency of activation- metries. (a) The final activation values observed after branch fusion no longer accurately reflect the individual contributions of neurons in each branch. (b) A near-zer zero value fails to represent the expressivity with non-ReLU activation function. (c) Normalization confuses ReDo by modifying the neuron's outlier output.
Figure 2:
Certain loss functions, especially those incorporating L.2 regularization, have been found to mitigate the network's s activity degradation [Kumar et al., 2023]. Another relevant line of research explores architectural adjustments such as scaling up [Nikishin et al., 2023] or topology growth [Liu et al., 2025a]. These methods can introduce modules that enhance the proportion of active, high-quality neurons. A separate and increasingly influential line of research focuses on directly manipulating model parameters to recover their activity. Resetting neural networks, either periodically or selectively, has been shown to be both effective and casy to implement [Farias and Jozefiak, 2025]. For instance, Nikishin al. [2022] demonstrates the benefits of periodic resets for continual learning.
Similarly, Ma et al. [2024] shows that selectively resetting specific layers can improve stability without sacrificing expressive capacity. At finer granularity. ReDO [Sokar et al., 2023] introduces a neuron-level resetting scheme that outperforms earlier coarse strategies in terms of stability and precision. However, ReDO's 's reliance on ielvaatio-bbsed quantization limits its applicability to more complex architectures [Lyle al., 2023, 2024]. To address this limitation, we introduce novel neuronal activity quantification approach based on gradient mugnitudes, enabling generalized activity estimation and recovery across arbitrary network designs. Notably, Ji al. [2024] provided the first empirical evidence in model-free reinforcement learning of strong correlation between the internal gradient dynamics of the agent's policy network and its learning capacity, offering compelling motivation for the present study.
4 Gradient Magnitude based Neuronal Activity Metric (GraMa)
In this section, we qualitatively investigate ReDo's limitations from the perspective of architectural composition Sec. 4.1. We then introduce GraMa (Sec. 4.2), novel neuronal activity metric that redefines the statistical objective from activation values to gradient magnitudes. Next, we perform an in-depth analysis of the characteristies of low learning capacity neurons using GraMa, and provide both theoretical and empirical analyses of the performance similarity between GraMa and ReDo on simple architectures. Finally, analyze the advantages of GraMa in complex architectures.
4.1 Misalignment of activation- neuron activity detection in modern architectures
The widely adopted activation- neu- activity measures [Sokar et al., 2023] operate on the assumption that a neuron's 's activation magnitude directly corresponds to its contribution to learning. Motivated by the development of more complex network structures in language and vision domains, recent trends in deep RL have moved be- yond simple serial MLP architectures with ReLU activations towards more sophisti- cated network designs for scaling. However, this assumption becomes increasingly frag- ile in modern complex architectures which are rapidly becoming standard practice. We reveal misalignment between activation values and actual learning potential, leading to inefficient neuron identification.
High learning capucily
High learning capacity
0.5
0.3
0.4
0.2
0.3
0.2
High expressivity
High expreesivity
(a) ReDo
(b) ReGraMa
Figure 3: ReDo fails reset neurons with low learning capacity (red), while also incorrectly resets the neurons with high learning capacity (dark blue), including those with both high learning capacity and high expressiveness (white). This plot shows proportion of resetting 4 neuron types during the same reset step in the Dog Walk task.

---

Motivating example. To systematically analyze this misalignment, we evaluate BRO-net -ne [Nauman al., 2024b], a recently proposed architecture, on the challenging Dog Walk task [Tassa al., 2018]. We categorized neurons into four quadrants based on their expressive capacity (measured by activation magnitude) and learning capacity (measured by gradient magnitude), which we introduce in more detail in Sec. 4.2. Neurons are ranked according to these two criteria, and we select the top 25% neurons in each ranking, so as to provide an understanding of the types of neurons reset by each method. The results in Fig. demonstrate that ReDo cannot accurately identify neurons that have genuinely lost their learning capacity (red) based on their activation values. More importantly, ReDo mistakenly resets significant number of neurons with high learning capacity that less expressive at the moment (dark blue).
Analysis. addition to the primary factor statistical objectives, we identify three key archi- tectural features that may undermine the reliability of activation-based neuron dormancy detection. Fig. provides an intuitive illustration of each. (i) Multi- baali--ranch network structures. In modern architectures like ResNets [He et al., 2015], information flows through multiple branches before being fused and passing through activation functions. This architectural pattern introduces a critical problem: the final activation values observed after branch fusion longer accurately reflect the individual contributions of neurons in each branch. Our experiments with Resnet-SAC [Shah and Kumar, 2021] shown in Fig. 4 (a) confirm this effect, where the results suggest that activation-basse methods lose their ability to identify dormant neurons and limits the performance of the agent. (ii) Non-ReLU activation function. The interpretability of activation values is highly dependent on the specific activation function used. While ReLU creates a clear distinction between "dead" (output=0) and "active" neurons, this clarity breaks down when considering alternative activation functions. For example, with Leaky ReLU, neurons may remain expressive and contribute to learning even when their pre-activation values are negative.
As a result, solely relying on activation values to measure neuron dormancy becomes unreliable and may lead to misidentification. Our experiments replacing ReLU with Leaky ReLU in SAC (Fig. 4 (b)) demonstrate this problem, where the activation-baxs based method (ReDo) shows considerable reset activity during early learning with lower performance, indicating that struggles to establish meaningful dormancy criteria. (iii) Normalization layers. Normalization techniques modify the distribution of neuron outputs before they pass through activation functions. This process adjusts outliers and rescales values across entire layers, causing post nost-nommalizaion values to lose direct correspondence with individual neuron functionality. Thus, activation values measured after normal- ization may no longer accurately reflect neuron's 's learning capacity. Our experiments incorporating layer normalization into SAC, Fig. 4 (c) reveal that the activation-bart method struggles maintain consistent neuron assessment, whereas our method consistently identifies relevant neurons throughout the learning process.
ReDo
RcDo
ReDo
ReGroMa
Parcentage of Raset
ReGraMa
Percentage of Recet
ReGraMa
Normaiized Percentage
X2
50
35
Stops
Stess
Steps
ReDo
ReDo
ReDo
ReCinMMa
Normellzed Score
ReGraMa
NOFFISIZZO Score
ReGraMa
Steps
Steas
The multi-branch structure
(b) Non-ReLU activation functions
(c) Normalization
Figure 4: Empirical validation corresponding the three cases of Fig. Top records the proportion of
reset neurons. Bottom row shows performance, Each curve represents average over seeds (Dog Walk).
5

---

CRONTOR
GraMa
641
ReDo
2.814
1-dormant neuron rallo
602
ReGraMaa
ReGronia
ACT3
Varila
Accurnsy
Vanila
CraMa
2034
Fpochs
Epochs
Medel deyth
Figure 5: (Left) Execution time comparison based on BRO-net (RTX3090 GPU); (Right) The results verify that ReGr aMa is as effective as ReDo on traditional network architectures. The backbone, hyperparameter settings and number of seeds are the same for all experiments.
4.2 Gradient Magnitude based Neuronal Activity Metric (GraMa)
Learning activity directly related to the gradient received by a neuron during training [Zhou and Ge, 2024]. A natural and simple way to enhance ReDo-like metrics is to count how many neurons in each network layer fail to obtain meaningful gradients from the current sample batch. Given an input
distribution D, let V [(x)]
denote the gradient magnitude of the neuron in layer l under input
D and H' be the number of neurons in layer l. Based on Eq. 1, we compute a learning capacity score for each neuron using the normalized average of the corresponding layer e, as shown in Eq. 2.
EXED [n)
G
(2)
F Eken BRED WV L(x)
GraMa determines that neuron i in layer L is inactive when G < т. GraMa has the same form as the dormant neuron ratio, but redefines its core signal, using gradient magni- tudes V [xx) rather than activation values (z). The pre-set threshold allows to detect neurons with outlier gradient magni- tude. Since gradient information is available in the tensor after backpropagation at each step, there is need to store the intermediate outputs each neuron during forward com- putation, which is required by activation-base based metrics. This makes GraMa lightweight, as verified in Fig. 5 (left).
Algorithm 1: ReGraMa
Input: 0, threshold т, frequency A, while < maximum training time do Update 0 with regular RL loss; if t mod A; == then for each layer do for eachneuron i do Calculate G²
> Eq.
if G S then L Reinitialize neuron i;
Resetting neurons guided by GraMa (ReGraMa). Neuron resetting is a simple technique widely used to preserve the learning capacity of deep RI. agents [Nikishin et al., 2022]. Our approach follows the ReDo pipeline [Sokar et al., 2023]. as outlined in Algorithm 1: during training, we periodically quantify the activity of neurons in all layers using GraMa; any neuron with G is considered inactive and reinitialized. Specifically, reinitialization involves resetting incoming weights to the original weight distribution, while outgoing weights we found that although LayerNorm layers contain only two learnable parameters, they tend to be quite inactive. As result, ReGraMa is deployed for these parameters in our official implementation, leading to performance improvements that warrant further analysis. Another avenue worth exploring the potential for resetting the input weights and biases of neurons.
zero. It is worth noting that, empirically,
The ratio of low learning capacity neurons is inversely related to performance. We control the number of reset neurons in ReGraMa to evaluate the performance of four agents with varying proportions of low learning capacity neurons on the Dog Walker task. Results in Fig. 6 (a) indicate that performance degrades significantly the ratio of low learning capacity neurons increases.
Neuron loss learning capacity irreversible. We use GraMa with a threshold of T = 0.0095 to sample 1000 inactive neurons in the pre-training period of vanilla agent. We then trace the change

---

progress
low cepacity
Proporiion
1007
0.001 0.00e boss tide aike usee waar udis 1o00
Distribution learning capacity
(h)
Figure 6: (a): Low learning capacity neurons have direct negative impact on agent performance. X-axis represents the normalized performance, and the Y-axis denotes the BRO-nets with different proportions of low learning capacity neurons. (h): Neuron loss of learning capacity is irreversible. x-axis denotes the number of neurons under each GraMa score, and the y-axis shows the training phase. (define 0.0095 active).
of their learning capacity scores and analyze the score distribution over time, as shown in Fig. (b). The results indicate that none of the sampled neurons exceed the threshold of 0.0095 as training progresses. This suggests that such neurons are unable to recover their learning ability independently, further underscoring the importance of resetting neurons with low learning capacity.
Equivalence of ReGraMa to ReDo in traditional architectures. We theoretically analyze the simi- larities between ReGraMa and ReDo traditional architectures (MLP with ReLU activations [Sokar et al., 2023]) and draw the following conclusions:
Theorem 1. If neuron is dormant (s = 0), then both Va.f f and G = 0.
Proof. From the dormant neuron formula (Eq, 1), we can conclude that:
ExeD (=)) 0.
hx(z) = This means the neuron is almost never activated on the dataset D. The derivative of the ReLU [Agarap, 2018] activation function is:
Since [n((z)) 2 0, [+(()) +
(hf(z) (z((2)
{
if 2f 0,
t if)
represents the output after passing through the activation function. Thus, if (f())) = 0. then (x) < 0, and during backpropagation, the gradient turns to zero:
0f dh{(z) (x) dh!
0f
ahf(x)
f = EXED
= EXED
- (x) = O.
(h)(x) 0z((x)
Takeaway. We prove that, in the traditional architecture (MLP with ReLU), neurons that are identified inactivate by ReDo will also be identified as such by ReGraMa.
Empirical verification. We trained a traditional fully connected network with ReLU on the CIFAR- 100 benchmark [Krizhevsky, 2009], following the continuous learning experimental setup in Dohare et al. [2024]. Every 15 epochs, a new category of data is added to the training set, requiring the agent classify samples from the whole data distribution. The two gray curves in Fig. (right) show that the vanilla agent's accuracy gradually declines as training progresses, indicating loss of learning ability. Meanwhile, the proportion of inactive neurons detected by GraMa increases over time and fluctuates with the same periodicity as the accuracy curve, with both stabilizing around epoch 150. To further validate ReGraMa, we conducted a two-part intervention study: (1) We reset neurons identified as inactive by the -- dormant neuron ratio (ReDo) whenever new data category is introduced. The resulting pink curves Fig. show that resetting dormant neurons can both improve performance and reduce GraMa ratio. (2) We then reset neurons flagged by ReGraMa as having low activity. This
7

---

intervention produced improvements similar to those of ReDo, as illustrated by the orange curves in Fig. These results suggest that ReGraMa as effective as the ReDo metric in identifying neuronal inactivity in simple network architectures.
ReGraMa identifies inactive neurons more ef- fectively in complex architectures. When moving to more complex architectures, reset- ting neurons based solely on activation values becomes less reliable. In deep networks with normalization layers, residual connections, or context-dependee features, activations can fluc- tuate substantially even for neurons that are not meaningfully contributing to learning. As result, activation magnitude alone is a noisy indicator of long-term inactivity. To examine this limitation, we compared activation-ban and gradient- based criteria by pruning the cor- responding neurons from a complex BRO-net agent and evaluating the resulting performance.
Nortatteed
ReGraMa pruning Standard
ReDo pruning Standard
Figure 7: Pruning neurons identified inactive by ReGraMa during the training has less impact the performance Dog Walk. Standard denotes vanilla agent. Curves show the average over four seeds.
The degree of post-pruning performance degra- dation reflects both the relevance of the pruned neurons and the reliability of the underlying metric. As shown in Fig. 7, pruning neurons identified by ReGraMa causes only minimal per- formance loss, indicating that gradient magnitude provides more stable and task-relevant measure of neuronal inactivity. This finding suggests that ReGraMa detects structural inactivity-neurons that remain consistently uninformative during optimization- rather than transient activation noise. Such stability is particularly advantageous in large, modular architectures, where distinguishing genuine inactivity from context-specific silence is critical for effective pruning and interpretability.
Experiments
We conduct series of experiments to investigate whether ReGraMa can mitigate neuronal activity loss and enhance performance. Specifically, we evaluate the effectiveness of ReGraMa across three representative and widely adopted architecture types: (i) the residual network-bard based policy (Sec. 1), (ii) the online policy parameterized by a diffusion model (Sec. 5.2), and (iii) the MLP policy featuring various activation functions (Sec. 5.3). Finally, we verify the robustness of ReGraMa with respect to the threshold (Appendix B).
AdCe
RaDe
NeDo
AcCrssaa Vantil
Racrstra
Warll
Veris
Veria
Kpyy
Soon
N00
Varila
Vuriia
Vuriaa
Verta
Figure 8: Performance and neuron inactivity with the default BRO-net size. (Top row) Episode return across four environments (Humanoid Stand, Dog Walk, Dog Stand, Humanoid Walk). (Bottom row) Corresponding proportion of inactive neurons. ReGraMa consistently achieves higher returns while maintaining fewer inactive neurons compared ReDo and the vamilla baseline, demonstrating effectiveness in stabilizing learning dynamics. Results are averaged over four seeds.
8

---

5.1 Residual Net-based Policy
Experiment setup. Recent studies [Nauman al.. 2024b, Lee et al., 2025a] have shown that integrating residual modules into deep RL agents can significantly improve representation capability on complex visual tasks. We choose BRO-net [Nauman al.. 2024b] as a representative baseline and evaluate all methods on four challenging tasks from the DeepMind Control Suite [Tassa et al., 201 8]. All the algorithm parameters follow the default settings. We set the empirical threshold r = 0.01 for ReGraMa and use the same ReDo's hyperparameters. The reset period is fixed at 1000 steps. Further details are provided in Appendix A.
Main results. Results in Fig. 8 show that ReGraMa can accurately reset the neurons with low activity in each stream of the multi-branch network, thus effectively maintaining the learning capacity of the deep RL agent on the four complex tasks and enabling continual learning. In contrast, ReDo performs poorly. This performance gap arises because, as discussed Sec. 4.1. ReDo misidentifies inactive neurons in multi-branch networks, undermining the effectiveness of its reset schedule.
ReGraMa
ReDo
Network scaling. To assess scalability, we evalu- ate both ReDo and ReGraMa under increased network depth using the BRO-net architecture across four chal- lenging environments: Dog Walk, Dog Stand, Hu- manoid Walk, and Humanoid Stand. As the network grows deeper, training stability and gradient signal propagation typically become harder to maintain, of- ten leading degraded performance lization of added capacity. In this context, we analyze whether the gradient-basee reset criterion of ReGraMa remains effective under larger model scales.
Baanngefron froel Recur
inefficient uti-
Depth Layers) Figure 9: ReGraMa is more robust under net- work scaling. Results averaged over four DMC tasks with 12 seeds per method.
Results show that ReGraMa consistently preserves stable performance improvements network depth increases, demonstrating that its gradient-drive measure generalizes well across scales. In contrast, activation-based resetting (ReDo) fails to exploit the additional representational capacity, showing marginal gains at best and even degrading performance in the 2x model. These findings indicate that ReGraMa scales more gracefully with model size, effectively leveraging deeper architectures without introducing instability.
5.2 Diffusion Model-Based Policy
Van
Experiment setup. Recent works have demon- strated that diffusion models, due their strong expressiveness over multi-modal distributions, can significantly improve RI. performance on complex control tasks [Chi et al., 2023]. We use DACER [Wang al., 2024], a recent online diffusion pol- icy, as a baseline and test ReDo and ReGraMa on two MuJoCo-v4 tasks from the original paper. DACER relies Unet backbone with Swish activations. All parameters follow official defaults. Hyperparameters for ReGraMa and ReDo are the same in Sec. 5.1, and the reset period is fixed 1,000 steps. Full details are in Appendix A.2.
ReDo
Normalized Score
Figure 10: Normalized scores for Ant and Walker2d. Boxes show 4 seeds, whiskers in- dicate min/max, the midline denotes the median.
Results. As shown in Fig. 10, ReGraMa maintains robust and consistent perfor- mance on this complex architecture, whereas ReDo provides only marginal im- provements. This discrepancy explained by the ratio of inactive neurons in Fig. 11, which reveals that ReDo fails to reset neu- rons that have lost learning capacity. In contrast, ReGraMa accurately identifies and resets low-activity neurons, preserving the agent's 's ability to learn and improving overall performance.
Cratta
Steps
Stepe
Figure 11: Proportion of inactive neurons during training across two MuJoCo tasks (Ant and Wdlker2d), ReGraMa maintains a consistently lower ratio of inactive neurons.

---

evaluate the robustness Epicode Return
Epitods
ReDo ReCinobee Vanila
ReDo BeTraMe Variaa
ReDo
BeEzahln
Varila
GraMa
(8) ReLU
(b) Sigmoid
(c)
Figure 12: Efffct of activation functions on performance and neuron inactivity. (Top row) Episode retum
across training under different activation functions (ReLU, Sigmoid, Tanh). (Bottom row) Corresponding proportion of inactive neurons. ReGraMa maintains higher performance and lower inactivity levels across all activations, indicating that gradient-hased resetting generalizes effectively beyond specific nonlinearities. Curves are averaged over four seeds.
5.3 Activation Function Variants
Experiment setup. Nadimpalli al. [2025] shows that saturated activation functions in the hidden layers may improve RI. agents' performance. Following their setup, we replace ReLU in SAC with Tanh and Sigmoid, keeping all other parameters as default. This enables of ReGraMa across various activation functions while minimizing the influence of extraneous factors on the experimental outcomes. We set for ReGraMa, and configure ReDo as Sec. 5. All methods are evaluated on the challenging Ant task. Hyperparameter are provided in Appendix A.3.
Results. Results in Fig. 12 show that ReGraMa accurately identifies low-quality neurons, mitigating neuronal inactivity and avoiding instability caused by false reset. While ReDo performs well under the ReLU, its performance degrades significantly with other activation functions, sometimes failing below vanilla SAC. In Fig. 12(c), although ReGraMa outperforms ReDo under the Tanh activation function, the proportion of inactive neurons gradually increases during training. We leave further investigation into whether this behavior stems from the perspective of the activation function reset mechanism to future work.
Discussion
This research focuses on the critical issue of neuronal activity loss during training in deep RL agents. show that the commonly used r-dormant neuron ratio (ReDo), which relies on neuron activations, struggles to capture learning activity modern, highly parameterized agents. This limitation arises because activation sparsity does not directly reflect a neuron's 's contribution to learning. We shift focus from activations to gradients, and introduce GraMa, simple, efficient, and vichitecture-agnosse metric based on gradient magnitude. GraMa enables accurate tracking of neuron-level learning dynamics across broad range of network types, including residual and diffusion-based policies, and substantially recovers learning activity via guiding neuron resetting (Sec. 4.2). Our findings suggest that even high-capacity policies in deep RL suffer from underutilization at the neuron level, which may hinder generalization, multi-task transfer, and continual adaptation as in supervised learning [Dohare et al., 2024]. By providing lightweight diagnostic and intervention tool, GraMa opens the door to more principled approaches for maintaining continuous learning ability in deep RL agents.
Limitations. Our evaluation is limited to three representative neural architectures due to computa- tional constraints. Future work will extend GraMa more complex settings, including large-scale transformer policies and multi-task environments. We aim for GraMa serve as practical tool for diagnosing and preserving learning capacity in deep RL agents.
10

---

This work is supported by the National Natural Science Foundation of China 62406266. The authors would like thank the reviewers for providing valuable feedback on the paper. We would also like to thank the Python community Van Rossum and Drake Jr [1995], Oliphant [2007] for developing tools that enabled this work, including NumPy Harris et [2020], Matplotlib Hunter [2007], Jupyter Kluyver et al. [2016], and Pandas McKinney [2013].
Acknowledgment
Broader Impact This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. ArXiv, abs/2303.07507, 2023. URL https:
//api. seapnissematicccchoo.. :rt.:/oorrpSS11:2235577755 257504763.
Abien Fred Agarap. Deep learning using rectified linear units (relu). ArXiv,
2018.
URL https: htappp::/app/ /api. semanti Ceemaanniissscllarrooo////orooo 49990355 4090379.
Jordan T. Ash and Ryan P. Adams. On the difficulty of warm-starting neural network training. ArXiv, abs/1910.08475, 2019. URL https://api. www.nte hwtp://rr.rrr/rroppus00000:/010
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learn- ing. ArXiv, abs/1912.06680, 2019. URL https: semantseenttt ticccholarr or ortg/or pusID: 209376771.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. ArXiv, abs/1606.01540, 2016. URL https://api. semanticscholar org.org///ooo :C:CCopsuII::11992222
Roger Creus Castanyer, Johan Obando-Ceron. Lu Li, Pierre-Lue Bacon, Glen Berseth, Aaron Courville, and Pablo Samuel Castro. Stable gradients for stable learning at scale in deep reinforce- ment learning, 2025. URL https:///aaxiii org. abs/2506. 15544.
Johan Samir Obando Ceron, Aaron Courville, and Pablo Samuel Castro. In value-basee deep reinforcement learning, pruned network is good network. In International Conference Machine Learning, pages 38495-38519. PMLR, 2024.
Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. ArXiv, abs/2303.0411 .0.133, 2023. URL htttp:://ppi.ssmanttccchhll wttpscch.......oooouossssilee.e. :ttg//orppsID::000 257378665.
Shibhansh Dohare, J. Fernando Kermandez-Garcia. Qingfeng Lan, Parash Rahman, Ashique Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632:768- 774, 2024. URL https: //api. semanticschollat www.erorgoooooooooe wtgg/CorpssID:....oooooo..pp 259251905.
Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy. ArXiv, abs/2502. 16112 2025. URL https://appi semanticscholar.co
JorpuSID:276408472
Mohamed Elsayed and Rupam Mahmood. Addressing loss plasticity and catastrophic forgetting in continual learning. In The Twelfth International Conference on Learning Representations, 2024.
URL https: //tttrei///ppereiiee.. wwwpppeewwwwweeeeeeeeeeeeeeeeeee nett/forrummidsss////ooo
Vivek Farias and Adam Daniel Jozefiak. Self-normalized resets for plasticity in continual learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
// //ooorreveew.ee//oorrrm/// wtt.:e/eoooouumiiiceeeuuuuzz
net
11

---

Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Coumaapeau, Eric Wieser, Julian Taylor, Sebastian Berg. Nathaniel Smith, al. Array program- ming with numpy. Nature, 585(78855)3357-3335 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015. URL https://app https://ap..semantisschooo
semanticscholar .or.aarrrrrgg/ge www.or///ooooo org/CorpusID: 206594692.
John Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(33):
90-95, 2007.
Tianying Ji, Yongyuan Liang, Yan Zeng, Yu Luo, Guowei Xu, Jiawei Guo, Ruijie Zheng, Furong Huang, Fuchun Sun, and Huazhe Xu. Ace Off-policy actor-critic with causality-aware en- tropy regularization, ArXiv, abs/2402. 14528, 2024. URL patpss//aaps semanticscollar org/ CorpusID:267782226
Arthur Juliani and Jordan T. Ash. A study of plasticity loss in on-policy deep reinforcement learning. In The Tiihth-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:: http:://pppeeevieeeeee ew. net/ Md--mmidi-MMMMMMMo
Thomas Kluyver, Benjain Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter Development Team. Jupyter Note- books-a publishing format for reproducible computational workflows. In IOS Press, pages 87-90.
2016. doi:
Alex Krizhevsky. Learning multiple layers of features from tiny images. In Arxiv, 2009. URL mttps://apl. semanticscholleee .at.aaarrrg////e mrgg/CorpuIID:: 8268744.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. In CoLLAs, 2023. URL https:/ps://ap semanticscholar. org/CorpusID:222 261 261076021.
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jacgul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. ArXiv, abs/2406.02596 2024. URL https: //api. semanti cichanntcccclarr... org/ Cor PoorpusI:: 270258586.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025a. URL https: /ttp::///ppereeeee http///ppee/eeeeeeeeeeeeeeeee/ee wtw.rrvveee.neeoooumeeeiiee net/forummid=jX.. iXLiDKsuD..
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jacgul Choo. Hyperspherical normalization for scalable deep reinforcement Iearning. ArXiv, abs/2502. 2025b. URL
https://api. www.nticscoolliae.... org/CorpusIDD 276558227....
Yang Li, Zhichen Dong, Yuhan Sun, Weixun Wang, Shaopan Xiong, Yijia Luo, Jiashun Liu, Han Lu, Jiamang Wang. Wenbo Su, Bo Zheng, and Jun-Feng Yan. Attention illuminates Ilm reasoning: The ceellan-and-nchoo rhythm enables fine-grained policy optimization. 2025. URL https: "lapi.semande i cscholar hwtphholarrrrgee//eeee Orgg/CorpusID: 2821 029660
Jiashun Liu, Johan Samir Obando Ceron, Aaron Courville, and Ling Pan. Neuroplastic expan- sion in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025a. URL https::/ http://v//ppeeeeiieeee nett/foruumidd22222227ff
Jiashun Liu, Johan S. Obando-Ceron, Han Lu, Yancheng He, Weixun Wang. Wenbo Su, Bo Zheng, Pablo Samuel Castro, Aaron C. Courville, and Ling Pan. Asymmetric proximal policy optimization: mini-critics boost IIm reasoning. 2025b. URL https: //api. semanticscholar. www.chollar.oo/oooppuuueee org/CorpusID: 281724081.
Rongrong Liu, Florent Nageotte, Philippe Zanne, Michel de Mathelin, and Birgitta Dresp. Deep reinforcement learning for the control robotic manipulation: focussed mini-review. ArXiv, abs/2102.04148, 2021. URL https:: //api.
12

---

Zihe Liu. Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong. Ju Huang, Jian Hu, Shengyi Huang, Siran Yang. Jiamang Wang, Wenbo Su, and Bo Zheng. Part i: Tricks traps? a deep dive into rl for IIm reasoning. ArXiv, 0is/2508.08221, 2025c. URL https: htpps///ap. api semanti www.nolcccccaarccc.ccooooaararre wwwschoaa..rrg/CCC. wrgg/orppsIDD: 280566935.
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Hao Jiang, Zifeng Gao, Yansong Tang. and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. ArXiv, abs/2505. 18719, 2025. URL https::/appl semanticscholar,
org/CorpusID: 27:::pppuuI::28044888000 278904856.
Lu Lu, Yanhui Su, and George Em Kamiaaakis. Collapse of deep and narrow neural nets. ArXiv, abs/1808. 0908.04947, 2018. URL nttps://apt. semant mnmaantiscchllrr www.cssa...oo//ooooo. www./orppssDD. 21981236.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, H. V. Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. ArXiv, abs/2402.18762, 2024. URL https://app semanticscholar wwt. ::tg:/orpsSD:::::0. 268063557.
Guozheng Ma, Lu Li, Sen Zhang. Zixuan Liu, Zhen Wang, Yixin Chen, Li Shen, Xueqian Wang, and Dacheng Tao. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviers net sttefforummmdddoR11s99
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Soft diffusion actor-critic: Efficient online reinforcement learning for diffusion policy. ArXiv, abs/2502.00361, 2025. URL https:
//apiissmanty semasniccccctihollrr
Cor 276894335599
Walter Mayor, Johan Obando-Ceron, Aaron Courville, and Pablo Samuel Castro.
The impact of
on-policy parallelized data collection deep reinforcement learning networks. In Forty-second International Conference on Machine Learning, 2025. URL http:::tp//pppott //pp:////e//eeeie...om.o net/forum? id=cnayzuZhSo.
Wes McKinney. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. O'Reilly Media, 1 edition, February 2013. ISBN 9789351 100065. URL http://tp// http:://www..eee amazon. com/exec
Kalyan Varma Nadimpalli, Shashank Reddy Chirra, Pradeep Varakantham, and Stefan Bauer. Evolving RL: Discovering new activation functions using LLMs. In Towards Agentic Al for Science: Hypothesis Generation, Comprehension, Quantification and Validation, 2025. URL
https://openrevieet het//forum?ic=H2xejuuu
Michal Nauman, Michat Bortkiewicz, Piotr Mitoš, Tomasz Trzciński, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning. ICML'24. JMgR. 2024a.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoš, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In NeurIPS 2024, 2024b. URL
Evgenii Nikishin, Max Schwarzer, Pierluca D' D'Oro, Pierre-Lue Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 2022. URL https://api.semanticcchhee semanticscholar Drgg/CorpssI:: 146811164.
Exgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and Andre Barreto. Deep reinforcement learning with plasticity injection. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL /ttpss//opeerrwwe wwt.wiwwwwweooommmiiiiiie jucDLW6G91.
Johan Obando Ceron, Marc Bellemare, and Pablo Samuel Castro. Small batch deep reinforcement learning, Advances in Neural Information Processing Systems, 36:26003-26024, 2023.
13

---

Johan Obando-Cerr Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. In Proceedings of the s1st International Conference on Machine Learning, pages 38520-38558 2024.
Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3): 10-20, 2007. doi: 10.1009//
Haoyuan Qin, Chennan Ma, Mian Deng, Zhengzhu Liu, Songzhu Mei, Xinwang Liu, Cheng Wang, and Siqi Shen. The dormant neuron phenomenon in multi-agent reinforcement learning value factorization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:////pppeeieeeeee wwwpeeeewwweeeeennneeeeeee met/forrmmidd=ANGrrrra
Allen Z. Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/openreviiie www.eeeewwwweeeeeeeeeeeeeeeeeeee dnntfforrmmmddmmmpatbbbbh22
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level efficiency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 30365-30380. PMLR, 23-29 mlr. wtwps///////seeeeeeeeeeeeeee wttp0//2cwwaaaeer23aa html.
Jul 2023. URL https: http::://proceedingg.
Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. ArXiv, 0as/2107.033880 2021. URL httpt::/ppppppt www.ssssccccccooarcrorooooo /t..:httti//aa.....aaaaccccooro.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300, 2024. URL https://app wwwanticscooolar.oooom org/
20741510
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145-32168. PMLR, 2023.
Ghada Sokar, Johan Obando-Ceron Aaron Courville, Hugo Larochelle, and Pablo Samuel Castro. Don't flatten, tokenize! unlocking the key softmoe's 's efficacy in deep rl. arXiv preprint arXiv:2410.01930. 2024.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. ArXiv, abs/1801 001880000000, 2018. URL https::///// /appi. semanticscholar. org/CorpusID: (atrrpusID:3355000 631 5299.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Yinuo Wang, Likun Wang. Yuxuan Jiang. Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang WU, Jingliang Duan, and Shengbo Eben Li. Diffusion actor-critic with entropy regulator. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: http///ppnrrveveeeeeeeeeee http:///pppeeeeeeeeeeeoorrr//eoe net/forum?id=10c1111 144r::i44111111
Timon Willi, Johan Samir Obando Ceron, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, and Pablo Samuel Castro. Mixture of experts in a mixture of rl settings. In Reinforcement Learning Conference.
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daum'è, e, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. ArXiv, abs/23 10. 19668, 2023, URL https: //api. http://ccc....oooooooooooo/ooaaa
14

---

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. ArXiv, abs/2107.09645, 2021. URL https://api. semanticscholar wwwee/orroooooeeeoo /
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang. Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Honglin Yu, Weinan Dai, Yuxuan Song, Xiang Wei, Haodong Zhou. Jingjing Liu, Wei Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yong- Xu Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system at scale. ArXiv, 0as/2503.144745 2025. URL https://api semanticscholar orgg/orrpsID:27711 2771100::27711222
Mo Zhou and Rong Ge. How does gradient descent learn features local analysis for regularized two-layer neural networks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:// nett//orrum/.ddx/xo001m2
15

---

A Experimental Details
A.1 Residual network based policy
BRO-net [Nauman et al., 2024b] is the first model-free algorithm to achieve near-optimal policies in the notoriously challenging Dog and Humanoid tasks. Its residual module-based architecture also gives it powerful ability to scale up, making it widely concerned [Leee et al., 2025b,a]. To this end, we choose BRO-net as the advanced agent for the multi-branch architecture to test the effectiveness of our metrics. And our implementation of BRO-net is based on the official implementation" 3.
BRO-net Architecture The detailed structure of the core block used BRO-net shown Tab. 1. The comprehensive Actor-Critic structure build by the above blocks is outlined in Tab. 2.
Table I: BroNetBlock Structure. Hy denotes the block's internal hidden dimension (c.g.. 256).
Step
Layer Configuration
1. FC Layer
Linear( Hs, H₂)
Norm + Act
LayerNorm( H4)) ReLU
3. FC Layer
Linear( H,, H₂))
Norm
LayerNorm(Ftb)
5. Residual
Output = Step 4 Output Block Input
Table Whole network architectures,
The structure BroNetBlock detailed in Table 1.
Layer
Actor Network
Critic Network (per Critic)
Fully Connected
(state dim,256)
(state dim action dim, 256) LayerNorm
LayerNorm
LayerNorm
Activation
ReLU
ReLU
BroNetBlock
Nx x BroNeiBlock
N X BroNetBlock
Fully Connected
(256, action dim)
(256, 1)
Activation
Tanh
None
Hyperparameter setting The shared hyperparameters the BRO-net algorithm utilized in all our experiments outlined in Tab. 3. Both Redo and Grama configured with the recommended values for (0.01 for Grama and 0.02 for ReDo) and the same reset frequency (every 1000 steps). To reproduce the learning curves shown in the main text, we advise using seeds ranging from 0 4. For the scale experiments, we increased the number of BroNetBlock from to 4 in both actor and critic.
Table 3: Hyperparameter settings for BRO
Hyperparameter
Value
Actor Learning Rate
1 x 10
Critic Learning Rate
1 10-3 3
Replay Ratio
2
Discount Factor (7)
0.99
Batch Size
128
Buffer Size
1 x 106
Actor BroNetBlock
1
Critic BroNetBlock
2
Reset Specific Parameters
Reset
0.01
Reset Frequency
1000
16

---

A.2 Diffusion model based policy
Network Architecture As one of the recent works to successfully construct online policies based on the diffusion model, Dacer has received extensive attention from the community and has been used as baseline by some recent studies [Dong al., 2025, Ma al., 2025]. To test the effectiveness of our metrics in the advanced diffusion model policies, we chose the official Swish activation function and U-net based Dacer baseline with complex architecure. We reproduce the version of the code that introduces the two neuronal metrics into the policy model, based on the official code of DACER*. Our detailed Structures were showen in Tab.
Table 4: Network Structures for DACER.
Layer
Actor Network
Critic Network
Fully Connected
(state dim + time embedding dim, 256)
(state dim + action dim, 256)
Activation
ReLU (256, 256)
ReLU
Fully Connected
(256, 256)
Activation
ReLU (256, action dim)
ReLU
Fully Connected
(256, 2)
Hyperparameter setting Our experiments adhere the hyperparameter listed in Tab. 5. for ReDo: 0.02; r for Grama: 0.01; reset frequency (every 1000 steps). To reproduce the learning curves shown in the main text, we advise using seeds ranging from 0 to 4.
Table 5: Hyperparameters DACER Training
Hyperparameter
Value
Actor Learning Rate
3 10 3 x 10-4
Critic Learning Rate
Alpha Learning Rate
3 x 10-2
Discount Factor (7)
0.99
Batch Size
256
Replay Buffer Size
106
Target Network Update Rate (r)
5 10-3
Policy Update Delay
2
Hidden Layer Size
256
Reward Scale
1.0
Reset Specific Parameters
Reset
0.01
Reset Frequency
1000
A.3 MLP-based SAC
Network Architecture We CleanRL for SAC (also Resnet SAC) implementation, which can be found at https:://st thub. cwm/ cww/wwwyzjnn/////coooooooooooozs (wwxyyjnn/leanrl.1 cleanrl. This library is reliable open-source resource for deep reinforcement learning, designed in PyTorch-friendee manner. And the detailed structure is shown in Tab. 6.
Hy Hyperparameter setting The shared hyp perparameeerrs for the SAC algorithm are summarized in Tab. 7. Note: We impose maximum reset percentage limitation of 5% exclusively for the Humanoid task.
17

---

Table 6: Network Structures for SAC
Layer
Actor Network
Critic Network
Fully Connected
(state dim, 256)
(state dim + action dim, 256)
Activation
ReLU
ReLU
Fully Connected
(256, 256)
(256, 256)
Activation
ReLU
ReLU
Fully Connected
(256, 2x action dim)
(256, 1)
Activation
Tanh
None
Table 7: Hyperparameters for the SAC
Hyperparameter
Value
Total Timesteps
3 x 106
Replay Buffer Size
1 x 10
Discount Factor (7)
0.99
Target Smoothing Coefficient (r)
0.005
Batch Size
256
Learning Starts
5 x 103
Policy Learning Rate
3 x 10-4
Q-Network Learning Rate
1 10-3
Policy Update Frequency
2
Target Network Update Frequency
1
Automatic Entropy Tuning
True
Reset Specific Parameters
Reset
0
Reset Frequency
1000
Max Reset percentage
5% (Humanoid)
B ReGraMa is more robust to the threshold т
To assess the robustness of both metrics across varying thresholds, we use BRO-net as the backbone and evaluate performance on the challenging DeepMind Control Suite Humanoid Walk task. By systematically varying the r following the recommended setup, we found that ReGraMa consistently outperformed ReDo (Fig. 13). This proves that, even with relaxed restrictions, ReGraMa has less tendency to reset incorrectly.
C Additional experiments
We further evaluate four challenging DeepMind Control (DMC) tasks with tensor-based observations. The results in Tab. 8 show that, under the BRO-net architecture, ReGraMa consistently outperforms all competing methods, highlighting its robustness in complex network architectures. ReBron [Qin et al., 2024], which explicitly accounts for both overactive and dormant neurons, also achieves strong performance, although it slightly degrades performance the Dog Walk task. In contrast, S&P [Ash and Adams, 2019] 9] and CBP [Dohare et al., 2024) provide modest but consistent improvements across tasks.
We additionally include new ablation studies in the updated appendix to investigate the interaction between ReGraMa and commonly used regularization techniques. Following the optimal weight decay configuration reported in Lyle et al. [2024], we evaluate the combination of ReGraMa with weight decay and L2 initialization (A 10 10-2) the DMC Quadruped Run task using DrQv2 [Yarats et al., 2021] for 3M environment steps.
The results in Tab. 9 indicate that combining ReGraMa with either L.2 initialization or weight decay individually further improves learning efficiency. Among these variants, 1.2 initialization, which preserves plasticity by regularizing the initial parameters, appears more compatible with our method.
18

---

ReGraMa
ReDo
Normalized performance
1.2
1.1
0.9
0.8
0.7
0.6
0.5
0
0.025
0.05
0.075
0.1
T
Figure 13: Sample uniformly from [0, 0.1.. Each bar is the average of 4 seeds.
In contrast, the simultaneous application of ReGraMa, L2 initialization, and weight decay does not yield additional gains. We leave deeper investigation of the underlying interactions between ReGraMa and other regularization strategies to future work.
Table 8: Normalized performance relative to the vanilla BRO-net policy on DMC tasks (mean + std over rums).
Method
Humanoid stand
Humanoid Run
Dog stand
Dog walk
vanilla BRO-net policy (baseline)
1.0
1.0
1.0
1.0
ReGraMa
0.03
1.16 0.08
1.12
1.08 0.04
ReDo
0.07
0.003
0.12
0.94 0.06
S&P
+
1.08 +
0.95
1.07 0.02
ReBorn
1.13 0.06
1.05
1.02
0.91 + 0.13
CBP
1.18 + 0.02
0.99 + 0.12
1.06
1.09 + 0.07
Table 9: Effect of combining ReGraMa with L2 initialization and weight decay on DMC Quadruped Run (3M steps, DrQv2).
Method
Quadruped Run 649.13 + 182.43
vanilla policy
ReGraMa
706 = 127.25
ReGraMa L2 init
742.36 127.31
ReGraMa + weight decay
751.31 + 94.26
ReGraMa + L.2 init weight decay
739.42 + 104.58
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations can be seen in Conclusion of main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Please refer to Sec.4 of the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We make our code available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
The code link can be found in abstret, and the hyperparameter setting can be found Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https:: //nips. cttppbccccppccceccccccccccccccct wetgggiiiddedduuiieeeeeeee iitobmisssinn ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The experimental detail can be found in Experiment part. The hyperparameter setting can be found in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are accompanied by error bars, confidence intervals. We detail the calculation of error bars, running steps and the number of seeds in the text description of each figure.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The Experiments Compute Resources can be found in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurIPS Code of Ethics https: //neuriss...pp..// http:////lllttttticcicleeeeeeee.
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: We make sure to preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is only about promoting technological innovation and does not have social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Ours paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package, and state which version of the asset is used (include a URL).
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy what should or should not be described.
220 for
26


Reviews:
[{"review_id": "9dmu6btASv", "reviewer": "Reviewer_AyZs", "summary": "This work proposes a metric for measuring inefficient learning neurons in single-agent reinforcement learning. Unlike ReDo's activation-based metric, this metric is gradient-based and can be applied to more complex networks and other activation functions besides ReLU. Good results have been obtained on many algorithms and benchmarks, such as MuJoCo and the DeepMind Control Suite.", "questions": "1. What is the distributional difference between neurons flagged as \"inactive\" by GraMa and \"dormant\" by ReDo? What is their overlap rate?\n\n2. Which part of neurons are reset to improve the performance of ReGraMa? Figure 3 shows that both ReGraMa and ReDo can reset neurons with low learning capacity and low expressivity. I want to know the impact of resetting these four types of neurons on performance. Such analysis is very important to prove the effectiveness of GraMa.\n\n3. The authors have shown that ReDo and ReGraMa are equivalent in MLP+ReLU. Why ReGraMa reduces GraMa significantly better than ReDo with exactly the same experimental settings in Figure 5?\n\n4. I checked the code you provided, and it seems that the reset is only performed on Linear, Conv2d, and LayerNorm. Although experiments were conducted on Residual network, Diffusion model, and MLP-based SAC architectures, most of them are combinations of MLP and ReLU. In critic networks, RNN is a commonly used structure, and I wonder if this method has any effect on this structure.\n\n5. The following methods [1,2,3] are all related to the plasticity or expressiveness of neural networks, such as CBP's utility is a discounted average product of a neuron's associated weights and activation. The related Baseline should not only include ReDo.\n\n[1] ReBorn: The Dormant Neuron Phenomenon in Multi-Agent Reinforcement Learning Value Factorization. Neurips 2024.\n\n[2] CBP: Loss of plasticity in deep continual learning. Nature 2024.\n\n[3] S&P: On warm-starting neural network training. Neurips 2020.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The gradient-based perspective is a meaningful departure from activation-based methods.\n\n2. The approach is simple, and the writing is clear and easy to understand.\n\n3. This metric is more lightweight and does not require additional overhead to store activation values.\n\nWeaknesses:\n\n1. The distinction between GraMa-detected \"inactive\" neurons and ReDo-detected \"dormant\" neurons is not rigorously analyzed.\n\n2. Although Theorem 1 proves that the roles of GraMa and ReDo are equivalent in simple architectures, there is no theoretical analysis in complex architectures with different activation functions.\n\n3. For complex network architectures and different activation functions, there are too few experiments as empirical verification, and they are not very convincing.", "was_revised": true, "final_justification": "The author's response addressed my concerns, but considering the other reviewers' comments, I maintain my score.", "rebuttal": "We thank the reviewer for the insightful suggestions and useful feedback on our work.  Please find our responses to each of the concerns below.\n\n**Weaknesses**\n\n> W1: The distinction between GraMa-detected \"inactive\" neurons and ReDo-detected \"dormant\" neurons is not rigorously analyzed. What is the distributional difference between neurons flagged as \"inactive\" by GraMa and \"dormant\" by ReDo? What is their overlap rate?\n\nThanks for raising this point! We have conducted additional experiments by performing a more thorough distributional analysis to examine the differences between neurons identified by each method. We evaluated this on the BRO-net architecture on DMC Humanoid Walk, and will include the full results in the new version.\n\n1. We measured the overlap between neurons flagged by ReGrama and ReDo. The average overlap rate is only **31.56%** (average of 3 runs), confirming that they identify fundamentally different neuron populations. \n2. We tracked the proportion of neurons reset by ReDo and ReGraMa throughout training. As shown in Table 1, ReDo performs aggressive resets early in training when activation patterns are naturally unstable (potentially leading to falsely identifying neurons). In contrast, ReGraMa shows a more progressive pattern (which resets more neurons as training evolves) that better aligns with the gradual loss of learning capacity (indicative of a more conservative and adaptive strategy).\n\nTable 1. Reset neuron ratios throughout training (average of 3 runs):\n\n| Method | during 1/4 training stage | during 2/4 training stage | during 3/4 training stage | during 4/4 training stage |\n| --- | --- | --- | --- | --- |\n| ReDo | $0.41\\pm 0.04$ | $0.24 \\pm 0.06$ | $0.32\\pm0.14$ | $0.35\\pm0.08$ |\n| ReGraMa | $0.13\\pm 0.07$ | $0.19\\pm0.03$ | $0.21\\pm0.07$ | $0.24\\pm0.11$ |\n1. We analyzed the layer-wise distribution for each method. Table 2 shows nearly opposite layer-wise preferences: ReDo predominantly resets neurons in input/output layers, whereas ReGraMa focuses more on the middle layers (where representational learning primarily occurs).\n\nTable 2. Layer-wise reset neuron ratios at step 500k (average of 3 runs):\n\n| Method | first 1/4 layers | [1/4, 2/4) layers | [2/4, 3/4) layers | last 1/4 layers |\n| --- | --- | --- | --- | --- |\n| ReDo | $0.35\\pm 0.04$ | $0.12 \\pm 0.07$ | $0.16\\pm0.09$ | $0.37\\pm0.08$ |\n| ReGraMa | $0.11\\pm 0.11$ | $0.38\\pm0.06$ | $0.33\\pm0.14$ | $0.18\\pm0.13$ |\n\n> W2: Although Theorem 1 proves … there is no theoretical analysis in complex architectures with different activation functions.\n\nThanks for bringing up this point. While Theorem 1 establishes equivalence for MLPs with ReLU activations, we acknowledge that comprehensive theoretical analysis for complex architectures (like ResNet) with arbitrary activation functions remains a significant open challenge in the field. Despite this, our approach has strong theoretical intuition: gradient magnitudes better adapt to any differentiable architecture’s computational graph, which can reflect neuron contribution regardless of the activation function. Our empirical results also strongly support this intuition, showing consistent performance improvements across diverse architectures and activation functions. We hope our work opens a new venue for future theoretical advances that can formally characterize dormancy detection across arbitrary architectures. \n\n> W3: For complex network architectures and different activation functions, there are too few experiments for empirical verification.\n\nWe have expanded our evaluation to address this concern and further substantiate our conclusions, which highlight the efficiency of ReGraMa.\n\n- We compare ReGraMa with four related methods based on RNN architectures, where ReGraMa consistently outperformed strong baselines including CBP, ReDo and S&P (refer to **Q4**)\n- We further compare ReGraMa with the four mentioned recent methods, i.e., ReBorn, CBP, S&P and ReDo, based on BRO-net, where ReGraMa demonstrates superior performance (refer to **Q5**).\n- We conducted experiments on non-ReLU activation setting by comparing with the four additional baselines, where ReGraMa consistently maintains its performance advantage (refer to **Q5**).\n\n**Questions**\n\n> Q1:\n\nPlease refer to our response to W1 above.\n\n> Q2: I want to know the impact of resetting these four types of neurons on performance. Such analysis is very important to prove the effectiveness of GraMa.\n\nTo analyze which neurons contribute to ReGraMa’s performance gains, we added a study isolating the impact of resetting each neuron type from Fig. 3. The table below shows the following key findings:\n\n1. Resetting neurons with high learning capacity and low expressivity harms performance, where these neurons are frequently misidentified by ReDo as dormant due to their low activity. Furthermore, resetting those with high learning capacity and high expressivity causes the most substantial performance degradation. \n2. Resetting neurons with low learning capacity and high expressivity yields marginal improvements, while resetting low learning capacity + low expressivity neurons provides the largest performance boost.\n\nThis analysis demonstrates that ReGraMa’s effectiveness stems from its ability to discriminate between neurons based on learning capacity. \n\nTable. Performance impact (percentage change relative to the vanilla BRO-net policy, avg of 3 runs):\n\n| Reset | Humanoid run |\n| --- | --- |\n| low learning capacity & low expressivity | $+18$% ($\\pm 4$%) |\n| low learning capacity & high expressivity | $+6$% ($\\pm 7$%) |\n| high learning capacity & high expressivity | $-17$% ($\\pm 9$%) |\n| high learning capacity & low expressivity | $-9$% ($\\pm 2$%) |\n\n> Q3: Why ReGraMa reduces GraMa significantly better than ReDo with exactly the same experimental settings in Fig. 5?\n\nThere appears to be a misinterpretation of Fig5 (right), where the first figure in Fig5 (right) shows that ReGraMa (orange) and ReDo (pink) show nearly overlapping GraMa metrics throughout training with only minimal differences (approximately 0.001), and the 2nd figure in Fig5(right) shows that both methods achieve comparable accuracy curves on CIFAR-100. The slight variations are attributed to randomness within the model rather than algorithmic distinctions, which is expected and consistent with our theoretical analysis.\n\nOn the other hand, Fig. 5 (left) demonstrates a different message, which is about the significant computational efficiency advantage of GraMa over $\\tau$-dormant neuron calculation.\n\nPlease kindly let us know if we understand this question correctly. \n\n> Q4: I checked the code you provided, ... Does ReGraMa work with RNNs, a common architecture in critic networks?\n\nWe further extended ReGraMa to GRU-based RNN architectures based on ReBorn’s official implementation [1]. ReGraMa achieves excellent effect as ReBorn, consistently outperforming most baselines. \n\nTable: Performance on RNN-based agents (avg. of 3 runs)\n\n| Method | HalfCheetah | Ant |\n| --- | --- | --- |\n| vanilla RNN policy | $13007.57\\pm 236.11$ | $6963.57\\pm 436.11$ |\n| ReGraMa | $14184.28\\pm 115.48$ | $7723.95\\pm 104.81$ |\n| ReDo | $13275.94\\pm 209.21$ | $7091.53\\pm 162.74$ |\n| S&P | $13191.73\\pm 127.31$ | $7306.83\\pm 265.36$ |\n| ReBorn | $14106.64\\pm 219.27$ | $7529.42\\pm 147.33$ |\n| CBP | $13452.71\\pm 232.04$ | $7254.41\\pm 322.53$ |\n\n[1]: The dormant neuron phenomenon in multi-agent reinforcement learning value factorization.\n\n> Q5: Other plasticity-aware baselines (e.g., CBP, ReBorn) should be included beyond ReDo.\n\nWe have extended the experiment on BRO-net and TanH activation-based methods for comparison based on 4 DMC tasks. \n\nThe results based on the BRO-net are summarized in Table a. ReGraMa consistently outperforms other methods and highlights its robustness in complex architecture. On the other hand, ReBorn, which considers both over-active and dormant neurons, shows moderate improvements and struggles with certain tasks (Dog walk), while S&P and CBP show slight improvements.\n\nTable a. Normalized score relative to baseline based on BRO-net (average of 3 runs):\n\n| Method | Humanoid stand | Humanoid run | Dog stand | Dog walk |\n| --- | --- | --- | --- | --- |\n| vanilla BRO-net (baseline) | $1.0$ | $1.0$ | $1.0$ | $1.0$ |\n| ReGraMa | $1.21\\pm 0.03$ | $1.16\\pm 0.08$ | $1.12\\pm 0.08$ | $1.08\\pm 0.04$ |\n| ReDo | $1.17\\pm 0.07$ | $0.96\\pm 0.03$ | $0.92\\pm 0.12$ | $0.94\\pm 0.06$ |\n| S&P | $1.05\\pm 0.12$ | $1.08\\pm 0.04$ | $0.95\\pm 0.07$ | $1.07\\pm 0.02$ |\n| ReBorn | $1.13\\pm 0.06$ | $1.05\\pm 0.07$ | $1.02\\pm 0.06$ | $0.91\\pm 0.13$ |\n| CBP | $1.18\\pm 0.02$ | $0.99\\pm 0.12$ | $1.06\\pm 0.04$ | $1.09\\pm 0.07$ |\n\nIn the non-ReLU setting, ReGraMa and CBP demonstrate strong and robust performance, where our method achieves more stable performance across different tasks.\n\nTable b.  Normalized score relative to baseline based on TanH based ResNet-SAC (average of 3 runs):\n\n| Method | Humanoid stand | Humanoid Run | Dog stand | Dog walk |\n| --- | --- | --- | --- | --- |\n| vanilla TanH based ResNet-SAC (baseline) | $1.0$ | $1.0$ | $1.0$ | $1.0$ |\n| ReGraMa | $1.29\\pm 0.13$ | $1.16\\pm 0.04$ | $1.23\\pm 0.06$ | $1.04\\pm 0.02$ |\n| ReDo | $0.94\\pm 0.11$ | $0.93\\pm 0.08$ | $1.14\\pm 0.07$ | $0.98\\pm 0.03$ |\n| S&P | $1.13\\pm 0.04$ | $0.84\\pm 0.13$ | $0.95\\pm 0.03$ | $1.08\\pm 0.05$ |\n| ReBorn | $1.09\\pm 0.12$ | $1.02\\pm 0.04$ | $1.15\\pm 0.01$ | $0.94\\pm 0.07$ |\n| CBP | $1.22\\pm 0.06$ | $1.03\\pm 0.05$ | $0.97\\pm 0.06$ | $1.06\\pm 0.12$ |\n\nThese comprehensive results across different neural plasticity methods further demonstrate ReGraMa's robustness across architectures, activation functions, and task domains.\n\n---\nThanks again for reviewing our work and constructive feedback! We hope our responses address your concerns. We are happy to provide further clarification if needed."}, {"review_id": "EnBJYABvma", "reviewer": "Reviewer_p3Yd", "summary": "The paper addresses the problem of neuronal activity loss in deep-RL agents. The authors argue that the standard definition of $\\tau$-dormant neurons, which relies on relative activation magnitudes, fails to work for modern architecture components such as residual connections, non-ReLU activations, and normalization layers. The paper's central insight is that a neuron’s learning capacity is better measured by the magnitude of its gradients than by its activations.\n\nThe paper introduces **GraMa** (Gradient-Magnitude Neuronal Activity metric) to measure learning capacity. The authors adapt the well-established ReDo algorithm to use the GraMa metric instead of activation magnitudes(**ReGraMa**). The paper provides:\n\n- an analysis of the failure cases of ReDo for BRO-net,\n- a theoretical proof that GraMa subsumes $\\tau$ neuron dormancy in ReLU MLP layers, and\n- an evaluation against ReDo using a BRO-net agent on DMC, a diffusion-policy (DACER) on MuJoCo, and SAC with varying activations.\n\nCompared to ReDo and a vanilla baseline, ReGraMa maintains lower inactive-neuron ratios and yields higher returns or faster learning.", "questions": "- How dependent is ReGrama on different optimizers? I assume the experiments use Adam, but this is not stated anywhere in the paper.\n- How does ReGraMa interact with parameter regularization, such as weight decay or L2 init [3]?\n\n[3] Kumar, Saurabh, Henrik Marklund, and Benjamin Van Roy. \"Maintaining plasticity in continual learning via regenerative regularization.\" (2023).", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "My biggest problem with the paper is the lack of novelty: Using gradient norms instead of activation magnitudes has already been proposed (see weaknesses). Therefore, I would require an exceptionally thorough evaluation using strong baselines or novel insights into RL training dynamics to recommend acceptance. While there are some interesting insights (neuron learning capacity loss is irreversible), overall, I find the evaluation to be lacking: ReDo does not work well outside NatureCNNAtari, which is known (see weaknesses), and the baselines are weak. For me to move to an accept, the paper would need to benchmark against more baselines (see weaknesses) and uncover more about gradient behavior during RL training. As it stands now, the contribution is just too incremental for a NeurIps paper.\n\n### Strengths\n\n- Clear writing and thorough explanations. Easy to understand.\n- Using gradient norms instead of activations is a natural extension of ReDo.\n- Great presentation including high-quality figures.\n- ReGraMa introduces two new hyperparameters, but the authors show convincingly that their settings work broadly.\n- Code is provided.\n\n\n### Weaknesses\n\n- The paper's central idea (use gradient norms instead of activations) has been developed before as \"gradient dormancy\" measure for variable-strength Shrink & Perturb resets [1]. Therefore, I cannot consider this a novel contribution of the author's paper.\n- The baselines are weak. The authors only benchmark against ReDo, which is known to not work well outside of the NatureCNN-Atari regime ([2], see Figure 12 in the Appendix). For the included benchmarks, there should be more baselines (not all are needed): Full resets (as BRO includes them), Plasticity Injection (also no catastrophic forgetting), and resets according to [1]. \n- I'm not convinced that ReGraMa provides that much of a benefit outside of ReLU-style activations (see Figures 12). When using TanH, which is most common for state-based continuous control, the benefits of ReGraMa are minimal over a vanilla baseline (Figure 12).\n- The authors claim that \"Recent studies [Nauman et al., 2024b, Lee et al., 2025a] have shown that integrating residual modules into deep RL agents can significantly improve representation capability on complex visual tasks\". BRO is not a pixel RL agent; this statement must be clarified.\n- The introduction mentions catastrophic forgetting as motivation to discard reset approaches, but then proceeds to not evaluate catastrophic forgetting at all. The paper only evaluates off-policy approaches, which mitigate catastrophic forgetting through their replay buffer. Therefore, I believe reset approaches to be a valid baseline for ReGraMa.\n- Sometimes the writing is rather verbose. For example, the introduction is rather lengthy.\n\n\n[1] Ji, Tianying, et al. \"Ace: Off-policy actor-critic with causality-aware entropy regularization.\"  (2024).\n\n[2] Nauman, Michal, et al. \"Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning.\" (2024).", "was_revised": true, "final_justification": "The authors have addressed all my concerns regarding the evaluation in their rebuttal. Therefore, I raise my score by one. During the rebuttal, the authors highlighted their conceptual contributions. However, I still believe that ACE has previously developed the main fundamental insights behind gradient-based reset approaches. In my eyes, this work provides an interesting, but ultimately limited extension to ACE's findings. Thus, I cannot recommend acceptance.", "rebuttal": "Thank you for your thoughtful review and detailed feedback! We carefully considered your suggestions and conducted additional experiments and analyses to address your concerns. Below we respond to each point.\n\n### Weaknesses\n\n> W1: The paper's central idea has been developed before as \"gradient dormancy\" measure for variable-strength Shrink & Perturb resets in ACE. Therefore, I cannot consider this a novel contribution of the author's paper.\n\nThank you for highlighting this relevant prior work! After a detailed comparison and new empirical analysis, we believe our contribution remains novel and significant. We have added a discussion of similarities and differences between ACE and our work to the revised manuscript.\n\n**Key Differences with ACE [1] in focus, scope, and mechanism and our new insights:**\n\n- ACE primarily proposes a novel causality-aware entropy term to improve exploration, with a secondary and empirical gradient dormancy component to prevent overfitting that guides Shrink & Perturb resets (which is implemented in a “global” way, and please find our new experimental analysis below for further explaining the difference in this part).\n- In contrast, our work provides a comprehensive framework that provides novel insights for understanding neuronal learning capacity in deep RL. We provide the first rigorous analysis of why popular activation-based metrics fail with modern deep RL architectures and identify new phenomena. We show how gradient information resolves these issues and the widely adopted ReDo measure can be restored effectively if properly guided by gradient magnitude through our principled approach. \n- *We provide detailed analysis and new perspectives that reinterpret the failure as stemming from ReDo’s statistical objective, not its mechanism, and examine how modern DRL architectural components affect this previously overlooked phenomenon.*\n\n**Additional empirical comparison:** \n\nFollowing the reviewer’s suggestion for a more thorough evaluation, we have expanded our comparisons to include more recent strong baselines across diverse architectures (see **W2**) and non-ReLU activations (see **W3**); and investigated the differences in learning behavior (see **W5**) as requested.  Uncover the neuron-level gradient behavior during training from the time and model perspective(see response to **W1** of Reviewer AyZs).\n\nWe compare ReGraMa and ACE's reset method regarding the generalization on diverse architectures. The consistent improvements demonstrated by ReGraMa across diverse architectures validate our theoretical insights and demonstrate their practical significance.\n\nTable: Advantages over base policy at 3M step on  Walker Walk (average of 3 runs):\n\n| Method | BRO-net based | Unet-DACER based |\n| --- | --- | --- |\n| Baseline | $1.0$ | $1.0$ |\n| ReGraMa | $1.18\\pm 0.11$ | $1.07\\pm 0.05$ |\n| ACE’s reset | $1.09\\pm 0.14$ | $0.94\\pm 0.08$ |\n\nIn summary, our work provides a rigorous, novel, and in-depth foundation, analysis, and comprehensive evaluation of neuronal learning capacity in deep RL, addressing key challenges previously unexplored in the literature.\n\n[1]:  Ji, Tianying, et al. \"Ace: Off-policy actor-critic with causality-aware entropy regularization.\" *arXiv preprint arXiv:2402.14528* (2024).\n\n> W2: The baselines are weak... there should be more baselines (not all are needed): Full resets, Plasticity Injection, and resets according to ACE. \n\nWe have added the related strong baselines including full resets, Plasticity Injection and Shrink & Perturb on 4 DMC tasks (6M steps). All methods follow the official implementations. Results show that ReGraMa improves learning performance across all 4 tasks. The radical approach of full resets magnified performance uncertainty, as evidenced by a high standard deviation across different seeds. Plasticity Injection performs the weakest, likely because abrupt architectural changes can interfere with learning in complex agents. Shrink & Perturb reset shows stable but relatively minor gains in 3 tasks.\n\nTable.  Performance improvement over baseline (Average of 3 runs):\n\n| Method | Quadruped Run | Humanoid Run | Walker Walk | Humanoid Walk |\n| --- | --- | --- | --- | --- |\n| vanilla BRO-net policy (baseline score) | $1.0$ | $1.0$ | $1.0$ | $1.0$ |\n| ReGraMa | $1.08\\pm 0.03$ | $1.16\\pm 0.07$ | $1.07\\pm 0.04$ | $1.12\\pm 0.08$ |\n| Shrink & Perturb reset | $1.05\\pm 0.06$ | $1.04\\pm 0.05$ | $1.09\\pm 0.03$ | $0.97\\pm 0.06$ |\n| Full Resets (every 300k) | $1.03\\pm 0.08$ | $1.13\\pm 0.16$ | $0.92\\pm 0.14$ | $0.91\\pm 0.07$ |\n| Plasticity Injection | $1.08\\pm 0.13$ | $0.94\\pm 0.11$ | $1.03\\pm 0.09$ | $1.08\\pm 0.09$ |\n\n> W3: When using TanH, which is most common for state-based continuous control, the benefits of ReGraMa are minimal over a vanilla baseline (Figure 12).\n\nWe replaced MuJoCo with a more challenging task (Walker Walk) based on SAC. The results below (normalized to vanilla SAC) show clear benefits from ReGraMa, outperforming both ReDo and the vanilla policy:\n\nTable. Advantages over vanilla policy (Average of 3 runs):\n\n| Method | Walker Walk |\n| --- | --- |\n| TanH-SAC (baseline score) | $1.0$ |\n| ReGraMa | $1.18\\pm 0.11$ |\n| ReDo | $0.93\\pm 0.14$ |\n\nBesides, we followed ReDo's setup in Appendix B and adopted the sparse-SAC to avoid the noise caused by parameter redundancy. The results show that our method significantly improves the policy performance.\n\nTable. Advantages over vanilla policy (Average of 3 runs):\n\n| Method | Ant |\n| --- | --- |\n| 0.5 width SAC (baseline) | $1.0$ |\n| ReGraMa | $1.14\\pm 0.04$ |\n| ReDo | $1.09\\pm 0.06$ |\n\n> W4: The authors claim that \"Recent studies [Nauman et al., 2024b, Lee et al., 2025a] … improve representation capability on complex visual tasks\". BRO is not a pixel RL agent; this statement must be clarified.\n\nWe would like to clarify that our claim is based on Table 8 of the official BRO-net paper (camera-ready, OpenReview NeurIPS’24), which evaluates BRO-net on image-based tasks. We will highlight that we use the same encoder setup (DrQv2-style convolutional encoder) and BRO’s official PyTorch wrapper.\n\n> W5: The introduction mentions catastrophic forgetting as motivation to discard reset approaches, but then proceeds to not evaluate catastrophic forgetting ... Therefore, I believe reset approache to be a valid baseline for ReGraMa.\n\nWe use DrQ as the backbone and compare the performance on the long-term training task Cheetah Run (8M steps). It can be seen that full resets may cause instability and oscillations. ReGraMa balances the restoration of network activity with forgetting by precisely resetting individual neurons.\n\nTable. DrQ on Cheetah Run (Average of 3 seed):\n\n| Method | during 1/5 training stage | during 2/5 training stage | during 3/5 training stage | during 4/5 training stage | during 5/5 training stage |\n| --- | --- | --- | --- | --- | --- |\n| DrQ | $478.39\\pm 78.96$ | $612.52\\pm 112.34$ | $549.17\\pm 136.81$ | $662.03\\pm 173.27$ | $637.61\\pm 193.68$ |\n| ReGraMa | $492.67\\pm 92.04$ | $641.31\\pm 107.93$ | $631.58\\pm 125.36$ | $653.03\\pm 173.27$ | $673.62\\pm 126.21$ |\n| Full Reset (every 2M step) | $495.14\\pm 108.35$ | $627.89\\pm 159.06$ | $514.73\\pm 154.57$ | $629.97\\pm 186.14$ | $652.37\\pm 177.35$ |\n\nWe also added PPO as the on-policy baseline to compare the above methods. ReDo reset has a relatively weak positive effect on the performance of neurons. Full resets caused performance instability, while ReGraMa significantly enhance learning efficiency.\n\nTable. PPO Performance on HalfCheetah (Average of 3 runs):\n\n| Method | HalfCheetah |\n| --- | --- |\n| vanilla PPO | $2604.71\\pm 487.61$ |\n| ReGraMa | $3418.38\\pm 336.14$ |\n| ReDo | $2641.31\\pm 394.27$ |\n| Full Reset (every 250k step) | $2313.85 \\pm 564.93$ |\n\n*We would like to emphasize that \"mitigate catastrophic forgetting\" is mainly used to introduce the contributions of related work, not to describe the approach proposed in this paper.*\n\n> W6: Sometimes the writing is rather verbose. For example, the introduction is rather lengthy.\n\nWe’ve streamlined the introduction and moved the detailed related work to its own dedicated section.\n\n### Questions\n\n> Q1: How dependent is ReGrama on different optimizers?\n\nWe uniformly use the Adam optimizer for fair comparison.\n\nWe have tested ReGraMa with different optimizers on Walker Walk. The optimizers include Adaptive Moment Estimation (Adam), Stochastic Gradient Descent (SGD), and Root Mean Square Propagation (RMSprop). Results show that ReGraMa might be more suitable for optimizers with adaptive learning rate, though future work could explore optimizer-reset interactions further.\n\nTable. Optimizer Dependence (Average of 3 seeds):\n\n| Method | Adam | SGD | RMSprop |\n| --- | --- | --- | --- |\n| SAC | $1.0$ | $1.0$ | $1.0$ |\n| ReGraMa | $1.07\\pm 0.05$ | $0.99\\pm 0.07$ | $1.08\\pm 0.03$ |\n\n> Q2: How does ReGraMa interact with parameter regularization, such as weight decay or L2 init [3]?\n\nThe effectiveness of the combination ReGraMa with weight decay (follow the optimal setting in [1]) and L2 init ($\\lambda=1e-2$) is tested using  DrQv2. Results show that integrating either method with ReGraMa can further enhance learning efficiency. L2 init may be more compatible with ReGraMa.\n\nTable. ReGraMa with Regularization (Average of 3 runs):\n\n| Method | Quadruped Run (3M step) |\n| --- | --- |\n| Vanilla Policy | $649.13\\pm 182.43$ |\n| ReGraMa | $706.59\\pm 127.25$ |\n| ReGraMa + L2 init | $747.36\\pm 127.31$ |\n| ReGraMa + Weight Decay | $721.31\\pm 94.26$ |\n| ReGraMa + L2 Init + W. Decay | $739.42 \\pm 104.58$ |\n\n[1]: Lyle, Clare, et al. \"Disentangling the causes of plasticity loss in neural networks.\" *arXiv* (2024).\n\n---\n\nThank you again for your constructive feedback. We hope these additional results address your concerns. Please let us know if further clarification is needed."}, {"review_id": "3naNqrpsvs", "reviewer": "Reviewer_7CkP", "summary": "Deep reinforcement learning often suffers from a loss of plasticity or neuronal activity, largely due to the non-stationarity of learning targets. A common metric for identifying such loss is the \\tau-dormant neuron ratio, which measures the relative activation strength of neurons across a dataset. Neurons with low activation are considered dormant and are reset to enable adaptation to new data---a strategy known as \\textit{ReDo}. In this paper, the authors argue that the \\tau-dormant ratio is limited to standard MLPs with ReLU activations and fails to capture plasticity in more complex architectures. To address this, they propose estimating plasticity using the relative magnitude of the gradient. (GraMa) They demonstrate that resetting neurons based on this new signal---an approach they call \\textit{ReGraMa}---results in improved performance.", "questions": "1) Can you address the questions regarding \"learning capacity\" and \"expressivity\"?\n2) Could you also plot the \\tau-dormant neuron ratio? \n3) Could you include the results mentioned in Weakness 5(c) to support the \"Analysis\" section?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The problem tackled in this paper is both relevant and impactful for the Deep RL community.\n2) It is clear, well-written, and the proposed approach is well motivated.\n3) The experiments are well-designed and validate the claims made by the authors.\n\nHowever, there are a few minor concerns/clarifications I would appreciate the authors addressing:\n1) Why does the gradient magnitude imply “learning capacity” and the activation magnitude imply “expressivity”? This is not well discussed in the paper.\n2) Figure 3: This seems like circular reasoning—you define \"learning capacity\" as the gradient magnitude and then evaluate GraMa, which is based entirely on that same metric. Doesn’t this place ReDo at an unfair disadvantage? That said, the results do show that the two methods reset different types of neurons.\n3) Suppose a neuron consistently receives a high-magnitude gradient (greater than \\tau) but in alternating directions (e.g., +c, -c, and so on). In such a case, can we truly say that the neuron is “learning'' or that it has high “learning capacity''?\n4) (Line 158) “Less expressive at the moment” — As I understand it, [Sokar et al., 2023] show that dormant neurons tend to remain dormant over time. Does this observation not apply to the BroNet architecture discussed in this context?\n5) Figure 4: \n\n    a) (Minor) I recommend adding the Resnet SAC architecture in the Appendix as well (since all the others are present)\n\n    b) The results show that ReGraMa is better at resetting neurons v/s ReDo, which is more aligned with the results presented in Section 4 (Experiments).\n\n    c) The analysis in Section 3 discusses how ReDo fails to identify dormant neurons in the settings considered (e.g., non-ReLU activations). To validate this claim, I suggest an experiment comparing the performance of SAC with ReLU and TanH activations, both with and without ReDo—i.e., SAC-ReLU, SAC-ReLU-ReDo, SAC-TanH, and SAC-TanH-ReDo. If ReDo improves performance in the ReLU setting but not with TanH, it would provide empirical support for the claim.\n\n6) Figure 7(a) —Are ReDo and ReGraMa equivalent in this setting (according to Theorem 1)? Shouldn't we then expect them to perform similarly? (ReGraMa gets a higher reward)\n\n7) I also recommend that the authors include plots of the \\tau-dormant ratio for some of the reported results. This would help verify whether ReDo is behaving as expected or diverging in practice---similar to the behaviour observed with GraMa in Figure~7(c), for example.\n\n8) Are “Low learning Capacity neuron ratio” (Figure 1) and GraMa the same?\n\n9) (Typos)\n    \n    a) (Line 228) z is the output _before_ passing through the activation?\n\n    b) (Line 513) “Structures were showen”", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our deep gratitude for your significant support for this article. Your detailed suggestions have enabled us to further improve our work. We hope that our response can address all your concerns.\n\n### Weakness\n\n> W1: Why does the gradient magnitude imply “learning capacity” and the activation magnitude imply “expressivity”? This is not well discussed in the paper.\n> \n\nThanks for the question. We have expanded our background section in the revised version, to provide a clearer explanation and foundations of *learning capacity* and *expressivity*.\n\n- Recent work [1] defines *learning capacity* as an agent’s ability to efficiently adapt its parameters to the current data distribution. Gradient magnitude directly quantifies this property; it reflects the effect of parameter updates prescribed by back-propagation upon receiving learning signals with new data. Hence, smaller magnitudes indicate lower learning capacity.\n- In [2,3], *expressivity* refers to the model's representational ability for the inputs. Empirical studies show that this correlates with the proportion of neurons whose activations exceed a threshold. Accordingly, subsequent work adopts mean activation magnitude as a practical proxy for expressivity.\n\n[1]: Liu et al., *Neuroplastic Expansion in Deep Reinforcement Learning*, ICLR 2025\n\n[2]: Sokar et al., *The Dormant Neuron Phenomenon in Deep Reinforcement Learning*, ICML 2023\n\n[3]: Ma et al., *Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning*, ICML 2025\n\n> W2: Figure 3: This seems like circular reasoning—you define \"learning capacity\" as the gradient magnitude and then evaluate GraMa, which is based entirely on that same metric. Doesn’t this place ReDo at an unfair disadvantage? That said, the results do show that the two methods reset different types of neurons.\n> \n\nWe use GraMa for analysis because ReDo tends to misidentify dormant neurons in complex architectures. Specifically, ReDo systematically flags a large fraction of healthy neurons as dormant while missing those that empirically harm performance. This misalignment is clearly visible in the reset-preference visualizations in Figures 1 and 3. As such, activation-based scores lose their statistical power in these cases. To facilitate direct visual comparison, we have added ReDo curves to the main experimental plots in the revised manuscript.\n\n> W3: Suppose a neuron consistently receives a high-magnitude gradient (greater than \\tau) but in alternating directions (e.g., +c, -c, and so on). In such a case, can we truly say that the neuron is “learning'' or that it has high “learning capacity''?\n> \n\nOur definition of learning capacity focuses on the *magnitude* of the gradient to reflect the agent’s \"*willingness\"* to update parameters. Even if the update direction oscillates, the neuron remains actively engaged in learning. Therefore, we consider large magnitudes, regardless of sign, as indicative of high capacity. In practice, measuring this is more tractable and stable than evaluating directional correctness on a per-neuron basis.\n\n> W4: (Line 158) “Less expressive at the moment” — As I understand it, [Sokar et al., 2023] show that dormant neurons tend to remain dormant over time. Does this observation not apply to the BroNet architecture discussed in this context?\n> \n\nIndeed! In the revised manuscript, we clarify that architectures with normalization (e.g., BRO-net) are already known to partially mitigate dormancy. In these cases, ReDo’s activation-based metric can occasionally “revive” dormant neurons, which is visible as minor oscillations in ReDo’s dormant-neuron curves. This transient rebound becomes more pronounced in deeper or more complex networks.\n\n> W5: (a,b) I recommend adding the Resnet SAC architecture in the Appendix as well (since all the others are present) ...\n> \n\nWe have added the Resnet SAC architecture in Appendix B of the revised manuscript and reorganized this point.\n\n> W5: (c) The analysis in Section 3 discusses how ReDo fails to identify dormant neurons in the settings considered (e.g., non-ReLU activations). To validate this claim, I suggest an experiment comparing the performance of SAC with ReLU and TanH activations, both with and without ReDo—i.e., SAC-ReLU, SAC-ReLU-ReDo, SAC-TanH, and SAC-TanH-ReDo. If ReDo improves performance in the ReLU setting but not with TanH, it would provide empirical support for the claim.\n> \n\nWe followed your suggestion and replicated the experimental setup from ReDo’s paper on the MuJoCo Humanoid task (1M steps, Table 6). We normalized all results against SAC-ReLU for clearer comparison. The table below shows the performance ratios (mean ± std across 3 runs).\n\nThe results in the table below match your conjecture. ReDo does improve the performance of the agent compared to vanilla ResNet-SAC under ReLU, but degrades it under TanH.\n\nTable: Advantages over SAC-ReLU (Average of 3 runs):\n\n| Method | Humanoid |\n| --- | --- |\n| SAC-ReLU | $1.0$ |\n| SAC-ReLU-ReDo | $1.14\\pm0.03$ |\n| SAC-TanH | $0.97\\pm0.05$ |\n| SAC-TanH-ReDo | $0.89\\pm0.06$ |\n\n> W6: Figure 7(a) —Are ReDo and ReGraMa equivalent in this setting (according to Theorem 1)? Shouldn't we then expect them to perform similarly? (ReGraMa gets a higher reward)\n> \n\nThank you for pointing this out. We apologize for the lack of clarity. The results in Figure 7 are based on BRO-net, not a standard MLP+ReLU architecture. Thus, GraMa and ReDo are no longer equivalent, and the superior robustness of ReGraMa is reflected in its higher reward. We clarify this distinction in the new version.\n\n> W7: I also recommend that the authors include plots of the \\tau-dormant ratio for some of the reported results.\n> \n\nWe have added $\\tau$-dormant ratio plots for ReDo in the revised manuscript to better visualize how it compares with GraMa under different conditions. \n\n> W8: Are “Low learning Capacity neuron ratio” (Figure 1) and GraMa the same?\n> \n\nYes, they are equivalent. We will explicitly clarify this in the revised version.\n\n> W9: (Typos)\n> \n\nWe have thoroughly revised the manuscript to fix typos, improve clarity, and correct inconsistent notation.\n\n### Questions\n\n> Q1: Can you address the questions regarding \"learning capacity\" and \"expressivity\"?\n> \n\nPlease refer to our response to **W1** above.\n\n> Q2: Could you also plot the \\tau-dormant neuron ratio?\n> \n\nYes, we will include these plots in the revised manuscript.\n\n> Q3: Could you include the results mentioned in Weakness 5(c) to support the \"Analysis\" section?\n> \n\nYes, please refer to our response to **W5(c)** and the accompanying experimental table.\n\n---\n\nThanks again for your valuable feedback. We appreciate your thoughtful suggestions and hope our clarifications are helpful. Please let us know if anything remains unclear."}, {"review_id": "7MpbiJSlya", "reviewer": "Reviewer_eJ85", "summary": "This paper introduces gradient magnitude as a novel measure to identify and reset inactive neurons in reinforcement learning. This measure called GraMa is lightweight and effective especially in complex networks.", "questions": "I know that this might be difficult, but could you come up with any kind of formal guarantees or explainations for GraMa? This would make the paper even stronger.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strong points:\nS1: The paper is well written. There are lots of illustrative figures. Important results are well explained, illustrated and summarized.\nS2: The code is available and well documented.\nS3: The proposed method is runtime-efficient and increases the performance in many cases, sometimes drastically.\n\nWeak points:\nW1: There are almost no formal guarantees nor formal explainations why GraMa works.\n\nAd W1: I guess it is very difficult to formally explore the properties, strengths and limitations of the method. At least, empirical evidence by well-conducted experiments show the merits of the method. The only theoretical exploration is the equivivalence to ReDo in traditional architectures. This is nice and easy to see but not very important, as more complex architectures are the central focus of this paper.\n\nMinor weak points:\n- The clarity of Fig. 1 can be improved. I suggest to remove the double title \"Complex Architecture Based Policy\" which is anyhow misleading in my view. Also revise the caption on the y-axis on the top subfigure. What is the unit here? That is not entirely clear. Also make sure that the axis labelling is readable, currently, it is too tiny.\n- The notation in Eq. 1 looks a bit overly complex. Consider simplifying. But if it is common and taken from the corresponding papers, you can also leave it as it is.\n- Fig. 3 is impressive but a bit more description would help the understanding. What are the light-blue and light-red areas?\n- What is the unit of measurement on the y-axis in the subfigure at the center of Fig. 5? Also, in Fig. 5 the axis labelling and the headings of the center and right-most subfigures are unreadable small.\n- The message in the text (Lines 215-220) does not really correspond to the visual information in Fig. 6. The figure suggest that during training, already inactive neurons become even less active. If this is the case it should be also stated in the text.\n- Sometimes the figures could be displayed closer to the places in the text where they are discussed, e.g., Fig. 5.\n- In Fig.9 the axis labelling is too small. Also, the meaning of 1x, 2x and so on on the x-axis is not clear. How many layers do these networks actually have?", "was_revised": true, "final_justification": "The rebuttal successfully addressed all my concerns.", "rebuttal": "We are grateful for your thoughtful review and for recognizing the importance of our research. Your suggestions encouraged us to significantly strengthen both the theoretical and empirical components of the paper.\n\n### Weakness\n\n> W1: There are almost no formal guarantees nor formal explainations why GraMa works. I know that this might be difficult, but could you come up with any kind of formal guarantees or explainations for GraMa? This would make the paper even stronger.\n> \n\nThanks for your valuable feedback. We appreciate your recognition of our paper’s strengths while acknowledging, as you noted, the inherent challenge of developing comprehensive and realistic theoretical guarantees for complex deep RL systems. \n\nAs the first study to comprehensively analyze gradient magnitudes for neural maintenance in deep RL, our work primarily establishes a strong empirical foundation across multiple modern architectural settings in deep RL, with a few theoretical justifications, though we fully agree that these could be better highlighted and expanded. \n\nBased on our continued theoretical explorations since submission, we have identified several promising future theoretical directions for our work, and we will include these discussions in the updated manuscript: 1) the relationship of resetting parameters with small gradient norms and suboptimal saddle point escape, which is a common challenge in deep RL optimization landscapes [1, 2], 2) the effect of selecting parameters with minimal gradient magnitudes for resetting on disruption to learned features while maximizing learning capacity. The developments in these theoretical perspectives can further help explain why GraMa consistently outperforms activation-based alternatives across diverse architectures and tasks, but remains particularly challenging due to the complex learning dynamics in deep RL systems. Our work also opens several promising avenues for future theoretical exploration including the connection to second-order optimization, and convergence guarantees in non-stationary environments. \n\nIn our revision, we will better highlight these theoretical insights with our paper's strong empirical focus and theoretical justifications. We believe our pioneering work not only establishes a practical method for neural maintenance but also provides valuable theoretical foundations for this emerging area of deep RL.\n\n[1]: Ge, Rong, et al. \"Escaping from saddle points—online stochastic gradient for tensor decomposition.\" COLT , 2015.\n\n[2]: Mishkin, Dmytro, et al. \"All you need is a good init.\" ICLR, 2015.\n\n### Minor Weakness\n\n> W1: I suggest removing the double title \"Complex Architecture Based Policy\" which is, anyhow, misleading in my view. Also revise the caption on the y-axis on the top subfigure. What is the unit here? That is not entirely clear. Also make sure that the axis labelling is readable. Currently, it is too tiny.\n> \n\nThanks for the suggestion. We incorporated recommended changes in the revision, where we removed the duplicate title, revised the caption to clarify the metric (the y-axis represents the score normalized by the peak ReGraMa performance, with the maximum value being 1.0), and enlarged axis labels for better readability. \n\n> W2: The notation in Eq. 1 looks a bit overly complex. Consider simplifying. But if it is common and taken from the corresponding papers, you can also leave it as it is.\n> \n\nThanks for raising this point! To ensure consistency with established literature [1], we retained the original notation, as it facilitates direct comparison with prior work. \n\n[1]: Sokar et al., *The Dormant Neuron Phenomenon in Deep Reinforcement Learning*, ICML 2023\n\n> W3: Fig. 3 is impressive but a bit more description would help the understanding. What are the light-blue and light-red areas?\n> \n\nThanks for highlighting this! We have enhanced the visualizations with clearer labels: light-blue areas now explicitly mark neurons with low learning capacity and low expressivity—the primary candidates for reset, while light-red indicates neurons with high learning capacity but low expressivity. We also added new panels that explicitly show the performance impact of resetting each of the four neuron types, thereby highlighting their distinct functional roles (detailed results are provided in Response W3 to Reviewer 1 due to space limitation).\n\n> W4: What is the unit of measurement on the y-axis in the subfigure at the center of Fig. 5? Also, in Fig. 5 the axis labelling and the headings of the center and right-most subfigures are unreadable small.\n> \n\nWe have enlarged all fonts and added explicit labeling showing that the y-axis represents percentage improvement. For intuitive interpretation, we normalized each ReGraMa score relative to the score of ReDo (ReDo’s score = 1.0), so the y-axis directly shows ReGraMa’s relative improvement over ReDo in percentage.\n\n> W5: The message in the text (Lines 215-220) does not really correspond to the visual information in Fig. 6. The figure suggest that during training, already inactive neurons become even less active...\n> \n\nIn the revised manuscript, we now state: “The figure indicates that neurons that are already inactive at the start of training become progressively less active as training proceeds.”\n\n> W6: Sometimes the figures could be displayed closer to the places in the text where they are discussed, e.g., Fig. 5.\n> \n\nGreat suggestion! We reorganized the layout so all figures appear adjacent to the corresponding discussion, eliminating the need for flipping pages and improving overall readability.\n\n> W7: In Fig.9 the axis labelling is too small. Also, the meaning of 1x, 2x and so on on the x-axis is not clear. How many layers do these networks actually have?\n> \n\nIn the revised Figure 9, we enlarged all axis labels and revised the x-axis labeling with a clear caption explaining: model sizes begin with two BRO-net blocks (as per the standard setting), and larger model scales are constructed by adding additional BRO-net blocks in sequence.\n\n### Question\n\n> Q1: \n\nPlease refer to W1.\n\n---\n\nThanks again for reviewing our work and constructive feedback! We hope our responses address your concerns.  We are happy to provide additional clarification if needed."}]

Author rebuttal:

Meta-review: This paper introduces gradient magnitude as a novel and practical metric to identify and reset inactive neurons in reinforcement learning. The proposed measure, GraMa, is both lightweight and highly effective, particularly in complex network architectures. The manuscript is well-written, and the key results are clearly explained, well-illustrated, and thoughtfully summarized. Notably, the method is runtime-efficient and significantly enhances performance in many scenarios, sometimes yielding substantial improvements.
While the paper offers limited formal guarantees or theoretical explanations for why GraMa works, its empirical effectiveness and innovative approaches make it a valuable contribution to the field.
Decision: Accept (poster)