Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
Authors: Simon Dufort-Labbé
Pierluca D'Oro
Evgenii Nikishin
Irina Rish
Pierre-Luc Bacon
Razvan Pascanu
Aristide Baratin
Venue: Accepted by TMLR ()

Abstract: When training neural networks, dying neurons —units becoming inactive or saturated— are traditionally seen as harmful. This paper sheds new light on this phenomenon. By exploring the impact of various hyperparameter configurations on dying neurons during training, we gather insights on how to improve upon sparse training approaches to pruning. We introduce Demon Pruning (DemP), a method that controls the proliferation of dead neurons through a combination of noise injection on active units and a one-cycled schedule regularization strategy, dynamically leading to network sparsity. Experiments on CIFAR-10 and ImageNet datasets demonstrate that DemP outperforms existing dense-to-sparse structured pruning methods, achieving better accuracy-sparsity tradeoffs while speeding up training up to 3.56$\times$. These findings provide a novel perspective on dying neurons as a resource for efficient model compression and optimization.

Published in Transactions on Machine Learning Research (02/2025)
Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
Simon Dufort-Labbé Mila, Université de Montréal
simon.tiffortllabeemmll...o queber
Pierluca D'Oro
Mila, Université de Montréal
Evgenii Nikishin
Mila, Université Montréal
Irina Rish
Mila, Université Montréal
Razvan Pascanu
Google Mind
Pierre-Luc Bacon
Mila, Université de Montréal
Aristide Baratin
boratina imm quebec
Samsung SAIL Montreal
Mila, Université de Montréal
Reviewed on OpenReview: https: // wwtp//veep...veeeeemmmmm net/ forum? id= nmBl euFzal
Abstract
When training neural networks, dying neurons units becoming inactive or saturated are traditionally seen as harmful. This paper sheds new light on this phenomenon. By exploring the impact of various hyperparameter configurations on dying neurons during training, we gather insights on how to improve upon sparse training approaches to pruning. We introduce Demon Pruning (DemP), method that controls the proliferation of dead neurons through a combination of noise injection on active units and a onc-cyele schedule regularization strategy, dynamically leading network sparsity. Experiments on CIFAR-10 and ImageNet datasets demonstrate that DemP outperforms existing dense-to-sparse structured pruning methods, achieving better accuracy-sparsity tradcoffs and accelcrating training by up to 3.56x. These findings provide novel perspective on dying neurons as a resource for efficient model compression and optimization. ion. The code for our experiments is available here
1 Introduction
In neural network training, dying thosc that saturate become inactive during the lcarning process have long been considered undesirable (Maas et al.) 2013 Xu et al. 2015), leading to performance degradation or loss of plasticity, especially in non-stationary settings (Lyle et al. 2023 Nikishin et al. 2022 Abbas et al. 2023). Past rescarch has proposed solutions such as alternative activation functions like Leaky
ReLU Swish (Maas al. 2013; Ramachandran et al. 2018). or methods to reset weights (D'Oro al. 2022 Dohare al. 2024). However, the role of dead neurons remains poorly understood.

---

Published in Transactions on Machine Learning Research (02/2025)
In this work, we reframe the phenomenon of neuron death and saturation resource for pruning, allowing us to reduce the size of neural networks dynamically during training. Inspired by Maxwell's Demon, which selectively filters molecules in thermodynamies (Maxwell 1872), we introduce Demon Pruning (DemP), a sparse training method that (i) promotes ncuron death by applying asymmetric Gaussian noise to the weights of active neurons and using a one-cycle regularization schedule, and (ii) iteratively removes dead neurons during training. This approach enables highly structured sparsity with minimal performance loss.
While DemP builds upon simple regularization-base methods (Zhou et al., 2016 Lym et al., 2019 Liu et al,
2017), unlike prior approaches passively inducing sparsity, DemP actively promotes unit saturation during training, yiclding progressively sparser subnetworks. Compared to existing methods (Lee et al.) 2023). key feature of DemP is the simplification of the pruning process: by inspecting neuron activations with just a few forward passes, dead units are easily identified and pruned at minimum computational costs.
Despite its simplicity, DemP consistently achieves superior accuracy-sparsity tradcoffs compared to strong baselines in structured dense- Lo-saree training, such as EarlyCrop (Rachwan el al. 2022) or SNAP (Verdenius et al. 2020 For instance, when pruning ResNet-18 CIFAR-10 ResNet-50 delivers up to a 2.5% improvement in accuracy 80% sparsity compared to those baselines, while also accelerating training by up to 1.23x on ImageNet. This positions DemP as an efficient method for model compression that balances both accuracy and computational cost. Finally, beyond its strong empirical performance, DemP integrates seamlessly into any training pipeline and can be readily combined with existing pruning techniques.
ImageNet, DemP
Our primary contributions are:
1. Insights into Neuron Saturation: We explore the mechanisms behind neuron saturation, high- lighting the significant roles of stochasticity and key hyperparameters such as learning rate, batch size, and regularization (Section 2 and Appendix Bi.
A Simple Structured Pruning Method:
We introduce DemP, a dynamic dense- to-saae training method that promotes neuron death, and prunes them in real-time, by applying decaying regularization and injecting noise early in training (Section 3 and Algorithm D
3. Training Speedup and High Compression: Extensive experiments CIFAR-10 and ImageNet show that DemP, despite its simplicity, surpasses strong comparable structured pruning baselines in aecuracy-copprssti tradeoffs, while achieving results comparable to unstructured pruning methods and offering significant training speedup (Section and Appendix
1.1 Related Work
Dead Neurons. It is widely recognized that neurons, especially for networks using ReL.U activations, can
saturate during training (Agarap) 2018 Trottier et al. 2017: Lu et al. 2019). Continual and reinforcement
learning literature linked the accumulation of dead units with plasticity loss (Sokar et al.) 2023 Lyle et al.
2023 Abbas et al.
2023 Dohare et al. 2024). Closer to our work, Evei 2018 noted the connection between the dying rate and the learning rate, and derived a pruning technique from it.
Mirzadeh et al. 2024 highlighted the strategic use of ReLU activations to achieve inference efficiency in transtormer-based models and proposed an approach leveraging post-activation sparsity to improve inference speed in large language models.
Structured Pruning. Pruning reduces the size and complexity of neural networks by removing redundant less important elements, be they neurons or weights (Leunn et al. 1989). Recent advances such as the Lottery Ticket Hypothesis (LTH) (Frankle & Carbin 2019) demonstrated the existence of subnetworks trainable to performance comparable to their dense counterpart. While unstructuued pruning imposes no constraints on which weights to remove, structured methods aim to eliminate entire structures within a network, such as channels, filters, or layers. This results smaller, faster models that are compatible with existing hardware accelerators and software libraries (Wen al. 2016 Li et al. 22017). Structured pruning methods often involve dense training period followed by sparse fine-tuning (Han et. al. 2015). Regularization--aer approaches for
2

---

Published in Transactions on Machine Learning Research (02/2025)
sparsification after dense training have been widely explored (Louizos et al. 2018 Liu et al. 2017 Ye et al.
2018
Ding et al. 2018 Wang et al. 2019 2021 J. The current SOTA for channel pruning at inference, such as Group Fisher (Liu et al., 2021), CafeNet-R (Su et al.) 2021). CHIP (Sui et al. 2021) or AutoSlim (Yu &
Huang 2019
all involve either an extended training period or an increased computational budget, negating
the potential benefits of pruning for training speedup.
Dynamic Pruning: To address these drawbacks, new methods aim to recover a sparse network under a fixed training budget. Regularization-baase methods have been adapted to this setting by leveraging group regularization to infiuence the training dynamics to yield sparser networks (Zhou et al. 2016 Lym et al., 2019), or remarking that batch normalization offers a natural grouping of units under the scale parameter that can be solely regularized (Liu et al.) 2017). The LTH inspired multiple methods seeking the winning
ticket without iterative pruming. Sparse-to-sparee sparee approaches (Lee et al. 2019 Wang et al.) 2020 pruming
weights before training- were adapted to work under structured constraints (Verdenius et al. 2020 Because
deep networks suffer increasingly from pruning at initialization (Frankle al.
2020). conse-to-sparse methods
perform pruning at later stage YYou et. al. 2020 Rachwan el al. 2022).
A recent trend, Dynamic Sparse Training (DST) Mocanu et al.
2018 Evei et al. 2020 initializes networks
sparsely and uses an iteratively updated pruning mask to balance performance and computational efficiency.
Initially focusing on unstructured sparsity, DST has recently been adapted for partiall--strucuun sparse
networks (Lasyy et al. 2024 Yin et al. 2023 J, yet still not exceeding 30% node sparsity.
Contextualizing DemP: Within the landscape of pruning techniques, DemP represents an advancement as dense-to-sparse end-to-end structured method, offering tangible speedup during both training and inference GPU hardware. It revisits carlier group regularization-baseed approaches monitoring activations similar to Hu et al. (2016) and leveraging increasing regularization in (Wang et al. (021] scale parameters (Liu) jtt al. 2017] by integrating fresh insights into its design to achieve competitiveness with more recent methods while preserving simplicity. Notably, addressing the phenomenon wherein noisier environments can make saturated regions more attractive (see Lemmass B.2 B.3 and Appendix D.I. DemP introduces asymmetric noise during optimization to further promole sparsity. Sparsity naturally emerges from the learning dynamics with DemP, simplifying the pruning process and eliminating the need for discrete pruning interventions which could disrupt training. Non-contributing neurons can be pruned as they become inactive, without compromising model performance (as demonstrated in Fig. 7p.. Recent unstructured pruning approaches manually tailor sparsity distribution across layers (Mocanu et al.)
2018 Evei et al. 2020), while structured
pruming mostly relies on uniform layer-wise pruning [Rachwan et al. 2022
DemP alleviates this by allowing
the learning dynamics to determine sparsity distribution across layers.
) DemP climinates the need for meticulous pruning timing (Wang et al. 2021 Rachwan et al. 2022 or
heuristic-based pruning schedules (Lee et al. 2023). Acknowledging that early pruning can be detrimental (Frankle et al.) 2020 Rachwan et al. 2022), DemP employs easy-to-tune regularization schedules to influence when units die. Additionally, iterative pruning (Verdenius et al. 2020) within single intervention is unnecessary, as units naturally become inactive during carly training with DemP.
Choice of Baselines: We highlight and benchmark against strong comparable baselines that use criteria
based on gradient flow to evaluate which nodes to prune (Verdenius et al. 2020 Wang et al. 2020 Rachwan
ct al. 2022). Other methods employing either L0 or L1 rogularization on subgroups of parameters to enforee
sparsity (Liu et al.] 2017 Louizos et al. 2018 You el. al. 2019 Lym et al. 2019) are omitted from the benchmark as they are outperformed by Rachwan et al. (2022).
2 Saturating Units: An Analysis
In this section, we offer theoretical insights into the occurrence of dead neurons and explore the impact of different training heuristics and hyperparametors. Given a neural network and set of n training data samples, we denote by af E RR" the vector of activations of the jth neuron in layer & for each training input. We adopt the following definition of dead neuron throughout the paper:
3

---

Published in Transactions on Machine Learning Research (02/202))
Dead Neurons Ratio
0.4
Activation Learning ride.
Swish 0.05
Swish 0:01
Lesky ReLU- 0.06
Lesky ReLU- 0.01
0.2
BaLU 0.05
ReLU 0..0
0.0
20000
40000
60000
Training Step
Figure 1: Dead neuron accumulation for a ResNet-18 trained on CIFAR-10 with different activation functions and values of the learning rate. We use a negative slope of a = 0.05 for Leaky ReLU and , = for Swish.
The j-th neuron in layer l is inactive if it consistently outputs zero on the entire training set, i.e. of = o A neuron that becomes and remains inactive during training is considered as dead 2
Definition:
Many modern architcctures use activation functions with a saturation region that typically includes 0 at its boundary. In this case, when a neuron becomes inactive during training, its incoming weights also receive zero or very smaly gradients, which makes it difficult for the neuron recover. In this paper, we mostly work with the Rectified Lincar Unit (RcLU) activation function, (xx) max(0, æ). In this case, the activity of neuron depends on the sign of the corresponding pre-activation feature.
2.1 Neurons Die During Training
We begin with some empirical observations. Applying the above criterion (Footnote 1 with threshold parameter 0.01, monitor the accumulation of dead neurons during training of Resnet-18 (He et al. 2016 on CIFAR-10 (Krizhevsky et al. 2009) with the Adam optimizer (Kingma & Ba 2015 with various learning rates and choices of activation functions.
The outcomes are shown in Fig. [ revealing a notable and sudden increase in the number of inactive neurons early in training. Morcover, a minimal portion of these inactive ncurons shows signs of recovery in later stages of training (see Fig. in Appendix C Overall, this results in significant. fraction of the 3904 neurons/filters in the convolutional layers of the ResNet-18 dying during training, particularly with a high learning rate. We note that this phenomenon is not exclusive to ReLU activations.
Intuition. Similar to Maxwell's demon thought experiment (Maxwell 1872 one can picture an asymmetric behavior of how weights can move across the boundary that delimitates the saturated versus the non-saturated state of a unit, akin to the demon's selective passage molecules. Neurons, or more specifically their weights, can move freely within the active region, but once they enter the inactive region their movement impeded, similar to how the demon can trap molecules in low-energy subspace. In this context, with a random component to their movement, neuron death can be influenced by various factors, c.g., noise from the data, being too close to the border, laking too large gradient steps. Once the neuron moves into the inactive zone, neurons can only be reactivated if the boundary itself shifts. This asymmetry makes it more likely for ncurons to die than to revive.
We formalize this analogy with simple theoretical models in Appendix B These models are meant to capture the multiplicative (i.e. -arameter-dependeent) nature of the gradient noise in SGD. Multiplicative noise in
In practice, espocially for non-ReLU activation functions, will be using the notion of c-inactivity, defined by the condition lafl for some threshold parameter Forr convolutional layers treat channels as neurons (see Appendix A Se due to regularization.
4

---

Published in Transactions on Machine Learning Research (02/2025)
1.00
1.00
Dead Neurons Ratio
Dead Neurons Ratio
0.75
0.75
0.50
0.50
Noise Variance
Rogilation Optimizss
0.0
0.0005
0.25
12. AdomW
0.25
L2 SCDM
Lasso SCDM
5e-05
0.001
12(1) AdonW
Laco)))) AdaM
0.0001
12 SGDM
Lasso( SGDM
0.00
0.00
0.000
0.002
0.004
0.006
0.008
0.010
0.000
0.001
0.002
0.003
0.004
Regularization Parameter (A)
Regularization Parameter (A)
Increased regularization during training increases the ratio of dead units, as showcased here for ResNet-18 trained on CIFAR-10. We use -(7) to denote when regularization is applied solely to the scale parameters of the normalization layers. Right: Augmenting training updates with asymmetric Gaussian noise, sampled from (2 o²) and applied to weights of live neurons only, also leads to higher levels of dead unit accumulation, also showcased for a ResNet-18 trained on CIFAR-10 with various values of the Lasso(7) regularization parameter.
Figure/ 2: Left:
stochastic processes is known to cause regions with lower noise magnilude to act as attractors OOksndall 2010). Essentially, lower noise regions (such as the inactive region of a saturating unit) tend to retain the system due to reduced impact, while higher noise regions push it away. We explicitly demonstrate this phenomenon in the theoretical framework of Appendix B
2.2 Factors Impacting Dying Ratios
Role of regularization. The above intuition suggests that maintaining activations close to zero increases the likelihood of neuron death, aligning them closer to the inactive region (the negative domain for ReLU networks). As illustrated in Fig. 2 training ResNet-18 with higher L2 Lasso regularization leads to sparser networks caused by an accumulation of dead units. Section B elaborattes on the approach with DemP, where we opt to apply regularization exclusively the scale parameters of the normalization layers, following Liu et al. (2017). This tends to better preserve performance and yields significantly improved accuracy-sparsity tradcoffs, as shown in Fig. in Appondix
CIFAR-10 (with Adam or SGD+momentum)
Role of noise. Our simple model suggests a pivotal role played by the noise the training process in the occurrence of dying neurons. To investigate this in simple setting, we train small MLPs on a subset of 10,000 images of the MNIST dataset in three different noisy regimes: vanilla SGD, pure SGD noise (obtained by isolating the noisy part of the minibatch gradient), and pure Gaussian noise. As observed in Fig. 9 in Appendix D.I. while pure SGD noise training yields dying ratios comparable to SGD, training with pure Gaussian noise is much less prone to dead neuron accumulation. We hypothesize that this difference is due to the asymmetry of SGD noise: since the gradients are for dead neurons, only (the weights of) live neurons are subject to noisy updates. In contrast, the weights of inactive neurons are updated under Gaussian noise training, which increases their probability of recovery. As we spell out in Section 3 DemP injects Gaussian noise exclusively weights of live neurons during training, which effective in encouraging neuron death (see Fig. 2 and Fig. g middle plot).
We note that other ways to control the SGD noise variance include varying the lcarning rate and/or the batch size. Fig. 10 (right) in Appendix D.1 illustrates how increasing the learning rate or reducing the batch size indeed increases dying ratios. However, proved more challenging to balance accuracy and sparsity by adjusting these hyperparamoters, as they also significantly affect overall performance. Instead, DemP maintains these parameters their default or task-optimized values, facilitating its integration into existing training routines.
5

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm DemP Algorithu
Input: Learning rate 7, pruning frequency т; regularization one-cycled schedule d₁ noise variance one-cycled schodule of learnable weights w-including normalization scale parameters r Initialize: weights wi including normalization scale parameters "Yo
for t= to T do
Compute gradient: ((w+) Regularization term: VR(m; 2 affecting only the normalization scale parameters
Sample asymmetric noise: fasym (f)
Update weights:
W1+1 Wt (V(()) + VR(TT)
+ Easym
if th then
Prune dead neurons
end if end for
Optimizer. We expect the choice of optimizer to affect the final count of dead neurons post-training (e.g., for adaptive optimizers, by altering the effective learning rate per parameter). Notably, we observe a significant discrepancy between using the Adam optimizer (Kingma & Ba 2015) and SGD with momentum (SGDM) (Fig. 2j. As also emphasized by Lyle et al. (2023) we altribute this difference primarily to the specific hyperparameter selection for Adam (B1, 3),e), which has a substantial impact on neuron death (further discussod in Appendix Results from experiments with both optimizers are presented in the next section.
Finally, the lotal training time and network width can also impact the occurrence of dying neurons and are explored respectively in Appendix D.2 and Appendix D.3
3 Demon Pruning
Leveraging insights from Section we introduce Demon Pruning (DemP), a structured pruning method that dynamically induces neuron saturation during training to achieve high levels of sparsity. DemP achieves this through two primary mochanisms: (1) a dynamic regularization schedule applied to normalization scale parameters and (2) asymmetric noise injection largeting active neurons. The combination of these mechanisms ensures sparse networks without compromising performance. DemP summarized in Algorithm
3.1 Regularization Schedule
Analyzing the pre-activations (z(!) after batch normalization of layer I's output: (F: a!!-)))
2 - o Fi Fi(W!!) al al-11)- -5--11 2291 =
+ +
(1)
20 +
suggests two strategies to push dlll toward zero: (1) regularizing all networks parameters, (2) regularizing the scale normalization parameters (y). We refrain from regularizing the offset parameters (8), to allow for large negative offsets that can help to get negative pre-activations, leading to dead neurons post ReL.U activation. By default, our method employs Ey-regularizatio R(y; A||rll on the scale parameters of normalization layers, in Liu al. (2017), with small modification. Based on the insights of Section 2 we adopt one-cycled schedule (Smith & Topin 2018) X for the regularization strength. This schedule involves initial warmup phase with a lincar growth to peak value X, followed by a cosine decay to fine-tune sparsity levels while maintaining performance.
Such a one-cycled strategy originally applied to learning rates effectively exploits the benefits of both warmup and docay phases (Fig. 4 While regularizing all network parameters is a viable alternative for inducing sparsity, applying regularization exclusively normalization scale parameters proves more effective and robust, shown in Fig. 2 and in Section 4.2

---

Published in Transactions on Machine Learning Research (02/2025)
Test. Accurncy
0.90
0.92
Adam
0.90
0.85
0.88
50
60
70
80
90
60
70
80
90
100
0.94
Test Accuracy
0.925
SGB
0.900
0.92
0.875
50
60
70
90
60
70
S0
90
100
Neuron Sparsity (%)
Weight Spansity (%)
CroPit-S
EarlyCroP-S
EarlySNAP
SNAP
Dem DemP-Lasso(?)
For ResNet-18 networks on CIFAR-10 trained with Adam (RcLU), DemP can find sparser solutions while maintaining better performance than other structured approaches. Higher levels of sparsity with DemP are obtained by increasing the peak strength of the added scheduled regularization, Bottom: With SGDM, DemP performance comparable, without significant differences between methods. Left: Neural sparsity, structured methods. Right: Weight sparsity, structured methods.
Figure 3: Top:
3.2 Noise Injection
To promote neuron death, DemP injects Gaussian noise selectively into active weights during training. At each iteration, the noise is sampled as Easym ~ NO0 of) where the variance of follows the same one-cycled schedule as the regularization strength. the targeted noise injection enhances sparsity without degrading performance (sce Fig. 6). The offectiveness of artificial noise further validated in Fig. 13 where it demonstrates the ability to sparsily networks even the absence of regularization.
3.3 Dynamic Pruning
DemP leverages targeted regularization and asymmetric noise to progressively accumulate inactive neurons, as described in Section 2 Inactive neurons are identified as those with outputs consistently equal to 0 across the dataset. criterion relaxed in practice to minibatches (Fig. 12 For Leaky ReLU and GeLU networks, neurons with consistently negative activations are similarly deemed inactive.
Instead of deferring pruning until training concludes, DemP dynamically prunes dead neurons every itcrations (default: 5000). This approach accelerates training by removing pruned neurons from the computation graph, reducing the embedding dimension. Importantly, it allows non-ReLU networks to recover without affecting final performance or sparsity levels (Fig.
DemP thus introduces two new hyperparameters: scheduled regularization on normalization scale parameters and artificial noise. These additions are orthogonal to standard hyperparameters like weight decay, ensuring compatibility with existing training pipelines. By retaining this orthogonality, DemP intogrates scamlessly into diverse setups, while leaving room for exploration of additional hyperparameters achieve greater sparsity. Additional implementation details and ablations provided in Appendix F

---

Published in Transactions on Machine Learning Research (02/2025)
4 Empirical Evaluation
Through extensive empirical evaluation across various benchmarks, we consistently observe that DemP achieves superior performance-sparsity tradeoffs, particularly high sparsity levels and when combined with Adam. Comparing with SOTA DST methods, such as Yin et al. (2023) and Lasby et al. (2024) that perform restructuration exposes the benefits of DemP for achieving GPU-sppported training FLOPs reduction (Table
2
Sctup: We focus our experiments on computer vision tasks, which is standard in pruning literature Gale et al. 2019). We train ResNet-18 and VGG-16 networks on CIFAR-10, and ResNet-50 networks on ImageNet
(He et al. 2016 Simonyan & Zisserman, 2015; Krizhevsky et al. 2009 Deng et al.) 2009). We follow the (2020) for ResNet architectures and use a setting similar Rachwan al, (2022) for the VGG to broaden the scope of our experiments. The results for pruning the VGG are reported
training regimes from Evci
in Appendix and training details are provided in Appendix
Our method is a structured pruning approach that removes entire neurons during training, transitioning from a dense network to a sparse one. The methods we compare with follow this paradigm. We employ
the following structured pruning baselines: Crop-it/Farly et al., 2020) and a modified version using the early pruming strategy from Rachwan al. (2022) (identified
(Rachwan al.} 2022) SNAP (Verdenius
EarlySNAP). We trained these baselines using the recommended configuration of the original authors; in particular, did not apply the regularization schedule used in our method.
4.1 Results
DemP demonstrates superior performance when paired with Adam optimizer, consistently outperforming all baselines at high sparsity across the board. The margin in test error can reach up to 2.25% when training CIFAR-10 and ImageNet (Fig, 3 and 15 and Table D Notably, the highest sparsification maintaining performance results in 2.45x faster training on ResNet-18 and 3.56x on VGG-16. The results are more contrasted with SGDM: DemP performs similarly to baselines on ResNet-18 18 benchmarks (see Fig, 3 and 15]
is outperformed at higher sparsities on VGG-16 (see Fig. 14 but slightly outperforms baselines when training
Table 1: Results when pruning a ResNet-50 trained on ImageNet with Adam at approximately 80% and 90% weight sparsity. DemP improves test accuracy by 2.85% and 2.14% at the respective sparsitics. Because structured pruning methods lack precise control over weight sparsity levels., we report. the closest. numbers obtained these target values. SNAP and CroPit-S achieve better speedup but do so by reducing the test accuracy below 30%. DemP achieves better speedups compared with methods maintaining accuracy above 60%. Standard deviation (+) was computed over three seeds.
Test accuracy
Neuron sparsity
Weight sparsity
Training time
Training FLOPs
Inference FLOPs
Method
Dense
74.98% t0.0s
L0x
1.0x (8.15-18)
1.0x (8.2.9)
28.28% 10.08
36.9%
81.4%
0.51x
0.32x
0.32x
SNAP
27.17% +0.07
56.0%
90.1%
0.48x
0.25x
0.25x
28.34%
+0.62
36.9%
81.4%
0.52x
0.32x
0.32x
CroPit-S
27.36% 10.16
53.2%
89.9%
0.47x
0.27x
0.27 x
Structured
68.67% $0.16
51.70%
80.37%
0.95x
0.63x
0.63x
EarlySNAP
63.80% #0.58
66.6%
90.06%
0.75x
0.46x
0.45x
68.26% +0.31
51.60%
79.97%
0.94x
0.66x
0.66x
EarlyCroP-S
64.20%
10.37
66.6%
90.37%
0.82x
0.51
0.50x
71.52%
19.09
61.83%
80.13%
0.81
0.57x
0.49x
DemP-L2
66.34% 10.16
74.1%
89.93%
0.61x
0.42x
0.34x
8

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Results for pruning a ResNet-50 trained on ImageNet with SGDM at approximately 80% and 90% weight sparsity, formatted similarly to Table 1 DemP outperforms comparable structured pruning methods and is also compared with selected unstructured and restructured approaches, including recent DST methods like RigL, SRigL and Chase. While these methods achieve better
tradeoffs in terms of theoretical FLOP reduction, their sparsification not fully GPU-amenable, leading to minimal training and inference speedup, as shown in the last column.
Test accuracy
Neuron sparsity
Weight sparsity
Training time
Training FLOPs
Inference FLOPs
GPU Inf. FLOPs
Method
Dense
75.62% +0.28
1.0x
1.0x (8.15-18)
1.0x (8.2.9)
1.0x (8.2:9)
26.05%
36.90%
81.40%
0.54x
0.33x
0.33x
0.33x
SNAP
24.84% 20.11
56.07%
90.10%
0.47x
0.25x
0.25x
0.25x
25.93% +0.06
36.90%
81.40%
0.50x
0.31x
0.31x
0.31x
CroPit-S
25.44%
53.20%
89.80%
0.49
0.26x
0.26x
0.26x
Structured
70.06% +0.16
38. 73%
79.97%
0.87x
0.57x
0.56x
0.36x
EarlySNAP
64.25% +0.10
57.23%
89.73%
0.78x
0.44x
0.44x
0.44x
71.45%
41.13%
79.77%
0.86x
0.61
0.61
0.61 x
EarlyCroP-S
67.04%
40.
58.77%
90.07%
0.80
0.48x
0.48x
0.48x
72.21%
20.10
53.50%
79.43%
0.88
0.62x
0.58x
0.58x
CemP-LLsso(1))
67.76% +0.12
68.51%
89.73%
0.69
0.46x
0.41x
0.41x
Dense
76.67%
Denses 4$
76.8 +0.00 %
1.0x
1.0x
1.0x
Mag'
75.53%
80%
1.0x
Salt 74.93%
80%
1.0x
Unstructured
72.9%20.30
80%
0.23x
0.23x
1.0x
SET*
69.6%
90%
0.10x
0.10x
1.0x
75.10%1o.oz
80%
0.42x
0.42x
1.0x
RigL (ERK)*
73.00%+oo.0
90%
0.25x
0.24x
1.0x
75.01%
22%
80%
0.36x
0.41x
=1.0x
Restructured
SRigL4
72.71%
<55%
90%
0.24x
0.24x
11.0x
75.27%
40%
80%
0.39x
0.37x
0.68x
Chase
74.03%
40%
90%
0.26x
0.23
0.67
values obtained from Lec al.
(2023)
evalues obtainod from Evei et al.
(2020)
values obtained from
Lasby al. 2024 neuron sparsity estimated from Fig. 3b
$values obtained from Yin et al,
2023
a ResNet-50 on ImageNet (refer to Table 2 Additionally, DemP provides significant speedup when training on ImageNet with both optimizers, surpassing baselines yet again (Tables and 2
Furthermore, as shown in Fig. 15 these conclusions hold when ResNet-18 trained with Leaky ReLU
activations instead of ReLU, highlighting DemP's adaptability across different activation functions.
9

---

Published in Transactions on Machine Learning Research (02/2025)
Adam
SGDM
0.925
Tost Accuracy
0.925
0.900
0.900
0.875
0.875
75
80
90
95
100
75
80
90 Weight Sparsity (%)
100
Weight Sparsity (%)
Constant
Cosine Deeay
One Cycle
Warmup
Figure 4: Impact of different schedules the regularization parameter for DemP, concluding that one-cycle scheduler is good default choice. Experiments were performed with ResNet-18 CIFAR-10, with Lasso(7) regularization, across three seeds. Higher sparsities are obtained by increasing the peak strength of the added scheduled regularization. Left: With Adam optimizer. Right: With SGDM optimizer.
4.2 Ablation
Regularizers. While the methodology works with both L2 and Lasso regularization, and either applied to the entire parameter space or specifically to the scale normalization parameters (7), we empirically found that Lasso regularization of the scaling worked best for SGDM (Fig. 5) For Adam, L2 regularization of the scaling slightly outperforms Lasso regularization of the scaling (Fig. 5) As such, applying regularization solely to the scale parameters beneficial to performance. For simplicity, we opted for Lasso regularization of the scale parameters in all subsequent experiments. Nevertheless, in the absence of normalization layers, traditional regularization methods remain viable for inducing sparsity in the model
Dynamic Pruning. To realize computational gain during training, we dynamically prune the NN at every steps (with a default of = 5000). Dead neurons are removed almost as soon as they appear, preventing their revival. This strategy enables faster training with no significant change in performance as shown in Fig, 7 We note that this smooth gradual pruning process compatible with our approach in part because there is no added cost for computing the pruning criterion.
Regularization Schedule. We test our hypotheses about regularization scheduling by comparing the one-cycle method with warmup, cosine decay, and constant schedules. Empirically, we confirm in Fig. 4 that using a one-cycle scheduler for the regularization parameter (A) is good strategy (Appendix F.4
Adam
SGDM
Test Accuracy
0.92
0.90
0.90
0.85
60
70
80
90
100
60
70
80
90
100
Weight Sparsity (%)
Weight Spansity (%)
L2
L2(7)
Lasso
Lasso(?)
Figure 5: ResNet-18 networks trained CIFAR-10 with different added regularization strategies, over three scods. ((7) denotes when regularization is only applied the scale parameters of the normalization layer. Higher sparsities are obtained by increasing the peak strength of the added scheduled regularization. Left: With Adam, using L2(y) regularization slightly outperforms other strategies. Right: Using SGDM, the differences in performance become more pronounced, with Lasso regularization applied to scale parameters providing a favorable balance between sparsity and performance.
10

---

Published in Transactions on Machine Learning Research (02/2025)
0.94
Tost Accuracy
0.7
0.0
5c-05
0.92
0.0001
0.6
0.0005
0.0
0.001
5e-05
0.5
60
70
80
90
100
20
40
60
80
100
Weight Sparsity (%)
Weight Sparsity (%)
Figure 6: Left: ResNet-18 networks trained on CIFAR-10 with SGDM over three seeds. Adding a very small =
amount of Gaussian noise 5o²=5 10 10-5) to the live neurons offers the best balance between performance and sparsity, Right: The addition of noise proves very impactful in increasing sparsity without affecting performance a ResNet-50 trained on ImageNet. Each symbol represents distinct peak regularization value for scale parameters, where each daia point represents distinnt seed.
Adam
SCDM
Tost Acouracy
0.90
0.90
0.85
0.85
0.80
40
60
80
100
20
80
100
Weight Sparsity (%)
Weight Sparsity (%)
Dynamic Pruning
False
True
Figure 7: Measuring the impact of dynamic pruning on the accuracy-sparsity tradeoff. The symbols denote different regularization strengths. Experiments were performed with ResNet-18 on CIFAR-10 10 and Lasso(y) regularization (three seeds). Left: With Adam, there are minimal differences between the two strategies, indicating that dynamic pruning does not affect performance. Right: With SGDM, dynamic pruning further improves sparsity without degrading performance, while also providing training speedup.
Added noise. Adding asymmetric noise is particularly beneficial to further increase sparsity (see Fig. G
The noise is kept small, with peak variance at o² = x 10 and following the same schedule as the added regularization for maximal effect. We believe it acts as the small additional push helping neurons that linger close to the inactive region boundary to cross it.
We conducted experiments assess the necessity of using asymmetric noise DemP (Fig. 13
We found that when all other aspects of DemP remained constant (including dynamic pruning), asymmetric noise not strictly essential. Yet, note that dynamic pruning mimics the effect of asymmetric noise: removal of dead neurons also removes the possibility of random revival. We decided to keep the added noise asymmetric due to significant differences observed in other settings (Fig. middle) where symmetric noise reduces dead neuron accumulation considerably due to neuron revival.
Additional ablations on the death criterion relaxation and weight decay are discussed respectively Appendix F.3 and Appendix F.3 We also discuss the choice of activation functions in Appendix F.7 showing that DemP offers similar performance when using Leaky ReLU activations (Fig. 15
11

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Using DemP to prune the MLP layers of ViT-B/16 model. Solely by injecting artificial noise
(A = 0), DemP removes up 90% of the neurons. We report the neural sparsity across the MLP layers but provide the equivalent weight sparsity for the entire network.
Test accuracy
Neuron sparsity
Weight sparsity
Training time
Method
Dense
78.75%
1.0x
DemP
71.38%
90.6%
59.4%
0.77x
DemP ReLU
77.62%
43.7%
29.6%
0.93x
4.3 Transformers
To assess the compatibility of DemP with transformer- basedd architectures, we conducted exploratory experi- ments on Vision Transformer (ViT) networks (Dosovitskiy et al., 2021). Our focus was on pruning the fully connected layers because they contain the majority of the parameters and use GELU activation functions. Notably, observed substantial number of neurons dying during regular training, which led to explore DemP's performance without regularization (A = 0), retaining only the added noise.
The exploratory results for a ViT-B/16 model trained on ImageNet are presented in Table 3 that 90% of the fully connected neurons were readily pruned using DemP. Additionally, replacing the GELU activations with ReLU (Mirzadeh et al. 2024) resulted in only a slight performance degradation while still allowing the pruning of 43% of the neurons. Further details are provided in Appendix G.2
We found
5 Conclusion
In this work, we have explored how various hyperparameter configurations such as the learning rate, batch size, regularization, architecture, and optimizer choices, collectively influence activation sparsity during neural network training. Leveraging this, we introduced Demon Pruning, dynamic pruning method that controls the proliferation of saturated neurons during training through combination of regularization and noise injection. Extensive empirical analysis on CIFAR-10 and ImageNet demonstrated superior accuracy-sparsity tradeoffs compared to strong structured dense-to-sparse training baselines.
The simplicity of our approach allows for versatile adaptation. Integrating our method with existing pruming techniques is straightforward. Specifically, since DemP acts on the training dynamics, can be used conjunction with other dynamical pruning methods (Evci et al. 2020 Lasby et al., 2024 Yin et al. 2023 We discuss limitations of our approach in Appendix
Broader Impacts Statement. Structured pruning methods, without specialized sparse computation primitives (Elsen et al. 2020 Gale et al. 2020) can elficiently leverage GPU hardware (Wen et al.) 2016)
compared to unstructured methods, which is crucial as deep learning models grow and environmental impacts escalate (Strubell et al.
2019 Lacoste et al. 2019 Henderson et al. 2020). Developing energy-officient
methods that can widely adopted essential.
However, while efforts to enhance the efficiency of deep learning training processes can reduce computational costs and energy requirements, they may inadvertently amplify concerns associated with the rapid advancement of AI. The swift progress in AI capabilities raises significant risks, from ethical dilemmas to information manipulation. By accelerating AT development and increasing its accessibility, research like ours may exacerbate ongoing issues.
Acknowledgments
This research was enabled in part by compute resources provided by Mila, the Digital Research Alliance of Canada, and NVIDIA.
12

---

Published in Transactions on Machine Learning Research (02/2025)
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, volune 232 of Proceedings of Machine Learning Research, pp. 620-63606 PMLR, 2023.
Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR, abs/1803.08873, 2018
Naman Agarwal, Rohan Anil, Elad Hazan, et al. Disentangling adaptive gradient methods from learning rates. CoRR, abs abs/20002.10 11803, 2020.
Russell W. Anderson, Chapter 7 biased random- walk learning: A neurobiological correlate to trial-and-error. In Newral Networks and Pattern Recognition, pp. 221-244. Academic Press, San Diego, 1998.
James Bradbury, Roy Frostig, Peter Hawkins, et al. JAX: composable transformations of Python NNmPy programs, 2018.
Xiang Cheng, Dong Yin, Peter Bartlett, el al. Stochastic gradient and Langevin processes. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1810-1819. PMLR, 13-18 Jul 2020.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Lc. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, volume 33, pp. 18613 18624, 2020.
Jia Deng, Wei Dong, Richard Socher, et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), pp- 248-20 253. IEEE Computer Society, 2009.
Xiaohan Ding, Guiguang Ding, Jungong Han, et al. Auto-balanced filter pruning for efficient convolutional neural networks. In Proceedings of the Thirty-Second AAAT Conference Artificial Intelligence, (AAAT-18), pp. 6797 6804. AAAI Press, 2018.
Shibhansh Dohare, J. Fernando Hernandez-Garcia,, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nat., 682(88066 7677774, 2024.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, et al. Sample-efficient reinforcement learning by breaking the replay ratio barrier. Tn The Fleventh International Conference on Learning Representations, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Erich Elsen, Marat Dukhan, Trevor Gale, et al. Fast sparse convnets. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 14629 14638, 2020.
Utku Evci. Detecting dead weights and units in neural networks. CoRR, abs/ 1806.06068, 2018.
Utku Evci, Trevor Gale, Jacob Mcnick, Rigging the lottery: Making all tickets winners. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Tearning Research, pp. 2943 2952. PMIRR 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, et al. Lincar mode connoctivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pp. 3259 3269. PMI.R, 2020.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, hlss/1902.09574 2019.
Trevor Gale, Matei Zaharia, Cliff Young, et al. Sparse gpu kernels for deep learning. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14. IEEE, 2020.
Priya Goyal, Piotr Dollár, Ross B. Girshick, et al. Accurate, large minibatch SGD: training imagenet in hour. CoRR, abs/ hbbs/770.00267. 2017.
Song Han, Jeff Pool, John Tran, et al. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems 28: Annual Conference on Newral Information Processing Systems 2015, pp. 1135 1143, 2015.
Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, pp. 1141-1150, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pp. 770-778. IEEE Computer Society, 2016.
Peter Henderson, Jieru Hu, Joshua Romoff, et al. Towards the systematic reporting of the energy and carbon footprints of machine learning. The Journal of Machine Learning Research, (1(1):10039-1008) 2020.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, et al. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the Thirty-Second AAAT Conference on Artificial Intelligence, (AAAI-18), pp. 3215 3222. AAAI Press, 2018.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, et al. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. CoRR, abs/1607 1b5/1660.00250, 2016.
Ioannis Karatzas and Steven E Shreve. Brownian motion and stochastic calculus. Springer New York, NY, 2014.
Nitish Shirish Keskar, Dheovatsa Mudigere, Jorge Nocodal, al. On largo-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 1CLR 2015, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, et al. Quantifying the carbon emissions of machine learning. CoRR, abs/1910.09970, 2019.
Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani Ioannou. Dynamic sparse training with structured sparsity. Tn The Twelfth International Conference on Learning Representations, ICLR 2024, 2024.
G.F. Lawler. Introduction to Stochastic Calculus with Applications. Taylor & Francis, 2016.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems 2, NIPS Conference, pp. 598-605. Morgan Kaufmann, 1989.
Joo Hyung Lee, Wonpyo Park, Nicole Mitchell, et al. Jaxpruner: concise library for sparsity research. 2023.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruming based on connection sensitivity. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Hao Li, Asim Kadav, Igor Durdanovic, et al. Pruning filters for efficient convnets. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Newral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, pp. 11669 11680, 2019.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pp. 7021-7032. PMLRR 2021.
Yinhan Liu, Myle Ott, Naman Goyal, et al. Roberta: A robustly optimized BERT pretraining approach. CoRR, abbs/1907.11699 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, et al. Learning efficient convolutional networks through network slimming. In JEEE International Conference on Computer Vision, ICCV 2017, pp. 2755-2763. IEEE Computer Society, 2017.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 1_ regularization. 2018.
Lu Lu, Yeonjong Shin, Yanhui Su, et al. Dying relu and initialization: Theory and numerical examples. arXiv preprint Xiv: 11111993300737 2019.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, et al. Understanding plasticity in neural networks. International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 23190 23211. PMLR, 2023.
Sangkug Lym, Esha Choukse, Siavash Zangench, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunctrain: fast neural network training by dynamic sparse model reconfiguration. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, pp. 16:1-36:13. ACM, 2019.
Andrew L. Maas, Awni Y. Hanmun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library.
con/pytorchhviiionn 2016.
Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks. CoRR, abs/1804.07612, 2018.
J.C. Maxwell. Theory of Heot. Textbooks of science. Longmans, Green, and Company, 1872. ISBN 9780598862662.
Seyed- Mirzadeh, Keivan Alizadeh- Vahid, Sachin Mehta, Carlo C. del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, et al. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature commmunications, 9(1):2383, 2018.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, et al. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828-1666 16847. PMTRR 2022.
Bernt Karsten Oksendal. Stochastic differential equations: an introduction with applications. 6th edition. Springer Berlin, Heidelberg, 2010.
Loucas Piliend-Vivien. Rethinking sgd's noise. httpss//ffaaccccccccccc//////acp
2022.
Accessed: 2023-08-26.
John Rachwan, Daniel Zügner, Bertrand Charpentier, et al. Winning the lottery ahead of time: Efficientt carly network pruning. In International Conference on Machine Learning, ICML 2022, volume 162 of Proceedings of Machine Learning Research, pp.
PMLR, 2022.
Prajit Ramachandran, Barret Zoph, and Quoe V. Le. Searching for activation functions. In 6th International Conference on Learning Represenlations, ICLR 2018, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates, 2018.
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, et al. Don't decay the learning rate, increase the batch sizc. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, et al. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 32145-322 32168. PMLR, 2023.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning NL.P Tn Proceedings of the 57th Conference of the Association for Computational Linguisties, ACL 2019, pp. 3645 3650. Association for Computational Linguistics, 2019.
Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Locally free weight sharing for network width search. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman A. Zonouz, and Bo Yuan. CHIP: channel independence-based pruning for compact neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, pp. 24604-24616, 2021.
Ludovie Trottier, Philippe Giguere, and Brahim Chaib-Draa. Parametric exponential linear unit for deep convolutional neural networks. In 2017 16th IEEE international conference on machine learning and applications (ICMLA), pp. 207-214. IEEE, 2017.
Stijn Verdenius, Maarten Stol, and Patrick Forré. Pruning via iterative ranking of sensitivity statistics. CoRR, hts//0006008966, 2020.
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse, Picking winning tickets before training by preserving gradient flow. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Huan Wang, Qiming Zhang, Yuehai Wang, et al. Structured pruning for efficient convnets via incremental regularization. In International Joint Conference on Neural Networks, IJCNN 2019, pp. 1-8. IEEE, 2019.
Huan Wang, Can Qin, Yulun Zhang, et al. Neural pruning via growing regularization. In Ith International Conference on Learning Representations, ICLR 2021, 2021.
Wei Wen, Chunpeng Wu, Yandan Wang, al. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, 6, pp.
2016.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Stephan Wojtowytsch. Stochastic gradient descent with noise of machine learning type part I: discrete time analysis. J. Nonlinear Sci., (3(3):45, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, al. Transformers: State-of-the-art natural language processing.
Tn Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45. Association for Computational Linguistics, October 2020.
Bing Xu, Naiyan Wang, Tianqi Chen, et al. Empirical evaluation of rectified activations convolutional 2015.
network. CoRR,
Jianbo Ye, Xin Lu, Zhe Lin, et al. Rethinking the weter--oooo---e---ioooonoe assumption in channel pruning of convolution layers. In 6th International Conference on Learning Represenlations, ICLR 2018, 2018.
Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, and Shiwei Liu. Dynamic sparsity is channel-level sparsity learner. In Advances in Newral Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, 2023.
Haoran You, Chaojian Li, Penglei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Zhonghui You, Kun Yan, Jinmian Ye, et al. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, pp. 2130-2141, 2019.
Jiahui Yu and Thomas S. Huang. Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers. CoRR, Abs/1903, 11728, 2019.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 TEFE/CVP International Conference on Computer Vision (ICCV), pp. 6022-6031, 2019.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Hao Zhou, Jose M. Alvarez, and Fatih Porikli. Less is more: Towards compact enns. In Computer Vision ECCV 2016, volume 9908, pp. 662 677. Springer, 2016.
17

---

Published in Transactions on Machine Learning Research (02/2025)
A Dead neurons in convolutional layers
In convolutional layers, ReLU is applicd element-wist to the pre-activation feature map. We consider individual neuron (channel) dead if all elements in the feature map after activation are 0. Formally, in this case, the definition from Section 2
becomes:
The j-th neuron/channel in the convolutional layer & is inactive if it consistently outputs a Teature map (post-activation) with elements summing zero on the entire training set, ie Σb, jju = 0. A neuron/channel that becomes and remains inactive during training considered as dead.
Definition:
B Biased Random Walk Model
This section aims to illustrate the role of asymmetric noise in neuron saturation through simple theorelical models, formalizing the intuition presented in Section 2
Sctup. We consider a network with parameter w trained to minimize the loss L(w) = E ((w), where (ww) is the loss function on sample i, using stochastic gradient descent (SGD) based methods. At each iteration, this requires an estimate of the loss gradient g(wr) := VI(w)) obtained by computing the mean gradient on a random minibatch b {1 nI...nJ. For simple SGD with learning rate n, the update rule takes the form
1
Wi+1 wi ng( by), ((w,b)
Vl.(w).
(2)
[6]
iEb
formalize the intuition from section 2.1 we follow a standard line of work (Cheng et al. 2020 taking the view of SGD in Eq. 2 biased random walk (Anderson 1998), described by the Langevin process,
wi+1 wi () E
33)
where the zero mean variable É(w,,) b) == v (g(w)) - ((w, (w, b)) represents the gradient noise. In the limit of small learning rate, Eq. 3 is also well approximated (Cheng et al. 2020 Theorem 2) by the stochastic differential equation (SDE),
dw,- -g(w:) + (Ww)dd.
(4)
where B, denotes standard Brownian motion and (ww) = VELlE(W, b)ε(w,
In SGD, a crucial characteristic of the gradient noise is its multiplicative nature, mcaning depends on the parameter. Systems with multiplicative noise exhibit a well-known property where regions with lower noise magnitude tend to as attractors [Oksendal 2010). Intuitively, the noise propels the system away from regions where it has a higher impact, leading to higher probability of staying regions where it has a lower impact. Mathematically, this manifests tendency for the invariant distribution associated with SDE in Eq. to have higher probability density in regions of lower noise magnitude.
We illustrate this phenomenon using two (drastically) simplified versions of the dynamics described by Eq. and !
B.1 Absorbing random walk
We consider one-dimensional absorbing random walk with a boundary at zero described by the SDE:
dw: { {
VndB. long as w.
(5)
otherwise
It models a system subject to noise an 'active' region w > O, which gets stopped at (-and remains there, hence -once it hits 0. It can be thought of a simplified description of a regime where the dynamics (Eq.Be 3) is dominated by noise, such e.g., neuron encoding features with very low correlation to the task.
18

---

Published in Transactions on Machine Learning Research (02/2025)
The survival probability at time is the probability that the system is still active t, ie we 0. It is
related to the distribution of the first hitting time at 0 of a standard Brownian motion, P(To > L), where
To = itfft 2 0 B, = 0}. A well-known property of Brownian motion (Karatzas & Shreve 2014) is
lim P > t) = 0, which shows that the system in Eq. 5 eventually dies with probability 1. More
generally, the following proposition specifies the dependence on learning rale and initialization:
Proposition B.1. Consider the system
S
initialized at wo > 0. The survival probability at time t >> 0 is
given by
2 wo/ V
Wo 22m
P(w; > 0| wo) erf
dw
(6)
Prop. B.1 (i) confirms that the system eventually dies almost surcly, since for all we > the survival
probability decays to 0 l +o0 (ii) implies that for any given finite horizon time L, the smaller the initialization, the more likely the system is to be dead at t, (iti) illustrates how noisier environment (i.e higher diffuusive coefficient y representing the learning rate) accelerates this dying process.
Proof. This is a standard application of the reflection property of Brownian motions (e.g., Lawler 2016 Let Wy be the solution of Eq. 5 with no boundary condition. For the same initial condition, wy and vù have the same distribution as long as W; 0. Let To = inf 2 0 i; = 0} be the first hitting time wy at The survival probability of wy can be expressed terms of the distribution function of Ta as
P(w: > wo) = P S
(7)
The reflection property states that P(T S t) 2P(w; s 0). To show this, let us first use the law of total
probability to decompose P(T. I t) as
P(To S 1) P(T < I, 0) P(To < S1, Wz > 0)
For the first term, we note that w, S 0 implies To < with probability 1, so P(To St S 0) = P(w, S 0). For the second term, we note that by the strong Markov property, W, := W Tr 2 To is a (scaled) standard Brownian motion, whose distribution symmetric about the origin: therefore P(w, > 0 S t) P(w; o s t). Thus,
P(To S, 0) P(w, > 0m%o S P) S))
P(w, 0 < (T <
P(ū, 0,T, t)
(9) where have used the fact that P(w: 0. As before, this last term equals P(w, S 0); and the reflection property is proved. Finally, dw, Bn is scaled Brownian motion with initial value wo, so w, is normally distributed with mean wo and variance rtt. Thus,
P(w, S 0,T, t)
mo! /ot 2 e - dw
P(w, > 0 w)) = - 2P S 0) = V nn
B.2 Geometrical random walk
The second example illustrates the stabilizing effects of multiplicative noise for systems such Eq. 3 near
unstable critical points (Oksendal 2010). Consider the SGD dynamics (Eq. 2 with diagonal quadratic approximation of the loss around 0, i.e., we assume
1
4(w) = (0) + w H H. = Diag(ha, ... haa)
(10)
19

---

Published in Transactions on Machine Learning Research (02/2025)
where H2 is some sample-dependent diagonal matrix. In such setting, model the dynamics of each
parameter by geometrical random walk,
W1+1 = Wt - 17h + Ge)
(11)
where h is one of the Hessian eigenvalues and Ga is a noise variable sampled from some zero mean distribution.
The case of a negative eigenvalue (h 0) is particularly interesting since it corresponds to an unstable direction (negative curvature) in the absence of noise. In what follows, to ensure the stability of the dynamics, assume that the noise variable Ç is bounded and the learning rate is small enough to ensure that nls1 < 1. We also assume the noise distribution is symmetric about 0.
Lemma B.2. Let μ = Ellog(1 (1 - 17(h + {))]. For all e >> 0 and ò > O, there exists to(e, such that for all 2 to; with probability ot least 5,
e 11000)1001 S |wel S elttt) lwol
(12)
In particular, wt - O w.h.p whenever < O.
Proof. This is consequence of the law of large numbers applied to the mean 2, 2g == log(1 + m/h + (j)); for all > 0, lim P( - 14| < E) = 1. This implies that w.h.p,
= =,= Σ 23 of i.i.d variables
+
(u-o) S τ² S ((to))
(13)
Now solving Eq.
yields
t-1 n1h + (i) wD E wo e5.
Wr- wo (1
II
(14)
j=1
j=1
Combining with Eq. 13 proves Lemma
B.2
Lemma B.3. There crists range of values for the learning rate for which < 0, making the direction
stable w.h.p, despite having h < 0.
Proof. This is consequence of the inequality (z) < x + for all -1. Applying this incquality to = (n + () and taking the average over ( gives the upper bound
2 2 (12 + +
- _h (hh + 3ho
n2
S -nh
(15)
3
where o² ;= E| and we used E[[] = E|s3 = 0 by symmety about O. Now, the sign of this bound coincides with the sign of the degree polynomials (n) == |I| - 3 (h² + + + "² [ 30²). We note that P(0) >
+ and that for a small enough ratio /h/o,, it has two positive roots bounding interval on which P(n) < 0. One way to see this is to compute the minimum
3 (h2 + 02)2
min P(n) = h
(16)
16 h1/2² + 302)
and to observe that it goes to -00 as |M| + 0+ for fixed o².
C Few Dead Neurons Revive
While empirical observations have shown a gradual accumulation of dead neurons (Fig. D we also observed that neurons can revive (Appendix D.2 To better assess the potential impact of reviving neurons on performance, measured the overlap ratio (IX Y// min([X], (Y]) between the historical set of dead neurons at previous itcrations and the set of dead neurons at the current iteration. This methodology directly follows Sokar et al. (2023). The results in Fig. g show that most neurons (over 90%) inactive al any point during training end up dead at the final iteration. This this--cuuped with our results showing that dying neurons can be dynamically pruned during training without impacting performance (Appendix F strongly suggests that neurons becoming inactive any point during training in ReLU networks do not contribute significantly to the final performance of the trained model.
20

---

Published in Transactions on Machine Learning Research (02/2025)
Adam
SGDM
1.0
1.000
Dead Neurons
0.8
0.999
0.998
0.6
20000
40000
60000
80000
20000
40000
60000
80000
Training Step
Training Step
Reg. Strength (A):
0.0001
0.001
le-05
0e-05
Figure 8: Overlap ratio of dead neurons during training, as measured across all layers of a ResNet-18 trained on CIFAR-10 at various maximal regularization strengths (Lasso(1))- Results are shown for training steps bigger than 15k when dead neurons become observable across all regularization strengths. The observation that most dying neurons remain dead justifies their early removal. Left: With Adam, observe that almost all (over 99%) neurons dying never revive. Right The picture is more nuanced with SGDM, yet we find that the majority of dying neurons are still dead when training finishes (~ 75%). Neural revival mostly happems when the learning rate is decayed, followed right after by a phase where most of the revived neurons die again.
Hyperparameters Impact, Additional Results
D.1 Noise, Learning Rate and Batch Size
To investigate the role of noise, trained 3-layer deep MLP (with layers of widths 100, 300, and 10 respectively) subset of 10 000 images of the MNIST dataset. To isolate the noise from minibatch (of size 1) gradient (i(w))) we subtract from it the full gradient (g(w))), taken over the entire training dataset. As such, the update of neuron j every time step given by wtt w n(g(w()) g(wf)).
4000
Doad Neuruns
300
200
2010
10
100
0
0
10000
20000
30000
10000
20000
30000
10000
20000
30000
Suee
Steps
Step-
0.025
0.05
0.075
0.1
Figure 9: Evolution of the number of dead neurons for 3-layer MLP (with layers of widths 100, 300, and 10 respectively) subset of MNIST. Left: The noisy part of the minibatch gradient is isolated and used exclusively to update the parameters. Noisy updates are sufficient to kill subset of neurons following standard initialization. Because SGD gradient. is 0 for dead neurons, there is an asymmetry: only live neurons are subject to noisy updates. Center: Gaussian noise is added to the parameters update, either asymmetrically (applied only to live neurons, plain lines) or symmetrically (dashed lines). Asymmetric noise is much more prone dead neuron accumulation while symmetric Gaussian noise can revive neurons, contrary to SGD noise, leading to much smaller accumulation. Right Standard SGD. Dead neurons accumulate quickly in noisy settings, but they plateau when the NN converges (leading to zero gradient). Results are averaged over three seeds.
21

---

Published in Transactions on Machine Learning Research (02/2025)
20
ADAM--01C
XDAM--0 (1
SCIDM-10 -62
Live Neucne Easto
4.3
Dead Neurons Variation 12
27
68
0.01
0.025
0.05
0.075 0.1
16.0
32.0 Batch Size
644
128.0
256.3
Finst Layer Wilth
Learning Rate
Figure 10: Left: An increased width leads to a higher ratio of neurons dying, independently of the optimizer. Measured on a ResNet-18 trained on CTFAR-10, without added regularization, across three scods. We use the number of channels the initial layer of the ResNet-18 to indicate the width, with 64 being the number of channels proposed by He et al. (2016) for the first convolution layer. Right: Varying the hyperparameters of ResNet-18 (CIFAR-10) impacts the number of dead neurons. The bar heights indicate the multiplicative ratio of dead ncurons compared to the base configuration (Ir = 0.05, bs 128 and X = 0). The number of training steps was kept constant when varying the batch size for fair comparisons. Quantities are averaged over three random seeds.
This approach has the benefit of preserving the asymmetric noise structure of SGD updates (Wojtowytsch) 2023 Illaud-Vivien 2022), where dead units are not affected by noise but live ones are. Compared to applying solely symmetric Gaussian noise at every time step, we notice a much sharper accumulation of dead neurons. The details are shown in Fig.
Diminishing the batch size or augmenting the learning rate also creates a noisier environment because both hyperparameters affect the noise covariance in SGD optimization (Keskar al., 2017 Masters & Luschi 2018 Goyal et al. 2017 He et al. 2019 Li ct al. 2019 Furthermore, /Smthh et al. (2018) shows that learning rale decay can be replaced with batch size growth, emphasizing the relationship between the two quantities. Because of their impact on noise, we should expect those quantities to affect the dying ratio of neurons. We confirm empirically this hypothesis in Fig. 10
D.2 Training Time
The relation with training time, asserting that the probability of neuron dying increases training progresses (Prop. B.1. docsn't entirely align with practical applications. Modern overparameterized architectures often can memorize the entire training dataset, achieving zero loss in the process. Given that the gradient signal is proportional to the loss, would concurrently diminish to zero for all neurons, preventing any further death.
We observe a pattern consistent with this idca (Fig. 1. where the total count of dead neurons spikes sharply in early training to then fluctuate slightly before stabilizing. The fluctuations demonstrate that neurons can indeed revive. However, additional experiments with ReLU networks revealed that most reviving neurons die again later (Fig. gh and that their dynamic elimination has negligible to impact performance (Fig. 7
D.3 Network Width
The widths of a neural network's layers also influence the ratio of live neurons (live neurons to total neurons in the network) post-training (see Fig. 10). Typically, this ratio incrcases with the width; however, the total number of live neurons continues rise with increased width. This phenomenon is somewhat anticipated as incorporating more neurons with random initialization in any given layer can only amplify the training noise, especially in the initial phase. Moreover, since initialization functions usually adjust their standard
deviation proportionally to the number of channels (a x V fan int out), widening the network places neurons closer to their inactive region right from the initialization. The connection between width and
22

---

Published in Transactions on Machine Learning Research (02/2025)
dead neurons maintains its significance neural network sizes are inclined to increase time with the availability of more computational resources. If this trend persists, the accumulation of dead neurons could potentially become increasingly pervasive.
E
Adam is Neuron Killer
The greater impact of Adam over the dying ratio compared to momentum must be due to the recond-moment term, which is the only significant difference with momentum. Recall that Adam update (Kingma & Ba 2015 is given by:
my = B1 -7--1 1 + (1-11 (1) -1)-9. 9t
2/ = B2 1/-1 + (1 32)
m
m = - Pi
Ut
02 1 - $
1]
Oc+1 0:
m
Vts +
Earlier, we hypothesized that the neurons ending up dead were the ones experiencing very small gradients, such that the noise dominated their update trajectories. If this is the case, of (the squared gradient) would be very small for those neurons' parameters, eventually leading to very small second-moment estimation iz- In such scenario, would end up dominating Vū,, effectively multiplying the learning rate by e (e is typically set to x 10-*). Moreover, as the decay (32 0.99) of ûz is usually slower than the one of m. (81 = 0.9), few sudden noisier updates would be sufficient to make huge random steps.
It worth noting that RL practitioners typically set epsilon to a higher value (Hessel el al. 2018), as has empirically been found perform better. Because of constant distribution shifts, rapid accumulation of dead neurons often occurs in RL tasks (Lyle et al. 2022 Sokar et al., 2023). Higher values should reduce the number of dead neurons induced by Adam optimizer, by making the eflective learning rate (#) smaller when dominates Vo, We confirm this hypothesis in Fig. 11 confirming the soundness of picking higher to improve performance and stability in RL.
Also notable, HuggingFace Transformers library (Wolf et al. 2020) default Adam parameter to 1 x 10-6, following RoBERTa example (Liu et al. 2211)). Manipulating the e parameter AdaGrad was also observed impact significantly transformer performance model (Agarwal al., (2020). Confirming that those heuristic choices are due to their impact on dead neuron accumulation would be quite interesting.
F Pruning Method Ablation
We validate and justify the heuristic choices made for our pruming method, summarized in Algorithm 1 via empirical observation exposed in this section. We used the same setup as before for a ResNet-18 trained on CIFAR-10.
F.1 Regularizer choice
In our empirical analysis, evaluated the effectiveness of various regularizers on the performance of ResNet-18 networks trained on CIFAR-10, using cither Adam or SGDM as optimizers. Our findings (Fig. 5) suggest that focusing regularization exclusively on scale parameters yields a more lavourable balance between sparsity and performance with both optimizers. While L2 regularization on scale parameters slightly enhances performance, the scenario changes with SGDM, where Lasso regularization on these parameters outperforms others by a wider margin. Consequently, for the sake of simplicity in our experiments, we have chosen consistently apply Lasso regularization to scale parameters.
23

---

Published in Transactions on Machine Learning Research (02/2025)
Learning Rate
12.5
F
0.005
0.01
Neuron Sponsity (%)
10.0
7.5
5.0
2.5
0.0
10
10
10
Adamm (c)
Figure 11: Sparsity decreases from 5.07+2.04% 0% when varying Adam parameter from 1e-10 to le-4, from 12.6%+1..5% to 0.47%11.00 when the learning rate is increased, supporting our claim that the usage Adam optimizer impacts the accumulation of dead neurons through parameter. We note that this additional evidence complements the findings from Lyle et al. 20022), which reported reduced plasticity loss with increasing Experiments were performed with RosNet-18 on CIFAR-10 over three sceds, without using the same training recipe as in Appendix
DemP,
F.2 Dynamic Pruning
To verify the impact of dynamic pruming, we measured if there were any performance discrepancies when it was enabled or not. Across runs, we varied the regularization strength while measuring accuracy and sparsity. The results, in Fig. show that enabling dynamic pruning does not affoct the final performance. The very slight variations between
fall well between the expected variance across different runs. This experiment reinforces the hypothesis that neurons that die and later revive during training do not contribute significantly to the learning process.
F.3 Death Criterion Relaxation
The definition we choose for a dead neuron asks for it to be inactive to the entire dataset. In practice, we found that this criterion could be relaxed and defaulted to using 512 (2048 for ImageNet) examples from the training dataset to mcasure the death state (measuring across multiple minibatches when necessary). Fig. 12 shows that using this proxy for tracking dead units is sufficient. With this relaxation of the criterion, pruning interventions become highly efficient, incurring at most the computational cost of a few forward propagation batches. To measure if a minibatch could be used to measure the death state instead of the entire dataset, tracked the number of dead neurons during training with both metrics in Fig. 12 We can that both curves closely track each other. More importantly, they match at the end of the training, indicating that overall the same amount of neurons would be removed when performing the death check over the minibatch. Dynamic pruning was disabled for this experiment.
F.4 Regularization Schedule
We also empirically lested different schedules over the regularization parameter in Fig. 4 trying to mitigate the impact high regularization by decaying the parameter throughout the training after a warmmp phase. We settled on using a one-cycle schoduler for the regularization strength because of slightly better performance in the higher sparsity level. However, we remark that all tested schedules over the regularization parameter remain sound with method.
24

---

Published in Transactions on Machine Learning Research (02/2025)
Adam
SGDM
1000
100
Dead Neurons
500
50
0
20000
40000 Training Step
60000
80000
0
20000
40000
60000
80000
Training Step
Entire dataset
512 examples
Figure 12: Instead validating the death state of neurons against the entire training dataset, proves sufficient use smaller datasct. The curves match throughout training, indicating that roughly the same number of neurons are removed with both strategies. Experiments were performed with ResNet- 18 and Lasso(7) regularization on CIFAR-10 over three seeds. Left: With Adam. Right: With SGDM.
F.5
Weight Decay
Our method defaults back to traditional regularization, with a term added directly to the loss, as opposed to the weight decay scheme proposed by [oshchilov & Hutter (2019). By doing so, the adaptive term in optimizers takes into account regularization, and neurons move more quickly toward the inactive region. From a pruning perspective, achieves a higher sparsity than weight decay for the same regularization strength. Note that we are referring here the added one-cycle regularization on scale parameters, which on top of potential constant weight decay applied as part of the original training recipe.
F.6 Added Noise
We measured empirically the impact of adding artificial asymmetric noise. As expected, adding too much noise hurts performance. However, when noise variance is small enough, it doesn't 't affect performance while helping neurons to cross over in the inactive region, as detailed in Fig. G The effect is particularly significant when training a ResNet-55..
Furthermore, we assess that increasing noise instead of regularization is also valid strategy for pruning, showcased in Fig. 13 However, it leads to worse tradcoff than when using increased regularization. Significantly, in this setting, with dynamic pruning enabled, notice major difference in performance between using asymmetric or symmetric noise.
F.7 Activation function
Dead neurons are naturally defined with ReLU activation functions, for which neurons can completely dcactivate. However, most activation functions, such as Leaky ReLU (Maas et al., 2013), also exhibit a "soft" saturated region. We postulate that neurons firing solely from the saturated region do not contribute much to the predictions and can be considered almost dead. We test this hypothesis by employing our method in network with Leaky ReLU activations (Fig. 15), removing neurons with only negative activation across a large minibatch. Again, our method can outperform baselines with Adam and offers a similar performance with SGDM.
G Additional Results
We report additional results in this section, including the pruning performance of DemP when pruning a VGG-16 (Fig. 14 trained with both Adam and SGDM and when pruning ResNet-50 trained with SGDM
25

---

Published in Transactions on Machine Learning Research (02/2025)
Adam
SGDM
*eeles
Ve
Tost Accuracy
0.8
0.8
0.6
0.6
0.4
0.4
20
40 Weight Sparsity (%)
60
0
20
40
60
80
Weight Sparsity (%)
Asymmetric Noise:
False
True
Figure 13: Weight sparsity for a ResNet-18 trained on CTEAR-10 while varying the variance the added noise, without adding any regularization. The experiments were made with dynamic pruning enabled and showcased that using asymmetrie noise over the live neurons only instead of symmetric noise across neurons does not impact the tradeoff. Left: With Adam, Right: With SGDM.
Tost Accuray
0.92
0.92
Adam
0.90
0.90
0.88
50
60
70
80
90
60
70
80
90
100
0.925
Tost Accuracy
0.90
SGDM
0.900
0.875
0.85
50
60
80
60
70
90
100
Neuron Sparsity (%)
Weight Sparsity (%)
SroPit-S
EarlyCroP
EarlySNAP
SNAP
DemP- Lasso(y)
Figure 14: The results on VGG-16 networks trained with Adam (Top) and SGDM (Bottom) on CIEAR-10. With Adam high sparsities, DemP can find subnetworks that better maintain performance. With SGDM, the performance instcad decays quickly compared to baseline when reaching = 95% weight sparsity (== 80% neural sparsity). Higher sparsities with DemP obtained by increasing the peak strength of the added scheduled regularization. Left: Neural sparsity, structured methods. Right: Weight sparsity, structured methods.
(Table 2) Results are similar, with DemP outperforming baselines with Adam at high sparsity. However, the performance of DemP with SGDM decay abruptly passed 95% weight sparsity, which may be due improper tuning of the method when using a one-cycle learning scheduler to train for small number of epochs (see A Appendix T.2 The results with Leaky ReLU are in Fig. 15
26

---

Published in Transactions on Machine Learning Research (02/2025)
Test. Accuracy
0.92
0.90
Adam
0.90
0.85
0.88
50
60
70
80
90
60
80
90
100
0.94
Test Aocuracy
0.925
SGDM
0.900
0.92
0.875
50
60
70
80
90
60
70
80
90
100
Neuron Spansity (%)
Weight Sparsity (%)
CroPit-S
EarlyCroP-S
EarlySNAP
SNAP
Dem DemP-l.asso(?)
Figure 15: Top: ResNet-18 networks with Leaky ReLU trained on CIFAR 10 with Adam. DemP again outperforms the baseline structured pruming methods when using Adam. Higher sparsities with DemP are obtained by increasing the peak strength of the added scheduled regularization. Bottom: With SGDM, DemP performance is again similar to other methods. Left: Neural sparsity, structured methods. Right: Weight sparsity, structured methods.
We included results from Lee et al. (2023) and Evci et al. (2020) in Table 2 to better illustrate the tradeoff between structured and unstructured pruning methods. While unstructured methods currently offer more potential to maintain performance at higher parameter sparsity, structured methods offer direct speedup advantages.
G.1 Comparison with Unstructured Methods
We employ the JaxPruner package (Lee et al. 2023 to illustrate further tradeoffs of our method against some unstructured methods. Our method capable of achieving similar performance unstructured ones for the ResNet-18 experiments (Fig. 16). The comparisons with the unstructured methods their default. configuration from JaxPruner, which was tuned for ResNet-50. We expect their performance models to be improved by tuning the pruning distribution, the pruning schedule, and the pruning iterations scheme (Lee et al. 2023). However, for those not interested in expensive tuning, our method becomes an interesting default choice.
smaller
G.2 Transformers Network
We explored the compatibility of DemP with transformer architectures, specifically the Vision Transformer (ViT-B/16) (Dosovitskiy et al., 2021) equipped with GeLU activation functions. Our focus was on pruning the MLP layers within the transformer blocks because these layers account for approximately 65% of the total parameters and contain activation functions, making them directly applicable to DemP.
This investigation led to two significant observations that, the best of our knowledge, have not been previously reported: (a) even when training the 'dense" version, 90% of the neurons within the MLP layers die during training, and (b) in the last MLP block, neurons tend to die quickly before gradually reviving. Tn
27

---

Published in Transactions on Machine Learning Research (02/2025)
Adam
SGDM
0.91
Test Aocuracy
0.90
0.92
0.85
0.90
60
70
80
90
100
60
70
80
90
100
Weight Spansity (%)
Weight Sparsity (%)
RigL
SET
Mag
Sal
DemP Lasso(>)
Figure 16: Comparison between DemP and unstructured pruming methods from Lee al. (2023). performed ResNet-18 trained on CIFAR-10 over three seeds. Higher sparsities with DemP are obtained by increasing the peak strength of the added scheduled regularization. Left: With Adam, DemP proves more effective than the unstructured methods left at their base configuration, even if it camnot achieve the same granularity in sparsity by being& structured method. Right: With SGDM, DemP performs similarly.
Layer
Leyer
Layer 12
3000
Live Nearons
2000
1000
0
20000
40000
20000
40000
0
20000 Training Steps
40000
Training Steps
Training Steps
DemP
DemP + ReLU
Dense
Figure 17: This figure illustrates the evolution of dead neurons during the training process for a 12-layer deep Vision Transformer (ViT-B/16). The rightmost plots depict the behavior in the last layer, where most neurons initially die but subsequently revive, exhibiting a qualitatively different pattern from the other layers. The leftmost and center plots show the typical behavior observed in the other layers, which more consistent with what has been seen other architectures.
other layers, neurons that die remain dead, consistent with our previous observations in other architectures. Those observations are reported in Fig. I7
To accommodate this behavior, we adapted DemP so that pruning begins after neuron revival occurs in the last. MTP layer. The exploratory results, reported in Table 3 based À 0 (no regularization), but with minimal additional noise applied. Inspired by Mirzadeh et al. (0024), also included results where the GeLU activation functions were replaced with ReLU. Although these results are preliminary, they illustrate the potential of DemP for transformer-bese based architectures. A promising future direction would be to use DemP in conjunction with the approach proposed by Mirzadoh et al. (2024). where additional ReLU activations are incorporated into transformer architectures.
H Limitations of our approach
The primary limitation of our work that, unlike most methods, the target sparsity level cannot be specified within the algorithm. Instead, DemP currently relies on hyperparameter search over the regularization strength (A) to achieve the desired sparsity. However, we confident. that our method can be adapted to this
28

---

Published in Transactions on Machine Learning Research (02/2025)
paradigm by contimuously increasing the regularization until the desired sparsity is achieved before terminating the process. This strategy is similar to the approach described by Waann et. al. (2021). Additionally, while our analysis demonstrated the impact of batch size and learning rate on final sparsity, for practical reasons, we decided to rely entirely on regularization strongth to control the sparsity level. This decision may constrain DemP to suboptimality.
In its current form, DemP' removes entire channels from convolutional networks. Adopting more fin-ggraind approach, such removing individual filters, could potentially lead to better performance. Combining DemP with unstructured approaches could also result in more competitive hybrid method.
Finally, DemP relies on activation functions with saturation regions to identify dead neurons. Consequently, DemP cannot be directly applied to prune layers that are not sandwiched between activation funct ions, such as the attention layers in transformer blocks. This is reflected in our experiments on ViT-B/ 16 networks (Section 4.3) where we limited DemP to pruning the fully-connected block within the network. Sparse training methods for Transformers remain underexplored and present unique challenges. A detailed study of DemP in this context, while valuable, is beyond the scope of this paper.
Implementation details
1.1 ResNet-18/ Net55
We mostly followed the training procedure of Evei et al. (2000) for the ResNet architectures.
ResNet-18. We train all networks for 250 epochs using a batch size of 128. The learning rate is initially set 0.005 for Adam, to 0.1 for SGDM, and is thereafter divided by 5 every 77 epochs. While varying regularization is used with our method, it on top of constant weight decay (0.0005) used across all methods, including ours. Random crop and random horizontal flips are used for data augmentation. The training utilized Nvidia RTX8000 GPU.
ResNet-50. We trained the ResNet-50 for 100 epochs, with a batch size of 256 instead of 4096. The initial learning rate is set to 0.005, before being decayed by a factor of 10 at epochs 30, 70, and 90. Label smoothing (0.1) and data augmentation (random resize to either 256 x 256 or 480 x 480, before randomily cropping to 224 x 224. Followed by random horizontal flip and input normalization) are also used. We again use Adam and SGDM, using constant weight decay (0.0001) for both. Training utilized a Nvidia A100 GPU.
1.2
VGG-16
We followed training procedure similar to Rachwan al. (2022). We used Adam and SGDM with respectively learning rate of 0.005 or 0.1 and batch size of 256, with the One Cycle Learning Rate scheduler (Smith & Topin 2018). The networks are trained for 80 epochs. CIFAR-10 images are normalized and resized to 64 x 64 before applying random crop and random horizontal flip for data augmentation. The training utilized a Nvidia RTX8000 GPU.
1.3 Vision Transformer (ViT-B/16)
We followed the training routine of Lasby et al. (2024), that is we added data augmentations following the standard TorchVision (maintainers contributors) 2016) VIT-B/16 training procedure for ImageNet. These augmentations include random cropping, resizing to 224x224 pixels, random horizontal flips, Ragnnnt [Cubuk et al.j 2020),, and normalization with typical RGB channel mean and standard deviation values. Additionally, applied either random mixup (Zhang et al. 2018 or random cutmix (Yun et al. 2019). with alpha parameters of 0.2 and 1.0 respectively.
We sampled 16 mini-batch steps with 256 samples per mini-batch and accumulated gradients before applying the optimizer, resulting in an effective mini-batch size of 4096. The model was trained for 150 epochs using an AdanW (Loshchilov Hutter 2019) optimizer with weight docay, label smoothing, and 31. S2 cocfficients of 0.3, 0.11, 0.9, and 0.999, respectively. We used cosine annealing with linear warmup for the learning rate scheduler, starting 9.9e-5 and warming up to 0.003 by epoch 16. Gradients were clipped to max T.2 norm
29

---

Published in Transactions on Machine Learning Research (02/2025)
of 1.0. Uniform sparsity applied across all layers, with AT set to 100 to update connectivity every 100 mini-batch steps. The training utilized Nvidia A100 GPU.
1.4 Structured Methods
We closely reimplement in JAX (Baddbuyy et al. 2018) the structured methods from Rachwan et al (2022 keeping all the lyperparameters specific to every method as is. The training hyperparameters are the same as specified in and 1.2
1.5
Unstructured Methods
For the unstructured methods, we rely Lee el al. (2023) implementations, using their method's configuration for pruning ResNet-50 for all our experiments. The training hyperparameters are the same as specified in
and 1.2
30


Reviews:
[{"review_id": "e7c9yVOR8J", "reviewer": "Reviewer_LUGQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The presentation quality is good. The paper is easy to follow with informative figures. \n- Analysis and observations on the impact factor of dying neurons are insightful and may inspire future research. It is also worth mentioning that the author analyzes the phenomenon both theoretically and empirically, which largely enhances the robustness and reliability of the research.\n- Compared to structured and selected unstructured pruning baselines, DemP shows optimal accuracy and efficiency. Both the training FLOPs and inference FLOPs can be reduced.\n- A comprehensive ablation study shows how the performance will change accordingly with different regularizers, regularization schedules, and pruning strategies.\nAn extended discussion on limitations is provided in the appendix, which should be encouraged.\n\nWeaknesses:\n- It's a bit overclaiming to say the DemP provides a \"novel\" perspective, as pruning by promoting dead neurons has been widely used in previous research [1][2].\n- The early exploration of transformer-based methods is good. It will be better to include more tasks in addition to image classification, such as detection/segmentation or tasks in NLP (following [3]).\n\n[1] Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life, IJCNN 2023\n\n[2] AP: Selective Activation for De-sparsifying Pruned Networks, TMLR 2023\n\n[3] Winning the Lottery Ahead of Time: Efficient Early Network Pruning, ICML 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper researched the dying neurons phenomenon (i.e. inactive or saturated neurons) and proposed a simple yet effective structured pruning-based method Demon Pruning (DemP) to tackle this challenge. Based on the empirical observations of the dying neurons ratio during training, the authors thoroughly verify the influence of various factors, including regularization, noise, and optimizer. DemP is proposed upon these analyses and can actively promote neuron death, which can then be pruned dynamically during training. The dense-to-sparse pruning is implemented by applying decaying regularization and injecting noise early training. DemP is verified on ResNet/ViT across CIFAR-10 and ImageNet datasets, demonstrating the effectiveness and efficiency with a high compression ratio and significant speedup compared to baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A recent pruning research HBGS[1] could be included in the discussion and comparison.\n\n[1] A Greedy Hierarchical Approach to Whole-Network Filter-Pruning in CNNs, TMLR 2024", "broader_impact_concerns": "All concerns have been sufficiently addressed in the Broader Impact Statement section", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9LUrMRBKEo", "reviewer": "Reviewer_Km7B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method demonstrates superior performance compared to baselines like EarlySNAP and EarlyCroPit, achieving better accuracy-sparsity tradeoffs while reducing both training and inference costs.\n2. The paper provides valuable insights into the phenomenon of neuron saturation.\n\nWeaknesses:\n1. The exact formulation of the regularization term $R$ is not explicitly provided, which makes it difficult to fully understand the proposed regularization strategy. Including this expression would improve the paper’s clarity. Moreover, the concept of asymmetric noise is not well-explained in Section 3. \n2. Most of the experiments are limited to CNN with ReLU activations. And no baseline comparisons involving Vision Transformers are included.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes DemP, which utilizes regularization and asymmetric noise injection in training to enforce sparse solutions by controlling the proliferation of dead neurons. The proposed approach achieves better accuracy-sparsity tradeoffs and faster training/inference compared to baselines such as SNAP, EarlySNAP, and EarlyCroPit.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. A more detailed discussion of regularization and asymmetric noise strengthen the paper.\n2. Provide baselines performance in Table 2.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XO9SEuH0Ab", "reviewer": "Reviewer_p6vp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- It is an interesting idea to introduce noise in response to the neuron activity at an update time in order to manipulate the activity/inactivity of the neuron.\n- Experiments was conducted on datasets and models of different scales. Also, this paper experimented with the Transformer architecture, which has recently been the subject of much discussion.\n\n**Weaknesses**\n- The research question needs to be clarified. I could not figure out what problem is being set in this work and what kind of answer DemP achieves to that problem.\n- This paper introduces DemP as a pruning method, but reading this, it seems that DemP is not a pruning method. The main part of DemP is the proposal for the weight update part of the equation (1) and does not discuss the pruning saliency or something like that.\n- Related to weaknesses 1 and 2 above, you must clarify what pruning methods DemP targets. After reading to its end, it is finally clear that structural pruning (channel-wise pruning?) and DST are DemP's targets. In my opinion, if you introduce equation (1), you can say that we use DemP, so you can also apply DemP to unstructured pruning, iterative pruning, and post-training pruning.\n- Also, the main manuscript is hard to read because of frequent directions to the appendix and various sections to see important results. TMLR has no page limit, so the position of figures and some theoretical results should be modified to make it more readable.\n- The comparison of methods needs to be more fair. This paper says that it adhered to the setting of the original paper for comparison, but this result does not tell me whether the good performance is due to one-cycled scheduling in Smith & Topin, 2018. Please also include the accuracy of previous studies on using one-cycled scheduling.\n- Table 1 appears to be an important result, but it is difficult to see which aspect of DemP is superior to the conventional methods. It would be easier to convey them if the axes (e.g., neuron sparsity and accuracy) were selected and represented in a scatterplot.\n- There is no comparison with other DST methods and DemP in the Transformer architecture. The result of this manuscript does not show the superiority of DemP over other DSTs in the Transformer, so I would like to see an evaluation of other DSTs in the Transformer. If the result of DemP with the Transformer is not good, please explain it in the limitation section.\n\n**Minor typo**\n- Section 2.2: similar to (Hu et al., 2016) -> similar to Hu et al., 2016\n- Maybe the em dash command (---) is wrong. Also, please unify whether spaces are inserted before or after the em dash. (By the way, I prefer not to insert a space.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the effect of some neural network settings on the output magnitude of each neuron experimentally. From this finding, this paper introduces DemP, a method for inducing neurons to become non-functional during training and making the sparse feature space.\nDemP achieves comparable or better accuracy on CIFAR-10 and ImageNet than traditional dynamic sparse training (DST) methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.\n\nWhile this paper presents an interesting proposal with promising results, the writing needs improvement.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a dynamic pruning method by leverages dying neurons during the training of deep neural networks. The method explored various hyperparameter configurations, such as regularization and asymmetric noise injection, to enforce sparsity while preserving model accuracy. The results demonstrate improved accuracy-sparsity tradeoffs compared to existing structured dense-to-sparse training methods.

The paper is reasonally well-written and has generally received favorable reviews. The reasons for acceptance are:

1)	It demonstrates the viability and effectiveness of pruning based solely on neuron saturation.

2)	It offers insights into the neuron saturation behavior through the lens of sparsity and dynamic pruning.

3)	The results validate the effectiveness of the proposed method.

On the weakness side, the paper primarily focuses on ReLU-based activations, which may limit its application scope. Additionally, the writing can be improved by providing clearer comparisons with existing pruning methods to better highlights the advantages of proposed approach. Nevertheless, the strengths of the paper outweigh the weakness, therefore, it is recommended for acceptance.
Decision: Accept (poster)