Sparse Training via Boosting Pruning Plasticity  with Neuroregeneration
Authors: Shiwei Liu
Tianlong Chen
Xiaohan Chen
Zahra Atashgahi
Lu Yin
Huanyu Kou
Li Shen
Mykola Pechenizkiy
Zhangyang Wang
Decebal Constantin Mocanu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Pruning Plasticity
sparse training
pruning during training
gradual pruning
dynamic sparse training
Neuroregeneration

Abstract: Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter usually struggles with insufficient performance. In comparison, during-training pruning, a class of pruning methods that simultaneously enjoys the training/inference efficiency and the comparable performance, temporarily, has been less explored. To better understand during-training pruning, we quantitatively study the effect of pruning throughout training from the perspective of pruning plasticity (the ability of the pruned networks to recover the original performance). Pruning plasticity can help explain several other empirical observations about neural network pruning in literature. We further find that pruning plasticity can be substantially improved by injecting a brain-inspired mechanism called neuroregeneration, i.e., to regenerate the same number of connections as pruned. We design a novel gradual magnitude pruning (GMP) method, named gradual pruning with zero-cost neuroregeneration (GraNet), that advances state of the art. Perhaps most impressively, its sparse-to-sparse version for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods with ResNet-50 on ImageNet without extending the training time. We release all codes in https://github.com/Shiweiliuiiiiiii/GraNet. 

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
Shiwei Liu't Tianlong Chen', Xiaohan Chen², Zahra Atashgahi', Lu Yin', Huanyu Kou*, Li Shen", Mykola Pechenizkiy Zhangyang Wang'. Decebal Constantin Mocanu 'Eindhoven University of Technology. Uniiversiyy of Texas at Austin University of Twente," " of Leeds, JD Explore Academy, University of Jyväskylä c {z. atashgahi, ,d momoamu) Sutwente...... ml, (khydoublel, (aataahenn@@@mmal.....
{s. 1iu3, mtt........mmpeeeeee..e wew.peeenizziiy/ttuee mnl ftianlong. chen, xiaohan. chen, atlaswang)/utexa. edu
Abstract
Works lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised lot attention currently on post-training pruning (iterative magni- tude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter usually struggles with insufficient performance. In comparison, a class of pruning methods that simultaneously enjoys the training/inference ef- ficiency and the comparable performance, temporarily, has been less explored. To better understand during-training pruning, we quantitatively study the effect of pruning throughout training from the perspective of pruning plasticity (the ability of the pruned networks to recover the original performance). Pruning plas- ticity can help explain several other empirical observations about neural network pruning in literature. We further find that pruning plasticity can be substantially improved by injecting brain-inspired mechanism called neuroregeneration. i.c., to regenerate the same number of connections as pruned. We design novel gradual magnitude pruning (GMP) method, named gradual pruning with zero- cost neuroregeneration (GraNet), that advances state of the art. Perhaps most impressively, its sparse-to-sparss version for the first time boosts the sparse-to- sparse training performance over various dense-to-sparse methods with ResNet- 50 on ImageNet without extending the training time. We release all codes https: thub. Somm/hiweill www.lliiiiiiiii wwwiiiiiii/rrrallae..ee..
traiinnntiinngggg pruning,
Introduction
Neural network pruning is the most common technique to reduce the parameter count, storage requirements, and computational costs of modern neural network architectures. Recently, post- training pruning [49, 29, 18, 47, 10, 54, 74, 5, 57, 75] and before-training pruning [31, 30, 67, 63, 11] have been two fast-rising fields, boosted by lottery tickets hypothesis (LTH) [10] and single- shot network pruning (SNIP) [31]. The process of post-training pruning typically involves fully pre-training dense network as well as many cycles of retraining (either fine-tuning [18, 17, 39] or rewinding [12, 54]). As the training costs of the state-of-the -fte-oof-the-rtt models, e.g., GPT-3 [4] and FixEfficientNee-L2 [64] have exploded, this process can lead to large amount of overhead cost.
Recently emerged methods for pruning at initialization significantly reduce the training cost by identifying a trainable sub-network before the main training process. While promising, the existing methods fail to match the performance achieved by the magnitude pruning after training [11].
Partial of this work have been done when Shiwei Liu worked as an intern at JD Explore Academy.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

GraNet
Gradual Pruning
Neuroregeneration
Number weights
Training time
Figure 1: Schematic view of GraNet. Left: Gradual pruning starts with a sparse subnetwork and gradually prune the subnetwork to the target sparsity during training. Right: We perform zero-cost neuroregeneration after each gradual pruning step. Light blue blocks/lines refer to the "damaged" connections and orange blocks/lines refer to the regenerated new connections.
Compared with the above-mentioned two classes of pruning, during-training pruning is a class of methods that reap the acceleration benefits of sparsity early the training and meanwhile achieve promising performance by consulting the information obtained during training. There are some works [77. 13. 33] attempting gradually prune the network to the desired sparsity during training. while they mainly focus on the performance improvement. Up to now, the understanding of during- training pruning has been less explored due to its more complicated dynamical process, and the performance gap still exists between pruning during training and full dense training.
To better understand the effect of pruning during the optimization process (not inference), study the ability of the pruned models to recover the original performance after short continued training with the current learning rate, which call pruning plasticity (see Section 3.1 for more formal definition). Inspired by the neuroregeneration mechanism in the nervous system where new neurons and connections are synthesized recover the damage in the nervous system [26, 41, 73], we examine if allowing the pruned network to regenerate new connections can improve pruning plasticity, and hence contribute pruning during training. We consequently propose a parameter-efficieet method to regenerate new connections during the gradual pruning process. Different from the existing works for pruning understanding which mainly focus on dense-to- to-sa-t training [42] (training dense model and prune it to the target sparsity), we also consider sparse-to sparse-to-spars to-ssarse training (training a
sparse model yet adaptively re-creating the sparsity pattern) which recently has received an upsurge of interest in machine learning [44, 3, 9, 48, 8, 37, 36).
In short, have the following main findings during the course of the study:
#1. Both pruning rate and learning rate matter for pruning plasticity. When pruned with low pruning rates (e.g., 0.2), both dense-to-sparse training and sparse-to-sparse training can easily recover from pruning. On the contrary, too many parameters are removed at one time, almost all models suffer from accuracy drops. This finding makes connection to the success of the iterative magnitude pruning [10, 54, 5, 6, 65], where usually a pruning process with a small pruning rate (e.g.. 0.2) needs to be iteratively repeated for good performance.
Pruning plasticity also gradually decreases as the learning rate drops. When pruning happens during the training phase with large learning rates, models can easily recover from pruning (up a certain level). However, pruning plasticity drops significantly after the second learning rate decay, Icading to situation where the pruned networks can not recover with continued training. This finding helps to explain several observations (1) for gradual magnitude pruning (GMP), it is always optimal to end pruning before the second learning rate drop [77, 13]; (2) dynamic sparse training (DST) benefits from a monotonically decreasing pruning rate with cosine or linear update schedule [8, 9]: (3) rewinding techniques [12, 54] outperform fine-tuning as rewinding retrains subnetworks with the original learning rate schedule whereas fine-tuning often retrains with the smallest learning rate.
#2. Neuroregeneration improves pruning plasticity. Neuroregeneration [41, 73] refers to the regrowth or repair of nervous tissues, cells, or cell products. Conceptually, it involves synthesizing new neurons, glia, axons, myelin, synapses, providing extra resources in the long term to replace

---

those damaged by the injury, and achieving a lasting functional recovery. Such mechanism is closely related to the brain plasticity [51], and we borrow this concept developing a computational regime.
We show that, while regenerating the same number of connections as pruned, the pruning plasticity is observed to improve remarkably, indicating more neuroplastic model being developed. However, it increases memory and computational overheads and seems to contradict the benefits of pruning- during-training. This however raises the question: can we achieve efficient newroregeneration during training with no extra costs? We provide an affirmative answer to this question.
#3. Pruning plasticity with neuroregeneration can be leveraged to substantially boost sparse training performance. The performance level under full continued training to the end. Imitating the neuroregeneration behavior [41, 73], we propose a new sparse training method - gradual pruning with zero-cost neuroregeneration (GraNet), which is capable of performing regeneration without increasing the parameter count.
findings of pruning plasticity can generalize to the final
In experiments, GraNet establishes the new state-of-the-art performance bar for dense-to-spp to-sare training and sparse-to-sparse training, respectively. Particularly, the latter for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods by a large margin without extending the training time, with ResNet-50 on ImageNet. Besides the consistent performance improvement. we find the subnetworks that GraNet learns are more accurate than the ones learned by the existing gradual pruning method, providing explanations for the success of GraNet.
2 Related Work
Post-Training Pruning, Methods that yield a sparse neural network from a pre-trained network by pruning the unimportant weights neurons, to the best of our knowledge, were proposed in (24] and [50]. After that, various pruning methods have emerged to provide increasingly efficient methods to identify sparse neural networks for inference. The pruning criterion includes weight magnitude [18, 10), gradient [61] Hessian [29, 19, 59), Taylor expansion [47, 46), etc. Low-rank decomposition [7, 23, 17, 71] are also used to induce structured sparsity in terms of channels or filters. Most of the above-mentioned pruning methods require many pruning and re-training cycles to achieve the desired performance.
Muring-Training Pruning. Instead of inheriting weights from a pre-trained model, some works attempt to discover werl-performing sparse neural networks with one single training process.
Gradual Magnitude Pruning (GMP), introduced in [77] and studied further in [13], gradually sparsifies the neural network during the training process until the desired sparsity is reached. Besides, [40] and [68] are prior works that enforce the network to sparse during training via Lo and L1 regulariza- tion, respectively. [60, 34, 55, 70, 28] moved further by introducing trainable sparsity heuristics to learn the sparse masks and weights simultaneously. These methods are all classified as dense-to-sparse training they start from dense network.
Dynamic Sparse Training (DST) [44, 3, 48. 8, 9, 36, 35, 25] is another class of methods that prune models during training. The key factor of DST is that it starts from a random initialized sparse network and optimizes the sparse topology as well as the weights simultaneously during training (sparse-to-sppp to-sparse training). Without an extended training time [37]. sparse-to-sparse training usually falls short of dense-to-seee to-ssarse training in terms of the prediction accuracy. For further details, see the survey of [43, 21].
Before Training Pruning. Motivated by SNIP [31], many works [67, 63, 6) have emerged recently to explore the possibility of obtaining a trainable sparse neural network before the main training process. [1] I] demonstrates that the existing methods for pruning at initialization perform equally well when the unpruned weights are randomly shuffled, which reveals that what these methods discover is the layer-wise sparsity ratio, rather than the indispensable weight values and positions. Our analysis shows that both the mask positions and weight values are crucial for GraNet.
3 Methodology for Pruning Plasticity
The primary goal of this paper to study the effect of pruning as well as neuroregeneration on neural networks during the standard training process. Therefore, we do not consider post- bost-training pruning and before-training pruning. Below, we introduce in detail the definition of pruning plasticity and the experimental design that we used to study pruning plasticity.
3

---

3.1 Metrics
Let us denote W, E IRd as the weights of the network and my E {0,1) 1] as the binary mask yielded from the pruning method at epoch I. Thus, the pruned network be denoted We my. Let T be the total number of epochs the model should be trained. Let CONTRAII ATTRAINN( (W, & @ my a) refers to the function that continues to train the pruned model for epochs with the learning rate schedule
Definition of Pruning plasticity, We define pruning plasticity as (COTRAIN"(WGGmm.e PRE is the test accuracy measured before pruning and (CoNTRAIN=(..Gmm.a) is the test aceuracy measured after & epoch of continued training CONTRAIN (W, @ mi; (t). Specifically, to better understand the effect of pruning on the current model status and to avoid the effect of learning rate decay, we fix the learning rate as the one when the model is pruned, i.c, a, This setting is also appealing to GMP [77, 13] and DST [44, 9, 48, 37] in which most of the pruned models are continually trained with the current learning rate for some time.
fPRE. where
Final performance gap. Nevertheless, we also investigate the effect of pruning on the final perfor- mance, that is, continually training the pruned networks the end with the remaining learning rate schedule CONTRAIN" '(W, @ m,, 1:t)) ). In this case, we report
(EINAL. where IANAL is the final test accuracy of the unpruned models.
3.2 Architectures and Datasets
We choose two commonly used architectures to study pruning plasticity, VGG-19 [58] with batch normalization on CIFAR- -00 [27], and ResNet-20 [20] on CIFAR-10.
We share the summary of the networks, data, and hyperparameters of dense-to-sparse training in Table 1. We use standard implementations and hyperparameters available online, with the exception of the small batch size for the ResNet-50 on ImageNet due to the limited hardware resources (2x Tesla V V100). All accuracies are in line with the basclines reported in the references [8. 11, 67, 9, 37].
Table I: Summary of the architectures and hyperparameters we study in this paper.
Model
Data
#Epoch
Batch Size
LR
LR Decay, Epoch
Weight Decay
Test Accuracy
ResNet-20
CIFAR-10
160
128
0.1 (8 = 0.9)
10x, [80, 120]
0.0005
92.41 +0004
CIFAR-10
160
128
0.1 (B 0.9)
10x, [80, 120]
0.0005
93.85
VGG-19
CIFAR-100
160
128
0.1 (ß = 0.9)
10x, (80, 120]
0.0005
73.43+00.08
CIFAR-10
160
128
0,1 (8 0.9)
10x, (80, 120]
0,0005
94.75-10.01
ResNet-50
CIFAR-100
160
128
0.1 (5 = 0.9)
10x, [80. 120]
0.0005
78.23-10.18
ImageNet
100
( 0.9)
10x, [30, 60, 90]
0.0004
76.80=0009
3.3 How to Prune, and How to Regenerate
Structured and Unstructured Pruning. We consider unstructured and structured pruning in this paper. Structured pruning prunes weights in groups, or removes the entire neurons, convolutional filters, or channels, enabling acceleration with the off-the-shelf hardware. In particular, we choose the filter pruning method used in Li al. [32]. Unstructured sparsity is a more promising direction not only due to its outstanding performance at extreme sparsities but the increasing support for sparse operation in the practical hardware [35, 14, 52, 76, 22]. For example, Liu al. [35] illustrated for the first time the true potential of DST, demonstrating significant training/inference efficiency improvement over the dense training. Different from prior conventions [77, 13, 33, where values of the pruned weights are kept, we set the pruned weights to zero to eliminate the historical information for all implementations this paper.
Magnitude pruning. We prune the weights with the smallest magnitude, as it has evolved as the standard method when pruning happens during training, c.g., GMP [77, 13] and DST [44, 9, 37]. We are also aware of other pruning criteria including but not limited to Hessian [29, 19, 59], Taylor expansion [47, 46), connection sensitivity [31], Gradient Flow [67]. Neural Tangent Kernel [38, 16].
One-shot pruning. To isolate the pruming effect at different training stages and to avoid the interaction between two iterations of pruning, can also be generalized in our setting, as our experimental design includes neural networks trained at various sparsities and each of them is further pruned with various pruning rates.
focus one-shot pruning. Please note that iterative pruning
Layer-wise pruning and global pruning. We study both the layer-wise magnitude pruning and global magnitude pruning for pruning plasticity. Global magnitude pruning prunes different layers

---

together and leads to non non-uniform sparsity distributions; layer-wise pruning operates layer by layer, resulting in uniform distributions.
Gradient-based regeneration. The simplest regeneration scheme is to randomly activate new connections [3, 44]. However, it would take a lot of time for random regeneration discover the important connections, especially for the very extreme sparsities, Alternatively, gradients, including those for the connections with zero weights, provide good indicators for the connection importance. For this reason, we focus on gradient- -based regeneration proposed in Rigged Lottery ( RRgL)) [9]. i.e. regenerating the same number of connections as pruned with the largest gradient magnitude.
3.4 Experimental Results
We study pruning plasticity during training with/without regeneration, for both dense training and sparse training. We report the results of ResNet-20 on CIFAR-10 with unstructured global pruning in the main body of the paper. The rest of the experiments are given in Appendix A. Unless otherwise stated, results are qualitatively similar across all networks. Concretely, we first pre-train networks at four sparsity levels, including O, 0.5, 0.9, and 0.98. The sparse neural networks are trained with uniform distribution (i.e., all layers have the same sparsity). We further choose four pruning rates, e.g., 0.2, 0.5, 0.9, and 0.98, to measure the corresponding pruning plasticity of the pre- pre-trained networks.
Pruning plasticity. We continue to train the pruned model for 30 epochs and report pruning plasticity in Figure 2. Overall, the learning rate schedule, the pruning rate, and the sparsity of the original models all have a big impact on pruning plasticity. Pruning plasticity decreases as the learning rate decays for all models with different sparsity levels. The models trained with a large learning rate 0.1 can easily recover, or exceed the original performance except for the extremely large pruning rate 0.98. However, the models obtained during the later training phases can recover only with the mild pruning rate choices, e.g., 0.2 (orange lines) and 0.5 (green lines).
We next demonstrate the effect of connection regeneration on pruning plasticity in the bottom row of Figure 2. It is clear to see that connection regeneration significantly improves pruning plasticity of all the cases, especially for the models that are over-prun over-pruned (purple lines). Still, even with connection regeneration, pruning plasticity suffers from performance degradation when pruning occurs after the learning rate drops.
Predzained Sparstywd
Pre trained Sparsity=3.5
trained Sparstyya..s
Pro teaitraned Sparsky-2.50 56
we Regemiration 1S3
Pruning Plastiity
0:
-40
-40
-40
-40
-50
-50
-60
-60
wr Regemiration PS0
Praning Piastrott
-20
-20
-20
100
115
229
235
169
152
300
-10
-40
number which model
Fruntig 0
Pruning nite
Pruning 1.0
Fruniin rate-0
Unstructured Pruning: Pruning plasticity (see Section 3.1 for definition) under a 30- epoch continued training with and without connection regeneration for ResNet-20 on C1. The vertical red lines refer to the points when the learning rate is decayed. "Pre-trained Sparsity" refers to the original sparsity of the pre- re-trained networks before pruning. The pruning method is the magnitude global pruning.
Figure 2:
Final performance gap. Compared with the current model status, people might be more interested in the effect of pruning on the final performance. We further measure the performance gap between the original test accuracy of the unpruned models and the final test accuracy of the pruned model under a full continued training CONTRAIN T-E O m, ((T)) in Figure 3.
We observe that, in this case, large learning rates do not enjoy large performance improvement, but still, the performance gap increases as the learning rate drops. It is reasonable to conjecture that the accuracy improvement of pruning plasticity with the large learning rate, 0. is due to the
5

---

wio Regeneration 1%4
25-
Pro sranedd Sparskyee
25
Pro tratii Sparsity 50
25
Pro trainod Sparsicyw 50
Pro tod Sparkky-3.5
Pinal Performance Cap Final Purformance
-25
-25
-30-
-30
-50
-50
wr Repenaration [N3
30
30
30-
10-
10
10
10
-10
-10
-10
63
109
312
123
242
300
119
129
135
100
115
135
340
211
14
number ofooch at whtch the model pruned
Pruning
Final performance gap between the unpruned models and the pruned models for ResNet-20 on CIFAR-10. The vertical red lines refer to the points when the learning rate is decayed. "Pre-trained Sparsity" refers to the original sparsity of the pre-trained networks before pruning. The pruning method is the magnitude global pruning.
Figure 3: Unstructured Pruning:
unconverged performance during the early phase of training. Besides, it is surprising to find that the final performance of extreme sparse networks (e.g.. the third column and the fourth column) significantly benefits from mild pruning. Again, the ability of the pruned model to recover from pruning remarkably improves after regenerating the connections back.
Gradual Pruning with Zero-Cost Neuroregeneration
So far, we have known that regenerating the important connections to the pruned models during training substantially improves pruning plasticity well as the final performance. However, naively regenerating extra connections increases the parameter count and conflicts with the motivation of gradual pruning.
Inspired by the mechanism of neuroregeneration in the nervous system, we propose a novel sparse training method which call gradual pruning with zero-cost neuroregeneration (GraNet). GraNet consults the information produced throughout training and regenerates important connections during training in a parameter-efficient fashion. See Appendix B.1 for the pseudocode of GraNet. We introduce the main components of GraNet below.
4.1 Gradual Pruning
We follow the gradual pruning scheme used in [77] and gradually sparsifies the dense network to the target sparsity level over n pruning iterations. Let us define is the initial sparsity, sy is the target sparsity, to is is the starting epoch gradual pruning, ty is the end epoch gradual pruning, and At is the pruning frequency. The pruning rate of each pruning iteration is:
8f (8: ss) t to nAt
t. E (lo, to At...,, to nd)..
(1)
We choose global pruning for our method as it generally achieves better performance than uniform pruning. We also report the performance of the uniform sparsity as used in [13] in Appendix C.3.
The conventional gradual pruning methods [77, 13] change the mask (not the weight values) to fulfill the pruning operation, so that the pruned connections have the possibility be reactivated in the later training phases. Despite this, since the weights of the pruned connections are not updated, they have a small chance receive sufficient updates to exceed the pruning threshold. This hinders the regeneration of the important connections.
4.2 Zero-Cost Neuroregeneration
The main difference between GraNet and the conventional GMP methods [77, 13] is the Zero-Cost Neuroregeneratiin. Imitating the neuroregeneration of the peripheral nervous system [41, 73] where new neurons and connections are synthesized to replace the damaged ones, we first detect and eliminate the "damaged" connections, and then regenerate the same number of new connections. By doing this, achieve connection regeneration without increasing the number of connections.

---

Concretely, we identify the "damaged" connections as the ones with the smallest weight magnitudes. Small magnitude indicates that either the weight's gradient small or a large number of oscillations occur to the gradient direction. Therefore, these weights have small contribution to the training loss and can be removed. Again, we use the gradient as the importance score for regeneration, same as the regrow method as used in Rigl.19)
Why we call "Zero-Cost Neuroregeneration". In addition not increasing the connection (parame- ter) count, the backward pass of our method is sparse most of the time even though our regeneration utilizes the dense gradient to identify the important connections. We perform neuroregeneration immediately after each gradual pruning step, meaning that the regeneration occurs only once every several thousand iterations. The extra overhead calculate the dense gradient can be amortized compared with the whole training costs. Compared with the methods [33, 69] that require updating all the weights in the backward pass, our method is much more training efficient, as around 2/3 of the training FLOPs is owing to the backward pass [9, 72].
Let us denote r as the ratio of the number of the regenerated connections the total number of connections; W is the network weight. We first remove r proportion of "damaged" weights with the smallest magnitude by:
W' = TopK (IW), 1 r)
Here (v, k) returns the weight tensor retaining the top k-proportion of elements from v. Imme- diately after that, we regenerate proportion of new connections based on the gradient magnitude:
W = W + TopK (B..ww-l r),
where |Bgg are the gradient magnitude of the zero weights. We perform Zero-Cost Neuroregener- ation layer by layer from the beginning of the training to the end.
GraNet can naturally generalize to the tonseee training scenario and the sparse-to-sparse training scenario by setting the initial sparsity level s; = 0 and > 0 in Eq. (1), respectively. For simplicity, we set s; = 0.5, to = 0, and y as the epoch when performing the first learning rate decay for the sparse-to-se to-ssas-sseeeeee- training. Different from the existing sparse-to- to-sparse training methods, i.e., SET [44], RigL [91, and ITOP [37]. in which the sparsity is fixed throughout training. GraNet starts from denser yet still sparse model and gradually prunes the sparse model to the desired sparsity. Although starting with more parameters, the global pruning technique of gradual pruning helps GraNet quickly evolve to a better sparsity distribution than Rigl. with lower feedforward FLOPSs and higher test accuracy. What's more, GraNet sparsifies all layers including the first convolutional layer and the last fully-connected layer.
4.3 Experimental Results
We conduct various experiments evaluate the effectiveness of GraNet. We compare GraNet with various dense-to-sparse methods and sparse-to-sparse methods. The results of Rigged Lottery (RigL) and GMP with CIFAR- 10/100 were reproduced by our implementation with PyTorch that the only difference between GraNet and GMP is the Lero-Cost Neuroregeneration. For each model, we divide the results into three groups from top to bottom: pruning at initialization, dynamic sparse training and dense-to-sparse methods. See Appendix B for more implementation details used in the experiments. GraNet (s; = 0.5) refers to the sparse-to-spant to-rse version and the and GraNet (s; = 0) refers to the dense-to-spare sparse version.
CIFAR-10/100. The results of CIFAR- are shared in Table 2. We can observe that performance differences among different methods on CIFAR- 10 are generally small, but still, GraNet (s: = 0) consistently improves the performance over GMP except for the sparsity 95%, and achieves the highest accuracy in 4 out of 6 cases. In terms of the more complex data CIFAR- the performance differences between the during-training pruning methods and before-training pruning methods are much larger. GraNet (s; = 0) again consistently outperforms GMP with all sparsities, highlighting the benefits of Zero-Cost Neuroregeneration. It maybe more interesting that GraNet (s, = 0) even outperforms the post-training method, subdifferential inclusion for sparsity (SIS), by large margin.
In terms of sparse-to-sparse training, our proposed GraNet (s; 0.5) has dominant performance over other methods. Especially at the very extreme sparsity 0.98, our method outperforms RigL by 1.40% and 2.22% with VGG-19 on CIFAR-10 and CIFAR-100, respectively.
ImageNet. Due to the small data size, the experiments with CIFAR-10/100 may not be sufficient to draw solid conclusion. We further evaluate our method with ResNet-50 on ImageNet in Table 3.
7

---

Table 2: Test accuracy of pruned VGG-19 and ResNet-50 on CIFAR-10/100. We mark the best sparse-to-spante training results in blue and the best dense-to-sparye training results in bold. The results reported with (mean + std) are run with three different random seeds by us. The rest are obtained from [66] and [67]. Note that the accuracy of Rigl. is higher than the ones reported in [66], as we choose a large update interval following the In-Time Over-Paramettriiztion strategy [37]. 8; refers to the initial sparsity of GraNet.
Dataset
CIFAR-10
CIFAR-100
Pruning ratio
90%
95%
98%
90%
95%
98%
VGG-19 (Dense)
93.85++0.05
73.43++0.08
SNIP [31]
93.63
93.43
92.05
72.84
58.46
GruSP [67]
93.30
93.04
92.19
71.95
71.23
68.90
SynFlow [63]
93.35
93.45
92.24
71.77
71.72
70.94
Deep-R [3]
90.81
89.59
86.77
66.83
SET [44]
92.46
91.73
89.18
72.36
RigL
93.38.10.11
93.06:10.09
91.9810.009
73.13:10.28
72.14:0.115
69. 69.82.10.09
GraNet (s, 0.5) (ours)
03.7
93.66.10.07
0.15
73.30.10.13
73.18:0.31
0..04
STR [28]
93.73
93.27
92.21
71.93
71.14
69.89
SIS [66]
93.99
93.31
93.16
72.06
71.85
71.17
GMP [13]
93.59=0.110
93.58+0.07
93.52-+0.33
73.10=+0.12
72.30=0.15
72.07+0.377
GraNet (5, = 0) (ours)
93.80+0.110
03.72++011
93.63++008
0.374++030
73.10=0.04
72.35-+0.26
ResNet-50 (Dense)
94.75+0.01
78.23++0.18
SNIP [31]
92.65
90.86
87.21
73.14
69.25
58.43
GraSP [67]
92.47
91.32
88.77
73.28
70.29
62.12
SynFlow [63]
92.49
91.22
88.82
73.37
70.37
62.17
RigL [9]
94.45-10.43
93.86 03.2655
93,26-10.2 22
06.30:10.33
76.03=0.34
75.06:10.27
GraNet (s, 0.5) (ours)
0.26
0..38
94.01
0..89
7.51
77.14:0.45
STR [28]
92.59
91.35
88.75
73.45
70.45
62,34
SIS [66]
92.81
91.69
90.11
73.81
70.62
62.75
GMP [13]
94.34-10.09
94.52:10.08
14.1940..0
76.91:10.23
76.42:10.51
75.58-10.20
GraNet (s, = 0) (ours)
94,49=0.08
94.44+0.00
94.34-0.17
77.29:10.45
76.71=0026
76.10=+0200
Table 3: Test accuracy of pruned ResNet-50 ImageNet dataset. The best results of DST methods are marked as blue and the best results pruning during training methods are marked in bold. The training/test FLOPs normalized with the FLOPs of dense model. s, refers the initial sparsity of GraNet.
Method
Top-1
FLOPs (Train)
FLOPs (Test)
TOP-1 Accuracy
FLOPs (Train)
FLOPs (Test)
Accuracy
Dense
7..8
Ix (3.2e18)
(8.2c9)
76.8
Ix (3.2e18)
(8.2c9)
Pruning ratio
80%
90%
Static (ERK)
72.1+0.04
0.42x
0.42
67.7+0.12
0.24x
0.24 x
Small- Dense
12.0
0.23x
0.23x
67.2:10.2
0.10x
0.10x
SNIP [31]
72.0=0.06
0.23x
0.23x
67.2+0012
0.10x
0.10x
SET [44]
72.910039
0.23
0,23x
69.6
0.10x
0.10x
DSR[48]
73.3
0.40x
0,40x
71.6
0.30x
0,30x
RigL (ERK) [9]
75.1=0005
0.42x
0.42x
73.0=0.044
0.25x
0.24
SNFS (ERK) [8]
1..2=0.11
0.61
0.42x
72.9
0.50x
0.24x
GraNet (8, 0.5) (ours)
76.0
0.37
0.35x
74.5
0.25x
0.20x
STR [28]
76.1
n/a
0.17x
74.0
n/a
0.08
DPF [33]
75.1
0.71
0.23x
n/a
n/a
n/a
GMP [13]
75.6
0.56x
0.23x
73.9
0.51
0.10x
GraNet (s; 0) (ours)
75.8
0.34x
0.28x
74.2
0.23x
0.16x
We only run this experiment once due to the limited resources. We set to 0 and ty = 30 for both GraNet (s, = and GraNet (s; = 0.5) on ImageNet. Again, GraNet (s; 0) outperforms GMP consistently with only half training FLOPs and achieves the highest accuracy among all the dense-to-sen tosa-se methods sparsity of 0.9. Surprisingly, GraNet (s; 0.5) significantly boosts the training performance, even over the dense-to-spars. training. Concretely, GraNet

---

(s; = 0.5) outperforms RigL by 0.9% and 1.5% at sparsity 0.8 and 0.9, respectively. To the best of our knowledge, this is the first time in the literature that sparse-to-sparse training reaches test accuracy of 76% with ResNet-50 on ImageNet at sparsity 0.8, without extension of training time. It is reasonable for GraNet (8; = 0.5) to achieve better accuracy than Rigl., since the denser models at the beginning help GraNet explore more the parameter space. According to the In- Time hypothesis [37]. the performance of sparse training methods is highly correlated with the total number of parameters that the sparse model has visited.
We further report the training/inference FLOPs required by all pruning methods. Compared with other dense-to-spane sp-srs methods, the final networks learned by GraNet (s; = 0) require more FLOPs to test, whereas the overall training FLOPs required by GraNet (s; 0) are smaller than others. Even though starting from a denser model, GraNet (s; = 0.5) requires less training and inference FL.Pss than the state-of-the -hrtt method, i.e., RigI.. The sparsity budgets for 0.9 sparse ResNeet-0 on ImageNet- 1 K learned by our methods are reported in Appendix D. We also report how FLOPS of the pruned ResNet-50 evolve during the course of training in Appendix E.
4.4 Effect of the Initial Sparsity
As we mentioned earlier, the denser initial network is the key factor in the success of GraNet. We conducted experiments to study the effect of the initial sparsity on GraNet with ResNet-50 on Ima- geNet. The initial sparsity is chosen from [0.0, 0.5, 0.6, 0.7, 0.8, 0.9] and the final sparsity is fixed as 0.9. The results are shared in Table 4. We can see the training FLOPs of GraNet are quite robust to the initial sparsity. Surprisingly yet reasonably, it seems that the the smaller the initial spansity is (up to 0.5), the better final sparsity distribution GraNet finds, with higher test accuracy and fewer feedfor- ward FLOPs. The lower feedforward FLOPs of the final network perfectly balance the overhead caused by the denser initial network.
Table 4: Effect of the initial sparsity GraNet with ResNet-50 on ImageNet. The training/test FLOPs normalized with the FLOPs of dense model.
Method
Top-I [%] Accuracy
FLOPs (Train)
FLOPs (Test)
GraNet
0.0
0.9
74.2
0.23
0.16x
GraNet
74.5
0.25x
0.20x
GraNet
0.6
0.9
74.4
0,25x
0,22x
GraNet
0.9
0.24x
0
GraNet
0.8
0.9
74.1
0,25
0,24x
Rigl.
0.9
0.9
73.0
0,25
0.24x
4.5 Performance of GraNet at Extreme Sparsities
In this section, we share the results of GraNet and RigL at extreme sparsities. The initial sparsity is set 0.5. When the final sparsity is relatively smaller (e.g.. 0.8, 0.9). GraNet requires a lower (or the same) number of training FLOPs than RigL, whereas GraNet requires more training FLOPs than RigL when the final sparsity extremely high (e.g., 0.95, 0.965). This makes sense since when the sparsity is extremely high, the saved FLOPSs count of the distribution discovered by GraNet is too small to amortize the overhead caused by denser initial models. Yet, the increased number of train- ing FL.OPs of GraNet leads to substantial accuracy improvement (> 2%) over RigL. The efficiency of GraNet (s; 0.5) comes from two important tech- nical differences compared with RigL: (I) better final sparse distribution discovered by global prun- ing: (2) a shorter period of gradual pruning time
Table 5: Comparison between GraNet and RigL at extreme sparsities with ResNet-50 on Ima- geNet. The training/test FLOOPs are normalized with the FI.OPs of a dense model.
Method
ey
Top-1 [%] Accuracy
FLOPs (Train)
FLOPs
(Test)
RigL GraNet
0.8
0.8
75.1
0.42x
0.42x
0.5
0.8
76.0
0,37x
0.35x
RigL GraNet
0.9
0.9
73.0
0,25x
0.24x
74.5
0.25x
0.20x
RigL GraNet
0.95
0.95
69.7
0,12x
0.12x
0.5
0.95
72.3
0.17x
0.12x
RigL GraNet
0.965
0.965
67.2
0.11x
0.11x
0.5
0.965
70.5
0.15x
0.09x
(the first 30 epochs for ResNet-50 on ImageNet). Although starting with more parameters, the global pruning enables GraNet to quickly (first 30 epochs) evolve to better sparsity distribution with lower test FLOPs than ERK. After 30 epochs of gradual pruning, the network continues to be trained with this better distribution for 70 epochs, so that the overhead in the early training phase with larger training FLOPs is amortized by the later and longer training phase with fewer training FLOPs.
4.6 Ablation Study of Random Reinitialization
Next, we ask whether what GraNet learned are the specific sparse connectivity or the sparse con- nectivity together with the weight values. We randomly reinitialize the pruned network with the

---

same mask and retrain it. The results are given in Figure 4. The performance of the reinitialized networks falls significantly short of the performance achieved by GraNet (s; = 0), indicating that what was learned by GraNet is the sparse connectivity together with the weight values. Besides, we find that the retraining performance of GraNet is higher than GMP. This further confirms that Zero-Cost Neuroregeneration helps the gradual pruning find more accurate mask positions.
VG219 un CIPAR-10
VOGIM SPAR100
94.00
73
93.50
93.25
72
93.00
71
92.75
70-
92.50
0.5
0.6
0.7
0.3
0.9
0.55
0.50
0.5
0.6
0.7
0.8
0.9
0.55
0.58
Sparsity
Sparsty
GraNer
--- GMPP
Renttaization th CDP mark
--- Renntia loatoon CMP made
Figure 4: Reinitialization ablation on subnetworks discovered by GMP and GraNet (s, 0).
4.7 Comparison between Re-training and Extended Training
In this section, we study if re-training techniques can further improve the performance of the subnetworks discovered by GraNet. The authors of Lottery Ticket Hypothesis (LTH) [10] introduced a retraining technique, even if they did not evaluate it as such, where the subnetworks discovered by iterative magnitude pruning can be re-trained in isolation to full accuracy with the original initializations. Later on, learning rate rewinding (LRR) [54] was proposed further to improve the re-training performance by only rewinding the learning rate. Since GraNet also utilizes magnitude pruning to discover subnetworks, it is natural to test if these re-training techniques can bring benefits to GraNet. As shown in Table 6, both re-training techniques do not bring benefits to GraNet. Instead of re-training the subnetworks, we find that simply extending the training time significantly boosts the performance of GraNet with similar computational costs.
Table 6: Effects of LTH and LRR on the subnetworks learned by GraNet. Methods with 2 x refer to extending the training steps by times. The results are reported with top-1 test accuracy [%]
Dataset
CIFAR-10
CIFAR-100
Pruning ratio
90%
95%
98%
90%
95%
98%
VGG-19 (Dense)
93.85+00005
73.43++0.08
GraNet (8 = 0)
93.80=0.10
93.72=0.11
93.6310.08
73.74:10.30
73.10+0.04
72.35:0.226
Lottery Ticket Hypothesis
93.63=0004
93.29=0005
92.46+0.08
72.97+0.25
71.76++022
69.28+00.36
Learning Rate Rewinding
93.84.10.14
93.72.10.06
93.53.10.04
73.71
73.24.10.24
72.50.10.22 0.26
GraNetzx (81 = 0)
0.03
93.98+0007
93.94+0.11
74.80+00.29
73.65+0.32
73.63+ 0.05
ResNet-50 (Dense)
94.7510.01
78.23
GraNet (& = 0)
94.49+0.08
94.44+0.01
94.34+0.17
77.29+00.45
76.71+0022
16.10+-0.2
Lottery Ticket Hypothesis
93.96110.10
93.7010.15
92.9411.114
15.74.10.19
74.3110.10
71.9910.08
Leamnngg Rate Rewinding
13.
9.,39++0.13
94.20++0.25
7..40+0.114
76.90:10.19
0..25
GraNetax (s; 0)
95.0910.115
94.84.10.11
94.69
78.18.10.20
18.17
77.15.10.29
Conclusion, and Reflection of Broader Impacts
In this paper, we re-emphasize the merit of during-training pruning. Compared with the recently proposed works, i.e., LTH and SNIP, traiinng pruning is an efficient yet performant class of pruning methods that have received much less attention. We quantitatively study pruning during training from the perspective of pruning plasticity. Inspired by the findings from pruning plasticity and the mechanism of neuroregeneration in the nervous system, further proposed a novel sparse training method, GraNet, that performs the cost-free connection regeneration during training. GraNet training.
advances the state of the art in both dense-to-spane so-sarse training and
Our paper re-emphasizes the great potential of during-training pruning in reducing the train- ing/inference resources required by ML models without sacrificing accuracy. It has a significant envi- ronmental impact on reducing the energy cost of the ML models and CO2 emissions [1, 53, 15, 56, 62].
10

---

6 Acknowledgement
This project partially financed by the Dutch Research Council (NWO). We thank the reviewers for the constructive comments and questions, which improved the quality of our paper.
References
[1] Z. Atashgahi, G. Sokar, T. van der Lee, E. Mocanu, D. C. Mocanu, R. Veldhuis, and M. Pech- enizkiy. Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. arXiv:2012. 2. 0.00660, 2020.
[2] B. Bartoldson, A. Morcos, A. Barbu, and G. Erlebacher. The generalization-stabiii tradeoff in neural network pruning. H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20852-20864. Curran Associates, Inc., 2020.
[3] G. Bellec, D. Kappel, W. Maass, and R. Legenstein. Deep rewiring: Training very sparse deep networks. International Conference on Learning Representations, 2018. arXiv:1711.05136 (2017).
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh. D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[5] T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, M. Carbin, and Z. Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16306616316, June 2021.
[6] P. de Jorge, A. Sanyal, H. Behl, P. Torr, G. Rogez, and P. K. Dokania. Progressive skeletonization: Trimming more fat from network at initialization. International Conference on Learning Representations, 2021. arXivics.CV/20066 CV/80.
[7] E. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Twenty-eighth Conference on Neural Information Processing Systems. UrXiv:1404.000 2014.
[8] T. Dettmers and L. Zettlemoyer. Sparse networks from scratch: Faster training without losing 2019.
performance. arXiv preprint
[9] U. Evei, T. Gale, J. Menick, Castro, and Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.
[10] Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. arXiv: 1 803.03635.
[11] Frankle, G. K. Dziugaite, Roy, and M. Carbin. Pruning neural networks initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021. HiXiv:2009008576
[12] J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. The lottery ticket hypothesis at scale. arXiv preprint arXiv: /903.01611, 2019.
[13] T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv preprint 1rXiv:1902.09574.0 2019.
[14] Gale, M. Zaharia, C. Young, and Elsen. Sparse gpu kernels for deep learning. arXiv preprint 20Xiv:2006.10901, 2020.

---

[15] E. García-Martien F. Rodrigues, G. Riley, and H. Grahn. Estimation of energy consumption in machine learning. Journal of Parallel and Distributed Computing, 34: 2019.
[16] Gebhart, U. Saxena, and P. Schrater. A unified paths perspective for pruning at initialization. arXiv preprini 10X1v:2101.10552, 2021.
[17] J. Guo, Y. Li, W. Lin, Y. Chen, and J. Li. Network decoupling: From regular to depthwise separable convolutions. arXiv preprint arXiv:1808.05511 2018.
[18] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1135-1143, 2015.
[19] B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[21] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 12(241):1-124, 2021.
[22] I. Hubara, B. Chmicl, M. Island, R. Banner, S. Naor, and D. Soudry. Accelerated sparse neural training: A provable and efficient method to find n: M transposable masks. arXiv preprint arXiv:2102.08124, 2021.
[23] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.338 2014.
[24] S. A. Janowsky, Pruning versus clipping in neural networks. Physical Review A, 39(12):6600, 1989.
[25] S. Jayakumar, R. Pascanu, J. Rae, S. Osindero, and E. Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744-20754, 2020.
[26] E. R. Kandel, J. H. Schwartz, T. M. Jessell, S. Siegelbaum, A. J. Hudspeth, and S. Mack. Principles of neural science, volume 4. McGraw-hill New York, 2000.
[27] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[28] A. Kusupati, V. Ramanujan, R. Somani, M. Wortsman, P. Jain, S. Kakade, and A. Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning, pages 5544 5555. PMLR, 2020.
[29] Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
[30] N. Lee, T. Ajanthan, S. Gould, and P. H. Torr. A signal propagation perspective for pruning neural networks at initialization. In International Conference on Learning Representations, 2020. 10Kiv:1906.0660
[31] N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. International Conference on Learning Representations, 2018.
[32] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. International Conference on Learning Representations, 2016.
[33] Lin, S. Stich, L. Barba, D. Dmitriev, and M. Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
[34] J. Liu, Z. Xu, R. Shi, R. C. C. Cheung, and H. K. So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In International Conference on Learning Representations, 2020.
12

---

[35] S. Liu, D. C. Mocanu, A. R. R. Matavalam, Y. Pei, and M. Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications, 33(7):2589-2604 2021.
[36] S. Liu, D. C. Mocan,, Y. Pei, and M. Pechenizkiy. Selfish sparse mn training. In Proceedings of the 39th International Conference on Machine Learning, pages 6893 6904. PMLR, 2021.
[37] S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy. Do we actually need dense over- parameterization? in-time parmmterizaaiionnn in sparse training. In Proceedings of the 39th International Conference on Machine Learning, pages 6989-7000. PMLR, 2021.
[38] Liu and Zenke. Finding trainable sparse networks through neural tangent transfer. In International Conference on Machine Learning, pages 6336-6347. PMLR, 2020.
[39] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. International Conference on Learning Representations, 2019.
[40] C. Louizos, M. Welling, and D. Kingma. Learning sparse neural networks through 1_0 regularization. International Conference on Learning Representations, 2018.
[41] M. Mahar and V. Cavalli. Intrinsic mechanisms of neuronal axon regeneration. Nature Reviews Neuroscience, 19(3)3333-337 2018.
[42] D. Mittal, S. Bhardwaj. M. M. Khapra, and B. Ravindran. Recovering from random pruning: On the plasticity of deep convolutional neural networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 848-857. IERE, 2018.
[43] D. C. Mocanu, E. Mocanu, T. Pinto, S. Curci, P. H. Nguyen, M. Gibescu, Ernst, and A. Vale. Sparse training theory for scalable and efficient agents. International Conference on Autonomous Agents and Multiagent Systems (AAMAS). arXiv:2103.01666, 2021.
[44] D. C. Mocanu, E. Mocanu, Stone, P. H. Nguyen, M. Gibeseu, and Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):3383, 2018.
[45] D. Molchanov, Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pages 2498-2500 PMLR, 2017.
[46] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz. Importance estimation for neural network pruning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 162--12772, 2019.
[47] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. International Conference on Learning Representutions, 2016.
[48] H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. International Conference on Machine Learning, 2019.
[49] M. C. Mozer and P. Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. In Advances in neural information processing systems, pages 107-115, 1989.
[50] M. C. Mozer and P. Smolensky. Using relevance to reduce network size automatically. Connec- tion Science, 1(1):3-16, 1989.
[51] G. Nagappan, H. Chen, and D.-Y. Wang. Neuroregeneration and plasticity: a review of the physiological mechanisms for achieving functional recovery postinjury. Military Medical Research, 7(1):1-16, 2020.
[52] Nvidia. Nvidia a00 tensor core gpu architecture. https:/wwwwavvida..cmcoootttttte 2020.
[53] D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104. 10350, 2021.
13

---

[54] A. Renda, J. Frankle, and M. Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. arXiv:2003.02333 02388.
[55] Savarese, H. Silva, and M. Maire. Winning the lottery with continuous sparsification. arXiv preprint
2019.
[56] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni. Green ai. arXiv preprint arXiv:1907.10597.
2019.
[57] Y. Shen, L.. Shen, H.-Z Huang, X. Wang, and W. Liu. Cpot: Channel pruning via optimal transport. arXiv preprint arXiv:2005. 10451, 2020.
[58] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2014.
[59] S. P. Singh and D. Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33, 2020.
[60] S. Srinivas,
A. Subramanya, and R. Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 145, 2017.
[61] N. Strom. Scalable distributed dnn training using commodity gpu cloud computing. In Sixteenth Anmual Conference of the International Speech Communication Association, 2015.
[62] E. Strubell, A. Ganesh, and A. McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 02244,, 2019.
[63] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems. w0Xv:2006.005667, 2020.
[64] H. Touvron, A. Vedaldi, M. Douze, and H. Jegou. Fixing the train-test resolution discrepancy. In H. Wallach, H. Larochelle, Beygelzimer, F. "'Alcéé-Buc E. Fox, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[65] S. Verdenius, M. Stol, and P. Forré. Pruning via iterative ranking of sensitivity statistics. arXiv preprint arXiv:2006.00880 2020.
[66] S. Verma and J.-C. Pesquet. Sparsifying networks via subdifferential inclusion. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10542-10552, PMLR, 18-24 Jul 2021.
[67] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference Learning Representutions, 2020.
[68] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[69] M. Wortsman, A. Farhadi, and M. Rastegari. Discovering neural wirings. H. Wallach, H. Larochelle, A. Beygelzimer, F. MAlché-Buc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[70] X. Xiao, Z. Wang, and S. Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural information processing systems, 32, 2019.
[71] Y. Xu, Y. Li, Zhang, W. Wen, B. Wang. Y. Qi, Y. Chen, W. Lin, and H. Xiong. Trained rank 2018.
pruning for efficient deep neural networks. arXiv preprint
[72] D. Yang, A. Ghasemazar, X. Ren, M. Golub, G. Lemieux, and M. Lis. Procrustes: a dataflow and accelerator for sparse deep neural network training. In 2020 53rd Annual IEFF/ACM International Symposium Microarchitecture (MICRO), pages 71 711-724. IEEE, 2020.
14

---

[73] G. Yiu and Z. He. Glial inhibition of ens axon regeneration. Nature Reviews Neuroscience, 7(8):617-621) 2006.
[74] H. You, C. Li, P. Xu, Y. Fu, Y. Wang, X. Chen, R. Baraniuk, Z. Wang, and Lin. Draw- ing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint 19Xiv:199.119999 2019.
[75] G. Yuan, L. Shen, and W.-S. Zheng. A block decomposition algorithm for sparse optimization. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 275-285, 2020.
[76] A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning n.m fine-grained structured sparse neural networks from scratch. In International Conference Learning Representations, 2021.
[77] M. H. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression, 2018.
15


Reviews:
[{"review_id": "wwzcFhXCEuC", "reviewer": "Reviewer_93rT", "summary": "This work aims to combine ideas presented in recent dynamic sparse training (DST) methods with gradual magnitude pruning. Additionally the paper has some novel experiments that confirm previous work on when-is-best to prune during training. Although an interesting and potentially impactful idea, the story and results need a bit more work.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Pros\n1) Results in Fig:2 are quite interesting. Though some clarification is needed (see below)\n2) The idea of combining pruning and re-generation is novel and authors do a good job on comparing with a large set of methods.\n\n# Cons\n1) \"For this reason, we determine to focus on gradient-based regeneration, i.e., regenerating the same number of connections as pruned with the largest gradient magnitude.\" This is sounds like what [rigl] does for training sparse networks. Is the method proposed for regeneration is practically same as rigl? If so it would be nice to called it so. If not, it would be nice to see a discussion on similarities and differences. My understanding is GraNet=GMP+RigL. It would be nice to make this clear.\n2) It needs to be clear which experiments uses global pruning (and thus non-uniform sparsity). For example on Table-3, GraNet-ST (ERK) requires 0.37× training flops which is less than rigl (erk). Shouldn't be the training flops of GraNet-ST always larger than rigl since it starts with more parameters? I am assuming global pruning is not used here, as if so; then the resulting distribution is not ERK anymore and this should be made clear. \n3) It should be make clear that GraNet-ST is different than other sparse training methods like set and rigl in the sense that it requires more sparsity/memory to begin with.  \n3) I couldn't see any ablations on the initial sparsity of GraNet-ST method. I think the value 0.5 is picked, which is a significant increase in parameter count if the final sparsity is >90%; it would be nice to see effect of initial sparsity. \n4) It might be better to use term 'sparsity' instead of 'density' on Figure 2. More importantly I am confused what \"pruning rate\" is referring to. Reading the text, it seems to refer sparsity of the network after pruning. Then it is not clear how do you prune a 98% sparse (2% dense) network to 50% sparsity. Do you only grow? This needs to be cleared I think. Experiments in Figure:2 looks quite interesting otherwise.\n\n## Minor\n1) \"To better understand the effect of pruning during the optimization proces\" [zhu2017] is a relevant work that studies the same.\n2) \"To the best of our knowledge, this is the first time in the literature that sparse-to-sparse training reaches a test accuracy of 76% with ResNet-50 on ImageNet at sparsity 0.8.\" [rigl] paper I think has 5x training runs with such results. If extended training runs are not considered; this should be explained. I think a better way to compare dense-to-sparse methods with DST methods is to use training flops on the x-axis.\n3) \"Perhaps most impressively, the latter for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods by a large margin with ResNet-50 on ImageNet.\" I guess large-margin is subjective but [rigl] shows the same. \n\n[rigl] https://arxiv.org/abs/1911.11134\n[zhu2017] https://arxiv.org/pdf/1710.01878.pdf\n\n## After Rebuttal\nI thank the authors for their strong rebuttal. I think all my concerns/questions are addressed and I believe this work is a timely work that inspire and impact future research. Therefore I support acceptance.\n\nOne final suggestion: \n- I think it might help the readers to name both versions of the method as GraNet as when initial sparsity=0 GraNet-ST is GraNet. Then authors can use sth like GraNet (s=0) to identify pruning version.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ftBba7gkhxr", "reviewer": "Reviewer_RYno", "summary": "This paper proposes a concept of pruning plasticity to study the effect of pruning during training. Using the insights obtained from the pruning plasticity, the authors improve the state-of-the-art sparse training performance. Since a large body of recent works has emerged to accelerate the training process, this work is helpful for this area.\n\nThe findings are interesting and make good connections with separately reported phenomena in the literature. Based on the insights from pruning plasticity, the authors further propose two gradual pruning methods which improve the performance over the existing methods. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n(1)\tThe paper does a thorough study of pruning during training, which is indeed recently ignored by researchers compared with after-training pruning and before-training pruning. The recently emerged before-training pruning (pruning at initialization) absolutely receive a lot of attention. This work, however, doesn’t follow the crowd and highlights the values of an existing technique - pruning during training. Due to the failure of the existing before-training pruning methods to capture the important masks pointed by [1], discovering either early-existing matching subnetworks or more accurate subnetworks during training have a great value for the community. \n(2)\tThe observations associated with pruning plasticity make good connections with several existing phenomena in the pruning literature, providing valuable insights to the people who want to use these techniques in reality. \n(3)\tWith the insights provided by pruning plasticity and the neuroregeneration from the nervous system, authors modify the existing gradual magnitude pruning methods and demonstrate the state of the art sparse training performance as well as the gradual pruning performance. \n\nQuality:\n(1)\tThe paper is clearly written and easy to follow. The authors provide an adequate related work review. \n(2)\tI like the random initialization analysis of the discovered masks between GMP and GraNet, making the overall paper more robust. \n(3)\tThe claim is well supported by the extensive experiments. \n\nSignificance:\n(1)\tWhile the idea of during-training pruning is not new, it indeed achieves the best performance-efficiency tradeoff compared with after-training pruning and before-training pruning. Works like this paper that provides better understanding and performance improvement of pruning during training are important for the ML community, providing good motivations for future hardware design. \n(2)\tThe proposed methods are sound and performant. Especially, the 0.9% and 1.5% accuracy improvement achieved by GraNet-ST over RigL with ResNet50 on ImageNet at sparsity of 80% and 90%, respectively, is quite impressive. It is the first study which, up to my knowledge, shows consistently that dynamic sparse training can outperform dense-to-sparse training with even fewer training FLOPs.\n\nImprovements:\n(1)\tThis paper only considers unstructured pruning. I suppose that the findings of pruning plasticity can also generalize to the structured setting. It would be nice to test pruning plasticity on the structured pruning. \n(2)\tThe authors should cite [2,3,4] for the zero-cost neuroregeneration.\n(3)\tIt is unclear to me if the pruned weights are still updated or not during the backward pass. This would cause a significant computational difference. It would be nice to make this point clearer.\n(4)\tI suggest the authors to put more emphasis on the dynamic sparse training part (GraNet-ST) rather than GraNet, which is more promising in terms of the accuracy-efficiency trade-off. The description of GraNet-ST in the current version is somehow unclear, especially for readers who are not familiar with sparse training.\n\nReference:\n[1] Frankle, Jonathan, et al. \"Pruning neural networks at initialization: Why are we missing the mark?.\" arXiv preprint arXiv:2009.08576 (2020).\n[2] Mocanu, Decebal Constantin, et al. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.\" Nature communications 9.1 (2018): 1-12.\n[3] Evci, Utku, et al. \"Rigging the lottery: Making all tickets winners.\" International Conference on Machine Learning. PMLR, 2020.\n[4] Dettmers, Tim, and Luke Zettlemoyer. \"Sparse networks from scratch: Faster training without losing performance.\" arXiv preprint arXiv:1907.04840 (2019).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "DpDjav09sQW", "reviewer": "Reviewer_VuBd", "summary": "This paper understands the during-training pruning from the perspective of training plasticity which is the ability of the pruned networks to recover the original performance. A technique named neuroregeneration is further shown to improve the training plasticity. Based on these findings, a gradual magnitude pruning (GMP) method with neuroregeneration (GraNet), and its dynamic sparse training variant are proposed. Experiments show the effectiveness of GradNet and GradNet-ST.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\nThe paper is well-written. The contributions are clearly presented.\n\nModel compression is an important topic in scaling deep models on mobile devices. It is essential to design effective channel pruning method.\n\nThe key idea of neuron regeneration is well-motivated.\n\nExtensive experiments are conducted to show the superiority of GradNet..\n\nCons\n\nIt seems that GradNet achieves marginal improvements over STR on the ImageNet dataset.\n\nDoes neuron regeneration work on a structured pruning framework?\n\nThe training plasticity is sensitive to hyper-parameters. It would be better to provide experiments to show how neuron regeneration helps training plasticity. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how to optimize the performance of sparse models which are pruned during the training process by combining during-training pruning with the growth criterion originally proposed in RigL. All reviewers felt the paper was clear and covered a timely and important topic, though there were some concerns regarding clarity and comparison to prior work which were resolved through the author discussion. As such, I recommend the paper be accepted. 
Decision: Accept (Poster)