NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization
Authors: Gen Li
Lu Yin
Jie Ji
Wei Niu
Minghai Qin
Bin Ren
Linke Guo
Shiwei Liu
Xiaolong Ma
Venue: ICLR 2024 poster (2024)
Keywords: Dynamic Sparse Training; Neuron Revitalization
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Dynamic Sparse Training (DST) employs a greedy search mechanism to identify an optimal sparse subnetwork by periodically pruning and growing network connections during training. To guarantee effectiveness, DST algorithms rely on high search frequency, which consequently, requires large learning rate and batch size to enforce stable neuron learning. Such settings demand extreme memory consumption, as well as generating significant system overheads that limit the wide deployment of deep learning-based applications on resource-constraint platforms. To reconcile such, we propose $\underline{Neur}$on $\underline{Rev}$italization framework for DST (NeurRev), based on an innovative finding that dormant neurons exist with the presence of weight sparsity, and cannot be revitalized (i.e., activated for learning) even with high sparse mask search frequency. These dormant neurons produce a large quantity of zeros during training, which contribute relatively little to the outputs of succeeding layers or to the final results. Different from most existing DST algorithms that spare no effort designing weight growing criteria, NeurRev focuses on optimizing the long-neglected pruning part, which awakes dormant neurons by pruning and incurs no additional computation costs. As such, NeurRev advances more effective neuron learning, which not only achieves outperforming accuracy in a variety of networks and datasets, but also promoting a low-cost dynamism at system-level. Systematical evaluations on training speed and system overhead are conducted on the mobile devices, where the proposed NeurRev framework consistently outperforms representative state-of-the-arts. Code will be released.

Published as a conference paper at ICLR 2024
NEURREV: TRAIN BETTER SPARSE NEURAL NETWORK PRACTICALLY VIA NEURON REVITALIZATION
Gen Li', Lu Yin ,Jie Ji', Wei Niu", Minghai Qin, Bin Ren", Linke Guo',
Shiwei Liu Xiaolong Ma²
Clemson University Eindhoven University of Technology University of Georgia
4 William & Mary" University of Aberdeen 6 University of Oxford
(gen, xi xiaolom} ) celemson.edu
ABSTRACT
Dynamic Sparse Training (DST) employs a greedy search mechanism to iden- tify an optimal sparse subnetwork by periodically pruning and growing network connections during training. To guarantee effectiveness, DST algorithms rely on high search frequency. which consequently, requires large learning rate and batch size to enforce stable neuron learning. Such settings demand extreme memory consumption, as well as generating significant system overheads that limit the wide deployment of deep learning- applications on resource-constraint plat- forms. To reconcile such, we propose an Neuron Revitalizationframewort for DST (NeurRev), based on an innovative finding that dormant neurons exist in the presence of weight sparsity and cannot be revitalized (i.e., activated for learning) even with high sparse mask search frequency. These dormant neurons produce a large quantity of zeros during training, which contribute relatively little to the outputs of succeeding layers to the final results. Different from most existing DST algorithms that spare no effort designing weight-growing criteria, NeurRev focuses on optimizing the long-neglected pruning part, which awakens dormant neurons by pruning and incurs no additional computation costs. As such, NerRev advances more effective neuron learning, which not only achieves outperformance accuracy in a variety of networks and datasets but also promotes low-cost dy- namism at the system level. Systematical evaluations on training speed and system overhead are conducted on mobile devices, where the proposed NeurRev frame- work consistently outperforms representative of-ee----thhtttttt Code available www.uu..cccccoooooo http:////g..hubbo.oooo/ooooeeoee
ICLR2024
INTRODUCTION
Deep Neural Networks (DNNs) have gained significant traction in a number of industries, such as advanced manufacturing (Nguyen et al.] 2021 Zhang et al.] [202]] Liu et al. 22220), connected healthcare (gguyen al., 2022 2021). and Augmented Reality/Virtual Reality (AR/VR) (Chen) et al. 2020 Eris et al. [202] l). etc. As the technologies promote a plethora of emerging applications, edge computing (Shi al. 2016 Abbas et al. 2017) becomes the core enabler facilitate the widespread of machine intelligence. However, as large models are prevailing in the recent deep learning era, matching the resource budget and the ever increasing DNN model complexity becomes unprecedented challenging (Li et al. 2019 Liu et al. 2022a Li et al. 2023).
With the potential to save time in both training and inference, dynamic sparse training (DST) Mooanu
et al. Mostafa & Wang 2019; Dettmers & Zettlemoyer 2019 Lym et al. 2019 Evei et al.
2020
Jayakumar et al., 2020) Yuan et al.) 2021 Liu et al.) 2021 a Hou et al. 2022: Jaiswal et al. 2022 Yin et al. 2024) has received tremendous attention for extending the scalability and facilitating larger design space of deep intelligence implementations (Liu et al. [022b]] Yuan et al. 2022) ). Due to its "always sparse" training regime and impressive training accuracy. DST maintains low memory footprint by avoiding the full dense model computation, and quickly becomes a preeminent approach in efficient training field.

---

Published as a conference paper at ICLR 2024
Beshwe332, VGG Al.lp
Number of 2010 Toweue magg
Uritcm ERK
Number al si--pre frigh coe mage
69.2
20.
93.4
52.5
512
Epochsipdate (USupaaee frequency)
Sparthy
Taahinng opochs
Figure 1: Preliminary results of DST. Left: accuracy changes with different DST update frequency. Middle: dormant neurons emerge and increase when network becomes sparse. Right: dormant neuron population increases during DST.
While DST has achieved superior performance, most DST algorithms are only focusing on algorithm level in pursue of high accuracy and theoretical computation reduction. From general optimization perspective, the effectiveness the algorithms cannot be guaranteed due to the random and heuristic nature of the dynamic network re-routing scheme, resulting in high frequency updates on network topology to ensure the greedy search works (Evci et al. 2020). As shown in Figure Ia), when the update frequency decreases, DST accuracy drops significantly. From system design perspective, high update frequency requires large learning rate and batch size to enforce stable neuron learningEEvci et al. (2020): [Liu et al. (2021c), incurring large system overhead on training devices, hence contradicting to the principles of deploying efficient training ubiquitous devices. Here, the system overhead comes from the time spent on the computation graph reconstruction after each model typology update on the edge. Considering the model with the highest accuracy in Figure a],, the total of 1,200 updates translate to 114 and 686 minutes system overhead when the sparse ResNet-32 and VGG-19 models compiled to executable code mobile devices (Niu al. 2020).
In accordance with the above observations, we find that the relatively low searching effectiveness is the key that causes high search frequency and any subsequent drawbacks. We identify the reason is that sparsity produces dormant neurons in the DST process, and such dormant neurons are extremely persistent to the pprune-grow" dynamism of DST once they are generated. These dormant neurons produce large number of zeros, which contribute relatively little to the output of succeeding layers or to the final results (Hu al.] 2016). Figure we demonstrate the number of dormant neurons emerged and increased when sparsity is gradually introduced into DNN training. And Figure Ic) shows that DST searching algorithm fails discourage the production of dormant neuron. On the contrary, dormant neuron population increases throughout training process, degrading the expressive power of neurons and wasting the valuable active weights in the network.
In this paper, we argue that dormant neurons are those convolution filters that not only contain negative values large magnitude but also result in negative outputs after convolution and are set to zero for ReLL outputs. Although these neurons have weights, they cannot be effectively updated during backpropagation. When the network is sparse, the impact such large negative weights is amplified since they account for larger proportion the remaining weights (see Figure (b). On the other hand, the frequent pruning operation in DST cannot surgically remove those harmful weights since they are heuristically pruned by small magnitude (Mocanu et al. 2018 Mostafa &
Wang 2019 Dettmers & Zettlemoyer 2019 Lym et al. 2019 Evei et al. 2020 Yuan et al. 2021
Liu et al. 2021c a).
To address the problem, we propose Neuron Revitalization framework for DST (NeurRev), which comprises an innovatively re-designed pruning operation that identify and surgically remove large negative weights. By doing so, NeurRev successfully reduces the number of dormant neurons as shown in Fi), which facilitate more effective learning. Please note that our proposed method not solely prune the negative weight based on their magnitudes, but also based on the pest-ReLU feature map. Therefore, what we suggest is essentially a data-driven approach, different from earlier heuristic pruning techniques, demonstrating its effecttivnness. We also demonstrate that NeurRev effectively reduces the dynamic update frequency, which enables efficient training on resource- constrained mobile devices. We also discuss the dominant practice of ReLU in software-hardwe co-design perspective, exhibiting the wide applicability of NeurRev in the efficient learning field. We summarize contributions follows:
2

---

Published as a conference paper at ICLR 2024
algorithı (prune-grow)
DST original praniing
Input feature map
CONV output
RcLU output
2ech
Convolution
Prune
Value
Nourfee jouns)
CONV output
outpat
Overview of NeurRev. In DST process showing in the left, NeurRev not only prunes the weight variable based on the magnitude of each weight, but also identifies and prunes the harmful weights that produce dormant neuron (i.e., all zero output) shown on the right.
Figure 2:
We find dormant neurons exist in the DST training process as sparsity is introduced and increased DNN model. We show the dormant neurons are persistent to the DST algorithms and degrading the learning effectiveness that eventually leading to impractical settings for efficient deployment.
We develop novel DST framework NeurRev, which identifies and surgically removes large negative weights revitalize the dormant neurons, and achieves outperforming accuracy on CIFAR-10/100 and ImageNet compared to the state-of-the-arts.
We deploy NeurRev on the saource-constraiigg mobile devices, and achieve 1.8-2.1 x acceleration dense training and maximum of 112x less overhead compared other sparse training methods.
We systematically evaluate ReLU activation function a software- hardware co-design perspec- tive, demonstrating NeurRev exhibits significant promise for extensive adoption in efficient learning field.
software-hrrdware
2 NEURON REVITALIZATION FOR DYNAMIC SPARSE TRAINING
In this section, We first describe the preliminary knowledge and notations of the dynamic sparse training algorithms. Then, we show that dormant neurons are produced when the neural network is sparse and put forward our algorithm to search and awake dormant neurons during sparse training. Our method is targeting practical implementation of efficient training on the edge device, therefore we also discuss NeurRev in seftware-hardwanr co-design perspective.
2.1 PRELIMINARIES
Sparse training seeks to minimize the loss function Σ, [(f(x;; 0z), yi) for a given dataset D with individual samples X; and targets yı- Here, f(X;; ; 0.) denoted sparse neural network. 0g is subset of dense weight 0 E N,,, where N is the total number of weights in the network.
Sparse initialization. Current works initialize sparse neural networks mainly following two schemes: uniform (Mostafa & Wang 2019: Dettmers & Zettlemoyer 2019) and Erdós-Rényt (ER) graph Mocanu et al. [018 Evei al. 20200 Liu al. a (0226). Under the uniform setting, the sparsity of each layer equal to the whole network sparsity. ER scheme will adaptively allocate sparsity based on the number of parameters of each layer, while ensuring the total sparsity at a certain value. Based on ER, Erdōs-Rényi-kee RenyR-keniiiee (ERK) (Evci et al.) 2020) additionally takes into account the information of the convolution kernel in assigning sparsity of each layer. Other sparsity distribution such dynamic sparse reparameterization (Mostafa & Wang) 2019) adjusts the layer-wise sparsity on-the-fly fly during training process. In this paper, we use the uniform and ERK sparsity since the fixed layer-wise sparsity ratio brings benefits the hardware, and they avoid reconfiguration of the hardware resources, which indicates lower overhead and stable throughput. The overall weight sparsity can be computed with ||0.|o, where given variable.
|/o denotes the number of non-zero elements in
10

---

Published as a conference paper at ICLR 2024
Training process. During the DST process, existing methods follows prune-gron regime to explore different network topologies (i.e., subnetworks), as shown in 2 Suppose the total number of training iteration is Tend, then conduct the DST step for the Tstop (< Tend) iterations at every ATp iteration. DST prunes the remaining weights and then grows same number of weights back from the inactive ones to maintain sparsity ratio. Among variety of DST algorithms, parameter growing is based on different metrics, such as gradient (Evci et al. 2020), momentum (Dettmers & Zettlmoyer 2019), or randomness (Yuan et al. 202T). On the other hand, pruning is solely based on the magnitude of the weights. Our innovation primarily resides in the meggeeelected pruning part, and random growing (Yuan et al., 202I) is adopted to reduce the peak memory consumption during the backpropagation.
2.2
NEURREV FOR DST FROM RATIONALE TO ALGORITHM
2.2.1 1
WHYY DORMANT NEURONS EXIST
In the case of image classification tasks, we mainly leverage ReL.U-based CNNs. The weights in convolutional and fully connected layers update according backpropagation and gradient descent algorithm in the training process. In the backpropagation process. we first compute the loss by cross entropy function. The loss L can be denoted as: L - Ex=1 y:log Ds where y/ is the ground truth label of the sample, ys is the predicted label of the model. The loss will be used in the
backpropagation process to update the model weights w/y using the gradient descent (GD) algorithm.
(!)
The GD algorithm can be expressed as the following equation: w!y w
a DC where BC is awly
awlj
(!) the gradient of wij and can be expressed as:
ac OC 0z. (2((+1)
= -((41) Oz! 02((41) = 8((+1)
a
Ow!y Oz, (1+1)
auly
aw!y
w/ represents the weights of neuron connections from the neuron i in layer / the neuron j in layer I ++1 1, and a stands for learning rate. o((+1) means the gradient values of unit in layer + 1, As shown in Figure under the sparsity constrains, the negative weights among the remaining weights have more dominant effects on generating dormant neurons, which produces numerous zeros in forward propagation, and leading to aj in equation I to be zero in backpropagation. As result, the gradient flow will be blocked by dormant neurons tnothe corresponding weights may not be updated.
(1)
2.2.2 SEARCH AND AWAKE DORMANT NEURONS
We present NeurRev, integrated algorithm where the operations are naturally coupled with the classic DST regime. Algorithm shows the outline of NeurRev, where the dormant neuron searching and awaking operations follow right after the occurrence of prune-grow step in DST, without incurring any additional update steps.
Search dormant neurons. Evaluating whether neuron is dormant neuron using layer output is computationally intensive due to the large volume of output feature maps. As we find that the gradients of dormant neurons are often zero, those neurons can hardly update weights in the training process. Based on this phenomenon, we can search for those dormant neurons according to their weight changes over a certain period of time. As shown in Algorithm [ non-zero weights after each NeurRev step, and compute the value differences AO with the newly updated weights during the next NeurRev step. Based on AA, we will sort and pick the p- 100% negative weights of total weights and record their indices for the awake step. Please note that the Prune &rrow& (+) function is the regular DST update step, but with the s - p target sparsity (i.e., more active weights) for the subsequent Search&Awake (-) operations.
we use O, to record the
Awake dormant neurons. Algorithm 2 demonstrates the steps to awake the dormant neurons by pruning the harmful weights recorded by the searching step. Our empirical observation indicates that the negative weights with zero changes (i.e., A = 0) are those with relatively large magnitudes, which proves our prior analysis in Section 1 As illustrated in Algorithm [ we awake the dormant neuron by pruning total number of -- |||| loo negative weights, thus reaching the target sparsity ratio s. By doing so, the neurons will be released from the dormant status and regain their learning ability.
4

---

Published as a conference paper at ICLR 2024
Algorithm 1: NeurRev for DST
Algorithm 2: Search and Awake
Input: 0s. fg s, ATp, T, P. Tatop
Input: 0. Os. A0,
Output: A sparse model satisfying the target
Output: A pruned sparse model.
parameter sparsity
Init: pruned weights =0
Init: Initialize O₂ according 0- 0..
sorted index = Wowrttt_iideeeeec ascending ) do
while 5 < Tstop do
if T mod 7Tp p == 0 then
for i in
A = abs(O, 05)
if [+] < then Prune(O.[i)
0. - ( Scarchsawekc(O.. A).p)
= pruned_ weights
0 = 0.
if seinee_weighhs--o p- ||0||o then exit ()
Continue sparse training from Tstop to Tend-
2.3 NEURREV IN SOFTWARE-HARDWARE CO-DESIGN PERSPECTIVE
The proposed NeurRev is mainly focused on optimizing the DST process of the ReLU-based neural network. In this section, we discuss the necessity of adopting NeurRev by evaluating the performance of ReLU activation function in
co-design perspective.
ReLU is a versatile activation function. ReLU activation function pervasive in variety of network
structures in computer vision tasks, such as ResNet (e et al. 2016). YOLO (Redmon et al.) 20166]
Redmon & Farhadi 2017 7 2018 Bochkovskiy et al. 2020), PointNet (Qi et al. 2017). Deeplab (Chen ct 2017),, etc. As other classic activation functions such sigmoid, hyperbolic tangent (tanh), LeakyReLU (Maas et al.) 2013), PReLU (He et al., 2015). etc. are also useful, ReLU is still the number one choice in the computer vision tasks due to its stable software-level performance and the ease of implementation at hardware-level. Although classic activation functions like sigmoid and tanh well supported in many edge platforms (e.g., AR.M-aased processor in mobile phones), they still suffer from limited accuracy due to gradient vanishing issue caused by saturated output range (van). Besides, extra operations such as exponentiation and division are introduced (e.g., sigmoid, tanh, ELU (Clevert et al.] 2016)), SELU (Klambauer et al. 2017) ,etc), which potentially increases the computation overhead.
Through rigorous investigation and more sophisticated searching algorithms (Ramachandran et al. 2017), new activation functions are developed to enhance the performance and capabilities of DNNs. However, the hardware implementation of such activation functions either employs the look-up table and Taylor expansion method (Timmons & Rice 2020), or uses the piecewise linear function method (Ngah et al. 2016,; Zhengbo et al. (2200) to approximate the non-linearity of the original activation functions, which essentially a trade-off between the degree of approximation and hardware performance. For instance, HardSwish (Howard al.) 2019) replaces the computationally expensive sigmoid with a piccewise linear analogue, but it degrades the accuracy during implementation. Lea,yReL.U, PReLU (He et al. 2015) and RReLU (Xu al. 2015) either experience unstable learning, or incorporate learnable parameters which bring higher implementation complexity. We present the experimental comparison of different activation functions in this software-hardwarc co-design perspective in Section 3.3]
Our framework. To support efficient and accurate training edge devices, especially the resource- constraint mobile devices, we use the compiler code generation (Niu et al. (2020) to convert DNN computation graph into static code (c.g., OpenCL or C++) for direct execution. During each sparse topology update, computation graph is reconstructed and the execution code is recompiled conse- quently, which incurs recompilation time overhead that elongates the total training time. Undoubtedly, the less topology update occurs, the less overhead there is for the overall training time. In our framework, NeurRev achieves notably low update frequency by fixing the most significant drawback that ReLU brings, while preserving the software- and hardware-level advantages (e.g., stable, easy implementation) that ReLU comprises. By enhancing the learning ability of sparse neurons, the searching for the optimal network topology can be more effective, thus reducing the number of iterations and hardware resources. Please note that the principle of awaking the dormant neuron can also be extended to non-Rel.U network and tasks.
5

---

Published as conference paper at ICLR 2024
Table 1: Test accuracy of pruned ResNer-32
CIFAR-10/100.
Sparsity Distribution
Datasets
CIFAR-10
CIFAR-100
Pruning ratio
90%
95%
98%
90%
95%
98%
ResNet-32
dense
94.88
94.88
94.88
74.94
LT 02
non-uniform
92.31
91.06
88.78
68.99
57.37
SNIP
non-uniform
92.59
91.01
87.51
68.89
65.02
57.37
GraSP
non-uniform
92.38
91.39
88.81
69.24
66.50
58.43
Deep-R
non-unifomn
91.62
89.84
86.45
66.78
63.90
58.47
SET 34
non-uniform
92.30
90.76
88.
69.66
67.41
62.25
DSR
35
non-uniform
92.97
91.61
88.46
69.63
68.20
61.24
RigL--110P
uniform
93.19
92.08
89.36
Rigl. 11
uniform
93.07
91.83
89.00
70.34
68.22
64.07
MEST+EM
58
uniform
92.55
91.15
89.22
70.44
68.43
64.59
NeurRev (ours)
uniform
93.31-::0.11
92.18=00.14
89.96:00.12
70.87=00.08
68.77:10.12
64.91 0.006
RigL (
ERK
93.55
92.39
90.22
70.62
68.47
64.14
RigL-ITOP
ERK
93.70
92.78
90.40
71.16
69.38
66.35
NeurRev
ERK
93.84:1:0009
9..9
90.84:10.12
7.06
69.92:10.05
66.82 0.07
Original DST
Neurfee towrs)
Original DS7
Neurflev (owns)
Original DS7
Newflev
Cat
Layer 3
Layer
Layar
Original DST
(ows)
Original
(ows)
Original
NowrRev (ours)
Bird
Layer
Layer 17
Layer
Figure 3: Neuron outputs at different layers of a ResNet-32 with 90% sparsity on CIFAR-10 images.
EXPERIMENTAL RESULTS
In this section, we carry out extensive experiments demonstrate the advantages of NeurRev. We implement our framework based on ResNet-32 (Wang et al. 2020) and VGG-19 (Simonyan & Zisserman 2014), training and testing on CIFAR- TO/T00 (Krizhevsky 2009). In order to show that NeurRev also applies to large datasets, we train and test our framework ImageNet (Russakovsky ct al. 2015) based on ResNet-34/50 (He al., 2016). The training speed results are obtained using Samsung Galaxy S21 with Snapdragon 888 chipset. We repeat software training experiments for times and report the mean and standard deviation of the accuracy.
Experiment Setting We follow the setting of GraSP (Wang et al. 2020) to set training epochs equal to 160 for CIFAR-10/100. For ImageNet, we use both standard 100 training epochs and extended training epochs make comprehensive comparison with other methods. We the SGD optimizer and apply the cosine annealing learning rate schedule during training. For CIFAR- 1 10/100, set batch size of 64 and the initial learning rate 0.1. For the ImageNet software results, set the batch size to 1024 and the learning rate to 1.024. The learning rate is scheduled to have linear warm-up for epochs before reaching the initial learning rate value. In sparsity distribution settings, we carry out experiments on both uniform and ERK distributions. For the uniformly sparse distribution, we the first convolution layer dense and the sparsity of the remaining convolutional layers to the value of overall sparsity. For ERK distribution, we follow the setting of RigL (Evci et al.) 2020). The detailed experiment setting and
selections can found Appendix A.
3.1 ACCURACY RESULTS
CIFAR-10 and CIFAR-100. The NeurRev results on ResNet-32 and VGG-19 are shown in Table and Appendix B. We compare NeurRev with different sparse training methods 90%, 95% and 98%

---

Published as a conference paper at ICLR 2024
Table 2: Results of ResNet-50 on ImageNet-
Sparsity Distribution
Top-1 Accuracy
Training FLOPs
Inference FLOPs
Top-1 Accuary
Training FLOPs
Inference FLOPs
Method
ResNet-50
dense
76.9
(xe18)
(xe9)
76.9
(xe18)
(xc9)
Sparsity
80%
90%
1.
12
72.6
Iaa
70.1
n/a
1.7
SNIP
non-uniform
69.7
1.67
62.0
1.9
GraSP
72.1
1.67
68.1
0.91
1.9
Deep-R
71.7
n/a
70.2
n/a
n/a
SNFS 23
73.8
n/a
72.3
n/a
non-uniform
SET 34
72.9
0.74
69.6
0.10
0.1
DSR 35
73.3
1.28
3.3
71.6
0.96
2.5
RigL.
75.1
1.34
73.0
0.80
2.0
RigL-sx
77.1
6.69
3,4
76.4
3.94
2.0
ERK
NeurRey
75.7=0.09
1.34
3,4
73.4=10.6
0.80
2.0
NeurRevax
77.6=0.10
4.01
3.4
76.7==0.99
2.38
2.0
RigL. [
74.6
0.74
1.7
72.0
0.39
0.9
MFST
58
75.4
0.74
1.7
72.6
0.39
0.9
Top-KAST
20
uniform
73.0
0.63
0.9
SpFDE
(59
75.4
0.74
NeurRey
75.9=0.17
0.74
73.3:10.11
0.9
MEST,.7
76.7
1.84
1.7
75.9
0.80
0.9
uniform
NeurRev1.5x
76.7=0.08
1.10
76.1=00100
0.48
0.9
RigLox
11
76.6
3.71
1.7
75.7
1.95
0.9
uniform
NeurRevax
77.2
2.28
1.7
76.0.10.1
0.96
0.9
sparsity. We have included the static sparse training methods such as the lottery ticket hypostasis (LTH) (Frankle & Carbin 2019, arXiv:1803.0366
SNIP (Lee et al.) 2019). and GraSP (Wang et 2020). and the representative DST methods (Bellec et al. M
al. 2018 Mostafa & Wang 2019, Evei et al. 2020; Yuan al. 2021 Liu al. 2021c). We can notice that the accuracy DST are generally higher than static sparse training methods, which indicates that accelerating training with sparsity should use dynamic updating scheme. Compare with the baselines, NeurRev achieves better accuracy in both uniform and ERK sparsity distributions. We attribute the outperforming accuracy of NeurRev to the better learning ability of each sparse neuron. As shown in Figure 3 we randomly pick neurons at different layers of ResNet-32 model. We can see that when the neuron is dormant (i.e., all or most of weights are zeros) in traditional DST, NeurRev can successfully revitalize it with meaningful output. Additionally, NeurRev enhance the active neurons by better expressing the output, which also proves the effectiveness and importance of our proposed method.
ImageNet-1K. Saable the the training FLOPs and accuracy results of ResNet-50 on ImageNet- IK dataset. Compared with the SOTAS, our result shows higher test accuracy at the same computa- tional cost level. To make fair comparison with different training recipes of RigL (Evci al. 2020) and MEST YYuan et al. 2021), we scale our training epochs to have the same or less overall training FLOPs for comparisons. We the original training data demonstrate the software-level accuracy. and the results show that NeurRev consistently outperforms other baselines in both original or scaled training recipes. Note that training an original ImageNet-1l K on mobile device not practical, we also scale the input to 128x 128 and evaluate the hardware performance. The results are shown in Section 3.2 More analysis and results for ResNet-34 are shown Appendix
3.2 SYSTEM OVERIEEAD AND TRAINING ACCELERATION OF NEURREV DEPLOYMENT
System Overhead Evaluation. For hardware performance on the mobile phone, we follow the original training settings such as training epochs in the cited baselines, but scale the batch size to match computation resources on the hardware. We use batch size of 64 for both CIFAR and ImageNet. Please note that even change the batch size hardware implementation, the results can still reproduce the original accuracy with gradient accumulation. Figure 4 reports the number of updates, overhead time, and accuracy for network topology updates on ResNet-32. We clearly that NeurRev has 12-96x less updates/overhead while achieving better accuracy compared to others.
More results on VGG-19 19 shown in Appendix D.
End-to-End Training Acceleration with System Overhead. Figure 5 shows the comparison results of different DST methods in terms of overall training time with regards to the system overhead. We

---

Published as a conference paper at ICLR 2024
2.23
130
Figure 5: Training acceleration and overhead comparison with dense model and other DST methods.
ResNes-32
VOG-19
ReaNet-32
VOG-19
Neti
0.0
0.25
90
0.9
3.95
0.50
0.
05
0.00
0.5
0.55
5.98
Suflware Accuncy
(b) Harrbaarr Accuracy
Figure 6: Software and hardware evaluations on different activation functions, (a) accuracy obtained with GPU server, (b) accuracy of hardware implementation simulation.
have evaluated both mobile CPU and GPU time with CIFAR and ImageNet data. Note that original ImageNet data is not practical for edge-level training, we scale the input size to 128x 128. To better demonstrate the results, we use training epochs of 100 and 30 for CIFAR and ImageNet, respectively. We include the dense model training time as one baseline, and the experimental results shows that NeurRev achieves 1.8-2.1 1.1x x better acceleration rate dense training compared with other methods. Note that the training time includes the system overhead, which in some cases, the total time in sparse training may exceed dense model training even with the present of sparsity and optimized hardware acceleration framework. Therefore, NeurRev is proved to be the one practical DST method for efficient training on edge devices.
3.3 EVALUATE NEURREV WITH DIFFERENT ACTIVATION FUNCTIONS
NeurRev is targeting ReI-Ubbseed DNN architectures to enhance DST training performance in both software and hardware levels. In this part, we demonstrate the advantages using ReLU in a software-harr wharwe co-design perspective. We replace ReLU with multiple activation func- tions while leaving all other settings intact. Figure ((a) shows the software accuracy different sparsity ratios, and ReLU has overall better performance. When the training is performed on mobile device as showing Figure b)), most of other activation functions need be implemented with piecewise linear approximation. Even with specif- ically designed hardware version (c.g., HardSwish, HardSigmoid, etc.), the accuracy on hardware still cannot match the ReL.Ubaaed DNN.
Figure 4: Overhead evalua- tion on ResNet-32.
3.4 ABLATION STUDY
Different Update Ratios With NeurRev. We manually set the update ratio for the sparse topology searching process. The update ratio is the proportion non-zero weights that are being pruned and grew at each update step. We set the update ratios ranging from 1% to 30% and test the accuracy. As we can see in Figure a[),, the higher the sparsity of the network, the more sensitive is to the update ratio. Therefore, when the network is highly sparse, the update ratio should be carefully set.
> Different Update Frequencies With NeurRev. We test NeurRev at different update frequencies 10 epochs/update. We
in Figure b). We set the update frequency from 100 iteration/update can clearly see that our framework performs well at a low update frequency (e.g.. 5, 10 epochs per update). Recall the results in Figure a)), we can conclude that NeurRev stable DST algorithm.
8

---

Published as a conference paper at ICLR 2024
RoNNa-32
VOG-19
RecNes332
VOG-19
33.0
D0% Sparsiy
5E% Saatiyy
55% Soutly
0:21
1.06
1.
0.23
6.30
0.03
0.65
0.10
1.00
0.22
014
1.14
(A) Nourlev Update Ratio
(b) NeurRev Updnt Frequency
Figure 7: Different update ratios and update frequencies of NeurRev.
RELATED WORKS
Most Sparse training works can be divided into two categories, static sparse training, and dynamic sparse training. Static sparse training determines the structure of the sparse network at the initial stage of training by using the pruning algorithm. Dynamic sparse training begins with sparse network structure that is randomly chosen at the beginning of training and alters the sparse network structure during the whole sparse training process in an effort find a better sparse structure.
Static Sparse Training Static sparse training defines the structure of a sparse network through the application of a pruning algorithm. The lottery ticket hypothesis (LTH) (Frankle & Carbin 2019
arXivv 18X3vv1803.00 03635 Renda et al. 20200. ar arXiv:2003.022 (00:20003002889) Chen et al. 202 You et al. 2020; Frankle ct al. 2020) uses iterative sognitude-baseed pruning (IMP) to pinpoint a subnetwork that can be trained again without losing accuracy. SNIP (Lee et al.] 2019) leverages the gradients of the training loss at initialization and prunes the network at early stage of training. GraSP (Wang et al. 2020) prune connections based on the gradient flow. SynFlow (Tanaka et al. (020) preserves the total flow of synaptic strengths which avoids layer collapse issue during pruning. FISH (Sung et al. 202I) obtains fixed subnetwork by pre-computing sparse mask with Fisher information. Static sparse training requires full dense model forward and backward computation, which exhibits high peak memory and extensive computation the hardware.
Dynamic Sparse Training Dynamic sparse training aims to minimize the computational and memory footprint during the training period. Sparse Evolutionary Training (SET) (Mocanu al., 2018) employs magnitude- -based pruning and random growth at the end of each training epoch, which is effective and easy implement. Following this, Deep R (Bellec et al. 2018) combines dynamic sparse parameterization with stochastic parameter updates for training. However, the main result of this framework were down on small and shallow networks. Then, Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang] 2019) proposes to distribute parameters between layers. Although it achieves great precision, additional training FLOPS are caused by the distributed weight that is gathered in the front layers. Sparse Networks from Scratch (SNFS) (Dettmers & Zettlemoyer] (019)) creates the sparse momentum algorithm, which finds the layers and weights that effectively reduce the error by using exponentially smoothed gradients (momentum). Same as SNFS, RigL (Evci) (ct al. 2020) compute dense gradients when the model need to be updated to grow new connections. Top-KAST (Jayakumar et al. 2020)) proposes scalable and constant sparse DST framework for effectiveness and efficiency. Based on KAST, Powerpropagation (Schwarz et al. 2021) proposes new wright-parameterissie unaffected by learning. ITOP (Liu et al., (021c) investigates the underlying DST mechanism and finds that the advantages of DST result from a time-based search for all potential factors. MEST (Yuan) et al. 202T) designs a memorr--cononee
for neural networks, leaving the low-magni parameters largely
sparse training framework targeting for accurate and fast execution on edge devices. AD/AC (Peste al.) (202I) proposes a co-training of dense and sparse models method, which generates accurate sparse dense model pairs at the end of the training process.
5 CONCLUSION AND DISCUSSION OF BROADER IMPACT
In this paper, we introduce neuron revitalization framework for DST (NeurRev) to solve the long- standing dormant neuron issue that has been overlooked by most of the DST frameworks. NeurRev optimizes DST from a unique angle, which is to revitalize the dormant neuron by weight pruning, thus achieving more effective learning during DNN training. Thanks to the reduced system overhead with NeurRev, dynamic sparse training becomes practical, especially devices. The research is scientific in nature, and we do not envision it generating any negative societal impact.
cesosrae-conttriitt edge

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENT
This work is partly supported by the National Science Foundation CCF-2312616, CCF-2008049, NASA 80NSSC23K 1393, and the Army Research Office W911NF-24-1-004, Any opinions. findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF, NASA, and Army Research Office. The work of Guo is partially supported by NSF under grant 2008049 and 231 2312616. The work of L. Guo is also sponsored by the Army Research Office and was accomplished under Grant Number W911NF-24-1- 0044. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Govermment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
URL
Nasir Abbas, Yan Zhang, Amir Taherkordi, and Tor Skeie. Mobile edge computing: survey. IEEE Internet of Things Journal, 5(1))450-44) 463500446,,200 2017.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018. URL
https::/openreviie net Foormmidde aid=iilddddBDD_
Alexey Bochkovskiy, Chien- Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
Dawei Chen, Linda Jiang Xie, BackGyu Kim, Li Wang, Choong Scon Hong, Li-Chun Wang, and Zhu Han. Federated learning based mobile edge computing for augmented reality applications. In 2020 international conference on computing, networking and communications (ICNC), pp. 767-773 IEEE, 2020.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-844 2017.
Xiaohan Chen, Yu Cheng, Shuohang Wang. Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. ACL-IJCNLP, 2021.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). ICLR, 2016.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprini 1rXiv:1907.048800 2019.
Mümtaz Cem Eriş, Burak Kantarci, and Sema Oktug. Unveiling the wireless network limitations in federated learning. In 2021 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), pp. 262-267. IEBE, 2021.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019. 100.::1303.03355..
URL htttps://ppsnrewee
net nwrrtfrrurm id=rJl -3RcF7
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3269-3269. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun- Yuan Kung. Chex: channel exploration for enn model compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 12287-12298,, 2022.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1314-1324, 2019.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi- Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv: 16077.0200 2016.
Ajay Kumar Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pp. 9833-9844. PMLR, 2022.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744-20754, 2020.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019.
En Li, Lickang Zeng, Zhi Zhou, and Xu Chen. Edge ai: On-demand accelerating deep neural network inference via edge computing. IEEE Transactions on Wireless Communications, 19(1)444-4577 2019.
Gen Li, Jie Ji, Minghai Qin, Wei Niu, Bin Ren, Fatemeh Afghah, Linke Guo, and Xiaolong Ma. Towards high-quality and efficient video super-resolution via spatial-temporal data overfitting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10259-10269, 2023.
Di Liu, Hao Kong, Xiangzhong Luo, Weichen Liu, and Ravi Subramaniam. Bringing ai to edge: From deep learning's perspective. Neurocomputing, 485:297- 2022a.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9923 2021a.
Shiwei Liu, Iftitahu Ni mah, Vlado Menkovski, Decebal Constantin Mocanu, and Mykola Pech- enizkiy. Efficient and effective training of sparse recurrent neural networks. Neural Computing and Applications, 33:9625-9636, 2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over pree-aaameteiiztiionn in-time over parameterization in sparse training. In International Conference Machine Learning, pp. 6989-7000) PMLR, 2021c.
11

---

Published as a conference paper at ICLR 2024
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint wrXIv:2222.02643 2022b.
Yi Liu, Sahil Garg. Jiangtian Nie, Yang Zhang, Zehui Xiong, Jiawen Kang, and M Shamim Hossain. Deep anomaly detection for time-scries data industrial iot: communication--ffiicee on-device federated learning approach. IEEE Internet of Things Journal, 8(8):6348-66) 63588, 2020.
Sangkug Lym, Esha Choukse, Siavash Zangench, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-13, 2019.
Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Atlanta, Georgia, USA, 2013.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1:1-112, 2018.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646-4655. PMLR, 2019.
Syahrulanuar Ngah, Rohani Abu Bakar, Abdullah Embong, Saifudin Razali, et al. Two-steps implementation of sigmoid function for artificial neural network in field programmable gate array. ARPN journal of engineering and applied sciences, 11(7):4882-4888 2016.
Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, Jun Li, Dusit Niyato, and H Vincent Poor. Federated learning for industrial internet of things in future industries. IEEE Wireless Communicationss 28(6):192-19)) 2021.
Dinh C Nguyen, Quoc-Viet Pham, Pubudu N Pathirana, Ming Ding, Aruna Seneviratne, Zihuai Lin, Octavia Dobre, and Won-Joo Hwang. Federated learning for smart healthcare: A survey, ACM Computing Surveys (CSUR), 55(3):1-37, 2022.
Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang, Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. Patdnn: Achieving real-time dnn execution on mobile devices with pattern-based weight pruning. In Proceedings of the Twenty- Frifth-Fiftth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 907-922, 2020.
Alexandra Peste, Eugenia lofinova, Adrian Vladu, and Dan Alistarh. Ac/de: Alternating com- pressed/decooppressed training of deep neural networks. Advances in Neural Information Process- ing Systems, 34:8557-85700 2021.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pp. 652-660, 2017.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: 02767, 2018
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
12

---

Published as a conference paper at ICLR 2024
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. arXiv:2003.02389,
j
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-255 2015.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Powerpropa- gation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems, 34:28889-28903, 2021.
Weisong Shi, Jie Cao, Quan Zhang, Youhuizi Li, and Lanyu Xu. Edge computing: Vision and challenges. IEEE internet of things journal, 3(5):637-666 646, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2014.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 14:24193-242055 2021.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptie flow. Advances in Neural Information Process- ing Systems. arXiv:2006. 055667 2020.
Nicholas Gerard Timmons and Andrew Rice. Approximating activation functions. arXiv preprint arXiv:2001.06370. 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020. URL
net forum?
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv: 1.550000553 2015.
Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang. Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu, et al. Dynamic sparsity is channel-level sparsity learner. Advances in Neural Information Processing Systems, 36, 2024.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing carly-bird tickets: Toward more efficient training of deep networks. In 8th International Conference on Learning Representations, 2020.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838-20850, 2021.
Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong, Sergey Tulyakov, Xulong Tang. Yanzhi Wang, and Jian Ren. Layer freezing & data sieving: Missing pieces of a generic framework for sparse training. arXiv preprint arXiv:2209, 11004, 2022.
Weiting Zhang, Dong Yang, Wen Wu, Haixia Peng. Ning Zhang, Hongke Zhang, and Xuemin Shen. Optimizing federated learning in distributed industrial iot: A multi-agent approach. IEEE Journal on Selected Areas in Communications, (9(12):3688-3 3703, 2021.
Chen Zhengbo, Tong Lei, and Chen Zuoning. Research and design of activation function hardware implementation methods. In Journal of Physics: Conference Series, volume 1684, pp- 012111 11. IOP Publishing, 2020.
13

---

Published as a conference paper at ICLR 2024
Appendix
A EXPERIMENTAL SETTING
The hyperparameter settings for NeurRev are listed in Table A.1
Table A.I: Hyperparameter settings.
Experiments
VGG-19 on CIFAR
ResNet-32 on CIFAR
ResNet-50/34 on ImageNet
Regular hyperparumeter settings
Training epochs (Tend)
160
160
100
Batch size
64
1024 for hardware)
Learning rale scheduler
cosine
Initial learning
0.1
1.024
Ending Jearning rate
4c-8
4c-8
Momentum
0.9
0.9
0.875
fg regularization
5e-4
le-4
3.05e-5
Warmup epochs
NeurRev hyperparameter settings
Search and Awake epochs
120
120
75
Search and Awake frequency
90%
Sparsity: 10%
90%
Sparsity: 10%
80% Sparsity: 15%
Update ratio
95%
Sparsity: 5%
95%
Sparsity: 5%
90% Spansity: 10%
Sparsity: 5%
98% Sparsity:
B RESULTS OF VGG-19 ON CIFAR-10/100
We test NeurRev on VGG-19 using CIFAR-10 and 100. The results are shown Table B. We demonstrate results on both uniform and ERK distributions, and achieve SOTA results on CIFAR- 10/100. In Figure D.I. we demonstrate more feature maps of different classes in CIFAR-10.
Table B.1: Test accuracy of pruned VGG-19 on CIFAR-10/100.
Sparsity Distribution
Datasets
CIFAR-10
CIFAR-100
Pruning ratio
90%
95%
98%
90%
95%
98%
VGG-19
dense
94,20
94,20
94.20
74.17
74.17
74.17
[2
non-uniform
93.51
92.92
92.34
72.78
71.44
68.95
S5
non-uniform
93.63
93.43
92.05
72.84
71.83
58.44 58.46
non-uniform
93.30
93.43
92.19
68.90
Docp-R
non-uniform
90.81
89.59
86.77
66.83
63.46
59.58
SET
non-uniform
92.46
91.73
89.18
72.36
69.81
65.94
DSR
non-uniform
93.75
93.86
93.13
72.31
71.98
70.70
RigL-ITOP
uniform
93.19
92.08
89,36
70.46
68.39
64.16
[1]
uniform
93.12
92.43
90.65
71.14
69.02
64.87
MEST+EM
uniform
93.07
92.59
90.55
71.23
69.08
64,92
NeurRey (ours)
uniform
93.55-1:0.31
92.92-10.24
90.96:10.22
71.22-0.29
69.54-1-025
65.31 ::036
Rigl 11
ERK
93.77
92.75
90.87
71.34
69.21
65.02
RigL-ITOP 29
ERK
93.81
92.81
90.53
71.46
69.58
66.72
NeurRey (ours)
ERK
03.88:+0.29
93.11+#0.44
91.17-+0.22
72.06-00.16
69.95-t0.25
67.03 0.227
C RESULTS OF RESNET-34 ON IMAGENET-IK
Table C shows the result of NeurRev on ImageNet- IK using ResNet-34 as backbone. The results include regular training with 100 epochs, 1.5x (150 epochs) and 3x (300 epochs).
14

---

Published as a conference paper at ICLR 2024
Table C.I: Results of ResNet-34 ImageNet-1K. K.
Sparsity Distribution
Method
Top-1 accuracy (%)
ResNet-34
dense
74.10
Sparsity ratio
80%
90%
NeurRev
uniform
72.71
70.30
NeurRev1,5x
uniform
73.02
70.56
NeurRev3x
uniform
73.32
71.12
NeurRev
ERK
73.03
70,68
NeurRev1.5x
ERK
73.27
70.92
NeurRevsx
ERK
74.06
71.87
Original DST
NewsRev (oun))
Original DST
Neurfler (ours)
Original DST
Neurfler (ours)
Ship
Layer
Layer
Layer 14
Original DS7
(oun)
Original
Neurleev (owns)
Original
Neurllev (oans)
Layer
Layer
Layer
Neurfav
Meuriier
Neurfier
Truck
Original DS7
NeurRev
Original DST
NourRev
Original DS7
Meurfev (ours)
Frog
Layar
Layer
Original D57
Neurftev (oum)
Original
Meunfee (nun))
Original
Neunflev (oun)
Dog
Layer
Layer
Layer
Figure D.I: Neuron outputs at different layers of a VGG-19 with 90% sparsity on CIFAR-10 images.
D HARDWARE OVERHEAD EVALUATION RESULTS OF VGG-19
In Table D we show the recompile overhead of some popular DST frameworks VGG-19. The right-side radar map serves as visual example to illustrate how our method maintains good speed with little recompilation overhead.
15

---

Published as a conference paper at ICLR 2024
Table D.I: Overhead evaluation VGG-19.
Model
VGG-19
Update Times
Recompilation Overhead (min)
Method
Accuracy
SET (Mocanu al. 2018
160
92.46
88.2
Top-KAST (Mostaaa & Wang 2019
1248
93.08
686.4
RigL (Evci čt al. 2020
936
93.12
468.1
180
Ours
26
93.55
13.1
Updata
Ours
13
93.47
6.1
Tines
16


Reviews:
[{"review_id": "JZv1qUKBgh", "reviewer": "Reviewer_Fgjo", "summary": "This paper identifies the issue of \"dormant neuron\" in the weight-sparse training process, which hinders the performance of DST-trained models. The paper proposes a delta-based criteria to search the dormant neurons and prune them to move them out of the dormant stage, therefore helping the convergence of the sparse model. Results on multiple models and datasets show the proposed method can make DST more stable and improve the final training performance.", "questions": "See Weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. From the novelty prespective, the paper makes novel observation on the dormant neuron, and provide novel solution of delta-based pruning criteria in DST\n2. From the quality prespective, the paper is technically sound. The proposed method is well motivated, and adequate experiments are performed to show the effectiveness of the proposed method\n3. The paper is overal clearly written and easy to follow\n4. The inclusion of runtime overhead on real hardware further improves the significance of this paper", "weaknesses": "1. A relavent previous work, \"Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask\" (NeurIPS 2019) may be worth discussing. The paper explored multiple pruning criteria for LTH, including the proposed movement criteria and a similar \"magnitude increase\" criteria.  \n2. The paper focus it's discussion on CNN models, exploring models with ReLU activations and some variants. However, transformer-based models with GeLU activation is dominating SOTA architectures. It would be great to also try the proposed method on transformer model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2Qlapg3Xp", "reviewer": "Reviewer_5GHY", "summary": "The paper presents a framework for Dynamic Sparse Training (DST), referred to as NeurRev (Neuron Revitalization). The paper addresses the dormant neuron issue in DST, a problem that has been generally overlooked in existing DST frameworks. By employing weight pruning, NeurRev revitalizes dormant neurons, leading to more effective learning during deep neural network (DNN) training. The paper also highlights the practicality of implementing NeurRev on resource-constrained edge devices due to reduced system overhead. The work is comprehensive and places itself well within the existing literature on sparse training methods, discussing both static and dynamic approaches.", "questions": "1. The Search and Awake process only prunes negative weights, what would happen if there are not enough non-zero negative weights to prune? How do you identify the proportion of dormant neurons to prune?\n2. Other activation layers such as leaky-Relu may not set the negative input to zero but also a very small magnitude. According to the NeurRev algorithm, it should probably work the same as the original ones. Are there any limitations on the types of DNN architectures where NeurRev can be applied? Is there any plan to extend NeurRev to other types of networks? More experiments would be helpful to understand its applicability to broader cases. \n3. Could you provide more details on the computational overhead introduced by the NeurRev framework? The evaluation setup is not very clear. Explain more details about the simulation. \n4. What are the computational complexities involved in NeurRev in terms of both time and space? Are there any trade-offs?\n5. How robust is NeurRev to different optimization algorithms? Is it specifically designed to work best with certain optimizers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work identifies the interesting dormant neuron problems. \n2. The authors also draw a connection between DST's update frequency and its system overhead. \n3. To address the above observations, this work innovatively uses post-Relu results to prune convolution weights.", "weaknesses": "1. The paper would gain considerable strength from a more comprehensive set of empirical benchmarks. Specifically, the inclusion of real-world scenarios or case studies would offer a more holistic assessment of the method's efficacy and applicability. Furthermore, the method's robustness to different data distributions remains unexplored, leaving these as gaps in the experimental design.\n2. The NeurRev is very limited as it seems to only work in Relu-based CNNs. Experiments only show results on the Resnet family. Would it work on more compressed networks such as Mobilenet?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MbCKYKg55Z", "reviewer": "Reviewer_AQcf", "summary": "The paper introduces NeurRev, a novel pruning method that targets dormant neurons by assessing the change in weights during training. The authors have demonstrated the effectiveness of NeurRev through extensive software results and simulations on edge devices, showcasing its superior performance in comparison to various baselines.", "questions": "1. Could the authors clarify why the weight change is zero for some weights? I ask the question sice the author mentions that weight decay is used (See appendix A).\n2. For Figure 5, how are the update frequency chosen for baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The edge device simulations is a strength, as it demonstrates the practical applicability of NeurRev in real-world scenarios.\n2. NeurRev is well-motivated and is supported by visualizations of neuron outputs. The experimental results further solidify NeurRev’s promising performance across different benchmarks.", "weaknesses": "My primary concern is about unfair comparison. It seems that results of Table 2 are mainly from the original papers. And I see a rarely-seen learning rate of 1.024 and an optimized cosine learning rate scheduler for the ImageNet. I am worried that the performance boost is due to the optimized training recipe. Could the authors confirm that at least Table 1 is conducted with the same training recipe?\n\nMinor issues:\n1. Page 8 Section 3.2: Figure D -> Figure 5. \n\nI am happy to adjust my rating if my questions are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)