Efficient Neural Network Training via Forward and Backward Propagation Sparsification
Authors: Xiao Zhou
WEIZHONG ZHANG
Zonghao Chen
SHIZHE DIAO
Tong Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: sparse training
deep learning
model compression
deep neural networks

Abstract: Sparse training is a natural idea to accelerate the training speed of deep neural networks and save the memory usage, especially since large modern neural networks are significantly over-parameterized.  However, most of the existing methods cannot achieve this goal in practice because the chain rule based gradient (w.r.t. structure parameters) estimators adopted by previous methods require dense computation at least in the backward propagation step.  This paper solves this problem by proposing an efficient sparse training method with completely sparse forward and backward passes. We first formulate the training process as a continuous minimization problem under global sparsity constraint. We then separate the optimization process into two steps, corresponding to weight update and structure parameter update. For the former step, we use the conventional chain rule, which can be sparse via exploiting the sparse structure.  For the latter step, instead of using the chain rule based gradient estimators as in existing methods, we propose a variance reduced policy gradient estimator, which only requires two forward passes without backward propagation, thus achieving completely sparse training. We prove that the variance of our gradient estimator is bounded. Extensive experimental results on real-world datasets demonstrate that compared to previous methods, our algorithm is much more effective in accelerating the training process, up to an order of magnitude faster. 

Efficient Neural Network Training via Forward and Backward Propagation Sparsification
Xiao Zhou' Weizhong Zhang" Zonghao Chen', Shizhe Diao', Tong Zhang
Hong Kong University of Science and Technology. TTinnguaa University xzhoubi@conneet. ust hk, tsinghua. edu. en, sdiaoaa@ust. hk, tongzhangdtooggzhag-- org
czh17@mails. taai7/mmalss.tsinhuuue
Abstract
Sparse training is a natural idea to accelerate the training speed of deep neural net- works and save the memory usage, especially since large modern neural networks are significantly wer-pprameterrized However, most of the existing methods cannot achieve this goal practice because the chain rule based gradient (w.r.f. structure parameters) estimators adopted by previous methods require dense computation least in the backward propagation step. This paper solves this problem by proposing an efficient sparse training method with completely sparse forward and backward passes. We first formulate the training process as a continuous minimiza- tion problem under global sparsity constraint. We then separate the optimization process into two steps, corresponding to weight update and structure parameter update. For the former step, we use the conventional chain rule, which can be sparse via exploiting the sparse structure. For the latter step, instead of using the chain rule based gradient estimators as in existing methods, propose variance reduced policy gradient estimator, which only requires two forward passes without backward propagation, thus achieving completely sparse training. We prove that the variance of our gradient estimator is bounded. Extensive experimental results worldworld datasets demonstrate that compared to previous methods, algo- rithm is much more effective in accelerating the training process, up to an order magnitude faster.
Introduction
In the last decade, deep neural networks (DNNs) [35, 11, 38] have proved their outstanding per- formance in various fields such as computer vision and natural language processing. However, training such large-sized networks is still very challenging, requiring huge computational power and storage. This hinders us from exploring larger networks, which are likely to have better performance. Moreover, it is widely-recognice parameterized, which means that a fully trained network can always be sparsified dramatically by network pruning techniques [9, 8, 25, 46, 20] into small sub-network with negligible degradation in accuracy. After pruning, the inference efficiency can be greatly improved. Therefore, a natural question is
property that modern neural networks are significantly over-
exploit this sparsity to improve the training efficiency?
The emerging technique called sparse network training [10] is closely related with our question, which can obtain sparse networks by training from scratch. We can divide existing methods into two categories, i.e., parametric and non-parametric, based on whether they explicitly parameterize network structures with trainable variables (termed structure parameters). Empirical results [24, 34, 44, 23] demonstrate that the sparse networks they obtain have comparable accuracy with those
"quual contribution
t Jointly with Google Rescarch
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

obtained from network pruning. However, most of them narrowly aim at finding sparse subnetwork instead of simultaneously sparsifying the computation of training by exploiting the sparse structure. As a consequence, it is hard for them to effectively accelerate the training process in practice on general platforms, e.g., Tensorflow [1] and Pytorch [31]. Detailed reasons are discussed below:
Non-parametric methods find the sparse network by repeating a two-stage procedure that alternates between weight optimization and pruning [10, 6]. or by adding proper sparsity- inducing regularizer on the weights to the objective [22, 41J. The two-stage methods prune the networks in weight space and usually require retraining the obtained subnetwork from scratch every time when new weights are pruned, which makes training process even more time-consuming. Moreover, the computation of regularized methods is dense since the gradients of a zero-valued weights/filters are still nonzero.
All the parametric approaches estimate the gradients based on chain rule. The gradient w.r.t. the structure parameters can be nonzero even when the corresponding channel/weight is pruned. Thus, to calculate the gradient via backward propagation, the error has to be propagated through all the neurons/channels. This means that the computation of backward propagation has be dense. Concrete analysis can be found in Section 3.
We notice that some existing methods [4, 28] can achieve training speedup by careful implementation. For example, the dense to sparse algorithm [28] removes some channels if the corresponding weights are quite small for a long time. However, these methods always need work with large model at the beginning epochs and consume huge memory and heavy computation in the early stage. Therefore, even with such careful implementations, the speedups they can achieve are still limited.
In this paper, we propose an efficient channel-level parametric sparse neural network training method, which is comprised of completely sparse (See Remark 1) forward and backward propagation. We adopt channel-level sparsity since such sparsity can be efficiently implemented on the current training platforms to save the computational cost. In method, first parameterize the network structure by associating each filter with a binary mask modeled as an independent Bernoulli random variable, which can be continuously parameterized by the probability. Next, inspired by the recent work [47], we globally control the network size during the whole training process by controlling the sum of the Bernoulli distribution parameters. Thus, we can formulate the sparse network training problem into a constrained minimization problem on both the weights and structure parameters (i.e., the probability). The main novelty and contribution of this paper lies in our efficient training method called completely sparse neural network training for solving the minimization problem. Specifically, to fully exploit the sparse structure, we separate training iteration into two parts, i.e., weight update and structure parameter update. For weight update, the conventional backward propagation is used to calculate the gradient, which can be sparsified completely because the gradients of the filters with zero valued masks are also zero. For structure parameter update, we develop a new variance reduced policy gradient estimator (VR-PGE). Unlike the conventional chain rule based gradient estimators (e.g., straight through[2). VR-PGE estimates the gradient via two forward propagations, which is completely sparse because of the sparse subnetwork. Finally, extensive empirical results demonstrate that our method can significantly accelerate the training process of neural networks.
The main contributions of this paper can be summarized as follows:
We develop an efficient sparse neural network training algorithm with the following three appealing features:
In our algorithm, the computation in both forward and backward propagations is completely sparse, i.e., they do not need to go through any pruned channels, making the computational complexity significantly lower than that in standard training.
During the whole training procedure, our algorithm works on small sub-networks with the targel sparsity instead of follows a dense-to- tosree scheme.
-Our algorithm can be implemented easily on widely-used platforms, e.g., Pytorch and Tensorflow, to achieve practical speedup.
We develop a variance reduced policy gradient estimator VR-PGE specifically for sparse neural network training. and prove that its variance bounded.
Experimental results demonstrate that our methods can achieve significant speed-up in training sparse neural networks. This implies that our method can enable us to explore larger-sized neural networks in the future.

---

Remark 1. We call sparse training algorithm completely sparse if both its forward and backward propagation do not need to go through any pruned channels. For such algorithms, the computational cost in forward and backward propagation cost can be roughly reduced to p² 100%, with being the ratio of remaining unpruned channels.
2 Related Work
In this section, we briefly review the studies on neural network pruning, which refers the algorithms that prune DNNs after fully trained, and the recent works sparse neural network training.
2.1 Neural Network Pruning
Network Pruning [10] is a promising technique for reducing the model size and inference time of DNNs. The key idea of existing methods [10, 8, 46, 20, 27, 13, 48, 40, 43, 32, 16] is to develop effective criteria (c.g. weight magnitude) to identify and remove the massive unimportant weights contained in networks after training. To achieve practical speedup on general devices, some of them prune networks in structured manner, i.e., remove the weights a certain group (e.g., filter) together, while others prune the weights individually. It has been reported in the literature [8, 25, 46, 20J that they can improve inference efficiency and reduce memory usage of DNNs by orders of magnitudes with minor loss in accuracy, which enables the deployment of DNNs on low-power devices.
We notice that although some pruning methods can be easily extended to train sparse networks, they cannot accelerate or could even slow down the training process. One reason is they are developed in the scenario that a fully trained dense network is given, and cannot work well on the models learned in the early stage of training. Another reason is after each pruning iteration, one has to fine tune or even retrain the network for lots of epoch to compensate the caused accuracy degradation.
2.2 Sparse Neural Network Training
The research sparse neural network training has emerged in the recent years. Different from the pruning methods, they can find sparse networks without pre-training a dense one. Existing works can be divided into four categories based on their granularity in pruning and whether the network structures are explicitly parameterized. To the best of our knowledge, no significant training speedups achieved in practice are reported in the literature. Table summarizes
representative works.
Table 1: Some representative works in sparse neural network training.
granularity
non canmparaettric
parametric
weight-level
[S. 6, 48, 22, 18, 29, 39, 30, 4]
[42, 37, 26, 47, 18]
channel-level
[41, 12]
[19, 24, 44, 26. 16]
Weight-level non-parametric methods, e.g., [6, 10, 48, 29, 30), always adopt two-stage training procedure that alternates between weight optimization and pruning. They differ in the schedules of tuning the prune ratio over training and layers. [10] prunes the weights with the magnitude below certain threshold and [48, 6] gradually increase the pruning rate during training. [30, 5] automatically reallocate parameters across layers during training via controlling the global sparsity.
Channel-level non- con-parametric methods [12, 41] are proposed to achieve a practical acceleration in inference. [41] is a structured sparse learning method, which adds group Lasso regularization into the objective function of DNNs with each group comprised of the weights a filter. [12] proposes soft filter pruning method. It zeroizes instead of hard pruning the filters with small & norm, after which these filters are treated the same with other filters in training. It obvious that these methods cannot achieve significant speedup in training since they need to calculate the full gradient in backward propagation although the forward propagation could be sparsified if implemented carefully.
Parametric methods multiply each weight/channel with binary [47, 44, 37, 42] or continuous [24, 26, 19, 18] mask, which can be either deterministic [24, 42] or stochastie [47, 44, 26, 37, 19, 18). The mask is always parameterized via continuous trainable variable, i.c., structure parameter. The
3

---

X2
X3
m my # 8,
W.2. my
W... my
8g w Nin
Xin,1
Xin2
Xin.3
Figure 1: A fully connected network. w is the weight matrix of 1st layer, my is the mask of i-th neuron; y. Pin and i;; are the output, input and preactivation. The 3rd neuron (in grey) is pruned.
sparsity achieved by adding -anduciy-nnuuiing regularizers on the masks. The novelties these methods lie in estimating the gradients w.r. structure parameters in training. To be precise,
Deterministic Binary Mask. [42] parameterizes its deterministic binary mask as a simple step function and estimates the gradients via sigmoid straight through estimator (STE) [2].
Deterministic Continuous Mask. [24] uses the linear coefficients of batch normalization (BN) continuous mask and enforces most of them to 0 by penalizing the objective with L norm of the coefficients. [18] defines the mask as soft threshold function with Icarnable threshold. These methods can estimate the gradients via standard backward propagation.
bernoulli random variable and the gradients w.r.. the parameters bernoulli distributions are estimated via STE. [47] estimates the gradients via Gumbel-Softmax trick [15], which is more accurate than STE.
Stochastic Binary Mask. [44, 37] model the mask
Stochastic Continuous Mask. [26. 19] parameterize the mask as a continuous function g(e, e), which is differentiable w.r.t. c, and parameter free noise, c.g., Gaussian noise (0,0 1). In this way, the gradients can be calcuated via conventional backward propagation.
Therefore, we can see that all of these parametric methods estimate the gradients of the structure parameters based on the chain rule in backward propagation. This makes the training iteration cannot be sparsified by exploiting the sparse network structure. For the details, please refer Section 3.
3 Why Existing Parameteric Methods Cannot Achieve Practical Speedup?
In this section, we reformulate existing parametric channel-level methods into a unified framework to explain why they cannot accelerate the training process practice.
Notice that convolutional layer can be viewed as a generalized fully connected layer, i.e., viewing the channels neurons and convolution of two matrices generalized multiplication (see [7]). Hence, for simplicity, we consider the fully connected network in Figure 1. Moreover, since the channels in CNNs are corresponding the neurons in fully connected networks, consider neuron-level instead of weight-level sparse training in our example.
As discussed in Section 2, existing methods parameterize the 4 kinds of mask in the following ways:
(i): m, d(s)); (ii): m, (s:);; (ii): m4 g(ss,,)) \O 1); (iv): (er)(: where the function d(s;) is binary, e.g., step function; (s;;) is a continuous function; g(s,, e) is differentiable w.r.t. s,. All existing methods estimate the gradient of the loss E(j. y) w.r.t. 8; based on chain rule, which can be formulated into a unified form below.
Specifically, we take the pruned neuron in Figure as an example, the gradient is calculated as
al(s. y)
0mg
ay ((j.(()) v)
(w..
Dig
0s3
forward
Existing parametric methods developed different ways to estimate 0m3 Actually, for cases (ii) and Uss
(iii), the gradients are well--efined and thus can be calculated directly. STE used to estimate the gradient in case (i) [42]. For cases (iv), [44, 37, 47] adopt STE and Gumbel-Softmax.

---

In (qn,((), the term (a) is always nonzero especially when .b is followed by BN. Hence, we can see that even for the pruned neuron the gradient tms dsa can be nonzero in all four cases. This means the backward propagation has to go though all the neurons/channels, leading to dense computation.
At last, we can know from (qn.(l) that forward propagation in existing methods cannot be completely sparse. Although w. " can be computed sparsely as in general models Zin could be sparse tensor of a layer with some channels being pruned, need to calculate it for each neuron via forward propagation to calculate RHS of (qn.(1). Thus, even if carcfully implemented, the computational cost of forward propagation can only be reduced to p 100% instead of p² 100% in inference.
That's why we argue that existing methods need dense computation at least in backward propagation. So they cannot speed up the training process effectively in practice.
Remark 2. The authors of GrowEfficient [44] confirmed that actually they also calculated the gradient of qe w.E., sg in their (6n via STE even if q. = 0. Thus need dense backward propagation.
4 Channel-level Completely Sparse Neural Network Training
Below, we present our sparse neural network training framework and the efficient training algorithm.
4.1 Framework of Channel-level Sparse Training
Given a convolutional network [æ; w)) let (Fe c E C be the set of filters with C being the set of indices of all the channels. To parameterize the network structure, we associate each Fe with a binary mask me which is an independent Bernoulli random variable. Thus, each channel is computed as
Cout. Win (Feme),
with being the convolution operation. Inspired by [47], avoid the problems, e.g., gradient vanishing, we parameterize m. directly on the probability se, i.c., me equals to and 0 with the probabilities se and 1 se. respectively. Thus, we can control the channel size by the sum of se- Following [47], we can formulate channel-leve sparse network training into the following framework:
N
min Epomle) L(w, m) = 88,8
1
Σ
( (x1; w, m)) ys)
s.t. RE.
[0, 9 : k),
where (x) is the training dataset, w is the weights of the original network, f (: is the pruned network, and :( is the loss function, e.g, cross entropy loss. K = p|C| controls the remaining channel size with p being the remaining ratio of the channels.
Discussion. We'd like to point out that although our framework is inspired by [47], our main contribution is the efficient solver comprised of completely sparse forward/backward propagation for Problem (2). Moreover, our framework can prune the weights in fully connected layers together, since we can associate each weight with independent mask.
4.2 Completely Sparse Training with Variance Reduced Policy Gradient
Now we present our completely sparse training method, which can solve Problem (2) via completely sparse forward and backward propagation. The key idea is to separate the training iteration into filter update and structure parameter update so that the sparsity can be fully exploited.
4.2.1 Filter Update via Completely Sparse Computation
It is easy to see that the computation of the gradient w.r.t the filters can be sparsified completely. To prove this point, we just need clarify the following two things:
We do not need to update the filters corresponding to the pruned channels. Consider a pruned channel i.e., m, 0, then due to the chain rule, we can have
de (f (x,; w. m)) 0e (f (x;; w, m)) Woout.c 0, DF.
0F.
Dxout,
5

---

the last equation holds since Dout.c O. This indicates that the gradient w.r.t the pruned filter Fe always O, and thus Fa does not need to be updated.
The error cannot pass the pruned channels via backward propagation. Consider a pruned channel we denote its output before masking as Dout,c = Iin Fe, then the error propagating through this channel can be computed as
de (f (x;; w, m)) ae (f (x;W,, w. m)) Ooout. 0.
Oootte
Dxotte
Dout.c
This demonstrates that to calculate the gradient w.r.t. the unpruned filters, the backward propagation does not need to go through any pruned channels.
Therefore, the filters can be updated via completely sparse backward propagation.
4.2.2 Structure Parameter Update via Variance Reduced Policy Gradient
We notice that policy gradient estimator (PGE) can estimate the gradient via forward propagation, avoiding the pathology of chain rule based estimators as dicussed in Section 3. For abbreviation, we denote L(w, m) as (mm)) since can be viewed as constant here. The objective can be written
+ Ep(mi is) ((m),
which can be optimized using gradient descent:
- s - -
with learning rate One can obtain a stochastic unbiased estimate of the gradient ( using PGE:
( = Ep(m|s) [(m) In p(m|ss,
(PGE)
leading to Policy Gradient method, which may be regarded as a stochastic gradient descent algorithm:
- r/((m))VV (mm(m)).
In (3), ((m) can be computed via completely sparse forward propagation and the computational cost of V. In pmm]ssm= = m x(1=a) is negligible, therefore PGE is computationally efficient.
However, in accordance with the empirical results reported in [33, 15]. we found that standard PGE suffers from high variance and does not work practice. Below we will develop Variance Reduced Policy Gradient Estimator (VR-PGE) starting from theoretically analyzing the variance of PGE.
Firstly, we know that this variance of PGE is
Ep(mls) L (3(m))) In pmm(mm)))l (
which can be large because C(m) is large.
Mean Field theory [36] indicates that, while [(m) can be large, the term [(m) - [(m') is small when and m' two independent masks sampled from same distributionpí m s) (see the appendix for the details). This means that we may consider the following variance reduced precondi- tioned policy gradient estimator:
Em'' Bmm''=p(m'le Em (mm((m)))) (C(m) [(mm)) (s)V lnp(m(s).
(VR-PGE)
where (8) is specific diagonal preconditioning matrix
Ha diag (1=8))²
(4)
with a E (0, 1) and being the element- product. It plays role as adaptive step size and is shown that this term can reduce the variance of the stochastic PGE term V. In (mmpmss)) The details can be found in the appendix. Thus (ss) can be optimized via:
ト 17 (C (m) L (m')) ( (s) Inp(m]s).
In our experiments, we set a to ! for our estimator VR-PGE, The theorem below demonstrates that VR-PGE can have bounded variance.
z

---

Algorithm 1 Completely Sparse Neural Network Training
Input: target remaining ratio p. dense network w. the step size 17, and parameter in (4).
1: Initialize w, let = =pl
2: for training epoch 1 = 1, ......T7 do
3: for each training iteration do
4: Sample mini batch of data B ((x1, (xn,
5: Sample m9 from p(mls)), 1,2.
6: Update s and up
(2) with 8 (Ls(Ru,, (Cs(,,,mm))) Ls(w, m (2))) Ha(s) m m u f- V wB mw)
a(1-a))
7: end for
8: end for
9: return A pruned network w o m by sampling a mask m from the distribution p(mlss)
Theorem 1. Suppose and m' are two independent masks sampled from the Bernoulli distribution p(ml)), then for any [ 1) and 8 E (0, 1)(c1, the variance is bounded for
(((m))_(mm) H (s) pmmlm)))
Finally, we provide complete view of our sparse training algorithm in Algorithm 1, which is essentially a projected stochastic gradient descent equipped with our efficient gradient estimators above. The projection operator in Algorithm 1 can be computed efficiently using Theorem 1 of [47].
Discussion. In our algorithm, benefited from our constraint on 8, the channel size of the neural network during training can be strietly controlled. This is in contrast with GrowEfficient [44], which ultilizes regularizer term to control the model size and has situations where model size largely drift away from desired. This will have larger demand for the GPU memory storage and have more risk that memory usage may explode, especially when we utilize sparse learning to explore larger models. Moreover, our forward and backward propagations are completely sparse, i.e., they do not need to go through any pruned channels. Therefore, the computational cost of each training iteration can be roughly reduced p² 100% of the dense network.
5 Experiments
In this section, we conduct series of experiments to demonstrate the outstanding performance of our method. We divide the experiments into five parts. In part one, we compare our method with several state-of-the-art methods on CIFAR-IO [17] using VGG-16 [35], ResNet-20 [11] and WideResNet-28.
[45] to directly showcase the superiority of our method. In part two, we directly compare with sate-of-theearr method GrowEfficient [44] especially extremely sparse regions, and on two high capacity networks VGG- 19 [35] and ResNet- [11] on CIFAR-10/100 [17]. In part three, conduct experiments large- dataset ImageNet [3] with Re [11] and MobileNetV1 [14] and compare with Erowiefieeent [44] across a wide sparsity region. In part four, present the crantccomputationll time as supplementary to justify the applicability of sparse training method into practice. In part five, we present further analysis epoch-wise train-cost dynamics and experimental justification of variance reduction of VR-PGE. Due to the space limitation, we postpone the experimental configurations, calculation schemes train-cost savings and train-computational time and additional experiments into appendix.
the conceptual train-cost savings
5.1 VGG-16, ResNet-20 and WideResNNet-2-- 10 on CIFAR-10
Table 2 presents T validation accuracy, parameters, FLOPs and train-cost savings comparisons with channel pruning methods L LI-Pruning [20]. SoftNet [12], ThiNet [27], Provable [21] and sparse training method GrowEfficient [44]. SoftNet can train from scratch but requires completely dense computation. Other pruning methods all require pretraining of dense model and multiple rounds of pruning and finetuning, which makes them slower than vanilla dense model training. Therefore the train-cost savings of these methods are below 1x and thus shown as ("-") in Table 2.
7

---

Table 2: Comparison with the channel pruning methods L1 LI-Pruning [20], SoftNet [12], ThiNet [27],
Provable [21] and one channel sparse training method GrowEfficient [44] on CIFAR-IO.
Model
Method
Val Acc(%)
Params(98) %6)
FLOPs(%)
Train-Cost Savings(X)
Original
92.9
100
100
1x
L.1-Pruning
91.8
19.9
19.9
SoftNet
92.1
36.0
36.1
VGG-16
ThiNet
90.8
36.0
36.1
Provable
92.4
5.7
15.0
Grow GrowEfficient
92.5
5.0
13.6
1.22x
Ours
92.5
4,4
8.7
x
Original
91.3
100
100
1x
L1-Pruning
90.9
55.6
55,4
SoftNet
90.8
53.6
50.6
ResNet-20
ThiNet
89.2
67.1
67.3
Provable
90.8
37.3
54.5
GrowEfficient
90.91
35,8
50,2
1.13x
Ours
90.93
35.1
36.1
2.09
Original
96.2
100
100
1x
LI-Pruning
95.2
20.8
49.5
WRN-28-10
Growenficient
95.3
9.3
28.3
1.17x
Ours
95.6
8.4
7.9
9.39
Restitt: 32 on CIFAR-10
RosNet-32 on CIFAR-100
VGG-19 CIFAR-10
VGG on CIFAR1000
10
50
40
20
20
25
7L0%% 750
FLO7% 1S11
PLOY' (5)
1L00% (50)
Grontflica ATd
Figure Comparison of Top-1 Validation Accuracy and Train-cost Savings CIFAR-10/100.
GrowEfficient [44] is a recently proposed state-of-the-art channel-level sparse training method showing train-cost savings compared with dense training, As described in Section 3, GrowEfficient features completely dense backward and partially sparse forward pass, making train-cost saving limited by - By contrast, the train-cost savings of our method is not limited by any constraint. The details of how train-cost savings are computed can be found in appendix.
Table 2 shows that our method generally exhibits better performance in terms of validation accuracy. parameters and particularly FLOPPs terms of train-cost savings, our method shows least 1.85 speed-up against GrowEfficient [44] and up to 9.39x speed-up against dense training.
5.2 Wider Range of Sparsity on CIFAR-10/100 on VGG-19 and ResNet-32
In this section, we explore sparser regions of training efficiency to present a broader comparision with state-of-the-art channel sparse training method GrowEfficient [44].
plot eight figures demonstrating the relationships between the Top-1 validation accuracy, FLOPPs and train-cos' savings. We find that method generally achieves higher accuracy under same FLOPs

---

Table 3: Comparison with the channel pruning methods L1- L1-Pruning [20], SoftNet [12], Provable [21]
and one channel sparse training method GrowEfficient [44] on ImageNet-IK.
Model
Method
Val Acc(%)
Params(58) %))
FLOPs(%)
Train-Cost Savings( x)
Original
77.0
100
100
1x
1.1-Pruning
74.7
85.2
77.5
SoftNet
74.6
58.2
ResNet-50
Provable
75.2
65.9
70.0
GrowEfficient
75.2
61.2
50.3
1.10x
Ours
76.0
48.2
46.8
1.60x
Ours
73.5
27.0
24.7
3.02x
Ours
69.3
10.8
10.1
7.36x
MobileNetVV
MobileNetV]
Epoch-wise Train-cost
Variance Comparison
3.5
0.0
60
Accurscy 780
3.0
0.6
40
2.5
CSE
2.0
0.4
20
1.5
30"4
0.2
40
20
50
1.00
FLOPS (96)
FLOPs 195)
Epoch
Layer Indox
GrooFificent
Ours
VR-PEF
Figure 3: Top-1 Validation Accuracy and Train-cost Savings on MobileNetV1 on ImageNet. Epoch- wise Train-cost and Variance Comparison VGG-19 on CIFAR-I0.
settings. To be noted, the train-cost savings of our method is drastically higher than GrowEfficient [44], reaching up to 58.8x when sparisty approches 1.56% on ResNet-32 on CIFAR-100, while the speed-up of GrowEfficient is limited by
5.3 ResNet-50 and MobileNetV1 on ImageNet-IK
In this section, we present the performance boost obtained by our method on ResNet-50 and Mo- bileNetV1 ImageNet- 1K [3]. Our method searches model with 76.0% Top-1 accuracy, 48.2 48.2% parameters and 46.8% FLOPs beating all compared state-of-the-ar methods. The train-cost saving comes up 1.60x and methods. Therefore we give a harder limit to the channel size and present sparser results on the same Table 3, reaching up 7.36x speed-up while still preserving 69.3% Top-1 accuracy. For the already compact model MobileNetV1, 1, we plot two figures in Figure comparing with Grow Efficient [44]. We find that our method is much stabler in sparse regions and obtains much higher train-cost savings.
prominent due to the accuracy constraint match up with compared
5.4 Actual Training Computational Time Testing
In this section, we provide actual training computational time on VGG-19 and CIFAR-10. The GPU in test is RTX 2080 Ti and the deep learning framework is Pytorch [31]. The intent of this section is to justify the feasibility of our method in reducing actual computational time cost, rather than staying in conceptual training FLOPs reduction. The computational time cost is measured by wall clock time, focusing on forward and backward propagation. We present training computational time in Table 4 with varying sparsity as Figure 2. It shows that the computational time savings increases steadily with the sparisty. We also notice the gap between the savings in FLOPS and computational time. The gap comes from the difference between FLOPs and actual forward/backward time. More specifically, forward/backwan time is slowed down by data-loading processes and generally affected by hardware latency and throughput, network architecture, etc. extremely sparse regions, the pure computational time of sparse networks only occupies little of the forward/backwwrd time and the cost of data management and hardware latency dominates the wall-clock time. Despite this gap, it can be expected that train-cost savings can be better translated into real speed-up in exploring large

---

models where the pure computational time dominates the forward/backwwrd time, which promises bright future for making training infeasibly large models into practice.
Table "raittcomputaational Time on VGG-19 with CIFAR-10. The computational time saving is not as prominent as train-cost savings while still achieving nearly an order of reduction, preserving 87.97% accuracy.
Model
Val Ac)(
Params %)
FL)Ps(%)
Train- Cost Savings( x)
Train- Soiin-Comuutaiional Time(min)
93.84
100.00
100.00
1.00x
21.85 (1.00x)
93,46
23.71
28.57
2.64
14.04 (1.55x)
93.11
12.75
19.33
3.89
10.43 (2.09x)
VGG-19
92.23
6.69
10.27
7.30x
6.83 (3.20x)
90.82
3.06
4.94
15.28x
4,86 (4,50x)
87.97
0.80
1.70
44.68x
2.95 (7.41x)
5.5 Further Analysis
[Epoch-wise Train-cost Dynamics of Sparse Training Process] We plot the train-cost dynamics Figure The vertical label the ratio of train-cost dense training, the inverse of train-cost savings. This demonstrates huge difference between our method and GrowEfficient [44]. The model searched by our method exhibits 92.73% Top-1 accuracy, 16.68% parameters, 14.28% FLOPs with 5.28 train-cost savings, while the model searched by GrowEfficient exhibits 92.47% Top-1 accuracy, 18. 18.08% parameters, 22.74% FLOPs with 1.21 X train-cost savings.
[Experimental Verification of Variance Reduction of VR-PGE against PGE] We plot the mean of variance of gradients of channels from different layers. The model checkpoint and input data selected randomly. The gradients are calculated in two approaches, VR-PGE and PGE. From the rightmost graph of Figure find that the VR-PGE reduces variance significantly, up to orders of magnitude.
Conclusion
This paper proposes an efficient sparse neural network training method with completely sparse forward and backward passes. A novel gradient estimator named VR-PGE is developed for updating structure parameters, which estimates the gradient via two sparse forward propagation. We theoretically proved that VR-PGE has bounded variance. In this way, in training and making the whole training process completely sparse. Emprical results demonstrate that the proposed method can significantly accelerate the training process of DNNs in practice. This enables us to explore sarger-sized neural networks in the future.
separate the weight and structure update
Acknowledgments and Disclosure of Funding
This work is supported by GRF 16201320.
10

---

References
[1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorflow: A system for large-scale machine lcarning. In 12th {USENIX) symposium on operating systems design and implementation ((OSDI) 16), pages 265-283, 2016.
[2] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 130.::33308.3333 2013.
[3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee. 2009.
[4] T. Dettmers and L. Zettlemooyr. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 1907, 2019.
[5] U. Evei, T. Gale, J. Menick, S. Castro, and F. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.
[6] J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611 2019.
[7] Y. Gu, W. Zhang, C. Fang. J. D. Lee, and T. Zhang. How characterize the landscape of overparameterized convolutional neural networks. In Advances Neural Information Processing Systems, volume 33, pages 3797-3807, 2020.
[8] Y. Guo, A. Yao, and Y. Chen. Dynamic network surgery for efficient dans. In Advances neural information processing systems, pages 1379-1387, 2016.
[9] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. International Conference on Learning Representations, 2016.
[10] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1135- 1143, 2015.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 770-778, 2016.
[12] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. Soft filter pruning for accelerating deep convolutional neural networks. In IJCAI International Joint Conference on Artificial Intelligence, 2018
[13] Y. Hc, X. Zhang, and J. Sun. Channel pruming for accelerating very deep neural networks. In Proceedings of the IEEE International Conference Computer Vision, pages 3389-1397, 2017.
[14] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04661 2017.
[15] E. Jang, S. Gu. and B. Poole. Categorical reparameterization with gumbel-softmat International Conference on Learning Representations, 2017.
[16] M. Kang and B. Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pages 5122-5131. PMLR, 2020.
[17] A. Krizhevsky. Learning multiple layers of features from tiny images. Master's 's thesis, University of Trons, 2009.
[18] A. Kusupati, V. Ramanujan, R. Somani, M. Wortsman, P. Jain, S. Kakade, and A. Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning, July 2020.
[19] C. Lemaire, A. Achkar, and P.-M. Jodoin. Structured pruning of neural networks with budget-aware regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 9108-9116 16, 2019.
[20] H. Li, A. Kaday, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. International Conference on Learning Representations, 2017.
[21] L. Liebenwein, C. Baykal, H. Lang, D. Feldman, and D. Rus. Provable filter pruning for efficient neural networks. In International Conference Learning Representations, 2019.
11

---

[22] B. Liu, M. Wang, H. Foroosh, M. Tappen. and M. Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 806-814, 2015.
[23] S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, 7. Wang, and D. C. Mocanu. Sparse training via boosting pruning plasticity with neuroregenerattio. arXiv preprint arXiv:2106. 10404, 2021.
[24] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pages 2736 2744, 2017.
[25] Z. Liu, M. Sun, Zhou, Huang, and Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
[26] C. Louizos, M. Welling, and D. Kingma. Learning sparse neural networks through 1_0 regularization. In International Conference on Learning Representations, 2018.
[27] J.-H. Luo, J. Wu, and W. Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058-5066, 2017.
[28] S. Lym, E. Choukse, S. Zangench, W. Wen, Sanghavi, and M. Frez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-13, 2019.
[29] D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1 12, 2018.
[30] H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646-4655. PMLR, 2019.
[31] A. Paszke, S. Gross, F. Massa, Lerer. J. Bradbury, G. Chanan. T. Killeen, Z. Lin, N. Gimelshein. L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8880 8037, 2019.
[32] A. Renda, J. Frankle, and M. Carbin, Comparing rewinding and fine-tuning neural network pruning. In International Conference on Learning Representations. 2019.
[33] D. J. Rezende, Mohamed, and Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[34] P. Savarese, H. Silva, and M. Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 33, 2020.
[35] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
[36] M. Song, A. Montanari, and P. Nguyen. A mean field view of the landscape of two-layers neural networks. Proceedings of the National Academy of Sciences, 75:E7665-E7671, 2018.
[37] S. Srinivas, A. Subramanya, and Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 138-145, 2017.
[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkorcit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[39] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
[40] Z. Wang, J. Wohlwend, and T. Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6151-666 6662, 2020.
[41] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pages 2074-2082, 2016.
[42] X. Xiao, Z. Wang, and S. Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. In Advances in Neural Information Processing Systems, pages 13681-13691, 2019.
12

---

[43] M. Ye, C. Gong, L. Nie. D. Zhou. A. Klivans, and Q. Liu. Good subnetworks provably exist: Pruning via greedy forward selection, In International Conference Machine Learning, pages 10820-10830. PMLR. 2020.
[44] X. Yuan, H. P. Savarese, and M. Maire. Growing efficient deep networks by structured continuous sparsification. International Conference on Learning Representations. 2020.
[45] S. Zagoruyko and N. Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[46] W. Zeng and Urtasun. MLPrune: Multi-layer pruning for automated neural network compression, 2019.
[47] X. Zhou, W. Zhang, H. Xu, and T. Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3599 3608, 2021.
[48] M. Zhu and S. Gupta. To prune, prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01810 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] This is
a sparse training algorithm aiming to speed up deep neural network training process,
with potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "kaH76vIlf2G", "reviewer": "Reviewer_X59J", "summary": "Network sparsity has historically targeted only inference.  Recent explorations in training with sparsity fall short of their potential, though, by not fully exploiting sparsity in the forward and backward propagation steps.  The authors decompose training a channel-wise sparse network into two parts: training the weights themselves (as usual) and learning the sparse structure.  Rather than using a chain-rule based gradient step for the structure, a unique policy gradient estimator with reduced variance is used; it is composed of two forward passes, eliminating the need for the chain rule to be used to learn the sparse structure.  Experiments show competitive accuracy for saved parameters and FLOPs for a number of networks, as well as actual time saved during training with the proposed technique for VGG19 on CIFAR-10.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\n\nThere's been lots of work in sparse training.  The authors present a nice survey in Section 2.2, but they've missed a number of works which do exploit sparsity in backpropagation [1-4].  It's likely that the recent works [3,4] could be trivially modified to support channel-pruning, rather than fine-grained sparsity.  What may be new here, and what the authors should clarify, is that the proposed approach is more theoretically grounded, without relying on heuristics to choose sparsity that isn't consistent between the forward and backwards passes.  Note: I'm not positive about this point; the burden remains on the authors to argue this convincingly!\n\nI'm not familiar with any other work on reducing the variance of policy gradient estimation, though, so at least this contribution was new to me.\n\n# Quality\nAside from the missing references that cast doubt on the claims of novelty, the experiments show that the technique does what it intends to do: variance of the gradients are reduced compared to the baseline PGE, accuracy is plausibly maintained as the number of nonzero parameters are reduced, and training time is reduced in practice.  In particular, reducing the training cost of ResNet50 on ImageNet1K by 1.6x is no small feat!  (In practice, this will be greatly reduced, of course.)\n\nRemark 2 is potentially misleading.  It suggests that the authors of GrowEfficient used a dense backprop procedure, even for zero parameters.  Is this just for simplicity of implementation, though?  If so, it's no cause for penalization, as computing with zeros is a common way of testing an algorithm; exploiting the zeros often follows.\n\nThere's no discussion of hyperparameters p and a, how sensitive results are to their values, and suggestions on how to select them.  A step in the right direction would be to detail (in the main text) the settings used for the experimental results.\n\n# Clarity\nI found some parts of the manuscript confusing and hard to follow.\n- Section 3 uses a fully-connected layer for simplicity (and it's understood that a convolution is a particular type of FC layer), but then Section 4 immediately switches to discussing in the context of convolutions.  Why the change?  Sticking with FC layers would help ease understanding.\n- Line 122: \"It is obvious that these methods cannot achieve significant speedup in training since they need to calculate the full gradient in backward propagation.\"  It's not obvious to me!  It sounds like the argument is that since the filters are not actually zero, neither are their gradients.  In the same sentence, though, it is said that \"forward propagation could be sparsified if implemented carefully.\"  If these not-quite-zero values can be exploited for fprop, why couldn't the same happen in bprop?\n- Similarly, the difference that gives true sparsity in the proposed method's backprop step, where prior efforts could not realize this sparsity, is not clear to me.  The authors suggest that since the output of the pruned channel is zero (Line 196), backprop can be sparse.  Is this not also the case in equation 1?  The term marked \"forward\" is zero for the pruned neuron, so the output should similarly be zero.\n\nIt'd be helpful if all equations were numbered for easy reference.\n\nThere were also a number of typos; another editing pass would be helpful to fix these (and, likely, others that I missed):\n- Line 70: \"… zero valued mask*s*\"\n- Line 74: \"propagation*S*\"\n- Line 121: \"filers\" -> \"filters\"\n- (Maybe not a typo, but a style thing) Figure 1's caption uses \"3-rd,\" when \"3rd\" is more common, or, in this instance, \"third\" would also be appropriate.\n- Algorithm 1, line 1: \"p1\" (is this supposed to be simply \"p\"?)\n- Section 5.5 mentions Top-1 accuracies for MobileNetv2 on ImageNet, but I think these numbers (90%+) are for Top-5.  (Why switch between Top-1 and Top-5 for Figure 3's results and discussion?)\n\n# Significance\nSaving training time while still generating a network that maintains accuracy is an important topic.  This work is no magic bullet, though, because accuracy doesn't quite match the dense baseline accuracy, even for the more modest savings.  However, it does seem much more promising than the work to which it is compared, in both accuracy vs. parameters and achieved training time savings, so it may be a foundation on which future work improves.  However, there are some baselines missing both qualitative and quantitative comparisons, so the importance of this work may be reduced with the added context.\n\n---\n\n[1] \"meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting,\" Sun et al.,  ICML 2017\n\n[2] \"Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training,\" Zhu et al., https://arxiv.org/abs/1806.00512\n\n[3] \"Sparse Weight Activation Training,\" Raihan and Aamodt, NeurIPS 2020\n\n[4] \"Top-KAST: Top-K Always Sparse Training,\" Jayakumar et al., NeurIPS 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There's no discussion of limitations in this work.  For example: \n- Can the proposed technique be applied to transient values, such as activations or attention layers (which have no learnable parameters), to reduce training (and inference) costs?  In recent thrusts for many domains, including image classification, such approaches are proving to be very successful.\n- Are there limitations on optimization schemes that will and will not work with the proposed technique?\n\nI'm sure the authors are better able to identify the limitations of their work than I am.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ho1B57bgdZ7", "reviewer": "Reviewer_ozBF", "summary": "Paper proposes a variational based pruning method where, with 2 structurally sparse forward passes and 1 sparse backward, the training can be significantly faster. Description is clear and theoretically sounds, equations are easy to follow, code is provided, results are encouraging. The paper might be a great benchmark and starting point for faster pruning-while-training with efficient inference. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "References need more work. For example many papers do not have the venue(journal) where it was published. \n\n# On sparse neural network training\n\nAuthors provide an overview of existing methods. However almost all structured pruning algorithms can be applied after few echoes, dense model obtained by removing channels, and then the rest of the training will continue with the small model. This approach will speed training as well. Are authors aware of such methods? For example [R1-R2]. These methods finish pruning with only 30% of training done, making the last 70% to be truly sparse.  \nClaiming to be the first in accelerating training via pruning might be challenged looking at previous work.  \n\n# Approach\n\nThe method theoretically sound. Equations are clear to be followed, Algorithm 1 explains the implementation clearly. Explanation and theory behind VR-PGE is interesting and clearly explained to be motivated by mean field theory.\n\nK (in line ) controls the pruning ration of the layer. If this is true then we have 1 extra parameter to finetune. \n\n# Results\n\nComparison shows better performance of the proposed approach over other pruning methods on CIFAR10/100. However, pruning network while training on CIFAR is not practical. Speeding up ImageNet training is more practical. \n\nWhen training ResNet-50 with accelerated strategy it will be nice to see a comparison with training a smaller model from scratch. For example, training ResNet-50 with uniform channel reduction initialization (only 50%, 75% of initial channels in the model should be 1.6x faster). One more obvious baseline is to finish training earlier for ResNet50. For example doing only half of training epochs. Cosine (or monotonic) learning rate decay will remove the need of hyper parameter search.\n\nWhen comparing results with pruning methods more other approaches should be added. For example, R3 has 76.4% with 50% FLOPs reduction. This method doesn't make pruning faster, \n\n# Overall\n\nThe approach is interesting, clearly explained and theoretically sound. Reviewer is not too much familiar with variation methods for pruning and might miss some important references. The approach of pruning aware training that speeds up training is not novel, hovewer the method is novel.\n\nQuestion if the pruning strategy is global (setting up K to be the neuron ration for the entire model, global pruning) or per-layer (setting K for every layer independently). In the former case cross-layer dependency might affect results and will need more hyper parameters. Code seems to indicate global pruning, just want to be sure.\n\nAlso additional baselines are required to see if the method outperform smaller model training from scratch in terms of training cost reduction. \nFor example ResNet-50 with uniform scaling, finishing training earlier (would be easy to target a particular 1.6x speed up, ImageNet preferably)\n\nFinally the baseline accuracy of ResNet50 on ImageNet is somewhat low. With the setting of cosine annulling, 4 GPUs we should expect ~77.2 accuracy of the full model. Did authors train the Original model with their setting or took from the model zoo. The latest might make comparison not correct. \n\nProviding code is great, however I found it a bit difficult to follow with too many arguments as input. Also for the final code release, would suggest to remove other than the proposed methods to make easy to follow. \n\n\n\n[R1] S. A. Aketi, S. Roy, A. Raghunathan, and K. Roy. Gradual channel pruning while training using feature relevance scores for convolutional neural networks. IEEE Access, 8:171924– 171932, 2020.\n\n[R2] Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2009–2018, 2020.\n\n[R3] EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning, ECCV2020", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "T8OdYMPxiJy", "reviewer": "Reviewer_CTnr", "summary": "This paper proposes an efficient sparse training method to solves the problem of accelerating the training speed of deep neural networks and save memory usage.  The authors first formulate the training process as a continuous minimization problem under global sparsity constraints.  Then, separating the optimization process into two steps, corresponding to weight update and structure parameter update. A variance-reduced policy gradient estimator is proposed to update the structure parameter. Finally, the experimental results are solid and strong. The problem has lots of meanings for engineering implementation.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The task is new. The method combinates some well-known techniques and also proposes its own technique part. It also makes a clear statement about the differences from previous contributions.\n\nQuality: The submission is technically sound.  Most claims are well supported (e.g., by theoretical analysis or experimental results). This is a complete piece of work. The authors are careful and honest about evaluating both the strengths and weaknesses of their work.\n\nClarity: The submission is clearly written and well organized.\n\nSignificance: The results are important and others would like to use the ideas and build on them. \nThe problem has lots of meanings for engineering implementation.  \nThe experimental results are solid and strong.\nThe analysis and the experiments of the proposed VR-PGE are solid and convincing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The only limitation of this paper is the complexity of the method implementation, especially for the VR-PGE part. The attached code about the training is a little bit complicated. Do the authors have any plan to release the code or make it a component of PyTorch?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "LCi9XmB06M0", "reviewer": "Reviewer_T3oD", "summary": "The authors propose a channel-level sparse training algorithm that can effectively utilize the channel sparsity to accelerate both forward and backward propagation process. They divide the sparse training process into weight updating step and mask updating step. And solve the problem separately. More specifically, they develop a variance-reduced policy gradient estimator to update the trainable pruning mask without the need for backpropagation, and hence a sparse computation on both forward and backward propagation.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThis paper shows an analysis that why previous parametric works are not able to utilize the sparsity to accelerate the training.\nThe authors evaluate their method on several representative DNNs and Datasets.\nIt is great that the authors also show actual training computational time rather than just using FLOPs to indicate the acceleration.\nTheir results seem decent when considering both accuracy and compression ratio.\n\nWeakness:\nIn the paper, the authors claim that they are the first to achieve training acceleration in practice. However, that is not correct or not rigorous. The existing work PruneTrain [1], and ClickTrain [2] can also achieve training acceleration. The author should also cite those papers and compare with them.\n\nFor the result comparison part, the accuracy of the original (dense) models is too low. For example, the accuracy of the original model for ResNet50 on ImageNet is 76.1%. I understand this is a widely used official model. But that model is too old, and it was probably obtained by some legacy training recipe. Your model is trained with warmup, label smoothing, and maybe cosine lr scheduler, which may significantly boost the model accuracy. To avoid misleading the audience, it is more reasonable to show the dense model accuracy obtained by using the same training settings as yours.\nFor a similar concern, the reference pruning works that you choose are also too old, which cannot represent the SOTA results (e.g., MetaPruning [3], DMCP [4]).\n\nMany references are missing publication information.\n\nYou mentioned the reference work NetAdapt in the caption of Table III, but you didn’t show the results.\n\nThe total training epochs and learning rate scheduler are missing. This may make your results less convincing.\n\nIt is good to see the accuracy and training speed if you directly train a small dense model with exactly the same model structure, layer width, and FLOPs as the remaining part (dense) of your sparse model. \n\n[1] PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration, SC’19\n[2] ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning, ICS’21\n[3] MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning, ICCV’19\n[4] DMCP: Differentiable Markov Channel Pruning for Neural Networks, CVPR’20\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "JS3C6s0lN8I", "reviewer": "Reviewer_CRsd", "summary": "This paper addresses the topic of sparse training of neural networks. They target channel-level sparsity with the goal of realizing practical training speedups with existing software and hardware. The authors study the limitations of existing techniques in terms of their ability to exploit sparsity during training and show a wide array of empirical results with their proposed technique.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The framing of this paper and it’s “truly sparse neural network training” technique is sensational and does not acknowledge the contributions of prior work on this topic. \n\nFirstly, It is not correct to say that no existing method can effectively accelerate training. There is a body of literature on sparse training that has developed techniques for exactly this. For example: SET [5], DeepR [6], SNFS [7], RigL [2] and Top-KAST [3]. The authors cite some of these techniques but don’t seem to acknowledge their contributions.\n\nSecond, it is not correct to say that sparse matrix multiplication cannot be implemented efficiently on widely-used platforms. Recent work has shown success in doing so for sparse deep learning workloads [8, 9].\n\nThird, when discussing channel-sparse algorithms, the authors state “It is obvious that these methods cannot achieve significant speedup in training since they need to calculate the full gradient in backward propagation”. This statement is an exaggeration. Even if the weight gradient computation needs to produce a dense gradient w.r.t. the sparsified weights the forward pass and backward gradient computation w.r.t. the input data to the linear operation can still take advantage of sparsity. That is to say 2/3rds of the FLOPs in the model still benefit from sparsity. This discrepancy leads me to believe that the authors could be underestimating the training-time FLOPs saving for other techniques in their experiments. \n\nDelving into this, the only existing technique for which the authors provide “training-cost savings” data in their experiments (Tables 2 & 3) is GrowEfficient. In their response, I’d like the authors of this work to clarify how they calculate the “training-cost savings” for GrowEfficient. Even if the weight gradients are calculated densely, the forward pass and half of the backward pass should benefit from sparsity.\n\nOn the topic of weight gradient computation, it’s not clear to me that GrowEfficient cannot exploit sparsity in these as well. As I understand it, GrowEfficient uses a similar Bernoulli random variable (q) on their weight matrices to induce sparsity. For cases where this value is 0, any gradient for the zeroed neuron should also be zero and thus GrowEfficient should also be able to exploit sparsity in their weight gradient computation. The authors of this work stated that “The authors of GrowEfficient [46] confirmed that actually they also calculated the gradient of qc w.r.t, sc in their Eqn.(6) via STE even if qc = 0.”. If this is true it would be correct to count the cost of computing a dense gradient for GrowEfficient. However, I was unable to confirm this from the GrowEfficient paper and basing conclusions on hearsay that is not documented in the literature feels tenuous. The clearest way to demonstrate this is true would be for the authors to re-implement the baseline technique, although I realize this is a considerable burden.\n\nOther Issues\n- In Table 1, many of these techniques are not “sparse training” algorithms, i.e. their aim is not to accelerate training with sparsity. For example, references 7, 50 and 28. Reference 19 is listed under both parametric and non-parametric techniques.\n- In general, the taxonomy of “non-parametric” and “parametric” sparsification methods is unclear and appears to be limited in scope. For example, it does not take into account details of the algorithm captured by existing taxonomies like “dense-to-sparse” v. “sparse-to-sparse”. I’d recommend the authors take a look at the taxonomies used in recent literature relating to sparsity. For example, Gale et. al [1], Evci et. al [2], Jayakumar et. al [3], and Hoefler et. al [4]. As I understand it, “parametric” techniques use some form of sparsity inducing regularization and “non-parameteric” techniques do something else. I don’t think a new term is necessary to make this distinction.\n- Ideally the authors would include training-cost savings for more techniques than GrowEfficient in their experiments. Given these theoretical savings rarely translate directly into runtime savings, it would improve the paper greatly if the authors included runtimes for more techniques than their own in Table 4.\n\nReferences\n[1] https://arxiv.org/abs/1902.09574\n[2] https://arxiv.org/abs/1911.11134\n[3] https://arxiv.org/abs/2106.03517\n[4] https://arxiv.org/abs/2102.00554\n[5] https://www.nature.com/articles/s41467-018-04316-3\n[6] https://arxiv.org/abs/1711.05136\n[7] https://arxiv.org/abs/1907.04840\n[8] https://arxiv.org/abs/1911.09723\n[9] https://arxiv.org/abs/2006.10901", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not identify potential negative societal impacts in this work.", "review_text": "", "extra_scores": {"ethical_concerns": "There are not ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3hrs"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes several effective schemes to ensuring the sparsity of a deep network being trained during backward and forward propagation. The reviewers recommendations are somewhat polarized and there are some serious disagreements between some reviewer and the authors regarding fair comparison with many existing sparsity promoting techniques. Despite some concerns, the proposed methods are relatively novel and effective and have been justified with adequate analysis and empirical evidences. The AC believes the quality of the paper is above the acceptance threshold. 
Decision: Accept (Poster)