Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
Authors: Junjie LIU
Zhe XU
Runbin SHI
Ray C. C. Cheung
Hayden K.H. So
Venue:  (2020)
Keywords: neural network pruning
sparse learning
network compression
architecture search

Abstract: We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly ﬁnd the optimal network parameters and sparse network structure in a uniﬁed optimization process with trainable pruning thresholds. These thresholds can have ﬁne-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same training epochs as dense models. Dynamic Sparse Training achieves prior art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence to the effectiveness and efﬁciency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.

Published as a conference paper at ICLR 2020
DYNAMIC SPARSE TRAINING: FIND EFFICIENT
SPARSE NETWORK FROM SCRATCH WITH TRAINABLE MASKED LAYERS
Junjie LIU', Zhe XU2, Runbin SHI Ray C.C. CHEUNG² Hayden K.H. SO l DDpartment of Electrical and Electronic Engineering, The University of Hong Kong Department of Electrical Engineering, City University of Hong Kong {ijliu, rbshi, h mku.hk zhexu22 cêmy. ci cityu. edu. z.cheung@cityu.edu .kk
ABSTRACT
We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network struc- ture unified optimization process with trainable pruning thresholds. These thresholds can have fine-grained layer-wise adjustments dynamically via back- propagation. We demonstrate that our dynamic sparse training algorithm can cas- ily train very sparse neural network models with little performance loss using the same number of training epochs as dense models. Dynamic Sparse Training achieves state the art performance compared with other sparse training algo- rithms various network architectures. Additionally, we have several surprising observations that provide strong evidence to the effectiveness and efficiency algorithm. These observations reveal the underlying problems of traditional three-stage pruming algorithms and present the potential guidance provided by our algorithm the design of more compact network architectures.
INTRODUCTION
Despite the impressive success that deep neural networks have achieved in wide range of chal- lenging tasks, the inference in deep neural networks highly memory-intensin and computation- intensive due the over- parameterization of deep neural networks. Network pruning (LeCun et al. (1990);(Hai) et al. (2135) Molchanov et al. (2017)) ) has been recognized as an effective approach to improving the inference efficiency in resource- limited scenarios.
Traditional pruning methods consist of dense network training followed with pruning and fine-tuning iterations. To avoid the expensive pruning and fine-tuning iterations. many sparse training meth- ods (Mocanu et al. 2018: Bellec et al.) 2017: Mostafa & Wang 2019 Dettmers & Zettlemoyer 2019) have been proposed, where the network pruning conducted during the training process. However, all these methods suffer from following three problems:
Coarse-grained predefined pruning schedule. Most of the existing pruming methods use prede- fined pruning schedule with many additional hyperparameters like pruning a% parameter each time and then fine-tuning for epochs with totally e pruning steps. hyperparameters for network architectures with various degrees of complexity. Therefore, usually fixed pruning schedule is adopted for all the network architectures, which means that very simple network architecture like LeNet-300-100 will have the same pruning schedule as a far more com- plex network like ResNet-152 152. Besides, almost all the existing pruning methods conduct epoch-wise pruning, which means that the pruning conducted between two epochs and no pruning operation happens inside each epoch.
non-trivial determine these
Failure to properly recover the pruned weights. Almost all the existing pruning methods conduct "hard" pruning that prunes weights by directly setting their values to 0. Many works (Guo et al. 2016 Mocanu et al. 2018 He al. 2018) have argued that the importance of network weights are not fixed and will change dynamically during the pruning and training process. Previously unimportant weights may tend be important. So the ability to recover the pruned weights is of

---

Published as a conference paper at ICLR 2020
high significance. However, directly setting the pruned weights to 0 results in the loss of historical parameter importance, which makes difficult to determine: 1) whether and when each pruned weight should be recovered, 2) what values should be assigned to the recovered weights. Therefore, existing methods that claim to be able to recover the pruned weights simply choose a predefined portion of pruned weights recover and these recover weights are randomly initialized initialized to the same value.
Failure to properly determine layer-wise pruning rates. Modern neural network architectures usually contain dozens of layers with a various number of parameters. Therefore, the degree of pa- rameter redundancy very different among the layers. For simplicity, some methods prune the same percentage of parameters at each layer, which is not optimal. To obtain dynamic layer-wise pruning rates, a single global pruning threshold or layer-wise greedy algorithms are applied. Using a single global pruning threshold is exceedingly difficult to assess the local parameter importance of the in- dividual layer since each layer has a significantly different amount of parameter and contribution to the model performance. This makes pruning algorithms based on single global threshold inconsis- tent and non-robust. The problem of layer-by-layer greedy pruning methods is that the unimportant neurons in early layer may have significant influence the responses in later layers, which may result in propagation and amplification of the reconstruction error Yuu et al. 2018).
We propose novel end-to-end sparse training algorithm that properly solves the above problems. With only one additional hyperparameter used to set the final model sparsity, our method can achieve dynamic fine-gr greenega pruning and recovery during the whole training process. Meanwhile, the layer- wise pruning rates will be adjusted automatically with respect to the change of parameter impor- tance during the training and pruning process. Our method achieves state-of-the-art performance compared with other sparse training algorithms. The proposed algorithm has following promising properties:
Step-wise pruning and recovery. A training epoch usually will have tens of thousands of training steps, which is the feed-forward and pook-propngation pass for single mini-batch, Instead of pruning between two training epochs with predefined pruning schedule, our method prunes and recovers the network parameter at each training step, which is far more fine- the-grained than existing methods.
Neuron-wise or filter-wise trainable thresholds. All the existing methods adopt a single pruning threshold for each layer the whole architecture. Our method defines threshold vector for each layer. Therefore, our method adopts neuron-wise pruning thresholds for fully connected and recurrent layer and filter-wise pruning thresholds for convolutional layer. Additionally, all these pruning thresholds are trainable and will be updated automatically via back-propagation.
Dynamic pruning schedule. The training process of deep neural network consists of many hyperparameters. The learning rate is perhaps the most important hyperparameter. Usually, the learning rate will decay during the training process. Our method can automatically adjust the layer-wise pruning rates under different learning rates to get the optimal sparse network structure. Consistent sparse pattern. Our algorithm can get a consistent layer-wise sparse pattern under different model sparsities, which indicates that our method can automatically determine the optimal layer-wise pruning rates given the target model sparsity.
2 RELATED WORK
Traditional Pruning Methods: LeCun et al. (1990) presented the early work about network pruning using second-order derivatives as the pruning criterion. The effective and popular training, pruning and fine-tuning pipeline was proposed by Han et al. (001)). which used the parameter magnitude as the pruning criterion. Narang et al.) (2017) extended this pipeline to prune the recurrent neural networks with complicated pruning strategy. Molchanov et al. (2016) introduced first-order Taylor term as the pruning criterion and conduct global pruning. Li ct al. (2016) used & regularization to force the unimportant parameters zero.
Sparse Neural Network Training: Recently, some works attempt to find the sparse network di- rectly during the training process without the pruning and fine-tuning stage. Inspired by the growth and extinction of neural cells in biological neural networks, Mocanu et al. (2018) proposed a prune- Morcann regrowth procedure called Sparse Evolutionary Training (SET) that allows the pruned neurons and
2

---

Deep neural network consists of set of parameters (W, : S sis C), where W; denotes the parameter matrix at layer i and C' denotes the number of layers in this network. For each fully connected layer and recurrent layer, the corresponding parameter W; E Re where c, is the output dimension and G is the input dimension. For each convolutional layer, there exists a convolution kernel K; E R x xaxh where Co the number of output channels, c, is the number of input channels, w and h are the kernel sizes. Each filter in convolution kernel K; can be flattened a vector. Therefore, a corresponding parameter matrix W; R can be derived from each convolution kernel K; E Rex xwxh, where = Ci X x h. Actually, the pruning process is equivalent to finding a binary parameter mask M, for each parameter matrix W,, Thus, a set of binary parameter masks {M; 1 < i S C} will be found by network pruning. Each element for these parameter masks M, is either 1 or 0. Published as a conference paper at ICLR 2020
connections to revive randomly. However, the sparsity level needs to be set manually and the ran- dom recovery of network connections may provoke unexpected effects on the network. DEEP-R proposed by [ellee et al.] (207)) used Bayesian sampling to decide the pruning and regrowth con- figuration, which is computationally expensive. Dynamic Sparse Reparameterization (Mostafa & Wang. 2019) used dynamic parameter reallocation to find the sparse structure. However, the prun- ing threshold can only get halved if the percentage of parameter pruned is too high or get doubled if that percentage is too low for certain layer. This coarse- gaanseegriieddddee adjustment of the pruning thresh- old significantly limits the ability of Dynamic Sparse Reparameterization. Additionally, predefined pruning ratio and fractional tolerance are required. Dynamic Network Surgery (Guo al.) 2016) proposed pruning and splicing procedure that can prune recover network connections according to the parameter magnitude but requires manually determined thresholds that are fixed during the sparse learning process. These layer-wise thresholds are extremely hard to manually set. Mean- while, Fixing the thresholds makes hard to adapt to the rapid change of parameter importance. Detmmes & Zettlemoyer (2019) proposed sparse momentum that used the exponentially smoothed gradients as the criterion for pruning and regrowth. A fixed percentage of parameters are pruned at each pruning step. The pruning ratio and momentum scaling rate need to be searched from relatively high parameter space.
DYNAMIC SPARSE TRAINING
3.1 NOTATION
3.2 THRESHOLD VECTOR AND DYNAMIC PARAMETER MASK
Pruning can be regarded as applying binary mask M to each parameter W. This binary parameter mask M preserves the information about the sparse structure. Given the parameter set (Wi, W2,... We). our method will dynamically find the corresponding parameter masks (M1 M2, Mc). To achieve this, for each parameter matrix W Rex a trainable pruning threshold vector t R is defined. Then we utilize unit step function S(z) shown in Fig- ure ([a a) to get the masks according to the magnitude of parameters and corresponding thresholds as present below.
Qi = F(W,,,
(5:=F(W,;; ()) = WWyl - 4 1 c, 1sjsc c
(1)
Mj = ((Qn)) 15i5 c. sssca
(2)
With the dynamic parameter mask M, the corresponding element in mask Mjj will be set to 0 if Wi needs to pruned. This means that the weight Wij masked out by the Mij to get sparse parameter W O M. The value of underlying weight Wy will not change, which preserves the historical information about the parameter importance.
For fully connected layer or recurrent layer with parameter W E Rexxx and threshold vector t E Rd= each weight Wyy will have a neuron-wise threshold t,, where Wij is the jth weight associated with the ith output neuron. Similarly, the thresholds filter-wise for convolutional layer. Besides, a threshold matrix or a single scalar threshold can also be chosen. More details are present in Appendix A.2
3

---

Published as a conference paper at ICLR 2020
3.3 TRAINABLE MASKED LAYERS
With the threshold vector and dynamic parameter mask, the trainable masked fully connected, con- volutional and recurrent layer are introduced as shown in Figure where X the input of current layer and Y is the output. For fully connected and recurrent layers, instead of the dense parameter W, the sparse product W M will be used in the batched matrix multiplication, where @ denote Hadamard product operator. For convolutional layers, each convolution kernel K IK X xexh can be flatten to get W R=xx= Therefore, the sparse kernel can obtained by similar process as fully connected layers. This sparse kernel will be used for subsequent convolution operation.
X
M
Parameter
Y
F(W. t)
S(x)
Element-wee wise product
Matrie-vector multipiiation of Convolution
Structure
gradient
Parformance gradient
Feed forward
Back-propagation
Figure I: Detailed structure trainable masked layer
unit step function Six)
unit impulse function Kx)
straight-through estimator STE(s)
long-tailed estimator Ho0
Figure 2: The unit step function S(x) and its derivative approximations.
In order to make the elements in threshold vector trainable via back- back-propagation. the derivative of the binary step function S(x) is required. However, its original derivative is impulse function whose value is zero almost everywhere and infinite at zero as shown in Figure ((b). Thus the origi- nal derivative of the binary step function S(x) cannot be applied in back-propagatioo and parameter updating directly. Some previous works (Hubara et al. y) Rastegari et al. (2016); Zhou et al. (2016)) demonstrated that by providing a derivative estimation, it possible to train networks con- laining such binarization function. A clip function called straight- estimator (STE) (Bengio ct al. 2013) was employed in these works and is illustrated Figure
Furthermore, Xu & Cheung (2019) discussed the derivative estimation in balance of tight approxi- mation and smooth Back-propagation We adopt this long-tailes higher-order estimator (xx) in our method. As shown in Figure d), it has wide active range between [-1, with a non-zero gradi- ent to avoid gradient vanishing during training. On the other hand, the gradient value near zero is piecewise polynomial function giving tighter approximation than STE. The estimator is represented
as
4/z%/
S x 0.4
0.4
s(x) = H(x)= dr
0.4,
0.4 otherwise
< x <
(3)
0.
With this derivative estimator, the elements in the vector threshold can be trained via back- propagation. Meanwhile, in trainable masked layers, the network parameter W can receive two branches of gradient, namely the performance gradient for better model performance and the struc- ture gradient for better sparse structure, which helps properly update the network parameter un- der sparse network connectivity. The structure gradient enables the pruned (masked) weights to be updated via back- paeckpropagation. The details about the feed-forward and back- propagaton of the trainable masked layer are present Appendix A.3
Therefore, the pruned (masked) weights, the unpruned (unmasked) weights and the elements in the threshold vector can all be updated via back- peok---rppgttion at each training step. The proposed method will conduct fine-grahe fine-grained step-wise pruning and recovery automatically.
4

---

Published as a conference paper at ICLR 2020
3.4 SPARSE REGULARIZATION TERM
Now that the thresholds of each layer are trainable, higher percentage of pruned parameter is de- sired. To get the parameter masks M with high sparsity, higher pruning thresholds are needed. To achieve this, we add sparse regularization term Lg to the training loss that penalizes the low thresh- old value. For a trainable masked layer with threshold t E Ry, the corresponding regularization exp(- (ft). Thus, the sparse regularization term La for a deep neural network
term is R = Eis =i= with C' trainable masked layers is:
L =
R
(4)
We use exp(-x) as the regularization function since it is asymptotical to zero as increases. Con- sequently, it penalizes low thresholds without encouraging them to become extremely large.
3.5 DYNAMIC SPARSE TRAINING
The traditional fully connected, convolutional and recurrent layers can be replaced with the corresponding trainable masked layers in deep neural networks. Then we can train a sparse neural network directly with back-propagation algorithm given the training dataset D { v1), (22, (ZN, yN)), the network parameter W and the layer-wise threshold t as follows:
J(W 1) C((x, ui); W,t) + aLs
(5)
W t" arg min J (WWtt)
(6)
W.t
where f(.) is the loss function, e.g. cross-entropy loss for classification and a is the scaling coef- ficient for the sparse regularization term, which can control the percentage of parameter remaining. The sparse regularization term L, tends to increase the threshold t for each layer thus getting higher model sparsity. However, higher sparsity tends increase the loss function, which reversely tends to reduce the threshold and level of sparsity. Consequently, the training process of the thresholds can be regarded as contest between the sparse regularization term and the loss function in the sense of game theory. Therefore, our method can dynamically find the sparse structure that properly balances the model sparsity and performance.
4 EXPERIMENTS
The proposed method is evaluated on MNIST, CIFAR-IO and ImageNet with various modern net- work architectures including fully connected, convolutional and recurrent neural networks. To quan- tify the pruning performance, the layer remaining ratio is defined to be hı = n/m, where is the number of elements equal model remaining ratio km is the overall ratio of the non-zero elements in the parameter masks for all trainable masked layers. The model remaining percentage defined as ki 100%. For all trainable masked layers, the trainable thresholds are initialized to zero since it is assumed that originally the network is dense. The detailed experimental sctup is present in Appendix
in the mask M and is the total number of elements in M. The
A. A.I
4.1 PRUNING PERFORMANCE ON VARIOUS DATASETS
MNIST. Table presents the pruning results of proposed method for Lenet-300-1 100 (LeCun et al., 1998). Lenet-5-Caffe and LSTM (Hochreiter & Schmidhuber 1997). Both LSTM models have two LSTM layers with hidden size 128 for LSTM-a and 256 for LSTM-b. Our method can prune almost 98% parameter with little loss of performance on Lenet-300-10 and Lenet-5-Cafte For the LSTM models adapted for the sequential MNIST classification, our method can find sparse models with better performance than dense baseline with over 99% parameter pruned.
CIFAR-10. The pruning performance of our method on CIFAR- 10 is tested on VGG (Simonyan & Zisserman 2014) and WideResNet (Zagoruyko & Komodakis 2016). We compare method
5

---

Published as a conference paper at ICLR 2020
Model Remaining Percentage (%)
Architecture
Dense Baseline (%)
Sparse Accuracy (%)
Difference
Lenet-300-
98.16 = 0.06
2.48 0.21
97.6910.14
-0.47
Lenet-5-Caffe
99.18 + 0.05
1.64 = 0.13
99.11
-0.07
LSTM-a
98.64 + 0.12
1.93 0.03
98.7011.06
+0.06
LSTM-b
98.87 + 0.07
0.98 = 0.04
98.89#0.11
+0.02
Table I: The pruning results MNIST for various architectures
with other sparse learning algorithms on CIFAR-10 as presented in Table [
The sute-of-the-ar
algorithms, DSR (Mostafa & Wang 2019 and Sparse momentum (Dettmers & Zettlemoyer 2019), are selected for comparison. Dynamic Sparse Training (DST) outperforms them in almost all the settings present in Table
Model Remaining Percentage (%)
Architecture
Method
Dense baseline
Sparse Accuracy
Difference
I0
93.36 = 0.04
-0.15
Sparse Momentum
93.51 + 0.05
93.00 + 0.07
-0.51
VGG-16
8.82
0.34
93.93 0.05
+0.18
DST (Ours)
93.75 + 0.21
3.76
0.53
93.02 0.37
-0.73
10
94.87 = 0.04
-0.56
Sparse Momentum
95.43 + 0.02
94.38 + 0.05
-1.05
10
94.93 + 0.04
-0.28
WideResNet-16-8
DSR
95.21 + 0.05
94.68 + 0.05
-0.53
9.86 1 0.22
95.05 1 0.08
-0.13
DST (Ours)
95.18 0.06
4.64 = 0.15
94.73 - 0.11
-0.45
Table 2: Comparison with other sparse training methods on CIFAR- 10.
ImageNet. For ResNet-50 on ImageNet (ILSVRC2012), we present the performance and compari- son with other methods Table Dynamic Sparse Training (DST) achieves better top-1 and top-5 accuracy with slightly higher model sparsity. The number of parameters for the dense ResNet-50 model 25.56 M.
Dense baseline Top-1 Top-5 (%)
Model Remaining
Sparse accuracy
Remaining number of parameters
Method
Difference
Percentage (%)
Top-1 Toe-l/Top- Topp-5 (%)
20
73.80/91 /9188
-1.10/-0.60
5.12M
Sparse Momentum
74.90 / 92.40
10
72.30 91.00
-2.60/11.40
2.56M
20
73.30792.40
-1.00740.0
5.14M
DSR
74.90/ / 92.40
10
71.60 90.50
-3.30/-1.90
2.56M
19.24
74.02 92.49
-0.737 40.11
5.0M
(Ours)
74.95 / 92.60
9.87
72.78 / 91.53
2-117/11.7
2.49M
Table 3: Comparison with other sparse training methods for ResNet-50 on ImageNet.
4.2 PRUNING PERFORMANCE ON VARIOUS REMAINING RATIO
By varying the scaling coefficient a for sparse regularization term, we can control the model re- maining ratios of sparse models generated by dynamic sparse training. The relationships between a, model remaining ratio and sparse model accuracy of VGG, WideResNet- 16-8 and -ideResNet-28-88 on CIFAR-10 10 are presented in Figure As demonstrated, the model remaining ratio keeps decreas- ing with increasing a. With moderate a value, it easy to obtain a sparse model with comparable or even higher accuracy than the dense counterpart. On the other side, if the value is too large that makes the model remaining percentage less than 5%, there will be noticeable accuracy drop. As the application
demonstrated in Figure 3 the choice of ranges from 10-9 to 10 Depending scenarios, we can either get models with similar or better performance as dense counterparts by a relatively small a get a highly sparse model with little performance loss by a larger a.

---

Published as a conference paper at ICLR 2020
eemaining rettio
Model remaahing radio
Tesi Accurecty (80)
Test Accuracy
30
30
alpha
alpha
alphs
(a) VGG-16
(b) WideResNet-16-3
(c) WideResNet-28-88
Figure 3: Test accuracy of sparse model on CIFAR-10 and model remaining ratio for different
5 DISCUSSION
5.1 FINE-GRAINED STEP-WISE PRUNING
Figure 4a) demonstrates the change of layer remaining ratios for Lenet-300-100 trained with DST at each training step during the first training epoch. And Figure (b) presents the change of layer remaining ratios during the whole training process (20 epochs). As present in these two figures, instead of decreasing by manually set fixed step size as in other pruning methods, our method makes the layer remaining ratios change smoothly and continuously at each training step. Meanwhile, as shown in Figure aa),, the layer remaining ratios fluctuate dynamically within the first 100 training steps, which indicates that DST can achieve step-wise fine-grained pruning and recovery.
Meanwhile, for multilayer neural networks, the parameters in different layers will have different relative importance. For example, Lenet-300-100 has three fully connected layers. Changing the parameter of the last layer (layer 3) can directly affect the model output. Thus, the parameter of layer 3 should be pruned more carefully. The input layer (layer 1) has the largest amount of parameters and takes the raw image as input. Since the images in MNIST dataset consist of many unchanged pixels (the background) that have no contribution to the classification process, it is expected that the parameters that take these invariant background pixels as input can be pruned safely. Therefore, the remaining ratio should be the highest for layer 3 and the lowest for layer 1 if Lenet-300-11 model pruned. To check the pruning effect of our method on these three layers, Lenet-300-11 100 model is sparsely trained by the default hyperparameters setting present in Appendix A. The pruning trends of these three layers during dynamic sparse training are present in Figure 4(b). During the whole sparse training process, the remaining ratios of layer and layer 2 keep decreasing and the remaining ratio of layer 3 maintains to be after the fluctuation during the first epoch. The remaining ratio of layer is the lowest and decrease to less than 10% quickly, which is consistent with the expectation. In the meantime, the test accuracy of the sparse model is almost the same as the test accuracy on the dense model in the whole training process. This indicates that our method can properly balance the model remaining ratio and the model performance by continuous fine-graines pruning throughout the entire sparse training procedure. similar training tendency can be observed in other network architectures. The detailed results for other architectures are present in Appendix A.4
Hoddl
Model remaining rotio
Test accurscy (16)
Layer remaining
Training step
Epoch
Epoch
(a) Layer remaining ratio during first epoch
(b) Layer remaining ratio during training
(c) Model remaining
ratio and accuracy
Figure 4: Change of layer remaining ratio, model remaining ratio and test accuracy for Lenet-300-100 with a 0.0005.
7

---

Published as a conference paper at ICLR 2020
5.2
DYNAMIC PRUNING SCHEDULE
Conv
Conv 4
FC2
Medell pernuriing 1et
Sperse accuracy
Dense eccurec
Layer remaining ratio
Model remaining ratio
0.8
0.5
Test Accuracy
-70
0.5
04-
50
0.2
0.0
100
140
170
Epoch
Epoch
(a) Remaining ratio of each layer
(b) Model remaining ratio and accuracy
Figure 5: VGG-16 16 trained with initial learning rate 0.1 and
10
Cane
Zone TC
Conv
Conv
FC2
Model remainig
Dense accuacy
Layer remaining ratio
Model remaining ratio
Test Accuracy (96)
70
0.2
02
20
80
100
140
100
20
60
50
100
120
160
Epoch
Epoch
(a) Remaining ratio of each layer
(b) Model remaining ratio and accuracy
Figure 6: VGG-16 trained with initial learning rate 0.01 and
10-6
Dynamic adjustment regarding learning rate decay. To achieve better performance, it is com- mon practice to decay the learning rate several times during the training process deep neural networks. Usually, the test accuracy will decrease immediately just after the learning rate decay and then tend toward flatness. A similar phenomenon is observed for VGG-16 on CIFAR-10 trained by DST present in Figure b)), where the learning rate decay from 0. to 0.01 at 80 epoch.
Like the layer 3 of Lenet-300-11 300--00, the second fully connected layer (FC 2) of VGG-16 the output layer, hence its remaining ratio is expected be relatively high. But surprising observation is that the remaining ratio of FC is quite low at around 0.1 for the first 80 epochs and increases almost immediately just after the 80 epoch present in Figure 5 5)a). We suppose that this is caused by the decay of the learning rate from 0.1 to 0.01 at 80 epoch. Before the learning rate decay, the layers preceding the output layer fail extract enough useful features for classification due to the relatively coarse-grained parameter adjustment incurred by high learning rate. This means that the corresponding parameters that take those useless features input can be pruned, which leads the low remaining ratio of the output layer. The decayed learning rate enables fine- grained parameter update that makes the neural network model converge to the good local minima quickly (Kawaguchi] 2016), where most of the features extracted by the preceding layers to be helpful for the classification. This makes previously unimportant network connections the output layer become important thus the remaining ratio of this layer gets abrupt increase.
There are two facts that support our assumptions for this phenomenon. The first fact the sud- den increase of the test accuracy from around 85% 92% just after the learning rate decay.
8

---

Published as a conference paper at ICLR 2020
Secondly, the remaining ratios of the preceding convolutional layers are almost unchanged after the remaining ratio of the output layer increases up to 1, which means that the remaining parameters in these layers necessary to find the critical features. The abrupt change of the remaining ratio of the output layer indicates that our method can dynamically adjust the pruning schedule regarding the change of hyperparameters during the training process.
Dynamic adjustment regarding different initial learning rate. To further investigate the effect of the learning rate, the VGG-16 model is trained by different initial learning rate (0.01) with other hyperparameters unchanged. The corresponding training curves are presented in Figure 6 As shown in Figure aa), the remaining ratio of FC increases up after around 40 epochs when the test accuracy reaches over 90%. This means that with the proper update of network parameters due to smaller initial learning rate, the layers preceding the output layer (FC 2) tend to extract useful features before the learning rate decay. It can be observed in Figure that the test accuracy only increases about 2% from 90% to 92% roughly after the learning rate decay the 80 epoch. Meanwhile, one can see that our method adopts different pruning schedule regarding different initial choices of training hyperparameters.
Model performance under dynamic schedule. Figure (b) and Figure bb) show the model re- maining ratio, test accuracy of sparse model (sparse accuracy) and test accuracy of dense counter- parts (dense accuracy) after each epoch during the whole training process. For the sparse model trained with initial learning rate 0.1, the final model remaining percentage is 8.82%. The test accuracy of this sparse model is 93.93%, which is better than 93.75% of the dense counterparts. Similarly, for the sparse model trained with initial learning rate 0.01, the final model remaining percentage 12.45%. The test accuracy of this sparse model is 92.74%, which is also better than 92.54% of the dense model.
Considering the model performance during the whole training process, when trained with initial learning rate 0.01, the sparse accuracy is consistently higher than dense accuracy during the whole training process as present in Figure Mcanwhile, one can see from Figure that the running average of sparse accuracy is also higher than that of dense accuracy during the whole training process when the initial learning rate is 0.1.
5.3 INFORMATION REVEALED FROM CONSISTENT SPARSE PATTERN
Many works have tried to design more compact models with mimic performance to over- parameterized models (He et al.) 2016 Howard et al., 2017 Zhang et al. 2018). Network archi- tecture search has been viewed as the future paradigm of deep neural network design. However, is extremely difficult to determine whether the designed layer consists of redundancy. Therefore, typical network architecture search methods rely on evolutionary algorithms (Liu et al. 2017) or reinforcement learning (Baker et al. 2016). which is extremely time-consuming and computation- ally expensive. Network pruning can actually be reviewed as a kind of architecture search process (Liu et al., [0188] Frankle & Carbin 2018)) thus the sparse structure revealed from network prun- ing may provide some guidance to the network architecture design. However, The layer-wise equal remaining ratio generated by the unified pruning strategy fails to indicate the different degrees of redundancy for each layer. And the global pruning algorithm is non-robust, which fails to offer consistent guidance.
Here we demonstrate another interesting observation called consistent sparse pattern during dynamic sparse training that provides useful information about the redundancy of individual layers as the guidance for compact network architecture design. For the same architecture trained by our method with various a values, the relative relationship of sparsity among different layers keeps consistent. The sparse patterns of VGG-16 on CIFAR-10 10 are present in Figure with three different a.
In all configurations, the last four convolutional layers (Conv 10-13) and the first fully connected layers (FC 1) are highly sparse. Meanwhile, some layers (Conv 3-7, FC 2) keep a high amount of parameters after pruning. This consistent sparse pattern indicates that these heavily pruned layers consist of high redundancy and large number of parameters can be reduced in these layers to get more compact models. This phenomenon also exists in other network architectures, which is present in detail in the appendix A.5) Therefore, with this consistent sparse pattern, after designing a new network architecture, our method can be applied to get useful information about the layer-wiss wise redundancy in this new architecture.

---

Published as a conference paper at ICLR 2020
0th01
Co
Genc
Layer remaining
Layer remaining
remaining
(a) a = 10" 6.66% remaining
(b) a 10 10-", 15.47% remaining
(c) a 10 35.35% remaining
Figure 7: The sparse pattern and percentage of parameter remaining for different choices of VGG-16
CONCLUSION
We propose Dynamic Sparse Training (DST) with trainable masked layers that enables direct train- ing of sparse models with trainable pruning thresholds. DST can be easily applied to various types of neural network layers and architectures to get highly sparse or better- setterperffrmed model than dense counterparts. With the ability reveal the consistent sparse pattern, method can also provide useful guidance to the design of the more compact network.
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. arXiv preprint urXiv:1611.02167 2016.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint 17X0v:171..05136, 2017.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 30o::3338.332 2013.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 197iv:190..04840, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /803.0360, 2018.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances In Neural Information Processing Systems, pp. 1379-1387, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 10.000149 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 1808. 06866, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for 2017.
mobile vision applications. arXiv preprint
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in neural information processing systems, pp. 4107-4115. 2016.
10

---

Published as a conference paper at ICLR 2020
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 594, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 2014.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-599889 0998-05,111 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (1:11)2:278233333 2324, 1998.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 2016.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search. arXiv preprint arXiv:1711.00436 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):2388, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 2498-2507. JMLR. org, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint 10XXv:6611106444 2016.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. arXiv preprint arXiv: 2019.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint 17X1v:1704.05119 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision (ECCV), pp. 325-542. Springer, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint vrXiv::409/ 1556, 2014.
Zhe Xu and Ray C. C. Cheung. Accurate and compact convolutional neural networks with trained binarization. In British Machine Vision Conference (BMVC), 2019.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and I Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194- 9203, 2018
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide residual networks.
arXiv preprint
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedingy of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6848-6856, 2018
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 0666666066.0660,,11 2016.
11

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 EXPERIMENTAL SETUP
MNIST: LeNet-300-10 100 and LeNet-5-Caffe trained using SGD with momentum of 0.9 and the batch size of 64 with 20 training epochs. The learning rate is 0.01 without learning rate decay. Meanwhile, the default scaling factor for sparse regularization term is 0.0005 for both network architectures. LSTM models are trained using Adam optimization scheme (Kingma & Baj 2014) with default Adam hyperparameter setting for 20 epochs. The batch size is 100 and the default learning rate is 0.001. Meanwhile, the default scaling factor for sparse regularization term is 0.001.
CIFAR-10 Models on CIFAR-10 10 are trained using SGD with momentum 0.9 and batch size of 64 with 160 training epochs. The initial learning rate is 0.1 and decayed by 0.1 at 80 and 120 epoch. The default scaling factor for sparse regularization term is 5 x 10-6 for all tested architectures.
ImageNet. ResNet-50 models are trained using SGD with momentum 0.9 and batch size of 1024 with 90 training epochs. The default initial learning rate is 0.1 and decayed by 10 at 30, 60, 80 epochs.
For all trainable masked layers, the trainable thresholds are initialized to zero. Additionally, we find that an extremely high scaling coefficient a will make the sparse regularization term dominates the training loss thus making the mask M to be all zero in certain layer, which may impede the training process. To prevent this, the pruning threshold will be reset to zero if more than 99% elements in the mask zero in certain layer despite the LSTM models. This mechanism makes the training process smoother and enables a wider range choice of a.
A.2 ANALYSIS OF DIFFERENT THRESHOLD CHOICES
Beside a threshold vector, threshold scalar & or threshold matrix E Reaxe can also be adopted for each trainable masked layer with parameter W e Rex.
Wi 4 W T for threshold matrix
for threshold scalar
Qu
All three choices of trainable thresholds are tested on various architectures. Considering the effects on network pruning, the matrix threshold has almost the same model remaining ratio compared with the vector threshold. The scalar threshold is less robust than the vector and matrix threshold. In terms of the storage overhead, the matrix threshold almost doubles the amount of parameter during the training process in each architecture. Meanwhile, the vector threshold only adds less than I 1% additional network parameter.
The extra trainable threshold will also bring additional computation, In both feed-forward and back- propagation phase, the additional computations are matrix-clement-wiss operations (O(n²)) which is apparently light-weighted compared to the original batched matrix multiplication (O(r"), For practical deployment, only the masked parameter W C M needs to be stored, thus overhead will be introduced. Therefore, considering the balance between the overhead and the benefit incurred by these three choices, the vector threshold is chosen for trainable masked layers.
A.3 MORE DETAILS ABOUT THE TRAINABLE MASKED LAYER
Feed forward process. Considering single layer in deep neural network with input and dense parameter W. The normal layer will conduct matrix-vector multiplication as W or convolution Conv( W, x). In trainable masked layers, since a sparse mask M is obtained for each layer. The sparse parameter W o M will be adopted in the corresponding matrix-vector multiplication or convolution as (W& @ M) and Conv W C M)
Back- poockproopggatio process. Referring to Figure I we denote P = W M and the gradient received by P in back-propag d.P Considering the gradients from right to left:
12

---

Published as a conference paper at ICLR 2020
The performance gradient is dP M
The gradient received by M dP @ W
The gradient received by Q is dP @ WO O (QQ,, where (Q) is the result of (x) applied Q elementwisely.
The structure gradient is dP W O HQQ) sgn( (W), where sgn sgn(W) is the result of sign function applied to W elementwisely.
The gradient received by the vector threshold is dt E R We denote dT - W (Q)).
then dT E x And we will have dt, Σド= Ti for Si co.
The gradient received by the parameter W is dW dP M d W o H(Q) O (W)
Since we add la regularization in the training process, all the elements in W are distributed within []., Meanwhile, almost all the elements in the vector threshold are distributed within [0, 1]. The exceptions are the situation as shown in Figure 3(a) and Figure 4(a) where the last layer get no weight pruned (masked). Regarding the process of getting Q, all the elements in Q are within 1]. Therefore (QQ) is dense matrix. Then W, (Q) and sgn( W) are all dense matrices and the pruned (masked) weights can receive the structure gradient dP W (Q) @ sgn (W)
A.4 CHANGE OF REMAINING RATIO IN OTHER NETWORK ARCHITECTURES
Here we present the change of remaining ratios during dynamic sparse training for the other tested architectures. Since WideResNet models only have one fully connected layer at the end as the output layer, the first five convolutional layers and the last fully connected layer are present. Fig- ure g demonstrates the corresponding result for WideRes Figure 9 demonstrates the cor- responding result for WideResNet- 16-10. And Figure IO demonstrates the corresponding result for WideResNett22888 The similar phenomenon can be observed in all these three network architectures for various
A.5 CONSISTENT SPARSE PATTERN IN OTHER NETWORK ARCHITECTURES
Here we present the consistent sparse pattern for the other tested architectures. Figure IT demon- strates the corresponding result for 16d And Figure 12 demonstrates the correspond- ing result for WideResNet-16-10.
Coew
Conn
Ceny R.
Coew
Cone4
Nocel remaining 1200
Sparse accursty
Denie #comtry
1.0
0.9
Model rermaininn ratio
08-
(%))
Layer Temaining
0.3
0.1
0.0
Epoch
Epoch
(a) Remaining ratio of each
(b) Model remaining ratio and accuracy
Figure 8: WideResNet-1664 trained by dynamic sparse training with a 10
13

---

Published as a conference paper at ICLR 2020
Cone3
canc
remaztig 1NT0
Spana WCEATREY
Laver remaining ratio
Model remaining ratio
(%)
0.7
64
0.--
0.5
0.-
0.1
0.0
Epoch
Epoch
(a) Remaining ratio of each layer
(b) Model remaining ratio and accurncy
Figure 9: WideResNet-166 16-10 trained by dynamic sparse training with a
Conu
Conk 4ons
Canv KC
Conw 2
Nocal remaiting 1ET
Sparne accurney
Demio acomwcy
Model remaining ratio
0.0
Test Accuracy (%)
0.7-
Layer remaining
Mwwn Mn
www.h vada
02-
0.1
6.1
0.0
240
100
Epoch
Epoch
(a) Remaining ratio of each
(b) Model remaining and accurncy
Figure 10: WideResNet-28-8 trained by dynamic sparse training with a =
019
099
079
0911
099
22 10 18. remaining
melo
rabio
(a)
10 9.86% remaining
(b)
10 14.47% remaining (c)
Figure 11: The sparse pattern and percentage of parameter remaining for different choices of on WRN-16-8
029
060
020
00%
Layer remainng ratio
Layar romaining rablo
Layer remaning
(a) 10 remaining
x 10 10.47% remaining
5 10 14.75% remaining
Figure 12: The sparse pattern and percentage of parameter remaining for different choices of a on WRN-16-10
14


Reviews:
[{"review_id": "BkxVwrQM5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for training networks with sparse parameter tensors. This involves achieving sparsity by application of a binary mask, where the mask is determined by current parameter values and a learned threshold. It also involves the addition of a specific regularizer which encourages the thresholds used for the mask to be large. Gradients with respect to both masked-out parameters, and with respect to mask thresholds, are computed using a \"long tailed\" variant of the straight-through-estimator.\n\nThe algorithm proposed in this paper seems sensible, but rather ad hoc. It is not motivated by theory or careful experiments. As such, the value of the paper will be determined largely by the strength of the experimental results. I believe the experimental results to be strong, though I am not familiar enough with this subfield to be confident there are not missing baselines.\n\nThere are many minor English language problems (e.g. with articles, prepositions, plural vs. singular forms, and verb tense), though these don't significantly interfere with understanding.\n\nRounding up to weak accept, though my confidence is low because I am basing this positive assessment on experimental results for tasks on which I am not well calibrated.\n\nmore detailed comments:\n\n\"using the same training epochs\" -> \"using the same number of training epochs\"\n\"achieves prior art performance\" -> \"achieves state of the art performance\"\n\"the inference of deep neural network\" -> \"inference in deep neural networks\"\n\nThis paper considered only sparsity of weights -- it might have been nice to also discuss/run experiments exploring sparsity of activations.\n\neq. 4 -- Can you say more about why this particular form is the right one for the regularizer? It seems rather arbitrary. (it will tend to be dominated by the smallest thresholds, and so would seem to encourage a minimum degree of sparsity in every layer)\n\nI appreciate the analysis in section 4.3.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkl9Cpy0Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Update after the rebuttal\nI appreciate the author's clarification in the rebuttal and the additional result on ImageNet, which addressed some of my concerns.\n\n# Summary\nThis paper proposes a trainable mask layer in neural networks for compressing neural networks end-to-end. The main idea is to apply a differentiable mask to individual weights such that the mask itself is also trained through backpropagation. They also propose to add a regularization term that encourages weights are masked out as much as possible. The result on MNIST and CIFAR show that their method can achieve the highest (weight) compression rate and the lowest accuracy reduction compared to baselines. \n\n# Originality\n- The idea of applying trainable mask to weights and regularizing toward masking out is quite interesting and new to my knowledge. \n\n# Quality\n- The performance seems to be good, though it would be more convincing if the paper showed results on larger datasets like ImageNet. \n\n- The analysis is interesting, but I am not fully convinced by the \"strong evidence to the efficiency and effectiveness of our algorithm\". For example, the final layer's remaining ratio is constantly 1 in Figure 3, while it starts from nearly 0 in Figure 4. The paper also argues that the final layer was not that important in Figure 4 because the lower layers have not learned useful features. This seems not only contradictory to the result of Figure 3 but also inconsistent of the accuracy being quickly increasing up to near 90% while the remaining ratio is nearly 0 in Figure 4. \n\n- If the motivation of the sparse training is to reduce memory consumption AND computation, showing some results on the reduction of the computation cost after sparse training would important to complete the story. \n\n# Clarity\n- The description of the main idea is not clear. \n\n- What are \"structure gradient\" and \"performance gradient\"? They are not mathematically defined in the paper.\n\n- I do not understand how the proposed method can \"recover\" from pruned connection, although it seems to be indeed happening in the experiment. The paper claims that the use of long-tailed higher-order estimator H(x) makes it possible to recover. However, H(x) still seems to have flat lines where the derivative is 0. Is H(x) in Equation 3 and Figure 2d are showing \"derivative\" or step function itself? In any cases, I do not see how the gradient flows once a weight is masked out. \n\n# Significance\n- This paper proposes an interesting idea (trainable mask), though I did not fully get how the mask is defined/trained and has a potential to recover after pruning. The analysis of the compression rate throughout training is interesting but does not seem to be fully convincing. It would be stronger if the paper 1) included more results on bigger datasets like ImageNet, 2) described the main idea more clearly, and 3) provided more convincing evidence why the proposed method is effective. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgGbhJ-iB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel network pruning algorithm  -- Dynamic Sparse Training. It aims at jointly finding the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds.  The experiments on MNIST, and cifar-10 show that proposed model can find sparse neural network models, but unfortunately with little performance loss.\nThe key limitation of the proposed model come from the experiments.\n(1) Nowadays, the nature and important question is that, one can not tolerate the degraded performance, even with sparse neural network. Thus, it is important to show that the proposed model can find sparse neural network models, and with increased performance. \n(2) Another weakness is that proposed model has to be tested on large scale dataset, e.g. ImageNet-2012.current two datasets are too small to support the conclusive results of this proposed model. \n(3) As for the model itself, I donot find very significant novelty. For example, Sec. 3.3 (TRAINABLE MASKED LAYERS) in general is quite following previous works’ designing principle. Thus, the novelty should be summarized, and highlighted in the paper.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper lies on the borderline. An accept is suggested based on majority reviews and authors' response.
Decision: Accept (Poster)