Sparse Networks from Scratch: Faster Training without Losing Performance
Authors: Tim Dettmers
Luke Zettlemoyer
Venue:  (2020)
Keywords: sparse learning
sparse networks
sparsity
efficient deep learning
efficient training

Abstract: We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer. Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms. Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network. 

Under review as a conference paper at ICLR 2020
SPARSE NETWORKS FROM SCRATCH: FASTER TRAINING WITHOUT LOSING PERFORMANCE
Anonymous authors Paper under double-blind review
ABSTRACT
We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse mo- mentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum mag- nitude of each layer. Within layer, sparse momentum grows weights according to the momentum magnitude zero-valued weights. We demonstrate state-of-the-art sparse performance error by a relative 8%, 15, and 6% compared to other sparse algorithms. Fur- thermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61 faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network.
MNIST, CIFAR-10 10, and ImageNet, decreasing the mean
INTRODUCTION
Current state-of-the-ar neural networks need extensive computational resources to be trained and can have capacities of close one billion connections between neurons (Vaswani et al.) 2017]] Devlin et al. 2018 Child et al., 2019). One solution that nature found to improve neural network scaling is to use sparsity: the more neurons brain has, the fewer connections neurons make with each other SancullanoHHuzzzee et al. 2010). Similarly, for deep neural networks, it has been shown that sparse weight configurations exist which train faster and achieve the same errors dense networks (Frankle and Carbin 2019). However, currently, these sparse configurations are found by starting from a dense network, which is pruned and re- re-trained repeatedly an expensive procedure.
In this work, we demonstrate the possibility of training sparse networks that rival the performance of their dense counterparts with single training run - no re-training is required. We start with random initializations and maintain sparse weights throughout training while also speeding up the overall training time. We achieve this by developing sparse momentum, an algorithm which uses the exponentially smoothed gradient of network weights (momentum) as a measure of persistent errors to identify which layers are most efficient at reducing the error and which missing connections between neurons would reduce the error the most. Sparse momentum follows cycle of (1) pruning weights with small magnitude, (2) redistributing weights across layers according to the mean momentum magnitude of existing weights, and (3) growing new weights to fill in missing connections which have the highest momentum magnitude.
We compare the performance of sparse momentum to compression algorithms and recent methods that maintain sparse weights throughout training. We demonstrate state-of- thethheart sparse performance on MNIST, CIFAR-10, and ImageNet- k. For CIFAR-10 10, we determine the percentage of weights needed to reach dense performance levels and find that AlexNet, VGG16, and Wide Residual Networks need between 35-50%, 5-10%, and 20-30% weights reach dense performance levels. We also estimate the overall speedups of training our sparse convolutional networks to dense performance levels on CIFAR-10 10 for optimal sparse convolution algorithms and naive dense convolution algorithms compared to dense baselines. For sparse convolution, we estimate speedups between 2.74x and 5.61x and for dense convolution speedups between 1.07x and 1.36x. your analysis, ablations demonstrate

---

Under review as a conference paper at ICLR 2020
that the momentum redistribution and growth components are increasingly important as networks get deeper and larger in size both are critical for good ImageNet performance.
RELATED WORK
From Dense to Sparse Neural Networks: Work that focuses on creating sparse from dense neural networks has an extensive history. Earlier work focused on pruning via second-order derivatives (LeCun et al. 1989 Karnin 1990; Hassibi and Stork 1992) and heuristics which ensure efficient
training of networks after pruning Chauvin 1988; Mozer and Smolensky 1988 Ishikawa, 1996). Recent work is often motivated by the memory and computational benefits of sparse models that enable the deployment of deep neural networks on mobile and low-energy devices. A very influential paradigm has been the iterative (1) train-dense, (2) prune, (3) re-train cycle introduced by Han et al. (2015). Extensions to this work include: Compressing recurrent neural networks and other models (Narang et al., 2017: Zhu and Gupta 2018; Dai et al.] 2018). continuous pruning and re- re-training (Guo et al. 2016). joint loss/pruning-cov cost optimization (Carreira-Perpinn and Idelbayev 2018). layer-by-layer pruning (Dong et al.] 2017). fast-switching growth-pruning cycles (Dai et al. 2017).
and soft weight-sharing (Ullrich et al. 20177). These approaches often involve re-training phases which increase the training time. However, since the main goal of this line of work is compressed model for mobile devices, it is desirable but not an important main goal to reduce the run-time of these procedures. This is contrary our motivation. Despite the difference in motivation, we include many of these dense-to-spars compression methods in our comparisons. Other compression algorithms include La regularization (Louizos et al./ 2018). and Bayesian methods (Louizos et al.) 2017 Molchanov et al. 2017). For further details, see the survey of Gale et al. (2019).
Interpretation and Analysis of Sparse Neural Networks: Frankle and Carbin (2099) show that "winning lottery tickets" exist for deep neural networks- sparse initializations which reach similar predictive performance as dense networks and train just as fast. However, finding these winning lottery tickets is computationally expensive and involves multiple prune and re-train cycles starting from dense network. Followup work concentrated on finding these configurations faster (Frankle) al. 2019 Zhou et al. 2019). In contrast, reach dense performance levels with a sparse network from random initialization with a single training run while accelerating training.
Sparse Neural Networks Throughout Training: Methods that maintain sparse weights throughout training through a reuue-redisrribut--re regrowth cycle are most closely related to our work. Bellec et al. (2018) introduce DEEP-R, which takes a Bayesian perspective and performs sampling for prune and regrowth decisions sampling sparse network configurations from a posterior. While theoretically rigorous, this approach is computationally expensive and challenging to apply large networks and datasets. Sparse evolutionary training (SET) (Mocanu et al.) 2018) simplifies prune-regrowth cycles by using heuristics: (1) prune the smallest and most negative weights, (2) grow new weights in random locations. Unlike our work, where many convolutional channels are empty and can be excluded from computation, growing weights randomly fills most convolutional channels and makes it challenging to harness computational speedups during training without specialized sparse algorithms. SET also does not include the cross-layer redistribution of weights which we find to be critical for good performance, shown in our ablation study. The most closely related work to ours is Dynamic Sparse Reparameterization (DSR) by Mostafa and Wang (2019), which includes cycle. However, DSR requires some specific layers to be dense.
the full prune-redistee rewistribute--rrgroonee Our method works in a fully sparse setting and is thus more generally applicable. More distantly related is Single-shot Network Pruning (SNIP) (Lee et al., 2019), which aims to find the best sparse network from a single pruning decision. The goal of SNIP is simplicity, while our goal is maximizing predictive and run-time performance. In our experiments, we compare against all four methods: DEEP-R, SET, DSR, and SNIP.
3 SPARSE LEARNING
We define sparse learning to be the training of deep neural networks which maintain sparsity through- out training while matching the predictive performance of dense neural networks. To achieve this, intuitively, we want to find the weights that reduce the error most effectively. This is challenging since most deep neural network can hold trillions of different combinations of sparse weights. Additionally,
2

---

Under review as a conference paper at ICLR 2020
during training, feature hierarchies are learned, efficient weights might change gradually from shallow to deep layers. How can we find good sparse configurations? In this work, we follow a divide-and-conguee strategy that is guided by computationally efficient heuristics. We divide sparse learning into the following sub-problems which can be tackled independently: (1) pruning weights, (2) redistribution of weights across layers, and (3) regrowing weights, defined in more detail below.
Sparse Momentum
Calculate mean magnitude momentum for existing weights
Prune smallest fraction of weights, p. for each layer
Regrow weights according momentum of missing weights
Remove
Regrow
16 Weights
16 Weights
8 Weights
Weights
16 Weights
28%
12%
22%
38%
Mean Momentum Contribution per layer
Removed Weights
Removed Weights
Redistribute
Figure 1: Sparse Momentum is applied at the end of each epoch: (1) take the magnitude of the exponentially smoothed gradient (momentum) of each layer and normalize to 1; (2) for each layer, remove p 20% of the weights with the smallest magnitude; (3) across layers, redistribute the removed weights by adding weights to each layer proportionate to the momentum of each layer; within layer, add weights starting from those with the largest momentum magnitude. Decay p.
3.1 SPARSE MOMENTUM
We use the mean magnitude of momentum M; of existing weights W; in each layer to estimate how efficient the average weight in each layer is at reducing the overall error. Intuitively, we want to take weights from less efficient layers and redistribute them to weight-efficient layers. The sparse momentum algorithm is depicted in Figure T In this section, we first describe the intuition behind sparse momentum and then present more detailed description of the algorithm.
The gradient of the error with respect a weight aw yields the directions which reduce the error AE
DE
at the highest rate. However, we use stochastic gradient descent, most weights of aW oscillate between small/large and negative/positive gradients with each mini-batch (Qian) 1999) good change for one mini-batch might be a bad change for another. We can reduce oscillations if we take the average gradient over time, thereby finding weights which reduce the error consistently. However, we want to value recent gradients, which are closer to the local minimum, more highly than the distant past. This can be achieved by exponentially smoothing ZE
SW the momentum M,:
ØE
M ++ = aM +1 aa)
aw,
where ar is smoothing factor, M, is the momentum for the weight W, in layer i; M, is initialized at f = 0 with 0.
Momentum is efficient at accelerating the optimization of deep neural networks by identifying weights which reduce the error consistently. Similarly, the aggregated momentum of weights in each layer should reflect how good each layer is reducing the error consistently. Additionally, the momentum of zero-valis zero-valued weights equivalent to missing weights in sparse networks - can used to estimate how quickly the error would change if these weights would be included in sparse network.
The details of the full training procedure our algorithm are shown in Algorithm[] See Algorithm in the Appendix for more detailed, source-code-lite description of sparse momentum.

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Sparse momentum algorithm.
Data: Layer
with: Momentum M,, Weight W,, binary Mask; prune rate Pi, density
for i <- 0 to k do
W; f xavierlnit(W) W
Mask; createMaakkForeeihhu (,
applyMask(W. Mask;)
end
for epoch +- 0 to numEpochs do
for f- 0 to numBatches do
batch + getBatch(j)
ØE
computeGraddeets((. batch)
7W
10
Updatc-Momentumí ow
Update (eights(M)
11
for i + 0 to & do
13
applyMask[ W. Mask;) end
15
end
16
tomentummn f- gerlotal/omeetum
totalPruned < getTolallPrunedWeehhsssS
18
for i 4 to & do
m, getMomenntumCCottiittttoono [M, Mask; totalMomentumm muugituudePruneWeiiee W, Mask; Ma,kk
20
regrowWeights( (W, Mask, totalPruned)
22
6 decayrruneraat
applyMask(W, (W. Mask,)
end
end
Before training, we initialize the network with a certain sparsity s: we initialize the network as usual and then remove fraction of weights for each layer. We train the network normally and mask the weights after each gradient update to enforce sparsity. We apply sparse momentum after each epoch. We can break the sparse momentum into three major parts: (a) redistribution of weights, (b) pruning weights, (c) regrowing weights. In step (a), we we take the mean of the element-wise momentum magnitude m, that belongs all nonzero weights for each layer and normalize the value by the total momentum magnitude all layers Ei=o m,- The resulting proportion is the momentum magnitude contribution for each layer. The number of weights to be regrow in each layer is the total number of removed weights multiplied by each layers momentum contribution: Regrow; = Total Removed m,, In step (b), prune proportion of p (prune rate) of the weights with the lowest magnitude for each layer. In step (c), we regrow weights by enabling the gradient flow of zero-valued (missing) weights which have the largest momentum magnitude.
Additionally, there are two edge-cases which we did not include in Algorithm| for clarity. (1) If we allocate weights be regrown than is possible for specific layer, for example regrowing 100 weights for a layer of maximum 10 weights, we redistribute the excess number of weights equally among all other layers. (2) For some layers, our algorithm will converge in that the average weight in layer has much larger momentum magnitude than weights in other layers, but at the same time, this layer is dense and cannot grow further. We do not want to prune weights from such important layers. Thus, for these layers, we reduce the prune rate proportional to the sparsity:
min(p, sparsity,).
After each epoch, we decay the prune rate Algorithm T in the same way learning rates are decayed. We use cosine decay schedule that anneals the prune rate to zero on the last epoch. See Appendix A. for analysis how decay schedule and starting prune rate affects training.
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTAL SETUP
For comparison, we follow three different experimental settings, one from Lee et al. (2019) and two settings follow Mostafa and Wang 2019); For MNIST (LeCun) 1998), we use a batch size of 100, decay the learning rate by a factor of 0.1 every 25000 mini-batches. For CIFAR-10 (Krizhevsky and Hinton 2009), we use standard data augmentations (horizontal flip, and random crop with reflective padding), batch size of 128, and decay the learning rate every 30000 mini-batches. We train for 100 and 250 epochs on MNIST and CIFAR-10, use a learning rate of 0.1, stochastic gradient descent with Nesterov momentum of 0.9, and we use weight decay of 0.0005. We use fixed 10% of the training data as the validation set and train on the remaining 90%. We evaluate the test set performance of our models on the last epoch. For all experiments on MNIST and CIFAR-IO. 10, we report the standard errors. Our sample size is generally between 10 and 12 experiments per methood/archieectuu/ssssssyy level with different random seeds for each experiment.
We use the modified network architectures of AlexNet, VGGI6, and LeNet-5 as introduced by Lee al. (2019). We consider two different variations of the experimental setup of Mostafa and Wang (2019) for ImageNet and CIFAR-10. The first follows their procedure closely, in that we run the networks in a partially dense setting where the first convolutional layer and downsampling convolutional layers are dense. Additionally, for CIFAR-10 10 the last fully connected layer is dense. In the second setting, we compare in a fully sparse setting - no layer is dense at the beginning of training. For the fully sparse setting we increase overall number of weights according to the extra parameters in the dense layers and distribute them equally among the network. The parameters in the dense layers make up 5.63% weights of the ResNet-50 network. We refer to these two settings the partially dense and fully sparse settings.
On ImageNet (Deng et al., 2009), we use ResNet-50 (He et al. 2016) with stride of for the 3x3 convolution in the bottleneck layers. We use batch size of 256, input size of 224, momentum of 0.9, and weight decay of 10-4, We train for 100 epochs and report validation set performance after the last epoch. We report results for the fully sparse and the partially dense setting.
For all experiments, we keep biases and batch normalization weights dense. We tuned the prune rate and momentum rate a searching the parameter space {0.2, 0.3, 0.4, 0.5, 0.6, 0.7} and {0.5, 0.6. 0.7, 0.8, 0.9, 0.95, 0.99] on MNIST and CIFAR-10 and found that p 0.2 and a 0.9 work well for most architectures. We use this prune and momentum rate throughout all experiments.
ImageNet experiments were run on 4x RTX 2080 Ti and all other experiments on individual GPUs.
Our software builds on PyTorch (Paszke et al. 2017) and is wrapper for PyTorch neural networks with a modular architecture for growth, redistribution, and pruning algorithms. Currently, no GPU- accelerated libraries that utilize sparse tensors exist, and as such we use masked weights to simulate sparse neural networks. Using our software, any PyTorch neural network can be adapted to be a sparse momentum network with less than 10 lines of code. We will open-source our software along with trained models and individual experimental results
RESULTS
Results in Figure 2 and T a comparison with model compression methods. On MNIST, sparse momentum the only method that provides consistent strong performance across both LeNet 300-100 and LeNet-5 Caffe models. Soft-weight sharing (Ullrich et al.) (017) and Layer-wise Brain Damage (Dong et al. (017) are competitive with sparse momentum for one model, but under- performs for the other model. For 1-2% of weights, variational dropout is more effective - but this method also uses dropout for further regularization while we only use weight decay. We can see that sparse momentum achieves equal performance to the LeNet-5 Caffe dense baseline with 8% weights.
On CIFAR-I0 in Table T we can see that sparse momentum outperforms Single-shot Network Pruning (SNIP) for all models and can achieve the same performance level as dense model for VGG16-D with just 5% of weights.
https: : htbp:://wwww /www dropbox.....oooo
5

---

Under review as a conference paper at ICLR 2020
LeNet-S Caffe on MNIST
LeNet 300-100 an MM
Dong
Goniz 2018
Layer-orise Brain Danage
Goree 2gis
via optimizaioon
3.5
Single Nat. Fruning
1.6
1.8
Gareers-Perinn Perpine 2018
2018
Soft weight sharing
Pyn. Network Surgery
Learn weig ceseccennctsoo
Variational
Test Enror
Targeted Dropout
Sporse Nomentum
Jes
Derise (100% Weights)
1.2
CarreisPPrrp 2018
Donag202
1.0
2016
iree 2017
Mo chaffos 2013
1wmn 2015
2015
10
10
Weights (16)
Weights (16)
Figure 2: Comparisons against compression methods MNIST with 95% confidence intervals.
Figure and Table show comparisons of sparse learning methods on MNIST and CIFAR that follows the experimental procedure Mostafa and Wang (2019) where some selected layers are dense. For LeNet 300-100 on MNIST, we can see that sparse momentum outperforms all other methods. For CIFAR- 10, sparse momentum is better than dynamic sparse in 4 out of 5 cases. However, in general, the confidence intervals for most methods overlap this particular setup for CIFAR-10 with specifically selected dense layers scems to be too casy to determine difference in performance between methods and we do not recommend this setup for future work. that sparse momentum outperforms all other methods on ImageNet (ILSVRC20)) accuracy measure. Dynamic sparse is better for the Top-5 accuracy with 20% weights. In the fully sparse setting, sparse momentum remains competitive and seems to find a weight distribution which works equally well for the 10% weights case. For 20% weights, the performance decreases slightly.
for the Top-1 1
Leket 300-100 MNST
95.0
on
98.6
$4.0
Test Accuracy
Test Accuncyy
98.4
93.5
92.5
Dynamic Sparse Sparse Momintum
92.9
SET
97.6
91.
Weights
Weights 1%6
Figure 3: Test set accuracy with 95% confidence intervals on MNIST and CIFAR at varying sparsity levels for LeNet 300-100 and WRN 28-2.
5.1 SPEEDUPS AND WEIGHTS NEEDED FOR DENSE PERFORMANCE LEVELS
We analyzed how many weights are needed achieve dense performance for our networks on CIFAR-10 and how much faster would we able to train such a sparse network compared to dense one. We do this analysis by increasing the number weights by 5% until the sparse network trained with sparse momentum reaches performance level that overlaps with 95% confidence interval of the dense performance. We then measure the speedup the model. For each network-density combination perform ten training runs with different random seeds to calculate the mean test error and its standard error.
To estimated the speedups that could be obtained using sparse momentum for these dense networks we follow two approaches: Theoretical speedups for sparse convolution algorithms which are proportional to reductions in FL.OPS and practical speedups using dense convolutional algorithms

---

Under review as a conference paper at ICLR 2020
Table CIFAR-IO test set error (dstandard error) for dense baselines, Sparse Momentum and SNIP
Sparse Error (%)
Model
Dense Error (%)
SNIP
Momentum
Weights (%)
AlexNet-s
12.95++0.056
14.99
14.27++0.22
10
AlexNet-b
12.85=0.068
14.50
13.56=0.094
10
VGG16-C
6.49:10.038
7.27
7.00==0.054
5
VGGI6-D
6.59:10.050
7.09
6.69-10.049"
5
VGGI6-like
6.50:10.054
8.00
7.0077
3
WRN-16-8
4.57::0022
6.63
5.62-10056
5
WRN-16-10
4.40:
6.43
5.24-
5
WRN-22-8
4.26-1-0.32
5.85
4.93+
5
95% confidence intervals overlap with dense model.
Table 2: Results for ResNet-50 on ImageNet.
Accuracy (%)
Model
Top-1
Top-5
Top-1
Top-5
Dense ResNet-50 (He et al. 2016)
74.9
92.4
74.9
92.4
Fully Sparse
10% weights
20%
Weights
DeepR (Bellec et al. 2018
x 70.2
90.0
71.7
90.6
SET (Mocanu et al.
2018)
x
70.4
90.1
72.6
91.2
Dynamic Sparse (Mostata and Wang 2019)
X 71.6
90.5
73.3
92.4
X
72.3
91.0
74.2
91.9
Sparse momentum
72.3
91.0
73.8
91.8
which are proportional to empty convolutional channels. For our sparse convolution estimates, we calculate the FLOPS saved for each convolution operation throughout training as well the runtime for each convolution. To receive the maximum speedups for sparse convolution, we then scale the runtime for each convolution operation by the FLOPS saved. While fast sparse convolution algorithm for coarse block structures exist for GPUs (Gray et al. (017), optimal sparse convolution algorithms for fine- -reiner patterns do not and need be developed enable these speedups.
The second method measures practical speedups that can be obtained with naive, dense convolution algorithms which are available today. Dense convolution unsuitable for the training of sparse networks but we include this measurement to highlight the algorithmic gap that exists to efficiently train sparse networks. For dense convolution algorithms, we estimate speedups as follows: If convolutional channel consists entirely of zero-valued weights we can remove these channels from the computation without changing the outputs and obtain speedups. To receive the speedups for dense convolution scale each convolution operation by the proportion of empty channels. Using these measures, we estimated the speedups for our models on CIFAR- 10. The resulting speedups and dense performance levels can be seen in Table
We see that VGG16 networks can achieve dense performance with relatively few weights while AlexNet requires the most weights. Wide Residual Networks need intermediate level of weights. Despite the large number of weights for AlexNet, sparse momentum still yields large speedups around 3.0x for sparse convolution. Sparse convolution speedups are particularly pronounced for Wide Residual Networks (WRN) with speedups as high as 5.61x. Dense convolution speedups are much lower and are mostly dependent on width, with wider networks receiving larger speedups. These results highlight the importance to develop optimized algorithms for sparse convolution.
Beyond speedups, we also measured the overhead of our sparse momentum procedure to be equivalent of slowdown to + compared to a dense baseline.
7

---

Under review as a conference paper at ICLR 2020
Table 3: Dense performance equivalents and speedups for sparse networks on CIFAR-10.
Speedups
Model
Weights (%)
Error %6)
Dense Convolution
Sparse Convolution (FLOPS Reduction)
(Empty Channels)
AlexNet-s
50
13.15::0.065
1.3lx
3.01x
AlexNet-b
35
13.00:10.065
1.21x
2.74x
VGGI6-C
10
6.64
1.32x
3,85x
VGGI6-D
5
6.49:E0.045
1.36x
3.51x
VGG16-like
5
6.46:40.036
1.32x
3.48x
WRN 16-8
30
4.72:40.051
1.07x
4.59x
WRN 16-10
25
4.56:L0.037
1.07x
4.41x
WRN 22-8
20
4.40
1.21x
5.61x
ANALYSIS
6.1
ABLATION ANALYSIS
Our method differs from previous methods like SET and Dynamic Sparse Reparameterization in two ways: (1) redistribution of weights and (2) growth of weights. To understand the performance contribution of these components, we perform ablations on CIFAR-10 for VGGI6-D with 5% weights, MNIST for LeNet 300-100 and LeNet-5 Caffe with 5% weights, and ImageNet for ResNet-50 with 10% weights in the fully sparse setting. The results can be seen in Table
Redistribution: Redistributing weights according to the momentum magnitude becomes increasingly important the larger a network is as can be seen from the steady increases in error from the small LeNet 330-10000000 the large ResNet-50 when no momentum redistribution is used. Increased test error is particularly pronounced for ImageNet where the Top-1 1 error increases by 3.42% to 9.7 % if no redistribution used.
Momentum growth improves performance over random growth by a large margin for ResNet-50 on ImageNet, but for smaller networks the combination redistribution and random growth seems to be sufficient to find good weights. Random growth without redistribution, however, cannot find good weights. These results suggest that with increasing network size a random search strategy becomes inefficient and smarter growth algorithms required for good performance.
Momentum growth:
Table 4: Ablation analysis for different growth and redistribution algorithm combinations for LeNet 300-100 and LeNet-5 Caffe on MNIST, VGG16-D on CIFAR- 10, and ResNet-50 on ImageNet.
Test error %
Redistribution
Growth
LeNet 300-100
LeNet-5 Caffe
VGG16-D
ResNet-50
momentum
momentum
1.5310.020
0.6910.021
6.69:1.049
27.07
momentum
random
+0.07++0022
-0.05=0.011
-0.19+00.00
+7.29
None
momentum
+0.0110.018
+0.32+0.071
+1.5440.101
+3.42
None
random +0.1110.020
+0.13+0.013
+1.49:0..147
+9.71
CONCLUSION AND FUTURE WORK
We presented our sparse learning algorithm, sparse momentum, which uses the mean magnitude of momentum to grow and redistribute weights. We showed that sparse momentum outperforms other sparse algorithms on MNIST, CIFAR-10 10, and ImageNet. Additionally, sparse momentum can rival dense neural network performance while accelerating training. Our analysis of speedups highlights the need for research into specialized sparse convolution and sparse matrix multiplication algorithms enable the benefits of sparse networks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Bellec, G., Kappel, D., Maass, W. and Legenstein, R. A. (2018). Deep rewiring: Training very sparse deep networks. CoRR, abs/1711.00 10./177..05116.
Carreira- PereiiaaPerrpnnn, M. A. and Idelbayev. Y. (2018). pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8532-8541.
algorithms for neural net
Chauvin, Y. (1988). A prok-pprpagation algorithm with optimal use of hidden units. In NIPS.
Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers. CoRR, abs/1 1111999440000 10509,
Dai, X., Yin, H., and Jha, N. K. (2017). Nest: A neural network synthesis tool based on a grow-and- prune paradigm. CoRR, abs/1711.0209
Dai, X., Yin, H., and Jha, N. K. (2018). Grow and prune compact, fast, and accurate Istms. CoRR, 0bs/1805.1179)
Deng, J., Dong. W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
Devlin, J., Chang. M.-W,, Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.0453)
Dong, X., Chen, S., and Pan, S. (2017). Learning to prune deep neural networks via layer-wise optimal brain surgeon. In NIPS.
Frankle, J. and Carbin, M. (2019). The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR 2019.
Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. (2019). The lottery ticket hypothesis at scale. CoRR, 900.00111....
Gale, T., Elsen, F.,, and Hooker, S. (2019). The state of sparsity in deep neural networks. CoRR, Abs/1902.09954.
Gray, S., Radford, A., and Kingma, P. (2017). Gpu kernels for block sparse weights.
Guo, Y., Yao, A., and Chen, (2016). Dynamic network surgery for efficient dnns. Advances In Neural Information Processing Systems, pages 1379-1387.
Han, S., Pool, J., Tran, J., and Dally, W. (2015). Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1135-1143.
Hassibi, B. and Stork, D. G. (1992), Second order derivatives for network pruning: Optimal brain surgeon. In NIPS.
He, K., Zhang, X., Ren, S., and Sun, (2016). Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Herculano- Houzel, S., Mota, B., Wong, P., and Kaas, J. H. (2010). Connectivity-drive white matter scaling and folding in primate cerebral cortex. Proceedings of the National Academy of Sciences of the United States of America, 107 44: 44:19008-13:
Ishikawa, M. (1996). Structural learning with forgetting. Neural Networks, 9:509-521.
Karnin, (1990). simple procedure for pruning back-propagation trained neural networks.
IEEE transactions neural networks, 2:239 42.
Krizhevsky. A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical report, Citeseer.

---

Under review as a conference paper at ICLR 2020
LeCun, Y. (1998). Gradient-based learning applied to document recognition.
LeCun, Y., Denker, J. S., and Solla, S. A. (1989). Optimal brain damage. In NIPS.
Lee, N., Ajanthan, T., and Torr, P. H. S. (2019). Snip: Single-shot network pruning based on connection sensitivity. In ICLR 2019.
Louizos, C., Ullrich, K., and Welling, M. (2017). Bayesian compression for deep learning. In Advances in Neural Information Processing Systems, pages 3288-3298.
Louizos, C., Welling, M., and Kingma, D. P. (2018). Learning sparse neural networks through lo regularization. CoRR, abs/1712.01312.
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, H., Gibescu, M., and Liotta, A. (2018). Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 91):23383.
Molchanov, D., Ashukha, A., and Vetrov, D. P. (2017). Variational dropout sparsifies deep neural networks. In International Conference MachineLearning (ICML).
Mostafa, H. and Wang, X. (2019). Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning (ICML).
Mozer, M. C. and Smolensky, P. (1988). Skeletonization: technique for trimming the fat from a network via relevance assessment. In NIPS.
Narang, S., Diamos, F., Sengupta, S., and Elsen, E. (2017). Exploring sparsity in recurrent neural networks. CoRR, abs/1704.05119.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., De Z., Lin, Z., Desmaison, A., Antiga,
L., and Lerer, A. (2017). Automatic differentiation in pytorch.
Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural networks: the journal of the International Neural Network Society, 12 1:145-151.
Simonyan, K., Vedaldi, A.. and Zisserman, A. (2013). Deep inside convolutional networks: Visualis-
ing image classification models and saliency maps. CoRR, abs/1312.6034,
Springenberg. J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A. (2014). Striving for simplicity: The all convolutional net. CoRR, abs/14 141.6/66...
Ullrich, K., Meeds, E.,, and Welling, M. (2017). Soft weight-sharing for neural network compression. CoRR, abs/1702.04000
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998 3998-6008.
Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In ECCV.
Zhou, H., Lan, J., Liu, R., and Yosinski, J. (2019). Deconstructing lottery tickets: Zeros, signs, and the supermask. arXiv preprint 1905.000
Zhu, M. and Gupta, S. (2018). To prune, or not to prune: Exploring the efficacy of pruning for model compression. CoRR, abx/1710.01878.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 SENSITIVITY ANALYSIS
Sparse momentum depends on two hyperparameters: Prune rate and momentum. In this section, we study the sensitivity of the accuracy of our models as we vary the prune rate and momentum. Since momentum parameter has an additional effect on the optimization procedure, we run control experiments for fully dense networks thus disentangling the difference in accuracy accounted by our sparse momentum procedure.
We run experiments for VGG-D and AlexNet-s with 5% and 10% weights on CIFAR-10. Results can be seen in Figure 4 We see that sparse momentum is highly robust to the choice of prune rate with results barely deviating when the prune rate in the interval between 0.2 to 0.4. However, we can see gradual linear trend that indicates that smaller prune rates work slightly better than larger ones. Cosine and linear prune rate annealing schedules do equally well. For momentum, confidence intervals for values between 0.7 and 0.9 overlap indicating that procedure is robust to the choice of the momentum parameter. Sparse momentum is more sensitive to low momentum values (50.6) while it is less sensitive for large momentum values (0.95) compared to dense control. Additionally, we test the null hypothesis that sparse momentum is equally sensitive to deviations from momentum parameter value of 0.9 as a dense control. The normality assumption was violated and data transformations did not help. Thus we use the non-parametra Wilcoxon Signed-rank Test. We find no evidence that sparse momentum more sensitive the momentum parameter than dense control, W (16) = 22.0, 0.58. Overall, we conclude that sparse momentum is highly robust to deviations of the pruning schedule and the momentum and prune rate parameters.
Prune Rate Parameter Sensiivitty
Nomentum Parameter Sensitivity
VGG Dense
7.4
12
Eroo
Cosine annealing
Unear srnealing
10
V-G16-D
6.4
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.50
0.60
0.70
0.80
0.30
0.85
Prune Rate
Momentum
Figure 4: Parameter sensitivity analysis for prune rate and momentum with 95% confidence intervals.
B ADDITIONAL ANALYSIS
B. DENSE VS SPARSE FEATURES
Are there differences between feature representations learned by dense and sparse networks? The answer to this question can help with the design of sparse learning algorithms and sparse architectures. In this section, we look at the features of dense and sparse networks and how specialized these features are for certain classes. We test difference between sparse and dense network features statistically.
For feature visualization, it is common to backpropagate activity to the inputs to be able to visualize what these activities represent (Simonyan et al.) 2013 Zeiler and Fergus 2014; SSpringenberg et al.. 2014). However, in case, each layer within our network, and as such we want to look at the magnitude the activity in a channel since unlike feature visualization diseriminators. For example, a face detector would induce positive activity for a 'person' class but might produce negative activity for 'mushroom' class. Both kinds of activity are useful.
more interested in the overall distribution of features for
not just interested in feature detectors but also
11

---

Under review as a conference paper at ICLR 2020
With this reasoning, develop the following convolutional channel-activatto analysis: (1) pass the entire training set through the network and aggregate the magnitude of the activation in each convolutional channel separately for each class; (2) normalize across classes to receive for each channel the proportion of activation which is due to each class; (3) look at the maximum proportion of each channel measure of class specialization: maximum proportion of 1/N, where No is the number of classes indicates that the channel is equally active for all classes in the training set. The higher the proportion deviates from this value, the more is channel specialized for partieular class.
We obtain results for AlexNet-s, VGG16-D, and WRN 28-2 on CIFAR-10 and use as many weights as needed to reach dense performance levels. We then test the null hypothesis, that there are no differences in class specialization between features from sparse networks and dense networks. Equal variance assumptions was violated for VGG-D and normality was violated for WRN-28-2, while all assumptions hold for AlexNet-s. For consistency reasons we perform non-parametric Kruskal Wallis one-way analysis of variance tests for all networks. For AlexNet-s, we find some evidence that features of sparse networks have lower class specialization compared to dense networks x5) = 4.43,7 p = 0.035, for VGG-D and WRN-28-2 we find strong evidence that features of sparse networks have lower class specialization than dense networks x²(13) 28.1, P < 0.001, (12) = 36.2, p < 0.001. Thus we reject the null hypothesis. These results increase our confidence that sparse networks learn features which have lower class specialization than dense networks.
Plots of the distributions of sparse vs. dense features for AlexNet-s, VGG16-D, and WRN 28-2 on CIFAR-10 10 in Figure [ These plots were selected to highlight the difference in distribution in the first layers and last layers of each network. We see the convolutional channels in sparse networks have lower class-specillization indicating they learn features which are useful for broader range of classes compared to dense networks. This trend intensifies with depth.
Overall, we conclude that sparse networks might be able rival dense networks by learning more general features that have lower class specialization.
C FURTHER RESULTS
C.1 TUNED RESNET-50 ON IMAGENET
We also tried a better version of the ResNet-50 in the fully sparse setting for which we use cosine learning rate schedule, label smoothing of 0.9, and warmup the learning rate. The results can be seen in Table 5
Table 5: Fully sparse ImageNet results.
Accuracy (%)
Model
Weights (%)
Top-1
Top-5
Tuned ResNet-50
100
77.0
93.5
10
72.9
91.5
Sparse momentum
20
74.9
92.5
30
75.9
92.9
D DETAILED SPARSE MOMENTUM ALGORITHM
For detailed NumPy-style algorithmic description of sparse momentum see Algorithm
12

---

Under review as a conference paper at ICLR 2020
AlexNet Conv2D Layer
AlexNet Conv2D Layer 5
Spa
Sparse
0.40
0.90
0.33
0.33
0.30
0.30
0.23
0.25
VGE Conv2DD Layer
VGG Conv2D Layer
Donse
Spa
Dense
Sparse
0.50
0.50
0.45
0.48
0.40
0.40
Cles Specialictio
0.25
0.20
0.26
0.15
0.18
0.30
0.16
10
Cheonel Count
ChennelCCom
WRN 28 Conv2D Layer
WRN 28 Conv2D Layer 23
Sparse
Sparse
0.40
0.40
0.33
0.33
0.30
0.30
0.23
0.25
Figure 5: Dense vs sparse histograms of class- specialiaeaiinoo for convolutional channels on CIFAR-10 10. class-ppecialization of 0.5 indicates that 50% the overall activity comes from a single class.
13

---

Under review as a conference paper at ICLR 2020
Algorithm 2: Sparse momentum algorithm in NumPy notation.
Data: Layer i to with: Momentum M,. Weight W,, binary Mask,; prune rate p
TotalMomentum O, TotalNonzero 0
** (a) Ca culate mean moment.um um contributions of all layers.
*/
for i f- to & do
MeanMomentum; + mean(abs M, [W; * 0)))
TotalMomentum TotalMomentum MeanMomentum; NonZero; = sum W; & 0)
TotalNonzero TotalNonzero NonZero,
end
for i + 0 to k do
LayerContribution, + MeanMonnentunn/TTuuMMomommmmmm
10
f getPruneRate(W ,. weights by finding the NumRemoveth smallest weight.
11
12 end
13 for i + to k do NumRemove; <- NonZero, p
14
PruneThreshold + sort abs W; [W; * #])) [NumRemove]]
15
16 Mask; [W, < PruneThreshold] <- 0 /// Stop gradient flow.
W; [W < PruneThhreshlle + 0
18 end
(c) Enable gradient flow of weights with largest momentum magnitude.
19
for: 4- to do
RegrowthTTrreshold +- sort(abs M; W; == 0))) [NumRegrowth,)
21
Z. = M, (W; = 0) // Only consider the momentum of missing weights.
Mask; + Mask (Z, > Regrowthhhrreshodd)) // | is the boolean OR operator
22
end
24 p- f PccaayPun))
applyMask()
14


Reviews:
[{"review_id": "SJgPTmQ5FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a method to train a sparse network and achieve \"dense-level\" performance. The method redistributes the sparsity according to momentum contribution of each layer during training after each epoch. Experiments on multiple datasets and architectures are conducted. \n\nWhile the method itself seems interesting, I do have have several important concerns.\n\n1. My biggest concern is about the experiments. Some results sometime seem questionable.\n\n1) Looking at the results for ImageNet ResNet50, the 20%-weights model achieve 0.7% test error loss compared with dense baseline. According to Table 6 in [1], using the simple pruning and fine-tuning method (Han et al. 2015), the 40%-weights model suffers from 0.06% error loss compared with dense baseline. I don't see a clear advantage of sparse momentum here. It seems possible that under the same sparsity level just pruning using Han et al. is better. Can the authors compare with Han et al. 2015 fairly at the same sparsity level, on multiple datasets? An apple-to-apple comparison is extremely necessary. \n\nAlso the baseline result differ: in [1] dense ResNet-50 achieve 76.15% but in this paper baseline result is 74.9%. Both papers use Pytorch. And Pytorch official number is actually 76.15% (see https://pytorch.org/docs/stable/torchvision/models.html ). In my experience that is reproduceble if you follow the official pytorch code of training ImageNet. What's the difference here? I think the authors should follow the standard way of training ImageNet to make the results more convincing.  Finally [1] is a relevant paper and should be discussed.\n\n2) I suggest that MNIST results be moved to Appendix since accuracy on MNIST is too easy to reach a high level, and the interpretation over it might not be convincing. Yet a lot of the analysis of the method's effectiveness is on MNIST.\n\n3) For CIFAR, the results of SNIP on VGG in Table 1 also seems inconsistent with their original paper. In their paper's Table 2, the VGG-like model achieve ~0.3% error reduction at 5% weights while in this paper's Table 1 it's a 0.5% error increase. Again, what is the difference? Is the SNIP method reimplemented? Putting that potentially flawed result aside, the performance on CIFAR is not better than other methods as well according to Figure 3.\n\n2. I don't think the general performance of the trained sparse models can be said to \"rival\" the dense model. It's a non-negligible margin in most times, for CIFAR and ImageNet. It's a little exaggerating to say that in the title, abstract and introduction.\n\n3. The benefit of the method seems unclear. It does not speedup training compared with the conventional training a dense model and then pruning pipeline (Han et al.). The test-time real speedup is also limited according to Table 3. The real benefit is the compressed model but that is also achievable by traditional pruning (Han et al.). \n\n4. The ablation study should compare with smarter baselines than random regrowth/no redistribution. Clearly each layer needs different level of sparsity so no redistribution is of course not a competitive one. But there might exist other criterion (e.g, weight magnitudes) than momentum which gives good results. The ablation study did not demonstrate why using momentum is justified.\n\n5. \"For dense convolution algorithms, we estimate speedups as follows: If a convolutional channel consists entirely of zero-valued weights we can remove these channels from the computation without changing the outputs and obtain speedups.\" How is it possible that all weights associated with a channel are all pruned, in a sparse pruning setting? A channel typically has at least hundreds or even thousands of weights connected with it, so even with 95% sparsity it's extremely unlikely (consider 0.95^1000). The estimate of speedup might be flawed.\n\nIn summary, those serious issues with experiments make me vote a rejection for the paper.\n\n[1] Rethinking the Value of Network Pruning, ICLR 2019.\n\n------------------------------------------------------------------------------------\nAfter author rebuttal:\n\nThank you for the response.\n\nThe authors view that their method is not comparable with Han et al. 15. because this work started training a sparse model while Han et al. 15 trains a dense model and then do pruning. But I'm not sure whether the rather limited actual (1.2-1.3x, and lower on Wide ResNet) speedup justify this argument. What's more, these speedup were said to be \"estimated\" so I'm not sure whether it's actual speedup. \n\nThe pruning ratio with dense performance is not remarkable: 50% sparsity ratio should be easily achievable with Han et al.\n\nExperiments on ResNet (not wide ResNet) were not provided and I assume ResNet would be harder to sparsify than AlexNet, VGG, Wide ResNet which may have more redundancy. I think thus the \"faster\" training part in the title is not entirely justified. \n\nI'm ok with other parts of the rebuttal and I will raise my score to weak reject. But I still wonder why we should use sparse training in the first place if it brings so little \"estimated\" speedup.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkg1FkmQqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a sparse momentum algorithm for doing efficient sparse training. The technique relies on identifying weights in a layer that do not have an effect on the error, pruning them, and redistributing and growing them across layers. The technique is compared against other recent algorithms on a range of models.\n\nThe paper is well written, and very easy to read. The proposed algorithm looks interesting and seems to empirically work well. \n\nI am, however, a bit confused with how the sparse momentum algorithm is discussed in this paper. In the paper, momentum is intuitively presented as an algorithm that reduces the variance of the noise in the gradients. However, there are a number of papers that show that this is not the case (and if anything it is the opposite). Further, a few recent papers show that momentum works better than SGD only for learning rates that are not too small, and this is because it averages out the gradients in the high-curvature directions (these are the directions where the gradients switch signs) and makes them stable in these directions, thus allowing larger steps in the low curvature directions. See for example the following two papers:\nMomentum Enables Large Batch Training. Samuel L Smith, Erich Elsen, Soham De. ICML Workshop on Physics for Deep Learning, 2019.\nWhich Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model. Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E Dahl, Christopher J Shallue, Roger Grosse. NeurIPS 2019.\n\nGiven these papers, can the authors comment on what they think is the reason for the effectiveness of their sparse momentum algorithm? These papers seem to indicate to me that an interesting (and important) ablation study would be to compare using just the gradients vs using momentum for the sparse training algorithm for both a small batch and a large batch. Without this ablation study, it is a bit unclear to me why/when this algorithm is working well, and this primarily explains my current score.\n\nThe experimental results look impressive, although I am not very aware of other work in sparse training, so unfortunately it is harder for me to properly evaluate the significance of the empirical results in this paper, and whether the numbers reported are indeed a significant improvement over current state-of-the-art algorithms. I do have a few questions about the design and the results of the experiments presented:\n\n1. Because the momentum of zero-valued weights are used, does that mean that the gradient over all weights are required to be taken at each step? How much does this affect training time in your experiments?\n\n2. How were the learning rates decided, and why are they kept fixed across methods? It seems feasible that the optimal learning rate could vary highly between methods?\n\n==========================================\n\nEdit after rebuttal:\nI thank the authors for the very detailed response and for doing the additional experiments requested. Due to the thoroughness of the response, I am increasing my score to a weak accept. I do however have a couple of comments regarding the author response:\n\n1. I do not agree that keeping the learning rate fixed across methods is the right approach. Many different things might interact with each other to change the optimal learning rate, and it is always more interesting to see the optimal performance attained by a method through a grid search. As long as the tuning budget across all methods are somewhat similar, I think the experiments would be more informative when doing a learning rate sweep. Further, while doing the sweep, the authors should be careful that the optimal learning rate does not lie on the edge of the grid search. This is what happens in the experiments reported in the authors' response titled \"Results from learning rate grid search\" where the optimal learning rate for all models lie at one of the extremes of the grid search (0.09).\n\n2. I would request the authors to slightly rewrite certain parts of their paper so as not to imply that momentum decreases the variance of the gradients in general. There are many papers (including the ones I mentioned in my review) that specifically show momentum does not have this effect. All of these papers however consider a different setting from the one considered in this paper (where there is a parameter redistribution step), so it is not immediately clear whether momentum has the same effect in this case. However, unless there is very specific evidence to show a variance reduction effect, I think putting in that intuition might be misleading. \nIt is interesting to see that momentum helps for small batches in this setting, and I agree that further investigation into this would be an interesting future direction. Note that there is a concurrent submission on sparse training (https://openreview.net/forum?id=ryg7vA4tPB&noteId=ryg7vA4tPB) that seems to show that higher momentum values (0.99) does better when considering large batches. So perhaps additional and more careful experiments need to be done about this before these additional experimental results can be put in the paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxWWfORYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm called Sparse Momentum for learning sparse neural networks. They claim to maintain sparse weights throughout training while achieving dense network performance levels. They also show their method improves training speed up to 5.61x faster training. The provides a decent motivation for why sparse networks can be helpful. The related work section is well summarized and they emphasize that the current work's primary motivation is to reduce training time while maintaining performance. They compare their method with other methods that also maintain sparse neural networks throughout training and involve single training phase, which is fair. Their method consists of primarily 3 phases - i) pruning weights ii) redistribution of weights iii) regrowing of weights based on the exponentially smoothed momentum term for each layer. The method is well explained and motivation is clear. Edge case was also explained for more clarity. However, there is something that needs more clarification in Pg-3, 3rd line, they say the 3 components of the algorithm i) ii) and iii) can be tackled independently with a divide and conquer strategy to gain some computational benefits. In my understanding, the method performs these 3 steps sequentially after each epoch. Not sure how divide and conquer strategy can be used here ?\n\nThe results were shown for MNIST & CIFAR-10 using AlexNet, VGG16 and LeNet-5 models. They show that the current algorithm is better than other proposed methods in the literature in most of the cases. The claims made in some cases that this method reaches dense network performance is not completely true though. For example, in Table 1, the only case where the proposed method reaches dense network's performance is for VGG16-D. In all the rest of the cases, the current method and dense network error differ by at least 5%. \n\nThe authors compare speed up results for training in two ways: theoretical speedups which are proportional to reduction in number of FLOPS and practical speedups using dense convolutional algorithms corresponding to completely empty channels. It's good that the authors have mentioned due to the current lack of optimal sparse matrix multiplication implementations these speedups cannot be garnered practically yet. The estimates based on FLOPs reduction or Empty Channel based look promising. They also show ablation study of how the redistribution and weight regrowth based on momentum is better than doing in a random fashion.\n\nOverall, I think the paper proposes an interesting idea of using momentum with promising results to learn sparse neural networks. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for training sparse neural networks that also provides a speedup during training, in contrast to methods for training sparse networks which train dense networks (at normal speed) and then prune weights.

The method provides modest theoretical speedups during training, never measured in wallclock time.   The authors improved their paper considerably in response to the reviews.  I would be inclined to accept this paper despite not being a big win empirically, however a couple points of sloppiness pointed out (and maintained post-rebuttal) by R1 tip the balance to reject, in my opinion.  Specifically: 

1) "I do not agree that keeping the learning rate fixed across methods is the right approach."  This seems like a major problem with the experiments to me.

2) "I would request the authors to slightly rewrite certain parts of their paper so as not to imply that momentum decreases the variance of the gradients in general."  I agree.
Decision: Reject