Starfire: Regularization-Free Adversarially-Robust Structured Sparse Training
Authors: Noah Gamboa
Kais Kudrolli
Anand Dhoot
Ardavan Pedram
Venue:  (2020)
Keywords: Structured Sparsity
Sparsity
Training
Compression
Adversarial
Regularization
Acceleration

Abstract: This paper studies structured sparse training of CNNs with a gradual pruning technique that leads to fixed, sparse weight matrices after a set number of epochs. We simplify the structure of the enforced sparsity so that it reduces overhead caused by regularization. The proposed training methodology explores several options for structured sparsity.

We study various tradeoffs with respect to pruning duration, learning-rate configuration, and the total length of training.
We show that our method creates a sparse version of ResNet50 and ResNet50v1.5 on full ImageNet while remaining within a negligible <1% margin of accuracy loss. To make sure that this type of sparse training does not harm the robustness of the network, we also demonstrate how the network behaves in the presence of adversarial attacks.  Our results show that with 70% target sparsity, over 75% top-1 accuracy is achievable. 

Under review as conference paper at ICLR 2020
STARFIRE: REGULARIZATION FREE ADVERSARIALLY ROBUST STRUCTURED SPARSE TRAINING
Anonymous authors authors Paper under double-blind review
ABSTRACT
This paper studies structured sparse training of CNNs with gradual pruning technique that leads to fixed, sparse weight matrices after set number epochss. We simplify the structure of the enforced sparsity so that it reduces overhead caused by regularization. The proposed training methodology explores several options for structured sparsity.
We study various tradeoffs with respect to pruning duration, learning-racd rate con- figuration, and the total length of training. We show that method creates a sparse version of ResNet50 and ResNet50v1.5 on full ImageNet while remaining within negligible <1% margin of accuracy loss. To ensure that this type of sparse training does not harm the robustness of the network, we also demon- strate how the network behaves in the presence of adversarial attacks. Our results show that with 70% target sparsity, over 75% top-1 accuracy is achievable.
INTRODUCTION
Pruning weights can compress network into smaller model so that the model can fit into faster/smaller memory and therefore result in execution speedups (Han et al. 2016 2015a To increase the accuracy Han et al. 2015b) Mao al. [011 explore training the network dense after pruning. The resulting network can maintain accuracy based on the specified level of
sparsity Mostafa & Wang 2019 Zhu & Gupta 2017 Han et al. 2015a).
Structured sparsity has been explored for RNNs and also CNNs where certain number of non- zeros is allowed across various cross-sections the weight tensors. These methods aim to speed up computation and reach some final level of sparsity for deployment. Narang et al. (2017) have shown promising results for structured training of RNNs while sparse CNNs could not achieve the same performance (Mao al. 2017).
Recent work has demonstrated that structurally sparse training can speed up execution on GPUs (He et al. 2017 Lym et al. 2019 Zhu & Guptal 2017). However, these training mecha- nisms add regularization and computational overhead to eliminate unnecessary weights. The regularization
modifies the original training and can be expensive in hardware. While forcing coarse- graii sparsity Lym et al. (2019) provides significant speedups, the final network contains an insufficient degree of sparsity for deployment on edge devices.
Mostafa & Wang (209)) show that with adaptive sparse training and dynamic reallocation of non- zeros sparsity levels up 80% can be achieved. However, even though an additional 10 epochs of training are required, an accuracy loss of around 1 1-2% is still observed. The main drawback is the overhead incurred while implementing such technique on the target platform. Continuous reconfiguration of the sparsity pattern is expensive as it does not allow for compression of weights during training.
To achieve speedups and a desired final degree of sparsity, we aim to apply the techniques in Han et al. (2015b) and Mao et al. (2017) at earlier stages in training at higher frequency within a period which we call the pruning era, usually a period of 20-30 epochs. During the pruning era, with fine granularity of at most kernel size, we exploit one of the three proposed sparsity regimes. Subsequently, we fix the mask for the rest of the training speed it up. Our motivation came from the insight that having a fixed sparse multilyy-cccumulate accumulate pattern allows weight compression during training and
compute and energy in hardware (Han et al. 2016).

---

Under review conference paper at ICLR 2020
Sparsity
Train Dense
Prunp
Strict Parameter Budget with Dynamic Reparameterization
Dynamic Sparse Reparameterization
80% %
Adaptive Sparse
60-70 %
Training
Dense
Pruning Era
Strict Parameter Budget
This work
Next Batch
Pruning Era
25-45
Prunetrain
Reac/ed
Yos
Epochs of Training
Train With Fixe Sparso Mask
0-20 40:50
90-100
(a)
(b)
Figure 1: (a) Our general sparsity mechanism in which we update the sparsity mask after each batch until reach desired level of sparsity. (b) Schedule comparison between this work, Mostafa & Wang (2099), and Lym et al. (2019). Our work has the shortest pruning era and more gradually reaches our final sparsity.
We explore the impact of various pruning granulariie,, sparsity levels, and learning-rate schedules on the network's convergence as well as adversarial robustness for CNNs like Resnet-50 (He et al. 2015 on ImageNet and tinylmagenet CCS231N 2015
Recent literature has shown that adversarial attacks are more successful on pruned neural net- works than they are on regular neural networks [Wang al. 2018). Given the danger of adversarial attacks in real world situations, we find that it is important to evaluate our sparsity techniques un- der adversarial robustness. We leverage the FGSM mechanism [Goodfellow et al. (014) evaluate the adversarial robustness on our sparse models. This paper makes the following contributions:
1. We propose a mechanism to train and prune a convolutional network during the earlier stages of training such that this sparsity can be harvested for the computational speedups. To do this, we fix the sparse weight masks for the remainder of the training.
2. For fully connected sparsification, we eliminate blocks of fully connected weights based on their connection the zeros in the previous convolutional layer.
3. We enforce structural, regularization free, magnitude sranitude-based pruning across two distinct dimensions and a combined version. These dimensions inside convolution window R x S and across input/output feature matrix (CK).
4. Our sparse models are as robust to adversarial FGSM attacks as fully dense models.
5. We demonstrate that early stage dense training is crucial for maintaining high accuracy.
6. The proposed technique is tolerant to sparsity levels of up to 60-70% with under 1 1% accuracy degradation. We can compensate by scheduling an extra learning rate drop and training for extra 10 epochs.
The rest of the paper is organized as follows. Sextions our pruning methodology. Section describes the experimental setup framework. Section 4 presents results and discusses their interpretation. DectionS3presee the related work. Section Gloncludee the paper.
PRUNING METHODOLOGY
Our proposed pruning mechanism works by always pruning the weights of smallest magnitude after each weight update. After a forward and backward pass (one batch update), the model is pruned. a weight already zero, the gradient is also set to zero. This means that once a weight becomes zero, it will remain zero for the rest of the training period.
This mechanism is similar to Han et al. 22015b), except that we only prune in the earlier stages of the training opposed to post training. Additionally, this work is similar to Narang al. (2017) although we set the sparsity threshold instead of using a heuristic to calculate it. We chose this pruning mechanism because of its negligible computational overhead.
2

---

Under review conference paper at ICLR 2020
Output Channel
Input Channel C
FC_Our
FC_Out
FC_in
FC_in
(b)
(c)
(d)
Figure This figure illustrates sparsity across different weight dimensions and different granulari- ties. (a) Window sparsity, which prunes the 5 smallest weights in a ax3 window. (b) CK pruning where whole RxS convolutional kernels are pruned. (c) Sparsity fully-connected weights. (d)
Block sparsity in fully-connected layer.
In our pruning algorithm, the sparsity threshold refers to the percentage of weights in the network that are currently pruned. Before during the first epoch of pruning, will have sparsity threshold of zero. As we continue training, we gradually increase the sparsity threshold so that by the final epoch pruning the network sparsity will have reached our final, desired threshold. Finally, we also define the pruning era to be the epochs between the first and final epochs of pruning depicted Figure
We evaluate the pruning mask after every training step until we reach the final epoch of pruning. After the final epoch, the pruned values the network will remain zero for the rest of training: no new pruning will occur, and only the non-zero weights will be updated.
2.1 PRUNING METHODOLOGY BY LAYER
Pruning the smallest magnitude weights in the entire network is inefficient because involves sorting the weights over the network. Instead, we prune the smallest magnitude weights or sum of weights, within a certain locale of the network. When pruning, we examine each layer individually and apply separate technique evaluate which weights to prune, depending on the type of layer we are currently pruning.
2.1.1 CONVOLUTIONAL LAYER PRUNING
Window pruning for 3x3 Convolutional Layers Figure 2a shows the result of a pruned 3x3 convolutional weight tensor under the window pruning method. this scheme, window layer pruning refers to pruning of weights within the x33 convolution kernels. We allow maximum number of non-zero values for each kernel the 3x3 convolutional layers and eliminate the weights of smallest magnitude.
Algorithm 1 CK Pruning Algorithm
sparsity_mask(Ah) for 0 in flayer do for all do for all k K do
oparsity_threshold)
kernel max,, max
end for cutoff_index = size(0,) seaarstt__treshhete
max(cutoff index, size(0,)) ) max_non non_zero 1)
cutoff. value =
largest value in kernel_max.
for all k in K do mask = 1 if kernel kernnel_ax,,ee cutoff_ else 0
end for
end end for
CK Pruning Methodology Figure 2b shows the result a pruned 3x3 convolutional weight tensor under the CK pruning method. In this scheme, the weights of certain layer can be viewed
3

---

Under review conference paper at ICLR 2020
Output Size
Channel
Output
GEMV
Weight
output
Flatten
POK
Matrix
PQ. K
Can beib
PQK
Input Channel
Figure 3: FC pruning taking advantage of the sparsity in the previous convolution layer. as CK matrix of RxS kernels. The CK pruning method involves pruning the 3x3 convolutions along the channel and kernel dimensions of each convolutional filter, i.e., we prune whole kernels (CK matrix of R windows) once and can ultimately prune all the input channels in an output channel. As defined by Algorithm [ we determine which filter to prune by examining the max of the magnitudes of all the weights in kernel, which is the max of nine weights. This max is used to evaluate whether the whole kernel should be pruned or not.
Combined Pruning Methodology To combine window and CK pruning, we introduce an intra- epoch combined pruming method, which we refer to hereafter as "intra-epor for short. As shown by appendix Algorithm in the Appendix, in a given epoch we first apply window pruning to each 3x3 convolutional layer at a fraction of the sparsity threshold for that epoch. Then, we prune the remaining fraction of the sparsity threshold with CK Pruning.
pruning" or "intra",
2.1.2 FULLY CONNECTED PRUNING
Like pruning for convolutional layers, we apply a two-tier pruning scheme from Mao al. (2017) for fully connected layers: micro-level pruning within block and macro-level pruning that eliminates entire blocks.
Block FC Pruning 2d refers pruning of individual blocks. Here, we prune an entire nxn (n<5) window within the dense layer and create coarse grained sparsity. To do this, we sum the magnitude of the weights each window and prune the windows with the smallest magnitude.
Fine FC Pruning Figure 2c refers to the pruning of individual weights. Here, prune the individual weights in the entire FC Layer, where we compare the magnitude of all the weights to each other.
The produced zero patterns in the last convolution layer allow for eliminating more weights in fully connected layer depicted in Figure If all the C windows for specific K; zeros, the output activation for the corresponding K; is also zero. The corresponding neurons in the following fully connected layer therefore receiving zero input activations and can be eliminated along with their associated weights. This enables us to get sparsity without having to evaluate the weights in the fully connected layer.
When pruning just the small weights in the FC layer, one can inadvertently cut off relevant con- nections between the input and output layers. Accordingly, we structure the pruning mechanism such that each output neuron should be influenced by the input. This means every column in the weight matrix of the fully connected layer Figure has least one non-zero element.
EXPERIMENTAL SETUP
To validate each type of pruning (window, CK, or intra-epocks we selected ResNet50 (He et 2015) vl and v1.5 with the ImageNet and/or -mng-emeeeNet [CS231N] 2015) datasets. We evaluated each pruning method by varying sparsity levels and pruning era.
We experimented with ResNet50 v1.5, in addition vl, to explore how changing the network structure would affect the top-1 accuracy. For window pruning, we tested with ResNet50v1 on Imageet well ResNet50vl and v1.5 ImageNet compare the impact of strided convolutions on our sparse training. Also, we experimented with the learning rate schedule of the training regime. Our typical schedule for ResNet50vl. 1.5 included learning rate drops at epochs 30,
4

---

Under review as conference paper at ICLR 2020
60, and 90, but experimented with placing the last drop epoch 80 instead. Unlike typical ResNet50 training, which uses a batch size of 256 and starts the learning rate at 0.1, we used batch size 64 as this is what could fit in our GPUs. As suggested by Krizhevsky (0144, we scaled the
starting learning rate by 2 to 0.05 in order to compensate for the smaller batch size.
3.1 SPARSE TRAINING EXPERIMENTS
For ResNet50v1 and Tiny-ImageNet, we did gradual pruning until epoch 10. We subsequently enforced the final sparsity requirement, set a maximum number of non-zero values each window/ kernel of each layer, and fixed this sparsity pattern for the rest of training. We chose the 10th epoch as the final epoch of pruning because we wanted to see if we could fix the sparsity mask early in the training process.
For ResNet50v1 and ImageNet, our goal was to start pruning as early as possible, while maintaining that 30th epoch would
high accuracy. We set our pruning era epochs 0-30. Our hypothesis be a suitable epoch to stop pruning, because this is where the learning rate is first decreased, in addition, there would be large drop in accuracy we stop pruning at epoch 20. However, this schedule did not perform well for ResNet50v1.5 and ImageNet and therefore we set our pruning era 30-50.
To test training using CK and intra- eetra-epoch pruning, we mainly used ResNet50vl and ResNet50v1.5 with ImageNet, but also performed CK pruning on ResNet50v1 and Tiny- ImageNe. We adopted similar approach to Han et al. (2015b) to train with CK or intra-epoch pruning by setting the first epoch of pruning to 20, 30, or 40 with a pruning era 20 or 30 epochs. Then, we continued to train the sparsified network until the final epoch.
3.2 ADVERSARIAL ROBUSTNESS
Since there was evidence that increasing sparsity lowers adversarial robustness (Wang et al. 2018 evaluated this robustness in models. To do so, we applied Fast Gradient Sign Method (FGSM) attacks defined in Goodfellow et al. (2014) on one of our sparse models, to generate its own adversarial examples, and measured the validation accuracy again. We used the same validation set as ImageNet and applied the attack's image transformation to each input image. Moreover, we experimented with variety of different in order to see how our accuracy decayed. Lastly, in our experiments we leveraged the examples provided in Pytorch tutorials
4 RESULTS
4.1 RESNET50 ON TINY-IMAGENET
From our experiments with Tiny-Imagen (shown Appendix in Table we see that even with up to 80% sparsity, both window and CK pruning able to achieve levels of accuracy comparable to the dense baseline. CK pruning performs even better than the baseline.
4.2 RESNET50 ON IMAGENET
Our ResNet50 v1.5 experiments (Table Appendix Figure 11 with the first epoch of pruning at epoch 30 show that all of our pruning methods are able to achieve over 73% accuracy, and we can achieve above 74% accuracy up 70% sparsity, Table Iablessshows that on ResNet50 v1, our methods can achieve between 0.1-0.3% less than the baseline.
By comparing the sparsity curves of the window, CK, and intra-epoch pruning runs in Figure (top right), we observe that the sparsity of window pruning is not as smooth as the other methods. This is likely indicative of the more rigid structure of CK and intra-epoch pruning, which causes the degree of sparsity to be much more uniform from epoch to epoch.
Figure (top left, bottom right) also shows on ResNetvl. 1.5, the window is slightly better than the CK and intra, which have similar performance, but the window is worse than the other two
.1mmm
5

---

Under review conference paper at ICLR 2020
Top-1 Accuracy
Network Sparsity
0.6
Window, Skart
CK,, Start 20
0.4
CI, Start 30
Dense CK, Start 30
Intm, Start
Windoww, Start 30
Intra,
CKC, Start 40
Windows
CKK Start 40, Era 30 Intra, Start 40, Fra
100
20
40
60
100
Epoch
Epoch
Top-1 Accuracy
29
Top- 1 Accuracy
Dense Window, Start
CK, Start 40,
Dense
70
Intra, Start 40,
70
CK, 20
Windoww, Start v1.5
CK, Start
CKK Start v1.5
CK, Start 40
65
Intra, Start 40, v1.5
80
Epoch
Epoch
Figure 4: top left: Convergence plots of all pruning methods with first epoch pruning 30, top right: Sparsity plot of all methods, bottom left: and CK starting pruning at different epochs, bottom right:
Resnet and v1.5 at 60% sparsity for epochs.
Model Sparsity (%)
40
60
70
Dense (76.29)
CK, start 40
75.82 (-0.46)
75.33 (-0.95)
74.92 (-1.36)
74.16 (-2.12)
CK, start 30
75.84 (-0.45)
(-0.45)
75.12 (-1.17)
74.72 (-1.56)
73.66 (-2.63)
CK, start 20
75.55 (-0.74)
74.71 (-1.58)
74.32 (-1.96)
72.79 (-3.50)
Intra, start 40
75.89 (-0.39)
75.38 (-0.90)
75.07 (-1.21)
74.02 (-2.27)
Intra, start 30
75.84 (-0.45)
75.16 (-1.12)
74.48 (-1.80)
73.66 (-2.63)
Intra, start 20
75.72 (-0.57)
74.75 (-1.53)
74.26 (-2.02)
72.97 (-3.32)
Window, start 0
73.63 (-2.65)
72.79 (-3.50)
70.25 (-6.04)
Window, start 30
75.45 (-0.84)
74.65 (-1.63)
73.31 (-2.98)
CK, start 40, era 40-70
75.52 (-0.77)
75.16 (-1.13)
Intra, start 40, 40-70
75.56 (-0.73)
75.14 (-1.15)
Table 1: Main results, top-1 accuracy on Resnet50 v1.5 after 100 epochs.
ResNetv1. Furthermore, starting the pruning era later improves performance (Figure (bottom left)).
Epoch 90
Epoch 100
Model
Sparsity [%]
Accuracy [%)
Sparsity [%)
Accuracy [%)
CK, start 40
60
74.60
60
75.36
Intra, start 40
60
74.68
60
75.37
Window, start
74.78
CK, start
80
73.66
PruneTrain (Lym et al. 2019
50
73.0
Dyn Sparse (Mostala Wang 2019)
80
73.3
Dyn Sparse (kernel granularity)
72.6
Table Main results, top-I I accuracy on ResNet50 after 90 and 100 epochs. Comparison with related work. PruneTrain sparsity is not explicitly stated, so we estimate their sparsity level from their inference FLOPs saved (1-FLOPs), Also, our experiments were run with batch size 64.
Table 3 demonstrates that sparsity mechanism can have minimal drop in adversarial robust- ness (approximately 1- 1-1.5%) compared to the dense baseline model, whereas other methods see more accuracy degradation [Wang et al. 2018).
The sparsity each layer, depicted in Figure emphasizes that early layers tolerate sparsity better, as they have consistently higher sparsity in the last xx1 convolutional layer of each residual block. This may be due to their vicinity to the residual connection, which provides additional information the layer.

---

Under review conference paper at ICLR 2020
Layer Sparsity
0.77
0.75
0.73
0ol
0.69
Conv 1
Group 1
Group 2
Group 3
Group 4
FC
Figure 5; Resnet50 v1.5 layer sparsity for CK, start 30, targetting 70% sparsity.
Model
Sparsity
== .05
c=
== .15
e .2
= .25
c .3
Dense
0
41.71
30.47
25.24
22.39
20.69
19.58
Intra, start 40
0.6
40.42
29.02
23.67
20.73
18.93
17.73
Intra, start 40
0.7
40.03
28.55
23.13
20.13
18.32
17.10
Window, start 30
0.6
40.52
29.12
23.77
20.81
18.99
17.78
Window, start 30
0.7
39.73
28.32
22.91
19.88
18.02
16.77
CK, start 40
0.6
40.51
29.21
23.95
21.08
19.34
18.18
CK, start
0.7
39.83
28.44
23.09
20.11
18.31
17.12
Table 3: Adversarial Robustness of Resnet50 v1.5 on Imagenet
4.3 DISCUSSION
Overall, we notice that there is a tolerance for sparsity (up to 70%), which yields around 1% accuracy loss compared to the dense baseline. However, this loss can be compensated by dropping the learning rate and performing another 10 epochs of training, which provides a 0.7-0.9% accuracy increase. With high levels of sparsity this extension is computationally cheap.
observed the early stages dense training are important for high accuracy, as longer periods of dense training consistently outperformed shorter ones. Moreover, widening the pruning slightly (10 epochs) improves the final convergence accuracy (by around 0.2%).
We also observed that pushing the learning rate drop schedule to earlier epochs aligning it with pruning era does not improve the final accuracy. However, pushing the last learning rate drop from epoch 90 80 improve the accuracy by around 0.1%. (See Appendix Tableßand Table
We postulate that window pruning performs worse for ResNetv1.5 compared to ResNetvl due to the strided nature of convolutions in ResNetv1.5.
RELATED WORK
To give a broad comparison stage, we extended Mostafa & Wang 2019 's table on alternative sparsity mechanisms in Table 4 with respect to characteristics of their mechanisms: train- ing/ cog/copprssioo focus, regularization, the period in which pruning is applied, strictness of parameter budget, and pruning granularity. We explain each of the columns below:
1. Training Focus:
Trying train while maintaining/increen The opposite is Compression Focus, i.e., methods that only seek to provide a smaller network for inference.
sparsity of the network.
2. Regularization:
Applying a regularization value to the loss, in order to find and prune irrelevant weights, while others use magnitude- based pruning.
3. Pruning Era: The period during training in which the pruning applied.
4. Strictness of Parameter Budget Era wrt to Pruning: A strict parameter budget is fixed to the size of the final sparse model. Mostafa Wang (209 have strict budget throughout training. Our method is only strict after the pruning era. Some networks do not have strict parameter budget and only prune weights that appear to be irrelevant and without sparsity target.
5. Pruning Granularity: The level of granularity within in the network at which values are pruned. For example, at the kernel level, we determine which values to prune by examin- ing only the weights in the kernel [Mao et al. 2017). See Figure for more information.
We chose these concepts because their characteristics can enable faster and lower-energy training. A strict parameter allows the hardware mapping to plan for fixed number of multilly-accumulate
7

---

Under review conference paper at ICLR 2020
Striet
Train/ Cmpres
Requires Regular
Parameter
Pruning
Budget
Granularity
Method
Focus
-ization
Era
Era
Sparsity
Window (This Work)
No
Beginning
Alter Pruning
non in Window
CK/lotra Cpoch (This Work)
T
No
Middle
After Pruning
Kernel
Evolutionary Mocamu et al 2018
T
No
Beginning
Throughout
non-structured
Zhu Gupa
T
No
Throughout
After Pruning
non-structured
Lottery Frankle & Carbin
2018
T
No
Throughout
Throughout
non-structt structuuctueedd
RNN Pruning (Naaang of 2017
T
No
Beginning
None
non-structured
NeST NeST of cli 2017
T
No
Throughout
None
non-structured
Prune Irain Dym al
2019
Yes
lhrooggouu
After Pruming
Layer'Channel
Dyn Sparse Mostata
Yes
Throughout
Throughout
non-/Kemnel
Deeplt
Bellee et al 2017
T
Yes
Throughout
Throughou
non-structured
Deep Comp
No
Throughout
non-struuctued
LI-Norm Channel eL ail 2016
C
Yes
Throughout
Channel
rhheeev & Lempitsky 20TT
C
Ves
End
non-structuree
Sparsity MMo 2017
Yes
Throughout
non-structured
SSL. [Wen et al 2016
C
Yes
Throughout
Channel/ Keennn///Kell/Leeee
Luub
End
Channel
LASSO regression ef
2017
C
Yes
End
Channel
Slimming
201
Yes
Throughout
SSS Huang Walne 2018
C
Yes
Throughout
Layer
PiA Stau et 2018
C
Yes
Throughout
Channel
Table 4: Comparison of Training Methods that yield sparse networks
operations. The granularity of the sparsity mechanism indicates how easy it is to adapt the mechanism to an existing hardware. The coarser the granularity, the more adaptable to existing hardware (Mao et al. 2017). Regularization, although useful in forcing the network to learn prunable weights, adds more irregularity to computation flow. The pruning era starting at the beginning of the training enables us to train with a compressed network.
Mao et 2017 explores pruning on range of granularities including window, kernel, and filter, and their effect on accuracy. They also qualitatively and quantitatively show that coarse- grain pruning, like kernel- or filter-level sparsity, more energy- efficient due fewer memory references. Similarly, our work surveys sparsity at the window, kernel, and filter levels. We improve on Mao al.'s work two ways. First, we show higher top-5 accuracy at higher sparsity levels on complex benchmark, ImageNet on ResNet50 (92.338% at 40% CK sparsity), and we also show high top-1 accuracy whereas Mao et al. only report top-5.
energy-efficien
Prunetrain [Lym et al. 2019 explores way create sparse channels and even layers to speed up training with around 1% drop in accuracy. However, this requires a shift in the training mechanism, including a regularization term that could effect how the mechanism scales to large and distributed settings and that must be computed throughout training. The resulting network is only around 50% sparse and the accuracy loss due sparse training is high enough that baseline network with same accuracy could result into same computational savings by just terminating training at much earlier stage/epoch.
In contrast to other pruning mechanisms, our proposed window, CK, and combined sparsity mechanisms have strict parameter budgets after the pruning era. The CK and combined schemes have channel -evell and kernel-level pruning granularities.
6 CONCLUSION AND FUTURE WORK
In this work, we introduced techniques to train CNNs with structured sparsity and studied the tradeoff's associated with various implementation options. We demonstrated ResNet50 with the full ImageNet dataset that the proposed sparse training method outperforms all related work and comparable to a dense model in terms of convergence accuracy. We also observed that delaying the start of enforced, gradual pruning to at least epoch 20 was necessary to reach high convergence accuracy, highlighting the importance of the early epochs dense training. Moreover, performing an additional 10 epochs of training provides substantial (around 1%) accuracy gains of the final model. In the future, we would like study the tradeoffs of sparse training on low- -we-prccision networks.
8

---

Under review as conference paper at ICLR 2020
REFERENCES
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert A. Legenstein. Deep rewiring: Training very sparse deep networks. CoRR, abs/1711.05136, 2017. URL
abs/ 05136
Stanford CS231N. Tiny imagenet visual recognition challenge, 2015. tiny- imagenet hwrokuapp.p.... com
URL https:/
Xiaoliang Dai, Hongxu Yin, and Niraj K Jha. Nest: A neural network synthesis tool based grow-and prruue paradigm. arXiv preprint 1rXiv:1111.0011,, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1303.03633 2018
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:4412.6572, 2014.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00114 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and William J. Dally. EIE: efficient inference engine on compressed deep neural network. CoRR, abs/ 160/.0122,,, 2016. URLpttp://aas //tp/v/aavvvvv/aa//////aalaaiavv wttpaawwwwv...aaaa///e 162.. 01528
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/ 1es/11112.388, 2015. URLhttp::/arxiie org/ wttpg//gg/////.//r////////// 03385
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEFE International Conference Computer Vision, pp. 1389- 1397,2017
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 304-320, 2018
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks, 2014.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2554-2564, 2016.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang, Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2017. 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-50666, 2017.
Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Mattan Erez, and Sujay Shanghavi. Prunetrain: Gradual structured pruning from scratch for faster neural network training. CoRR, 2019. /ttt: http///xvvvv/// http:///rwxi..oooobb/////b/b 901. 09290
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang. and William J Dally. Explor- ing the granularity of sparsity in convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 13-20, 2017.
9

---

Under review as conference paper at ICLR 2020
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 9(1):2383, 2018.
Hesham Mostafa and Xin Wang. Parameter efficient de convolutional neurol neural by dynamic sparse reparameterization. CoRR, abs/ 1902.05967, 2019. URL httpptt///axxx
azg/abs/1902 05967
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recur- rent neural networks. arXiv preprint arXiv:1704.05113 2017.
Xavier Suau, Luca Zappella, Vinay Palakkode, and Nicholas Apostoloff. Principal filter analysis for guided network compression. arXiv preprint arXiv:1807.10558; 2018.
Lei Sun. Resnet on tiny imagenet. Submitted on, 14, 2016.
Luyu Wang, Gavin Weiguang Ding, Ruitong Huang, Yanshuai Cao, and Yik Chau Lui. Adversarial robustness pruned neural networks. ICLR 2018, 2018.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082. 2016.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv e-prints, 2017. URL https::: / /ui adsabs harvard. abss/2017arxX1111101132
10

---

Under review conference paper at ICLR 2020
7 APPENDIX
7.1 DETAILS OF PRUNING ALGORITHMSS
Here we provide full descriptions of our other pruning methods and our general methodology sparse training.
Sparse Training Methodology Algorithm 2 shows how we modify normal training in order to train sparsely.
Algorithm 2 Pruning Algorithm
current iter= while training do
if current_ire iter first epoch of pruming and current j_tertterr-- < last epoch of pruning then mask = geenerate_sparsiyy mmaiiiyy__aki((( 0, current_ite, sparsity threshold)
end if
pruned mask 0
= forward (pstrdd___s((((_ pruned. 2:
= weight update( 9. 0 prunedd)
current iter current iter end while
Window Pruning Methodology
Agorithm3showe how we prune with window sparsity.
Algorithm 3 Window Pruning Algorithm
generate Lenerate_window.cnn saaddowssparsity www.sssttaass(OOOeet mask sparsity threshold):
for 0 in dlayer do
for all c in C do
for all k in K do
cutoff ind__=nndxxx- size( sparsity_threshoet threshold
n ===mxxcctofffineeee _index, size(0,..)) mmax_non_zae - 1)
cutoff_valuu = th largest value in 0g.
for all i,j in R,S do
maski,j.e.k = j.j.k vutoff_value, else
end for end for end for end for
Combined Pruning Methodology To combine Window and CK pruning, we introduce intra- epoch pruning. As shown by Algorithm in given epoch first apply Window Pruning to each 3x3 convolutional layer at a fraction of the sparsity threshold for that epoch. Then, we prune the remaining fraction of the sparsity threshold with CK Pruning. The idea being that kernels that lose many of their parameters during window pruning can be fully pruned during the CK pruning phase. Our intuition is that first pruning parameters within kernels guides the subsequent CK pruning towards the less important kernels. Thus, we pick out better kernels to prune. We also gain more structured sparsity but sacrifice the precision of window pruning.
Algorithm 4 Intra-Epoch Pruning Algorithm
generate intra epoch_sparsity mask(0 sparsity sparsis seearsstyythreshold): for 0 in Olayer do
window s_ceow_msss------ generate __k______________e sparsity_maskkl sparsity_threeeeeeeeee (hreehhldd)
ck_mask generate_dd setee_ccsss__tt_eeeeeeee (0aiit0__aa(O,,,_ sparsity_thresholdd
mask window mask and ck. mask end for
11

---

Under review conference paper at ICLR 2020
For completeness, also tried another method of combining called inter-epoch pruning, which involved splitting the pruning era into CK pruning and window pruning phases. However, from our initial experiments determined that intra-epoch pruning, performed better (though it was more compurationally expensive) than inter-epoch pruning. With inter-epoch pruning we were only able achieve 74.1% top-1 accuracy with first epoch of sparsity of 40 and final sparsity of 40% on Resnet50v1.5 and Imagnet. The same setup trained with intra-epoch pruning achieved 74.9% accuracy. Thus, we pursued intra-epoch pruning as our method combine the two sparsification methods.
7.2
ADDITIONAL DETAILS ON EXPERIMENTAL SETUP
This section goes into more detail on the exact details of the models and dataset combinations we sued for experimentation.
7.2.1 RESNET50 ON INN-IIMAENET
For this training domain, we trained using the Tiny-imagenet dataset CS231N 2015) with resnet50 He et al. 2015). However, changed the training mechanism in order get vali- date our results. Each network we train for 40 epochs, with a batch size of 64. Additionally, we use the Adam optimizer to train with learning rate set 0.001 and momentum weight decay set to 0.0001, and we anneal the learning rate to 0.0001 after 30 epochs of training in order to converge faster. We apply the same image transforms as on full Imagenet.
0.9. We also use
We chose this optimization method because we felt that it achieved good overall accuracy at baseline level and represents the results in Sun 2016 in their vanilla model. We do not use the same preprocessing image transforms in the report Sun (0016). Moreover, we wanted quick way to estimate how our method would perform on full Imagenet.
7.2.2 RESNET50 ON IMAGENET
Here, we train each network for 90 epochs with reduced batch size of 128 instead 256 because 256 would not fit GPU in addition our pruning layers. We found that changing the batch size to 128 but retaining all other hyperparameters as specified in He et al. (2055) we were able to achieve the same benchmark 74.94% accuracy as the paper. We train for 90 epochs with SGD with momentum set to 0.9 and weight decay is 10-1. We set the initial learning rate to be 0.1 and then anneal the learning rate to 0.01 at epoch 30 and then finally to 0.001 at epoch 60.
For dataset transformations, we perform the same transformations as This means that during training we perform a random sized crop to size 224x224, randomly flip the image horizontally, and normalize the image. The batches are shuffled during training. For validation, we resize the image to 256 and then center crop to size 224x224 and then normalize.
7.2.3 RESNET50V1.5 .5 ON IMAGENET
We train our model for 90/100 epochs and use SGD with momentum (0.9) to optimize. The standard model says that learning rate should 0.1 for 256 batch size, but since that didn't fit in our GPUs with our sparsity mechanism, we used batch size 64 and linearly scaled the learning rate to be 0.05. We set the learning rate decay such that we multiply by 0.1 after 30, 60, and 90 epochs. We have weight decay set to 1 x 14 Resnet1.5 is defined in detail here
7.3
MISCELLANEOUS RESULTS
7.3.1 RESNET50 ON IMNGIIMAENET
Our models actually perform better than the baseline with the following configurations: window pruning with 60% sparsity as well as CK pruning with 20%, 60% and 80% sparsity. The number of
For_pytorch
12

---

Under review conference paper at ICLR 2020
epochs required to reach the converge to the final accuracies is the same for CK and earlier for window at 40% and 60% sparsity.
Window
Model Spursity
Epoch of
True sparsity
Epoch Convergence
Accuracy [%)
Convergence
Accuracy
True sparsity [%)
52,03
40
0.01
51.84
31
0.021
20
50.97
0.48
52.40
31
0.207
600
52.09
40
0.69
52.56
31
0.602
51.18
52.07
0.800
Table 5: Best accuracy for different sparsity levels for Resnet50 Tiny-Imagent with Window and CK Pruning. Both methods see high accuracy close to, or even above the baseline in the cases of Window 60%, CK 20%, 60% and 80%. CK converges to the final epoch the same epoch as the baseline while window converges at the same epoch or earlier in the cases of 40% and 60%.
52
52
Accuracy
Accuracy
40
46
41
44
Epoch
Epoch
Figure 6: Convergence plots of Resnet50 on Imay-em with Window (left) and CK Pruning (right). all sparsity levels we are the near or above the baseline. Though the differences between models is small, 60% seems to performs the best and is a peak with respect to 40% and 80%.
0.8
Network Sparsity
74
Accuracy
0.6
72
40%
0.4
40%%
70
80%
605
70%
0.2
705
68
80%
00%
Epoch
Epoch
Figure 7: Convergence and sparsity plots of Resnet50v1.5 on Imagenet with CK Pruning, first epoch of sparsity 30. As the amount of sparsity increases, the accuracy of the model seems to decrease semi-linearly. We can achieve up to 70% while still being above 74% sparsity.
Model Sparsity (%)
40
60
70
80
Dense (75.25)
CK, start 40
74.93 (-0.32)
74.21 (-1.04)
73.34 (-1.91)
CK, start 30
74.96 (-0.29)
74.25 (-1.00)
73.83 (-1.42)
CK, start 20
74.74 (-0.5))
73.84 (-1.41)
73.37 (-1.88)
72.49 (-2.76)
Intra, start 40
74.91 (-0.34)
74.75 (-0.50)
74.36 (0.89)
73.22 (-2.03)
Intra, start 30
74.94 (-0.31)
74.41 (-0.84)
73.87 (-1.38)
Intra, start 20
74.73 (-0.52)
73.65 (-1.60)
72.22 (-3.03)
Window, start 0
72.63 (-2.62)
71.79 (-3.46)
69.48 (-5.77)
Window, start 30
74.28 (-0.97)
73.88 (-1.37)
72.63 (-2.62)
CK, start 40, era 40-70
74.67 (-0.58)
74.34 (-0.91)
Intra, start 40, era 40-70
74.77 (-0.48)
74.38 (-0.87)
Table Main results, top-1 accuracy on Resnet50 v1.5 after epochs.
13

---

Under review conference paper at ICLR 2020
Network Sparsity
Accuracy
60
0.5
00%
0.4
00%
40
90%
607
70%
0.2
70%
80%
00%%
Epoch
Epoch
Figure 8: Convergence and sparsity plots of Resnet50v1.5 on Imagenet with Bpora Pruning, first epoch of sparsity 30. The left plot shows training over the full 100 epochs instead of zooming in on the tail end of training. This allows us to observe the importance of the learning rate drops epoch 30 and 60. The drop epoch 90 does have small increase, well.
0.8 A
Network Sparsity
Accuracy
60
0.6
0.4
00%
60%
70%
70%
80%
80%
Epoch
Epoch
Figure 9: Convergence and sparsity plots of Resnet50v1.3 on Imagenet with Window Pruning, first epoch of sparsity = 30. With window, likw with CK, the impact of the learning rate drops at epochs 30 and 60 is big. Note that the sparsity of the window is not as smooth as CK, showing that it is less uniformly sparse from epoch to epoch.
Top- Accuracy
Dense
Window, Start
CK, Start 20 Intra,
CK, Start
Intra, Start 30
Window, Start 30 CK, Start Intra,
40
60
80
100
Epoch
Start
Top 1op-5.A6 Accuracy
992
Windows
Start
90
Intra, Start 20
30
88
Intra, Start 30
Windune Start 30
Start
Intra, Start 40
Epoch
CK. Start Era 30
Lnttin, Start 40, Fra 30
Figure 10: Convergence plots of Resnet50v1.5 1.5 on Imagenet 60%, top-1 and top-5 accuracy. For both top-1 an top-5, window pruning starting at does not perform as well as the other methods. The rest mostly clustered around 75% top-1 accuracy and 92% top-5 accuracy.
14

---

Under review conference paper at ICLR 2020
74
74
Accuracy
Accuracy
72
20
70
68
100
Epoch
Epoch
Network Sparsity
74
Accuracy
0.6
60%
60%
70%
20%
68
00%
00%
Epoch
Epoch
Figure Convergence plots with CK (top left), intra (top right), window (bottom left), and sparsity plot with window (bottom right). Resnet50vl.5 on Imagenet with first epoch of sparsity = 30.
Model Sparsity (%)
40
60
70
80
Dense
1.04
CK, start 40
0.89
0.73
0.71
0.83
CK, start 30
0.87
0.86
0.90
CK, start 20
0.80
0.87
0.96
0.30
Intra, start 40
0.98
0.64
0.72
0.79
Intra, start 30
0.89
0.75
0.61
Intra, start 20
0.99
1.10
0.75
Window, start 0
1.01
1.00
0.76
Window, start 30
1.17
0.77
0.67
CK, start 40, era 40-70
0.75
0.82
Intra, start 40, era 40-70
0.79
0.76
Table 7: Gain in top-1 accuracy on Resner50 v1.5 from epoch 90 to epoch 100. This table shows more explicitly that the benefit of the additional 10 epochs of training from epoch 90 100 is about 0.8-1.1%.
Experiment @ 60% Sparsity
Accuracy (Improvement) [%]
CK, start
75.39 (0.05)
CK, start
75.06 (-0.06)
CK, start 20
74.80 (0.09)
Intra, start 40
75.46 (0.08)
Intra, start 30
75.19 (0.03)
Intra, start
74.88 (0.13)
Window, start 0
73.89 (0.26)
Table Results when we changed final learning rate drop from epoch to 80, top-1 accuracy on Resnet50 v1.5 at epoch 100. This change does provide 0.1-0.2% improvement is some cases, but the benefit is relatively small.
15


Reviews:
[{"review_id": "r1xPmfUG9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates methods to train neural networks so the final network has sparse weights, both in convolutional layers and in fully connected layers. In particular, the paper focuses on modifying the training so that the network is first trained without sparsification for a certain number of epochs, then trained to be increasingly sparse, and then fine-tuned with a fixed sparsity pattern at the end.\n\nWhile I find the overall approach of the paper interesting, currently the experiments are not systematic enough to derive clear insights from the paper. Hence I unfortunately recommend rejecting the paper at this point. I hope the authors find time to conduct more systematic experiments for a future version of the paper.\n\nConcretely, the following would be interesting experiments / questions:\n\n- How effective is the proposed training method on architectures other than ResNets?\n\n- What happens if the \"pruning era\" is made longer, started substantially earlier, or started substantially later? Currently it is not clear if the epoch 30 - 50 pruning era is (approximately) optimal and how much performance varies with begin and end of the pruning era.\n\n- Due to the small variation between some of the methods, it would be good to investigate how robust the ordering is when the experiment is re-ran with different random seeds etc.\n\n\nIn addition, I have the following suggestions:\n\n- The authors may want to remove or enhance the adversarial robustness evaluation. Currently the authors only evaluate robustness against FGSM, but it is well known that iterative attacks such as PGD are more effective.\n\n- Instead of \"intra-epoch pruning\" or \"intra\", the name \"combined\" may be more clear for the combined method.\n\n- In the description of the experimental setup, it could be good to specify what GPUs were used (since this lead to the smaller batch size).\n\n- It could be helpful for the reader to discuss how predictive results on Tiny-ImageNet are for results on ImageNet.\n\n- In Table 2, it would be good to add context by comparing to prior work with sparsity level 60% and some of the compression-focused methods from Table 4.\n\n- In the comparison to Mao et al. (2017), it could be good to clarify that they also work with ResNet models on ImageNet.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1lup7X-5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY\n-------\n\nThis paper explores a series of incremental variations of existing pruning techniques for compressing Resnet-50 for ImageNet. Specifically, it proposes concentrating all pruning during an early \"era\" of training (the first 20-50 epochs out of 100 total). It also explores hybrids between sparse pruning and structured pruning. Finally, it considers the adversarial robustness of the resulting networks to the FGSM attack. \n\nThis paper makes no novel proposals and experiments are minimal. There are no clear takeaways from the results of these experiments. The goals of the paper are unclear, and it is difficult to compare this paper to existing work. \n\nThis paper has no clear motivation and makes no tangible contributions to the literature and, therefore, I recommend a rejection. \n\nCONTRIBUTIONS\n-------------\n\n1) A study of the appropriate window (\"pruning era\") for pruning Resnet-50 on ImageNet and TinyImageNet\n2) A study of the tradeoffs between various forms of structured and unstructured pruning.\n3) An analysis of the adversarial robustness of the pruned networks.\n\n\nDETAILED COMMENTS\n------\n\nPROBLEMS ADDRESSED\n\nIt was challenging to discern the specific problems that this paper sought to address and, relatedly, the goals that the paper sought to achieve. The introduction of the paper lists a wide variety of problems in the existing literature:\n\n1) Paragraph 3: Structured sparsity introduces \"regularization and computational overhead.\"\n2) Paragraph 3: \"Coarse-grained sparsity\" cannot eliminate enough parameters to perform well on \"edge devices.\"\n3) Paragraph 4: Dynamic sparsity techniques require more training epochs.\n4) Paragraph 4: Dynamic sparsity techniques do not preserve network accuracy (1-2 percentage point drop at 80% sparsity).\n5) Paragraph 4: Dynamic sparsity requires reconfiguring the sparsity pattern frequently, which is computationally expensive.\n\nThe paper does not justify the fact that any of these are actually problems, nor does it make any attempt to quantify the extent of these problems. Moreover, the proposed techniques do not resolve any of these problems. Corresponding to the numbers above:\n\n1) The paper never measures this overhead nor justifies that it is a problem in practice. Meanwhile, the techniques proposed in the paper introduce substantial overhad of their own, including training for an extra ten epochs. It is possible that the techniques proposed in this paper have worse overhead than the techniques that are criticized in the introduction. Since the paper provides on numbers either way, it is impossible to tell. In short, computational cost is a key part of the author's argument despite the fact that there is no empirical support for any of these claims.\n\n2) I believe the paper means that, in order to get to sufficient levels of sparsity to work on \"edge devices,\" accuracy drops unacceptably far. What does the paper mean by \"edge devices,\" what are sufficient levels of sparsity, and what does it mean for accuracy to drop unacceptably far? The paper has numbers for the proposed methods, so it should be possible to make this comparison if such baselines are explicit.\n\n3) The proposed techniques also require the same number of additional training epochs, so this complaint is unaddressed.\n\n4) The proposed techniques show a 2-3 percentage point drop at 80% sparsity (Table 1), which is actually worse than the technique that the authors criticize.\n\n5) The proposed techniques require pruning after every single training step during the \"pruning era.\" This is likely to be more computationally expensive than any of the other gradual pruning and dynamic sparsity techniques listed, which prune at intervals of hundreds or thousands of iterations. In addition, the authors never justify why changing the sparsity pattern frequently throughout training will affect performance. On GPUs with modern frameworks, I see no reason why this should matter so long as the sparsity pattern does not change too frequently (although that is exactly what this paper proposes to do during the \"pruning era\").\n\n\nGOALS\n\nIt was also challenging to discern the goals of the paper. Was it:\n\n1) To produce the smallest possible trained networks with the highest possible accuracy?\n\n2) To reduce the cost of obtaining a pruned network for inference-time? (Or to reduce the cost of obtaining a sufficiently efficient pruned network for inference-time?)\n\n3) To reduce the cost of training neural networks in general by pruning them during training?\n\nIn the introduction and the related work section, these goals go unstated, making it difficult to determine how this paper compares to existing work. The comparisons provided in the paper focus on specific aspects of each related work rather than the entire picture. For example, in comparison to Mao et al., the authors claim better accuracy at one sparsity level, implying goal 1. However, for to Lym et al., the paper focuses on the computational costs of training the network, implying goal 3.\n\n\nUNJUSTIFIED CLAIMS ABOUT NEURAL NETWORK COMPUTATION\n\nThroughout the paper, there are a number of unjustified claims about which neural network configurations will perform better on contemporary hardware. Considering computational efficiency appears to be a key element of the paper's argument, these claims require citations or - particularly when various configurations are compared to one another - empirical support. Some examples:\n\n* Section 1, Paragraph 3: \"The regularization term [of structured sparsity] modifies the original training and can be expensive in hardware.\"\n* Section 1, Paragraph 3: \"The final network [from Lym et al. 2019] contains an insufficient degree of sparsity for deployment on edge devices.\"\n* Section 1, Paragraph 4: \"Continuous reconfiguration of the sparsity pattern is expensive as it does not allow for compression of weights during training\"\n* Section 1, Paragraph 5: \"having a fixed sparse multiply-accumulate pattern allows weight compression during training and can save compute and energy in hardware\"\n* Section 5, Paragraph 2: \"A strict parameter allows the hardware mapping to plan for a fixed number of multiply-accumulate operations.\"\n* Section 5, Paragraph 2: \"Regularization, although useful in forcing the network to learn prunable weights, adds more irregularity to computation flow.\"\n\n\nPRUNING TECHNIQUES\n\n* Recomputing the pruning mask at every training step seems gratuitously inefficient.\n* Sorting the weights in the entire network shouldn't be particularly inefficient if it isn't done on every single iteration. (Section 2.1 paragraph 1)\n* Why do you maintain the same number of weights in each convolutional filter with window pruning? (Presumably for performance reasons, but you never say that.)\n* None of the pruning methods are novel. They're simply various permutations of structured and unstructured magnitude pruning as proposed by many others in the literature.\n\n\nEXPERIMENTS\n\n* Section 3.1 Paragraph 2: It appears that you are exploring the best \"pruning era.\" If you are to do so, you will have to sweep over (1) the length of the pruning era (2) the starting epoch of the pruning era, and (3) the shape of the function used to determine sparsity. Instead, it sounds like you tried two arbitrary pruning eras (0-30 and 30-50). Likewise, in Paragraph 3, you test only a small number of possible scenarios.\n* Section 3.1 is generally hard to parse. It is unclear what you are studying. The ideal pruning era? The relative performance of the pruning methods introduced in section 2? \n* How many times did you replicate each experiment? You should ideally include at least 3 (and preferrably 5) replicates with mean and stddev reported.\n* What baselines are you including? You should include a random pruning baseline and you should ideally replicate any methods that you compare to.\n\n\nRESULTS\n\n* Section 4.1: The data you refer to is in an appendix even though it is crucial to the main body of the paper. The appendices should contain material that is nonessential for making sense of the paper.\n* Section 4.2 Paragraph 1: Are these numbers good? A standard sparse pruning technique (Gale et al. 2019, https://arxiv.org/pdf/1902.09574.pdf) achieve 70% sparsity without any change in accuracy. Please include baselines comparing to other methods in the literature.\n* Table 2: It is difficult to compare the results in these papers. PruneTrain aims to reduce the cost of training and measures cost reductions in FLOPS. If you intend to compare against this paper, you should quantify the cost of training using your method against that of PruneTrain. Merely presenting sparsity and accuracy numbers is insufficient. Likewise for the dynamic sparsity experiments. What is your goal in showing this comparison, and did Mostafa and Wang share that goal when they justified their technique?\n* You do not describe the hyperparameters for Intraepoch pruning (the balance between window and CK - last paragraph of 2.1.1)\n\n\nADVERSARIAL ROBUSTNESS\n\nConsidering the fact that this paper focuses on proposing new variations of existing pruning techniques, any discussion of adversarial robustness seems to be (1) out of place and (2) an afterthought. If the authors delete a half-page of content (one phrase from the abstract, a paragraph and bullet from the introduction, and a paragraph each from sections 3 and 4), this content could be removed with minimal impact to the paper's main contributions. The content on adversarial robustness is cursory, uses a weak and out-of-date attack (FGSM), and does not compare to any other pruning methods. In fact, the one comparison is to the results in a paper (Wang et al, 2018) that looks at both FGSM and PGD (a stronger attack) on completely different networks and tasks (MNIST and CIFAR10). The paper would be stronger if content on adversarial robustness was removed entirely.\n\n\nOTHER MINOR COMMENTS\n\n* The title includes the word \"starfire,\" but it never appears again in the paper. The paper proposes no specific technique, so there isn't anything to name.\n* Use the \\ begin{appendix} command before you create the appendices and the \\ end{appendix} command when you are done. You can then use \\section normally and each section so-created will appear with a letter rather than a number.\n* Figure 4 is very hard to read.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJgK9BkUYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a strategy to prune a convolutional neural network during training. To speed up training, the proposed method prunes the weights with the smallest magnitude during only a small number of epochs at the beginning of training, later on continuing training with a fixed sparsity pattern. Several granularity levels for convolutional and fully-connected layers are studied. Furthermore, the robustness of the resulting pruned networks to adversarial attacks is investigated.\n\nOriginality:\n- As acknowledged at the beginning of section two, the general pruning strategy used here is very similar to that introduced by Narang et al., 2017. While the authors argued that the threshold is computed in a different manner, it also increases gradually during training, as in Narang et al., 2017.\n- I acknowledge that Narang et al., 2017 focuses on RNNs, while here the focus is on CNNs. However, the originality of the different pruning strategies used here for convolutional and fully-connected layers is very limited. In essence, these strategies directly follow those studied by Mao et al., 2017.\n- The study of robustness to adversarial attacks, while interesting, is also not novel per se, as the idea of performing such a study was proposed in Wang et al.,  2018. I acknowledge that the conclusions drawn here differ from those in Wang et al., 2018. However, there are no explanations for this different behavior.\n\nMethodology:\n- While the beginning of Section 2 states that the pruning threshold gradually increases during training, the specific way this is achieved is not clearly explained.\n- The pruning strategies depicted by Fig. 2, whether for convolutional layers or for fully-connected ones, never aim to remove entire output channels. However, the only way to truly end up with a smaller network is to remove entire channels and/or layers, as argued in Wen et al., 2016 and in Alvarez & Salzmann, NIPS 2016, as well as studied in Mao et al., 2017 via the filter-level granularity. It is unclear to me how speed would be affected by having a network with the same number of channels and layers, but many parameters set to zero. \n\nExperiments:\n- The experiments show the good behavior of the proposed algorithm in terms of sparsity vs accuracy tradeoff. However, while the introduction seems to focus on the benefits of the proposed method in terms of training speed, these benefits are not demonstrated in the experiments, where no timings (neither for training not for inference) are reported.\n- As mentioned above, it is not clear to me that the speedup will be significant if the sparsity pattern does not remove entire channels, but I am willing to be proven wrong.\n\nSummary:\nMy main concern about this paper is its novelty, as the method essentially uses the method of Narang et al., 2017, albeit with a different threshold, with the sparsity patterns of Mao et al., 2017. The experiments demonstrate that the method is effective at pruning, but do not provide any timings to evaluate the resulting speedups.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper concerns a training procedure for neural networks which results in sparse connectivity in the final resulting network, consisting of an "early era" of training in which pruning takes place, followed by fixed connectivity training thereafter, and a study of tradeoffs inherent in various approaches to structured and unstructured pruning, and an investigation of adversarial robustness of pruned networks.

While some reviewers found the general approach interesting, all reviewers were critical of the lack of novelty, clarity and empirical rigour. R2 in particular raised concerns about the motivation, evaluation of computational savings (that FLOPS should be measured directly), and felt that the discussion of adversarial robustness was out of place and "an afterthought".

Reviewers were unconvinced by rebuttals, and no attempts were made at improving the paper (additional experiments were promised, but not delivered). I therefore recommend rejection. 
Decision: Reject