STARFIRE: REGULARIZATION FREE ADVERSARIALLY ROBUST STRUCTURED SPARSE TRAINING
Year: 2019

Abstract:
This paper studies structured sparse training of CNNs with a gradual pruning technique that leads to fixed, sparse weight matrices after a set number of epochs. We simplify the structure of the enforced sparsity so that it reduces overhead caused by regularization. The proposed training methodology explores several options for structured sparsity. We study various tradeoffs with respect to pruning duration, learning-rate configuration, and the total length of training. We show that our method creates a sparse version of ResNet50 and ResNet50v1.5 on full ImageNet while remaining within a negligible <1% margin of accuracy loss. To ensure that this type of sparse training does not harm the robustness of the network, we also demonstrate how the network behaves in the presence of adversarial attacks. Our results show that with 70% target sparsity, over 75% top-1 accuracy is achievable.

text: This paper studies structured sparse training of CNNs with a gradual pruning technique that leads to fixed, sparse weight matrices after a set number of epochs. We simplify the structure of the enforced sparsity so that it reduces overhead caused by regularization. The proposed training methodology explores several options for structured sparsity.
We study various tradeoffs with respect to pruning duration, learning-rate configuration, and the total length of training. We show that our method creates a sparse version of ResNet50 and ResNet50v1.5 on full ImageNet while remaining within a negligible <1% margin of accuracy loss. To ensure that this type of sparse training does not harm the robustness of the network, we also demonstrate how the network behaves in the presence of adversarial attacks. Our results show that with 70% target sparsity, over 75% top-1 accuracy is achievable.

heading: 1 INTRODUCTION
text: Pruning weights can compress a network into a smaller model so that the model can fit into faster/smaller memory and therefore result in execution speedups (Han et al., 2016; 2015a). To increase the accuracy Han et al. (2015b) and Mao et al. (2017) explore training the network dense after pruning. The resulting network can maintain accuracy based on the specified level of sparsity (Mostafa & Wang, 2019; Zhu & Gupta, 2017; Han et al., 2015a).
Structured sparsity has been explored for RNNs and also CNNs where a certain number of nonzeros is allowed across various cross-sections of the weight tensors. These methods aim to speed up computation and reach some final level of sparsity for deployment. Narang et al. (2017) have shown promising results for structured training of RNNs while sparse CNNs could not achieve the same performance (Mao et al., 2017).
Recent work has demonstrated that structurally sparse training can speed up execution on GPUs (He et al., 2017; Lym et al., 2019; Zhu & Gupta, 2017). However, these training mechanisms add regularization and computational overhead to eliminate unnecessary weights. The regularization term modifies the original training and can be expensive in hardware. While enforcing coarse-grain sparsity Lym et al. (2019) provides significant speedups, the final network contains an insufficient degree of sparsity for deployment on edge devices.
Mostafa & Wang (2019) show that with adaptive sparse training and dynamic reallocation of nonzeros sparsity levels up to 80% can be achieved. However, even though an additional 10 epochs of training are required, an accuracy loss of around 1-2% is still observed. The main drawback is the overhead incurred while implementing such technique on the target platform. Continuous reconfiguration of the sparsity pattern is expensive as it does not allow for compression of weights during training.
To achieve speedups and a desired final degree of sparsity, we aim to apply the techniques in Han et al. (2015b) and Mao et al. (2017) at earlier stages in training at higher frequency within a period which we call the pruning era, usually a period of 20-30 epochs. During the pruning era, with fine granularity of at most a kernel size, we exploit one of the three proposed sparsity regimes. Subsequently, we fix the mask for the rest of the training to speed it up. Our motivation came from the insight that having a fixed sparse multiply-accumulate pattern allows weight compression during training and can save compute and energy in hardware (Han et al., 2016).
We explore the impact of various pruning granularities, sparsity levels, and learning-rate schedules on the network’s convergence as well as adversarial robustness for CNNs like Resnet-50 (He et al., 2015) on ImageNet and tinyImagenet (CS231N, 2015).
Recent literature has shown that adversarial attacks are more successful on pruned neural networks than they are on regular neural networks (Wang et al., 2018). Given the danger of adversarial attacks in real world situations, we find that it is important to evaluate our sparsity techniques under adversarial robustness. We leverage the FGSM mechanism (Goodfellow et al., 2014) to evaluate the adversarial robustness on our sparse models. This paper makes the following contributions:
1. We propose a mechanism to train and prune a convolutional network during the earlier stages of training such that this sparsity can be harvested for the computational speedups. To do this, we fix the sparse weight masks for the remainder of the training.
2. For fully connected sparsification, we eliminate blocks of fully connected weights based on their connection to the zeros in the previous convolutional layer.
3. We enforce structural, regularization free, magnitude-based pruning across two distinct dimensions and a combined version. These dimensions are inside convolution window R ×S and across input/output feature matrix (C K ).
4. Our sparse models are as robust to adversarial FGSM attacks as fully dense models.
5. We demonstrate that early stage dense training is crucial for maintaining high accuracy.
6. The proposed technique is tolerant to sparsity levels of up to 60-70% with under 1% accuracy degradation. We can compensate by scheduling an extra learning rate drop and training for an extra 10 epochs.
The rest of the paper is organized as follows. Section 2 explains our pruning methodology. Section 3 describes the experimental setup framework. Section 4 presents results and discusses their interpretation. Section 5 presents the related work. Section 6 concludes the paper.

heading: 2 PRUNING METHODOLOGY
text: Our proposed pruning mechanism works by always pruning the weights of smallest magnitude after each weight update. After a forward and backward pass (one batch update), the model is pruned. If a weight is already zero, the gradient is also set to zero. This means that once a weight becomes zero, it will remain zero for the rest of the training period.
This mechanism is similar to Han et al. (2015b), except that we only prune in the earlier stages of the training as opposed to post training. Additionally, this work is similar to Narang et al. (2017) although we set the sparsity threshold instead of using a heuristic to calculate it. We chose this pruning mechanism because of its negligible computational overhead.
In our pruning algorithm, the sparsity threshold refers to the percentage of weights in the network that are currently pruned. Before or during the first epoch of pruning, we will have a sparsity threshold of zero. As we continue training, we gradually increase the sparsity threshold so that by the final epoch of pruning the network sparsity will have reached our final, desired threshold. Finally, we also define the pruning era to be the epochs between the first and final epochs of pruning depicted in Figure 1b.
We evaluate the pruning mask after every training step until we reach the final epoch of pruning. After the final epoch, the pruned values in the network will remain zero for the rest of training; no new pruning will occur, and only the non-zero weights will be updated.

heading: 2.1 PRUNING METHODOLOGY BY LAYER
text: Pruning the smallest magnitude weights in the entire network is inefficient because it involves sorting the weights over the network. Instead, we prune the smallest magnitude weights or sum of weights, within a certain locale of the network. When pruning, we examine each layer individually and apply a separate technique to evaluate which weights to prune, depending on the type of layer we are currently pruning.

heading: 2.1.1 CONVOLUTIONAL LAYER PRUNING
text: Window pruning for 3x3 Convolutional Layers Figure 2a shows the result of a pruned 3×3 convolutional weight tensor under the window pruning method. In this scheme, window layer pruning refers to pruning of weights within the 3×3 convolution kernels. We allow a maximum number of non-zero values for each kernel in the 3×3 convolutional layers and eliminate the weights of smallest magnitude.
Algorithm 1 CK Pruning Algorithm
generate_ck_sparsity_mask(θl ayer , sparsity_threshold): for θ in θl ayer do
for all c in C do for all k in K do
kernel_maxc,k = max(θc,k ) end for cutoff_index = size(θc ) ∗ sparsity_threshold n = max(cutoff_index, size(θc ) − max_non_zero − 1) cutoff_value = nth largest value in kernel_maxc for all k in K do
maskc,k = 1 if kernel_maxc,k > cutoff_value, else 0 end for
end for end for
CK Pruning Methodology Figure 2b shows the result of a pruned 3×3 convolutional weight tensor under the CK pruning method. In this scheme, the weights of a certain layer can be viewed
as a CK matrix of R×S kernels. The CK pruning method involves pruning the 3×3 convolutions along the channel and kernel dimensions of each convolutional filter, i.e., we prune whole kernels (CK matrix of R×S windows) at once and can ultimately prune all the input channels in an output channel. As defined by Algorithm 1, we determine which filter to prune by examining the max of the magnitudes of all the weights in a kernel, which is the max of nine weights. This max is used to evaluate whether the whole kernel should be pruned or not.
Combined Pruning Methodology To combine window and CK pruning, we introduce an intraepoch combined pruning method, which we refer to hereafter as “intra-epoch pruning” or “intra”, for short. As shown by appendix Algorithm 4 in the Appendix, in a given epoch we first apply window pruning to each 3×3 convolutional layer at a fraction of the sparsity threshold for that epoch. Then, we prune the remaining fraction of the sparsity threshold with CK Pruning.

heading: 2.1.2 FULLY CONNECTED PRUNING
text: Like pruning for convolutional layers, we apply a two-tier pruning scheme from Mao et al. (2017) for fully connected layers: micro-level pruning within a block and macro-level pruning that eliminates entire blocks.
Block FC Pruning Figure 2d refers to pruning of individual blocks. Here, we prune an entire n×n (n<5) window within the dense layer and create coarse grained sparsity. To do this, we sum the magnitude of the weights in each window and prune the windows with the smallest magnitude.
Fine FC Pruning Figure 2c refers to the pruning of individual weights. Here, we prune the individual weights in the entire FC Layer, where we compare the magnitude of all the weights to each other.
The produced zero patterns in the last convolution layer allow for eliminating more weights in fully connected layer as depicted in Figure 3. If all the C windows for a specific Ki are zeros, the output activation for the corresponding Ki is also zero. The corresponding neurons in the following fully connected layer are therefore receiving zero input activations and can be eliminated along with their associated weights. This enables us to get sparsity without having to evaluate the weights in the fully connected layer.
When pruning just the small weights in the FC layer, one can inadvertently cut off relevant connections between the input and output layers. Accordingly, we structure the pruning mechanism such that each output neuron should be influenced by the input. This means every column in the weight matrix of the fully connected layer in Figure 3 has at least one non-zero element.

heading: 3 EXPERIMENTAL SETUP
text: To validate each type of pruning (window, CK, or intra-epoch) we selected ResNet50 (He et al., 2015) v1 and v1.5 with the ImageNet and/or Tiny-ImageNet (CS231N, 2015) datasets. We evaluated each pruning method by varying sparsity levels and pruning era.
We experimented with ResNet50 v1.5, in addition to v1, to explore how changing the network structure would affect the top-1 accuracy. For window pruning, we tested with ResNet50v1 on Tiny-ImageNet as well as ResNet50v1 and v1.5 on ImageNet to compare the impact of strided convolutions on our sparse training. Also, we experimented with the learning rate schedule of the training regime. Our typical schedule for ResNet50v1.5 included learning rate drops at epochs 30,
60, and 90, but we experimented with placing the last drop at epoch 80 instead. Unlike typical ResNet50 training, which uses a batch size of 256 and starts the learning rate at 0.1, we used batch size 64 as this is what could fit in our GPUs. As suggested by Krizhevsky (2014), we scaled the starting learning rate by 1p
4 = 12 to 0.05 in order to compensate for the smaller batch size.

heading: 3.1 SPARSE TRAINING EXPERIMENTS
text: For ResNet50v1 and Tiny-ImageNet, we did gradual pruning until epoch 10. We subsequently enforced the final sparsity requirement, set a maximum number of non-zero values in each window/kernel of each layer, and fixed this sparsity pattern for the rest of training. We chose the 10th epoch as the final epoch of pruning because we wanted to see if we could fix the sparsity mask early in the training process.
For ResNet50v1 and ImageNet, our goal was to start pruning as early as possible, while maintaining high accuracy. We set our pruning era to epochs 0-30. Our hypothesis was that 30th epoch would be a suitable epoch to stop pruning, because this is where the learning rate is first decreased, in addition, there would be a large drop in accuracy if we stop pruning at epoch 20. However, this schedule did not perform well for ResNet50v1.5 and ImageNet and therefore we set our pruning era to 30-50.
To test training using CK and intra-epoch pruning, we mainly used ResNet50v1 and ResNet50v1.5 with ImageNet, but also performed CK pruning on ResNet50v1 and Tiny-ImageNet. We adopted a similar approach to Han et al. (2015b) to train with CK or intra-epoch pruning by setting the first epoch of pruning to 20, 30, or 40 with a pruning era of 20 or 30 epochs. Then, we continued to train the sparsified network until the final epoch.

heading: 3.2 ADVERSARIAL ROBUSTNESS
text: Since there was evidence that increasing sparsity lowers adversarial robustness (Wang et al., 2018), we evaluated this robustness in our models. To do so, we applied Fast Gradient Sign Method (FGSM) attacks defined in Goodfellow et al. (2014) on one of our sparse models, to generate its own adversarial examples, and measured the validation accuracy again. We used the same validation set as ImageNet and applied the attack’s image transformation to each input image. Moreover, we experimented with a variety of different ² in order to see how our accuracy decayed. Lastly, in our experiments we leveraged the examples provided in Pytorch tutorials 1.

heading: 4 RESULTS

heading: 4.1 RESNET50 ON TINY-IMAGENET
text: From our experiments with Tiny-Imagenet (shown in Appendix in Table 5), we see that even with up to 80% sparsity, both window and CK pruning are able to achieve levels of accuracy comparable to the dense baseline. CK pruning performs even better than the baseline.

heading: 4.2 RESNET50 ON IMAGENET
text: Our ResNet50 v1.5 experiments (Table 1 and Appendix Figure 11) with the first epoch of pruning at epoch 30 show that all of our pruning methods are able to achieve over 73% accuracy, and we can achieve above 74% accuracy up to 70% sparsity. Table 2 shows that on ResNet50 v1, our methods can achieve between 0.1-0.3% less than the baseline.
By comparing the sparsity curves of the window, CK, and intra-epoch pruning runs in Figure 4 (top right), we observe that the sparsity of window pruning is not as smooth as the other methods. This is likely indicative of the more rigid structure of CK and intra-epoch pruning, which causes the degree of sparsity to be much more uniform from epoch to epoch.
Figure 4 (top left, bottom right) also shows on ResNetv1.5, the window is slightly better than the CK and intra, which have similar performance, but the window is worse than the other two on
1https://pytorch.org/tutorials/beginner/fgsm_tutorial.html
ResNetv1. Furthermore, starting the pruning era later improves performance (Figure 4-(bottom left)).
Table 3 demonstrates that our sparsity mechanism can have a minimal drop in adversarial robustness (approximately 1-1.5%) compared to the dense baseline model, whereas other methods see more accuracy degradation (Wang et al., 2018).
The sparsity of each layer, depicted in Figure 5, emphasizes that early layers tolerate sparsity better, as they have consistently higher sparsity in the last 1×1 convolutional layer of each residual block. This may be due to their vicinity to the residual connection, which provides additional information to the layer.

heading: 4.3 DISCUSSION
text: Overall, we notice that there is a tolerance for sparsity (up to 70%), which yields around 1% accuracy loss compared to the dense baseline. However, this loss can be compensated by dropping the learning rate and performing another 10 epochs of training, which provides a 0.7-0.9% accuracy increase. With high levels of sparsity this extension is computationally cheap.
We observed the early stages of dense training are important for high accuracy, as longer periods of dense training consistently outperformed shorter ones. Moreover, widening the pruning era slightly (10 epochs) improves the final convergence accuracy (by around 0.2%).
We also observed that pushing the learning rate drop schedule to earlier epochs or aligning it with pruning era does not improve the final accuracy. However, pushing the last learning rate drop from epoch 90 to 80 can improve the accuracy by around 0.1%. (See Appendix Table 8 and Table 1)
We postulate that window pruning performs worse for ResNetv1.5 compared to ResNetv1 due to the strided nature of convolutions in ResNetv1.5.

heading: 5 RELATED WORK
text: To give a broad comparison stage, we extended Mostafa & Wang (2019)’s table on alternative sparsity mechanisms in Table 4 with respect to characteristics of their mechanisms: training/compression focus, regularization, the period in which pruning is applied, strictness of parameter budget, and pruning granularity. We explain each of the columns below:
1. Training Focus: Trying to train while maintaining/increasing sparsity of the network. The opposite is Compression Focus, i.e., methods that only seek to provide a smaller network for inference.
2. Regularization: Applying a regularization value to the loss, in order to find and prune irrelevant weights, while others use magnitude-based pruning.
3. Pruning Era: The period during training in which the pruning is applied.
4. Strictness of Parameter Budget Era wrt to Pruning: A strict parameter budget is fixed to the size of the final sparse model. Mostafa & Wang (2019) have a strict budget throughout training. Our method is only strict after the pruning era. Some networks do not have a strict parameter budget and only prune weights that appear to be irrelevant and without a sparsity target.
5. Pruning Granularity: The level of granularity within in the network at which values are pruned. For example, at the kernel level, we determine which values to prune by examining only the weights in the kernel (Mao et al., 2017). See Figure 2 for more information.
We chose these concepts because their characteristics can enable faster and lower-energy training. A strict parameter allows the hardware mapping to plan for a fixed number of multiply-accumulate
operations. The granularity of the sparsity mechanism indicates how easy it is to adapt the mechanism to an existing hardware. The coarser the granularity, the more adaptable it is to existing hardware (Mao et al., 2017). Regularization, although useful in forcing the network to learn prunable weights, adds more irregularity to computation flow. The pruning era starting at the beginning of the training enables us to train with a compressed network.
Mao et al. (2017) explores pruning on a range of granularities including window, kernel, and filter, and their effect on accuracy. They also qualitatively and quantitatively show that coarsegrain pruning, like kernel- or filter-level sparsity, is more energy-efficient due to fewer memory references. Similarly, our work surveys sparsity at the window, kernel, and filter levels. We improve on Mao et al.’s work in two ways. First, we show higher top-5 accuracy at higher sparsity levels on a complex benchmark, ImageNet on ResNet50 (92.338% at 40% CK sparsity), and we also show high top-1 accuracy whereas Mao et al. only report top-5.
Prunetrain (Lym et al., 2019) explores a way to create sparse channels and even layers to speed up training with around a 1% drop in accuracy. However, this requires a shift in the training mechanism, including a regularization term that could effect how the mechanism scales to large and distributed settings and that must be computed throughout training. The resulting network is only around 50% sparse and the accuracy loss due to sparse training is high enough that a baseline network with same accuracy could result into same computational savings by just terminating training at much earlier stage/epoch.
In contrast to other pruning mechanisms, our proposed window, CK, and combined sparsity mechanisms have strict parameter budgets after the pruning era. The CK and combined schemes have channel-level and kernel-level pruning granularities.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this work, we introduced techniques to train CNNs with structured sparsity and studied the tradeoffs associated with various implementation options. We demonstrated on ResNet50 with the full ImageNet dataset that the proposed sparse training method outperforms all related work and is comparable to a dense model in terms of convergence accuracy. We also observed that delaying the start of enforced, gradual pruning to at least epoch 20 was necessary to reach high convergence accuracy, highlighting the importance of the early epochs of dense training. Moreover, performing an additional 10 epochs of training provides substantial (around 1%) accuracy gains of the final model. In the future, we would like to study the tradeoffs of sparse training on low-precision networks.

heading: 7 APPENDIX

heading: 7.1 DETAILS OF PRUNING ALGORITHMS
text: Here we provide full descriptions of our other pruning methods and our general methodology sparse training.
Sparse Training Methodology Algorithm 2 shows how we modify normal training in order to train sparsely.
Algorithm 2 Pruning Algorithm
current_iter = 0 while training do
if current_iter > first epoch of pruning and current_iter < last epoch of pruning then mask = generate_sparsity_mask( θ, current_iter, sparsity threshold ) end if θpr uned = mask ⋂ θ ŷ = forward_pass( θpr uned , x ) θ = weight_update( y, ŷ, θpr uned ) current_iter = current_iter + 1
end while
Window Pruning Methodology Algorithm 3 shows how we prune with window sparsity.
Algorithm 3 Window Pruning Algorithm
generate_window_sparsity_mask(θl ayer , sparsity_threshold): for θ in θl ayer do
for all c in C do for all k in K do
cutoff_index = size(θc,k ) ∗ sparsity_threshold n = max(cutoff_index, size(θc,k ) − max_non_zero − 1) cutoff_value = nth largest value in θc,k for all i,j in R,S do
maski , j ,c,k = 1 if θi , j ,c,k > cutoff_value, else 0 end for
end for end for
end for
Combined Pruning Methodology To combine Window and CK pruning, we introduce intraepoch pruning. As shown by Algorithm 4, in a given epoch we first apply Window Pruning to each 3×3 convolutional layer at a fraction of the sparsity threshold for that epoch. Then, we prune the remaining fraction of the sparsity threshold with CK Pruning. The idea being that kernels that lose many of their parameters during window pruning can be fully pruned during the CK pruning phase. Our intuition is that first pruning parameters within a kernels guides the subsequent CK pruning towards the less important kernels. Thus, we pick out better kernels to prune. We also gain more structured sparsity but sacrifice the precision of window pruning.
Algorithm 4 Intra-Epoch Pruning Algorithm
generate_intra_epoch_sparsity_mask(θl ayer , sparsity_threshold): for θ in θl ayer do
window_mask = generate_ck_sparsity_mask(θ, sparsity_threshold) ck_mask = generate_ck_sparsity_mask(θ, sparsity_threshold) mask = window_mask and ck_mask
end for
For completeness, we also tried another method of combining called inter-epoch pruning, which involved splitting the pruning era into CK pruning and window pruning phases. However, from our initial experiments we determined that intra-epoch pruning, performed better (though it was more computationally expensive) than inter-epoch pruning. With inter-epoch pruning we were only able to achieve 74.1% top-1 accuracy with a first epoch of sparsity of 40 and a final sparsity of 40% on Resnet50v1.5 and Imagnet. The same setup trained with intra-epoch pruning achieved 74.9% accuracy. Thus, we pursued intra-epoch pruning as our method to combine the two sparsification methods.

heading: 7.2 ADDITIONAL DETAILS ON EXPERIMENTAL SETUP
text: This section goes into more detail on the exact details of the models and dataset combinations we sued for experimentation.

heading: 7.2.1 RESNET50 ON TINY-IMAGENET
text: For this training domain, we trained using the Tiny-imagenet dataset CS231N (2015) with resnet50 He et al. (2015). However, we changed the training mechanism in order to get validate our results. Each network we train for 40 epochs, with a batch size of 64. Additionally, we use the Adam optimizer to train with learning rate set to 0.001 and momentum set to 0.9. We also use weight decay set to 0.0001, and we anneal the learning rate to 0.0001 after 30 epochs of training in order to converge faster. We apply the same image transforms as on full Imagenet.
We chose this optimization method because we felt that it achieved a good overall accuracy at a baseline level and represents the results in Sun (2016) in their vanilla model. We do not use the same preprocessing or image transforms in the report Sun (2016). Moreover, we wanted a quick way to estimate how our method would perform on full Imagenet.

heading: 7.2.2 RESNET50 ON IMAGENET
text: Here, we train each network for 90 epochs with a reduced batch size of 128 instead of 256 because 256 would not fit on a GPU in addition to our pruning layers. We found that changing the batch size to 128 but retaining all other hyperparameters as specified in He et al. (2015) we were able to achieve the same benchmark 74.94% accuracy as the paper. We train for 90 epochs with SGD with momentum set to 0.9 and weight decay is 1×10−4. We set the initial learning rate to be 0.1 and then anneal the learning rate to 0.01 at epoch 30 and then finally to 0.001 at epoch 60.
For dataset transformations, we perform the same transformations as 2. This means that during training we perform a random sized crop to size 224x224, randomly flip the image horizontally, and normalize the image. The batches are shuffled during training. For validation, we resize the image to 256 and then center crop to size 224x224 and then normalize.

heading: 7.2.3 RESNET50V1.5 ON IMAGENET
text: We train our model for 90/100 epochs and use SGD with momentum (0.9) to optimize. The standard model says that learning rate should 0.1 for 256 batch size, but since that didn’t fit in our GPUs with our sparsity mechanism, we used batch size 64 and linearly scaled the learning rate to be 0.05. We set the learning rate decay such that we multiply by 0.1 after 30, 60, and 90 epochs. We have weight decay set to 1×10−4. Resnet1.5 is defined in detail here 3.

heading: 7.3 MISCELLANEOUS RESULTS

heading: 7.3.1 RESNET50 ON TINY-IMAGENET
text: Our models actually perform better than the baseline with the following configurations: window pruning with 60% sparsity as well as CK pruning with 20%, 60% and 80% sparsity. The number of
2https://github.com/pytorch/examples/tree/master/imagenet 3https://ngc.nvidia.com/catalog/model-scripts/nvidia:resnet_50_v1_5_
for_pytorch
epochs required to reach the converge to the final accuracies is the same for CK and earlier for window at 40% and 60% sparsity.

Review summaries:
This paper introduces a strategy to prune a convolutional neural network during training. To speed up training, the proposed method prunes the weights with the smallest magnitude during only a small number of epochs at the beginning of training, later on continuing training with a fixed sparsity pattern. Several granularity levels for convolutional and fully-connected layers are studied. Furthermore, the robustness of the resulting pruned networks to adversarial attacks is investigated.

This paper explores a series of incremental variations of existing pruning techniques for compressing Resnet-50 for ImageNet. Specifically, it proposes concentrating all pruning during an early "era" of training (the first 20-50 epochs out of 100 total). It also explores hybrids between sparse pruning and structured pruning. Finally, it considers the adversarial robustness of the resulting networks to the FGSM attack.