Starfire: Regularization-Free Adversarially-Robust Structured Sparse Training

This paper studies structured sparse training of CNNs with a gradual pruning technique that leads to fixed , sparse weight matrices after a set number of epochs . We simplify the structure of the enforced sparsity so that it reduces overhead caused by regularization . The proposed training methodology explores several options for structured sparsity . We study various tradeoffs with respect to pruning duration , learning-rate configuration , and the total length of training . We show that our method creates a sparse version of ResNet50 and ResNet50v1.5 on full ImageNet while remaining within a negligible < 1 % margin of accuracy loss . To ensure that this type of sparse training does not harm the robustness of the network , we also demonstrate how the network behaves in the presence of adversarial attacks . Our results show that with 70 % target sparsity , over 75 % top-1 accuracy is achievable . 1 INTRODUCTION . Pruning weights can compress a network into a smaller model so that the model can fit into faster/smaller memory and therefore result in execution speedups ( Han et al. , 2016 ; 2015a ) . To increase the accuracy Han et al . ( 2015b ) and Mao et al . ( 2017 ) explore training the network dense after pruning . The resulting network can maintain accuracy based on the specified level of sparsity ( Mostafa & Wang , 2019 ; Zhu & Gupta , 2017 ; Han et al. , 2015a ) . Structured sparsity has been explored for RNNs and also CNNs where a certain number of nonzeros is allowed across various cross-sections of the weight tensors . These methods aim to speed up computation and reach some final level of sparsity for deployment . Narang et al . ( 2017 ) have shown promising results for structured training of RNNs while sparse CNNs could not achieve the same performance ( Mao et al. , 2017 ) . Recent work has demonstrated that structurally sparse training can speed up execution on GPUs ( He et al. , 2017 ; Lym et al. , 2019 ; Zhu & Gupta , 2017 ) . However , these training mechanisms add regularization and computational overhead to eliminate unnecessary weights . The regularization term modifies the original training and can be expensive in hardware . While enforcing coarse-grain sparsity Lym et al . ( 2019 ) provides significant speedups , the final network contains an insufficient degree of sparsity for deployment on edge devices . Mostafa & Wang ( 2019 ) show that with adaptive sparse training and dynamic reallocation of nonzeros sparsity levels up to 80 % can be achieved . However , even though an additional 10 epochs of training are required , an accuracy loss of around 1-2 % is still observed . The main drawback is the overhead incurred while implementing such technique on the target platform . Continuous reconfiguration of the sparsity pattern is expensive as it does not allow for compression of weights during training . To achieve speedups and a desired final degree of sparsity , we aim to apply the techniques in Han et al . ( 2015b ) and Mao et al . ( 2017 ) at earlier stages in training at higher frequency within a period which we call the pruning era , usually a period of 20-30 epochs . During the pruning era , with fine granularity of at most a kernel size , we exploit one of the three proposed sparsity regimes . Subsequently , we fix the mask for the rest of the training to speed it up . Our motivation came from the insight that having a fixed sparse multiply-accumulate pattern allows weight compression during training and can save compute and energy in hardware ( Han et al. , 2016 ) . We explore the impact of various pruning granularities , sparsity levels , and learning-rate schedules on the network ’ s convergence as well as adversarial robustness for CNNs like Resnet-50 ( He et al. , 2015 ) on ImageNet and tinyImagenet ( CS231N , 2015 ) . Recent literature has shown that adversarial attacks are more successful on pruned neural networks than they are on regular neural networks ( Wang et al. , 2018 ) . Given the danger of adversarial attacks in real world situations , we find that it is important to evaluate our sparsity techniques under adversarial robustness . We leverage the FGSM mechanism ( Goodfellow et al. , 2014 ) to evaluate the adversarial robustness on our sparse models . This paper makes the following contributions : 1 . We propose a mechanism to train and prune a convolutional network during the earlier stages of training such that this sparsity can be harvested for the computational speedups . To do this , we fix the sparse weight masks for the remainder of the training . 2 . For fully connected sparsification , we eliminate blocks of fully connected weights based on their connection to the zeros in the previous convolutional layer . 3 . We enforce structural , regularization free , magnitude-based pruning across two distinct dimensions and a combined version . These dimensions are inside convolution window R ×S and across input/output feature matrix ( C K ) . 4 . Our sparse models are as robust to adversarial FGSM attacks as fully dense models . 5 . We demonstrate that early stage dense training is crucial for maintaining high accuracy . 6 . The proposed technique is tolerant to sparsity levels of up to 60-70 % with under 1 % accuracy degradation . We can compensate by scheduling an extra learning rate drop and training for an extra 10 epochs . The rest of the paper is organized as follows . Section 2 explains our pruning methodology . Section 3 describes the experimental setup framework . Section 4 presents results and discusses their interpretation . Section 5 presents the related work . Section 6 concludes the paper . 2 PRUNING METHODOLOGY . Our proposed pruning mechanism works by always pruning the weights of smallest magnitude after each weight update . After a forward and backward pass ( one batch update ) , the model is pruned . If a weight is already zero , the gradient is also set to zero . This means that once a weight becomes zero , it will remain zero for the rest of the training period . This mechanism is similar to Han et al . ( 2015b ) , except that we only prune in the earlier stages of the training as opposed to post training . Additionally , this work is similar to Narang et al . ( 2017 ) although we set the sparsity threshold instead of using a heuristic to calculate it . We chose this pruning mechanism because of its negligible computational overhead . In our pruning algorithm , the sparsity threshold refers to the percentage of weights in the network that are currently pruned . Before or during the first epoch of pruning , we will have a sparsity threshold of zero . As we continue training , we gradually increase the sparsity threshold so that by the final epoch of pruning the network sparsity will have reached our final , desired threshold . Finally , we also define the pruning era to be the epochs between the first and final epochs of pruning depicted in Figure 1b . We evaluate the pruning mask after every training step until we reach the final epoch of pruning . After the final epoch , the pruned values in the network will remain zero for the rest of training ; no new pruning will occur , and only the non-zero weights will be updated . 2.1 PRUNING METHODOLOGY BY LAYER . Pruning the smallest magnitude weights in the entire network is inefficient because it involves sorting the weights over the network . Instead , we prune the smallest magnitude weights or sum of weights , within a certain locale of the network . When pruning , we examine each layer individually and apply a separate technique to evaluate which weights to prune , depending on the type of layer we are currently pruning . 2.1.1 CONVOLUTIONAL LAYER PRUNING . Window pruning for 3x3 Convolutional Layers Figure 2a shows the result of a pruned 3×3 convolutional weight tensor under the window pruning method . In this scheme , window layer pruning refers to pruning of weights within the 3×3 convolution kernels . We allow a maximum number of non-zero values for each kernel in the 3×3 convolutional layers and eliminate the weights of smallest magnitude . Algorithm 1 CK Pruning Algorithm generate_ck_sparsity_mask ( θl ayer , sparsity_threshold ) : for θ in θl ayer do for all c in C do for all k in K do kernel_maxc , k = max ( θc , k ) end for cutoff_index = size ( θc ) ∗ sparsity_threshold n = max ( cutoff_index , size ( θc ) − max_non_zero − 1 ) cutoff_value = nth largest value in kernel_maxc for all k in K do maskc , k = 1 if kernel_maxc , k > cutoff_value , else 0 end for end for end for CK Pruning Methodology Figure 2b shows the result of a pruned 3×3 convolutional weight tensor under the CK pruning method . In this scheme , the weights of a certain layer can be viewed as a CK matrix of R×S kernels . The CK pruning method involves pruning the 3×3 convolutions along the channel and kernel dimensions of each convolutional filter , i.e. , we prune whole kernels ( CK matrix of R×S windows ) at once and can ultimately prune all the input channels in an output channel . As defined by Algorithm 1 , we determine which filter to prune by examining the max of the magnitudes of all the weights in a kernel , which is the max of nine weights . This max is used to evaluate whether the whole kernel should be pruned or not . Combined Pruning Methodology To combine window and CK pruning , we introduce an intraepoch combined pruning method , which we refer to hereafter as “ intra-epoch pruning ” or “ intra ” , for short . As shown by appendix Algorithm 4 in the Appendix , in a given epoch we first apply window pruning to each 3×3 convolutional layer at a fraction of the sparsity threshold for that epoch . Then , we prune the remaining fraction of the sparsity threshold with CK Pruning . 2.1.2 FULLY CONNECTED PRUNING . Like pruning for convolutional layers , we apply a two-tier pruning scheme from Mao et al . ( 2017 ) for fully connected layers : micro-level pruning within a block and macro-level pruning that eliminates entire blocks . Block FC Pruning Figure 2d refers to pruning of individual blocks . Here , we prune an entire n×n ( n < 5 ) window within the dense layer and create coarse grained sparsity . To do this , we sum the magnitude of the weights in each window and prune the windows with the smallest magnitude . Fine FC Pruning Figure 2c refers to the pruning of individual weights . Here , we prune the individual weights in the entire FC Layer , where we compare the magnitude of all the weights to each other . The produced zero patterns in the last convolution layer allow for eliminating more weights in fully connected layer as depicted in Figure 3 . If all the C windows for a specific Ki are zeros , the output activation for the corresponding Ki is also zero . The corresponding neurons in the following fully connected layer are therefore receiving zero input activations and can be eliminated along with their associated weights . This enables us to get sparsity without having to evaluate the weights in the fully connected layer . When pruning just the small weights in the FC layer , one can inadvertently cut off relevant connections between the input and output layers . Accordingly , we structure the pruning mechanism such that each output neuron should be influenced by the input . This means every column in the weight matrix of the fully connected layer in Figure 3 has at least one non-zero element .

Summary: This paper introduces a strategy to prune a convolutional neural network during training. To speed up training, the proposed method prunes the weights with the smallest magnitude during only a small number of epochs at the beginning of training, later on continuing training with a fixed sparsity pattern. Several granularity levels for convolutional and fully-connected layers are studied. Furthermore, the robustness of the resulting pruned networks to adversarial attacks is investigated.