Adaptive Activation-based Structured Pruning

1 INTRODUCTION . Deep neural networks ( DNNs ) have substantial compute and memory requirements . As deep learning becomes pervasive and moves towards edge devices , DNN deployment becomes harder because of the mistmatch between resource-hungry DNNs and resource-constrained edge devices . DNN pruning is a promising approach ( Li et al . ( 2016 ) ; Han et al . ( 2015 ) ; Molchanov et al . ( 2016 ) ; Theis et al . ( 2018 ) ; Renda et al . ( 2020 ) ) , which identifies the parameters ( or weight elements ) that do not contribute significantly to the accuracy and prunes them from the network . Recently , works based on the Lottery Ticket Hypothesis ( LTH ) have achieved great successes in creating smaller and more accurate models through iterative pruning with rewinding ( Frankle & Carbin ( 2018 ) ) . However , LTH has only been shown to work successfully with unstructured pruning which , unfortunately leads to models with low sparsity and difficult to accelerate on commodity hardware such as CPUs and GPUs ( e.g. , Hill et al . ( 2017 ) shows directly applying NVDIA cuSPARSE on unstructured pruned models can lead to 60ˆ slowdown on GPU compared to dense kernels . ) Moreover , most pruning methods require users to explore and adjust multiple hyper-parameters , e.g. , with LTH-based iterative pruning , users need to determine how many parameters to prune in each round . Tuning the pruning process is time consuming and often leads to sub-optimal results . We propose activation-based , adaptive , iterative structured pruning to find the “ winning ticket ” models that are at the same time hardware efficient and to automatically meet the users ’ model accuracy , size , and speed requirements . First , we propose an activation-based structured pruning method to identify and remove unimportant filters in an LTH-based iterative pruning ( with rewinding ) process . Specifically , we properly define an attention mapping function that takes a 2D activation feature maps of a filter as input , and outputs a 1D value used to indicate the importance of the filter . This approach is more effective than weight-value based filter pruning because activationbased attention values not only capture the features of inputs but also contain the information of convolution layers that act as feature detectors for prediction tasks . We then integrate this attention- based method into the LTH-based iterative pruning framework to prune the filters in each round and find the winning ticket that is small , accurate , and hardware-efficient . Second , we propose adaptive pruning that automatically optimizes the pruning process according to different user objectives . For latency-sensitive scenarios like interactive virtual assistants , we propose FLOPs-guaranteed pruning to achieve the best accuracy given the maximum amount of compute FLOPs ; For memory-limited environments like embedded systems , we propose modelsize-guaranteed pruning to achieve the best accuracy given the maximum amount of memory footprint ; For accuracy-critical applications such as those on self-driving cars , we propose accuracyguaranteed pruning to create the most resource-efficient model given the acceptable accuracy loss . Aiming for different targets , our method adaptively controls the pruning aggressiveness by adjusting the global threshold used to prune filters . Moreover , it considers the difference in each layer ’ s contributions to the model ’ s size and computational complexity and uses a per-layer threshold , calculated by dividing each layer ’ s remaining parameters or FLOPs by the entire model ’ s remaining parameters or FLOPs , to prune each layer with differentiated level of aggressiveness . Our results outperform the related works significantly in all cases targeting accuracy loss , parameters reduction , and FLOPs reduction . For example , on ResNet-56 with CIFAR-10 dataset , without accuracy drop , our method achieves the largest parameter reduction ( 79.11 % ) , outperforming the related works by 22.81 % to 66.07 % , and the largest FLOPs reduction ( 70.13 % ) , outperforming the related works by 14.13 % to 26.53 % . In addition , our method enables a pruned model the reach 0.6 % or 1.08 % higher accuracy than the original model but with only 30 % or 50 % of the original ’ s parameters . On ResNet-50 on ImageNet , for the same level of parameters and FLOPs reduction , our method achieves the smallest accuracy loss , lower than the related works by 0.08 % to 3.21 % ; and for the same level of accuracy loss , our method reduces significantly more parameters ( 6.45 % to 29.61 % higher than related works ) and more FLOPs ( 0.82 % to 17.2 % higher than related works ) . 2 BACKGROUND AND RELATED WORKS . Unstructured vs . Structured Pruning . Unstructured pruning ( LeCun et al . ( 1990 ) ; Han et al . ( 2015 ) ; Molchanov et al . ( 2017 ) ) is a fine-grained approach that prunes individual unimportant elements in weight tensors . It has less impact to model accuracy , compared to structured pruning , but unstructured pruned models are hard to accelerate on commodity hardware . Structured pruning is a coarse-grained approach that prunes entire regular regions of weight tensors according to some rule-based heuristics , such as L1-norm ( Li et al . ( 2016 ) ) , average percentage of zero ( Molchanov et al . ( 2016 ) ) , and other information considering the relationship between neighboring layers ( Theis et al . ( 2018 ) ; Lee et al . ( 2018 ) ) . Compared to unstructured pruning , it is more difficult to prune a model without causing accuracy loss using structured pruning , because by removing entire regions , it might remove weight elements that are important to the final accuracy ( Li et al . ( 2016 ) ) . However , structured pruned models can be mapped easily to general-purpose hardware and accelerated directly with off-the-shelf hardware and libraries ( He et al . ( 2018b ) ) . One Shot vs. Iterative Pruning . One-shot pruning prunes a pre-trained model and then retrains it once , whereas iterative pruning prunes and retrains the model in multiple rounds . Both techniques can choose either structured or unstructured pruning techniques . Recently , works based on the Lottery Ticket Hypothesis ( LTH ) have achieved great successes in creating smaller and more accurate models through iterative pruning with rewinding ( Frankle & Carbin ( 2018 ) ) . LTH posits that a dense randomly initialized network has a sub-network , termed as a winning ticket , which can achieve an accuracy comparable to the original network . At the beginning of each pruning round , it rewinds the weights and/or learning rate of the sub-network to some early epoch of the training phase of the original model to reduce the distance between the sub-network and original model and increase the change of finding the winning ticket . However , most of LTH-based works considered only unstructured pruning , e.g. , Iterative Magnitude Pruning ( IMP ) ( Frankle & Carbin ( 2018 ) ; Frankle et al . ( 2019 ) ) , which , as discussed above , is hardware-inefficient . It is non-trivial to design an iterative pruning method with structured pruning . To understand the state of iterative structured pruning , we experimented with IMP ’ s structured pruning counterpart—L1norm structured pruning ( ILP ) ( Li et al . ( 2016 ) ) which removes entire filters depending on their L1norm value . We observed that ILP can not effectively prune a model while maintaining its accuracy , e.g. , ILP can prune ResNet-50 by at most 11.5 % of parameters when the maximum accuracy loss Algorithm 1 Adaptive Iterative Structured Pruning Algorithm is limited to 1 % on ImageNet . Therefore , directly applying iterative pruning with existing weightmagnitude based structured pruning methods does not produce accurate pruned models . In this paper , we study how to produce small , accurate , and hardware-efficient models based on iterative structured pruning with rewinding . Automatic Pruning . For pruning to be useful in practice , it is important to automatically meet the pruning objectives for diverse ML applications and devices . Most pruning methods require users to explore and adjust multiple hyper-parameters , e.g. , with LTH-based iterative pruning , users need to determine how many parameters to prune in each round . Tuning the pruning process is time consuming and often leads to sub-optimal results . Therefore , we study how to automatically adapt the pruning process in order to meet the pruning objectives without user intervention . Some works ( Zoph et al . ( 2018 ) ; Cai et al . ; Ashok et al . ( 2017 ) ) use reinforcement-learning algorithms to find computationally efficient architectures . These works are in the area of Neural Architecture Search ( NAS ) , among which the most recent is AutoML for Model Compression ( AMC ) ( He et al . ( 2018b ) ) . AMC enables the model to arrive at the target speedup by limiting the action space ( the sparsity ratio for each layer ) , and it finds out the limit of compression that offers no loss of accuracy by tweaking the reward function . But this method has to explore over a large search space of all available layer-wise sparsity , which is time consuming when neural networks are large and datasets are complicated . 3 METHODOLOGY . Algorithm 1 illustrates the overall flow of the proposed adaptive activation-based structured pruning . To represent pruning of weights , we use a mask Mr t0 , 1ud for each weight tensor W rt Rd , where r is the pruning round number and t is the training epoch . Therefore , the pruned network at the end of training epoch T is represented by the element-wise product Mr dW rT . The first three steps are to train the original model to completion , while saving the weights at Epoch k. Steps 4–10 represent a pruning round . Step 4 prunes the model ( discussed in Section 3.1 ) . Step 5 ( optional ) and 6 perform rewinding . Step 7 retrains the pruned model for the remaining T ´ k epochs . Step 8 evaluates the pruned model according to the pruning target . If the target is not met , Step 9 resets the weights to an earlier round . Step 10 calculates the pruning threshold for the next pruning round following the adaptive pruning policy ( discussed in Section 3.2 ) . 3.1 ACTIVATION-BASED FILTER PRUNING . To realize iterative structured pruning , one can start with the state-of-the-art structured pruning methods and apply it iteratively . The widely used structured pruning method—L1-norm based structured pruning Renda et al . ( 2020 ) removes filters with the lowest L1-norm values , whereas the most recent method—Polarization-based structured pruning Zhuang et al . ( 2020 ) improves it by using a regularizer on scaling factors of filters and pruning filters whose scaling factors are below than a threshold . These two methods both assume that the weight values of a filter can be used as an indicator about the importance of that filter , much like how LTH uses weight values in unstructured pruning . However , we observe that weight-based structured pruning methods can not produce accurate pruned models . For example , to prune ResNet-56 on CIFAR-10 with no loss in top-1 accuracy , L1-norm based structured pruning can achieve at most only 1.15ˆmodel compression , and Polarization-norm based structured pruning can achieve at most only 1.89ˆ inference speedup . The reason is that , some filters , even though their weight values are small , can still produce useful non-zero activation values that are important for learning features during backpropagation . That is , filters with small values may have large activations . We propose that the activation values of filters are more effective in finding unimportant filters to prune . Activations like ReLu enable non-linear operations , and enable convolutional layers to act as feature detectors . If an activation value is small , then its corresponding feature detector is not important for prediction tasks . So activation values , i.e. , the intermediate output tensors after the non-linear activation , not only detect features of training dataset , but also contain the information of convolution layers that act as feature detectors for prediction tasks . We present a visual motivation in Figure 1 ( a ) . The figure shows the activation output of 16 filters of a convolution layer on one input image . The first image on the left is the original image , and the second image is the input features after data augmentation . We observe that some filters extract image features with high activation patterns , e.g. , the 6th and 12th filters . In comparison , the activation outputs of some filters are close to zero , such as the 2nd , 14th , and 16th . Therefore , from visual inspection , removing filters with weak activation patterns is likely to have low impact on the final accuracy of the pruned model . There is a natural connection between our activation-based pruning approach and the related attention-based knowledge transfer works ( Zagoruyko & Komodakis ( 2016 ) ) . Attention is a mapping function used to calculate the statistics of each element in the activations over the channel dimension . By minimizing the difference of the activation-based attention maps from intermediate layers between a teacher model and a student model , attention transfer enables the student to imitate the behaviour of the teacher . Our proposed pruning method builds upon the same theory that activation-based attention is a good indicator of filters regarding their ability to capture features , and it addresses the new challenges in using activations to guide automatic structured pruning . In the following , we describe how to prune filters based on its activation feature maps in each round of the iterative structured pruning process . Figure 1 ( b ) shows the inputs and outputs of a 2D convolution layer ( referred to as conv2d ) , followed by an activation layer . For the ith conv2d layer , let Xi Rn i´1ˆhi´1ˆwi´1 denote the input features , and F ij R ni´1ˆkiˆki be the jth filter , where hi´1 and wi´1 are the height and width of the input features , respectively , ni´1 is the number of input channels , ni is the number of output channels , and ki is the kernel size of the filter . The activation of the jth filter F ij after ReLu mapping is therefore denoted by A i j R hiˆwi . The attention mapping function takes a 2D activation Aij R hiˆwi of filter F ij as input , and outputs a 1D value which will be used as an indicator of the importance of filters . We consider three forms of activation-based attention maps , where p ě 1 and aik , l denotes every element of Aij : 1 ) Mean of attention values raised to the power of p : FmeanpAijq “ 1hiˆwi řhi k “ 1 řwi l “ 1 ˇ ˇ ˇ aik , l ˇ ˇ ˇ p ; 2 ) Max of atten- Algorithm 2 Accuracy-guaranteed Adaptive Pruning 1 : Input : Target Accuracy Loss AccLossTarget 2 : Output : A small pruned model with an acceptable accuracy 3 : Initialize : T “ 0.0 , λ “ 0.005 . 4 : for pruning round r ( r ě 1 ) do 5 : Prune the model using T rrs ( Refer to Algorithm 3 ) 6 : Train the pruned model , evaluate its accuracy Accrrs 7 : Calculate the accuracy loss AccLossrrs : AccLossrrs “ Accr0s ´Accrrs 8 : if AccLossrrs ă AccLossTarget then 9 : if the changes of model size are within 0.1 % for several rounds then 10 : Terminate 11 : else 12 : λrr ` 1s “ λrrs 13 : T rr ` 1s “ T rrs ` λrr ` 1s 14 : end if 15 : else 16 : Find the last acceptable round k 17 : if k has been used to roll back for several times then 18 : Mark k as unacceptable 19 : Go to Step 15 20 : else 21 : Roll back model weights to round k 22 : λrr ` 1s “ λrrs { 2.0pN ` 1q ( N is the number of times for rolling back to round k ) 23 : T rr ` 1s “ T rks ` λrr ` 1s 24 : end if 25 : end if 26 : end for tion values raised to the power of p : FmaxpAijq “ maxl “ 1 , hiˆwi ˇ ˇ ˇ aik , l ˇ ˇ ˇ p ; and 3 ) Sum of attention values raised to the power of p : FsumpAijq “ řhi k “ 1 řwi l “ 1 ˇ ˇ ˇ aik , l ˇ ˇ ˇ p . We choose FmeanpAijq with p equals to 1 as the indicator to identify and prune unimportant filters , and our method removes the filters whose attention value is lower than the pruning threshold . See Section 5 for an ablation study on these choices .

Summary: This paper proposes iterative structured pruning methods using activation-based attention feature maps and an adaptive threshold selection strategy. Inspired by attention transfer, Activation-based attention feature maps are constructed as the important evaluation of filters in each layer. Adaptive threshold selection strategy decides the number of removed filters, which satisfies one of three adaptive pruning policies. Experimental results on CIFAR-10 and ImageNet with ResNet architectures show performance gains over several state-of-the-art methods.