Neural Pruning via Growing Regularization

1 INTRODUCTION . As deep neural networks advance in recent years LeCun et al . ( 2015 ) ; Schmidhuber ( 2015 ) , their remarkable effectiveness comes at a cost of rising storage , memory footprint , computing resources and energy consumption Cheng et al . ( 2017 ) ; Deng et al . ( 2020 ) . Neural network pruning Han et al . ( 2015 ; 2016 ) ; Li et al . ( 2017 ) ; Wen et al . ( 2016 ) ; He et al . ( 2017 ) ; Gale et al . ( 2019 ) is deemed as a promising force to alleviate this problem . Since its early debut Mozer & Smolensky ( 1989 ) ; Reed ( 1993 ) , the central problem of neural network pruning has been ( arguably ) how to choose weights to discard , i.e. , the weight importance scoring problem LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Molchanov et al . ( 2017b ; 2019 ) ; Wang et al . ( 2019a ) ; He et al . ( 2020 ) . The approaches to the scoring problem generally fall into two groups : importance-based and regularization-based Reed ( 1993 ) . The former focuses on directly proposing certain theoretically sound importance criterion so that we can prune the unimportant weights once for all . Thus , the pruning process is typically one-shot . In contrast , regularization-based approaches typically select unimportant weights through training with a penalty term Han et al . ( 2015 ) ; Wen et al . ( 2016 ) ; Liu et al . ( 2017 ) . However , the penalty strength is usually maintained in a small regime to avoid damaging the model expressivity . Whereas , a large penalty strength can be helpful , specifically in two aspects . ( 1 ) A large penalty can push unimportant weights rather close to zero , then the pruning later barely hurts the performance even if the simple weight magnitude is adopted as criterion . ( 2 ) It is well-known that different weights of a neural network lie on the regions with different local quadratic structures , i.e. , Hessian information . Many methods try to tap into this to build a more accurate scoring LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Wang et al . ( 2019a ) ; Singh & Alistarh ( 2020 ) . However , for deep networks , it is especially hard to estimate Hessian . Sometimes , even the computing itself can be intractable without resorting to proper approximation Wang et al . ( 2019a ) . On this problem , we ask : Is it possible to exploit the Hessian information without knowing ∗Corresponding author . their specific values ? This is the second scenario where a growing regularization can help . We will show under a growing regularization , the weight magnitude will naturally separate because of their different underlying local quadratic structure , therein we can pick the unimportant weights more faithfully even using the simple magnitude-based criterion . Corresponding to these two aspects , we will present two algorithms based on a growing L2 regularization paradigm , in which the first highlights a better pruning schedule1 and the second explores a better pruning criterion . Our contributions . ( 1 ) We propose a simple yet effective growing regularization scheme , which can help transfer the model expressivity to the remaining part during pruning . The encouraging performance inspires us that the pruning schedule may be as critical as the weight importance criterion and deserve more research attention . ( 2 ) We further adopt growing regularization to exploit Hessian implicitly , without knowing their specific values . The method can help choose the unimportant weights more faithfully with a theoretically sound basis . In this regard , our paper is the first to show the connection between magnitude-based pruning and Hessian-based pruning , pointing out that the latter can be turned into the first one through our proposed growing regularization scheme . ( 3 ) The proposed two algorithms are easy to implement and scalable to large-scale datasets and networks . We show their effectiveness compared with many state-of-the-arts . Especially , the methods can work seamlessly for both filter pruning and unstructured pruning . 2 RELATED WORK . Regularization-based pruning . The first group of relevant works is those applying regularization to learn sparsity . The most famous probably is to use L0 or L1 regularization Louizos et al . ( 2018 ) ; Liu et al . ( 2017 ) ; Ye et al . ( 2018 ) due to their sparsity-inducing nature . In addition , the common L2 regularization is also explored for approximated sparsity Han et al . ( 2015 ; 2016 ) . The early papers focus more on unstructured pruning , which is beneficial to model compression yet not to acceleration . For structured pruning in favor of acceleration , Group-wise Brain Damage Lebedev & Lempitsky ( 2016 ) and SSL Wen et al . ( 2016 ) propose to use Group LASSO Yuan & Lin ( 2006 ) to learn regular sparsity , where the penalty strength is still kept in small scale because the penalty is uniformly applied to all the weights . To resolve this , Ding et al . ( 2018 ) and Wang et al . ( 2019c ) propose to employ different penalty factors for different weights , enabling large regularization . Importance-based pruning . Importance-based pruning tries to establish certain advanced importance criteria that can reflect the true relative importance among weights as faithfully as possible . The pruned weights are usually decided immediately by some proposed formula instead of by training ( although the whole pruning process can involve training , e.g. , iterative pruning ) . The most widely used criterion is the magnitude-based : weight absolute value for unstructured pruningHan et al . ( 2015 ; 2016 ) or L1/L2-norm for structured pruning Li et al . ( 2017 ) . This heuristic criterion was proposed a long time ago Reed ( 1993 ) and has been argued to be inaccurate . In this respect , improvement mainly comes from using Hessian information to obtain a more accurate approximation of the increased loss when a weight is removed LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) . Hessian is intractable to compute for large networks , so some methods ( e.g. , EigenDamage Wang et al . ( 2019a ) , WoodFisher Singh & Alistarh ( 2020 ) ) employ cheap approximation ( such as K-FAC Fisher Martens & Grosse ( 2015 ) ) to make the 2nd-order criteria tractable on deep networks . Note that , there is no a hard boundary between the importance-based and regularization-based . Many papers present their schemes in the combination of the two Ding et al . ( 2018 ) ; Wang et al . ( 2019c ) . The difference mainly lies in their emphasis : Regularization-based method focuses more on an advanced penalty scheme so that the subsequent pruning criterion can be simple ; while the importance-based one focus more on an advanced importance criterion itself . Meanwhile , regularization paradigm always involves iterative training , while the importance-based can be one-shot LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Wang et al . ( 2019a ) ( no training for picking weights to prune ) or involve iterative training Molchanov et al . ( 2017b ; 2019 ) ; Ding et al . ( 2019a ; b ) . Other model compression methods . Apart from pruning , there are also many other model compression approaches , e.g. , quantization Courbariaux & Bengio ( 2016 ) ; Courbariaux et al . ( 2016 ) ; Rastegari et al . ( 2016 ) , knowledge distillation Buciluǎ et al . ( 2006 ) ; Hinton et al . ( 2014 ) , low- 1By pruning schedule , we mean the way to remove weights ( e.g. , removing all weights in a single step or multi-steps ) , not the training schedule such as learning rate settings , etc . rank decomposition Denton et al . ( 2014 ) ; Jaderberg et al . ( 2014 ) ; Lebedev et al . ( 2014 ) ; Zhang et al . ( 2015 ) , and efficient architecture design or search Howard et al . ( 2017 ) ; Sandler et al . ( 2018 ) ; Howard et al . ( 2019 ) ; Zhang et al . ( 2018 ) ; Tan & Le ( 2019 ) ; Zoph & Le ( 2017 ) ; Elsken et al . ( 2019 ) . They are orthogonal to network pruning and can work with the proposed methods to compress more . 3 PROPOSED METHOD . 3.1 PROBLEM FORMULATION . Pruning can be formulated as a transformation T ( ∗ ) that takes a pretrained big model w as input and output a small model w1 , typically followed by a fine-tuning process F ( ∗ ) , which gives us the final output w2 = F ( w1 ) . We do not focus on F ( ∗ ) since it is simply a standard neural network training process , but focus on the process of w1 = T ( w ) . The effect of pruning can be further specified into two sub-transformations : ( 1 ) M = T1 ( w ) , which obtains a binary mask vector M that decides which weights will be removed ; ( 2 ) T2 ( w ) , which adjusts the values of remaining weights . That is , w1 = T ( w ) = T1 ( w ) T2 ( w ) =M T2 ( w ) . ( 1 ) For one-shot pruning , there is no iterative training at T1 . It depends on a specific algorithm to decide whether to adjust the remaining weights . For example , OBD LeCun et al . ( 1990 ) and L1norm pruning Li et al . ( 2017 ) do not adjust the kept weights ( i.e. , T2 is the identity function ) while OBS Hassibi & Stork ( 1993 ) does . For learning-based pruning , both T1 and T2 involve iterative training and the kept weights will always be adjusted . In the following , we will present our algorithms in the filter pruning scenario since we mainly focus on model acceleration instead of compression in this work . Nevertheless , the methodology can seamlessly translate to the unstructured pruning case . The difference lies in how we define the weight group : For filter pruning , a 4-d tensor convolutional filter ( or 2-d tensor for fully-connected layers ) is regarded as a weight group , while for unstructured pruning , a single weight makes a group . 3.2 PRUNING SCHEDULE : GREG-1 . Our first method ( GReg-1 ) is a variant of L1-norm pruning Li et al . ( 2017 ) . It obtains the mask M by L1-norm sorting but adjusts the kept weights via regularization . Specifically , given a pre-trained model w and layer pruning ratio rl , we sort the filters by L1-norm and set the mask to zero for those with the least norms . Then , unlike Li et al . ( 2017 ) which removes the unimportant weights immediately ( i.e. , one-shot fashion ) , we impose a growing L2 penalty to drive them to zero first : λj = λj + δλ , j ∈ { j |M [ j ] = 0 } , ( 2 ) where λj is the penalty factor for j-th weight ; δλ is the granularity in which we add up the penalty . Clearly , a smaller δλ means this regularization process smoother . Besides , λj is only updated every Ku iterations , which is a buffer time to let the network adapt to the new regularization . This algorithm is to explore whether the way we remove them ( i.e. , pruning schedule ) leads to a difference given the same weights to prune . Simple as it is , the scheme can bring significant accuracy gains especially under a large pruning ratio ( Tab . 1 ) . Note that , we intentionally set δλ the same for all the unimportant weights to keep the core idea simple . Natural extensions of using different penalty factors for different weights ( such as those in Ding et al . ( 2018 ) ; Wang et al . ( 2019c ) ) may be worth exploring but out of the scope of this work . When λj reaches a pre-set ceiling τ , we terminate the training and prune those with the least L1norms , then fine-tune . Notably , the pruning will barely hurt the accuracy since the unimportant weights have been compressed to typically less than 11000 the magnitude of remaining weights .

Summary: This paper explores how the basic L2 regularization can be exploited in a growing fashion for better deep network pruning. The authors proposed two algorithms in this work: (1) The first (called GReg-1) is a variant of the L1-norm based filter pruning method [Ref1]. The important/unimportant filters are decided by their L1-norms. Later the unimportant ones are forced to zero through the proposed rising penalty scheme. (2) The second algortihm (called Greg-2) imposes the rising L2 regularization on all the filters. It is theoretically shown in the paper that this makes the parameters to separate to different degrees according to their local curvatures (ie, Hessian values). The method takes advantage of this by driving the weights into two groups with stark magnitude difference and then prunes by the simple L1-norm criterion.  The two methods are demonstrated effective on CIFAR10/100 and ImageNet benchmarks in the comparison with many state-of-the-art methods.