Neural Pruning via Growing Regularization

1 INTRODUCTION . As deep neural networks advance in recent years LeCun et al . ( 2015 ) ; Schmidhuber ( 2015 ) , their remarkable effectiveness comes at a cost of rising storage , memory footprint , computing resources and energy consumption Cheng et al . ( 2017 ) ; Deng et al . ( 2020 ) . Neural network pruning Han et al . ( 2015 ; 2016 ) ; Li et al . ( 2017 ) ; Wen et al . ( 2016 ) ; He et al . ( 2017 ) ; Gale et al . ( 2019 ) is deemed as a promising force to alleviate this problem . Since its early debut Mozer & Smolensky ( 1989 ) ; Reed ( 1993 ) , the central problem of neural network pruning has been ( arguably ) how to choose weights to discard , i.e. , the weight importance scoring problem LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Molchanov et al . ( 2017b ; 2019 ) ; Wang et al . ( 2019a ) ; He et al . ( 2020 ) . The approaches to the scoring problem generally fall into two groups : importance-based and regularization-based Reed ( 1993 ) . The former focuses on directly proposing certain theoretically sound importance criterion so that we can prune the unimportant weights once for all . Thus , the pruning process is typically one-shot . In contrast , regularization-based approaches typically select unimportant weights through training with a penalty term Han et al . ( 2015 ) ; Wen et al . ( 2016 ) ; Liu et al . ( 2017 ) . However , the penalty strength is usually maintained in a small regime to avoid damaging the model expressivity . Whereas , a large penalty strength can be helpful , specifically in two aspects . ( 1 ) A large penalty can push unimportant weights rather close to zero , then the pruning later barely hurts the performance even if the simple weight magnitude is adopted as criterion . ( 2 ) It is well-known that different weights of a neural network lie on the regions with different local quadratic structures , i.e. , Hessian information . Many methods try to tap into this to build a more accurate scoring LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Wang et al . ( 2019a ) ; Singh & Alistarh ( 2020 ) . However , for deep networks , it is especially hard to estimate Hessian . Sometimes , even the computing itself can be intractable without resorting to proper approximation Wang et al . ( 2019a ) . On this problem , we ask : Is it possible to exploit the Hessian information without knowing ∗Corresponding author . their specific values ? This is the second scenario where a growing regularization can help . We will show under a growing regularization , the weight magnitude will naturally separate because of their different underlying local quadratic structure , therein we can pick the unimportant weights more faithfully even using the simple magnitude-based criterion . Corresponding to these two aspects , we will present two algorithms based on a growing L2 regularization paradigm , in which the first highlights a better pruning schedule1 and the second explores a better pruning criterion . Our contributions . ( 1 ) We propose a simple yet effective growing regularization scheme , which can help transfer the model expressivity to the remaining part during pruning . The encouraging performance inspires us that the pruning schedule may be as critical as the weight importance criterion and deserve more research attention . ( 2 ) We further adopt growing regularization to exploit Hessian implicitly , without knowing their specific values . The method can help choose the unimportant weights more faithfully with a theoretically sound basis . In this regard , our paper is the first to show the connection between magnitude-based pruning and Hessian-based pruning , pointing out that the latter can be turned into the first one through our proposed growing regularization scheme . ( 3 ) The proposed two algorithms are easy to implement and scalable to large-scale datasets and networks . We show their effectiveness compared with many state-of-the-arts . Especially , the methods can work seamlessly for both filter pruning and unstructured pruning . 2 RELATED WORK . Regularization-based pruning . The first group of relevant works is those applying regularization to learn sparsity . The most famous probably is to use L0 or L1 regularization Louizos et al . ( 2018 ) ; Liu et al . ( 2017 ) ; Ye et al . ( 2018 ) due to their sparsity-inducing nature . In addition , the common L2 regularization is also explored for approximated sparsity Han et al . ( 2015 ; 2016 ) . The early papers focus more on unstructured pruning , which is beneficial to model compression yet not to acceleration . For structured pruning in favor of acceleration , Group-wise Brain Damage Lebedev & Lempitsky ( 2016 ) and SSL Wen et al . ( 2016 ) propose to use Group LASSO Yuan & Lin ( 2006 ) to learn regular sparsity , where the penalty strength is still kept in small scale because the penalty is uniformly applied to all the weights . To resolve this , Ding et al . ( 2018 ) and Wang et al . ( 2019c ) propose to employ different penalty factors for different weights , enabling large regularization . Importance-based pruning . Importance-based pruning tries to establish certain advanced importance criteria that can reflect the true relative importance among weights as faithfully as possible . The pruned weights are usually decided immediately by some proposed formula instead of by training ( although the whole pruning process can involve training , e.g. , iterative pruning ) . The most widely used criterion is the magnitude-based : weight absolute value for unstructured pruningHan et al . ( 2015 ; 2016 ) or L1/L2-norm for structured pruning Li et al . ( 2017 ) . This heuristic criterion was proposed a long time ago Reed ( 1993 ) and has been argued to be inaccurate . In this respect , improvement mainly comes from using Hessian information to obtain a more accurate approximation of the increased loss when a weight is removed LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) . Hessian is intractable to compute for large networks , so some methods ( e.g. , EigenDamage Wang et al . ( 2019a ) , WoodFisher Singh & Alistarh ( 2020 ) ) employ cheap approximation ( such as K-FAC Fisher Martens & Grosse ( 2015 ) ) to make the 2nd-order criteria tractable on deep networks . Note that , there is no a hard boundary between the importance-based and regularization-based . Many papers present their schemes in the combination of the two Ding et al . ( 2018 ) ; Wang et al . ( 2019c ) . The difference mainly lies in their emphasis : Regularization-based method focuses more on an advanced penalty scheme so that the subsequent pruning criterion can be simple ; while the importance-based one focus more on an advanced importance criterion itself . Meanwhile , regularization paradigm always involves iterative training , while the importance-based can be one-shot LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Wang et al . ( 2019a ) ( no training for picking weights to prune ) or involve iterative training Molchanov et al . ( 2017b ; 2019 ) ; Ding et al . ( 2019a ; b ) . Other model compression methods . Apart from pruning , there are also many other model compression approaches , e.g. , quantization Courbariaux & Bengio ( 2016 ) ; Courbariaux et al . ( 2016 ) ; Rastegari et al . ( 2016 ) , knowledge distillation Buciluǎ et al . ( 2006 ) ; Hinton et al . ( 2014 ) , low- 1By pruning schedule , we mean the way to remove weights ( e.g. , removing all weights in a single step or multi-steps ) , not the training schedule such as learning rate settings , etc . rank decomposition Denton et al . ( 2014 ) ; Jaderberg et al . ( 2014 ) ; Lebedev et al . ( 2014 ) ; Zhang et al . ( 2015 ) , and efficient architecture design or search Howard et al . ( 2017 ) ; Sandler et al . ( 2018 ) ; Howard et al . ( 2019 ) ; Zhang et al . ( 2018 ) ; Tan & Le ( 2019 ) ; Zoph & Le ( 2017 ) ; Elsken et al . ( 2019 ) . They are orthogonal to network pruning and can work with the proposed methods to compress more . 3 PROPOSED METHOD . 3.1 PROBLEM FORMULATION . Pruning can be formulated as a transformation T ( ∗ ) that takes a pretrained big model w as input and output a small model w1 , typically followed by a fine-tuning process F ( ∗ ) , which gives us the final output w2 = F ( w1 ) . We do not focus on F ( ∗ ) since it is simply a standard neural network training process , but focus on the process of w1 = T ( w ) . The effect of pruning can be further specified into two sub-transformations : ( 1 ) M = T1 ( w ) , which obtains a binary mask vector M that decides which weights will be removed ; ( 2 ) T2 ( w ) , which adjusts the values of remaining weights . That is , w1 = T ( w ) = T1 ( w ) T2 ( w ) =M T2 ( w ) . ( 1 ) For one-shot pruning , there is no iterative training at T1 . It depends on a specific algorithm to decide whether to adjust the remaining weights . For example , OBD LeCun et al . ( 1990 ) and L1norm pruning Li et al . ( 2017 ) do not adjust the kept weights ( i.e. , T2 is the identity function ) while OBS Hassibi & Stork ( 1993 ) does . For learning-based pruning , both T1 and T2 involve iterative training and the kept weights will always be adjusted . In the following , we will present our algorithms in the filter pruning scenario since we mainly focus on model acceleration instead of compression in this work . Nevertheless , the methodology can seamlessly translate to the unstructured pruning case . The difference lies in how we define the weight group : For filter pruning , a 4-d tensor convolutional filter ( or 2-d tensor for fully-connected layers ) is regarded as a weight group , while for unstructured pruning , a single weight makes a group . 3.2 PRUNING SCHEDULE : GREG-1 . Our first method ( GReg-1 ) is a variant of L1-norm pruning Li et al . ( 2017 ) . It obtains the mask M by L1-norm sorting but adjusts the kept weights via regularization . Specifically , given a pre-trained model w and layer pruning ratio rl , we sort the filters by L1-norm and set the mask to zero for those with the least norms . Then , unlike Li et al . ( 2017 ) which removes the unimportant weights immediately ( i.e. , one-shot fashion ) , we impose a growing L2 penalty to drive them to zero first : λj = λj + δλ , j ∈ { j |M [ j ] = 0 } , ( 2 ) where λj is the penalty factor for j-th weight ; δλ is the granularity in which we add up the penalty . Clearly , a smaller δλ means this regularization process smoother . Besides , λj is only updated every Ku iterations , which is a buffer time to let the network adapt to the new regularization . This algorithm is to explore whether the way we remove them ( i.e. , pruning schedule ) leads to a difference given the same weights to prune . Simple as it is , the scheme can bring significant accuracy gains especially under a large pruning ratio ( Tab . 1 ) . Note that , we intentionally set δλ the same for all the unimportant weights to keep the core idea simple . Natural extensions of using different penalty factors for different weights ( such as those in Ding et al . ( 2018 ) ; Wang et al . ( 2019c ) ) may be worth exploring but out of the scope of this work . When λj reaches a pre-set ceiling τ , we terminate the training and prune those with the least L1norms , then fine-tune . Notably , the pruning will barely hurt the accuracy since the unimportant weights have been compressed to typically less than 11000 the magnitude of remaining weights .

Summary: The authors propose regularization-based pruning methods with the penalty factors uniformly increased over the training session. The first algorithm (GReg-1) sorts the filters by L1-norm and only applies the increasing regularization to the “unimportant” filters; the second one (GReg-2) applies the increasing regularization to all the filters. The experiments are very extensive and convincing to support the claimed contributions.