Layer-adaptive Sparsity for the Magnitude-based Pruning

1 INTRODUCTION . Neural network pruning is an art of removing “ unimportant weights ” from a model , with an intention to meet practical constraints ( Han et al. , 2015 ) , mitigate overfitting ( Hanson & Pratt , 1988 ) , enhance interpretability ( Mozer & Smolensky , 1988 ) , or deepen our understanding on neural network training ( Frankle & Carbin , 2019 ) . Yet , the importance of weight is still a vaguely defined notion , and thus a wide range of pruning algorithms based on various importance scores has been proposed . One popular approach is to estimate the loss increment from removing the target weight to use as an importance score , e.g. , Hessian-based approximations ( LeCun et al. , 1989 ; Hassibi & Stork , 1993 ; Dong et al. , 2017 ) , coreset-based estimates ( Baykal et al. , 2019 ; Mussay et al. , 2020 ) , convex optimization ( Aghasi et al. , 2017 ) , and operator distortion ( Park et al. , 2020 ) . Other approaches include on-the-fly1 regularization ( Louizos et al. , 2018 ; Xiao et al. , 2019 ) , Bayesian methods ( Molchanov et al. , 2017 ; Louizos et al. , 2017 ; Dai et al. , 2018 ) , and reinforcement learning ( Lin et al. , 2017 ) . Recent discoveries ( Gale et al. , 2019 ; Evci et al. , 2020 ) demonstrate that , given an appropriate choice of layerwise sparsity , simply pruning on the basis of weight magnitude yields a surprisingly powerful unstructured pruning scheme . For instance , Gale et al . ( 2019 ) evaluates the performance of magnitudebased pruning ( MP ; Han et al . ( 2015 ) ; Zhu & Gupta ( 2018 ) ) with an extensive hyperparameter tuning , and shows that MP achieves comparable or better performance than state-of-the-art pruning algorithms that use more complicated importance scores . To arrive at such a performance level , the authors introduce the following handcrafted heuristic : Leave the first convolutional layer fully dense , and prune up to only 80 % of weights from the last fully-connected layer ; the heuristic is motivated by the sparsity pattern from other state-of-the-art algorithms ( Molchanov et al. , 2017 ) and additional experimental/architectural observations . Unfortunately , there is an apparent lack of consensus on “ how to choose the layerwise sparsity ” for the magnitude-based pruning . Instead , the layerwise sparsity is selected mostly on an algorithm-byalgorithm basis . One common method is the global MP criteria ( see , e.g. , Morcos et al . ( 2019 ) ) , ∗Work done at KAIST 1i.e. , simultaneously training and pruning where the layerwise sparsity is automatically determined by using a single global threshold on weight magnitude . Lin et al . ( 2020 ) propose a magnitude-based pruning algorithm using a feedback signal , using a heuristic rule of keeping the last fully connected layer dense . A recent work by Evci et al . ( 2020 ) proposes a magnitude-based dynamic sparse training method , adopting layerwise sparsity inspired from the network science approach toward neural network pruning ( Mocanu et al. , 2018 ) . Contributions . In search of a “ go-to ” layerwise sparsity for MP , we take a model-level distortion minimization perspective towards MP . We build on the observation of Dong et al . ( 2017 ) ; Park et al . ( 2020 ) that each neural network layer can be viewed as an operator , and MP is a choice that incurs minimum ` 2 distortion to the operator output ( given a worst-case input signal ) . We bring the perspective further to examine the “ model-level ” distortion incurred by pruning a layer ; preceding layers scale the input signal to the target layer , and succeeding layers scale the output distortion . Based on the distortion minimization framework , we propose a novel importance score for global pruning , coined LAMP ( Layer-Adaptive Magnitude-based Pruning ) . The LAMP score is a rescaled weight magnitude , approximating the model-level distortion from pruning . Importantly , the LAMP score is designed to approximate the distortion on the model being pruned , i.e. , all connections with a smaller LAMP score than the target weight is already pruned . Global pruning2 with the LAMP score is equivalent to the MP with an automatically determined layerwise sparsity . At the same time , pruning with LAMP keeps the benefits of MP intact ; the LAMP score is efficiently computable , hyperparameter-free , and does not rely on any model-specific knowledge . We validate the effectiveness of LAMP under a diverse experimental setup , encompassing various convolutional neural network architectures ( VGG-16 , ResNet-18/34 , DenseNet-121 , EfficientNet-B0 ) and various image datasets ( CIFAR-10/100 , SVHN , Restricted ImageNet ) . In all considered setups , LAMP consistently outperforms the baseline layerwise sparsity selection schemes . We also perform additional ablation studies with one-shot pruning and weight-rewinding setup to confirm that LAMP performs reliably well under a wider range of scenarios . Organization . In Section 2 , we briefly describe existing methods to choose the layerwise sparsity for magnitude-based pruning . In Section 3 , we formally introduce LAMP and describe how the ` 2 distortion minimization perspective motivates the LAMP score . In Section 4 , we empirically validate the effectiveness and versatility of LAMP . In Section 5 , we take a closer look at the layerwise sparsity discovered by LAMP and compare with baseline methods and previously proposed handcrafted heuristics . In Section 6 , we summarize our findings and discuss future directions . Appendices include the experimental details ( Appendix A ) , complexity analysis ( Appendix B ) , derivation of the LAMP score ( Appendix C ) , additional experiments on Transformer ( Appendix D ) , and detailed experimental results with standard deviations ( Appendix E ) . 2 RELATED WORK . This section gives a ( necessarily non-exhaustive ) survey of various layerwise sparsity selection schemes used for magnitude-based pruning algorithms . Magnitude-based pruning of neural networks dates back to the early works of Janowsky ( 1989 ) ; LeCun et al . ( 1989 ) , and has been actively studied 2i.e. , using a global threshold for LAMP score again under the context of model compression since the work of Han et al . ( 2015 ) . In Han et al . ( 2015 ) , the authors propose an iterative pruning scheme where the layerwise pruning threshold is determined by the standard-deviation-based heuristic . Zhu & Gupta ( 2018 ) propose a uniform pruning algorithm with a carefully tuned gradual pruning schedule combined with weight re-growths . Gale et al . ( 2019 ) refine the algorithm by adding a heuristic constraint of keeping the first convolutional layer fully dense and keeping at least 20 % of the weights surviving in the last fully-connected layer . MP has also been widely used in the context of “ pruning at initialization. ” Frankle & Carbin ( 2019 ) combine MP with weight rewinding to discover efficiently trainable subnetworks : for small nets , the authors employ uniform layerwise sparsity , but use different rates for convolutional layers and fully-connected layers ( with an added heuristic on the last fully-connected layer ) ; for larger nets , authors use global MP . Morcos et al . ( 2019 ) consider transferring the “ winning ticket ” initializations , using the global MP . Evci et al . ( 2020 ) proposes a training scheme for sparsely initialized neural networks , where the layerwise sparsity is given by the Erdős-Rényi kernel method ; the method generalizes the scheme initially proposed by Mocanu et al . ( 2018 ) to convolutional neural networks . We note that there is a line of results on the trainable layerwise sparsity ; we refer the interested readers to the recent work of Kusupati et al . ( 2020 ) for a concise survey . However , we do not make direct comparisons to these methods , as our primary purpose is to deliver an easy-to-use layerwise sparsity selection scheme without requiring the modification of training objective , or an extensive hyperparameter tuning . We also note that we focus on the unstructured sparsity . While such unstructured pruning techniques have been considered less practical ( compared to structured pruning ) , several recent breakthroughs provide promising methods to bridge this gap ; see Gale et al . ( 2020 ) ; Elsen et al . ( 2020 ) . 3 LAYER-ADAPTIVE MAGNITUDE-BASED PRUNING ( LAMP ) . We now formally introduce the Layer-Adaptive Magnitude-based Pruning ( LAMP ) score . Consider a depth-d feedforward neural network with weight tensors W ( 1 ) , . . . , W ( d ) associated with each fully-connected/convolutional layer . For fully-connected layers , corresponding weight tensors are twodimensional matrices , and for 2d convolutional layers , corresponding tensors are four-dimensional . To give a unified definition of the LAMP score for both fully-connected and convolutional layers , we assume that each weight tensor is unrolled ( or flattened ) to a one-dimensional vector . For each of these unrolled vectors , we assume ( without loss of generality ) that the weights are sorted in an ascending order according to the given index map , i.e. , |W [ u ] | ≤ |W [ v ] | holds whenever u < v , where W [ u ] denote the entry of W mapped by the index u.3 The LAMP score for the u-th index of the weight tensor W is then defined as score ( u ; W ) : = ( W [ u ] ) 2∑ v≥u ( W [ v ] ) 2 . ( 1 ) Informally , the LAMP score ( Eq . 1 ) measures the relative importance of the target connection among all surviving connections belonging to the same layer , where the connections with a smaller weight magnitude ( in the same layer ) have already been pruned . As a consequence , two connections with identical weight magnitudes have different LAMP scores , depending on the index map being used . Once the LAMP score is computed , we globally prune the connections with smallest LAMP scores until the desired global sparsity constraint is met ; the procedure is equivalent to performing MP with an automatically selected layerwise sparsity . To see this , it suffices to check that ( W [ u ] ) 2 > ( W [ v ] ) 2 ⇒ score ( u ; W ) > score ( v ; W ) ( 2 ) holds for any weight tensor W and a pair of indices u , v. From the definition of the LAMP score ( Eq . 1 ) , it is easy to see that the logical relation ( 2 ) holds . Indeed , for the connection with a larger weight magnitude , the denominator of Eq . 1 is smaller , while the numerator is larger . We note that the global pruning with respect to the LAMP score is not identical to the global pruning with respect to the magnitude score |W [ u ] | ( or ( W [ u ] ) 2 , equivalently ) . Indeed , in each layer , there 3This “ order ” among weights is required to handle the weights with equal magnitude . exists exactly one connection with the LAMP score of 1 , which is the maximum LAMP score possible . In other words , LAMP keeps at least one surviving connection in each layer . The same does not hold for the global pruning with respect to the weight magnitude score . We also note that the LAMP score is easy-to-use . Similar to the vanilla MP , the LAMP score does not have any hyperparameter to be tuned , and is easily implementable via elementary tensor operations . Furthermore , the LAMP score can be computed with only a minimal computational overhead ; the sorting of squared weight magnitudes required to compute the denominator in Eq . 1 is already a part of typical vanilla MP algorithms . For more discussions , see Appendix B .

Summary: This paper presents a novel technique  (layer-adaptive magnitude based pruning, or LAMP) for pruning neural network weights (pruning can be beneficial in terms of overfitting prevention as well as other practical considerations).  LAMP evaluates weights in each layer in terms of the ratio of the magnitude of the weight to the sum of magnitudes of all surviving weights in the layer. The weight which evaluates as least important across all layers is pruned and then the process is repeated until the desired sparsity is achieved. The method is motivated theoretically as minimizing the distortion in the input/output mapping implemented by the weights of the layer. Experimental results on several benchmarks are presented.