Global Magnitude Pruning With Minimum Threshold Is All We Need

1 INTRODUCTION . Neural network pruning remains an important area from both practical perspective ( deployment in real world applications ) and academic perspective ( understanding how to create an efficient architecture ) . It is a long standing area of exploration ( LeCun et al. , 1990a ; Hassibi & Stork , 1993a ) , and was reinvigorated by Han et al . ( 2015a ) . Since then , much work has been done on trying to find different ways of pruning neural networks , such as magnitude-based , gradient or second order based , and regularization-based methods amongst many others . In this work , we shed light on an often overlooked method that has been seen as a mediocre baseline by the community — global magnitude pruning ( GP ) , and show that it can achieve SOTA pruning performance . We demonstrate that GP by itself is a strong pruning algorithm and outperforms SOTA pruning algorithms on benchmarks like ResNet-50 and MobileNet-V1 on ImageNet . We also investigate the pruning behavior of GP and find that a simple addition to GP can raise its performance even more . In contrast to the idea of sparsifying each layer of a neural network to the maximum possible level , we find that preserving a certain amount of weights in each layer actually leads to a better pruning scheme , achieving higher accuracy at the same sparsity level . We call this the Minimum Threshold ( MT ) . When combined with GP , this technique enhances the pruning performance in most cases . We conduct a range of experiments to showcase the above and also study detailed ablations to isolate the effects of GP and MT . We obtain SOTA accuracies on all four sparsity targets on ResNet-50 on ImageNet . We obtain SOTA accuracies on other architectures and datasets tested as well . Finally , GP with MT ( GPMT ) is very simple conceptually and very easy to implement . It is a one-shot pruning method in which the weights to be pruned are decided in one-go without needing any iterative or gradual phases . 2 RELATED WORK . Compression of neural networks has become an important research area due to the rapid increase in size of neural networks ( Brown et al. , 2020 ) , the need for fast inference on edge devices , e.g. , a quadrotor ’ s onboard computer ( Camci et al. , 2020 ) , and concerns about the carbon footprint of training large neural networks ( Strubell et al. , 2019 ) . Over the years , several compression techniques have emerged in the literature ( Cheng et al. , 2017 ) , such as quantisation , factorisation , attention , knowledge distillation , architecture search and pruning ( Almahairi et al. , 2016 ; Ashok et al. , 2017 ; Iandola et al. , 2016 ; Pham et al. , 2018b ) . Quantisation techniques which restrict the bitwidth of parameters ( Rastegari et al. , 2016 ; Courbariaux et al. , 2016 ) and tensor factorisation and decomposition which aim to break large kernels into smaller components ( Mathieu et al. , 2013 ; Gong et al. , 2014 ; Lebedev et al. , 2014 ; Masana et al. , 2017 ) are popular methods . However , they need to be optimised for specific architectures . Attention networks ( Almahairi et al. , 2016 ) have two separate networks to focus on only a small patch of the input image . Training smaller student networks in a process called knowledge distillation ( Ashok et al. , 2017 ) has also proved effective , although it can potentially require a large training budget . Architecture search techniques , such as new kernel design ( Iandola et al. , 2016 ) or whole architecture design ( Pham et al. , 2018a ; Tan et al. , 2019 ) have also become popular . Nevertheless , the large search space size requires ample computational resources to do the architecture search . Different from all these approaches , we focus on pruning deep neural networks in this work . As compared to other categories , pruning is more general in nature and has shown strong performance ( Gale et al. , 2019 ) . Many pruning techniques have been developed over the years , which use first or second order derivatives ( LeCun et al. , 1990b ; Hassibi & Stork , 1993b ) , gradient based methods ( Lee et al. , 2018 ; Wang et al. , 2020 ) , sensitivity to or feedback from some objective function ( Molchanov et al. , 2017 ; Liu et al. , 2020 ; Lin et al. , 2020 ; de Jorge et al. , 2021 ) , distance or similarity measures ( Srinivas & Babu , 2015 ) , regularization-based techniques ( Kusupati et al. , 2020 ; Savarese et al. , 2020 ; Wang et al. , 2021 ) , and magnitude-based criterion ( Ström , 1997 ; Zhu & Gupta , 2018 ; Park et al. , 2020 ; Evci et al. , 2020 ; Lee et al. , 2021 ) . Han et al . ( 2015b ) discovered a key trick to iteratively prune and retrain the network , thereby preserving high accuracy . Gale et al . ( 2019 ) adopt simple , magnitudebased pruning but employ gradual pruning that requires high computational budget and preset sparsification schedules . Runtime Neural Pruning ( Lin et al. , 2017 ) attempts to use reinforcement learning ( RL ) for compression by training an RL agent to select smaller sub-networks during inference . He et al . ( 2018 ) design the first approach using RL for pruning . However , RL training approaches typically require additional RL training budgets ( Gupta et al . ( 2020 ) ) or iterative pruning to achieve good accuracy ( He et al . ( 2018 ) ) . In this work , we focus on a simple , effective , yet quite overlooked pruning method — global magnitude pruning ( GP ) . Although first proposed in 1990s ( Hoefler et al . ( 2021 ) ) , it has largely been ignored in recent years , generally being relegated to the position of a baseline for comparison ( Zhu & Gupta , 2018 ; Blalock et al. , 2020 ; Lee et al. , 2021 ) rather than a strong pruning technique . A few recent works use it as one in a possible pool of pruning techniques ( See et al. , 2016 ; Frankle & Carbin , 2018 ; Gohil et al. , 2020 ) but never study it in detail or adopt it as the main pruning method . We delve deep into GP and showcase that it can achieve SOTA results with the addition of the MT technique . We present SOTA results on various architectures and datasets including ResNet-50 and MobileNet-V1 on ImageNet , and include comprehensive ablation studies and insights on the workings of GP with MT ( GPMT ) . An advantage of GPMT is that it is a very simple and reliable approach . There are a few levels of simplicities and robustness to GPMT . Firstly , it is conceptually very simple and easy to implement . It does not require any complex pruning frameworks like RL ( He et al. , 2018 ) or sparsification schedules ( Zhu & Gupta , 2018 ) . Secondly , it is one-shot and does not require any iterative procedure . Thirdly , it is easily generalizable across architectures and datasets , as shown in the experiments . Lastly , it is data-independent and does not access the dataset for determining the pruning mask . 3 METHOD . We conduct unstructured weight pruning using magnitude pruning . Below we describe the key components of our algorithm in more detail . 3.1 GLOBAL MAGNITUDE PRUNING ( GP ) . GP is a magnitude based pruning approach whereby weights bigger than a given threshold are kept and weights smaller than the threshold are pruned . Formally , for a given threshold t and each individual weight w in any layer , the new weight wnew is defined as follows : wnew = { 0 |w| < t , w otherwise . ( 1 ) In contrast to layer-wise pruning , the threshold is not set on a per layer basis but rather a single threshold is set for the entire network . In this aspect , GP is much more efficient than layer-wise pruning because the threshold does not need to be searched for every layer . On the other hand , uniform pruning refers to setting the same sparsity target for each layer . Thus , every layer is pruned by the same percentage . 3.2 MINIMUM THRESHOLD ( MT ) . The Minimum Threshold ( MT ) refers to a fixed number of weights that are preserved in every layer of the neural network post pruning . The MT is a scalar value that is fixed before the start of the pruning cycle . The weights in a layer are sorted by their magnitude and the top MT number of weights are preserved . For instance , an MT of 500 implies that 500 of the largest weights in every layer need to be preserved post pruning . If a layer is smaller than the MT number , then it implies that all the weights of that layer must be preserved . Therefore , the MT is a very simple concept to apply and also computationally inexpensive . This corresponds to- min ‖Wl‖0 = { σ if m ≥ σ , m otherwise ( 2 ) where Wl ∈ Rm denotes the weight vector for layer l , σ is the MT value in terms of the number of weights and min ‖Wl‖0 indicates the number of non-zero elements in Wl . We explain in the below section how the actual pruning using MT is implemented . 3.3 THE PRUNING WORKFLOW . The pruning pipeline for GP and GP with MT ( GPMT ) is straightforward . It consists of pruning the original model followed by fine-tuning for a few epochs . It is one-shot , therefore , the pruning & fine-tuning cycle does not need to be repeated multiple times . In terms of the pruning procedure itself , for GP , it consists of doing one pass over the network and pruning the weights according to their magnitude to reach the specified sparsity target . For GPMT , it consists of two steps . Firstly , the model is pruned using GP . Secondly , the pruned model is evaluated to check if the MT condition is met by all layers or not . If the condition is not met by a layer , then its sparsity ratio is reduced to meet the MT . The slack arising from the decrease in sparsity is then redistributed amongst the other layers which do not violate the MT condition . The redistribution is done in proportion to their existing sparsities so as to preserve their relative sparsities . This finishes the pruning cycle and the network is then fine-tuned . Fig . 1 explains this procedure , while Algorithm 1 gives the pseudocode . 4 EXPERIMENTS . Below we describe experiments related to ablations on global magnitude pruning ( GP ) and Minimum Threshold ( MT ) , comparison with state-of-the-art algorithms and experiments on non-vision domains . We report hyper-parameters and training related information for all the experiments in the appendix ( section A.4 ) . 4.1 ISOLATING AND UNDERSTANDING IMPACT OF GP AND MT OVER DIFFERENT ARCHITECTURES . We conduct detailed ablations to isolate and measure the impact of standalone GP and GP with MT ( GPMT ) as compared to a uniform pruning baseline . We ablate on multiple architectures and sparsity targets . In addition , we report results averaged over multiple runs where each run uses a different pre-trained model to provide more robustness . We first prune a WRN-22-8 model on CIFAR-10 at 95 % sparsity . We then fine-tune the model for a few epochs and report the final accuracy . We experiment with different pruning schemes , i.e. , uniform pruning , GP and GPMT ( see Section 3 for details ) . We find that GP outperforms uniform pruning . Furthermore , adding MT improves the performance even more , see Table 1 . Next , we do an experiment on the highly efficient MobileNet-V2 architecture to see if the above conclusion holds on it too . We find that indeed GP beats uniform pruning in this situation as well , and adding MT improves performance even further ( Table 2 ) . This shows that GP by itself is superior to uniform pruning and adding MT aids GP even more .

Summary: This paper revisits Global Pruning (GP) and makes a case to consider it seriously as part of the pruning literature. The authors also present an addition to GP, that claims, to further increase accuracy and reliability called Minimum Threshold (MT). MT is the constraint put on every layer of the neural network to ensure they are not pruned beyond a certain limit which might result in catastrophic accuracy drops.   The paper presents experiments on CIFAR-10 with a few CNN architectures along with current pruning baselines (layer-wise mostly. They also show results on ImageNet using ResNet50 and MovileNetV1. Following these experiments, the authors claim that GP or GPMT are SOTA for pruning despite their simplicity.   The paper also includes some discussion on how output patterns of each layer change with different pruning schemes and a note about how to set the optimal MT values.   While I agree with the sentiment of the paper, I do not think the paper is presenting anything new (even from a benchmarking perspective) but rather is missing the point about what makes GP vs layer-wise pruning a worthy trade-off often. I will list my concerns in the main review.