Global Magnitude Pruning With Minimum Threshold Is All We Need

1 INTRODUCTION . Neural network pruning remains an important area from both practical perspective ( deployment in real world applications ) and academic perspective ( understanding how to create an efficient architecture ) . It is a long standing area of exploration ( LeCun et al. , 1990a ; Hassibi & Stork , 1993a ) , and was reinvigorated by Han et al . ( 2015a ) . Since then , much work has been done on trying to find different ways of pruning neural networks , such as magnitude-based , gradient or second order based , and regularization-based methods amongst many others . In this work , we shed light on an often overlooked method that has been seen as a mediocre baseline by the community — global magnitude pruning ( GP ) , and show that it can achieve SOTA pruning performance . We demonstrate that GP by itself is a strong pruning algorithm and outperforms SOTA pruning algorithms on benchmarks like ResNet-50 and MobileNet-V1 on ImageNet . We also investigate the pruning behavior of GP and find that a simple addition to GP can raise its performance even more . In contrast to the idea of sparsifying each layer of a neural network to the maximum possible level , we find that preserving a certain amount of weights in each layer actually leads to a better pruning scheme , achieving higher accuracy at the same sparsity level . We call this the Minimum Threshold ( MT ) . When combined with GP , this technique enhances the pruning performance in most cases . We conduct a range of experiments to showcase the above and also study detailed ablations to isolate the effects of GP and MT . We obtain SOTA accuracies on all four sparsity targets on ResNet-50 on ImageNet . We obtain SOTA accuracies on other architectures and datasets tested as well . Finally , GP with MT ( GPMT ) is very simple conceptually and very easy to implement . It is a one-shot pruning method in which the weights to be pruned are decided in one-go without needing any iterative or gradual phases . 2 RELATED WORK . Compression of neural networks has become an important research area due to the rapid increase in size of neural networks ( Brown et al. , 2020 ) , the need for fast inference on edge devices , e.g. , a quadrotor ’ s onboard computer ( Camci et al. , 2020 ) , and concerns about the carbon footprint of training large neural networks ( Strubell et al. , 2019 ) . Over the years , several compression techniques have emerged in the literature ( Cheng et al. , 2017 ) , such as quantisation , factorisation , attention , knowledge distillation , architecture search and pruning ( Almahairi et al. , 2016 ; Ashok et al. , 2017 ; Iandola et al. , 2016 ; Pham et al. , 2018b ) . Quantisation techniques which restrict the bitwidth of parameters ( Rastegari et al. , 2016 ; Courbariaux et al. , 2016 ) and tensor factorisation and decomposition which aim to break large kernels into smaller components ( Mathieu et al. , 2013 ; Gong et al. , 2014 ; Lebedev et al. , 2014 ; Masana et al. , 2017 ) are popular methods . However , they need to be optimised for specific architectures . Attention networks ( Almahairi et al. , 2016 ) have two separate networks to focus on only a small patch of the input image . Training smaller student networks in a process called knowledge distillation ( Ashok et al. , 2017 ) has also proved effective , although it can potentially require a large training budget . Architecture search techniques , such as new kernel design ( Iandola et al. , 2016 ) or whole architecture design ( Pham et al. , 2018a ; Tan et al. , 2019 ) have also become popular . Nevertheless , the large search space size requires ample computational resources to do the architecture search . Different from all these approaches , we focus on pruning deep neural networks in this work . As compared to other categories , pruning is more general in nature and has shown strong performance ( Gale et al. , 2019 ) . Many pruning techniques have been developed over the years , which use first or second order derivatives ( LeCun et al. , 1990b ; Hassibi & Stork , 1993b ) , gradient based methods ( Lee et al. , 2018 ; Wang et al. , 2020 ) , sensitivity to or feedback from some objective function ( Molchanov et al. , 2017 ; Liu et al. , 2020 ; Lin et al. , 2020 ; de Jorge et al. , 2021 ) , distance or similarity measures ( Srinivas & Babu , 2015 ) , regularization-based techniques ( Kusupati et al. , 2020 ; Savarese et al. , 2020 ; Wang et al. , 2021 ) , and magnitude-based criterion ( Ström , 1997 ; Zhu & Gupta , 2018 ; Park et al. , 2020 ; Evci et al. , 2020 ; Lee et al. , 2021 ) . Han et al . ( 2015b ) discovered a key trick to iteratively prune and retrain the network , thereby preserving high accuracy . Gale et al . ( 2019 ) adopt simple , magnitudebased pruning but employ gradual pruning that requires high computational budget and preset sparsification schedules . Runtime Neural Pruning ( Lin et al. , 2017 ) attempts to use reinforcement learning ( RL ) for compression by training an RL agent to select smaller sub-networks during inference . He et al . ( 2018 ) design the first approach using RL for pruning . However , RL training approaches typically require additional RL training budgets ( Gupta et al . ( 2020 ) ) or iterative pruning to achieve good accuracy ( He et al . ( 2018 ) ) . In this work , we focus on a simple , effective , yet quite overlooked pruning method — global magnitude pruning ( GP ) . Although first proposed in 1990s ( Hoefler et al . ( 2021 ) ) , it has largely been ignored in recent years , generally being relegated to the position of a baseline for comparison ( Zhu & Gupta , 2018 ; Blalock et al. , 2020 ; Lee et al. , 2021 ) rather than a strong pruning technique . A few recent works use it as one in a possible pool of pruning techniques ( See et al. , 2016 ; Frankle & Carbin , 2018 ; Gohil et al. , 2020 ) but never study it in detail or adopt it as the main pruning method . We delve deep into GP and showcase that it can achieve SOTA results with the addition of the MT technique . We present SOTA results on various architectures and datasets including ResNet-50 and MobileNet-V1 on ImageNet , and include comprehensive ablation studies and insights on the workings of GP with MT ( GPMT ) . An advantage of GPMT is that it is a very simple and reliable approach . There are a few levels of simplicities and robustness to GPMT . Firstly , it is conceptually very simple and easy to implement . It does not require any complex pruning frameworks like RL ( He et al. , 2018 ) or sparsification schedules ( Zhu & Gupta , 2018 ) . Secondly , it is one-shot and does not require any iterative procedure . Thirdly , it is easily generalizable across architectures and datasets , as shown in the experiments . Lastly , it is data-independent and does not access the dataset for determining the pruning mask . 3 METHOD . We conduct unstructured weight pruning using magnitude pruning . Below we describe the key components of our algorithm in more detail . 3.1 GLOBAL MAGNITUDE PRUNING ( GP ) . GP is a magnitude based pruning approach whereby weights bigger than a given threshold are kept and weights smaller than the threshold are pruned . Formally , for a given threshold t and each individual weight w in any layer , the new weight wnew is defined as follows : wnew = { 0 |w| < t , w otherwise . ( 1 ) In contrast to layer-wise pruning , the threshold is not set on a per layer basis but rather a single threshold is set for the entire network . In this aspect , GP is much more efficient than layer-wise pruning because the threshold does not need to be searched for every layer . On the other hand , uniform pruning refers to setting the same sparsity target for each layer . Thus , every layer is pruned by the same percentage . 3.2 MINIMUM THRESHOLD ( MT ) . The Minimum Threshold ( MT ) refers to a fixed number of weights that are preserved in every layer of the neural network post pruning . The MT is a scalar value that is fixed before the start of the pruning cycle . The weights in a layer are sorted by their magnitude and the top MT number of weights are preserved . For instance , an MT of 500 implies that 500 of the largest weights in every layer need to be preserved post pruning . If a layer is smaller than the MT number , then it implies that all the weights of that layer must be preserved . Therefore , the MT is a very simple concept to apply and also computationally inexpensive . This corresponds to- min ‖Wl‖0 = { σ if m ≥ σ , m otherwise ( 2 ) where Wl ∈ Rm denotes the weight vector for layer l , σ is the MT value in terms of the number of weights and min ‖Wl‖0 indicates the number of non-zero elements in Wl . We explain in the below section how the actual pruning using MT is implemented . 3.3 THE PRUNING WORKFLOW . The pruning pipeline for GP and GP with MT ( GPMT ) is straightforward . It consists of pruning the original model followed by fine-tuning for a few epochs . It is one-shot , therefore , the pruning & fine-tuning cycle does not need to be repeated multiple times . In terms of the pruning procedure itself , for GP , it consists of doing one pass over the network and pruning the weights according to their magnitude to reach the specified sparsity target . For GPMT , it consists of two steps . Firstly , the model is pruned using GP . Secondly , the pruned model is evaluated to check if the MT condition is met by all layers or not . If the condition is not met by a layer , then its sparsity ratio is reduced to meet the MT . The slack arising from the decrease in sparsity is then redistributed amongst the other layers which do not violate the MT condition . The redistribution is done in proportion to their existing sparsities so as to preserve their relative sparsities . This finishes the pruning cycle and the network is then fine-tuned . Fig . 1 explains this procedure , while Algorithm 1 gives the pseudocode . 4 EXPERIMENTS . Below we describe experiments related to ablations on global magnitude pruning ( GP ) and Minimum Threshold ( MT ) , comparison with state-of-the-art algorithms and experiments on non-vision domains . We report hyper-parameters and training related information for all the experiments in the appendix ( section A.4 ) . 4.1 ISOLATING AND UNDERSTANDING IMPACT OF GP AND MT OVER DIFFERENT ARCHITECTURES . We conduct detailed ablations to isolate and measure the impact of standalone GP and GP with MT ( GPMT ) as compared to a uniform pruning baseline . We ablate on multiple architectures and sparsity targets . In addition , we report results averaged over multiple runs where each run uses a different pre-trained model to provide more robustness . We first prune a WRN-22-8 model on CIFAR-10 at 95 % sparsity . We then fine-tune the model for a few epochs and report the final accuracy . We experiment with different pruning schemes , i.e. , uniform pruning , GP and GPMT ( see Section 3 for details ) . We find that GP outperforms uniform pruning . Furthermore , adding MT improves the performance even more , see Table 1 . Next , we do an experiment on the highly efficient MobileNet-V2 architecture to see if the above conclusion holds on it too . We find that indeed GP beats uniform pruning in this situation as well , and adding MT improves performance even further ( Table 2 ) . This shows that GP by itself is superior to uniform pruning and adding MT aids GP even more .

Summary: A very simple and effective pruning method is proposed. Instead of layer-wise pruning, a global threshold is used to prune weights according to their magnitude. In addition, to avoid over-pruning, a minimum number of parameters is preserved for each layer after pruning. Experiments on CIFAR-10 and ImageNet validate the effectiveness of proposed method.