GLOBAL MAGNITUDE PRUNING WITH MINIMUM THRESHOLD IS ALL WE NEED
Year: 2021

Abstract:
Neural network pruning remains a very important yet challenging problem to solve. Many pruning solutions have been proposed over the years with high degrees of algorithmic complexity. In this work, we shed light on a very simple pruning technique that achieves state-of-the-art (SOTA) performance. We showcase that magnitude based pruning, specifically, global magnitude pruning (GP) is sufficient to achieve SOTA performance on a range of neural network architectures. In certain architectures, the last few layers of a network may get over-pruned. For these cases, we introduce a straightforward method to mitigate this. We preserve a certain fixed number of weights in each layer of the network to ensure no layer is over-pruned. We call this the Minimum Threshold (MT). We find that GP combined with MT when needed, achieves SOTA parameter-accuracy trade-off on all datasets and architectures tested including ResNet-50 and MobileNet-V1 on ImageNet. Code is available at https://github.com/GPMT-Authors/Global-PruningWith-Minimum-Threshold.

heading: 1 INTRODUCTION
text: Neural network pruning remains an important area from both practical perspective (deployment in real world applications) and academic perspective (understanding how to create an efficient architecture). It is a long standing area of exploration (LeCun et al., 1990a; Hassibi & Stork, 1993a), and was reinvigorated by Han et al. (2015a). Since then, much work has been done on trying to find different ways of pruning neural networks, such as magnitude-based, gradient or second order based, and regularization-based methods amongst many others. In this work, we shed light on an often overlooked method that has been seen as a mediocre baseline by the community — global magnitude pruning (GP), and show that it can achieve SOTA pruning performance.
We demonstrate that GP by itself is a strong pruning algorithm and outperforms SOTA pruning algorithms on benchmarks like ResNet-50 and MobileNet-V1 on ImageNet. We also investigate the pruning behavior of GP and find that a simple addition to GP can raise its performance even more. In contrast to the idea of sparsifying each layer of a neural network to the maximum possible level, we find that preserving a certain amount of weights in each layer actually leads to a better pruning scheme, achieving higher accuracy at the same sparsity level. We call this the Minimum Threshold (MT). When combined with GP, this technique enhances the pruning performance in most cases.
We conduct a range of experiments to showcase the above and also study detailed ablations to isolate the effects of GP and MT. We obtain SOTA accuracies on all four sparsity targets on ResNet-50 on ImageNet. We obtain SOTA accuracies on other architectures and datasets tested as well. Finally, GP with MT (GPMT) is very simple conceptually and very easy to implement. It is a one-shot pruning method in which the weights to be pruned are decided in one-go without needing any iterative or gradual phases.

heading: 2 RELATED WORK
text: Compression of neural networks has become an important research area due to the rapid increase in size of neural networks (Brown et al., 2020), the need for fast inference on edge devices, e.g., a quadrotor’s onboard computer (Camci et al., 2020), and concerns about the carbon footprint of
training large neural networks (Strubell et al., 2019). Over the years, several compression techniques have emerged in the literature (Cheng et al., 2017), such as quantisation, factorisation, attention, knowledge distillation, architecture search and pruning (Almahairi et al., 2016; Ashok et al., 2017; Iandola et al., 2016; Pham et al., 2018b).
Quantisation techniques which restrict the bitwidth of parameters (Rastegari et al., 2016; Courbariaux et al., 2016) and tensor factorisation and decomposition which aim to break large kernels into smaller components (Mathieu et al., 2013; Gong et al., 2014; Lebedev et al., 2014; Masana et al., 2017) are popular methods. However, they need to be optimised for specific architectures. Attention networks (Almahairi et al., 2016) have two separate networks to focus on only a small patch of the input image. Training smaller student networks in a process called knowledge distillation (Ashok et al., 2017) has also proved effective, although it can potentially require a large training budget. Architecture search techniques, such as new kernel design (Iandola et al., 2016) or whole architecture design (Pham et al., 2018a; Tan et al., 2019) have also become popular. Nevertheless, the large search space size requires ample computational resources to do the architecture search. Different from all these approaches, we focus on pruning deep neural networks in this work. As compared to other categories, pruning is more general in nature and has shown strong performance (Gale et al., 2019).
Many pruning techniques have been developed over the years, which use first or second order derivatives (LeCun et al., 1990b; Hassibi & Stork, 1993b), gradient based methods (Lee et al., 2018; Wang et al., 2020), sensitivity to or feedback from some objective function (Molchanov et al., 2017; Liu et al., 2020; Lin et al., 2020; de Jorge et al., 2021), distance or similarity measures (Srinivas & Babu, 2015), regularization-based techniques (Kusupati et al., 2020; Savarese et al., 2020; Wang et al., 2021), and magnitude-based criterion (Ström, 1997; Zhu & Gupta, 2018; Park et al., 2020; Evci et al., 2020; Lee et al., 2021). Han et al. (2015b) discovered a key trick to iteratively prune and retrain the network, thereby preserving high accuracy. Gale et al. (2019) adopt simple, magnitudebased pruning but employ gradual pruning that requires high computational budget and preset sparsification schedules. Runtime Neural Pruning (Lin et al., 2017) attempts to use reinforcement learning (RL) for compression by training an RL agent to select smaller sub-networks during inference. He et al. (2018) design the first approach using RL for pruning. However, RL training approaches typically require additional RL training budgets (Gupta et al. (2020)) or iterative pruning to achieve good accuracy (He et al. (2018)).
In this work, we focus on a simple, effective, yet quite overlooked pruning method — global magnitude pruning (GP). Although first proposed in 1990s (Hoefler et al. (2021)), it has largely been ignored in recent years, generally being relegated to the position of a baseline for comparison (Zhu & Gupta, 2018; Blalock et al., 2020; Lee et al., 2021) rather than a strong pruning technique. A few recent works use it as one in a possible pool of pruning techniques (See et al., 2016; Frankle & Carbin, 2018; Gohil et al., 2020) but never study it in detail or adopt it as the main pruning method. We delve deep into GP and showcase that it can achieve SOTA results with the addition of the MT technique. We present SOTA results on various architectures and datasets including ResNet-50 and MobileNet-V1 on ImageNet, and include comprehensive ablation studies and insights on the workings of GP with MT (GPMT).
An advantage of GPMT is that it is a very simple and reliable approach. There are a few levels of simplicities and robustness to GPMT. Firstly, it is conceptually very simple and easy to implement. It does not require any complex pruning frameworks like RL (He et al., 2018) or sparsification schedules (Zhu & Gupta, 2018). Secondly, it is one-shot and does not require any iterative procedure. Thirdly, it is easily generalizable across architectures and datasets, as shown in the experiments. Lastly, it is data-independent and does not access the dataset for determining the pruning mask.

heading: 3 METHOD
text: We conduct unstructured weight pruning using magnitude pruning. Below we describe the key components of our algorithm in more detail.

heading: 3.1 GLOBAL MAGNITUDE PRUNING (GP)
text: GP is a magnitude based pruning approach whereby weights bigger than a given threshold are kept and weights smaller than the threshold are pruned.
Formally, for a given threshold t and each individual weight w in any layer, the new weight wnew is defined as follows:
wnew = { 0 |w| < t, w otherwise.
(1)
In contrast to layer-wise pruning, the threshold is not set on a per layer basis but rather a single threshold is set for the entire network. In this aspect, GP is much more efficient than layer-wise pruning because the threshold does not need to be searched for every layer. On the other hand, uniform pruning refers to setting the same sparsity target for each layer. Thus, every layer is pruned by the same percentage.

heading: 3.2 MINIMUM THRESHOLD (MT)
text: The Minimum Threshold (MT) refers to a fixed number of weights that are preserved in every layer of the neural network post pruning. The MT is a scalar value that is fixed before the start of the pruning cycle. The weights in a layer are sorted by their magnitude and the top MT number of weights are preserved. For instance, an MT of 500 implies that 500 of the largest weights in every layer need to be preserved post pruning. If a layer is smaller than the MT number, then it implies that all the weights of that layer must be preserved. Therefore, the MT is a very simple concept to apply and also computationally inexpensive. This corresponds to-
min ‖Wl‖0 = { σ if m ≥ σ, m otherwise
(2)
where Wl ∈ Rm denotes the weight vector for layer l, σ is the MT value in terms of the number of weights and min ‖Wl‖0 indicates the number of non-zero elements in Wl. We explain in the below section how the actual pruning using MT is implemented.

heading: 3.3 THE PRUNING WORKFLOW
text: The pruning pipeline for GP and GP with MT (GPMT) is straightforward. It consists of pruning the original model followed by fine-tuning for a few epochs. It is one-shot, therefore, the pruning & fine-tuning cycle does not need to be repeated multiple times. In terms of the pruning procedure itself, for GP, it consists of doing one pass over the network and pruning the weights according to their magnitude to reach the specified sparsity target. For GPMT, it consists of two steps. Firstly, the model is pruned using GP. Secondly, the pruned model is evaluated to check if the MT condition is met by all layers or not. If the condition is not met by a layer, then its sparsity ratio is reduced to meet the MT. The slack arising from the decrease in sparsity is then redistributed amongst the other layers which do not violate the MT condition. The redistribution is done in proportion to their existing sparsities so as to preserve their relative sparsities. This finishes the pruning cycle and the network is then fine-tuned. Fig. 1 explains this procedure, while Algorithm 1 gives the pseudocode.

heading: 4 EXPERIMENTS
text: Below we describe experiments related to ablations on global magnitude pruning (GP) and Minimum Threshold (MT), comparison with state-of-the-art algorithms and experiments on non-vision domains. We report hyper-parameters and training related information for all the experiments in the appendix (section A.4).

heading: 4.1 ISOLATING AND UNDERSTANDING IMPACT OF GP AND MT OVER DIFFERENT ARCHITECTURES
text: We conduct detailed ablations to isolate and measure the impact of standalone GP and GP with MT (GPMT) as compared to a uniform pruning baseline. We ablate on multiple architectures and sparsity targets. In addition, we report results averaged over multiple runs where each run uses a different pre-trained model to provide more robustness. We first prune a WRN-22-8 model on CIFAR-10 at 95% sparsity. We then fine-tune the model for a few epochs and report the final accuracy. We experiment with different pruning schemes, i.e., uniform pruning, GP and GPMT (see Section 3 for details). We find that GP outperforms uniform pruning. Furthermore, adding MT improves the performance even more, see Table 1. Next, we do an experiment on the highly efficient MobileNet-V2 architecture to see if the above conclusion holds on it too. We find that indeed GP beats uniform pruning in this situation as well, and adding MT improves performance even further
(Table 2). This shows that GP by itself is superior to uniform pruning and adding MT aids GP even more.

heading: 4.2 STATE-OF-THE-ART COMPARISON ON CIFAR-10
text: We also conduct experiments to compare GP and MT to SOTA pruning algorithms on the CIFAR-10 dataset. We compare with various algorithms including SNIP (Lee et al., 2018), SM (Dettmers & Zettlemoyer, 2019), DSR (Mostafa & Wang, 2019) and DPF (Lin et al., 2020). We start off with the original model having the same initial accuracy as the other algorithms. For the WRN-28-8 model we find that GP alone beats all the SOTA pruning algorithms at two out of the three sparsity levels (Table 5). If we incorporate MT, then the accuracy increases further, and the algorithm is able to beat all the SOTA algorithms at all the sparsity levels. For ResNet-32 as well (Table 6), both GP and GPMT outperform all the other algorithms at 90% sparsity.
0
500
1000
1500
2000
2500
3000
3500
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
N u
m b
er o
f re
m ai
n in
g w
ei gh
ts
Layer index
Without MT (N.A.)
With MT (82.40% accuracy)

heading: 4.3 STATE-OF-THE-ART COMPARISON ON IMAGENET
text: We also test another architecture on ImageNet, MobileNet-V1, which is a much smaller and more efficient architecture than ResNet-50. We start with the same original accuracy as the other models. For MobileNet-V1, GP alone gets good results at 75% sparsity and outperforms SOTA algorithms by 2.44% on a constant parameter budget basis (Table 8). On combining with MT, GPMT also surpasses the SOTA algorithms at 90% sparsity by 2.43%. Since MobileNet-V1 is a very efficient architecture (having only 4.2 million parameters compared to 25.6 million for ResNet-50), MT especially helps at higher sparsities when the layers are pruned very aggressively.

heading: 4.4 GENERALIZING TO OTHER DOMAINS AND RNN ARCHITECTURES
text: We experiment with the GP rule on other domains and non-convolutional networks as well to measure the generalizability of the algorithm on different domains and network types. We experiment on a FastGRNN model (Kusupati et al. (2018)) on the HAR-2 Human Activity Recognition dataset (Anguita et al. (2013)). More details on the dataset and the experimental setup can be found in Section A.3. We test the GP rule under different network configurations. We find that GP surpasses other pruning algorithms on all the configurations (Table 9) and successfully prunes the model on a very different architecture and domain.

heading: 5 INSIGHTS ON GP AND MT

heading: 5.1 NETWORK ARCHITECTURES CAN AFFECT PRUNING PERFORMANCE
text: We find that network architectures can affect the results of how the pruning algorithm performs quite a lot, especially in the high sparsity domain. We take the case of the high sparsity experiments
between MobileNet-V2 and WRN-22-8, as mentioned in Section 4.1. In the case of just using GP, we find that WRN-22-8 is still able to learn, however, accuracy crashes for MobileNet-V2 and the model is unable to learn anything, getting a chance accuracy rate of 10%. The reason for this wide discrepancy in learning behavior lies in the shortcut connections (He et al., 2015). Both WRN-22-8 and MobileNet-V2 use shortcut connections, however, their placement is different. Referring to Fig. 4, WRN uses identity shortcut connections from Layer 20 to Layer 23. Identity shortcut connections are simple identity mappings and do not require any extra parameters, and hence, they do not count towards the weights. However, MobileNet-V2 uses a convolutional shortcut mapping from Layer 52 to Layer 57 and hence, it does add to the model’s weights, and thus, it is pruned by the pruning algorithm. Both the models have the preceding two layers before the last layer, completely pruned. However, because WRN uses identity mappings, it is still able to relay information to the last layer, and the model is still able to learn.
Method Top-1 Acc Params. Sparsity
ResNet32
93.83 ± 0.12 % 0.46M 0.00%
SNIP 90.40 ± 0.26% 0.046M 90% SM 91.54 ± 0.18% 0.046M 90% DSR 91.41 ± 0.23% 0.046M 90% DPF 92.42 ± 0.18% 0.046M 90% GP 92.67 ± 0.03% 0.046M 90% GP + MT 92.74 ± 0.06% 0.046M 90% SNIP 87.23 ± 0.29% 0.023M 95% SM 88.68 ± 0.22% 0.023M 95% DSR 84.12 ± 0.32% 0.023M 95% DPF 90.94 ± 0.35% 0.023M 95% GP 90.65 ± 0.13% 0.023M 95% GP + MT 90.58 ± 0.24% 0.023M 95%
Method Top-1 Acc Params. Sparsity FLOPs
ResNet-50 77.0% 25.6M 0.00% 4.09G
GMP 75.60% 5.12M 80.00% 818M DSR*# 71.60% 5.12M 80.00% 1.23G DNW 76.00% 5.12M 80.00% 818M SM 74.90% 5.12M 80.00% - SM + ERK 75.20% 5.12M 80.00% 1.68G RigL* 74.60% 5.12M 80.00% 920M RigL + ERK 75.10% 5.12M 80.00% 1.68G DPF 75.13% 5.12M 80.00% 818M STR 76.19% 5.22M 79.55% 766M GP 76.84% 5.12M 80.00% 1.13G GP + MT 76.75% 5.12M 80.00% 1.28G
GMP 73.91% 2.56M 90.00% 409M DNW 74.00% 2.56M 90.00% 409M SM 72.90% 2.56M 90.00% 1.63G SM + ERK 72.90% 2.56M 90.00% 960M RigL* 72.00% 2.56M 90.00% 515M RigL + ERK 73.00% 2.56M 90.00% 960M DPF# 74.55% 4.45M 82.60% 411M STR 74.73% 3.14M 87.70% 402M GP 75.28% 2.56M 90.00% 704M GP + MT 75.21% 2.56M 90.00% 926M
Method Top-1 Acc Params. Sparsity FLOPs
ResNet-50 77.0% 25.6M 0.00% 4.09G
GMP 70.59% 1.28M 95.00% 204M DNW 68.30% 1.28M 95.00% 204M RigL* 67.50% 1.28M 95.00% 317M RigL + ERK 70.00% 1.28M 95.00% 600M STR 70.97% 1.33M 94.80% 182M STR 70.40% 1.27M 95.03% 159M STR 70.23% 1.24M 95.15% 162M GP 71.56% 1.20M 95.30% 437M GP + MT 71.57% 1.20M 95.30% 438M
GMP 57.90% 0.51M 98.00% 82M DNW 58.20% 0.51M 98.00% 82M STR 61.46% 0.50M 98.05% 73M GP 61.80% 0.50M 98.05% 257M GP + MT 61.90% 0.50M 98.05% 257M
Table 7: Results on ResNet-50 on ImageNet. GP and GPMT outperform SOTA pruning algorithms
by upto 1.3% accuracy. * and # imply that the first and last layer are dense respectively.
1x1 conv, in:160, out:960
3x3 conv, in:960, out:960
1x1 conv, in:960, out:160
1x1 conv, in:160, out:960
3x3 conv, in:960, out:960
1x1 conv, in:960, out:320
1x1 conv, in:320, out:1280
Fully connected, in:1280, out:10
1x1 conv, in:160, out:320
[50]
[51]
[52]
[53]
[54]
[55]
[57]
[58]
[56]
3x3 conv, in:512, out:512
3x3 conv, in:512, out:512
Fully connected, in:512, out:10
[19]
[20]
[21]
[22]
[23]
3x3 conv, in:512, out:512
3x3 conv, in:512, out:512
WideResNet-22-8 MobileNet-V2
Shortcut layer Downsampling layer
Figure 4: Difference in architectures between WRN and MobileNet. WRN does not have any prunable residual connections in the last layers (dotted lines) while MobileNet does. This leads to different pruning behaviors on the two architectures.
Pruning algorithms can be susceptible to such catastrophic network disconnection issues especially in the high sparsity domain. Fortunately, the MT can easily overcome this issue. Retaining a small MT of 0.02% was sufficient for the MobileNet-V2 model to avoid disconnection and learn successfully. Hence, retaining a small MT can help in the learning dynamics of models in high sparsity settings.

heading: 5.2 GP PRESERVES OUTPUTS BETTER THAN UNIFORM PRUNING
text: To understand why GP performs better than uniform pruning, especially at the layer-by-layer level, we measure the Frobenius output distortion between the original model and the pruned model (Park
et al., 2020). We compare the difference in the layer-wise outputs between the original model and the pruned model, and then compute the norm of this difference. Thus, a smaller norm indicates that the output of the pruned model is closer to the original model and a larger norm indicates that the output is further away. We compute and plot this layer-wise output distortion on the WRN-22-8 model on CIFAR-10 for the GP and uniform pruning models as compared to the original model (Fig. 5). As can be seen from the figure, the distortion is much higher for the uniform pruning model. Layers 1 and 2 are especially impacted, with the distortion being around 3x that of GP in layer 1. Thus, GP is able to preserve outputs much closer to the original model as compared to uniform pruning. This points towards a new direction for empirically understanding the underlying differences between pruning algorithms.

heading: 5.3 WHEN IS MT REQUIRED AND HOW TO DETERMINE THE OPTIMAL VALUE
text: We find that whether MT is required or not depends on the neural network architecture family. For instance, in our experiments on multiple architectures, we found generally that ResNet architectures did not require MT (tested on ResNet-32 and ResNet-50). Conversely, MobileNet and WideResNet architectures did require MT (tested on MobileNet-V1, MobileNet-V2, WideResNet-22-8 and WideResNet-28-8). Hence, we hypothesize that the requirement for MT depends upon the base family architecture used. We also found that in architectures that require MT, a decent rule of thumb value for MT is 0.05% of the total weights. Increments of 0.05% may be tried, i.e., 0.1%, 0.15% or 0.2%, if need be. We never had to go beyond 0.2%. In cases of high sparsity, where there is not enough capacity in the network to support 0.05% MT, lower values of MT can be tried, e.g., 0.02%, 0.01%, 0.005% or 0.002%. We found that the above-stated values were sufficient to get good performance across models and we never had to undertake a more exhaustive MT search. In most cases, we only tried two to three MT values and achieved good results.

heading: 6 LIMITATIONS AND FUTURE WORK
text: A limitation of the GP method is that it can incur higher FLOPs compared to some layer-wise baselines (Table 7). This is because it prunes the last layers more with respect to the initial layers (that have a higher FLOPs to parameter ratio) compared to some baselines. In applications where FLOPs reduction is equally important as parameter reduction, a way to mitigate this would be to add constraints on the FLOPs as well along with the parameters, and thus jointly optimize for both FLOPs and parameters. We will look at this in future work. Another exciting extension to GP is to do GP gradually instead of one-shot. As demonstrated in Table 4, doing GP gradually enhances the accuracy performance of GP. We believe this increase in performance is transferable to other datasets and architectures as well, and is another topic that we will look at in the future.

heading: 7 CONCLUSION
text: Many methods have been proposed for neural network pruning over the years. In this paper, we focus on an often overlooked pruning method — global magnitude pruning (GP). We show that GP by itself is a strong pruning method and achieves SOTA performance on ResNet-50 and MobileNetV1 on ImageNet. Furthermore, we investigate the pruning behavior of GP in depth and find that in certain cases, e.g., at high sparsities, a few layers in the network may be over-pruned. To rectify this, we propose a novel yet very simple mitigation mechanism called Minimum Threshold (MT). MT preserves a certain fixed amount of weights in every layer post pruning and ensures no layer is over-pruned. We find that adding MT on top of GP leads to higher performance across many architectures and datasets. We achieve SOTA results on CIFAR-10 and ImageNet using the above pruning technique. GP with MT (GPMT) is also easy to implement, is one-shot and is easily portable across architectures and datasets. GP can be further extended to be done gradually instead of oneshot and can lead to more performance gains. A limitation of GPMT is that it can incur higher FLOPs compared to some layerwise methods and a possible mitigation for this is to add a FLOPs constraint on top of the parameter constraint when doing pruning. Overall, we find that GPMT can be used as a useful baseline for future pruning works given it’s simplicity and performance.

heading: 8 REPRODUCIBILITY
text: To help with reproducibility, we have released our codebase. It can be found at https://github.com/GPMT-Authors/Global-Pruning-With-Minimum-Threshold. Furthermore, we report in detail the experimental setup including all the hyper-parameters for all our experiments in the appendix (Section A.4). The above two items ensure that all our experiments are reproducible. Furthermore, our method is conceptually very simple and hence, very easy to implement. It does not require additional algorithm-specific hyper-parameters or any architecture-specific tweaking. We use standard PyTorch libraries (Paszke et al. (2019)) without any modification to the model layers. These make the GPMT method straightforward to implement, and hence, easy to replicate. Lastly, we show that GPMT gets stable results across multiple runs and pre-trained models (Figs. 6 and 7) and hence, aids in reproducibility.

heading: A APPENDIX
text: A.1 SIMPLE ONE-SHOT APPROACH
A.2 PSEUDO-CODE
Algorithm 1 Pseudo-code for GPMT Load a pre-trained DNN with weight vectors Wl for each layer l Set a target sparsity, κ Set an MT value, σ Concatenate and sort all Wl into a vector W= {w0, w1, ..., wn} where wi ≤ wi+1 k = κ100 · n Zero-mask all elements from w0 to wk in W across all Wl Slack pruning budget, Σ = 0 for l in layers do
if min ‖Wl‖0 < σ then Restore the (σ −min ‖Wl‖0) # of highest magnitude weights in Wl Σ = Σ + σ −min ‖Wl‖0
end if end for Distribute Σ into each layer where min ‖Wl‖0 > σ, proportionally to their sparsities κl Fine-tune the pruned model
A.3 SPARSIFYING FASTGRNN ON HAR-2 DATASET
HAR-2 dataset that is used in the FastGRNN pruning experiment is a binarized version of the 6- class Human Activity Recognition dataset. From the full-rank model with rW = 9 and rU = 80 as suggested on the STR paper (Kusupati et al., 2020), we apply GP on the matrices W1 and W2. To do this, we find the weight mask by ranking the columns of W1 and W2 based on their absolute sum, then we prune the 9− rnewW lowest columns and 80− rnewU lowest columns from W1 and W2
respectively. In the end, we fine-tuned this pruned model by retraining it with FastGRNN’s trainer and applying the weight mask on every epoch.
A.4 HYPER-PARAMETERS AND EXPERIMENTAL SETUP
No data augmentation is done apart from standard data pre-processing. Difference in batch size for training and testing in some experiments is due to GPU RAM availability. Averages reported over three runs.
MT values
Experiment Sparsity MT value
Table 1 95% 0.05%
Table 2 40% 0.05%
Table 3 99.9% 0.002%
Table 4 98% 0.02%
Table 5 90% 0.05% 95% 0.10% 97.5% 0.05%
Table 6 90% 0.05%95% 0.05%
Table 7
80% 0.05% 90% 0.05% 95.3% 0.005% 98.05% 0.005%
Table 8 75% 0.05%90% 0.2%
Setup for Table 1
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Nesterov
Training 30 256 0.9 5e-4 0.1 Step decay (Step size 25, gamma 0.1) Yes Finetuning 80 1800 0.9 5e-4 0.1 Step decay (Step size 40, gamma 0.1) Yes
Setup for Table 2
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Nesterov
Training 200 450 0.9 5e-4 0.1 Step decay (Step size 25, gamma 0.56) Yes Finetuning 200 325 0.9 5e-4 0.031 Step decay (Step size 25, gamma 0.56) Yes
Setup for Table 3
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Nesterov
Training 30 256 0.9 5e-4 0.1 Step decay (Step size 25, gamma 0.1) Yes Finetuning 80 64 0.9 5e-4 0.1 Step decay (Step size 40, gamma 0.1) Yes
Setup for Table 4
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Nesterov
Training 200 450 0.9 5e-4 0.1 Step decay (Step size 25, gamma 0.56) Yes Finetuning 200 64 0.9 5e-4 0.1 Step decay (Step size 25, gamma 0.56) Yes
Setup for Table 5
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Nesterov
Training 200 128 0.875 5e-4 0.1 Cosine LR Yes Finetuning (GP 90%) 200 128 0.9 0 0.0512 Cosine LR Yes Finetuning (GP + MT 90%) 200 128 0.9 5e-4 0.0064 Cosine LR Yes Finetuning (GP 95%) 200 128 0.9 2e-5 0.0256 Cosine LR Yes Finetuning (GP + MT 95%) 200 128 0.9 5e-4 0.0064 Cosine LR Yes Finetuning (GP 97.5%) 200 128 0.9 0 0.0128 Cosine LR Yes Finetuning (GP + MT 97.5%) 200 128 0.9 6e-5 0.0256 Cosine LR Yes
Setup for Table 6
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Nesterov
Training 300 128 0.9 0.001 0.05 Cosine LR No Finetuning (GP 90%) 300 128 0.9 0.001 0.01 Cosine LR No Finetuning (GP + MT 90%) 300 128 0.9 0.001 0.01 Cosine LR Yes Finetuning (GP 95%) 300 128 0.9 1e-5 0.01 Cosine LR No Finetuning (GP + MT 95%) 300 128 0.875 0.0005 0.005 Linear LR No
Setup for Table 7
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Label Smoothing
Training 100 256 0.875 0.000031 0.256 Cosine LR (warmup=5) 0.1 Finetuning (GP 80%) 100 256 0.875 0.000023 0.0256 Cosine LR (warmup=5) 0.1 Finetuning (GP + MT 80%) 100 256 0.875 0.000023 0.0256 Cosine LR (warmup=5) 0.1 Finetuning (GP 90%) 100 256 0.875 0.000007 0.1024 Cosine LR 0.1 Finetuning (GP + MT 90%) 100 256 0.875 0.000007 0.0512 Cosine LR 0.1 Finetuning (GP 95.3%) 100 256 0.95 0.0 0.0512 Cosine LR 0.05 Finetuning (GP + MT 95.3%) 100 256 0.95 0.0 0.0512 Cosine LR 0.05 Finetuning (GP 98.05%) 100 256 0.95 0.0 0.0512 Cosine LR 0.05 Finetuning (GP + MT 98.05%) 100 256 0.95 0.0 0.0512 Cosine LR 0.05
Setup for Table 8
Stage Epochs Batchsize Momentum Weight Decay Initial LR LR Scheduler Label Smoothing
Training 100 256 0.875 3.1e-5 0.256 Cosine LR (warmup=5) 0.1 Finetuning (GP 75%) 120 256 0.875 1e-5 0.0512 Cosine LR 0.1 Finetuning (GP + MT 75%) 120 256 0.875 1e-5 0.0512 Cosine LR 0.1 Finetuning (GP 90%) 120 256 0.875 1e-5 0.0256 Cosine LR 0.1 Finetuning (GP + MT 90%) 120 256 0.875 0 0.0512 Cosine LR 0.1
Setup for Table 9
Stage Epochs Batchsize Initial LR
hd Optimizer
Training 300 100 0.0064 80 Adam Finetuning (9,8) 300 64 0.5 80 Adam Finetuning (9,7) 300 100 0.5 80 Adam Finetuning (8,7) 300 100 0.55 80 Adam

Review summaries:
This paper revisits Global Pruning (GP) and makes a case to consider it seriously as part of the pruning literature. The authors also present an addition to GP, that claims, to further increase accuracy and reliability called Minimum Threshold (MT). MT is the constraint put on every layer of the neural network to ensure they are not pruned beyond a certain limit which might result in catastrophic accuracy drops.   The paper presents experiments on CIFAR-10 with a few CNN architectures along with current pruning baselines (layer-wise mostly. They also show results on ImageNet using ResNet50 and MovileNetV1. Following these experiments, the authors claim that GP or GPMT are SOTA for pruning despite their simplicity.   The paper also includes some discussion on how output patterns of each layer change with different pruning schemes and a note about how to set the optimal MT values.   While I agree with the sentiment of the paper, I do not think the paper is presenting anything new (even from a benchmarking perspective) but rather is missing the point about what makes GP vs layer-wise pruning a worthy trade-off often. I will list my concerns in the main review.

The paper revisits a traditional model compression method, global magnitude pruning (GP), and shows that GP can achieves state-of-the-art. The paper further improves GP by introducing minimum threshold (MT). Experiments on ResNet show that GP+MT achieves better accuracy with the same sparsity ratio compared to GP.  

A very simple and effective pruning method is proposed. Instead of layer-wise pruning, a global threshold is used to prune weights according to their magnitude. In addition, to avoid over-pruning, a minimum number of parameters is preserved for each layer after pruning. Experiments on CIFAR-10 and ImageNet validate the effectiveness of proposed method.