Global Magnitude Pruning With Minimum Threshold Is All We Need
Authors: Manas Gupta
Vishandi Rudy Keneta
Abhishek Vaidyanathan
Ritwik Kanodia
Efe Camci
Chuan-Sheng Foo
Jie Lin
Venue: ICLR 2022 Submitted (2022)
Keywords: Pruning
Model Compression
One-shot
Global Magnitude Pruning

Abstract: Neural network pruning remains a very important yet challenging problem to solve. Many pruning solutions have been proposed over the years with high degrees of algorithmic complexity. In this work, we shed light on a very simple pruning technique that achieves state-of-the-art (SOTA) performance. We showcase that magnitude based pruning, specifically, global magnitude pruning (GP) is sufficient to achieve SOTA performance on a range of neural network architectures. In certain architectures, the last few layers of a network may get over-pruned. For these cases, we introduce a straightforward method to mitigate this. We preserve a certain fixed number of weights in each layer of the network to ensure no layer is over-pruned. We call this the Minimum Threshold (MT). We find that GP combined with MT when needed, achieves SOTA performance on all datasets and architectures tested including ResNet-50 and MobileNet-V1 on ImageNet. Code available on github.

Under review as a conference paper at ICLR 2022
GLOBAL MAGNITUDE PRUNING WITH MINIMUM
THRESHOLD IS ALL WE NEED
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network pruning remains very important yet challenging problem solve. Many pruning solutions have been proposed over the years with high de- grees of algorithmic complexity. In this work, we shed light on a very simple prun- ing technique that achieves state-of-the-ar (SOTA) performance. We showcase that magnitude based pruning, specifically, global magnitude pruning (GP) is suf- ficient to achieve SOTA performance on a range of neural network architectures. In certain architectures, the last few layers of network may get over over-pruned. For these cases, we introduce straightforward method to mitigate this. We preserve certain fixed number of weights in each layer of the network to ensure no layer is over- weer-pruned. We call this the Minimum Threshold (MT). We find that GP com- bined with MT when needed, achieves SOTA parameter- accuurcy trade-off on all datasets and architectures tested including ResNet-500 and MobileNet-V VI on Im- ageNet. Code available at http:///ggtbbb.oom////PrAAtttttt
Pruning-
INTRODUCTION
Neural network pruning remains important area from both practical perspective (deployment in real world applications) and academic perspective (understanding how to create an efficient archi- tecture). It is long standing area of exploration (LeCun al. 1990al Hassibi & Stork 1993a), and was reinvigorated by Han et al.] (2015a). Since then, much work has been done on trying to find different ways of pruning neural networks, such as magnitude-basedr gradient second order based, and regularization-bat often overlooked method that has been seen as mediocre baseline by the community - global magnitude pruning (GP), and show that it can achieve SOTA pruning performance.
methods amongst many others. In this work, we shed light on an
We demonstrate that GP by itself algorithms on benchmarks like ResNet-50 and MobileNet-Vi V1 on ImageNet. We also investigate the pruning behavior of GP and find that a simple addition to GP can raise performance even more. In contrast to the idea of sparsifying each layer of neural network to the maximum possible level, we find that preserving a certain amount of weights each layer actually leads to better pruning scheme, achieving higher accuracy the same sparsity level. We call this the Minimum Threshold (MT). When combined with GP, this technique enhances the pruning performance in most cases.
strong pruning algorithm and outperforms SOTA pruning
We conduct range of experiments to showcase the above and also study detailed ablations to isolate the effects of GP and MT. We obtain SOTA accuracies on four sparsity targets on ResNet-50 on ImageNet. We obtain SOTA accuracies other architectures and datasets tested well. Finally, GP with MT (GPM')) is very simple conceptually and very easy to implement. is a one-shot pruning method in which the weights to be pruned are decided in one-go without needing any iterative or gradual phases.
2 RELATED WORK
Compression of neural networks has become important research area due to the rapid increase in size neural networks (Brown et al.] 2020). the need for fast inference on edge devices, e.g., quadrotor's onboard computer (Camei al. 2020), and concerns about the carbon footprint of

---

Under review as a conference paper at ICLR 2022
training large neural networks (Strubell et al., 2019). Over the years, several compression techniques
have emerged in the literature (Cheng et al. 2017). such as quantisation, factorisation, attention,
knowledge distillation, architecture search and pruning (Almahairi et al. 2016 Ashok et al. 2017
landola et al. 2016 |Pham et al.) 2018b).
Quantisation techniques which restrict the bitwidth of parameters (Rastegari al. 2016 Cour-
bariaux et a 2016) and tensor factorisation and decomposition which aim to break large kernels
into smaller components (Mathieu et al. 2013] Gong et al.) 2014 Lebedey al. 2014 Masana et al. 2017) are popular methods. However, they need to be optimised for specific architectures.
Attention networks (Almahairi et al., 2016) have two separate networks focus on only small
patch of the imput image. Training smaller student networks in a process called knowledge distilla- tion (Ashok al. 2017) has also proved effective, although it can potentially require a large training budget. Architecture search techniques, such as new kernel design (landola et al. 2016) or whole architecture design (Pham et al. [0]8aa Tan et al., 2019) have also become popular. Nevertheless, the large search space size requires ample computational resources to do the architecture search. Different from all these approaches, we focus on pruning deep neural networks this work. As compared to other categories, pruning is more general in nature and has shown strong performance (Gale et al. 2019).
Many pruning techniques have been developed over the years, which use first second order deriva- tives
(LeCun et al. [990b] Hassibi & Stork 1993b), gradient based methods (Lee et al. 2018: Wang
et al.
2020), sensitivity to or feedback from some objective function (Molchanov al. [017]] Liu
ict al.
2020: Lin et al. 2020) de Jorge et al. 2021), distance or similarity measures (Srinivas &
BBau
2015).
y, regularization-basel techniques (Kusupati et al., 2020; Savarese et al.) [2020] Wang
cct al.
202T). and magnitude-based criterion (Ström 1997; Zhu & Gupta. 2018 Park et al. 2020
Evei et al.] 2020 Lee et al., (02I). HHn et al. (2015b) discovered a key trick to iteratively prune and retrain the network, thereby preserving high accuracy. Gale et al. (2019) adopt simple, magnitude-
based pruning but but employ gradual pruning that requires high computational budget and preset sparsi- fication schedules. Runtime Neural Pruning (Lin et al.] (017) attempts to use reinforcement learning (RL) for compression by training an RL agent to select smaller sub-networks during inference. [He] ct al. (2018) design the first approach using RL for pruning. However, RL training approaches typ- ically require additional RL training budgets (Gupta et al. (220 or iterative pruning to achieve good accuracy (He et al. (2018)).
In this work, we focus on a simple, effective, yet quite overlooked pruning method - global mag- nitude pruning (GP). Although first proposed in 1990s (Hoefler et al., (2021)), it has largely been ignored in recent years, generally being relegated to the position of a baseline for comparison (Zhu & Gupta 2018 Blalock al. 2020) Lee et al., (02T) rather than a strong pruning technique. A few recent works use it as one in a possible pool of pruning techniques (See et al. 2016 Frankle & Carbin 2018 Gohil al. 2020) but never study it detail or adopt it as the main pruning method. We delve deep into GP and showcase that it can achieve SOTA results with the addition of the MT technique. We present SOTA results on various architectures and datasets including ResNet-50 and MobileNet-V1 V1 on ImageNet, and include comprehensive ablation studies and insights on the workings of GP with MT (GPMT).
An advantage of GPMT is that it is a very simple and reliable approach. There are a few levels of simplicities and robustness to GPMT. Firstly, it is conceptually very simple and easy to implement. does not require any complex pruning frameworks like RL (He al., 2018)) or sparsification sched- ules (Zhu & Gupta. 2018). Secondly, it is one-shot and does not require any iterative procedure. Thirdly, it is easily generalizable across architectures and datasets, as shown in the experiments. Lastly, it data-independent and does not access the dataset for determining the pruning mask.
3 METHOD
We conduct unstructured weight pruning using magnitude pruning. Below we describe the key components of our algorithm in more detail.
2

---

Under review as a conference paper at ICLR 2022
3.1 GLOBAL MAGNITUDE PRUNING (GP)
GP is a magnitude based pruning approach whereby weights bigger than a given threshold are kept and weights smaller than the threshold are pruned.
Formally, for given threshold and each individual weight w in any layer, the new weight Wnew is defined as follows:
0
| <
Wnew
otherwise.
(I)
In contrast to layer-wise pruning, the threshold is not set per layer basis but rather single threshold set for the entire network. In this aspect, GP is much more efficient than layer-wise pruning because the threshold does not need to be searched for every layer. On the other hand, uniform pruning refers to setting the same sparsity target for each layer. Thus, every layer is pruned by the percentage.
3.2 MINIMUM THRESHOLD (MT)
The Minimum Threshold (MT) refers to fixed number of weights that are preserved in every layer of the neural network post pruning. The MT is a scalar value that is fixed before the start of the pruning cycle. The weights in a layer are sorted by their magnitude and the top MT number of weights are preserved. For instance, an MT of 500 implies that 500 of the largest weights in every layer need to be preserved post pruning. that all the weights of that layer must be preserved. Therefore, the MT is very simple concept to apply and also computationally inexpensive. This corresponds to-
layer smaller than the MT number, then it implies
o if 2 o,
min [
(2)
m otherwise
where Wi E In denotes the weight vector for layer a is the MT value terms of the number of weights and min |Wiloo indicates the number of non-zero elements in W.. We explain in the below section how the actual pruning using MT is implemented.
3.3 THE PRUNING WORKFLOW
The pruning pipeline for GP and GP with MT (GPMT) is straightforward. It consists of pruning the original model followed by fine-tuning for few epochs. It is one-shot, therefore, the pruning & fine-tuning cycle does not need to be repeated multiple times. In terms of the pruning procedure itself, for GP, it consists of doing one pass over the network and pruning the weights according to their magnitude to reach the specified sparsity target. For GPMT, it consists of two steps. Firstly, the model is pruned using GP. Secondly, the pruned model is evaluated to check the MT condition met by all layers not. If the condition is not met by layer, then its sparsity ratio is reduced to meet the MT. The slack arising from the decrease in sparsity is then redistributed amongst the other layers which do not violate the MT condition. The redistribution is done in proportion their existing sparsities so as to preserve their relative sparsities. This finishes the pruning cycle and the network is then fine-tuned. Fig. explains this procedure, while Algorithm gives the pseudocode.

---

Under review as a conference paper at ICLR 2022
Global Pruning (GP) Target Sparsity 60%
Minimum Threshold (MT)
MT = 10%
15-3
13
20-20= 0 weights
12-2=
12 -2
0+ 6
Weights
25
20 weights
12
remaining
weights
weights
weights
weights
Figure I: An example workflow over three-layer network with 60 weights. Thicker lines represent the weights with high magnitude. The network is first pruned to 60% sparsity globally, then MT of 10% (6 weights) is activated per layer. Since the last layer is completely pruned by GP. MT recovers the minimum number of connections there. distributes the slack arising from the drop in sparsity to the other layers in proportion to their existing sparsities, i.e., 2 weights for Layer 1 (sparsity 20%) and 4 weights for Layer 2 (sparsity 52%). Red color indicates changes in weights due GP and yellow indicates changes due to MT.
WRN- 22-8 on CIFAR-10
Method
Sparsity
Starting Acc.
Pruned Acc.
Uniform Pruning
95%
94.07% + 0.05% 94.07% 0.05%
94.16% + 0.10%
GP
95%
43% + 0.02%
GP + MT
95%
94.07% + 0.05%
94.64% + 0.14%
Table I: GP improves performance on Uniform Pruning and adding MT improves performance even further.
MobileNet-V2 CIFAR-10
Method
Sparsity
Starting Acc.
Pruned Acc.
Uniform Pruning
40%
94.15% + 0.23%
03.76% t 0.18%
GP GP + MT
40%
94.15% + 0.23% 94.15% 0.23%
94.07% + 0.14%
40%
94.21 0.12%
Table 2: Same trend is seen for MobileNet as well where GP outperforms Uniform Pruning and adding MT improves performance even further.
EXPERIMENTS
Below we describe experiments related to ablations on global magnitude pruning (GP) and Mini- mum Threshold (MT), comparison with state-of-the-at algorithms and experiments non-vision domains. We report hhper-yypaanaaaee parameters and training related information for all the experiments in the appendix (section A.4).
4.1 ISOLATING AND UNDERSTANDING IMPACT OF GP AND MT OVER DIFFERENT ARCHITECTURES
We conduct detailed ablations isolate and measure the impact of standalone GP and GP with MT (GPMT) compared to a uniform pruning baseline. We ablate on multiple architectures and sparsity targets. addition, we report results averaged over multiple runs where each run uses a different pre-trained model to provide more robustness. We first prune WRN-22-8 model on CIFAR-I0 at 95% sparsity. We then fine-tune the model for few epochs and report the final accuracy. We experiment with different pruning schemes, i.e., uniform pruning, GP and GPMT (see Section for details). We find that GP outperforms uniform pruning. Furthermore, adding MT improves the performance MobileNet- V2 architecture to see if the above conclusion holds on too. We find that indced GP beats uniform pruning in this situation well, and adding MT improves performance even further
more, Table Next, we do an experiment on the highly efficient
4

---

Under review as a conference paper at ICLR 2022
(Table 2 more.
This shows that GP by itself is superior to uniform pruning and adding MT aids GP even
WRN-22-8 CIFAR-IO
Method
Sparsity
Starting Acc.
Pruned Acc.
GP GP + MT
99.9%
94.07% + 0.05% 94.07% + 0.05%
67.68% + 0.78%
99.9%
68.42% + 0.58%
Table MT improves performance on WideResNet-22-8 even in the high sparsity regime at 99.9%
sparsity.
MobileNet-V2 on CIFAR-10
Method
Sparsity
Starting Acc.
Pruned Acc.
GP GP
98.0%
94.15% + 0.23%
10% (Unable learn) 82.97% + 0.57% 87.36% % + 0.18%
98.0%
94.15% + 0.23%
Gradual GP
98.0%
NA
Table Adding MT conducting GP gradually enables the MobileNet model to learn and achieve good classification performance in the high sparsity regime.
We then conduct experiments under much tougher conditions, increasing the sparsity rates on both the above-mentioned models see how the algorithm performs. We find that at 99.9% sparsity, the WRN is able to get decent accuracy and adding MT improves performance even at this sparsity rate (Table B}. For MobileNet however, using GP only, accuracy drops 10% and the model is not able to learn. We discuss this in detail in Section 5.1 However, on adding the MT, the model is able to learn and the accuracy jumps back to 83% (Table Alternatively, conducting GP gradu- ally (Zhu Gupta 2018) also enables the model learn well and achieves 87.36% accuracy, We provide layer-wise weight snapshot for both the models before and after applying MT, to illustrate what MT does (Fig. and Fig. 3). Thus, in cases of high sparsity, MT desirable to have.
3030
Number of terraining weigghss
With
3000
(59.34% accurwc))
2000
.20
Figure 2: Weights remaining in WRN-22-8 model after pruning at 99.9% sparsity. MT helps retain weights in layers that have low magnitude weights and are heavily pruned, e.g., Layers 20, 21, and 22.
4.2 STATE-OF-THE-ART COMPARISON ON CIFAR-10
We also conduct experiments to compare GP and MT SOTA pruning algorithms on the CIFAR- 10 dataset. We compare with various algorithms including SNIP (Lee et al. 2018), SM (Dettmers & Zettlemoyer 2019). DSR (Mostafa Wang 2019) and DPF (Lin et al. 2020). We start off with the original model having the same initial accuracy the other algorithms. For the WRN-28-8 model we find that GP alone beats all the SOTA pruning algorithms at two out of the three sparsity levels (Table 15. If we incorporate MT, then the accuracy increases further, and the algorithm is able to beat all the SOTA algorithms at all the sparsity levels. For ResNet-32 as well (Table both GP and GPMT outperform all the other algorithms 90% sparsity.

---

Under review as a conference paper at ICLR 2022
0:20
Number of remaining Wrights
Withost IRAI (27 acnurnn)
2530
00:0
1530
Layer indox
Figure 3: For MobileNet- V2 at 98% sparsity as well, MT is essential to retain some weights in the heavily pruned layers (Layers 55, 56, and 57) and allow the model to learn successfully.
4.3 STATE-OF-THE-ART COMPARISON ON IMAGENET
We apply GP and MT ResNet-50 on the ImageNet dataset well and compare to the SOTA pruning algorithms. We compare with SOTA algorithms like GMP (Zhu & Gupta 2018 DSR (Mostafa & Wang 2019). DNW (Wortsman
al. 2019), SM (Dettmers & Zettlemoyer 2019),
RigL (Evci et al. 2020). DPF (Lin et al. 2020) and STR (Kusupati et al., (020). We start with the same original accuracy the other models. We find that GP alone gets good performance and surpasses all the other algorithms at all the sparsity levels (see Table 7 keeping the numbers of parameters constant. Adding MT leads roughly the
performance as GP (slightly higher and slightly lower certain cases) and this is explained by the fact that all layers in ResNet-50 always maintain enough weights (i.c., 1,000) at all sparsity levels. GP incurs higher FLOPs compared to some baselines as it prunes the last layers more vs. the initial layers (which have higher FLOPs to parameters ratio) compared to some baselines. Optimization for parameters vs. FLOPs is usually trade-off against each other and joint optimization of parameters and FLOPS can be implemented as future work to reduce the FLOPs of GP. GP still outperforms other baselines like DSR, SM, SM ERK and RigL- ERK on both accuracy and FLOPs.
We also test another architecture on ImageNet, MobileNet-VL. which is a much smaller and more efficient architecture than ResNet-50. We start with the same original accuracy as the other models. For MobileNet-Ve V1, GP alone gets good results 75% sparsity and outperforms SOTA algorithms by 2.44% on constant parameter budget basis (Table 8 On combining with MT, GPMT also surpasses the SOTA algorithms 90% sparsity by 2.43%. Since MobileNet-Vi is very efficient architecture (having only 4.2 million parameters compared to 25.6 million for ResNet-50), MT especially helps at higher sparsities when the layers are pruned very aggressively.
4.4 GENERALIZING TO OTHER DOMAINS AND RNN ARCHITECTURES
We experiment with the GP rule on other domains and non-convolutional networks as well to mea- sure the generalizability of the algorithm on different domains and network types. We experiment on FastGRNN model (Kusupati et al. (2018) on the HAR-2 Human Activity Recognition dataset (Anguita et al. (2013)). More details on the dataset and the experimental setup can found in Section A.3 We test the GP rule under different network configurations. We find that GP surpasses other pruning algorithms on all the configurations (Table 9 and successfully prunes the model on
very different architecture and domain.
INSIGHTS ON GP AND MT
5.1 NETWORK ARCHITECTURES CAN AFFECT PRUNING PERFORMANCE
We find that network architectures can affect the results how the pruning algorithm performs quite lot, especially in the high sparsity domain. We take the case of the high sparsity experiments

---

Under review as a conference paper at ICLR 2022
between MobileNet- V2 and WRN-22-8, as mentioned in Section 4.1 In the case of just using GP, we find that WRN-22-8 is still able to learn, however, accuracy crashes for MobileNet-N V2 and the model is unable to learn anything, getting a chance accuracy rate of 10%. The reason for this wide discrepancy in learning behavior lies in the shortcut connections (He et al. 2013). Both WRN-22-8 and MobileNet-V2
shorteut connections, however, their placement different. Referring Fig. 4. WRN uses identity shortcut connections from Layer 20 to Layer 23, Identity shortcut connections simple identity mappings and do not require any extra parameters, and hence, they do count towards the weights. However, MobileNet- V2 uses convolutional shortcut mapping from Layer 52 to Layer 57 and hence, it does add the model's ' weights, and thus, it pruned by the pruning algorithm. Both the models have the preceding two layers before the last layer, completely pruned. However, because WRN uses identity mappings, still able to relay information to the last layer, and the model is still able to learn.
Method
Top-1 Ace
Params. Sparsity
Method
Top-1 Acc
Params. Sparsity
WRN-28-
96.06%
23.3M
0.0%
ResNet-
93.83 + 0.12 %
0.46M 0.00%
8
32
SNIP
95.49 + 95.49 0.21%
2.33M
90%
SNIP
90.40 + 0.26%
0.046M 90%
SM
95.67 0.14%
2.33M
90%
SM
91.54 + 0.18%
0.046M 90%
DSR
95.81 + 0.10%
2.33M
90%
DSR
91.41 + 0.23%
0.046M
90%
DPF
96.08 + 0.15%
2.33M
90%
DPF
92.42 + 0.18%
0.046M
90%
GP
96.30 1 0.03 %
2.33M
90%
GP
92.67 1: 0.03% %
0.046M 90%
GP + MT
96.44 1 0.09 %
2.33M
90%
GP + MT
92.74 1 0.06%
0.046M 90%
SNIP
94.93 + 0.13%
1.17M
95%
SNIP
87.23 + 0.29%
0.023M 95%
SM
95.64 + 0.07%
1.17M
95%
SM
88.68 + 0.22%
0.023M 95%
DSR
95.55 + 0.12%
1.17M
95%
DSR
84.12 + 0.32%
0.023M 95%
DPF
95.98 + 0.10%
1.17M
95%
DPF
90.94 + 0.35%
0.023M 95%
GP
96.16 - 0.02 %
1.17M
95%
GP
90.65 + 0.13%
0.023M 95%
GP MT
96.27 + 0.06 %
1.17M
95%
GP MT
90.58 + 0.24%
0.023M 95%
SNIP
94.11 + 0.19%
0.58M
97.5%
SM
95.31 + 0.20%
0.58M
97.5%
Table 6: Results of pruning algorithms on
DSR
95.11 + 0.07%
0.58M
97.5%
ResNet-32 on CIFAR-10, GPMT outperforms all
DPF
95.84 1 0.04%
0.58M
97.5%
the other algorithms at 90% sparsity.
95.68 - 0.08%
0.58M
97.5%
GP MT
95.89 + 0.02%
0.58M 97.5%
Table 5: Results of SOTA pruning algorithms on WideResNet-28-8 on CIFAR-10, GP + MT (GPMT) outperforms all the other algorithms.
Method
Top-1 Ace
Params.
Sparsity FLOPs
Method
Top-1 Acc
MobileNet-VI
71.95%
4.21M
0.00%
569M
FastGRNN
96.10%
80
GMP
67.70%
1.09M
74.11%
163M
Vanilla Training
94.06%
8
STR
68.35%
1.04M
75.28%
101M
STR
95.76%
8
70.74%
1,04M
75.28%
177M
GP
95.89%
8
GP + .T
70.79%
1.04M
75.28%
204M
Vanilla Training
93.15%
7
GMP
61.80%
0.46M
89.03%
82M
STR
95.62%
7
STR
61.51%
0.44M
89.62%
40M
GP
95.72%
7
GP
59.49%
0.42M
90.00%
93M
Vanilla Training
94.88%
7
GP + MT
63.94%
0.42M
90.00%
154M
STR
95.59%
7
GP
95.62%
7
Table 8: Results of pruning algorithms on MobileNet-VI on ImageNet. GPMT surpasses the SOTA algorithms by 2.4% accuracy.
Table Results of pruning algorithms on FastGRNN on HAR-2 dataset. GP outperforms other pruning algorithms in all the different network configurations.

---

Under review as a conference paper at ICLR 2022
Method
Top-1 Acc
Params. Sparsity FLOPs
Method
Top-1
Params. Sparsity FLOPs
Acc
ResNet-50
77.0%
25.6M
0.00%
4.09G
ResNet-50
77.0%
25.6M
0.00%
4.09G
GMP
75.60%
5.12M
80.00%
818M
GMP
70.59%
1.28M
95.00%
204M
DSR*
71.60%
5.12M
80.00%
1.23G
DNW
68.30%
1.28M
95.00%
204M
DNW
76.00%
5.12M
80.00%
818M
RigL*
67.50%
1.28M
95.00%
317M
SM
74.90%
5.12M
80.00%
RigL + ERK
70.00%
1.28M
95.00%
600M
SM ERK
75.20%
5.12M
80.00%
1.68G
STR
70.97%
1.33M
94.80%
182M
RigT.*
74.60%
5.12M
80.00%
920M
STR
70.40%
1.27M
95.03%
159M
Rigl. + ERK
75.10%
5.12M
80.00%
1.68G
STR
70.23%
1.24M
95.15%
162M
DPF
75.13%
5.12M
80.00%
818M
GP
71.56%
1.20M
95.30%
437M
STR
76.19%
5.22M
79.55%
766M
GP + MT
71.57%
1.20M
95.30%
438M
GP
76.84%
5.12M
80.00%
1.13G
GMP
57.90%
0.51M
98.00%
82M
GP + MT
76.75%
5.12M
80.00%
1.28G
DNW
58.20%
0.51M
98.00%
82M
GMP
73.91%
2.56M
90.00%
409M
STR
61.46%
0.50M
98.05% 73M
DNW
74.00%
2.56M
90.00%
409M
GP
61.80%
0.50M
98.05%
257M
SM
72.90%
2.56M
90.00%
1.63G
GP + MT
61.90%
0.50M
98.05% 257M
SM + ERK
72.90%
2.56M
90.00%
960M
RigL*
72.00%
2.56M
90.00%
515M
RigL+ + ERK
73.00%
2.56M
90.00%
960M
DPF#
74.55%
4.45M
82.60%
41
STR
74.73%
3.14M
87.70%
402M
GP
75.28% %
2.56M
90.00%
704M
GP MT
75.21%
2.56M
90.00%
926M
Table 7: Results on ResNet-50 on ImageNet. GP and GPMT outperform SOTA pruning algorithms by upto 1.3% accuracy. * and # imply that the first and last layer are dense respectively.
WidaittsNNtt-2 22-8
Mubiletten-V2 V2
I00 Icl tew.ee
[S]
140
Global Pruning (94.40% accuracy) Pruning accuracyl
10 Idloettoon Ea3ia
200
Output
8g
221
Shortsst layer
Dowmserlly pinee
Index
Telyenmeene instii0.eei
Figure Difference in architectures be- tween WRN and MobileNet. WRN does not have any prunable residual connections in the last layers (dotted lines) while Mo- bileNet does. This leads to different pruning behaviors on the two architectures.
Figure 5: The frobenius output distortion is lower for GP compared to uniform pruning on a layer by layer basis. Thus, GP preserves outputs closer to the original unpruned model compared to uniform pruning. Results WRN-22-8 on CIFAR- 10 dataset.
Pruning algorithms can be susceptible to such catastrophic network disconnection issues especially in the high sparsity domain. Fortunately, the MT can easily overcome this issue. Retaining a small MT of 0.02% was sufficient for the MobileNet- V2 model to avoid disconnection and learn success- fully. Hence, retaining small MT can help the learning dynamics of models in high sparsity settings.
5.2 GP PRESERVES OUTPUTS BETTER THAN UNIFORM PRUNING
To understand why GP performs better than uniform pruning, especially at the layer-by-layer level, measure the Frobenius output distortion between the original model and the pruned model (Park)
8

---

Under review as a conference paper at ICLR 2022
et al. 2020). We compare the difference in the layer-wise outputs between the original model and the pruned model, and then compute the norm of this difference. Thus, a smaller norm indicates that the output of the pruned model is closer to the original model and larger norm indicates that the output is further away. We compute and plot this layer-wise output distortion on the WRN-22-8 model on CIFAR-10 for the GP and uniform pruning models as compared to the original model (Fig. 5). As can be seen from the figure, the distortion is much higher for the uniform pruning model. Layers and are especially impacted, with the distortion being around 3x that of GP in layer 1. Thus, GP is able to preserve outputs much closer to the original model as compared to uniform pruning. This points towards new direction for empirically understanding the underlying differences between pruning algorithms.
5.3 WHEN IS MT REQUIRED AND HOW TO DETERMINE THE OPTIMAL VALUE
We find that whether MT is required or not depends on the neural network architecture family. For instance, in our experiments on multiple architectures, we found generally that ResNet architectures did not require MT (tested on ResNet-32 and ResNet-50). Conversely, MobileNet and WideRes- Net architectures did require MT (tested on MobileNet-Vi. MobileNet- V2, WideResNet-222 WideResNet-255 28--8). Hence, we hypothesize that the requirement for MT depends upon the base family architecture used. We also found that in architectures that require MT, decent rule of thumb value for MT is 0.05% of the total weights. Increments of 0.05% may be tried, i.e., 0.1%, 0.15% or 0.2%, if need be. We never had to go beyond 0.2%. In cases of high sparsity, where there is not enough capacity in the network to support 0.05% MT, lower values of MT can be tried, e.g., 0.02%, 0.01% %6 0.005% or 0.002%. We found that the above-stated values were sufficient to get good perfor- mance across models and we never had to undertake more exhaustive MT search. In most cases, we only tried two to three MT values and achieved good results.
and
6 LIMITATIONS AND FUTURE WORK
limitation of the GP method that it can incur higher FLOPs compared to some layer-wise base- lines (Table 77 This because it prunes the last layers more with respect the initial layers (that have a higher FLOPs parameter ratio) compared to some baselines. In applications where FLOPs reduction is equally important as parameter reduction, way mitigate this would be to add con- straints on the FLOPS as well along with the parameters, and thus jointly optimize for both FLOPs and parameters. We will look at this in future work. Another exciting extension to GP is to do GP gradually instead of one-shot. As demonstrated in Table 4 doing GP gradually enhances the accu- racc performance of GP. We believe this increase in performance is transferable to other datasets and architectures as well, and is another topic that we will look at in the future.
7 CONCLUSION
Many methods have been proposed for neural network pruning over the years. In this paper, we focus on an often overlooked pruning method - global magnitude pruning (GP). We show that GP by itself is a strong pruning method and achieves SOTA performance on ResNet-50 and MobileNet- VI on ImageNet. Furthermore, we investigate the pruning behavior of GP in depth and find that in certain cases, c.g., at high sparsities, a few layers in the network may be over-pruned. To rectify this, we propose novel yet very simple mitigation mechanism called Minimum Threshold (MT). MT preserves a certain fixed amount of weights in every layer post pruning and ensures no layer is over-pruned. We find that adding MT on top of GP leads to higher performance across many architectures and datasets. We achieve SOTA results on CIFAR-10 and ImageNet using the above pruning technique. GP with MT (GPMT) is also easy to implement, is one-shot and is easily portable across architectures and datasets. GP can be further extended to be done gradually instead of one- shot and can lead to more performance gains. A limitation of GPMT is that it can incur higher FLOPs compared to some layerwise methods and a possible miligation for this is to add FLOPs constraint on top of the parameter constraint when doing pruning. Overall, we find that GPMT can be used as useful baseline for future pruning works given it's simplicity and performance.
9

---

Under review as a conference paper at ICLR 2022
8 REPRODUCIBILITY
To help with reproducibility, we have released our codebase. It can be found at SiggggttbboooonGMM///tttoooeooee PrrnGlooaa--rung--WWe -re-i-rnntnnn--iiiiiiii Www.iiii-mmmmsmhmmaae
Furthermore, report in detail the experimental setup including all the hyper-parameters for all our experiments in the appendix (Section A.4). The above two items ensure that all our experiments are repro- ducible. Furthermore, our method is conceptually very simple and hence, very easy to implement. It does not require additional cetoortttm-ppccfiii
or any prchitecture-speciff tweak- ing. We use standard Py PoTrhhh libraries (Paszke et al. (2019)) without any modification to the model layers. These make the GPMT method straightforward to implement, and hence, easy to replicate. Lastly, we show that GPMT gets stable results across multiple runs and pre-trained models (Figs. and 7 and hence, aids in reproducibility.
REFERENCES
Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, and Aaron Courville. Dynamic capacity networks. In International Conference on Machine Learning, pp- 2549-2558, 2016
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L., Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. 21th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2013, Bruges, Belgium, 2013.
Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, and Kris M. Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning, 2017.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint arXiv:2003.03033, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
Efe Camei, Domenico Campolo, and Erdal Kayacan. Deep reinforcement learning for motion plan- ning of quadrotors using raw depth images. In 2020 international joint conference on neural networks (LJCNN), pp. 1-7. IEEE, 2020.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. survey of model compression and acceleration for deep neural networks, 2017.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to or 11 2016.
Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Grégory Rogez, and Puneet K. Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. In International Conference Learning Representations, 2021. URLhttps: wwpp///eeeeeeeeeeeee/ee/e. nwt. forum??
id 9GsFOUyUP1
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. CoRR, abs/ htbb//999700444 2019. URL.ttpp:/aaxxi wttp://p.w../b.....t/t///b/t 1907..4440000
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2943 2952. PMLR, 13-18 Jul 2020. URL actps: ptteps:://poccedinggs.tt
preas/v119/ / 20a. html
10

---

Under review as a conference paper at ICLR 2022
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, arXiv preprint arXiv:1803.036:: 2018.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv 2019.
preprint
Varun Gohil, S. Deepak Narayanan, and Atishay Jain. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. ReScience C, 6(2), 2020. URL Accepted at NeurlPS 2019 Re-
/ttp:/s////petee..eeeeeeeee/ producibility Challenge.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net-
works using vector quantization, 2014.
Manas Gupta, Siddharth Aravindan, Aleksandra Kalisz, Vijay Chandrasekhar, and Lin Jie. Learning to prune deep neural networks via reinforcement learning. International Conference on Machine Learning (ICML) AutoML Workshop. 2020.
Song Han, Jeff Pool, John Tran, and William Dally.
Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015a. URL
ccc/paper/211///1eee
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume Morgaan-Kuffmann 1993a. URL httpps://pprcceeennggss neurips.
cc/paper/199/ 992//9999222 (tt.e//000.......030030020000000
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In S. J. Hanson, J. D. Cowan, and C. L. Giles (eds.), Advances in Neural Information -aagaan,a 1993b.
Processing Systems 5, pp. 164-171,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/15 112.03388, 2015. URL http: http:://aaxxi..oorr/b/////11111 wttp://///bb///...033933 URL
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration mobile devices. In European Conference on Computer Vision (ECCV), 2018.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint ar arXiv:2102.0055. 2021.
Forrest N. landola, Song Han, Matthew W. Moskewicz. Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and j0.5mb model size, 2016.
Aditya Kusupati, Manish Singh, Kush Bhatia, Ashish Jith Sreejith Kumar, Prateek Jain, and Manik Varma. Fastgrnn: fast, accurate, stable and tiny kilobyte sized gated recurrent neural network. In NewrlPS, 2018
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Confer- Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5554 PMLR, 13-18 Jul 2020. URL http::/pppt http://pprceedingg mlr r. ptwss//1ss//////// kusupat 20a. html

---

Under review as a conference paper at ICLR 2022
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding up convolutional neural networks using fine-tuned cp-decomposition. International Conference on Learning Representations, 2014.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touret- zky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1990a. URL httpps:: http://ooooeddno.....iiiessi...p wttpaapppppppppppppe.aaeaaamaeee hctpppaaaapp/999///eee 6c682bbacce113552231113237777333
pdf
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. Touretzky (ed.),
Advances in Neural Information Processing Systems 2, pp- 598 605. Morgan-Kaufmann, 1990b.
n .npss.ppp....... httppp/scccpppppe////20ee22 URL http:///w..pppp.raoopoooooraaaer
Jaeho Lee. Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for
the magnitude-based pruning. In International Conference on Learning Representations, 2021.
URL htt.t:ttss///pptteee
net/ wtt.ewweeeoeeeeeeeeeeeeeeee forum? d (JCTkdF
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Tom. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv: 1810 1810.02340, 2018.
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou Runtime neural pruning. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2181-2191. Curran Associates, Inc., 2017. URL runt - aeeeennural-ae -
httt:://apper: wwwpearssnnpppa www.. httpipssccpppppr/ee///
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL https:
/oopnneeveww ferstt/foummti S.e
Junjie Liu, Zhe XU, Runbin SHI, Ray C. C. Cheung, and Hayden K.H. So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In International Confer- ence on Learning Representations, 2020. URL https:://peneevie..rrt www.nnteffooodmdd SJIbGJrtDDB
Mare Masana, Joost van de Weijer, Luis Herranz, Andrew D. Bagdanov, and Jose M. Alvarez. Domain-adaptive deep network compression. 20/7 IEEE International Conference on Computer Vision (ICCV), Oct 2017. doi: 10.1109/icc.22017.46 URL http://dx.... http://xxxddiio..o.. /ot.://///..doo..g/11..... org/10.
1109/ICCV 2017.660
Michael Mathieu, Mikael Henaif, and Yann LeCun. Fast training of convolutional networks through ffts, 2013.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. International Conference on Learning Represen- tations, 2017.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Proceedings of the 36th International Conference on Machine Learning, pp. 4646 4655. PMLR, 2019.
Sejun Park, Jacho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: far-sighted alternative of magnitude-based pruning. International Conference on Learning Representations, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL www.hhtpp.. http::///pprrrsse.uuee http:ppp..cpppp....pp/ 2015-pytorchs an wtw.aaa-atttteeettttteeeeee otg.a//eeeeee.eeiioooooeeeoereer deep- waw.amiinng-bbbar......m Librar.r......
pddf
12

---

Under review as a conference paper at ICLR 2022
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 4095-41044 Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018a. PMLR. URL httphhttp://prceeeeiiie http.eedeenn..........rss//////a
mi wwwpppprssssspppeeeeeeeee 8a html
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018b.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. Lecture Notes Computer Science, pp. 525 542, 2016. ISSN 1611-3344 doi: 10. 10111109778-3331-466600 dx. doi. wrg/r////... 10. 1007/978888888 -- ---1------------ 0-493-043202
32. URL http://
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 2020.
Abigail See, Minh- Thang Luong, and Christopher D Manning. Compression of neural machine translation models via pruning. arXiv preprint arXiv: 09274, 2016.
Suraj Srinivas and Venkatesh Babu. Data-free parameter pruning for deep neural networks. Procedings of the British Machine Vision Conference 2015, 2015. doi: 10.5244/c.29.11. URL
http: //dx doi www.orgg////.. 5244/C.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Nikko Ström Sparse connection and pruning in large dynamic artificial neural networks, 1997.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-ave aware neural architecture search for mobile. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), Jun 2019, doi: 10. 10.1109/cvpr. 2019.00293, URt.htpp:// wwwpddp...w........oooo.o//
doi www.orog11.. 10.9//1..11///PPRR.... 2019 002.9
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference Learning Representations, 2020. URL
htttps://openrevier net www.mrid--oooo SkgsACKKP
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations, 2021. URL URI/https:: ptatttps://openeevies net
Ts [s_
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL ott.://cccoorrraaaaaaaccaeeeeeee
M. Zhu and S. Gupta. prune, or not prune: exploring the efficacy of pruning for model compression. ICLR Workshop, abs/1771.00877/ 2018.
A APPENDIX
A.1 SIMPLE ONE-SHOT APPROACH
A.2 PSEUDO-CODE
13

---

Under review as a conference paper at ICLR 2022
Number remaining weights
0230
Funil fun minunA
2530
20:0
Layer
Figure 6: Layer-wise pruning results produced by GP on MobileNet- V2 model on CIFAR-10. Prun- ing is done three different pre-trained models and the pruning results across the three runs are very stable.
Number of remaining weights
3030
BRin Rin
2:3
200
1330
1030
Layor
Figure 7: similar trend is observed for the case of GPMT on MobileNet- V2 model on CIFAR-10 well. Pruning results on the three different pre-trained models very stable.
Algorithm Pseudo-code for GPMT
Load a pre-trained DNN with weight vectors Wj for each layer /
Set target sparsity,
Set an MT value, Concatenate and sort all Wi into a vector W= {wi. W1, ... w. where w; S w141
A: 100
Zero-mask all elements from wp wy W across all Wi
Slack pruning budget, Σ = (
for / layers do
if min Willo then
Restore the (a min Willo) of highest magnitude weights in Wi
= Σ + min [Wlloo
end if end for
Distribute Σ into each layer where min |Wiloo > a,, proportionally to their sparsities Fine-tune the pruned model
A.3 SPARSIFYING FASTGRNN ON HAR-2 DATASET
HAR-2 dataset that is used in the FastGRNN pruning experiment is binarized version of the 6- class Human Activity Recognition dataset. From the full-rank model with I'w = 9 and Ty = 80 suggested the STR paper (Kusupati et al. 2020), apply GP the matrices W, and W2. To do this, we find the weight mask by ranking the columns of W, and W, based on their absolute new lowest columns from W, and W2
sum, then prune the 9 new lowest columns and 80
14

---

Under review as a conference paper at ICLR 2022
respectively. In the end, fine-tuned this pruned model by retraining it with FastGRNN's trainer and applying the weight mask on every epoch.
A.4 HYPER-PARE PARAMETERS AND EXPERIMENTAL SETUP
No data augmentation is done apart from standard data pre-processing. Difference in batch size for training and testing in some experiments is due to GPU RAM availability. Averages reported over three runs.
MT values
Experiment
Sparsity
MT value
Table
95%
0.05%
Table
40%
0.05%
Table
99.9%
0.002%
Table
98%
0.02%
90%
0.05%
Table
95% %
0.10%
97.5%
0.05%
90%
0.05%
6
95% %
0.05%
80%
0.05%
90%
0.05%
Table
95.3%
0.005%
98.05%
0.005%
75%
0.05%
Table
90%
0.2%
Setup for Table
Stage
Epochs Batch- size
Mom-
Weight
Initial
LR Scheduler
Nes-
entum
Decay
LR
terov
Training
5e-4
Step decay (Step size 25, gamma 0.
Finetuning
80
1800
0.9
5c-4
0.1
Step decay (Step size 40, gamma 0.1)
Yes
Setup for Table
Stage
Epochs
Batch- size
Mom- entum
Weight Decay
Initial
LR Scheduler
Nes-
LR
terov
Training
200
450
0.9
5e-4
0.1
Step decay (Step size 25, gamma 0,56)
Yes
Finetuning
200
325
0.9
5e-4
0.031
Step decay (Step size 25, gamma 0.56)
Yes
15

---

Under review as a conference paper at ICLR 2022
Sctup for Table
Stage
Epochs
Batch- size
Mom- entur
Weight Decay
Initial LR Scheduler
Nes-
LR
terov
Training
256
0.9
Se-4
0.1
Step decay (Step size gamma 0.1)
Yes
Finetuning
5e-4
Step decay (Step size 40, gamma 0.1)
Setup for Table
Stage
Epochs
Batch- size
Mom- entum
Weight
Initial
LR Scheduler
Nes-
Decay
LR
terov
Training
200
5e-4
Step decay (Step size 25, gamma 0.56)
Yes
Finetuning
200
64
0.9
5e-4
0.1
Step decay (Step size 25, gamma 0.56)
Yes
Sctup for Table
Stage
Epochs
Batch- size
Mom- entum
Weight
Initial
LR Scheduler
Nes-
Decay
LR
terov
Training
200
128
0.875
5c-4
0.1
Cosine LR
Yes
Finetuning (GP 90%)
200
128
0.9
0,0512
Cosine LR
Yes
Finetuning (GP MT 90%)
200
128
0.9
5e-4
0,0064
Cosine LR
Yes
Finetuning (GP 95%)
200
128
0.9
2e-5
0.0256
Cosine LR
Yes
Finetuming (GP
95%)
200
0.9
5c-4
0.0064
Cosine LR
Yes
Finetuming (GP 97.5%)
200
128
0.9
0.0128
Cosine LR
Yes
Finetuning (GP MT 97.5%
200
128
0.9
6e-5
0,0256
Cosine LR
Yes
Sctup for Table 6
Stage
Epochs
Batch- size
Mom- entum
Weight
Initial
LR Scheduler
Nes-
Decay
LR
terov
Training
300
128
0.9
0.001
0.05
Cosine LR
No
Finetuning (GP 90%)
300
128
0.9
0.001
0.01
Cosine LR
No
Finetuning (GP MT 90%)
300
128
0.9
0.001
0.01
Cosine LR
Yes
Finetuning (GP 95%)
300
0.9
1c-5
0.01
Cosine LR
No
Finetuning (GP MT 95%)
300
128
0.875
0.0005
0.005
Linear LR
No
Setup for Table
Stage
Epochs
Batch- size
Mom- entum
Weight
Initial LR Scheduler
Label
Decay
LR
Smooth
ing
Training
100
256
0.875
0.000031
0.256
Cosine
LR
0.1
(warmup=5)
Finetuming (GP 80%)
100
256
0.875
0.000023
0.0256
Cosine
LR
0.1
(warmup=5)
Finetuning (GP + MT 80%)
100
256
0.875
0.000023
0.0256 Cosine
LR
0.1
(warmup=S)
Finetuning (GP 90%)
100
256
0.875
0.000007
0.1024 Cosine LR
0.1
Finetuning (GP MT 90%)
256
0.875
0.000007
0.0512 Cosine LR
Finetuning (GP 95.3%)
100
256
0.95
0.0
0.0512 Cosine LR
0.05
Finetuning (GP MT 95.3%)
100
256
0.95
0.0
0,0512 Cosine LR
0.05
Finetuning (GP 98.05%) ))
100
256
0.95
0.0
0,0512 Cosine LR
0.05
Finetuning (GP+ MT 98.05%)
100
256
0.95
0.0
0.0512 Cosine LR
0.05
16

---

Under review as a conference paper at ICLR 2022
Setup for Table
Stage
Epochs Batch- size
Mom- entum
Weight
Initial
LR Scheduler
Label
Decay
LR
Smooth-
ing
Training
100
256
0.875
3.1c-5
0.256
Cosine
LR
0.1
(warmup=5)
Finetuning (GP 75%)
120
256
0.875
le-5
0.0512
Cosine LR
0.1
Finetuming (GP + 75%)
0.875
0.0512
Cosine LR
0.1
Finetuming (GP 90%)
120
256
0.875
le-5
0.0256
Cosine LR
0.1
Finetuning (GP + MT 90%)
120
256
0.875
0
0.0512
Cosine LR
0.1
Setup for Table
Stage
Epochs
Batch- size
Initial
Optimizer
I.R
Training
300
100
0.0064
Adam
Finetuning (9,8)
300
64
0.5
80
Adam
Finctuning (9,7)
300
100
0.5
80
Adam
Finetuning (8,7)
300
100
0.55
80
Adam
17


Reviews:
[{"review_id": "yk3qGpNokH2", "reviewer": "Reviewer_YX1q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) My major concern is that the paper does not clearly explain differences between GP in this paper and traditional GP methods. Based on Equation (1), I think GP in this paper is the same as GP adopted in Han et al.(2015a). If GP in this paper can achieve SOTA, Han's method can also achieve SOTA. My suggestion is that the authors provide qualitative or quantitative comparison with previous GP methods.\n\n2) The proposed MT is not a sufficient improvement. First, GP+MT increases accuracy on MobileNet, while slightly decreasing accuracy on ResNet. Second, I am not sure whether MT is sensitive to selected threshold. \n\n3) Since CIFAR-10 is a tiny-scale dataset, I think experiments on CIFAR-10 cannot sufficiently validate that for WideResNet-22-8, GP+MT can further improve accuracy compared to GP with the same sparsity ratio.", "summary_of_the_paper": "The paper revisits a traditional model compression method, global magnitude pruning (GP), and shows that GP can achieves state-of-the-art. The paper further improves GP by introducing minimum threshold (MT). Experiments on ResNet show that GP+MT achieves better accuracy with the same sparsity ratio compared to GP.\n\n", "summary_of_the_review": "The paper does not clearly explain differences between GP in this paper and traditional GP methods. The proposed MT is not a sufficient improvement. So my rating is \"5: marginally below the acceptance threshold\"", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "GQ7wdtM1_9J", "reviewer": "Reviewer_MHY2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will go sequentially in the paper but will start with the strengths follow it up with the weaknesses: \n\nOverall, the writing quality is OK and is easy to follow. The information has been well put together for a new reader to understand without having to worry about the background. The authors should be appreciated for that.  \n\nStrengths:\n1) The fact that authors revisited GP to show its power is a very nice step. GP has been (according to me) the strongest baselines for pruning for a given parameter budget even now (has been acknowledged in multiple papers). \n2) The idea of MT makes sense and is a valid fix for the overly pruned layers in general.\n3) The experiments on ImageNet are commended because the insights from there a more valuable than other things presented in this paper. \n4) The discussion on network architectures, outputs and settings of MT are interesting in general. \n5) I really liked the related work, it is thorough from a literature review standpoint and would recommend new readers to go through it for a nice picture of the field \n\nWeaknesses:\nThis might sound a bit critical, but I would appreciate it if the authors understand the merit of whatever I am trying to put forward here. \n\n1) No one ever claimed GP is not sufficient to achieve SOTA performance on pruning. GP is extremely powerful no matter what gives the highest accuracy (often) for the same #params even at the highest sparsity levels. GP is not forgotten or overlooked by any means. The reason why people work on layer-wise pruning is slightly different and I will come to it later. \n2) While the authors mentioned people of LTH community probably used GP as one of the potential pruning techniques, GP was the only one that resulted in strong tickets at higher sparsities, making IMP explicitly work on GP no matter what other factors are. \n3) MT comes into the picture only when there is one-shot pruning and GP has been shown to work great even with gradual schedules (IMP is an example) and will not suffer from pruning out entire layers just because of the gradual processing. I would even argue setting the small pruning schedule is data-independent and has a similar cost as finding the optimal MT as done in this paper. MT makes sense for one-shot pruning, but one-shot pruning itself is problematic (will explain in a bit).\n4) Table 1 is a poor representation, I would say, STR and GMP are extremely easy to implement. GMP could be done in One-shot and GMP is data-independent no matter what. I would like to hear authors' thoughts on this. \n5) Coming to the setup, the biggest problem I have with the way GP or GPMT is being used is the application on top of a pre-trained network. Eg. On imagenet the fine-tuning of GP or GPMT takes 20 extra epochs, which isn't the case with all the baselines in the discussion. All of them are pruning while training baseline that only takes 100 epochs at best. I personally have adapted GP to be pruning while training like with IMP and GP does amazingly well, so the aspect of one-shot is overshadowed by the extra cost involved in fine-tuning. I don't know how we can make a case for these trade-offs to make the comparison fair.\n6) The big one, the paper completely disregards the compute cost during inference (FLOPs) for these pruned models. The biggest difference GP and layer-wise pruning methods have is the inference costs involved. Please see STR or RigL papers to see about that axis of comparison. GP often has 2x more inference FLOPs than uniform-sparsity with very little gain in accuracy. If uniform sparsity was adjusted for that compute costs, the accuracy will be much higher. This paper, while being a study on GP, completely overlooks this aspect of GP or GPMT.\n7) While the results on CIFAR-10 are good for debugging and understanding, the insights aren't particularly transferable. I would also not read too much into 40% sparsity results or even 95% sparsity because the networks involved are so heavily overparameterized that even at that sparsity level CIFAR-10 seems like a simple task. However, I would like to say Table 5 is a real thing for one-shot pruning, but will not happen for gradual GP. I would recommend authors implement gradual GP to see what it brings to the table, along with reduced training costs (debatable because one can think of pre-trained models as free). \n8) Figures 2 and 3 just reinforce what I am saying above. \n9) The gains in Tables 6 and 7 are not significant enough to make sweeping claims on SOTA while also not measuring FLOPs.\n10) The use of \"huge margin\" is highly discouraged in Table 8 because the gains are very insignificant and trust me the FLOPs are going to be so exorbitantly high that the accuracy vs # params doesn't matter in that case. \n11) As mentioned in the paper, MT doesn't seem to bring anything to the table for ImageNet experiments, just because of how the weight norms across layers change. MT is probably not a valuable addition in this case. \n12) I would not get into the discussion of output preservation because, it doesn't necessarily mean that the model is doing well if its outputs are closer to some other model, the yardstick itself is flaky here. \n13) Lastly, the discussion of MT tries to make it feel like it is not complex or data-dependent. But the same paper says GMP schedule is complex. I would say searching for MT is as good as coming up with a cubic pruning pattern which will even help GP when done in a gradual fashion.\n\nOverall, I do not think the paper brings any new observations into the limelight and also ignores the issues present with whatever has been presented. The paper is not ready for publication, but I am happy to chat with the authors to help them understand and (myself) gain perspective if I am missing something. \n\n", "summary_of_the_paper": "This paper revisits Global Pruning (GP) and makes a case to consider it seriously as part of the pruning literature. The authors also present an addition to GP, that claims, to further increase accuracy and reliability called Minimum Threshold (MT). MT is the constraint put on every layer of the neural network to ensure they are not pruned beyond a certain limit which might result in catastrophic accuracy drops. \n\nThe paper presents experiments on CIFAR-10 with a few CNN architectures along with current pruning baselines (layer-wise mostly. They also show results on ImageNet using ResNet50 and MovileNetV1. Following these experiments, the authors claim that GP or GPMT are SOTA for pruning despite their simplicity. \n\nThe paper also includes some discussion on how output patterns of each layer change with different pruning schemes and a note about how to set the optimal MT values. \n\nWhile I agree with the sentiment of the paper, I do not think the paper is presenting anything new (even from a benchmarking perspective) but rather is missing the point about what makes GP vs layer-wise pruning a worthy trade-off often. I will list my concerns in the main review.", "summary_of_the_review": "The paper does not propose anything novel (which is fine) but the observations are also not new while the paper ignores most of the issues and tries to underplay other baselines. \n\nThe paper is not ready to be published but is an excellent starting point for new readers. \n\n\n=---------------------------------------------------------------------------------------\n\nAfter an extremely long discussion with the authors and providing them with things I felt were necessary for fixing the experiments, claims etc., I think the paper is not ready to be published. The authors should revisit most aspects of the paper if they want to make it a benchmark paper for GP and claim so. None of the insights and contributions are novel given my discussion and experience. I  hope the authors understand that it is better to publish a ready paper than a paper changing around a lot at this point. \n\nI vote for rejection.\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-VmicmQ5F25", "reviewer": "Reviewer_5P4J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Layer-wise sparsity ratios used to be tricky hyper-parameters. A global threshold and a minimum number of preserved weights eliminate layer-wise sparsity ratios. In this context, the proposed method is practical. \n\n* Although the empirical results seem promising despite of the simplicity. it is not straightforward to understand why we should use global threshold across layers and why MT is crucial to ensure superior performance for certain models. For example, how does over-pruning impact the performance? \n\n* A related study [1] argues that layer-wise sparsity is of importance. A comparison (both theoretical and empirical) should make the result stronger. \n\n\n[1] Lee, Jaeho, et al. \"Layer-adaptive Sparsity for the Magnitude-based Pruning.\" International Conference on Learning Representations. 2020. ", "summary_of_the_paper": "A very simple and effective pruning method is proposed. Instead of layer-wise pruning, a global threshold is used to prune weights according to their magnitude. In addition, to avoid over-pruning, a minimum number of parameters is preserved for each layer after pruning. Experiments on CIFAR-10 and ImageNet validate the effectiveness of proposed method. ", "summary_of_the_review": "The simplicity and effectiveness of proposed method are important and interesting. The findings will inspire more studies on the nature of network pruning. However, it is not easy for readers to understand where the effectiveness of proposed method comes. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper claims that one of the most common (and obvious) pruning methods in the literature today (global magnitude pruning) is "overlooked" and "seen as a mediocre baseline by the community." As an active member of the pruning research community myself, I can attest that this is simply not true. I am in strong agreement with reviewer MHY2 and - after reading the discussion around that review and the paper itself in detail - I confidently recommend rejection.

Magnitude pruning itself dates back decades, at least to the work of Janowski (Pruning vs. Clipping in Neural Networks, 1988). The paper is correct that *global* magnitude pruning (in which all weights are compared in a layer-agnostic manner) was largely ignored in favor of layer-wise magnitude pruning (i.e., pruning all layers by the same amount) in much of the work that popularized magnitude pruning (e.g., Han et al., 2015). However, global magnitude pruning has become much more popular since that time. In work establishing the lottery ticket hypothesis, Frankle and Carbin (The Lottery Ticket Hypothesis) use it in certain cases and - later - in all cases (Frankle et al., Linear Mode Connectivity and the Lottery Ticket Hypothesis). In the past several years, global pruning in general has become the de facto way to use all new pruning heuristics (e.g., SNIP: Single-Shot Network Pruning based on Connection Sensitivity; Picking Winning Tickets Before Training by Preserving Gradient Flow; Pruning Neural Networks without Any Data by Iteratively Conserving Synaptic Flow). Moreover, other papers have specifically advocated that global magnitude pruning is state-of-the-art within recent years at this very conference: Comparing Rewinding and Fine-Tuning in Neural Network Pruning (Renda et al., ICLR 2020 oral): "We propose a pruning algorithm...that matches state-of-the-art tradeoffs between Accuracy and Parameter-Efficiency across networks and datasets:...globally prune the 20% of weights with the lowest magnitudes." (This paper does not cite Renda et al. despite the fact that it is a prominent paper that directly contradicts the purported problem that the paper relies on to support the significance of the findings.)

In short, in the pruning literature, the idea that global pruning, magnitude pruning, or global magnitude pruning is overlooked or is not recognized as a strong baseline is simply preposterous. The reason that global magnitude pruning has "largely been ignored in recent years, generally being relegated to the position of a baseline for comparison" is because it is a simple technique whose efficacy has long been known and established - exactly what a good "baseline for comparison" should be.

The paper has narrowed its claims somewhat during the discussion and revision period, advocating for a one-shot global magnitude pruning strategy that "does not require any complex pruning frameworks like RL or sparsification schedules [or]...iterative procedure." To do so, however, the proposed method replaces each of these "complex" hyperparameters with another set: whether or not to use a minimum threshold (MT) and where to set it. Even if the approach isn't iterative, the hyperparameter search necessary to set it almost certainly is, and it is unclear whether searching for the MT value is any more efficient than the other approaches. The costs of this hyperparameter search need to be measured. And iterative pruning's costs can often be mitigated by making pruning gradual, something the paper considers superficially in the revisions.

Finally, as reviewer MHY2 observes, one of the primary reason papers *don't* use global magnitude pruning is that, although it leads to higher sparsities than layerwise magnitude pruning, it also often leads to higher FLOP counts. Although FLOP counts are a terrible indicator of real-world speedup, they are a much higher-fidelity indicator than parameter-count, which neglects the fact that - in convolutional networks - a small number of parameters can lead to vastly more FLOPs if they operate on larger activation maps (i.e., before the activation maps have been downsampled). In the revisions, the paper gives a token nod (and a superficial dismissal) to this fact in Sections 4.3 and 6, but the paper needs to fully acknowledge this point by measuring and discussing its consequences. "Look[ing] at this in future work" is not enough.

Due to these many concerns, I strongly recommend rejection.
Decision: Reject