Layer-adaptive Sparsity for the Magnitude-based Pruning
Authors: Jaeho Lee
Sejun Park
Sangwoo Mo
Sungsoo Ahn
Jinwoo Shin
Venue: ICLR 2021 Poster (2021)
Keywords: network pruning
layerwise sparsity
magnitude-based pruning

Abstract: Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on ``how to choose,'' the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation.
Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection.
Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity

Published as a conference paper at ICLR 2021
LAYER-ADAPTIVE SPARSITY FOR THE MAGNITUDE-BASH PRUNING
Jacho Lee
Sejun Park *AAIST AI
Sangwoo Mo *MBZUAI
Sungsoo Ahn Jinwoo Shin
KKAIST EE
(jaeho-Lee, se jun. park, swmo, jinwoos tekaist peter ahnsmozuai
ABSTRACT
Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, simple magnitude-basse pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on "how to choose." the layerwise sparsities are mostly selected algorithm-by algorithm, often resorting to handcrafted heuristics an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global prun- ing, coined layer-adaptive magnitude-based pruning (LAMPP) score; the score is a rescaled version of weight magnitude that incorporates the model-level l2 distortion incurred by pruning, and does not require any hyperparameter tuning heavy computation. Under various image classification setups, LAMP consistently outper- forms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriente layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-base pruning in this case. Code: http://wwwwooooccoooooooo/aaaaaa
INTRODUCTION
Neural network pruning is an art of removing "unimportant weights" from a model, with an intention to meet practical constraints (Han et al., 2015), mitigate overfitting (Hanson & Pratt, 1988), enhance interpretability (Mozer & Smolensky, 1988), deepen understanding (Frankle & Carbin, 2019). Yet, the importance of weight is still a vaguely defined notion, and thus a wide range of pruning algorithms based various importance scores has been proposed. One popular approach is to estimate the loss increment from removing the target weight to use as an importance score, e.g., Hessian-based approximations (LeCun et al., 1989; Hassibi & Stork, 1993; Dong et al., 2017), coreset-basee based estimates (Baykal et al., 2019; Mussay et al., 2020), convex optimization (Aghasi al., 2017), and operator distortion (Park et al., 2020). Other approaches include on- I regularization (Louizos et al., 2018; Xiao et al., 2019). Bayesian methods (Molchanov et al., 2017; Louizos et al., 2017; Dai et al., 2018), and reinforcement learning (Lin et al., 2017).
neural network training
Recent discoveries (Gale et al., 2019; Evci et al., 2020) demonstrate that, given appropriate choice of layerwise sparsity, simply pruning on the basis of weight magnitude yields a surprisingly powerful unstructured pruning scheme. For instance, Gale et al. (2019) evaluates the performance of magnitude- based pruning (MP; Han et al. (2015); Zhu & Gupta (2018)) with an extensive hyperparameter tuning, and shows that MP achieves comparable or better performance than state-of-the-as pruning algorithms that use more complicated importance scores. To arrive at such performance level, the authors introduce the following handcrafted heuristic: Leave the first convolutional layer fully dense, and prune up to only 80% of weights from the last fully-connected layer; the heuristic motivated by the sparsity pattern from other state-of-the-an a algorithms (Molchanov al., 2017) and additional experimennta//arciiccttt
observations.
Unfortunately, there is an apparent lack of consensus on "how choose the layerwise sparsity" for the magnitude-based pruning. Instead, the layerwise sparsity is selected mostly on an algorithm-by- algorithm basis. One common method is the global MP criteria (see, e.g., Morcos et al. (2019)).
Work done at KAIST
iic.. simultancously training and pruning

---

Published as a conference paper at ICLR 2021
of
1
Vi
Mf vf Ø
42 V/2
17 7744
Mg V/3
0
of + up
+
18 18
(sorted) Weight Magnitudes
LAMP score
Global Pooling and Pruning
The LAMP score is a squared weight magnitude, normalized by the sum of all "surviving weights" in the layer. Global pruning by LAMP is equivalent to the layerwise magnitude- -based pruning with an automatically chosen layerwise sparsity.
Figure 1:
where the layerwise sparsity is automatically determined by using a single global threshold on weight magnitude. Lin al. (2020) propose magnitude-based pruning algorithm using feedback signal, using a heuristic rule of keeping the last fully connected layer dense. A recent work by Evci et al. (2020) proposes magnitude-baut based dynamic sparse training method, adopting layerwise sparsity inspired from the network science approach toward neural network pruning (Mocanu et al., 2018).
Contributions. In search of a "go-to" layerwise sparsity for MP, we take a model-level distortion minimization perspective towards MP. We build on the observation of Dong et al. (2017); Park al. (2020) that each neural network layer can be viewed as an operator, and MP is choice that incurs minimum l distortion to the operator output (given worst-case input signal). We bring the perspective further to examine the "model-level" distortion incurred by pruning a layer; preceding layers scale the input signal to the target layer, and succeeding layers scale the output distortion.
Based on the distortion minimization framework, we propose a novel importance score for global pruming, coined LAMP Addative Magnitude-based Pruning). The LAMP score is rescaled weight magnitude, approximating the model-level distortion from pruning. Importantly, the LAMP score is designed approximate the distortion the model being pruned, i.e., all connections with a smaller LAMP score than the target weight is already pruned. Global pruning" with the LAMP score is equivalent to the MP with an automatically determined layerwise sparsity. At the same time, pruning with LAMP keeps the benefits of MP intact; the LAMP score is efficiently computable, hyperparameter- free, and does not rely on any model-specific knowledge.
We validate the effectiveness of LAMP under diverse experimental setup, encompassing various convolutional neural network architectures (VGG-16. ResNet-18/344 DenseNet- 121, EfficientNet-BO) and various image datasets (CIFAR- 10/100, SVHN, Restricted IetgeNet). In all considered setups, LAMP consistently outperforms the baseline layerwise sparsity selection schemes. We also perform additional ablation studies with one-shot pruning and weight-rewinding setup to confirm that LAMP performs reliably well under a wider range of scenarios.
Organization. In Section 2, we briefly describe existing methods to choose the layerwise sparsity for magnitude-based pruning. In Section 3, we formally introduce LAMP and describe how the lz distortion minimization perspective motivates the LAMP score. In Section 4, empirically validate the effectiveness and versatility of LAMP. In Section 5, we take a closer look at the layerwise sparsity discovered by LAMP and compare with baseline methods and previously proposed handcrafted heuristics. In Section 6, we summarize our findings and discuss future directions. Appendices include the experimental details (Appendix A), complexity analysis (Appendix B), derivation of the LAMP score (Appendix C), additional experiments on Transformer (Appendix D), and detailed experimental results with standard deviations (Appendix E).
2 RELATED WORK
This section gives (necessarily non-exhaustive) survey of various layerwise sparsity selection schemes used for magnitude-bast pruning algorithms. Magnitude-basee pruning of dates back to the early works of Janowsky (1989); LeCun al. (1989), and has been actively studied
neural networks
i.c. using global threshold for LAMP score
2

---

Published as a conference paper at ICLR 2021
again under the context of model compression since the work of Han et al. (2015). In Han et al. (2015),
the authors propose an iterative pruning scheme where the layerwise pruning threshold is determined heuristic. Zhu & Gupta (2018) propose a uniform pruning algorithm with a carefully tuned gradual pruning schedule combined with weight re- re-growths. Gale et al. (2019)
by the sanddarddeviationn
refine the algorithm by adding heuristic constraint of keeping the first convolutional layer fully dense and keeping at least 20% of the weights surviving in the last fully-connected layer.
MP has also been widely used in the context of "pruning at initialization." " Frankle & Carbin (2019) combine MP with weight rewinding to discover efficiently trainable subnetworks: for small nets, the authors employ uniform layerwise sparsity, but use different rates for convolutional layers and connecconnected layers (with an added heuristic on the last fully-connected layer); for larger nets, authors use global MP. Morcos et al. (2019) consider transferring the "winning ticket" initializations, using the global MP. Evei et al. (2020) proposes a training scheme for sparsely initialized neural networks, where the layerwise sparsity is given by the Erdös-Rény kernel method; the method generalizes the scheme initially proposed by Mocanu et al. (2018) to convolutional neural networks.
We note that there is a line of results on the trainable layerwise sparsity; we refer the interested readers to the recent work of Kusupati et al. (2020) for concise survey. However, we do not make direct comparisons to these methods, as our primary purpose is to deliver an easy-to-use layerwise sparsity selection scheme without requiring the modification of training objective, or extensive hyperparameter tuning.
We also note that we focus the unstructured sparsity. While such unstructured pruning techniques have been considered less practical (compared to structured pruning). several recent breakthroughs provide promising methods to bridge this gap; see Gale et al. (2020); Elsen et al. (2020).
LAYER-ADAPTIV MAGNITUDE-BASER PRUNING (LAMP)
We now formally introduce the Layer-Adaptive Magnitude- Pruning (LAMP) score. Consider a depth-d feedforward neural network with weight tensors W ...., W7()) associated with each bully-comeecee//convlluue
layer. For fully-connected layers, corresponding weight tensors are two- dimensional matrices, and for 2d convolutional layers, corresponding tensors are four-dimensional. To give a unified definition of the LAMP score for both fully-connected and convolutional layers, we assume that each weight tensor unrolled (or flattened) to a one-dimensional vector. For each of these unrolled vectors, assume (without loss of generality) that the weights are sorted in an ascending order according to the given index map, i.c., |W [u] I [Wli] holds whenever u < v, where W [u] denote the entry of W mapped by the index u.3
The LAMP score for the u-th index of the weight tensor W is then defined as
(W(x))2
score(u; W) = Evzu (W(o)))
Informally, the LAMP score (Eq. 1) measures the relative importance of the target connection among all surviving connections belonging to the same layer, where the connections with a smaller weight magnitude (in the same layer) have already been pruned. As a consequence, two connections with identical weight magnitudes have different LAMP scores, depending on the index map being used.
Once the LAMP score is computed, we globally prune the connections with smallest LAMP scores until the desired global sparsity constraint is met; the procedure is equivalent to performing MP with an automatically selected layerwise sparsity. To see this, it suffices to check that
(W(u))² > (Wlo))² + score(u; (u; W) > score(u; W)
(2)
holds for any weight tensor W and a pair of indices u, v. From the definition of the LAMP score (Eq. 1), it is easy to see that the logical relation (2) holds. Indeed, for the connection with larger weight magnitude, the denominator of Eq. is smaller, while the numerator is larger.
We note that the global pruning with respect to the LAMP score is not identical to the global pruning with respect to the magnitude score [W[u [W]l (or (W [ul)?, cquivalently). Indeed, in each layer, there
TThi "order" among weights is required handle the weights with equal magnitude.
3

---

Published as a conference paper at ICLR 2021
exists exactly one connection with the LAMP score of 1, which is the maximum LAMP score possible. In other words, LAMP keeps at least one surviving connection in each layer. The same does not hold for the global pruning with respect to the weight magnitude score.
We also note that the LAMP score is easy-to-use, Similar to the vanilla MP, the LAMP score does not have any hyperparameter to be tuned, and is easily implementable via elementary tensor operations. Furthermore, the LAMP score can be computed with only a minimal computational overhead; the sorting of squared weight magnitudes required to compute the denominator in Eq. is already a part of typical vanilla MP algorithms. For more discussions, see Appendix B.
3.1 DESIGN MOTIVATION: MINIMIZING OUTPUT l DISTORTION
The LAMP score (Eq. 1) is motivated by the following observation: The layerwise MP is the solution of the layerwise minimization of Frobenius distortion incurred by pruning. which can be viewed as relaxation of the output & distortion minimization with respect to the worst-case input. This observation leads us to the speculation "Reducing the pruning-incurred Ez distortion of the model output with respect to the worst-case output may be beneficial to the performance of the retrained model (and perhaps that is why MP works well in practice)." This speculation is not entirely new; the optimal brain damage (OBD; (LeCun et al., 1989)) is also designed around similar philosophy of loss minimization, without a complete understanding on how the benefit of loss minimization seems to pertain after retraining.
Nevertheless, we use this speculation as guideline to design LAMP as a natural extension of layerwise MP global pruning scheme with an automatically determined layerwise sparsity. To make arguments a bit more formal, consider depth-d fully-connected* neural net, whose output given the input æ is
fa; W(ld)) w(th (o.. ()(.))....)
(3)
where a denotes the ReLU activation and W; denotes the weight matrix for the i-th layer, and (1:d)) (W(1) ...., W( denotes the of weight matrices.
Viewing MP as a relaxed layerwise & distortion minimization. We first focus on a single fully- connected layer (instead of full model), and consider the problem of minimizing the pruning-incurred & distortion in the layer output, given the worst-case input signal. We then observe that the problem can be relaxed to the minimization of Frobenius distortion in the weight tensor, whose solution coincides with the layerwise MP. Formally, let & E IR" be an input vector to fully-comnected layer with the weight tensor W E Rmxn, We want prune the tensor to W M @ W, where M is an m x n binary matrix (i.e., having only Os and is as its entries) satisfying some predefined sparsity constraint M|o imposed by the operational constraints (e.g., model size requirements). We wish to find the pruning mask M that incurs the minimum l2 distortion in the output given the worst-case bo-bdddd input, i.c.
min Keary Mals
sup |[WE - (M @ W)ll22
(4)
l1lla11
The minimax distortion (4) upper-b the minimum expected l2 distortion for any distribution of & supported on the unit ball, and thus can be viewed as a data-oblivious version of the pruning algorithms designed for loss minimization (using squared loss). By the definition of the spectral norm," Eq. 4 is equivalent to
min M hrary LMllSS
I - WII..
(5)
where || denotes the spectral norm. Using the fact that ||A]| < (where || || denotes the Frobenius norm), the optimization (5) can be relaxed to the Frobenius
||A|| holds for any matrix A
1 'An extension to convolutional neural network is straightforward: see, e.g., (Sedghi et al., 2019).
5 "which is the operator norm with respect the & norms, i.c., supe ||
||
'The inequality is simple consequence of the Cauchy-Schwarl inequality: ||Az|| E, Agz,) Σ,C, A (, z || where subscripts denote weight indices.
4

---

Published as a conference paper at ICLR 2021
distortion minimization
min AT Merry SM
| - M WI =
min Mt Neary IMllss
Σ KCl.....
(1- My
(6)
JE
where Wij, Maj denote (ith entries of W, M, respectively. From the right-hand side of Eq. 6, we see that the layerwise MP, i.e., setting M,j for (i,j) pairs with top-k largest Waj, is the optimal choice to minimize the Frobenius distortion incurred by pruning. This observation motivates us to view the layerwise MP as the (approximate) solution of the output &g distortion minimization procedure, and speculate the connection between the small output distortion and the favorable performance of the prumed-retrainng subnetwork (given the unreasonable effectiveness of seemingly- na've MP as demonstrated by Gale al. (2019)).
LAMP: greedy, relaxed minimization of model output distortion. Building on this speculation,
now ask the following question: "How can we select the layerwise sparsity of MP have small model-level output distortion?" ?d To formalize, we consider the minimization
sup [x; W(ld))) - f(x; ((1:)) ||2
min 1 ||
where x: denotes the model-level sparsity constraint imposed by the operational requirements and W(O) = M( O (()) denotes the pruned version of the i-th layer weight matrix.
Due to the nonlinearities from the activation functions, it is difficult to solve Eq. 7 exactly. Instead, we consider the following greedy procedure: At each step, we (a) approximate the distortion incurred by pruning single connection, (b) remove the connection with the smallest score, and then (c) go back to step (a) and re-compute the scores based on the pruned model.
Once assume that only connection is pruned at single iteration of the step (a), we can the following upper bound of the model output distortion to relax the optimization (7): With W; denoting pruned version of W,, we have
W - W ||
sup [x; ((lld)) f(z; ((1-1),, W( w(++1d)) |
II
IF
[WOle
(8)
(see Appendix C for a derivation). Despite the sub-optimalities from the relaxation, considering the right-hand side of Eq. provides two favorable properties. First, the right-hand side free of any activation function, and is equivalent the layerwise MP. Second, the score be computed in advance, i.c., does not require re-computing after pruning each connection. In particular, the product term = || does not affect the pruning decision, and the denominator can be pre-computed with the cumulative sum (WV(0))())) [v] for each index for W This computational trick leads us the LAMP score (1).
4 EXPERIMENTS & ANALYSES
To empirically validate the effectiveness of the proposed method, we compare LAMP with following layerwise sparsity selection schemes for magnitude-base pruning:
Global. A global threshold on the weight magnitudes imposed on every layer meet the global sparsity constraint, and the layerwise sparsity is automatically determined according to this threshold; see, e.g., Morcos et al. (2019).
Uniform. Every layer is pruned to have identical layerwise sparsity levels, which is in turn equal to the global sparsity constraint; see, e.g., Zhu & Gupta (2018).
Uniform+. Same Uniform, but impose two additional constraints: (1) we keep the first convolutional layer unpruned, and (2) retain at least 20% connections alive in the last fully- connected layer; this heuristic rule is proposed by Gale al. (2019).
5

---

Published as a conference paper at ICLR 2021
Paak 90%
Peak
92%
Unpruned
80%
Tost Tost Agqunis)
80%
775%
88%
Glabal
Global
Unitorm
Unitorm
Uniorr
70%
Uniforme
84%
LAMP (Ours)
Erdos-Rányl kar.
Erdbs Fānyl kar. LAMP (Ours)
65%
10.0%
5.0%
2.0%
1.0%
0.5%
0.2%
20%
Fraction Survived Weights
Fradion of Survived Weights
VGG-16
(b) ResNet-20
96%
80%
80%
Linpruncd
88%
Imprunoo
85%
866%
80%
Teat
79%
882%
70%
Global
Unilorm
Global
Unilorm+
78%
Unitorme
65%
Erdbs-Ränyl kar.
Erdöe-Féryl ker.
LAMP (Ours)
LAMP (Ours)
60%
20%
10%
5%
2%
1%
50%
20%
10%
5%
2%
1%
Fraction of Survived Weights
Fraction Survived Weights
(c) DenseNet-121
(d) FfficientNet-BO
Figure 2: Sparsity-accuracy tradeoff curves VGG-16, ResNet-18, DenseNet- 121, and EfficientNet- BO. All models are iteratively pruned and retrained with CIFAR-10 dataset.
Erdös-Rényi kernel. An extension of Rénōi method (originally given by Mocanu et al. (2018)) accounting for convolutional layers, proposed by Evei et al. (2020). The numbers of n'' + +
nonzero parameters of sparse convolutional layers are scaled proportional where denotes the number of neurons at layer I, and wi, h' denotes the width and height of the Ith layer convolutional kernel.
n - w
As default setup, we perform five independent trials for each baseline method, where in each trial iterative pruning-and-retrainee
(Han et al., 2015): we prune 20% of the surviving weights each iteration. For the Restricted Imageeee experiment, we provide the result from four trials. For clear presentation, only report the average on the figures appearing the main text. Standard deviations for five-seed results in Section 4.1 will be given in Appendix E. Also, in Appendix D, we report additional experimental results for language modeling tasks (Penn Treebank and WT-2) on Transformers (Vaswani et al., 2017).
Summary of observations. From the experimental results (Figs. 4), we observe that LAMP con- sistently outperforms all other baselines, in terms of the pparsity-accurae tradeoff. The performance gap between LAMP and baseline methods seems more pronounced with modern network archi- tectures, e.g., EfficientNet-BO We also observe that LAMP performs well under weight rewinding setup, while the strongest baseline Fardös-Rény: kernel) seems to be sensitive to such rewinding.
4.1
MAIN RESULTS
Our main experimental results are on image classification models. We explore a diverse set of model architectures and datasets, with base setup of VGG-16 (Simonyan & Zisserman, 2015) trained on CIFAR-10 10 (Krizhevsky & Hinton, 2009) dataset. In particular, our experiments cover the following models and datasets.

---

Published as a conference paper at ICLR 2021
70%
54%
53%%
Unioorr
Unitm
647
Linkerr
Unifor
Unikatte
LAMP (Oum)
AMP (Dun)
LAVP (OuR)
Pradiin
th
0.0%
0.
10.7%
Prection Survind Winght
90%
1.0%
0.P%
10
6.0%
Pradion Sarwiee Weihrs
2-5
3.5
(a) SVHN
(b) CIFAR-100
(c) Restricted ImageNet (4 seeds)
Figure 3: Sparsity- accuracy tradeoff curves of pruned models trained for SVHN and CIFAR-100 (on VGG-16) and Restricted ImageNet (on ResNet-34).
Models. We consider four network architectures for image classification experiments: (1) VGG-16 (Simonyan& & Zisserman, 2015) adapted for CIFAR- 10 to have batch normalization layers and one fully-connected layer (as used in Liu al. (2019); Frankle Carbin (2019)): (2) ResNet-20/34 (He et al., 2016); (3) DenseNet-121 (Huang et al., 2017); (4) EfficientNet-Bo (Tan & Le, 2019). For all four models, we prune the weight tensors for fully-connected and convolutional units. Biases and batch normalization layers are kept unpruned.
Datasets. We consider the following datasets; CIFAR-10/100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), and Restricted ImageNet (Tsipras et al., 2019). All datasets except for Restricted ImageNet are used for training VGG-16; 16; Restricted ImageNet is used for training ResNet-34.
Other details. Detailed experimental setup is given in Appendix A.
In Fig. 2, we provide aparsity-accurace tradeoff curves for four different model architectures trained on CIFAR-10 10 dataset. The first observation is that LAMP achieves the best tradeoff; in all four models, LAMP consistently outperforms baseline methods. We also observe that Erdös-Réng kernel method also outperforms other baselines in VGG-16, ResNet-20, and EfficientNet-BBl BO, but fails to do so on DenseNet-121 Furthermore, the gap between LAMP and the Réngi kernel method seems widen the model architecture gets notable especially in EfficientNet-BO, where mobile inverted bottleneck convolutional layers replace traditional convolutional modules. In particular, LAMP achieves 38.1% 1% test accuracy when only 1.44% of all weights survive, while Erdōs-Rén Rényi kernel achieves 77.8%. Lastly, we observe that the heuristic of Gale et al. (2019) seems to provide an improvement over the Uniform MP.
complicated; the gap between two methods is
In Fig. 3, we present the tradeoff curves for three additional datasets: SVHN, CIFAR- 100, and Restricted ImageNet. Similar Fig. 2, we observe that LAMP outperforms all other baselines and Erdös-Rény: kernel remains to be the most competitive baseline.
4.2 ABLATIONS: ONE-SHOT PRUNING, WRIGHT REWINDING, AND SNIP
Modern magnitude-bast based pruning algorithms are often used in combination with customized pruming schedules (c.g., Zhu & Gupta (2018)) or weight rewinding (c.g., Frankle & Carbin (2019); Renda et al. (2020)). As sanity check that LAMP perform reliably well along with such techniques, we conduct following additional experiments.
One-shot pruning. As an extreme case of pruning schedule, we test the scheme where we only run a single training- prrniing--rnig------ pruning-retraining cycle, instead of iterating multiple cycles. We test the one-shot pruning on VGG-16 trained with CIFAR-10 10 dataset.
Weight rewinding. After pruning, we rewind the remaining weights to their values in the early epoch, in the "lottery ticket" experiments by Frankle & Carbin (2019). We perform iterative magnitude-based pruning (IMP) on VGG-16, using the warm-up step and the training schedule described in Frankle et al. (2020).
SNIP. As additional experiment, test whether LAMP can provide a general-purpose layerwise sparsity for pruning schemes other than MP. We test under the "pruning at initialization" setup with SNIP scores (Lee et al., 2019). Basclines similarly modified to the SNIP score. We Conv-6 model on CIFAR-10 10 dataset (see Frankle & Carbin (2019) for more details the model) with batch size of 128.
7

---

Published as a conference paper at ICLR 2021
25%
60%
50%
92%
500%
Teaf 422-1000
Untom
Unlors
(AMP (Dun)
MMP (oun)
74h.
LAMP (Oun)
19.0%
0.5%
50%
0.70
50%
27%
Fradional Surveed
Prucion Survieed Woights
Fradiono Surnved
(a) One-shot pruning
(b) Weight rewinding
(c) SNIP Score
Figure 4: Sparsity- tradeoff curves under one-shot pruning, weight rewinding, and the SNIP setup. One-shot pruning and the weight-rewindie experiments are done on VGG-16 16 trained on CIFAR- 10 dataset. SNIP experiment performed on Conv-6 trained on CIFAR-10.
Again, other experimental details are given at the Appendix A.
Results for all three experiments are given in Fig. 4. On one-shot pruning, we confirm that LAMP comfortably leads other baselines, as in the iterative pruning case. We note that the gap between one-shot pruning and iterative pruning is quite small for LAMP: when 1.15% of all prunable weights survive, iterative LAMP brings only 1. .09% accuracy gain over one-shot LAMP. By contrast, the gain from iteration for Uniform MP is 41.62% at the same sparsity level.
On the weight-rewinding experiment, observe that LAMP remains beneficial over the baseline methods. We also remark that the Global baseline tend to perform well in this case, even outper- forming the Erdös-Rényl Rényi kernel method in the low-sparsity regime. This phenomenon seems to be connected to the observation Zhou et al. (2019) that the initial weights and final weights of a model highly correlated; global MP may help preserving connections with larger initial magnitudes, which play important roles in terms of signal propagation at initialization (Lee et al., 2020).
On the SNIP experiment, observe that LAMP achieves similar performance the Global SNIP. Recalling that the SNIP score is designed for global pruning (Lee et al., 2019), such high performance LAMP unexpected. We suspect that this is because LAMP is also designed for "output distortion minimization," which shares similar spirit with the "gradient distortion minimization."
5 LAYERWISE SPARSITY: GLOBAL MP, REDÖS-RÉNTE KERNEL, AND LAMP
With the effectiveness of LAMP confirmed, we take closer look at the layerwise sparsity discovered by LAMP. We focus answering questions: Q1. Does layerwise sparsity distilled from LAMP behave similarly to the heuristics constructed from experiences, e.g. the one given by Gale et al. (2019)? Q2. Is there any other defining characteristic of L.MP sparsity patterns, which may help us guide the design of (sparse) network architectures?
In Fig. 5, we plot the layerwise survival rates and the number of nonzero weights discovered by itera- tively pruning VGG-16 (trained on CIFAR-10), by Global MP, Erdös-Rényi kemel, and LAMP. Lay- erwise survival rates are given for the global survival rates of (51.2%, 26.2%, 13.4%, 6.87%, 3.52%) (from lighter to darker). Number of nonzero weights are plotted for the pruned models with total (3.52%, 1.80% 0.92%, 0.47%. 0.24%) fraction of all weights surviving.
We observe that LAMP sparsities share similar tendency to sparsity levels given by the Erdös-Rémp kernel method. particular, both methods tend to keep the first and layer layers of the neural network relatively dense; this property is reminiscent of the handcrafted heuristic given in Gale et al. (2019): keep the first convolutional layer unpruned, and prune at most 80% from the last fully-connected layer. While Global MP also keeps large fraction of the last fully-connected layer unpruned, the first convolutional layer gets pruned quickly. LAMP sparsities differ from Erdös-Rén Rényi kernel sparsities in two senses.
Although LAMP demonstrates its tendency to keep the first and the last layers relatively unpruned, this tendency is softer. When 3.52% of weights survive, LAMP keeps 79% and 62% of weights unpruned from the first and last layers (respectively). while Erdös- Rénós-Rényiiee kernel does not prune any weight from those two layers.
8

---

Published as a conference paper at ICLR 2021
0.4
02
10 21 24
32
PC
14000
12000
30000
13000
3000
4000
10000
10000
Lajer Fritx
34
(a) Global
(b) Erdōs-Rééní kernel
(c) LAMP (Ours)
Figure 5: Layerwise statistics of VGG-16 iteratively pruned CIFAR-10. Top: Layerwise survival rate for models with (51.2%, 26.2%, 13.4%, 6.87%, 3.52%) weights surviving. Bottom: Number of nonzero weights for models with (3.52%, 1.80%, 0.92%, 0.47%, 0.24%) weights surviving.
LAMP tends keep the number of nonzero weights relatively uniform throughout the layers at extreme sparsity levels (indeed, the first observation can be understood as consequence of the second observation). In contrast, Erdös-Rénp Rényi kernel method keeps the relative ratio constant, regardless of the global sparsity level.
Following the second observation, we conjecture that having a similar number nonzero connections in each layer may be essential condition to guarantee maximal memory capacity (sce, e.g., Yun et al. (2019)), given a global sparsity constraint on the neural network. Theoretical investigations of the approximability by such sparse neural networks may be interesting future research direction, potentially leading to a more principled and robust pruning algorithms.
As additional remark, note that the layerwise sparsity discovered by LAMP behaves similarly to that of AMC (He et al., 2018), which uses a reinforcement learning agent to search over the space of all available layerwise sparsity. We provide further details in Appendix F.
CONCLUSION
In this paper, we investigate under-explored problem on the layerwise sparsity for magnitude- based pruning scheme. The proposed method, coined LAMP Slayer-Adaptiv Magnitude- based Pruning), motivated from the & distortion minimization perspective on magnitude-based pruning, and provides a consistent performance gain on wide range of models and datasets. Furthermore, LAMP performs reliably well when combined with one-shot pruning schedule weight rewinding, which makes it an attractive candidate as a "go-to" layerwise sparsity for the magnitude- based pruning. Taking deeper layerwise sparsities, observe that LAMP automatically recovers the handcrafted rules for the layerwise sparsity. Furthermore, we observe that LAMP tend to keep the number of nonzero weights relatively uniform throughout the layers, sparsity levels. We conjecture that such uniformity in the number of unpruned weights may be an essential condition for maximal expressive power of sparsified neural networks.
look
consider more extreme
ACKNOWLEDGMENTS
This work supported in part by Institute of Information & Communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0) 9-0000075, Artificial Intelligence Graduate School Program (KAIST)) in part by Samsung Advanced Institute of Technol- ogy (SAIT), and in part by the Defense Challengeable Future Technology Program the Agency for Defense Development, Republic of Korea.

---

Published as a conference paper at ICLR 2021
REFERENCES
A. Aghasi, A. Abdi, N. Nguyen, and J. Romberg. Net-trim: Convex pruning of deep neural networks with performance guarantee. In Advances Neural Information Processing Systems, 2017.
C. Baykal, L. Liebenwein, I. Gilitschenski, D. Feldman, and D. Rus. Data-dependent coresets for compressing neural networks with applications to generalization bounds. In International Conference on Learning Representations, 2019.
B. Dai, C. Zhu, B. Guo, and D. Wipf. Compressing neural networks using the variational information bottleneck. In Proceedings of International Conference on Machine Learning, 2018.
X. Dong, S. Chen, and S. J. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 2017.
E. Elsen, M. Dukhan, T. Gale, and K. Simonyan. Fast sparse ConvNets. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In Proceedings of International Conference on Machine Learning, 2020.
J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of International Conference Machine Learning, 2020.
T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. In ICML ODML- CDNNR Workshop, 2019.
T. Gale, M. Zaharia, C. Young, and E. Elsen. Sparse GPU kernels for deep learning. Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, 2020.
S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, 2015.
S. J. Hanson and L. Y. Pratt. Comparing biases for minimal network construction with back- propagation. In Advances in Neural Information Processing Systems, 1988.
B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, 1993.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Y. He, J. Lin, H. Wang, L.JJ. Li, and S. Han. AMC: AutoML for model compression and acceleration mobile devices. In European Conference on Computer Vision, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
S. A. Janowsky. Pruning versus clipping in neural networks. Physical Review A, 1989.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
A. Kusupati, V. Ramanujan, R. Somani, M. Wortsman, P. Jain, S. Kakade, and A. Farhadi. Soft thresh- old weight reparametrization for learnable sparsity. In Proceedings of International Conference on Machine Learning, 2020.
Y. LeCun, S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989.
N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2021
N. Lee, T. Ajanthan, S. Gould, and P. H. S. Tomr A signal propagation perspective for pruning neural networks at initialization. In International Conference on Learning Representations, 2020.
J. Lin, Y. Rao, J. Lu, and J. Zhou. Runtime neural pruning. In Advances in Neural Information Processing Systems, 2017.
Lin, S. U. Stich, Barba, D. Dmitriev, and M. Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
Z. Liu, M. Sun. T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization, In International Conference on Learning Representations, 2019.
C. Louizos, K. Ullrich, and M. Welling, Bayesian compression for deep learning. In Advances in Neural Information Processing Systems, 2017.
C. Louizos, M. Welling, and D. P. Kingma. Learning sparse neural networks through lo regularization. In International Conference on Learning Representations, 2018.
M. Marcus, B. Santorini, and M. Marcinkiewicz. Building larger annotated corpus of English: The Penn treebank. Computational Linguistics, 1993.
S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2016.
D. C. Mocanu, E. Mocanu, Stone, H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature Communications, 2018.
D. Molchanov, A. Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of International Conference Machine Learning, 2017.
A. S. Morcos, H. Yi, M. Paganini, and Y. Tian. One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Information Processing Systems, 2019.
M. Mozer and Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. In Advances in Neural Information Processing Systems, 1988.
B. Mussay, M. Osadchy, V. Braverman, S. Zhou, and D. Feldman. Data-indenpendent neural pruming coresets. In International Conference on Learning Representations, 2020.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Deep Learning and Unsupervised Feature Learning Workshop, 2011.
B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In Conference Learning Theory, 2015.
S. Park, J. Lee, S. Mo, and J. Shin. Lookahead: A far-sighted alternative of magnitude-based pruning.
In International Conference on Learning Representations, 2020.
A. Renda, J. Frankle, and M. Carbin. Comparing fine-tuning and rewinding in neural network pruning. In International Conference on Learning Representations, 2020.
V. Sanh, T. Wolf, and A. M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. arXiv preprint
2020.
H. Sedghi, V. Gupta, and P. M. Long. The singular values of convolutional layers. In International Conference on Learning Representations, 2019.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
In International Conference on Learning Representations, 2015.

---

Published as a conference paper at ICLR 2021
M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, In Proceedings of International Conference on Machine Learning, 2019.
D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2019.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t... Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
X. Xiao, Z. Wang, and S. Rajasekaran. AutoPrune: Automatic network pruning by regularizing auxiliary parameters. In Advances in Neural Information Processing Systems, 2019.
C. Yun, S. Sra, and A. Jadbabaie, Small ReLU networks are powerful memorizens: a tight analysis of memorization capacity. In Advances in Neural Information Processing Systems, 2019.
H. Zhou, Lan, Liu, and J. Yosinki. Deconstructing lottery tickets: Zeros, signs, and supermasks.
In Advances in Neural Information Processing Systems, 2019.
M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In ICLR Workshop track, 2018.
12

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL SETUPS
For any implementational details not given this section, we refer to the code at:
https: http://www......oooooo//aaaaaaaa
Optimizer. With an exception of the weight rewinding experiment, we use A (Loshchilov Hutter, 2019) with learning rate 0.0003: use vanilla Adam with learning rate 0.0003 for the weight rewinding experiment, following the setup of Frankle & Carbin (2019). For other hyperparameters, we follow the PyTorch default setup: 3 = (0.9, 0.999), wd = 0.01, = 10-8.
Pre-processing. CIFAR-10/100 dataset is augmented with random crops with padding of 4 and random horizontal flips. We normalize both training and test datasets with constants
(0.4914, 0.4822, 0.4465), (0.237,0.3 0.223,.0..2..
SVHN training dataset augmented with random crops with a padding 2. normalize both training and test datasets with constants
(0.4377, 0.4438, 0.4728), (0. 198, 0.201, 0.197).
Restricted ImageNet training dataset augmented with Random resized crop to size 224 and random horizontal flips. Restricted ImageNet test dataset is resized to 256 and then center-cropped to size 224. We normalize both training and test datasets with constants
(0.485, 0.456, 0.406), (0.229, 0.224, 0.225).
Models. Some of the models used for CIFAR-10 10 datasets are originally developed for ImageNet: VGG-16, DenseNet-121, EfficientNet-BO The models are adapted for CIFAR-IO/II datasets by modifying only the average pooling and the (first) fully-connected layer (and convolutional layers) to fit the 32 x 32 resolution of the input image.
Table 1: Optimization details.
Dataset
Model
Initial training iter.
Re-training iter.
Batch size
SVHN
VGG-16
40000
30000
100
CIFAR-10
(VGG-16, EfficientNet-BN
50000
40000
100
CIFAR-10
DenseNett-121
80000
60000
CIFAR-100
VGG-16
60000
50000
100
Restricted ImageNet
ResNet-34
80000
80000
128
CIFAR-10 10
Conv-6 (SNIP)
50000
40000
128
13

---

Published as a conference paper at ICLR 2021
B COMPUTATIONAL AND IMPLEMENTATIONAL ASPECTS OF LAMP
In this section, we discuss the computational complexity of LAMP and describe how LAMP can be implemented. We consider depth-d neural network, with n, number of connections in i-th layer. We also let n >i=1 Ti:-
The global MP is comprised of two steps:
Step 1. Pool and sort the weight magnitudes of all connections, taking On log n) operations.
Step 2. Assign Os to connections with smallest weights until the desired sparsity level is achieved, taking (n)) computations.
The total computational cost is of order (n n). Similarly, we see that performing MP with any pre-determined sparsity levels incurs O n, log n,) computations.
LAMP, on the other hand, can done in four steps.
Step 1. Weight magnitudes are squared and sorted for each layer, which can be done with computation cost of O n, log vı,).
Step 2. LAMP score denominators E.en. [[] for each layer is computed by summing-ande
storing the squared weight magniuudes according to descending order; takes (n)) computations.
Step 3. The LAMP score is computed by dividing squared weight magnitudes by the denominators,
using (n) steps.
Step 4. We sort and prune in global MP, taking Onn log n) steps.?
We observe that step 4 is the dominant term, which is shared by the global MP. The first three steps can be easily implemented in Py as follows.
def Samp_score (we L
normelizer veim)t.norm.) 2
weight sarted_idx weight. abs lo..aabsss...................... vview (-1) .cort (deacend)
neightt_cumsum__t = (sortad waight ** 2) cumsum (dim=0)
weight_cumsum torch.zeros (weight_ www.gutttuuu..mmmmmom temp. shape) weightt_cunsum1:: weight__umsum. ilen weeightt_emeunnnee
sorted_weight /= (nermalizer neight_aquaro. (etrram_mmmmmmmmmm .agrt
score Lozch, zeros (veight__unsuut) cuaeum.ahapee score [sorted_idd idx sorted_weight
score. view tweight. shape!
Thhis cost can he further reduced, recalling that the LAMP scores in each layer are already sorted. All that remains is merging step.
14

---

Published as a conference paper at ICLR 2021
C DERIVATION OF INEQUALITY (8)
In this section, we prove the following inequality.
sup (x; ((ll))) fa; W(1:-1),, W( w(+ s || W |
II j+i
This inequality is simplified and modified version of what is popularly known as "peeling" procedure in the generalization literature (e.g., (Neyshabur et al., 2015)), and we present the proof only for the sake of completeness. We begin by peeling the outermost layer as
(1; (f; W(.d)) f(x; W(x-1) W(, W(++1d))]l
(10)
=| (o([,; os((;; W(1--)) Wi,, w(b+ldd1)))
(11)
< |F [f w( off (( wi,, w
(12)
< || [(r, f(x; w(1--1), W(, w
(13)
where we have used Cauchy-Schwand inequality for the first inequality, and the 1-Lipschitzness of ReLU activation with respect to le norm. We keep on peeling until we get
ll; ((lld)) f(x; ((ii-)) 1), W( w(++ llle $ (II [[ (z; w(x))) - f(;; ( 1l) JS1
(14)
(15)
The second multiplicative term on the right- right-hand side requires slightly different treatment. Via Ctuchy-Schwarz we get
[If(x; ((1:)) - fa; ((1--11) w = - w -
of W(
(16)
S <|IW(0 1/'0 w(1:-1)) ))122
(17)
The activation functions from this point require zero-in-zero-out (ZIZO; o (0) 0) property to be peeled, in addition the Lipschitzness. Indeed, we can proceed as
[] ( 1))la = [lo(f(x; (())))))) - (0)]l2 S [x; 1)) - 0|1 (x; ((1:-1))
(18)
(19)
Iteratively applying the Cauchy-Schwarl inequality and the step Eq. (19), we arrive at Eq. (9).
15

---

Published as a conference paper at ICLR 2021
D EXPERIMENTAL RESULTS ON LANGUAGE MODELING
In this section, we apply the proposed LAMP and baseline layerwise sparsities on language modeling tasks. We note, however, that it is still unclear whether magnitude-basee pruning approaches achieve state-of-the-art results for pruning natural language processing tasks (see, e.g., Sanh et al. (2020), for more discussions). In particular, NLP model architectures (e.g,, embedding layers) and learning pipelines (e.g., more prevalent use of pre- vre-trained models) add more complications to model pruning, obscuring how pruning algorithms should be fairly compared. We also note that we do not test Uniform+ baseline for this task, as the heuristic is proposed explicitly for image classification models, not language models (Gale et al., 2019).
Model. We consider the simplified version of Transformer-Xi (Dai et al., 2018) architecture, which has six self-attention layers (original has 16) and removed some regularizing heuristics, such as exponential moving average and cosine annealing. Similar to the setting of Sanh et al. (2020), we focus on pruning self-attention layers of the model (approximately 7.57M parameters).
Optimizer. We use AdamW (Loshchilov & Hutter, 2019) with learning rate 0.0003, following the PyTorch default setup for other hyperparameters: 3 = (0.9, 0.999), wd 0.01, 10-- We use batch size 20 and maximum sequence length 70. We train the initial unpruned model for 200,000 iterations and re-train the model after pruning for 50,000 iterations.
Datasets. We use Penn Treebank (Marcus et al., 1993) and WikiText-2 (Merity et al., 2016) datasets.
We provide the experimental results in Tables 2 and We observe that LAMP consistently achieves the near-best performance among all considered methods. We note, however, that the gain is marginal. We suspect that this marginal-ity is due to the fact that widths of the Transformer layers are relatively uniform compared to image classification models. We do not observe any notable pattern among the baseline methods.
Table 2: Test perplexities of Transformer-X. iteratively pruned and trained on Penn Treebank. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Surv. weights
26.2%
21.0%
16,8%
13.4%
10.8%
8.59%
6.87%
5.50%
4.40%
3.52%
Global
71.78=034
73.23:003
73.67
76.23
78.21-031
83.98
92. 30.004
Uniform
=1.57:044
71.74=039
73.27aus
74.18
76.98=241
79.25
81.79
84.95
$9.41
92.44ause
Erdös- Rényi ket.
71.53-033
71.94:000
73,564001
74.436920
77.15
79.38-505
82.03
85.01
89, $9,31
92.75
LAMP (Ours)
70.5
71.79
73.11.00
73.74.031
76.23
18127
10.73
83.60
88.06
91.24
Table 3: Test perplexities of Transformer-Xi XL iteratively pruned and trained on WT-2. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Surv. weights
26.2%
21.0%
16,8%
13.1%
10.8%
8.59%
6.87%
5.50%
4,40%
3.52%
Global
87.62,335
90.43:05
a1
94
96.34.03
100.07..
107.95.e77
112.20,a3
117.03,0a
Uniform
87.88
90.461009
91.58ac2s
95.01.021
96.72
100.83
15
108.31
112.7laa2
117.01:000
Erdós Rényi ket.
88.01
90.62
91.99
95,40sa3
97.13sem
100.83
101.4
108.51
112.56:047
116.69:004
LAMP (Ours)
87.73
90.76
91.47ac33
94,58
96.10
99.62.2319
103.01
106.60-344
110.87:0a3
115.31:0M
16

---

Published as a conference paper at ICLR 2021
E DETAILED EXPERIMENTAL RESULTS (WITH STANDARD DEVIATIONS)
Table 4: Test accuracies of VGG-16 iteratively pruned and trained on CIFAR-10. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Surv, weights
6.87%
2.81%
L80%
1.15%
0.74%
0.47%
0.30%
0.19%
0.12%
Global
91 91.30s
90.30
89,28
85.55-234
81.56=133
54,5 =1.58
41.91
31.93=2124
21.87
11.72=229
Uniform
91.47e9u19
90. 78tuuz
88.61=um
81.17:444
55.68=/223
3S. S1
26.11
16.75
11. 11.58e20
9,95
Uniforme
91 14
91. 20s0c
90.16am12
89 89
87 25
86 S3amns
84, 84
82.41:004
74 41.44asa
24.46 1494
Frelä-Réngi
92.34,005
91.99
91.66.070
91.15
90.55gn
89.51g00
88.21
86.73,035
8485
81.90.04h
EAMP (Curr)
9M
92.061071
91.71.00
91.66.2331
91.07201
90.49.071
29.64
38.75.00
87.07
84.90_00
Table 5: Test accuracies of ResNet-20 iteratively pruned and trained on CIFAR-10. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Surv. weights
20.97%
13.42%
8.59%
5.50%
3.52%
2.25%
1.44%
0.92%
0,59%
0.38%
Global
87.48e9.2
86.97a039
86.294sus
85.02
83.15=a.
52
76, 76,28:003
70.69amc
47.47a11:
12.02e3.
Uniform
87.24
86.70.u3:
86.09
84.53:1044
82.05=::2
64.24:1430
47.97
20.45:1m
13.35:216
Uniform
87 37.30=aa2
87.00
86.274aus
85,00=a015
83.23-8.1
80.40ea21
76. 10:44
69.31
52.06-251
20.19:544
Erdös Rényi ker.
87.49
16.83
85.84
84.08
81.76,037
78.70.025
74.40
66.42:146
10.90.12s
LAMP (Ours)
87.54
87.12
16.561921
85.64
84.18
81,361025
78.63
74.20
67.01
:1.24:x
Table 6: Test accuracies of DenseNet- 121 iteratively pruned and trained on CIFAR-10. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Sure weights
5.50%
4,40%
3.52%
2,825%
2.25%
1,80%
1.44%
1.18%
0.92%
0.74%
Global
90.16.0
89. 89.52,a 1
88.8 13.8
88 g0 1
86.8 86.85anu7
85.32g015
27.688883
45, 45.30g3375
49.68 as
20,96./144
Uniform
90.21=611
10
88, 88.412009
87 a7
86 83:444
85.00=
82. 16
10.13
66.1621222
48.71=2142
Unilorm+
90.25:001
30
89.03-0077
87.40
86.26
84.555..45
81,87:877
60.25
58.91:136
Erdls Rényi ker.
90,21:029
89.79
88.92
88 28.20
87.25
86 86.22
84.10
$1.82:000
59.06-2541
59.07
LAMP (Ours)
90.89
90
89.72
89.12
88.39
87.75
86.53
85.13-531
a2.92
79.23
Table 7: Test accuracies of EfficientNet-Bo -O iteratively pruned and trained on CIFAR-10, 10. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Surv. weights
41.0%
26.2%
16.8%
10.7%
6,87%
4,40%
2.82%
1.80%
1.15%
0.74%
Global
19.66
89.55 3035
18.8
87.64
14.36
79 29:25
11.09
10.
10:00
10.00:a1e
Unifoem
$8.99:001
88.26
86.48:10ss
a3.40
23.65423.16
10.83::333
10.00
10.00=aue
10.00
10.00=a1e
Uniform
89. 19.18.0075
88.03.00
14
86,71
84. 16
36. 17
10.45.00
10:00..00
10.19_1.
10.001.0m
10.00.
Erdes- Rényi
89154.0.17
90.09
20.0
89.62
88.82
87
87.08:032
84.72
81.53.044
11.31
13.40:31M
LAMP (Ours)
89.52-0511
30
89.97
90.21
89.91
39.79
89.30.000
88.51 =837
86.79.075
65.76-23100
Table 8: Test accuracies of VGG-16 iteratively pruned and trained on SVHN. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Surv, weights
6.87%
4.40%
2,82%
1.00%
1.15%
0.74%
0.47%
0.30%
0.19%
0.12%
Global
95 75
95 46anin
24.91.00
61.066:3291
49.51
14
15.37-545
7.79
14,72g104
Uniform
95 78
05.521n00
95. am11
91.98.c11
64
15m
29.54,1131
19.63-751
13.731607
17.94_455
Uniform
95.70.015
42.611
95.05-211
94.57ams
93.87am00
93.07a31s
92.25.025
74
1436
Erdfs- Rényi ker.
96,00
95.89
95.59eaus
95.47=230
95.16suas
94.81ass
94,30=a31
93.74-017
93.48au27
93.06:025
LAMP (Ours)
95.98 15.15
95.87 201ss
95.74
95
95.35
95.16
94.71=331
94.24:011
94.05
93.97-01
Table 9: Test accuracies of VGG- 16 iteratively pruned and trained on CIFAR-100. Bold denotes the pruning scheme with accuracy within one standard deviation from the highest.
Surv. weights
13.42%
8.59%
5.50%
3.52%
2.25%
1.44%
0.92%
0.59%
0.38%
0.24%
Global
68.02..022
67:20..11
102s
65.68
63.25 10:37
60.18
13.52,100
33.66.11100
7,80
4.29
Uniform
67.92-005
67.45
67.4
66.41
63
63
57.62 62a20
48 18
32.
10
14.99-166
3.94
3.35
Uniforrm
67.94_03e
67.11.0
66.37 300
65.5 65.511991
63.431n
60.76
$7.9310
55.40.4.0
$1.9011 9
33
Erdis- Rényi
68.07
67.92.03
67.45.0mm
67.09
18.861n1s
14.15
62.04.077
60.07
58.58,n05
55 .5.8
LAMP (Ours)
68.07:5033
22
67.84
67.37 25
66.35
65.25 ec:s
63,46:00
62.37
60.77:002
58.05:142
17

---

Published as a conference paper at ICLR 2021
20.0%
Loyar Ardea
Lugger
(a) Global
Erdös Erdös-Rényi kernel
(c) LAMP (Ours)
Figure 6: Layerwise survival rates of one-shot pruned ResNet-50 trained on ImageNet dataset. The global survival rates are (80.0%, 41.0%, 21 31.0% (from lighter darker).
F PEAKS AND CRESTS OF LAMP SPARSITY
In this section, we compare the layerwise sparsity discovered by LAMP (and other non-uniform baselines) to the sparsity discovered by AMC (He et al., 2018), which uses reinforcement learning agent explicitly search for the optimal layerwise sparsity. In particular, "peaks and crests" phenomenon observed for AMC-ddscoveredd layerwise sparsity also takes place the layerwise sparsity induced by LAMP: He al. (2018) observe that the layerwise sparsity of ResNet- 50 (trained on ImageNet) and pruned by AMC exhibits what they call peaks and crests, i.e. the sparsity oscillates between high and low periodically; see Figure 3 of He et (2018). The authors speculate that such phenomenon takes place, because AMC automatically learns that 3 x 3 convolution has more redundancy than 1 X convolution and can be pruned more.
focus on whether the
To see if LAMP also automatically discovers such "peaks and crests," we prune the ImageNet- pretrained ResNet-50 model with one-shot LAMP, global MP, and Erdös-Rényi kernel. The layerwise survival ratios are reported in Fig. 6. From the figure, observe that LAMP also discovers different sparsity ratio for X 1 convolution and 3 x convolution layers; pruned more. Such pattern more noticeable in the later layers, where more weights pruned. In other words, LAMP discovers a layerwise sparsity similar to that of AMC, even without training a reinforcement learning agent. We note, however, slight discrepancy in the setting: AMC reports the sparsity from an iterative pruning setup, and the results in Fig. 6 are from a one-shot pruning setup.
convolutional filters are
In the model pruned with global MP, such pattern does not stand out. In the model pruned with Erdös-Rény: kernel method, peaks and crests are extremely evident, even more than those discovered by AMC.
18

---

Published as a conference paper at ICLR 2021
Peak
Unpruned
Appuracy
85%
Global
80%
Unitorm
Uniforme
Erdbe-Féénll ker.
LAMP (Ours)
10.0%
5.0%
2.0%
1.0%
0.5%
0.2%
Fraction of Survived Weights
Figure 7: Performance of LAMP and baselines on VGG-16 trained on CIFAR-10 dataset, using the training schedules from Liu et al. (2019).
G LAMP WITH AN EXTENSIVE TRAINING SCHEDULE
this section, we depart from the standardized training setup in the main text which uses Adam with weight decay Loshchilov & Hutter (2019), without any learning rate decay. Instead, we validate the effectiveness of LAMP using the optimizer and hyperparameters extensively tuned for the maximum performance of the model. In particular, VGG-16 on the CIFAR- dataset. The experimental result is reported Fig. 7. Similar to the experiments appearing in the main text, we observe that LAMP continues outperform match the performances of the baselines.
configurations from Liu et al. (2019) for training
19


Reviews:
[{"review_id": "lF5LHgMrP-v", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# After Rebuttal: Score Lowered from 7 to 6\n\n## Concerns Addressed\n\nI appreciate the effort the reviewers put into revising the paper to include the settings I suggested.\n\nI am generally pleased with the revisions the authors made to the paper (especially Section 3), and I appreciate their attention to these details.\n\n## Remaining Concern: Settings in the Main Body are Poorly-Tuned and May Overstate Results\n\nI am concerned by one aspect of Figure 2: the unpruned accuracies for VGG-16 and ResNet-20 are much lower than they should be. VGG-16 should get 94% accuracy on CIFAR-10 (vs. 91% in the plot), and ResNet-20 should get 92% accuracy (vs. 86% accuracy in the plot).  This is because the paper uses Adam to train all networks without any learning rate drops, whereas the typical learning rate schedules for VGG-16 and ResNet-20 use SGD with momentum with learning rate drops.\n\nThis important difference raises the concern of whether the results shown in the paper will translate into fully-tuned, large-scale settings. As evidence of this concern, Appendix G does show a fully-tuned VGG-16 getting standard accuracy. In this setting, LAMP is no better than global magnitude pruning until very extreme sparsities.\n\n**I have lowered my score on the basis that the results in Figure 2 may overstate the value of LAMP in well-tuned settings. I no longer have unequivocal confidence that LAMP is an improvement that should be adopted in general. I implore the authors to replace the experiments in Figure 2 with well-tuned versions of these networks that achieve SOTA accuracies.**\n\n## Overall: Score Lowered from 7 to 6\n\nI am less confident in the method's significance in well-tuned settings, and I can no longer unequivocally trust the empirical evaluation in the paper. I still support acceptance, but only tentatively.\n\n# Overall\n\nI think the technical results are excellent. The authors should be commended for making a productive contribution to the pruning literature. My biggest concerns are :\n(1) The mathematical derivations are unreadable and impossible for future researchers to build on, so the authors *must* expand this section and make it crystal clear if I am to continue to recommend acceptance (they should use the extra page for this)\n(2) The authors should include a network that isn't severely overparameterized for CIFAR-10. I recommend ResNet-20 or ResNet-56.\n(3) The authors need to verify that their statements about prior papers are correct. I have noted a few things that need to be addressed below. In many cases, prior work was unclear about how it handled global/layerwise pruning decisions, and the authors should mention when there is ambiguity.\n\nIn addition, other smaller (but important) changes I would like to see include:\n(4) The authors should ideally include results on all of ImageNet, since subsets of ImageNet tend to show very different results from the full task. However, I understand that this is very expensive and that not everyone has the resources to do so, and I understand if the authors are not in a position to accomplish this.\n(5) The authors should use weight-rewinding as described below in the manner of \"Linear Mode Connectivity and the Lottery Ticket Hypothesis\" (Frankle et al.). Conv-6 is not a setting whose results correspond to larger-scale settings.\n(6) I think the authors should revise the title to clarify that they are proposing a better way to select the layerwise rates for magnitude pruning. This paper makes a valuable contribution, and that should come through from the title so readers can determine this easily.\n\n# Score\n\nI have issued the paper a 7 on the assumption that the authors will make substantial revisions to Section 3 and take advantage of the extra page of space to do so. If they do not address my major concerns (and ideally my minor concerns), I will lower my score.\n\n# Questions\n\nWhy is \"minimizing the l2 distortion for the worst-case input signal\" a reasonable design choice to make? The authors assert that it is what they aim to do, but they never explain why this is a good idea. I see why this makes sense at a high level, but adding a couple of sentences to this effect would be valuable, and I would like to hear from the authors directly.\n\n\"Minimizing l2 distortion for the worst-case input signal is equivalent to minimizing the spectral norm distortion.\" Please justify this. It's not immediate. I assume that the norm on the right side of (4) is the spectral norm, but please clarify.\n\n\"The optimization (4) can be relaxed to a Frobenius distortion minimization.\" Please justify this. It is not immediate. In general, this math is dense and the paper goes through it too quickly, and I wasn't able to follow it.\n\nWhere does the sparsity constraint k come from? \n\nIf I understand \"MP: layerwise distortion minimization\" correctly (which I don't think I do), it's meant to say that pruning the lowest-magnitude weights is a way of minimizing the l2 distortion for the worst-case input signal for a single layer in isolation? If this is the statement, I'm concerned that there's a <= in the derivation, since this suggests these two are not necessarily equivalent. And I don't understand why k is in there if we're pruning to a specific target sparsity. In general, I'm quite confused by (1) what you're trying to show here and (2) the actual details of this derivation.\n\nThe section on the LAMP score is completely uninformative. You need to include more of the derivation in the main body of the paper, especially explaining why it's appropriate to use the relaxation from the generalization theory literature (that again includes a <= suggesting that the two sides are not equivalent), what these quantities mean, what a \"damage score\" is, and how this leads to the LAMP scores. Right now, this paper suffers from - at best - obfuscation via math; worse, I highly doubt it would be possible for a reader (or someone working on follow-up work) to understand or reproduce your derivations/rationale. This is a crucial flaw in the paper.\n\nThe models you use are quite overparameterized in general. I would prefer to see a model like ResNet-20 or ResNet-56 on CIFAR-10, which is much less overparameterized for the specific task. \n\nIn general, the results look very impressive.\n\nI am concerned that these graphs zoom in on the lowest sparsities. Importantly, it is difficult to tell what the unpruned accuracy looks like for these models.\n\nFor weight-rewinding, do not use Conv-6. It is a toy network with little correspondence to real behavior. Use a larger-scale network like VGG-1`6 or ResNet-20 and rewind to iteration 1000 as Frankle et al. do in \"Linear Mode Connectivity and the Lottery Ticket Hypothesis.\"\n\n# Things to Address\n\nVerify that the details of the way that prior work has pruned networks is correct.\n\nEnsure the purpose and derivations in Section 3.1 are explained in a way that is crystal clear to a reader. I'm a mathematically well-informed reader who is an expert on pruning, and I can't follow the derivations or what you're trying to say with those derivations. This is due to unclear writing; I'm not a poorly-informed reader.\n\n# Other Notes\n\n\"The iterative pruning scheme of Han et al. uses global magnitude pruning.\" I believe this is incorrect. Han et al say that \"the pruning threshold is chosen as a quality parameter multiplied by the standard deviation of a layer's weights.\" \n\n\"Frankle & Carbin (2019)...employ uniform layerwise sparsity.\" I believe this is incorrect. Frankle & Carbin use uniform sparsity for MNIST but use global pruning for larger-scale networks (ResNet-18 and VGG-19).", "extra_scores": {}, "extra_text": {}}, {"review_id": "c3LTo2vo3So", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n- The authors propose LAMP, a layerwise adaptive magnitude-based pruning method.  The authors conduct extensive experiments on CIFAR10/CIFAR100/SVHN and Penn Treebank to validate the method.\n\nPros\n- somewhat novel pruning method, based on new weight score\n- extensive experiments on image and language datasets\n\nCons\n- In Equation (2), the authors point out that LAMP score is align with the order of weight squares. Therefore, one can directly prune the network based on weight squares. Why is it necessary to prune the network based on LAMP?\n- The comparisons are not sufficient. The authors should compare other \"pruning-retraining\" methods, like network slimming [1], soft filter pruning [2], etc. Though they focus on structured pruning, the core idea can be borrowed and adapted for unstructured pruning.\n- Lacking of experiments on large-scale datasets and large models, for example, on ImageNet. The performance of pruning methods  can be very sensitive and versatile when only evaluating on small datasets and models. And usually, a pruning method can be invalid when testing on ImageNet models.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UC_pxmKtHsj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe paper proposes LAMP, an importance score for unstructured pruning that incorporates layerwise statistics such that the resultant scores for each connection can be compared globally, cutting down on the hyperparameter space for magnitude pruning from the relatively standard practice of requiring hand-specified layerwise pruning rates. LAMP is motivated with a distortion analysis: LAMP is shown to be equivalent to minimizing an upper bound on the supremum of the change in model predictions of unit vectors. LAMP is compared against layerwise pruning rates obtained by standard uniform layerwise and global pruning, along with the less standard Erdos-Renyi kernel method, showing that LAMP can achieve higher accuracy for equivalent pruning rates in a specific experimental setup across several different networks.\n\n# Strengths\n\n- The problem the paper addresses is well-motivated: magnitude pruning rests on a poorly motivated and poorly understood set of heuristics.\n- The approach proposed by the paper has several desirable properties: it is hyperparameter-free, relatively cheap to compute, and handles some corner cases of global magnitude pruning (e.g., LAMP is invariant to rescaling of a layer in a net with BatchNorm).\n- Within the scope of the empirical evaluation in the paper, LAMP is demonstrated to outperform uniform and global magnitude pruning, the two most common approaches in the literature.\n\n# Weaknesses\n\n- Theoretical justification of LAMP: it's unclear how useful or strong the bound in Equation 7 is, especially given that the network is re-trained after pruning. These types of bounds often seem to be weak or vacuous in practice; further, there's no explicit connection between the bound and the loss, let alone the loss after re-training.\n- Empirical evaluation: the training and re-training regimes used by the paper are very non-standard, raising questions about the generality of the technique and the comparison to prior work. Specifically, the choice of AdamW with a fixed learning rate and the seemingly arbitrarily chosen durations of re-training are out of line with standard practice for many of these networks; these choices mean that it is impossible to compare the accuracy/sparsity tradeoffs achieved by LAMP to the results of various other papers that claim state-of-the-art pruning results with local/global magnitude pruning.\n\n# Overall recommendation\n\n6: Weak accept\n\n# Questions / suggestions for authors\n\nI would be willing to raise my score to a 7 with a more thorough comparison against prior work with more standard hyperparameters. Specifically, if the authors reported the results a comparison of using LAMP with the same networks and training/re-training schedule as any of [1,2,3,4], or any other specific setting that claims to be state-of-the-art, I would raise my score.\n\n# Other comments and suggestions\n\n- The notation of Equation 1 is a little bit confusing: I'm assuming that $W[i]$ is the $i$th element in some unrolled version of $W$, and that $W^2$ denotes element-wise squaring, but it'd be good to clarify this non-standard notation.\n- It would be good to have some more details on the networks, especially since many are being used in non-standard settings (e.g., VGG and ResNet-18 on CIFAR). Specifically: how are these networks adapted to these datasets, and what are the base accuracies for each networks on each dataset?\n- A deeper comparison to the layerwise sparsities (and overall pereformance) of [2] would be appreciated. For example: Section 5 of this paper notes that \"LAMP tends to keep the number of nonzero weights relatively uniform throughout the layers at extreme sparsity level.\" Section 4.2 of [2] notes \"The peaks and crests show that the RL agent automatically learns to prune $3 \\times 3$ convolutional layers with larger sparsity.\" Is there a reason for this discrepancy between the findings of each paper?\n- More details on the experimental methodologies of Figures 4(b,c), along with motivations for these experiments, would be appreciated. I am particularly confused by 4(c) -- I do not see standard SNIP scores on this plot, unless \"Global\" here refers to SNIP scores and not global magnitude pruning (in which case, how does standard global magnitude pruning perform here?). It would be worthwhile to include full methodological details for each of these experiments, including the training and re-training (where applicable) schemes, the specific point in training at which the importance scores are calculated, and the specific importance score used by each line on the plot.\n- Minor point: I don't believe that [1] used global magnitude pruning; I believe a better citation would be [5] (though there are also other prior/contemporary examples of global heuristics, like [6] and [7])\n\n# References used in review:\n\n[1] Song Han, Jeff Pool, John Tran, William Dally. \"Learning both weights and connections for efficient neural networks\".\n[2] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, Song Han. \"AMC: AutoML for Model Compression and Acceleration on Mobile Devices\"\n[3] Trevor Gale, Erich Elsen, Sara Hooker. \"The State of Sparsity in Deep Neural Networks\"\n[4] Alex Renda, Jonathan Frankle, Michael Carbin. \"Comparing Rewinding and Fine-tuning in Neural Network Pruning\"\n[5] Jonathan Frankle, Michael Carbin. \"The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks\"\n[6] Yann Le Cun, John S. Denker, Sara A. Solla. \"Optimal Brain Damage\"\n[7] Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr. \"SNIP: Single-shot Network Pruning based on Connection Sensitivity\"\n\n\n# Update post author response\n\nThanks to the authors for the detailed response. The authors have satisfactorily responded to my main criticisms of the paper (primarily about the non-standard evaluation regime, and secondarily about the motivation and strength of the theoretical results), so I’m raising my score to a 7, though I do think the paper would be further improved with references to Appendix G in the main body of the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NQqKMDRXNgB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel technique  (layer-adaptive magnitude based pruning, or LAMP) for pruning neural network weights (pruning can be beneficial in terms of overfitting prevention as well as other practical considerations).  LAMP evaluates weights in each layer in terms of the ratio of the magnitude of the weight to the sum of magnitudes of all surviving weights in the layer. The weight which evaluates as least important across all layers is pruned and then the process is repeated until the desired sparsity is achieved. The method is motivated theoretically as minimizing the distortion in the input/output mapping implemented by the weights of the layer. Experimental results on several benchmarks are presented.\n\nPros:\n\nThe experimental results are strong,  with LAMP consistently winning vs. competing techniques on 4 benchmarks problems. \n\nThe method is elegant, requiring no hyperparameter tuning and minimal computation. \n\nThe theoretical justification (mapping-distortion-minimization) makes a lot of sense. \n\nCons:\n\nMy only objection is that all of the experiments seem to be done on image datasets. It seems possible that deep learning networks applied to non-image data might not do as well under LAMP as they do for images. 'Under diverse datasets' in the abstract seems like an exaggeration. \n\nFurther comments:\n\nI found a couple of typos.\n\nP3 while such unstrutured pruning -> while such unstructured pruning\nP5 Global ‘on every layers’ -> on every layer\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a layer-wise magnitude-based tuning method through the adoption of LAMP score, motivated by minimizing the model output distortion. The new importance score differs from vanilla magnitude-based score in that it incorporate more layer-wise information. Extensive experiments are conducted on image and language models to show the improved accuracy upon prior arts under same model compression ratio. Ablation study is also provided to further explain the intuition and comparison of LAMP with other pruning methods. 

Though the experiments are extensive, some reviewers raised questions that only image datasets are tested. In the rebuttal, the authors addressed more on Appendix D which provides non-image results, and also modified the abstract to highlight the efficacy on image data. In all, given the extensive empirical evaluation on various datasets and model architectures, the improvement of LAMB over prior methods seems convincing. Nevertheless, we urge the authors to include more experimental results, for example ResNet-18 on ImageNet as promised to Reviewer 1, to make the results more solid. It is also suggested to include and discuss some relevant papers mentioned by the reviewers in the final version.  
Decision: Accept (Poster)