On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
Authors: Michela Paganini
Jessica Forde
Venue:  (2020)
Keywords: Pruning
Lottery Tickets
Science of Deep Learning
Experimental Deep Learning
Empirical Study

Abstract: We examine how recently documented, fundamental phenomena in deep learn-ing models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics  of  pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of  trainable, high-sparsity sub-networks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based un-structured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.

Under review as a conference paper at ICLR 2020
ON ITERATIVE NEURAL NETWORK PRUNING, REINI- TIALIZATION, AND THE SIMILARITY OF MASKS
Anonymous authors Paper under double-blind review
ABSTRACT
We examine how recently documented, fundamental phenomena in deep learn- ing models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through set of common iterative pruning techniques, to address questions of uniqueness of trainable, high-sparsity sub- networks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-base un- structured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.
INTRODUCTION
Deep neural architectures have seen dramatic increase in size over the years (Amodei & Hernan- dez 2018). Although not entirely understood, it is known that over-parametrized networks exhibit high generalization performance. with recent empirical evidence showing that the generalization gap tends to close with increased number of parameters (Zhang et al. 2017 Belkin et al.) 20118 Hastic
al. 2019: Allen -he et al. 2018 Du et al. 2018 Du Lee 2018), contrary to prior belief, also depending on the inductive bias and sensitivity to memorization of each base architecture (Zhang al. 2019).
While advantageous under this point of view, the proliferation of parameters in neural architectures may induce adverse consequences. The computational cost to train some state-of-the-art models has raised the barrier to entry for many researchers hoping to contribute. Because of limited memory, time, and compute, and to enable private, secure, on-device computation, methods for model com- pression have seen rise in popularity. Among these are techniques for model pruning, quantization, and distillation.
Pruning, in particular, has been literature (Breiman et al.) 1984). with work such as that of Mingers (1989) comparing the effects of different tree pruning techniques. In analogy this prior line of work, we provide empirical comparison of pruning methods for neural networks. As explicitly pointed out in LeCun et al.) (1990b), there no obvious, natural order in which neural network parameters should be removed. Indeed, as Schaffer (193)) suggests, identifying the best network pruning technique should give way understanding what accounts for their success.
overfitting avoidance method since the early decision tree
Revived by the recent observation of the existence of sub-networks with favorable training properties within larger over-paranne over-parametrized models ('lottery' or 'winning tickets' (Frankle & Carbin 2018 ). and in the spirit of Zhou et al.] (2019). this work investigates the dependence of the properties of these sparse, lucky sub-networks on the choice of pruning technique. We set out to answer the questions: do pruning methods other than magnitude-based unstructured pruning give rise to win- ning tickets? If so, do they identify the same lucky sub-network, or are there many equiperforming winning tickets within the same parametrized network? This allows us to categorize pruning methods based on the similarity of masks they generate. We develop methods to further analyze the nature of trainable sparse networks found through common iterative pruning techniques, and provide new insight towards a deeper understanding of their properties and of the processes that generate them. Without addressing issues related scalability and emergence of lottery tickets in

---

Under review as a conference paper at ICLR 2020
large-scale domains, focus instead the empirical characterization of weight evolution and emergence of distinctive connectivity patterns in small architectures, such as LeNet (LeCun et al, [9900), simple datasets, such as MNIST (LeCun et al. 1994). This also helps develop analysis techniques for sanity checks via visual inspection. We then apply the same techniques to larger architectures and more complex tasks to explore whether findings hold in different regimes.
1.1 CONTRIBUTIONS
This work provides empirical evidence showing that:
1. there can exist multiple different lucky sub-networks (lottery tickets) within an over- parametrized network;
2. it is possible to find a lucky sub-network through variety of choices of pruning techniques;
3. lottery ticket-style weight rewinding, coupled with unstructured pruning, gives rise con- nectivity patterns similar to the ones obtained with structured pruning along the input di- mension, thus pointing to an input feature selection type of effect. This is not the case when rewinding is substituted by finetuning;
4. random structured pruning outperforms random unstructured pruning, meaning that net- works are more resistant to the removal of random units/channels than to the removal of random individual connections;
5. different iterative pruning techniques learn vastly different functions of their input, and sim- ilarly performing networks make different mistakes on held-out test sets, hinting towards the utility of ensembling this setting:
6. weight stability to pruning correlates with performance, and can be induced through the use of suitable pruning techniques, without late resetting.
RELATED WORK
Identifying important units in neural network is an open challenge in machine learning, driven by desires for interpretability and model reduction, among others. Various proxies for importance have been proposed as direct consequence of the lack of precise mathematical definition of this term. Different objectives lead different mathematical formulations of importance: concept cre- ation, output attribution, conductance, saliency, activation, explainability, connectivity density, dundancy, noise robustness, stability, and more (Berglund et al. 2013 [Amjad et al. 2018 Zhou
al. 2018 Dhamdhere al. 2018 Shrikumar et al. (017).
State-of-the-ee -het---rt results in the field have seemed to strongly correlate with increased number of pa- rameters, leading the common plexity of the tasks at hand. In practice. however, most common neural network architectures can be dramatically pruned down in size (Hanson & Pratt 1989 LeCun et al. 1990b Hassibi & Stork
of vastly wver-parametrizer models, compared to the com-
1993 Han et al. 2015 Guo et al. 2016||Yang et 2016 to et al. 2017
Tung et al. 2017 Zhang
al. 2018 Ayinde al. 2019 Mussay al. 2019).
Unlike pruned pre-trained and then finetuned models, smaller untrained models are, in general, empirically harder to train from scratch. Frankle & Carbin 2018) conjecture that there exists least one sparse sub-network, a winning ticket, within an over- parametrized network, that can achieve commensurate accuracy commensurate training time with fewer parameters, retrained from the same initialization used to find the pruning mask. To find this lucky sub-network, they iteratively prune the lowest magnitude unpruned connections after a fixed number of training iterations (LeCun) et al.) 1990b] Han et al., 2015). Interestingly, while pruned weights are set zero, unpruned weights are rewound to their initialization values, prior to the next iteration of training and pruning.
Researchers have encountered difficulties in scaling this method larger models. Liu et al. (2018) were unable to successfully train small sparse sub-networks found through this procedure, when starting from VGG-16 (Simonyan & Zisserman 2014] and ResNet-50 (He et al.) 2015) trained on CIFAR-10 (Krizhevsky) 22099). and instead observed that random initialization outperformed weight rewinding. Similarly, Gale et al. (2019) were unable to successfully find sub-network
2

---

Under review as a conference paper at ICLR 2020
with these properties within ResNet-50 trained ImageNet (Russakovsky al. 2013) and Trans- former /Vaswani et al. 2017) trained on WMT 2014 English to-Germar
Slight modifications to the original procedure have been proposed in order to favor the emergence of winning tickets in larger models and tasks. Zhou et al. 2019) challenge the idea that unstructured, magnitude-basee pruning is necessary for the emergence of winning tickets. They instead prune weights based on their change in magnitude between initialization and the end of training, and reset the surviving weights to their sign initialization times the standard deviation of all original weights in the layer. Frankle & Carbin (0018) note that larger models such as ResNet- 18 and VGG-19 require selecting unimporannt weights globally instead of layer by layer, and recommend the use of learning rate warmup. Frankle et al. (2019) further add that late resetting of the unpruned parameters to values achieved early in training appears to be superior to rewinding the weights all the way to their initial values. MMorcos et al. (2019) confirm the findings on the efficacy of global pruning and late resetting.
Although hard to find in larger regimes, when found, lottery tickets have been shown to possess generalization properties that allow for their reuse in similar tasks, thus reducing the computational cost of finding task- and dataset-dependent sparse sub-networks (Morcos et al. 2019).
On the contrary, alternative approaches to neural architecture optimization have taken a construc- tionist approach, starting from minimal set of units and connections, and growing the network by adding new components (Stanley & Mikkkulaneen 2002) Xie et al.) [2019). Finally, other meth- ods combine both network growing and pruning. in analogy the different phases of connection formation and suppression in the human brain (Floreano et al. (2008] Dai et al. 2019).
METHOD
All networks in this section are trained for 30 epochs using SGD with constant learning rate 0.01, batch size of 32, without explicit regularization. The pruning fraction per iteration is held con- stant at 20% of remaining connections/units per layer. All experiment and analysis code is publicly available
3.1 PRUNING METHODS
This works explores a variety of pruning techniques that may differ along the following axes:
Neuronal Importance Definition: In magnitude-hased pruning, units/connections are removed based on the magnitude of synaptic weights. Usually, low magnitude parameters are removed. As (rarer) alternative, one can consider removing high magnitude weights instead (Zhou et al. 2019). Non-magnitude-based pruning techniques, instead, can be based, among others, on activations, gra- dients, or custom rules for neuronal importance.
Local vs. global: Local pruming consists of removing fixed percentage of units/connections from each layer by comparing each unit/connection exclusively to the other units/connections in the layer. On the contrary, global pruning pools all parameters together across layers and selects a global frac- tion of them to prune. The latter is partieularly beneficial in the presence of layers with unequal parameter distribution, by redistributing the pruning load more equitably. A middle-ground ap- proach is to pool together only parameters belonging to layers the same kind, to avoid mixing, say, convolutional and fully-connected layers.
Unstructured vs. structured: Unstructured pruning removes individual connections, while struc- tured pruning removes entire units or channels. Note that structured pruning along the input axis is conceptually similar to input feature importance selection. Similarly, structured pruning along the output axis is analogous output suppression.
In this work, we compare: magnitude-bast based {L1, random} unstructured (US), {L1, Lz, L- ran- dom} structured (S), and hybrid pruning. The hybrid techniques consists of pruning convolutional
"Translation Task ACI. 2014 Ninth Workshop on Statistical Machine Translation http:://tttt/ ptttranssall in taak htmm) html)
statmt wogg wmt 1 4/ //m/
in anonymized form, http::/tw...ccccoooaoaaaaaaaaaaa
3

---

Under review as a conference paper at ICLR 2020
dusud- 27% uia-1
GEPPAA
LESEEEE 2744
La straored
7004
bonns
2e aisel
byond
210C300 Drined Welgats
Pranionur Proteed Vomplii
Figure 1: Test accuracy on the MNIST test set for SGD-trained LeNet models, pruned using six different pruning techniques, and rewound to initial weight values after each pruning itera- tion.
Figure 2: Test accuracy on the MNIST test set for SGID-trained LeNet models, pruned using four different pruning techniques corresponding the four colors. The transparent curves corre- spond to finetuning; the dark ones to rewinding.
layers with L1 structured pruning and cully-comnected layers with Lı unstructured pruning. "fc- only" identifies experiments in which only the fully-connected layers are pruned. Structured pruning performed along the input axis. All techniques are local. We implement these pruning techniques and train our models using PyTorch (Paszke et al. 2017).
3.2 FINETUNING VS. REINITIALIZING
point of contention in the literature revolves around the necessity to rewind weights after each pruning iteration, opposed simply finetuning the pruned model. Here avoid performance- based arguments favor of a study of how this choice affects the nature of left-over connectivity structure. For comparisons with alternative rewinding techniques, see Appendix A Unless other- wise specified, all results refer to setup with full weight rewinding. Note that when rewinding the weights according to the strategies proposed in Frankle & Carbin (2018) and Zhou et al. (2019), all parameters in the network get affected by this procedure, including blases. On the other hand, in this work, pruning is not applied biases.
4 RESULTS
In all figures, unless otherwise specified, the error bars and shaded envelopes correspond to one standard deviation (half up, half down) from the mean, over 5 experiments with seeds 0-4.
We begin by exploring the performance of sub-networks generated by different iterative pruming techniques starting from base LeNet architecture, where lottery tickets are known to uncontrover- represents the test accuracy after exactly 30 epochs of training (not the best test accuracy achieved across the 30 epochs). Multiple techniques able to identify trainable sub-networks up to high levels of sparsity. For note on how the fraction of pruned weights computed, see Appendix
sially exist and be easy to find. Each point in Fig. [
Different types of magnitude-basee structured pruning seem to perform only marginally better than pruming random channels, leading us to conclude that either the channels learn highly redundant transformations and therefore equivalent under pruning, there exists hierarchy of importance among channels but it not correlated to any of the L,, norms tested in these experiments.
We quantify the overlap in sub-networks found by two different pruning methods started with pairwise-identica initializations by computing the Jaccard similarity coefficient (also known as In- tersection over Union, IoU) between the masks (Jaccard 1901).
Although the different structured pruning techniques find sub-networks that perform similarly (Fig I. deeper investigation into the connectivity structure of the sub-networks they find shows,
4

---

Under review as a conference paper at ICLR 2020
6 structured, amount=00% axis=1
random unstructured, amount-20%
4 structured, amount=20% axis=]
random structured, am0unt=20%, axis=]
E; unstructured, amount=20%
structured, amount-20% axis=1
bybrid
Jonowed
Figure 3: Jaccard distance between the masks (i.e. connectivity structures) found by pruning LeNet using Lg-structured pruning, and those found by other pruning methods listed in the legend, con- ditional on identical seed for meaningful comparison in light of neural network degeneracy. The comparison is conducted for each layer individually. L1 -sstructured pruning yields the most similar masks L2-structured pruning, as expected.
Figure 4: Jaccard distance between the mask for the second convolutional layer found by pruning LeNet using Ln-unstructured pruning, and the mask found by other pruning methods listed in the legend above, conditional identical seed. Surprisingly, Lu-unstructured behaves more like L1- structured pruning than a random unstructured pattern. This behavior is further investigated in Fig.
as evident from the per-layer growth of the Jaccard distance with pruning iterations in Fig. 3 that there exist multiple lucky sub-networks with similar performance, yet little to no overlap.
The study of the Jaccard distance between masks surfaces another interesting phenomenon: as shown in Fig. 4 the connectivity patterns obtained from magnitudebassed unstructured pruning are relatively more similar to those that one would expect from structured pruning along the in- put dimension, especially in the convolutional layers, than random unstructured patterns. This suggests that a form of input (or, at times, especially in larger models, output) feature selection is automatically being learned. This observation is further confirmed via visual inspection of the prun- ing masks; the first two columns of Fig. 5 show the weights and masks in the second convolutional layer of LeNet, obtained by applying structured and unstructured Lı pruning. Unstructured pruning ends up automatically removing entire rows of filters corresponding to unimportant input channels. Structured pruning, instead, in this case, does because, in these experiments, it explicitly in- structed to prune along the input dimension. This suggests that the low- www-aagit weights pruned by L1 -uuustruuttueed pruning may be those that process low-importance hidden representations and whose outputs contribute the least to the network's 's output, which in turn results in lower gradients and smaller weight updates, causing these weights to remain small, and thus be subject to pruning.
Not only do the different pruning methods lead to different masks, but they also lead to sub-networks that learn partially complementary solutions, opening up the opportunity for the ensembling dif- ferent sparse sub-networks. Table 2 records the average accuracy (over the 5 experiment seeds) of sub-networks obtained at each pruning iteration through a set of pruming techniques, as well as the average accuracy obtained after simply averaging all their predictions, or the predictions of struc- tured and unstructured L pruning. Note that, after each pruning iterations, the levels of sparsity in the sub-networks will be different as they depends the type of pruning applied.
5

---

Under review as a conference paper at ICLR 2020
(a) 2nd iteration of L1 S
(b) 5th iteration of L1-
(c) 5th iteration of L1- (d) 5th iteration of L1-
pruning with rewinding
US pruning with rewind- ing
US pruning with o
US pruning with finetun- ing
sign(w;) reinitialization
3na2'
..t
2 L.
vex
(e) 5th iteration of L1-S pruning with rewinding
(f) 10th iteration of
(g) 10th iteration of L1- (h) 10 iteration L1-
US pruning with rewind- ing
US pruning with a US pruning with finetun-
sign(w,) reinitialization ing
Figure 5: Masked weights in the 2nd convolutional layer of LeNet. The two represent two different pruning iterations. The columns represent four different pruning and weight treatments. This convolutional layer is composed of 6 input channels and 16 output channels, with 3z3 filters. Masked out weights appear in gray color. Active, positive weights are depicted in red, negative weights in blue.
4.1 FINETUNING VS. REINITIALIZING
The "Lottery Ticket Hypothesis" Frankle & Carbin (2018) postulates that rewinding weights the initial (or early-stage (Frankle et al., (219]) values after pruning is key to identifying lucky sub- networks. In this section, attempt test this hypothesis in the experimental setting of small networks (LeNet) and simple tasks (MNIST), and we investigate, at the individual parameter level, how the this procedure ends up differing from finetuning after pruning.
The emphasis here is not on performance, rather on understanding. focus on extracting state-of-the-ar performing lottery tickets, in favor, instead, of minimizing the influence of exogenous choices and the reliance on ad-hoc heuristics, whose primary goal is last-mile performance gains.
Therefore, we decide not to
Finetuning is found to yield pruned networks that are comparable in performance with reinitialized regime) when given the same training budget,
networks (at least in the sasple-tass-smll--netteee and when the networks are pruned using the same technique. It is known, however, that higher quality lottery tickets can be obtained by rewinding the weight values to their value after a small number of training iterations (Frankle et al. 2019). This was not experimented with in this work.
When finetuning with unnstrruutrreed pruning, the magnitude of weights tends to continue growing. achieving higher final values at later pruning iterations than their reinitialized counterparts.
Despite achieving similar average performance in this simplified scenario, the interesting observa- tion lies in the stark difference between masks created by the same pruning technique when the weights are rewound or finetuned. Fig. 5 shows that finetuning removes the natural tendency of Ly-unstructured pruning with reinitialization to approximate the behavior of L1-structured pruning- We can therefore primarily attribute the emergence of that interesting connectivity pattern to the choice of reinitializing the weights. In terms of mask structure, the difference between the effects of finetuning and lottery-ticket reinitialization is observed to grow similarly across all layers and logarithmically with the number of pruning iteration (see Appendix
5 WEIGHT EVOLUTION AND STABILITY TO PRUNING
Frankle al. (2019) introduce the practice of late resetting, i.e. rewinding weights to their value after a small number of training iterations, as opposed to their value at initialization, and claim that its effectiveness is due it enforcing form of weight stability pruning. In other words, weight values after a few batches of training may be sufficiently informative to then drive the optimization down the same promising direction, from which the high-quality solution found by the unpruned

---

Under review as a conference paper at ICLR 2020
(a) Hybrid (b) L1 US (c) Rand US (d) Rand
(e) L2
(1)
(g) L (h) fe-only
Figure 6: Weight values (y-axis) after 30 epochs of training various consecutive sparsity levels (x-axis), for weights in the 2nd convolutional layer in the LeNet architecture (seed: 0). Lines are terminated when the weight is pruned.
(a) Hybrid (b) La US
Rand US (d) Rand
L2 S
L
(g) L- (h) fe-only
Figure 7: Weight values (y-axis) after 30 epochs of training various consecutive sparsity levels (x-axis), for weights in the 3rd fully-connected layer in the LeNet architecture (seed: 0). Lines are terminated when the weight is pruned.
model can be reached, even in high-sparsity scenarios. In practice, this would correspond all unpruned weights converging to approximately the same value after each training stint.
While there exists multitude of good solutions for convergence, the initially found by the unpruned network is certainly high-quality one that we know should be reachable from the initial conditions. In way, being able to find this solution even in the sparse regime hints at the fact that the adopted pruning strategy has not interfered with and modified the optimization landscape to the point that this solution is no longer easy to reach.
We therefore ask whether weight stability to pruning is indeed advantageous, in terms of pruned network performance, and, if so, whether this property can be induced in more natural ways than by manual late resetting. To answer this, we track individual and group weight evolution over time in aver-parameterized and ander-parameterized networks (i.e. unpruned and pruned, with increas- ing sparsity). We look for differences in weight value after training when the weight is part of large/small pool unpruned weights.
As shown in Fig. mance (see T for performance evaluation), and observe that the most competitive pruning techniques also exhibit the highest degree of weight stability, when weights reset to their true initial value (no late resetting).
anddn
we empirically find correlation between weight stability and perfor-
In general, pruning techniques with unstructured component tend exhibit higher weight sta- bility over pruning iterations. These weights tend to converge to solutions in which their ordinality varies only minimally from one pruning iteration the next. This is especially true for the 'hybrid' pruning technique, in which weights in convolutional layers seem to converge to approximately the same value after each set of 30 allotted epochs of training from rewinding.
In fully-connected layers, even when using "stable" pruning techniques, the magnitude of unpruned weights the end training (with constant training budget of 30 epochs from the initial conditions) grows as sparsity grows, likely to compensate for the contribution of missing signal from pruned connections to the output of downstream units. a parallel to the familiar decrease in the standard deviation of the weight distribution of common initialization schemes as the number of incoming and/or outgoing connections increases, here we confirm that, as we prune individual connections, the ideal, final weight configuration is composed of values with higher overall magnitude and higher standard deviation. We hypothesize, but do not get to test in this work, that, when pruning with rewinding, rescaling the initial weights by a factor that accounts for the decrease in the number of incoming and/or outgoing connections could be beneficial to achieve better, faster convergence in sparsified networks.

---

Under review as a conference paper at ICLR 2020
0.10
Linf stroctured, amoun-- 20% axcs-1
L1 sirectoned 2005 axio-
13 instrustrrdd 20%
random stractured, smount-2e axis-] randors urat sectuureet amount 20%
a030/16 -W
0.04
0.00
D02V1
COBV2
2:1
D22
3:3
Figure 8: Absolute difference between weight values pruning iteration and i - 1, averaged over all iterations and all weights in each layer. The lower, the more stable.
5.1 Do THESE OBSERVATIONS HOLD LARGER NETWORKS AND DOMAINS?
These empirical results have been observed hold in AlexNet and VGG- 11 architectures on MNIST and CIFAR-10 (see Appendix D). In even larger models, the nature of lottery tickets is still con- tended, and special care is required in designing ad-hoc training and pruning procedures to facilitate their discoverability. These larger regimes are not subject to study in this current work.
CONCLUSION
We show evidence against the uniqueness of winning tickets in variety of networks and tasks, by identifying different lucky sub-networks of competitive performance within the same parent net- work, while controlling for degeneracy by fixing experimental seeds.
We also provide empirical results showing that rewinding weights to the original values at initializa- tion after each pruning iteration yields sparsified networks that may not only be superior to finetuned sparse models from a performance perspective, but also, practice, appear to possess structural advantages that might make them more suitable for hardware-level implementations with inference- time speed-up effects.
With the explicit intent not to adopt any special training tricks suggested in the literature induce the emergence of lottery tickets in state-of-the-ar deep architectures, this work intentionally restricts its action smaller models and domains, begin more thorough investigation of the phenomenology of pruned networks.
We offer methods experimentally analyze and compare the effects of different pruning techniques on the performance of neural networks and on the nature of the masks generated by each. This can be further extended in the future better understand the role late resetting, learning rate warm-up, and any new pruning method or related training trick, in terms of the similarity of masks obtained through each pruning procedure modification. We also conjecture that early monitoring of weight stability after small number of pruning iterations (as opposed to, or in addition to the monitoring of the spectrum of the Hessian) might serve as a powerful indicator of the likelihood of the iterative pruning procedure to result in the discovery of lottery ticket.
Valuable extensions would target machine learning applications beyond image classification, and study the interplay of different optimization strategies and pruning techniques in the evolution of weights and emergence of connectivity structure.
We hope these experimental results will guide theoretical work in this sub-field to match observa- tions. An open dialog between the theoretical and experimental communities can lead to faster and more robust understanding of deep learning phenomena, and allow practitioners to select the most suitable methodology for their application based on more grounded understanding rather that mere performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Z. Allen-Zhu, Y. Li, and Y. Liang. Learning and Generalization in Overparameterized Neural Net- works, Going Beyond Two Layers. arXiv e-prints, November 2018.
Rana Ali Amjad, Kairen Liu, and Bernhard C Geiger. Understanding individual neuron importance using information theory. April 2018. URL http:/ttt//ee /ttp:///axivvvo org/ 2a//g/abbb18406679
Dario Amodei and Danny Hernandez. Al and compute. May 2018. URL httts::/openaai
com/bl og/ / a and compute/
Babajide O. Ayinde, Tamer Inanc, and Jacek M. Zurada. Redundant feature pruning for accelerated
inference in deep neural networks. Neural Networks, 118:148 158, 2019. ISSN 0893-6080. doi: htps://doi.org/Iin http:////....../o/oeeeeeeee.e110
URL http: / /www. sci encedi rect. com/
M. Belkin, D. Hsu, S. Ma, and Mandal. Reconciling modern machine learning and the bias- variance trade-off. arXiv e-prints, December 2018.
Mathias Berglund, Tapani Raiko, and KyungHyun Cho. Measuring the usefulness of hidden units in boltzmann machines with mutual information. In Minho Lee, Akira Hirose, Zeng-Guang Hou, and Rhee Man Kil (eds.), Neural Information Processing, pp. 482-489, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-42054-22
L. Breiman, J. Friedman, C.J. Stone, and R.A. Olshen. Classification and Regression Trees. The Wadsworth and Brooks-Cole statistics-probbbblity series. Taylor & Francis, 1984. ISBN 9780412048418. /ttp::t:: httttppss/boooksoooogeeeeee gwwgle. com/ books broooks?dd-Jwae JwQx WOmSyQC
Xiaoliang Dai, Hongxu Yin, and Niraj Jha. Nest: A neural network synthesis tool based on grow- and-prune paradigm. IEEE Transactions on Computers, 2019.
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How Important Is a Neuron? arXiv e-prints, art, arXiv: 1805. 805.12233, May 2018.
Simon S. Du and Jason D. Lee. On the Power of oaeapaaramniiizationn in Neural Networks with Quadratic Activation. arXiv e-prints, art. arXiv:1803.01200 Mar 2018.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient Descent Provably Optimizes Oct 2018.
Over-parameterrized Neural Networks. arXiv e-prints, art.
Dario Floreano, Peter Dün, and Claudio Mattiussi. Neuroevolution: from architectures to learn- ing. Evolutionary Intelligence, 1(1):47-62, Mar 2008 ISSN 1864-5917. doi: 10. 10.1007/ s12065-007-0000-4.
Jonathan Frankle and Michael Carbin. The Toe networks. March 2018. URL UBL/attp://arxx...0///////111 036353333333333
Finding sparse, trainable neural
Jonathan Frankle, Gintare Karolina Dziugaite, M Roy and Michael Carbin. The lottery ticket hypothesis scale. March 2019. URI.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. February 2019. URL (Rtppt//:/aaxii www. org/ abs / 1wt...90....00 00119.92.979444
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic Network Surgery for Efficient DNNs. arXiv e-prints, art, arXiv: 1611164888..34... Aug 2016
Han, J Pool, Tran, and Dally. Learning both weights and connections for efficient neural network. Adv. Nearal Inf. Process. Syst., 2015. URL http:://///ppppeeiissicsscr///pp 1844-eermn ng- both weehhhwwei ghts- and ott.::/ceennnnooooooooaaaaaaaaaa www.raiiiicceetteee--e-e-eeeeeee
Song Han. Huizi Mao, and William J. Dally. Deep Compression: Compressing Deep Neu- ral Networks with Pruning, Trained Quantization and Huffman Coding. arXiv e-prints, art. 1100v:1510.00149 Oct 2015.

---

Under review as a conference paper at ICLR 2020
Stephen José Hanson and Lorien Y Pratt. Comparing biases for minimal network construction with propaproppgttiin. Advances in neural information processing systems, pp. 177-185, 1989.
Babak
Hassibi and David G. Stork. Second order derivatives for network prun- In J. Hanson, J. D. Cowan, and C. L. Giles (eds.), Advances in Neural Information Processing Systems 5. pp. 164 Morgan-Kaufmann
ing: Optimal brain surgeon.
URL http::// ni niw. cttpps.cccpeer 647 oooo-eeord-order--aee derorddrrrrriiieie vaativatio ves for newwwrk- ng- - -braibbbrrraggge
171.
1993.
pdf
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in High- Dimensional Ridgeless Least Squares Interpolation. arXiv e-prints, art. arXiv:1903.08560, Mar 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. December 2015. http:////aaxxi wttp://///.bb....o..///// /tttt:///aart..oorr/a/////11
03385
Paul Jaccard. Etude de la distribution florale dans une portion des alpes et du jura. Bulletin de la Societe Vaudoise des Sciences Naturelles, 37:547-579, 01 1901. doi: 1599/eall-2664555
Krizhevsky. Learning multiple layers of features from tiny images. 2009.
URL
http://citeseerx. psu.psu.u...
1.2.222.
9204rep-repletyee--e
Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, E Howard, Wayne Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a Back-Propagation network. In D S Touretzky (ed.), Advances in Neural Information Processing Systems 2. pp. 396-404. MargannKaaffannn, 1990a. URL http:
pdf
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Touretzky (ed.), Advances in Neural Information Processing Systems 2. pp. 598 605. MlogannKKuufman 1990b. URL http://awa..aaaaaaaccc/aaa/aaaap
daiinnddamgee-ae pdf
Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. The MNIST database of handwritten digits., 1994. URL http::/yaannnnntt http:///annnnnnn...mmmmm com/ exdb/mnist/
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the value of 05270
network pruning. October 2018.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A Filter Level Pruning Method for Deep
Neural Network Compression. arXiv e-prints, art. arXiv: 10Xivv1107...om htl..1707706342,10 00342, Jul 2017.
John Mingers. An empirical comparison of pruming methods for decision tree induction. Machine Learning, 4(2):227-243, Nov 1989 ISSN 1573-0565, doi: 10.1022344102264110993.. URL
htttss://ooi. www.ddi..oggg//.. 1023 //
Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuand ong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. arXiv e-prints, art. Mixiv:190.002777 Jun 2019.
B. Mussay, S. Zhou, V. Braverman, and D. Feldman. On Activation Function Coresets for Network Pruning. arXiv e-prints, July 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. ImageNet large scale visual recogmition challenge. Jnt. J. Comput. Vis., 115())211-255 Decem- ber 2015. URL btt.tttss::. //ooi http:///ooooo.......0/... wtwp///1.1000/////.
10

---

Under review as a conference paper at ICLR 2020
Cullen Schaffer. Overfitting avoidance bias. Machine learning, 10(2):153 1993.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Courning-Volumme 70, pp. 3145-3153, JMLR. org, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for Large-Scale image recognition. September 2014.
1556
Kenneth O. Stanley and Risto Mikkullinen. Evolving neural networks through augmenting topolo- gies. Evolutionary Computation, 10(2):99-227, 2002. URL hatp:://nnncc cs www.ccs..ttteeee utw.as.eeeeo edu/
?stanley:ec02
Frederick Tung, Srikanth Muralidharan, and Greg Mori. Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization. arXiv e-prints, art. arXiv:1707,09102, Jul 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. Ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I Guyon, U V Luxburg, Bengio, H Wallach, Fergus, S Vishwanathan, and R Garnett (eds.), Advances ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL httppssccpppppeeppppppep a
Neu-
http::/
attention. is all aal-you- you syoo-neeee need. pdf
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring Randomly Wired Neural Networks for Image Recognition. arXiv e-prints, art. arXiv:1904.01550 Apr 2019.
Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing Energy- Convolutional Neural Networks using Energy- Aware Pruning. arXiv e-prints, art. arXiv:1611.0511 Nov 2016.
Chiyuan Zhang, Qianli Liao, Alexander Rakhlin, Karthik Sridharan, Brando Miranda, Noah Golowich, and Tomaso Poggio. No 067 april 4, 2017 theory of deep learning iii General- ization properties of sgd by. 2017.
Chiyuan Zhang. Samy Bengio, Moritz Hardt, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterization. February 2019. URL http:://rxiv:
Strg/bbs/ii 04698
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang- A systematic dnn weight pruning framework using alternating direction method of multipliers. In The European Conference on Computer Vision (ECCV), September 2018.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in CNNs via ablation. June 2018.
02891
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. May 2019. URL ULLT::p:///axxxv.oorr/////10...

---

Under review as a conference paper at ICLR 2020
L, structured, amount-20%, axis=1, reinit
structured, amount-20%, axis=1, reinit
4, structured, amount=20% axis=1, finetune
La structured, emount=20% exism1, finetune
4 unstructured, amount=20%, reinit unstructured, amount=20% finetune
hybrid, reinit
hybrid, finetune
laper
Ioper comy?
1.
1.4
06
Prasing
Prusing Terabis
Jayer.
layer:
layer:
10
Jacoed Biacin park 3rth--
Jecced dissson th "aaas" raak
100-
0.
60
00
Trusigg
Trusdag
Prostog Darshoo
Figure 9: Jaccard distance between the masks found by pruning LeNet and rewinding weights, and those found by finetuning after pruning, conditional on identical pruning techniques and seeds. The comparison is conducted for each layer individually, Note the logarithmic scale on the x-axis.
A FINETUNING VS. REINITIALIZING (CONTINUED)
The Jaceard distance, or any other similarity measure among masks (e.g., the Hamming distance), can be adopted to quantify the effects of the choice of finetuning or reinitializing weights after pruning. As expected, the nature the connectivity structure that emerges in an iterative series of pruning and weight handling steps depends not only on the pruning choice but also on how weights are handled after pruning. Finetuning yields significantly different masks from rewinding, for all pruning techniques, and the difference (quantified in terms of the Jacecard distance) appears to grow logarithmically in the number of pruning iterations (Fig.
A.1 SIGN-BASED BASED REINITIALIZION
Does the exact weight value initialization carry any significance, or all we care about its sign?
As proposed by Zhou et al. (2019), we experiment with reinitializing the pruned sub-network by resetting each unpruned parameter
L. -ign(w:). where w; the weight value of parameter at initialization, and L., is the empirical standard deviation of the weights in layer L that contains the parameter i. This is in contrast to rewinding to the initial weight values wi, as originally suggested by Frankle & Carbin (2018), or finetuning without weight resetting.
The overall test accuracy of our experiments (Fig. TOp seems to support the observation of Zhou al. (2019) that, as long the sign matches the sign at initialization (and special care is taken in re-scaling the standard deviation), resetting the weights to different values won't affect the sub- networks trainability and performance. Removing the factor of causes the network not to converge; as expected, keeping the weights within a numerically favorable range is important, and naively focusing on the sign alone leads to poor performance.
Adopting mask similarity to test for equivalence among reinitialization techniques, the method of Zhou al. (2019) is found induce structure that differs more substantially from the connectiv-
12

---

Under review as a conference paper at ICLR 2020
0.80
11 dinutuadd musie
0.80
10002t-11 s1a-1, frstzze
aiuutued mustg
0.0
11 untrad (al
E050121 315 xpss
12 inuuteed musee
am00rt: 10% ass tistzze
12 siuuteed smase
lybesl,
057 0.5
proned
Figure 10: Test accuracy comparison the MNIST for SGD-trained LeNet models, pruned using four different pruning techniques corresponding to the four colors, and reinitialized using three different techniques corresponding the line styles.
0.2
0.4
0.5
0.6
0.7
0.8
0.9
0.95
0.97
0.98
0.99
AlexNet
61,100
48,880
36,660
30,550
24.440
18.330
12,220
6.110 13,286
3.065
1.833
1,222
VOGH
132,863
106,290
79,718
66,431
53,145
39,859
26,572
6,633
3,985
2,657
1,328
Table 1: Number of parameters (in thousands) left in each model at each pruning fraction.
ity patterns generated by the method of Frankle & Carbin (2018) than simple finetuning does, across most layers and a set of common choices of magnitude-base pruning techniques (Fig. II
B EFFECTIVE PRUNING RATE
subtle implementation detail involves the way in which the fraction of pruned weights com- puted, especially in convolutional layers pruned with structured pruning techniques. Take, for ex- ample, the LeNet architecture used in this work. When applying pruning to produce results like the ones displayed in Fig. [ there are two ways define the number of weights that get pruned (displayed along the x-axis). The definition used in the main portion of the paper uses the fraction of weights explicitly pruned by the decision rule that produces the mask for each layer. However, this doesn't t account for the implicitly pruned weights, i.e. those weights that, due to downstream pruning following layers, are now disconnected from the output of the neural network. For this reason, they receive no gradient and their value never changes from that at initialization. They can technically take on any value, including 0, without affecting the output of the neural network. In other words, they could effectively be pruned without any loss of performance. If include these weights the effective fraction of pruned weights, Fig. [ (now converted into Fig. [1( for visual coherence) is modified into Fig. 12(b When looking at effective sparsity, structure pruning appears to be competitive, especially at high pruning fractions, because its effective sparsity naturally tends to be higher. This also suggests that since the effective pruning rate per iteration is higher than 20% for structured pruning techniques, one could experiment with lowering it, to avoid aggressive pruning and consequent loss in performance.
The number of parameters corresponding to various fractions of pruned weights in the models in- vestigated in this work listed in Table
C PRUNING AFTER TRAINING EPOCH
Is necessary to train for many epochs before pruning to find high quality masks?
13

---

Under review as a conference paper at ICLR 2020
(a) First
convolutional (b) First
convolutional (c)
First
convolutional (d) First
convolutional
layer
layer
layer
layer
(e) Second convolutional (f) Second convolutional (g) Second convolutional (h)
Second convolutional
layer
layer
layer
layer
(i) First fully-connected (j) First fully-connected (k) First fully-connected (I) First fully-connected layer
layer
layer
layer
(m) Second fully- (n)
Second
fully- (o) Second
fully- (p) Second fully-
connected layer
connected layer
connected layer
connected layer
(q) Third fully-connected (r) Third fully-connected (s) Third fully-connected (t) Third fully-connected layer
layer
layer
layer
Figure 11: Growth of the Jaccard distance between the mask found by rewinding after pruning and three other techniques of handling weights (in order from highest lowest opacity the figures: finetuning, rewinding to sign(w;), and rewinding to sign(w,) (w)). Each column of sub-figures cor- responds to a pruning technique (same color code in the rest of the paper); each row corresponds to layer in LeNet. Note the logarithmic scale on the x-axis.
14

---

Under review as a conference paper at ICLR 2020
L-tnr structured
I-nn stractured
Test Accurocy
amonnt 110--000 20% asa
Test Accurscy
arsount 20% a
1.1 instructurnd amount=20y 20%
1.1 instricturedd amoonnt 20% strectured
12 structured
amount-
arsount-20%
Ic-oaly unstrwluued amormt-20%%
foouly urstructared Arsennt-208
Tybrid
hyarid
mandom structured
amoent-20%%,
amountt22X%
random unstructured
random unstractured
amouunt 200%
amoontt 20%
Praction of Pruned Waights
Effecttive Praction of Pruned Waights
(a) Number of pruned weights corresponds loca- tions where the mask is explicitly zero.
(b) Unchanged units included in the effective num- ber of pruned weights.
Figure 12:
Test accuracy achieved after 30 training iterations by LeNet models trained with SGD and pruned with the methods listed in the legend. Each dot corresponds to a version of the model the end of training. Each line connects models obtained by iterative pruning from the same seed. The two plots differ in the way the fraction of pruned weights is computed.
(a) 5th iteration of L1-US pruning with rewinding epoch)
(b) 5th iteration of UU pruning with rewinding (30 epachs)
(c) th iteration of L1-US prun-
(d) 10th iteration of LI prun-
ing with rewinding epoch) ing with rewinding (30 epachs)
Figure 13: Difference in mask structure in the second convolutional layer that emerges when pruning after a single epoch (left) versus 30 epoch (right) training (seed: 0).
To speed up the winning ticket search, we evaluate the option of shortening the training phases to only 1 epoch of training per iteration.
We compare the masks obtained with this quicker method the masks found after training the network at each pruning iteration for 30 epochs (Fig. [3). After a single epoch of training, the ordinality of weight magnitudes has not yet settled to the solution corresponding to the values after 30 epochs of training. Although the majority of ranking swaps among weights, which can bring parameters from lowest magnitude quartile all the way to the top one, and vice versa, happen in the first few epochs of training, minor movements later on in training might still be highly relevant in obtaining robust ranking for magnitude- mrage pruning.
D COMPLEMENTARITY OF LEARNED SOLUTIONS
Pruning the same network using different pruning techniques gives rise to sparse sub- networkss that differ not only in structure but also in the learned function that they compute. Given sufficient com-
15

---

Under review as a conference paper at ICLR 2020
8000
R000
2t sattaaaee
6U30
K00U
lyteld
belend
4030
1000
2000
2000
saly estratesd
sap sadtrattaee
(a) 18 pruning iteration
(b) 20 pruning iteration
Figure 14: Number of examples in the MNIST test set over which the sub-networks obtained through each pruning technique agree on the prediction, on average (over experimental seeds).
Pruming leeration
S
L US
hybrid
nindom US
mendom S
S
fo-unly Ly US
all
bybrid Ic folonly
L.g
97.4
97.5
97.8
97.8
97.4
97.3
97.1
97.9
2
98.
95.5
96.5
97.8
97.5
96.
96.4
96.2
91.7
98.1
97.9
95.8
95.6
97.9
97.6
95.6
95.
95.4
91.8
97.9
97.9
95.0
95.1
97.6
97.5
91.8
94.0
94
97.8
97.7
97.9
91.7
97.4
97.5
91.0
91.3
97.6
97.4
97.8
89.5
90.9
97.3
91.4
45.8
83.2
$3.9
91,5
90.2
97.7
87.6
97.0
97.3
14.0
83.7
86.2
97.2
97.5
77.2
71.3
95.9
96.8
11.3
60.1
76,3
97.0
96.6
97.1
12
72.5
72.5
94.5
96.1
11.3
455
74,1
96.9
96.3
97.0
69.0
65.1
90.2
95.8
11.3
41.1
65.6
96,1
95.7
96
55.7
72.7
94.1
11.3
18.5
48.3
94.7
94.6
95.7
$7.1
43.7
69.5
92.5
11.3
11.3
24.7
93.3
94.0
94.7
$8.3
11.3
11.3
21.9
91.9
19
30.2
33.9
23.0
76.4
11.3
11.3
11.9
$4.3
89.3
90.3
20
29.4
33.1
21.2
71.9
11.3
11.3
11.7
73.6
85.0
86.0
Table 2: Sub-network accuracies at each pruning iteration. Ensembling of sub-networks obtained through different pruning techniques can yield higher performance, hinting the complementarity of information learned by each sub-network.
pute and memory budget, one can consider combining the predictions made by each sub-network to boost performance. On the left side of Table for each pruning iteration, the average accuracy of pruned LeNet model is listed, along with, on the right, the accuracies obtained by simply averaging the predictions of all eight individual sub-networks ("all") and by averaging the predictions obtained from the three most promising pruning techniques (last column).
The similarity of solutions can also be explored by looking at the heat maps of the agreement in average class prediction across sub-networks obtained through different pruning techniques. For reference, Fig. 14 these visualizations for the 18th and 20th pruning iterations.
E ALEXNET AND VGG ON MNIST AND CIFAR-10
In this section, confirm qualitative observations, previously reported on LeNet models, the structure of connectivity patterns that emerge from the application of L, unstructured pruning in the context of Alex and VGG-11 architectures.
16

---

Under review as a conference paper at ICLR 2020
st conv AlexNet
(b) 1st conv in VGG-11
(c) 2nd conv in AlexNet
(d) 2nd conv in VGG-11
000 5 1.
(e) 3rd conv AlexNet
(1 3rd conv VGG-11
g tth conv in AlexNet
(h) conv in VGG-11
Figure 15: Binary weight masks for the first four convolutional layers of AlexNet trained on MNIST (left) and VGG-11 trained on CIFAR-10 (right), the 20th and last pruning iteration for Lı unstruc- tured pruning (seed: 0). Despite the unstructured nature of the pruning technique, structure emerges along the input and output dimensions, which resembles the effect of structured pruning.
We train two individual sets of experiments starting from the base AlexNet model, one on MNIST, one on CIFAR-I0. VGG models are trained on CIFAR-IO exclusively.
Fig. 15 shows the binary masks obtained in the first four convolutional layers of the two models after 20 pruning iterations, with pruning rate of 20% of remaining connections at each iteration. Here, the reported AlexNet properties refer to the MNIST-trained version. Preferential structure along the input and output dimensions (in the form rows or columns of unpruned filters) is visible across the various layers, although visual inspection becomes hard and inefficient the number of parameters per layer grows.
Qualitative visualizations of weight values at convergence over the series of 20 pruning iterations (Fig. [6) support arguments around weight stability as an indicator of the likelihood of identify- ing competitive sub-network. In this small set of experiments, in which L unstructured pruning
17

---

Under review as a conference paper at ICLR 2020
appears remain competitive at high sparsity levels, weights remain more likely preserve their ordinality when subject to L1 unstructured pruning. On the other hand, hybrid pruning, while exhibiting weight evolution characteristics that bridge the behavior of structured and unstructured pruning, is dominated by the effects of L1 structured pruning of the convolutional layers. We sus- pect that too high pruning fraction in the pruning techniques with structured components may be responsible for the associated drop in performance. Regardless, the claim that weight stability is sufficient (though perhaps not necessary) for sub-network performance high sparsity appears to hold in these architectures and tasks. As in previous experiments with smaller models, we want to shy away from performance- based comparisons that attempt to determine the "best" pruning tech- nique, and instead reiterate that, irrespective of its value, performance seem to correlate positively with weight stability. However, this not yet sufficient to determine direct causal link between the two properties.
18

---

Under review as a conference paper at ICLR 2020
015
305
0.0
300
410
406
4393
(a) conv in AlexNet pruned with L1-S pruning
(b) 1st conv in AlexNet pruned with hybrid pruning
(c) 1st conv in AlexNet pruned with L1-US pruning
0.2
210
-13
(d) Last fo-layer in AlexNet
(e) Last fc-layer AlexNet
(f) Last fc-layer in
AlexNet
pruned with L pruning
pruned with hybrid pruning
pruned with LTUUS pruning
09
-02
48
19
12
(g) 1st conv in VGG-11 pruned with S pruning
(h) 1s5 conv in VGG-11 pruned
(i) 1st conv in VGG-11 pruned
with hybrid pruning
with L, US pruning
611
631
-311
15
( Last fc-layer in VGG-11 pruned with Li-S pruning
(k) Last fe-layer in VGG-11 pruned with hybrid pruning
()) Last fe-layer in VGG-11
pruned with LUS pruning
Figure 16: Weight values (y-axis) after 30 epochs of training at various consecutive sparsity levels (x-axis), for weights in the convolutional layer (first row) and last fully-connected layer (second row) in the AlexNet architecture, and the 1a convolutional layer (third row) and last fully-connected layer (fourth row) in the VGG-11 architecture (seed: 0). Each column corresponds to one of the following pruning techniques: L1 structured pruning, hybrid L, structured (in conv layers) and L unstructured (in fully-con connecco layers) pruming, and L, unstructured pruning.
19


Reviews:
[{"review_id": "r1e9ZDlfYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There are major problems with this paper. It is concerned with the examination of pruning experiments for a LeNet on the MNIST dataset.  I fail to see how anything useful can be derived from this, as MNIST is a completely trivial dataset and LeNet is a very old, small architecture which does not at all resemble the massive overparameterised models that we care about.\n\nFrom a narrative perspective, I am not sure what the key point is, what should the reader take home? What should they take account of when performing network pruning?\n\nIn terms of presentation, some of the figures are unreadable (figure 4). Figure 15 looks like noise. The writing is good however, if a bit grandiloquent.\n\nI dislike writing short reviews, but I fear this paper falls too far short of ICLR standard.\n\nPros:\n- Well written\n\nCons:\n- Experiments are weak\n- Unclear narrative; what's the one key message?\n\nI have to give this paper a reject as the experiments conducted are far too weak, and there is little evidence anything found here will, say, generalise to a ResNet/DenseNet on ImageNet.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BJeo-bA4qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nThis paper compares network pruning masks learned via different iterative pruning methods. Experiments on LeNet + MNIST show (a) different methods can achieve similar accuracy, (b) pruned sub-networks may differ significantly despite identical initialization, (c) weight reinitialization between pruning iterations yields more structured convolutional layer pruning than not reinitializing, and (d) pruning methods may differ in the stability of weights over pruning iterations.\n\n*Rating*\nThere are interesting bits of data in this paper, but the overall story is somewhat muddled and some inferences seem to be insufficiently supported by data (1-2 below). In addition, the text would benefit from better organization and presentation (3-4 below) and replications on other datasets and architectures (5 below). As a result, my rating is currently weak reject.\n\n(1) *Overlap in pruned sub-networks*: In the middle of Sec. 4, Fig 3-5 examine the similarity of pruning masks between methods. It seems clear from several of the plots that multiple methods produce identical layer-wise masks, e.g. Fig 3(a), while others show a wide variance. The overlap in lines makes this difficult to assess at times: perhaps a table would communicate it better? Also, are Fig 3-4 depicting the Jaccard distance between masks of unpruned or pruned weights? Is the ordering of training samples fixed in addition to network initialization? Is reinitialization used between iterations? Also, Fig 5 seems to contradict the conclusion that methods tend to learn different masks, since the structures are noticeably similar.\n\n(2) *Weight stability during pruning*: It is difficult to discern a conclusion in Sec 5. First, a clarification on the figures: are lines for pruned weights terminated where they are pruned? If so, this would be helpful to state. The 4th paragraph claims, \"we empirically find a correlation between weight stability and performance\", but this is not at all obvious from Figures 6-7. I'm not sure what a more stable evolution looks like. Hybrid is shown to be accurate in Fig 1, but the conv. weights in 6(a) are a spaghetti tangle and the FC weights in 7(a) are constantly increasing in magnitude. Perhaps a mathematical formulation for stability (perhaps based on average standard deviation of each weight's values over training) with a table of values for each method/layer would help to clarify.\n\n(3) *Organization*: Since the paper has many intertwined observations, a better organization would be helpful. Consider mirroring the structure of Sec 1.1 in a combined Sec. 4-5 with clear paragraph headers summarizing each conclusion.\n\n(4) *Presentation*: Figure is too small throughout to read from a printed copy (or even on a screen without significant zooming). Several results could be presented with less ambiguity in tabular form, as noted above.\n\n(5) *Replications*: The paper presents results only a single set of experiments using the MNIST dataset with the LeNet architecture. While this isn't a fatal issue, it is a significant weakness.\n\n*Notes*\nFig 1 and 2: What spacing is used for the x- and y- axes?\nFig 8: Perhaps scale vertically by the standard deviation of the weights?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeL9xKqtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper study the lottery ticket hypothesis by observing the properties of lottery tickets. In particular, the authors tested several different pruning techniques by varying evaluation criteria (L_1, L_2, L_-\\infty and random) and pruning structures (structured, unstructured and hybrid). The authors perform experiments mainly on LeNet with the MNIST dataset and analyze the observations.\n\nOverall, I think that the observations presented in the paper are not significant due to the following reasons.\n\nFirst, the paper consists of the list of observations but how the observations extend to is not clearly described. There are no guidelines how to utilize the observations in future research (e.g., how they can be used for verifying the lottery ticket hypothesis or how they affect to existing pruning techniques) while some observations might be trivial or not very interesting (e.g., contribution 1 and contribution 2) for me.\n\nSecond, the observations are only presented for LeNet and MNIST and it is non-trivial whether they extend to large scale models. The authors present VGG11 and AlexNet results in Appendix but they are not large enough to verify their hypothesis for practice. The authors mentioned that larger models are not their subject, but this significantly reduces the confidence of the observations.\n\nOther comments:\nI think that Figure 5 is not well described. Explicitly noting the meaning of color in the figure would be better.\n\nTexts in Figure 7 are too small to read.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an observational work with experiments for comparing iterative pruning methods.

I agree with the main concerns of all reviewers:

(a) Experimental setups are of too small-scale or with easy datasets, so hard to believe they would generalize for other settings, e.g., large-scale residual networks. This aspect is very important as this is an observational paper.
(b) The main take-home contribution/message is weak considering the high-standard of ICLR.

Hence, I recommend rejection. 

I would encourage the authors to consider the above concerns as it could yield a valuable contribution.
Decision: Reject