Pruning Neural Networks at Initialization: Why Are We Missing the Mark?
Authors: Jonathan Frankle
Gintare Karolina Dziugaite
Daniel Roy
Michael Carbin
Venue: ICLR 2021 Poster (2021)
Keywords: Pruning
Sparsity
Lottery Ticket
Science

Abstract: Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.

Published as a conference paper at ICLR 2021
PRUNING NEURAL NETWORKS AT INITIALIZATION: WHY ARE WE MISSING THE MARK?
Jonathan Frankle
Gintare Karolina Dziugaite Element Al
Daniel M. Roy University of Toronto Vector Institute
Michael Carbin
MIT CSAIL
MIT CSAIL
ABSTRACT
Recent work has explored the possibility of pruning neural networks at initializa-
tion. We assess proposals for doing SNIP (Lee et al. 2019), GraSP (Wang) et al. 2020p, SynFlow (Tanaka et al., 2020). and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, we find that they remain below the accuracy of magnitude pruning after training. We show that, un- like magnitude pruning after training, randomly shuffling the weights these meth- ods prune within each layer or sampling new initial values preserves or improves accuracy, As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire prune at initialization, or both.
INTRODUCTION
Since the 1980s, we have known that it possible to eliminate a significant number of parameters from neural networks without affecting accuracy at inference-time (Reed. 1993] Han et al.) 2015). Such neural network pruning can substantially reduce the computational demands of inference when conducted in a fashion amenable to hardware (Li et al. 2017) or combined with libraries (Elsen et al.
2020) and hardware designed to exploit sparsity (Cerebras) 2019 NVIDIA 2020
Toon 2020).
When the goal is to reduce inference costs, pruning often occurs late in training (Zhu & Gupta 2018
Gale et al. 2019) or after training (LeCun et al. 1990; Han et al. 2015). However, as the financial, computational, and environmental demands of training (Strubell et al., (019) have exploded, re- searchers have begun to investigate the possibility that networks can be pruned early in training or even before training. Doing so could reduce the cost of training existing models and make it possible to continue exploring the phenomena that emerge at larger scales (Brown et al. 2020).
There is reason believe may be possible to prune early in training without affecting final accu- racy, Work on the lottery licket hypothesis (Frankle & Carbin 2019; Frankle et al. 2020a) shows that, from early in training (although often after initialization). there exist subnetworks that train in isolation to full accuracy (Figure red line). These subnetworks are as small as those found by inference-focuse prospect that it may be possible to maintain this level of sparsity for much or all of training. However, this work does not suggest a way to find these subnetworks without first training the full network.
pruning methods after training (Appendix E Renda et al., (020)) raising the
The pruning literature offers starting point for finding such subnetworks efficiently. Standard networks are often so overparameterized that pruning randomly has little effect on final accuracy at lower sparsities (green line). Moreover, many existing pruning methods prune during training (Zhu Gale et al., 2019), even if they were designed with inference in mind (orange line).
& Gupta 2018:
Recently, several methods have been proposed specifically for pruning at initialization. ctt al. 2019) aims to prune weights that are least salient for the loss. GraSP (Wang et al.
SNIP (Lee)
2020 aims
to prune weights that most harm least benefit gradient flow. SynFlow (Tanaka et al.
2020 aims
to iteratively prune weights with the lowest "synaptic strengths" " in a data-independent manner with the goal of avoiding layer collapse (where pruning concentrates on certain layers).
In this paper, we assess the efficacy of these pruning methods at initialization. How do SNIP, GraSP, and SynFlow perform relative each other and naive baselines like random and magnitude pruning?

---

Published as a conference paper at ICLR 2021
Ro-le- Daippeo Wark o5 Aearcy
130
Rasim
Grodad Procicz (Gal et 2089)
11
20
Exc ol Staeas
50
70
352 Gra50 siaponedi,, Sydiher, and Miggitade hriaaizace
Lomey Taker (Fonnle ctal 2330 Rester 2000
Figure I: Weights remaining at each training step for methods that reach accuracy within one per- centage point of ResNet-50 on ImageNet. Dashed line is a result that is achieved retroactively.
How does this performance compare to methods for pruning after training? Looking ahead, are there broader challenges particular to pruning at initialization? Our purpose is to clarify the state of the art, shed light on the strengths and weaknesses of existing methods, understand their behavior in practice, set baselines, and outline an agenda for the future. We focus at and near matching sparsities: those where magnitude pruning after training matches full accuracy| We do so because: (1) these are the sparsities typically studied in the pruning literature, and (2) for magnitude pruning after training, this is a tradeoff-free regime where we do not have balance the benefits of sparsity with sacrifices in accuracy. Our experiments (summarized in Figure 2 and findings are as follows:
The state of the art for pruning initialization. The methods for pruning at initialization (SNIP, GraSP. SynFlow, and magnitude pruning) generally outperform random pruning. No single method SOTA: there a network, dataset, and sparsity where each pruning method (including magnitude pruning) reaches the highest accuracy. SNIP consistently performs well, magnitude pruning is sur- prisingly effective, and competition increases with improvements we make to GraSP and SynFlow.
Magnitude pruning after training outperforms these methods. Although this result is not nec- essarily surprising (after all, these methods have less readily available information upon which to prune), it raises the question of whether there may be broader limitations the performance achiev- able when pruning at initialization. In the rest of the paper, we study this question, investigating how these methods differ behaviorally from standard results about pruming after training.
Methods prune layers, not weights. The subnetworks that these methods produce perform equally well (or better) when randomly shuffle the weights they prune in each layer; it is therefore pos- sible to describe a family of equally effective pruning techniques that randomly prune the network in these per-layer proportions. The subnetworks that these methods produce also perform equally well when we randomly reinitialize the unpruned weights. These behaviors are not shared by state- of-the-art weight- trgunn-p methods that operate after training: both of these ablations (shuffling and reinitialization) lead to lower accuracy (Appendix F Han et al. 2015 Frankle & Carbin 2019).
These results appear specific to pruning initialization. There are two possible reasons for the comparatively lower accuracy of these methods and for the fact that the resulting networks are insen- sitive the ablations: (1) these behaviors are intrinsic to subnetworks produced by these methods or (2) these behaviors are specific to subnetworks produced by these methods at initialization. We eliminate possibility (1) by showing that using SNIP, SynFlow, and magnitude to prune the network after initialization leads to higher accuracy (Section (6) and sensitivity to the ablations (Appendix [F) This result means that these methods encounter particular difficulties when pruming at initialization.
Looking ahead. These results raise the question of whether it is generally difficult to prune at initialization in a way that is sensitive to the shuffling or reinitialization ablations. If methods that maintain their accuracy under these ablations are inherently limited in their performance, then there may be broader limits on the accuracy attainable when pruning at initialization. Even work on lottery tickets, which has the benefit of seeing the network after training, reaches lower accuracy and is unaffected by these ablations when pruning occurs at initialization (Frankle et al. 2020a).
Although accuracy improves when using SNIP, SynFlow, and magnitude after initialization, it does not match that of the full network unless pruning occurs nearly halfway into training (if at all). This means that (1) it may be difficult to prune until much later in training or (2) we need new methods designed to prune early in training (since SNIP, GraSP, and SynFlow were not intended to do so).
Tanaka et al. design SynFlow avert layer collapse, which occurs higher sparsities than consider. However, they also evaluate our sparsities, so we helieve this reasonable setting to study SynFlow.
2

---

Published as a conference paper at ICLR 2021
Method
Early Prusing Methods
Baseline Methods Magnimde (After)
Ablations
SNIP GraSP SynFlow
Magnirude
Random
ETR
OOter
Reinit Shuffle
Invert
SNIP
x
X
X
GraSP
SynPlow
Comparisons in the SNIP, GraSP, and SynFlow papers. Does not include MNIST. SNIP lacks baselines beyond MNIST. GraSP includes random, LTR, and other methods; it lacks magnitude at init and ablations. SynFlow has other methods at init but lacks baselines or ablations.
Figure 2:
2 RELATED WORK
Neural network pruning dates back to the 1980s (survey: Reed 1993). although it has seen a recent resurgence (survey: [Blllock et al. 2020). Until recently, pruning research focused on improving efficiency of inference. However, methods that gradually prune throughout training provide oppor- tunities to improve the efficiency of training as well (Zhu & Gupta 2018 Gale et al. 2019).
Lottery ticket work shows that there are subnetworks before (Frankle & Carbin 2019) or early in training (Frankle et al.] 2020a) that can reach full accuracy. Several recent papers have proposed efficient ways find subnetworks at initialization that can train to high accuracy. SNIP (Lee al.
2019), GraSP (Wang et al. 2020), SynFlow (Tanaka et al. (2020), and NTT (Liu & Zenke 2020) prune initialization. De Jorge et al. (2020) Verdenius et al. (2020) apply SNIP iteratively; Cho et al. (2020) use SNIP for pruning for inference. Work on dynamic sparsity maintains a pruned net- work throughout training but regularly changes the sparsity pattern (Mocanu al. [018 Dettmers & Zettlemoyer, 2019; [Evci et al.) 2019); to match the accuracy of standard methods for pruning after training. EEci et al. needed to train for five times as long is standard for training the unpruned networks. You et al. (2020) prune after some training; this research is not directly comparable to any of the aforementioned papers as it prunes channels (rather than weights as in all work above) and does so later (20 epochs) than SNIP/GraSP/SynFlow (0) and lottery tickets (1-2).
3 METHODS
Pruning. Consider a network with weights wy E Rede in each layer E {1,..., D}. Pruning pro- duces binary masks m {0, 1)d A pruned subnetwork has weights we C me, where @ is the element-wise product. The sparsity E [0, 1] of the subnetwork is the fraction of weights pruned: E. me/> De de. We study pruning methods (rune(W,s) s) that prune to sparsity using two operations. First, score(W') issues scores E Rd to all weights W = (WI...., WL)) Second, remove(Z, Z, s) converts scores Z (z1..... 21) into masks my with overall sparsity s. Pruning may occur in one shot (score once and prune from sparsity 0 to s) or iteratively (repeatedly score un- pruned weights and prune from sparsity- s " to 8 2 over iterations n E [1.....N]).
Re-training after pruning. After pruning at step t of training, we subsequently train the network further by repeating the entire learning rate schedule from the start (Renda et al.) 2020). Doing so ensures that, no matter the value of f, the pruned network will receive enough training to converge.
Early pruning methods. We study the following methods for pruning early training.
Random. This method issues each weight random score žς Uniform(O, 1) and removes weights with the lowest scores. Empirically, prunes each layer to approximately sparsity 8. Random pruning is a naive method for early pruning whose performance any new proposal should surpass.
Magnitude. This method issues each weight its magnitude ,, = |wz| as its score and removes those with the lowest scores. Magnitude pruning is a standard way to prune after training for inference Janowsky|[188 Han et al., 2015) and is an additional naive point of comparison for early pruning.
SNIP LLe et al. 2019). This method samples training data, computes gradients ge for each layer, issues scores žạ = 1gt O wyl, and removes weights with the lowest scores in one iteration. The justification for this method is that it preserves weights with the highest "effect the loss (either positive or negative)." For full details, see Appendix B In Appendix G we consider a recently-
proposed iterative variant of SNIP de Jorge et al.; 2020 Verdenius et al. 2020).
GraSP Wang et al. 2020). This method samples training data, computes the Hessian- -radieent product by for each layer, issues scores zạ wuy hy, and removes weights with the highest scores
3

---

Published as a conference paper at ICLR 2021
in one iteration. The justification for this method is that it removes weights that "reduce gradient flow" and preserves weights that "increase gradient flow." For full details, see Appendix C
SynFlow (Tanaka et al. 2020). This method replaces the weights we with |we]. It computes the sum R of the logits on an imput of I's and the gradients dft of R. It issues scores 24 = 1 removes weights with the lowest scores. It prunes iteratively (100 iterations). The justification for this method is that it meets criteria that ensure (as proved by Tanaka et al.) it can reach the maximal sparsity before a layer must become disconnected. For full details, see Appendix D
_dwg @ wel and
ait
dun
Benchmark methods. We use two benchmark methods to illustrate the highest known accuracy attainable when pruning to particular sparsity in general (not just at initialization). Both methods reach similar accuracy and match full accuracy at the same sparsities (Appendix E Note: we use one-shot pruning, so accuracy is lower than in work that uses iterative pruning. We do so to make a fair comparison to the early pruning methods, which do not get to train between iterations.
Magnitude pruning after training. This baseline applies magnitude pruning to the weights at the end of training. Magnitude pruning is standard method for one-shot pruning after training (Renda) et al. 2020). We compare the early pruning methods at initialization against this baseline.
Lottery ticket rewinding (I.TR). This baseline uses the mask from magnitude pruning after training and the weights from step t. Frrankle al. (2020a) show that, for early in training and appropriate sparsities, these subnetworks reach full accuracy. This baseline emulates pruning at step t with an oracle with information from after training. Accuracy improves for t > O, saturating early in training (Figure 6 blue). We compare the early pruning methods after initialization against this baseline.
Sparsities. We divide sparsities into three ranges (Frankle et al., 2020a). Trivial sparsities are the lowest sparsities: those where the network is so overparameterized that randomly pruning at initialization can still reach full accuracy. Matching sparsities are moderate sparsities: those where the benchmark methods can match the accuracy of the unpruned network. Extreme sparsities those beyond. We focus on matching sparsities and the lowest extreme sparsities. Trivial sparsities are addressed by random pruning. Extreme sparsities require making subjective or context-specific tradeoffs between potential efficiency improvements of sparsity and severe drops in accuracy.
Networks, datasets, and replicates. We study image classification. It is the main (SNIP) or sole (GraSP and SynFlow) task in the papers introducing the early pruning methods and in the papers introducing modern magnitude pruning (Han et al.) 2015) and LTR (Frankle et al. [2220a). We use ResNet-20 and VGG-16 on CIFAR- 10, ResNet- 18 on Tiny ImageNet, and ResNet-50 on ImageNet. See Appendix A for hyperparameters. We repeat experiments five (CIFAR- 10) or three (Tinylma- geNet and ImageNet) times with different seeds and plot the mean and standard deviation.
PRUNING AT INITIALIZATION
In this section, we evaluate the early pruning methods at initialization. Figure shows the perfor- mance of magnitude pruning (green), SNIP (red), GraSP (purple), and SynFlow (brown) at initial- ization. For context, it also includes the accuracy of pruning after training (blue), random pruning initialization (orange), and the unpruned network (gray).
Matching sparsities. For matching sparsities SNIP, SynFlow and magnitude dominate depending on the network. On ResNet-20, all methods are similar but magnitude reaches the highest accuracy. On VGG-16, SynFlow slightly outperforms SNIP until 91.4% sparsity, after which SNIP overtakes it; magniudde and GraSP are at most 0.4 and 0.9 percentage points below the best method. On 18, SNIP and SynFlow remain even until 79.0% sparsity, after which SNIP dominates; magnitude and GraSP are at most 2.1 and 2.6 percentage points below the best method. On ResNet- 50, SNIP, SynFlow, and magnitude perform similarly, 0.5 to percentage points above GraSP.
DSSIP,
In some cases, methods are able to reach full accuracy at non-trivial sparsities. On VGG-16, SNIP and SynFlow do so until 59% sparsity (vs. 20% for random and 93.1 % for magnitude after training). On ResNet-18. 18, SNIP does so until 89.3% sparsity and SynFlow until 79% sparsity (vs. 20% for random and 96.5% for magnitude after training). On ResNet-20 and ResNet-50, no early pruning
Matching sparsities are those where magnitude pruning after training matches full accuracy. These are sparsities < 73.8%, 93.14 %., 96.5%, and 67.2% for ResNet-20, VGG-16, ResNet- 18, and ResNet-50.
4

---

Published as a conference paper at ICLR 2021
Uupcuned Network
Magnitude After Training
Random
Magnitude
SNIP
GrSPP
SynElow
Spaass 29)
Figure 3: Accuracy early pruning methods when pruning at initialization to various sparsities.
methods reach full accuracy non-trivial sparsities; one possible explanation for this behavior is that these settings are more challenging and less overparameterized in the sense that magnitude pruning after training drops below full accuracy at lower sparsities.
Extreme sparsities. At extreme sparsities the ordering of methods is the same on VGG-16 but changes on the ResNets. On ResNet-20,, magnitude and SNIP drop off, GraSP overtakes the other methods at 98.2% sparsity, and SynFlow performs worst; ResNet-50 shows similar behavior except that SynFlow performs best. On ResNet- 18, GraSP overtakes magnitude and SynFlow but not SNIP.
Summary. No one method is SOTA in all settings and sparsities. SNIP consistently performs well, with SynFlow frequently competitive. Magnitude is surprisingly effective against more compli- cated heuristies. GraSP performs worst matching sparsities but does better extreme sparsities. Overall, the methods generally outperform random pruning; however, they cannot match magnitude pruning after training in terms of either accuracy or the sparsities at which they match full accuracy.
5 ABLATIONS AT INITIALIZATION
In this section, we evaluate the information that each method extracts about the network at initial- ization in the process of pruning. Our goal to understand how these methods behave in practice and gain insight into why they perform differently than magnitude pruning after training.
Randomly shuffling. We first consider whether these pruning methods prune specific connections. To do so, we randomly shuffle the pruning mask my within each layer. If accuracy is the same after shuffling, then the per-weight decisions made by the method can replaced by the per-layer fraction of weights it pruned. If accuracy changes, then the method has determined which parts of the network to prune at a smaller granularity than layers, c.g., neurons or individual connections.
Overall. All methods maintain accuracy or improve when randomly shuffled (Figure 4 orange line). In other words, the useful information these techniques extract is not which individual weights to re- move, but rather the layerwise proportions by which to prune the network Although layerwise pro- portions are an important hyperparameter for inference-focuseee Gale al., 2019), proportions alone are not sufficient explain the performance of those methods. For example, as the bottom row of Figure Ssoows, shows, magnitude pruning after training makes pruning decisions specific particular weights; shuffling in this manner reduces performance (Frankle & Carbin 2019). In general, we know of no state-of-the-att weight-pruning methods (among the many that exist (Blalock al. 2020)) with accuracy robust to this ablation. This raises the question of whether the insensitivity to shuffling for the early pruning methods may limit their performance.
pruning methods (He et al.) 2018
Extreme sparsities those beyond which magnitude pruning after training reaches full accuracy. "Note that there are certainly pathological cases where these proportions alone are not sufficient to match the accuracy of the pruning techniques. For example, at the extreme sparsities studied by Tanaka et al. (2020). pruning randomly could lead the network to hecome disconnected. An unlucky random draw could also lead the network to become disconnected. However, we do observe these behaviors in any of experiments.
5

---

Published as a conference paper at ICLR 2021
Magaltude
SNIP
GraSp
20
SynFlow
85 122
00 25 560
200
200
100
Magulude After
Figure 4: Ablations subnetworks found by applying magnitude pruning, SNIP, GraSP, and Syn- Flow at initialization. (We ran limited ablations on ResNet-50 due to resource limitations.)
Magnitude pruming. Since the magnitude pruning masks can be shuffled within each layer, its prun- ing decisions sensitive only the per-layer initialization distributions. These distributions chosen using He initialization: normal with per-layer variance determined by the fan-in or fan-out (He al. 2015). These variances alone, then, are sufficient information to attain the performance of magnitude couning-performance often competitive with SNIP, GraSP, and SynFlow. Without this information, magnitude pruning performs worse (purple line): each layer is initialized with variance 1, it will prune all layers by the same fraction no differently than random pruning. This does not affect SNIPf GraSP, or SynFlow, showing a previously unknown benefit of these methods: they maintain accuracy in a case where the initialization is not informative for pruning in this way.
SynFlow. On ResNet-20 and ResNet- 18, SynFlow accuracy improves at extreme sparsities when shuffling. We connect this behavior pathology of SynFlow that we term neuron collapse: Syn- Flow prunes entire neurons (in this case, convolutional channels) at a higher rate than other methods (Figure 5) At the highest matching sparsities, SynFlow prunes 31 %, 52%, 69%, and 29% of neu- rons on 20, VGG-16, and ResNet-18, and ResNet-50. In contrast, SNIP prunes 5%, 11%, 32%, and 7%: GraSP prunes 1%, 6%, 14%, and 1 1%; and magnitude prunes 0%, 0%, 1%, and 0%. Shuffling SynFlow layerwise reduces these numbers to 1%, 0%, 3.5%, and 1 (orange line).
We believe neuron collapse inherent to SynFlow. From another angle, SynFlow works follows: consider all paths p (w/²) (e) } from any input node to any output node. The SynFlow gradient dit for weight is the sum of the products Ie Ie |upe) ( | of the magnitudes on all paths containing w. This
dw
the weight's 's contribution to the network's 's (f1,1) path norm (Neyshabur et al. (2155), connection we demonstrate in Appendix D.S. Once an outgoing (or incoming) weight pruned from a neuron, all dit incoming (or outgoing) weights are in fewer paths, decreasing des they are more likely to be pruned on the next iteration, potentially creating a vicious cycle that prunes the entire neuron. Similarly, SynFlow heavily prunes skip connection weights, which participate in fewer paths (Appendix
Reinitialization. We next consider whether the networks produced by these methods are sensitive to the specific initial values of their weights. That is, is performance maintained when sampling new
$ Note that initialization still affect the performance of SNIP. In particular, Lec 2020) demonstrate a scheme for adjusting the initialization of a SNIP-pruned network after pruning that leads to higher accuracy. "This number remains high for ResNet-50 because nearly half of the pruned neurons are in layers that get pruned entirely, specifically skip connections that downsample using 1xl convolutions (see Appendix D).

---

Published as a conference paper at ICLR 2021
Dedic Sea AO
Fecher (mgple Nac.eeeeeeee--eee 1007
Mrgahooe
SNIP
Go5P
Syfoow
Sysflow (Str0le) Ligersie))
Figure 5: Percent of neurons (conv. channels) with sparsity 2 5% at the highest matching sparsity.
VGG Sori--FTAa... 1D 90.5 quadly
Lobey
Systkm
Figure 6: Accuracy of early pruning methods when pruning at the iteration on the x-axis. Sparsities are the highest matching sparsities, LTR prunes after training and initializes to the weights from the specified iteration. Vertical lines are iterations where the learning rate drops by 10x.
initialization for the pruned network from the same distribution as the original network? Magnitude pruning after training and LTR are known to be sensitive this ablation: when reinitialized, pruned networks train to lower accuracy (Figure 4 bottom row; Appendix F Han al. 2015 Frankle & Carbin 2019). and we know of no state-of-the-ar weight-pruning methods with accuracy robust to this ablation. However, all early pruning techniques line): accuracy is the same whether the network is trained with the original initialization or a newly sampled initialization. As with random shuffling, this raises the question of whether this insensitivity to initialization may pose limit to these methods that restricts performance.
unaffected by reinitialization (green
Inversion. SNIP, GraSP, and SynFlow are each based on hypothesis about properties of the net- work or training that allow a sparse network to reach high accuracy. Scoring functions should rank weights from most important to least important according to these hypotheses, making it possible to preserve the most important weights when pruning to any sparsity. In this ablation, we assess whether the scoring functions successfully do so: we prune the most important weights and retain the least important weights. If the hypotheses behind these methods are correct and they are accu- rately instantiated as scoring functions, then inverting this manner should lower performance.
Magnitude, SNIP, and SynFlow behave as expected: when pruning the most important weights, accuracy decreases (red line). In contrast, GraSP's accuracy does not change when pruning the most important weights. This result calls into question the premise behind GraSP's s heuristic: one can keep the weights that, according to Wang et al. (2020). decrease gradient flow the most and get the same accuracy as keeping those that purportedly increase it the most. Moreover, we find that pruning weights with the mowest-magnitude GraSP scores improves accuracy (Appendix H..
Summary. When using the methods for pruning initialization, it is possible reinitialize layerwise shuffle the unpruned weights without hurting accuracy. This suggests that the useful part of the pruning decisions of SNIP, GraSP. SynFlow, and magnitude initialization are the layerwise proportions rather than the specific weights or values. In the broader pruning literature, shuffling and reinitialization are associated with lower accuracy, and-as an example-we show that this is the case for magnitude pruning after training. Since these ablations do not hurt the accuracy of the methods for pruning at initialization, this raises the question of whether these methods may be confined to this lower stratum of performance.
This behavior under the ablations also raises questions about the reasons why SNIP, GraSP, and SynFlow are able to reach the accuracies in Section 4 Each paper poses hypothesis about the qualities that would allow a pruned network to train effectively and derives a heuristic to determine which specific weights should be pruned based on this hypothesis. In light of our ablations, it is unclear whether the results achieved by these methods are attributable to these hypotheses. For example, SNIP aims to "identify important connections" (Lee et 2019); however, accuracy does not change if the pruned weights are randomly shuffled. SynFlow focuses on paths, "taking the layer-layer interactions of parameters into account" (Tanaka al. 2020); accuracy improves in some
7

---

Published as a conference paper at ICLR 2021
cases if we discard path information by shuffling and is maintained if we alter the "synaptic strengths flowing through each parameter" by reinitializing. In addition to these concerns, GraSP performs identically when inverted. Future early pruning research should use these and other ablations to evaluate whether the proposed heuristics behave according to the claimed justifications.
6 PRUNING AFTER INITIALIZATION
In Section 4 we showed that pruning at initialization leads to lower accuracy than magnitude pruning after training. In Section | we showed that this accuracy is invariant to ablations that hurt the accuracy of magnitude pruning after training. In this section, we seek distinguish whether these behaviors are (1) intrinsic to the pruning methods or (2) specific to using the pruning methods initialization. We have already shown evidence in support of (2) for magnitude pruning: the subnetworks it finds are less accurate and maintain their accuracy under shuffling and reinitialization at initialization but not when pruning after training (Figure in Section Moreover, LTR performs best when pruning early in training rather than at initialization (Frankle et al. (020a), potentially pointing to broader difficulties specific to pruning at initialization.
To eliminate possibility (1), we need to demonstrate that there are circumstances where these meth- ods reach higher accuracy than they do when pruning at initialization and where they are sensitive to the ablations. We do so by using these methods to prune later in training: we train for 1: iterations, prune using each technique, and then train further for the entire learning rate schedule (Renda et al.) 2020 If accuracy improves when pruning at iteration l >> and the methods become sensitive to the ablations, then we know that the behaviors we observe in Sections and are not intrinsic to the methods. We note that SNIP, GraSP, and SynFlow were not designed to prune after initialization; as such, we focus on whether accuracy improves rather than the specific accuracy itself.
Figure 6 shows the effect of this experiment on accuracy. We include random pruning baseline and L.TR is the best accuracy we know to be possible when applying a pruning mask early in training. Random pruning also provides control to show that pruning after the network has trained for longer (and has reached higher accuracy) does not necessarily cause the pruned network to reach higher accuracy: no matter when in training we randomly prune, accuracy the same.
lower
Magnitude, SNIP, and SynFlow improve as training progresses, with magnitude and SNIP approach- ing LTR and SynFlow close behind. This means that the performance gap in Section is not intrinsic to these methods, but rather is due to using them at initialization. Moreover, in Appendix(F] we show that, as accuracy improves, the subnetworks are increasingly sensitive to the ablations, further evi- dence that these behaviors are specific to initialization.
Interestingly, LTR reaches higher accuracy than the pruning methods early in training: magnitude pruning does not match the accuracy of LTR at iterations IK, 2K, and 1K until iterations 25K, 26K, and 36K on R2sNet-20, VGG-16, 16, and ResNet-18. If this gap in accuracy reflects a broader challenge inherent to pruming early in training, then these results suggest it may be difficult to prune, not just at initialization, but for large period after.
7 DISCUSSION
The state of the art. We establish the following findings about pruning at initialization.
Surpassing random pruning. All methods surpass random pruning at some or all matching sparsities, and, in certain settings, some methods maintain full accuracy at non-trivial sparsities.
No single method is SOTA at initialization. Depending on the network, dataset, and sparsity, there setting where each early pruning method reaches the highest accuracy. Our enhancements to GraSP (Figure 13 in Appendix Hy and SynFlow (Figure further tighten the competition.
Data is not currently essential al initialization. SynFlow and magnitude pruning are competitive at initialization without using any training data. Like robustness to shuffling and reinitialization, however, data-independence may only be possible for the limited performance of current methods. In contrast, magnitude pruning after training and LTR rely on data for both pruning and initialization.
Due the expense of this experiment, we examine single sparsity: the most extreme matching sparsity.
8

---

Published as a conference paper at ICLR 2021
Below the performance of pruning after training. All methods for pruning at initialization reach lower accuracy than magnitude pruning after training.
The challenge of pruning at initialization. It is striking that methods that use such different signals (magnitudes; gradients; Hessian; data or lack thereof) reach similar accuracy, behave similarly under ablations, and improve similarly (except GraSP) when pruning after initialization.
Ablations. None of the methods examine are sensitive to the specific weights that are pruned or their specific values when pruning at initialization. When pruning after training, these ablations are associated with lower accuracy. This contrast raises the question of whether methods that maintain accuracy under these ablations are inherently limited in the accuracy they can reach, and whether this is property of these methods or of pruning at initialization in general. It may yet be possible to design a method that reaches higher accuracy and is sensitive to these ablations at initialization, but it may also be that the failures of four methods to do so (and their improvements after initialization in (action[oand Appendix F point to broader challenges in pruning at initialization.
Why is it challenging? We do not identify a cause for why these methods struggle to prune in a specific fashion at initialization (and often only at initialization), and we believe that this is an important question for future work. Perhaps there are properties of optimization that make pruning specific weights difficult or impossible at initialization (Evci et al. 2019). For example, training occurs in multiple phases (Gur-Ari al., 2018 Jastrzebski et al., 2020 Frankle et al.) 2020b); perhaps it is challenging to prune during this initial phase. Further study of this question may reveal whether this challenge related to shared characteristics of these methods or whether it is intrinsic to pruning at initialization.
Looking ahead. We close by discussing the implications of our findings for future pruning research.
Why prune early in training? There many reasons to do so. This includes the scientific goals of studying the capacity needed for learning (Frankle & Carbin 2019) and the information needed to prune (Tankka et al. 2020) and the practical goals of redueing the cost of finding pruned networks for inference (Lee et al. (011)) and "saving resources at training time" (Wang et al. 2020).
We focus on the practical goal of reducing the cost of training, since our results show that pruning at initialization is not drop-in replacement for pruning after training. We find that existing methods for pruning at initialization require making tradeoff: reducing the cost of training entails sacrificing some amount of accuracy. Looking ahead, we believe the most compelling next step for pruning research is developing tradeoff-frre free ways reduce the cost of training.
Pruning after initialization. In Section 6 we show that SNIP. SynFlow, and magnitude improve gradually after initialization but LTR improves much faster. However, these methods were designed for initialization; focusing early in training may require new approaches. Alternatively, it may be that, even at iterations where LTR succeeds in Section O the readily available information is not sufficient reach this performance without consulting the state of the network after training. One way to avoid this challenge is to dynamically change the mask to exploit signals from later in training (Mocanu et al. 2018 Dettmers& & Zettlemoyer 2019 Evci et al. 2020)..
New signals for pruming. It may be possible to prune at initialization or early in training, but signals like magnitudes and gradients (which suffice late in training) may not be effective. Are there differ- ent signals we should use early in training? Should we expect signals that work early in training to work late in training (or vice versa)? For example, second order information should behave differ- ently at initialization and convergence, which may explain why GraSP struggles after initialization.
Measuring progress. We typically evaluate pruning methods by comparing their accuracies at certain sparsities. In the future, we will need to extend this framework to account for tradeoffs in differ- ent parts of the design space. At initialization, against decreases in accuracy. This is especially important for methods like SynFlow and FORCE (de Jorge al. 2020), which are designed to maintain diminished but non-random accuracy the most extreme sparsities. In this paper, we defer this tradeoff by focusing on matching sparsities.
must weigh the benefits of extreme sparsities
When pruning after initialization, we will need to address an additional challenge: method that prunes to sparsity s at step against a method that prunes to sparsity s' > s at step t' > t. To do so, we will need to measure overall training cost. That might include measuring the area under the curve in Figure I. FLOPs (as, e.g., and energy consumption software and hardware optimized for pruned neural networks.
comparing a
e.g.,Evvo Evei et al. (2019) do), or real-world training time

---

Published as a conference paper at ICLR 2021
REFERENCES
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Proceedings of Machine Learning and Systems 2020, pp. 129-146. 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint vrXiv:2005.14100 2020.
Cerebras.
Cerebras wafer scale engine: An introduction, 2019.
URL
www http://:ww...aaaa...e..//////aaa
Cerebras- -aw-ass-Waaarr--aalaaae 6oz.0224444aace------------1----
pdf
Minsu Cho, Ameya Joshi, and Chinmay Hegde. Espn: Extremely sparse pruned networks. arXiv
preprint arXiv:2006. 157220066.771,, 2020.
Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006. 09081, 2020.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 1x7..:/90.008844 2019.
Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14629-14638, 2020.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv:1906. 10732, 2019.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, 2020a.
Jonathan Frankle, David Schwab, and Ari S. Morcos. The early phase of neural network training. In International Conference on Learning Representations, 2020b.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019.
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv: 1812.0475, 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing luvan-Level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yihui He. Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li. and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 784-800, 2018.
10

---

Published as a conference paper at ICLR 2021
Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, 39(12)66000, 1989.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho and Krzysztof Geras The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Tom. A signal prop- agation perspective for pruning neural networks at initialization. In International Confer- wttpeeiiwwwweeeeeeeeeeeeeeeeeeee fertmforrm?i
ence on Learning Representations, 2020. URL
FFFFF
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017.
Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent trans- fer. In International Conference Machine Learning. 2020.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1111, 2018.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
NVIDIA. Nvidia a100 tensor core gpu architecture, 2020.
URL
https:
7/www. wwtp://www..e//oooooteaaaaaaaeea
nvid wWW.l--mmeer--aaclllotttete-oopo
Russell Reed. Pruning algorithms-a survey. IEEE transactions on Neural Networks, 4(5):740-747,
1993.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650 2019.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05440 2020.
Nigel Toon.
Introducing
2nd
generation ipu systems for
at
scale, 2020.
https: h/tw://wwwwww.... www.ggraphcore.cco iicirreaiiiii ai/posts/ introducing second generation ipu systems for ai scale
URL
Stijn Verdenius, Maarten Stol, and Patrick Forré. Pruning via iterative ranking of sensitivity statis- tics. arXiv preprint arXiv:2006.00890 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by
preserving gradient flow. In International Conference on Learning Representations, 2020.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing carly-bird tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations, 2020.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07146 2016.
Michael H. Zhu and Suyog Gupta. To prune, or not to prume: Exploring the efficacy of pruning for model compression, 2018. URL
wwwp://wwwwweeeteeteeeeeeo d=$1 IN69AT-

---

Published as a conference paper at ICLR 2021
CONTENTS OF THE APPENDICES
Appendix
A
The details of the networks, datasets, and hyperparameters we use in our experiments.
Appendix
B
Details of our replication of SNIP.
Appendix
Details of our replication of GraSP.
Appendix
D Details of our replication of SynFlow.
Appendix Plots the baselines: random pruning, magnitude pruning at initialization, magnitude pruning after training, and LTR.
Appendix The shuffling and reinitialization ablations from Section conducted when the pruning methods are applied after training.
Appendix The result of performing SNIP iteratively over 100 iterations rather than one shot.
The three different variants of GraSP we study: pruning the weights with the lowest scores, highest scores, and magent-m scores.
Appendix
Appendix 1 comparison of the early pruning methods including our improvements GraSP and SynFlow.
Appendix The layerwise proportions in which each methods prunes the networks.
Appendix K
Experiments from the main body and the appendices on the LeNet-300-100 fully- connected network for MNIST.
Appendix L Experiments from the main body and the appendices on a modified version of ResNet- 18 based on the network used by Tanaka al. (2020) in the SynFlow paper.
Appendix ??. The ICLR 2021 concurrent work policy.
Appendix M An analysis of the effective sparsities of the networks when taking into account weights that are disconnected but that have not explicitly been pruned.
12

---

Published as a conference paper at ICLR 2021
A
NETWORKS, DATASETS, AND TRAINING
We use the following combinations of networks and datasets for image classification:
Network
Dataset
Appears
Notes
LeNei-300-100
Appendix
shuffling ablation fully- sonnected network.
ResNet-20
CIFAR-10
Main Body
VGG-16
CIFAR-10
Main Body
ResNet-18
TinyImageNet
Main Body
Madined ResNet-
Medified ImageNNet
Appendix
To match the ResNet. ISTnnelageeNe experiment in the SynFlow paper.
ResNet-50 50
IntgeNet
Main Body
A.1 NETWORKS
The networks are designed as follows:
LeNet-330-100 is a fully-connected network with two hidden layers for MNIST. The first hidden layer has 300 units and the second hidden layer has 100 units. The network has ReLU activations.
ResNer-20 is the CIFAR-IO version ResNet with 20 layers as designed by He et al. (2016). We place batch normalization prior to activations. We use the ResNet-20 implementation from the
OpenLTH repository
VGG-16 16 is CIFAR-10 network described by Lee et al. (2019). The first two layers have 64 channels followed by 2x2 max pooling; the next two layers have 128 channels followed by 2x2 max pooling; the next three layers have 256 channels followed by 2x2 max pooling; the next three layers have 512 channels followed by max pooling; the final three layers have 512 channels. Each channel uses 3x3 convolutional filters. VGG-16 has batch normalization before each ReLU activation. We use the VGG-16 implementation from the OpenLTH repository.
ResNet- 18 and ResNet-50 are the ImageNet version of ResNet with 18 and 50 layers as designed byHHe al. (2016). We use the Torch Vision implementations of these networks.
Modified ResNet-18 is a modified version of ResNet-18 designed to match the version used by Tanaka et al. 2020) in the SynFlow paper[] This version has been modified specifically for IinelmageNet: the first convolution has filter size 3x3 (rather than 7x7) and the max-pooling layer that follows has been eliminated. For more discussion, see Appendix D
A.2 DATASETS
CIFAR-10 augmented by normalizing per-channel, randomly flipping horizontally, and ran- domly shifting by up to four pixels in any direction.
TinylmageNet is augmented by normalizing per channel, selecting a patch with random as- pect ratio between 0.8 and 1.25 and random scale between 0.1 and 1, cropping 64x64, and randomly flipping horizontally.
Modified TinylmageNet is augmented by normalizing per channel, randomly flipping horizon- tally, and randomly shifting by up four pixels any direction.
ImageNet is augmented by normalizing per channel, selecting patch with random aspect ratio between 0.8 and 1,25 and a random scale between 0.1 and 1, cropping to 224x224, and randomly flipping horizontally.
A.3 TRAINING
Natwook
Dateett
Fpochs
Racc
Opi.
Mom
[JR
Drup
Wright Dony
Initilication
Bars Fp
Rexind Ber
LaNet
SINIST
40
128
SCDb
0.1
Kaining Normal
469
Ro
CIFAR-13
150
9000
epocts
Kaining Nermal
1000)
CIFAR-1D
opocis
Kaining
ResNa-11
spocha
Moditted ResNes
Modileed TipplnsgeNe::
503
0.2
Kaining Noreal
1000
RooNas-00
ImageNa
90
1024
90D
0.9
0.4
spedoo Mi, 90
1o-4
Kaiming Nermal
[35]
6355
1th htaps//ttttttt///ttttttt--t
13

---

Published as a conference paper at ICLR 2021
B REPLICATING SNIP
In this Appendix, we describe and evaluate our replication of SNIP (Lee et al. 2019)
B. ALGORITHM
SNIP introduces virtual parameter of E 0, as coefficient for each parameter wy. Initially, SNIP assumes that = 1.
SNIP assigns each parameter w; a score 8; 8eg
and prunes the parameters with the lowest scores.
This algorithm entails three key design choices:
1. Using the derivative of the loss with respect to as a basis for scoring.
2. Taking the absolute value of this derivative.
3. Pruning weights with the lowest scores.
Lee al. (2019) explain these choices follows: "if the magnitude of the derivative is high (re- gardless of the sign), it essentially means that the connection o; has a considerable effect on the loss (either positive negative) and it has to be preserved allow learning wy."
B.2 IMPLEMENTATION DETAILS
Algorithm. We can rewrite the score as follows. Let ag be the incoming activation that is multiplied by wt. Let be the pre-activation of the neuron which wi serves an input.
0L 0c;
0L 0z
aL
ØL 0z
OL
=
0, =
Wi =
02 dc,
Oz
Oz Ow;
Ow:
In summary, we can rewrite as the gradient of w; multiplied by the value of w,. This is the formula use in implementation.
Selecting examples for SNIP. Lee et al. 2019 use single mini-batch for SNIP. To create the mini- batch that use for SNIP, we follow the strategy used Wang al. (2020) for GraSP: create mini-batch composed of ten examples selected randomly from each class.
Reinitializing, After pruning, Lee al. (2099) reinitialize the network We do not reinitialize.
Running on CPU. To avoid any risk that distributed training might affect results, we run all SNIP computation CPU and subsequently train the pruned network TPU.
10see discussion ReenReview.
14

---

Published as a conference paper at ICLR 2021
B.3 NETWORKS AND DATASETS
Lee et al. consider the following networks for computer vision:
SNIP Name
Our Name
Dataset
Gitllu Replicated Notes
LeNet-300-100 LeNet-300-100
MNIST
X
Fully-connected
LeNet-5
MNIST
Convolutional
AlexNet-s
CIFAR-10
AlexNet-b
CIFAR-10 10
VGG-C
CIFAR-10 10
X
VGG-D D but some layers have 1xl convolutions VGG-like but with two fully-connected layers VGG-D but with fully-connected layer
VGG-D
VGG2-16
CIFAR-10
VGG-like
VGG-16
CIFAR-10
WRN-16-8
WRN-14-8
CIFAR-10 10
WRN-16-10
WRN-14-10
CIFAR-10
WRN-22-8
WRN-20-8
CIFAR-10
Table The networks and datasets examined in the SNIP paper (Lee et al. 2019).
B.4 RESULTS
GitHub repository associated with the papet" includes all of those networks except for the wide ResNets (WRNs). We have made our best effort to replicate subset of these networks in our research framework. Table shows the results from our replication. Each of our numbers the average across five replicates with different random seeds.
Name
Unpruned Accuracy Reported
Sparsity
Pruned Accuracy
Ours
Reported
Ours
VGG-16
91.7%
93.6%
97%
92.0% (+0.3)
92.1% (-1.5)
VGG2-16
93.2%
93.5%
95%
92.9% (-0.3)
92.3% 1.2)
WRN-14-8
93.8%
95.2%
95%
93.4% (-0.4)
93.4% 1.8)
WRN-14-10
94.1%
95.4%
95%
93.6% 0..5)
93.9%
(- 1..4)
WRN-20-8
93.9%
95.6%
95%
94.1% % (+-0.3)
94.3% (-1.2)
Table 2: The performance of SNIP as reported in the original paper and in our reimplementation.
Unprumed networks. The unpruned networks implemented by Lee et al. (019)) appear poorly tuned such that they do not achieve standard performance levels. The accuracy of our VGG-16 is 1.9 percentage points higher, and the accuracies of our wide ResNets are between 1.3 and 1.7 percentage points higher. In general, implementation details of VGG-style networks for CIFAR-10 vary widely (Blalock et al., 2020), so some differences are to be expected. However, ResNets for CIFAR-10 10 are standardized (He et al. 2016) Zagoruyko & Komodakis (2166), and our accuracies are identical to those reported by Zagoruyko & Komodakis) in the paper that introduced wide ResNets.
Pruned networks. After applying SNIP, our accuracies more closely match those reported in the paper. However, since our networks started at higher accuracies, these values represent much larger drops in performance than reported in the original paper. Overall, our results after applying SNIP appear to match those reported in the paper, giving us some confidence that our implementation is correct. However, since the accuracies of the unpruned networks SNIP are lower than standard values, it is difficult to say for sure.
B.5 RESULTS FROM GRASP PAPER
The paper that introduces GraSP WWag al. 2020 also replicates SNIP In Table compare their reported results with ours on the networks described in Table 4 in Appendix C
" "htpps: / //qi thub. com nommonamhon ee/ ininnniin pniipppublc/ Athoughh Wang et al. (2020) have released open-source implementation of GraSP, this code does not include their implementation of SNIP. We not certain which hyperparameters they used for SNIP.
15

---

Published as a conference paper at ICLR 2021
Name
Unpruned Accuracy Reported
Results
Ours
Sparsity
Reported
Ours
90%
93.6% (-0.6)
93.5% (-0.0)
VGG-19
94.2%
93.5%
95%
93.4% (-0.8)
93.4% (-0.1)
98%
92.1% (-2.1)
diverged
90%
92.6% (-2.2)
92.5% (-2.0)
WRN-32-2
94.8%
94.5%
95%
91.1% (-3.7)
91.0% (-3.5)
98%
87.5% (-7.3)
87.7% (-6.8)
60%
74.0% (-1.7)
73.9% (-2.3)
5e
75.7%
76.2%
80%
69.7% (-6.0)
71.2% (-5.0)
90%
62.0% (-13.7)
65.7% (-10.5)
Table 3: The performance of SNIP as reported in the original paper and in our reimplementation.
Unpruned networks. See Appendix C.4 for a full discussion the unpruned networks. Our VGG- 19, WRN-32-2, and ResNet-50 reach slightly different accuracy than those of Wang et al. 2020) but the performance differences are much smaller than for the unpruned networks Table 2
Pruned networks. Although performance of our unpruned VGG-19 network is lower than that of Wang et al., the SNIP performances are identical at 90% and 95% sparsity. This outcome similar our results in Appendix B.4 where we found similar pruned performances despite the fact that unpruned performances differed. At 98% sparsity on VGG-19, three of our five runs diverged.
On ResNet-50 for ImageNet, unpruned and SNIP accuracies higher than those reported in by (Wang et al., 2020). This may be a result of different hyperparameter choices for training the network; see Appendix C.3 for full details. This may also be result of different hyperparameter choices for SNIP. We may use a different number of examples than Wang et al. to compute the SNIP gradients and may select these examples differently (although, since Wang al. did release their SNIP code, we cannot be certain); see Appendix C.2 for full details.
16

---

Published as a conference paper at ICLR 2021
C REPLICATING GRASP
In this Appendix, we describe and evaluate our replication of GraSP Wang et al. 2020)
C.I ALGORITHM
Scoring parameters. GraSP designed to preserve the gradient flow through the sparse network that results from pruning. To do so, it attempts to prune weights in order to maximize the change in loss that takes place after the first step of training. Concretely, let AL(W) the change in loss due to the first step of training 13
AL(w) L(w + 17 VL(w)) L(w)
where 7) is the learning rate. Since GraSP focuses gradient flow, it takes the limit as v goes to 0:
L(w + n VL(w)) - L(w) VL(w) VL(W))VL(u (L(w)
(L(w) lim 9-0
7
The last expression emerge by taking first-order Taylor expansion of L(w n VL(w)).
GraSP treats pruning the network as a perturbation transforming the original parameters w into perturbed parameters + 5. The effect of this perturbation the change in loss of the network can be measured by comparing (L( + 5) and A ((w)::
C()) = AL(W + 6) AL(w) = VLww + +0) VL(W + VL(w) TL(W))VL(uu (L
(2)
Finally, GraSP takes the first-order Taylor approximation of the left term about w, yielding:
C()) ~ V VLL - -L(()) VL
+ 28 (LL(w\VL(w) 0(16112)
-8
where H is the Hessian and is the gradient. Pruning individual parameter wy at index involves creating vector 8(8 where (( ww and o/4) for j * The resulting pruned parameter vector is 8(9); this vector is identical to except that w; has been set to O. Using the analysis above, the effect pruning parameter wy in this manner on the gradient flow approximated by C(-8(0)) w. (Hg); GraSP therefore gives each weight the following score:
8; = C(-8()) = --w;HHg)
Using scores to prune. To use 8; for pruning, Wang et al. make the following interpretation:
GraSP uses [Equation 3 as the measure of the importance of each weight. Specifically, if C(8) is negative, then removing the corresponding weights will reduce gradient flow, while if it is positive, it will increase gradient flow.
In other words, parameters with lower scores more important (since removing them will have a less beneficial or more detrimental impact on gradient flow) and parameters with higher scores less important (since removing them will have more beneficial less detrimental impact on gradient flow). Since the goal of GraSP is to maximize gradient flow after pruning, it should prune "those weights whose removal will not reduce the gradient flow," i.e.. those with the highest scores.
We believe this quantity should instead be specified as L(w) L(w 1 VL(W)). The gradient update goes in the negative direction, so we should subtract the expression V ((w) from the original initialization w. We expect loss to decrease after taking this step, sn-if we want A (Lw)) to capture the improvement in loss need subtract the updated loss from the original loss.
17

---

Published as a conference paper at ICLR 2021
C.2 IMPLEMENTATION DETAILS
Algorithm. To implement GraSP, we follow the PyTorch implementation provided by the authors on GitHut (which computes the Hessian-gradient product according Algorithm 2 of the paper).
Selecting examples for scoring parameters. To create the mini-batch that we use to compute the GraSP scores, we follow the strategy used by Waan et al. (2020) for the CIFAR-I0 networks in their implementation: we randomly sample ten examples from each class. We use this approach for both CIFAR-I0 and ImageNet: on ImageNet, this means we use 10,000 examples representing all ImageNet classes.
not entirely clear how Wang al. select the mini- for the ImageNet networks in their experiments. In their configuration files, Wang et al. appear to use one example per class (1000 in total covering all classes). In their ImageNet implementation (which ignores their configuration files), they use 150 mini-batches where the batch size is 128 (19,200 examples covering an uncertain number classes).
Reinitializing, We do not reinitialize after pruning.
Running on CPU. To avoid any risk that distributed training might affect results, we run GraSP computation on CPU and subsequently train the pruned network on TPU.
C.3 NETWORKS AND DATASETS
Wang et al. consider the networks for computer vision in Table below. They use both CIFAR-10 and CIFAR- for all CIFAR-10 10 networks, while we only CIFAR-10. Note that our hyperpa- rameters for ResNet-50 on ImageNet differ from those in the GraSP implementation (likely due to different hardware): we use larger batch size (1024 vs. 128), higher learning rate (0.4 vs. 0.1).
GraSP Name Our Name
Dataset
Gitllub Replicated Notes
VGG-19
VGG-19
CIFAR-10
ResNet-32
WRN-32-2
CIFAR-10
/ Wang et al. twice the standard width.
ResNet-50
ResNet-50
ImageNet
/ ResNets for CIFAR-10 10 and ImageNet are different.
VGG-16
ImageNet
X
x
VGGs for CIFAR- 10 and ImageNet are different.
Table The networks and datasets examined in the GraSP paper (Wang al. 2020
C.4 RESULTS
The GitHub implementation of GraSP by Wang et al. (2020) includes all of the networks from Sable-excey VGG-16. We have made our best effort to replicate these networks in our research
framework. Table C.2 shows the results from our replication. Each of our numbers the average across five replicates with different random seeds.
Unprumed networks. Our unpruned networks perform similarly those of Wang al. 2020 Although we made every effort to replicate the architecture and hyperparameters of the GraSP implementation of VGG-19, our average accuracy 0.7 percentage points lower [5] Accuracy on WRN-32-2 is closer, differing by only 0.3 percentage points. Accuracy on ResNet-50 for ImageNet higher by half a percentage point, likely due to the fact that we use different hyperparameters.
Pruned networks. Our pruned VGG-19 and WRN-32-2 also reach lower accuracy than those of Wang et a this difference is commensurate with the difference between the unpruned networks. On VGG-19, the accuracies of our pruned networks are lower than those of Wang et al. by 0.5 to 0.6 percentage points, matching the drop 0.7 percentage points for the unpruned network. Similarly, on WRN-32-2, the accuracies of pruned networks are lower than those of [Wann et al. by 0.2 to 0.5 percentage points, matching the drop of 0.3 percentage points for the unpruned networks. In both cases, the decrease in performance after pruning (inside the parentheses in Table (5) is nearly identical between the two papers, differing by no more than 0.2 percentage points at any sparsity.
"htpps: / //g thub. com/ a wllewaancg//raaae GraSp
VVGG networks for CIFAR-10 notoriously difficult to replicate (Blalock
2020)
18

---

Published as a conference paper at ICLR 2021
Name
Unpruned Accuracy Reported
Sparsity
Pruned Accuracy
Ours
Reported
Ours
90%
93.3% (-0.9)
92.8% (-0.7)
VGG-19
94.2%
93.5%
95%
93.0% (-1.2)
92.5% (-1.0)
98%
92.2% (-2.0)
91.6% (-1.9)
90%
92.4% (-2.4)
92.2% (-2.3)
WRN-32-2
94.8%
94.5%
95%
91.4% (-3.4)
90.9% (-3,6)
98%
88.8% (-6.0)
88.3% (-6.2)
60%
74.0% (-1.7)
73.4% ( (-2.8)
ResNet-50
75.7%
76.2%
80%
72.0% (.0)
71.0% (-5.2)
90%
68.1% 1% (-7.6)
67.0% (-9.2)
Table 5: The performance of GraSP as reported in the original paper and in our reimplementation.
We conclude that the behavior of our implementation matches that of Wang et al. although we are starting from slightly lower baseline accuracy.
The accuracy of our pruned ResNet-50 networks is less consistent with that of Wang et al. Despite starting from higher baseline, our accuracy after pruning is lower by 0.6 to 1.1 percentage points. These differences are potentially due to different hyperparameters: as mentioned previously, we select examples for GraSP differently than Wang et al., and we train with different batch size and learning rate.
19

---

Published as a conference paper at ICLR 2021
D REPLICATING SYNFLOW
In this Appendix, we describe and evaluate our replication of SynFlow (Tanaka et al. 2020
D.I ALGORITHM
SynFlow is an iterative pruning algorithm. It prunes to sparsity over the course of N iterations, pruning from sparsity " to sparsity 8 on each iteration E {I,..., N). On each iteration, it issues scores to the remaining, unprumed weights and then removes those with the lowest scores.
Synflow scores weights as follows:
1. It replaces all parameters we with their absolute values |we/-
2. It forward propagates an input of all ones through the network.
3. It computes the sum of the logits R.
4. It computes the gradient of A with respect to each weight (w]: dR
du.
5. It issues the score
dw
w for each weight.
Tanaka et al. (2020) explain these choices as follows. Their goal is create a pruning technique that "provably reaches Maximal Critical Compression," i.e., pruning technique that ensures that the network remains connected until the most extreme sparsity where possible do so. As they prove, any pruning technique that is iterative, issues positive scores, and is conservative (i.e., the sum of the incoming and outgoing scores for a layer are the same), then it will reach maximum critical compression (Theorem 3). The iterative requirement is that scores recalculated after each parameter is pruned; their experiments, Tanaka et al. use 100 iterations in order to make the process more efficient.
D.2 IMPLEMENTATION DETAILS
Iterative pruning. We use 100 iterations, the same as Tanaka et al. (022)) use (as noted in the appendices).
Input size. We input size that is the same the input size for the corresponding dataset. For example, for CIFAR-10, we use an input that is 32x32x3; for ImageNet, we use an input that is 224x224x3.
Reinitializing, After pruning, Tanaka et al. 22200) do not reinitialize the network.
Running on CPU. To avoid any risk that distributed training might affect results, run all SynFlow computation on CPU and subsequently train the prumed network on TPU.
Double precision floats. We compute the SynFlow scores using double precision floating point numbers. With single precision floating point numbers, the SynFlow activations explode on net- works deeper than ResNet-44 (CIFAR-10) 10) and ResNet-18 (ImageNet).
D.3 NETWORKS AND DATASETS
Tanaka al. consider the following settings for computer vision:
SynPlow Name Our Name
Dataset
GitHub Replicated Notes
VGG-11
VGG-11
CIFAR-10
A shallower version of VGG- 16 network nerwork
VGG-11
CIFAR-100
VGG- II
VGG-11 (Modified)
TinylmageNet
Modified for ImagenaeeNet Identical X
VGG-16
VGG-16
CIFAR-IO
VGG-16 networr
VGG-16
VGG-16
CIFAR-100
Identical to our VGG-16 network
VGG-16
VGG-16 (Modified)
TinyImageNer
Modified for TinyllageNee Modified ImageNet ReaNet-18; Modified ImageNet ResNet-18. 18; first cone is 3x.3 stride I: no max pool Modified ImageNet ResNet-18. 33; is 3x3 stride pool
ReaNet-18 (Mdiifeed
CIFAR-10
max-pool
ResNer-18 18
ResNet-Ik (Modified
CIFAR-100
R8sNet-18
ResNet-18 (Modifie) TinyImageNet
Table 6: The networks and datasets examined in the SynFlow paper (Tanaka al.
2020
20

---

Published as a conference paper at ICLR 2021
Of the settings that replicated, unpruned network performance is follows:
Network
Dataset
Reported Replicated Notes
VGG-11
CIFAR-10
-992
92.0%
Hyperparameters and augmentation are identical to ours
VGG-16
CIFAR-10
<.44%
Hyparparameters and augmentation idcutical to ours
ResNes-18 (Modified)
CIFAR-10 10
955
93.7%
Hyperpurameters reported by Tanaka et 1r=00001, batch size= 128, drop factors() 2)
RosNer-18 (Modified)
CIFAR-10 10
94.6%%
Hyperparmmeters reported by Lanaka
Ie=0 batch sicc=256, drop factor=0.1)
ResNet-18 (Modified)
NeylmageNeet
Hyperpurameters reportdd by lanaka 1l=0.01,
size= 128, epochs=100) Modined hyperparmmetes (1202 mach 256- epochs-
ResNes-18 18 (Modified) IingenggeNer
64%
Table 7: Top-1 accuracy of unpruned networks as reported by Tanaka et al. (020)) and as replicated. Tanaka al. showed plots rather than specific numbers, so reported numbers are approximate.
The VGG-11 and VGG-16 CIFAR-10 results are identical between our implementation and that of Tanaka et al.
We modified the standard ImageNet ResNet-18 18 from Torch Vision to match the network of Tanaka it al.] We used the same data augmentation and hyperparameters on TinylmageNet, but accuracy was much lower (58.8% vs. 64%). By increasing the learning rate from 0.01 0.2, increasing the batch size to 256, and increasing the number of training epochs to 200, we were able to match the ac- curacy reported by Tanaka al., We also used the same data augmentation and hyperparameters on CIFAR-10, 10, but accuracy was lower (95% vs. 93.6%). Considering that we needed different hyper- parameters on both datasets, we believe that there is an unknown difference between our ResNet- 18 implementation and that of Tanaka et al.)
D.4 RESULTS
Tanaka et al. compare to random pruning, magnitude pruning at initialization, SNIP, and GraSP at 13 sparsities evenly space logarithmically between 0% sparsity and 99.9% sparsity (compression ratio 10²). In Figure 7 we plot the same methods at sparsities between 0% and 99.9% at intervals of additional 50% sparsity (e.g., 50% sparsity, 75% sparsity, 87.5% sparsity, etc.).
Tanaka et al. present graphs rather than tables of numbers. As such, in Figure 7 we compare our graphs (left) the graphs from the SynFlow paper (right). On the VGG-style networks for CIFAR- 10, our results look nearly identical to those of Tanaka et al.) in terms of the accuracies of each method, the ordering of the methods, and when certain methods drop to random accuracy. The only difference is that SNIP encounters layer collapse in our experiments, while it does not in those of Tanaka et al.] Since these models share the same architecture and hyperparameters as in the SynFlow paper and the results look very similar, we have confidence that our implementation of SynFlow and the other techniques matches that of Tanaka et al.
Our ResNet- 18 experiments look quite different from those of Tanaka et al., The ordering of the lines is different, SynFlow not the best performing method at the most extreme sparsities, and magnitude pruning does not drop to random accuracy. Considering the aforementioned challenges replicating Tanaka et al. performance on the unpruned ResNet-18, we attribute these differences to unknown difference in our model training configuration. Possible causes include different hyperparameters (which may cause both the original model and the pruned networks to perform dif- ferently). Another possible cause is a different initialization scheme: we initialize the parameters of BatchNorm uniformly between and 1 and use He normal initialization based on the fan-in of the layer (both PyTorch defaults) while Tanaka al. initialize the y parameters to and use He nor- mal initialization based on the fan-out of the layer. Although these differences in the initialization scheme are small, they could make substantial difference for methods that prune at initialization. This difference in results, despite the fact that both unpruned ResNet- 18 networks reach the same accuracy on IanylmageNet, suggest that there may be a significant degree of brittleness, at least at the most extreme sparsities.
(The GraSP implementation Tanaka et al. 2020) contains a bug: it uses batch normalization in evaluation mode rather than training mode, which may also explain some of the differences we see in GraSP performance, especially on Modified ResNet-18.)
21

---

Published as a conference paper at ICLR 2021
100
V66-11 (CIFAR-10))
10
100
VGG-16 (CIFAR-10)
101
ResNee- 18 (CIFAR-10))
50
Mudilled RosNet- (Tiny (onggngaNNe
191 9922
100
10
10a
10a
Sponity (9)
Compression ratio
Random
Magnitude
SNIP
GaSP
SynFlow
Figure 7: Synflow replication experiments.
22

---

Published as a conference paper at ICLR 2021
D.5 A CONNECTION BETWEEN SYNFLOW AND PATH NORM PRESERVATION
In this section, we elucidate a connection between SynFlow and a particular group-norm that has been studied in the setting of generalization bounds for neural networks.
For this subsection, fix a neural network architecture, represented by a directed, acyclic graph (V, E) with neurons V and connections E' x V. We will assume that the architecture layered in the sense that each neuron is connected to every neuron in the layer before and after and no other neurons. A neural network N (V, E, W) is an architecture and weights W (WE) associated to every connection. A path p (U1,.... ppp)) E V is finite sequence of con- nected neurons. i.e. (E, u(+1) ture. For path p = (V1,..., plpt), we write v p (resp., E p) for E {Vi..... Uppl (resp., E {(11, (2)..... (1/p)-1; Ppo1))
E for all i. Let P denote the set of all paths in the architec-
aD) C V and O = (oi,.... ok} C V denote the input and output neurons, respectively. For 0, E V. let P(v) denote the set of all paths E P such that v E and let P(e, v²) denote the set of all paths in P starting at and ending at v'. For E, let P(e) denote the set of all "complete" paths E P such that e E p. where "complete" means that, for some a E A and O, P E o)a,o)
Let A
The (f1.1) path norm p(N) of a ReLU network / is the sum, over all paths from an input neuron an output neuron, of the product of weights of each connection in p, i.e.,
x(N) Σ Σ
II PEP
We
AEA, (EP(a.
The path norm is special case of group norms studied Neyshabu et al. 2015), towards bounding the Rademacher complexity of norm- boundeddded classes neural networks with ReLU activations. As such, the path norm can be interpreted properties of the path norm is its invariance to a certain type of balanced rescalings of the weights that also leave the output of ReLU networks invariant. In particular, multiplying one layer of weights by e while multiplying another layer by its reciprocal 1/c leaves the path norm invariant.
capacity measure for such networks. One of the key
In order to relate the path norm to SynFlow, we relate the path norm certain gradients. Consider the input equal to (I...., 1) and let N be a network whose weights W have all been made positive by setting each weight to its absolute value, In such a network, on input (I...., 1). the preactivation
value fe at every neuron v E V is positive, and so ReLU activations can be ignored/dropped. For every output neuron o, the gradient of fo with respect to a weight w. is
Qfo
Σ
II We'.
(5)
Dw.
PEP(O))P(O) e'Ep,
It it then straightforward to verify that, writing Eg for all connections in the E'th layer,
Of
m(N)
Σ |wel Σ EEE
Ow.
oEO
We will now show that SynFlow can be to prune weights that maximally preserve the path norm locally. Taking the derivative of the path norm with respect to an individual weight gives
Op(N) = Σ II Ow
We'
PEP(c) e'Ep,e'se
Consider ReLU network AV, and let N-p be the subnetwork with all the paths through connection removed. In other words, N is the network after pruning the weight w.. We can approximate how removing the weight w. determines the path norm of the resulting subnetwork A In the first-order Taylor series approximation, removing the weight w. decreases the path norm by
[wel| Op(N)
Σ
II e'Ep
Owe
(EP(c)
23

---

Published as a conference paper at ICLR 2021
which, by (5) can be rewritten as
afo Dwe
Wce DEO
We see that D.3 is identical to SynFlow's 's weight score function. In particular, the highest-scoring weight according to SynFlow the weight such that, an infinitesimal perturbation has the smallest impact on the path norm.
24

---

Published as a conference paper at ICLR 2021
E BASELINES
In Figure 8 we show the four baseline methods to which we compare SNIP, GraSP, and SynFlow: random pruning at initialization, magnitude pruning initialization, magnitude pruning after train- ing, and lottery ticket rewinding. Lottery ticket rewinding reaches different accuracies depending on the iteration and 6000 for ResNet-20, VGG-16, ResNet- 18, and ResNet-50; as shown in Figure 6 these are the iterations where accuracy improvements saturate.
which we the state of the pruned network. We use 1000, 2000, 1000,
VGG 100
101NCE13115 (5)
63.2
91.1
$5.0
0.1
19.1
13.38
85.
80.5
10.
63.0
BodNem E1
Rociee 90 (IroggaNee))
Tee Anxoeer
30
5311
112
50.1
Spardty 633
Speciity (9)
Unppunne Network
Magnitude Allez Training
LTR
Magnildd
Random
Figure 8: Accuracy of the baseline methods.
25

---

Published as a conference paper at ICLR 2021
F ABLATIONS AFTER INITIALIZATION
In this appendix, we perform the ablations (Section 5 on the pruning methods when applied after initialization (Section ( Our goal is a continuation of Section to study whether the pruning methods become more sensitive to the ablations as pruning occurs later in training.
F.1 ABLATIONS AT THE END OF TRAINING
In Figure we perform the ablations from SectionSSwhae pruning at the end of training. This figure where we performed these ablations when pruning initialization.
parallels Figure 4
Magnitude. Our results in Figure 9 confirm the well-known result that shuffling or reinitializing subnetworks found via magnitude pruming after training causes accuracy to drop (Han et al. 2015 Frankle et al., 2020a). On ResNet-20., shuffling and reinitializing perform similarly; they match full accuracy until 49% sparsity vs. 73.8% for the unmodified network. On VGG-16 and ResNet-18, random pruning performs better initially, after which reinitializing overtakes at higher sparsities.
SNIP. On SNIP, there smaller differences in accuracy between the unmodified network and the ablations. On ResNet-20, the unmodified network performs slightly better. On VGG-16, it performs approximately well when randomly shuffled. On ResNet-18, there more substantial dif- ferences. These results indicate that SNIP is not inherently robust to these ablations, but rather that the point in training at which SNIP is applied plays role. Note that, the highest sparsities on ResNet-20 and VGG-16, SNIP did not consistently converge, leading to the large error bars.
GraSP. On GraSP, the ablations have limited effect on the performance of the network, similar to pruning at initialization. On ResNet-20, the unmodified networks and the ablations perform the same. On VGG-16, the shuffling ablation actually outperforms the unmodified network (which performs the same as when reinitialized) at lower sparsities. On ResNet- 18, all three experiments perform similarly at lower sparsities, and shuffling performs lower at extreme sparsities.
SynFlow. On SynFlow, the ablations do affect performance, but in different ways different networks. On ResNet-20, shuffling improves accuracy; on VGG-16 and ResNet-18 it decreases accuracy at higher sparsities.
F.2 ABLATIONS AFTER INITIALIZATION
In Figure TO we perform the ablations at all points in training at a single sparsity for each network. This figure shows the ablations corresponding Figure Section 6
Magnitude. Magnitude pruning becomes sensitive to the ablations early in training (after iteration 5,000 for ResNet-20, 10,000 for VGG-16, and 4,000 for ResNet-18). The performance of the abla- tions does not improve after the very earliest part of training; the performance without the ablations continues improving after this point, while the performance of the ablations remains the same after this point. On VGG-16, shuffling outperforms reinitialiation, suggesting the initialization matters more tan the structure of the sparse network. On ResNet-18, the reverse true.
SNIP. The behavior of SNIP is very similar to the behavior of magnitude.
GraSP. GraSP improves little when pruning after initialization. It only sensitive the shuffling ablation on VGG-16 and R8sNet-18, and the changes in accuracy under this ablation are small.
SynFlow. SynFlow sensitive to the ablations, but different ways on different networks. ResNet-20 and VGG-16, the accuracy under shuffling improves alongside the accuracy of unmod- ified SynFlow. On ResNet-18, 18, the accuracy under shuffling is lower and does not improve. In all cases, accuracy is lower under reinitialization.
F.3 SUMMARY
Overall, these results support our hypothesis that it may be especially difficult to prune in a connection-specific or imitialization-sensiti manner initialization. At initialization, magnitude pruning, SNIP, and SynFlow maintain or improve upon their accuracy under random shuffling and reinitialization. After initialization, shuffling and reinitialization hurt accuracy to varying degrees.
26

---

Published as a conference paper at ICLR 2021
Redia Ab.CIAA
VOG 16 ICEAAR-O))
Rodies- (Taelaaeeee
Magnitude
To Accaroo 150
92
155
03.5
SNIP
45
35
13.2
90
17.2
14.2
360
5917
35.2
80.7
05.0
07.2
000
1901
713
10.7 911
992
GraSP
0.0
30.0
560
138
232
29
91.1
05.0
07.2
08.2
00
360
500
598
452
803
98.1
15.6
07.2
69.2
061
560
550
755
802 80.3 92.1
00.2 952
SynFlow
10.1
03.2
93.1
91.2
15.1
Reinlly 05)
Spendyy (5)
Spanay
Uspezzed
Stulffled Layernie
Reinitatized
Figure 9: Ablations on subnetworks found by applying magnitude, SNIP, GraSP, and SynFlow after training. (We did not run this experiment on 5o due to resource limitations.)
27

---

Published as a conference paper at ICLR 2021
Magnitude
SNIP
GraSP
SynFlow
Unmodified
Shuffled Layerwise
Reinitialized
Figure 10: Accuracy of early pruning methods and ablations when pruning the iteration the x-axis. Sparsities are the highest matching sparsities. Vertical lines are iterations where the learning rate drops by 10x.
28

---

Published as a conference paper at ICLR 2021
G ITERATIVE SNIP
In the main body of the paper, we consider SNIP as it was originally proposed by Lee et al. (2019) pruning weights from the network in one shot. Recently, however, de Jorge et al. (020) and Verde- nius et al. (2000) have proposed variants of SNIP in which pruning occurs iteratively (like SynFlow). In iterative variants of SNIP, the SNIP scores are calculated, some number of weights are pruned from the network, and the process repeats multiple times with new SNIP scores calculated based on the pruned network. In this appendix, we compare one-shot SNIP (the same experiment as in the main body of the paper) and iterative SNIP (in which we prune iteratively over 100 iterations, with each iteration going from sparsity
to sparsity 8 the same strategy as we use for SynFlow).
Figure 11 below shows the performance of SNIP (red) and iterative SNIP (green) at the sparsities we consider. At these sparsities, making SNIP iterative does not meaningfully alter performance. is possible that making SNIP iterative matters most at the especially extreme sparsities studied by de Jorge et al. (2020) and Tanaka et al. (2020) rather than at the matching and relatively less extreme sparsities we focus on in this paper.
Figure 12 shows the shuffling and reinitialization ablations for SNIP (top) and iterative SNIP (bot- tom). In both cases, the ablations do not meaningfully affect accuracy.
1000 20
Rucher 1o Clack aptier
10
56
Unprined Nawerk
Magainule Alker Training
Random
SNIP
Derrativ SNIP
Figure 11: A comparison of SNIP and iterative SNIP (100 iterations).
RoNNo 20 (CTPAR 100
VOG-C U8 (CIFAR-1))
Jealios 18 (hayykagghhcc
SNIP
85
45
80
35
90.1
06.2
99.2
Iteative SNIP
30
200
65
40
35
Netook
Shalllal
Weritialood
Innettal
Isil A70
Figure 12: A comparison of the ablations for SNIP (from the main body, Figure and for iterative SNIP. (ResNet-50 is not included due to computational limitations.)
29

---

Published as a conference paper at ICLR 2021
H VARIANTS OF GRASP
In Figure [3] we show three variants of GraSP: pruning weights with the highest scores (the ver- sion of GraSP from Wang et al.., pruning weights with the lowest scores (the inversion experiment from Section D and pruning weights with the lowest magnitude GraSP scores (our proposal for an improvement GraSP shown in Figure 14. This figure is intended make the comparison between these variants clearer, figure is too crowded for these distinctions to be easily visible.
VGS to (CIFAR-10)
1h0 MoxERey
50.
10
160
23.0
815
90.1
59.0
16.2
13.8
53.5
BocNee (ExegaNeo
T20 A0X
71
12
182
-2.0
25.0
25.8
3322
00.3
90.1
50.0
47.2
18.2
3.0
510
$1.
00.1
Spardty
Umpraned Nerwark
GraPP
GmSP Inverted
GraNP Magnimric
Figure 13: Accuracy of three different variants of GraSP
30

---

Published as a conference paper at ICLR 2021
1 COMPARISONS TO IMPROVED GRASP AND SYNFLOW
In Figure below, we compare the early pruning methods with our improvements to GraSP and SynFlow. This figure is identical to Figure except that modify GraSP prune the weights with the lowest-magnitude GraSP scores (Appendix) H and we modify SynFlow to randomly shuffle the per-layer pruning masks after pruning (Section
W0T 10-10
Diperced Nevock Touning
Resken
Magnitde
5NBP
Gradf Magritak:
118
SysRow Eageei))
Figure 14: The best variants of pruning methods at initialization from our experiments. GraSP and SynFlow have been modified as described in Section 5
31

---

Published as a conference paper at ICLR 2021
J LAYERWISE PRUNING PROPORTIONS
In Figure 05 on the following page, we plot the per-layer sparsities produced by each pruning method for the highest matching sparsity. Each sparsity is labeled with the corresponding layer name: layers are ordered from input (left) to output (right) with residual shortcut/downsample connections placed after the corresonding block. We make the following observations.
Different layerwise proportions lead to similar accuracy. At the most extreme matching sparsity, the early pruning methods perform in relatively similar fashion: there gap of less than 1, 1.5, 2.5, and percentage point between the worst and best performing early pruning methods on ResNet-20, VGG-16, ResNet-18 18, and ResNet-50. However, the layerwise proportions are quite different between the methods. For example, on ResNet-20, SynFlow prunes the early layers to less than 30% sparsity, while GraSP prunes to more than 60% sparsity. SNIP and SynFlow tend to prune later layers in the network more heavily than earlier layers, while GraSP tends to prune more evenly.
On the ResNets, the GraSP layerwise proportions most closely resemble those of magnitude pruning after training, despite the fact that GraSP is not the best-performing method at the highest matching sparsity on any network.
These results are further evidence that determining the layerwise proportions in which prune (rather than the specific weights to prune) may not be sufficient to reach the higher performance of the benchmark methods. The early pruning methods produce diverse array of different layerwise proportions, yet performance is universally limited.
Skip connections. When downsampling the activation maps, the ResNets use 1xl convolutions their skip connections. On ResNet-20, these layer names include the word shortcut; on ResNet- 18 and ResNet-50, they include the word downsample. SynFlow prunes these connections more heavily than other parts of the network; in contrast, all of the other methods prune these layers to similar (ResNet-50) ResNet-50, SynFlow entirely prunes three of the four downsample layers, eliminating the residual part of the ResNet for the corresponding blocks.
much lower (ResNet-20 and ResNet-18) sparsities than adjacent layers. On
The output layer. All pruning methods (except random pruning) prune the output layer at lower rate than the other layers. These weights are likely disproportionately important to reaching high accuracy since there are so few connections and they directly control the network outputs.
32

---

Published as a conference paper at ICLR 2021
Role: SHLLPA IE soundt
Layer
VOG SATEAAAEEA-- qurdss
RuNe-:RiToofeaga 55 cueiyy
Mashie--0 qunntty
Magsital Traating
Magnitaic
Figure 15: Per-layer sparsities produced by each pruning method at the highest matching sparsity.
33

---

Published as a conference paper at ICLR 2021
K LENET-300-108 FOR MNIST
In this appendix, show selected experiments from the main body of the paper and the appen- dices on the fully-connected LeNet-300- 100 for MNIST. We did not include this setting in the main body of the paper because results on MNIST often do not translate to other, larger-scale settings. The reason why we include this experiment here is to evaluate our ablations in the context fully-connected network. Unlike convolutional network, each weight of the first layer of fully- connected network tied to a specific pixel of the input, so it is possible that randomly shuffling the pruned weights may have different effect in this setting.
K.1
FIGURE 3
For the standard versions of the methods, SNIP performs best and magnitude pruning at initialization performs worst, although all of the methods perform similarly lower sparsities. Compared to the larger convolutional networks, there is little distinction between the performance of magnitude pruning after training and the early pruning methods just small fraction of percent. This result is further evidence that LcNet-300-1 100 on MNIST is not representative of larger-scale settings.
Unperned Nerract
Magrinde Afler Tnining
Random
Magrital:
Qua2?
SystHow
K.2 FIGURE
The ablations vary by network. For magnitude pruning, shuffling and reinitializing do not affect accuracy. For SNIP, shuffling and reinitializing do not affect accuracy except at the higher sparsities. For GraSP, shuffling and reinitializing increase accuracy at lower sparsities and hurt accuracy at higher sparsities; unlike the other networks, inverting GraSP does hurt accuracy. Finally, SynFlow is unaffected by shuffling at lower sparsities and is unaffected by reinitializing at all sparsities.
Recall that the motivating research question for this section was whether random shuffling would have a different effect on fully-connected network in which each connection in the first hidden particular pixel of the input. For this reason, one possible hypothesis that random shuffling will hurt accuracy in manner that it did not for the convolutional networks.
layer is specific
These ablations show that randomly shuffling only hurts accuracy (compared to the unmodified pruning strategies) at the highest sparsities if at all. At these sparsities, the network has been pruned to the point where accuracy begins plummet. We conclude that the hypothesis about the effect of shuffling on fully-connected network does hold, but only at the most extreme sparsities. For comparison, shuffling affects the accuracy of magnitude pruning after training (Figure below, upper left plot) at much lower sparsities.
Magnitude
SNNP
63
SymFPlo Fmm
GraSp
000
34

---

Published as a conference paper at ICLR 2021
K.3 FIGURE 5
1-50--1--11C (NNST- 99.35 23007
Masstade
SNEP
GraPP
Spaflow
SeaFlon (Skafiial Lagervise)
K.4 FIGURE g
X01
Ungronal Network
Magnitade Alla Triiing
LIR
Magnilukk
Randon
K.5 FIGURE 9
Magnitude
SNIP
01 003
M1
SynFlow
GruSP
Oigodd Toally
Traking
K.6 FIGURE
Unlike the convolutional networks, iterative SNIP improves accuracy the most extreme sparsi- ties. It is possible that it will have the same effect for the convolutional networks at more extreme sparsities than wstudy in this paper.
1/h6 THIR SPETT
Unpraned Network
Magelltuee Training
Random
SNIP
SNPP
35

---

Published as a conference paper at ICLR 2021
K.7 FIGURE B
Urgrunal Natwork
CraSP
GraPP trvente
GraSP Magnitale
K.8 FIGURE 15
9036 speit
Mastialk Trangg
Radan
Magztakk
SNIP
GsSP
Sor
36

---

Published as a conference paper at ICLR 2021
L MODIFIED RESNET-18 FOR TINYIMAGENET
In this appendix, we show the experiments from the main body of the paper on the modified version of ResNet-18 TinyImageNe modeled after the setting used by Tanaka al. (2020) evaluate SynFlow. This is the same configuration as we describe in Appendix A
We include this experiment because the TinylmageNet benchmark is less standardized than other benchmarks the we include in the paper (c.g., ResNets, VGGs, CIFAR-10, and ImageNet). We use this appendix validate that, for a very different realization of the TinylmageNet benchmark (modified ResNet-18, different augmentation, and higher accuracy), our main findings hold.
L.1 FIGURE
For the standard versions of the early pruning methods, magnitude pruning at initialization performs best at lower sparsities and GraSP performs best at higher sparsities. Surprisingly, SynFlow is no better than random pruning. At the higher accuracy this variant TinylmageNer reaches, magnitude pruning after training is matching only at much lower sparsities.
Unprused Netwodk
Magninude After
Randon
Magrindd
GraSP
NyntHow
L.2 FIGURE
As in the main body of the paper, we find that all methods maintain improve upon their accuracy when randomly shuffling (Figure 4 SynFlow shows dramatic improvements, and SNIP improves as well. All methods maintain their performance when randomly reinitializing. Only magnitude pruning degrades in performance when changing the initialization distribution to have fixed vari- ance. Finally, GraSP maintains its performance when inverting, while the other methods degrade in performance.
Magaittde
SNIP
Spa Flow
GraSP
Ondilal
Saflk
Renidioal
37

---

Published as a conference paper at ICLR 2021
L.3 FIGURE
As in the main body of the paper. SynFlow leads to neuron collapse, and randomly shuffling reduces the extent of this neuron collapse 5
30
Magrisde
SNZP
CraSP
SysTlow
Sysflns (Staliial layemie)
L.4 FIGURE
Unprened Netwock
Macninuee Alee Trrining
LTR
Mapnitde
Randum
L.5 FIGURE
Magnitude
SNIP
SynFFum
GriSP
Noak
Liprnie Nar
B.
L.6 FIGURE 13
Pruning the weights with the magntttt GraSP scores does not change performance, whereas it improves performance in some cases the main body of the paper.
other
Nawwodk
GraSP Erventel
GraSP Magnitude
38

---

Published as a conference paper at ICLR 2021
L.7 FIGURE 15
As in Appendix SynFlow prunes skip connection weights with a higher propensity than other methods (Figure 15
Musited Rechoud Mappaamchinn
200
Layer
Magshaaoo Afe Trikingg
Radon
Migntade
Syr
39

---

Published as a conference paper at ICLR 2021
M COMPARING THE ACTUAL AND EFFECTIVE SPARSITY OF RANDOMLY SHUFFLED NETWORKS
In this appendix, evaluate the following hypothesis:
is possible that the early pruning methods leave some weights unprumed but disconnected, which means the "effective sparsity" (the sparsity when also eliminating disconnected weights) of these networks is higher than the "actual sparsity" (the sparsity when taking into account only connec- tions that are explicitly pruned). Furthermore, when randomly shufflingg, some of these disconnected weights may become reconnected. As such, random shuffing might appear to maintain performance simply because it has lower effective sparsity than the unmodified pruning technique.
To evaluate this hypothesis, we compare the effective sparsities and actual sparsities for the early pruning methods with and without randomly shuffling.
Methods. To determine which weights are disconnected, we use the following procedure:
1. Set each weight in the network to 1 if it is unpruned or 0 if it is pruned.
2. Forward-propagate a single example comprising all I's.
3. Compute the sum of the logits.
4. Compute the gradients with respect to this sum.
5. Prune any unpruned weight with a gradient of O. Since these weights did not receive any gradient, they are disconnected from the output of the network.
effective parameters in the shuffled network Evaluation. For each actual sparsity, we compute the ratio of ettective parameters in the unmodiñied network"
This allows us to determine the extent to which the shuffled network is larger or smaller than the unmodified pruned network.
Results. Figure 16 we show this ratio for all pruning methods and all sparsities on our four main networks. In the table below, summarize our results at the highest matching sparsity and highest sparsity that we consider.
Network
Pruning Method
Highest Matching Sparsity Sparsity
Highest Sparsity We Consider Sparsity
Ratio
Ratio
Magnitude SNIP
1.0000
1.0016
1.0006
1.0353
ResNet-20 (CIFAR-10)
73.8%
98.2%
GraSP
1.0000
1.0197
SynFlow
1.0000
0.9921
Magnitude SNIP
1.0000
0.9998
1.0007
1.0058
VGG-16 (CIFAR-10)
93.1%
98.2%
GraSP
1.0003
1.0U36
SynFlow
1.0000
1.0001
Magnitude
1.0003
1.0000
SNIP
1.0026
1.0227
ResNet-18 (TinylmageNet)
96.5%
98.2%
GraSP
1.0011
1.0030
SynHlow
1.0001
1 1.0001
Magnitude
1.0000
0.9979
SNIP
1.0003
1.0016
ResNet-50 (ImageNet)
67.2%
94.5%
GraSP
1.0000
1.0009
SynFlow
1.0000
1.0001
At the highest matching actual sparsity, the effective sparsity the shuffled network was larger by negligible amount: at most 1.0026x (i.e., 0.26%) larger than the unmodified network and in most cases closer to 1.0005x (i.e., 0.05%) larger. At more extreme sparsities, this ratio was higher but still small: at most 1.0353x (i.e., 3.5% larger) and in most cases closer 1.005x (i.e., 0.5%) larger. We conclude that, although random shuffling does often restore some disconnected parameters, the actual fraction of parameters it restores is minuscule and beyond the matching sparsities focus on. These differences are so small that the effective sparsities round to the same values as the actual sparsities on the x-axis labels of our plots.
40

---

Published as a conference paper at ICLR 2021
-.0
100
1.01
1.00
-000
303
15.5
13.2
30.3
10.1
33.0
102
182
340
350
Spensiy
Spenit (90
Aucii 12 (Lapkangehcc)
Baallu222 Baaqplic
1.55
110
-100
Magnitude
1.42
SNIP
180
1.80
GraSP
SynFlow
Spers (90
Spunii (70
Figure 16: For each actual sparsity (x-axis), the ratio of the effective parameter-count of the ran- domly shuffled ablation to the effective parameter-count of the network pruned with the unmodified pruning method. Note that we have zoomed into the range of ratios between 0.99 and 1 1.05.
41


Reviews:
[{"review_id": "rLklmQxfPz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an empirical work assessing some of the recent works of early pruning methods in an effort to understand why early pruning methods perform worse than pruning after training methods. The paper is written very well, and some of the findings are interesting leaving important questions to address. Some major concerns remain though.\n\nThe paper proposes to achieve the following goal: to understand why early pruning methods perform worse than pruning after training methods.To achieve this, what the paper chooses to do is basically to measure their final performances and compare them to each other, and then to check for some ablation studies.\n\nObviously, the results (Figure 3) show which performs better than the other (i.e. pruning after training > pruning early), meaning that this result itself in terms of the final performance does not suffice to prove any cause or effect to explain the behaviors in comparison aimed to understand.\n\nIn the ablation studies (Section 5), the paper appears to be focused on studying and debunking the early pruning methods rather than finding *why* pruning after training is better than pruning early. For example, the result of random shuffling (early pruning methods for finding layerwise proportions) and the failure of GraSP when inverted are quite interesting, but they remain empirical without contributing directly to fulfill the purpose; some of the conclusions drawn from the ablation studies are not convincing (e.g., lack of specificity or insensitivity to initialization may limit performance); some of the claims are not expected to generalize either (e.g., shuffling to fully-connected layers, initializing with different standard deviations of the normal distribution).\n\nIn fact, an intuitive ablation setting is to test for the effect of training with the same saliency measure, as done in Section 6, but unfortunately the result out of this seems to be unhelpful for figuring out the potential cause of poor performances for pruning early methods. The results however show that early pruning methods improve after some training, indicating that training to some degree before pruning helps to increase the performance of the pruned model. One of the potentially many reasons that pruning early methods is underperformed by pruning after training methods could merely be the fact that the former is not trained before it gets pruned as opposed to the latter yielding the observed performance gap quite obviously. Anyway, the fact that they (after some training) still do not perform better than LTR (which requires oracle information) does not contribute to proving anything for why pruning early methods is underperformed by pruning after training either.\n\nIn short, this work does not explain \"why all methods fall short of magnitude pruning after training\" in a convincing manner, and the contributions are considered not quite significant overall.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ksnRmL86yP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nGenerating a pruned network falls into two broad categories: 1) spend some extra time and effort to train or fine-tune the pruned model after first training a dense version, or 2) cut out that extra time and effort by generating a sparse network \"from scratch.\"  While approach (1) has historically given the best accuracy, recent advances (such as the lottery ticket hypothesis) suggest that there are sparse networks hidden in the initialization that don't need to first be trained, if only we could divine the structure of those models.  Approach (2) seeks to do just this: determine the connectivity as close to initialization possible.  However, even the best results taking this second path fall short when compared to the accuracy of the former path - why is this?  The submission pokes at three recent techniques to pull out some commonalities that are *not* shared with (1), suggesting possible issues that need to be overcome to improve accuracy, and proposes a set of experiments and comparisons that should be part of any new technique that claims to discover a good sparse mask at initialization.\n\n##########################################################################\n\nReasons for score: \n\nOverall, this submission raises some important questions (why is from-scratch pruning falling short of SOTA accuracy?), empirically shows the performance of some leading techniques (and that they fall short in a well-motivated range of sparsities), and, via ablation studies, points towards some potential reasons.  More importantly, these findings are interesting:\n- There's no single SOTA method for sparse-from-scratch training\n- There's a need for consistent reporting in this area, and the ablation studies performed have been shown to lead to useful insights; adopting them as standard for future work seems fruitful.\n\nI think these are enough to warrant an \"above-the-threshold\" rating, but a higher rating would require empirical results on more networks on large-scale data sets, or the inclusion of techniques the submission itself suggests might be a promising direction forward in the current gauntlet of tests.\n\n##########################################################################\n\nPros:\n\n+ The organization of the paper makes it easy to follow the logical progression and points being raised.\n+ The direct comparisons of three recent techniques (SNIP, GraSP, and SynFlow) on different networks and data sets fills in some gaps in the literature.\n+ Further, the ablation studies performed on these techniques yield surprising results, both in isolation (inverting GraSP improves accuracy!) and when compared to magnitude pruning after training (these three are invariant to shuffling and re-initialization).\n\n##########################################################################\n\nCons:\n\n- Experiments on large data sets are limited to RN50 on ImageNet.\n- The three particular techniques chosen (SNIP, GraSP, SynFlow) aren't particularly motivated - why these three, and not other recent techniques?  (A potential answer may be that there's no training before pruning is finished, but why is this important?)\n- Mostly tongue-in-cheek: the submission doesn't answer all the questions it raises, unfortunately.\n\n##########################################################################\n\nQuestions:\n\n- What benefit do the three at-initialization \"static\" pruning techniques have over those that reduce training FLOPs and memory requirements but allow the sparse mask to change dynamically, like RigL (Evci et al., 2020) and sparse momentum (Dettmers and Zettlemoyer, 2019)?  Is there a reason they do not belong in the current lineup?  The overhead of occasionally updating the mask shouldn't be too imposing.\n\n- In the final paragraph, it is suggested that it may be tricky to compare the training cost of \"a method that prunes to sparsity s at step t against a method that prunes to sparsity s' < s at step t' > t.  If method A prunes to a higher sparsity at an earlier time step, shouldn't it cost strictly less than method B, which prunes to a lower sparsity later in training?\n\n##########################################################################\n\nMinor suggestions:\n\nFigure 2 is never referenced in the text (that I could find), and LTR isn't defined until the following page.\n\n\n##########################################################################\n\nUpdates are appreciated\n\nHi, Authors,\n\nI appreciate the updates you've made to the paper and the responses to my questions.  You're quite correct that RN50 and ImageNet is sufficient to illustrate deficiencies.\n\n(I'd still want to see broader experiments for claims of some new method overcoming these deficiencies, though!  When broadening scope to other tasks, I'd expect the authors of prior methods designed for vision tasks would be okay with use of their methods as baselines if there are no methods designed specifically for those new tasks.)\n\nWith this in mind, I'll update my rating to a 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UdcCYV85z1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary \n\nThe paper provides an extensive empirical analysis of Pruning-at-Initialization (PaI) techniques and compares it against two pruning methods after (or during) training. This comparison sheds some light on why pruning at initialization is inherently hard. Furthermore, the comparison among PaI methods with various ablations shows some inherent properties that are common to PaI methods and the benefits/drawbacks of certain methods. With these experiments, certain conclusions are reached among them an important one is that PaI methods only determine what is the fraction of weights to be pruned in each layer rather than which weights to prune.\n\n## Strengths\n\n1. Quantitative comparison of PaI methods is an important contribution to the community. In addition, the paper extensively analyses them with various ablations (eg, shuffle the layerwise masks, reinitialization) which uncovers various properties of PaI methods previously not conveyed clearly in the respective papers.\n\n2. Extensive analysis shows that there is a performance gap of PaI methods compared to Pruning-after-Training (PaT) methods (understandably) and attempts to explore the potential reasons. This analysis uncovers some inherent difficulties of PaI. The analysis provided in this paper could serve as a guide to better understand and improve PaI methods.\n\n3. Overall the paper is clearly written and sufficient details are provided in the appendix.\n\n## Weaknesses\n\nThe main weaknesses of the manuscript in my opinion are as follows:\n\n1. Some conclusions of this paper are unsubstantiated or should be rephrased:\n\t- First, the experiments mainly convey the inherent difficulties of PaI rather than the issues with specific methods themselves. To address this concern, the methods (SNIP, GraSP, SynFlow) are performed during training and showed that they perform inferior to LTR in Fig. 6. However, this comparison is unfair as LTR has additional information which is obtained after training (pruning mask is obtained after training but applied during training) and those PaI methods are not specifically designed to be performed during/after training (even so some perform reasonably well). In short, it is not clear that the performance of PaT can be matched by PaI given only the information at initialization. Note that, having access to the pruning mask after training defeats the purpose of PaI since given an already trained network, one might as well simply prune at the end (no need to retrain from scratch). I believe, Sec. 7 has some discussions about this (not complete) but should come early in the paper and emphasized. Please clarify and rephrase certain parts (especially in the introduction). \n\t- One of the main conclusions: \"PaI methods only determine what is the fraction of weights to be pruned in each layer rather than which weights to prune\" should be rephrased. In fact, it is possible to have a pathological case (eg, disconnected layers at a given pruning ratio) where knowing the optimal pruning ratio is not sufficient to obtain matching accuracy. These pathological cases are not observed in practice due to the random component (initialization or shuffling) and it should be emphasized. Also, since the weight initialization is iid Gaussian, there may not be sufficient information for PaI methods to select each weight individually but rather select them as a group in each layer or whole network.\n\t- Another hypothesis that the performance gap between PaI and PaT is due to the robustness of PaI methods to shuffling, reinitialization, etc, are also not clear. I understand that there is a correlation exists that some PaT methods are not robust to such variations. But it is NOT clearly demonstrated that being robust to such variations is necessarily a bad thing for PaI. In fact, one would think it is a good thing that PaI methods are robust to such variations given that there is not a lot of information available at initialization (note initialization is iid) to perform effective pruning and it seems these methods are robust and perform competitively to unpruned networks.\n\n2. Inconsistency of SNIP being independent to initialization:\n\t- In the last paragraph of page 5, it is mentioned that SNIP is independent to what initialization is used. However, there is a follow-up work of SNIP showing that it could be beneficial for SNIP if the network is initialized to have good signal propagation [a]. Please discuss this in the context to avoid any misinterpretations.\n\n3. Issue with random shuffling:\n\t- The random shuffling experiment needs some refinement in my opinion. I believe PaI methods such as SNIP, GraSP or SynFlow might have some unpruned weights which are not updated during training (ie, they are disconnected in the signal propagation path). In my personal experience, I observed that there are about 1-2% of unpruned disconnected weights in the network (in particular layers this value could be up to 10%) for SNIP (meaning they can be removed) depending on the network. This means the effective sparsity is slightly lower than what is observed in SNIP (not mentioned in the original paper though) and presumably in other methods as well. This observation would be applicable to random shuffling as well. I mean, after random shuffling there might be some unpruned disconnected weights and they could be removed before training, leading to higher effective sparsity. Furthermore, the existence of unpruned disconnected weights could be the reason for similar behaviour even after random shuffling in each layer. \n\nI found the experiments in this paper to be thorough but I think the deduced conclusions are slightly off with respect to the observations in the experiments. I believe this paper will be a good contribution to the pruning literature if the conclusions are tightened.\n\n## Minor Comments\n\n1. Related work: SNIP is not a follow-up of LTH but rather they are concurrent works published in ICLR 2019.\n2. Abstract: I think \"undermines\" might be a strong word given that there are questions regarding deduced conclusions.\n\n## References\n\n- [a] Lee, N., Ajanthan, T., Gould, S. and Torr, P.H., 2020. A signal propagation perspective for pruning neural networks at initialization. ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ZDWAD4Lrly", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A recent trend of 'pruning at initialization' in neural network pruning has left me baffled. It's counter-intuitive that neural networks can be pruned at initialisation, improving results for the training done thereafter. Nitpicking semantics, one could hardly even call this a pruning technique, since there is no a-priori knowledge of the dataset distilled in the network. Perhaps it's more aptly referred to as a method of sparse initialisation methods.\nThe authors of this paper seem to have had the same doubts when it comes to the pruning at initialization literature, and put the magnifying glass on these methods that have recently been published. In an extensive and comprehensive study, they show that pruning at initialization methods do naught but set per-layer sparsity rates, where the sparse initialization might as well have been random.\n\nWriting these types of papers is important. It's essentially a survey paper, reproducing results from other papers, and testing their claims. In moving forward in this field, this type of work is crucial in filtering out the sense from the nonsense. Although the paper does not provide any new shiny optimization method, or stellar new GAN with a funny name, I highly laude the authors for working on this survey, and I believe the impact on the model efficiency community of this work is significant.\n\nNow on to the nitty-gritty of the paper. The key argument that shuffling the weights within a layer, essentially functioning like a re-initialization with a given per-layer sparsity ratio, is solid, convincing and damning. The inversion arguments equally so. The only qualm I have with the results are that there are not more on different architectures... but I hardly think that's necessary to make the point. If these methods were to work as intended, it should show on the common computer vision architectures we work with.\n\nThe paper is well-written and well-structured. Clear and concise, with an extensive appendix highlighting more background information, and showing that the authors know the field very well. They covered every angle I could think of to put a crowbar in the paper and pry open some problems. \n\nI would have liked to see a discussion on why we would even want to prune at initialization, because the arguments in the paper only really come to light if we consider the utility of the methods. Sure, pruning at initialization is worse than pruning, but perhaps there are reasons to prune at initialization anyway. Are we looking at this as a research exercise? A quest to improve our understanding of networks? Is it done for sparse training? If it's the latter, a discussion on this, and perhaps a comparison to other sparse training methods would be in order.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyses several approaches to pruning at initialization, compared to after training. There was a large gap in reviewers appreciation of the paper, but I think that the pros outdo the cons as the paper show a lot of insights overall. I recommend accepting the paper.
Decision: Accept (Poster)