Pruning Neural Networks at Initialization: Why Are We Missing the Mark?

1 INTRODUCTION . Since the 1980s , we have known that it is possible to eliminate a significant number of parameters from neural networks without affecting accuracy at inference-time ( Reed , 1993 ; Han et al. , 2015 ) . Such neural network pruning can substantially reduce the computational demands of inference when conducted in a fashion amenable to hardware ( Li et al. , 2017 ) or combined with libraries ( Elsen et al. , 2020 ) and hardware designed to exploit sparsity ( Cerebras , 2019 ; NVIDIA , 2020 ; Toon , 2020 ) . When the goal is to reduce inference costs , pruning often occurs late in training ( Zhu & Gupta , 2018 ; Gale et al. , 2019 ) or after training ( LeCun et al. , 1990 ; Han et al. , 2015 ) . However , as the financial , computational , and environmental demands of training ( Strubell et al. , 2019 ) have exploded , researchers have begun to investigate the possibility that networks can be pruned early in training or even before training . Doing so could reduce the cost of training existing models and make it possible to continue exploring the phenomena that emerge at larger scales ( Brown et al. , 2020 ) . There is reason to believe it may be possible to prune early in training without affecting final accuracy . Work on the lottery ticket hypothesis ( Frankle & Carbin , 2019 ; Frankle et al. , 2020a ) shows that , from early in training ( although often after initialization ) , there exist subnetworks that can train in isolation to full accuracy ( Figure 1 , red line ) . These subnetworks are as small as those found by inference-focused pruning methods after training ( Appendix E ; Renda et al. , 2020 ) , raising the prospect that it may be possible to maintain this level of sparsity for much or all of training . However , this work does not suggest a way to find these subnetworks without first training the full network . The pruning literature offers a starting point for finding such subnetworks efficiently . Standard networks are often so overparameterized that pruning randomly has little effect on final accuracy at lower sparsities ( green line ) . Moreover , many existing pruning methods prune during training ( Zhu & Gupta , 2018 ; Gale et al. , 2019 ) , even if they were designed with inference in mind ( orange line ) . Recently , several methods have been proposed specifically for pruning at initialization . SNIP ( Lee et al. , 2019 ) aims to prune weights that are least salient for the loss . GraSP ( Wang et al. , 2020 ) aims to prune weights that most harm or least benefit gradient flow . SynFlow ( Tanaka et al. , 2020 ) aims to iteratively prune weights with the lowest “ synaptic strengths ” in a data-independent manner with the goal of avoiding layer collapse ( where pruning concentrates on certain layers ) . In this paper , we assess the efficacy of these pruning methods at initialization . How do SNIP , GraSP , and SynFlow perform relative to each other and naive baselines like random and magnitude pruning ? How does this performance compare to methods for pruning after training ? Looking ahead , are there broader challenges particular to pruning at initialization ? Our purpose is to clarify the state of the art , shed light on the strengths and weaknesses of existing methods , understand their behavior in practice , set baselines , and outline an agenda for the future . We focus at and near matching sparsities : those where magnitude pruning after training matches full accuracy.1 We do so because : ( 1 ) these are the sparsities typically studied in the pruning literature , and ( 2 ) for magnitude pruning after training , this is a tradeoff-free regime where we do not have to balance the benefits of sparsity with sacrifices in accuracy . Our experiments ( summarized in Figure 2 ) and findings are as follows : The state of the art for pruning at initialization . The methods for pruning at initialization ( SNIP , GraSP , SynFlow , and magnitude pruning ) generally outperform random pruning . No single method is SOTA : there is a network , dataset , and sparsity where each pruning method ( including magnitude pruning ) reaches the highest accuracy . SNIP consistently performs well , magnitude pruning is surprisingly effective , and competition increases with improvements we make to GraSP and SynFlow . Magnitude pruning after training outperforms these methods . Although this result is not necessarily surprising ( after all , these methods have less readily available information upon which to prune ) , it raises the question of whether there may be broader limitations to the performance achievable when pruning at initialization . In the rest of the paper , we study this question , investigating how these methods differ behaviorally from standard results about pruning after training . Methods prune layers , not weights . The subnetworks that these methods produce perform equally well ( or better ) when we randomly shuffle the weights they prune in each layer ; it is therefore possible to describe a family of equally effective pruning techniques that randomly prune the network in these per-layer proportions . The subnetworks that these methods produce also perform equally well when we randomly reinitialize the unpruned weights . These behaviors are not shared by stateof-the-art weight-pruning methods that operate after training ; both of these ablations ( shuffling and reinitialization ) lead to lower accuracy ( Appendix F ; Han et al. , 2015 ; Frankle & Carbin , 2019 ) . These results appear specific to pruning at initialization . There are two possible reasons for the comparatively lower accuracy of these methods and for the fact that the resulting networks are insensitive to the ablations : ( 1 ) these behaviors are intrinsic to subnetworks produced by these methods or ( 2 ) these behaviors are specific to subnetworks produced by these methods at initialization . We eliminate possibility ( 1 ) by showing that using SNIP , SynFlow , and magnitude to prune the network after initialization leads to higher accuracy ( Section 6 ) and sensitivity to the ablations ( Appendix F ) . This result means that these methods encounter particular difficulties when pruning at initialization . Looking ahead . These results raise the question of whether it is generally difficult to prune at initialization in a way that is sensitive to the shuffling or reinitialization ablations . If methods that maintain their accuracy under these ablations are inherently limited in their performance , then there may be broader limits on the accuracy attainable when pruning at initialization . Even work on lottery tickets , which has the benefit of seeing the network after training , reaches lower accuracy and is unaffected by these ablations when pruning occurs at initialization ( Frankle et al. , 2020a ) . Although accuracy improves when using SNIP , SynFlow , and magnitude after initialization , it does not match that of the full network unless pruning occurs nearly halfway into training ( if at all ) . This means that ( 1 ) it may be difficult to prune until much later in training or ( 2 ) we need new methods designed to prune early in training ( since SNIP , GraSP , and SynFlow were not intended to do so ) . 1Tanaka et al . design SynFlow to avert layer collapse , which occurs at higher sparsities than we consider . However , they also evaluate at our sparsities , so we believe this is a reasonable setting to study SynFlow . 2 RELATED WORK . Neural network pruning dates back to the 1980s ( survey : Reed , 1993 ) , although it has seen a recent resurgence ( survey : Blalock et al. , 2020 ) . Until recently , pruning research focused on improving efficiency of inference . However , methods that gradually prune throughout training provide opportunities to improve the efficiency of training as well ( Zhu & Gupta , 2018 ; Gale et al. , 2019 ) . Lottery ticket work shows that there are subnetworks before ( Frankle & Carbin , 2019 ) or early in training ( Frankle et al. , 2020a ) that can reach full accuracy . Several recent papers have proposed efficient ways to find subnetworks at initialization that can train to high accuracy . SNIP ( Lee et al. , 2019 ) , GraSP ( Wang et al. , 2020 ) , SynFlow ( Tanaka et al. , 2020 ) , and NTT ( Liu & Zenke , 2020 ) prune at initialization . De Jorge et al . ( 2020 ) and Verdenius et al . ( 2020 ) apply SNIP iteratively ; Cho et al . ( 2020 ) use SNIP for pruning for inference . Work on dynamic sparsity maintains a pruned network throughout training but regularly changes the sparsity pattern ( Mocanu et al. , 2018 ; Dettmers & Zettlemoyer , 2019 ; Evci et al. , 2019 ) ; to match the accuracy of standard methods for pruning after training , Evci et al . needed to train for five times as long is standard for training the unpruned networks . You et al . ( 2020 ) prune after some training ; this research is not directly comparable to any of the aforementioned papers as it prunes channels ( rather than weights as in all work above ) and does so later ( 20 epochs ) than SNIP/GraSP/SynFlow ( 0 ) and lottery tickets ( 1-2 ) . 3 METHODS . Pruning . Consider a network with weights w ` ∈ Rd ` in each layer ` ∈ { 1 , . . . , L } . Pruning produces binary masks m ` ∈ { 0 , 1 } d ` . A pruned subnetwork has weights w ` m ` , where is the element-wise product . The sparsity s ∈ [ 0 , 1 ] of the subnetwork is the fraction of weights pruned : 1− ∑ ` m ` / ∑ ` d ` . We study pruning methods prune ( W , s ) that prune to sparsity s using two operations . First , score ( W ) issues scores z ` ∈ Rd ` to all weights W = ( w1 , . . . , wL ) . Second , remove ( Z , s ) converts scores Z = ( z1 , . . . , zL ) into masks m ` with overall sparsity s. Pruning may occur in one shot ( score once and prune from sparsity 0 to s ) or iteratively ( repeatedly score unpruned weights and prune from sparsity s n−1 N to s n N over iterations n ∈ { 1 , . . . , N } ) . Re-training after pruning . After pruning at step t of training , we subsequently train the network further by repeating the entire learning rate schedule from the start ( Renda et al. , 2020 ) . Doing so ensures that , no matter the value of t , the pruned network will receive enough training to converge . Early pruning methods . We study the following methods for pruning early in training . Random . This method issues each weight a random score z ` ∼ Uniform ( 0 , 1 ) and removes weights with the lowest scores . Empirically , it prunes each layer to approximately sparsity s. Random pruning is a naive method for early pruning whose performance any new proposal should surpass . Magnitude . This method issues each weight its magnitude z ` = |w ` | as its score and removes those with the lowest scores . Magnitude pruning is a standard way to prune after training for inference ( Janowsky , 1989 ; Han et al. , 2015 ) and is an additional naive point of comparison for early pruning . SNIP ( Lee et al. , 2019 ) . This method samples training data , computes gradients g ` for each layer , issues scores z ` = |g ` w ` | , and removes weights with the lowest scores in one iteration . The justification for this method is that it preserves weights with the highest “ effect on the loss ( either positive or negative ) . ” For full details , see Appendix B . In Appendix G , we consider a recentlyproposed iterative variant of SNIP ( de Jorge et al. , 2020 ; Verdenius et al. , 2020 ) . GraSP ( Wang et al. , 2020 ) . This method samples training data , computes the Hessian-gradient product h ` for each layer , issues scores z ` = −w ` h ` , and removes weights with the highest scores in one iteration . The justification for this method is that it removes weights that “ reduce gradient flow ” and preserves weights that “ increase gradient flow. ” For full details , see Appendix C. SynFlow ( Tanaka et al. , 2020 ) . This method replaces the weights w ` with |w ` | . It computes the sum R of the logits on an input of 1 ’ s and the gradients dRdw ` of R. It issues scores z ` = | dR dw ` w ` | and removes weights with the lowest scores . It prunes iteratively ( 100 iterations ) . The justification for this method is that it meets criteria that ensure ( as proved by Tanaka et al . ) it can reach the maximal sparsity before a layer must become disconnected . For full details , see Appendix D. Benchmark methods . We use two benchmark methods to illustrate the highest known accuracy attainable when pruning to a particular sparsity in general ( not just at initialization ) . Both methods reach similar accuracy and match full accuracy at the same sparsities ( Appendix E ) . Note : we use one-shot pruning , so accuracy is lower than in work that uses iterative pruning . We do so to make a fair comparison to the early pruning methods , which do not get to train between iterations . Magnitude pruning after training . This baseline applies magnitude pruning to the weights at the end of training . Magnitude pruning is a standard method for one-shot pruning after training ( Renda et al. , 2020 ) . We compare the early pruning methods at initialization against this baseline . Lottery ticket rewinding ( LTR ) . This baseline uses the mask from magnitude pruning after training and the weights from step t. Frankle et al . ( 2020a ) show that , for t early in training and appropriate sparsities , these subnetworks reach full accuracy . This baseline emulates pruning at step t with an oracle with information from after training . Accuracy improves for t > 0 , saturating early in training ( Figure 6 , blue ) . We compare the early pruning methods after initialization against this baseline . Sparsities . We divide sparsities into three ranges ( Frankle et al. , 2020a ) . Trivial sparsities are the lowest sparsities : those where the network is so overparameterized that randomly pruning at initialization can still reach full accuracy . Matching sparsities are moderate sparsities : those where the benchmark methods can match the accuracy of the unpruned network . Extreme sparsities are those beyond . We focus on matching sparsities and the lowest extreme sparsities . Trivial sparsities are addressed by random pruning . Extreme sparsities require making subjective or context-specific tradeoffs between potential efficiency improvements of sparsity and severe drops in accuracy . Networks , datasets , and replicates . We study image classification . It is the main ( SNIP ) or sole ( GraSP and SynFlow ) task in the papers introducing the early pruning methods and in the papers introducing modern magnitude pruning ( Han et al. , 2015 ) and LTR ( Frankle et al. , 2020a ) . We use ResNet-20 and VGG-16 on CIFAR-10 , ResNet-18 on TinyImageNet , and ResNet-50 on ImageNet . See Appendix A for hyperparameters . We repeat experiments five ( CIFAR-10 ) or three ( TinyImageNet and ImageNet ) times with different seeds and plot the mean and standard deviation .

Summary: Generating a pruned network falls into two broad categories: 1) spend some extra time and effort to train or fine-tune the pruned model after first training a dense version, or 2) cut out that extra time and effort by generating a sparse network "from scratch."  While approach (1) has historically given the best accuracy, recent advances (such as the lottery ticket hypothesis) suggest that there are sparse networks hidden in the initialization that don't need to first be trained, if only we could divine the structure of those models.  Approach (2) seeks to do just this: determine the connectivity as close to initialization possible.  However, even the best results taking this second path fall short when compared to the accuracy of the former path - why is this?  The submission pokes at three recent techniques to pull out some commonalities that are *not* shared with (1), suggesting possible issues that need to be overcome to improve accuracy, and proposes a set of experiments and comparisons that should be part of any new technique that claims to discover a good sparse mask at initialization.