Pruning Neural Networks at Initialization: Why Are We Missing the Mark?

1 INTRODUCTION . Since the 1980s , we have known that it is possible to eliminate a significant number of parameters from neural networks without affecting accuracy at inference-time ( Reed , 1993 ; Han et al. , 2015 ) . Such neural network pruning can substantially reduce the computational demands of inference when conducted in a fashion amenable to hardware ( Li et al. , 2017 ) or combined with libraries ( Elsen et al. , 2020 ) and hardware designed to exploit sparsity ( Cerebras , 2019 ; NVIDIA , 2020 ; Toon , 2020 ) . When the goal is to reduce inference costs , pruning often occurs late in training ( Zhu & Gupta , 2018 ; Gale et al. , 2019 ) or after training ( LeCun et al. , 1990 ; Han et al. , 2015 ) . However , as the financial , computational , and environmental demands of training ( Strubell et al. , 2019 ) have exploded , researchers have begun to investigate the possibility that networks can be pruned early in training or even before training . Doing so could reduce the cost of training existing models and make it possible to continue exploring the phenomena that emerge at larger scales ( Brown et al. , 2020 ) . There is reason to believe it may be possible to prune early in training without affecting final accuracy . Work on the lottery ticket hypothesis ( Frankle & Carbin , 2019 ; Frankle et al. , 2020a ) shows that , from early in training ( although often after initialization ) , there exist subnetworks that can train in isolation to full accuracy ( Figure 1 , red line ) . These subnetworks are as small as those found by inference-focused pruning methods after training ( Appendix E ; Renda et al. , 2020 ) , raising the prospect that it may be possible to maintain this level of sparsity for much or all of training . However , this work does not suggest a way to find these subnetworks without first training the full network . The pruning literature offers a starting point for finding such subnetworks efficiently . Standard networks are often so overparameterized that pruning randomly has little effect on final accuracy at lower sparsities ( green line ) . Moreover , many existing pruning methods prune during training ( Zhu & Gupta , 2018 ; Gale et al. , 2019 ) , even if they were designed with inference in mind ( orange line ) . Recently , several methods have been proposed specifically for pruning at initialization . SNIP ( Lee et al. , 2019 ) aims to prune weights that are least salient for the loss . GraSP ( Wang et al. , 2020 ) aims to prune weights that most harm or least benefit gradient flow . SynFlow ( Tanaka et al. , 2020 ) aims to iteratively prune weights with the lowest “ synaptic strengths ” in a data-independent manner with the goal of avoiding layer collapse ( where pruning concentrates on certain layers ) . In this paper , we assess the efficacy of these pruning methods at initialization . How do SNIP , GraSP , and SynFlow perform relative to each other and naive baselines like random and magnitude pruning ? How does this performance compare to methods for pruning after training ? Looking ahead , are there broader challenges particular to pruning at initialization ? Our purpose is to clarify the state of the art , shed light on the strengths and weaknesses of existing methods , understand their behavior in practice , set baselines , and outline an agenda for the future . We focus at and near matching sparsities : those where magnitude pruning after training matches full accuracy.1 We do so because : ( 1 ) these are the sparsities typically studied in the pruning literature , and ( 2 ) for magnitude pruning after training , this is a tradeoff-free regime where we do not have to balance the benefits of sparsity with sacrifices in accuracy . Our experiments ( summarized in Figure 2 ) and findings are as follows : The state of the art for pruning at initialization . The methods for pruning at initialization ( SNIP , GraSP , SynFlow , and magnitude pruning ) generally outperform random pruning . No single method is SOTA : there is a network , dataset , and sparsity where each pruning method ( including magnitude pruning ) reaches the highest accuracy . SNIP consistently performs well , magnitude pruning is surprisingly effective , and competition increases with improvements we make to GraSP and SynFlow . Magnitude pruning after training outperforms these methods . Although this result is not necessarily surprising ( after all , these methods have less readily available information upon which to prune ) , it raises the question of whether there may be broader limitations to the performance achievable when pruning at initialization . In the rest of the paper , we study this question , investigating how these methods differ behaviorally from standard results about pruning after training . Methods prune layers , not weights . The subnetworks that these methods produce perform equally well ( or better ) when we randomly shuffle the weights they prune in each layer ; it is therefore possible to describe a family of equally effective pruning techniques that randomly prune the network in these per-layer proportions . The subnetworks that these methods produce also perform equally well when we randomly reinitialize the unpruned weights . These behaviors are not shared by stateof-the-art weight-pruning methods that operate after training ; both of these ablations ( shuffling and reinitialization ) lead to lower accuracy ( Appendix F ; Han et al. , 2015 ; Frankle & Carbin , 2019 ) . These results appear specific to pruning at initialization . There are two possible reasons for the comparatively lower accuracy of these methods and for the fact that the resulting networks are insensitive to the ablations : ( 1 ) these behaviors are intrinsic to subnetworks produced by these methods or ( 2 ) these behaviors are specific to subnetworks produced by these methods at initialization . We eliminate possibility ( 1 ) by showing that using SNIP , SynFlow , and magnitude to prune the network after initialization leads to higher accuracy ( Section 6 ) and sensitivity to the ablations ( Appendix F ) . This result means that these methods encounter particular difficulties when pruning at initialization . Looking ahead . These results raise the question of whether it is generally difficult to prune at initialization in a way that is sensitive to the shuffling or reinitialization ablations . If methods that maintain their accuracy under these ablations are inherently limited in their performance , then there may be broader limits on the accuracy attainable when pruning at initialization . Even work on lottery tickets , which has the benefit of seeing the network after training , reaches lower accuracy and is unaffected by these ablations when pruning occurs at initialization ( Frankle et al. , 2020a ) . Although accuracy improves when using SNIP , SynFlow , and magnitude after initialization , it does not match that of the full network unless pruning occurs nearly halfway into training ( if at all ) . This means that ( 1 ) it may be difficult to prune until much later in training or ( 2 ) we need new methods designed to prune early in training ( since SNIP , GraSP , and SynFlow were not intended to do so ) . 1Tanaka et al . design SynFlow to avert layer collapse , which occurs at higher sparsities than we consider . However , they also evaluate at our sparsities , so we believe this is a reasonable setting to study SynFlow . 2 RELATED WORK . Neural network pruning dates back to the 1980s ( survey : Reed , 1993 ) , although it has seen a recent resurgence ( survey : Blalock et al. , 2020 ) . Until recently , pruning research focused on improving efficiency of inference . However , methods that gradually prune throughout training provide opportunities to improve the efficiency of training as well ( Zhu & Gupta , 2018 ; Gale et al. , 2019 ) . Lottery ticket work shows that there are subnetworks before ( Frankle & Carbin , 2019 ) or early in training ( Frankle et al. , 2020a ) that can reach full accuracy . Several recent papers have proposed efficient ways to find subnetworks at initialization that can train to high accuracy . SNIP ( Lee et al. , 2019 ) , GraSP ( Wang et al. , 2020 ) , SynFlow ( Tanaka et al. , 2020 ) , and NTT ( Liu & Zenke , 2020 ) prune at initialization . De Jorge et al . ( 2020 ) and Verdenius et al . ( 2020 ) apply SNIP iteratively ; Cho et al . ( 2020 ) use SNIP for pruning for inference . Work on dynamic sparsity maintains a pruned network throughout training but regularly changes the sparsity pattern ( Mocanu et al. , 2018 ; Dettmers & Zettlemoyer , 2019 ; Evci et al. , 2019 ) ; to match the accuracy of standard methods for pruning after training , Evci et al . needed to train for five times as long is standard for training the unpruned networks . You et al . ( 2020 ) prune after some training ; this research is not directly comparable to any of the aforementioned papers as it prunes channels ( rather than weights as in all work above ) and does so later ( 20 epochs ) than SNIP/GraSP/SynFlow ( 0 ) and lottery tickets ( 1-2 ) . 3 METHODS . Pruning . Consider a network with weights w ` ∈ Rd ` in each layer ` ∈ { 1 , . . . , L } . Pruning produces binary masks m ` ∈ { 0 , 1 } d ` . A pruned subnetwork has weights w ` m ` , where is the element-wise product . The sparsity s ∈ [ 0 , 1 ] of the subnetwork is the fraction of weights pruned : 1− ∑ ` m ` / ∑ ` d ` . We study pruning methods prune ( W , s ) that prune to sparsity s using two operations . First , score ( W ) issues scores z ` ∈ Rd ` to all weights W = ( w1 , . . . , wL ) . Second , remove ( Z , s ) converts scores Z = ( z1 , . . . , zL ) into masks m ` with overall sparsity s. Pruning may occur in one shot ( score once and prune from sparsity 0 to s ) or iteratively ( repeatedly score unpruned weights and prune from sparsity s n−1 N to s n N over iterations n ∈ { 1 , . . . , N } ) . Re-training after pruning . After pruning at step t of training , we subsequently train the network further by repeating the entire learning rate schedule from the start ( Renda et al. , 2020 ) . Doing so ensures that , no matter the value of t , the pruned network will receive enough training to converge . Early pruning methods . We study the following methods for pruning early in training . Random . This method issues each weight a random score z ` ∼ Uniform ( 0 , 1 ) and removes weights with the lowest scores . Empirically , it prunes each layer to approximately sparsity s. Random pruning is a naive method for early pruning whose performance any new proposal should surpass . Magnitude . This method issues each weight its magnitude z ` = |w ` | as its score and removes those with the lowest scores . Magnitude pruning is a standard way to prune after training for inference ( Janowsky , 1989 ; Han et al. , 2015 ) and is an additional naive point of comparison for early pruning . SNIP ( Lee et al. , 2019 ) . This method samples training data , computes gradients g ` for each layer , issues scores z ` = |g ` w ` | , and removes weights with the lowest scores in one iteration . The justification for this method is that it preserves weights with the highest “ effect on the loss ( either positive or negative ) . ” For full details , see Appendix B . In Appendix G , we consider a recentlyproposed iterative variant of SNIP ( de Jorge et al. , 2020 ; Verdenius et al. , 2020 ) . GraSP ( Wang et al. , 2020 ) . This method samples training data , computes the Hessian-gradient product h ` for each layer , issues scores z ` = −w ` h ` , and removes weights with the highest scores in one iteration . The justification for this method is that it removes weights that “ reduce gradient flow ” and preserves weights that “ increase gradient flow. ” For full details , see Appendix C. SynFlow ( Tanaka et al. , 2020 ) . This method replaces the weights w ` with |w ` | . It computes the sum R of the logits on an input of 1 ’ s and the gradients dRdw ` of R. It issues scores z ` = | dR dw ` w ` | and removes weights with the lowest scores . It prunes iteratively ( 100 iterations ) . The justification for this method is that it meets criteria that ensure ( as proved by Tanaka et al . ) it can reach the maximal sparsity before a layer must become disconnected . For full details , see Appendix D. Benchmark methods . We use two benchmark methods to illustrate the highest known accuracy attainable when pruning to a particular sparsity in general ( not just at initialization ) . Both methods reach similar accuracy and match full accuracy at the same sparsities ( Appendix E ) . Note : we use one-shot pruning , so accuracy is lower than in work that uses iterative pruning . We do so to make a fair comparison to the early pruning methods , which do not get to train between iterations . Magnitude pruning after training . This baseline applies magnitude pruning to the weights at the end of training . Magnitude pruning is a standard method for one-shot pruning after training ( Renda et al. , 2020 ) . We compare the early pruning methods at initialization against this baseline . Lottery ticket rewinding ( LTR ) . This baseline uses the mask from magnitude pruning after training and the weights from step t. Frankle et al . ( 2020a ) show that , for t early in training and appropriate sparsities , these subnetworks reach full accuracy . This baseline emulates pruning at step t with an oracle with information from after training . Accuracy improves for t > 0 , saturating early in training ( Figure 6 , blue ) . We compare the early pruning methods after initialization against this baseline . Sparsities . We divide sparsities into three ranges ( Frankle et al. , 2020a ) . Trivial sparsities are the lowest sparsities : those where the network is so overparameterized that randomly pruning at initialization can still reach full accuracy . Matching sparsities are moderate sparsities : those where the benchmark methods can match the accuracy of the unpruned network . Extreme sparsities are those beyond . We focus on matching sparsities and the lowest extreme sparsities . Trivial sparsities are addressed by random pruning . Extreme sparsities require making subjective or context-specific tradeoffs between potential efficiency improvements of sparsity and severe drops in accuracy . Networks , datasets , and replicates . We study image classification . It is the main ( SNIP ) or sole ( GraSP and SynFlow ) task in the papers introducing the early pruning methods and in the papers introducing modern magnitude pruning ( Han et al. , 2015 ) and LTR ( Frankle et al. , 2020a ) . We use ResNet-20 and VGG-16 on CIFAR-10 , ResNet-18 on TinyImageNet , and ResNet-50 on ImageNet . See Appendix A for hyperparameters . We repeat experiments five ( CIFAR-10 ) or three ( TinyImageNet and ImageNet ) times with different seeds and plot the mean and standard deviation .

Summary: A recent trend of 'pruning at initialization' in neural network pruning has left me baffled. It's counter-intuitive that neural networks can be pruned at initialisation, improving results for the training done thereafter. Nitpicking semantics, one could hardly even call this a pruning technique, since there is no a-priori knowledge of the dataset distilled in the network. Perhaps it's more aptly referred to as a method of sparse initialisation methods.