On the Predictability of Pruning Across Scales

1 INTRODUCTION . For decades , neural network pruning—eliminating unwanted parts of the network—has been a popular approach for reducing network sizes or computational demands of inference ( LeCun et al. , 1990 ; Reed , 1993 ; Han et al. , 2015 ) . In practice , pruning can reduce the parameter-counts of contemporary models by 2x ( Gordon et al. , 2020 ) to 5x ( Renda et al. , 2020 ) with no reduction in accuracy . More than 80 pruning techniques have been published in the past decade ( Blalock et al. , 2020 ) , but , despite this enormous volume of research , there remains little guidance on important aspects of pruning . Consider a seemingly simple question one might ask when using a particular pruning technique : Given a family of neural networks ( e.g. , ResNets on ImageNet of various widths and depths ) , which family member should we prune ( and by how much ) to obtain the network with the smallest parametercount such that error does not exceed some threshold k ? As a first try , we could attempt to answer this question using brute force : we could prune every member of a family ( i.e. , perform grid search over widths and depths ) and select the smallest pruned network that satisfies our constraint on error . However , depending on the technique , pruning one network ( let alone grid searching ) could take days or weeks on expensive hardware . If we want a more efficient alternative , we will need to make assumptions about pruned networks : namely , that there is some structure to the way that their error behaves . For example , that pruning a particular network changes the error in a predictable way . Or that changing the width or depth of a network changes the error when pruning it in a predictable way . We could then train a smaller number of networks , characterize this structure , and estimate the answer to our question . We have reason to believe that such structure does exist for pruning : techniques already take advantage of it implicitly . For example , Cai et al . ( 2019 ) create a single neural network architecture that can be scaled down to many different sizes ; to choose which subnetwork to deploy , Cai et al . train an auxiliary , black-box neural network to predict subnetwork performance . Although this black-box approach implies the existence of structure , it does not reveal this structure explicitly or make it possible to reason analytically in a fashion that could answer our research question . Outside the context of pruning algorithms , such structure has been observed—and further codified explicitly—yielding insights and predictions in the form of scaling laws . Tan and Le ( 2019 ) design the EfficientNet family by developing a heuristic for predicting efficient tradeoffs between depth , width , and resolution . Hestness et al . ( 2017 ) observe a power-law relationship between dataset size and the error of vision and NLP models . Rosenfeld et al . ( 2020 ) use a power scaling law to predict the error of all variations of architecture families and dataset sizes jointly , for computer vision and natural language processing settings . Kaplan et al . ( 2020 ) develop a similar power law for language models that incorporates the computational cost of training . Inspired by this line of work , we address our research question about pruning by developing a scaling law to predict the error of networks as they are pruned . To the best of our knowledge , no explicit scaling law holding over pruning algorithms and network types currently exists . In order to formulate such a predictive scaling law , we consider the dependence of generalization error on the pruning-induced density for networks of different depths and width trained on different dataset sizes . We begin by developing a functional form that accurately estimates the generalization error of a specific model as it is pruned ( Section 3 ) . We then account for other architectural degrees of freedom , expanding the functional form for pruning into a scaling law that jointly considers density alongside width , depth , and dataset size ( Section 4 ) . The basis for this joint scaling law is an invariant we uncover that describes ways that we can interchange depth , width , and pruning without affecting error . The result is a scaling law that accurately predicts the performance of pruned networks across scales . Finally , we use this scaling law to answer our motivating question ( Section 7 ) . The same functional form can accurately estimate the error for both unstructured magnitude pruning ( Renda et al. , 2020 ) and SynFlow ( Tanaka et al. , 2020 ) when fit to the corresponding data , suggesting we have uncovered structure that may be applicable to iterative pruning more generally . And now that we have established this functional form , fitting it requires only a small amount of data ( Appendix 5 ) . In summary , our contributions are as follows : • We develop a scaling law that accurately estimates the error when pruning a single network . • We observe and characterize an invariant that allows error-preserving interchangeability among depth , width , and pruning density . • Using this invariant , we extend our single-network scaling law into a joint scaling law that predicts the error of all members of a network family at all dataset sizes and all pruning densities . • In doing so , we demonstrate that there is structure to the behavior of the error of iteratively pruned networks that we can capture explicitly with a simple functional form . • Our scaling law enables a framework for reasoning analytically about pruning , allowing us to answer our motivating question and similar questions about pruning . 2 EXPERIMENTAL SETUP . Pruning . We study two techniques for pruning neural networks : iterative magnitude pruning ( IMP ) ( Janowsky , 1989 ; Han et al. , 2015 ; Frankle et al. , 2020 ) in the main body of the paper and SynFlow ( Tanaka et al. , 2020 ) in Appendix E. We describe IMP in detail here and SynFlow in Appendix A. IMP prunes by removing a fraction—typically 20 % , as we do here—of individual weights with the lowest magnitudes at the end of training.1 We choose these weights globally throughout the network , i.e. , without regard to specific layers . We use per-weight magnitude pruning because it is generic , well-studied ( Han et al. , 2015 ) , and matches the sparsity/accuracy tradeoffs of more complicated methods ( Gale et al. , 2019 ; Blalock et al. , 2020 ; Renda et al. , 2020 ) . Pruning weights typically reduces the accuracy of the trained network , so it is standard practice to further train after pruning to recover accuracy . For IMP , we use a practice called weight rewinding , in which the values of unpruned weights are rewound to their values at epoch 10 and the training process is repeated from there to completion . To achieve density levels below 80 % , this process is repeated iteratively—pruning by 20 % , rewinding , and retraining—until a desired density level is reached . Renda et al . ( 2020 ) demonstrate that IMP with weight rewinding achieves state-of-the-art tradeoffs between sparsity and accuracy . For a formal statement of this pruning algorithm , see Appendix A. Datasets . We study the image classification tasks CIFAR-10 and ImageNet . Our scaling law predicts the error when training with the entire dataset and smaller subsamples . To subsample a dataset to a size of n , we randomly select n of the training examples without regard to individual classes such 1We do not prune biases or BatchNorm , so pruning 20 % of weights prunes fewer than 20 % of parameters . that in expectation we preserve the original dataset distribution ( we always retain the entire test set ) . When performing iterative pruning , we maintain the same subsample for all pruning iterations . Networks . We study three families of neural networks : ResNets for CIFAR-10 , ResNets for ImageNet , and ( in Appendix E ) VGG-style networks for CIFAR-10.2 We develop a scaling law that predicts the error ( when pruned ) of an entire family of networks with varying widths and—in the case of the CIFAR-10 ResNets—depths . To vary width , we multiply the number of channels in each layer by a width scaling factor . To vary depth of the CIFAR-10 ResNets , we vary the number of residual blocks . We refer to a network by its depth l ( the number of layers in the network , not counting skip connections ) and its width scaling factor w. Notation and terminology . Throughout the paper , we use the following notation and terminology : • DN = { xi , yi } Ni=1 is a labeled training set with N examples . A subsample of size n is a subset of DN containing n examples selected uniformly at random . • l and w are , respectively , the depth ( i.e. , the number of layers , excluding skip connections ) and the width scaling factor of a particular network . • A collection of networks that vary by width and depth are a network family . • s is the sparsity of a pruned network ( i.e. , the fraction of weights that have been pruned ) and d , 1− s is the density ( i.e. , the fraction of weights that have not been pruned ) . • ( d , l , w , n ) is the test error of a network with the specified density , depth , width scaling factor , and dataset size . • np ( l , w , n ) = ( 1 , l , w , n ) is the test error of the unpruned network with the specified depth , width scaling factor , and dataset size . When clear from context , we omit ( w , l , n ) and write np . • ̂ ( np , d | l , w , n ) is an estimate of the error of a pruned model for a scaling law that has been fit to a specific network with the specified depth , width scaling factor , and dataset size ( Section 3 ) . • ̂ ( np , d , l , w , n ) is an estimate of the error of a pruned model with the specified depth , width scaling factor , and dataset size for a scaling law that has been fit to a network family ( Section 4 ) . Dimensions . In developing scaling laws , we vary four different dimensions : dataset subsample size ( n ) and network degrees of freedom density ( d ) , network depth ( l ) , and width scaling factor ( w ) . We consider the following ranges of these values in our experiments in the main body of the paper : Network Family Ntrain Ntest Densities ( d ) Depths ( l ) Width Scalings ( w ) Subsample Sizes ( n ) CIFAR-10 ResNet 50K 10K 0.8i , i ∈ { 0 , . . . , 40 } 8 , 14 , 20 , 26 , 50 , 98 2i , i ∈ { −4 , . . . , 2 } Ni , i ∈ { 1 , 2 , 4 , 8 , 16 , 32 , 64 } ImageNet ResNet 1.28M 50K 0.8i , i ∈ { 0 , . . . , 30 } 50 2i , i ∈ { −4 , . . . , 0 } Ni , i ∈ { 1 , 2 , 4 } We use sanity checks to filter infeasible or unusual configurations from this table . In many cases , networks become disconnected before we reach the lowest density ( e.g. , ∼30 % of the CIFAR-10 ResNet configurations ) . We also eliminate configurations where increasing width or depth of the unpruned network lowers test accuracy ( e.g. , 144 of the 294 CIFAR-10 ResNet configurations of l , w , and n ) ; these are typically unusual , imbalanced configurations ( e.g. , l = 98 , w = 116 ) . Of the 12,054 possible CIFAR-10 ResNet configurations , about 8,000 are eliminated based on these sanity checks .

Summary: This paper studies how to estimate the performance of pruned networks using regression models. The authors first empirically observe that there exist three distinct regions of sparsity: (1) In the low-sparsity regime, pruning does not decrease the accuracy (2) In the mid-sparsity regime, a linear relationship between the sparsity and the accuracy is observed (3) In the high-sparsity regime, pruning does not decrease the accuracy again. Under this observation, the authors proposed a regression model called the rational family and empirically verified its performance. The authors further extended this model to incorporate the network width and depth under some empirical observation called the error-preserving invariant. The authors performed experiments to verify different perspectives of the proposed functional form.