The Generalization-Stability Tradeoff in Neural Network Pruning

1 INTRODUCTION . Pruning weights and/or convolutional filters from deep neural networks ( DNNs ) can substantially shrink parameter counts with minimal loss in accuracy ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Han et al. , 2015a ; Li et al. , 2016 ; Molchanov et al. , 2017 ; Louizos et al. , 2017 ; Liu et al. , 2017 ; Ye et al. , 2018 ) , enabling broader application of DNNs via reductions in memory-footprint and inference-FLOPs requirements . Moreover , many pruning methods have been found to actually improve generalization ( measured by model accuracy on previously unobserved inputs ) ( Narang et al. , 2017 ; Frankle & Carbin , 2018 ; You et al. , 2019 ) . Consistent with this , pruning was originally motivated as a means to prevent over-parameterized networks from overfitting to comparatively small datasets ( LeCun et al. , 1990 ) . Concern about over-parameterizing models has weakened , however , as many recent studies have found that adding parameters can actually reduce a DNN ’ s generalization-gap ( the drop in performance when moving from previously seen to previously unseen inputs ) , even though it has been shown that the same networks have enough parameters to fit large datasets of randomized data ( Neyshabur et al. , 2014 ; Zhang et al. , 2016 ) . Potential explanations for this unintuitive phenomenon have come via experiments ( Keskar et al. , 2016 ; Morcos et al. , 2018 ; Yao et al. , 2018 ; Belkin et al. , 2018 ; Nagarajan & Kolter , 2019 ) , and the derivation of bounds on DNN generalization-gaps that suggest less overfitting might occur as parameter counts increase ( Neyshabur et al. , 2018 ) . This research has implications for neural network pruning , where a puzzling question has arisen : if larger parameter counts don ’ t increase overfitting , how does pruning parameters throughout training improve generalization ? To address this question we first introduce the notion of pruning instability , which we define to be the size of the drop in network accuracy caused by a pruning iteration ( Section 3 ) . We then empirically analyze the instability and generalization associated with various magnitude-pruning ( Han et al. , 2015b ) algorithms in different settings , making the following contributions : 1 . We find a tradeoff between the stability and potential generalization benefits of pruning , and show iterative pruning ’ s similarity to regularizing with noise—suggesting a mechanism unrelated to parameter counts through which pruning appears to affect generalization . 2 . We characterize the properties of pruning algorithms which lead to instability and correspondingly higher generalization . 3 . We derive a batch-normalized-parameter pruning algorithm to better control pruning stability . 2 RELATED WORK . There are various approaches to pruning neural networks . Pruning may be performed post-hoc ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Han et al. , 2015b ; Liu et al. , 2017 ) , or iteratively throughout training , such that there are multiple pruning events as the model trains ( Hochreiter & Schmidhuber , 1997 ; Narang et al. , 2017 ; Zhu & Gupta , 2017 ) . Most methods prune parameters that appear unimportant to the function computed by the neural network , though means of identifying importance vary . Magnitude pruning ( Han et al. , 2015b ) uses small-magnitude to indicate unimportance and has been shown to perform competitively with more sophisticated approaches ( Gale et al. , 2019 ) . Many pruning studies have shown that the pruned model has heightened generalization ( Narang et al. , 2017 ; Frankle & Carbin , 2018 ; You et al. , 2019 ) , consistent with the fact that pruning may be framed as a regularization ( rather than compression ) approach . For example , variational Bayesian approaches to pruning via sparsity-inducing priors ( Molchanov et al. , 2017 ; Louizos et al. , 2017 ) can describe weight removal as a process that reduces model description length , which in theory may help improve generalization ( Rissanen , 1978 ) . Similarly , the idea that models may be described more succinctly at flat minima has motivated pruning in service of flat minimum search ( Hochreiter & Schmidhuber , 1997 ) . Though Dinh et al . ( 2017 ) notes , however , that flatness can be arbitrarily modified by reparameterizing the function , and sharp minima can generalize well . VC dimension ( a measure of model capacity ) has motivated the use of iterative pruning to improve generalization ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ) . Overfitting can be bounded above by an increasing function of VC dimension , which itself often increases with parameter counts , so fewer parameters can lead to a guarantee of less overfitting ( Shalev-Shwartz & Ben-David , 2014 ) . Unfortunately , such bounds can be so loose in practice that tightening them by reducing parameter counts need not translate to better generalization ( Dziugaite & Roy , 2017 ) . Rather than support parameter-count-based arguments for generalization in DNNs , our results suggest iterative DNN pruning may improve generalization by creating various noisy versions of the internal representation of the data , which unpruned parameters try to fit to , as in noise-injection regularization ( Srivastava et al. , 2014 ; Poole et al. , 2014 ) . Dropout creates particularly similar noise , as it temporarily sets random subsets of layer outputs to zero ( likely changing an input ’ s internal representation every epoch ) . Indeed , applying dropout-like zeroing noise to a subset of features during training can encourage robustness to a post-hoc pruning of that subset ( Leclerc et al. , 2018 ; Gomez et al. , 2018 ) . Iterative DNN pruning noise ultimately differs , however , as it is : applied less frequently , not temporary ( except in algorithms with weight re-entry ) , usually not random , and less well studied . 3 APPROACH . Given a neural network and set of test data , let t be the top-1 test accuracy , the fraction of test data examples correctly classified multiplied by 100 . We define a pruning algorithm ’ s instability on pruning iteration i in terms of t measured immediately before ( tpre , i ) and immediately after ( tpost , i ) pruning : instabilityi = tpre , i − tpost , i . In other words , the instability is the size of the accuracy drop caused by a particular pruning event . Pruning Iterationsi− 1 i i+ 1 Train Test ( tpre , i−1 ) Prune Test ( tpost , i−1 ) Train Test ( tpre , i ) Prune Test ( tpost , i ) This measure is related to a weight ’ s importance ( sometimes referred to as “ saliency ” ; LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ) to the test accuracy , in that less stable pruning algorithms target more important sets of weights ( all else equal ) . The stability of a pruning algorithm may be affected by many factors . Our experiments ( Section 4 ) explore the effects of the following : pruning target , pruning schedule , iterative pruning rate , and model . The remainder of this section provides an overview of these factors and demonstrates a need for a novel pruning target , which we derive . 3.1 PRUNING TARGET . In all of our experiments , we use iterative magnitude pruning ( Han et al. , 2015b ) , which removes weights according to some magnitude-based rule , retrains the resulting smaller network to recover from the pruning , and repeats until the desired size reduction is met . We denote pruning algorithms that target the smallest-magnitude parameters with an `` S '' subscript ( e.g . pruneS ) , random parameters with an `` R '' subscript , and the largest-magnitude parameters with an `` L '' subscript . The usual approach to pruning involves removing parameters that have the smallest magnitudes ( Li et al. , 2016 ; Gale et al. , 2019 ) , or , similarly , those parameters least important to the loss function as determined by some other metric ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Molchanov et al. , 2016 ; 2017 ; Louizos et al. , 2017 ; Ye et al. , 2018 ; Yu et al. , 2018 ; You et al. , 2019 ) . 3.1.1 IDENTIFYING IMPORTANT BATCH-NORMALIZED PARAMETERS . The correlation between parameter magnitude and importance weakens in the presence of batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) . Without batch normalization , a convolutional filter with weights W will produce feature map activations with half the magnitude of a filter with weights 2W : filter magnitude clearly scales the output . With batch normalization , however , the feature maps are normalized to have zero mean and unit variance , and their ultimate magnitudes depend on the BN affine-transformation parameters γ and β . As a result , in batch normalized networks , filter magnitude does not scale the output , and equating small magnitude and unimportance may therefore be particularly flawed . This has motivated approaches to use the scale parameter γ ’ s magnitude to find the convolutional filters that are important to the network ’ s output ( Ye et al. , 2018 ; You et al. , 2019 ) . Here , we derive a novel approach to determining filter importance/magnitude that incorporates both γ and β . To approximate the expected value/magnitude of a batch-normalized , post-ReLU feature map activation , we start by defining the 2D feature map produced by convolution with BN : M = γBN ( W ∗ x ) + β . We approximate the activations within this feature map as Mij ∼ N ( β , γ ) . This approximation is justified if central limit theorem assumptions are met by the dot products inW ∗x , and we empirically show in Figure A.1 that this approximation is highly accurate early in training , though it becomes less accurate as training progresses . Given this approximation , the post-ReLU feature map R = max { 0 , M } has elements Rij that are either 0 or samples from a truncated normal distribution with left truncation point l = 0 , right truncation point r =∞ , and mean µ where µ = γ φ ( λ ) − φ ( ρ ) Z + β , λ = l − β γ , ρ = r − β γ , Z = Φ ( ρ ) − Φ ( λ ) , and φ ( x ) and Φ ( x ) are the standard normal distribution ’ s PDF and CDF ( respectively ) evaluated at x . Thus , an approximation to the expected value of Rij is given by E [ Rij ] ≈ Φ ( λ ) 0 + ( 1− Φ ( λ ) ) µ . We use the phrase `` E [ BN ] pruning '' to denote magnitude pruning that computes filter magnitude using this derived estimate of E [ Rij ] . E [ BN ] pruning has two advantages . First , this approach avoids the problematic assumption that filter importance is tied to filter ` 2 norm in a batch-normalized network . Accordingly , we hypothesize that E [ BN ] pruning can grant better control of the stability of the neural network ’ s output than pruning based on filters ’ ` 2 norms . Second , the complexity of the calculation is negligible as it requires ( per filter ) just a handful of arithmetic operations on scalars , and two PDF and CDF evaluations , which makes it cheaper than a data-driven approach ( e.g . approximating the expected value via the sample mean of feature map activations for a batch of feature maps ) .

Summary: This paper mainly studies the relationship between the generalization error and mean/variance of the test accuracy. The authors first propose a new score for pruning called E[BN]. Then, the authors observe the generalization error and the test accuracy mean/variance for pruning large score weights and small score weights for VGG11, ResNet18, and Conv4 models. From these experiments, the authors observe that pruning large score weights generates instable but high test accuracy and smaller generalization gap compared to pruning small score weights. The authors additionally study some other aspects of pruning (e.g., pruning as a noise injection) and conclude the paper.