On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks

1 INTRODUCTION . Deep neural architectures have seen a dramatic increase in size over the years ( Amodei & Hernandez , 2018 ) . Although not entirely understood , it is known that over-parametrized networks exhibit high generalization performance , with recent empirical evidence showing that the generalization gap tends to close with increased number of parameters ( Zhang et al. , 2017 ; Belkin et al. , 2018 ; Hastie et al. , 2019 ; Allen-Zhu et al. , 2018 ; Du et al. , 2018 ; Du & Lee , 2018 ) , contrary to prior belief , also depending on the inductive bias and sensitivity to memorization of each base architecture ( Zhang et al. , 2019 ) . While advantageous under this point of view , the proliferation of parameters in neural architectures may induce adverse consequences . The computational cost to train some state-of-the-art models has raised the barrier to entry for many researchers hoping to contribute . Because of limited memory , time , and compute , and to enable private , secure , on-device computation , methods for model compression have seen a rise in popularity . Among these are techniques for model pruning , quantization , and distillation . Pruning , in particular , has been seen as an overfitting avoidance method since the early decision tree literature ( Breiman et al. , 1984 ) , with work such as that of Mingers ( 1989 ) comparing the effects of different tree pruning techniques . In analogy to this prior line of work , we provide an empirical comparison of pruning methods for neural networks . As explicitly pointed out in LeCun et al . ( 1990b ) , there is no obvious , natural order in which neural network parameters should be removed . Indeed , as Schaffer ( 1993 ) suggests , identifying the best network pruning technique should give way to understanding what accounts for their success . Revived by the recent observation of the existence of sub-networks with favorable training properties within larger over-parametrized models ( ‘ lottery ’ or ‘ winning tickets ’ ( Frankle & Carbin , 2018 ) ) , and in the spirit of Zhou et al . ( 2019 ) , this work investigates the dependence of the properties of these sparse , lucky sub-networks on the choice of pruning technique . We set out to answer the questions : do pruning methods other than magnitude-based unstructured pruning give rise to winning tickets ? If so , do they identify the same lucky sub-network , or are there many equiperforming winning tickets within the same over-parametrized network ? This allows us to categorize pruning methods based on the similarity of masks they generate . We develop methods to further analyze the nature of trainable sparse networks found through common iterative pruning techniques , and provide new insight towards a deeper understanding of their properties and of the processes that generate them . Without addressing issues related to scalability and emergence of lottery tickets in large-scale domains , we focus instead on the empirical characterization of weight evolution and emergence of distinctive connectivity patterns in small architectures , such as LeNet ( LeCun et al. , 1990a ) , on simple datasets , such as MNIST ( LeCun et al. , 1994 ) . This also helps develop analysis techniques for sanity checks via visual inspection . We then apply the same techniques to larger architectures and more complex tasks to explore whether findings hold in different regimes . 1.1 CONTRIBUTIONS . This work provides empirical evidence showing that : 1. there can exist multiple different lucky sub-networks ( lottery tickets ) within an overparametrized network ; 2. it is possible to find a lucky sub-network through a variety of choices of pruning techniques ; 3. lottery ticket-style weight rewinding , coupled with unstructured pruning , gives rise to connectivity patterns similar to the ones obtained with structured pruning along the input dimension , thus pointing to an input feature selection type of effect . This is not the case when rewinding is substituted by finetuning ; 4. random structured pruning outperforms random unstructured pruning , meaning that networks are more resistant to the removal of random units/channels than to the removal of random individual connections ; 5. different iterative pruning techniques learn vastly different functions of their input , and similarly performing networks make different mistakes on held-out test sets , hinting towards the utility of ensembling in this setting ; 6. weight stability to pruning correlates with performance , and can be induced through the use of suitable pruning techniques , even without late resetting . 2 RELATED WORK . Identifying important units in a neural network is an open challenge in machine learning , driven by desires for interpretability and model reduction , among others . Various proxies for importance have been proposed as a direct consequence of the lack of precise mathematical definition of this term . Different objectives lead to different mathematical formulations of importance : concept creation , output attribution , conductance , saliency , activation , explainability , connectivity density , redundancy , noise robustness , stability , and more ( Berglund et al. , 2013 ; Amjad et al. , 2018 ; Zhou et al. , 2018 ; Dhamdhere et al. , 2018 ; Shrikumar et al. , 2017 ) . State-of-the-art results in the field have seemed to strongly correlate with increased number of parameters , leading to the common use of vastly over-parametrized models , compared to the complexity of the tasks at hand . In practice , however , most common neural network architectures can be dramatically pruned down in size ( Hanson & Pratt , 1989 ; LeCun et al. , 1990b ; Hassibi & Stork , 1993 ; Han et al. , 2015 ; Guo et al. , 2016 ; Yang et al. , 2016 ; Luo et al. , 2017 ; Tung et al. , 2017 ; Zhang et al. , 2018 ; Ayinde et al. , 2019 ; Mussay et al. , 2019 ) . Unlike pruned pre-trained and then finetuned models , smaller untrained models are , in general , empirically harder to train from scratch . Frankle & Carbin ( 2018 ) conjecture that there exists at least one sparse sub-network , a winning ticket , within an over-parametrized network , that can achieve commensurate accuracy in commensurate training time with fewer parameters , if retrained from the same initialization used to find the pruning mask . To find this lucky sub-network , they iteratively prune the lowest magnitude unpruned connections after a fixed number of training iterations ( LeCun et al. , 1990b ; Han et al. , 2015 ) . Interestingly , while pruned weights are set to zero , unpruned weights are rewound to their initialization values , prior to the next iteration of training and pruning . Researchers have encountered difficulties in scaling this method to larger models . Liu et al . ( 2018 ) were unable to successfully train small sparse sub-networks found through this procedure , when starting from VGG-16 ( Simonyan & Zisserman , 2014 ) and ResNet-50 ( He et al. , 2015 ) trained on CIFAR-10 ( Krizhevsky , 2009 ) , and instead observed that random initialization outperformed weight rewinding . Similarly , Gale et al . ( 2019 ) were unable to successfully find a sub-network with these properties within ResNet-50 trained on ImageNet ( Russakovsky et al. , 2015 ) and Transformer ( Vaswani et al. , 2017 ) trained on WMT 2014 English-to-German1 . Slight modifications to the original procedure have been proposed in order to favor the emergence of winning tickets in larger models and tasks . Zhou et al . ( 2019 ) challenge the idea that unstructured , magnitude-based pruning is necessary for the emergence of winning tickets . They instead prune weights based on their change in magnitude between initialization and the end of training , and reset the surviving weights to their sign at initialization times the standard deviation of all original weights in the layer . Frankle & Carbin ( 2018 ) note that larger models such as ResNet-18 and VGG-19 require selecting unimportant weights globally instead of layer by layer , and recommend the use of learning rate warmup . Frankle et al . ( 2019 ) further add that late resetting of the unpruned parameters to values achieved early in training appears to be superior to rewinding the weights all the way to their initial values . Morcos et al . ( 2019 ) confirm the findings on the efficacy of global pruning and late resetting . Although hard to find in larger regimes , when found , lottery tickets have been shown to possess generalization properties that allow for their reuse in similar tasks , thus reducing the computational cost of finding task- and dataset-dependent sparse sub-networks ( Morcos et al. , 2019 ) . On the contrary , alternative approaches to neural architecture optimization have taken a constructionist approach , starting from a minimal set of units and connections , and growing the network by adding new components ( Stanley & Miikkulainen , 2002 ; Xie et al. , 2019 ) . Finally , other methods combine both network growing and pruning , in analogy to the different phases of connection formation and suppression in the human brain ( Floreano et al. , 2008 ; Dai et al. , 2019 ) . 3 METHOD . All networks in this section are trained for 30 epochs using SGD with constant learning rate 0.01 , batch size of 32 , without explicit regularization . The pruning fraction per iteration is held constant at 20 % of remaining connections/units per layer . All experiment and analysis code is publicly available2 . 3.1 PRUNING METHODS . This works explores a variety of pruning techniques that may differ along the following axes : Neuronal Importance Definition : In magnitude-based pruning , units/connections are removed based on the magnitude of synaptic weights . Usually , low magnitude parameters are removed . As a ( rarer ) alternative , one can consider removing high magnitude weights instead ( Zhou et al. , 2019 ) . Non-magnitude-based pruning techniques , instead , can be based , among others , on activations , gradients , or custom rules for neuronal importance . Local vs. global : Local pruning consists of removing a fixed percentage of units/connections from each layer by comparing each unit/connection exclusively to the other units/connections in the layer . On the contrary , global pruning pools all parameters together across layers and selects a global fraction of them to prune . The latter is particularly beneficial in the presence of layers with unequal parameter distribution , by redistributing the pruning load more equitably . A middle-ground approach is to pool together only parameters belonging to layers of the same kind , to avoid mixing , say , convolutional and fully-connected layers . Unstructured vs. structured : Unstructured pruning removes individual connections , while structured pruning removes entire units or channels . Note that structured pruning along the input axis is conceptually similar to input feature importance selection . Similarly , structured pruning along the output axis is analogous to output suppression . In this work , we compare : magnitude-based { L1 , random } unstructured ( US ) , { L1 , L2 , L−∞ , random } structured ( S ) , and hybrid pruning . The hybrid techniques consists of pruning convolutional 1Translation Task - ACL 2014 Ninth Workshop on Statistical Machine Translation ( URL : https : //www . statmt.org/wmt14/translation-task.html ) 2in anonymized form , at github.com/iclr-8dafb2ab/iterative-pruning-reinit layers with L1 structured pruning and fully-connected layers with L1 unstructured pruning . “ fconly ” identifies experiments in which only the fully-connected layers are pruned . Structured pruning is performed along the input axis . All techniques are local . We implement these pruning techniques and train our models using PyTorch ( Paszke et al. , 2017 ) .

Summary: This paper study the lottery ticket hypothesis by observing the properties of lottery tickets. In particular, the authors tested several different pruning techniques by varying evaluation criteria (L_1, L_2, L_-\infty and random) and pruning structures (structured, unstructured and hybrid). The authors perform experiments mainly on LeNet with the MNIST dataset and analyze the observations.