An empirical study of a pruning mechanism

1 INTRODUCTION . Deep learning is currently one of the most powerful machine learning methods . It requires neural network to train , which usually takes a few to hundreds times more weights than training data ( He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ; Huang et al. , 2017 ; Karen & Andrew , 2015 ) . Usually , in common regimes , a greater number of weights leads to better performance ( Zagoruyko & Komodakis , 2016 ) . However , paradoxically , neural networks are also compressible . Many of the recent pruning methods aim to maximally compress the networks ( Han et al. , 2015 ; Liu et al. , 2017 ; He et al. , 2019 ; You et al. , 2019 ) , however , there are few works that investigate why and how the pruning mechanism works ( Frankle et al. , 2019 ; Elesedy et al. , 2020 ) . In this work , we empirically investigate a standard framework for network pruning : pretraining a large network and then pruning and retraining it . The framework has been commonly used based on heuristics , i.e . finding a good minima with a larger network and retaining it with careful pruning and retraining ( Han et al. , 2015 ; Liu et al. , 2017 ) . We investigate the heuristic in two parts , i.e. , one for the pretraining phase and the other for the pruning and retraining phase . For the pretraining phase , the reason for training the large network to obtain a good minima is investigated . Since the neural network is generally compressible , the pretrained large network can be pruned to a smaller one . However , a network with the same number of weights as that of the pruned network can not achieve similar performance when trained from scratch ( Frankle & Carbin , 2018 ) . We conjecture that this comes from the networks not utilizing all of their weights . Thus we hypothesize : if trained from scratch , there is a utility imbalance among the weights in neural network . For investigation , the measures for the weight utility and the utility imbalance are proposed . Thereafter , the cause of the utility imbalance and the characteristics of the weight utility in various conditions are examined . For the pruning and retraining phase , we verify the heuristic that once a good minima is obtained with the large network , it can be retained by careful pruning and retraining ( Han et al. , 2015 ; Renda et al. , 2020 ) . Our investigation is based on the loss surface visualization on a two dimensional plane formed by three points in the weight space where each point represents the pretrained network , the pruned network , and the pruned and retrained network . We examine ( 1 ) the dynamics of the network on the loss surface throughout pruning and ( 2 ) the validation accuracy of the networks over varying pruning methods and retraining methods . Contributions .. • The utility imbalance among the weights increases during optimization . • The neural networks utilize the weights in proportion to their size . • If a pretrained network is carefully pruned and retrained , then the pruned-and-retrained network shares the same loss basin with the pretrained network . 2 WEIGHT UTILITY ANALYSIS FOR THE PRETRAIN MECHANISM . Then why do we have to train a large network and then prune to a smaller one ? Why not just train the smaller one to get the performance we need ? Why is it difficult ? The investigation about the questions starts with a hypothesis : let Nlarge be a large network that does not utilize all of its weights , and thus can be easily compressed into a smaller network Npruned with minimal loss change . And let Nsmall be a network trained from scratch , whose number of weights is comparable to that ofNpruned , which is sufficient to achieve a similar level of loss to those ofNlarge orNpruned . However , Nsmall generally performs worse , because Nsmall does not utilize all of its weights either . Therefore , we hypothesize that the neural network does not utilize all of its weights when trained from scratch in general . And we refer to the phenomenon which the neural network utilize the weights unevenly as utility imbalance . Thus , Main Hypothesis . If trained from scratch , there is utility imbalance among the weights in a neural network . And we empirically measure the utility of weights as : Definition 1 ( Utility measure ) . LetW be a set of total weights in a networkN , Ws be a subset ofW , and X be a dataset . Suppose fW ( x ) and fW\Ws ( x ) are probability mass functions resulting from a softmax layer , where x ∼ X is an input and fW\Ws ( x ) is obtained by zeroing out the weights in Ws . Then , the utility of Ws can be measured as U ( Ws ) = E x∼X [ dKL ( fW ( x ) , fW\Ws ( x ) ) ] , where dKL is KL-divergence . For reference , the way of the measurement , i.e. , network ablation , was similarly done in ( Casper et al. , 2019 ; 2020 ; Meyes et al. , 2019 ; Cheney et al. , 2017 ) . We also define the utility imbalance as : . Definition 2 ( Utility imbalance ) . For Wi ⊂ W and Wj ⊂ W , we say there is δ utility imbalance between the subsets of weights when |U ( Wi ) − U ( Wj ) | > δ . Where we empirically measure the utility imbalance by : Definition 3 ( Utility imbalance measure ) . For a set of the randomly drawn subsets of W , i.e. , { Wi } Ni=1 , Wi ⊂W , which we refer to as a sample set , we empirically measure the utility imbalance of the set by the standard deviation of { U ( Wi ) } Ni=1 . We empirically show that the utility imbalance among the weights exists in Figure 1 . Hereafter , the cause of the utility imbalance and the characteristics of the weight utility are discussed in Section 2.1 and 2.2 , respectively . 2.1 THE CAUSE OF THE UTILITY IMBALANCE . In this section , we investigate the cause of the utility imbalance among the weights . The cause driven from initialization and optimization is discussed in Section 2.1.1 and 2.1.2 , respectively . 2.1.1 FROM INITIALIZATION . The utility imbalance among the weights can be given by initialization . The Lottery Ticket Hypothesis ( Frankle & Carbin , 2018 ) showed that there is a fortuitous subset of weights at initialization , namely winning tickets , which achieves the commensurate accuracy with the original network within the commensurate training time when trained in isolation . Since they uncovered the winning tickets by pruning the weights with the smallest magnitude after pretraining the whole network , we can say that the weights in the winning ticket are the ones that become the largest when trained as a whole . And since the weights with the larger magnitude tend to be more utilized than the others ( Han et al. , 2015 ) , it can be inferred that the weights in the winning ticket are the well-initialized ones that can become the most utilized ones after trained as a whole . However , the winning tickets become less effective when the subnetworks are trained to be far apart from the initialization , e.g. , trained with a larger learning rate or using ResNet ( He et al. , 2016 ) or Batch Normalization ( Ioffe & Szegedy , 2015 ; Hoffer et al. , 2018 ) , where the gradients flowing through the network are larger ( Frankle & Carbin , 2018 ) . 2.1.2 DURING OPTIMIZATION . The utility imbalance among the weights can also be intensified during optimization . Although Frankle & Carbin ( 2018 ) conjectured that SGD may seek out and train the winning tickets , they did not give further evidence . Here , we conduct an experiment to measure the utility imbalance during training . We classified CIFAR10 dataset ( Krizhevsky , 2009 ) using a vanilla convolutional neural network ( vanilla CNN ) ( the architecture of the network is specified in Section A.2.1 ) , trained by SGD with zero momentum and weight decay . The network was trained for 200 epochs with initial learning rate of 0.1 , which is reduced by 10× at 50 % and 75 % of the training epochs . When we measured the utility of a subset ( refer to Definition 1 ) , the weights in the subset were zeroed out for the exclusion , and KL-divergence was used to measure the distance between the probability outputs of the original network and the ablated network . For each subset , the distance was measured by averaging over the total dataset . To measure the utility statistics , e.g. , utility imbalance , we used a set of 500 randomly selected subsets ( N = 500 in Definition 3 ) which we refer to as sample set . We only used the training data in the experiment for two reasons : ( 1 ) it is more straightforward to interpret and ( 2 ) in our experiments , the validation accuracy was higher when the training loss was lower , thus we regarded that the probability output of the training data represented the validation accuracy to some degree . From the left figure in Figure 2 , we can see that the utility imbalance , i.e. , the standard deviation of the utility of the subsets in the sample set , increases during training . Additionally , the mean of the utility increases ( Figure 2 , middle ) , which implies that the network is utilizing the weights more effectively and is becoming sensitive to the ablation as training proceeds . Because the utility given the amount of ablation ( |δ| ) can be interpreted as |∆f ( W+δ ) ∆δ | , it is also inferred that the loss surface is becoming sharper . It corresponds with the previous work ( refer to Ghorbani et al. , 2019 , Figure 3 ) , where the scale of the eigenvalues of the Hessian with respect to the weights grows significantly over training . The increase in the average utility can be the reason for the increase in the utility imbalance – if statistically analyzed , the standard deviation of data grows larger when the data is scaled to be larger ; or if intuitively interpreted , the network outputs with respect to the different ablation sets differ to a greater extent as the loss surface becomes sharper . Moreover , we can see that the minimum of the utility in the sample set also increases ( Figure 2 , right ) , which infers that each of the weights is struggling to be utilized more , rather than SGD purposely differ the utility among the weights . This still holds when we trained the network much longer , i.e. , 5000 epochs , where we put the result in Figure 11 . We also conducted the experiments with the most popular architecture for classification , i.e. , ResNet20 ( He et al. , 2016 ) and Batch Normalization ( Ioffe & Szegedy , 2015 ) ; and more advanced optimization techniques , i.e. , momentum and weight decay . Please refer to Figure 12 and Figure 13 for the results . The results also hold for the above statements .

Summary: This paper dives into why small pruned networks don’t train as well as large networks. They come up with a measure of weight utilization and claim that networks of all sizes only use a portion of their weights during training, and the imbalance increases during optimization. Additionally, they visualize the accuracy surface on the plane defined by the pretrained, pruned, and retrained networks and find that the retrained networks end up in the same basin as the pretrained networks.