The Sooner The Better: Investigating Structure of Early Winning Lottery Tickets

1 INTRODUCTION . Deep Neural Networks ( DNNs ) achieve superior accuracy in a wide spectrum of applications through the use of very large and deep models ( Goodfellow et al . ( 2016 ) ) . These high-capacity but complex models , however , pose a tremendous challenge for their deployment , particularly in resource-constrained edge environments . Over the years , many techniques have been developed to compress the models to a compact counterpart to alleviate the computational costs . Among these techniques , pruning less important parameters to obtain a compact sub-network has emerged to be a popular and efficient approach ( Cun et al . ( 1990 ) ; Han et al . ( 2015 ) ) . In search of these sub-networks , new intuitions are also built up for understanding the DNN working mechanism , one example of which is the recently proposed `` lottery ticket hypothesis '' by Frankle & Carbin ( 2018 ) . The lottery ticket hypothesis states that , once a sub-network is found to match the accuracy of the original neural network , the sub-network ( i.e. , lottery ticket ) together with its initialized weights can be trained in isolation and still achieve accuracy comparable to the original network within a similar number of iterations . This conjecture intrigues discussions on a series of topics , such as the importance of initialization scheme ( Liu et al . ( 2018 ) , Zhou et al . ( 2019 ) ) , the role of over-parameterization in training ( Frankle et al . ( 2019 ) ) and even the transferability of the `` winning ticket '' ( S. Morcos et al . ( 2019 ) ) . However , all these discussions start from the point that the winning ticket has been obtained after painfully long iterative pruning procedures , which often take up to thousands of epochs . When and how a winning ticket can be found in the course of pruning procedures has not been studied ; most of the prior works use a traditional way of repeated cycles of pruning and retraining , which makes such study less practical . In this work , we provide insights to find the winning tickets early . We start with an interesting observation that there exist many lottery tickets that can achieve equally good accuracy much before the regular training schedule finishes . This leads to our in-depth investigation of the weight magnitude ; we discover that a model saturates early but not too early under stochastic gradient descent ( SGD ) optimization . To understand this characteristic of the winning lottery tickets , we conjecture that pruning a premature model causes the loss of capability in learning complex patterns , leading to accuracy degradation . We confirm this conjecture with the empirical evidence as well as the quantitative analysis based on the memorization capacity . Using this analysis framework , we further provide a reasoning behind the success of the gradual pruning over the one-shot pruning . Based on these insights , we identify the early winning tickets for various ResNet architectures on both CIFAR10 and ImageNet , achieving state-of-the-art accuracy at a high pruning rate without expensive iterative pruning . In the case of ResNet50 on ImageNet , this comes to the winning ticket of 75.31 % Top-1 accuracy at 80 % pruning rate obtained within only 22 % of the total epochs for iterative pruning . This promising outcome not only sheds light on understanding the optimization behavior of the pruned models but also enables performance gain for fast training of the pruned models . 2 RELATED WORK . The lottery ticket hypothesis was first proposed in Frankle & Carbin ( 2018 ) where the presence of a trainable subnetwork that achieves compelling final accuracy with the inherited initial values for the un-pruned connections is demonstrated . This paper also argued that the initial weight of the original network is essential for maintaining good accuracy when the model is sparsified . This claim has been extended to the over-parameterized neural networks on the larger datasets in Frankle et al . ( 2019 ) with the notion of `` rewinding '' ; the authors claim that rewinding of the weights not to the initial values but to the values after a few epochs can stabilize the accuracy of the winning lottery tickets . As a follow-up work , Zhou et al . ( 2019 ) studied the critical components of the lottery tickets such as zeros , signs and the super-mask . Also , S. Morcos et al . ( 2019 ) investigated the transferability of winning tickets obtained in one dataset to the network of a similar structure for the other datasets . But none of these focused on the structure of the winning lottery tickets ; it involves several repetitions of the full training schedule for iterative pruning and retraining , often taking hundreds to thousands of epochs . In this work , we demonstrate that many winning tickets can be found in the early stage of the baseline training schedule , drastically reducing the computational effort to obtain them . Most work on the lottery ticket hypothesis , including this work , rely on magnitude-based weight pruning for identifying unimportant weights to be pruned ( usually via global sorting ) . Such an intuitive method was first proposed in Han et al . ( 2015 ) and became popular . Afterwards , more complex pruning methodologies have been presented to improve pruning performance , such as having different pruning criteria ( Li et al . ( 2017 ) , Wen et al . ( 2016 ) , Liu et al . ( 2017 ) ) or different pruning granularity ( Mao et al . ( 2017 ) , Molchanov et al . ( 2019 ) ) . While these attempts offer insights on training pruned models , there is little discussion about the interesting interplay between the pruning criteria and the structure of a model pruned by it . In this work , we reveal that the pruned structure obtained based on the weight magnitude has tangible impact on the final accuracy , and further propose a way to quantitatively distinguish good structures for pruning . There have been various strategies to apply pruning to the neural networks . Iterative pruning by Han et al . ( 2015 ) involves several repetition of pruning ( with gradual increase of the pruning rate ) and retraining . One can increase the repetition cycles arbitrarily large to achieve good accuracy at high pruning rate ; Frankle & Carbin ( 2018 ) employed iterative pruning with hundreds to thousands of pruning epochs to match the baseline accuracy for the challenging neural networks . On the other hand , gradual pruning introduced by Zhu & Gupta ( 2017 ) determines the pruning rate and frequency via a polynomial equation as a function of the starting and ending epochs as well as the target pruning rate . Although it provides a systematic pruning schedule , there is lack of discussion about when to start the gradual pruning . Lastly , Lee et al . ( 2018 ) proposes a method of pruning weights at initialization . This method takes most advantage in performance since a pruned model can be obtained without any expensive retraining procedures . However , its effectiveness has not been demonstrated for the challenging neural networks on large datasets such as ImageNet . In contrast , we propose a mechanism to identify the winning lottery ticket early in the course of baseline training so that we can avoid costly iterative pruning while maintaining the baseline accuracy . 3 STRUCTURE OF EARLY WINNING TICKETS . In this section , we extend the lottery ticket hypothesis by Frankle & Carbin ( 2018 ) ; Frankle et al . ( 2019 ) to discuss the early winning tickets . The lottery ticket hypothesis can be summarized as : for a given network of f ( x ; w0 m0 ) with the initial weight w0 and the mask of all ones m0 , there exists a winning lottery ticket mf where |mf |/|m0| = 1 − p % ( p is the pruning rate ) and training of f ( x ; wv mf ) for 0 < v f achieves test accuracy comparable to the baseline f ( x ; wf m0 ) . There are two main components of a winning lottery ticket : the sparsified structure mf and the weight that initializes it , wv . In the previous work , mf has been obtained only after expensive iterative pruning . We characterize the structure of the early lottery ticket ms where s f , then propose a strategy for finding it early . 3.1 EXTENDED LOTTERY TICKET EXPLORATION . Frankle et al . ( 2019 ) suggests that a lottery ticket found after a baseline training can achieve the accuracy of the baseline model if it is initialized with the weight of the baseline model after a few epochs , which is called rewinding . We extend this exploration toward different structures obtained at different epochs of the baseline training ( via magnitude-based weight pruning ) . Fig . 1 shows the validation accuracy after retraining of a ResNet20 model on CIFAR10 pruned at different epochs of its baseline training . The same learning rate schedule of 0.1 reduced by 10x at epoch 120 and 160 is used for both the baseline training and retraining , and the total number of epochs is 200 . The rows ( = v ) correspond to the different rewinding epoch , whereas the columns ( = s ) correspond to the different epoch that we apply one-shot pruning ( pruning rate= 80 % ) . For example , the validation accuracy along the lottery ticket configurations of ( s = 200 , v = 0 ∼ 200 ) resembles the phenomena of `` rewinding '' observed in Frankle et al . ( 2019 ) . Interestingly , Fig . 1 further demonstrates that the winning tickets emerge at much earlier epochs of the baseline training ; the lottery ticket configurations of ( s ≥ 100 , v ≥ 5 ) achieve almost the same accuracy as the accuracy of ( s = 200 , v = 200 ) . The accuracy then gradually decreases as s ≤ 80 . This result implies two important aspects : 1 ) a winning ticket can be found in the middle of the baseline training so that one can avoid expensive iterative pruning used in the prior work , and 2 ) the winning ticket , however , does not emerge arbitrarily early in the process of training . In the following sections , we investigate the characteristics of these early winning tickets . In particular , we focus on their structure , as the weight initialization is not the major factor provided a proper rewinding . 3.2 ANALYSIS ON WEIGHT MAGNITUDE . As the first step of understanding the characteristics of the early winning tickets , we focus on the important quantity of pruning , weight magnitude . At pruning , we determine the structure of the sparsified model based on the rank of the weight magnitude ( via global sorting ) . Therefore , the change in the weight magnitude during training has large impact on the lottery ticket structure . In Fig . 2a , we measure the standard deviation of weight ( Wstd ) for each layer in the course of training ResNet20 on CIFAR10 ( note : the mean of weight is typically near zero ) . The same learning rate schedule is used as above . The first thing to note is that the change in Wstd has strong correlation with the learning rate change . In particular , different layers show different rate of change in Wstd when the learning rate is 0.1 , but from the second learning rate ( after epoch 120 ) , all the weights follow a very similar decreasing trend . This trend in Wstd can be understood via steps of stochastic gradient descent . From the typical setting of weight update with momentum and weight decay , we have : vt+1 = mvt + ( λwt + wg , t ) , wt+1 = wt − ηvt+1 . ( 1 ) where wt , wg , t and vt are the weight , gradient and momentum at step t , respectively ; m is the momentum factor , η is the learning rate and λ is the weight decay factor . After n steps , vt+n = m nvt + n∑ k=1 ( mn−kwg , t+k−1 ) + λ n∑ k=1 ( mn−kwt+k−1 ) , ( 2 ) wt+n = ( 1− ηλ ) nwt − η n∑ k=1 ( ( 1− ηλ ) n−k ( mvt+k−1 + wg , t+k−1 ) ) . ( 3 ) From this derivation , we can see that the two factors determine the mode of change in Wstd . When learning rate is high , the gradient terms play the major role in weight update . On the other hand , if the gradient activity becomes low , e.g. , when the learning rate is low and the gradients oscillate around zero , we can further simplify Eq . 3 . Assume that mnvt approaches to zero when n is relatively large , wt+k−1 ≈ wt , and ∑n k=1 ( ( 1 − ηλ ) n−kwg , t+k−1 ) and∑n k=1 ( m n−kwg , t+k−1 ) approach to zero as the gradients oscillate around zero , we have vt+i ≈ λwt1−m . Then wt+n is approximated as , wt+n ≈ ( 1− ηλ ) nwt − η n−1∑ i=0 ( ( 1− ηλ ) imλwt 1−m ) . ( 4 ) Note that ( 1− ηλ ) n ≈ 1− nηλ since ηλ 1 and ∑n−1 i=0 ( ( 1− ηλ ) i ) ≈ n. Thus , wt+n ≈ ( 1− nηλ ) wt − nηλmwt 1−m = ( 1− nηλ 1−m ) wt . ( 5 ) In other words , when the gradient activity is low , the change in Wstd is dominated by the both the learning rate and the weight decay . As an example , in Fig . 2a , Wstd is decreased with the slope following Eq . 5 when learning rate is low ; Wstd of layer 10 is decreased by 1.5e-3 in 782 updates of CIFAR10 ( with λ =1e-4 and m = 0.9 ) , confirming the slope from Eq . 5 . These two modes governing the change of weight magnitude are critical for understanding the behavior of pruning . In particular , the interplay between learning rate and weight decay causes the per-layer pruning rate to converge after enough number of epochs . Fig . 2b shows the pseudo per-layer pruning rate , where we just measure the layer-wise pruning rate without really pruning out the weights in the model , for the same CIFAR10 training experiment . As the figure shows , the per-layer pruning rates saturate around epoch 100 , indicating that pruning before that would select the weights based on the premature model . Moreover , although the weight magnitude is an important factor , we discovered that the ranking of individual weight does not play a critical role in winning lottery tickets . Specifically , we empirically show that there exist many winning lottery tickets that are vastly different in terms of its sparse structure mv . Fig . 3 shows the hamming distance of the sparse structure of the lottery tickets at different configuration ( s , v ) . Note that the two distant lottery tickets ( e.g. , ( s = 200 , v = 100 ) and ( s = 100 , v = 100 ) ) show large hamming distance of 0.159 ( where x/y = 200/100 ) while achieving the equally good accuracy as shown in Fig . 1 , indicating that the ranking of the weights itself can not explain the quality of the structure of the lottery ticket .

Summary: This paper attempts an in depth study of the lottery ticket hypothesis. The lottery ticket hypothesis holds that sparse sub-networks exist inside dense large models and that the sparse sub-networks achieve at least as good an accuracy as the underlying large model. These sub-networks are discovered by training and iteratively pruning the dense model. This paper investigates the epoch at which pruning should occur as well as the epoch at which weights should be rewound when retraining. Then, the authors conduct experiments with different pruning strategies (one-shot vs. gradual) in an attempt to find such sparse models (or "winning tickets") earlier than they otherwise would have been found.