Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?

1 Introduction . In recent years , the Lottery Ticket Hypothesis ( LTH ) [ 1 ] has drawn great attention and thorough research efforts . As an important study to investigate the initialization state and network topology of the deep neural networks ( DNNs ) , LTH claims the existence of a winning ticket ( i.e. , a properly pruned subnetwork together with original weight initialization ) that can achieve competitive performance to the original dense network , which highlights great potential for efficient training and network design . Unfortunately , among the various researches on the lottery ticket hypothesis [ 2 , 3 , 4 , 5 , 6 , 7 , 8 ] , there are many inconsistencies regarding the settings of training recipe , and they further lead to the controversies over the conditions for identifying winning tickets . We revisit and analyze the definition of the original lottery ticket hypothesis and find that the quality of training recipe is a critical factor for the network performance , which in fact , is largely missing in previous discussions . In the standard LTH setup [ 1 ] , key training hyperparameters such as learning rate and training epochs were not scrutinized nor exhaustively tuned . The winning ticket can be identified in the case of small learning rate , but can fail to emerge at higher initial learning rates especially in deeper networks . For instance , in [ 1 ] , the winning tickets can be identified only in the case of small learning rate 0.01 , with ResNet-20 and VGG-19 on CIFAR-10 . At larger learning rates , however , [ 9 ] reveals that the “ winning ticket ” has no accuracy advantage over the random reinitialization , which contradicts with † Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the LTH definition . On the other hand , the settings in [ 1 ] train 78 epochs for ResNet-20 on CIFAR-10 . Such insufficient training causes a relatively low pretraining accuracy . When pruned iteratively , the subnetwork accuracy can easily match that pretraining accuracy of the original network . Under such experimental conditions , the existence of the winning ticket is questionable . In addition to all the problems caused by the experimental conditions , the huge computational consumption to find a winning ticket becomes another research barrier and the practical main drawback , limiting the observations made on LTH . For instance , to reach around 90 % overall sparsity ratio , iterative magnitude-based pruning ( IMP ) in [ 1 ] requires totally 11 iterations ( 20 % of the weights are pruned in each iteration ) . It adds up to 1,760 total training epochs if each iteration consumes 160 epochs . On the other hand , as an efficient pruning method , one-shot magnitude-based pruning ( OMP ) prunes a pretrained DNN model to arbitrary target sparsity ratio in one shot , which greatly saves training efforts . However , OMP is rarely considered in the related literature , and is often deemed as “ weak '' without full justification . Based on the above reasons , we feel we can not confidently draw arguments , before we are able to evaluate LTH comprehensively in regards of key factors such as different network structures , network dimensions , and training dataset sizes . In this paper , we dive deeper into the underlying condition of the lottery ticket hypothesis . We raise the following questions : ( 1 ) What makes the comprehensive condition to define the lottery ticket hypothesis ? ( 2 ) Do winning tickets exist across the major DNN architectures and/or applications under such definition ? and ( 3 ) What are the intrinsic reasons for their existence or non-existence ? To answer the above questions , we present our rigorous definition of the lottery ticket hypothesis , which specifies settings of the training recipe , the principles for identifying winning tickets , and the rationality on examining the winning ticket existence . Under this rigorous definition , we perform extensive experiments with many representative DNN models and datasets . The relationships between winning tickets and various factors are quantitatively analyzed . We empirically study the patterns through our analysis , and develop a guideline to ease the process of obtaining the winning ticket . Our findings open up many new questions for future work . We summarize our contributions as follows : I . We point out that the usage of inappropriately small learning rates , insufficient training epochs , and other inconsistent and implicit conditions for identifying winning ticket in the literature , are the main reasons that cause controversies in the lottery ticket studies . II . We propose a more rigorous definition of the winning ticket , and evaluate the proposed definition on different training recipe , DNN architecture , network dimension , and the training data size . Somehow surprisingly , we find that under the new rigorous definition , no “ rigorous ” winning tickets are found by current methods , while there do exist winning tickets under a slightly looser definition . III . We find that when residual connections exist in the network , using a relatively small learning rate is more likely to find ( close to ) winning tickets . When no residual connection exists , the IMP method may not be necessary because OMP can achieve equivalent performance . IV . We also find that when a smaller learning rate is not favorable , initialization is likely to make no difference in finding the winning ticket ( e.g. , lottery initialization is not necessary ) . We quantitatively analyze the patterns , and present a guideline to help identify winning tickets . 2 Re-defining Lottery Ticket Hypothesis . 2.1 Notations and Preliminary . In this paper , we follow the notations from [ 1 , 5 ] . Detailed notations and functions are listed in Table 1 . Based on Table 1 , we provide several key LTH-related settings along with descriptions . Consider a network function f ( · ) that is initialized as f ( x ; θ0 ) where x denotes input training samples . We define the following settings : • Pretraining : We train the network f ( x ; θ0 ) for T epochs , arriving at weights θT and network function f ( x ; θT ) . • Pruning : Based on the trained weights θT , we adopt OMP ( θT , s ) or IMP ( θT , s ) to generate a pruning mask mO , mI ∈ { 0 , 1 } |θ| . Note that for IMP , the same θ0 is used in each iteration to ensure fairness to OMP . • Lottery ticket with OMP ( LT-OMP ) : We directly apply mask mO to initial weights θ0 , resulting in weights θ0 mO and network function f ( x ; θ0 mO ) . • Lottery ticket with IMP ( LT-IMP ) : We apply mI to initial weights θ0 , and get f ( x ; θ0 mI ) . • Random reinitialization with OMP ( RR-OMP ) : We apply mask mO to the random reinitialized weights θ′0 , and get network function f ( x ; θ ′ 0 mO ) . • Random reinitialization with IMP ( RR-IMP ) : We apply mI to random reinitialized weights θ′0 , and get f ( x ; θ′0 mI ) . • Small-dense training ( SDT ) : We construct a small-dense network that has the same depth and reduced width compared to the original network , and initialized by θSD , i.e . f ( x ; θSD ) . Original definition of the winning ticket : The original lottery ticket hypothesis [ 1 ] claims that there exists subnetwork f ( x ; θ0 m ) in a randomly initialized dense network f ( x ; θ0 ) , that once trained for T epochs ( or fewer ) will result in similar accuracy as f ( x ; θT ) , under a non-trivial sparsity ratio . Additionally , the accuracy of f ( x ; ( θ0 m ) T ) should be noticeably higher than f ( x ; ( θ′0 m ) T ) . Note that ( θ0 m ) T and ( θ′0 m ) T are the initial and the randomly reinitialized weights of the sparse subnetwork trained for T epochs , respectively . When the above conditions are met , ( θ0 m ) can be considered the Winning Ticket . We define a network is well-trained , if it is trained using a sufficient training recipe ( i.e. , an appropriate learning rate and sufficient training epochs ) . However , in many prior works such as [ 1 ] , the pretraining of the lottery ticket experiments used an insufficient training recipe ( i.e. , inappropriately small learning rate and fewer training epochs ) , which leads to non-optimal pretraining accuracy at relatively low levels . Apparently , a higher pretraining accuracy is more difficult for a subnetwork to match or “ win the ticket ” , even by using a sufficient training recipe . We further revisit the LT-IMP and RR-IMP experiments using ResNet-20 on CIFAR-10 dataset , at three different learning rates over a range of different sparsity ratios ( [ 1 ] uses the small learning rate 0.01 ) . We train the subnetworks with the same training recipe in pretraining , and we also adopt the settings in [ 1 ] to reproduce the results . Our preliminary results are shown in Figure 1 . Through Figure 1 ( a ) , 1 ( b ) , our first observation is that , under either training recipe , the “ winning ticket ” exists in smaller learning rates ( e.g. , 0.005 and 0.01 ) , but does not exist at a relatively larger learning rate ( e.g. , 0.1 ) . For instance , in the cases of the initial learning rate of 0.005 and 0.01 , we find a noticeable accuracy gap between LT-IMP and RR-IMP using both training recipes , and the LT-IMP accuracy is close to the pretraining accuracy with a reasonable sparsity ratio ( e.g. , 50 % or above ) . This is similar to the observations found in [ 1 ] on the same network and dataset . On the other hand , in the case of the initial learning rate of 0.1 , the LT-IMP has a similar accuracy performance as the RR-IMP , and can not achieve the accuracy close to the pretrained DNN with a reasonable sparsity ratio , thus no winning ticket condition is satisfied . Through Figure 1 ( c ) , our second observation is that , at the same learning rate , the winning ticket defined in [ 1 ] can be identified by using an insufficient training recipe , but fails to satisfy the winning ticket condition when the network is well-trained . For instance , in the case of initial learning rate of 0.005 , [ 1 ] uses approximately 78 epochs for training the network , which achieves 88.0 % pretraining accuracy , 87.1 % on LT-IMP and 80.3 % on RR-IMP , respectively . The LT-IMP accuracy is close to the pretraining accuracy , and outperforms RR-IMP , thus it is claimed in [ 1 ] that the winning ticket is found . However , when we train the network with a sufficient number of epochs ( 160 in our settings ) , the accuracy of pretraining , LT-IMP , and RR-IMP are 89.6 % , 87.4 % , and 82.9 % , respectively . In this case , the accuracy gap between pretraining and LT-IMP is not small enough to claim that they are “ similar ” , thus in fact no winning ticket is found . Takeaway : The above two observations indicate that the winning tickets are more likely to exist at a small learning rate or at an insufficient training epochs , but may not exist at a relatively large learning rate or sufficient training epochs ( also observed in [ 9 ] ) . However , we would like to point out that using a relatively large learning rate ( e.g. , 0.1 ) and sufficient training epochs ( e.g. , 160 , which is the standard settings on CIFAR-10 ) result in a notably higher accuracy for the pretrained DNN ( 92.3 % vs. 88.0 % ) . This point is largely missing in the previous discussions , and questions whether the previously identified “ winning tickets ” are meaningful enough .

Summary: This paper goes deep into the investigation of the lottery ticket hypothesis, and found out that the training recipe that the original findings were based on had been insufficient. With a revised recipe and through extensive experiments, the author empirically studied the patterns of whether and when a "jackpot ticket" and "secondary tickets" can be identified. The major factor turns out to be the learning rate with which to train the subnetworks. The paper then ends with guidelines on how to find winning tickets.