The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively . Based on these results , we articulate the lottery ticket hypothesis : dense , randomly-initialized , feed-forward networks contain subnetworks ( winning tickets ) that—when trained in isolation— reach test accuracy comparable to the original network in a similar number of iterations . The winning tickets we find have won the initialization lottery : their connections have initial weights that make training particularly effective . We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations . We consistently find winning tickets that are less than 10-20 % of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10 . Above this size , the winning tickets that we find learn faster than the original network and reach higher test accuracy . 1 INTRODUCTION . Techniques for eliminating unnecessary weights from neural networks ( pruning ) ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Han et al. , 2015 ; Li et al. , 2016 ) can reduce parameter-counts by more than 90 % without harming accuracy . Doing so decreases the size ( Han et al. , 2015 ; Hinton et al. , 2015 ) or energy consumption ( Yang et al. , 2017 ; Molchanov et al. , 2016 ; Luo et al. , 2017 ) of the trained networks , making inference more efficient . However , if a network can be reduced in size , why do we not train this smaller architecture instead in the interest of making training more efficient as well ? Contemporary experience is that the architectures uncovered by pruning are harder to train from the start , reaching lower accuracy than the original networks.1 Consider an example . In Figure 1 , we randomly sample and train subnetworks from a fully-connected network for MNIST and convolutional networks for CIFAR10 . Random sampling models the effect of the unstructured pruning used by LeCun et al . ( 1990 ) and Han et al . ( 2015 ) . Across various levels of sparsity , dashed lines trace the iteration of minimum validation loss2 and the test accuracy at that iteration . The sparser the network , the slower the learning and the lower the eventual test accuracy . 1 “ Training a pruned model from scratch performs worse than retraining a pruned model , which may indicate the difficulty of training a network with a small capacity. ” ( Li et al. , 2016 ) “ During retraining , it is better to retain the weights from the initial training phase for the connections that survived pruning than it is to re-initialize the pruned layers ... gradient descent is able to find a good solution when the network is initially trained , but not after re-initializing some layers and retraining them. ” ( Han et al. , 2015 ) 2As a proxy for the speed at which a network learns , we use the iteration at which an early-stopping criterion would end training . The particular early-stopping criterion we employ throughout this paper is the iteration of minimum validation loss during training . See Appendix C for more details on this choice . In this paper , we show that there consistently exist smaller subnetworks that train from the start and learn at least as fast as their larger counterparts while reaching similar test accuracy . Solid lines in Figure 1 show networks that we find . Based on these results , we state the lottery ticket hypothesis . The Lottery Ticket Hypothesis . A randomly-initialized , dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations . More formally , consider a dense feed-forward neural network f ( x ; θ ) with initial parameters θ = θ0 ∼ Dθ . When optimizing with stochastic gradient descent ( SGD ) on a training set , f reaches minimum validation loss l at iteration j with test accuracy a . In addition , consider training f ( x ; m θ ) with a mask m ∈ { 0 , 1 } |θ| on its parameters such that its initialization is m θ0 . When optimizing with SGD on the same training set ( with m fixed ) , f reaches minimum validation loss l′ at iteration j′ with test accuracy a′ . The lottery ticket hypothesis predicts that ∃m for which j′ ≤ j ( commensurate training time ) , a′ ≥ a ( commensurate accuracy ) , and ‖m‖0 |θ| ( fewer parameters ) . We find that a standard pruning technique automatically uncovers such trainable subnetworks from fully-connected and convolutional feed-forward networks . We designate these trainable subnetworks , f ( x ; m θ0 ) , winning tickets , since those that we find have won the initialization lottery with a combination of weights and connections capable of learning . When their parameters are randomly reinitialized ( f ( x ; m θ′0 ) where θ′0 ∼ Dθ ) , our winning tickets no longer match the performance of the original network , offering evidence that these smaller networks do not train effectively unless they are appropriately initialized . Identifying winning tickets . We identify a winning ticket by training a network and pruning its smallest-magnitude weights . The remaining , unpruned connections constitute the architecture of the winning ticket . Unique to our work , each unpruned connection ’ s value is then reset to its initialization from original network before it was trained . This forms our central experiment : 1 . Randomly initialize a neural network f ( x ; θ0 ) ( where θ0 ∼ Dθ ) . 2 . Train the network for j iterations , arriving at parameters θj . 3 . Prune p % of the parameters in θj , creating a mask m. 4 . Reset the remaining parameters to their values in θ0 , creating the winning ticket f ( x ; m θ0 ) . As described , this pruning approach is one-shot : the network is trained once , p % of weights are pruned , and the surviving weights are reset . However , in this paper , we focus on iterative pruning , which repeatedly trains , prunes , and resets the network over n rounds ; each round prunes p 1 n % of the weights that survive the previous round . Our results show that iterative pruning finds winning tickets that match the accuracy of the original network at smaller sizes than does one-shot pruning . Results . We identify winning tickets in a fully-connected architecture for MNIST and convolutional architectures for CIFAR10 across several optimization strategies ( SGD , momentum , and Adam ) with techniques like dropout , weight decay , batchnorm , and residual connections . We use an unstructured pruning technique , so these winning tickets are sparse . In deeper networks , our pruning-based strategy for finding winning tickets is sensitive to the learning rate : it requires warmup to find winning tickets at higher learning rates . The winning tickets we find are 10-20 % ( or less ) of the size of the original network ( smaller size ) . Down to that size , they meet or exceed the original network ’ s test accuracy ( commensurate accuracy ) in at most the same number of iterations ( commensurate training time ) . When randomly reinitialized , winning tickets perform far worse , meaning structure alone can not explain a winning ticket ’ s success . The Lottery Ticket Conjecture . Returning to our motivating question , we extend our hypothesis into an untested conjecture that SGD seeks out and trains a subset of well-initialized weights . Dense , randomly-initialized networks are easier to train than the sparse networks that result from pruning because there are more possible subnetworks from which training might recover a winning ticket . Contributions .. • We demonstrate that pruning uncovers trainable subnetworks that reach test accuracy comparable to the original networks from which they derived in a comparable number of iterations . • We show that pruning finds winning tickets that learn faster than the original network while reaching higher test accuracy and generalizing better . • We propose the lottery ticket hypothesis as a new perspective on the composition of neural networks to explain these findings . Implications . In this paper , we empirically study the lottery ticket hypothesis . Now that we have demonstrated the existence of winning tickets , we hope to exploit this knowledge to : Improve training performance . Since winning tickets can be trained from the start in isolation , a hope is that we can design training schemes that search for winning tickets and prune as early as possible . Design better networks . Winning tickets reveal combinations of sparse architectures and initializations that are particularly adept at learning . We can take inspiration from winning tickets to design new architectures and initialization schemes with the same properties that are conducive to learning . We may even be able to transfer winning tickets discovered for one task to many others . Improve our theoretical understanding of neural networks . We can study why randomly-initialized feed-forward networks seem to contain winning tickets and potential implications for theoretical study of optimization ( Du et al. , 2019 ) and generalization ( Zhou et al. , 2018 ; Arora et al. , 2018 ) . 2 WINNING TICKETS IN FULLY-CONNECTED NETWORKS . In this Section , we assess the lottery ticket hypothesis as applied to fully-connected networks trained on MNIST . We use the Lenet-300-100 architecture ( LeCun et al. , 1998 ) as described in Figure 2 . We follow the outline from Section 1 : after randomly initializing and training a network , we prune the network and reset the remaining connections to their original initializations . We use a simple layer-wise pruning heuristic : remove a percentage of the weights with the lowest magnitudes within each layer ( as in Han et al . ( 2015 ) ) . Connections to outputs are pruned at half of the rate of the rest of the network . We explore other hyperparameters in Appendix G , including learning rates , optimization strategies ( SGD , momentum ) , initialization schemes , and network sizes . Notation . Pm = ‖m‖0|θ| is the sparsity of mask m , e.g. , Pm = 25 % when 75 % of weights are pruned . Iterative pruning . The winning tickets we find learn faster than the original network . Figure 3 plots the average test accuracy when training winning tickets iteratively pruned to various extents . Error bars are the minimum and maximum of five runs . For the first pruning rounds , networks learn faster and reach higher test accuracy the more they are pruned ( left graph in Figure 3 ) . A winning ticket comprising 51.3 % of the weights from the original network ( i.e. , Pm = 51.3 % ) reaches higher test accuracy faster than the original network but slower than when Pm = 21.1 % . When Pm < 21.1 % , learning slows ( middle graph ) . When Pm = 3.6 % , a winning ticket regresses to the performance of the original network . A similar pattern repeats throughout this paper . Figure 4a summarizes this behavior for all pruning levels when iteratively pruning by 20 % per iteration ( blue ) . On the left is the iteration at which each network reaches minimum validation loss ( i.e. , when the early-stopping criterion would halt training ) in relation to the percent of weights remaining after pruning ; in the middle is test accuracy at that iteration . We use the iteration at which the early-stopping criterion is met as a proxy for how quickly the network learns . The winning tickets learn faster as Pm decreases from 100 % to 21 % , at which point early-stopping occurs 38 % earlier than for the original network . Further pruning causes learning to slow , returning to the early-stopping performance of the original network when Pm = 3.6 % . Test accuracy increases with pruning , improving by more than 0.3 percentage points when Pm = 13.5 % ; after this point , accuracy decreases , returning to the level of the original network when Pm = 3.6 % . At early stopping , training accuracy ( Figure 4a , right ) increases with pruning in a similar pattern to test accuracy , seemingly implying that winning tickets optimize more effectively but do not generalize better . However , at iteration 50,000 ( Figure 4b ) , iteratively-pruned winning tickets still see a test accuracy improvement of up to 0.35 percentage points in spite of the fact that training accuracy reaches 100 % for nearly all networks ( Appendix D , Figure 12 ) . This means that the gap between training accuracy and test accuracy is smaller for winning tickets , pointing to improved generalization . Random reinitialization . To measure the importance of a winning ticket ’ s initialization , we retain the structure of a winning ticket ( i.e. , the mask m ) but randomly sample a new initialization θ′0 ∼ Dθ . We randomly reinitialize each winning ticket three times , making 15 total per point in Figure 4 . We find that initialization is crucial for the efficacy of a winning ticket . The right graph in Figure 3 shows this experiment for iterative pruning . In addition to the original network and winning tickets at Pm = 51 % and 21 % are the random reinitialization experiments . Where the winning tickets learn faster as they are pruned , they learn progressively slower when randomly reinitialized . The broader results of this experiment are orange line in Figure 4a . Unlike winning tickets , the reinitialized networks learn increasingly slower than the original network and lose test accuracy after little pruning . The average reinitialized iterative winning ticket ’ s test accuracy drops off from the original accuracy when Pm = 21.1 % , compared to 2.9 % for the winning ticket . When Pm = 21 % , the winning ticket reaches minimum validation loss 2.51x faster than when reinitialized and is half a percentage point more accurate . All networks reach 100 % training accuracy for Pm ≥ 5 % ; Figure 4b therefore shows that the winning tickets generalize substantially better than when randomly reinitialized . This experiment supports the lottery ticket hypothesis ’ emphasis on initialization : the original initialization withstands and benefits from pruning , while the random reinitialization ’ s performance immediately suffers and diminishes steadily . One-shot pruning . Although iterative pruning extracts smaller winning tickets , repeated training means they are costly to find . One-shot pruning makes it possible to identify winning tickets without this repeated training . Figure 4c shows the results of one-shot pruning ( green ) and randomly reinitializing ( red ) ; one-shot pruning does indeed find winning tickets . When 67.5 % ≥ Pm ≥ 17.6 % , the average winning tickets reach minimum validation accuracy earlier than the original network . When 95.0 % ≥ Pm ≥ 5.17 % , test accuracy is higher than the original network . However , iterativelypruned winning tickets learn faster and reach higher test accuracy at smaller network sizes . The green and red lines in Figure 4c are reproduced on the logarithmic axes of Figure 4a , making this performance gap clear . Since our goal is to identify the smallest possible winning tickets , we focus on iterative pruning throughout the rest of the paper .

Summary: It was believed that sparse architectures generated by pruning are difficult to train from scratch. The authors show that there exist sparse subnetworks that can be trained from scratch with good generalization performance. To explain the difficulty of training pruned networks from scratch or why training needs the overparameterized networks that make pruning necessary,  the authors propose a lottery ticket hypothesis: unpruned, randomly initialized NNs contain subnetworks that can be trained from scratch with similar generalization accuracy.  They also present an algorithm to identify the winning tickets.