Drawing Early-Bird Tickets: Toward More Efficient Training of Deep Networks

1 INTRODUCTION . The recent record-breaking predictive performance achieved by deep neural networks ( DNNs ) motivates a tremendously growing demand to bring DNN-powered intelligence into numerous applications ( Xu et al. , 2020 ) . However , the excellent performance of modern DNNs comes at an often prohibitive training cost due to the required vast volume of training data and model parameters . As an illustrative example of the computational complexity of DNN training , one forward pass of the ResNet50 ( He et al. , 2016a ) model requires 4 GFLOPs ( FLOPs : floating point operations ) of computations and training requires 1018 FLOPs , which takes 14 days on one state-of-the-art NVIDIA M40 GPU ( You et al. , 2018 ) . As a result , training a state-of-the-art DNN model often demands considerable energy , along with the associated financial and environmental costs . For example , a recent report shows that training a single DNN can cost over $ 10K US dollars and emit as much carbon as five cars in their lifetimes ( Strubell et al. , 2019 ) , limiting the rapid development of DNN innovations and raising various environmental concerns . The recent trends of improving DNN efficiency mostly focus on compressing models and accelerating inference . An empirically adopted practice is the so-called progressive pruning and training ∗Correspondence should be addressed to : Zhangyang Wang and Yingyan Lin . routine , i.e. , training a large model fully , pruning it , and then retraining the pruned model to restore the performance ( the process can be iterated several rounds ) . While this has been a standard practice for model compression ( Han et al. , 2015 ) , some recent efforts start empirically linking it to the potential of more efficient training . Notably , the latest series of works ( Frankle & Carbin , 2019 ; Liu et al. , 2018b ) reveals that dense , randomly-initialized networks contain small subnetworks which can match the test accuracy of original networks when trained alone themselves . These subnetworks are called winning tickets . Despite their insightful findings , there remains to be a major gap between the winning ticket observation and the goal of more efficient training , since winning tickets were only identified by pruning unimportant connections after fully training a dense network . This paper closes this gap by demonstrating the Early-Bird ( EB ) tickets phenomenon : the winning tickets can be drawn very early in training and with aggressively low-cost training algorithms . Through a range of experiments on different DNNs and datasets , we observe the consistent existence of EB tickets and the cheap costs needed to reliably draw them , and develop a novel mask distance metric to detect their emergence . After being identified , re-training those EB tickets ( using standard training ) leads to comparable or even better final accuracies , compared to either standard training , or re-training the “ ground-truth ” winning tickets drawn after full training as in ( Frankle & Carbin , 2019 ) . Our observations seem to coincide with the recent findings by ( Achille et al. , 2019 ; Li et al. , 2019 ) about the two-stage optimization trajectory in training . Taking advantage of EB tickets , we propose an efficient DNN training scheme termed EB Train . To our best knowledge , this is the first step taken towards exploiting winning tickets for a realistic efficient training goal . Our contribution can be summarized as follow : 1 . We discover the Early-Bird ( EB ) tickets , and show that they 1 ) consistently exist across DNN models and datasets ; 2 ) can emerge very early in training ; and 3 ) stay robust under ( and sometimes even favor ) various aggressive and low-cost training schemes ( in addition to early stopping ) , including large learning rates and low-precision training . 2 . We propose a practical , easy-to-compute mask distance as an indicator to draw EB tickets without accessing the “ ground-truth ” winning tickets ( drawn after full training ) , fixing a major paradox for connecting winning tickets with the efficient training goal . 3 . We design a novel efficient training framework based on EB tickets ( EB Train ) . Experiments in state-of-the-art benchmarks and models show that EB Train can achieve up to 5.8× ∼ 10.7× energy savings , while maintaining the same or even better accuracy , compared to training with the original winning tickets . 2 RELATED WORKS . Winning Ticket Hypothesis . The lottery ticket hypothesis ( Frankle & Carbin , 2019 ) first points out that a small subnetwork , called the winning ticket , can be identified by pruning a fully trained dense network ; when training it in isolation with the same weight initialization once assigned to the corresponding weights in the dense network , one can restore the comparable test accuracy to the dense network . However , finding winning tickets hinged on costly ( iterative ) pruning and retraining . ( Morcos et al. , 2019 ) studies the reuse of winning tickets , transferable across different datasets . ( Zhou et al. , 2019 ) discovers the existence of supermasks that can be applied to an untrained , randomly-initialized network . ( Liu et al. , 2018b ) argues that the weight initialization might make less difference when trained with a large learning rate , while the searched connectivity is more of the winning ticket ’ s core value . It also explores the usage of both unstructured and ( more hardware-friendly ) structured pruning and shows that both lead to the emergence of winning tickets . Another related work ( Lee et al. , 2019 ) prunes a network at single-shot with one mini-batch , in which the irrelevant connections are identified by a connection sensitivity criterion . Comparing to ( Frankle & Carbin , 2019 ) , the authors show their method to be more efficient in finding the good subnetwork ( not the winning ticket ) , although its re-training accuracy/efficiency is found to be inferior , compared to training the “ ground truth ” winning ticket . Other Relevant Observations in Training . ( Rahaman et al. , 2019 ; Xu et al. , 2019 ) argue that deep networks will first learn low-complexity ( lower-frequency ) functional components , before absorbing high-frequency features : the former being more robust to perturbations . An important hint can be found in ( Achille et al. , 2019 ) : the early stage of training seems to first discover the important connections and the connectivity patterns between layers , which becomes relatively fixed in the later training stage . That seems to imply that the critical sub-network ( connectivity ) can be identified independent of , and seemingly also ahead of , the ( final best ) weights . Finally , Li et al . ( 2019 ) demonstrates that training a deep network with a large initial learning rate helps the model focus on memorizing easier-to-fit , more generalizable pattern faster and better – a direct inspiration for us to try drawing EB tickets using large learning rates . Efficient Inference and Training . Model compression has been extensively studied for lighterweight inference . Popular means include pruning ( Li et al. , 2017 ; Liu et al. , 2017 ; He et al. , 2018 ; Wen et al. , 2016 ; Luo et al. , 2017 ; Liu et al. , 2018a ) , weight factorization ( Denton et al. , 2014 ) , weight sharing ( Wu et al. , 2018a ) , quantization ( Hubara et al. , 2017 ) , dynamic inference ( Wang et al. , 2018b ; 2019b ; Shen et al. , 2020 ) , network architecture search ( Zoph & Le , 2017 ) , among many others ( Wang et al. , 2018c ; d ) . On the other hand , the literature on efficient training appears to be much sparser . A handful of works ( Goyal et al. , 2017 ; Cho et al. , 2017 ; You et al. , 2018 ; Akiba et al. , 2017 ; Jia et al. , 2018 ; Gupta et al. , 2015 ) focus on reducing the total training time in paralleled , communication-efficient distributed settings . In contrast , our goal is to shrink the total resource cost for in-situ , resource-constrained training , as ( Wang et al. , 2019a ) advocated . ( Banner et al. , 2018 ; Wang et al. , 2018a ) presented low-precision training , which is aligned with our goal and can be incorporated into EB Train ( see later ) . 3 DRAWING EARLY-BIRD TICKETS : HYPOTHESIS AND EXPERIMENTS . We hypothesize that the winning tickets can emerge at a very early training stage , which we term as an Early-Bird ( EB ) ticket . Consider a dense , randomly-initialized network f ( x ; θ ) , f reaches a minimum validation loss floss at the i-th iteration with a test accuracy facc , when optimized with stochastic gradient descent ( SGD ) on a training set . In addition , consider subnetworks f ( x ; m⊙ θ ) with a mask m ∈ { 0 , 1 } indicates the pruned and unpruned connections in f ( x ; θ ) . When being optimized with SGD on the same training set , f ( x ; m ⊙ θt ) reach a minimum validation loss f ′loss with a test accuracy f ′acc , where θt denotes the weights at the t-th iteration of training . The EB tickets hypothesis articulates that there exists m such that f ′acc ≈ facc ( even ≥ ) , i.e. , same or better generalization , with t≪ i ( e.g. , early stopping ) and a sparse m ( i.e. , much reduced parameters ) . Section 3 addresses three key questions pertaining to the EB ticket hypothesis . We first show via an extensive set of experiments , that EB tickets can be observed across popular models and datasets ( Section 3.1 ) . We then try to be more aggressive to see if high-quality EB tickets still emerge under “ cheaper ” training ( Section 3.2 ) . We finally reveal that EB tickets can be identified using a novel mask distance between consecutive epochs , thus no full training needed ( Section 3.3 ) . 3.1 DO EARLY-BIRD TICKETS ALWAYS EXIST ? . We perform ablation simulations using two representative deep models : VGG16 ( Simonyan & Zisserman , 2014 ) and pre-activation residual networks-101 ( PreResNet101 ) ( He et al. , 2016b ) , on two popular datasets : CIFAR-10 and CIFAR-100 . For drawing the tickets , we adopt a standard training protocol ( Liu et al. , 2018b ) for both CIFAR-10 and CIFAR-100 : the training takes 160 epochs in total and the batch size of 256 ; the initial learning rate is set to 0.1 , and is divided by 10 at the 80th and 120th epochs , respectively ; the SGD solver is adopted with a momentum of 0.9 and a weight decay of 10−4 . For retraining the tickets , we keep the same setting by default . We follow the main idea of ( Frankle & Carbin , 2019 ) , but instead prune networks trained at much earlier points ( before the accuracies reach their final top values ) , to see if reliable tickets can still be drawn . We adopt the same channel pruning in ( Liu et al. , 2017 ) for all experiments since it is hardware friendly and aligns with our end goal of efficient training ( Wen et al. , 2016 ) . Figure 1 reports the accuracies achieved by re-training the tickets drawn from different early epochs . All results consistently endorse that there exist high-quality tickets , at as early as the 20th epoch ( w.r.t . a total of 160 epochs ) , that can achieve strong re-training accuracies . Comparing among different pruning ratios p , it is not too surprising to see over-pruning ( e.g. , p = 70 % ) makes drawing good tickets harder , indicating a balance that we need to calibrate between accuracy and training efficiency . Two more striking observations from Figure 1 : 1 ) there consistently exist EB tickets drawn at certain early epoch ranges , that outperform those drawn in a later stages , including the “ ground-truth ” winning tickets drawn at the 160th epoch . That intriguing phenomenon implies the possible “ overcooking ” when networks try to identify connectivity patterns at later stage ( Achille et al. , 2019 ) , that might hamper generalization ; 2 ) some EB tickets are able to outperform even their unpruned , fully-trained models ( e.g. , the dashlines ) , potentially thanks to the sparse regularization learned by EB tickets .

Summary: The authors further study the lottery ticket hypothesis formulated by Frankle and Carbin. They demonstrate that the sparsity pattern corresponding to a lottery ticket for a given initialization can be uncovered via low-cost training. By doing so, they propose a method to: 1) first identify the lottery ticket efficiently and 2) exploit the sparsity of the resulting network to train it at a lower cost.