Drawing Early-Bird Tickets: Toward More Efficient Training of Deep Networks

1 INTRODUCTION . The recent record-breaking predictive performance achieved by deep neural networks ( DNNs ) motivates a tremendously growing demand to bring DNN-powered intelligence into numerous applications ( Xu et al. , 2020 ) . However , the excellent performance of modern DNNs comes at an often prohibitive training cost due to the required vast volume of training data and model parameters . As an illustrative example of the computational complexity of DNN training , one forward pass of the ResNet50 ( He et al. , 2016a ) model requires 4 GFLOPs ( FLOPs : floating point operations ) of computations and training requires 1018 FLOPs , which takes 14 days on one state-of-the-art NVIDIA M40 GPU ( You et al. , 2018 ) . As a result , training a state-of-the-art DNN model often demands considerable energy , along with the associated financial and environmental costs . For example , a recent report shows that training a single DNN can cost over $ 10K US dollars and emit as much carbon as five cars in their lifetimes ( Strubell et al. , 2019 ) , limiting the rapid development of DNN innovations and raising various environmental concerns . The recent trends of improving DNN efficiency mostly focus on compressing models and accelerating inference . An empirically adopted practice is the so-called progressive pruning and training ∗Correspondence should be addressed to : Zhangyang Wang and Yingyan Lin . routine , i.e. , training a large model fully , pruning it , and then retraining the pruned model to restore the performance ( the process can be iterated several rounds ) . While this has been a standard practice for model compression ( Han et al. , 2015 ) , some recent efforts start empirically linking it to the potential of more efficient training . Notably , the latest series of works ( Frankle & Carbin , 2019 ; Liu et al. , 2018b ) reveals that dense , randomly-initialized networks contain small subnetworks which can match the test accuracy of original networks when trained alone themselves . These subnetworks are called winning tickets . Despite their insightful findings , there remains to be a major gap between the winning ticket observation and the goal of more efficient training , since winning tickets were only identified by pruning unimportant connections after fully training a dense network . This paper closes this gap by demonstrating the Early-Bird ( EB ) tickets phenomenon : the winning tickets can be drawn very early in training and with aggressively low-cost training algorithms . Through a range of experiments on different DNNs and datasets , we observe the consistent existence of EB tickets and the cheap costs needed to reliably draw them , and develop a novel mask distance metric to detect their emergence . After being identified , re-training those EB tickets ( using standard training ) leads to comparable or even better final accuracies , compared to either standard training , or re-training the “ ground-truth ” winning tickets drawn after full training as in ( Frankle & Carbin , 2019 ) . Our observations seem to coincide with the recent findings by ( Achille et al. , 2019 ; Li et al. , 2019 ) about the two-stage optimization trajectory in training . Taking advantage of EB tickets , we propose an efficient DNN training scheme termed EB Train . To our best knowledge , this is the first step taken towards exploiting winning tickets for a realistic efficient training goal . Our contribution can be summarized as follow : 1 . We discover the Early-Bird ( EB ) tickets , and show that they 1 ) consistently exist across DNN models and datasets ; 2 ) can emerge very early in training ; and 3 ) stay robust under ( and sometimes even favor ) various aggressive and low-cost training schemes ( in addition to early stopping ) , including large learning rates and low-precision training . 2 . We propose a practical , easy-to-compute mask distance as an indicator to draw EB tickets without accessing the “ ground-truth ” winning tickets ( drawn after full training ) , fixing a major paradox for connecting winning tickets with the efficient training goal . 3 . We design a novel efficient training framework based on EB tickets ( EB Train ) . Experiments in state-of-the-art benchmarks and models show that EB Train can achieve up to 5.8× ∼ 10.7× energy savings , while maintaining the same or even better accuracy , compared to training with the original winning tickets . 2 RELATED WORKS . Winning Ticket Hypothesis . The lottery ticket hypothesis ( Frankle & Carbin , 2019 ) first points out that a small subnetwork , called the winning ticket , can be identified by pruning a fully trained dense network ; when training it in isolation with the same weight initialization once assigned to the corresponding weights in the dense network , one can restore the comparable test accuracy to the dense network . However , finding winning tickets hinged on costly ( iterative ) pruning and retraining . ( Morcos et al. , 2019 ) studies the reuse of winning tickets , transferable across different datasets . ( Zhou et al. , 2019 ) discovers the existence of supermasks that can be applied to an untrained , randomly-initialized network . ( Liu et al. , 2018b ) argues that the weight initialization might make less difference when trained with a large learning rate , while the searched connectivity is more of the winning ticket ’ s core value . It also explores the usage of both unstructured and ( more hardware-friendly ) structured pruning and shows that both lead to the emergence of winning tickets . Another related work ( Lee et al. , 2019 ) prunes a network at single-shot with one mini-batch , in which the irrelevant connections are identified by a connection sensitivity criterion . Comparing to ( Frankle & Carbin , 2019 ) , the authors show their method to be more efficient in finding the good subnetwork ( not the winning ticket ) , although its re-training accuracy/efficiency is found to be inferior , compared to training the “ ground truth ” winning ticket . Other Relevant Observations in Training . ( Rahaman et al. , 2019 ; Xu et al. , 2019 ) argue that deep networks will first learn low-complexity ( lower-frequency ) functional components , before absorbing high-frequency features : the former being more robust to perturbations . An important hint can be found in ( Achille et al. , 2019 ) : the early stage of training seems to first discover the important connections and the connectivity patterns between layers , which becomes relatively fixed in the later training stage . That seems to imply that the critical sub-network ( connectivity ) can be identified independent of , and seemingly also ahead of , the ( final best ) weights . Finally , Li et al . ( 2019 ) demonstrates that training a deep network with a large initial learning rate helps the model focus on memorizing easier-to-fit , more generalizable pattern faster and better – a direct inspiration for us to try drawing EB tickets using large learning rates . Efficient Inference and Training . Model compression has been extensively studied for lighterweight inference . Popular means include pruning ( Li et al. , 2017 ; Liu et al. , 2017 ; He et al. , 2018 ; Wen et al. , 2016 ; Luo et al. , 2017 ; Liu et al. , 2018a ) , weight factorization ( Denton et al. , 2014 ) , weight sharing ( Wu et al. , 2018a ) , quantization ( Hubara et al. , 2017 ) , dynamic inference ( Wang et al. , 2018b ; 2019b ; Shen et al. , 2020 ) , network architecture search ( Zoph & Le , 2017 ) , among many others ( Wang et al. , 2018c ; d ) . On the other hand , the literature on efficient training appears to be much sparser . A handful of works ( Goyal et al. , 2017 ; Cho et al. , 2017 ; You et al. , 2018 ; Akiba et al. , 2017 ; Jia et al. , 2018 ; Gupta et al. , 2015 ) focus on reducing the total training time in paralleled , communication-efficient distributed settings . In contrast , our goal is to shrink the total resource cost for in-situ , resource-constrained training , as ( Wang et al. , 2019a ) advocated . ( Banner et al. , 2018 ; Wang et al. , 2018a ) presented low-precision training , which is aligned with our goal and can be incorporated into EB Train ( see later ) . 3 DRAWING EARLY-BIRD TICKETS : HYPOTHESIS AND EXPERIMENTS . We hypothesize that the winning tickets can emerge at a very early training stage , which we term as an Early-Bird ( EB ) ticket . Consider a dense , randomly-initialized network f ( x ; θ ) , f reaches a minimum validation loss floss at the i-th iteration with a test accuracy facc , when optimized with stochastic gradient descent ( SGD ) on a training set . In addition , consider subnetworks f ( x ; m⊙ θ ) with a mask m ∈ { 0 , 1 } indicates the pruned and unpruned connections in f ( x ; θ ) . When being optimized with SGD on the same training set , f ( x ; m ⊙ θt ) reach a minimum validation loss f ′loss with a test accuracy f ′acc , where θt denotes the weights at the t-th iteration of training . The EB tickets hypothesis articulates that there exists m such that f ′acc ≈ facc ( even ≥ ) , i.e. , same or better generalization , with t≪ i ( e.g. , early stopping ) and a sparse m ( i.e. , much reduced parameters ) . Section 3 addresses three key questions pertaining to the EB ticket hypothesis . We first show via an extensive set of experiments , that EB tickets can be observed across popular models and datasets ( Section 3.1 ) . We then try to be more aggressive to see if high-quality EB tickets still emerge under “ cheaper ” training ( Section 3.2 ) . We finally reveal that EB tickets can be identified using a novel mask distance between consecutive epochs , thus no full training needed ( Section 3.3 ) . 3.1 DO EARLY-BIRD TICKETS ALWAYS EXIST ? . We perform ablation simulations using two representative deep models : VGG16 ( Simonyan & Zisserman , 2014 ) and pre-activation residual networks-101 ( PreResNet101 ) ( He et al. , 2016b ) , on two popular datasets : CIFAR-10 and CIFAR-100 . For drawing the tickets , we adopt a standard training protocol ( Liu et al. , 2018b ) for both CIFAR-10 and CIFAR-100 : the training takes 160 epochs in total and the batch size of 256 ; the initial learning rate is set to 0.1 , and is divided by 10 at the 80th and 120th epochs , respectively ; the SGD solver is adopted with a momentum of 0.9 and a weight decay of 10−4 . For retraining the tickets , we keep the same setting by default . We follow the main idea of ( Frankle & Carbin , 2019 ) , but instead prune networks trained at much earlier points ( before the accuracies reach their final top values ) , to see if reliable tickets can still be drawn . We adopt the same channel pruning in ( Liu et al. , 2017 ) for all experiments since it is hardware friendly and aligns with our end goal of efficient training ( Wen et al. , 2016 ) . Figure 1 reports the accuracies achieved by re-training the tickets drawn from different early epochs . All results consistently endorse that there exist high-quality tickets , at as early as the 20th epoch ( w.r.t . a total of 160 epochs ) , that can achieve strong re-training accuracies . Comparing among different pruning ratios p , it is not too surprising to see over-pruning ( e.g. , p = 70 % ) makes drawing good tickets harder , indicating a balance that we need to calibrate between accuracy and training efficiency . Two more striking observations from Figure 1 : 1 ) there consistently exist EB tickets drawn at certain early epoch ranges , that outperform those drawn in a later stages , including the “ ground-truth ” winning tickets drawn at the 160th epoch . That intriguing phenomenon implies the possible “ overcooking ” when networks try to identify connectivity patterns at later stage ( Achille et al. , 2019 ) , that might hamper generalization ; 2 ) some EB tickets are able to outperform even their unpruned , fully-trained models ( e.g. , the dashlines ) , potentially thanks to the sparse regularization learned by EB tickets .

Summary: This paper presents a method to speed up training of deep neural networks. The main contribution is a method to quickly identify winning lottery tickets (denoted early-bird, or EB by the authors), without running the model to convergence. The authors present interesting preliminary experiments that motivate their method, and show that it works on two image recognition datasets using two models.