Drawing Early-Bird Tickets: Toward More Efficient Training of Deep Networks
Authors: Haoran You
Chaojian Li
Pengfei Xu
Yonggan Fu
Yue Wang
Xiaohan Chen
Richard G. Baraniuk
Zhangyang Wang
Yingyan Lin
Venue:  (2020)

Abstract: (Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as Early-Bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 5.8x ~ 10.7x energy savings while maintaining comparable or even better accuracy as compared to the most competitive state-of-the-art training methods, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training.

Published as a conference paper at ICLR 2020
DRAWING EARLY-BIRD TICKETS: TOWARDS MORE EF- FICIENT TRAINING OF DEEP NETWORKS
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Richard G. Baraniuk & Yingyan Lin'
Department of Electrical and Computer Engineering
Rice University
Houston, TX 77005, USA
[hy34, c1114, px5, yf22, yw68, yingyan.lin, richb]erice.es
Xiaohan Chen & Zhangvang Wang"
Department of Computer Science and Engineering
Texas A&M University
College Station, TX 77843, USA
{chernxh, at.laswang @tamu.
ABSTRACT
(Frankle Carbin 2019) shows that there exist winning tickets (small but crit- ical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracy to the latter in a similar number of it- erations. However, the identification of these winning tickets still requires the costly train-prume sein-prune-retrain process, limiting their practical benefits. In this paper. we discover for the first time that the winning tickets can be identified at a very early training stage, which we term as Early-Bird (EB) tickets, via low- cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding on the existence of EB tickets is consistent with re- cently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose mask distance metric that can be used to identify EB tickets with a low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets val- idate: the existence of EB tickets and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training tickets can achieve up to 5.8x 10.7x energy savings while maintaining com- compared to the most competitive state-of-
EB
parable or even better accuracy the-art art training methods, demonstrating promising and easily adopted method for tackling the often cost- cost- prot-prohibitive deep network training. Codes available hattps://githun cww/ RICE- EIC/Early http://rllyyBBr--lcccceee
INTRODUCTION
The recent record-breed breaking predictive performance achieved by deep neural networks (DNNs) mo- tivates a tremendously growing demand to bring DNN INN-powered intelligence into numerous appli- cations Xu al. 2020). However, the excellent performance of modern DNNs comes often prohibitive training cost due to the required vast volume of training data and model parameters. As an illustrative example of the computational complexity of DNN training, one forward pass of the ResNet50 (He et al., 201 model requires GFLOPs (FLOPs: floating point operations) of com- putations and training requires 10" FLOPs, which takes 14 days on one state-of tfate-of-the-art NVIDIA M40 GPU YYou et al.) 2018). As a result, training a state-of-the-art DNN model often demands considerable energy, along with the associated financial and environmental costs. For example, recent report shows that training a single DNN can cost over $10K US dollars and emit as much carbon five cars in their lifetimes (Strubell et al. 2019), limiting the rapid development of DNN innovations and raising various environmental concerns.
The recent trends of improving DNN efficiency mostly focus compressing models and acceler- ating inference. An empirically adopted practice is the so-called progressive pruning and training
Correspondence should be addressed Zhangyang Wang and Yingyan Lin.

---

Published as a conference paper at ICLR 2020
routine, i.e,, training large model fully, pruning it, and then retraining the pruned model to restore the performance (the process can be iterated several rounds). While this has been a standard prac- tice for model compression (Han et al. 2015), some recent efforts start empirically linking it to the potential of more efficient training. Notably, the latest series of works (Frankle & Carbin 2019 Liu et al. 2018b) reveals that dense, candomly-initialized networks contain small subnetworks which can match the test accuracy of original networks when trained alone themselves. These subnetworks are called winning tickets. Despite their insightful findings, there remains to be major gap between the winning ticket observation and the goal of more efficient training, since winning tickets were only identified by pruning unimportant connections after fully training dense network.
This paper closes this gap by demonstrating the Early-Bird (EB) tickets phenomenon: the winning tickets can be drawn very early in training and with aggressively low-cost training algorithms. Through a range of experiments on different DNNs and datasets, we observe the consistent existence of EB tickets and the cheap costs needed to reliably draw them, and develop a novel mask distance metric to detect their emergence, After being identified, re-training those EB tickets (using standard training) leads to comparable or even better final accuracies, compared to either standard training, or re-training the trrounddtruth" winning tickets drawn after full training as in (Frankle & Carbin 2019). Our observations seem to coincide with the recent findings by (Achille et al. [2019]]Li et al, 2019) about the two-stage optimization trajectory in training. Taking advantage of EB tickets, we propose an efficient DNN training scheme termed EB Train. To our best knowle mowwledge, this is the first step taken towards exploiting winning tickets for a realistic efficient training goal.
Our contribution can be summarized as follow:
1. We discover the Early-Bird (EB) tickets, and show that they 1) consistently exist across DNN models and datasets; 2) can emerge very early in training; and 3) stay robust under (and sometimes even favor) various aggressive and low-cost training schemes (in addition to early stopping), including large learning rates and low-precision training.
2. We propose a practical, casy-to-compute mask distance as an indicator draw EB tickets without accessing the "ground-truth" winning tickets (drawn after full training), fixing a major paradox for connecting winning tickets with the efficient training goal.
3. We design novel efficient training framework based on EB tickets (EB Train). Exper- iments in state-of-the-ar benchmarks and models show that EB Train can achieve up to 5.8x 10.7x energy savings, while maintaining the same even better accuracy, com- pared to training with the original winning tickets.
2 RELATED WORKS
Winning Ticket Hypothesis. The lottery ticket hypothesis (Frankle & Carbin, 2019) first points out that a small subnetwork, called the winning ticket, can be identified by pruning a fully trained dense network; when training it in isolation with the same weight initialization once assigned to the corresponding weights in the dense network, one can restore the comparable test accuracy to the dense network. However, finding winning tickets hinged on costly (iterative) pruning and re- training. (Morcos et al., 2019) studies the reuse of winning tickets, transferable across different datasets. (Zhou al. 2019) discovers the existence of supermasks that can be applied to an un- trained, randomly-initiaa lized network. (Liu et al. 201 argues that the weight initialization might make less difference when trained with a large learning rate, while the searched connectivity is more of the winning ticket's core value. It also explores the usage of both unstructured and (more hardware-friendly) structured pruning and shows that both lead to the emergence of winning tickets.
Another related work (Lee et al. 2019) prunes network at single-shot with one mini-batch, in which the irrelevant connections are identified by a connection sensitivity criterion. Comparing to (Frankle & Carbin 2019), the authors show their method to be more efficient in finding the good subnetwork (not the winning ticket), although its re-training accuracy/effccinnyy is found to be inferior, compared to training the "ground truth" winning ticket.
Other Relevant Observations in Training, (Rahaman et al. 2019 Xu et al. 2019) argue that
deep networks will first learn low-complexity freuu-fre functional components, before ab- sorbing lighfrreqeeccy features: the former being more robust to perturbations. An important hint can be found in (Achille et al. 2019): the early stage of training seems to first discover the impor- tant connections and the connectivity patterns between layers, which becomes relatively fixed in the
2

---

Published as a conference paper at ICLR 2020
later training stage. That seems to imply that the critical sub-network (connectivity) can be identi- fied independent of, and seemingly also ahead of, the (final best) weights. Finally, Li et al. (2019) demonstrates that training deep network with large initial learning rate helps the model focus on memorizing easier-to-fit, more generalizable pattern faster and better- a direct inspiration for us to try drawing EB tickets using large learning rates.
Efficient Inference and Training. Model compression has been extensively studied for lighter-
weight inference. Popular means include pruning (Li et al. 2017 Liu et al.) 2017: He et al.
2018
Wen et al. 2016; [uo et al.] 2017; Liu et al.) 2018a), weight factorization (Denton et al.] 2014).
weight sharing (Wu et al., (2018a), 8a), quantization (Hubara et al. 2017), dynamic inference (Wang
(ct al., 20188; 2019b; [Shen et al. 2020), network architecture search (Zoph & Le| 2017). among
many others Wang et al. 2018c dp. On the other hand, the literature efficient training appears
to be much sparser. A handful of works (Goyal et al. 2017; Cho et al.) 2017 You et al., 2018
Akiba al. [017 Jia al. 2018 Gupta et al.,) 2015) focus reducing the total training time in paralleled, communication-eficciii
distributed settings. In contrast, our goal is to shrink the total
resource cost for in-situ, resource-constrained training, (Wang al. (019a) advocated. (Banner iet al. 2018 Wang et al. 2018a) presented low-precision training, which is aligned with our goal and can be incorporated into EB Train (see later).
3 DRAWING EARLY-BIRD TICKETS: HYPOTHESIS AND EXPERIMENTS
We hypothesize that the winning tickets can emerge very early training stage, which we term as an Early-Bird (EB) ticket. Consider a dense, randomly-initializer network (ax; 0), f reaches a minimum validation loss fross at the i-th iteration with a test accuracy facrs when optimized with stochastic gradient descent (SGD) on a training set. In addition, consider subnetworks f(z; m O f) with a mask m E {0, 1} indicates the pruned and unpruned connections in [(x;;] f). When being optimized with SGD on the same training set, f (x; m O 0,) reach a minimum validation loss hass with a test accuracy fare. where 6r denotes the weights at the t-th iteration of training. The EB tickets hypothesis articulates that there exists m such that Sace = Sace (even 2), i.e., same or better generalization, with << i (e.g., early stopping) and a sparse (i.e., much reduced parameters).
Section 3 addresses three key questions pertaining to the EB ticket hypothesis. We first show via an extensive set of experiments, that EB tickets can be observed across popular models and datasets (Section 3. We then try to be more aggressive to see if high-quality EB tickets still emerge under "cheaper" training (Section 3.2). We finally reveal that EB tickets can be identified using a novel mask distance between consecutive epochs, thus no full training needed (Section 3.3).
3.1
DO EARLY-BIRD TICKETS ALWAYS EXIST?
We perform ablation simulations using two representative deep models: VGG16 (Simonyan & Zis- serman 2014) and pre-activation residual networks- 101 (PrcResNet101) (He et 2016b), on two popular datasets: CIFAR- 10 and CIFAR- 100. For drawing the tickets, we adopt a standard training protocol (Liu et al. 2018b) for both CIFAR-IO and CIFAR- the training takes 160 epochs in total and the batch size of 256; the initial learning rate is set to 0.1, and is divided by 10 at the 80th and 120th epochs, respectively: the SGD solver is adopted with momentum of 0.9 and weight decay of 10 For retraining the tickets, we keep the same setting by default.
We follow the main idea of (Frankle & Carbin 2019). but instead prune networks trained at much earlier points (before the accuracies reach their final top values), see if reliable tickets can still be drawn. We adopt the same channel pruning in (Liu et al. 2017) for all experiments since it is hardware friendly and aligns with our end goal of efficient training (Wen et al., 2016). Figure ree- ports the accuracies achieved by re-training the tickets drawn from different early epochs. All results consistently endorse that there exist high-quality tickets, at as early as the 20th epoch (w.rt. a total of 160 epochs), that can achieve strong re-training accuracies. Comparing among different pruning ratios p. it is not too surprising to see over over-pruning (c.g., = 70%) makes drawing good tickets harder, indicating balance that we need to calibrate between accuracy and training efficiency.
Two more striking observations from Figure 1) there consistently exist EB tickets drawn at certain early epoch ranges, that outperform those drawn in later stages, including the "ground-truth winning tickets drawn at the 160th epoch. That intriguing phenomenon implies the possible "over- cooking" when networks try to identify connectivity patterns at later stage (Achille et al. 2019.
3

---

Published as a conference paper at ICLR 2020
Retrain accuracy (%)
PreResNet101
VGG16
PreResNet101
VGG16
72
11
Unprased basuline
92
Utapraned basslina
Urpruned basalive
69
Ungraned Daaeline
p-m 30%
pm 20%
50%
p
911
30
100
150
5D
100
130
50
100
130
300
150
Epoch subnetwork drawn from
Epoch subnetwork drawn from
Epech subnetwork drawn from
Epoch subnetwork drawn from
CIFAR-10
CIFAR-100
Figure 1: Retraining accuracy vs, epoch numbers at which the subnetworks are drawn, for both PreResNet101 and VGG16 on the CIFAR-10/100 datasets, where p indicates the channel pruning ratio and the dashed line shows the accuracy of the corresponding dense model on the same dataset, x denotes the retraining accuracies of subnetworks drawn from the epochs with the best search accuracies, and error bars show the minimum and maximum of three runs.
that might hamper generalization; 2) some EB tickets able to outperform even their unpruned, fully-trained models (c.g., the dashlines), potentially thanks to the sparse regularization learned by EB tickets.
Table 1: The retraining accuracy of subnetworks drawn at different training epochs using different learning rate schedules, with a pruning ratio of 0.5. Here [0, 100] represents [O,n-0.01, 100 while [80, 120] denotes (oanotes[[0,,aa [80,R=0000 120 J. for compactness.
Retrain acc. (%) (CIFAR-100)
Retrain acc. (%) (CIFAR- 10)
LR Schedule
10
20
40
final
10
20
40
final
[0. 100]
66.70
67.15
66.96
69.72
92.88
93.03
92.80
92.64
VGG16
[80, 120]
71.11
71.07
69.14
69.74
93.26
93.34
93.20
92.96
[0. 100]
69.68
69.69
69.79
70.96
92.41
92.72
92.42
93.05
PreResNet101
[80, 120]
71.58
72.67
72.67
71.52
93.60
93.46
93.56
93.42
3.2 DO EARLY-BIRD TICKETS STILL EMERGE UNDER LOWER- COST TRAINING?
For EB tickets, only the important connections (connectivity) found by them matter, while the weights are to be re-trained anyway. One might hence come up with an idea, that more aggres- sive and "cheaper" training methods might be applicable further shrink the cost of finding EB tickets (on top of the aforementioned early stopping of training thanks to the identification of EB tickets ), as long as the same significant connections still emerge. We experimentally investigate the impacts of two schemes (which are using appropriate large learning rates and training in lower precision): EB tickets are observed to survive well under both of them.
Appropriate large learning rates are important to the emergence of EB tickets. We first vary the learning rate schedule in the above experiments. The original schedule is denoted as [80 120 J. i.e., starting from 0.1, decayed to 0.01 at the 80th epoch, and fur- ther decayed to 0.001 at the 120th epoch. In comparison, we test a new learning rate sched- ule [O..n-o.m 100, ::00000000 starting from 0.01 (the Oth epoch), and decayed to 0.001 at the 100th epoch. After drawing tickets, we re-train them all using the same learning rate schedule [80g8~0.01: 120 R--00.0000 for sufficient training and a fair comparison. We can from Table thhat high-quality EB tickets always emerge earlier when searching with the schedule of a larger learning rates, i.c., [80
120_nt-o.001]] whose final accuracies are also better. Note that the earlier emergence of good EB tickets contributes to lowering the training costs. It was observed that larger learning rates are beneficial for training the dense model fully to draw the winning tickets in (Frankle & Carbin 2019 Liu et al. 2018b): here we show this benefit extends to EB tickets too. More experiments with larger learning rates can be found in Appendix A.2
Low-precision training does not destroy EB tickets. We next examine the emergence of EB tickets within a ofate-of-the-art low-precision training algorithm (Wu et al. 2018h). where all model weights, activations, gradients and errors are quantized to 8 bits throughout training. Note that here we only apply low-precision training to the stage of identifying EB tickets, and afterwards the tickets are trained in the same full-precision in Section 3.1. Figure 2 shows the retraining accuracy and total number of FLOPs (EB ticket search (8 bits) + retraining (32 bits floating points)) for VGGI6 and CIFAR-10/100 for identifying EB tickets: the general trends resemble the full tree-iici training cases in Figure except the emergence of good EB tickets seem to become even earlier up to initial epochs. In this
We can that EB tickets still emerge when using aggressively low-precision
4

---

Published as a conference paper at ICLR 2020
VGG16@CIFAR-10
VGG16@CIFAR-10
VGG16
VGC160CIFAR-IDE
10%
10%
AcC-92.60%
briann FLOPS (P)
AcC.899.0%
L8 ticset
EB tiset
Winning Uicket
Winniy
accuracy
1,58X
10 8
Acc.69.789
2,69X
2A7
Acc 92.53%
Spech with search Dronased baselee
best Searth
Urpexned bosetine
70%
10 Figure 2: Retraining accuracy and total training FLOPs comparison epoch number which the subnetwork drawn, when using 8 bits precision during the stage of identifying EB tickets based on the VGG16 model and CIFAR-10/ AARR-00100 datasets, where p indicates the channel-wise pruning ratio and the dashed line shows the accuracy of the corresponding dense model on the same dataset.
85
250
Epoch subnetwork 65
subnetwork drawn
subnetwork
subnebwork
way, it will lead cost savings in finding EB tickets, since low-precision updates can aggressively save energy compared to their full-precision baseline, as shown in the Table 2
3.3 How TO IDENTIFY EARLY-BIRD TICKETS PRACTICALLY?
Distance between Ticket Masks. For each time of pruning, we define a binary mask the drawn ticket (pruned subnetwork) w.r.t. the full dense network. We follow (Liu al. 2017) to consider the scaling factor r in batch normalization (BN) layers as indicators of the corresponding channels' significance. Given target pruning ratio p. then prune the channels with top p-percentage smallest values. Denote the pruned channels as O while the kept ones as 1, the original network be mapped into binary "ticket mask". For any sub-networks pruned from the same dense model, we calculate their mask distance via the Hamming distance between their two ticket masks.
Detecting EB Tickets via Mask Distances. We first visualize and observe the global behaviors of mask distances between consecutive epochs. Figure 3 plots the pairwise mask distance matrices (160 160) of the VGG16 and PreResNet101 experiments on CIFAR-100 (from Section 3.1), at different pruning ratios p, where (i. (..j)-th element in a matrix denotes the mask distance between subnetworks drawn from the i-th and j-th epochs in that corresponding experiment (160 epochs in total for all). For the ease of visualization, all elements in matrix are linearly normalized between O and 1; Therefore, the resulting matrix, a lower value (close to 0) indicates a smaller mask distance and is highlighted with warmer color (same hereinafter).
Figure demonstrates fairly consistent behaviors. Taking VGGI6 on CIFAR-100 (p = 0.2) for an example: 1) at the very beginning, the mask distances change rapidly between epochs, manifested by the quickly "cooling" colors (yellow - green), from the diagonal line (lowest since that is comparing epoch's 's mask with itself), to off-diagonal (comparing different epochs; the deviates from the diagonal, the more distant the two epochs are away from each other); 2) after ~ 10 epochs, the off off-diagonal elements become "yellow" too, and the color transition becomes smoother from diagonal to off-diagonal, indicating the masks change mildly only after passing this point; 3)
element
VGG16 (p 0.2)
VG616 (p
VGG16 (0-0.))
VGG16 (p- 0.8)
25
0.2
Epoch subnetwork drawn from
0.4
75
0.5
0.6
0.6
125
0.8
0.8
0.9
0.8
150
PreResNet101 (p 0.2)
PreResNet101 (p= 0.4)
PreResNet101 (p= 0.61
PreResNet101 (p 0.8)
0.2
0.2
s0
0.4
100
50
Epech subnetwork drawn
Figure 3: Visualization of the pairwise mask distance matrix for VGG16 and PreResNet101 on CIFAR-100.

---

Published as a conference paper at ICLR 2020
after -80 epochs, the mask distances almost remain unchanged across epochs. Similar trends observed in other plots too. seems to concur the hypothesis in (Achille et al., 2019) that a network first learns important connectivity patterns and then fixs them and further tune their weights.
Our observation that the ticket masks quickly become stable and hardly changed in early training stages supports drawing EB tickets. We therefore measure the mask distance between the consec- utive epochs, and draw EB tickets when such distance is smaller than a threshold c. Practically, to improve the reliability of EB tickets, we will stop draw EB tickets when the last five recorded mask distances are all smaller than e, to avoid some irregular fluctuation in early training stages. In Figure 3] the red lines indicate the identification of EB tickets when e set to 0.1.
EFFICIENT TRAINING VIA EARLY BIRD TICKETS
In this section, present effi- cient DNN training scheme, termed EB Train, which leverages 1) the existence of EB tickets and 2) the proposed mask distance that can de- tect the emergence of EB tickets for time- and energy-efficient train- ing. We will first provide con- ceptual overview, and then describe the routine of EB Train, and finally show the evaluation performance of EB Train by benchmarking it with statc-of-the-an methods of ob- taining compressed DNN models based on representative datasets and DNNs.
Progressive Pruning and Training
Trained Model
training
(train N'' epochs, N 160) Train (Proposed)
6% 20% training (train N opochs, e.g., Nep 10)
Figure 4: high-level overview the commonly adopted progressive pruning and training scheme and our EB Train.
4.1 WHY IS EB TRAIN MORE EFFICIENT?
EB Train vs. Progressive Pruning and Training. Figure 4 illlustrates an overview of our proposed EB Train and the progressive pruning and training scheme, e.g., as in (Frankle & Carbin 2019). In particular, the progressive pruming and training scheme adopts three-step routine of 1) training a large and dense model, 2) pruning it, and 3) then retraining the pruned model restore perfor- mance, and these three steps can be iterated (Han et al. 2015). The first step often dominates (e.g., occupy 75% training FLOPs when using the PreResNet model and CIFAR- 10 dataset) in terms of training energy and time costs as it involves training large and dense model. The key feature of our EB Train scheme is that replaces the aforementioned steps I and 2 with a lower-cost step of detecting the EB tickets, i.e., enabling early stopping during the time- and energy-dominant step of training the large and dense model, thus promising large savings both training time and en- ergy. For example, assuming the first step of the progressive pruning and training scheme requires N epochs to sufficiently train the model for achieving the target accuracy, the proposed EB Train needs only NEB epochs to identify the EB tickets by making use of the mask distance that can detect the emergence of EB tickets. where NEB < N (e.g., NEB/A N = 12.5% in the experiments summarized in Figure and NEB/
6.25% in the experiment summarized in Table
Algorithm 1:
The Algorithm for Searching EB Tickets Initialize the weights W, scaling factor r. pruning ratio p, and the FIFO queue Q with length /; while (epoch) < Emax do
1:
2:
3: Update W and using SGD training:
4: Perform structured pruning based on r2 towards the target ratio pt
5:
Calculate the mask distance between the current and last subnetworks and add to Q.
6:
f=t+1
7:
if 1MMxx Q) < then
8:
=t Return f(x; W) (EB ticket);
9:
10: end if
end while

---

Published as a conference paper at ICLR 2020
PreResNet101
VGG16
PreResNet101
VGG16
Acc.992.83%
30%
Acc.:93. 05%
10%
18
Acc.72.88%
20%
Acc.i71.289
Total trianing FLOPS
p 70%
50%%
p- 50%
30%
16
p- 70%
50%
-0%%
50%
14
68 ticket with
EB ticket with
6B ticket with
6a ticket with
10:
2.17X
A9c.193.81%
ticket
20
Acc.193.39%
Bikket
2.33X
Acc.173.08%
ticket
20
2,40X
Acc.:72.277
ticket
0:
15
10
4
10 Epoch subnetwork
10 60
20 BC
20 Epoch subnetwork
Epoch subnebwork
Epoch subnetwork
CIFAR-10
CIFAR-100
Figure 5: The total training FLOPs vs. the epochs at which the subnetworks are drawn from, for both the PreResNet101 and VGG16 models on the CIFAR- 10 and CIFAR- 100 datasets, where indicates the channel-wise pruning ratio for extracting the subnetworks. Note that the EB tickets at all cases achieve comparable or higher accuracies and consume less FL.OPs than those of the "ground-truth" winning tickets (drawn after the full training of 160 epochs).
4.2 HOw TO IMPLEMENT EB TRAIN?
From Figure I we can see that the EB Train scheme consists of training (searching) step to identify EB tickets and a retraining step to retrain the EB tickets for achieving the target accuracy. Algorithm|] I the step of searching for EB tickets. Specifically, the EB searching process 1) first initializes the weights W and scaling factors 2) iterates the structured pruning process in (Liu et al.] (017)) to calculate the mask distances between the consecutive resulting subnetworks, storing them into first-in-fist-ouu (FIFO) queue with a length of 1 5; and 3) exits when the maximum mask distance in the FIFO is smaller than a specified threshold c (default 0.1 with the normalized distances of [0,1]) The output is the resulting EB ticket denoted as f fa;; me O W), which will be retrained further to reach the target accuracy. Note that unlike the lottery ticket training in (Frankle & Carbin 2019). EB Train inherits the unpruned weights from the drawn EB ticket instead of rewinding to the original initialization, as it has been shown that deeper networks are not robust reinitializion with untrained weights (Frankle et al. 2020).
4.3 How DOES EB TRAIN PERFORM COMPARED TO STATE- CEIE--FF-THH---TEE ART TECHNIQUES?
Experiment Setting, We consider training the VGG16 and PreResNet101 models on both CIFAR- 10/100 and ImageNet datasets following the basic setting of (Liu et al., 2018b) (i.e., training 160 epochs for CIFAR and 90 epochs for ImageNet). We measure the training energy costs from real- device operations as the energy cost consists of both computational and data movement costs, the latter of which is often dominant but can not captured by the commonly used metrics, such as the number of FLOPs (Chen et al. 2017). we evaluate the proposed techniques against the baselines in terms of accuracy and real measured energy consumption. Specifically, all the energy consumption of pree-precision models are obtained by training the corresponding models in an embedded GPU (NVIDIA JETSON TX2). The GPU measurement setup can be found in the Appendix. Note that the energy measurement results include the overhead of using the mask distance to detect the emergence of EB tickets, which is found negligible as compared to the total training cost. For example, for the VGG16 model, the overhead caused by computing mask distances is S 0.029% in memory storage size, s 0.026% in FLOPs, and 0.07% in real- meaauredd energy costs. This because 1) the mask distance evaluation only involves simple hamming distance calculations and 2) each epoch only calculates the distance once.
We first compare the computational savings with the baselines using a progressive pruning and training scheme (Liu al., 2017) in terms of the total training FLOPs, when drawing subnetworks at different epochs. Figure S summarizes the results of the PreRes- Ne1101/VGG16 models and CIFAR-10/100 datasets, corresponding to the same set of experiments as in Figure I We see that EB Train can achieve 2.2 2.4x FLOPs reduction over the baselines, while leading to comparable or even better accuracy (- 0.81% ~ + 2.38% over the baseline).
Results and Analysis.
Table compares the retraining accuracy and consumed FLOPs/energy of EB Train with four state- of-the-art progressive pruning and training techniques: the original lottery ticket (LT) training (Fran- kle & Carbin 2019), network slimming (NS) (Liu et al.) 2017). ThiNet (Luo et al. (017) and SNIP (Lee et al. 2019). While all of them involve the process of training a dense network, pruning it, and retraining, they apply different pruning criteria, e.g., NS imposes Lq-sparsity on channel-wise
7

---

Published as a conference paper at ICLR 2020
Table 2: Comparing the accuracy and energy/FLOPs EB Train (including its variants), NS Liu et al. 2017)) LT (Frankle & Carbin 2019), SNIP (Lee et al. 2019). and ThiNet (Luo et al. 2017
Retrain
Energy cost (KJ)FLLPPs (P)
Setting
Methods
p=30%
p=50%
p-70%
p=30%
p=50%
p=70%
LT (one-shol)
93.70
93.21
92.78
6322/14.9
6322/14.9
6322/14.9
SNIP
93.76
93.31
92.76
3161/7,40
3161/7.40
3161/7.40
NS
93.83
93.42
92.49
5270/13.9
4641/12.7
4211/11.0
ThiNet
93.39
93.07
91.42
3579/13.2
2656/10.6
1901/8.65
PreResNet
EB Train (re-init)
93.88
93.29
92.39
2817/7.75
2382/7.05
1565/3.77
-101
EB Train (FF)
93.91
93.90
92.49
2370/6.50
1970/5.70
1452/3.50
CIFAR-10
EB Train (I.F)
93.48
93.31
92.24
2265/6.45
1667/5.39
1338/3.44
EB Train (LL)
93.24
92.85
92.12
489.46.45
410.9/5.39
281.8/3.4
EB Train Improw.
0.08
0.48
-0.29
6.5x/1.11
6.5x/1.4x
6.7x/2.2X
LT (one-shot)
93.18
93.25
93.28
746.2/30.3
746.2/30.3
746.2/30.3
SNIP
93.20
92.71
92.30
373.1/15.1
373.1/15.1
373.1/15.1
NS
93.05
92.96
92.70
617.1/27.4
590.7/25.7
553.8/23.8
ThiNet
92.82
91.92
90.40
298.0/22.6
383.9/19.0
380.1/16.6
VGG16
EB Train (re-init)
93.11
93.23
92.71
290.4/14.44
237.3/12.0
200.5/9.45
CIFAR-10
EB Train (FF)
93.39
93.26
92.71
256.4/12.7
213.4/10.8
184.2/9.85
EB Train (LF)
93.20
93.19
92.91
250.
199.4/10.7
170.3/8.51
EB Train (LL)
93.25
93.13
92.60
56.1/12.8
43.1/10.7
36.5/8.51
EB Train Improw.
0.19
0.01
0.57
6.6x/1.2x
8.6x/14x
10.2x/1.8x
(one-shot)
71.90
71.60
69.95
6095/14.9
6095/14.9
6095/14.9
SNIP
72.34
71.63
70.01
3047/7.40
3047/7.40
3047/7.40
NS
72.80
71.52
68.46
4851/13.7
4310/12.5
3993/10.3
ThiNet
73.10
70.92
67.29
3603/13.2
2642/10.6
1893/8.65
PreResNet
EB Train (re-init)
73.23
73.36
71.05
2413/7.35
2016/6.25
1392/3.53
-101
EB Train (FF)
73.52
73.15
72.29
2020/6.40
1769/5.45
1294/3.28
CIFAR-100
EB Train (LF)
73.41
73.02
70.72
2038/6.42
1614/5.45
1171/2.99
EB Train (LL)
73.04
71.82
69.45
434.4/6422
366.5/5.45
247.3/2.99
EB Train Improw.
0.42
1.73
2.28
7.0x/1.2x
7.2x/1.4x
7.6x/2.5x
10%
p=30%
p=50%
p=10%
p=30%
p=50%
LT (one-shot)
72.62
71.31
70.96
741.2/30.3
741.2/30.3
741.2/30.3
SNIP
71.55
70.83
70.35
370.6/15.1
370.6/15.1
370.6/15.1
NS
71.24
71.28
69.74
636.5/29.3
592.3/27.1
$67.8/24.0
ThiNet
70.83
69.57
67.22
632.2/27.4
568.5/22.6
381.4/19.0
VGG16
EB Train (re-init)
71.65
71.48
69.66
345.3/16.3
300.0/13.7
246.8/10.6
CIFAR-100
EB Train (FF)
71.81
72.17
71.28
287.7/14.1
262.2/12.2
221.7/9.85
EB Train (LF)
71.60
71.50
70.27
270.5/14.0
262.7/12.8
208.7/10.1
EB Train (I.L)
71.34
70.53
69.91
54.6/14.0
64.//1288
44,4/10.1
EB Train Improv.
0.81
0.86
6.8x/1.11
5.8x/1.2x
10.7x/1.5x
scaling factors from BN layers, and ThiNet greedily prunes the channel that has the smallest effect the next layer's activation values. For EB Train, by default follow (Liu al. 2017) in- herit the same weights when re-training the searched ticket, adopting floating points for both the search and retrain stages (i.e., EB Train FF). We also notice existing debates (Liu the initialization re-use, and thus also compare with a variant by re-training the ticket from a new random initialization, termed as EB Train (re-init). The comparisons in Table 2 further show that inheriting weights from the EB tickets favor the generalization of retraining as compared to both the random initialization and "over-cooked" weights, aligning well with the recent discussion between rewinding and fine-tuning (Renda et al., 2020). Furthermore, of the low-precision training method (Yang et al. 2019a) and obtain experiment results of another two variants of EB Train: 1) EB Train with low-pre wwwpprccision search and full-precision retrain (i.c., EB Train LF), and 2) EB Train with both low-precision search and retrain (i.c., EB Train LL).
2018b) on
apply the proposed EB Train on top
Table demonstrates that EB Train consistently outperforms all competitors in terms of saving training energy and computational costs, meanwhile improving the final accuracy in most cases. We EB Train Improv. to record the performance margin (either accuracy energy/computation) be- tween EB Train and the strongest competitor among the four state-of- the-art baselines. Specifically, EB Train with full-precision floating point (FP32) search and retrain outperforms those pruning
8

---

Published as a conference paper at ICLR 2020
Table 3: Comparing the accuracy and total training FLOPs of EB Train, Network Slimming (Liu et al. 2017), ThiNet (Luo et al., 2017), and SFP (He et al. 2018). The "Acc. Improv." is the accuracy of the pruned model minus that of the unpruned model, so a positive number means the
pruned model has a higher accuracy.
Pruning ratio
Top-1 Acc. (%)
Top-1 Acc. Improv. (%)
Top-S Acc. (%)
Top-5 Aoc. Improv. (%)
Total Training
Total Training Energy (MJ)
Models
Methods
FLOPs (P)
Unpruned
69.57
89.24
1259.13
98.14
10%
69.65
+0.08
89.20
-0.04
2424.86
193.51
NS
30%
67.85
-1.72
88.07
-1.17
2168.89
180.92
ResNet18 ImaageNet
SFP
30%
67.10
-2.47
87.78
-1.46
1991.94
158.14
10%
69.84
+0.27
89.39
+0.15
1177.15
95.71
EB Train
30%
68.28
-1.29
88.28
-0.96
952.46
84.65
Unprumed
75.99
92.98
2839.96
280.72
30%
72.04
-3.55
90.67
-2.31
4358.53
456.13
ThiNet
50%
71.01
-4.58
90.02
-2.96
3850.03
431.73
ResNet50
70%
68.42
-7.17
88,30
-4.68
3431.48
416.44
ImageNet
30%
73.86
-1.73
91.52
-1.46
2242.30
232.1 18
EB Train
73.35
-2.24
91.36
-1.62
1718.78
188.
70%
70.16
-5.43
89.55
-3.43
890.65
121.15
methods by up to 1.2 - 4.7x and 1.1 4.5x in terms of the energy consumption and computa-
tional FLOPs, while always achieving comparable or even better accuracies, across three pruning ratios, two DNN models and two datasets. Moreover, EB Train with both low-precision (8 bits block floating point (FP8)) search and retrain outperforms the baselines by up to 5.8 24.6x and 1.1 ~ 5.0x in terms of energy consumption estimated using measuredd unit energy from YYnn al. 2019b) and computational FLOPs. Note that FP8 and FP32 are counted the same FL.OPs count units (Sohoni et al.) (019). In addition, comparing with the re-init variant endorses the effectiveness of initialization inheritance EB Train. As an additional highlight, EB Train nat- urally Icads to more efficient inference of the pruned DNN models, unifying the boost of efficiency throughout the full learning lifecycle.
ResNet18/50 on ImageNet. To study the performance of EB Train in harder dataset, we conduct experiments using ResNet1 8/50 and ImageNet, and compare its resulting accuracy, total training FLOPSs and energy cost with those of the method in (He et al. (016 NS (Liu et al.) 2017). ThiNet (Luo et al. 2017) and SFP (He et al. 2018) as summarized Table We have three observations: 1) When training ResNet18 on ImageNet, EB Train achieves a better accuracy (+0.27%) over the unpruned one while introducing 10% channel sparsity: 2) EB Train outperforms other baselines for both ResNet18 and ResNet50 by reducing the total training costs while achieving comparable or better accuracy. Specifically, EB Train achieves a reduced training FLOPs of 51.5% 56.1% and 48.6% 74.0% and reduced training energy of 46.5% 53.2% and 49.1 1% 70.9%, while leading to better top-1 accuracy of +0.19% ~ +1. +1.18% and +1.74% ~ +2.34% for ResNet18 8 and ResNet50, respectively. For example, when training ResNet50 on ImageNet, EB Train with pruning ratio of 70% achieves better (+1.74%) accuracy over ThiNet while saving 74% total training FLOPs and 71% training energy: 3) Different from the results on CIFAR-10/100 shown in Table 2 all methods performed on ImageNet (a harder dataset) start to yield accuracy reductions (- (-1.72% ~ -3.55%) for BesNet18/50 with a pruning ratio only 30%, compared with the unpruned one. Note that in Table 3 the unpruned results are based on the official implementation in (He et al. 2016 The SFP results are obtained from their original paper (He et al., (0188), which does not provide results at a pruning ratio of 10%; all the ThiNet results under various pruning ratios obtained from the original paper (Luo et al. 2017); and the NS results are obtained by conducting the experiments ourselves.
DISCUSSION AND CONCLUSION
We have demonstrated that winning tickets can be drawn the very early training stage, i.e., EB tickets exist, in both the standard training and several lower-cost training schemes. That motivates a practical success of applying EB tickets to efficient training, whose results compare favorably against -fate-of-the-arts.. Moreover, experiments show that EB Train with low-precision search and retraining achieve more efficient training. We believe many promising problems still remain open to be addressed. An immediate future work is to test low-precision EB Train algorithms on larger models/datasets. We are also curious whether more lower-cost training techniques could be asso- ciated with EB Train. Finally, sometimes high pruning ratios (e.g, p 0.7) can hurt the quality of EB tickets and the retrained networks. We look forward to automating the choice of p for different models/datasets, unleashing higher efficiency.
9

---

Published as a conference paper at ICLR 2020
6 ACKNOWLEDGEMENT
The work is supported by the National Science Foundation (NSF) through the Real-Time Machine Learning (RTML) program (Award number: 1937592, 1937588).
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep net- works. In International Conference on Learning Representations, 2019. URL https://
Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet- 50 on imagenet in 15 minutes. CoRR, abs/1711.04322 2017.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, pp. 5145 5153, 2018.
Y. H. Chen, T. Krishna, J. S. Emer, and V. Sze. Eyeriss: An energy-efficient reconfigurable accelera- tor for deep convolutional neural networks. IEEE Journal Solid-State Circuits, 52(1):127-138, Jan 2017. ISSN 0018-9200
Minsik Cho, Ulrich Finkler, Sameer Kumar, David S. Kung, Vaibhav Saxena, and Dheeraj Sreedhar. Powerai ddl. CoRR, abs/1708.02288. 2017.
Emily I. Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 1269-1277. Curran Associates, Inc., 2014.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. International Conference on Learning Representations, 2020. URI. https: 7/openrevi ew. wttt:/fooumiiddieem 1
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training im- agenet in hour. CoRR, abs/1706.02677, 2017. URL 02677
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746,
2015.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.001 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016a. URL https:///i httt::://ggttubb.cmm/aaaaoooo www.m//aaaaoooarcccccccooooocaee fb. reanet.
Eorch
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference on Artificial Intelligence (IICAI), pp. 2234-2240, 2018
Itay Hubara, Matthieu Courbariaux. Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Leaming Research, (((1))666-6886 2017.
10

---

Published as a conference paper at ICLR 2020
Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, et al. Highly scalable deep learning training system with mixed-precision Training imagenet in four minutes. arXiv preprint urXiv:1807.111 1205, 2018.
Namhoon Lee. Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single- shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. arXiv preprint arXiv: 1907, 190vv:190.004555, 2019.
Sicong Liu, Yingyan Lin, Zimu Zhou, Kaiming Nan, Hui Liu, and Junzhao Du. On-demand deep model compression for mobile devices: A usage-driven model selection framework. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pp. 389 400. ACM, 2018a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv: 1810.05270, 2018b.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. 2019.
NVIDIA Inc. NVIDIA Jetson TX2.
https: www. nvidia. comm//en-ss aut hwwpa/saaaaaiieeeeeeeeeeeeeeeeee wttpeeedddsssttttte//tttto---eee jetss//jjstt--teeeeeeee accessed 2019-09- 01.
systems/
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 09-15 Jun 2019.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruming. In International Conference on Learning Representations, 2020. URL https:
//open/evvew..et//e
Jianghao Shen, Yonggan Fu, Yue Wang, Pengfei Xu, Zhangyang Wang, and Yingyan Lin. Fractional Skipping: Towards Finer-Grained Dynamic Inference. In The Thirty-Forth AAAI Conference on Artificial Intelligence, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszezynski, Jian Zhang, and Christopher Ré. Low-memory neural network training: A technical report. arXiv preprint 1xXiv:1904.106111 2019.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and considertin for deep learning in NLP. CoRR, 000/1906602243, 2019. URL http://arriiv.ott/bbb///
02243
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems, pp. 7675-7684, 2018a. Sa.

---

Published as a conference paper at ICLR 2020
Yue Wang, Tan Nguyen, Yang Zhao, Zhangyang Wang, Yingyan Lin, and Richard Baraniuk. Energ- ynet: Energy-efficient dynamic inference. 2018b.
Yue Wang, Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang. E2-train: Training state-of-the-art cnns with over 80% energy savings, In Advances in Neural Information Processing Systems, pp. 5139-5151, 2019a.
Yue Wang, Jianghao Shen, Ting-Kuei Hu, Pengfei Xu, Tan Nguyen, Richard Baraniuk, Zhangyang Wang, and Yingyan Lin. Dual dynamic inference: Enabling more efficient, adaptive and control- lable deep inference. arXiv preprint arXiv: 1907 1907.0453. 2019b.
Yunhe Wang, Chang Xu, XU Chunjing, Chao Xu, and Dacheng Tao. Learning versatile filters for efficient convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1608-1618, 2018c.
Yunhe Wang. Chang Xu, Chao Xu, and Dacheng Tao. Packing convolutional neural networks in the frequency domain. IEEE transactions on pattern analysis and machine intelligence, 41(10): 2495 2510, 2018d.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. arv preprint arXiv: 0926,, 2018a.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations, 2018b. URL https:
7/ //openreview. net. wttp:forummiicoo wwt.:eeteeoooommmmmmtmttt d.
Pengfei Xu, Xiaofan Zhang, Hao Cong, Yang Zhao, Yongan Zhang, Chaojian Li, Zetong Guan, Deming Chen, and Yingyan Lin. AutoDNNchip: An automated dnn chip predictor and builderfor both fpgas and asics. In 28th ACM/SIGDA International Symposium on Field-Programmnt Programmable Gate Arrays, 2020.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06552. 2019.
Guandao Yang. Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Christo- pher De Sa. SWALP: Stochastic weight averaging in low precision training. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, mir press/v97/
California, USA, pp. 7015-7024, 2019a. URL yangl 9d. html
Yukuan Yang, Shuang Wu, Lei Deng, Tianyi Yan, Yuan Xie, and Guoqi Li. Training high- performance and large-scale deep neural networks with full 8-bit integers. arXiv preprint arXiv: 2019b.
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. Proceedings the 47th International Conference on Parallel Processing, pp. 1. ACM, 2018.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, 2019.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017. URL https://arxiv..rr Mos/1611.00 01578
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 MEASUREMENT OF ENERGY COST
Figure 6 shows our GPU measurement setup, in which the GPU board connected laptop and power meter. In particular, the training settings are downloaded from the laptop to the GPU board, and the real-measured energy consumption obtained via the power meter and runtime measurement for the whole training process.
Edge GPU
Training Settings KDNN, epoch, learning etc.)
1203
SSH
Measurement Result (accuracy, runtime)
Figure 6: The energy measurement setup (from left right) with a MAC Air latptop, an embedded GPU (JETSON TX2 (NVIDIA Inc.)), and a power meter.
A.2 APPROPRIATE LARGE LEARNING RATE FAVORS THE EMERGENCE EB TICKETS
Here we show the effect of larger learning rates setting. The original schedule is denoted as [80Læ-00001 120 L2-0.o000 i.c. starting from 0.1, decayed to 0.01 at the 80th epoch, and further decayed to 0.001 at the 120th epoch. In comparison, from 0.5 and 0.2 for VGG16 and PreResNet101 datasets, respectively. After drawing tickets, we re- train them all using the same learning rate schedule with their searching phase for sufficient training and a fair comparison. We see from Table 4 that high-quality EB tickets also emerge early when searching with the larger initial learning rate, whose final accuracies even better.
test larger initial learning rate, staring
The retraining accuracy of subnetworks drawn different training epochs using different inital learning rate, where the pruning ratio is 0.5.
Table 4:
Retrain acc. (CIFAR-100)
Retrain acc.(%) (CIFAR-10)
LR Initial
10
20
40
final
10
20
40
final
0.1
71.11
71.07
69.14
69.74
93.26
93.34
93.20
92.96
VGG16
0.5
70.69
71.65
71.94
71.58
93.49
93.45
93.44
93.29
0.1
71.58
72.67
72.67
71.52
93.60
93.46
93.56
93.42
PreResNet101
0.2
72.58
72.90
72.86
72.71
93.40
93.46
93.87
93.69
13


Reviews:
[{"review_id": "rk5HxSNKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to speed up training of deep neural networks. The main contribution is a method to quickly identify winning lottery tickets (denoted early-bird, or EB by the authors), without running the model to convergence. The authors present interesting preliminary experiments that motivate their method, and show that it works on two image recognition datasets using two models.\n\nThis paper addresses an under-explored, but very important problem in AI: the increasing cost of training models. The authors present interesting evidence about the potential to detect EBs early on. The experiments presented in Figures 1 and 3 are convincing and will be of interest to the community. The proposed method seems to work, at least on the setups explored by the authors. I am leaning towards acceptance, but am concerned with the following: \n\n1. The authors experiment with a limited set of datasets (CIFAR-10 is a relatively easy task), and with a set of non-competitive baselines (SOTA for CIFAR-10/100 is 99%/91.3%, see https://benchmarks.ai/cifar-10{,0}). I would have liked to see whether the proposed method translates to harder datasets and stronger models.\n\n2. I might be missing something here, but to the best of my understanding the large learning rate part (page 4) does not demonstrate the benefits of increasing the learning rate, but the problems with *decreasing* it. The two might seem like the same thing, but in fact they're not: the authors claim the [80,120] policy is standard, and use it when training the subnetwork, so showing that [0,100] is inferior does not present a way to improve over the current approach, but evidence that the other approaches are inferior.\n\nOther questions: \n1. In Figure 1, it seems that the extracted subnetworks are doing very well even after 0 epochs. Does this mean that a trained version of a random subnetwork could reach within 1-2 points of the unpruned model? or is it pruned after training for 1 epoch?\n\n2. If I understand correctly, Figure 5 should be illustrating the proposed method, which automatically identifies the early stopping point. In that case, I am not sure why the plot is a function of the epoch.\n\n3. Do the authors have any intuition as to the sharp decrease in the 70% graph in Figures 1 and 2 around epoch 50?\n\nWriting: \n\n1. The language used by the authors is sometimes exaggerated. Expressions such as \"bold guess\" (section 3.2), \"innovative ... scheme\" (section 4) and comparisons to Winston Churchill would be better left out of the manuscript. \n\n2. Typos and such: \n-- several across the paper. For instance: \n- Intro: After *bring* identified (should be \"being\")\n- Related work: when training *it* isolation (in)\n\n-- Missing venue for Frankle and Corbin (2019)\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkx7VRIhFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors further study the lottery ticket hypothesis formulated by Frankle and Carbin. They demonstrate that the sparsity pattern corresponding to a lottery ticket for a given initialization can be uncovered via low-cost training. By doing so, they propose a method to: 1) first identify the lottery ticket efficiently and 2) exploit the sparsity of the resulting network to train it at a lower cost.\n\nThe contribution is however a bit incremental in my opinion. The original LT paper was not focused on efficiency, and it is not a far stretch to try to find the tickets sooner during the training. On the other hand, the experiments are well conducted (especially 4.3) and even if the original idea is simple, it is of interest to see it tested as clearly. All in all, I found this paper convincing and worth reading, and I think it should be accepted.\n\nPositive points:\n- The literature review is sufficient and present with great clarity the latest results.\n- The problem tackled is of great interest and has potentially impactful applications.\n- The authors focus on hardware friendly types of pruning.\n- The paper is well written and enjoyable.\n- The algorithm used to compute the EB tickets seems a bit ad hoc, but in my opinion sufficient as a first approach.\n\n\nNitpicking:\n- Not sure why Max(Q) > eps is in the while condition (return if Max(Q) < eps should be sufficient)\n- The treatment of the mask distance in figure 3 is confusing. It is not obvious why the authors are plotting 1-distance, and the legend of the figure suggest that a mask has a distance of 1 with itself. Recommend plotting d instead of 1-d and invert the color bar instead if they feel so inclined (yellow=0).\n- Abstract: “consistently exist across models and datasets” a bit of a strong claim as only cifar is used.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJxIRropKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper empirically analyzed the wide existence of \"early-bird tickets\", e.g., the \"lottery tickets\" emerging and stabilizing in very early training stage. The potential connection to (Achille et al., 2019; Li et al.,2019) reads interesting. \n\nThe authors made several contributions in addition to the observation: (1) the EB tickets stay robust under large learning rates (while early stopping) and low-precision training; (2) the EB tickets can be detected using epoch-wise consecutive comparison (mask distance), rather than comparing with some oracle ticket; (3) the application of EB ticket towards energy efficient training, which is interesting as this is perhaps the first practical application demonstrated of lottery ticket. \n\nWhile I like how the paper connects theory hypothesis to real applications, the experiments need to be solidified in a few aspects:\n\n1) Figures 1 and 2, why a few plunges of curves (say Fig 2.a, p = 70%)? Does it imply the training might not be stable?\n\n2) Table 1, the authors test two lr schedules to show \"large learning rates favor the emergence of EB Tickets\". Yet the choice of lr matters a lot and can be tricky. Why the authors pick the two specific learning rate schedules? Why are they \"comparable\"? What if being more aggressive in choosing larger lr, say starting from lr= 0.5?\n\n3) The low precision EB ticket is not actually applied or evaluated in Section 4. It would have been interesting to see.\n\n4) I fail to find the 4.7 times energy saving as claimed in abstract from Table 2?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies small but critical subnetworks, called winning tickets, that have very similar performance to an entire network, even with much less training. They show how to identify these early in the training of the entire network, saving computation and time in identifying them and then overall for the prediction task as a whole. 

The reviewers agree this paper is well-presented and of general interest to the community. Therefore, we recommend that the paper be accepted.
Decision: Accept (Spotlight)