Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
Authors: Xiaolong Ma
Geng Yuan
Xuan Shen
Tianlong Chen
Xuxi Chen
Xiaohan Chen
Ning Liu
Minghai Qin
Sijia Liu
Zhangyang Wang
Yanzhi Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Lottery ticket hypothesis

Abstract: There have been long-standing controversies and inconsistencies over the experiment setup and criteria for identifying the "winning ticket" in literature. To reconcile such, we revisit the definition of lottery ticket hypothesis, with comprehensive and more rigorous conditions. Under our new definition, we show concrete evidence to clarify whether the winning ticket exists across the major DNN architectures and/or applications. Through extensive experiments, we perform quantitative analysis on the correlations between winning tickets and various experimental factors, and empirically study the patterns of our observations. We find that the key training hyperparameters, such as learning rate and training epochs, as well as the architecture characteristics such as capacities and residual connections, are all highly correlated with whether and when the winning tickets can be identified. Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis. Our codes are publicly available at: https://github.com/boone891214/sanity-check-LTH.

Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
Xiaolong Ma Geng Yuan Xuan Shen', Tianlong Chen', Xuxi Chen', Xiaohan Chen',
Ning Liu", Minghai Qin, Sijia Liu", Zhangyang Wang", Yanzhi Wang"
Northeastern University, 2 University of Texas at Austin
Midea Group, Michigan State University [ma xiaol, yanz. wang. @nort hwwstern.ennn.....
Abstract
There have been long-standing controversics and inconsistencies over the exper- iment setup and criteria for identifying the "winning ticket" in literature. To reconcile such, we revisit the definition of lottery ticket hypothesis, with compre- hensive and more rigorous conditions. Under our new definition, show concrete evidence to clarify whether the winning ticket exists across the major DNN architec- tures and/or applications. Through extensive experiments, analysis on the correlations between winning tickets and various experimental factors, and empirically study the patterns of our observations. We find that the key training hyperparameters, such as learning rate and training epochs, as well the architecture characteristics such capacities and residual connections, are all highly correlated with whether and when the winning tickets can be identified. Based on our analysis, summarize guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis. Our codes are publicly available at: https::/ /tetta://qithhne hwtpgbithuuuu..e hwtp://cc///ooonnnnic...m http://///aaitttceeeeeee -
perform quantitative
Introduction
In recent years, the Lottery Ticket Hypothesis (LTH) has drawn great attention and thorough research efforts. As important study to investigate the initialization state and network topology of the deep neural networks (DNNs), LTH claims the existence a winning ticket (i.c., a properly pruned subnetwork together with original weight initialization) that can achieve competitive performance to the original dense network, which highlights great potential for efficient training and network design.
Unfortunately, among the various researches on the lottery ticket hypothesis 2 3 there are many inconsistencies regarding the settings of training recipe, and they further lead the controversies over the conditions for identifying winning tickets. We revisit and analyze the definition of the original lottery ticket hypothesis and find that the quality of training recipe is critical factor for the network performance, which in fact, is largely missing in previous discussions.
6
7 8).
In the standard LTH setup [.. key training hyperparameters such as learning rate and training epochs were not scrutinized nor exhaustively tuned. The winning ticket can be identified in the case of small learning rate, but can fail emerge higher initial learning rates especially in deeper networks. For instance, in [.. the winning tickets can be identified only in the case of small learning rate 0.01, with ResNet-20 and VGG-19 on CIFAR-10. 10. At larger learning rates, however, 9 reveals that the "winning ticket" has no accuracy advantage over the random reinitialization. which contradicts with
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the LTH definition. On the other hand, the settings in [ train 78 epochs for ResNet-20 on CIFAR- 10. Such insufficient training causes a relatively low pretraining accuracy. When pruned iteratively, the subnetwork accuracy can easily match that pretraining accuracy of the original network. Under such experimental conditions, the existence of the winning ticket is questionable.
In addition to all the problems caused by the experimental conditions, the huge computational consumption to find winning ticket becomes another research barrier and the practical main drawback, limiting the observations made on LTH. For instance, to reach around 90% overall sparsity ratio, iterative magnitude- based pruning (IMP) in [ requires totally 11 iterations (20% of the weights are pruned in each iteration). It adds up to 1,760 total training epochs if each iteration consumes 160 epochs. On the other hand. as an efficient pruning method, one-shot magnitude- based pruning (OMP) prunes a pretrained DNN model to arbitrary target sparsity ratio in one shot, which greatly saves training efforts. However, OMP rarely considered in the related literature, and often deemed as "weak" without full justification. Based on the above reasons, we feel we cannot confidently draw arguments, before we are able to evaluate L.TH comprehensively in regards of key factors such as different network structures, network dimensions, and training dataset sizes.
In this paper, we dive deeper into the underlying condition of the lottery ticket hypothesis. We raise the following questions: (1) What makes the comprehensive condition to define the lottery ticket hypothesis? (2) Do winning tickets exist across the major DNN architectures and/or applications under such definition? and (3) What are the intrinsic reasons for their existence or non -oneexisteece?
To answer the above questions, we present our rigorous definition of the lottery ticket hypothesis, which specifies settings of the training recipe, the principles for identifying winning tickets, and the rationality examining the winning ticket existence. Under this rigorous definition, we perform extensive experiments with many representative DNN models and datasets. The relationships between winning tickets and various factors are quantitatively analyzed. We empirically study the patterns through our analysis, and develop guideline to ease the process of obtaining the winning ticket. Our findings open up many new questions for future work. We summarize our contributions as follows:
I. We point out that the usage of inappropriately small learning rates, insufficient training epochs, and other inconsistent and implicit conditions for identifying winning ticket in the literature, are the main reasons that cause controversies in the lottery ticket studies.
II. We propose more rigorous definition of the winning ticket, and evaluate the proposed definition on different training recipe, DNN architecture, network dimension, and the training data size. Somehow surprisingly. we find that under the new rigorous definition, "rigorous" winning tickets are found by current methods, while there do exist winning tickets under a slightly looser definition.
III. We find that when residual connections exist in the network, using a relatively small learning rate is more likely to find (close to) winning tickets. When residual connection exists, the IMP method may not be necessary because OMP can achieve equivalent performance.
IV. We also find that when a smaller learning rate is not favorable, initialization likely to make no difference in finding the winning ticket (e.g., lottery initialization is not necessary). We quantitatively analyze the patterns, and present guideline to help identify winning tickets.
2 Re-defining Lottery Ticket Hypothesis
2.1 Notations and Preliminary
Detailed notations and functions are listed in T Based on Table we provide several key LTH-related settings along with descriptions.
In this paper, we follow the notations from 1 5)
Consider network function f(-) that is initialized as f(; 0o) where denotes input training samples. We define the following settings:
Pretraining: We train the network far 0o) for T epochs, arriving at weights Or and network function f2; 0r).
Pruning: Based the trained weights Gr, adopt OMP s) or IMP s) to generate pruning mask mo, my (0, 13101 Note that for IMP, the same 0g used in each iteration to ensure fairness to OMP.

---

Table 1: Summary of notations and functions.
Notation
Description
T
T the total numher of training epochs.
0o, 0z. 8%
0g ~ Do denotes initial weights used for training. 9e is the weights that trained from Pg for t epochs where t v T'. 0- ~ Dg denotes a random reinitialization that is different from Og.
A sparse mask E {0, |0 is obtained from certain pruning algorithm.
m
is the sparsity ratio, which is defined the percentage of pruned weights in the DNN model.
0SD
GDD denotes the weight in small-dense model that has the same number of non-zero parameters as pruned model, i.e. @SD ~ D mm|l
OMP s) IMP 8)
One-shot Magnitude- baagd Pruning [0] that prunes 0r and returns m, i.e. mo OOMP s).
It prunes & x 100% of weights in a one-time operation manner.
Iterative Magnitude- based Pruning 11 that prunes fr and returns m, i.c. my +- (0 8). IMP(.) prunes 20% of remaining weights per iteration until arriving at target sparsity 5
Lottery ticket with OMP (LT-OMP): We directly apply mask mo to initial weights do resulting in weights On O mo and network function [x; 0o @ mo).
Lottery ticket with IMP (LT-IMP): We apply my to initial weights G.. and get f(z; 0o @ my).
Random reinitialization with OMP (RR-OMP): We apply mask mo to the random reinitialized weights 8/ and get network function fx; 0g @ mo).
Random reinitialization with IMP (RR- IMP): We apply my to random reinitialized weights 8.. and get f(; 0 O m1).
Small-dense training (SDT): We construct a small-dense network that has the same depth and reduced width compared to the original network, and initialized by OSD i.e. [ax; (SD).
The original lottery ticket hypothesis ① claims that there exists subnetwork f(x; 0o @ m) in a randomly initialized dense network f(x; 0g). that once trained for I'' epochs (or fewer) will result in similar accuracy as f(x 0r), under non-trivial sparsity ratio. Additionally, the accuracy of f(x; (Og 3 m)r) should be noticeably higher than f(x; (% 3 m)r). Note that (og @ m)r and (8) O m)r are the initial and the randomly reinitialized weights of the sparse subnetwork trained for T epochs, respectively. When the above conditions are met, (0g O m) can be considered the Winning Ticket.
Original definition of the winning ticket:
We define network well-trained, if trained using sufficient training recipe (i.e., an appropriate learning rate and sufficient training epochs). However, in many prior works such as 1. the pretraining of the lottery ticket experiments used insufficient training recipe (i.e., inappropriately small learning rate and fewer training epochs), which leads to non- non-optimal pretraining accuracy at relatively low levels. Apparently, a higher pretraining accuracy more difficult for subnetwork to match or "win the ticket", even by using a sufficient training recipe.
We further revisit the LT- IM-MMP and RR- IMP experiments using ResNet-20 CIFAR-10 dataset, at three different learning rates over a range of different sparsity ratios (II uses the small learning rate 0.01). We train the subnetworks with the same training recipe in pretraining, and we also adopt the settings in ① to reproduce the results. Our preliminary results are shown in Figure
Through Figure ((a)
[(b) our first observation is that, under either training recipe, the "'winning ticket" exists in smaller learning rates (c.g., 0.005 and 0.01). but does not exist at relatively larger learning rate (c.g., 0.1). For instance, in the cases of the initial learning rate of 0.005 and 0.01, we find a noticeable accuracy gap between LT-IMP and RR-IMP using both training recipes, and the LT-IMP accuracy is close to the pretraining accuracy with reasonable sparsity ratio (e.g., 50% or above). This is similar to the observations found in 1 on the same network and dataset. On the other hand, in the case of the initial learning rate of 0.1, the LT-IMP has similar accuracy performance as the RR-IMP, and cannot achieve the accuracy close to the pretrained DNN with a reasonable sparsity ratio, thus no winning ticket condition is satisfied.
Through Figure [(c) our second observation that, at the same learning rate, the winning ticket defined in can be identified by using an insufficient training recipe, but fails to satisfy the winning
3

---

R1
Acoracy (76)
E3-347111
6165
6.0 kominng rat
6207
3.11
01
6006
6:0
lompgro
lorning p0
(a) sparsity ratio = 0.59
(b) sparsity ratio = 0.832
(c) sparsity ratio s 0.914
Figure I: Preliminary results of ResNet-20 on CIFAR-I0 dataset with different learning rates and sparsity ratios. We train the network using 160 epochs, while 1 uses 78 epochs. Please refer and Appendix A for the full results of all sparsity levels.
ticket condition when the network is well-trained. For instance, in the case initial learning rate of 0.005, [ uses approximately 78 epochs for training the network, which achieves 88.0% pretraining accuracy, 87.1 % on LT-IMP and 80.3% on RR-IMP, respectively. The LT-IMI accuracy close to the pretraining accuracy, and outperforms RR-IMP, thus it is claimed in [ that the winning ticket is found. However, when we train the network with sufficient number epochs (160 in our settings), the accuracy of pretraining. LT-IMP, and RR-IMP are 89.6%, 87.4%, and 82.9%, respectively. In this case, the accuracy gap between pretraining and IMP not small enough to claim that they are "similar", thus in fact winning ticket is found.
The above two observations indicate that the winning tickets are more likely to exist at small learning rate or at an insufficient training epochs, but may not exist at a relatively large learning rate sufficient training epochs (also observed in 99) However, we would like to point out that using a relatively large learning rate (e.g., 0.1) and sufficient training epochs (e.g., 160, which the standard settings on CIFAR- 10) result a notably higher accuracy for the pretrained DNN (92.3% vs. 88.0%). This point is largely missing in the previous discussions, and questions whether the previously identified "winning tickets" are meaningful enough.
Takeaway:
2.2 A Rigorous Definition of the Lottery Ticket Hypothesis
The above discussion reveals the inconsistency of identifying the winning ticket under different conditions. We provide more rigorous definition of lottery ticket hypothesis to reconcile the long-standing winning ticket identification discrepancy between experiment setting Our goal is to investigate the precise conditions when winning ticket exists and how to identify them.
The lottery ticket hypothesis rigorous definition. Under non-trivial sparsity ratio, there exists an identically initialized subnetwork that when trained in isolation with decent learning rate can reach similar accuracy with the well-trained original network using the same or fewer iterations, while showing clear advantage in accuracy compared to randomly reinitialized subnetwork well as an equivalently parameterized small- dense network.
The principles for the identification of the winning tickets. From our preliminary results in Figure we recognize that the pretraining of the randomly initialized dense network fæ; fu) with different initial learning rates achieves varying accuracy. Based on this observation and the rigorous definition of lottery ticket hypothesis, we list the conditions for identifying winning ticket follows:
① non-trivial sparsity ratio and a sufficient training epochs T are adopted for the subnetwork. SDT of (x; OSP) shows clear accuracy drop compared to the well-trained subnetwork.
There exists a learning rate such that the subnetwork f(x; (0g O m)r) achieves notably higher accuracy (with clear gap) than (a; (% m)r) trained with any learning rates.
There exists a learning rate such that the subnetwork f [x; (0g O m)r) achieves accuracy similar to or higher than the pretrained network f (x; 0r) at the same learning rate.
We also provide mathematical construct in Appendix C.

---

Jackpot
Secondary prize
Condition
A
Jackpot winning ticket
2
3
Secondary prize
dicket
Learning rate
Learning rule Suhnetwork fox: (fgom))) s.e. Small dense f(x:07)
Pretrain ((x:e,)
Subnetwork f(c (6,m)),)
(a)
(b)
(c)
Figure 2: An illustration of the principles for identification of the winning tickets.
@ There exists a learning rate such that the subnetwork f(; (m)) achieves accuracy similar to higher than the well-trained original network f2; 0r) (i.e., trained with an appropriate learning rate and sufficient number of training epochs).
Our listed conditions complete the long missing but necessary aspects for identifying the winning ticket. @ formally recognizes the practical significance of the winning tickets, that a found network topology of the winning ticket should benefit the training/infernce speed. commonly acknowl- edged that the overall sparsity ratio of the non-structured sparsity should exceed approximately 60% to deliver on-device acceleration. avoids a situation where the accuracy the winning ticket is comparable to that of small-dense network due to the weer-parameterization of network, which ensures the necessity of the winning ticket existence. different learning rates, which is missing in previous discussions. @ is the original condition for identifying winning ticket in previous works, but it does not consider the best pretraining accuracy at desirable learning rate. @ takes the desirable training recipe into consideration, which is different from existing works and becomes the most crucial condition in our definition. We define "similar accuracy" within 0.5% accuracy drop for CIFAR-10, 10, 1% for CIFAR-100 100 and Tiny-ImageNet, and 1.5% for ImageNet-16 K, and a "clear gap" between [(;; (0g ④ m)r) and [(;; (% @ m)r) (condition ( should be accuracy difference over 0.5%
takes into account of the influences by
We summarize the principles for identifying the winning tickets in Figure (a).
In the case that subnetwork (x; (fg m)r) satisfies the condition @ 3 as Figure (b) shows, we call (fg @ m) as Jackpot winning ticket, for has the potential to completely match the best performance of the original dense network.
On the other hand, the original "winning ticket" discussed in [ achieves the pretraining accuracy that is clearly lower than the best pretraining accuracy as Figure 2 (c). this case, condition @ are satisfied while the condition @ is not, and consider
secondary prize ticket.
We distinguish our definition of the lottery ticket hypothesis from the weight rewinding technique [5] 12. Lottery ticket hypothesis, on one hand, a study of initialization a neural network, while weight rewinding, on the other hand, studies the trade-off between accuracy and subnetwork searching cost. Despite the difference, we can generalize the weight rewinding technique into the winning ticket identification principle, which is shown in Appendix B. Detailed experimental evaluations of weight rewinding can also be found in Appendix D.
and network topology for
Sanity Checks for Lottery Tickets: Evaluation, Analysis and Guideline
Based the rigorous definition of the lottery ticket hypothesis, evaluate the lottery tickets with different types of network architectures, datasets with different sizes, and different learning rates. Detailed analysis demonstrated for deeper understanding of the lottery ticket hypothesis.
3.1 A Comprehensive Study Under the Rigorous Definition
Networks and datasets: this section, we evaluate the lottery ticket hypothesis with various
combinations of networks and datasets. We choose different network architectures among ResNet
series [14. VGG [15]. and MobileNet-vl [66. Specifically, the ResNet-32 is a wide version [17]
with width multiplier of 2. CIFAR-10/100 (18), Imay-Ima [19 and ImageNet-1K [20) all
evaluated. Table listss the details of the networks and datasets in the experiments we perform.
Our guantitative criteria for accuracy gaps no different from many previous efforts
5

---

Table 2: Dataset and network we evaluate using the re-definition of the lottery ticket hypothesis.
Dataset
CIFAR-I0
CIFAR-I00
Tiny-ImageNet
ImageNet- K
#lmages
50K/10K
S0K/10K
100K/10K
1.28M/50K
#Classes
10 32 x 32
100
200 64
1000
Img Size
224 x 224
Network
RN-20 0.27M
RN-32
MBNet-vl
RN-18 11.22M
VGG-16 14.72M
RN-18 11.68M
RN-50 25.56M
RN-18 11.69M
RN-50
#Params.
1.66M
3.21M
25,56M
Table Summary of the observations of all experiments.
Dataset
RN20
RN32
MBNet-vl
RN18
VGG-16
Dataset
RN18
RN50
CIFAR-10
<
Y
X
Tiny-ImageNet
x
CIFAR-100
<<C
x
x
ImageNet-IK
x
Jackpot
Secondary
Prefer small Ir
Prefer IMP
+ Yes *No AA boundary
Experimental setups: In this paper, we conduct our experiments using different learning rates. We empirically set the (initial) learning rate from extremely small to normal, then to very large based on the network and dataset. At each learning rate, we conduct a series of experiments described in Section 2.T] and each experiment is run three times. For IMP(-), we follow the settings in [ that 20% of the weights are pruned in each iteration. For OMP(-), We train the network for 160 epochs and
directly prune the network
to the same sparsity ratio as (.)( (.). On CIFAR-10/100 the learning rates decrease by a factor of 10 after 80 and 120 epochs. On ImageNet-1K, IK. We train the network for 90 epochs and cosine annealing learning rate schedule is used. We conduct our experiments NVIDIA A100 with 8 GPUs. Detailed experiment settings are listed in Appendix E.
We plot the accuracy vs. learning rate curves for all experiments we run, and demonstrate them in Figure 3 Due to the space limits, we put the full results for all other networks, datasets and sparsity ratios Appendix F.1. Based on the results, we summarize the observations answer the following questions with detailed analysis. For the following discussion, if not otherwise specified, we use LT to denote the setting of the subnetwork training with LT-IMP IMP or LT-OMP, and RR for RR-IMP RR-OMP.
Table and
Do Jackpot winning tickets exist in our evaluation?
We carefully examine all the results. Unfortunately, under the rigorous definition of the lottery ticket hypothesis and current ticket searching methods (IMP(.) and OMP(.)) no clear Jackpot winning tickets are found, and even tickets that merely reach the boundary of conditions rarely exist. According to the experiments and the preliminary results in Section 2.1 do notice accuracy improvement for both pretraining and subnetwork training with a sufficient training recipe. However, the accuracy gap between pretrained network and subnetwork is still non-negligible. For instance, consider the case using ResNet-20 on CIFAR-10 10 at = 0.914 in Figure 33 the Jackpot winning ticket is not identified, because the highest accuracy of the subnetwork by LT-IMP has a noticeable gap (> 0.5%) compared to the highest pretraining accuracy. Take VGG-16 on CIFAR-IO at s = 0.914 as another example, although the subnetwork achieves similar accuracy with pretraining, there is no accuracy gap (< 0.5%) between LT and RR, thus no tickets are found.
Recall the principles for identifying the winning ticket, all the cases verified at the best suited learn- ing rate, and please note that if there exists any non-trivial sparsity ratio (please check Appendix F. for results all sparsity ratios) that makes the subnetwork meet the conditions, winning ticket exist for this network. Under the rigorous definition, the odds for getting a Jackpot winning ticket is low, but we believe the Jackpot winning ticket is likely be existing with an appropriate size and trained using desirable learning rate (please check Appendix F.2 for more details). For instance, in Figure the case of MobileNet-vi on CIFAR-10 10 at s 0.832 reaches the boundary of Jackpot winning ticket conditions, as the accuracy gaps between LT and pretraining, and between RR and LT are both around 0.5%.
call the Jackpot
network
Do secondary prize tickets exist in our evaluation?
Yes. secondary prize tickets exist in most of the networks on small datasets. Note that the "winning tickets" found in previous works are (at most) similar to the secondary prize tickets based on our definition. Again, we use ResNet-20 at 0.914 as an example. In Figure secondary prize ticket

---

RodNet CLEAR 10 500022
KesNer 20 CIEAR 10 10540914
KosNer 20 CIPAR 50
12-
gap > 0.5%
Accimay (50
10
Accuney (10
gap > 0.5%
13:34
11:00
12
52
0.005
0.66
0.3
02
0.005
6.05
0.35
5,05
3.01
2.1
lcaming
lcarning
[earnnng
ReaNos32 CIFAR-10 5-0.322
ReoNet-32 CIFAR-10
ReaNes32 CIFAR-I0 5-0.945
Acoraty (50
Accuaay (%0)
Accuasy (56)
11:07
33:06
5,50
lesming
Teamiing
lcornin
MobilNNo-V VI CIFAR-IO $33
MobileNat-V1 CIEAR- 5-0.914
MobileNat-V1 CIFAB- 10 540.44
0.5%
0.5%
Accuetty (10
Accoracy (50)
ACCEBTY
007
44
CIPAR-I0
VOCI
gap 0.55%
Actenny (50
sap 0.55%
Accuacy 156)
11:01F
8
23
301
26
0.205
0.53
0.10
0.1
1.13
0.2
0.000
0.00
0.45
0.1
0.15
0.2
2,05
0.61
2155
9.1
0.15
0.2
leaming FARO
leamingg E002
Lennniing 2012
RcdNc-50 Imagenet
RoNNct-0
5-0.532
RoNNct50 Inngrre 5-0.914
15
Accinsy P9)
ACouacy (50
Accuray (50)
17:077
0.2
1.2
1.5
1.0
82
2.4
1.5
1.5
2.0
0.
0.4
1.0
1.6
2.5
lesming 7ote
leaming 1312
Leoming 13te
Figure 3: Lottery ticket experiments with different networks, datasets and (initial) learning rates. CIFAR-10 results are ordered by network size. ResNet-50 results on ImageNet-IK are also included.
exists in the green box, because the LT accuracy is similar with the pretraining accuracy at the same learning rate (0.005), while accuracy gap (> 0.5%) between LT and RR exists. However, the capacity of the network (in our cases, the number weights a network) determines the maximum sparsity which secondary prize ticket be found. For instance, relatively small network ResNet-20 can identify the secondary prize ticket at maximum sparsity ratio of 0.914 on CIFAR- 10, while larger networks such as ResNet-32, ResNet-19 18 and VGG-16 can identify secondary prize tickets on sparsity ratio of 0.945 or higher (refer to Appendix F.I). But on medium and large-scale dataset as -mageNet and ImageNet-IK, no clear secondary prize tickets are identified using ResNet- 18 or ResNet-50, We believe larger network may be able to identify one on ImageNet- 1
Which pruning method is better, IMP, OMP or it does not matter?
Comparing the results regarding network structures, we find that when residual connections exist in the network, TMP is more preferable than CMP, and when there are no residual connections the TMP has no advantages over OMP. To further investigate it with "apple-to-apple" comparison, we construct "ResNet-32-like" network, by removing all residual connections from ResNet-32 while
7

---

-2
25
1
(a) IME, 1=00.1,, 9cc-92.9% (b) IMP. 1=0.1. Acc=91.4%
(c) OMP, 1=0.001, Acc-91.8%
(d) OME, Ir0.11, Acc=91.66
Figure 4: Training trajectories along the loss surface contours of ResNet-32 on CIFAR-10 sparsity ratio of 0.945.
60
-50
(a) IMP, 1==0.01, Acc=89.4%
(b) IMP, 1=0.11 Acc=89.6% 6%
(c) OMP, 1r=0001, Acc=89.7%
(d) OMP, 1r Acc=90.7%
Figure 5: Training trajectories along the loss contours of ResNet-32-like network without residual connections CIFAR-10 at sparsity ratio of 0.945.
leaving all else intact. We then evaluate the accuracy of IMP and OMP ResNet-32, versus the newly constructed ResNet-32-like network. We also visualize both optimization trajectories along the contours of the loss surface, using the classical method in [2 22).
According to Figure 4 that the residual connections exist in ResNet-32, subnetwork using IMP explores much smoother route than using OMP as its contour is smoother and close-to-convex (a larger landscape area with mild variance, and larger basin in the middle of it (22). which indicates that the optimization route may be smooth towards local minima.
When there are residual connection Figure 3 shows, however, we do see much difference between IMP and OMP. Compare to the IMP method in Figure g the advantages of the IMP to CMP diminished. Note that the landscape will become much more rugged if residual connections are removed from a network (22). We conjecture that in our constructed no-residual ResNet-32, the optimization becomes too difficult and neither TMP nor OMP effective enough to explore smooth route towards local minima: hence much difference observed between them.
What learning rate is more likely to help identifying the winning tickets?
We notice that when residual connections exist, the subnetwork achieves higher accuracy at relatively small learning rate, while a larger learning rate is more preferable in training of subnetwork without residual connections. In Figure 4 as the residual connection makes the landscape become much smoother (22),
subnetwork trained with small initial learning rate 0.01 achieves larger contour and larger basin in the middle, while the contour and basin area with large learning rate 0.1 are relatively small. We conjecture that the optimization is much easier for smaller initial learning rate on a smooth loss surface, leading to better network performance. Without residual connections (as Figure 5p. the above observations ResNet-32 creates a more rugged landscape, thus a small initial learning rate 0.01 is more likely to stuck in sub-optimal local minima, while a large initial learning rate is unlikely to, therefore the SGD process is more likely to find desired path to high quality solutions.
exactly the opposite. Note that the no-residual
When does Og benefit subnetwork training?
We find that the secondary prize tickets are more likely be found relatively small learning rate. To analyze the reason, we use correlation indicator (0, 0') to quantify the number of overlapped indices of the top-p 100% large-magnitude weights between two different sets of weights. We say
the correlation between 0 and 0' is weak if R0, 0') R and when (0, 0') > p. the correlation is positive. The detailed definition and explanation of the correlation indicator shown Appendix G. We evaluate the correlations between (0p O m) and (0r C m)) and between (% 3 m) and (0r O m)

---

regarding different learning rates on ResNet-20 and VGG-16 as Figure 6 shows. When using relatively small learning rate, we find that the accuracy of f(x; (0, @ m)r) is closer to pretraining accuracy than fa(:;; (%) @ m)r) does. In this case, the correlation between (0g @ m) and (0r @ m) positive while (% @ m) and (0r o m) weak. When the correlation between (0g @ m) and (Or @ m) is positive, the weights that are large in magnitude in pretraining network are likely to also be large in a trained subnetwork, thus a relatively close accuracy is observed. When the correlation does not exist, using 0o or 0g in the subnetwork makes no difference to the final accuracy.
Does the size of the dataset affects the patterns for the winning tickets identification?
ReNNe.2 CIFAR-10
V0G-15 CIFAR-0
Lebey Miceet
Labory Scka
We find the patterns for the identified winning tick- ets are different on a relatively large-scale dataset, such Tiny- Ima-eemeeNet and ImageNet-IK. For all the ResNet architectures we evaluate, OMP outperforms IMP, and small learning rates are not preferable in training a subnetwork. We provide more discussion in Appendix F.3.
Juninn TOEK
12.5
2,9
Lonningttt
Lonntageee oe
Does weight rewinding improve the accuracy?
Figure 6: Correlation between weights in sub- network and pretrained network with different learning rates. The subnetwork we use has 0.832 and we set p = 0.1.
We find the weight rewinding technique [ consis-
tently improves the subnetwork accuracy. We gener- alize the weight rewinding technique into the winning ticket identification principles, and perform series of experiments. Due to space limits, the results are discussed in Appendix D.
3.2 How to Quickly Win Prize in Lottery Game - A Guideline
In this section, we summarize the patterns we find through the extensive experimental results, and present in the form of a guideline help quickly identify the Jackpot winning ticket and secondary prize ticket (both referred as ticket below for simplicity). Our guideline is presented as follows:
1. On a small dataset using networks with residual connections, IMP is better than OMP. When the network has no residual connections, IMP has no advantages over OMP.
2. On a small dataset using networks with residual connections, the subnetwork prefers a relatively small learning rate to find the tickets. When the network has no residual connections, small learning rate is not preferable.
3. When the network is redundant (e.g., a large network on a small-scale dataset), the maximum sparsity that a ticket can be found is relatively high, and vice versa.
4. When the (sub)network prefers large learning rates, using different initialization yields the similar accuracy in subnetwork training.
4 Ablation Study on Subnetwork Training with Different Learning Rates
In the lottery ticket hypothesis studies, it is standard setting to use the same learning rate in pretraining (for finding the mask by pruning thereafter) and subnetwork training (for training the sparse model) [ 5 12). In this paper, for each learning rate we have evaluated, the pretraining and subnetwork training also adopt the same learning rate setting. However. it does not consider the possibility that subnetwork may prefer different learning rate than it is used in pretraining. One key observation in (23] suggests that it is desirable to use different learning rates during pretraining and subnetwork training, and that doing so may lead to the well-performing lottery tickets.
According to our principle and @ for identifying winning tickets, any learning rate that satisfying the conditions would make successful Jackpot winning ticket or secondary prize ticket. Therefore, the rigorous definition of lottery ticket hypothesis and the principles for identifying winning tickets are valid (when consider different combinations of learning rates) and can hold true for future research. In Table we evaluate two series of experiments with two different pretraining learning rates using ResNet-20. We find that using different learning rates in pretraining and subnetwork training slightly

---

Table 4: Ablation results using ResNet-220 on CIFAR-I0 at sparsity 0.914. The shaded area indicates the learning rate that finds the better subnetwork accuracy.
Pretraining (Ace %): 0.01 (90.3)
Pretraining (Ace %): 0.1 (92.4)
LT
IMP Acc (%)
OMP Ace (%)
LT Ir
IMP Acc (%)
OMP Acc (%)
0.001
87.5
83.3
0.01
85.3
85.8
0.005
89.7
85.3
0.05
86.6
87.4
0.01
89.4
86.5
0.1
87.3
87.2
0.05
87.9
87.2
0.15
86.7
87.3
benefits the accuracy (e.g., 89.7% 89.4% IMP accuracy in the case of pretraining using learning rate of 0.01, or 87.4% vs. 87.2% OMP accuracy in the case of pretraining using learning rate of 0.1) but is not changing our previous observations. The results further strengthen our claim that the Jackpot winning ticket might exist in network when trained using desirable learning rate.
Related Works
Lottery Ticket Hypothesis. The lottery ticket hypothesis and the definition of the "winning ticket" firstly proposed in M. Concurrent work (9] finds that the identical initialized weights will not provide any advantage over training with randomly initialized weights at relatively large learning rates. Later works [9] (24 also confirm that the matching subnetworks at nontrivial sparsity are hard find in more challenging tasks. The following works 1 extend the subnetwork training from initial weights the weights early stage of pretraining (rewinding). and improve the accuracy more challenging tasks at nontrivial sparsity. Concretely. [12] makes key observation that subnetworks are stable SGD noise early stage of training. which explains why rewinding technique succeeds in LTH. In this paper, we recognize rewinding technique as successful approach to achieve dense network accuracy for the subnetworks, but our study focus on the effects and their rationales of different network characteristics and the experimental conditions in LTH.
Besides computer vision tasks, the lottery ticket hypothesis is also investigated in many other tasks (24 25
6, 7 26 27 28, 29] 30, 31). Other works [6 32) further extend the lottery ticket hypothesis a pre-trained BERT model. On object detection task, 333 proposes a guidance find task aask-specffic winning tickets for object detection, instance segmentation, and keypoint estimation. 35 have studied the lottery ticket hypothesis in unsupervised learning reveal how well the tickets are transformed between different datasets.
34
Find Winning Ticket at Early Stage of Training. The potential of training sparse network from initialization suggested by the lottery ticket hypothesis has motivated the study of deriving the "winning tickets" at an early stage of training, thereby accelerating training process. There is number of work in this direction. [4, 36) conduct retraining process after searching sub-network topology for few epochs. [37] examines the network state during early iterations of training, and analyzes the weight distribution and its reliance on the dataset. SNIP (38) finds the sparse mask based on the saliency score of each weight that is obtained after training the dense model for only few iterations. GraSP 39 prunes weights based on preserving the gradient flow in the network.
6 Conclusion and Discussion of Broader Impact
In this paper, we investigate the underlying condition and rationale behind the lottery ticket hypothesis. By revisiting the original definition, we find out that the current controversies over this topic is largely related to the quality of the training recipe. We propose a rigorous definition of the lottery ticket hypothesis, as well as the principles for identifying the true "Jackpot winning ticket" or "secondary prize ticket". We perform sanity checks for the lottery tickets through extensive experiments over mul- tiple deep models on different datasets, and empirically study the patterns we observe by quantitative analysis. Meanwhile, we develop guideline based on our summarized patterns, which potentially facilitates the research process on the topic the lottery ticket hypothesis. The research is scientific in nature and we do not envision it to generate any negative societal impact.
10

---

Acknowledgment
This work is partly supported by the National Science Foundation CCF-1919117 and 22532222 and Army Research Office/Army Research Laboratory via grant 1911NFF20-1-016 (YIP) to Northeastern University. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF or ARO.
References
[1] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations (ICLR), 2018.
[2] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference Computer Vision (ECCV), pages 67-82, 2018.
[3] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in Neural Information Processing Systems, 2020.
[4] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. International Conference on Learning Represenlations (ICLR), 2020.
[5] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. International Conference on Learning Representations (ICLR), 2020.
[6] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 2020.
[7] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. CVPR, 2021.
[8] Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data is more. In International Conference on Machine Learning, pages 12380-12390.
PMLR, 2021.
[9] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. International Conference on Learning Representations (ICLR), 2020.
[10] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
[11] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations (ICLR), 2016.
[12] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[13] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 1xXiv:1710.0187. 2017.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
[16] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. (Dt:::704.486/1 2017.
[17] Sergey ko and Nikos Komodakis. Wide residual networks. arXiv preprint
dxXiv:160..07146. 2016.

---

[18] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[19] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7,2015. 2015.
[20] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009.
IEEE Conference on, pages 248-255. IEEE, 2009.
[21] Eliana Lorch. Visualizing deep network training trajectories with pea. ICML Workshop on Visualization for Deep Learning, 2016.
[22] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 2018.
[23] Janice Lan, Rudy Chin, Alexei Baevski, and Ari S. Morcos. Uncovering the impact of hyperpa- rameters for global magnitude pruning, 2021.
[24] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv
preprint arXiv: 09574497.474 2019.
[25] Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data is more. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 12380-12390 PMLR, 18-24 Jul 2021.
[26] Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nip. arXiv preprint arXiv: 19t6:::900.02111 02760, 2019. [27] Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. [GAN]s can play lottery tickets
too. In International Conference on Learning Representations, 2021.
[28] Haoyu Ma, Tianlong Chen, Ting-Kuel Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2101.03223 2021.
[29] Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, and Jingjing
Liu. Playing lottery tickets with vision and language. arXiv preprint arXiv:2104.1 /832, 2021.
[30] Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. arXiv preprint arXiv:2102.067900 102.06790, 2021.
[31] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. citfa-data-efficiettt gan training: Drawing lottery ticket first, then training it toughly. arXiv preprint arXiv:2103.00397. 2021.
[32] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets winning. EMNLP, 2020.
[33] Sharath Girish, Shishira R Maiya, Kamal Gupta, Hao Chen, Larry Davis, and Abhinav Shrivas- tava. The lottery ticket hypothesis for object recognition. CVPR, 2021.
[34] Ari S Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. Advances in neural information processing systems, pages 0942-494,, 2019.
[35] Rahul Mehta. Sparse transfer learning via winning lottery tickets. Workshop on Learning Transferable Skills (NeurIPS), 2019.
[36] Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via carly-bird lottery tickets. arXiv preprint arXiv:2101.0000 2021.
[37] Jonathan Frankle, David J Schwab, and Ari S Morcos. The early phase of neural network training. International Conference on Learning Representations (ICLR), 2020.
[38] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR), 2019.
[39] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow, In International Conference on Learning Representations (ICLR), 2020.
12


Reviews:
[{"review_id": "uT1zRWIarTB", "reviewer": "Reviewer_cJrs", "summary": "This paper tries to reconcile the controversies around lottery ticket hypothesis. Their study reveals that small learning rate, insufficient training epochs etc. are the main reasons for these controversies. They define a rigorous version of LTH and show that under different training recipes, network architectures and train data size, no rigorous winning tickets are found by the current methods. Their study highlights that using small learning rate on DNNs with residual connections is more likely to find winning ticket and in conditions where small learning rate is not favourable then any special initialisation has no impact on chances of finding a winning ticket. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies a very important problem and tries to find answers to some of the inconsistencies around LTH. The findings are novel, very insightful and valuable to the LTH and pruning research. Finding a winning ticket is an expensive process and guidance provided by this paper on several hyper-parameters is quite useful. Most part of the paper is well written and has sufficient clarity. I feel the definition of rigorous LTH can be made more clear, possibly using some mathematical constructs. The experiments are done over different datasets( CIFAR-10/100, Tiny-Imagenet) and different architectures (ResNet, VGG, MBNet etc.), which makes their findings more credible. I think the claims provide important hints about LTH however, since the claims are only based on experimental results, so we need to be open to situations where some of their claims may not hold as well. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the definition of rigorous LTH can be made more clear. In addition to the empirical explanations, it would be great if these claims can be explained using theory. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "o5mQJJG8F1O", "reviewer": "Reviewer_Dhgf", "summary": "This paper goes deep into the investigation of the lottery ticket hypothesis, and found out that the training recipe that the original findings were based on had been insufficient. With a revised recipe and through extensive experiments, the author empirically studied the patterns of whether and when a \"jackpot ticket\" and \"secondary tickets\" can be identified. The major factor turns out to be the learning rate with which to train the subnetworks. The paper then ends with guidelines on how to find winning tickets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper challenges the established LTH, reproduced and improved the original training recipe, and comes up with a revised definition, and recommended guidelines on obtaining winning tickets based on an extensive set of empirical study. Overall I think such a sanity check endeavor is of great value to the community. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author claims that they found a better training recipe than used in original LTH. And Figure 1 is the only figure that showed such superiority. However it is on three hand-picked sparsity values. One plot I feel like is missing and could really drive this point home is one that plots accuracy vs. sparsity like in the original LTH and many follow-up works. So instead of picking three sparsity levels, the three subplots would be the three LRs, and each subplot would have x axis being the sparsity level or \"percentage of weight remaining\" like in all the other LTH papers.\n\nAnother limitation that invites questions is the ImageNet result. According to paper, \"Tiny-ImageNet and ImageNet-1K, no clear secondary prize tickets are identified using ResNet-18 or ResNet-50.\" That is not quite believable. One possibility is that the author adopted the original rewinding recipe used for CIFAR-10, where the weights are rewound to initial values. However a follow-up work on LTH on larger models [1] demonstrated that it is necessary to rewind to a later point during training for IMP to succeed. I suggest the author study this paper and many other papers on finding winning tickets large-scale models and datasets, which almost all proved the existence of winning tickets in ImageNet and ResNets.\n\n[1] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the\nlottery ticket hypothesis. arXiv preprint arXiv:1903.01611, 2019.\n\nErrors, modifications or improvements needed:\n - In 2.1 when defining a network function f (·), I don't think input x should be included, that is, it should just be f (\\theta) as opposed to f(x, \\theta). Both because the function shouldn't depend on input, but also throughout the section x is always the same and hence repetitive. \n - Figure 2 the difference between (b) and (c) is unclear. According to the definition, the highlighted difference between \"jackpot\" and \"secondary\" should be whether the original network is well trained (with appropriate LR and length of training) or not. The plots though seem to only highlight whether the crossing point is at a lower or higher LR. Therefore it doesn't seem consistent with the definition. \n\nMinor:\nEach figure and table should be self-sufficient, in that the caption should convey what the reader is supposed to get out of. Sometimes that requires repeating what's said in the main text but it should be very much encouraged. For example Figure 6 is confusing to me as what observation or conclusion we are supposed to draw.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "A9Iku5nBmwK", "reviewer": "Reviewer_277e", "summary": "This paper critically evaluates the lottery ticket hypothesis under different hyper parameter settings. They note that the learning rate used has a substantial impact on the quality of the unpruned model as well as the lottery ticket networks. They propose a stronger form of LTH by requiring the lottery ticket subnetworks to be able to match the best performing unpruned network. They find that using a larger learning rate results in a better unpruned model, but also results in an inability to find equally performant pruned networks.\n\nThey show that IMP is better than OMP only for residual networks, and residual networks prefer a smaller learning rate on small datasets, and when larger learning rates are preferred, random reinitialization performs no worse than lottery tickets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n- There are some novel observations, including a systematic look at the impact of learning rate on LTH, comparing iterative magnitude pruning (IMP) and one-shot pruning (OMP), as well as the investigation into the loss contours for residual connections. However, the overall originality of the contribution is limited. Some results already exist in prior work. For example:\n    - The definition of secondary prize tickets seem very similar to what is used in prior work, and observations regarding them have already been well established in those papers. The failure to train LT from initialization has also been studied in [2].\n    - It's well known that initial weights and final weights are correlated, and that LT train to the same basin as the original network, which makes the analysis regarding correlation indicator unsurprising.\n\nQuality:\n- The experimental settings in this paper are comprehensive, which I highly appreciate. \n\n- The stricter definition proposed in this paper requires LTs to be found at the best hyper parameter settings for the unpruned network. However, it doesn’t consider the possibility that training the subnetworks would require different optimal hyper parameters. In Lan et al [1], they show that it is optimal to use different learning rates during pre-training (for finding the mask) and during LT training (for training the sparse model), and that doing so leads to well-performing lottery tickets. Though the authors cannot be faulted for not citing non-conference papers, the observations in Lan et al suggest that different learning rates between pruned and unpruned models should also be considered in this work. Intuitively, if a jackpot lottery ticket satisfying the specified conditions can be found by using different learning rates, it should be considered as a successful lottery ticket. Without running those experiments, it raises doubt regarding the soundness of this paper’s main message.\n\n- In section “Which pruning method is better, IMP, OMP , or it does not matter?”, how are the IMP/OMP trained? Is there an unpruned model trained without residual connections, and new lottery ticket masks are derived from that model? If so, how does the loss contours compare for the unpruned models with or without residual connections? I wonder if the smoothing effect is inherent to ResNets, and IMP simply preserves it, or is it specific to sparse models? Do you see the same non-smooth behavior in other non-ResNet model architectures (both unpruned and pruned)? Do these other model architectures prefer larger learning rates (unlike ResNets)?\n\n- No justification is provided for the thresholds of 0.5%, 1%, and 1.5%, other than prior work - please cite the relevant prior work there. \n\n- Missing citations for Stabilizing the Lottery Ticket Hypothesis [2], as many similar experiments have been conducted there. Any differences in results should be noted.\n\nClarity:\n- The paper is overall well written. \n- Regarding secondary prize ticket, is it identical to the definition proposed in the original LTH? \n- In the section \"When does theta_0 benefit subnetwork training?\" Could the same conclusions be drawn by simply looking at the correlation of the weights between theta_0 and theta_T?\n\nSignificance:\n- I think the position of this paper is significant and an important contribution to lottery ticket work. I believe that the stricter form of lottery tickets is necessary in future work and can help remove inconsistencies in this research direction. However, this is contingent on resolving the issues I raised above, especially regarding the hyperparameter optimization performed for lottery ticket networks.\n\n[1] https://openreview.net/forum?id=XkI_ggnfLZ4\n\n[2] https://arxiv.org/pdf/1903.01611v3.pdf\n\n===================================================\nUpdate post rebuttal:\n\nI want to thank the authors for the very thorough response, which helped to address my original concerns. I hope the authors can incorporate many of these discussion points in the final manuscript. On the original reading it seemed like the conclusion was that no jackpot tickets can be found, but the rebuttal provides a more nuanced stance, which I hope can be better reflected in the final paper. I also would encourage the authors to provide a more thorough study of decoupled learning rates (as the authors already indicated that they will do), as it is an important factor in determining final performance and should really be explored in a paper that aims to provide general guidelines for LT training.\n\nAssuming these changes will be incorporated in the final paper, I will raise my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the impact of hyperparameters on lottery ticket performance and uses these analyses to provide guidance as to how to best select these hyperparameters. All 3 reviewers praised the paper for its comprehensive and compelling experiments as well as its clarity. There was some concern regarding the stringent definitions used in this work, but clarifications in the discussion were sufficient to resolve these concerns so long as the authors add additional discussion to the manuscript. Given the rigor of this work and its relevance to the lottery ticket community, I recommend this paper is accepted.
Decision: Accept (Poster)