Considering Layerwise Importance in the Lottery Ticket Hypothesis
Authors: Benjamin Vandersmissen
Jose Oramas
Venue: Submitted to ICLR 2023 (2023)
Keywords: Lottery Ticket Hypothesis

Abstract: The recently-introduced Lottery Ticket Hypothesis (LTH) posits that it is possible to extract a sparse trainable subnetwork from a dense network using iterative magnitude pruning.
By iteratively training the model, removing the connections with the lowest global weight magnitude and rewinding the remaining connections, sparse networks can be extracted that, when fully trained, reach a similar or better performance than their dense counterpart.

Intuitively, this approach of comparing connection weights globally removes a lot of context about the connection weights and their relations to other connections in their layer as the weight distributions in layers throughout the network often differ significantly.

In this paper we study a number of different approaches that try to recover some of this layer distributional context by computing an importance value for each connection that is dependent on the weights of the other connections in the same layer. We then generalise the LTH to use weight importances rather than weight magnitudes.

Experiments using these importance metrics on several architectures and datasets,  reveal interesting aspects on the structure and emergence of Lottery tickets. 

We find that given a repeatable training procedure, applying different importance metrics lead to distinct performant lottery tickets with little overlapping connections.

Under review as a conference paper at ICLR 2022
CONSIDERING LAYERWISE IMPORTANCE IN THE LOTTERY TICKET HYPOTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
The recently-introdueed Lottery Ticket Hypothesis (LTH) posits that it is possi- ble to extract a sparse trainable subnetwork from a dense network using iterative magnitude pruning. By iteratively training the model, removing the connections with the lowest global weight magnitude and rewinding the remaining connec- tions, sparse networks can be extracted. These sparse networks are referred to lottery tickets and when fully trained, they reach a similar or better performance than their dense counterpart. Intuitively, this approach of comparing connection weights globally removes a lot of context about the relations between connection weights in their layer, as the weight distributions in layers throughout the network often differ significantly. In this paper, we study a number of different approaches that aim at recovering some of this layer distributional context by computing a connection importance value that is dependent on the weights of the other connec- tions in the same layer. We then generalise the LTH to consider weight importance values rather than weight magnitudes.
Experiments using these importance metrics on several architectures and datasets, reveal interesting aspects on the structure and emergence of Lottery tickets. We find that given a repeatable training procedure, applying different importance met- ries leads to distinct performant lottery tickets with little overlapping connections which strongly suggests that lottery tickets are not unique.
INTRODUCTION
The recent trend in machine learning chase higher benchmark scores by adding additional param- eters, has led to an explosive increase in the size of neural network architectures. A prime example of this phenomenon are the GPT models. While the first model in the family (Radford et al.) 2018) has 117 million parameters, the latest model (Brown et al. 2020) already has a whopping 175 billion parameters which amounts poses new problems.
1000 times increase. However, this explosive rise in parameters
Training a single transformer model with parameter count of 213 millon still orders of magni- tude smaller than GPT-3 - using Neural Architecture Search emits as much CO2 as five cars during their lifetime (Strubell et al. 2020). Furthermore larger models typically need specialised hardware and a lot of computing power for training and inference, which constrain the ability of the model to run on mobile devices, thus limiting powerful models to well-funded institutions. Finally, research has shown that these large models are typically overparameterized and encode lot of redundant information that can be removed (Denil et al. 2013).
To alleviate these issues, numerous approaches have been studied to scale down the number of pa- rameters in a model, while still preserving (roughly) the same performance. This can be achieved by. c.g., designing parameter-efficienn network structures (Sandler et 201,2018), or sparsifying exist- ing neural network structures via pruning (le Cun 1990 Hassibi & Stork 1993 Han et al. 2015
Louizos et al. 2018 Molchanov et al. 2017).
Until recently, was thought to be difficult to train sparse neural networks from scratch (Evci et al. 2019), which was further strengthened by the finding that over parameterized network architectures are proven to lead to an optimal global minimum when training (Zou Gui 2019). As such the classical way reduce parameter count via the train-prune-fince finetune loop, in which model is

---

Under review as a conference paper at ICLR 2022
first trained to completion, then redundant connections are pruned and finally the resulting network is finetumed
The recently introduced lottery ticket hypothesis (LTH) (Frankle & Carbin 2019) challenged this notion and introduced a procedure to extract a sparse trainable network a lottery ticket (LT) from dense network. This is done using pruning criterion in the form of the global weight magnitude in combination with a gradual pruning procedure.
In this paper we study refinement on this criterion by adding a notion of layerwise importance, which we introduce in [section 2] We do this by considering number of different weight rescaling methods, such that the comparisons comparisons between different importance measures and the baseline are reported in /section 3 In addition, we shine light the observable differences in the generated LTs (scction 4), and determine how LT's emerge and differ when considering identical training conditions (section 5). A brief overview of related work laid out in |section 6land finally, the paper concluded section 7
more calibrated between layers. Quantitative and qualitative
The key observations of our study are that: i) given a fixed weight initialization. it is possible to ex- tract different lottery tickets that have similar performance, but differ significantly their structure, ii) these tickets have noticeable amount of common connections which have low-variance across tickets, and iii) these stable common connections survive the L.TH procedure even when the other weights in the model are reinitialized. Together these observations suggest that these connections might be a promising avenue towards finding LTs more efficiently.
2 THE LOTTERY TICKET HYPOTHESIS
The Lottery ticket hypothesis uses iterative Global Magnitude Pruning (GMP) (Han et al. 2015), which prunes individual connections that have the lowest weight magnitudes in a network. By repeating this process multiple times, it is possible to obtain a highly sparse network that when trained still reaches commensurate accuracy.
Later work by Frankle et al. (2020) introduced modification to the LTH procedure by rewinding to parameters at iteration = k < m, rather than resetting to the initial parameters at t = 0. By rewinding a later iteration, the performance of the found lottery tickets was improved for complex networks at high sparsities. In the literature, this procedure is usually referred to as Lottery Ticket Rewinding (LTR) rather than the LTH. We will adopt this naming in the rest of the document.
While the exact mechanism behind the success of lottery tickets not fully understood, one hy- pothesis. posited by Evei al. (2022). is that due to the LTH/LTR procedure the resulting networks are already in the same loss basin as the fully trained dense network and as such the ticket can still converge to a performant solution during training. Pseudocode for the LTR in its initial form can be found in Algorithm
Algorithm The LTR procedure
1: Initialize a model M with parameters d,
2: Pretrain M for iterations resulting parameters 0
3: for + O. do
4: Train M for m-k iterations, resulting in parameters 8mmm
5: Opooted t- (abs(((..)
6: p 4 j-th percentile of 0ponted
7: Prune all connections with abs(0.mm
8: Rewind parameters of M to 8
9: end for
weak aspect of globally pruning is that the only factor that determines whether a connection is pruned, is the magnitude of the connection weight. As such, connection weights from different layers are compared on global scale, rather than within the layer. This disregards more complex factors such as the weight distribution within a layer and the number of remaining connections in the layer. In fact, due to the commonly used Kaiming Normal initialization (He et al.) 2015), different layers are already initialized different weight distributions the standard deviation is inversely
2

---

Under review as a conference paper at ICLR 2022
Initialised it)
Pretrained (1000 it)
Fully Trained (78200 it)
Conv
[-0.47, 0.41]
[-0.57, 0.78]
[-1.07, 0.90]
Downsamplel
[-0.65, 0.66]
[-0.64, 0.63]
[-1.01, 1.11]
Conv17
[-0.11, 0.10]
[-0.12, 0.10]
[-0.15, 0.19]
FC
[-0.30, 0.27]
[-0.29, 0.39]
[-0.45, 1.19]
Table 1: Weight distributions [minimum and maximum] of selected layers in ResNet-18 network at different steps in the training procedure.
proportional the size of the layer, and as illustrated in Table I| they also converge to different distributions. However, it should be noted that this difference in standard deviation actually already introduces small implicit bias in the weights such that the weights are somewhat dependent on the size of the layer. Using importance metrics makes this bias both more explicit and stronger, dependent on which metric used.
Disregarding these previously-mentione factors might come at the cost of accuracy loss in the resulting ticket.
worst case this can even lead phenomenon called "layer-collapse" (Tanaka et al. 2020). critical failure case of neural network pruning in which information can no longer flow from the input layer to the output layer of the network. This finds its origin in one or more intermediate layers being completely destroyed, thus rendering the resulting predictions invalid.
Proposal. Taking these issues into account, we propose a modification the LTR procedure in the form of assigning importance value to each connection representing its importance within the layer. This importance score still dependent on the weight magnitude, but also considers the weight magnitudes of all other unpruned connections within the layer. In doing so, it injects an additional notion of the layer distribution the pruning process. Intuitively, this should alleviate the issue with layer-collapse and should improve the performance of the LTR procedure. Furthermore, it is possible to view this modification as generalization of the LTR procedure, where the purely magnitude-base
approach uses the identity mapping to calculate importance values. The only computational overhead incurred in our method is the calculation of importance scores, which negligible. A pseudo code overview of the modified procedure can be found in /Algorithm 2)
Algorithm The modified LTR procedure
1: Initialize a model M with parameters 0g
2: Pretrain M for iterations resulting parameters 0
3: for + 1, do
4: Train M for m-k iterations, resulting in parameters 8m.m
5: score; &
iiwpriiseiimotteeeeelaale (f
6: Scorepooted + Pool (score,)
7: + j-th percentile of Scorepanted
8: Prune all connections with abs cbs(scoree) < p
9: Rewind parameters to 0,,k
10: end for
In the same context layerwise importance calculation, Lee et al. (202T) introduced metric called the LAMP score, which follows model-level distortion minimization perspective by calculating the T.2 norm of the connections w.r.t. the other unpruned connections in the layer. The authors also demonstrated that this results in more performant lottery tickets very high sparsity levels. Intuitively, adding a layerwise importance score would allow the LTR procedure focus more on the layers with more connections and more redundancy, and such avoid removing connections from sparser layers, thus limiting the impact on model performance.
view the use of importance scores in the LTR procedure as an heuristic to calculate layerwise pruning ratios in the context of local pruning rather global pruning.
Additionally, also noticed in Lee et al. (2021)
3

---

Under review as a conference paper at ICLR 2022
2.1 CONSIDERED IMPORTANCE METRICS
To calculate a layerwise importance value for a given connection at position j in layer i, we need access the weight matrix W;,-
W
L normalization importance(i..))
W.::
W
2 normalization imporlanncet..))
(Lee al. 2021
EW?
exp(Wen)
Softmax imporlance((..)) Σ exp(WW..)
W - min (.))
Min-Max normalization :importncc((..)) =
max W min W.:)
The goal of the considered importance measures is to make weight values from different layers comparable. To accomplish this, there are two main approaches.
The first approach, as followed by L1, L22 and SoftMax is the sum-to-one principle, to rescale the importances of layer such that the total importance of layer is one. Then the importances themselves still determined by (an operation on) the magnitude, but rescaled.
The second approach, followed by Min-Max normalization determines the importances such that each layer has the same lower and upper boundaries.
Importance Distributions. In case of the softmax function, we notice that the importance dis- tribution is very narrow and the lower bound is exactly 1/[W], while for the other functions the distribution is more broad with a lower bound of O. This is due to a combination of two factors: (1) the Kaiming Normal initialization which has a mean of 0 and standard deviation that is inversely proportional to the size of the layer and (2) the exponential function for which ex R if |x]| < 1. Together this nullifies the dependence on the weights themselves almost completely in the softmax function when applied on large layers. L2 is another measure that transforms the relevance distri- bution to be more narrower than the weight distribution due to the quadratic function. The other measures, L1 and Min-Max, do not fundamentally impact the relevance distribution, only re-scale (and re-center) the distribution.
EXPERIMENTS
Our experiments, based the openlth 1t library (Frankle 2020). We follow the configurations
specified in Frankle et al. (2021) unless otherwise stated.
Networks. We adopt three different types of Neural Network architectures: Fully Connected Net-
works 30lelle--30000000 & Zisserman (2015) and Residual Networks He et al. (2016) (ResNet-18 & ResNet-20).
et al. (1998)) ), classical Convolutional Networks (VGG-16 Simonyan
Datasets. We adopt three different datasets commonly used the LTH/LTR literature: MNIST, CIFAR-10 and TinylmageNet. MNIST is used in conjunction with LeNet-300-100, CIFAR-10 used with both VGG-16 and ResNet-20, and TinyImageNet is used with ResNet-18.
For detailed description of the used training configuration, we refer the reader Appendix We consider the Lottery Ticket Hypothesis with weight rewinding (LTR) introduced in Frankle et al. (2020). The authors noted that inclusion of rewinding increased the performance the procedure on all types of models and datasets, but more specifically on complexer datasets and models.
3.1 QUANTITATIVE EVALUATION
The goal of the initial experiments to determine whether using different importance methods results in Lottery tickets with similar or better accuracy. To this end, caleulate the Top-1 accuracy results for Lottery Tickets at different sparsity levels. Results are listed in tabular form in [Table 2 (averaged over runs), in graphical form in Appendix B.T To enable comparison with other

---

Under review as a conference paper at ICLR 2022
Table 2: Average top-1 accuracy of different network dataset combinations over 3 runs. Boldfaced results highlight the best result for a given amount of pruning steps.
(a) LeNet-300-100 on MNIST
Steps (sparsity)
Magniuide
SoftMax
MinMax
Dense network
98.11% + 0.08
91% + 0.08
98.11% + 0.08
98.11%
0.08
98.11% = 0.08
5 (67.23%)
98.00% = 0.09
98.12%
0.10
98.04%
+ 0.13
98.00% = 0.08
98.02% = 0.09
(89.26%)
0.06
98.15%
0.05
15 (96.48%)
97.94%
1 0.08
98.12%
1. 0.15
97.95%:
1. 0.10
97.90%
0.05
97.77%
1 0.17
(98.85%)
97.23%
0.15
97.44%
97.47%
+ 0.10
0.17
97.41%
24 (99.57%%
90.62% + 0.29
90.89%
+ 0,22
91,05%
1,09
91.29%
0.79
91.82%
0.16
(b) ResNett-2
CIFAR-10
Steps (sporsity)
Magnitude
L1 91.67%
L2 91.67% + 0.40
SoftMax
MinMax
Dense network
91.67% + 0.40
+ 0:40
91.67% + 0.40
91.67% = 0.40
5 (67.27%%
92.01%
0.14
91.37%
0.20
91.65%
0.25
90.15% + 0.92
91.64%
10 26%)
90.90%
0.15
90.56%
0.22
90.86% + 0.39
87.75% + 0.33
90.51%
0.05
15 (96, 48%)
85.59% 1.00
84.76%
0.62
86.28% 1: 0.45
84.00% 1: 0.54
83.28%
0.32
20 85%)
77.07%
1.57
76.31% 1011%:1000
76.68% 0.63
76.37%
+ 1.06
74.36%
0.57
25 (99 62%)
63.366 + 0.55
63.40%
1: 0.77
62.78% 0.46
63.07%
1: 0.55
50.17%
= 6.78
88%)
45. 36.66%
2.96
1.49
43.65%
35 (99. 96%)
22.58% % 4.57
21.60% + 4.79
19.59 + 6.37
17.75% + 3.20
14.62%
4.00
40 (99 99%)
12.87% 4.97
10.00% 0.00
12.45% + 3.55
12.29% + 2.00
13.34%
2.89
(c) ResNet-18 on ImageegeNeet
Stops (sporsity)
Magnitude
L1
L2
SoftMax
MinMax
Dense network
49.53%
0.56
49.53%
0.56
49.53%
+ 0.56
49.53% + 0.56
49.53% + 0.56
5 (67.27%)
49.99%
= 0.10
50.96%
= 0.19
30.45%
+ 0.40
50.9%% = 0.18
50.38% = 0.47
10 (39 26%)
49.98%
0.41
50.84%
0.65
50.49%
+: 0.50
50.69%
+ 0.73
50.24%
0.68
15 (96, 48%,
48.72%
1. 0.20
49.76%
0.56
49.41%
1 0.09
45,M03
1 4.42
47.77%
1 0.37
20 (98 85%)
46.36% + 0.52
47.58%
1: 0.24
47.26%
0.52
40.48%
+ 3.52
43.49%
+ 0.17
25 62%)
38.54% 0.48
37.37%
0.35
37.00%
0.54
32.51%
+ 2.57
34.23%
1.19
30 (99 88%)
24.36%
+ 1.00
25.12%
0.71
25.88%
- 0.47
21.67%
+ 1.40
23.34%
+ 0.12
40 (99 99%)
4.62% 0.25
6.48%
= 0.30
6.83%
0.36
4.69% 0.72
5.31% + 0.62
(d) VGG16
CIFAR-10
Steps (sparsity)
Magnitude
L1
L2
SoltMax
MinMax
Dense network
93.32% = 0.08
93.52%
+ 0.08
93.32%
+ 0.08
93.52% = 0.08
93.32% + 0.08
(67.23%%
93.57%
1:
93.49% 0.03
10 (89. 26%)
93.75%
0.03
93.78%
0.25
93.57% 1.5%==00 0.06
64.99%
47.64
93.67%
+ 0.11
15 (96.48%)
93.69%
0.10
93.76%
1: 0.04
93.66%
0.18
64.8%%
47.36
93.47%
1: 0.14
20 85%)
93.53% + 0.10
93.24%
0.14
93.03%
105%%:00
63.72%
46.54
92.83%
+ 0.02
25 (99 62%)
91.8%% 0.14
90.94%
- 0.31
91.48%
0.30
60.66% = 43.87
91.81%
+ 0.09
35 (99 96%)
10.00% 0.00
40.99%
1.12
29.54% 1. 6.67
24.21% 17.48
14.38% 1: 7.59
studies, we adopted the commonly used pruning ratio of 20%. Additionally, we will make our code public at g tittubliinn-eeette to foster reproducibility of the reported results.
From these experiments, we can distill few key observations on which we elaborate the following sections. First, the magnitude, L2 and L2 criterions all produce LTs of similar performance, the only noticeable difference is in the VGG16 + CIFARIO experiment. Second, Min-Max normalization and SoftMax seem to keep up with magnitude pruning during the initial pruning phase, but suffers during the later iterations. This seem to be the trend except in the simplest scenario.
3.2 SIMILARITY BETWEEN L1, L2 AND MAGNITUDE
We notice that the top-1 accuracy of both magnitude, and the L1 and La normalization are pretty much identical at the earlier pruning ratios, contradicting our initial intuition. Our hypothesis is that this lack of difference indicates that the approaches finding lottery tickets are less strict than initially thought, as introducing the notion of importance does not seem to have large impact on the LTT procedure. We elaborate further on this more in the rest of the paper, where we conduct a deeper study on some differences between the found lottery tickets.
3.3 ISSUES WITH MIN-MAX
Due to the definition of Min-Max normalization, at least one connection in each layer will have an importance of O and at least connection in each layer has importance of 1. This will mean that
5

---

Under review as a conference paper at ICLR 2022
Prusing twokicns
Puining
Pietro
Pruking
Pluting
Moded Shrrmood
Mens Jetetoo
Mede strittd
Uaak Salosbord
magnilude
sofmmex
minmax
12 (Lee 2021)
Figure How the layer sparsities evolve during the LTH procedure with the ResNet- 18 model trained on TinylmageNet. Layers of interest annotated the X-axis.
during each pruning iteration, connections will be pruned from each layer. Eventually, this means that the the thinner layers will have too much capacity removed which will affect the final accuracy. This phenomenon can also be seen in the additional figures of of[Apeexii
3.4 ON LAYER-COLLARE
One of the possible symptoms of Layer-collapse is that the predictions of the model revert to random chance. Indeed, we notice in the later iterations the LTH procedure on VGG16 with CIFAR-10, that the top-1 accuracies of the magnitude criterion well run of the softmax criterion are cquivalent to random chance.
Additionally, Layer-collapse also occurs in the ResNet experiments, albeit less noticeable. More specifically this occurs with the SoftMax and Magnitude criterions, but due to the presence of skip connections the flow of information is still preserved when the main connection is fully pruned. As such, the resulting top-1 accuracies are not nearly as affected.
4 A CLOSER LOOK
In this section we take closer look at the generated LT's from the experiments. We focus our study on how they differ in structure and how they evolve throughout the pruning process.
Taking a look at how the sparsity of different network layers evolves at different intervals in the LTH procedure (see Figure for the plots from ResNet18 trained on TinylmageNet), we can clearly that during the initial pruning iterations there are large differences both between different layers and between importance measures. Even though these large differences exist between the tickets generated by different importance measures, established in the previous section that there is significant difference the top-1 accuracy of Magnitude, L, and L2. This is an indication that different kinds of structures exist in network, that when trained still achieve competitive

---

Under review as a conference paper at ICLR 2022
05
04
0.3
overiapoing
ye3
D.gee etai 20011
Ower apping
0000
P22G
0100
0505
3:000
0.155
8.159
0:375
D993
2002
Figure 2: Left: The distribution of standard deviations, Right: The number of sign flips for over- lapping and non-overlapping connections 96.48% sparsity for the selected importance measures.
accuracy. As such, we might conclude that there are multiple ways to find Lottery Tickets (on which expand in (section 5). later iterations, the difference between structures is still noticeable, but much less pronounced, which is expected as the number of pruned connections vastly outnumbers the number of unpruned connections.
Additionally, we can find an interesting pattern in the generated structures. We find that in each case, certain layers are pruned (significantly) less than most other layers the network. Specifically, these layers are often layers that intuitively are more important to the quality of the training procedure. In the case of ResNet- 18, these layers are both the input and output layer of the network, well the 1x1 convolutions used in the skip connections.
5 MULTIPLE LTS PER INITIALIZATION
In this experiment we determine whether a single initialization may lead to multiple Lottery Tickets. This requires making the LTR procedure deterministic and repeatable, i.e., all randomness in the training procedure is dependent on set random seed. This ensures that the only impact on the quality and structure of the found ticket the importance measure applied for the pruning step.
We consider the LTs found by the following importance measures: magnitude, I and L2. We limit ourselves to these measures, as they demonstrate a matching accuracy with the orginal dense network the Tiny ImageNet dataset at higher sparsity level (96.48% ) than the other measures. As such, we also do this experiment on the same configuration namely ResNet18 and Tinylmagenet.
We find that the resulting tickets share little amount of connections. Only 0.33% of the total (pruned and unpruned) connections are overlapping, while 9.34% of the unpruned connections are overlap- ping between different settings. We also find that some layers have noticeably larger fraction of overlapping connections, more specifically the first few convolutional layers as well as the linear classification layer. This can be attributed the intuition that these layers have less connections and as such are much less overparameterized.
Looking the overlapping connections in the first convolutional layer (Figure 3), which 68.78% of the remaining connections, compared to the non-overlapping connections, we can notice that the overlapping connections on average have larger difference between the initial and final weight value. Please see the appendix for an extended set of plots. We also study whether these overlapping connections converge the same weight in each pruning step up until the winning ticket is found. To do this, we have two simple metrics. For the first metric, we take the set of all weight values a single connection has at each pruning iteration and consider the standard deviation of that set as an indication of the robustness of that connection. We repeat this process for all connections in the ticket and find that on average the overlapping connections have lower standard deviation than the non-overlapping connections. In Figure Moreover, worth noting that these trends are consistent over other sparsity levels as well (see subsection B.2
present results for sparsity level of 96.48%.

---

Under review as a conference paper at ICLR 2022
The second metric also apply is counting the number of sign flips. Once again consider- ing the set of weight values a connection has during the pruning steps, we define a sign flip, the sign of least instance in the set is differing from the others. Here too, we find that the overlapping connections have less sign flips than the non-overlapping connections. These re- sults are presented in Figure with results for other sparsity levels reported in the appendix.
As final sanity check, we do partial reinitializa- tion test, where rather than starting the LTH/LTR procedure from the initial parameters Bu, we pre- serve the weights of fn that correspond to the over- lapping connections at the 96.84% sparsity level, while reinitializing the weights of the remaining connections using the same distribution (Kaiming Normal). We then restart the LTR procedure with the studied importance measures using this initial- ization with the same deterministic training proce- dure as the initial initialization. The goal is to de- termine whether these overlapping connections still (mostly) unpruned given a different initializa- tion, and determine as such whether the overlap- ping property is dependent on the total initializa- tion, or whether these connections are capable by themselves to steer an initialization in similar, per- formant lottery ticket.
Overlep
94
20 Pruning Iterations
Figure 3: The evolution of overlapping con- nections within the first convolutional layer
We find that on average, for three random runs, the amount of remaining connections is higher than the expected value, assuming that each connection has the same chance to be in the ticket. More specifically, for the initial few convolutional layers, we see increases over the expected amount ranging from +8.5% to +70.0%. In the fully connected layer, can even higher increases ranging from +440% to +480%. Finally, these effects were not nearly as noticeable in the x convolutions, with only -3.2% to +20.0%.
These results suggest that these connections are, by grace of their individual initializations, more likely to be part of the ticket. This also indicates that there likely a minimum amount of connec- tions in some layers that are necessary for performant ticket, with those mostly concentrated in the initial layers and the classification layer.
RELATED WORK
6.1 PRUNING NEURAL NETWORKS
Neural networks be trimmed of their fat by neural network pruning. These pruning methods can be grouped on a number of different axes iterative pruning vs single-shot pruning, local pruning vs global pruning, data-driven vs data-free and more. Here, we limit ourselves and give brief overview of different method categories that have some relation with respect to the analysis conducted in this paper. For an in-depth review, we refer the reader to Hoefler et al. 202T).
baaserion-based. This group constitutes the original line of research, which follows the train-prunce prune- finetune approach, where the network is first trained on the dataset, then pruned using a criterion and finally the pruned network is finetuned on the task to regain some accuracy. Pioneered by le Cun (1990); Hassibi Stork 1993), there has recently been a resurgence in popularity (Han al. 2015
Li ct al.
2017
Zhao et al. 2019 Dong et al. 2017 Yu et al. 2018).
Foresight Pruning. Inspired by the LTH, this line of research tries to prune a network at initial-
ization by using small subset of the training data to select promising connections, after which the sparse network will be fully trained. (Wang et al. 2020 Lee et al., 2019 Wang et al. 2019 Tanaka
et al. 2020) Recent work (Liu et al. 2022) asserted that these methods are functionally distinct to the LTH, because rather than generating sparse structures that depend on the initial weights, these methods learn efficient layerwise sparsity ratios.
8

---

Under review as a conference paper at ICLR 2022
Pruning and Regrowing. This line of research has been inspired by biological processes in which new pathways in the brain grow during the lifetime and involves methods that learn both weights and connections during the training process. (Guo al. 2016 Bellec et al. 2018 Mocanu et al. 2018 Dai et al. 2019)
Regularization--aase Uses regularization term the weights of network during training that will be minimized, such that a sparsity is automatically learned during the training process. (Huang
& Wang 2018 Alvarez & Salzmann 2016 Srinivas et al. 2017 Liu al. 2017)
6.2 THE LOTTERY TICKET HYPOTHESIS
Introduced by Frankle & Carbin 2019), the Lottery Ticket Hypothesis introduces a method to extract sparse trainable networks from dense network. Initially, this method was criticised due to the reliance on low learning rates (Liu et al., 2019) and the fact that it could not be reproduced in more complex settings (Gale et al., 2019). However, these concerns were mitigated with the introduction of weight rewinding (Frankle et al. 2020). These initial papers sparked a large body of follow-up work that we will briefly summarize in the next paragraphs.
Empirical and Theoretical properties. A first line of follow-up work was focussed on the prop- erties of the LTH. Zhou et al.) (2019) systematically explored different components of the LTH procedure. Renda et al. (2020) compares the impact of finetuning and rewinding and introduces the aspect of learning rate rewinding. Maene al. (202T) introduces a theoretical proof for the effec- tiveness of the LTH, while Evci et al. (2022) studied the gradient flow in the LTH procedure. Given the characteristics of our study, our work is situated in this category.
Transfering LTs. A second line of research involved transfering lottery tickets. This includes the transferability of lottery tickets between different datasets (Morcos et al. 2019 SSolen Sheppard 2019 Mchta 2019; Sabatelll et al. 2021 Desai et al.. 2021 Chen et al. [222aa) and even between different models (Chen et al.] 2021d).
Other domains and models. A third line of research studied the application of the LTH approach which was initially introduced for Convolutional Networks
on other types of models and
domains. (Chen et al. [202cc] Diffenderfer & Kailkhura 2021 Chen al. 2020: Kalibhat et al.
2021 Chen et al. 2021b)
Strong LTH. Finally, recently lot of effort has been poured in the so-called 'strong Lottery Ticket Hypothesis' (Ramanujan et al. (020) and proving its existence first in MLPs, then CNNs and even
in Equivariant Networks. (Chijiwa al. 2021 da Cunha et al. 2022 Malach et al., 2020 Orseau ct al. 2020 Pensia et al. 2020; Ferbach et al. 2022) The strong Lottery Ticket Hypothesis dictates that fully trained network can approximated by pruning sufficiently large randomly-initfalized
network, rather than extracting a sparse network from trained network as in the 'original' Lottery
Ticket Hypothesis formulation.
7 CONCLUSION
We studied modification the Lottery Ticket Hypothesis by introducing the notion of layerwise importance in the procedure. While we did not note a significant difference in performance when using certain importance measures, we emphasize that the LTH is invariant in a certain sense the layerwise pruning ratios, as long as the lowest magnitude weights are removed from each layer.
Furthermore, we noticed that given a fixed initialization, tickets that are dependent on the importance measure that is used. These Lottery Tickets all have roughly the same top-1 accuracy, but differ significantly in their structure. Looking at the remaining connections in the tickets, we noticed that there exist noticeable amount of overlapping connections between the tickets, namely mostly in the first convolutional layers and in the output layer. We observed that these overlapping connections consistently have lower variance between different lottery tickets, suggesting that these connections are somehow more stable and converge more often to roughly the same value. Finally, we have shown with a partial reinitialization test, that these connections even survive when the other weights are reinitialized. This suggests that we should focus attention on these specific connections to find LT's more efficiently.
possible extract different lottery

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jose M. Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. 2016.
Guillaume Bellec, David Kappel. Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers. Advances in neural information processing systems, 33:1877-1901, 200.
Tianlong Chen, Jonathan Frankle, Shiyu Chang,
Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834-15846 2020.
Tianlong Chen, Jonathan Frankle. Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 202 doi: 10. 10.110CCVVR44437..200 20.21.446.00.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, pp. 1695-1706. PMLR, 2021b.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In International Conference on Learning Representations, 2021c
Xiaohan Chen, Yu Cheng, Shuohang Wang. The Gan, Jingjing Liu, and Zhangyang Wang. The elastic lottery ticket hypothesis. Advances in Neural Information Processing Systems, 34, 2021d.
Daiki Chijiwa, Shin'y ya Yamaguchi, Yasutoshi Ida, Kenji Umakoshi, and Tomohiro Inoue. Pruning randomly initialized neural networks with iterative randomization. Advances in Neural Informa- fion Processing Systems, 34, 2021.
Arthur da Cunha, Emanuele Natale, and Laurent Viennot. Proving the strong lottery ticket hypothe- sis for convolutional neural networks. In International Conference on Learning Representations, 2022.
Xiaoliang Dai, Hongxu Yin, and Niraj K. Jha. Nest: A neural network synthesis tool based on grow-and-prune paradigm. IEEE Transactions on Computers, 68, 2019. ISSN 10.1109/tc.21192299 14448999999998..
doi:
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Predict- ing parameters in deep learning. Advances neural information processing systems, 26, 2013.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. In DeepLo@ EMNLP-IJCNLP 2019 - Proceedings of the 2nd Workshop on Deep Learn- ing Approaches for Low-Resource Natural Language Processing Proceedings, 2021. doi: 10.18653/11/d1-coo
James Diffenderfer and Bhavya Kailkhura. Multi-prize lottery ticket hypothesis: Finding accurate binary neural networks by pruning randomly weighted network. In International Conference on Learning Representations, 2021.
Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 2017- December, 2017. ISSN 10495258.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. In Workshop Deep Phenomena, 2019.
Utku Evci, Yann Dauphin, Yani Ioannou, and Cem Keskin. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the Thirty- Sixth AAAI Conference on Artificial Intelligence, pp. 3082-3101, 2022.
10

---

Under review as a conference paper at ICLR 2022
Damien Ferbach, Christos Tsirigotis, Gauthier Gidel, al. A general framework for proving the equivariant strong lottery ticket hypothesis. arXiv preprint arXiv:2206.042277 04270, 2022.
Jonathan Frankle. Openlth: A framework for lottery tickets and beyond, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learming Representations, ICLR 2019, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pp- 3259-3269 PMLR, 2020. ISBN 2640-3498.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural net- works initialization: Why are missing the mark? In International Conference on Learning Representations, 2021.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. 2016.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Advances Neural Information Processing Systems, volume 2015-January, 2015.
B. Hassibi and D. Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in Neural Information Processing Systems, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pailern recognition, pp- 770-778, 2016.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. jmir.org, 23: 1-124, 2021.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. volume 11220 LNCS, 2018. doi: (100779788--030-01270-000
Neha Mukund Kalibhat, Yogesh Balaji, and Soheil Feizi. Winning lottery tickets in deep generative models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8038- 8046, 2021.
Yan le Cun. Optimal brain damage. Advances in Neural Information Processing Systems, 2, 1990.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 (6(11):2278-2)) 2324, 1998. doi: 10.1109/5.726791.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In International Conference Learning Representations, 2021.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H.S. Torr. Snip: Single-shot network pruning based on connection sensitivity. 7th International Conference on Learning Representations, ICLR
2019.
Hao Li, Hanan Samet, Asim Kadav, Igor Durdanovic, and Hans Peter Graf. Pruning filters for efficient convnets. 5th International Conference on Learning Representations, ICLR 2017- Con- ference Track Proceedings, 2017.
Shiwei Liu, Tianlong Chen. Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. International Conference on Learning Representations, 2022.
11

---

Under review as a conference paper at ICLR 2022
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. volume 2017-October, 2017. doi: 10.1 N0.1109//CCV2201..2999
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference Learning Representations, ICLR 2019, 2019.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 10 regularization. In 6th International Conference on Learning Representations, ICLR 2018 Conference Track Proceedings, 2018.
Jaron Maene, Mingxiao Li, and Marie-Francinn Moens. Towards understanding iterative magnitude pruning: Why lottery tickets win, 2021.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 6682-6691. PMLR, 3 2020.
Rahul Mehta. Sparse transfer learning via winning lottery tickets, 2019.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature Communications, 9, 2018. ISSN 20411723, doi: 10.10388/441477-18--043333
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In 34th International Conference on Machine Learning, ICML 2017, volume 5, 2017.
Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers. In Advances Neural Information Processing Systems, volume 32, 2019.
Laurent Orseau, Marcus Hutter, and Omar Rivasplata. Logarithmic pruning is all you need. In Advances in Neural Information Processing Systems, volume 2020-December, 2020.
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Op- timal lottery tickets via subsetsum: Logarithmic mee-paraamttrizztionn is sufficient. In Advances in Neural Information Processing Systems, volume 2200Deeeeberr 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden randomly weighted neural network? In Proceedings of the IEEE Com- puter Suciety Conference Computer Vision and Pattern Recognition, 2020. doi: 10.1 109/ CVPR442600.2020.011199
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Matthia Sabatelli., Mike Kestemont., and Pierre Geurts. On the transferability of winning tickets in non-natural image datasets. In Proceedings of the 16th International Joint Conference on Com- puter Vision, Imaging and Computer Graphics Theory and Applications Volume 5: VISAPP, pp- 59-69. SciTePress, 2021. ISBN 978-989-758--888 doi: 10.5200001019630030000.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
12

---

Under review as a conference paper at ICLR 2022
Ryan Van Soelen and John W. Sheppard. Using winning lottery tickets in transfer learning for convolutional neural networks. In Proceedings of the International Joint Conference on Neural Networks, volume 2019-July, 2019. doi: 2LL.19/ICCNN.219.885222240
Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. pp. 138-145, 2017.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In AAAI, pp. 13693-136999 2020. URL https://aaai,
Hidenori Tanaka, Daniel Kunin, Daniel L.K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, volume 2020-December, 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. International Conference on Learming Representations, 2019.
Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, and Xiaolin Hu. Pruning from scratch. AAA/ 2020 34th AAAI Conference on Artificial Intelligence, 2020. ISSN 2159- 5399. doi:
16.1/a9aaaii 11.0///aa...34017.......... v34107.6910.
Ruichi Yu, Ang Li, Chun Fu Chen, Jui Hsin Lai, Vlad L. Morariu, Xintong Han, Mingfei Gao, Ching Yung Lin, and Larry S. Davis. Nisp: Pruning networks using neuron importance score propagation. Proceedings the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2018. ISSN 10636919. doi: 1091//9//
Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. Variational convolutional neural network pruning. Proceedings of the IEFE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June, 2019. ISSN 10636919, doi: 10.1109/ CVPR.2019.00299.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, volume 32, 2019.
Difan Zou and Quanquan Gu. An improved analysis of training over parameterizee deep neural networks. Advances in neural information processing systems, 32, 2019.
13

---

Under review as a conference paper at ICLR 2022
Supplementary Material
14

---

Under review as a conference paper at ICLR 2022
Supplementary Material
A NETWORKS, DATASETS AND TRAINING
Nerwerk
Donaser
Lpodts
Bacch
Ogc
Mam.
LX
LR Deop
Waight Decay
Inmilication
lte Up
Rewind herr
LeNet
MINIS
125
50.0
Nornal
469
0
ResNeed0
CHRRR-10
238
9000
0.9
10s
Karning Nornal
191
1030
SGD
Kanning
2000
ReiNect
TayimageNet
SGD
opechs 150
Kanning Nomnll
1020
B ADDITIONAL FIGURES
B.1 ACCURACY CURVES
The curves in Figure 4 illustrate the evolution of the top-1 accuracy of Lottery Tickets at differ- ent sparsities. These figures provide a more general overview than the tables the tables found in section 3 but are harder extract fine-grained details from.
0.80
0.8
0.98
Acaurae (too-1)
Acdurecy (tpp1)
0.6
0.04
Imporfance magnitude
044
Importance
0.92
mirmax
minmax
0.90
12 (Lee al 2021)
10
15
20
10
20 Pruning Iterations
15
Pruning ilerationa
(a) LeNet300-11 1000000000000000 MNIST
(b) VGG16 CIFAR10
0.5
Acaurasy (0o-1)
Accuracy (0p-1)
0.8
0.4
Importance magnitude
Importance magnitude
softmax
softmax
mimox
mimmax
2021)
2021)
10
25
40
10
25
30
Pruning Iterabons
Pruning itaration
(c) ResNet20 on CIFAR10
(d) ResNet18 on TinyImageNet
Figure 4: The accuracy curves for the different training configurations used in the main paper. The mean and standard deviation are given for 3 random seeds. The dashed line represents the mean accuracy of the dense network.
15

---

Under review as a conference paper at ICLR 2022
B.2 MULTIPLE LTs PER INITIALIZATION
owarlapping
yes
LU00 dt 2021)
1,40 0T 2021
magnoude
magretdd
overlapping
0.00
0.05
0.95
13:15
0.20
0.25
0.30
0.33
0440
2.00
005
D.10
0:15
0.20
0.25
0.30
033
0400
6000
600V
(a) 99.62% pruned
(b) 99.96% pruned
Figure 5: Additional distributions of standard deviations of both overlapping and non-overlapping connections for the (L1, L2, Magnitude] importance measures
0.30
035
025
0.30
025
f08
overlapping
10pe
0.20
overtapping
0.10
0.1b
0.10
0.10
nagnilude
12 (Lew 2021)
magnilude
12 (Lew 20211
importance
importance
(a) 99.62% pruned
(b) 99.96% pruned
Figure 6: Additional Sign flips of both overlapping and non-overlapping connections for the (L1, L2, Magnitude) importance measures
16

---

Under review as a conference paper at ICLR 2022
B.3
OVERLAP CURVES
Layer mode come sengge
Layer a modkk leyert condt Loww1.wrigf weight
Layer ti mode Jayer we ght
Layer ti mode layert, 1.tan weight
Layar - mode layar1 conu segtt
Layer - ricel byar? canc weight
Layar - mode Jayard D convt ow get
Layer mocel opert drwnoomple Diwe ght
Layar - mode Jaye2.11 convt wegtt
Layer e recelt lear2. Lond 1.o22. weight
Layer e mode Jagerd D. convt tww get
Layer a mode laye type11..ccom Coaan weight
Layar mocoel baprd deoocompll Bwaigt
Layar " model byach rood weight
Layer mode layad) candd ow
Layar mode byyant D wnigh
Layar Layes cond weggh
Layer mode layani B.coeraam Canggt
Layer = mode Jayant, Laanet owr
Layer made layank fanddd wright
Layer modd Joweight
Layor
Pritin
Figure 7: Additional curves measuring the proportion of overlapping connections between tickets
17

---

Under review as a conference paper at ICLR 2022
C LAYER SPARSITY
When studying the layerwise sparsities generated by the different pruning criteria (see Figure we can notice a number of interesting observations. Primarily, we notice that for each pruning criterion, both the input and output layer are pruned proportionally much less than other layers, though this wasn't explicitly encoded in the criterion. While the exact mechanism behind this isn' t fully understood and is out-of-scope, process and the fact that these layers are vital for the flow of information in the network. From all considered importance measures, the magnitude measure prunes the least of those layers.
theorize that this is due combination of the training
When considering ResNets, we can additionally notice that most of the other criterions (I1, L.2, SoftMax) also prune the Ixl convolutions less. These Ixl convolutions located in the bottleneck modules and often contain less redundancy than the other 3x3 convolutions.
Pruning ratios of different layers.
Pruning ratios of different layers.
1.000
0.195
0.8
Pruring ro0e
0.900
Pruning rabe
0.6
meg
0.800
magnitude
softmax
0.925
[Lae et 2021)
02-
softmas
baodine
12 (Lee et al. 2020)
dhertcuz22
baseline
Layers
Leyers
(a) Resnet20 on CIFAR-10 10
(b) VGG-16 on CIFAR-10
100
098
Spardyy
038
Importance mag
0.5
softmax
12 fLae al. 2021)
model conv)
moddttayeez downseempeee.
meoded Layer downsamplee
modat downsentio
Layers
(c) ResNet-18 18 on TinylmageNet
Figure 8: The layerwise pruning ratio of different layers in a LT compared to the total pruning ratio.
The x-axis is (roughly) corresponding model depth and layers of interest are indicated.
18

---

Under review as a conference paper at ICLR 2022
Laya spacity of macelcrriiinooo
Lavenbbe spantly racel laye dldooarrril
Layoralia spariity d'madellayeri servi
10/R POSM
(a) convO (input)
(b) Downsample
(c) mid conv
Laye city moocen Lay stanpia soiige
Lapinaite City loyer roee moaet
12 suartyy &raige
Seadiry
(d) Downsample 3
(e) last conv
( fo (outp)
Figure 10: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the ResNet18 network.
D LAYER SPARSITY IN TIME
Losarnie spantily of leyar conuweigle
ya la spunity CERANICR
Layerdia querty Hl orrighh
2871)
Spodle
(a) convO (input)
(b) conv8
(c) conv
Figure 9: The evolution of layer sparsity during the LT extracting process, visualised in a number of selected layers of the VGG16 network.
In Appendix C we showed that some layers are disproportionally pruned less by different pruning criteria. In this section, will take look at how the sparsity of these layers evolves throughout the pruning process and compare them to 'normal' layers in both VGG16 (see Figure 9h and ResNet- 18 (see Figure TOP.
can be clearly seen that generally the deeper the layer is within the model, the faster this layer is pruned. This has the effect that often the higher layers are 'ahead' of the global sparsity, while the lower layers are 'behind' of the global sparsity. The main exceptions here being the 1xl convolu- tional layers, however we can still notice that the later Ixl convolutions are pruned faster than the 1x1 convolutions.
Additionally, we can notice some differences in behaviour of the different pruning criteria, namely that softmax has spiky evolution, meaning that at some timesteps, no connections are pruned in layer, while at other timesteps large amounts of connections are pruned. The other criteria follow a much smoother approach, where at least every timestep some connections are removed from each layer, with Min-Max often following the global sparsity best.
19


Reviews:
[{"review_id": "reqL0uzWd5O", "reviewer": "Reviewer_NGvs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to modify the LTH procedure by using a score that measures the layerwise importance in order to select the weights to be pruned. The authors propose a few possible scores and evaluate their performance on a set of experiments. The experiments show that in most of the cases the proposed method does not improve with respect to the standard LTH that uses magnitude in order to select the weights to be pruned. The authors also perform a comparison of lottery tickets obtained with the different scores showing that hey have a noticeable amount of overlapping connections and such connections seems to be more stable.", "summary_of_the_review": "Even though the paper is well written and it provides a solid experimental evaluation of the proposed method, I believe that the novel theoretical contribution is very limited and the proposed method does not exhibit a significant gain with respect to the state of the art. ", "strength_and_weaknesses": "The paper is well written and easy to follow. However, I think that the novel contribution is limited (a similar idea of layerwise importance of the weights is already present in Tanaka et al. 2020). In addition, the results of the experiments do not show a significant gain using the proposed method and I haven't found the comparison of different tickets particularly insightful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "dxCLJf4pcA", "reviewer": "Reviewer_GSCn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new Lottery Tickets procedure by introducing a layerwise importance score to find the lottery tickets. Additionally, this work demonstrates several intriguing observations: 1) from a fixed initialization, there are different lottery tickets that significantly differ in their structure. 2) these tickets have some common connections, which survive the LTH procedure even when the other weights are reinitialized.", "summary_of_the_review": "None", "strength_and_weaknesses": "### Strength:\n\n1. This paper is well-written and easy to follow.\n2. Both quantitative and qualitative results are reported.\n3. Experiments are conducted across different datasets and architectures.\n\n### Weakness:\n\n1. Only small-scale datasets are considered in the experiments.\n2. I'm a little bit confused about the main contribution of this work. Using a layerwise importance score can improve the performance of identified lottery tickets or some observations that suggest lottery tickets are not unique. With different mini-batch sequences during training, we may also obtain significantly different lottery tickets from the same initialization.\n3. Using layerwise importance score, can the LTH procedure identifies the winning ticket with higher sparsity?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DYc3El4cCh", "reviewer": "Reviewer_8dwk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the popular lottery ticket hypothesis (LTH) framework, which discovers sparse, trainable subnetworks via iterative magnitude pruning (IMP). Typically, IMP is performing in a global manner, in which weight magnitudes are compared with all other weights within a network when determining which weights should be pruned/discarded. Alternatively, the authors argue that comparing weights on a per-layer basis is more appropriate, as importances/magnitudes of weights oftentimes differ on a layer-to-layer basis throughout the network. Instead, they propose an alternative procedure that evaluates the importance of weights within each layer. In particular, this method (1) adopts the same weight magnitude and (2) normalizes this magnitude in a per-layer fashion using a few different techniques. This technique is evaluated for 4 different types of normalization on computer vision experiments. The authors then go on to show that different layer-based pruning techniques can be used to find winning tickets with different structures. ", "summary_of_the_review": "To begin, I want to emphasize that my current review reflects my initial impression of the paper. I am fully open to discussion with authors/other reviewers, and my final score will be mostly determined by this discussion.\n\nI believe the problem the authors study and the proposed approach are interesting. However, I think the paper--in its current form--falls short of providing enough valuable insight to the problem of improving upon global IMP for LTH. Currently, the main weaknesses of this paper in my opinion are:\n* Lack of large-scale experiments. This is very necessary for LTH as mentioned above. \n* Lack of novelty in layer-wise importance schemes. The work could be greatly improved by exploring more schemes (possibly inspired by work in the pruning literature mentioned above).\n* Improvements over normal LTH procedures are not consistent. I am not convinced the layer-wise importance approach truly provides a clear benefit.\n\nNonetheless, the paper is well-written and motivated, and I strongly encourage the authors to continue working in solving the problems I outline above. I especially enjoy the analysis of lottery ticket overlap at the end of the paper. I believe by furthering extending this analysis, thinking more deeply about other layer-wise techniques that could be used/compared-to, and expanding the empirical scope to more domains (e.g., ImageNet, transformer tasks, or other larger-scale cases) the authors will greatly improve the paper and provide a valuable contribution to the community. ", "strength_and_weaknesses": "Strength:\n* The LTH problem considered is popular and the focus on an intuitive/simple issue with typical approaches to LTH (i.e., global pruning leads and layer collapse). I believe this is an interesting direction that the authors motivate well. \n* I believe the observation that many different structures of lottery tickets can be discovered within a network has already been made (see e.g., https://arxiv.org/abs/1912.05671), but the authors do provide some extra analysis that goes beyond previous work that is interesting/useful. I really enjoy this extra analysis, and I think the authors could emphasize this more!\n\nWeakness:\n* Because the paper is purely empirical, I believe that large-scale experiments on ImageNet are very necessary in this case. This is because, for the LTH, it is widely recognized that many observations made on LTH for smaller-scale experiments do not hold at scale (see https://arxiv.org/abs/1810.05270 and https://arxiv.org/abs/1902.09574). Thus, in this work, performing the larger-scale experiments (e.g., ImageNet) is necessary. \n* Accuracy improvements of the proposed procedure are small and there is not a single procedure that performs best. The proposed methods are outperformed by magnitude pruning (i.e., the normal LTH procedure) in many of the considered cases. \n* Several pruning works consider pruning in a layer-normalized fashion (see https://proceedings.mlsys.org/paper/2020/hash/d2ddea18f00665ce8623e36bd4e3c7c5-Abstract.html and references, just search “layer wise vs. global pruning” in the paper). None of these techniques are compared against within the analysis. I believe comparing against a few of the SOTA layer-based pruning techniques would be useful. \n\nSmall Stuff:\n* Figure 1 is difficult to parse. It might be better to use a different subplot for each layer, then plot the sparsity as a function of the pruning iteration. \n* L1/L2 normalization of per-layer importance are basically identical (even in the results). It might be better to just use one or the other and state that they perform very similarly. \n* In the introduction, your posing of LTH makes it seem like it solves the problem of the train-prune-finetune loop. It might be good to state that LTH still requires this loop (i.e., the network needs to be pre-trained to discover the winning ticket still). \n* “Using importance metrics makes this bias both more explicit and stronger, dependent on which metric is used.”: I am not quite sure what this means here or why this would be the case. It might be nice to explain a bit more or maybe provide a citation. \n* “The only computational overhead incurred in our method is the calculation of importance scores, which is negligible.”: Whether this computational cost is negligible depends on the importance score you choose/implement, right?\n\nQuestions:\n* Is there any motivation for the importance metrics that you consider, or are you simply trying a bunch of different approaches and trying to determine the best strategy empirically? Are there other approaches beyond these defined importances that could be useful?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject