The Elastic Lottery Ticket Hypothesis
Authors: Xiaohan Chen
Yu Cheng
Shuohang Wang
Zhe Gan
Jingjing Liu
Zhangyang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Lottery Ticket Hypothesis

Abstract: Lottery Ticket Hypothesis (LTH) raises keen attention to identifying sparse trainable subnetworks, or winning tickets, which can be trained in isolation to achieve similar or even better performance compared to the full models. Despite many efforts being made, the most effective method to identify such winning tickets is still Iterative Magnitude-based Pruning (IMP), which is computationally expensive and has to be run thoroughly for every different network. A natural question that comes in is: can we “transform” the winning ticket found in one network to another with a different architecture, yielding a winning ticket for the latter at the beginning, without re-doing the expensive IMP? Answering this question is not only practically relevant for efficient “once-for-all” winning ticket ﬁnding, but also theoretically appealing for uncovering inherently scalable sparse patterns in networks. We conduct extensive experiments on CIFAR-10 and ImageNet, and propose a variety of strategies to tweak the winning tickets found from different networks of the same model family (e.g., ResNets). Based on these results, we articulate the Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or dropping) and re-ordering layers for one network, its corresponding winning ticket could be stretched (or squeezed) into a subnetwork for another deeper (or shallower) network from the same family, whose performance is nearly the same competitive as the latter’s winning ticket directly found by IMP. We have also extensively compared E-LTH with pruning-at-initialization and dynamic sparse training methods, as well as discussed the generalizability of E-LTH to different model families, layer types, and across datasets. Code is available at https://github.com/VITA-Group/ElasticLTH.

The Elastic Lottery Ticket Hypothesis
Xiaohan Chen'* I+ Yu Cheng²
Shuohang Wang Zhangyang Wang'
Zhe Gan
Jingjing Liu
University of Texas at Austin
Microsoot Corporation TTiinguua University (xiaohan. chen, atlaswang lyu.cheng, shuowa, zhe chee.ann @mi crosoft. com JJLiuiudair. tsinghua. edu. on
edu
Abstract
Lottery Ticket Hypothesis (LTH) raises keen attention to identifying sparse train- able subnetworks, winning tickets, which can be trained in isolation to achieve similar or even better performance compared to the full models. Despite many efforts being made, the most effective method to identify such winning tickets still Iterative Magnitude- brsssidude-bedddddd Pruning (IMP), which is computationally expensive and has to be run thoroughly for every different network. A natural question that comes in is: can we "Iransform" " the winning ticket found in one network to another with a different architecture, yielding a winning licket for the latter the beginning, without re-doing the expensive IMP? Answering this question is not only practically relevant for efficient "once-for-all" winning ticket finding, but also theoretically ap- pealing for uncovering inherently scalable sparse patterns in networks. We conduct extensive experiments on CIFAR-10 and ImageNet, and propose a variety of strate- gies to tweak the winning tickets found from different networks of the same model family (e.g., ResNets). Based on these results, we articulate the Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or dropping) and re-ordering layers for one network, its corresponding winning ticket could be stretched (or squeezed) into subnetwork for another deeper (or shallower) network from the same family, whose performance is nearly the same competitive as the latter's 's win- ning ticket directly found by IMP. We have also extensively compared E-LTH with pruning-ur pruning-at-initialee at-initialization and dynamic sparse training methods, as well as discussed the generalizability of E-LTH different model families, layer types, and across datasets. Code is available at wilhttp::// ptstps://get //ggthutttt////////////e com/VITA GrouppElasttie
Introduction
Lottery Ticket Hypothesis (LTH) [13 suggests the existence of sparse subnetworks in over- parameterized neural networks at their random initialization, early training stage, or pre-trained initialization 35 non-zero parameters compared with the original dense networks, but can achieve similar or even better performance when trained in isolation. The discovery undermines the necessity of over- parameterized initialization for successful training and good generalization of neural networks [46] 31]. That implics the new possibility train highly compact subnetwork instead of prohibitively large without compromising performance, potentially drastically reducing the training cost.
3 2
Such subnetworks, usually called winning tickets, contain much fewer
However, the current success of LTH essentially depends on Iterative Magnitude-based Pruning (IMP), which requires repeated eycles of training networks from scratch, pruming and resetting the remaining parameters. IMP makes it extremely expensive and sometimes unstable to find winning
Work was done when the author interned Microsoft.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

ResNet 32 - 56 (CIFAR-10) 10)
ResNet 56 - 32 (CIFAR-10)
92
Test Test Acouracy (56
90
90.0
88
86
87.5
85.0
0.0
36.0
59.0
73.8
83.2
89.3
93.1
95.6 97.2
0.0
36.0
59.0
73.8
29..8833.889988888888 93.1 95.6 97.2
VGG 13 19 (CIFAR-10)
VGG 19 13 (CIFAR-10)
Unpruned
IMP
Test Accuracy (%)
Reinit
93
Random
Magnitude
92
SNIP
92
GraSP
91
ETTs
0.0 36.059.0 2.8.88328893333 95.6
36.0 73.8 2.3.289.399. 93.1 95.6 97.2
Sparsity (%)
Sparsity (%)
Figure 1: Validating Elastic Lottery Ticket Hypothesis (E-LTH) between two pairs of networks (ResNet-32 and ResNet-56, VGG-13 and VGG-19) trained on the CIFAR-IO dataset. We transform the winning tickets in the source models at different levels of sparsity using the proposed Elastic Ticket Transformations (ETTs), and compare with other pruning methods (please refer to Section 4 for those methods' details), including the state-of- the art pruning-ar setg--t-iiiiiin at-initialization methods (e.g.. SNIP and GraSP). Substantial accuracy gaps can be observed between ETTs and other pruning methods, corroborating that ETTs provide high-quality tickets the target models that are comparable with the winning tickets found by the expensive IMP. All results are based on three independent runs.
tickets at scale, with large models and large datasets 15 To alleviate this drawback, many efforts have been devoted to finding more efficient alternatives to IMP that can identify sparse trainable
subnetworks at random initialization, with little-to-no training (7 40 38 16]. Unfortunately. they all see some performance gap when compared the winning tickets found by IMP, with often different structural patterns [16. Hence, IMP remains to be the de facto scheme for lottery ticket finding.
Prior work 33 found that winning ticket of one dense network can generalize across datasets and optimizers, beyond the original training setting where it was identified. Their work provided a new perspective of reducing IMP cost to only find generic, dataset-independent winning ticket for each backbone model, then transferring and re-training it on various datasets and downstream tasks. Compared to this relevant prior work which studies the transferablity of winning ticket in the same network architecture, in this paper, we ask an even bolder question:
Can we transfer the winning ticket found for one network to other different network architectures?
This question not only has strong practical relevance but also arises theoretical curiosity. On the practicality side, if its answer is affirmative, then we will perform only expensive IMP for one network and then automatically derive winning tickets for others. It would point to tantalizing possibility of once-for-all lottery ticket finding, and the extraordinary cost of IMP on a "source architecture" is amortized by transferring find winning tickets by IMP on a small source architecture and then transfer it to a much bigger target architecture, leading to drastic savings compared to directly performing IMP on the latter. Another use case is to compress a larger winning ticket directly to smaller ones, in order to fit in the resource budgets on different platforms. On the methodology side, this new form of transfcrability would undoubtedly shed new lights on the possible mechanisms underlying the LTH phenomena by providing another perspective for understanding lottery tickets through their transferablity, and identify shared and transferable patterns that make sparse networks trainable [17] 38 [4]. Moreover, many deep networks have regular building blocks and repetitive structures, lending to various model-
range of "target architectures". promising application is to first
based interpretations such as dynamical systems or unrolled estimation [19 empirical methods seem to remind of those explanations too.
42
20
Our explored
1.1 Our Contributions
We take the first step to explore how to transfer winning tickets across different architectures. The goal itself would be too daunting if no constraint is imposed on the architecture differences. Just like

---

general transfer learning, it is natural to hypothesize that two architectures must share some similarity so that their winning tickets may transfer: we stick to this assumption this preliminary study.
We focus our initial agenda on network architectures from the same design family (e.g., a series of models via repeating or expanding certain building blocks) but of different depths. For a winning ticket found on one network, we propose various strategies to "stretch" it into a winner ticket for a deeper network of the same family, or "squeeze" it for a shallower network. We then compare their performance with tickets directly found on those deeper or shallower networks. We conduct extensive experiments on CIFAR-10 with models from the ResNet and VGG families, and further extend to ImageNet. Our results seem to suggest an affirmative answer to our question in this specific setting,
We formalize our observations by articulating the Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or dropping) and re-ordering layers for one network, its corresponding winning ticket could be stretched (or squeezed) into subnetwork for another deeper (or shallower) network from the same family, whose performance is nearly the same as the latter's winning ticket directly found by IMP. Those stretched or squeezed winning tickets largely outperform the sparse subnetworks atiinitializationn approaches (27), and show competitive efficiency to state-of-the-ar
found by dynamic sparse training M. We also provide intuitive explanations for the preliminary success.
Lastly, we stress that our method has a pilot-study nature, and is not assumption-fred The assumption that different architectures come from one design "family" might look restrictive. However, point out many thate-of-the-art deep models are designed in "families", such as ResNets [22],
MobileNets (23), EfficientNets (37,, and Transformers [39]. Hence, we see practicality in the current results, and we are also ready to extend them to more general notions- - which we discuss in Section 5
2
Related Work
2.1 Lottery Ticket Hypothesis Basics
The pioncering work [13] pointed out the existence of winning tickets at random initialization, and showed that these winning tickets can be found by IMP. Denote f(; 0) deep network parameterized by 0 and x as its input. A sub-network of f can be characterized by a binary mask m, which has exactly the same dimension 8. When applying the mask m to the network, obtain the sub-network fx;; 9 @ m), where @ is the Hadamard product operator.
For a network initialized with fo. the IMP algorithm with rewinding [I5] 14 works as follows: (1) initialize m as an all-one mask; (2) train [(x Op @ m) for r steps to get 0,: (3) continue to
fully train [(;;; 0, @ m) to obtain a well-trained 0; (3) remove a small portion p E (0, 1) of the remaining weights with the smallest magnitudes from 0 ( m and update m; (5) repeat (3)-(4) until certain sparsity ratio is achieved. Note that when
O, the above algorithm reduces to IMP without rewinding [13. Rewinding is found to be essential for successfully and stably identifying winning tickets in large networks 055 [4]. 44] identified Early-Bird Tickets that contain structured sparsity, which emerge at the early stage of the training process. B3 3 studied the transferability of winning tickets between datasets; the former focuses on showing one winning ticket to generalize across datasets and optimizers: and the latter investigates LTH in large pre-trained NLP models, and demonstrates the winning ticket transfcrability across downstream tasks
2.2 Pruning in the Early Training, and Dynamic Sparse Training
parallel line of works study the pruning of networks either the initialization or the early stage of training 66, so that the resulting subnetworks can achieve close performance to the dense model when fully trained. SNIP 277 proposed to prune weights that are the least salient for the loss in the one-shot manner. GraSP 40 further exploited the second-order information of the loss at initialization and
prune the weights that affect gradient flows least. (38] unified these methods under a newly proposed invariant measure called synaptic saliency, and showed that pruming iteratively instead of in one-shot is essential for avoiding layer collapse. The authors then propose an iterative pruning method called SynFlow based on synaptic saliency that requires no access to data.
However, it is observed in [6] that vanilla magnitude- based pruning 21 is as competitive as the above carefully designed pruning methods in most cases, yet all are inferior to IMP by clear margins. methods such as SNIP and GraSP were suggested to identify no more than layer-wise pruning ratio configuration. That was because the SNIP/GraSP masks were found to be insensitive to mask shuffling within each layer, while LTT masks were impacted lot by the same. Those suggest that existing carly- firli-pruning methods are not yet ready to replace IMP.
The
3

---

Elastic Ticket Transformation
Target Decper Network
Source Ticket
Block
Block 2
Block
Block
Stretching
Stage
Squeezing
Winning Ticket
Residual Block Conv BN Cony HN
Target Shallower Network
Weight:
Mask:
Figure 2: An illustration of Elastic Lottery Ticket Hypothesis, using ResNet-type network as an example, which includes five residual blocks in each stage. Note that we hide "Block 0" who downsamples the input features as we will always preserve this block, without replicating or dropping. On the left, we use IMP to find winning tickets of the source network. On the right, can either stretch the source ticket to a deeper network, or squeeze to a shallower network, using the proposed Elastic Ticket Transformation (ETT). In both cases, have number of strategies for mindfully replicating, dropping and re-ordering the blocks.
Instead of searching for static sparse subnetworks, another line of influential works, called dynamic sparse training (DST), focuses on training sparse subnetworks from scratch while dynamically changing the connectivity patterns. DST was first proposed in [32]. Following works improve DST by parameter redistribution [34] [9] and gradient-based methods [T]. A recent work [30] suggested that successful DST needed explore the training of possible connections sufficiently.
2.3 Network Growing
Broadly related this paper also includes the research on network growing. Net2Net 4 provided growing operations that preserve the functional equivalence for widening and deepening the network. Network Morphism [4] further extended Net2Net to more flexible operations that change the network architecture but maintains its functional representation. Network Morphism is used in [10] to generate several neighbouring networks to select the best one based on some training and evaluation. However, this requires comparing multiple candidate networks simultaneously. A more recent work, FireFly 43
proposed a joint optimization framework that grows the network during training.
Despite our similar goal of transferring across architectures, the above methods do not immediately fit into our case of winning tickets. We face the extra key hurdle that we need transfer (stretch or squeeze) not only general weights, but also the sparse mask structure while preserving approximately the same sparsity. For example, Net2Net and FireFly add (near) identity layers as the initialization for the extra layers, which are already highly sparse and incompatible with IMP without modification. Network Morphism decomposes one layer into two by alternating optimization, which will break the sparse structure of the source winning tickets.
3 Elastic Lottery Ticket Hypothesis
The overall framework of E-LTH is illustrated in Figure At the core of E-LTH are a number of Elastic Ticket Trunsformations (ETTs) proposed by to convert the sparse architecture of winning ticket to its deeper or shallower variants. We present rules of thumb for better performance, as well as discussions and intuitive explanation for why they do/do not work. For the simplicity of illustration, we will use the ResNet family as examples this section.
Suppose we have identified a winning ticket f= m²) on a source network using IMP, characterized by the rewinding weights 8; and the binary mask m,, where r the rewinding steps and s stands for "source". Our goal is to transform the ticket f into a target network Ist (g(0r, m²), h(,,, m²)) with a different depth directly, avoiding running the expensive IMP on the target network again. Here, the superscript stands for "target". g(.,)) and h((,-) are the transformation mapping for rewinding weights and the mask, respectively, taking the source weights and source mask as inputs.
Terminology:

---

3.1 Stretching into Deeper Tickets
We first present ETTs for stretching winning ticket, which select certain layers to replicate, both in the (rewinding) weights and the sparse mask. Below, we discuss the major design choices.
Minimal unit for replication. Intuitively, "layer" is the most natural choice for the minimal unit in neural networks. Normalization layers (e.g., Batch Normalization layers (24)) are widely used in modern networks and play essential roles in the successful training of the networks. Therefore, in this paper, ETTs consider a linear or convolutional layer and the normalization layer associated with it whole "layer". We find this to work well for VGG networks [36].
However, in ResNet [22 or its variants, which consist of multiple residual blocks, we consider the residual block as the minimal unit, because they are the minimal repeating structure. Moreover, the residual blocks, which are composed of two or three convolutional layers with normalization layers and shortcut path, can be interpreted from different perspectives. For example, one residual block is interpreted as one time step for a forward Euler discretization in dynamical systems [42), or as one iteration that refines latent representation estimation [19]. More discussions are in Section 3.3
Invariant components. To ensure generalizability, we prefer minimal modification to the architec- tures involved. Hence, we will keep several components of the networks unaffected during stretching:
The number of stages, which are delimited by the occurrences of down-sampling blocks. Adding new stages change the dimensions of intermediate features and create new layers with totally different dimensions, which are hard to be transferred from source tickets.
The down-sampling blocks. In ResNet networks, the first block of each stage is down- sampling block. In ETTs, we directly transplant the hown-ssmpling blocks to the target network without modification and keep the one-to- me-to-one relationship with the stages.
The input and output layers. Thanks to the first invariance, the input and output layers have consistent dimensions in the source and target networks and thus can be directly reused.
Which units to replicate? Let us take ResNet-20 (source) and ResNet-32 (target) as example. Each stage of 2e contains one down-sampling block, Bi and two normal blocks, Bi and Bi. while ResNet-32 has four normal blocks for each stage. To stretch winning ticket of ResNet-20 to ResNet-32, we need to add two residual blocks in each stage. Then, should we replicate both Bi and B5 once, or only replicate Bi (or B5) twice? If the latter, which one?
For the first question, ETTs we choose to replicate more unique source blocks and for fewer times, i.e. we prefer to replicate both Bf and B once in the above example. As will be explained in Section 3.3] this could be understood as a uniform interpolation dynamical system, a natural way to go finer-resolution along time. Another practical motivation for the choice is that such strategy can better preserve the sparsity ratio of each stage and thus the overall network sparsity. If B; and BA have different sparsity ratios (we observe that usually later blocks are sparser), replicating only one of them for several times will either increase or decrease the overall sparsity; in comparison, replicating both Bí and B proportionally maintains the overall sparsity.
the discretization steps in the
If we compare replicating either Bi or B5, the former will lead to better performance due resultant lower sparsity (see above explained), and the latter causing lower performance due to effective higher sparsity. Both are understandable and could be viewed as trade-off options.
How to order the replicated units? Following the ResNet-20 to ResNet-32 example above, we replicate Bi and B5 once for each of them, resulting in two possible ordering of the replicated blocks: (1) B + Bi + B + Bi + B5. which we call appending as the replicated blocks are appended as whole after the last replicated block; (2) B5 - Bi - Bi - B + BA, which we call interpolation as each replicated block is inserted right after its source block. We conduct experiments using both ordering strategies and observe comparable performance.
3.2 Squeezing into Shallower Tickets
Squeezing the winning tickets in a source network into a shallower network is the reverse process of stretching, and now we need to decide which units to drop. Therefore, have symmetric design choices as ticket stretching, besides following the same minimal unit and invariances.
To squeeze winning ticket from ResNet-32 into ResNet-20, we need to drop two blocks in each stage. The first question is: should we drop consecutive blocks, e.g., Bf. B3 or Bg, Bi. which corresponds to the inverse process of the appending ordering, drop non-consecutive blocks, e.g..
5

---

Bi, Bi or B2, Bi, which corresponds to the inverse process of the interpolation ordering above? The second question is: in either case, should we drop the earlier or the later blocks?
According to the ablation study in Section 4.1 ETTs are not sensitive to whether we drop blocks consecutively or intervals; however, it is critical that we do not drop too many early blocks.
3.3 Rationale and Preliminary Hypotheses
We draw two perspectives that may intuitively explain the effectiveness of ETTs during stretching and squeezing. Note that both explanations are fairly restricted to the ResNet family, while E-LTH seems to generalize well beyond ResNets. Hence they are only our preliminary hypotheses. and further theoretical understandings of ETTs will be future work.
Dynamical systems perspective: ResNets have been interpreted as discretization of dynamical systems 42 20 1] Each residual block in the network can be seen as one step of a forward Euler
discretization, with an implicit time step size of an initial value ordinary differential equation (ODE). Under this interpretation, adding a residual block right after each source block, that copies the weights and batch normalization parameters from the source block, can be seen as a uniform interpolation of this forward Euler discretization, by doubling the number of time steps while halving the implicit step size. Under the same unified view, if we replicate only Bf (or Bi), that could be seen as performing non-uniform interpolation, that super-solver only one time interval but not others. Without pre- assuming which time step is more critical, the uniform interpolation is the plausible choice, hence providing another possible understanding of our design choice in Section 3.1
Unrolled estimation perspective: [19] interprets ResNets unrolled iterative estimation process: each stage has a latent representation for which the first block in this stage generates a rough estimation and the remaining blocks keep refining it. This perspective provides a strong motivation to keep the first block of each stage untouched, as it contributes to the important initial estimate for the latent representation. Replicating or dropping the remaining residual blocks will incrementally affect the latent representation estimation. The interpolation method for stretching then corresponds to running every refining step for multiple times and the appending method corresponds to re-running (part of) the refining process again for better estimation. It is also implied by [19] that since each residual block is incremental, removing blocks only has mild effect on the final representation. providing intuition for the effectiveness of squeezing tickets by dropping blocks.
4 Numerical Evaluations
We conduct extensive experiments on CIFAR-10 10 (26 and then ImageNet 77. transferring the winning tickets across multiple models from ResNet family and VGG family. The implementation and experiment details are presented in Appendix. We run all experiments three times independently with different seeds. Besides IMP and the unpruned dense models, we will also compare ETTs with state-of-the pruning-at-ine iniitaallllii methods, including SNIP (27, GraSP [40] and One-shot Magnitude- based pruning which was suggested by [16] as a strong baseline. We also include two common pruning baselines in LTH works: (1) Reinitialization (Reinit), which preserves the identified sparse mask but reinitialize the rewinding weights; and (2) Random Pruning, which keeps the rewinding weights yet permuting the masks, only preserving the layerwise sparsity ratios.
Throughout this paper. we use "sparsity" or "sparsity ratio" to denote the portion of zero elements in networks due to pruning. Thus, the higher the sparsity ratio. the more parameters pruned. Note that ETTs may cause (slightly) misaligned sparsity ratios with IMP and other pruning methods because replicating and dropping blocks may result in the sparsity changes in a less controllable way. For a fair comparison, all pruning methods will match the sparsity ratios of the winning tickets generated by IMP, to the best possible extent. We apply pruning methods to the rewinding weights instead of initialization because the former was found to substantially improve LTH performance [16]
4.1 A Thorough Ablation Study on ResNet-32 and Resnet-56
In this subsection, we conduct an ablation study about the rules of thumb for constructing better ETTs discussed in Section 3 We investigate the selection of replicated squeezed units when apply ETTs to stretch or squeeze a winning ticket, and the order of replicated units for the stretching case. All results reported are the average of three independent runs of experiments.
Do we really find winning tickets? To show that the new network generated by ETTs is a winning ticket, we compare the performance of ETTs with the following baselines when transforming ResNet-

---

ResNet 56 (CIFAR-10)
ResNet 32 - 56 (CIFAR-10)
94
Test TestAccurscy (50
Test TestAucuray (%5
62
80-
91
59.0
73.8
83.2 Sparsity (%)
89.3
93.1
0.0
36.0
59.0
73.8
89.3 93.1 95.6 97.2 Sparsity (%)
83.2
Random Rain?
ETTs
Random Mask
Random Reinit Mask
Combine SNIP Mask
IMP
Uppruned
Interpolation 1234
Appencing 1254
Interpoiation
Interpoation 1234
-Random
Appending
Appending
Figure 3: Ablation study to show that ETTs do find winning tickets on the target networks com- pared to other baselines.
Figure 4: Ablation study on replicating unique units on CIFAR-10, using ETTs to transform tickets in ResNet-32 to ResNet-56.
32 winning tickets into ResNet-56: (1) we transform the mask of the source winning ticket but perform weight reinitialization under that mask; (2) we preserve the weight initialization of the source network but select random mask; and (3) we use random weights with random mask on the target network. The results are shown in Figure 3 where we can see ETTs are among the best at all non-trivial sparsity levels (2 50%) and consistently better than other baselines at high sparsity ratios (> 80%). We also include another baseline which copies the masks from the source winning ticket for existing blocks but uses SNIP masks for the extra blocks ResNet-56. The superiority of ETTs over this bascline shows that only using the original winning tickets without ETTs is not good enough, corroborating the necessity of ETTs from another perspective.
Should we replicate more unique units? ResNet-32 has five residual blocks in each stage, notated as [Bo.... B.). As we discussed Section 3] we leave the downsampling block Bg alone and only play with the rest four. We consider three options, i.e., replicating (1) all four blocks; (2) the first two blocks: (3) and the first block (B1). We run for each option both the appending and the interpolation methods (note that appending interpolating the first block yields the same resulting model).
The results are shown in Figure where the numbers in the legend are the indices of replicated blocks. We can see that replicating four blocks is slightly better than replicating two for both appending and interpolation and much better than only replicating the first by clear gap. The gaps enlarge when the sparsity ratios grow higher. At lower sparsity ratios, the differences between those options are smaller. We also include an option, "Interpolation 1234- Random", in which we randomly permute the masks of the replicated blocks (but preserves the masks of the original blocks). That shows better than its random- partoonm-rnniggggoo variant all sparsity ratios and the advantage more evident at high sparsity ratios.
The carlier or the later units? The next question in ETTs that follows is: should we replicate earlier units in stretching for better performance, or the later ones? And similarly, should we drop earlier units when squeezing the tickets? Is one option always better than the other? For the stretching part, we try to replicate (B1, B2). (Be, Ba), (B3, B,) using appending and interpolation methods. Results in the top subfigure of Figure 5 show the advantage of replicating earlier blocks than the later ones with high sparsity, while the differences are less obvious in the low sparsity range. For squeezing, we also try different options of dropping blocks as shown in the bottom subfigure of Figure 5 and find that ETTs are not sensitive to dropping consecutive or non-consecutive blocks, which consistent to our observation on the stretching experiments, as long as we do not drop earlier blocks too much we can see a significant decrease in performance when we drop the first four blocks the same time.
4.2 More Experiments on CIFAR-10
Experiments on more ResNets. In addition to the transformation between ResNet-32 and ResNet- 56, we apply ETTs that use appending operations to more ResNet networks to transfer the winning tickets across different structures. More complete results are presented in Figure 6 We can see that the tickets generated by ETT' from different source networks clearly outperform other pruning methods, with the only exception of transferring then smallest ResNet- to large target networks.
Another (perhaps not so surprising) observation we can draw from Figure is that ETTs usually work better when the source and target networks have a smaller difference in depth. For example, when ResNet-44 is the target network, ETT tickets transformed from ResNet-32/56 outperform
7

---

ResNet 32 + 56 (CIFAR-10)
ResNet 56 + 32 (CIFAR-10)
94
Test Test Accuracy (%)
92
92
90
90
88
88
0.0
36.0
59.0 73.8 83.2 89.3 93.1 95.6 97.2
36.0 59.0 73.8 83.2 89.3 93.1 95.6 97.2
Sparsity (%)
Sparsity (%)
Unpruned
Interpolation 23
Appending
Unprumed
Drop 5678
Drop 2468
Interpolation
Appending
Drop
Drop 1357
Interpolation
Appending
Figure 5: Results of the ablation study on replicating/droppingg earlier or later units of the networks. Left: stretching the winning tickets of ResNet-32 into ResNet-56 by appending/nnterplating different residual blocks. Right: squeezing the tickets of ResNet-55 into ResNet-32 by dropping four blocks at different positions. Results are the average of three independent runs with error bars.
Reshit 14 (Target, CIFAR-10)
ReaNet 20 (Tangst, CIFAR-10)
Unprured
IMP
Reinit
ETTs
Random
ETTs 20
Magnitude
ETTs
84
SNIP
ETTs 44
GraSP
ETTs
Reshie CIFAR.10)
ReaNet CIFAR.10)
ResNet CIFAR11))
02
54
88
250
738
832 2
23.1
25.5 $7.2
360
Spanit
240
Figure 6: Results of the experiments on CIFAR-10 using 5 different ResNet networks: ResNet- 14, ResNet-20, ResNet-32, ResNet-44, ResNet-56, left to right, top to bottom. ETTs < n in the legend stands for the ticket transformed from ResNet-n using ETTs, which is also absent in the subfigure where ResNet-n is the target network. Curves for pruning methods are dashed.
those transformed from ResNet-14/20 by notable gaps. Similar comparisons can also be drawn on other target networks. On the other hand, the tickets transformed from ResNet- 14 have the best performance than other source networks on ResNet-20, and are nearly as competitive as IMP. However, ResNet-32, the performance of ETTs from ResNet- 14 lies between ETTs from other source models and pruning methods. When the target model goes to ResNet-44 or beyond, ETTs from ResNet-14 perform no better than pruning-at pruning-at-initaalinn at-initialization methods. This implies that our replicating or dropping strategies potentially still introduce noise, which may be acceptable within a moderate range, but might become too amplified when replicating or dropping too many times.
Experiments on VGG networks. We also apply ETTs to VGG networks, and report another comprehensive suite of experiments of transforming across VGG-13, VGG-19 and VGG-19 networks. Here we directly replicate/drop "layers", instead of residual blocks. The results shown in Figure [ convey similar messages as the ResNet experiments, where we see very competitive performance with IMP, even at high sparsity ratios, and significant gaps when compared against other pruning counterparts, especially SNIP and GraSP.
4.3 Experiments on ImageNet
Next, we extend the experiments to ImageNet. We adopt three models: ResNet-18, ResNet-26, and ResNet-34. We select sparsity ratio 73.79% as the test bed, at which level IMP with rewinding can successfully identify the winning tickets [15 Our results are presented in Table where we can see ETT' work effectively when transforming tickets to ResNet-26 and ResNet-34, yielding accuracies comparable to the full model and the IMP-found winning ticket, which are much higher than other
8

---

VSG-13 (Target, 1IFAR-10)
V0G-16 (Target, CIFAR-10)
VGG-19 (Target, CIFAR
$4.00
16.0
$3.5
23.75
00.3
$3.5
$3.25
$3.00
09.2
$3.0
50.3
62.9
42.25
82.9
12.00
30.0
590
73.
10.3
883
A31 25.0
67.2
0.0
360
0610
13.3
10.
(6-3
R72
300
580
71.0
83.
66.1
9511
Utgruned
ETTs
ETTs
Reint
Random
Magnitude
SNIP
GraSP
Figure 7: Results of the experiments on CIFAR-I0 with VGG networks VGG-13, -66 and -19. ETTs 6- n the legend stands for the ticket transformed from VGG-n using ETTs, which is also absent in the subfigure where VGG-n is the target network. Curves for pruning methods are dashed.
pruning counterparts. Transforming to ResNet- 18 seems slightly more challenging for ETTs, but its superiority over other pruning methods is still solid.
Table 1: E-LTH for ResNets on ImageNet.
Linear Mode Connectivity ReaNet-32
90
Network
Res-18 18
Res-26
Res-34
80
Full Model
69.96%
72.56%
73.77%
70-
IMP ETTs 14
IMP
70.22%
72.94%
74.20%
80
50
ETTs 20
Reinit 62.88%
66.97%
68.36%
Teal
Random
63.10%
67.07%
68.68%
40
Magnitude
64.96%
68.51%
69.74%
30
SNIP
62.23%
67.51%
69.35%
20
GraSP
62.85%
67.24%
69.23%
0.1
0.5
Alpha
ETTs 18
71.29%
71.86%
Figure 8: Linear mode connectivity property on ResNet-32. We compare winning tickets directly found on ResNet-32 using IMP and E-LTH tickets transferred from on ResNet-14 and -20.
ETTs + 26
68.17%
73.37%
ETTs 34
68.08%
72.47%
4.4 Linear Mode Connectivity of E-LTH
observed that winning tickets are stably optimized into a linearly connected minimum under different samples of SGD noises (e.g., data order and augmentation), which means that linear interpolations of two independently trained winning tickets have comparable performance. In contrast, linear interpolations for other sparse subnetworks have severely degraded performance. Here, we observe such "linear mode connectivity" property on tickets transferred from ResNet- 14 and ResNet-20 to ResNet-32, shown in Figure 8 We can see that IMP is stable at the linear interpolations of two trained IMP tickets, as observed in [4]. ETTs from ResNet-22 20 are less stable than IMP in terms of linear mode connectivity, with maximal 8.72% accuracy drop, but much more stable than ETTs from ResNet-14 14, with a maximal 69. 99.72% drop. This is consistent with our observation on model accuracies ETTs have better linear mode connectivity when the source and target networks have a smaller difference in depth, thus having better performance.
14
Possibility to Transfer General Pruned Solutions. The above experiments indicate lottery tickets transfer better via ETTs when they stay in the same basin, which was found be the same basin as the pruning solution
Implication:
12]. That implies ETTs might be able to transfer pruned solutions this is out of the current work's 's focus, but would definitely be our future work.
in general 35
4.5 Comparison to State-of-the-Art Dynamic Sparse Training Methods
Dynamic sparse training (DST) is an uprising direction to train sparse networks from scratch by dynamically changing the connectivity patterns, while keeping the overall sparse ratios (thus com- putation FLOPs) low. However, a crucial difference between E-LTH and DST is that E-LTH is a "one-for-all" method. For example, if we run IMP on ResNet-34 once, we then obtain tickets for ResNet-20, ResNet-44 and more "for free" simultaneously, by applying ETTs. In contrast, DST methods have to run each architecture independently from scratch. Therefore, any overhead of E-LTH is amortized by transferring to many different-deptt architectures: that is conceptually similar to LTH's value in pre- pre-training [2] 3 [8 E-LTH hence has strong potential in scenarios where

---

Table 3: Comparing E-LTH with DPF 28 using 20/3/5-20/32/6 on CIFAR-10, 10, using ResNett14/2003 as the source models respectively.
Table 2: Comparing E-LTH and RigL- 5x on ImageNet using ResNet- 18/26/34 (use ResNet-26 the source model).
Accuracy Drop
Backbone
Dense Acc.
Sparsity
DPF
E-LTH
Method
E-LTH
RigL-5x
90%
-1.60%
-1.64%
ResNet-20
92.48%
95%
4.47%
-4.8%%
ResNet-18
68.17%
69.59%
ResNet-226
72.94%
72.66%
ResNet-32
93.83%
90%
-1.41%
-0.96%
ResNet-34
73.37%
73.88%
95%
-2.9%
-2.33%
90%
-0.56%
-0.50%
ResNet-56
94.51%
95%
-1.77%
-1.66%
have different hardware constraints various types of devices can flexibly adapt one ticket to satisfy different constraints without applying IMP or DST methods repeatedly.
To quantitatively understand our relative costs, we compare E-LTH with two recent state-of-ther -herttr-tt DST algorithms, Rigl. [[]] and DPF [28] To compare E-LTHH with Rigl., we use ResNet- 18/26/34 on ImageNet (ResNet-26 as source model). We set the sparsity ratio to 73.79% so that it is consistent with Section[4.3) For result clarity, we have normalized all FLOPs numbers, by the FLOPPs of one-pass pass standard dense training ResNet-34]] Table indicates comparable accuracies between E-LTH and RigL on three models. RigL. uses 0.69x normalized FLOPs to train R 1.0x FLOPs for ResNet-26, and 1.30x FLOPs for ResNet-34. Meanwhile, E-LTH first uses 2.77x FLOPs to find the mask once on ResNee-226 then after applying the (transferred) mask, only takes 0.1 1.14/.0200x 20x/0.2xxx additional FLOPs to train ResNet- 18/26/34, respectively. Taking together, E-LTH and RigL use similar total FLOPs on training the three models; but note that E-LTH will become more cost-effective as the found mask is transferred more networks.
To compare E-LTH with DPF, we conduct experiments on CIFAR-10 to compare the accuracies and the FLOPs needed to obtain sparse networks with 90% and 95% sparsity levels. On one hand, we can see from the results in Table 3 that E-LTH achieves accuracy results fully on par with DPF, and even better at higher sparsities of ResNet-32 and ResNet-56. On the other hand, to train sparse models with sparsity ratios 90% and 95% on five networks of ResNet-14, -20, -32, -44, and -56, E-LTH needs total of 1.26 billion F while DPF requires 1.49 billion. Note that DPF is dynamic training method and calculates the gradients for all weights (including zero weights), which increases their FLOPs cost; in contrast. E-LTH utilizes static sparsity patterns and thus saves computations during back- pec-prrppggttion by skipping the calculation of the gradients of zero weights.
Conclusions and Discussions of Broader Impact
This paper presents the first study of the winning ticket transferability between different network ar- chitectures (within the same design family), and concludes with the Elastic Lottery Ticket Hypothesis (E-LTH). E-LTH has significantly outperformed SOTA pruning-at-ine muninn-at-initializaiinn methods, e.g., SNIP and GraSP. Our finding reveals inherent connections between lottery ticket subnetworks derived from similar models, and suggests brand-new opportunities in practice, such efficient ticket finding for large networks or adaptation under resource constraints. For further discussions about the limitation, deeper questions and future directions. please see the Appendix. We do not that this work will directly impose any negative social risk. Besides the intellectual merits, the largest potential societal in this work is helping understand lottery tickets and sparse network training. also provides a solution to flexibly deploy networks with different complexity,
impact that
Acknowledgment
Z.W. in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML).
DDifferent from what was calculated in the appendix of [1I, here we use different sparsity ratio (73.79%) than the original RigL paper (90%) for fair comparison, and uses the default 5x training steps for the best of RigL performance, which results in (1-73.79% 5x5 = 1.30x FLOPs. Similar calculation applies ResNet-18 18 and 26. comparison, E-LTH uses normal training steps (1x).
'We run 13 iterations of IMP on 32 to generate winning tickets with 90% and 95% sparsity ratios and transfer these two tickets to the other four architectures using ETTs.
10

---

References
[1] Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level residual networks from dynamical systems view. In International Conference on Learning Representations, 2018.
[2] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self superviue pre- vre-training in computer vision models. arXiv preprint 1rXiv20022.06908 2020.
[3] Tianlong Chen, Jonathan Frankle. Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin, The lottery ticket hypothesis for pre-trained bert networks, 2020.
[4] Tianqi Chen, lan Goodfellow, and Jonathon Shlens. Net2net: Accelerating leamnnng via knowledge transfer. arXiv preprint wrxiv:551.0066/ 2015.
[5] Xiaohan Chen. Cheng. Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint arXiv:2101.0m0 000663, 2020.
[6] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and accelcration for deep neural networks. arXiv preprint #rXiv1770..9928: 2017.
[7] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 lece, 2009.
[8] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04844 2019.
[9] Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6 2020.
[10] Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter, Simple and efficient architecture search for convolutional neural networks. arXiv preprint #rXiv:/71..0552 2017.
[11] Utku Evei, Trevor Gale, Jacoh Menick, Pahlo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMI.R 2020.
[12] Utku Evei, Yani A loannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. arXiv preprint arXtv:2010.0353. 2020.
[13] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainahle neural networks. In ICI.R, 2019.
[14] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[15] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.0160 2019.
[16] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprini arXiv:2009.085 2020.
[17] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:I902.00 09574.90......00 2019.
[18] Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, and Jingjing Liu. Playing lottery tickets with vision and language. arXiv preprint arXiv:2104.11822 2021.
[19] Klaus Greff, Rupesh K Srivastava. and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016.
[20] Eldad Haber. Lars Ruthotto, Elliot Holtham, and Seong-Hwan Jun. Learning across mallss-ull methods for convolution neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[21] Song Han. Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 1r1.v:1510.00149. 2015.
11

---

[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[23] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 3:Xiv:/704.0440 2017.
[24] Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456. PMLR, 2015.
[25] Arthur Jacot, Franck Gabriel, and Clément Hongler. Ncural tangent kernl: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8580-88 8589, 2018.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr SNIP: SINGLE NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learning Representations. 2019.
[28] Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
[29] Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei. and Mykola Pechenizkiy. Selfish sparse rnn training. arXiv preprint arXiv:2101 09048, 2021.
[30] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parameterization? in-time over-paramcterization in sparse training. arXiv preprint (tdi::202200880 2021.
[31] Zhuang Liu, Mingjee Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
[32] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 91): 2018.
[33] Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian, One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In NeurlPS, pages 1932-4942, 2019.
[34] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646-4655. PMIRR, 2019.
[35] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruming. arXiv preprint arXiv:2003.02389 2020.
[36] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1.409..556, 2014.
[37] Mingxing Tan and Quoc Lee. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML,, 2019.
[38] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, arXiv preprint ar arXiv:2066.00 054467 2020.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 1xxiv:1706. 037662
[40] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
[41] Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In International Conference on Machine Learning, pages 564 572. PMLR, 2016.
[42] E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1): 2017.
[43] Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. Advances in Neural Information Processing Systems, 33, 2020.
12

---

[44] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang. and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. ICLR, 2020.
[45] Jiaxuan You, Jure Leskovec. Kaiming He, and Saining Xie. Graph structure of neural networks. In International Conference on Machine Learning, pages 10881-10891. PMLR, 2020.
[46] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and supermask. NewrIPS, pages 1597-3607) 2019.
13


Reviews:
[{"review_id": "z-xAF6o__OV", "reviewer": "Reviewer_r4hi", "summary": "This paper focused on exploring the transferability of a subnetwork obtained from the given dense network. Following the Lottery Ticket Hypothesis (LTH), a new hypothesis called Elastic Lottery Ticket Hypothesis (E-LTH) was proposed along with a corresponding validation method––Elastic Ticket Transformations (ETT). Extensive experiments on different benchmarks are provided to prove the effectiveness of ETT.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is drafted with a strong motivation resulting in an interesting research topic --- the transferability of a winning ticket. The previous surprising findings in LTH require expensive computational costs to benefit real-world network training. This paper tries to alleviate this disadvantage by transferring the obtained winning ticket to other network structures, showing a good potential for practical application and theoretical exploration.\n\nThe experiments are comprehensive. In addition to the comparisons between networks in the same family based on several strong baselines, this work also provides other experimental analyses, such as the comparisons with dynamic sparse training and the analysis of linear mode connectivity. \n\nSome experimental settings are unclear. As mentioned in L257-L258, the sparsity ratio will be different. It is better to provide detailed comparisons and how the “best possible extent” (L259) was made to compensate for the difference.\n\nSome potential missing baselines. I am curious about the extreme situation of both “stretching” and “squeezing.” How about only stretching the last unit to finish transferring from lower to deeper network, or even stretching randomly picked unit.\n\nThe value of “squeezing” network: “stretching” network will benefit the winning ticket finding for a larger network based on a smaller one. However, based on the results (e.g., ResNet56 to ResNet32 in Fig. 4), the structural changes in winning tickets will also cause a performance drop for the converse scenario. It is better to elaborate on the potential benefits of this situation. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Suffice. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "NnJjHaOga6", "reviewer": "Reviewer_9fut", "summary": "This paper describes a method for adapting lottery tickets (LT) found in one network architecture to a related architecture of different size. The motivation for this work is to study the connection between LTs of different architectures, and to reduce the cost of performing iterative magnitude pruning (IMP). If LTs can transfer between architectures, then it may be possible to find lottery ticket only once and adapting the size of the model as needed, without having to perform IMP for each model architecture. They restrict to adaptation to models of the same family (only differing in depth), and show that the adapted lottery tickets outperform pruning-at-initialization and RigL.\n\nNetworks are viewed at the level of \"units\", which are layers (+normalization), or residual blocks for ResNets. The adaptation method is to replicate nearby units of a smaller source model for each additional layer, or drop nearby units of a larger source model for each extra layer. They perform ablation studies on various choices for the replication/dropping of units.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- This work appears to be very original, as it is the first work I have seen for adapting one lottery ticket to multiple architectures. \n\nQuality:\n- My major issue with this work is the claim that lottery tickets (LT) can be stretched or shrunk to different architectures. This is a bold claim and the current experiments leave room for doubt. To claim that the new network is a LT, we should see that 1) reinitialization under the LT mask, 2) LT weights with random mask, and 3) random weights with random mask all perform worse than the LT (while keeping the source blocks the same). However, the \"Interpolation 1234 - Random\" experiment in Figure 3 shows that it is not significantly different from \"Interpolation 1234\", raising serious doubts about whether the approach generates \"lottery tickets\" or just a way to stretch or shrink existing models. No other experiments are conducted to establish that the new network is indeed a LT, which also raises questions.\n\n- It’s unclear how much of the gain is due to simply having access to the LT on the original source model. For pruning-at-initialization baselines, what if you copy the same weights as LTH for existing blocks, and only perform pruning for the new layers using pruning-at-initialization techniques? This would be a more appropriate baseline to compare to. The current experiments disadvantages other baselines by leveraging a prior LT model only for ETTs. \n\n- Are the baselines (unpruned, IMP) initialized in the same way as the elastic LTH? I.e. Weights are repeated for certain blocks? Or is it done with random initializations? If the latter, I would be curious to see what happens when you replicate blocks in the same way at initialization. How much of the benefit of ELTH comes from replicating initial weights (if any)?\n\n- The claims that ETTs may transfer to general pruned solutions seems unfounded (also unclear what \"general pruned solutions\" mean here, so I may be misreading). To make the claim that LTs derived from different model architectures are connected, and that they stay in the same basin, the authors could look at linear mode connectivity between IMP solution on a target model and the ETT solutions. \n\nClarity:\n- The paper is generally well written. My only comment is that the takeaway of section 4.4 confuses me. I hope the authors can elaborate further on what the implication of their linear mode connectivity result is.\n\nSignificance:\n- The paper raises a question that would be significant to the community. However, the results so far do not make a compelling case for it. Nonetheless, I would be convinced otherwise if my earlier concerns were sufficiently addressed. \n\n\n=================================================================\nUpdates post rebuttal:\nThe authors have provided additional experiments and clarifications that addressed my earlier concerns. Although the performance difference between ELT and the appropriate baselines suggested in the review is small, it seems consistent, and would be interesting to the community given the general novelty of this work. Assuming that these new results will be added to the paper accordingly, I will increase my score from 4 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "MRxSltREdv", "reviewer": "Reviewer_GtXb", "summary": "This paper transfer the winning ticket found for one model to other models, the author claims it can reduce the pruning cost for a large-scale model. The motivation is clear.  \n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "However, I still have some concerns about this manuscript.\n\nThe readability of Fig3 4 5 6 is poor.\n\nThe author does not present a principal method to grow the depth and increase the width, and the depth and width are critical dimensions for a large-scale model.\n The SOTA dynamic method should include the lastest method(e.g. DWN[1] DFP[2] STR[3] ...)\n\nBesides, this paper only focuses on unstructured sparsity. In fact, the unstructured sparsity is hard to have speed gain emphasized by[4, 5], The authors are suggested conducting experiments on channel/filter pruning[6, 7] or fine-grained structured sparsity [4, 8, 9, 10].\n\ntypo: line 358 \"onlu\".\n\nOverall, the author proposes a framework to reduce **the IMP cost** via transferring \"winner ticket\" to other neural models. I go through the whole paper. I cannot find any training time report in the experimental evaluation. Therefore, it is not convincing to support the claim with the experiment of this manuscript.    \n\n[1] Discovering Neural Wirings\n\n[2] Dynamic Model Pruning with Feedback\n\n[3] soft threshold weight reparameterization for learnable sparsity\n\n[4] Accelerating Sparse Deep Neural Networks\n\n[5] PCNN: pattern-based fine-grained regular pruning towards optimizing CNN accelerators\n\n[6] Neural Pruning via Growing Regularization\n\n[7] Rethinking the Value of Network Pruning\n\n[8] Learning N: M fine-grained structured sparse neural networks from scratch\n\n[9] 1xN Block Pattern for Network Sparsity\n\n[10] PCNN: pattern-based fine-grained regular pruning towards optimizing CNN accelerators", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates whether lottery tickets can transfer across architectures of different depths, following up on prior work which showed transfer across datasets for the same architecture. Reviewers all found the clarity and originality to be strong and the topic is of significant interest. There were some concerns regarding the soundness of the claims presented, but these were resolved through additional experiments performed during the rebuttal period. I would strongly encourage the authors to include these experiments in the final version of the paper. Overall, I think this paper presents a valuable contribution and I recommend acceptance. 
Decision: Accept (Poster)