Dual Lottery Ticket Hypothesis
Authors: Yue Bai
Huan Wang
ZHIQIANG TAO
Kunpeng Li
Yun Fu
Venue: ICLR 2022 Poster (2022)
Keywords: Dual Lottery Ticket Hypothesis
Sparse Network Training

Abstract: Fully exploiting the learning capacity of neural networks requires overparameterized dense networks. On the other side, directly training sparse neural networks typically results in unsatisfactory performance. Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. Concretely, it claims there exist winning tickets from a randomly initialized network found by iterative magnitude pruning and preserving promising trainability (or we say being in trainable condition). In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark, then go from a complementary direction to articulate the Dual Lottery Ticket Hypothesis (DLTH): Randomly selected subnetworks from a randomly initialized dense network can be transformed into a trainable condition and achieve admirable performance compared with LTH --- random tickets in a given lottery pool can be transformed into winning tickets. Specifically, by using uniform-randomly selected subnetworks to represent the general cases, we propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH. Concretely, we introduce a regularization term to borrow learning capacity and realize information extrusion from the weights which will be masked. After finishing the transformation for the randomly selected subnetworks, we conduct the regular finetuning to evaluate the model using fair comparisons with LTH and other strong baselines. Extensive experiments on several public datasets and comparisons with competitive approaches validate our DLTH as well as the effectiveness of the proposed model RST. Our work is expected to pave a way for inspiring new research directions of sparse network training in the future. Our code is available at https://github.com/yueb17/DLTH.

Published as a conference paper at ICLR 2022
DUAL LOTTERY TICKET HYPOTHESIS
Yue Bai', Huan Wang', Zhiqiang Tao?, Kunpeng Li', and Yun Fu! 'Nottheastern University, Boston, MA, USA
Santa Clara University, Santa Clara, CA, USA
MMeta Research, Burlingame, CA, USA (bai. mei.yues yue, wang. (amaam))mmrtheetterr.cmm
ztaosscu. edu, kunpengl i fb. com, yunfušece. new.ecceeeee.ee edu
ABSTRACT
Fully exploiting the learning capacity of neural networks requires overparameter- ized dense networks. On the other side, directly training sparse neural networks typically results in unsatisfactory performance. Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its ca- pacity. Concretely, it claims there exist winning tickets from randomly initialized network found by iterative magnitude pruning and preserving promising trainabil- ity (or we say being in trainable condition). In this work, regard the winning ticket from LTH as the subnetwork which is in trainable condition and its perfor- mance
benchmark, then go from complementary direction articulate the Dual Lottery Ticket Hypothesis (DLTH): Randomly selected subnetworks from randomly initialized dense network be transformed into a trainable condi- tion and achieve admirable performance compared with LTH - random tickets in given lottery pool can be transformed into winning tickets. Specifically, by using mniform-randomll selected subnetworks to represent the general cases, we propose simple sparse network training strategy, Random Sparse Network Trans- formation (RST), to substantiate our DLTH. Concretely, we introduce regulariza- tion term to borrow learning capacity and realize information extrusion from the weights which will be masked. After finishing the transformation for the randomly selected subnetworks, we conduct the regular finetuning to evaluate the model us- ing fair comparisons with LTH and other strong baselines. Extensive experiments on several public datasets and comparisons with competitive approaches validate our DLTH as well as the effectiveness of the proposed model RST. Our work is expected to pave a way for inspiring new research directions of sparse network training in the future. Our code is available https://github..co//yee////
INTRODUCTION
While wee-pprraeteriized networks perform promisingly challenging machine learning tasks Zagoruyko & Komodakis (2016); Arora et al. (2019); Zhang et al. (2019), (0019, they require a high cost of computational and storage resources Wang al. (2020a); Cheng et al. (2017); Deng et al. (2020). Recent pruning techniques aim to reduce the model size by discarding irrelevant weights of well-trained models based different criteria Gale et al. (2019); He et al. (2017); Han et al. (2015a;b). Decisive weights are preserved and finetuned for obtaining final compressed model with acceptable performance loss. Following this line, several series of research works have been done to explore effective pruning criterion for better final performances. For example, regularization based pruning approaches Liu al. (2017); Han et al. 5a;b) leverage a penalty term during training for network pruning. Also, many researches take advantages of Hessian information to build more proper criteria for pruning LeCun et al. (1990); Hassibi & Stork (1993); Wang et al. (2019a); Singh & Alistarh (2020). However, regular pruning methods still requires full pretraining with high com- putational and storage costs. To avoid this, pruning at initialization (PI) attempts determining the sparse network before training and maintains the final performances. For instance, Single-shot Net- work Pruning (SNIP) Lee et al. (2018) uses a novel criterion called connection sensitivity to measure the weights importance and decide which weights should be removed. Gradient Signal Preservation

---

Published as a conference paper at ICLR 2022
70
Top1
66
DLTH Others: CIFAR10
DLTH Others: CIF AR100
86
D
DUTHUter
LTH Jter
LTHuiter
EB
EB
94
Scratch
Scratch
L1 (Prctrain based)
(Pretrain basod)
Baseline (Full modell)
Baseline (Full modell
0.5
0,7
6.9
0.95
0.98
0.5
0.7
0.9
0.95
0.98
Sparsity Ratio
Sparsity Ratio
(a) DLTH V. Others on CIFAR1O.
(b) DLTH v. Others on CIFARI0O.
Figure 1: Comparisons between DLTH with other training strategies. 3 times average scores with standard deviation shown as shadow are plotted. Both (a)(bb ours superiority in all non-pretrain based approaches (solid lines). Two pretrain based methods (dash lines) are plotted for reference. (This figure is best viewed in color.)
demonstrate
(GraSP) Wang et al. (2020a) regards the gradient flow as an importance factor and correspondingly use it to design the PI criterion.
Regular pruning and PI techniques both achieve promising results on sparse network training with considerable model compression. Nevertheless, they only focus on designating criteria to find a specific sparse network but ignore exploring the internal relationships between the dense network and its eligible subnetwork candidates. which impedes full understanding of the sparse network training. Lottery Ticket Hypothesis (LTH) Frankle & Carbin (2018) hits this problem with the first shot. It claims a trainable subnetwork exists in the randomly initialized dense network and can be found by pruning a pretrained network. In other words, as long as the dense network is initialized, the good subnetwork has been implicitly decided but needs to be revealed by the pruning process. The LTH has been validated by training the subnetwork from scratch with mask obtained from corresponding iterative magnitude pruning Han al. (2015b;a). The subnetwork is regarded as the winning ticket among the lottery pool given by the dense network. Accordingly, we treat the subnetworks which can reach the performance of winning ticket from LTH as the ones who are in trainable condition or with promising trainability. This valuable discovery naturally describes the relationship between a random initialized dense network and the trainable subnetwork hidden in it.
However, LTH only focuses on finding one sparse structure at the expense of full pretraining, which not universal both practical usage and investigating the relationship between dense and its subnetworks for sparse network training. In our work, we go from a complementary perspective of LTH and propose a new hypothesis called Dual Lottery Ticke! Hypothesis (DLTH). It studies a more challenging and general case of investigating the internal relationships between a dense network and its sparse counterparts and achieves promising performances for sparse network training (see Fig. I). randomly selected subnetwork from a randomly initialized
Specifically, DLTH is described dense network can be transformed into a trainable condition and achieve admirable performance using regular finetuning compared with LTH. In other word, random ticket in a given lottery pool can be turned into a winning ticket. (Please note, we consider the common cases based on uniformly random selection for weights obtain subnetworks, not including certain extreme situations such as the disconnected subnetworks.) For convenience, in the following sections, we omit this note and refer to "randomly selected subnetwork" as the common cases mentioned above. To validate our DTLH, we design Random Sparse Network Transformation (RST) to transform randomly selected subnetwork into a trainable condition. RST simply introduces a gradually increased regularization term to achieve information extrusion from extra weights (which are set to be masked) to the target sparse structure. In this way, we construct a closed-loop research perspective from the dual side of LTH as shown in Fig. 2. Compared with LTH, our DLTH considers a general case studying a randomly selected subnetwork instead of a specific one- with a simple training strategy RST proposed to validate it. The comprehensive relationship between dense network and its sparse counterparts is inquired for studying sparse network training. Our contributions can be summarized as follows:
2

---

Published as a conference paper at ICLR 2022
We investigate novel view of studying sparse network training by presenting Dual Lot- tery Ticket Hypothesis (DLTH), a dual problem of Lottery Ticket Hypothesis (LTH), artic- ulated as: A randomly selected subnetwork from randomly initialized dense network can be transformed into a trainable condition and achieve admirable finetuning performance compared with L.T..
We propose a simple sparse network training strategy, Random Sparse Network Transfor- mation (RST). to achieve promising performance and validate our DLTH. Concretely, we realize the information extrusion from the extra weights which are set to be masked to en- hance the target sparse structure by leveraging a gradually increased regularization term. In this way, the randomly selected subnetwork will be transformed into trainable condition.
Extensive and fair comparisons based on benchmark datasets with competitive approaches using the same finetuning schedule shows our RST obtains promising and consistent per- formances, solidly demonstrating its effectiveness and validating DLTH.
Our DLTH, compared with LTH, more general and challenging study the relationship between dense network and its sparse counterparts. We expect our work inspires novel perspective to study sparse neural network in more flexible and adjustable way.
2 RELATED WORK
Network pruning is one of the relevant research topics. In this section, we group the existing pruning methods into After-Training Pruning and Pre-Training Pruning as follows. In addition, several dynamic sparse network training strategies are proposed to adaptively adjust and train the sparse network referred as Dynamic Sparse Network Training below.
After-Training Pruning. Most pruning methods prunes a pre-trained network Louizos et al. (2017); Liu al. (2017); Ye et al. (2018); Han et al. (2015b;a). Algorithms utilize different ranking strategies to pick redundant weights with low importance scores, and remove them to achieve pruning along with acceptable performance drop. Pioneer magnitude-basee method Han et al. (2015b;a) regards weights with low values unnecessary ones, which is straightforward but may remove important low-value weights. Hessian-based approaches measure the weight importance by computing the their removal effects on the final loss LeCun et al. (1990); Hassibi & Stork (1993). Recently pub- lished technique also achieves Hessian-free pruning Wang et al. (2020b) by adding regularization which is more computational friendly. In addition, pruning process can be also conducted during the network training. To name few, a special dropout strategy is utilized in Srinivas & Babu (2016) to adjust the dropout during training and obtain a pruned network after training. A To norm regu- larization based method is proposed for network pruning Louizos et al. (2017). Above algorithms obtaining sparse network from a trained dense network are seen as Bher--raining Pruning.
Pre-Training Pruning. This research direction investigates how to prune randomly initialized network without any training. Pruning at initialization is one typical direction deriving sparse net- work by remove part of randomly initialized weights Lee et al. (2018; 2019); Wang al. (2020a). For instance, Single- Shot Neural Network Pruming (SNIP) algorithm Lee et al. (2018) first proposes learnable sparse network strategy according to the computed connection sensitivity. An orthogo- nal initialization method is proposed to investigate pruning problem in signal propagation view Lee al. (2019). Gradient Signal Preservation (GraSP) considers to preserve the gradient flow as an efficient criterion to achieve pruning at initialization. On the other hand, Lottery Ticket Hypothe- sis (LTH) claims that the winning ticket subnetwork exists in the randomly initialized network and can be found by deploying conventional pruning algorithm. Then the selected sparse network can be efficiently trained from scratch and achieve promising performance. Based on LTH, Early-Bird Bird (EB) ticket You et al. (2020) proposes an efficient way to find winning ticket. Above algorithms obtaining sparse network from a random dense network are seen as Pre-Truining Pruning.
Sparse Network Training. The sparse network can be dynamically decided and adaptively adjusted through the training to improve the final performance. Sparse Evolutionary Training (SET) Mo- canu et al. (2018) proposes an evolutionary strategy to randomly add weights on sparse network and achieve better training. Dynamic Sparse Reparameterizztion (DSR) Mostafa & Wang (2019) presents a novel direction to dynamically modify the parameter budget between different layers. In this way, the sparse structure can be adaptively adjusted for more effective and efficient usage. Sparse Networks from Scratch (SNFS) Dettmers & Zettlemoyer (2019) designates momentum
3

---

Published as a conference paper at ICLR 2022
Comparisons between DLTH and other settings. Base model, One/Multipl controllabilic Transformatio inn, and Pretrain represent where to pick subnetwork, if the setting finds one specific subnetwork or studies randomly selected subnetworks, if sparse structure is controllable, if the selected subnetwork needs transformation before finetuning, and pretraining dense network is needed, respectively.
Table I:
Settings
Basc model
Onc/Multiple
Controllability
Transformation Pretrain
Conventional
Pretrained Initialized Initialized
One
Uncontrollable
No
Yes
LTH
One One Uncontrollable
Uncontrollable
No
Yes
No No
DLTH (ours)
Initialized
Multiple
Controllable
Yes No
based approach as criterion to grow weights and empiricalty proves it benefits the practical learning performance. Deep Rewiring (DeepR) Bellec al. (2017) introduces the sparsity during the training process and augments the regular SGD optimization by involving random walk in the parameter space. can achieve effectively training on very sparse network relying on theoretical foundations. Rigging the Lottery (RigL) Evci et al. (2020) enhances the sparse network training by editing the network connectivity along with the optimizing the parameter by taking advantages of both weight magnitude and gradient information.
3 LOTTERY TICKET PERSPECTIVE OF SPARSE NETWORK
Lottery Ticket Hypothesis (LTH) Frankle & Carbin (2018) articulates the hypothesis: dense ran- domly initialized networks contain subnetworks, which can be trained in isolation and deliver per- formances comparable to the original network. These subnetworks are seen as winning tickets with good trainability and are discovered by iterative magnitude pruning, which requires training dense network. The mask of the pruned network illustrates the sparse structure of winning tickets.
Problem Formulation. We start from introducing general network pruning problem. The neural network training can be seen sequence of parameter updating status using stochastic gradient descent (SGD) Wang et al. (2021); Bottou (2010):
{w (1),, (())
(k)
(K)),
where w is the model parameter with superscript as training iteration number. For a general case, the sparse network structure can be mathematically described by binary mask m which has the same tensor shape as w. The process of obtaining can be formulated as a function m = Fm (w; D), where D the training data. Further, the weights of the sparse structure are different based on different strategies Hassibi & Stork (1993); Wang et al. (2020b), which is given by w" Fw (w; D). The final pruned network can be integrated as
= Fm (wm D) Fic( D):
where wmm and t represent different parameter conditions for Fm and F.. The conventional pruning requires that km = kw K. The LTH needs km K, kw O, and F, I, where is the identical mapping representing model directly inherits the randomly initialized weights.
Against with traditional view that directly training sparse network cannot fully exploit network ca- pacity Wang et al. (2020b;a), LTH validates there exists sparse network (winning ticket) with better trainability than other subnetworks (other tickets). Specifically, the picked winning ticket will be trained (finetuned) in isolation to evaluate the model performance and illustrate its admirable trainability. However, to uncover this property still needs first pruning a pretrained model to obtain mask and the mask must match with the original dense network winning ticket matches with the given lottery pool. LTH provides a novel angle to understand and reveal the connections between random dense network and its subnetworks with admirable trainability. However, LTH only vali- dates there exists one specific subnetwork and still requires pruning on pre- pre-trained model to find it, which is relatively restricted case.
4

---

Published as a conference paper at ICLR 2022
Lottery Ticket Hypothesis
Dual Lottery Ticket Hypothesis
Randomly initialized
Randomly initialized
Pre-train
Randomly pick
Pre-trained network
Random subnetwork
Winning ticket exists
Random ticket wins
Pruned network
Transformed network
Prune
Transform
Pruning pre-trained dense network
Transform random subnetwork
Figure Diagram of Lottery Ticket Hypothesis (LTH) and Dual Lottery Ticket Hypothesis (DLTH).
4 DUAL LOTTERY TICKET HYPOTHESIS
We follow the perspective of LTH and investigate sparse network training from a complementary di- rection. We articulate Dual Lottery Ticket Hypothesis (DLTH). We elaborate our DLTH and discuss its connections and differences with LTH and other related research topics.
Dual Lottery Ticket Hypothesis (DLTH). A randomly selected subnetwork from randomly initialized dense network can be transformed into a trainable condition, where the transformed subnetwork can be trained in isolation and achieve better at least comparable performance LTH and other strong baselines.
Formally, we follow the general definition provided in Sec. 3 for problem formulation. Our DLTH requires hi = 0, k.,, and allows F,, being random mapping. F,, represents our proposed RST which will be detailed introduced in Sec. 5
Comparisons with LTH. LTH proves there exists an appropriate subnetwork randomly ini- tialized dense network whose weights can be easily trained and overcome the difficulty of training sparse network. The weights sparse structure are picked from the initial network but it requires praning pre-trained dense network to obtain the mask. On the contrary, DLTH claims randomly selected subnetwork can be transformed into a proper condition without pruning pre-trained model where we adjustably pre-define the mask and further conduct efficient sparse network training.
On the one hand, these two hypotheses mathematically form a dual statement, which lies a com- plementary direction. On the other hand, based
definition in Sec. sparse network has two characteristics: 1) network structure; 2) network weights. In this way, LTH can be seen as finding structure according weights, because prunes the pretrained network to find mask using weight magnitude ranking; DLTH can be seen as finding weights based on a given structure, be- cause transforms weights for a randomly selected sparse network. Therefore, we name our work as DLTH. We clarify the advantages of DLTH compared with LTH from three angles: 1) Although LTH and DLTH are twins in mathematics, our DLTH considers more general scenario (exist specific subnetwork versus random selecied subnelwork) study the relationship between dense and sparse networks: 2) DLTH more valuable for practical usage, because generally allows to transfer random subnetwork into trainable condition instead of pruning pretrained model to identify one good subnetwork. Further, the transformation is also more efficient than pretraining dense network in LTH, detailedly discussed in Sec. 6; 3) DLTH allows flexible sparse network selection but LTH must use the pruned mask and the sparse structure is not controllable.
Comparisons with Pruning at Initialization (PI). Pl is another relevant research topie, which lects sparse network from scratch according to well-designed criterion. The subnetwork with better trainability is picked for following finetuning. Unlike LTH finding sparse candidate through
5

---

Published as a conference paper at ICLR 2022
ventional pruning, PI chooses candidates without training dense network. Following our definition in Sec. 3, PI requires km = 0, w/ = 0. Fm is determined by different algorithms and F = I. Similarly, PI also aims to obtain specific sparse network based criteria and the selected sparse structure is not controllable. Our work is not for choosing one specific subnetwork but transferring random subnetwork into good condition. In this way, the sparse structure will be flexible and under our control. We summarize these comparisons in Tab. 1.
As LTH and Pl use finetuning to evaluate final performance, our DLTH also conducts the same finetuning on the transformed subnetwork to make fair comparison with other methods.
5 RANDOM SPARSE NETWORK TRANSFORMATION
Being analogous to the real-world lottery game where the winning ticket exists in the lottery pool but hard to find, randomly picking a subnetwork from a random dense network is not wise and needs to be transformed leveraging on the existing information from other parts of network (other tickets). We follow this analogy to propose Random Sparse Network Transformation (RST) to obtain trainable sparse network. Specifically, we employ a widely-used regularization term Wang et al. (2020b; 2019b) to borrow information from other weights which are set to be masked. It benefits the sparse network training by extruding information from other weights to the target sparse structure.
Information Extrusion. We use a regularization term to realize information extrusion. Given a randomly initialized dense network f(x; () with randomly selected subnetwork represented by mask m E {0, 1}, the information extrusion can be achieved as optimizing following loss:
LR = [(fxx;)) 1 11
(3)
where loss LR contains a regular training loss L on given data D and a L2 regularization term added on 0" 0* is the part of parameter 0 which being masked by m = 0. In this way, the unmasked weights, referred as 0*, are regularly trained by SGD, while the masked weights 0* are still involved for network training but their magnitudes are gradually suppressed via the increasing regularization. The trade-off parameter X is set to gradually increase from a small value during optimizing Lx:
Ap + 7h
< Ab= Ap
Ap
Ap+1
Xpp
Ab:
where Ap is the regularization term at p-th updating. 7) is the given mini-step for gradually increasing value. A is the bound to limit the À increasing. In this way. the magnitude of 6* is diminished from regular to small scale along with training process. Hence, the importance of weights is dynamically adjusted from a balanced (equal for all weights) to an imbalanced situation (0* becom- ing less important with diminishing magnitude). The information of 0* is gradually extruded into 0 Analogically, the random ticket is transformed to winning ticket by leveraging information from other tickets in the whole lottery pool. After sufficient extrusion, the 0 will have very limited effect on the network and we remove them obtain the final sparse network, which will be finetuned to obtain the final test performance for model evaluation.
6 EXPERIMENTAL VALIDATION
Experiments are based on ResNet566ResNetl et (2016) on CIFAR CIFARIO/CIFARIO Krizhevsky et al. (2009), and a ImageNet subset Deng et (2009) to compare our method with Lottery Ticket Hypothesis (LTH) Frankle & Carbin (2018) and other strong baselines. We set five sparsities: 50%, 70%, 90%, 95%, and 98%. Specifically, we leave the first convolutional layer and the last dense layer as complete, and prune the weights in other middle layers with the consistent sparsity ratio. We run each experiment three times and report their means with standard deviations.
Comparison Approaches. We use several comparison methods: L1 et. (2016) is the regular LI pruning based on pretrained network. LTH Frankle & Carbin (2018) is the lottery ticket performance based on one-shot pruning strategy. LTH-Iter follows the iterative magnitude pruning strategy used in LTH. EB You et al. (2020) is the Early-Bird ticket for LTH with one-shot pruning strategy. Scratch represents training random subnetwork from scratch without pretraining. RST and RST-Itter

---

Published as a conference paper at ICLR 2022
Table 2: Performance comparison of ResNet56 on CIFARIO and CIFARTO0 datasets using 50%, 70%, 90%, 95%, and 98% sparsity ratios. Except for the L.1 row, the highest/second highest perfor- mancs are emphasized with red/blud fonts.
ResNet56 + CIFARIO: Baseline accuracy:
93.50%
Pruning ratio
50%
70%
90%
95%
98%
LI Li et al. (2016) (rettrai-bbsee)
93.33:10.08
92.83++0.39
91.67=0.11
90.13=0.21
84.78=10.27
LTH Frankle & Carbin (2018)
92.67:1..25
91.88+0.35
89.7810.35
88.05
83,85=00.55
LTTH Iter-5 Frankle & Carbin (2018)
92.68++0.39
92.50++0.15
90.24+0.27
88.10-0.36
83.91=0.15
EB You at (2020)
92.76+00.21
91.61+0.60
89.50+0.60
88.00-0.38
83.74+0.35
Seratch
92.497-0.35
92.14+0027
F9.89-F6.122
87.41-00.31
82.7T-00:40
RST (ours)
92.34:10112
92.27:10.24
90.41:10005
08,24-0.08
83.77:00447
RST Iter-5 (ours)
93,41+0.16
92.67++0.02
90.43++0.21
88.40-0.14
83.97++0.09
ResNet56 CIFARI00: Baseline accuracy: 72.54%
Pruning ratio
50%
70%
90%
95%
98%
L1 Li al. (2016) (Pretrain-baseel
71.96+0.11
71.59+0.22
68.29+-0.0)
64.74-0.22
50.04+1.552
LTH Frankle & Carbin (2018)
69.95+-0.47
68.24+0..0
65.66-F0..7
60.97-0.30
552.77-50:44
LTH Iter-5 Frankle Carbin (2018)
10.57+0.15
69.54+0.46
64.84:0.11
60.45-0.61
53.83++0.09
EB You el. (2020)
70.27+00.59
69.1644.36
64.01110.42
60.09-0.33
13.14:1.044
Scratch
70.96:1:.25
68.59:1.355
64.62:10.2
59.93=0.24
50.80=10.55
RSI (ours)
7T.1370.448
69.85-
66.17:60118
61.66-0.37
544.TT-0037
RST Iter-5 (ours)
71.39:10.34
70.48:0.19
65.6550.15
61.71=0.36 54.46:10.32
method in one-shot and iterative way, respectively. We use uniform-randonee randoolly selected subnetworks to represent the general cases for our RST and validate the DLTH.
Implementations. All experiments use the same finetuning schedule for faimess. Experiments on CIFARIO/CIFAR 100 are optimized by SGD with 0.9 momentum and 5e-4 weight decay using 128 batch size. Total number of epochs 200 with 100.001/00000 learning rates starting at 0/100/150
epochs, respectively. Those for ImageNet are optimized by SGD with 0.9 momentum and le-4 weight decay using 256 batch size. Total number of epochs is 90 with 0.110.01/00.001/00.000 learning rates starting at 0/30/60/75 epochs, respectively.
We use the same schedule as finetuning to pretrain the network for L1 and LTH. For LTH-Iter, we set the total number of epochs 50 with 0. 1/0.01/000000 learning rates starting at 0/25/37 epochs, respectively, for each mini-pretraining. Other configurations are kept as the same. For EB, we use its original early stop point, 25 epochs (1/8 of total epochs), and keep other settings the same.
For RST, we set the start point λn and bound Å as and 1, respectively. The mini-step is set as 1e-4 and the À is increased every 5 iterations termed as U.. Each iteration optimizes the parameter with 64 batch size and le-3 learning rate. After reaches the bound, we continue the extrusion with 40,000 iterations for model stability termed v.- For RST-Iter, in order to keep faimess with LterIter, we accordingly diminish the transformation process by setting X being increased every iterations and 10,000 iterations for model stability. More detailed comparisons are provided in Sec. 6.1. Since our transformation uses 1e-3 learning rate, we add small warm-up up slot in finetuning for our methods: 10 epochs and epochs with le-2 learning rate, for FF/CIAO/IFARRI and ImageNet, respectively, at the beginning then back to 0.1 as mentioned above.
6.1 BASELINE COMPARISONS
Tab. shows the comparisons on CIFARIO/CIFARIO0. The full accuracy (first line) is trained by regular SGD on random dense network. For consistency. L1, LTH, LTH Iter-5, and EB inherit the same initial weights as the full accuracy model. Scratch, RST, and RST Iter-5 use the same initial weights as above and they also share the same randomly selected subnetwork from the dense one. Only LI requires pretrained model and others use random dense network. Note that, once the subnetwork is randomly decided, it is kept intact without any structural modification. Since LI uses pretrained network, it should have the highest performance. Thus, we exclude the LI and emphasize the rest rows using red/blue fonts the highest/second highest performances.
Performance Analysis. In Tab. 2, we find: 1) Our method generally outperforms other approaches; 2) Compared with LTH which uses wisely picked subnetwork, our method achieves promising
7

---

Published as a conference paper at ICLR 2022
sults on randomly picked subnetwork, which solidly validates our DLTH. Our method also pushes the performance close to the LI with comparable results and even better in certain cases; 3) Com- pared with scratch, our method consistently achieves better performance; 4) Iterative strategy bene- fits both LTH and RST.
Iterative Cycle Number Choice. Using itera- tive magnitude pruning is a key factor for LTH which the benchmark for our work. Hence, we first conduct an ablation study on LTH to choose the most competitive iteration number (cycle number) for a persuasive comparison. We use CIFAR1O with 0.7 sparsity as an exam- plar to set the iterative numbers in Fig. 3. We run 3 times and plot the mean with shadow area as standard deviation. LTH performs better us- ing more iterative cycles; our DLTH takes less advantages of more cycles but still outperforms LTH. To throughly demonstrate our superiority, LTH) for both LTH and our DLTH. The ablation study and further analysis of DLTH cycle number is provided in Fig. 4.
$20
120
Top3 Accuracy
90.4
320
LTH DLTH
116
21A
5
Number Cycles
Figure 3: LTH DLTH on CIFARIO with 0.7 sparsity to choose cycle number.
we set the cycle number as 5 (the best choice for
ImageNet Results. We compare RST with LTH on a ImageNet subset in Tab. 3. ImageNet subset contains 200 out of 1000 classes from the original dataset. We use the one-shot strategy for LTH/RST and our RST performs better.
Table 3: Test Accuracy of R on ImageNet Subset.
ResNet18+ + ImageNet Subset: Baseline: 78.96%
Sparsity
50%
70%
LTH Frankle & Carbin (2018) 77.87
76.48
RST (ours)
78.24
76.80
Discussion of DLTH v. LTH. As a dual format of LTH, DLTH allows randomly
selecting subnetwork instead of pruning pretrained network for mask. RST transfers the picked ran- dom ticket for winning, instead of directly employing obtained mask to represent the winning ticket in LTH. This pair of hypothesis use the same finetuning schedule and we compare their differences as "pretrain + prune" versus "randomly select transform": 1) Pretraining requires the same com- putational resource as finetuning in our experiments. LI pruning can be done efficiently without any parameter updating: 2) DLTH randomly selects subnetwork with extra cost (even no sorting cost). The transformation can be seen as an optimization process. Based on the details in the imple- mentations section above, the extra training cost can be calculated as c. ((Ap/)) (h)/n))r v. +u,) ),,N,,,,, )o,,/N 5//D. where N₂ and ND are batch size and number of training samples. Specifically, for RST, this number ss 115 epochs (CIFAR as example). We conclude: 1) Since RST still maintains the full network during transformation, the required memory space is the same as pretraining in LTH; 2) Complete LTH and RST require 400 epochs and 315 epochs training cost (both including 200 finetuning epochs). The extra computational cost of RST is lower than LTH. 3) 5-cycle iterative setting, for RST Iter-5: there are C, = ((1/ee4))*1 1+ 1 (0000)66/5000)) = 25.6 epochs each cycle and 5 25.6 = 1 128 epochs for total before finetuning: for LTH Iter-5: this number is 5*50 250 epochs. RST Iter-5 is still more efficient than LTH Iter-5.
(0l6-)))55-00000))44/550000
Exactly saying goes: There ain't It such thing a free lunch, finding trainable sparse structure in a random dense network needs extra cost. As the calculations above, these cost of both LTH and RST are in reasonable scope (ours is more efficient). However, our hypothesis considers a more general and valuable side of this pair-wise hypotheses.
Iterative Number Ablation Study for RST. Fig. 3 shows the LTH benefits from the more iterations (cycles) but our RST obtains relatively robust performance with a little improvement. We show the ablations for RST here and leave the detailed discussions in the supplementary material. Fig. 4 visualizes the performance variations of different cycle numbers. We find using iteration strategy (cycles: 2,3,4,5) can outperform the one-shot strategy (cycle: 1) for most cases. However, the overall performances show robustness about iteration number with little improvement and inapparent correlation between cycle numbers and final performances.
8

---

Published as a conference paper at ICLR 2022
50.00
50.40
350
-5.00
36A
42.4
10.73
10.30
35.3
7003
-4.75
427
342
$2.85
+0.30
702
23.2
Werd
06.33
30.53
401
535
15.0
32.33
00.35
803
624
of Cydax
Mumber Cycle
(a) Iterative cycle number ablation on CIFARIO.
(b) Iterative cycle number ablation on CIFAR 100.
Figure 4: performance on CIFAR CAAARO/IIFRRIII 100 with sparsity ratio from 0.5 0.98 and cycle number from to 5. Different colors represent different sparsity ratios. (This figure is best viewed in color.)
Ablation study for iterative cycle number of DLTH. (a)/(b) show the times average
Comparison with GraSP. We compare RST with Gradient Signal Preservation (GraSP) Wang et al. (2020a). a representative Pruning Initialization (PI) approach, to further validate our DLTH. Ex- periments are based on CIFARI COARROO///AARIII 100 datasets (see Tab. 4) using ResNet56 and ImageNet subset (see Tab. 5) using R on different sparsities. GraSP selects a subnetwork from ran- dom dense network to finetune. To be fair, RST inherits the weights of the same random dense network and follows the same layer-wise sparsity ratio derived by GraSP. RST achieves better per- formances using the layer-wise sparsity from GraSP, which means RST has generalibility to handle different sparse structures and further validates our proposed DLTH.
Table 4: Comparison with GraSP CIFARIO and CIFAR 110AR100 datasets.
ResNet56 CIFARIO: Baseline accuracy: 93.50%
Sparsity
50%
70%
90%
95%
98%
GraSP Wang et al. (2020a)
91.72
90.75
89.93
88.62
84.46
RST (ours)
92.69:10.13
92.03:10.28
90.88:1.11
89.14:0.27 84.95:10005
ResNet56 + CIFAR100: Baseline accuracy: 72.54%
Spursity
50%
70%
90%
95%
98%
GraSP Wang al. (2020a)
67.88
67.38
64.21
59.39
45.01
RST (ours)
70. 0.
0.500
6436--033
60.20-0.122
35.93--03366
Table 5: Top-1 and top-5 accuracy comparison with GraSP on ImageNet Subset.
ResNet18 + Imagenet Subset
Sparsity
70%
90%
95%
Metric
Acc
Acce5
Acc@1 I
AccW5
Acc@0
Acc@P
GraSP Wang et al. (2020a)
75.20
92.17
73.64
91.15
69.70
89.62
RST (ours)
76.95
92.87
74.49
91.52
70.33
90.01
7 CONCLUSION AND DISCUSSION
We propose Dual Lottery Ticket Hypothesis (DLTH) dual problem of Lottery Ticket Hypoth- esis (LTH). We find that a randomly selected subnetwork in randomly initialized dense network can be transformed into appropriate condition with admirable trainability. Compared with LTH, our DLTH considers a more challenging and general case about studying sparse network training, being summarized as follows: 1) It is executable transferring a randomly selected subnetwork of a randomly initialized dense network into format with admirable trainability; 2) The flexibil- ity of selecting sparse structure ensures us having the controllability of the subnetwork structure instead of determined by pruning method; 3) Our DLTHH studies sparse network training in a com- plementary direction of LTH. It investigates general relationships between dense network and its sparse subnetworks which is expected to inspire following research for sparse network training. We propose a simple strategy, Random Sparse Network Transformation (RST), to validate our DLTH. Specifically, we naturally involve regularization term and leverage other weights to enhance sparse network learning capacity. Extensive experiments on several datasets with competitive comparison methods substantially validate the DLTH and effectiveness of the proposed RST.

---

Published as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In /CML, 2019.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv: 1.05/1,,1/36 2017. 4
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT, pp- 177-186. Springer, 2010.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09280 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 6, 12
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprini arXiv:1907.04887 2019.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In ICML, 2020. 4
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint /803.0333, 2018. 2, 4, 6, 7, 8, 12, 13, 14
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint 1xXx::9902.09574 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 10.00144 2015a. 1.2.3
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.0266 2015b. 2,
Babak Hassibi and David G Stork. Second order derivatives for network pruming: Optimal brain surgeon. Morgan Kaufmann, 1993. 1,3, 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6, 12
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. NeurIPS, 1990. 1,3
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Tor. Snip: Single-shot network pruning based connection sensitivity. arXiv preprint arXiv:1810.02344 2018. 1,
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr signal propagation perspective for pruning neural networks at initialization. arXiv preprint 1XX6v:1906.066 2019.
Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608.0866008777 2016. 6, 7, 13, 14
10

---

Published as a conference paper at ICLR 2022
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017. 1,
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint 171.v:1712.01//11 2017.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1-12, 2018. 3
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML,, 2019.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. XrXiv preprint arXlv:2004.14340, 2020.
Suraj Srinivas and Venkatesh Babu. Generalized dropout. arXiv preprint 1rXiv:1611.06791,,000
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. In ICML, 2019a.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020a. 1, 2, 3, 4, 9, 12,
Huan Wang, Qiming Zhang, Yuehai Wang, Yu, and Haoji Hu. Structured pruning for efficient convnets via incremental regularization. In IJCNN, 2019b. 6
Huan Wang. Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. arXiv preprint arXiv:2012.09924 2020b. 3, 4, 6
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Emerging paradigms of neural network pruning. arXiv preprint arXiv:2103. 06460, 2021. 4
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the mmaller-orrm-ees--nfoon informative assumption in channel pruming of convolution layers. arXiv preprint arXiv: 2018.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard Baraniuk. Drawing early-bird tickets: Toward more efficient training of deep networks. In ICIR, 2020. 3, 6,
Sergey Zagoruyko and Nikos Komodakis. 11Xiv:1600.00714. 2016.
Wide residual networks.
arXiv preprint
Guodong Zhang, James Martens, and Roger Grosse. Fast convergence of natural gradient descent for overparameterized neural networks. arXiv preprint 1eXiv:1905.10900 2019. I
A ANALYSIS OF ITERATIVE NUMBER ABLATION STUDY FOR DLTH
We analyze the ablation pattern from implementation aspect of iterative strategy for both LTH and RST. Then, we provide more experimental details. For both LTH and RST with iteration strategy, the corresponding subnetwork is selected or pruned iteratively until it reaches the given sparsity. However, the iterative operations for LTH and RST are different. For each iteration of LTH, obtains the pretrained model and uses LI pruning to decide the subnetwork based on the weight magnitude. In this way, the subnetwork elaboratively selected within each iteration and LTH progressively enhanced to achieve higher performance. On the other side, within each iteration, RST randomly selects weights to add regularization and removes them at the end of this iteration. This iteration strategy does improve the performances for most cases (see comparison between RST and RST Iter-5 in Tab. 2). because the extra weights are gradually removed through each iteration and the whole transformation is more stable. However, compared with LTH enhanced by iteratively using LI pruning, this progressive enhancement for RST is relatively weak. We further analyze this comparison based on the experiments of ablation study.

---

Published as a conference paper at ICLR 2022
Table 6: Cycle number ablation study for RST ResNet56 on CIFARIO and CIFARIO0 datasets.
ResNet56
CIFARIO: Baseline accuracy: 93.50%
CIFARI00: Baseline accuracy: 72.54%
Cycles
50%
70%
90%
95%
98%
50%
70%
90%
95%
98%
1 93.02
92.50
90.03
88.05
83.75
71.53
70.44
65.57
61.38
53.51
2
93.47
92.79
90.17
88.14
83.49
71.51
70.47
65.66
61.69
54.04
3
93.57
92.61
90.36
88.33
83.78
71.63
70.01
65.83
61.84
54.17
4
93.54
92.63
90.46
88.14
83.64
71.70
65.78
61.56
53.65
5
93.41
92.67
90.43
88.40
83.97
71.39
70.48
65.65
61.71
54.46
Mcan
93.40
92.64
90.29
88.21
83.73
71.55
70.35
65.70
61.64
53.97
Std
0.20
0.09
0.16
0.13
0.16
0.11
0.18
0.09
0.16
0.35
Experimentally, Tab. 6 provides performance means and standard deviations of each cycle number for RST ablation study on CIFARI CEAARO/CIFARIII 100 datasets. We find RST obtains relatively robust per- formances among different cycle numbers. However, it generally achieves better performances with small standard deviations compared with other methods in Tab. 2. We conclude RST is adaptable and robust for iteration strategy and even obtains better performances using fewer cycles compared with LTH, which can further save the computational costs.
In order to make fair comparisons with LTH, we implement RST with an iteration strategy following the fashion in LTH and use the same iterative number. For how to wisely leverage the iteration strategy for RST to obtain further improvement, we leave it into our future work.
B DETAILED EXPERIMENTAL COMPARISON
As an extension of experimental comparisons the tables Baseline Comparisons section, we provide the performances of each run for L1, LTH, Scratch, and RST with their average and standard deviation shown in Tab. 7 and Tab. 8, which demonstrate the consistent effectiveness of our proposed RST. The column of Std is the same values in the tables of our main text.
C VISUALIZATIONS OF PERFORMANCE COMPARISON
We provide visualizations of performance comparisons. The comparison between Lottery Ticket Hypothesis (LTH)) Frankle & Carbin (2018) and our Dual Lottery Ticket Hypothesis (DLTH) using the proposed Random Sparse Network Transformation (RST) CIFAR10 and CI- FARI00 Krizhevsky et al. (2009) are shown Fig. 5 and Fig. 6. The comparisons between our RST and training from scratch CIFAR 10 and CIFAR 100 Krizhevsky et al. (2009) datasets are shown in Fig. and Fig. We plot the 100 to 200 finetuning epochs for a clear visualization.
D IMPLEMENTATION DETAILS
We use 4 NVIDIA Titan XP GPUs to perform our experimental evaluations. Each experiment on CIFAR MAAARO//IFARIO 100 dataset requires around 6 hours one GPU and each experiment Ima- geNet Deng et al. (2009) subset requires around 12 hours on two GPUs. Further, the section of comparison between our RST and GraSP Wang et al. (2020a) algorithm, sity ratio obtained from GraSP. We provide the detailed layer-wise ratio for CIFARIO and CIFAR100 shown in Fig. and Fig. 10, respectively.
the layer-wise spar-
E CURRENT LIMITATION AND FUTURE WORK
Our Dual Lottery Ticket Hypothesis (DLTH) paves complementary direction to explore Lottery Ticket Hypothesis (LTH) problem. Current experimental results and analysis substantially validate proposed DLTH. However, there are still some remaining problems. For example, we vali- date the random sparse network can be manipulated into a good trainable condition in this work. However, which type of sparse structure
be transformed in high efficiency and which fits the practical deployment appropriately to benefit computational acceleration? These questions are still open and will continually explore them in future work.
12

---

Published as a conference paper at ICLR 2022
Table 7: Different rums details of test aceuracy comparisons CIFARIO dataset using ResNet56.
Our method generally achieves better performance and validates the proposed DLTH.
ResNet56 CIFAR10: Bascline accuracy: 93.50%
Pruning ratio 50 %
Run #
Run # 2
Run #
Mean
L1 Li et al. (2016)
93.40
93.22
93.36
93.33:10.08
LTH Frankle & Carbin (2018)
92.99
92.65
92.37
92.67:1..25
Scratch
92.47
99.07
92.93
92.49+0.35
RST (ours)
92.43
97.
92.42
92.34
Pruning ratio 70%
Run #
Run # 2
Run # 3
Mea
L1 Li et al. (2016)
93.26
92.31
92.93
92.83110339
LTH Frankle & Carbin (2018)
92.34
91.49
91.82
91.88.10.35
"Srracch"
92.30
92.07
9[.84
92.14++0277
RST (ours)
92.26
91.98
92.57
92.27710.4
Pruning ratio 90%
Run
Run #
Run
Mta
LI Li et al. (2016)
91.62
91.57
91.82
91.67.10.11
LTH Frankle & Carbin (2018)
89.30
89.89
14
89.7810.35
Scratch
89.97
89.73
89.98
89.8940.12
RST (ours)
90.39
90.35
90.48
90.4110.05
Pruning ratio 95%
Run #
Run #
Run # 3
Sta
L1 Li ct al. (2016)
90.19
90.35
89.85
90.13+0.21
LTH Franklc & Carbin (2018)
88.41
87.35
88.40
88.05+0..50
Scratch
87.66
86.97
87.59
87.41+0.31
RST (ours)
88.36
88.21
88. 16
88.2440..08
Pruning ratio 98%
Run #
Run # 2
Run # 3
Mca
L1 Li ct al. (2016)
85.15
84.67
84.52
84.78:10277
LTH Frankle & Carbin (2018)
84.11
84.35
83.08
83.85+0.55
Scratch
82.82
82.17
83. 14
82.71+10.40
RST (ours)
83.94
83.133
34.24
"83.7710.47
13

---

Published as a conference paper at ICLR 2022
Table 8: Different runs details of test accuracy comparisons CIFARI00 dataset using ResNet56.
Our method generally achieves better performance and validates the proposed DLTH.
ResNet56 CIFARI00: Baseline accuracy: 72.54%
Pruning ratio 50 %
Run #
Run # 2
Run #
Mean
L1 Li et al. (2016)
71.95
71.85
72.09
71.96:1.100
LTH Frankle & Carbin (2018)
70.08
69.32
70.46
69.9510.47
Scratch
71.18
7T1.99
70.61
70.96+0..5
RST (ours)
70.56
771.74"
71.08
77.13340.48
Pruning ratio 70%
Run #
Run #
Run #
MeaniiStd
L1 Li et al. (2016)
71.35
71.54
71.87
71.5911.21
LTH Frankle & Carbin (2018)
69.08
67.95
67.70
68.24:10.60
"Srracch"
68.6
69.01
68. 16
68.59++0.3
RST (ours)
69.81
69.59
76.16
69.8510.23
Pruning ratio 90%
Run
Run #
Run 3
Mean
LI Li et al. (2016)
68.55
68.25
68.07
68.2911.200
LTH Frankle & Carbin (2018)
65.08
66.23
65.67
65.66.10.47
Scratch
64.35
64.18
65.34
64.62+0.32
RST (ours)
66.42
65.98
66.
66.17110.18
Pruning ratio 95%
Run #
Run #
Run #
Sca
L1 Li ct al. (2016)
64.57
64.61
65.05
64.74+
LTH Franklc & Carbin (2018)
60.55
61.22
61.15
60.97++0.30
Scratch
59.89
59.66
60.24
59.93++024
RST (ours)
61.77
62.05
61.17
61.66.10.37
Pruning ratio 98%
Run #
Run # 2
Run #
Mcan
L1 Li ct al. (2016)
52.19 19
48.88
49.04
50.04:1..2
LTH Frankle & Carbin (2018)
53.38
52.53
52.39
52.77+0.44
Scratch
51.36
51.00
50.05
50.80-10.55
RST (ours)
53.84
553.85
34.63
55..1140.37
14

---

Published as a conference paper at ICLR 2022
Pruning raflo.0.5
Pruning (a16:0.77
Pruniig ralio 0.9
Pruning refo:3.05
Pruning rallo.0.38
90
90
85
85
05
85
50
80
X 80
70
75
75
76
70
65
70
LTH DL
70
LTH DLTH
70
LTH DLTH
63
LTH DLTH
60
LTH
OLTH
08
65
65
80
55
150
160
160
Epoch
Epoch
Epoch
Epoch
Epoch
Figure 5: Visualization of test accuracy comparison of LTH and DLTH on CIFARIO using ResNet56. Our DLTH general outperforms LTH and achieves comparable performance for 98% pruning ratio.
Pruning
Pruning
Pruning rato:0.9
Fruning rato:0.95
75
70
55
65
65
50
45
1 60
A/4 60
55
ACC
40
50
50
LTx D1
45
30
DLT4
0.TM
DLTH
41
45
40
75
150 Fpoch
200
150
100
150
Epoch
Epoch
Epoch
Figure 6: Visualization of test accuracy comparison of LTH and DLTH on CIFAR100 using ResNet56. Our DLTH general outperforms LTH for all sparsities.
Pruniin 12.
Pruning talo:0
Pruning ralio
Pruning raliot
Pruning ratic,0,96
80
00
75
75
75
70
Scraech
70
Scryeth
70
65
60
Scrench
Di
Ex TH
73
Di T1
Dil TH
86
65
65
60
86
150
150
150
1:0
100
150
Epoch
Epoch
Epoch
Epoch
Epoch
Figure 7: Visualization of test accuracy comparison of training from scratch and DLTH on CIFARIO using ResNet56. Our training strategy consistently surpasses training from scratch on different pruning ratios. The performance gain increases when the pruning ratio becomes larger.
Pruning r.5
Pruning r
Pruning ratio 0.9
Pruning rall:0.05
Pruning rallo.0.95
65
70
65
60
50
65
60
65
55
65
60
45
35
50
Scraihn DL
60
Spratch D4.TH
45
Scrsshh DLTH
40
Scratih DLH
30
Scraahh DL.TH
45
45
40
25
75
1550
160
150
Epoch
Epoch
Epoch
Epoch
Epcch
Figure 8: Visualization of test accuracy comparison of training from scratch and DLTH on CI- 100 using ResNet56. Our training strategy consistently surpasses training from scratch on different pruning ratios. The performance gain increases when the pruning ratio becomes larger.
15

---

Published as a conference paper at ICLR 2022
Overal Sparsty:0.5
Overal Sparshyy3.7
Oversl Sparsiyy
Overall Scorsty
Overal Sparshy
Layervitee Sparity
Layerwite Sponsity
Layereete Spariity
8
Spensity
Layer Layerwise Sparsity
%
8
80
8
8
2
8
50
50
Layer Indox
Layer Indax
Layar Index
Layer Index
Layar Index
Figure 9: ResNet56 layer-wise sparsity ratio of different overall sparsity CIFARIO dataset. These layer-wise ratios are obtained by GraSP Wang et al. (2020a) algorithm. The last layer the final fully-connected layer whose value is relatively special compared with others. The layer-wise spar- sity generally starts from around 50% and increases when layer index increases for different overall sparsity (except for 50% whose layer-wise sparsity are distributed around 50% for all layers).
Overal Spanstyy
Overall Spansty:d.7
Overal Sparsty
Overall Scarsty 0.98
Ovenall Spanity
Sparshy
Layer- was Spanshy
Layor woo Sparsity
80
Layon W.S Spansity
30
30
80
40
8
50
50
Layer
Layer Inders
Layer Index
Layer
Layer Index
Figure 10: ResNet56 layer-wise sparsity ratio of different overall sparsity CIFAR100 dataset. These layer-wise ratios are obtained by GraSP Wang et al. (2020a) algorithm. The last layer the final fully-connected layer whose value is relatively special compared with others. The layer-wiss sparsity generally starts from around 50% and increases when layer index increases for different overall sparsity (except for 50% whose layer-wise sparsity distributed around 50% for all layers).
16


Reviews:
[{"review_id": "zDbK-BqLcUE", "reviewer": "Reviewer_W7x5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\n- The method is well-motivated. The sparse training techniques are now more and more important as the model sizes are increasing.\n- The paper is overall clearly written, and the concept is easy to understand.\n- The experiment results show the superiority of the proposed method compared to other methods like vanilla LTH. \n\nCons:\n\n- I am not sure if this work can be seen as a complement/supplement/extension to the LTH. One key point in LTH is \"trained in isolation\", which means that the sparse sub-networks should be extracted out. In this work, however, the subnetwork emerges late, and all the connections in models are alive during the training. Also, there is also no such phase as re-training. Based on the comments above, I think this paper is more like a pruning paper aiming at compressing models and training models at the same time. \n- If the above comments are correct, then I think the authors should provide a comparison between existing compressing methods. More specifically, pruning-after-training methods seem to be the correct way to look at. \n- Some notions should be explicitly defined and some need correction. For example, the \\bar{\\theta^*} was never pre-defined. Also, the second condition in Equation (4) may never be satisfied. Changing \"=\" to be \"\\lt\" would be better. \n\nMinor: Some citation formats should have parentheses. Consider using \\citep.\n\n \n\n", "summary_of_the_paper": "The paper proposes a new pruning method that resembles distillation - keeping pruned connections within the pruned network and using them when generating the models' output. The method works better than the currently popular topic - LTH, and works better than some pruning-at-initialization methods such as GraSP. ", "summary_of_the_review": "My final score will be mostly based upon discussion with authors regarding my points outlined above. Currently, I think the paper is somehow off-topic, but I am open to hearing from the authors and willing to change my view if I had made mistakes. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lWfAa_KeCIv", "reviewer": "Reviewer_mwor", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of exploring a transferable LTH is interesting, which is complementary to the original LTH. The empirical study is clean and shows promising results on some datasets. In addition, the paper also compares DLTH with other techniques such as Pruning at Initialization (PI). The general writing is good. \n\nHowever, I have a few concerns about the current version:\n1. The empirical study is only on small backbones and datasets, how about larger ones (ImageNet, etc.)?\n2. The paper only introduces the regularization method to transform the RST to winning tickets. It is better to have a deep dive into what kind of approaches could work. \n3. The paper ignores some related but important works, such as [a,b,c]. It would be better to compare with them, e.g., RigL. \n    a. Rigging the lottery: Making all tickets winners.\n    b. The Elastic Lottery Ticket Hypothesis.\n    c. Drawing early-bird tickets: Toward more efficient training of deep networks.", "summary_of_the_paper": "This paper studies the Lotter Ticket Hypothesis (LTH) and proposes a Dual Lottery Ticket Hypothesis (DLTH). DLTH describes that any ticket in a given lottery pool can be transformed into a winning ticket. The paper uses a regularization-based method for this transformation. The experiments have been conducted on CIFAR10/100 with ResNets, which indicates a consistent empirical result with DLTH. ", "summary_of_the_review": "Overall, this is an interesting and good paper. If the authors can address my concerns in the rebuttal phase. I will consider making the recommendation for the paper. \n\n-----------------------------------------------------------------------------------------\nMost of my concerns have been addressed by the authors. I change my score and recommendation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kF0hPb079vR", "reviewer": "Reviewer_XCnY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Two main concerns I have on this work are the way it is posed and the writing issue.\n\nFor how this work is posed, as the authors mentioned in multiple places in the write-up, DLTH is about a proper \"transformation\" of any ticket in the lottery pool into the winning ticket. Correspondingly, the authors proposed the RST as the transformation above to substantiate DLTH. However, according to the formulation in eqn. (3) and (4), Random Sparse Network Training seems to a better name for RST as far as I can see because it looks more like a general (potentially a uniformly better one) training method for sparse neural networks from scratch. For example, we could for sure apply RST to the winning tickets found by IMP directly. Will combining IMP + RST give us a better performance than RST + any tickets? This experiment will give us a clue of which one of the following is more true (or both not true):\n\n1) a winning ticket achieves the best performance regardless of the training method and any transformed ticket with RST can not achieve better; OR\n2) the winning tickets are still better subnetworks and can also be improved in terms of accuracy with RST?\n\nIf the first case is more true, then the contribution of this work would be much more interesting to me. If the second is more true, however, I think this work is not posed well. It is not about transforming any ticket into a winning one, but about proposing a general better training method for sparse neural networks. Let alone that the core regularization technique in RST is not new.\n\nAnother concern of mine is the writing. Firstly, the introduction and discussion of literature before Section 4 seem to be too long to me. Secondly, Section 3 is so disconnected with Section 4 and Section 5. Even the notations are not consistent in Section 3 and 5. The authors made efforts to explicitly define the transformation of weight and mask but lacked discussion about them in Section 4 and 5, like how RST is related to them, making RST just a training technique with regularization, totally losing connection to Section 3. I think this issue is also related to my first concern about the posing of this work. My suggestion is to shorten the contents before Section 3 and add richer discussion in Section 3, 4 and 5 to make them more logically consistent and connected.", "summary_of_the_paper": "This work proposes the so called Dual Lottery Ticket Hypothesis which claims that every ticket in the ticket pool, i.e., every random sparse subnetwork in a randomly initialized dense neural network, can be transformed into a winning ticket with admirable trainability. This \"transformation\" proposed in the notation of Random Sparse Network Transformation (RST), in practice, comes in with the form of a squared L2 norm regularization on the masked weights, making them still involved during training while extruding the information contained in them and transferring into the unmasked ones. The empirical experiments show good empirical performance of RST compared to the vanilla LTH, pruning-at-initialization methods and LTH variants such as EB-LTH.", "summary_of_the_review": "My evaluation for this work is that it is below the acceptance threshold, but I tend to give a score 4 (but I don't have option). Major reason is the posing of the DLTH claim and the contribution. I also think the writing could be fundamentally improved. I still think the technique and the idea are interesting and encourage the authors to try the next venue with enough improvements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NmiYkfFEkV", "reviewer": "Reviewer_jr1G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The concept of the Dual Lottery Ticket Hypothesis is interesting and worth studying.\n2. The construction of the paper is clear and easy to follow. \n\nWeakness:\n\n1. As formulated in the abstract, the DLTH demonstrates that any ticket in a given lottery pool can be transformed into a winning ticket. And the definition of winning tickets is a subnetwork, which can be trained from scratch and match the performance of the dense counterpart. But on CIFAR-10/100 and ImageNet with a 50-98% pruning ratio, the performance of all the subnetworks still remains a unignorable gap against the dense model. Personally, I think it is more meaningful to investigate the proposed method in the setting when subnetworks identified by both LTH and DLTH are winning tickets.\n\n2. As suggested in [1], the connections of neurons in individual layers in initial tickets can be totally rearranged without performance drop, and only the layer-wise sparsity level matters. The RST method follows the same layer-wise sparsity ratio derived by GraSP. Is the proposed RST still effective when the layer-wise sparsity level is also randomly arranged?\n\n3. The related work about sparse network training needs to be re-organized, which only tells several works without any logic.\n\n[1] Su et al. Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot, NeurIPS 2020\n\nMinor:\n\n1. The caption of Figure 3 covers part of the y label.", "summary_of_the_paper": "This paper extends the Lottery Ticket Hypothesis(LTH) to a more challenging case and proposes the concept of Dual LTH: Any randomly selected subnetwork of a randomly initialized dense network can be transformed into a winning ticket. And the authors propose Random Sparse Network Transformation(RST) to accomplish the transformation process. Experiment results on CIFAR-10/00 and part of ImageNet with ResNet-18/56 demonstrate the effectiveness of the RST and validate the Dual LTH.", "summary_of_the_review": "Overall, it's interesting to investigate the proposed Dual Lottery Ticket Hypothesis. However current results are a little bit insufficient to validate the DLTH. Specifically, no winning tickets have been identified in current experiments and whether DLTH can be extended to the setting when layer-wise sparsity level can also to randomly arranged.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HK8OlSydXIS", "reviewer": "Reviewer_vhgZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major Pros:\nThat randomly selected sub-networks, can attain same performance as dense and comparable sparsity IMP derived LT has reached significant attention. The main idea presented here at tackling how a random network may be 'transformed' via RST is interesting.\n\n\nMajor Cons:\n1. The main claim: \" *any* randomly picked subnetwork .... \" is incorrect. Examples where such a claim will fail are easy to construct: a choice of subnetwork which is disconnected, or lost its residual connection(s) or is woefully bottlenecked ...\n\n2. The results with respect to cycle dependency are very noisy raising suspicion with respect to their statistical significance and/or the representativeness of the regime in which they were conducted. Why are they noisy (e.g. figure 5).\nSpecifically, for high sparsity levels --- performance in iterative magnitude pruning depends strongly on number of iterations (I assume this is what is meant by cycles?).\n\nStyle and clarity (not affecting score):\nThroughout the paper much is left to the reader to infer burdening the clarity of the paper and its decipherability:\n- At the onset of the main claim: what is a \" trainable condition \"\n- Information Extrusion mentioned but not defined (as early as the abstract), only to be exemplified via implementation in section 5.\n- 'Cycles' are not defined\n- \"As mentioned in implementation section, the extra training cost can be calculated...\" the reader is referred to an equation in the Discussion of DLTH v. LTH section of the optimization cost , but no such equation is to be found in that reference. \n\n\n\n*Updated review in light of author responses [Nov 26, 2021]:*\n\nI have reviewed the updated manuscript following several iterations with the authors:  https://anonymous.4open.science/r/DLTH_Updated-3020/2022_ICLR_DLTH_Updated.pdf.\n\nThe updated manuscript changes the erroneous original claims (\"any network can be transformed to trainable condition\") and precisely modifies and scopes them (\"random ticket wins\"). In particular, giving room for a key novelty in the paper in the form of RST.\n\nIssues with respect to the regime in which the conclusions are made remain, including an insufficient analysis of the sensitivity to iteration and duration of extrusion. These issues *should be expanded further in the final manuscript by additional experiments* in the regime where sensitivity is more pronounced (e.g. reaching higher sparsity levels where error materially increases --- and high sensitivity to pruning details is typically exhibited). Some concerns with respect to the generalizability of these results also remain.\n\nHowever, with the updated discussions and revisions as well as the correction of the core claims, the paper rises to the level of acceptance --- assuming addressing the above issues, which the authors demonstrate willingness to do.\nI thus accordingly update my score and thank the authors for their efforts in addressing concerns and for the open and constructive discussion and modifications. \n", "summary_of_the_paper": "The paper claims that \"Any randomly picked subnetwork in a dense randomly initialized neural network can be transformed into a trainable condition, where the subnetwork can be trained in isolation and achieves better at least comparable performance to LTH and other strong baselines.\"\n\nIn order to demonstrate how such a transformation may be achieved the paper introduces RST which gradually penalizes the magnitude (L2) of the non-selected parameters during training (upon which they are zeroed out explicitly) in order to arrive at the randomly selected sparse sub-network. \n\n", "summary_of_the_review": "The paper's main claim as presented seems to be false --- it is expected not to hold true that *any* randomly picked subnetwork is transformable to a network as performant as the original dense network if trained. \nFor the particular method presented (RST) dependency on cycles leaves utility / insight unclear.\n\n*Updated review in light of author responses [Nov 26, 2021]:* see above update. claims corrected, RST novel contribution, experiments for high sparsity in order to address high-sensitivity regime expected. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend this paper for acceptance but I do so with significant reservations. Since this metareview will be public for all time, I direct this metareview to future readers of this paper so that they can weigh its merits and drawbacks in a clear-minded way.

This paper proposes a "dual lottery ticket hypothesis." For those unfamiliar, the original lottery ticket hypothesis (Frankle & Carbin, ICLR 2019) states approximately that any randomly initialized neural network contains a subnetwork that can be trained in isolation to full accuracy in the same number of steps as the original network. That is, $\forall$ neural networks, $\exists$ a subnetwork such that $Accuracy(Train($subnetwork$)) \geq Accuracy(Train($network$))$ for a standard, fixed training procedure $Train$. (For the sake of posterity, note that this claim was supported on small-scale neural networks but there is not evidence that it holds in general; only that it holds on the state of networks *early* in training. See *Linear Mode Connectivity and the Lottery Ticket Hypothesis* by Frankle et al. 2020.) To support this claim, Frankle & Carbin develop a procedure that finds such subnetworks, demonstrating that they exist in certain settings.

As far as I understand, the dual lottery ticket hypothesis states that, $\forall$ subnetworks of a neural network, $\exists$ a setting of the weights such that $Accuracy(Train($subnetwork$)) \geq Accuracy(Train($network$))$. Like the original lottery ticket paper, this paper shows that such subnetworks exist: it trains the subnetwork with an L2 penalty on all of the weights except those of the subnetwork, allowing them to gradually fade away and leaving a new setting of the weights for the subnetwork that then allows it to train in isolation to full accuracy (like those subnetworks found in the original lottery ticket hypothesis paper).

The reason that I have reservations about this approach is that the subnetwork found by the dual lottery ticket hypothesis procedure contains fully trained weights. This is novel but - to me - much less surprising and interesting: a randomly sparse subnetwork can be set with trained weights such that, after all of the other weights are fully pruned away, it can recover full accuracy. On the one hand, this is almost reminiscent of a standard pruning procedure where the network is both trained and pruned until a sparse subnetwork reaches full accuracy, with the dense network needed for much or all of training. On the other hand, the impressive part is that this can be done with a *randomly selected* sparse network rather than one chosen by a pruning heuristic. To me, that is the most interesting part of the paper. (And, for those readers wondering why specifically this paper is distinct from standard pruning, this is it.)

I wonder about the significance of this finding given that the subnetwork is set by training (not by random initialization or a tiny amount of training as in work on the lottery ticket hypothesis), but it's a novel idea and I think future scholars and future research should be the judge of that significance, not me or the reviewers. The novelty alone merits publication, and we will have to wait and see about the significance. Thus, I weigh in favor of acceptance, although with reservations.
Decision: Accept (Poster)