Title: WINNING THE LOTTERY WITH CONTINUOUS SPARSIFICATION

Abstract: The Lottery Ticket Hypothesis from Frankle & Carbin (2019) conjectures that, for typically-sized neural networks, it is possible to find small sub-networks which train faster and yield superior performance than their original counterparts. The proposed algorithm to search for "winning tickets", Iterative Magnitude Pruning, consistently finds sub-networks with 90 -95% less parameters which train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning. In this paper, we propose Continuous Sparsification, a new algorithm to search for winning tickets which continuously removes parameters from a network during training, and learns the sub-network's structure with gradient-based methods instead of relying on pruning strategies. We show empirically that our method is capable of finding tickets that outperforms the ones learned by Iterative Magnitude Pruning, and at the same time providing faster search, when measured in number of training epochs or wall-clock time.

1. INTRODUCTION:

Although deep neural networks have become ubiquitous in fields such as computer vision and natural language processing, extreme overparameterization is typically required to achieve state-ofthe-art results (Xie et al., 2017; Devlin et al., 2018) , causing higher training costs and hindering applications where memory or inference time are constrained. Recent theoretical work suggest that overparameterization plays a key role in both the capacity and generalization of a network (Neyshabur et al., 2018) , and in training dynamics (Allen-Zhu et al., 2019) . However, it remains unclear whether overparameterization is truly necessary to train networks to state-of-the-art performance.

At the same time, empirical approaches have been successful in finding less overparameterized neural networks, either by reducing the network after training (Han et al., 2015; 2016) or through more efficient architectures that can be trained from scratch (Iandola et al., 2016) . Recently, the combination of these two approaches lead to new methods which discover efficient architectures through optimization instead of design (Liu et al., 2019; Savarese & Maire, 2019) . Nonetheless, parameter efficiency is typically maximized by pruning an already trained network.

The fact that pruned networks are hard to train from scratch (Han et al., 2015; 2016) suggests that, while overparameterization is not necessary for a model's capacity, it might be required for successful network training. Recently, this idea has been put into question by Frankle & Carbin (2019) , where heavily pruned networks are trained faster than their original counterparts, often yielding superior performance.

A key finding is that the same parameter initialization should be used when re-training the pruned network. A winning ticket, defined by a sub-network and a setting of randomly-initialized parameters, is quickly trainable and has already found applications in, for example, transfer learning (Morcos et al., 2019; Mehta, 2019; Soelen & Sheppard, 2019) , making the search for winning tickets a problem of independent interest.

Currently, the standard algorithm to find winning tickets is Iterative Magnitude Pruning (IMP) (Frankle & Carbin, 2019; Frankle et al., 2019) , which consists of a repeating a 2-stage procedure that alternates between parameter optimization and pruning. As a result, IMP relies on a sensible choice for pruning strategy, and is time-consuming: finding a winning ticket with 1% of the original parameters in a 6-layer CNN requires over 20 rounds of training followed by pruning, totalling over 1000 epochs (Frankle & Carbin, 2019) . Choosing a parameter's magnitude as pruning criterion has also shown to be sub-optimal in some settings (Zhou et al., 2019) , leading to the question of whether better winning tickets can be found by different pruning methods. Moreover, at each iteration, IMP resets the parameters of the network back to initialization, hence considerable time is spent on re-training similar networks with different sparsities.

With the goal of speeding up the search for winning tickets in deep neural networks, we design a novel method, Continuous Sparsification, which continuously removes weights from a network during training, instead of following a strategy to prune parameters at discrete time intervals. Unlike IMP, our method approaches the search for sparse networks as a 0 -regularized optimization problem (Louizos et al., 2017) , resulting in a method that can be fully described in the optimization framework. To approximate 0 -regularization, we propose a smooth re-parameterization, allowing for the subnetwork's structure to be directly learned with gradient-based methods. Unlike previous works, our re-parameterization is deterministic, proving more convenient for the tasks of pruning and ticket search, while also yielding faster training times.

Experimentally, our method offers superior performance when pruning VGG to extreme regimes, and is capable of finding winning tickets in Residual Networks trained on CIFAR-10 at a fraction of time taken by Iterative Magnitude Pruning. In particular, Continuous Sparsification successfully finds tickets in under 5 iterations, compared to 20 iterations required by Iterative Magnitude Pruning in the same setting. To further speed up the search for sub-networks, our method abdicates parameter rewinding, a key ingredient of Iterative Magnitude Pruning. By showing superior results without rewinding, our experiments offer insights on how ticket search should be performed.

2. RELATED WORK:



2.1. LOTTERY TICKET HYPOTHESIS:

The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) states that for a network f (x; w), w ∈ R d , and randomly-initialized parameters w 0 ∼ D, there exists a sparse sub-network, defined by a configuration m ∈ {0, 1} d , m 0 d, that, when trained from scratch, achieves higher performance than f (x; w) while requiring fewer training iterations. The authors support this conjecture experimentally, showing that such sub-networks indeed exist: in particular, they can be discovered by repeatedly training, pruning, and re-initializing the network, through a procedure named Iterative Magnitude Pruning (IMP; Algorithm 1) (Frankle et al., 2019) . More specifically, IMP alternates between: (1) training the weights w of a network, (2) removing a fixed fraction of the weights with the smallest magnitude (pruning), and (3) rewinding: setting the remaining weights back to their original initialization w 0 .

The sub-networks found by IMP, which indeed train faster and outperform their original, dense networks, are called winning tickets, and can generalize across datasets (Mehta, 2019; Soelen & Sheppard, 2019) and training methods (Morcos et al., 2019) . In this sense, IMP can be a promising tool in applications that involve knowledge transfer, such as transfer or meta learning. Zhou et al. (2019) perform extensive experiments to re-evaluate and better understand the Lottery Ticket Hypothesis. Relevant to this work is the fact that the authors propose a method to learn the binary mask m in an end-to-end manner through SGD, instead of relying on magnitude-based pruning. The authors show that learning only the binary mask and not the weights is sufficient to achieve competitive performance, confirming that the learned masks are highly dependent on the initialized values w 0 , and are also capable of encoding substantial information about a problem's solution.

2.2. SPARSE NETWORKS:

The core aspect of searching for a winning ticket is finding a sparse sub-network that attains high performance relative to its dense counterpart. One way to achieve this is through pruning methods (LeCun et al., 1990) , which follow a strategy to remove weights from a trained network while minimizing negative impacts on its performance. In Han et al. (2015) , a network is iteratively trained and pruned using parameter magnitudes as criterion: this iterative, two-stage algorithm is shown to outperform "one-shot pruning": training and pruning the network only once.

Other methods attempt to approximate 0 regularization on the weights of a network, yielding onestage procedures that can be fully described in the optimization framework. In order to find a sparse Algorithm 1 Iterative Magnitude Pruning (Frankle et al., 2019) 1: Initialize w ← w 0 ∼ D and m ← 1 d 2: Minimize L(f (x; m w)) until w T is produced 3: Set m i = 0 for the active weights with smallest magnitudes (|w T,i | ≤ τ and m i = 1) 4: If satisfied, output ticket f (x; m w k ) 5: Otherwise, set w ← w k and go back to step 2 Algorithm 2 Iterative Stochastic Sparsification (inspired by Zhou et al. (2019) )

1: Initialize w ← w 0 ∼ D, s ← s 0 2: Minimize E m∼Ber(σ(s)) [L(f (x; m w))] + λ σ(s) 1 until w T and s T are produced 3: If satisfied, output ticket f (x; m w k ), m ∼ Ber(σ(s T )) 4: Otherwise, set w ← w k , s i ← -∞ for s i,T < s i,0

, and go back to step 2 setting m ∈ {0, 1} d of a network f (x; m w), Srinivas et al. (2016) and Louizos et al. (2017) use a stochastic re-parameterization m ∼ Bernoulli(g(s)) with s ∈ R d and g : R → [0, 1] applied element-wise. First-order methods, coupled with gradient estimators, are then used to train both w and s to minimize the expected loss. This approach performs continuous parameter removal during training in an automatic fashion: any component s i of s that assumes a value during training where g(s i ) = 0 effectively removes w i from the network. Moreover, approximating 0 regularization has the advantage of not requiring a pruning strategy, which might be arbitrarily complex.

3. METHOD:

Designing a method to quickly find winning tickets requires an efficient way to sparsify networks: ideally, sparsification should be done as early as possible in training, and the number of removed parameters should be maximized without harming the model's performance. In other words, sparsification must be continuously maximized following a trade-off with the performance of the network. This goal is not met by Iterative Magnitude Pruning: sparsification is done at discrete time steps, only after fully training the network, and optimal pruning rates likely depend on the model's performance and current sparsity: factors which are typically not accounted for -note that these are inherent characteristics of magnitude-based pruning.

In light of this, we turn to 0 -regularization methods for learning sparse networks, which consist of optimizing a clear trade-off between sparsity and performance. As we will see, performing sparsification continuously is not only straightforward, but done automatically by the optimizer.

3.1. CONTINUOUS SPARSIFICATION BY LEARNING DETERMINISTIC MASKS:

We first frame the search for sparse networks as a loss minimization problem with 0 regularization (Louizos et al., 2017; Srinivas et al., 2016) :

min w∈R d L(f (x; w)) + λ • w 0 (1)

where λ ≥ 0 controls the sparsity of the solution, and, with a slight abuse of notation, L(f (x; w)) denotes the loss incurred by the network f (x; w) (e.g., the cross-entropy loss over a training set). As 0 regularization is typically intractable, we re-state the above minimization problem as:

min w∈R d m∈{0,1} d L(f (x; m w)) + λ • m 1 (2)

which uses the fact that, for m ∈ {0, 1} d , m 0 = m 1 . The 1 term can be minimized with subgradient descent, however the m ∈ {0, 1} d constraint makes the above problem combinatorial and poorly suited for local search methods like SGD.

Algorithm 3 Continuous Sparsification

1: Initialize w ← w 0 ∼ D, s ← s 0 , β ← β 0 2: Minimize L(f (x; σ(β • s) w)) + λ σ(βs) 1 while increasing β, producing w T , s T , and β T 3: If satisfied, output ticket f (x; b(s T ) w k ) 4: Otherwise, set s ← min(β T • s T , s 0 ), β ← β 0 , (optionally, w ← w 0 )

, and go back to step 2

-2.5 0.0 2.5 z -0.5 0.0 0.5 1.0 1.5 b(z) -2.5 0.0 2.5 z b (z) -2.5 0.0 2.5 z σ(βz) β = 200 β = 4 β = 1 -2.5 0.0 2.5 z σ (βz)

Figure 1 : Illustration of our proposed re-parameterization m = σ(βs), where σ(z) = 1 1+e -z is the sigmoid function and β acts as a temperature. As β increases, σ(βz) approaches b(z), which can can be used to frame a 0 -regularized problem (Equation 4). Note that the gradients of σ(βs) vanish as β increases, suggesting that β should be annealed slowly during training.

We can avoid the binary constraint m ∈ {0, 1} d by re-parameterizing m as a function of a newlyintroduced variable s ∈ R d . For example, Louizos et al. (2017) propose a stochastic mapping s → m and use gradient methods to minimize the expected total loss, while using estimators for the gradients of s (since m is still binary). Having a stochastic mask (or, equivalently, a distribution over subnetworks) poses an immediate challenge for the task of finding tickets, as it is not clear which ticket should be chosen once a distribution over m is learned. Moreover, relying on gradient estimators often causes gradients to have high variance, requiring longer training to reach optimality. Alternatively, we consider a deterministic parameterization m = b(s), where s ∈ R d =0 and b : R =0 → {0, 1} is applied element-wise:

EQUATION

Applying this re-parameterization to Equation 2 yields:

min w∈R d s∈R d =0 L(f (x; b(s) w)) + λ • b(s) 1 (4)

Clearly, the above problem is again intractable, as it is still equivalent to the original 0 problem in Equation 1. More specifically, the step function b(z) is non-convex, and having zero gradients make gradient-based optimization ineffective. Instead, we consider the following smooth relaxation of b(•):

m := σ(β • s) (5)

where β ∈ R >0 , and σ is the sigmoid function σ(z) = 1 1+e -z , applied element-wise. By controlling β, which acts as a temperature parameter, we effectively interpolate between σ(s), a smooth function well-suited for SGD, and lim β→∞ σ(β • s) = b(z), our original goal, which brings computational hardness to the problem. Figure 1 illustrates this behavior. Note that, if L(f (x; w)) is continuous in w, then:

min w∈R d s∈R d =0 lim β→∞ L(f (x; σ(βs) w)) + λ • σ(βs) 1 = min w∈R d s∈R d =0 L(f (x; b(s) w)) + λ • b(s) 1 (6)

Although gradient methods will become ineffective as β → ∞ due to vanishing gradients of s, we can increase β while optimizing s and w with gradient descent. That is, our loss at each iteration will be a function of β as follows:

L β (w, s) = L(f (x; σ(βs) w)) + λ • σ(β • s) 1 (7)

How does the soft mask m = σ(β • s) behave as we minimize L β (w, s) while increasing β? As β → ∞, every negative component of s will be mapped to 0, effectively removing its correspondent weight parameter from the network. While analytically the weights will never truly be zeroed-out, limited numerical precision has the fortunate side-effect of causing actual sparsification to the network during training, as long as β is increased to a large enough value.

In a nutshell, we learn sparse networks by minimizing L β (w, s) for T parameter updates with gradient descent while jointly annealing β: producing w T , s T and β T , which is ideally large enough such that,

numerically 1 , σ(β T • s T ) = b(s T ).

In case m is truly required to be binary (as in the task of finding tickets), the dependence on numerical imprecision can be avoided by directly outputting m = b(s T ) at the end of training.

Finally, note that minimizing L β while increasing β is not generally equivalent to minimizing the original 0 -regularized problem. Informally, the former aims to solve lim β→∞ min w,s L β (w, s), while the 0 problem is min w,s lim β→∞ L β (w, s).

3.2. TICKET SEARCH THROUGH CONTINUOUS SPARSIFICATION:

The method presented above offers a direct alternative to magnitude-based pruning when performing ticket search, but a few considerations must follow. Most importantly, when searching for winning tickets, there is a strict constraint that the learned mask m be binary: otherwise, one can also learn the magnitude of the weights, defeating the purpose of finding sub-networks that can be trained from scratch. To guarantee that the output mask satisfies this constraint regardless of numerical precision, we always output b(s T ) instead of σ(β T • s T ).

Additionally, we also incorporate two techniques from successful methods for learning sparse networks and searching for winning tickets. First, motivated by Han et al. (2015) , where it is shown that iteratively pruning a network yields improved sparsity compared to pruning it only once, we enable "kept" weights -those whose corresponding component of s is positive after many iterationsto be removed from the network at a later stage. More specifically, when β becomes large after T gradient descent updates, the gradients of s vanish and weights will no longer be removed from the network. To avoid this, we set s ← min(β T • s T , s 0 ), effectively resetting the soft mask parameters for the remaining weights while at the same time not interfering with weights that have been removed. This is followed by a reset on the temperature, β ← β 0 , to allow training of s once again.

Second, we perform parameter rewinding, following Frankle & Carbin (2019) , which is a key component of Iterative Magnitude Pruning. More specifically, after T gradient descent steps, we reset the weight values back to an earlier stage w ← w k , where k T . Even though experimental results in Frankle & Carbin (2019) suggest that rewinding is necessary for successful ticket search, we leave rewinding as an optinal component of our algorithm: as we will see empirically, it turns out that ticket search is possible without rewinding weights. Our proposed algorithm to find winning tickets is presented as Algorithm 3, and referred simply as "Continuous Sparsification".

4. EXPERIMENTS:

Our experiments aim at comparing different methods on the task of finding winning tickets in neural networks, hence our evaluation focuses on the generalization performance of each ticket (sub-network) when trained from scratch (or from an iterate in early-training). Additionally, we measure the cost of the search procedure: the number of training epochs to find tickets with varying performance and sparsity.

Besides comparing our proposed method to Iterative Magnitude Pruning (Algorithm 1), we also design a baseline method, Iterative Stochastic Sparsification (ISS, Algorithm 2), motivated by the procedure in Zhou et al. (2019) to find a binary mask m with gradient descent in an end-to-end fashion. More specifically, ISS uses a stochastic re-parameterization m ∼ Bernoulli(σ(s)) with s ∈ R d , and trains w and s jointly with gradient descent and the straight-through estimator (Bengio et al., 2013) . When ran for multiple iterations, all components of the mask parameters s which have decreased in value from initialization are set to -∞, such that the corresponding weight is permanently removed from the network. While this might look arbitrary, we observed empirically that ISS was unable to remove weights quickly without this step unless λ was chosen to be large -in which case the model's performance decrease in exchange for sparsity. The hyperparameters used in this section were chosen based on analysis presented in Appendix (...), where we study how the pruning rate affects IMP, and how λ, s 0 and β T interact in CS.

4.1. CONVOLUTIONAL NEURAL NETWORKS:

We train a neural network with 6 convolutional layers on the CIFAR-10 dataset (Krizhevsky, 2009) , following Frankle & Carbin (2019) . The network consists of 3 blocks of 2 resolution-preserving convolutional layers followed by 2 × 2 max-pooling, where convolutions in each block have 64, 128 and 256 channels, a 3 × 3 kernel, and are immediately followed by ReLU activations. The blocks are followed by fully-connected layers with 256, 256 and 10 neurons, with ReLUs in between. The network is trained with Adam (Kingma & Ba, 2015) with a learning rate of 0.0003 and a batch size of 60.

Learning a Supermask: As a first baseline, we consider the task of learning a "supermask" (Zhou et al., 2019) : a binary mask m that, when applied to a network with randomly initialized weights, yields performance competitive to that of training its weights. This task is equivalent to pruning a randomly-initialized network, or learning an architecture that performs well prior to training with a fixed initialization. We compare ISS and CS , where each method is run for a single iteration composed of 100 epochs. When ran for a single iteration, ISS is equivalent to the algorithm proposed in Zhou et al. (2019) to learn a supermask, referred here as simply Stochastic Sparsification. We control the sparsity of the learned masks by varying s 0 between -5 and 5 for Stochastic Sparsification (which showed to be more effective than varying λ), while for Continuous Sparsification we vary λ between 10 -11 and 10 -7 (which results in stable and consistent training, unlike varying s 0 ). SS uses SGD with a learning rate of 100 to learn its mask parameters, while CS uses Adam with 3 × 10 -4 .

Results are presented in Figure 2 : CS outperforms SS in terms of both training speed and the quality of the learned mask. In particular, CS finds masks with over 75% sparsity that yield over 75% test accuracy, while the performance of masks found by SS decrease when sparsity is over 50%. Moreover, CS makes faster progress in training, showing that optimizing a deterministic mask is indeed faster than learning a distribution over masks through stochastic re-parameterizations.

Finding Winning Tickets: We run IMP and ISS for a total of 30 iterations, each consisting of 40 epochs. Parameters are trained with Adam (Kingma & Ba, 2015) with a learning rate of 3 × 10 -4 , following Frankle & Carbin (2019) . For IMP, we use pruning rates of 15%/20% for convolutional/dense layers. We initialize the Bernoulli parameters of ISS with s 0 = 1, and train them with SGD and a learning rate of 20, along with a 1 regularization of λ = 10 -8 . For CS , we anneal the temperature from β 0 = 1 to β 0 = 250 following an exponential schedule (β t = 250 t T ), training both the weights and the mask with Adam and a learning rate of 3 × 10 -4 .

To test whether our method is capable of finding winning tickets in a limited amount of time, we limit each run of CS to 4 iterations only, in contrast with IMP and ISS which are run for 30. We perform 6 runs of CS , each with a different value for the mask initialization s 0 : -0.05, -0.03, -0.02, -0.01, -0.005, 0, keeping λ = 10 -10 , such that sparsification is not enforced during training, but heavily biased at initialization. In order to evaluate how consistent our method is, we repeat each run with 3 different random seeds so that error bars can be computed.

Figure 3 (left) presents the quality of tickets found by each method, measured by their test accuracy when trained from scratch. To illustrate the quality of the tickets that can be found by Continuous Sparsification, we plot the Pareto curve (green) of the tickets founds with the 6 different values for s 0 . With s 0 = -0.03, in only 2 iterations CS finds a ticket with over 77% sparsity (first marker of purple curve) which outperforms every ticket found by IMP in its 30 iterations. The Pareto curve of CS strictly dominates IMP for tickets with more less than 97% sparsity, where ticket performance is superior or similar to the original dense network.

In terms of computational time, the total cost to run CS with the 6 different values for s 0 is lower than performing a single run of IMP for 30 iterations, even though CS takes 15% extra time per epoch due to the mask parameters. This shows the potential of our model even in the setting where a specific sparsity is desired for the tickets. When run in parallel, CS takes less wall-clock time to find all tickets in the Pareto curve than to run IMP for 5 iterations.

4.2. FINDING WINNING TICKETS IN RESIDUAL NETWORKS WITHOUT REWINDING:

Searching for tickets in realistic models is not as straightforward as finding tickets in a small CNN, and might require new strategies. Frankle et al. (2019) show that IMP fails at finding winning tickets in ResNets (He et al., 2016) unless the learning rate is smaller than the recommended value, leading to worse overall performance and defeating the purpose of ticket search. However, the authors propose a slight modification to IMP that enables search for winning tickets to be successful on complex networks: instead of training from scratch, tickets are initialized with weights from early training.

With this in mind, we evaluate how Continuous Sparsification performs in the time-consuming task of finding winning tickets in a ResNet-202 (He et al., 2016) trained on CIFAR-10: a setting where IMP might take over 10 iterations (850 epochs) to succeed. We follow the setup in Frankle & Carbin (2019) and Frankle et al. (2019) : in each iteration, the network is trained with SGD, a learning rate of 0.1, and a momentum of 0.9 for a total of 85 epochs, using a batch size of 128. The learning rate is decayed by a factor of 10 at epochs 56 and 71, and a weight decay of 0.0001 is applied to the weights (for CS , we do not apply weight decay to the mask parameters s). The two skip-connections that perform 1 × 1 convolutions and the output layer are not removable: for IMP, their parameters are not pruned, while for CS their weights do not have a correspondent mask m nor mask parameters s.

When training the returned tickets in order to evaluate their performance, we initialize their weights with the iterates from the end of epoch 2 (780 parameter updates), similarly to Frankle et al. (2019) . Unlike when searching for winning tickets in the 6-layer CNN, IMP performs global pruning, removing 20% of the remaining parameters with smallest magnitude, ranked globally (across different layers). IMP runs for a total of 30 iterations, while CS is limited to only 5 iterations for each run. The sparsity of the tickets found by CS is controlled by varying the mask initialization s 0 ∈ {-0.3, -0.2, -0.1, -0.05, -0.03, 0, 0.03, 0.05, 0.1, 0.2, 0.3} (a total of 11 values). To allow for even faster ticket search, we run CS without parameter rewinding: that is, the weights w are transferred from one iteration to another, removing the need to re-train the network as the method progresses through iterations. For both CS and IMP, each run is repeated with 3 different random seeds.

The results presented in Figure 3 (right) show that CS is able to successfully find winning tickets with varying sparsity in under 5 iterations. Once again, the Pareto curve strictly dominates IMP, and variance across runs is smaller than IMP's. Most notably, CS is capable of quickly sparsifying the network in a single iteration (first marker of each purple curve), and typically finds better tickets than IMP after only 2 rounds (compare blue curve and second marker of each purple curve), regardless of sparsity. When run in parallel, 2 iterations suffice for CS to find tickets that outperform the ones found by IMP.

We observed that not performing rewinding caused the performance of tickets with high sparsity to quickly degrade after 2 or more iterations of CS. We speculate that, when rewinding is not performed between iterations, the distance between w k and the parameter iterates produced by gradient descent w t increase significantly with the number of iterations. This in turn can result in the learned mask m T to be highly sub-optimal for weight values w k (k T ) which are used to re-train the ticket. This suggests that in order to avoid re-training the network and hence make the search for winning tickets more efficient, rewinding should not be performed between iterations. In this case, the search must complete quickly, before performance degradation occurs due to "overtraining", requiring optimal ways to perform sparsification without negatively impacting the model's performance.

4.3. PRUNING VGG:

Our experiments show that Continuous Sparsification is capable of finding tickets quickly and consistently, and we attribute its success to its deterministic re-parameterization of the binary mask. Here, we evaluate our method a pruning technique, to better assess whether our proposed re-parameterization is advantageous only in terms of training time, or also in respect to the quality of the learned masks.

For this task, we train a VGG (Simonyan & Zisserman, 2015) on the CIFAR-10 dataset, following the protocol in Frankle & Carbin (2019) : the network is trained with SGD and an initial learning rate of 0.1, which is decayed by a factor of 10 at epochs 80 and 120. After 160 training epochs, the network is sparsified and then fine-tuned for 40 epochs with a learning rate of 0.001. We evaluate previously described methods when executed for a single iteration (one-shot pruning): Continuous Sparsification, Magnitude Pruning (IMP with 1 iteration) (Han et al., 2015) , and Stochastic Sparsification (ISS with 1 iteration), which is similar to methods in Zhou et al. (2019) , Srinivas et al. (2016), and Louizos et al. (2017) . 100.0 51.4 26.5 13.7 7.1 3.7 1.9 1.0 0.5 0.3 0.2 0.1 Percentage of weights remaining (%) At the sparsification step, IMP performs global pruning, ISS fixes the binary mask m to be the maximum likelihood one under Ber(σ(s)) (which performed better than sampling from the distribution), and CS changes the parameterization of the mask from σ(βs) to b(s) (or, equivalently, weights w i where s i < 0 are removed).

We use a momentum of 0.9, a weight decay of 0.0001 (not applied to s), and a batch-size of 64.

Following Frankle & Carbin (2019) , sparsification is not applied to batch normalization nor the final linear layer.

To evaluate each method when finding masks with different sparsity levels, we run IMP with global pruning rates 50%, 75%, 80%, 85%, 90%, 95%, 97.5%, 98%, 98.5%, 99%, 99.5%, 99.75%, and ISS and CS with initial mask values -0.3, -0.25, -0.2, -0.15, -0.1, -0.05, -0.01, -0.005, -0.001, 0. Results are shown in Figure 4 : both magnitude pruning and stochastic 0 regularization (Stochastic Sparsification) fail at removing over 98% of the weights without severely degrading the performance of the model. On the other hand, Continuous Sparsification successfully removes 99.7% of the parameters in the convolutional layers while still yielding over 90% test accuracy. When taken to the extreme, our method is capable of removing 99.85% of the weights and still yield over 83% accuracy.

The dramatic performance difference between stochastic and continuous sparsification shows that our proposed deterministic re-parameterization is key to achieve superior results in both network pruning and ticket search. The fact that it outperforms magnitude pruning, a standard technique in the pruning literature, suggests that further exploration of 0 -based methods could yield significant advances in pruning techniques.

5. DISCUSSION:

With Frankle & Carbin (2019) , we now realize that sparse sub-networks can indeed be successfully trained from scratch, putting in question the belief that overparameterization is required for proper optimization of neural networks. Such sub-networks, called winning tickets, can be potentially used to significantly decrease the required resources for training deep networks, as they are shown to transfer between different, but similar, tasks (Mehta, 2019; Soelen & Sheppard, 2019) .

Currently, the search for winning tickets is a poorly explored problem, where Iterative Magnitude Pruning (Frankle & Carbin, 2019) stands as the only algorithm suited for this task, and it is unclear whether its key ingredients -post-training magnitude pruning and parameter rewinding -are the correct choices for the task. Here, we approach the problem of finding sparse sub-networks as an 0 -regularized optimization problem, which we approximate through a smooth, parameterized relaxation of the step function. Our proposed algorithm for finding winning tickets, Continuous Sparsification, removes parameters automatically and continuously during training, and can be fully described by the optimization framework. We show empirically that, indeed, post-training pruning might not be a sensible choice for finding winning tickets, raising questions on how the search for tickets differs from standard network compression. With this work, we hope to further motivate the problem of quickly finding tickets in overparameterized networks, as recent work suggests that the task might be highly relevant to transfer learning and mobile applications. Next, we consider the fixed settings (s 0 = -0.2, λ = 10 -10 ), (s 0 = 0.05, λ = 10 -12 ), (s 0 = -0.3, λ = 10 -8 ), and proceed to vary the final temperature β T between 50 and 200. Figure 6 shows the results: in all cases, a larger temperature of 200 yielded better accuracy. However, it decreased sparsity compared to smaller temperature values for the settings (s 0 = -0.2, λ = 10 -10 ) and (s 0 = -0.3, λ = 10 -8 ), while at the same time increasing sparsity for (s 0 = 0.05, λ = 10 -12 ).

While larger temperatures appear beneficial and might suggest that even higher values should be used, note that, the larger β T is, the earlier in training the gradients of s will vanish, at which point training of the mask will stop. Since the performance for temperatures between 100 and 200 does not change significantly, we recommend values around 150 or 200 when either pruning or performing ticket search. Lastly, we vary the initial mask value s 0 between -0.3 and +0.3, with hyperpameter settings (β T = 100, λ = 10 -10 ), (β T = 200, λ = 10 -12 ), and (β T = 100, λ = 10 -8 ). Results are given in Figure 7 : unlike the exploration on λ and β T , we can see that s 0 has a strong and consistent effect on the sparsity of the found tickets. For this reason, we suggest proper tuning of s 0 when the goal is to achieve a specific sparsity value. Since the percentage of remaining weights is monotonically increasing with s 0 , we can perform binary search over values for s 0 to achieve any desired sparsity level. In terms of performance, lower values for s 0 naturally lead to performance degradation, since sparsity quickly increases as s 0 becomes more negative.

:

We observed in our experiments that a final temperature of 500 is sufficient for iterates of s when training with SGD with 32-bit precision. The required temperature is likely to depend on the how s is numerically represented, as in reality our method relies on numerical imprecision.

We used the same network asFrankle & Carbin (2019) andFrankle et al. (2019), where it is referred as a ResNet 18.

Figures and Tables:

Figure 3: Test accuracy of tickets found by different methods on CIFAR-10. Error bars depict variance across 3 runs. Left: Performance of tickets found on a 6-layer CNN, when trained from scratch. Right: Performance of tickets found on a ResNet 20, when rewinded to the second training epoch. In both experiments, tickets found by CS outperform ones found by IMP. In most cases, CS successfully finds winning tickets in 2 iterations (purple curves).

Figure 5: Impact on relative test accuracy and sparsity of tickets found in a ResNet 20 trained on CIFAR-10, for different values of λ and fixed settings for β T and s 0 .

Figure 6: Impact on relative test accuracy and sparsity of tickets found in a ResNet 20 trained on CIFAR-10, for different values of β T and fixed settings for λ and s 0 .

Learning a binary mask with weights frozen at initialization with Stochastic Sparsification (SS, Algorithm 2 with one iteration) and Continuous Sparsification (CS), on a 6-layer CNN on CIFAR-10. Left: Training curves with hyperparameters for which masks learned by SS and CS were both approximately 50% sparse. CS learns the mask significantly faster while attaining similar early-stop performance. Right: Sparsity and test accuracy of masks learned with different settings for SS and CS: our method learns sparser masks while maintaining test performance, while SS is unable to successfully learn masks with over 50% sparsity.

Table content: <table><tr><td/><td/><td/><td/><td/><td>Stochastic Sparsification Continuous Sparsification</td><td/><td/></tr><tr><td>80 85 Figure 2: 100.0 60 65 70 75 Test Accuracy (%) Iterative Magnitude Pruning 51.4 26.5 Percentage of weights remaining (%) 13.7 7.1 Iterative Magnitude Pruning (reinit) Iterative Stochastic Sparsification Continuous Sparsification Continuous Sparsification (reinit) Continuous Sparsification (Pareto)</td><td>3.7</td><td>1.9</td><td>Test Accuracy (%)</td><td>92 94 86 88 90 100.0 82 84</td><td>51.4 Iterative Magnitude Pruning 26.5 Percentage of weights remaining (%) 13.7 7.1 Iterative Magnitude Pruning (reinit) Continuous Sparsification Continuous Sparsification (reinit) Continuous Sparsification (Pareto)</td><td>3.7</td><td>1.9</td></tr></table>

Peer review (iclr2020):
To the authors of paper 2504: I have posted a private comment for the reviewer/AC discussion period based on your author responses and your revised paper. I want you to be able to see my full response, but I can*t post additional public comments to the paper. As such, I*m editing my review with exactly what I sent to the AC. Thank you to the authors for your thoughtful rebuttal and for updating the paper with both new text and (more impressively) new experiments. I read all three of your rebuttals and re-read the revised paper in detail. My comment to the AC is at the bottom of this message. I don*t know if you will get an alert that the review was updated, but I hope you get a chance to take a look. =========================================== EXECUTIVE SUMMARY OF REVIEW Summary of paper: This paper proposes a technique that simultaneously trains a neural network and learns a pruning mask. The goal of this technique is to make it faster to retroactively find sparse subnetworks that, from a point early in training, could have trained in isolation to the same performance as the full network (*winning tickets*). The best subnetworks found by this technique outperform those found by existing techniques [1] at every sparsity level. Summary of review: The technique introduces several new hyperparameters whose values are asserted without describing the extent of the search necessary to find them; it unclear whether the cost of the search cancels out the efficiency gains. In addition, the proposed technique is inconsistent across runs. The sparsity and accuracy of the subnetworks it produces vary greatly, and there are no hyperparameters to explicitly control these outcomes. It is also unclear whether results vary from run to run even with the same hyperparameters. As such, this technique is not clearly a cost reduction as compared to existing approaches in [1] when it comes to studying lottery tickets. Moreover, the paper implicitly proposes a new pruning technique, and it should be evaluated against other related techniques in the pruning literature. Finally, the evaluation needs more rigor. Conclusion: It is difficult to determine whether the technique is an improvement over existing methods since the true costs of using it in practical workflows are unclear. Weak reject. Opportunity to improve score: Include a more detailed analysis of the overall costs of finding winning tickets at a target sparsity using the proposed technique, particularly including the costs of hyperparameter search necessary to do so. Include multiple replicates of experiments, experiments on more networks, and information about the variance of performance across runs with the same hyperparameters. Compare against other state-of-the-art pruning techniques. PROBLEM STATEMENT AND PROPOSED SOLUTION Problem: Winning tickets are currently expensive to find. The best known procedure [1, 2] is *iterative magnitude pruning* (IMP), which involves repeatedly training a network to completion, pruning by a fixed percentage, and *rewinding* weights to an early iteration of training until the network reaches the desired level of sparsity. To reach sufficient sparsity on standard networks, this procedure must be repeated 10 or more times. Goal: To propose a procedure that finds winning tickets more efficiently. Significance: A more efficient procedure would make it easier to study the lottery ticket phenomenon. (Whether studying that phenomenon is, itself, significant is debatable.) Personally, I have extensive experience using IMP to find winning tickets, so techniques to reduce the cost of finding winning tickets would be very valuable for my work. Proposed solution: The authors propose *continuous sparsification* (CS), which makes it possible to learn which weights to prune simultaneously with the weights themselves. Accompanying each parameter w in the network is a second parameter s. The actual weight used in the network is w * sigmoid(s * beta), where beta is a temperature hyperparameter. If the learned value of s is such that sigmoid(s * beta) is approximately 0 at the end of training, then the parameter has been pruned. The value of beta increases exponentially throughout training, meaning the output of the sigmoid will be closer to a hard 0 or 1, producing a pruning mask. To ensure sparsification happens, a regularization term lambda |sigmoid(beta * s)| is added. When run over multiple iterations, values of s are reset to their original values for weights that are not pruned. In addition, weights are either rewound (as in IMP) or left at their final values (as in [3]). Novelty: The paper is slightly novel. The technique is a variation of those proposed in [4] and [5], but the changes are meaningful. This is the first known use for finding winning tickets, but any pruning technique could hypothetically be used for this purpose. In effect, the paper proposes a new pruning technique that is primarily evaluated for its efficacy in finding winning tickets. TECHNICAL REVIEW * This technique introduces several new hyperparameters: lambda, initial and final betas, and the initial values for s. The paper suggests good values for these hyperparameters for both networks considered. These values were presumably found through hyperparameter search of some kind. How extensive was this hyperparameter search, and what is the range of *good* combinations of values? This is not just a methodological footnote; this paper*s stated goal is to improve the efficiency of finding winning lottery tickets, and - if good hyperparameters are hard to find - then that defeats the purpose of a more efficient technique. IMP, while far less efficient epoch for epoch as compared to CS, requires no hyperparameter search; global pruning 20% of parameters per iteration seems to work well in general [2]. In a revised version of the paper, I would be eager to learn more about this set of tradeoffs, since that is what matters in practice. * The authors only study two networks: a toy convolutional network and a small Resnet. It is hard to draw broad conclusions from such a limited set of examples. I would be particularly interested in seeing how this technique performs on a large-scale network for ImageNet (e.g., Resnet-50), since these are the situations where IMP becomes particularly cost-prohibitive. If the technique works well in these settings, it would enable lottery ticket research at much larger scales than is currently possible. If the technique works as efficiently at this scale, then doing could even be feasible during the rebuttal period. (I acknowledge getting experiments working on ImageNet is no small undertaking in terms of both engineering time and cost, but it would improve my confidence to see those results.) * The authors did an admirably careful job replicating the networks in [1], which include a variety of nonstandard hyperparameters. * Did the paper study Resnet-18 (a network designed for ImageNet with 11.2M parameters) or Resnet-20 (a network designed for CIFAR-10 with 272K parameters)? Frankle et al. [1, 2] describe Resnet-20 in their appendices but mistakenly refer to it as Resnet-18 throughout both papers, so I wanted to clarify. Based on the final test accuracy of the network, it appears to be Resnet-20; if so, I*d urge you to call it as such and note in a parenthetical or footnote that it*s the same network as in [1, 2] but with Frankle et al*s mistaken name corrected. * Are the values of s0 for CS sampled from a distribution, or are they all the same, fixed value? * Do the extra parameters lead to longer wall-clock training times? If you are making an argument about a more efficient technique, this is an important consideration. * Figure 2 includes the same graph twice. I believe a different graph should appear on the left, and I am eager to take a look at it in a revised version of the paper. * It does not appear that multiple replicates of each experiment were run with different network initializations in Figure 3. There don*t appear to be any error bars on Figure 3, suggesting that this only represents a single initialization. Considering the wide variance in performance achieved by continuous sparsification across runs in Figure 3 (right), this graph needs to include multiple runs and error bars. (I assume the multiple runs shown in Figure 3 (right) are with different hyperparameters?) * What is the performance of CS as a pruning technique? By finding winning tickets, CS is also implicitly pruning the network. Is this competitive with L0 Regularization [4]? Is it more efficient than iterative magnitude pruning as in [3]? If this is a more efficient technique for finding winning tickets, it is also liable to be a more efficient pruning method, which seems like even broader impact for the proposed technique. Alternatively, if this technique is less effective than comparable work (especially [4]) as a pruning method, then it is possible comparable work (especially [4]) might also produce better winning tickets, in which case the importance of this work is diminished. In an updated version of the paper, I would be interested in seeing an evaluation of CS as a pruning technique independent of the lottery ticket hypothesis (and compared to standard techniques in the pruning literature as such). I have a hard time seeing any reason why new techniques for finding lottery tickets are any different than new pruning techniques, and they should be evaluated as such in the context of the broader literature on pruning. * The last paragraph of Section 4.1 is missing important details that are necessary to evaluate the utility of continuous sparsification in comparison to IMP. *In only two iterations, CS finds a ticket with over 77% sparsity...* - for which hyperparameters, and how many hyperparameters had to be explored to find these values? How was the pareto curve obtained? How many different runs were necessary to create it? How many total epochs of training did it take to find that curve? If obtaining this pareto curve required many runs of CS, then it may not be any more efficient than running IMP in practice. The pareto curves appear to make CS look misleadingly effective, since they hide many of the actual costs involved (e.g., hyperparameter search, number of separate runs that were conducted to produce the curve, etc.). Greater transparency of this aspect of the paper would go a long way toward increasing my confidence that the findings are an improvement over IMP. * The sparsity and accuracy of subnetworks found by CS appears to vary widely from run to run as shown in Figure 3 (right). The final sparsity appears to be a function of the values of s0, lambda, and beta, potentially along with luck from the optimization process. For the same values of s, lambda, and beta, how widely do the final sparsities and accuracies vary? If I wanted to find a winning ticket with a particular sparsity using CS, what would the procedure look like for doing so? Would I have to sweep across values of these hyperparameters, or is there a more straightforward way to do so? The practical usefulness of the procedure hinges on these questions. * *We associate the performance drop of highly sparse tickets found by our method from the second iteration onwards to the lack of weight rewinding.* Why didn*t you also try it with weight rewinding? That seems like an easy way to evaluate this hypothesis. For both networks, it would be interesting to see the performance of CS with and without rewinding (analogous to Appendix B in [1]). * In the literature so far, the only winning tickets to be examined are those from IMP [1, 2]. CS is a different technique, and it likely finds different winning tickets. Do these winning tickets have different properties from [1, 2]? Can they be reinitialized? Can they be rewound earlier? These comparisons seem like an interesting scientific opportunity. * There are a few additional comparisons that I think are vital to include in the paper to appropriately contextualize results. They*re in the bullets below. * First comparison: it would be useful to include random reinitialization or random pruning baselines as in [1, 2] simply to make it easier for the reader to contextualize the performance of other sparse subnetworks. * Second comparison: what happens if you run IMP such that each iteration prunes to the same sparsity as achieved by each iteration of CS? Perhaps pruning by a fixed amount per iteration in IMP is wasteful, and one can prune more aggressively during earlier iterations as CS naturally appears to do. In other words, one way of explaining the advantage of CS would be that it prunes more aggressively. Is this indeed the case? I would be very curious to know. * Third comparison: What are the range of results achieved if IMP and CS are run *one-shot* (pruning after just one iteration; in the case of IMP, pruning directly to a desired sparsity)? That is, how well can these techniques do with just a single iteration? WRITING The writing is excellent. The prose is clear, and I was able to fully understand a relatively sophisticated technique on the first read through the paper. Writing of this quality is rare, and the authors should be commended for it. OVERALL: Weak Reject The problem statement is that IMP is not efficient. The paper claims that CS is more efficient. However, the paper does not present a convincing case that CS is, on the whole, more efficient when taking into account hyperparameter search to get CS to work, hyperparameter search to target a particular sparsity, potential variance across runs of CS, potential additional training costs of CS, and the possibility that IMP might be able to work comparable well given a more aggressive pruning schedule. In addition, the evaluation needs more experiments, including multiple replicates for each experiment and more networks (ideally one on ImageNet). Finally, I am unclear on what distinguishes CS from any other pruning technique, and it should be evaluated in the context of the broader pruning literature. If the authors clarify that the overall cost of CS (including all of the factors listed above) is lower than for IMP, address technical concerns about the evaluation, and evaluate CS as a pruning technique in an updated version of the paper, I will update my score accordingly. [1] Frankle & Carbin. *The Lottery Ticket Hypothesis.* ICLR 2019. [2] Frankle et al. *Stabilizing the Lottery Ticket Hypothesis.* Arxiv. [3] Han et al. *Learning both Weights and Connections for Efficient Neural Networks.* NeurIPS 2015. [4] Louizos et al. *Learning Sparse Neural Networks through L0 Regularization.* ICLR 2018 [5] Zhou et al. *Deconstructing Lottery Tickets: Signs, Zeros, and the SuperMask.* NeurIPS 2019. ===================================== COMMENT TO THE AC AFTER READING REBUTTALS AND REVISED PAPER TLDR I believe the technique is novel and should be published, regardless of whether it is actually more efficient than IMP for practical workflows. However, I believe the current evaluation is inadequate, both in the experiments and in the way data is presented. Namely, it is impossible to actually compare the costs of CS and IMP in the scenarios the authors evaluate, despite the fact that efficiency is the authors* claimed contribution. From the data as presented, it is unclear whether CS is actually more efficient than IMP for scientific use-cases. I*m not particularly concerned if there isn*t an efficiency advantage - it*s an innovative contribution regardless. My concern is that it is impossible to compare the costs of these techniques given the current presentation of data. That speaks to flaws in the evaluation section. I therefore maintain my score: weak reject. The technique deserves to be published, but the paper in its current form does not. INTRODUCTION After reading the original submission, I had the following questions: 1) Right now, the only way to find winning lottery tickets is through training the network and pruning it. IMP can be instantiated with any pruning technique, and the authors are really proposing a new pruning technique for use in the IMP framework. How does continuous sparsification perform as a pruning technique for network compression (i.e., independent of the lottery ticket hypothesis)? 2) How efficient is continuous sparsification (CS) in the scientific use cases where one would seek to find winning lottery tickets? In my research experience, there are two such use cases: (a) producing a winning lottery ticket with a specific sparsity and (b) producing winning lottery tickets across the full range of different sparsities. In their reubttal, the authors addressed these underlying questions: AS A PRUNING TECHNIQUE *We have added new experiments showing that our method yields competitive results when pruning VGG trained on CIFAR, outperforming both magnitude pruning and stochastic l0 regularization* My response: The initial results that the authors present are quite impressive on a VGG-style network for CIFAR-10. However, I find it concerning that the authors study pruning on this specific model but no others, particularly because they focus on a different network (namely, Resnet-20) in all other experiments in the paper. Since the authors are, in essence, proposing a new pruning technique, I would like to see it comprehensively evaluated as such on a range of networks against other baselines (as the authors do in Figure 4 for one network - those baselines are fine to me). EFFICIENCY OF CONTINUOUS SPARSIFICATION - It increases the cost of each individual network training run slightly: *Continuous Sparsification on a 1080 Ti: our method resulted in 15% extra wall-clock time per training epoch.* - PRODUCING WINNING TICKETS ACROSS THE FULL RANGE OF SPARSITIES: Throughout the paper, the authors present *pareto curves* showing the highest accuracy achieved by CS subnetworks at various sparsities. I find these pareto curves misleading: they hide the fact that CS had to be run several times with different hyperparameters to produce these curves. For example, in Figure 6 (right), the authors produce the pareto curves by training Resnet-20 (by my count) 22 times (each point along the purple lines). The corresponding IMP curve appears to have 14 points. In other words, to get winning lottery tickets across all sparsities, CS must be run for more iterations than IMP - it appears to be less efficient, contradicting the authors* core claim. The best argument in favor of CS is that one could perform each of these runs in parallel if sufficient GPUs were available, meaning less wall-clock time would be required. One caveat to this analysis: the authors state that they run CS *without rewinding,* meaning that the second iteration of CS requires less training time than fully training the network (as IMP requires). The authors do not state how long they train when they *aren*t* rewinding, so it is impossible to compare the efficiency of CS with IMP. All they say is that it *allow[s] for even faster ticket search.* They also do not study IMP without rewinding, which would be a helpful baseline for comparing to CS without rewinding. ***In short, if the authors are making an argument that one technique is more efficient than the other on an epoch-for-epoch basis, they need to actually plot the epochs required by each technique.*** - PRODUCING A WINNING TICKET AT A SPECIFIC SPARSITY: My concern about this use case is that there is a non-intuitive relationship between the initialization of the sparsity parameters and the final sparsity of the network. As the authors state in the rebuttal: *To achieve a desired sparsity, one can either perform runs in parallel with different values for s_0, or perform sequential binary search if the goal is to minimize the overall computational cost and not wall-clock time.* In other words, there is no precise way to target a particular sparsity other than trying many hyperparameter configurations. The authors do not provide any concrete costs of using CS in this way in comparison to IMP, and - from a usability perspective - this is a challenging workflow. OTHER CONCERNS - The authors only study CS on one toy network (the six-layer convolutional network, which - in my experience - is a particularly easy setting compared to deeper networks) and one *real* network (Resnet-20 on CIFAR-10). I would like to see results on other networks for CIFAR-10 (for example, the VGG network in Section 4.3). - More importantly, I would like to see results on an ImageNet network. If CS makes finding winning lottery tickets more efficient as the authors claim, then finding winning tickets efficiently on an ImageNet network should be an excellent demonstration of their contribution. This scenario has stretched IMP to its breaking point, as the authors note. ARGUMENTS IN FAVOR OF ACCEPTING + The authors propose a new pruning technique that appears to improve upon the increasingly popular L0-regularization technique. + With the right hyperparameters, the proposed technique makes it possible to find winning lottery tickets more efficiently than existing methods (i.e., IMP with magnitude pruning). + The winning tickets found by the proposed technique reach higher accuracy than IMP winning tickets and produce winning tickets at more extreme sparsities, improving upon our knowledge of the existence of winning lottery tickets. ARGUMENTS IN FAVOR OF REJECTING - The authors perform only minimal evaluation of their method as a pruning technique. It is possible that this is a missed opportunity to show an additional contribution. It is also possible that other, existing pruning techniques outperform CS at both pruning and finding winning lottery tickets. - The technique is hard to use for the two existing use cases for finding winning tickets. In particular, there is no way to search for winning tickets at a specific sparsity. - It is unclear whether CS is actually more efficient than IMP on an epoch-for-epoch basis, even though this is the main claimed contribution. The authors do not disclose - let alone plot - the number of training epochs required to find (a) winning tickets at a specific sparsity and (b) winning tickets at a range of sparsities, so it is impossible to make these comparisons. Meanwhile, the pareto curves the authors present are misleading, since they are the amalgamation of many separate runs of CS. - The authors study their technique on only one *real* network (Resnet-20) for finding winning tickets and a separate *real* network (VGG-19) for pruning. The authors do not show how CS performs in other challenging settings, especially ImageNet. - This work is only valuable if we believe that *lottery ticket hypothesis* work is valuable. In other words, this is a narrow contribution to an already-narrow area of study. This is one reason why pitching CS as a pruning technique would make this a stronger paper. I personally believe that *lottery ticket* work is valuable area of study, but I understand that it may not be seen as such in the broader ICLR community. CONCLUSION I believe the technique deserves to be published regardless of whether it is actually more efficient than IMP. It is a novel contribution to both our knowledge of pruning and of finding winning lottery tickets. The paper is exceptionally well written, and - as such - I believe it will help to inspire other research in this area. However, I do not believe that the current paper - namely, the current evaluation - should be published. The paper presents no concrete data on the comparative costs of performing CS and IMP even though the core claim is that CS is more efficient. The paper does not disclose enough detail to compute these costs, and it seems like CS is more expensive than IMP for standard workflows. Moreover, the current presentation of the data through *pareto curves* is misleadingly favorable to CS. I also believe that the paper needs experiments on ImageNet and needs a more thorough evaluation as a pruning technique beyond the lottery ticket hypothesis. I therefore retain my current score of *weak reject,* though I am eager to hear the thoughts of other reviewers, and I am open to changing my score.