Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
Authors: Mansheej Paul
Brett W Larsen
Surya Ganguli
Jonathan Frankle
Gintare Karolina Dziugaite
Venue: NeurIPS 2022 Accept (2022)
Keywords: data pruning
linear mode connectivity
iterative magnitude pruning
loss landscape geometry
lottery ticket hypothesis
sparsity

Abstract: A striking observation about iterative magnitude pruning (IMP; Frankle et al. 2020) is that—after just a few hundred steps of dense training—the method can find a sparse sub-network that can be trained to the same accuracy as the dense network. However, the same does not hold at step 0, i.e. random initialization. In this work, we seek to understand how this early phase of pre-training leads to a good initialization for IMP both through the lens of the data distribution and the loss landscape geometry. Empirically we observe that, holding the number of pre-training iterations constant, training on a small fraction of (randomly chosen) data suffices to obtain an equally good initialization for IMP. We additionally observe that by pre-training only on "easy" training data, we can decrease the number of steps necessary to find a good initialization for IMP compared to training on the full dataset or a randomly chosen subset. Finally, we identify novel properties of the loss landscape of dense networks that are predictive of IMP performance, showing in particular that more examples being linearly mode connected in the dense network correlates well with good initializations for IMP. Combined, these results provide new insight into the role played by the early phase training in IMP.

Lottery Tickets on a Data Diet:
Finding Initializations with Sparse Trainable Networks
Mansheej Paul
Brett W. Larsen
Surya Ganguli
Jonathan Frankle
Gintare Karolina Dziugaite
Stanford
Meta Al
MIT
Mo
HHarvard
"Google Brain
"Mila; McGill
Abstract
A striking observation about iterative magnitude pruning (IMP; Frankle et al. 10 is that after just few hundred steps of dense training the method can find a sparse sub-network that can be trained to the same accuracy as the dense network. However, the same does not hold at step 0, i.e., random initialization. work, we seek to understand how this early phase of pre-training leads to good initialization for IMP both through the lens of the data distribution and the loss landscape geometry. Empirically training iterations constant, training on a small fraction (randomly chosen) data suffices to obtain equally good initialization for IMP. We additionally observe that by pre-training only on "easy" training data we can decrease the number steps necessary to find good initialization for IMP compared training the full dataset or a randomly chosen subset. Finally, we identify novel properties of the loss landscape of dense networks that are predictive of IMP performance, showing in particular that more examples being linearly mode connected the dense network correlates well with good initializations for IMP. Combined, these results provide new insight into the role played by the early phase training in IMP.
this
observe that, holding the number of pre-
1 Introduction
Modern deep neural networks are often trained in the massively over-parameterized regime. Though these networks can eventually be pruned, quantized, or distilled into smaller networks, the resources required for the initial per-parameterized training poses a challenge to the democratization and sustainability AI. This raises fundamental question: under what circumstances can we efficiently train sparse networks? Recent work on the lottery ticket hypothesis [ IQ] has shown that, after just few hundred steps of pre-traiie pre-training, a dense network contains a sparse sub-network that can be trained without any loss in performance. Finding this sparse sub-network currently requires multiple rounds of training to convergence, pruning, and rewinding to the pre-train point, procedure termed iterative magnitude pruning (IMP, Figure I
y IO]), Remarkably, even after all these rounds of training, we do not find trainable sparse sub-networks we rewind to the random initialization; the first few hundred steps of dense network training is essential for finding sparse networks through IMP. In this work, we seek to understand this very short but critical phase of pre-training. particular, we investigate the effect of training data and number of steps used during pre-training on the accuracy achieved by IMP. We also explain how certain properties of the loss landscape allow us to predict whether will find matching initialization (i.c.. initialization from which IMP "succeeds" in finding a subnetwork that can match the accuracy of the unpruned network, to be formalized later).
IMP proceeds in three phases: an initial pre-training phase where (1) the dense weights are trained for few hundred steps, followed by (2) the resource-intensin mask search phase, during which we
Equal contribution. Correspondence to: [mansheej,
edu; gádz@google.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Phote Pre-training
WD
keendeck
Pruning Bound
Train Loss
WT
Barrier
WT
Phose Sparas Training
Figure 1: Left: Three phases of iterative magnitude pruning (IMP) with weight rewinding [T A dense network is trained in the pre-training phase for 4, iterations, where I, is referred to as the rewinding iteration, and W.,, is the parameters of the network at the rewinding point. The mask search phase produces sparse subnetwork at a desired sparsity level by iteratively training, pruning the smallest magnitude weights, and rewinding to wt,- The sparse training phase trains the final sparse subnetwork to convergence, starting with weights wt., Right: Illustration of computing the train loss barrier for initialization Wty WT and w/r are trained with different data order.
iteratively find a sparse mask by training the network and rewinding the weights repeatedly, finally, ending with (3) the sparse training phase, when the sparse masked network is trained. The weights learned after the pre-training phase-which serve as the initialization for the mask search phase- are referred to as the pre-trained initialization (see Figure
In this work, we focus studying the pre-training phase. To probe sufficient information needed to arrive at a matching initialization for IMP, we modify the pre-training phase by training with different pruned datasets and varying the number of pre-training steps; we then compare the performance of the resulting sub-networks across a range of sparsities. Following Paul et al. [23]. we prune the datasets both randomly and according to EL2N scores, which estimate example difficulty by measuring early training performance in an ensemble of networks. We next turn to characterizing the loss landscape properties of matching initialization for IMP. Frankle et al. [IO] investigate the relationship between successful initializations for IMP and linear mode connectivity on the sparse initialization for the entire dataset; here, we investigate this relationship a per-example basis. This enables us to identify signatures of the dense loss landscape correlated with matching initialization. Thinking more broadly, we explore the relationship between sparse trainability and training stability. Gilmer et al. [3 demonstrate that, in hyperparameter regimes in which early training unstable, learning rate warmup helps stabilize training. We build on this by investigating if the same pruned datasets which decreased the amount of pre-training required to find a matching initialization for IMP also decrease the required duration of learning rate warmup.
the dense initialization on
Overall, this empirical analysis provides new insight into the important and mysterious role that pre-training the dense network plays in IMP. Intriguingly, finding matching initialization for IMP is an example of a problem in which the dominant paradigm of "more data, more training" is not optimal. Indeed, by carefully choosing the training examples, we can not only use very small subset of the data but also reduce the training time required to solve this problem. From a scientific perspective, this suggests that different phases of training play different roles in the optimization process. Identifying and characterizing these phases will not only lead to a deeper understanding of the optimization of deep neural networks but also allow us to design training strategies that are optimal for each phase. From the practical standpoint, data loading is often an expensive part of the training process and can be bottleneck especially early in training. By identifying the essential subset for this phase of training, our work may enable strategies to eliminate this bottleneck.
Contributions. We find empirical evidence for the following statements:
In the pre-training phase, only a small fraction of the data is required to find matching initialization for IMP: On standard benchmarks, across all sparsity levels we evaluated, we find that we can match accuracy by training on small fraction of all of the available training data, selected randomly. (As vary the amount of training data in pre- pre-training phase, the number

---

of training iterations is held fixed.) Note that this observation changes if random label noise is introduced, in which case it becomes important select easy (small EL2N score) examples.
The length of the pre-training phase can be reduced if we train only on the easiest examples: Informally, training on a small subset of "casy-to-learn" training examples produces a better rewinding point than training all data for the same number of iterations;
The quality of a pre-trained initialization for IMP correlates with more examples being linearly mode connected in the dense network: This result complements the empirical evidence produced by Frankle et al. 100 connecting linear mode connectivity and the performance of IMP.
Training on easy data, which reduces the amount of pre-training required to find a matching initialization for IMP, does not reduce the amount of training required to stabilize the network via learning rate warmup (I33): The role played by data during IMP pre-training is thus different than that during learning rate warmup.
2 Background, Methods, and Related Work
We consider standard neural network training on image classification. Let {(x,, y yn)) j== denote training data, let w, E R. denote model parameters (weights) of the neural network, and let W1, W.,... be the iterates of (some variant) of SGD, minimizing the training loss, i.e., average cross-entropy loss over the training data. For a given training example Xn, let f(w, x) TR denote the logit outputs of the network for weights w and p(w, x) = (f(w, x)) be the probability vector returned by passing the logits through the softmax operation a. By the loss (error) landscape, we mean the training loss (error), viewed as a function of the parameters. By training and test error, we mean the average 0-1 classification loss.
Lottery ticket subnetworks. The lottery ticker hypothesis states that any standard neural network "contains [at initialization] subnetwork that is initialized such can match the test accuracy of the original network after training for at most the same number of iterations." Although such matching subnetworks (those that can train to completion on their own and reach full accuracy by following the same procedure as the unpruned network) are not known to exist in general at random initialization, they have been shown to exist after pre-training the dense network for a short amount of time (Phase in Figure L before pruning
trained in isolation it
27
Empirical evidence for this phenomenon comes via procedure that finds such subnetworks retroac- tively after training the entire network. This procedure, called Iterative Magnitude Pruning [IMP: 10 is based on standard iterative pruning procedures [1]. and can be decomposed into three phases (Figure D. outlined Algorithm
Algorithm 1 IMP rewinding to step i, and N iterations.
1: Create network with randomly initialization Wo Rd.
2: Initialize pruning mask to m 1d
3: Train wo for 4, steps to W
Phase I: Pre-Preaiie Pre-Training
4: for {1,.... N) do
> Phase 2: Mask Search
5: Train the pruned network m we, to completion. (6 the element-wise product)
6: Prune the lowest magnitude 20% of weights after training.
Let m|i] = 0 if the corresponding weight is pruned.
7: Train the final network ı @ W... Measure its accuracy.
> Phase 3: Sparse Training
This procedure reveals the accuracy of pre-training the dense network for I,, iterations, pruning, and training the pruned network thereafter. Phase can be understood as an (expensive) oracle for choosing weights to prune at t,. Although IMP is too expensive to use practical way to speed up training, it provides a window into a possible minimal number of parameters and operations necessary to successfully train a network to completion in practice, In our work, we extend this line of thinking, pursuing the minimal amount of data necessary to find and train these subnetworks. This is especially tantalizing due to the potential positive interactions between sparsity and minimizing the data necessary for training. The result is a deeper inquiry into the minimal recipe for successful training and, thereby, into the fundamental nature of neural network learning in practice.
In this respect, the closest work to ours is an experiment a larger compendium by Frankle al. showing that the standard pre- pre-training phase could be replaced by much longer self soff-supervieed phase.
3

---

Like our experiments in Section that work aims to study what makes a matching initialization. Our approach and findings are substantially different, however: we reduce the number of examples rather than changing the labels, and we show that, not only are a small set of examples (starting at R 2%) sufficient for pre-training, but also that they make it possible to pre-train in fewer steps.
There are many other ways to obtain pruned neural networks [e.g.. aspect of work on the lottery ticket hypothesis (and the one that makes the right starting point for our inquiry) is that its goal is to uncover a minimal path from initialization to trained network, regardless of the cost of doing so. The aforementioned procedures target real-world efficiency for training and/or inference.
16 20 29]. The distinctive
Linear Mode Connectivity and the Loss Barrier. We investigate the error landscape using the parent-child methodology and instability analysis of Frankle et al. [10] and Fort et al. [ Writing err(w) for the test error at the weights w, the (test) loss barrier between two networks w and w' is [0, [err(a w + (1 - a a))') (a err(w) + (1 - (t)err(w)))] This quantity measures the maximum increase in loss above the average loss along the linear path connecting the two networks on the loss landscape.
The loss barrier iteration (with parent weights wx) the loss barrier between the (children) weights wT and w'.. where and w/r are copies of w, trained to completion with the same procedure but different random seeds (minibatch order, GPU noise, data augmentation, etc.). See Figure 1 right for a visualization. Empirically, the loss barrier is achieved near 2. and so we compute it this way. The onset of linear mode connectivity (LMC) is defined to be the iteration & such that, for all > f. the error barrier t is zero. We follow Draxler et al. A Garipov et al. [12], and Frankle et al. [10 in considering the 0-1 loss barrier to be zero if it is less than 2%. Similar to Paul al. 23 we also measure the eгror (0-1 loss) or cross-entres cros-enntoppy loss barrier on individual training examples to explore the loss landscape corresponding to different subpopulations.
Ranking training examples. We define "easy/hard data" the data that is ranked low/high, respec- tively, by the EL,2N score introduced by Paul et al. (23 FL.2N scores depend on the margin early in training, and, loosely speaking, higher average margin early in training means lower importance for generalization of the final trained model. This connection to margin suggests that easy data is learned first (has higher margin early in training, maintained throughout the rest of training). EL2N scores were derived from the size of the loss gradient, and are thus are highly correlated with the magnitude of the gradient.
Definition 2.1 (EL.2N Score) The EL2N score of a training sample (x.y) at iteration is defined as Ellp( (1p(W,,3) x) -yl2,, where the expectation is taken over wt conditioned on the training data.
To calculate EL2N scores for dataset, we follow the process outlined in 23 In particular, do the following:
1. Independently train K 10 networks from different random initializations for iterations.
2. For each example and each network, we calculate the 1.2 norm of the error vector defined as [|p(x) y/12 where y is the one-hot encoding of the label, and p(x) are the softmax outputs of the network evaluated on example x.
3. For each example, the EL2N score is the average of the error vector L2 norm across the K networks.
In our experiments (Section B) we vary the data that is accessible in the pre- pre-training phase of IMP defined above. We either choose the data that we feed to the algorithm at random while preserving class balance, or based on the EL2N scores.
Stabilizing early training via learning rate warmup. Learning rate warmup period can be seen as a form of pre-training. allowing one to eventually train higher learning rates and larger batch sizes. Training with large batches and high learning rates is desirable in practice, as it allows for more efficient GPU utilization and may reduce the total number of updates needed to achieve the desired accuracy. Gilmer et al. [K] empirically observe that learning rate warmup essentially improves the optimization by allowing the initial optimization trajectory to navigate to "flatter" optimization landscape, i.c., one with smaller highest loss Hessian eigenvalue.

---

3
The Role of Training Data Selection in Pre-Training
As can be seen in Figure - when training sparse networks using IMP with rewind step t, = O, the final test accuracy of the sparse networks falls off rapidly with increasing sparsity. However, we increase the rewind step fr. the network performance improves across all sparsity levels and at a rewind step t", the network performs as well as better than the dense network at high sparsities. Informally, training the dense network for t* steps creates a matching initialization for IMP. But what does the network learn in these first t* steps? In this section, take the first step towards answering this question by investigating which subsets of the training data are sufficient for finding a matching initialization. In order compare networks trained on different subsets of data for different numbers of iterations, we introduce the notion of a matching initialization with the following definitions.
Definition 3.1 Let w be the dense network weights after training on subset of the training data, S until rewind step I. Then for two data subsets {SS S" rewind times {t, and a given range of sparsities, w is said dominate (weakly dominate) w if sparse networks obtained from IMP with wy as the initialization achieve better (no-worse) accuracy than those obtained from IMP with w as the initialization.
CIFAR-10 ResNet-20
CITAR-100 ResNet-32
CINIC-10 + ResNet-S5
Subset Size 3200 (6.4%). 1"
Subset Size 2048 (4.1%)) 1" 800
Subset Size 51 128. (28.4%)) e" 800
052
0.84
2-39
047
4:52
033
0.44
(ooeid E.
(kowbellt PM)
iTawind R. P50
(Rowkel L. 17/
-+0- (Rawird 221
Figure 2: For given rewind step 6, /2, training small fraction of random data during the pre-training phase of IMP leads to matching initializations (compare the solid green with circles and dashed orange curves) across dataset, network, and hyperparameter configurations. Using just the easiest training examples during this phase produces a matching initialization for rewind point t" in just t* 22 steps (compare the solid red with triangles and dashed blue curves). Pre-training on the hardest examples is detrimental to the performance of the initialialization (solid pink curve with crosses). IMP with rewinding to initialization (dashed yellow curve) and the dense model (dashed grey curve) are used as baselines. For each dataset + network configuration, we present the best performing easy data subset size. For a sweep across subset sizes, see Figure 3 and the Appendix
For network trained on the full dataset for & steps, we write w.. In Figure we see that W,- dominates w which in turn dominates wp. We investigate which data subsets S and rewind steps t lead to networks w that dominate w,- and W such networks are called matching initializations.
Definition 3.2 A dense network w is a matching initialization for rewind time t" if w/ weakly dominates w.-.
We empirically find that certain surprisingly small subsets S and rewind step t, < tt lead to matching initializations for rewind time t*.
Experimental design. To evaluate the effect of the training subset size and composition on the quality of the pre-trained initialization, we train ResNett20/ResNet 3eeeeeeeett on subsets of CEAAR--O/CIFAR-II 1AOEAAA-O//IIIIIIUUI -10. respectively. The subset size M varied and subsets are chosen as follows: (i) M randomly selected examples, distributed equally among all classes; (ii) the easiest M examples; (iii) the hardest M examples. The easiest examples are those with the smallest EL.2N scores and the hardest are the examples with the largest EL2N scores
23
NNote that due to the stochasticity in training and risk measurements, we ignore small deviations in the final test accuracy. Thus one rewinding point could weakly dominate another one even if at some sparsity levels their mean performance "crosses" over while approximately remaining within the standard error of one another.
5

---

8% Weights Remaining Pruning Roundst
10. 7% Weights Remaining
9% Weights Remaining
11.0 Pruning Rounds)
(12 Pruning Rounds)
CIFAR-10 ResNet-20 = 400
3.14
4.8
5294
664
1385
31830
314
68
0234
25499700
443
1360
3968
386
1254
465
1985
3.0
CIFAR-100 ResNet-32 2' 800
3.40
1.50
0.00
CINIC-10 + Net-66 ('m 800
3.50
1700
M0 1200
Dulasst Sic
1:50
Dutautt Sice
10.00
12807
Dutaust Dos
LOEE
48 Ddte
Mardiio Subie
Dlavid
DUITERS
Figure 3: A summary of the the dependence on subset size for the style of experiments described in Figure The first column represents the performance across subset size for the fixed sparsity 16.8% weights remaining or 8 rounds of pruning. The subsequent columns show the same for 10.7% (10 pruning rounds) and 6.9% (12 pruning rounds) respectively. The horizontal lines correspond to baseline runs at rewind steps 0, t* /2, and t using all the data. For CINIC-10 (bottom row), rewind step and the hardest data subsets are not visible in some cases because their accuracies fall below the range displayed.
Due to the significant computational demands of performing IMP with multiple pre-training schemes and replicates, we focus on a targeted set of pre-training iterations f,. In particular, we study the pre-training iteration f, t where training on all examples leads IMP to find spare sub-networks that perform as well as the dense network for large range sparsities (t* 400 for CIFAR-10 and 800 for CIFAR-100 and CINIC-10). We also study the more challenging pre-training iteration of t, - where pre- pre-training all data does not yield matching initialization. When M examples are not enough to train for t, iterations without replacement, we make multiple passes over the M examples necessary. Figure 2 shows the best performing easy data subset for each dataset across the full range of sparsities; Figure shows the performance across subset size at three fixed sparsities.
Randomly chosen examples. Pre-training the dense networks on small, randomly chosen subsets S can lead to initializations for IMP, w that dominate initializations Wt,, trained on the entire training set for the same number of steps. In Figure 2 we see that for all dataset + network combinations, pre-training the dense network on small random subset (solid green curve with circles; sizes ranging from 4.1 11% for CIFAR- 100 to 28.4 28.4% for CINIC-10) for 4, = 1* /2 leads to initializations that (weakly) dominate those that were obtained from training the network for the same number of steps on all the data. This observation leads to surprising suggestion: in these experiments, the subset size is smaller than the total number of images seen during the pre-training phase; for the particular goal of finding a matching initialization of IMP, multiple passes through the same small dataset can be more beneficial than seeing more random data.
Easiest examples (lowest EL2N scores). By pre-training on just the easiest examples (identified by lowest EL.2N scores, solid red curve with triangles Figure 2 we can obtain matching initializations in fewer steps compared to training on the full dataset. In Figure 2 we see that for all three dataset and network combinations and for the subset sizes shown, the initialization obtained from training on the casiest examples for t, t" /2 steps leads matching initializations for t".

---

CIFAR-10 ResNet-20
CIFAR-10 - ResNet-20
10% Rendorizeed Labels During Pre-Training
50% Rendomized Lebels During Pre-Training
0.92
0.92
0.91
0.91
2.50
3.69
6.69
3.87
0.67
0.55
6.65
Weigghs Rernaining
% Weights Remaining
T/S
(Tavnd 0. 21
(Raind 1.1/d
Figure 4: Pre-training on a random subset or all data is not robust to label noise during this initial phase of IMP. However, pre-trate pre-training with the easiest data as scored by EL2N scores computed from the corrupted dataset is robust. In both the left (10% randomized labels) and right (50% randomized labels), pre-training on the easiest data for L* /2/2 iterations dominates all other pre-training schemes, including training on all data for 1" iterations. Results for additional subset sizes are included in Appendix B
Hardest examples (highest EL,2N scores). Conversely, pre-training on the hardest examples (solid pink curve with crosses in Figure 2) yields worse accuracies than pre-training all examples or a random subset. In fact, on CIFAR-10 the hardest examples perform little better than using no pre-training at all. Interestingly, when training a dense network, these hardest examples are crucial for obtaining a network with good generalization properties [23]. This suggests that while the hard example may be key later in training, repeated passes through easier examples should be the focus during the very early stages of training to quickly find good initialization for IMP.
Randomized labels during pre-training. Pre-training on a all data or random subset is not robust to corruption with random label noise (28] during the pre-training phase. As seen in Figure 4 the higher the percentage of randomized labels, the lower the performance of these data subsets, and in particular, the pre-trained rewinding point becomes no better than a random initialization when 50% of the labels randomized. On the other hand, training on easiest data with EL2N scores computed on the corrupted dataset is robust to this noise (solid red curve with triangles in Figure + This is because examples with randomized labels are hard ((23) according to this metric, and thus the easiest examples will select a subset of largely uncorrupted data.
Summary. Taken together, our results suggest that, finding matching initialization for IMP at rewinding step t" is an interesting problem in which "more data, more training" is not optimal: is neither necessary to train on all the data nor to train for the full t steps. In fact, we can get away with training on a surprisingly small dataset for as little as half the number of steps if we make multiple passes through the right examples, here the easiest examples defined by lowest EL2N scores.
4 Pre-training through the lens of Linear Mode Connectivity
Frankle al. [TC observed a strong indicator of whether a sparse sub-network generated by IMP would be able to match the accuracy of the dense network is whether or not that sparse sub-network train the same linearly connected mode (i.e. the train loss barrier is close 0, see Figure D.. Here we ask what properties of the dense network at the rewind point might also be predictive of this property. In the architectures + datasets we consider, the onset of linear mode connectivity (LMC) occurs in the dense network later than the first rewind time which produces good initialization for IMP (i.e. i > t,), and thus, we consider two generalizations to the notion of LMC. First, we look at the train loss barrier on per-example basis meaning that instead of considering the full loss landscape we separately look at the performance of the linear interpolation on each example (the original notion of train loss barrier is obtained by averaging these values). Second, we then look at the distribution of these per-example train loss barriers as a continuous measure of the state of the network rather than simply considering whether their average is O. We demonstrate in general that these measures of the loss landscape of the dense network correlate well with the IMP performance of the pre-trained initialization. We additionally show that pre- pre-training on easy data results in
7

---

All Data (600 (terations]
All Data (500 Iteration)
3.57
2209
3.40
5000
02:0
6600
04:00
6.
4003
40:00
2001
10:00
2200
Randem Examples (400 (teresions)
2048 Easiest
(400 Iteratoor)
18
Smallest EL2N Scores
15
28
3:50
120
Pretein Aar.ccoecyy
650
ain oin
1690
3300
4200
5:03
430
500
0103
2000
2002
50
Tram Barier
Lot Burker
Pian Barier
(a) CIFAR-100, ResNet-32, 1" 800 and 4 = 12/2 400.
All Data (200 neraainns)
All Data 1400 iterations
500002
>22000
Bur Bumber of Exmmpe
200033
322000
0.805
100033
100000
0.852
farson
Tower Tastenn
-2.5
13.0
255
603000
75.60 Random Pox moles (POD Perations)
25.0K Cas
(200 Parasoon))
Smalest E12N Scores
0.664
Pretrsin lest Accuracy
Campol
Number bommes
503000
300033
0.052-
803000
466000
203000
200000
0.8
-3.0
-2.5
5.3 Tran Lons Barmer
2.0
2.5
7.5
10.0
Train Loss Barterr
20.0 0.600
0.5
Newn Trxin Loss Daerie"
3.0
3.5
2.
25
(b) CINIC-10, ResNet-56, L" 400 and 1, =0= 200.
Figure 5: Left and center columns: Pre-training on all the data for 1" (top right) vs. /2/2 (top left) steps leads to a distribution shift to smaller per-examples train loss barriers between children runs of the dense network. When pre-trained for just 12/2 iterations but with random subset of data, the distribution is similar to pre-training on all data for L* /2 steps. However, when pre-trained on a subset of easy examples for 2 iterations. the distribution of train loss barriers displays a shift comparable to pre-training on all the data for t* iterations. As seen in Figure 2 and the Appendix, pre-training with this random subset matches pre-training with all data for t t steps while pre-training with the easiest data for t"/2 matches pre-training on all data for t* steps. This observation suggests that the per-example distribution of train loss barriers may be useful signature for predicting the IMP performance of dense initializations. Right column: Scatter plots of final test accuracy vs. pre-train accuracy of the dense initialization (top) and the mean train loss barrier (bottom) across a variety of dataset pruning strategies for pre-training. Final test accuracy is at 8.6% weights remaining (11 pruning rounds) for 10F and 21 % weights remaining (7 pruning rounds) for CINIC- 10. We observe that the mean train loss barrier is better correlated with the final test accuracy than the pre-train accuracy of the dense initialization.
distribution with a large fraction of examples linearly mode connected faster than pre- pre-training on all data. See Appendices B and C for additional experiments and an exploration of why this occurs.
The distribution of per-example loss barriers. the left and center columns of Figure 5 we plot histograms of per-example loss barriers for various rewinding points and data subsets for CIFAR-100 (panel a) and CINIC-10 10 (panel b). There is a clear shift in the empirical distribution of the per-example loss barriers not only the mean is shifting previously observed, but also the mode. Pre-training
8

---

on all data for t" iterations has the same effect on the distribution of the loss barriers as pre-training on the easy examples for t* /2 iterations, two procedures which produce matching IMP initializations.
Given the connection between the error barrier and IMP sub-network performance, it is natural to ask whether examples which achieve low loss barrier early in training with the full dataset are more important for producing a matching initialization than high loss barrier examples. We call this ranking of the data by the per-example loss barrier at a given iteration the LMC score, and the blue squares in the seatter plot are the result of training on a subset of the data determined by this score. Our empirical findings suggest that using this score does not match the performance of using easy data (see Appendix D
The correlation between final test accuracy and mean train loss barrier. The right column of Figure 5 shows scatter plot of final test aceuracy of the sparse trained network two different properties of the dense initialization: test accuracy of this pre-train point (top) and the average train loss barrier from 3 pairs of children runs (bottom). Here we observe the same result across the different training conditions: a correlation between the train loss barrier when spawning from wr,, and the final test accuracy of the sparse sub-networks. The correlation is weaker for the pre-train test accuracy, suggesting that the loss landscape properties hold more import for determining the success of IMP. Furthermore, we see that training on the easiest examples produces rewinding points with smaller train loss barriers than training on random subsets.
5 The Role of Training Data in Learning Rate Warmup
To this point, we have studied the early phase of training through the lens of the lottery ticket hypothesis. However, the dynamics of the early phase of training have important implications beyond the sparse regime considered by lottery ticket research. For example, learning rate warmup a nearly ubiquitous part of training state-of-the-art models. Warmup can be viewed as an accommodation for the idiosyncrasies of the earliest part of training and a pre-training strategy that prepares the network to train at the full learning rate. In this section, we extend our previous experiments to dense training with learning rate warmup by ablating the training data during this phase to assess whether easy data alone suffices, similar to our observations in the pre- veeetraiiing phase for IMP (Section B3
Easy data does not reduce the amount of training required to stabilize the network via learning rate warmup. The effects of performing learning rate warmup with easy, random, and hardest. subsets of the data appear in Figure O Intriguingly. the most striking performance drop is observed when doing learning rate warmup with easy data, especially at longer pre-training times. Pre-training on the random and hardest data produces the same performance for sufficiently large subset sizes. The role played by data during IMP pre-training is thus different than that during learning rate warmup; the easy data does not add stability benefits during the later. We hypothesize that this is because learning rate warmup happens over a much larger fraction of the total training time than IMP pre-training.
Easiest Examples
Random Examples
Hardest Examples
0.92
Test Accuracy
0.93
0.916
0.916
0.90
0.912
0.912
0.89
Fretrain Num Exdeet
Rnot ain Rondom Subset Size
Fretrain Num Hardert
50000
50002
50000
5120
0.908
5120
0.308
5120
0.88
20240
10243
20240
25500
25502
25500
0.904
0.904
100
200
400
500
1600
3200
100
200
400
800
1500
3200
100
200
430
800
1600
3200
Batches Warmup
Belches Warmuu
Batches Warmup
Figure 6: Training with different data subset sizes and pruning strategies for the learning rate warmup period for ResNet-20 on CIFAR-10. In the large batch size and learning rate regime, learning rate warmup is essential for stabilizing training; we sweep over different learning rate warmup periods with a batch size of 1024. In contrast to pre-training for IMP, training on easiest examples in this initial phase leads to worse performance than training with all data, random subsets, or the hardest. data. We note that the learning rate warmup period is a much longer fraction of training than IMP pre-training which lasts for only few hundred steps.

---

Discussion and algorithmic implications. The benefits of training on easy data faster learning and increased stability to SGD noise-are are only present in the very early phase of learning. Paul et al. [43] have shown that, if we continue to train on easy data only, the network will converge to lower final test accuracy. So a natural question is what other scenarios do the benefits of training on easy data extend to? We began to investigate this question here by studying training at high learning rates where learning can become unstable early in training and is usually stabilized by learning rate warmup 13. Our results show that training only on easy examples does not help and in fact hurts performance in this case. Two possible reasons for this behavior are (1) learning rate warmup is often used for the first 10% of training, which may be too long for easy examples be helpful (the pre- pre-training phase in iterative magnitude pruning where easy examples are clearly beneficial can be as little 2% of training) and (2) the instability in this case case-loss diverging to infinity is not reduced by methods that improve stability to SGD noise. Despite the negative results of this experiment, we think that, when combined with our previous results, the core scientific findings in our work are interesting and broadly useful to the community. In particular, some algorithmic implications follows: (1) While developing algorithms for training sparse networks or finding matching networks at initialization, researchers might want to incorporate methods or search for initializations that improve stability to SGD noise. (2) Recent work in curriculum learning found that curricula provide marginal benefits unless there are additional constraints such as a training budget (26] Our work helps narrow the time window in which training on easy examples is beneficial and provides another example in which, for the constrained scenario of finding matching sparse networks, simple curriculum of training on easy data first beneficial. Thus, these findings may help in the design of better training curricula. (3) In the training of large language models, a common problem is that loss spikes and training becomes unstable. A hypothesis for why this phenomenon occurs is that the network is presented with bad batch or set of bad batches that destabilize training. Since this is related instability due to SGD noise (bad batches are randomly sampled) it may be possible to improve training stability by injecting easy examples into the training process.
6 Discussion
Recent empirical evidence has shown that deep neural network optimization proceeds in several distinet phases of training & []. Understanding the role that data plays in these different phases can help us characterize what is being learned during them. Since data loading is often a bottleneck, this understanding also has the potential enable more efficient training schemes. In this work, we have considered two essential phases of early training: pre-training for IMP which enables sparse optimization and learning rate warmup which stabilizes network training. Our experiments identify what data is sufficient for each of these procedures, and as they both occur early in training, these finding can be used design more efficient data loaders for streaming datasets. Furthermore, as IMP is computationally expensive procedure, understanding how it works is essential for designing better algorithms with the same performance. To this end, we identify loss landscape properties of the dense network initialization for IMP that are predictive of successful spare training. Though this work does not provide an improved algorithm for obtaining sparse networks, we believe our results provide essential guidance for researchers pursuing algorithms that perform pruning early in training (i.e. finding sparse masks without training to convergence).
Acknowledgements
The experiments for this paper were funded by Google Cloud research credits. S.G. thanks the James S. McDonnell and Simons Foundations, NTT Research, and an NSF CAREER Award for support while at Stanford. This work was done in part while G.K.D. was visiting the Simons Institute for the Theory of Computing. The authors would like to thank Daniel M. Roy for feedback on multiple drafts.
10

---

References
[1] R. Baldock, H. Maennel, and B. Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34, 2021.
[2] T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang. Z. Wang, and M. Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in Neural Information Processing Systems, 2020.
[3] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, Cinic-10 is not imagenet or cifar-10. arXiv preprint vrXiv:1810.03550 2018.
[4] F. Draxler, K. Veschgini, M. Salmhofer, and F. Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, pages 1309-1318.
PMLR, 2018.
[5] F. Draxler, K. Veschgini, M. Salmhofer, and F. Hamprecht. Essentially no barriers in neural network energy landscape. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1309-1318. PMLR, 10-15 Jul 2018. htttt:ss////poceedeeee
mlr. www.:mmes.././/mmmmmmmmmmmmm prwssr
drax Traxleria er 18a.8B..1 html
[6] R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2 10. 10.0699, 2021.
[7] U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952 PMLR, 2020.
[8] S. Fort, G. K. Dziugaite, M. Paul, S. Kharaghani, D. M. Roy, and S. Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. arXiv preprint arXiv:2010. 15/ 10, 2020.
[9] Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1rXiv:18033 1803.000666 803.03635, 2018.
[10] J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proc. Int. Conf. Machine Learning (ICML), 2020.
[11] J. Frankle, D. J. Schwab, and A. S. Morcos. The early phase of neural network training. In International Conference on Learning Representutions, 2020. URL ltttps://openrevieee
net/forum? 1d=Hk11iRii iRNFFs
[12] T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in Neural Information Processing Systems, 31, 2018.
[13] J. Gilmer, B. Ghorbani, A. Garg, S. Kudugunta, B. Neyshabur, D. Cardoze, G. E. Dahl, Z. Nado, and O. Firat. A loss curvature perspective on training instabilities of deep learning models. In International Conference on Learning Representations, 2021.
[14] S. Han, J. Pool, Tran, and J. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, 2015.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[16] S. A. Janowsky. Pruning versus clipping in neural networks. Physical Review A, (9(1))6600,, 1989.
[17] N. M. Kalibhat, Y. Balaji, and S. Feizi Winning lottery tickets in deep generative models. arXiv preprint arXiv:2010.023350 2020.
[18] A. Krizhevsky, V. Nair, and G. Hinton. The cifar-10 10 dataset. online: http:///www. es. toronto. edu/kriz/cifar. html, 55:5, 2014.
[19] R. Kuditipudi, X. Wang. H. Lee, Y. Zhang, Z. Li, W. Hu, R. Ge, and S. Arora. Explaining
landscape connectivity of low-cost solutions for multilayer nets. Advances in Neural Information Processing Systems, 32, 2019.
[20] Y. LeCun, S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pages 598-605, 1990.

---

[21] K. Mangalam and V. U. Prabhu. Do deep neural networks learn shallow learnable examples first? ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena. 2019. URL wtt..htts:/// http:::///peeeeieeeeee net/ forumm??dd
[22] V. Nagarajan and J. Z. Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
[23] M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning data diet: Finding important examples early in training. arXiv preprint arXiv:2107.07075. 2021.
[24] M. Toneva, A. Sordoni, R. T. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. empirical study of example forgetting during deep neural network learning. arXiv preprint 1r 0.../....59, 2018.
[25] M. A. Vischer, R. T. Lange, and H. Sprekeler. On lottery tickets and minimal task representations in deep reinforcement learning. arXiv preprint arXiv:2105.016 2021.
[26] X. Wu, E. Dyer, and B. Neyshabur. When do curricula work? International Conference on Learning Representations, 2021. URL httptttt//eeerreeee//
[27] H. Yu, S. Edunov, Y. Tian, and A. S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in and nlp. International Conference on Learning Representations, 2020. /ttpptttss//oppeereee openrevee. http:///eaareeeeeeeeeeerrrrrrr// nett/orrum?I--SIxxRRR
[28] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/161 1.03530, 2016.
[29] M. Zhu and S. Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 171iv 2017.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section !
(b) Did you describe the limitations of your work? [Yes] See Section as well as Section (Experimental design).
(c) Did you discuss any potential negative societal impacts your work? [Yes] We do not any negative societal impact, as our work provides better understanding of existing algorithms.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]] See Appendix A
for a link to the code.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Appendix
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] It does not.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ywQAK50ThJi", "reviewer": "Reviewer_xZ2E", "summary": "This paper studies iterative magnitude pruning (IMP), which is used to find a sparse sub-network that can train to achieve the accuracy dense networks can achieve. The authors found that empirically:\n- Training on a small fraction of randomly chosen data is sufficient to find a good initialization \n- Training on easy training data can reduce the number of steps to find a good initialization\n- Mean train loss barrier is predictive of final test accuracy. \n\n", "questions": "- B.1 refutes the hypothesis that training on easy data is doing gradient clipping. Are there any other hypotheses?  \n- How are the results in setion 5 related to finding good sparse initialization? Does the observation that easy data can reduce lr warmup time tell us anything about why easy data can reduce the rewinding time? \n- What’s the definition of level in B.2? How much weights are left for each level? \n\nTypos:\nL179: “for”, L453: “at least”\n", "limitations": "This paper has some discussion of the limitations in section 6.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nFinding initializations with sparse trainable networks is a fundamental question in neural network training. The observations made by the paper are interesting and can provide insights into understanding sparse trainable networks. In particular, it shows that using easy data is sufficient and can reduce rewinding time, which can be practically useful. \n\nWeakness:\n\n- The writing of this paper can be improved. I think the connection between the sections can be made stronger. For example, in section 5, the authors found that easy data does not reduce the amount of training needed in the learning rate warmup, but did not talk much about how this phenomenon is related to the previous sections.\n- I think this paper could delve deeper into the observations made. While the observations made by the authors are novel, the authors did not give much explanation of the phenomena oberserved. I think the paper can be made much stronger if the authors could propose and investigate some explanations of the phenomena and discuss more how the observations can facilitate new algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gl_PZFr7mEN", "reviewer": "Reviewer_jz9s", "summary": "The paper deals with performing pruning with respect to data. The emphasis of this work is in understanding the dynamics of the learning process. More specifically how do the early stages of the training help you with performing pruning based on the neural network weights.", "questions": "Did you try this on different modalities of data ?\nHow does the loss landscape look like before and after the pruning ? Did you try and visualize it ?", "limitations": "No There is no discussion on it (or perhaps I missed it in the supplementary material ) ", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is extremely clearly written. The claims are very crisp and clear. More importantly the impact of this is pretty huge, if this will hold at scale for very large data sets. \nThe method section is well explained and the (continually hammered in ) connection to the lottery ticket hypotheses gives good context to the solution \nAlbeit there are a few things that are still very empirical/ad hoc: \nline 136 - is the 2% a stiff criteria ? Or did you try to change it. More importantly what would have been very convincing if this would have been tried on several modalities of data. I have a sneaking suspicion that for tabular/time series this would not hold.\nLine 147 - Is P the probability of the network, the wights ?\n\nA few typos and assortment of other things\nline 115, missing a space\nline 118 - double dashes \n* sometimes authors relay way too much on references for example EL2N context some of the benchmarks of GWMP other methods etc...\n** This is extremely nitpicking but, the reviewer is colorblind, such it took me a but to correlate in Fig 5 Easiest examples and low EL2N, can you please make it consistent between the title of the figures and legend of the scatter plots ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XRD4OCmA3VB", "reviewer": "Reviewer_exXP", "summary": "This paper focuses on the pre-training phase of the IMP procedure. The authors show that only a small fraction of data is required for finding a matching initialization. In particular, the authors show that the length of such pre-training is reduced if we use as training set only the easiest examples. Then, they also introduce two measures of the loss landscape of the dense network that correlate well with the IMP performance of the pre-trained initialization. The authors also investigate if the same behavior can be observed during learning rate warmup.", "questions": "The authors present all the results using the ResNet architecture. It would be interesting to show if the same behavior is confirmed also with other architectures (such as VGG or WideResNet).  I understand that a thorough validation with different architectures will require a significant additional number of experiments. However, I think that it is still interesting to verify the most important assumptions with other architectures in a limited number of experiments.\nI found the experiment with randomized labels (pag. 7) not very fair because, when you perform training on easiest data, you select almost only data with uncorrupted labels.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides new insights into the role of the pretraining phase for finding matching lottery tickets. I think that the empirical evidence shown in this paper can significantly contribute to the research in this field, reducing the amount of data and time required for this training phase. The contributions presented in this paper also help to shed light into the behavior of the network during such pretraining phase.\nThe paper is well written and easy to follow. The experimental validation of the assumptions presented in the paper is solid.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Nqa08ucuKk1", "reviewer": "Reviewer_pW7h", "summary": "This paper studies the role of data in finding early-stage lottery tickets. With carefully-designed experiments, the authors showed that training on a small fraction of easy data suffices to get a good initialization for IMP and in particular it takes fewer pre-train iterations. In addition, the authors proposed to use the distribution of per-example loss barriers to predicting the quality of IMP initialization. Overall, these findings offer interesting insights into the early phase neural network training dynamics.", "questions": "See my comments in the weaknesses section.\n\nOther questions and suggestions:\n- For experiments with only a subset of data, do you also use the same subset of data for IMP or the whole dataset?\n- I encourage the authors to dig a little deeper into why easy data would help. I believe this would make the paper much stronger.", "limitations": "The authors discussed the limitations at the end of the paper. I don't see any negative societal impact of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and quite easy to follow.\n- Although the ingredients are not new, the idea of studying the role of data in finding good lottery tickets is novel. I believe some of the findings of this work are interesting to many people in our community.\n- Experiments are well-designed and support the authors' claims very well.\n\nWeaknesses:\n- All experiments are done on relatively small datasets. Quite often, the results on CIFAR do not transfer to larger datasets like ImageNet. One or two runs on ImageNet would make the results more convincing and I believe nowadays most academic labs can afford to run a few experiments on ImageNet.\n- The authors only used one single metric to rank the training data. It would be great if the authors could try out one or two other metrics. In addition, it seems that the data selection procedure could be expensive. I think the authors should add some discussions in the main paper about that.\n- The authors discovered an interesting phenomenon but didn't give any explanation of why easy data could help.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept