Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies iterative magnitude pruning (IMP), which is used to find a sparse sub-network that can train to achieve the accuracy dense networks can achieve. The authors found that empirically:
- Training on a small fraction of randomly chosen data is sufficient to find a good initialization 
- Training on easy training data can reduce the number of steps to find a good initialization
- Mean train loss barrier is predictive of final test accuracy. 


strengths_and_weaknesses: Strength:

Finding initializations with sparse trainable networks is a fundamental question in neural network training. The observations made by the paper are interesting and can provide insights into understanding sparse trainable networks. In particular, it shows that using easy data is sufficient and can reduce rewinding time, which can be practically useful. 

Weakness:

- The writing of this paper can be improved. I think the connection between the sections can be made stronger. For example, in section 5, the authors found that easy data does not reduce the amount of training needed in the learning rate warmup, but did not talk much about how this phenomenon is related to the previous sections.
- I think this paper could delve deeper into the observations made. While the observations made by the authors are novel, the authors did not give much explanation of the phenomena oberserved. I think the paper can be made much stronger if the authors could propose and investigate some explanations of the phenomena and discuss more how the observations can facilitate new algorithms.
questions: - B.1 refutes the hypothesis that training on easy data is doing gradient clipping. Are there any other hypotheses?  
- How are the results in setion 5 related to finding good sparse initialization? Does the observation that easy data can reduce lr warmup time tell us anything about why easy data can reduce the rewinding time? 
- What’s the definition of level in B.2? How much weights are left for each level? 

Typos:
L179: “for”, L453: “at least”

limitations: This paper has some discussion of the limitations in section 6.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper deals with performing pruning with respect to data. The emphasis of this work is in understanding the dynamics of the learning process. More specifically how do the early stages of the training help you with performing pruning based on the neural network weights.
strengths_and_weaknesses: The paper is extremely clearly written. The claims are very crisp and clear. More importantly the impact of this is pretty huge, if this will hold at scale for very large data sets. 
The method section is well explained and the (continually hammered in ) connection to the lottery ticket hypotheses gives good context to the solution 
Albeit there are a few things that are still very empirical/ad hoc: 
line 136 - is the 2% a stiff criteria ? Or did you try to change it. More importantly what would have been very convincing if this would have been tried on several modalities of data. I have a sneaking suspicion that for tabular/time series this would not hold.
Line 147 - Is P the probability of the network, the wights ?

A few typos and assortment of other things
line 115, missing a space
line 118 - double dashes 
* sometimes authors relay way too much on references for example EL2N context some of the benchmarks of GWMP other methods etc...
** This is extremely nitpicking but, the reviewer is colorblind, such it took me a but to correlate in Fig 5 Easiest examples and low EL2N, can you please make it consistent between the title of the figures and legend of the scatter plots ?
questions: Did you try this on different modalities of data ?
How does the loss landscape look like before and after the pruning ? Did you try and visualize it ?
limitations: No There is no discussion on it (or perhaps I missed it in the supplementary material )
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the pre-training phase of the IMP procedure. The authors show that only a small fraction of data is required for finding a matching initialization. In particular, the authors show that the length of such pre-training is reduced if we use as training set only the easiest examples. Then, they also introduce two measures of the loss landscape of the dense network that correlate well with the IMP performance of the pre-trained initialization. The authors also investigate if the same behavior can be observed during learning rate warmup.
strengths_and_weaknesses: The paper provides new insights into the role of the pretraining phase for finding matching lottery tickets. I think that the empirical evidence shown in this paper can significantly contribute to the research in this field, reducing the amount of data and time required for this training phase. The contributions presented in this paper also help to shed light into the behavior of the network during such pretraining phase.
The paper is well written and easy to follow. The experimental validation of the assumptions presented in the paper is solid.
questions: The authors present all the results using the ResNet architecture. It would be interesting to show if the same behavior is confirmed also with other architectures (such as VGG or WideResNet).  I understand that a thorough validation with different architectures will require a significant additional number of experiments. However, I think that it is still interesting to verify the most important assumptions with other architectures in a limited number of experiments.
I found the experiment with randomized labels (pag. 7) not very fair because, when you perform training on easiest data, you select almost only data with uncorrupted labels.
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the role of data in finding early-stage lottery tickets. With carefully-designed experiments, the authors showed that training on a small fraction of easy data suffices to get a good initialization for IMP and in particular it takes fewer pre-train iterations. In addition, the authors proposed to use the distribution of per-example loss barriers to predicting the quality of IMP initialization. Overall, these findings offer interesting insights into the early phase neural network training dynamics.
strengths_and_weaknesses: Strengths:
- The paper is well-written and quite easy to follow.
- Although the ingredients are not new, the idea of studying the role of data in finding good lottery tickets is novel. I believe some of the findings of this work are interesting to many people in our community.
- Experiments are well-designed and support the authors' claims very well.

Weaknesses:
- All experiments are done on relatively small datasets. Quite often, the results on CIFAR do not transfer to larger datasets like ImageNet. One or two runs on ImageNet would make the results more convincing and I believe nowadays most academic labs can afford to run a few experiments on ImageNet.
- The authors only used one single metric to rank the training data. It would be great if the authors could try out one or two other metrics. In addition, it seems that the data selection procedure could be expensive. I think the authors should add some discussions in the main paper about that.
- The authors discovered an interesting phenomenon but didn't give any explanation of why easy data could help.
questions: See my comments in the weaknesses section.

Other questions and suggestions:
- For experiments with only a subset of data, do you also use the same subset of data for IMP or the whole dataset?
- I encourage the authors to dig a little deeper into why easy data would help. I believe this would make the paper much stronger.
limitations: The authors discussed the limitations at the end of the paper. I don't see any negative societal impact of this work.
review_title: null

Meta-review: metareview: This paper presents comprehensive experiments studying the role of data in finding lottery tickets in the early stage of training. All reviewers liked the paper and agreed that the paper has novel and insightful results worth sharing with the community.
confidence: Certain
Decision: Accept