Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes an algorithm that finds sparse subnetworks at initialization, trainable to accuracy comparable or better than IMP with warm-up.
strengths_and_weaknesses: Strength:
This paper proposes an algorithm that finds winning tickets at initialization and validates the algorithm on several tasks.
The experiments are solid.
Weakness:
The novelty is limited. 
The proposed method is a combination of different techniques.
questions: If I understand the algorithm correctly, about half of the weights would be masked at initialization as score vector p is initialized randomly, which seems a little bit weird to me. According to Algorithm 1, only effective weights receive gradients, so if a weight is masked, it will be masked forever. So, could the authors explain the design behind this or correct my misunderstanding?
limitations: The authors address the limitations adequately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work aims to find lottery tickets at initialization efficiently by a proposed Gem-Miner. 
Previous studies either find bad tickets at initialization (ineffectiveness) or find good tickets after initialization (inefficiency). 
To tackle the open problem, the Gem-Miner is proposed to discover rare gems, which can find good enough tickets at initialization even before weight training. 
The Gem-Miner can be essentially viewed as a bag of tricks for finding tickets. 
Experimental results show that rare gems perform far better than baselines in terms of both performance metrics and sanity checks. 
In addition, rare gems hold the advantage of a faster identification process.

strengths_and_weaknesses: Pros:
1. The open problem is interesting and is somewhat solved by the proposed method.
2. Appealing experiments results in terms of both effectiveness and efficiency.

Cons:
1. There are several presentation flaws as listed in the Questions below.
2. This work focuses on a specific pruning method (i.e., Edge-Popup), and I would glad to see further results of arming Gem-Miner to other pruning methods. 
3. This work can be viewed as a bag of tricks for finding good tickets at initialization. However, ablation studies are not thoroughly conducted for a better understanding. There is a related section describing how Gem-Miner resolves Edge-Popups failings by adding components to Edge-Popup, yet without adequate connections between these components and tricks introduced by Gem-Miner.
questions: Q1:
The authors state that "exactly the same number of epochs" yet "faster than IMP" at the same time in Line 61, which seems ambiguous to me. Would "IMP" consume longer training time for an epoch?
If I understand correctly, the "IMP" in Table 1 indicates "IMP with warmup" instead since it passes sanity checks as stated in Line 40. Then it can be acknowledged that Gem-Miner is faster than "IMP with warmup". However, this would be controversial to the above statement.

Q2:
"after E epochs of weight training" in Line 232 indicates that a pre-training stage is required, so I wonder whether other pruning methods require a pre-training stage if the pre-training is required by Edge-Popup. Would this lead to an unfair comparison?
limitations: Hyper-parameters are sensitive in Gem-Miner but are not listed in this paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes GEM-MINER to find lottery tickets at initialization. The found subnetworks are trainable and can achieve comparable or better performance compared to existing approaches (e.g., IMP), and requires much less time (19x faster compared to IMP). The approach can pass the sanity checks proposed by prior work. The experiments with different datasets, different model architectures, and different sparisities show the effectiveness of the proposed approach. The paper claims it resolves the open problem of hunting for subnetworks at initialization trainable to SOTA accuracy.
strengths_and_weaknesses: Strengths:
- The paper is well-written and very easy to understand.
- The paper conducts extensive and solid experiments to study the effectiveness of the proposed method. The results are strong and exciting, and also provide insightful analysis (e.g., why EP failed).
- The approach is simple and effective. I believe that the paper can facilitate more research in this field.

Weaknesses:
I don’t see a major concern with this paper. 

Minor: why is sparisity defined as the ratio of non-zero weights instead of zero weights? (Based on the current definition, 95% sparsity actually means a very dense model!)

questions: - Can GEM-MINER be applied to NLP tasks?
limitations: The limitations have been well discussed in the paper in an explicit section (e.g., it may require careful hyper-parameters tuning). I don’t see other major limitations of the approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of searching trainable subnetworks at initialization (winning tickets). The authors propose a method called GEM-MINER, which maximises the accuracy of subnetworks before weight training. GEM-MINER can discover subnetworks comparable to IMP with warmup (a widely-used method to find winning ticket subnetworks) while being 19 $\times$ faster. GEM-MINER outperforms the Edge-popup, which is another method that also optimizes the subnetwork structure before weight training. The subnetworks found by GEM-MINER also pass the "sanity check", outperforming random sparse subnetworks.
strengths_and_weaknesses: ### Strengths
* Searching for subnetworks at initialization with better after-training performance is an important problem in the lottery ticket hypothesis (LTH). This paper proposes a new method to tackle this problem and achieves promising results.
* The main reasons (i.e., global pruning and gradual pruning) for the effectiveness of the proposed method are well explained in the ablation study (Tab.3), which can be used as default setups in follow-up works.
* Extensive experiments are conducted to demonstrate the effectiveness of the proposal, covering multiple tasks and backbone models.
* The paper is generally well-written and easy to follow.

### Weaknesses
1. Although the proposal is a new method to search trainable subnetworks at initialization, its core ideas (global pruning and gradual pruning) have already been studied in the literature on model compression [1]. This undermines the technical novelty of this paper.
[1] Movement Pruning: Adaptive Sparsity by Fine-Tuning. In NeurIPS 2020.

2. The effectiveness of global pruning and gradual pruning is a useful finding. However, it is unclear how these two factors contribute to identifying more trainable subnetworks at initialization. Further analysis is required.

3.  Although extensive experiments are conducted, the empirical studies can still be improved in terms of comprehensiveness. For example:
* Comparing with other "pruning at initialization" methods (e.g., SNIP, GraSP and SynFlow).
* The paragraph "Convergence of accuracy and sparsity" can benefit from adding results of Edge-popup and SNIP, etc.

4. Some arguments made in this paper are questionable. For example, "EP subnetworks are not conducive to further fine-tuning" does not always hold. At 5% sparsity, the accuracy of EP subnetworks can be improved from 70.69 to 84.34, according to Fig.2.

5. Limitations of the GEM-MINER are well discussed. However, the authors did not provide any ways of solutions.

### Typos
* In the upper line of Fig.1 (the pipeline of IMP), it seems that the "0.5% sparse Subnetwork at Warm Initialization" should have the same color as the "Partially trained dense network" for the same weights.
* CIFAR-100 is not mentioned in the paragraph "Tasks" (line 226-230), while appears in Fig.4.
* The caption of Fig.4 mentions ImageNet, while only TinyImageNet can be found in the titles of 2nd and 3rd columns.
* Line 272 mentions Fig.4c, which cannot be found in Fig.4.

questions: * Why not compare with SNIP, GraSP and SynFlow?
* Could you remark on how to resolve the limitations you discussed in line 311-321?
limitations: Please refer to the Strengths And Weaknesses and Questions.
review_title: null

Meta-review: metareview: This paper proposes GEM-MINER to find lottery tickets at initialization. It tries to maximize the accuracy of subnetworks before weight training, and can discover subnetworks comparable to IMP with warmup while being much faster. The approach can pass the sanity checks proposed by prior work.

It received scores of 4568. All the reviewers agree that the problem studied in this paper is important for lottery ticket hypothesis (LTH). The reviewers appreciate the authors' extensive experiments and thorough analysis. Most of the concerns have been well addressed, though Reviewer BBr3 still showed concerns on why 'Rare Gem' can solve the problems others could not solve, and commented that more insightful discussions behind the finding are needed. 

Overall, the AC thinks that this paper presents valuable findings, and the responses to reviewers' comments are adequate, therefore, the AC would like to recommend acceptance of the paper.
confidence: Certain
Decision: Accept