Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary 

The paper provides an extensive empirical analysis of Pruning-at-Initialization (PaI) techniques and compares it against two pruning methods after (or during) training. This comparison sheds some light on why pruning at initialization is inherently hard. Furthermore, the comparison among PaI methods with various ablations shows some inherent properties that are common to PaI methods and the benefits/drawbacks of certain methods. With these experiments, certain conclusions are reached among them an important one is that PaI methods only determine what is the fraction of weights to be pruned in each layer rather than which weights to prune.

## Strengths

1. Quantitative comparison of PaI methods is an important contribution to the community. In addition, the paper extensively analyses them with various ablations (eg, shuffle the layerwise masks, reinitialization) which uncovers various properties of PaI methods previously not conveyed clearly in the respective papers.

2. Extensive analysis shows that there is a performance gap of PaI methods compared to Pruning-after-Training (PaT) methods (understandably) and attempts to explore the potential reasons. This analysis uncovers some inherent difficulties of PaI. The analysis provided in this paper could serve as a guide to better understand and improve PaI methods.

3. Overall the paper is clearly written and sufficient details are provided in the appendix.

## Weaknesses

The main weaknesses of the manuscript in my opinion are as follows:

1. Some conclusions of this paper are unsubstantiated or should be rephrased:
	- First, the experiments mainly convey the inherent difficulties of PaI rather than the issues with specific methods themselves. To address this concern, the methods (SNIP, GraSP, SynFlow) are performed during training and showed that they perform inferior to LTR in Fig. 6. However, this comparison is unfair as LTR has additional information which is obtained after training (pruning mask is obtained after training but applied during training) and those PaI methods are not specifically designed to be performed during/after training (even so some perform reasonably well). In short, it is not clear that the performance of PaT can be matched by PaI given only the information at initialization. Note that, having access to the pruning mask after training defeats the purpose of PaI since given an already trained network, one might as well simply prune at the end (no need to retrain from scratch). I believe, Sec. 7 has some discussions about this (not complete) but should come early in the paper and emphasized. Please clarify and rephrase certain parts (especially in the introduction). 
	- One of the main conclusions: "PaI methods only determine what is the fraction of weights to be pruned in each layer rather than which weights to prune" should be rephrased. In fact, it is possible to have a pathological case (eg, disconnected layers at a given pruning ratio) where knowing the optimal pruning ratio is not sufficient to obtain matching accuracy. These pathological cases are not observed in practice due to the random component (initialization or shuffling) and it should be emphasized. Also, since the weight initialization is iid Gaussian, there may not be sufficient information for PaI methods to select each weight individually but rather select them as a group in each layer or whole network.
	- Another hypothesis that the performance gap between PaI and PaT is due to the robustness of PaI methods to shuffling, reinitialization, etc, are also not clear. I understand that there is a correlation exists that some PaT methods are not robust to such variations. But it is NOT clearly demonstrated that being robust to such variations is necessarily a bad thing for PaI. In fact, one would think it is a good thing that PaI methods are robust to such variations given that there is not a lot of information available at initialization (note initialization is iid) to perform effective pruning and it seems these methods are robust and perform competitively to unpruned networks.

2. Inconsistency of SNIP being independent to initialization:
	- In the last paragraph of page 5, it is mentioned that SNIP is independent to what initialization is used. However, there is a follow-up work of SNIP showing that it could be beneficial for SNIP if the network is initialized to have good signal propagation [a]. Please discuss this in the context to avoid any misinterpretations.

3. Issue with random shuffling:
	- The random shuffling experiment needs some refinement in my opinion. I believe PaI methods such as SNIP, GraSP or SynFlow might have some unpruned weights which are not updated during training (ie, they are disconnected in the signal propagation path). In my personal experience, I observed that there are about 1-2% of unpruned disconnected weights in the network (in particular layers this value could be up to 10%) for SNIP (meaning they can be removed) depending on the network. This means the effective sparsity is slightly lower than what is observed in SNIP (not mentioned in the original paper though) and presumably in other methods as well. This observation would be applicable to random shuffling as well. I mean, after random shuffling there might be some unpruned disconnected weights and they could be removed before training, leading to higher effective sparsity. Furthermore, the existence of unpruned disconnected weights could be the reason for similar behaviour even after random shuffling in each layer. 

I found the experiments in this paper to be thorough but I think the deduced conclusions are slightly off with respect to the observations in the experiments. I believe this paper will be a good contribution to the pruning literature if the conclusions are tightened.

## Minor Comments

1. Related work: SNIP is not a follow-up of LTH but rather they are concurrent works published in ICLR 2019.
2. Abstract: I think "undermines" might be a strong word given that there are questions regarding deduced conclusions.

## References

- [a] Lee, N., Ajanthan, T., Gould, S. and Torr, P.H., 2020. A signal propagation perspective for pruning neural networks at initialization. ICLR.
review_title: Some conclusions of this paper are unsubstantiated or should be rephrased

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:

Generating a pruned network falls into two broad categories: 1) spend some extra time and effort to train or fine-tune the pruned model after first training a dense version, or 2) cut out that extra time and effort by generating a sparse network "from scratch."  While approach (1) has historically given the best accuracy, recent advances (such as the lottery ticket hypothesis) suggest that there are sparse networks hidden in the initialization that don't need to first be trained, if only we could divine the structure of those models.  Approach (2) seeks to do just this: determine the connectivity as close to initialization possible.  However, even the best results taking this second path fall short when compared to the accuracy of the former path - why is this?  The submission pokes at three recent techniques to pull out some commonalities that are *not* shared with (1), suggesting possible issues that need to be overcome to improve accuracy, and proposes a set of experiments and comparisons that should be part of any new technique that claims to discover a good sparse mask at initialization.

##########################################################################

Reasons for score: 

Overall, this submission raises some important questions (why is from-scratch pruning falling short of SOTA accuracy?), empirically shows the performance of some leading techniques (and that they fall short in a well-motivated range of sparsities), and, via ablation studies, points towards some potential reasons.  More importantly, these findings are interesting:
- There's no single SOTA method for sparse-from-scratch training
- There's a need for consistent reporting in this area, and the ablation studies performed have been shown to lead to useful insights; adopting them as standard for future work seems fruitful.

I think these are enough to warrant an "above-the-threshold" rating, but a higher rating would require empirical results on more networks on large-scale data sets, or the inclusion of techniques the submission itself suggests might be a promising direction forward in the current gauntlet of tests.

##########################################################################

Pros:

+ The organization of the paper makes it easy to follow the logical progression and points being raised.
+ The direct comparisons of three recent techniques (SNIP, GraSP, and SynFlow) on different networks and data sets fills in some gaps in the literature.
+ Further, the ablation studies performed on these techniques yield surprising results, both in isolation (inverting GraSP improves accuracy!) and when compared to magnitude pruning after training (these three are invariant to shuffling and re-initialization).

##########################################################################

Cons:

- Experiments on large data sets are limited to RN50 on ImageNet.
- The three particular techniques chosen (SNIP, GraSP, SynFlow) aren't particularly motivated - why these three, and not other recent techniques?  (A potential answer may be that there's no training before pruning is finished, but why is this important?)
- Mostly tongue-in-cheek: the submission doesn't answer all the questions it raises, unfortunately.

##########################################################################

Questions:

- What benefit do the three at-initialization "static" pruning techniques have over those that reduce training FLOPs and memory requirements but allow the sparse mask to change dynamically, like RigL (Evci et al., 2020) and sparse momentum (Dettmers and Zettlemoyer, 2019)?  Is there a reason they do not belong in the current lineup?  The overhead of occasionally updating the mask shouldn't be too imposing.

- In the final paragraph, it is suggested that it may be tricky to compare the training cost of "a method that prunes to sparsity s at step t against a method that prunes to sparsity s' < s at step t' > t.  If method A prunes to a higher sparsity at an earlier time step, shouldn't it cost strictly less than method B, which prunes to a lower sparsity later in training?

##########################################################################

Minor suggestions:

Figure 2 is never referenced in the text (that I could find), and LTR isn't defined until the following page.


##########################################################################

Updates are appreciated

Hi, Authors,

I appreciate the updates you've made to the paper and the responses to my questions.  You're quite correct that RN50 and ImageNet is sufficient to illustrate deficiencies.

(I'd still want to see broader experiments for claims of some new method overcoming these deficiencies, though!  When broadening scope to other tasks, I'd expect the authors of prior methods designed for vision tasks would be okay with use of their methods as baselines if there are no methods designed specifically for those new tasks.)

With this in mind, I'll update my rating to a 7.
review_title: Official blind review #3

final_score: rating: 4: Ok but not good enough - rejection
review_content: This is an empirical work assessing some of the recent works of early pruning methods in an effort to understand why early pruning methods perform worse than pruning after training methods. The paper is written very well, and some of the findings are interesting leaving important questions to address. Some major concerns remain though.

The paper proposes to achieve the following goal: to understand why early pruning methods perform worse than pruning after training methods.To achieve this, what the paper chooses to do is basically to measure their final performances and compare them to each other, and then to check for some ablation studies.

Obviously, the results (Figure 3) show which performs better than the other (i.e. pruning after training > pruning early), meaning that this result itself in terms of the final performance does not suffice to prove any cause or effect to explain the behaviors in comparison aimed to understand.

In the ablation studies (Section 5), the paper appears to be focused on studying and debunking the early pruning methods rather than finding *why* pruning after training is better than pruning early. For example, the result of random shuffling (early pruning methods for finding layerwise proportions) and the failure of GraSP when inverted are quite interesting, but they remain empirical without contributing directly to fulfill the purpose; some of the conclusions drawn from the ablation studies are not convincing (e.g., lack of specificity or insensitivity to initialization may limit performance); some of the claims are not expected to generalize either (e.g., shuffling to fully-connected layers, initializing with different standard deviations of the normal distribution).

In fact, an intuitive ablation setting is to test for the effect of training with the same saliency measure, as done in Section 6, but unfortunately the result out of this seems to be unhelpful for figuring out the potential cause of poor performances for pruning early methods. The results however show that early pruning methods improve after some training, indicating that training to some degree before pruning helps to increase the performance of the pruned model. One of the potentially many reasons that pruning early methods is underperformed by pruning after training methods could merely be the fact that the former is not trained before it gets pruned as opposed to the latter yielding the observed performance gap quite obviously. Anyway, the fact that they (after some training) still do not perform better than LTR (which requires oracle information) does not contribute to proving anything for why pruning early methods is underperformed by pruning after training either.

In short, this work does not explain "why all methods fall short of magnitude pruning after training" in a convincing manner, and the contributions are considered not quite significant overall.
review_title: Not convincing enough to support the claim

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: A recent trend of 'pruning at initialization' in neural network pruning has left me baffled. It's counter-intuitive that neural networks can be pruned at initialisation, improving results for the training done thereafter. Nitpicking semantics, one could hardly even call this a pruning technique, since there is no a-priori knowledge of the dataset distilled in the network. Perhaps it's more aptly referred to as a method of sparse initialisation methods.
The authors of this paper seem to have had the same doubts when it comes to the pruning at initialization literature, and put the magnifying glass on these methods that have recently been published. In an extensive and comprehensive study, they show that pruning at initialization methods do naught but set per-layer sparsity rates, where the sparse initialization might as well have been random.

Writing these types of papers is important. It's essentially a survey paper, reproducing results from other papers, and testing their claims. In moving forward in this field, this type of work is crucial in filtering out the sense from the nonsense. Although the paper does not provide any new shiny optimization method, or stellar new GAN with a funny name, I highly laude the authors for working on this survey, and I believe the impact on the model efficiency community of this work is significant.

Now on to the nitty-gritty of the paper. The key argument that shuffling the weights within a layer, essentially functioning like a re-initialization with a given per-layer sparsity ratio, is solid, convincing and damning. The inversion arguments equally so. The only qualm I have with the results are that there are not more on different architectures... but I hardly think that's necessary to make the point. If these methods were to work as intended, it should show on the common computer vision architectures we work with.

The paper is well-written and well-structured. Clear and concise, with an extensive appendix highlighting more background information, and showing that the authors know the field very well. They covered every angle I could think of to put a crowbar in the paper and pry open some problems. 

I would have liked to see a discussion on why we would even want to prune at initialization, because the arguments in the paper only really come to light if we consider the utility of the methods. Sure, pruning at initialization is worse than pruning, but perhaps there are reasons to prune at initialization anyway. Are we looking at this as a research exercise? A quest to improve our understanding of networks? Is it done for sparse training? If it's the latter, a discussion on this, and perhaps a comparison to other sparse training methods would be in order.
review_title: An excellent tour-de-force review which carefully understands, investigates and scrutinizes recent work in neural network pruning from scratch

Meta-review: The paper analyses several approaches to pruning at initialization, compared to after training. There was a large gap in reviewers appreciation of the paper, but I think that the pros outdo the cons as the paper show a lot of insights overall. I recommend accepting the paper.
Decision: Accept (Poster)