Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the effectiveness of random pruning for sparse training. Specifically, the paper adopted several random pruning methods(ERK, Uniform, Uniform+) and compare the results with non-random Pruning methods(SNIP,GraSP). They are conducted on different ResNet scaling models, with  CIFAR-10/100 and ImageNet dataset under different  sparsity ratios. Many observations and conclusions were proposed in the experiments. Results show that random pruning can be quite effective for sparse training especially on larger or wider models.  In some cases, it can outperform well-versed pruning approaches and match the performance of dense networks. 
main_review: Strengths:
1. The paper draws attention to the commonly used baseline method-random pruning and finds that it can be very effective for training a sparse network.
2. This paper explores and evaluates various random pruning methods with different layer-wise sparsity ratios. Adopting these layer-wise sparsity ratios as standard sparse initialization for future work seems necessary.
3. Experiments are promising, and analyses are very detailed.
4. The findings of the gradient norm seem consistent and correct, making good connections with previous works.
5. The paper also assesses random pruning from other perspectives for broader evaluation.

Weaknesses:
1. The effectiveness of random pruning has been investigated in previous works before, although not as detailed as in this paper.
2. There is not much novelty in the method itself. Mainly using multiple existing pruning methods to support their claim.

Questions for Author: 
1. In the paper, you mentioned two categories (static and dynamic sparse training), and states that random pruning lies in the static sparse training. In the dynamic sparse training prune-and-grow scheme, if you randomly prune and grow the weights, can it be considered random pruning? 
2. Can these observations and conclusions potentially apply to other popular models such as VGG or Transformer, etc.? 
3. Reference styles are inconsistent, for example (Evci et al. , 2020a ) and Lee et al. (2018)
summary_of_the_review: This paper re-evaluates the underrated baseline of random pruning in sparse training. The paper is well-written and well-structured. Although the paper does not provide new pruning techniques,  I believe the conclusions from the paper can contribute to the sparse training domain.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses pruning at initialization (PaI) and they argue random pruning can be quite strong actually, whose performance was under-rated in the past. They present abundant experiments to show random pruning can perform on par with other PaI methods with dedicated pruning criteria. They further show random pruning networks can outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness.
main_review: Strengths:

1.	As said in the paper, random pruning is the very baseline for all pruning methods. Its performance was believed to be quite bad in the past, while this paper shows evidence on the contrary. This appears quite surprising and interesting.
2.	They have extensive experiments trying to support their claim. 
3.	They further show random pruning networks can outperform dense counterparts in other favorable aspects. Potentially, this can make random pruning more useful in practice.

Weaknesses:

This paper appears to present surprising results (as indicated by the title -- The “Unreasonable” Effectiveness). However, the claims and contributions in this paper need to be rectified and re-evaluated.  

First, a more rigorous way to treat the terminologies is needed. Pruning can be split into pruning a pretrained model (which is the traditional way), and pruning at initialization (i.e, pruning a randomly initialized network). What this paper really discussed and compared is pruning at initialization (PaI). PaI is only a relatively small track in pruning currently. And importantly, it has been shown PaI seriously underperforms the traditional pruning (pruning a pretraiend model), see [*1]. So the title should be “The Unreasonable Effectiveness of Random Pruning at Initialization”, not the most general pruning case. 

Otherwise, if you want to claim random pruning can be on par with other pruning methods (let’s take the most simple magnitude pruning as example) under the traditional pruning case, please try to add this result: Prune a pretrained ResNet50 with 90% sparsity on ImageNet, compare random pruning with magnitude pruning (extensive previous methods actually have shown the latter is better).
Then, if we are talking about pruning at initialization, the so-called “unreasonable effectiveness” is actually not that unreasonable, provided you’ve seen [*1], which is cited as Frankle et al. (2020b) in the paper but I don’t think it is well discussed. [*1] has quite a lot of overlap with this paper.

Specifically, in [*1], they propose several sanity-check rules to evaluate whether a kind of pruning criterion is really effective or not: Randomly shuffling, Reinitialization, Inversion, in their Sec. 5. Particularly, for the random shuffling, they shuffled the masks obtained by some PaI methods (like SNIP, SynFlow, GraSP) and found  after mask random shuffling, these methods can achieve comparable accuracy (“All methods maintain accuracy or improve when randomly shuffled”). Randomly shuffling masks is essentially the same as random pruning. That is, in [*1], it has been shown that random pruning can perform on par with other PaI methods like SNIP. So the major point of this paper is actually reiterating what has been found in [*1]. 

Even similarly, in [*1], after discovering randomly shuffling masks does not hurt accuracy, they thus proposed “In other words, the useful information these techniques extract is not which individual weights to remove, but rather the layerwise proportions by which to prune the network”. Then look at the 2nd contribution claimed in this paper: “We further identify that appropriate layer-wise sparsity ratios can be an important booster for the performance of random pruning”. They are basically the same.
Given above, this paper has non-trivial overlaps with [*1]. Although it presents more empirical evidence to show random pruning can be useful in some cases (out-of-distribution detection, uncertainty estimation, and adversarial robustness), yet the major claims and points are already established in [*1].

Also, in the experiments, some are questionable. E.g., they compare pruned WideResNet50 to ResNet50 (Sec. 4.2). Although they have similar #params, notably they are two different networks. The comparison is not an apple-to-apple comparison (not scientifically valid), no matter how surprising the result may look. To claim random pruning is effective, what should be done is to compare randomly pruned WideResNet50 to a dense WideResNet50, not ResNet50. If the authors can provide this result and still show they have similar accuracy, I will improve my rating.

[*1] Pruning Neural Networks at Initialization: Why are We Missing the Mark?. ICLR, 2021.

summary_of_the_review: This paper has substantial overlap with a previous paper. Also, they have unfair comparisons in their experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Sparse training has traditionally relied on carefully crafted sparse patterns or pruning criteria. This paper reports a novel discovery that without any delicate pruning criteria or careful sparsity structures, training a randomly pruned sparse network from scratch can match the performance of its dense equivalent.
main_review: Strengths: 
-	This paper systematically revisits the underrated baseline of sparse training – random pruning. The authors present a series of very interesting findings with solid experimental supports. 
-	The findings first highlight a hidden gem role of larger models – their high tolerance to sparse training. As the networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios. This is an inspiring new insight of broad potential impact in current ML trend, especially how to train larger models more efficiently.
-	Another meaningful discovery is that, echoing prior arts, the layer-wise sparsity ratio or “effective width” seems to be a critical factor for high performant sparse training. Borrowing ratios from SNIP could lead to pushing the performance of a completely random sparse Wide ResNet-50 over the strong baseline of densely trained ResNet-50 on ImageNet. However, this part lacks deep understanding of its underlying cause – see first bullet under “questions”.
-	The effectiveness of ERK on ImageNet (with the help of sparsity ratios) is surprising and impressive – perhaps the strongest I’ve seen in relevant empirical literature. It can outperform SNIP/Grasp for many cases, both in accuracy and in robustness/other aspects. 
-	Codes are included as supplementary, and experiments have included sufficient details to be reproducible. 


Questions or Suggestions:
-	One weakness I feel about this paper is its interpretation part in Section 4.3. Especially, I find it hard to comprehend what “early-stage gradient gaps” mean or can reflect for sparse network optimization. The authors basically described the phenomenon then stopping without digging more insights.
-	Moreover, if early training dynamics are indeed essential for improving sparse training, can this be combined with dynamic sparse training (e.g. Evci 2020a)? As one possible way to balance randomness with learned connectivity, could there be some dynamic sparsity ratio re-allocation during training? The authors are encouraged to discuss more.
-	A minor limitation of this work is that random sparsity will only work so well when the network is large/wide enough, as the authors also pointed out. 
-	(Optional) With SNIP/Grasp being evaluated, it is recommended that the authors can compare with SynFlow and some more recent sparse training baselines too. Including comparison to RigL would be interesting too.

summary_of_the_review: The work questions the common wisdom of sparse training and found properly trained randomly pruned sparse networks from scratch can match dense networks in many ways. This finding is counter-intuitive yet important. The paper could yet be strengthened further by providing more interpretations of its observations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper revisits the OG and the simplest pruning baselines out there: Random Pruning. Random Pruning randomly zeros out the weights in a network with a target budget and is often used at a layer level based on the chosen sparsity ratios. 

This paper is a detailed study into Random Pruning. As put forward in contributions, the discussion includes the dependence on network size, the importance of sparsity ratios, and lastly the general-purpose aspects like robustness and OOD, etc of these pruned networks. 

The observations seem to point to the fact that as the model size increases the pruning method becomes obsolete and everything converges to the same solution. They also show that sparsity ratios matter more than anything else for pruning techniques. These two observations have been made in the past in different settings but still are valid observations. 

Lastly, the paper extensively tries to evaluate the robustness aspects of these models + sparsity ratios for CIFAR. 


main_review: I will go sequentially and start with strengths and proceed to weaknesses:

The writing of the is really good and I appreciate the authors for their efforts. 
I also loved the extensive related work which will act as a great stepping stone for new researchers the same way Gale et al 2019 did in the past 2 years. 

Strengths:
1) The revisiting of Random Pruning makes a lot of sense give the more complex pruning methods based on Hessians etc becoming a norm. The evaluation of this simple baseline should give perspective to new readers about what components affect the final deliverables most. 
2) Very well crafted related work section 
3) THe methodology is very clear and makes a clear distinction of where Random Pruning falls into the literature of learnable sparsity etc.,
4) The sparsity ratios being discussed are well explained are extensive. 
5) The experimental setup is extremely clear and reproducible. 
6) The results on CIFAR are extensive with various architectures and variants among them along with a very nice section 4.1 listing the observations that come of these evaluations. 
7) While not as extensive similar observations were made on ImageNet with Wide ResNet50 as the base architecture. 
8) The observations made on the importance of pruning ratios, the indifference of pruning methods at larger model sizes are valuable (while not novel). 
9) The experiments trying to understand random pruning via gradient flow is interesting and give some perspective as to why certain pruning ratios might be doing better. 
10) While only on CIFAR, I love the extent of broader evaluation on uncertainty, OOD, Robustness, etc for these random pruning models with different architectures and pruning ratios.
11) The plots are very clean and authors should be appreciated. The takeaways are explicitly mentioned making it easy for first-time readers. 
12) The appendices are well laid out and detailed.

Weaknesses:
1) As I mentioned earlier, the method itself isn't novel and the certain observation of pruning larger models being better than dense counterparts of the same size isn't novel. Gale et al., 2019, Wallace et al 2020, Kusupati et al 2020 make the same observation but even Zhu & Gupta 2018 make similar observations and it is often observed that pruning large models is called large-sparse is better than small-dense. That is the same observation being made here. 
2) The argument about the sparsity ratios being super important is not new either. There are papers like Su et al NeurIPS 2020 which argue that sparsity ratios are all that matter even in the LTH space and randomly changing the weights still gives them the tickets needed to win the lottery. This paper is missing in the related work but I am pretty sure (I am probably blanking out) there are papers that talk about the importance of sparsity ratios. Even papers like RigL and STR make a case for why sparsity ratios are important in general. 
3) While on the topic of sparsity ratios, I also noticed that the benchmarked pruning results don't have inference cost involved in these cases. For example, I would love to see the FLOPs for Fig 4 (right) to see what the FLOPs are for pruned WRN vs a Dense ResNet. 
4) I have no qualms about the broader evaluation except that I would love to see them on ImageNet even if it is for a single architecture. Is that something possible?

Overall, I think I don't see as much novelty in this paper (which is fine) however even the observations made are not completely novel. The evaluation is thorough and the observations are clean. However, I want to hear from other reviewers and the authors during the discussion before I move my decision towards acceptance. This is a great benchmarking paper, but at times looks like stitching across insights from other papers. Looking forward to the discussion. 


summary_of_the_review: As mentioned in the paper, the technique isn't novel however, the detailed study can benefit quite a few in the community given the potential ramifications of such observations. I would say it is a great benchmarking paper that would help people like "The State of Sparsity" (Gale et al., 2019) did, but again it doesn't make a super-strong case for acceptance. I am on the fence with a 5.  However, I am willing to change my mind after rebuttal and discussion with other reviewers. 

I still think this paper is a good asset and will be a great one if the same scale of experiments is done on ImageNet instead of CIFAR and will prove to be a solid exploratory work in that case.

Overall, I am happy with the rebuttal. I want to see these updates and results in the main paper and not the appendix and you can do this as revision in the next few days. 

I am increasing my score to 6 and will talk to other reviewers but push for an acceptance.
review_title: null

Meta-review: ### Summary

This paper builds on previous work on sparse training that shows the many modern sparse training techniques do no better than a random pruning technique that selects layer wise rations, but otherwise randomly selects which weights within a layer to remove.  The key difference in this work is to take these existing results and scale the size of the network to show that as the size of the network increases, the smaller -- as measured in pruning ratio -- a matching subnetwork becomes.

### Discussion 

#### Strengths

Places an emphasis on simple techniques

#### Weakness

Significant overlap with previous work. Prior already demonstrated the equivalence of random pruning and contemporary pruning at initialization techniques.

### Recommendation

I recommend Accept (poster). However, I do want to stress that there is a significant overlap with previous work. The paper does appropriately attribute observations to previous work. However, there is some risk that readers may misinterpret the title and claim results as a wholly new observation about random pruning, where the reality is instead much more nuanced. Given that the work points to new methodological directions on considering scaling the network as an additional parameter to consider in pruning observations, I do believe these results -- even if narrower in scope that can be interpreted -- provide value to the community.
Decision: Accept (Poster)