Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a new benchmark and results for evaluating sparsity methods on diverse tasks called SMC-Bench.
Throughout difficult tasks the paper finds that existing works on sparsity may fail at very early in the level of sparsity in comparison to what is known previously in the literature for rather simpler tasks such as imagenet classification or language modeling tasks.
This result can imply a few different things including ineffectiveness of existing methods and the need for new ones or even the need of re-thinking the benefit of sparsity itself.

strength_and_weaknesses: Strength
- This paper provides a range of new and difficult evaluation tasks for pruning and serves as a new benchmark.
- Overall the paper is well written.

Weaknesses
- The results achieved do not seem to provide significantly new or unknown information in the community. They basically show that iterative magnitude based pruning after training works well. The discovery is where pruning methods start to fail, but for more complex and difficult tasks it is quite obvious and expected that they will fail earlier. In this sense, the paper quite exaggerates the benefit of this new benchmark.
- This paper serves to provide empirical results but also draw comparisons and conclusions based only upon it rather than any analysis of methods themselves; it is like A is better than B because of the performance, but without providing why. Perhaps it suits the purpose of creating a benchmark though, but to point out a limitation.
- All evaluated methods are saliency-based pruning methods, among which all methods are magnitude based approaches except for random and SNIP. This is definitely a degrading factor, considering that throughout the paper the authors address in words a lot of different pruning approaches even including variational approaches and sparsity-inducing regularization based approaches.
- The benchmark is missing one critical aspect of evaluating sparsity methods: lack of consideration of the total computations needed to achieve the trained performant sparse networks. Precisely, given a network model, how many forward/backward passess (or gradient computations) are needed for the entire process of finding and training the sparse network result, for each and every method being compared? Why is this important? It is because the benchmark claims to serve as a new standard evaluation protocol providing comparisons that one is better than the other. With different budgets or without even mentioning it properly is simply not fair for that matter. Potential evaluation protocols may include (1) for a fixed budget of computations, how good one method compares to the others in terms of final accuracy or sparsity level, or (2) for a fixed target accuracy/sparsity, how efficiently/faster one method achieves the goal compared to the others. It would also be better to include, at least in words level, how simple or difficult it is to tune the algorithms or hyperparameters (which includes the winning process in the LTH). Without these considerations, this benchmark may only provide a partial result.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, original, and reproducible.
summary_of_the_review: This paper develops a new benchmark for evaluating pruning. There still remain concerns to be resolved for the paper to serve as a new pointer for sparsity research.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new set of benchmarks dubbed "SMC-Bench" (Sparsity May Cry). The authors correctly remark that the ever-growing research in sparsifying neural networks is often (if not always) evaluated on limited set of benchmarks which are often one of the very well understood, and "relatively easy" datasets.

The authors then walk us over the different ways of sparsifying a neural network - providing a good summary of the current state of the art.

The paper continues with the introduction of the dataset, and providing experiments that author use to support their claims. 
strength_and_weaknesses: The paper attaches an obvious question that seems not to have been addressed yet.  It's obvious that the authors spent an enormous amount of time and effort to collect all the results.

The authors promise to open-source their dataset, however no more details are provided. Additionally one would expect a well organized list of the "10 open source repositories" used for creating the benchmark. The open-sourcing and hopefully stream-lining future benchmarks is the most valuable part of the authors' work.

I would also expect the authors to evaluate the networks on the "weak"/"easy" datasets - to confirm that their implementations/runs are consistent with what was claimed in the intro
clarity,_quality,_novelty_and_reproducibility: The manuscript is mostly very clear - tho, as mentioned above, I'd expect a bit more details (or better organized/prezented) list of used benchmarks, and more info about open-sourcing.

The reproducibility is limited with the current state - but hopefully will be trivial after the authors open-source.
summary_of_the_review: Please see above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors assemble a large-scale, challenging, and more diverse benchmark, SMC-Bench, for pruning and sparse training algorithms. A careful evaluation indicates that SMC-Bench significantly challenges current SOTA sparse algorithms, exposes their limitations and points to new research opportunities. 
strength_and_weaknesses: Strength:
- This paper raises a very interesting new angle on the uprising field of sparse neural networks. Given the enormous that sparse neural networks have achieved, the authors ask a timely question: are current benchmarks sufficiently challenging for them, and (if indeed so) are we too optimistic in declaring the prevailing win of sparsity? The authors choose a few latest datasets and tasks that require substantial reasoning and structure information. This new benchmark will make an exciting addition to several existing sparse NN benchmarks such as Frankle et al. (2021).

- Comparing a large range of before-training, during-training and after-training pruning methods (including both static and dynamic), the authors observe several surprising phenomena that are absent in the current evaluation. Most alarmingly, all of the SOTA sparse algorithms bluntly fail to perform on SMC-Bench, sometimes at significantly trivial sparsity e.g., 5%. The failure does not appear specific to one approach but unanimously across all sparse NN approaches that were evaluated. That points us to some rather fundamental rethinking of both sparse NN training and evaluation currently.

- The authors also identified that model “prunability” is intimately related to task difficulty: those models trained on difficult tasks suffer more from pruning compared to easier tasks. Also, some best pruning algorithms start to perform indistinguishably from the random pruning baseline on the new challenging benchmark, and iterative pruning does not necessarily generalize better than one-shot pruning either. All those findings offer brand-new knowledge of broad interest to the sparse NN community.

Weaknesses:
- The paper did not identify a root cause for the failure of SNNs on SMC-Bench, although exploring then excluding a few factors. I feel one main reason is probably intimately related to the “lazy training” dynamics emerging in sufficiently overparameterized models, e.g., after training weights will only change very little from initialization. 
Moreover, in one of their sanity checks on whether pruning embedding layers or not, the authors found SNIP to significantly underperform others because it prunes too much of the pre-trained embedding while others not. I am wondering whether we could ad-hoc skip that embedding, for all pruning methods and then compare them?  

- I strongly suggest the authors to add to their comparison one or more second-order pruning methods. such as Optimal Brain Damage, Fisher Pruning, or the recent Optimal BeRT Surgeon. 

- Also missed in the current benchmark is the structured pruning family: will they be more robust or fragile when task difficulty and data volume scale up? How about sparse MoE methods?

clarity,_quality,_novelty_and_reproducibility: Clarity: easy to read and understand

Quality: solid and comprehensive work

Novelty: high

Reproducibility: good (the authors promised to open-source)

summary_of_the_review: This paper proposes an interesting method and the major concerns lie on more comparisons in the experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces “Sparsity May Cry” Benchmark (SMC-Bench), a collection of carefully curated 4 diverse tasks with 12 datasets, for a more general evaluation and unveiling the true potential of sparse algorithms. Evaluation reveals several important and unusual findings. 
strength_and_weaknesses: Overall, I enjoy this work a lot and think it will make a big/good splash in the sparsity community. But I also believe its scientific rigor needs to be enhanced in several aspects before being treated as an indeed solid benchmark. See details below.
 
First, it might be too early to declare sparsity will cry! For example, only first-order, unstructured pruning has been so far tested, which represent a strong yet still restricted subclass of modern pruning methods. I would definitely suggest the authors to pick representatives from other pruning families, such as second-order pruning.

Second, how the authors could justify their chosen datasets and tasks are indeed more challenging than ImageNet? IMHO, the sole fact that current pruning methods work better on ImageNet than those cannot directly support the claim, since it could also be some task types that are just more “pruning (un)friendly”. 

Similarly, how the authors can justify their chosen models (admittedly very large already) are sufficiently “overparameterized” for those new difficult tasks? For example, would it be possible that, if continuing to increase the model size on those tasks, the generalization performance will keep improving, while pruning algorithms also start to gain back their effectiveness?

clarity,_quality,_novelty_and_reproducibility: Writing is very clear. High technical quality and novelty as a “new perspective” benchmark in sparsity and pruning.
summary_of_the_review: Overall, I would tend to accept this paper. If the author can address my concerns. I would be more convinced.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new benchmark for evaluating the performance of sparse neural networks. The authors argue that existing benchmarks are too simple and do not reflect (all) real use cases of sparsity. To address this issue, the authors introduce the "Sparsity May Cry" Benchmark (SMC-Bench). 

The authors evaluate a range of state-of-the-art sparse algorithms on this benchmark, and find that many of them perform poorly, even at low levels of sparsity. This observation highlights the need for the development of more generic sparse algorithms.

One reviewer raised the important point that the benchmark does not adequately consider the total computational cost of training a sparse network. This hasn't been addressed by the Authors. I agree it is a key point, but at the same time the benchmark is sufficiently challenging to help drive the field even with this limitation.

Another important weakness is lack of breadth in evaluated algorithms. Authors have partially addressed this during rebuttal. I believe Authors have shown sufficient evidence that the benchmark is challenging, and future work can evaluate more diverse methods.

The key strength of the paper is helping define a new challenge for the field, which will likely help drive future progress.

Overall, the paper presents a valuable contribution to the field by introducing a new, challenging benchmark for evaluating sparse neural networks. Its acceptance (ultimately the least supportive reviewer also leaned towards accepting the paper) by all reviewers highlights the importance of this work and the need for further research in this area. 

Thank you for your submission. Please remember to address all reviewers' comments in the final camera ready version.
justification_for_why_not_higher_score: The results are impactful but not enough to warrant the "Accept with Oral" category. It is hard to predict whether the paper will impact the field going forward in a very significant manner, as there are some concerns such as lack of computational time as metric or exclusion of certain methods.
justification_for_why_not_lower_score: The paper is likely to have a sizeable impact on the field by focusing it on more challenging tasks.
Decision: Accept: notable-top-25%