Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper understands the during-training pruning from the perspective of training plasticity which is the ability of the pruned networks to recover the original performance. A technique named neuroregeneration is further shown to improve the training plasticity. Based on these findings, a gradual magnitude pruning (GMP) method with neuroregeneration (GraNet), and its dynamic sparse training variant are proposed. Experiments show the effectiveness of GradNet and GradNet-ST.
main_review: Pros

The paper is well-written. The contributions are clearly presented.

Model compression is an important topic in scaling deep models on mobile devices. It is essential to design effective channel pruning method.

The key idea of neuron regeneration is well-motivated.

Extensive experiments are conducted to show the superiority of GradNet..

Cons

It seems that GradNet achieves marginal improvements over STR on the ImageNet dataset.

Does neuron regeneration work on a structured pruning framework?

The training plasticity is sensitive to hyper-parameters. It would be better to provide experiments to show how neuron regeneration helps training plasticity. 

limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a concept of pruning plasticity to study the effect of pruning during training. Using the insights obtained from the pruning plasticity, the authors improve the state-of-the-art sparse training performance. Since a large body of recent works has emerged to accelerate the training process, this work is helpful for this area.

The findings are interesting and make good connections with separately reported phenomena in the literature. Based on the insights from pruning plasticity, the authors further propose two gradual pruning methods which improve the performance over the existing methods. 

main_review: Originality: 
(1)	The paper does a thorough study of pruning during training, which is indeed recently ignored by researchers compared with after-training pruning and before-training pruning. The recently emerged before-training pruning (pruning at initialization) absolutely receive a lot of attention. This work, however, doesn’t follow the crowd and highlights the values of an existing technique - pruning during training. Due to the failure of the existing before-training pruning methods to capture the important masks pointed by [1], discovering either early-existing matching subnetworks or more accurate subnetworks during training have a great value for the community. 
(2)	The observations associated with pruning plasticity make good connections with several existing phenomena in the pruning literature, providing valuable insights to the people who want to use these techniques in reality. 
(3)	With the insights provided by pruning plasticity and the neuroregeneration from the nervous system, authors modify the existing gradual magnitude pruning methods and demonstrate the state of the art sparse training performance as well as the gradual pruning performance. 

Quality:
(1)	The paper is clearly written and easy to follow. The authors provide an adequate related work review. 
(2)	I like the random initialization analysis of the discovered masks between GMP and GraNet, making the overall paper more robust. 
(3)	The claim is well supported by the extensive experiments. 

Significance:
(1)	While the idea of during-training pruning is not new, it indeed achieves the best performance-efficiency tradeoff compared with after-training pruning and before-training pruning. Works like this paper that provides better understanding and performance improvement of pruning during training are important for the ML community, providing good motivations for future hardware design. 
(2)	The proposed methods are sound and performant. Especially, the 0.9% and 1.5% accuracy improvement achieved by GraNet-ST over RigL with ResNet50 on ImageNet at sparsity of 80% and 90%, respectively, is quite impressive. It is the first study which, up to my knowledge, shows consistently that dynamic sparse training can outperform dense-to-sparse training with even fewer training FLOPs.

Improvements:
(1)	This paper only considers unstructured pruning. I suppose that the findings of pruning plasticity can also generalize to the structured setting. It would be nice to test pruning plasticity on the structured pruning. 
(2)	The authors should cite [2,3,4] for the zero-cost neuroregeneration.
(3)	It is unclear to me if the pruned weights are still updated or not during the backward pass. This would cause a significant computational difference. It would be nice to make this point clearer.
(4)	I suggest the authors to put more emphasis on the dynamic sparse training part (GraNet-ST) rather than GraNet, which is more promising in terms of the accuracy-efficiency trade-off. The description of GraNet-ST in the current version is somehow unclear, especially for readers who are not familiar with sparse training.

Reference:
[1] Frankle, Jonathan, et al. "Pruning neural networks at initialization: Why are we missing the mark?." arXiv preprint arXiv:2009.08576 (2020).
[2] Mocanu, Decebal Constantin, et al. "Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science." Nature communications 9.1 (2018): 1-12.
[3] Evci, Utku, et al. "Rigging the lottery: Making all tickets winners." International Conference on Machine Learning. PMLR, 2020.
[4] Dettmers, Tim, and Luke Zettlemoyer. "Sparse networks from scratch: Faster training without losing performance." arXiv preprint arXiv:1907.04840 (2019).

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work aims to combine ideas presented in recent dynamic sparse training (DST) methods with gradual magnitude pruning. Additionally the paper has some novel experiments that confirm previous work on when-is-best to prune during training. Although an interesting and potentially impactful idea, the story and results need a bit more work.
main_review: # Pros
1) Results in Fig:2 are quite interesting. Though some clarification is needed (see below)
2) The idea of combining pruning and re-generation is novel and authors do a good job on comparing with a large set of methods.

# Cons
1) "For this reason, we determine to focus on gradient-based regeneration, i.e., regenerating the same number of connections as pruned with the largest gradient magnitude." This is sounds like what [rigl] does for training sparse networks. Is the method proposed for regeneration is practically same as rigl? If so it would be nice to called it so. If not, it would be nice to see a discussion on similarities and differences. My understanding is GraNet=GMP+RigL. It would be nice to make this clear.
2) It needs to be clear which experiments uses global pruning (and thus non-uniform sparsity). For example on Table-3, GraNet-ST (ERK) requires 0.37× training flops which is less than rigl (erk). Shouldn't be the training flops of GraNet-ST always larger than rigl since it starts with more parameters? I am assuming global pruning is not used here, as if so; then the resulting distribution is not ERK anymore and this should be made clear. 
3) It should be make clear that GraNet-ST is different than other sparse training methods like set and rigl in the sense that it requires more sparsity/memory to begin with.  
3) I couldn't see any ablations on the initial sparsity of GraNet-ST method. I think the value 0.5 is picked, which is a significant increase in parameter count if the final sparsity is >90%; it would be nice to see effect of initial sparsity. 
4) It might be better to use term 'sparsity' instead of 'density' on Figure 2. More importantly I am confused what "pruning rate" is referring to. Reading the text, it seems to refer sparsity of the network after pruning. Then it is not clear how do you prune a 98% sparse (2% dense) network to 50% sparsity. Do you only grow? This needs to be cleared I think. Experiments in Figure:2 looks quite interesting otherwise.

## Minor
1) "To better understand the effect of pruning during the optimization proces" [zhu2017] is a relevant work that studies the same.
2) "To the best of our knowledge, this is the first time in the literature that sparse-to-sparse training reaches a test accuracy of 76% with ResNet-50 on ImageNet at sparsity 0.8." [rigl] paper I think has 5x training runs with such results. If extended training runs are not considered; this should be explained. I think a better way to compare dense-to-sparse methods with DST methods is to use training flops on the x-axis.
3) "Perhaps most impressively, the latter for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods by a large margin with ResNet-50 on ImageNet." I guess large-margin is subjective but [rigl] shows the same. 

[rigl] https://arxiv.org/abs/1911.11134
[zhu2017] https://arxiv.org/pdf/1710.01878.pdf

## After Rebuttal
I thank the authors for their strong rebuttal. I think all my concerns/questions are addressed and I believe this work is a timely work that inspire and impact future research. Therefore I support acceptance.

One final suggestion: 
- I think it might help the readers to name both versions of the method as GraNet as when initial sparsity=0 GraNet-ST is GraNet. Then authors can use sth like GraNet (s=0) to identify pruning version.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper studies how to optimize the performance of sparse models which are pruned during the training process by combining during-training pruning with the growth criterion originally proposed in RigL. All reviewers felt the paper was clear and covered a timely and important topic, though there were some concerns regarding clarity and comparison to prior work which were resolved through the author discussion. As such, I recommend the paper be accepted. 
Decision: Accept (Poster)