Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a sparse training method based on fine-grained block sparsity. The main insight is that shuffled block sparsity (where entries in the weight matrices are permuted to form blocks) can be hardware efficient and yield speed up. The method proposed consists of block-pruning and block- growing at regular intervals (i.e., dynamic block sparsity). Both block-pruning and block-growing can be done efficiently without computing gradient for the whole dense matrix. Experiments on ResNets & VGG on CIFAR and ImageNet show that at 0.50 sparsity, the proposed method can match the accuracy of the dense models.
strengths_and_weaknesses: Strengths:
1. The block-wise grow criterion is clever and well-motivated. It avoids having to compute the gradient for the entire dense matrix. Instead, it estimates the gradient magnitude by multiply the magnitude of the output gradient and the input. Maybe micro-experiment to show how close this heuristic is compared to the expensive gradient of the dense matrix would make it even more convincing.
2. The accuracy vs block sizes ablation is informative. It reveals that block size of 16x16 can yield speedup while maintaining accuracy of the dense model.

Weaknesses:
1. Lack of validation on wallclock-time speedup. Does the benchmark use channel first or channel last? fp32 or fp16? These details significantly change the speed of the dense model.
What is the overall model speedup, either in inference or training?
The code seems to just use mask to multiply with a dense weight matrix so I'm not sure how one could get wall-clock time speedup. 
2. Lack of comparison with existing block-sparse methods (e.g., Block Pruning For Faster Transformers. Lagunas et al. EMNLP 2021). Does the shuffled block sparsity perform better / faster than just block sparsity?
3. The concept of shuffled block should to be explained more clearly. Is the entire 16x16 block either zero or nonzero? Or are there both zeros and nonzeros in the same block?
How important is the shuffle step? How does one decide which rows / columns to shuffle?
questions: 1. What is the overhead of block dropping? The paper claims it improves accuracy but there's no experiments to show this.

limitations: Limitations are discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies dynamic sparse training (DST) with fine-grained structured pruning. 
Starting with a randomly initialized nonstructured model, the shuffled blocking procedure groups active weights into several blocks. During the training interval, it drops some weights with the smallest magnitudes, and grows some weights with the proposed block-wise grow criterion. With experiments on CIFAR and ImageNet, this paper demonstrates that the proposed framework is better than structured sparse lottery ticket (SSLT) [3] and RigL [6]. 

strengths_and_weaknesses: Strengths

- This paper firstly studies the DST with fine-grained structured models. Compared with previous works, the proposed method no longer requires specialized hardware (sparse tensor cores on GPU) [16] or pre-trained dense models [3]. 
- The proposed growing algorithm is novel and practical as it avoids the calculation of dense gradients and dense weights, which makes the training purely sparse. 
- This paper clearly discusses the difference between the proposed method and the Transposable N: M Sparsity [16]. 
- The paper is in general well written and easy to follow.

Weaknesses

- More details about the hardware acceleration should be provided. 
Since this paper focuses on sparse training with fine-grained structured pruning, the total training time saving and memory saving should be reported. Meanwhile, this paper should compare the speedup with some structured pruning methods in Figure 9. 

- In Table 2, Figure 6, and Figure 7, the error bar should be presented. 
 

questions: This paper is an overall OK submission. Nevertheless, the lack of acceleration comparison refrains me to give a better score. I would be happy to improve my rating as long as the authors address the above weaknesses in the rebuttal.   

limitations: No, this paper does not discuss the limitation and potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a hardware-friendly sparse training method with shuffled block structures, which can be accelerated by the common hardware (GPU). The acceleration can be achieved by quickly obtaining the index of the grouped weights during the forward and backward pass. The research topic is very important for the sparse training community. In spite of the promising results of dynamic sparse training (DST), a major problem of DST is that the promising efficiency can not be fully transferred to the common hardware. Therefore, any good contribution to the topic is important. However, several major concerns need to be clarified as written below.
strengths_and_weaknesses: ## Strengths

(1) This paper proposes a structured sparse training method by grouping the unstructured elements together during training. It is surprising to see that the grouped DST can still match the performance of the dense model while achieving up to 5x acceleration. 

(2) The criterion used in the paper does not require to calculate the dense gradients. 

(3) Related work is relatively extensive.

##  Weakness

(1) The authors claim that they are the first to apply group sparsity to dynamic sparse training. However, I think they have missed one prior work (https://arxiv.org/pdf/2108.06277.pdf) which has explored this direction, even though the method is distinct. I suggest adjusting their claim. 

(2) The description of the methodology is relatively dry and hard to follow. More effort should be put to improve it.

(3) My major concern is that the authors did not mention which part of the results are produced with masks and which part of the results are produced with the proposed method? It has been claimed in the abstract that the algorithm can match the accuracy while reducing training time by 5x, but no actual wall-clock training time has been reported in the paper. More specifically, (1) does the proposed method support end-to-end dynamic sparse training? As far as I understand, the dense matrix implementation in Pytorch or Tensorflow is so well-optimized, that even we can achieve acceleration by quickly selecting the index of grouped weights, it is difficult to be quicker than the mask-version DST in terms of the real wall-clock time. (2) Was the averaged speed up reported in figure 9 be evaluated with one feed-forward pass or evaluated with the end-to-end training? This question is important to verify the real contribution of the proposed method.

(4) I am curious about the real speedup of the proposed grow criterion? It would be nice to provide how many flops can be saved by using this instead of calculating the dense gradient? In reality, how fast it is compared with dense gradient on GPU?
questions: Please refer to the above weakness. 
limitations: No
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores sparse training for fine-grained block sparse models. Detailedly, the authors leverage the classic prune-and-regrow workflow in a block-aware manner with a new grow criterion that does not rely on dense gradient calculation of weights. Experiments on ImageNet and CIFAR-10/100 show that the proposed block-wise sparse training can achieve satisfying performance and noticeable speedup at moderate sparse rates. 
strengths_and_weaknesses: Strengths:
1. The reviewer appreciates the motivation and contribution of this work. The exploration of hardware-friendly sparse training is remarkable and could benefit the community a lot.
2. The proposed block-wise growing criterion is well-designed and can effectively reduce the overhead of sparse training, despite that the ablation studies are missing. I recommend the authors compare traditional dense gradient criterion with the proposed block-wise grow criterion to further demonstrate their view.
3. This paper is well-written and easy to follow.

Weakness:
1. The authors claim that the proposed block-wise sparsity can match the performance of unstructured sparsity in most cases. However, DSB seems to lag far behind the current state-of-the-art unstructured methods. For example, STR[1] achieves 76.12% top-1 accuracy at 81% sparsity, while DSB only achieves 74% top-1 accuracy at 75% sparsity.
2. The core advantage of the proposed DSB falls in acceleration and computation reduction on general hardware. However, the authors only compare DSB with unstructured methods. In contrast, structured sparsity methods[2,3,4] can also eliminate the training and inference costs. Thus, it is necessary to add comparisons with structured methods to show whether the DSB can obtain a better trade-off between speedup and performance maintenance.

Minor: 
1. What is SDB In Tab.2?  I recommend the authors add a relevant caption.

[1] Soft Threshold Weight Reparameterization for Learnable Sparsity. In ICML,2020.
[2] Hrank: Filter pruning using high-rank feature map. In CVPR, 2020.
[3] Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In ICCV, 2021
[4] Channel Pruning via Automatic Structure Search. In IJCAI, 2020.
questions: 1. The authors claim that DSB accelerates the computation on traditional computing units that are broadly supported by hardware. However, I notice that their provided codebase is a stimulated version, i.e., weights are multiplied with binary masks to represent pruned model. It would be nice for the authors to provide a running demo or code that can be deployed on general hardware, which could also make this paper more convincing and easy-to-implement.
limitations: The authors do not discuss the limitations of their work as well as the negative societal impact.
review_title: null

Meta-review: metareview: This paper studies dynamic sparse training (DST) with fine-grained structured pruning. A topic of practical relevance, the authors proposed hardware-friendly shuffled blocking and simultaneously prune-and-grow. While this paper seems to be a mild extension from Chen et.al. 2022, the contributions are clear. One main concern raised is whether evaluating the end-to-end speedup with layer-wise CUDA execution time is a correct practice. I agree with the authors, that is common in the ML literature and more real evaluation requires system-level engineering (often beyond the scope of an algorithm paper). 

Three out of four reviewers acknowledged they were convinced by the authors' rebuttal. One reviewer mainly questioned one missing reference (while acknowledging “distinct”) + several algorithm step clarifications, to which the authors delivered point-by-point responses. Therefore, given the overall sentiment among reviewers, I’m recommending accept.
confidence: Certain
Decision: Accept