Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

The paper introduces a new sparse training algorithm (SR-STE) based on the straight through estimator which is specially designed for the hardware constraints of Nvidia A100 GPU. Auxiliary, in order to study better this algorithm, the paper introduces also a metric (SAD) to measure the changes in the sparse network topology during training. The contributions have real added value as they show that sparse neural networks can actually benefit of hardware designed to consider sparsity. The experiments on CNNs and Transformers support the claims.

Strong Points

•	SR-STE can take real advantage of A100 sparse capabilities.

•	The algorithm is designed for the general form of n:m sparsity and, likely, can be used also for the next generation of GPUs with sparse networks support.

•	It is the first study which, up to my knowledge, shows consistently that sparse networks can outperform dense networks also for convolutional layers.

•	The paper is written in a clear manner and well structured.

Weak points

•	The proposed SAD metric seems to work just on the same network during its own training process, as it does not consider the fact that at different training runs the hidden neurons may develop differently.

•	The related work is not complete

•	The paper has a number of sloppy written passages and inconsistencies in the mathematical formalism.

During the rebuttal, I would suggest to the authors to consider the following comments:

1) Improve the mathematical formalism and perform a careful proof-read to make sure that all notations are consistent. For instance page 5,  2nd paragraph, E_f is actually \epsilon_f?;  page 5, third paragraph, the last equation has two equal terms, and so on.

2) I believe that it would be informative to discuss the SAD metric limitations and advantages in comparison with other state-of-the-art metrics suitable to measure the distance between two sparse topologies, e.g. NNSTD https://arxiv.org/abs/2006.14085 .

3)  Related work. the 4th paragraph on page 2 about the one-stage scheme to obtain efficient sparse networks is missing the first two references from 2017 which introduced sparse training with dynamic topology: https://arxiv.org/abs/1707.04780, https://arxiv.org/abs/1711.05136 . Auxiliary, I believe that also this work has to be cited https://arxiv.org/abs/1907.04840 . 

4) The last phrase of the above mentioned paragraph and the next one are, up to my knowledge, accurate just for CNNs. It has been shown already that for other types of neural networks (e.g. MLPs), sparse networks can easily and constantly outperform dense networks (e.g. https://arxiv.org/abs/1905.09397). Please make this discussion more accurate and informative.

5)  Please read the paper thoroughly to address all typos and small inconsistencies. E.g., In Tables 1 and 2 “ours” appear just in some cases of SR-STE; In Table 3, STR appears twice with exactly the same sparsity level, but different number of parameters (more clarifications would make easier the reader job).
review_title: SR-STE and happy SAD

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors proposed a new method for training N:M fine-grained structured sparse networks from scratch. The authors found that the SAD metric, which measures the number of weights whose pruning state is changed, became higher if the existing STE is used to train sparse networks and this metric had the positive relationship with accuracy drop. To reduce the SAD, SR-STE which gives higher weight decay coefficient to the pruned weights is applied to train sparse networks, improving the accuracy of sparse networks trained from scratch.

SR-STE which is penalizing the pruned weights is somewhat simple, but the results in the paper are strong. The proposed method supports to increase the accuracy of N:M fine-grained structured sparse networks, and it can be applied in many diverse tasks regardless of pruning method like general unstructured pruning. I think it is a good technique to easily apply when pruning states of each weight are continuously changed.

However, weight-sorting process which occurs in every training step can decrease the training speed compared to other threshold-based pruning method. Could you compare the actual consuming time to train sparse network for each training method? 

Comment: Below are some typos I think.

1. 3.2. Analysis: I think |^{S}{SAD}^{l}_{0:t}-^{D}{SAD}^{l}_{0:t}| instead of |^{D}{SAD}^{l}_{0:t}-^{D}{SAD}^{l}_{0:t}| is likely to be the right expression in context.
2. 3.2. Analysis: The sentence "This formula measures the number of neurons that are pruned~" is unfamiliar to me, because "neuron" is usually used when depicting activation, not weight of artificial neural network. I think "weight" is more matched expression.
3. The stated GFLOPs of ResNet-50 (2:8 sparse pattern, SR-STE method) are different in Table 1 (1.02) and Table 3 (0.1). I think the former is right. 
review_title: Simple technique, but nice results 

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose to train N:M structured sparse neural networks from scratch.  To improve the effectiveness of the training process, a sparse-refined straight through estimator (SR-STE) is proposed based on vanilla STE. Finally, the authors define Sparse Architecture Divergence (SAD) to indicate the topology change of the sparse network. The novelty of the proposed SR-STE is not enough, and the defined SAD lacks relevant theoretical support. Despite the good performance, I am concerned about the technical novelty of this paper. 
(1) The main contribution of the proposed SR-STE lies in a sparse-refined regularization term. It seems like a mitigation measure, which cannot completely eliminate the negative impact caused by the approximate measures taken in the STE. 
(2) And in my opinion, the analysis of SAD lacks relevant theoretical support and is not very convincing. I expect more explanations for SAD.

There are still several issues that need to be addressed. Firstly, the layout of this paper is not very suitable, and it is not very convenient in the process of finding the corresponding figures. Secondly, there are several narrative errors in this papere, which need to be checked carefully, e.g., in the third-to-last line of Section 1, I think it should be 'we propose a sparse refined term to enhance the effectiveness on training the sparse neural networks from scratch'; in the second-to-last line of Section 3.2, it should be '\left | ^{S}\textrm{SAD}_{0:t}^{l}- ^{D}\textrm{SAD}_{0:t}^{l}\right |'.




review_title: In this paper, the authors propose to train N:M structured sparse neural networks from scratch.  To improve the effectiveness of the training process, a sparse-refined straight through estimator (SR-STE) is proposed based on vanilla STE. Finally, the authors define Sparse Architecture Divergence (SAD) to indicate the topology change of the sparse network. The novelty of the proposed SR-STE is not enough, and the defined SAD lacks relevant theoretical support. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: To maintain advantages of both unstructured and structured sparsity, the paper improves previous STE method based on the proposed SAD. The experiments on five types of applications show its outstanding performance.

pros.
1. The motivation of this paper, maintaining advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity, is clear and persuasive.
2. To validate outstanding performance of the proposed method, the authors did experiments on five different types of applications: image classification, object detection, instance segmentation, optical flow, and machine translation.
3. The authors provide detailed explanations on physical meaning of their proposed SAD.

cons.
1. In Section Introduction, I think the authors should formally explain meaning of "N:M" when "N:M" is mentioned for the first time.
2. In your github code train_imagenet.py, implementation of "devkit" package on 19th/20th/22nd line can not be found. I think your proposed method is mainly implemented in devkit.sparse_optimizer, while you do not include the most important code.
3. In Figure 1, physical meaning of variables R and C is not introduced.
4. The original STE is used for unstructured sparsity, while you improved STE for N:M structured sparsity. Why does your improvement encourage structured sparsity? 
5. According to Table 2, the classification performance gap between STE and SR-STE is minor. So I think SR-STE is not a significant improvement compared to the original STE.  I am not 100% sure about my opinion on performance, since comparison between STE and SR-STE is not provided for the experiments on other applications (detection, segmentation, optical flow, and machine translation).
review_title: Clear motivation and detailed explanations on SAD, but I have concerns on structured sparsity and performance improvement

Meta-review: Reviewers like the simplicity of the approach and the fact that it works well.  
Decision: Accept (Poster)