Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a method for a stochastic unbiased masking of the gradients such that half of them are 0 and they can be used to accelerate the matrix multiplications in an accelerator such as a GPU. In particular, the method induces a form of sparsity called N:M sparsity where N out of M consecutive (in memory) elements are 0. The authors propose to use a masking such that the gradients are still unbiased but also such that they have the minimum variance. Thorough experiments show that using their approximate 2:4 sparsity algorithm as well as the exact 1:2 algorithm, allows training a variety of neural network architectures on both images and text with minimal loss in final performance, if any. Moreover, the authors show that their algorithm can be combined with similar algorithms for the forward and backward pass as well as quantization without significant drop in performance.
strength_and_weaknesses: Strengths
------------

- The theoretical analysis is well written and intuitive
- Accelerating GEMMs is a very important topic of research as it affects all modern neural network architectures
- Thorough experimental evaluations show that the method indeed does not hurt performance and can be used to train a variety of models

Weaknesses
--------------

- The main weakness of the paper is the lack of real world performance improvements due to lack of custom kernels or support from currently available accelerators. The speedup that is mentioned is a best case scenario and makes the method seem more useful than it is probably.
- Another weakness is the lack of comparisons with MSE based sparsity for the gradients. Even though previous work has shown that unbiased estimators are more important for the gradients than the weights, it would strengthen the paper significantly if instead of the greedy estimator the paper also compared with MSE optimal biased estimators.

Typos
------- 

- eq. 8 typo it should be $sign(a_2) \left(\|a_1\| + \|a_2\|\right)$
- Fig. 2 typo $a_4 >> \max(a_1, a_2, a_3)$
clarity,_quality,_novelty_and_reproducibility: The paper is well written with clear analyses and provides code for the proposed method.
summary_of_the_review: The method is very intuitive and straightforward. It may be slightly incremental work, however I believe it is a clear contribution.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper develops a new scheme for pruning the gradients passed through a network during back-propagation. In particular, the proposed method designs a stochastic pruning operator which is unbiased and provides the minimum variance output. Both 1:2 (2 inputs, 1 non-zero output) and 2:4 (4 inputs, 2 non-zero outputs) variants of this operator were developed. In testing, networks pruned with the proposed operator demonstrated little-to-no degradation in performance across a range of network architectures and classification problems.

strength_and_weaknesses: # Strengths

The paper provides a novel method for pruning the back-propagation gradients.

The proposed method does not hurt network performance, whereas naive greedy baselines hurt performance significantly.

The paper is well-written and reasonably easy to follow.

The paper provides a significant discussion of the implications the paper's results have on hardware acceleration and design

# Weaknesses
The paper doesn't make a strong case for why the purning operator needs to be unbiased and only points vaguely to Chmiel et al.'s finding that "for the neural gradients... it is critical to use unbiased quantization (i.e., Bias[θ(a)] = 0)."

For completely understandable reasons (no hardware support for pruning during training), the paper is unable to demonstrate any actual training time reductions. Hopefully this changes as new hardware is developed.

# Suggestion
The introduction of the MSE of the pruning operation describes $a$ as scalar, rather than a vector. The resulting discussion is then confusing as there's no stochasticity in $a$ and one is left asking why not just let $\theta(a)=a$. I suggest introducing $a$ as a vector from the start, as this will make the resulting discussion easier to follow -- there are many choices for the pruning operator which would produce different MSEs. 

clarity,_quality,_novelty_and_reproducibility: Besides the above suggestion, the paper is generally easy to follow.

The results demonstrate the proposed 2x pruning procedure doesn't hurt performance significantly.

The proposed pruning operator is, to my knowledge, novel and seems simple to reproduce from the provided equations.

summary_of_the_review: The paper provides a simple gradient pruning procedure that could accelerate future hardware systems. While the form of the pruning operator could use further motivation, I'm overall in favor of this paper's publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper aims to accelerate training by pruning the calculated gradients. More specifically, the authors propose 1:2 and 2:4 minimum-variance unbiased estimators and show their effectiveness on a number of tasks, where accuracy is not severely affected by the proposed pruning scheme.
strength_and_weaknesses: The paper is well written and proposes very attractive improvements to training speed without affecting the accuracy.
Given a relatively general title, I'd be curious to see some results on using the approximate method in setups with different sparsity setups, e.g, 4:8, and also higher sparsity, e.g., 1:4.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and the results are presented in a clear way.
Proposed pruning scheme seems straightforward to implement, but having access to a reference implementation may still be valuable.
summary_of_the_review: A well written paper with interesting and relevant results for practical applications of DL.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper investigates application of N:M sparsity for the neural gradients (i.e. loss gradients with respect to the intermediate neural layer outputs), by using a masking mechanism such that the gradients are still unbiased and have minimum variance. Experimental results confirm that using the proposed approximate 2:4 and the exact 1:2 algorithms, allows training a variety of neural network architectures on both images and text with minimal loss in final performance. The proposed idea is novel and the experimental results are convincing. The main downside is that the paper is unable to demonstrate any actual training time reductions, as existing hardwares do not support N:M sparsity for gradients. Still, the proposed method is valuable and as such hardwares are expected to be developed in near future. Hence, I recommend acceptance.
justification_for_why_not_higher_score: The paper is unable to demonstrate any actual training time reductions.
justification_for_why_not_lower_score: The proposed method is novel and is supported by extensive experiments, and is of importance for the ICLR community.
Decision: Accept: notable-top-25%