Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors show that gradients from backpropagation are lognormally distributed empirically across several popular architectures like (Bert, ResNet18, MobileNetV2, VGG16, DenseNet121) over widely used datasets (CoLa, MRPC, ImageNet and CIFAR100). Using this result, they propose schemes for gradient quantization and gradient pruning that are theoretically principled and outperform existing methods in the literature. 

I was very impressed by the principled nature of the authors' approach. Proving the lognormal distribution using a KS test and using it elegantly to find analytical formulations for optimal bit division in quantization and threshold parameter for sparsity in gradient pruning was masterful. 

* In the appendix, the KS test shows that logLaplace distribution is also a good fit for the gradient distribution. Can the authors provide any intuition as to why a log normal might be better? 

* Clearly for pruning, logNormal distribution seems to work better than logLaplace. It’d be interesting to see the results of using LogLaplace distribution for quantization and verify that it yields worse results there too.

* Can the authors elaborate on time/memory complexity, and how well their methods made improvements for training time?


review_title: new insights on gradient quantization

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This work proposed a very interesting idea that the back-propagated errors have log-normal distributions. The authors could extend this intriguing observation into computation efficient algorithms; reduced-precision floating-point quantization or the pruning of the back-prop error, which are very interesting. The authors also provide details of derivations as well as the data analysis along with several small ideas (in Appendix) to strongly support their ideas.

The biggest shortcoming would be the relatively shallow evaluation of the proposed method. In particular, since the main ideas of this work relies on a very strong assumption that the back-propagated errors are log-normal distributions, (and I guess it would not be possible to analytically prove this assumption) it would be necessary to provide thorough investigations on various deep learning applications to validate this assumption. Table 1 seems to be a good start, but it is a little disappointing to see that a very limited set of image classification models are tested with the proposed method in Table 3 and Fig 5. 

In fact, one concern about the proposed method is that the back-prop error might evolve drastically throughout the training periods, and there is no guarantee for it to maintain its shape. Therefore, it would be critical for the authors to convince that the proposed method work in a wide range of applications (at least the kinds shown in Table 1, since this table claims that the evaluated models here show log-normal shapes).

In summary, this work seems to be interesting, but it should be supported by a much more thorough evaluation.


review_title: Interesting observation led to nice computation savings, but need more evaluation

final_score: rating: 7: Good paper, accept
review_content: The authors suggest an interesting finding where the gradient distribution in each layer is close to log-normal distribution instead of normal distribution. Given the suggestion, the authors propose two closed-form analytical methods to produce a better low-precision floating-point format and an optimized sparsity threshold for gradient pruning. While such distribution assumption with associated analytical methods seems promising, there are a few suggestions and questions I want to address for the paper:

1. There are many previous studies on quantizing gradients for efficient distributed training, such as QSGD [1] and signSGD [2]. Although most of those studies do not reduce the computational complexity for backpropagation, they still serve as well-performed gradient quantitation methods. I think it's better to distinguish from those previous studies explicitly in the paper.

2. It's better to involve more motivations and explanations about why gradient quantization is important for reducing the computational and memory burden of neural gradients.

3. In the context of the real hardware design, it would be hard to get a computational reduction by using the proposed method which determines the optimal floating-point representation. Real hardware design (e.g. GPU architecture) requires a global floating representation across all datasets so it is impossible to design hardware with a specific floating representation for a specific dataset. Therefore, I don't think determining the best floating-point format for each dataset is useful in practice.

4. Also I would like to address some technical questions regarding the setting and experiment:

a. How does weight update work with the quantized gradient? Do you perform the quantized gradient over the full-precision weight?

b. What kind of optimizer you used for the experiments?

c. How do you handle the precision of the error term (defined in WAGE paper [3])? 

Overall, I think the paper is interesting and it would be worth being accepted after addressing the suggestions above.


[1] Alistarh, Dan, et al. "QSGD: Communication-efficient SGD via gradient quantization and encoding." Advances in Neural Information Processing Systems. 2017.

[2] Bernstein, Jeremy, et al. "signSGD with majority vote is communication efficient and fault tolerant." arXiv preprint arXiv:1810.05291 (2018).

[3] Wu, Shuang, et al. "Training and inference with integers in deep neural networks." arXiv preprint arXiv:1802.04680 (2018).

*************
I want to thank the authors for their detailed comments and explanations. I agree with (3) that a global floating representation is not required, but different settings of bitwidth may require additional data paths designed in the hardware (therefore a unified bitwidth for exponent and mantissa is recommended). Given revised related work and additional practical implications, I would like to champion this paper. 
review_title: interesting paper for suggesting a better distribution of gradients, but more details and practical implications should be covered

final_score: rating: 7: Good paper, accept
review_content: Following the author response, I have updated my score to an accept. The key contribution is likely to be of interest to the ICLR community and the paper is well executed. 

---

The paper proposes a principled approach to the compression of intermediate gradients by fitting a log-normal distribution to the individual derivatives, which they show is a good fit for the studied models. Although simple a posteriori, the proposed approach is elegant and the arguments are presently clearly and well illustrated.

The weak points of the submission are a lack of clarity for some of the assertions and a lack of motivating example or application. The current writing takes for granted that quantizing intermediate gradients is good. The claimed benefits of quantization is to "alleviate data-throughput requirements and accelerate the training process"
and "reduce both bandwidth and memory footprint, as well as computation time". Those benefits are more obvious in a distributed setting, where the main bottleneck to training is the communication of gradients. This reasoning does not directly apply to intermediate gradients. Quantization and low-prevision might very well provide those benefits in the non-distributed setting, but the current exposition does not make it concrete, which might limit its impact.

My initial recommendation is towards acceptance if those points and some smaller points below can be adressed during the discussion period. The submission is well executed and its contribution is likely to be of further theoretical and practical interest.

Smaller points;
- The introduction mentions that 2/3 of the underlying computation involve intermediate gradients. Please make this number concrete through an example, as this seems highly dependent on the architecture and data type.
- One of the main points of the introduction is that the intermediate gradients are fundamentally different from weight gradients. This assertion should be supported by data in the submission, maybe as an addition to Figure 2.
- The term `neural gradient` being undefined in the abstract and title is an issue. While its meaning can be inferred, it is not standard terminology and needs to be defined in the abstract or replaced.
- The term `dynamic range` would benefit from a technical definition.




Minor points, not part of the main decision
- Something went wrong in Equation (7).  $\tau$ seems to be used instead of what was originally planed as $\epsilon'$?
- Some `\refs` seem broken in Section 4.3 as they point _to_ section 4.3
- The acronym `BF16` does not seem defined in text.
- The value of the Kolmogorov–Smirnov test and Table 1 is limited by a lack of interpretability, since it is not aimed at a p-value. Adding more candidate distributions to communicate the range of possible values would help. 
- A short definition of the tradeoff between mantissa and exponent might help a general audience.
- Some of the references point to preprint versions when the work has been published, for example [Cambier et al. 2020], or does not give appropriate bibliographical information, for example [Choi et al. 2018a]. Please take the time 
review_title: Review

Meta-review: This work makes the observation that gradients in neural network training are approximately distributed according to a log-normal distribution. This observation is then used to compress and sparsify the gradients, which can be useful in distributed optimization of neural nets. The reviewers indicate that this contribution is novel and useful and they do not find any major issues with the presented work. I recommend accepting the paper for a poster presentation.
Decision: Accept (Poster)