Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: I think the use of QPyTorch for the experiments here invalidates the results since the intermediate matrix multiplies are done in single precision (FP32), and so are more optimistic than a pure 16-bit implementation. (This is both according to the authors Sec 4, experiment setup; and according to the QPyTorch paper arxiv:1910.04540, Sec 3 intro.) For these kinds of experiments to be meaningful, they have to be done on native 16-bit hardware which luckily is becoming more common, e.g., Google's TPUs or the newer NVIDIA GPUs.

There are two other problems. First, it is not clear how stochastic rounding would be implemented in hardware. Doing it for every MAC operation could likely be even more expensive than just doing 32-bit MAC operations, since it involves the generation of random numbers, division, etc. Second, Kahan summation takes up twice the weight storage, so a more detailed calculation is needed to compare any hardware/energy savings to use that instead of just 32-bit.

As an aside, it may be interesting in Figure 1 to zoom in on the initial part of training to understand where the difference between 32-bit and standard 16-bit comes from in early training since at that point, the gradients are generally larger than later on in training.

review_title: Serious problems with the approach

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper explores the possibilities of reducing the precision of the weight update operation (i.e. AXPY ops) from 32 bit to 16 bit in today’s BFloat16 training framework. To enable 16 bit update, the authors proposed two techniques, i.e. stochastic rounding and Kahan summation. The authors use a simple least-squares regression model to theoretically explain the model accuracy degradation due to nearest rounding, then experimentally demonstrate the effectiveness of two techniques on a range of deep learning models and dataset. 

Strong points:

This paper is very well written. The problem is well addressed, and the solutions are well explained. The authors also provided both theoretical analysis and comprehensive experimental results. 


Week points:

1). Novelty: Both the problem addressed, and the solutions proposed in this paper have been reported in recent publications. In particular, both (https://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers) and (https://papers.nips.cc/paper/8736-hybrid-8-bit-floating-point-hfp8-training-and-inference-for-deep-neural-networks) investigated the reduced precision of weight update from 32 bit to 16 bit or less. Moreover, the former introduced stochastic rounding and the later introduced round-off residue which basically the same as the Kahan accumulation technique discussed in this paper. Although, both papers discussed this topic in the FP8 training frameworks, while this paper in the BFloat16 framework, the basic concepts are the same. The authors of this paper summarized the techniques nicely, however, the novelty limited.

2). On the same note, theoretical analysis on the impact of rounding mode on quantized weight update were also discussed in recent publications, such as (https://arxiv.org/abs/1706.02379 and https://openreview.net/forum?id=ryM_IoAqYX). It would have been nice to include these discussions as background knowledge and to distinguish this work from others. 

To clarify:

1). The proposed Kahan Summation method created a second tensor to store/accumulate the quantization error. Both weight and rounding error tensors are in 16 bit which in total, effectively, is 32 bit. Since AXPY is a very fast operation, this method does not seem to save much in terms of memory or speed. 

2). Today, SGD is often used with momentum, could the authors comment on the precision of momentum accumulation. And how about other popular optimizers, such as Adam?

3). The analysis, i.e. Theroem 1 and 2, is based on a simple least-square regression model. Can this theory generalize to deep learning models?

4).  On Table 2, last two rows of last column, the value seems to be inconsistent with other data in the same row. 

review_title: A good submission but lack novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This work reinvigorates half precision training as an alternative to either full single precision or mixed half and single precision. The authors demonstrate that the nearest rounding is the culprit for the worse performance of half precision training compared to single precision, due to cancelling small updates. They then propose two known techniques that can mitigate this effect, stochastic rounding and Kahan summation. Empirically, they demonstrate on an extensive suite of tasks that the proposed adjustments lead to half precision performance that is almost on par to single precision. 

### Pros
- Solid experimental evaluation and results on a large variety of tasks
- Both modifications that need to done are simple and straightforward

### Cons
- Limited novelty

### Recommendation 
Overall, even though the paper does not have a lot of novel content, the experimental evaluation is thorough and demonstrates that the improvements close the gap to single precision training. Therefore I am keen on accepting this work, although admittedly not by much. 

### Detailed feedback
The paper is relatively well written and easy to follow. The authors nicely motivate their modifications which clearly show that the gap between half and single precision training is almost closed. My main point of criticism is that the novelty of this paper is relatively small, as the techniques proposed for performing the quantized weight update are, firstly, not new and, secondly, have been used in previous works for similar reasons. More specifically, performing stochastic rounding for the weight update in order to make progress when the magnitude of the update is small (thus rounded to zero) was also proposed at [1], which also showed successful 16 bit training (albeit for outdated tasks). As for Kahan summation; that has been proposed before for training quantised neural networks at [2] (again as a means to avoid making no progress when the parameter update is small). As a result, the true contributions of this work lie on the extensive empirical evaluation along with the theoretical analysis of rounding in a simple linear model. 

As for other feedback and questions
- At section 3.1 you first assumption A1 seems peculiar, as you state that you work on a least squares regression problem, but you assume that the model is overparametrized. How can this be the case? In a linear model, the amount of parameters is bounded by the dimensionality of the input (and it doesn’t seem that you perform any feature expansion for x). I believe that a more reasonable statement would be to assume that the actual data are generated by a linear model, hence there exists a true solution w^*.
- Stochastic rounding requires the generation of random numbers; can this step also be done accurately in bfloat16?
- From the evaluation it seems that Kahan summation performs better but it increases the memory size by a factor of 2; how does this fare memory wise to having the weights in single precision and is it a worthy tradeoff?
- I believe that comparisons against other methods for quantised training that use fixed point instead of floating point would be interesting, e.g., [3, 4]. This would highlight the differences between these two formats and show whether the more expensive floating point operations are necessary.

[1] Deep Learning with Limited Numerical Precision, S. Gupta, A. Agrawal, K. Gopalakrishnan, P. Narayanan, 2015
[2] Training Deep Neural Networks in Limited Precision, H. Park, J. H. Lee, Y. Oh, S. Ha, S. Lee, 2018
[3] Training and Inference with Integers in Deep Neural Networks, S. Wu, G. Li, F. Chen, L. She, 2018
[4] Per-Tensor Fixed-Point Quantization of the Backpropagation Algorithm, C. Sakr, N. Shanbhag, 2018
review_title: Limited novelty, but extensive empirical evaluation

final_score: rating: 3: Clear rejection
review_content: During the rebuttal, I concluded that this submission is highly confusing, rather misleading.  I was led to believe the authors are in fact talking 'pure 16b MAC' - meaning 16b FP multiplies and 16b accumulate. After reading their responses to R4, I now learnt that they in fact are using 32b accumulates as is already standard practice in BF16 - If so, they have little or nothing new to offer. Rounding discussions the paper focuses on become highly secondary.  BF16 is already well-understood and accepted. Their writeup was highly misleading to say the least. I change my rating based on this.  

They use a representative set of benchmarks which include, CNN-based Resnet, recommendation proxy DLRM, and NLP proxy BERT. 

Novelty is limited, but critical: They reiterate prior observations that accurate weight updates are more critical for maintaining accuracy than forward-backward gradient updates. Former is what suffers when round-to-nearest cancels out small updates. Stochastic rounding has also been published before and shown to still miss the accuracy mark with >0.,1% accuracy gap for some important benchmarks. Novel part of this work is suggesting a Kahan summation based software fix on top of stochastic rounding to overcome the accuracy loss.

Primary issue: This is a very confusing and misleading writeup.  Authors should have clearly spelt out that they are in fact talking about BF16 - which is well-publicized for years now as 16b mule and 32b accumulate - and not labelled it as 'pure 16b MAC' - which is already known to exist as well, and proven to be not sufficient for DL training.  In light of this, this work has very little value-add. I change my rating now to 'clear reject' for their misleading writing style.

review_title: This submission rehashes BF16 labeling it incorrectly as 'pure 16b' and creating the wrong perception that they are first to question the so-called 'folklore' of 16b MAC not enough for training.

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees with reviewers that the paper has limited novelty and could be more clear about mix precision training. Therefore this paper is rejected.

Thank you for submitting the paper to ICLR. 

Decision: Reject