Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper has a short analysis on rounding schemes, comparing nearest-rounding to stochastic rounding for quantized training. It also introduces LUQ, a new quantized training scheme with a FP4 format for the gradients. In order to make this work efficiently in hardware, they introduce a new hardware block that does multiplication-less gradient calculations for the backward pass. 
main_review: This paper is interesting, but leaves me with too many questions and uncertainties on all three of it's contributions to make a proper judgment. It would be great if the authors could address my questions below for clarity:

I appreciate the rounding scheme analysis w.r.t. the MSE. I haven't seen that specifically before, and it's insightful to understand why stochastic rounding on the forward pass is not necessarily a good idea. There is one small caveat that should be mentioned, and that is that noise sometimes helps for proper convergence and regularization. Dropout adds noise, and increases the MSE, but still sometimes helps. It would be good if the authors rephrased this section slightly to take this into account properly. 

The conclusion on page 3 is much too fast and needs to be worked out significantly more for clarity. Paraphrasing: "Unbiased gradients are necessary for convergence, therefore gradients should be quantized with stochastic rounding". I can't find the statements regarding the biased gradients in Bottou 2010, perhaps the authors could point me to where exactly this is mentioned. 
I would also like to understand better what the bias of the gradients actually means, and why this would be so detrimental. If you talk about the bias of the forward pass given the stochastic distribution, I understand. However, it's unclear to me what the effect of this is on the gradients themselves. It would be good if the authors worked this out. What are the gradients biased with respect to? The FP32 gradients? Is that even the correct thing to compare to?
To me, figure 1 is insufficient evidence that a 'bias' is necessarily the issue in training. This effect would have to be disentangled from other effects that adding noise to a network would have on training. As it stands, the difference could come from just the addition of noise.
Similarly, The argument that RDN should be used in the forward pass goes much too fast for me as well. It would be nice if the authors showed explicitly how the linearity causes a non-biased estimate of the gradients in the forward pass. 

Section 3.1 Extra sampling would also linearly increase the cost of the method. So why would this be better than simply training more iterations on different data as opposed to the same data? A discussion of the overhead/complexity of this method already here would be much appreciated. For the results section, I also don't see how the overhead comes into play with the comparison the raw LUQ method itself.

Section 3.2 Wouldn't we generally want the activations in a lower bit-width as well?

Method - It is unclear to me if the weights in this scheme are kept in a quantized format, or if they are in floating points with the common 'shadow-weight' approach. Please take more care in describing this properly. If the weights are in floating point, that would mean a very significant overhead during training, as the floating points would have to be loaded somewhere, then quantized, then an operation is to be applied to them. In full quantized training, you would want to have the weights quantized as well, but I see no mention of this. Similarly, are the updates to the weights quantized? See e.g. the WAGE paper.

Underflow Threshold - What are you taking the maximum over? If you are taking a maximum over all the values in the tensor you are quantizing, you will run into problems with quantization as described in the In-Hindsight quantization paper (https://arxiv.org/pdf/2105.04246.pdf). Essentially, for activation quantization, if you need any statistic of a tensor to do the quantization, you have to write significant amounts of data to memory, which is very slow. You might as well not have done activation/gradient quantization at that point. 

Overhead of SMP and FNT. This number shouldn be 8x compared to FP16. Compared to FP32 it would be 16x at least. 

Main results. It seems you are using at least a different ResNet-18 model compared to the Ultra-low paper, as the baseline accuracy is 69.7% for your model, but Ultra-low reports 69.4. Could some difference not be explained based on diffent models? The delta in performance is the same, roughly 0.7 for ResNet18. Also, how does this method compare to other works before it like WAGE (https://arxiv.org/pdf/1802.04680.pdf) and other papers that are based on this?

You mention that the shortcut connection in ResNets and the depth-wise layers are kept in full precision. For the depth-wise layer, does that mean the input activations to it, or the output activations? For both, I don't think this is necessarily trivial to do. It's not a given that a fixed point training engine also has a full-precision engine on it's die. Having that would significantly increase the die size. Similarly, although the compute is small of this solution, the data movement is not. Data movement is a very important metric on efficient devices (the ones we would be doing quantized training for). This needs to be addressed in the context of overall efficiency.

MF-BPROP - I definitely appreciate the hardware implementation, as if you'd have just the FP7 calculation MAC array, you might as well just do INT8 computations all the way as opposed to going throug the trouble of quantizing everything to INT4/FP4. But this section does highlight one problem with this method: Dedicated hardware is necessary for this method to work efficiently. This greatly limits the scope of this work in it's practicality. And this brings me back to the above data movement questions that were glossed over. The suggested method is not a general method for quantized training and more of a 'hardware and software go hand-in-hand' kind of method, but many of the practical details of the hardware implementation and costs/overhead are left out. Please comment on this.

Type-o's
page 3 round-to-zero | nearest
page 4 avoid of clipping | -of
Page 5 The a different samples | -a
Page 6 4-bit training in various DNN models | in -> on


summary_of_the_review: This is an interesting paper with some interesting insights, but the paper tries to do too much in too small a space. This leads me to have too many open questions for a good rating. The paper would likely have been better if it honed in on a single aspect that they present, and did so in a clearer fashion with more bases covered.
The conclusions on page 3 are drawn much too fast, and are unclear. The authors have to convince the reader a lot harder to make the claims they make.
On the one hand the paper's main method only seemingly work more efficiently with very dedicated hardware implementations, but several important aspects of efficient hardware implementations of training, such as data movement, are seemingly ignored. 

I would be willing to increase my rating if the authors addressed my above questions, rewrote section 2 significantly with a more convincing analysis, and added a better complexity analysis of the overhead of their method in terms of data movement.

*** post-rebuttal ***

Given the author's excellent response to my questions, and the questions of the other authors, I am increasing my review to a 5. For a 6 or higher I definitely need the efficiency conundrum resolved, as for now I just don't see how to resolve the dichotomy between being either a generally applicable method on common-day and general hardware, or a method that is really aimed at hyper-efficient dedicated implementations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on reducing the computational footprint of training neural networks using quantization. They investigate the importance of having unbiased gradients and show that stochastic rounding is important for the backwards pass. They propose a logarithmic unbiased quantization (LUQ) scheme which achieves SOTA results for 4-bit training on ImageNet. Results are further improved with multiple sampling (SMP) and a final full precision fine-tuning (FNT) at the cost of extra compute overhead. Finally they introduce a HW block which exploits LUQ by avoiding multiplication which can reduce the multiplier area by 5 times.
main_review: 
Strong points:
* Theoretically well-motivated choice of quantization scheme for forward and backward path.
* The logarithmic unbiased quantization method is theoretically sound and well explained.
* Strong results for 4 bit training of various ImageNet models.
* Clear ablation studies for the contribution of SMP and FNT in table 1.
* Most parts of the paper are well written and easy to follow.

Weak points:
* It is unclear what the contributions of unbiased stochastic pruning and logarithmic unbiased rounding in LUC are. An ablation study would be insightful on this.
* The SMP method for reducing variance requires further clarification. It is unclear where and when this exactly is applied. For example, are the multiple samples only used to calculate the weight update (similar to the second sample in Sun et al.) or is the average of multiple samples also used in back propagating to he other layers. If the latter, then this seems to likely imply that a higher bit width needs to be used in the matmul of the next layer. How does the multiple samples compare in performance and complexity to using a different bit width?
* FNT: full-precision fine-tuning makes the comparison to other low-bit training methods unfair. But results are also show without it and show competitive results.
* The transform to Standard FP7 lacks detail and explanation. It is hard to understand how the input/output table is constructed.
* While the paper is easy to read and clearly written, many small details are missing, especially on the experimentation part (see questions below).

Questions:
* What is the HW impact of rounding to nearest power? This seems significantly more complicated than adding uniform noise as it is the case for uniform quantization.
* How do they define the pruning threshold? Are they learned/updated? I could not find any details on this in the paper.
* How do you deal with BN?
* What quantization approach is applied in the forward pass (weights and activations)? Rounding to nearest is clear, but how are the ranges defined/learned etc.

Editorial notes: 
* Few appreciations introduced are unclear what they stand for. E.g SMP, FNT, RDN. The later I assume comes from rounding-to-nearest, but would than not RTN be the right appreciation?
* Typo below equation 9 (rounding-to-zero).
* Page 3: “will not make help making the loss estimate unbiased”
summary_of_the_review: Overall a well written paper that is easy to follow (except a few parts highlighted above). The proposed approach is somewhat novel, it combines stochastic rounding with logarithmic quantization and stochastic pruning. The main strength are that most parts are well motivated and the strong 4 bit training results. The main weaknesses are around SMP and some other parts that are unclear. Overall the contributions slightly outweigh the weaknesses of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes techniques to quantize the gradients in the back-propagation algorithm down to 4 bits. To do so, the authors propose the [1,3,0] radix-2 FP4 format, as well as bias removal and variance reduction techniques. In order to achieve SOTA accuracy, the method requires an additional stage of high precision fine-tuning.
main_review: While this paper is interesting and presents promising results, I did have many questions listed hereafter:

Introduction: Radix-4 representation is criticized arguing that conversion to radix-2 requires an explicit multiplication to modify exponent and mantissa. This is untrue, in the paper by Sun et al. that the authors cite, the radix-4 format does not use a mantissa, it only uses 3 exponent bits, and therefore converting to radix-2 simply requires appending the exponent field by a zero.

Section 2: In the calculation of rounding variance and MSE, why is the input x considered deterministic?

In Section 2, conclusion, the following claim is made:  "the forward phase should be quantized deterministically (using RDN) since stochastic rounding will not make help making the loss estimate unbiased (due to the non-linearity of the loss and activation functions) while unnecessarily increasing the mean-square-error". Here the authors are claiming that the use of non-linear activations and loss will negate the fact that SR is unbiased. Why is that the case? Is this a known result from prior arts? If so, the authors should add a reference as was done in this same paragraph regarding the works of Chen et al., and Bottou.

The LUQ proposed in eq. (11)-(12) is not unbiased when n=b-1, i.e., in the quantization region corresponding to the largest magnitudes. Usually, such boundary cases do not matter, but given that here there are only 16 regions, and that the data is assumed to be heavy-tailed (most of the data would actually fall in this final region), this issue might be significant. Can the authors comment on what happens in the final quantization region?

Related to the above, this is more of a suggestion. Rather than setting an underflow value as quantization meta-parameter, why isn't the max of the tensor used instead? This would avoid the above issue. Have the authors considered that? Did tuning an underflow hyperparameter yield better results?

In eq.(14), why is a subtraction of a 1/2 term needed when RDN is defined as per eq.(5)? Also this equation is improperly typeset.

One final question: The issue of quantization bias occurring in gradients and weight updates has previously been studied in the paper below. Can the authors compare their findings and check if their conclusions are consistent with prior arts?
Sakr et al., Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm, in ICLR 2019.

The experimental results, and crucially, the implementation details in the Appendix look good.

Finally, there are many typos and grammatical errors throughout the paper. I urge the authors to perform a spell check.
summary_of_the_review: The paper is interesting, tackles an important problem, and presents promising results. Therefore, I do not recommend a clear reject. However, there are unfortunately many issues I though were unclear and hence raised in my detailed review. Therefore, I cannot recommend acceptance of the paper if these issues are not addressed. Hence, for now I am rating the paper as a weak reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes some techniques to train a deep neural network in 4-bit and provides a theoretical analysis. The experimental results support the effectiveness of the proposed method. Although some part of the method still needs high precision, the Reviewer thinks that it could open a door to training the deep neural network with ultra-low bits.
main_review: ** Strength **
1. Overall the paper is well written, quite thorough, and well-cited. The theoretical analysis related to stochastic rounding and rounding-to-nearest are well justified.
2. The paper achieved superior training results in 4-bit training.
3. The authors not only provided the 4-bit training algorithm but also suggested dedicated hardware blocks.
** Weaknesses **
1. Some parts of the proposed method still need high-precision
2. Some experimental results, MobileNet-V2 and ResNext-50, are not provided because of the time deadline.
3. The hardware implementation costs are evaluated in terms of logical area, but I think it should also be analyzed in terms of memory area since the multiple sampling (SMP) and underflow threshold computation parts need additional memories.

** minor comments **
1. In Figure 1 (b) and (c), which rounding methods are adopted for Bwd and Fwd, respectively.
2. In order to reduce variance, the author performs multiple samplings and averages them, but as the sampling increases, does not it get closer to the RDN?
3. There is a typo on page 5. (The a different samples -> The different samples)
4. The authors mentioned that "we increase all network parts to full-precision, except the weights" for the high-precision fine-tuning. If activation is also trained with full precision here, after fine-tuning, the activation precision should be lowered back to 4 bits.  Was there any performance degradation at this point?
  
summary_of_the_review: This paper proposed several techniques for effective 4-bit training. Some parts of the proposed method still require high-precision, but it might be a good starting point for full 4-bit training in the near future.
review_title: null

Meta-review: This paper proposes a method for 4-bit quantized training of NNs (forward and backward), obtaining SOTA 4-bit training quantization, motivated by an analysis of rounding schemes (an important aspect) in quantized training. The main concerns from the reviewers were that the approach was not practical (both a general concern, and of specific note here since the word is used in the title and motivation of the work), due to lack of compatibility with (current) general purpose hardware, and lack of suitability of the approach for specialized hardware, so it is unclear what the actual use case is for the approach. The authors argued that (1) this is not a problem on some hardware and (2) that past works have not been held to this standard. I did not find the authors to provide a strong argument during the discussion period to address these concerns.
Decision: Reject