Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose 2 impactful methods to aid in the design of numeric precision for neural network training: a method to quickly determine which formats work for weights and activations using angular deviation of gradients between low precision and FP32, and a hysteresis method for dealing with low precision representations.  
main_review: While there are few nitpicks I have (these should be fixed prior to publishing), the ideas in the paper are well-founded and useful.  I believe there is a meaningful contribution to the field here.

In the intro, there really needs to be a citation for Koster et al (2017) on Flexpoint. That was one of the first papers discussing these issues and one of first implementations of different numerics in modern hardware.

In the second paragraph of the introduction, the line "When a value is represented using a fixed number of bits, there is a trade-off between dynamic range and precision" is not quite correct.  This is only true for floating-point formats, so simply stating that will clear this up.

In the "Performance degradation in from-scratch training" section: The line "the neural network is trained from scratch while all the values in the network - not only parameters, but also other variables in the network (e.g., activation, error, and weight gradient) " is not necessarily true and different components are done differently even in this paper. I'm not sure why there is so much text devoted to differentiating quantized vs from-scratch training, but I think this could removed.  The paper is about from-scratch training so just state that.

Under Numeric Formats, I'm not sure what a "symmetric" format is and why 2's complement is asymmetric.  Please define or remove.

Under Effect of quantized activation, it says "...suggesting that the misalignment angle is a better metric to predict the ranking of various formats based on the training performance." I think this is true, but more due to the shape of the curve rather than the (small) effect on spearman correlations.  It might be clearer to state this; the angle differences are high for even small losses.

The hysteresis method has some great results and might really be the key to unlocking lower precision.  Great work!
summary_of_the_review: The paper presents 2 very important ideas that I believe are of high utility to low precision efforts. The idea of weight hysteresis could actually spawn many new methods that lead to better hardware going forward.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to find an optimal quantization format based on error angle estimation and hardware overhead. The authors also present an hysteresis-based quantization method to reduce fluctuation of exponent values such that training (from the scratch) using only 4-bit weights can result in negligible amount of accuracy degradation for ResNet-18 on ImageNet. Experimental results are provided for ResNet-18, MobileNetv2, 2-layer LSTM, Transformer, and MobileNetV2+SSDLite. For 8-bit quantization, FP134 is chosen and such quantization is also applied to BatchNorm layers to reduce memory consumption.
main_review: Introduction discusses the difference between quantized models (after fine-tuning a pre-trained model) and from-scratch training for quantization (to gain speed-up for training) clearly and the direction of research to investigate why from-scratch training methods show increased degradation in accuracy is certainly important.

Unfortunately, this reviewer finds the following serious concerns in this paper.

(1) The authors argue that measuring the misalignment angle is better than the magnitude of the error. But in Section 2.3, it is not clear why the noise in the opposite direction is harmful. Similar to stochastic variation, noise on gradients can show some benefits if the amount of error is right such that regularization effects can be obtained. Moreover, as shown in Figure 2, reducing the magnitude of error also tends to present the training loss change. Is it difficult or impossible to find FP134 as an optimal one with magnitude-based error measurement? Supporting data to validate the claim (that the angle is better than the magnitude) needs to be provided.

(2) This paper seems to suggest a design methodology of NPU in the form of ASIC. Then, the authors need to prove that FP134 can be applied to a wide range of models. Such a particular format would show significant accuracy degradation if model size increases. What would be the limit of such a format? Moreover, the authors show the superiority of FP134 using ResNet-18 model while a few additional experimental results using some simple Transformers, such a strong argument to choose FP134 as an optimal one needs to consider what kind of limitations would be provided. For example, schemes in Table 3 might not be good for ResNet-18 but may be good for ResNet-101. This reviewer is not sure whether FP134 is a customized one for small models such as ResNet-18.

(3) A quantization technique using Hysteresis is interesting. But more detailed discussions and theories why Hysteresis is important for 4-bit Log W need to be included. Is Hysteresis generally helpful for other quantization formats and larger models as well?


summary_of_the_review: Overall, even though this reviewer finds some interesting ideas including Hysteresis and experimental results are good for ResNet-18, the followings need to be addressed.

(a) FP134 cannot be found by measuring the magnitude of the error?

(b) Do the authors suggest FP134 as a format to be applied to a wide range of models (especially even with large model size?). If so, please provide supporting data and theories.

(c) Most experimental data are only for ResNet-18.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method to predict the performance of different numeric formats, which allows determining the optimal data format for various neural network architectures, datasets, and tasks efficiently. By comparing 498 formats in total, the authors find an optimal 8-bit format suitable for various models. To improve the performance of from-scratch training, the authors further propose hysteresis quantization to mitigate the ﬂuctuation issue. Experiments on 8-bit and 4-bit training demonstrate the effectiveness of the proposed method.
main_review: **Contribution:**

1.	The authors propose a metric to evaluate the performance of different numeric formats. Using the proposed metric, the authors further find an optimal 8-bit numeric format suitable for various models.

2.	The authors find that the performance degradation of 8-bit training is due to the fluctuation issue of quantized weights. To solve this, the authors propose a hysteresis quantization scheme to improve the performance of from-scratch training.

3.	Experiments on 8-bit and 4-bit training show the promising performance of the proposed method. 

**Questions and points needed to be improved:**

1.	In Figure 2, the improvement of spearman’s correlation of the proposed metric over the magnitude of the error is marginal (0.9283 vs. 0.9215). It seems that the magnitude of the error is a good metric to measure the performance degradation. What are the advantages of the proposed metric? More explanations and results are required.

2.	In Section 3.1, the authors state that the amount of change in the quantized weight due to the fluctuation is not necessarily proportional to the weight gradient. To mitigate the fluctuation issue above, the authors propose hysteresis quantization scheme. However, Figure 5 can not show the effect of hysteresis quantization. The number of $Q_w$ changes is the same in Figure 5(a) and Figure 5(b). More explanations are required.

3.	The idea of changing the rounding function in network quantization is similar to AdaRound[1]. It would be better for the authors to add more discussions on the difference between the proposed method and AdaRound.

4.	In Table 4, many notations are unclear. What do “X” and “O” denote? What do dw and x denote? More explanations are required. 

**Reference:**

[1] Up or Down? Adaptive Rounding for Post-Training Quantization. ICML 2020.


summary_of_the_review: Experiments show promising performance. However, there are still some concerns regarding the proposed method. Some notations in the experiments are not clear. More explanations are required.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission starts from an interesting point that various quantized training environments may require different formats for training accurate deep neural networks. They present a metric based on the misalignment between $\frac{\partial \ell}{\partial w}+noise$ and $\frac{\partial \ell}{\partial w}$ to determine the optimal format. To mitigate the fluctuation issue caused by network quantization, they propose a hysteresis quantization scheme to avoid frequent changes of quantized points. The experiment results fully support the effectiveness of the proposed methods.
main_review: pros.
1. This paper addresses a very relevant topic. Though model compression has been widely discussed in the recent literature, efficient training with quantized networks is an approach worth exploring.
2. The technique part of this paper is simple and easy to follow. It is clear that the Hysteresis setting contributes to a consistent improvement over the standard approach on both image recognition and NLP tasks.

cons.
1. There are some related works missing in comparisons such as Flexpoint [1] and HFINT [2]. Especially, HFINT first notices the difference in weights distribution between CNN and NLP models/tasks then presents an adaptive floating-point format with hardware implementations. Compared with the static FP143/FP134 mode used in the current draft, they use an on-the-fly adaptive format. In light of this, it would be better to include a HARDWARE EVALUATION section to verify the effectiveness of the proposed "data flow quantization".
2. Please denote "weight gradients" as $G_{W}$ or $G^{W}$. $WG$ commonly refers to matrix multiplication.
3. Figure 2: How to distinguish 8-,7-,6-bit quantization formats? 
4. It seems that Figure 8 (in A.3) has almost the same tendency as Figure 2. In general, F134 performs the best. Why introduce a more complicated metric to determine the optimal format? Note that the authors conduct all experiments under the setting of FP134.
4. How to obtain $WG+N_{\Delta E}$ and $WG+N_{\Delta A}$?
5. Table1: Why does "1-input" consume a much larger area than conventional MAC? Besides, why conventional MAC consumes a smaller area along with the decrease of mantissa, yet Multi-way MAC behaves in the opposite way? I expect more results on LUTs, DSPs, BRAMs, and Power except for Area.
6. Is hysteresis quantization compatible with uniform quantization?
7. Does FP134 still involve a scaling factor $s$ during training?

ref.
* [1] Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks. NIPS2017
* [2] Algorithm-Hardware Co-Design of Adaptive Floating-Point Encodings for Resilient Deep Learning Inference. DAC2020

supplementary:
1. Please include a **README** file in the supplementary material. 
2. How to understand the magic number $555555543210$ used for "data format for data flow quantization"?
summary_of_the_review: Overall, the idea of hysteresis quantization seems neat and new. However, the effectiveness of the performance indicator needs to be further clarified. The empirical success of FP130 with hysteresis seems to further weaken the motivation of finding the optimal data format. 

The authors are encouraged to address the weakness. I will consider changing the final rating according to the authors' feedback.
review_title: null

Meta-review: This paper introduces a method to determine which precision to use for the weights, as well as a quantisation method using hysteresis to improve performance with low-precision weights, including 4-bits.
Reviewers tend to agree that the two points presented are useful and can have a large impact on the field.
Generally, reviewers pointed out that motivations, notations and experimental studies could be improved. This has been partly addressed by the authors.
I recommend to accept this paper for ICLR 2022.
Decision: Accept (Poster)