Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper analyzes the relationship between relative quantization errors and fixed-point formats for zero-centered normal distributions and finds a linear model which fits the best exponent length of the fixed-point data type given a standard deviation. These insights are then unified with parameterized clipping activation (PACT) to normalize incoming floating point data into the desired fixed point range. To handle the network with sole 8-bit multiplications a forward pass in floating precision is used to compute batch norm statistics for the main 8-bit forward/backward pass. Additional adjustments are made between successive layers and residual layers which rely on reusing some statistics of the previous layer.

main_review: Pro: 
This is a very strong paper and I have not seen any paper of such quality in this space for some time. The analysis of relative error and normal distributions gives a strong empirical and theoretical basis for the approach used in this work and provides deep insight into what the optimal fixed-point data type in each layer would look like. This analysis alone will inform future research in this area. Usually, such data types would be learned. It is very creative to abandon this approach and instead start from the analysis. Later this approach is merged with learned quantization through unification with PACT.

Overall, the results are outstanding and appear to be reliable given the extensive analysis of the approach.

Con:
Some details might not be accessible to readers that do not have an extensive background, especially pre-estimation of the fraction length and the sharing of the clipping thresholds for residual connections. It might be beneficial to trim the conclusion and add more context for these sections.


==============================UPDATE===================================

All the other reviewers take a very hardware-oriented perspective. I feel like they are not seeing the broader picture of this work which goes much beyond its initial applications. Many insights of this work are directly applicable for mixed-precision hardware where most, but not all operations are done in some 8-bit data type. From my own experience, I know that most 8-bit training methods fail when scaled up to models with billions of parameters.  For that area, quantization precision is the most important unsolved problem. This is a very important problem since transformer models keep getting larger and larger and accelerating training through 8-bit methods can make it more feasible to train these models. This work is one of the only ones in this space that makes some headway by its excellent analysis and impressive results. Many of the insights of the paperw are immediately applicable to my research. 

Overall, I disagree with the perspective of the other reviewers and I still believe this is an outstanding paper. As such, I keep my score of 10.
summary_of_the_review: The approach taken in this paper is very creative and the insights gained from this paper will inform research in this space for quite some time. As such, I strongly recommend acceptance of this work. I would be happy for this work to be selected as an oral presentation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a low-precision DNN inference models with 8-bit fixed point. To realize the number of fraction bits, the author uses the variance of DNN parameters and combines it with PACT approach in QAT. The new approach is evaluated in various neural networks such as MobileNet V1/V2 and ResNet18/50 on ImageNet for image classification and the result are mostly par with the state of the art approaches. 
main_review: Strengths:
1- The proposed numerical format is evaluated in variant of benchmarks which mean the proposed approach can be deployed and generalized for different models
2- Combining the PACT and fixed point is interesting approach.

Weaknesses:
1- The correlation between standard division and fraction bit-width is interesting but it is obvious and in my opinion it is not required. The fraction length can be selected based on dynamic range and I think the reason behind the constant in the equation 1 is the correlation between standard division and dynamic range (DR=3ST). If we closely look at the figure 3, we find most of these standard divisions ( more than 1 ) is not useful for your case study, since dynamic range of most parameters is less than one for most of the layers. This small dynamic range of parameters explains why most of the layers parameters are using 6 to 8 bit fractions in figure 2. This brings a doubt on the motivation of why using the fixed point instead of INT8 approach?
2- The state of approach like dyadic quantization [1] and linear quantization with bit shift [2] performs DNN inference with 8-bit quantization without 32-bit INT multiplication. How is your approach different compared to these studies?
3- The author claims the performance of his approach is better than state of the art. However, the 32-bit baseline is different and the degradation accuracy should be reported in Tables 1 and 2. For instance, in ResNet18 result the degradation of accuracy in HAWQ-V3 is less than the proposed approach. 
4- The new approach needs to compare with pervious work for 4-bit quantization. It is difficult to understand the advantages of this approach in compared to pervious work in 8-bit? 
[1] Yao, Zhewei, et al. "Hawq-v3: Dyadic neural network quantization." International Conference on Machine Learning. PMLR, 2021.
[2] Langroudi, Hamed F., et al. "Cheetah: Mixed low-precision hardware & software co-design framework for dnns on the edge." arXiv preprint arXiv:1908.02386 (2019).
summary_of_the_review: As a reviewer mentioned, the benefits and motivation of the proposed works is not obvious and the experimental result have not shown the advantages of this approach compared to state of the art works in 8-bit integer quantization. Therefore, I recommended this manuscript is marginally rejected.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes a novel quantization framework that involving only fix-point 8-bit multiplication for DNN execution. The paper first highlights the advantages of the fixed-point numeric format. The paper then conducts some statistical study and derive an empirical formula to relate the fraction length of the fix-point representation with the standard deviation of the value distribution. After that, the paper introduces a novel approach to determine the right format for each layer during the forward propagation of the training. The proposed solution, F8Net, has been evaluation on ImageNet using multiple DNN structures (e.g., MobileNet, ResNet). 

main_review: strengths:
+ The authors consider a practical issue from the perspective of DNN hardware implementation. The problem is well-defined.
+ The paper is easy to understand. The solution is presented clearly. 

Weakness:

- I am not convinced that 8-bit fix-point format is cheaper to implement than the other numeric formats. Quantization has been studied very extensively by the ML community, and numerous numeric formats have been applied to facilitate the implementation of the DNN. (e.g., binary quantized DNN, Logarithm quantization [a1], block floating point [a2], etc). The authors should better motivate this fix-point format by providing some hardware evidence. Without this, it is hard to convince the reviewer about the usefulness of this work.
- Equation 1 is based on the empirical approximation with Gaussian distribution, not the real DNN trace. Given the importance of this equation, authors should better justify the correctness of this empirical approximation with real DNN trace.
- This work assumes the ReLU is used for nonlinear operation. How to handle other types of nonlinear operations (e.g., leaky ReLU)? 
- In the paper, the author mentioned that they use previously stored activation fractional length for pre-estimating the fractional length for the following layer. What does the previously stored activation mean? Is it the activation from the last training batches?
- Observation 1 and 2 is obvious. It is nice to have some real empirical results to demonstrate this,  but I think they should be described very briefly.

[a1] Oh, Sangyun, et al. "Automated Log-Scale Quantization for Low-Cost Deep Neural Networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[a2] Darvish Rouhani, Bita, et al. "Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point." Advances in Neural Information Processing Systems 33 (2020).
summary_of_the_review: Overall, I think this paper lacks some motivation for the fix-point format. Given the fact that quantization has been studied extensively by the previous work, I am not convinced that 8-bit fix-point can achieve a better hardware efficiency than the previous work. Some hardware results may be helpful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new quantization flow to train DNNs using only 8-bit fixed-point multiplications. They show that 8-bit fixed point can represent different exponent ranges based on fractional length, thus choosing the right fractional length is critical. They then empirically derive a formula to calculate the optimal fractional length for a tensor based on its standard deviation. The authors combine PACT (learnable clip threshold) with fixed-point quantization, and propose a two-pass method to handle batch norm. They can achieve a small accuracy improvement (<1%) when training from scratch or fine-tuning on ImageNet using a variety of small models (ResNet-18 and MobileNet variants), compared to other quantization-aware training methods.

main_review: Strengths:
 - The paper is well written and easy to understand. Technical details are clear.
 - The idea seems practical and straightforward to implement in real systems

Weaknesses:
 - The paper claims that the quantized training flow uses only 8-bit fixed-point multiplications. However, calculating the fractional length requires the standard deviation, which needs to be done in float? The two-pass method for batch norm also uses float in the first pass. It's not clear if the technique is suitable on a specialized accelerator that performs only fixed-point multiplication. Instead the technique seems to target quantization-aware training (QAT) on GPUs for producing a quantized model. If this is the case, the results seem weak - the improvement over SOTA is very small, and the models tested are all small scale. The results are not enough to convince me that this is a significant contribution to QAT literature.
 - The novelty of the formulas to compute the fractional length seems weak. It's common in QAT to scale the tensor by dividing by the max value or dividing by some clip value. The scaled tensor then has range [-1, 1] and maps to a fixed-point representation with no integer portion. It's not clear what is gained by using stddev to estimate the fractional length instead. I would like to see some small experiments specifically comparing these ways to handle the dynamic range of a float tensor. In my mind they should achieve similar accuracy results.
 - It's not clear to me what combining PACT with quantization achieves. PACT is already meant to be used with quantization, and it's not clear what we gain by combining the two instead of doing PACT, followed by quantization. Again, some small experiments comparing the two would be useful.
 
summary_of_the_review: The authors propose a method to estimate the optimal fractional length of a fixed-point format given a float tensor, as well as other minor improvements to a quantization-aware training flow such as (1) combining PACT and quantization, (2) a two-pass method to handle batch norm. However, the novelty of the proposed methods is low, and it's not clear to me what improvements they offer over existing standards. The end-to-end accuracy improvement of the authors' technique is fairly small. As a result I rate the paper marginally below acceptance. If the authors can provide experiments that show specifically how each of their proposals improve over existing standard, then I would consider raising my score.

EDIT: The authors have addressed most of my concerns and I will raise my score to a 6. I still think that the formulas to compute the fractional length are not novel. The idea of estimating the dynamic range of the tensor and using that to determine the quantization range is well known.
review_title: null

Meta-review: This paper proposes an approach for 8-bit fixed point training of NNs, based on a careful analysis of quantization error in fixed-point methods. They present convincing and thorough empirical results in addition to a detailed analysis providing insights about their method. Reviews for this paper were quite split. One reviewer was a strong advocate, asserting that the paper will have substantial impact in the area, and that the authors’ approach of minimizing quantization error for fixed-point training is of substantial practical interest. Other reviewers were concerned that the proposed method was not novel enough, and that the proposed approach was not practical enough to work in realistic hardware use cases. The authors provided substantial detailed responses addressing the majority of reviewers’ concerns, and after following the discussion in detail I agree with the reviewer advocating for the paper, that the paper presents a practical, novel approach with valuable insights for the field from their analysis and results. 

I indicated I am certain about this decision, but I would be ok with the paper being bumped down from oral to poster.
Decision: Accept (Oral)