Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper investigates how the choices that can be made to leverage the FP8 format (i.e. including the important choice of the number of bits for the mantissa and exponent), and these investigations deliver analytical results on which settings for better performance. Following their analytic contributions, this work also shows how these findings translate to real networks, provide an efficient implementation for FP8 simulation, and a new algorithm that enables the learning of both the scale parameters and number of exponent bits in the FP8 format. The main conclusion from this work is that FP8 format is more suitable than INT8 in terms of the accuracy, when doing post-training quantization for a wide range of networks, the FP8 format is better than INT8 in terms of accuracy. Also, this work delivers a recommendation that the choice of the number of exponent bits is driven by the severity of outliers in the network (under FP8 format). A sensitivity study on quantization-aware training is also carried out.
strengths_and_weaknesses: Strengths:
- A sensible insight and the solution is convincing (for the accuracy)
- A method to make simulated quantization fast
- Evaluation results are sufficient to back up the current claim

Weaknesses:
- Limited insights (i.e., this work essentially describes a usage of minifloats for DNNs)
- The proposed method is only tailored to an efficient quantization procedure
- No discussions on the potential impacts on the inference performance
- Low reproducibility (No source codes available)
questions: - Can your method benefit the quantization-aware training in terms of the efficiency? If yes, how?
- Can you comment on the potential impacts on the inference performance, given the solution and your recommendations (i.e. 5M2E & 4M3E)?
limitations: - The comparison is performed between FP8 and INT8 fake quantization, and I suggest the authors to make it clear upfront.

- The accuracy is expected to be good by using FP8 format, since it can have a wider coverage than INT8. However, this can bring huge overheads in terms of the inference performance (see next), and it shall not be overlooked.

- The inference performance is hard to benefit from the concluded 5M2E and 4M3E: Minifloats are commonly used in embedded devices, which need to be emulated in software. These overheads are extra burdens to the inference over INT8 fake quantization.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper first analyzes the effect of the number of bits for the mantissa and exponent in FP8 data format, and then proposes a method to learn them with gradient descent. Empirical results on various tasks with various models show that the proposed FP8 quantization has a clear performance improvement for post-training quantization but performs similarly or even slightly worse than previously commonly used uniform INT8 quantization.
strengths_and_weaknesses: The paper is overall well written and easy to follow. The discussions on the effect of using different #bits for the mantissa and exponent to the quantization error are also intuitive. Extensive experiments on various tasks and various domains are performed to validate the efficacy 
of the proposed method.

Still some parts of the method and experiment sections need some more clarifications.
- Lines 119-120: Why the MSE can be approximated by (7). Are there any intuitions, explanations, or references?
- Table1: most of the time, "Fixed" performs similarly to "flex bias" and "flexible" on all models except DeepLabV3. Are there any explanations for this observation?

questions: Please address the questions above.
limitations: I did not find extra sections discussing about the limitations or broader impact, but some are talked about in the related work and conclusion. 
It would be better if the authors can discussion about them in more detail.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: FP8 is discussed both in the context of quantization-aware training and in the context of post-training quantization.  A very nice analysis of the expected quantization error is provided by the authors when the input to the quantizer is normal/ uniform/student's distribution. One of the most important and striking conclusions is that FP8 is a better format for inference than INT8, even though the latter has been the standard format up to now. In my opinion, this is a critical conclusion since switching between floating-point formats (used for training) and integer formats (used for inference) is a pain. 


strengths_and_weaknesses: Strengths: 
1. The paper provides a rigourous approach to proving that FP8 outperforms INT8 in inference. 5M2E is found to minimize MSE of normal distributions, and thus is of particular interest. This is a well-written, novel paper with clear practical implications. 
2. The results of the analysis and simulation are in close agreement. 

Weakness: 
1. I do not find the FP simulator to be novel or interesting. 
2. It was a great work by the authors to provide MSE based on clipping value (set via exponent bias). Next, they should find the derivative, set it to zero, and find the roots or, if that is not possible, numerically solve it. Otherwise, analytical expressions have less practical value. 
questions: 1. Bert-base opts for 3M4E in table 1. This may be due to heavy-tailed distributions of weights and activations. Could you please estimate the type of distribution Bert has in the FWD pass? Could a plot be included?
2. What task does Bert base use here (word masking)? I'm not familiar with the baseline accuracy of  "83.06" in table 1 (baseline accuracy should be ~86). 

XXXXX
Thank you for your detailed response. Could you please share the code? 


 
limitations: The current work has no negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the FP8 format quantization algorithm is realized based on a common FP32 deep learning framework for post-training quantization and quantization-aware training. The effects of choosing different bits for the mantissa and exponent on the quantization error and the final accuracy of the model are studied. Experiment results show that FP8 format is better than INT8 format.
strengths_and_weaknesses: ## Strength

1. This paper did a good work on clearly describing the floating-point format and quantization process.
2. This paper realized the quantization simulation based on the FP32 framework.
3. Experimented with models on various tasks.

## Weakness

I think the major weakness of this work is the lack of novelty and insufficient comparison with baselines. Please see the question section for more detailed discussion.

### Novelty
1. Using floating-point low-bit format in neural network quantization is very common. For example, group-wise quantization [1][2][3], different quantization parameters for different layers [4][5], special schemes for tensor with outliers [6], and QAT scheme for reducing quantization error [7][8] are well discussed in the literature.
2. It seems the experiments do not bring new findings or knowledge beyond existing studies. 

### Insufficient comparison
3. There are many other low-bit quantization schemes in the literature (as shown above), this paper should compare with more baselines other than the classical INT8 quantization scheme.

## References
[1] Köster, Urs et al. "Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks." NIPS (2017).

[2] Rouhani, Bita Darvish et al. "Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point." NeurIPS (2020).

[3] Zhong, Kai et al. "Exploring the Potential of Low-bit Training of Convolutional Neural Networks." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2022).

[4] Banner, Ron et al. "Post training 4-bit quantization of convolutional networks for rapid-deployment." NeurIPS (2019).

[5] Dong, Zhen et al. "HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision." 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019): 293-302.

[6] Zhao, Ritchie et al. "Overwrite Quantization: Opportunistic Outlier Handling for Neural Network Accelerators." ArXiv abs/1910.06909 (2019).

[7] Choi, Jungwook et al. "PACT: Parameterized Clipping Activation for Quantized Neural Networks." ArXiv abs/1805.06085 (2018).

[8] Yamamoto, Kohei. "Learnable Companding Quantization for Accurate Low-bit Neural Networks." 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 5027-5036.


questions: 1. What is the major contribution and novelty of this paper? I think the authors should make it more clear. Is it the "chief conclusion" saying that FP8 format has a larger representation range and so has a smaller quantization error on the tensor with outlier? This is a somewhat trivial knowledge. Is it the innovation of this paper to conduct “fast” FP8 quantization simulation in the FP32 framework? If so, the implementation of this paper should be compared with other quantitative simulation frameworks.

2. Line 93, "it has been shown to be indicative of the final loss in a neural network". Can it be confirmed by experiments in this paper?

3. This paper proposes to select the appropriate bits for mantissa and exponent for each layer (best flexible in the experiment? ) Is this off-line selection obtained by traversing the selection of mantissa and exponent bits for the weight and activation of each layer respectively? When selecting the quantization format for each layer, do you aim to minimize the MSE of the tensor, or minimize the scalar product quantization error, or maximize the final model score? If MSE is minimized, will the result with the largest model score be obtained?

4. "best flexible" does not seem to be significantly better than “best flex bias”. The tensor is divided into blocks, and each block adopts different exponent bias, which is equivalent to the integer format of block quantization, also called share exponent. This idea has been put forward in Flexpoint and other work [1][2][3]. Is the block +FP8 scheme used in this paper better than block + INT8? Further consider that the actual deployment requires hardware design. The use of best flex bias or best flexible requires support for various m/e combinations on hardware, while block + INT8 only needs to support one.

5. There should not be a "(PTQ)" under the "INT8" in Table 2.

6. From the incomplete results in Table 2, QAT is not necessary for best flexible? In other words, applying QAT to the best fixed scheme or even INT8 scheme is enough to achieve the optimal accuracy. Is the complex FP8 format such as best flexible necessary?

7. Because the colors in Figure 5 overlap, new colors are generated, which is difficult to distinguish. Maybe change to a line chart?

8. Line 146, "we fit Gaussian distributions in the weights and activations sample, and compute the expected MSE analytically in Fig. 3 (right)" If we are experimenting with the data in the actual model, why should we fit the Gaussian distribution? We should directly use the original data to calculate the quantization error.

9. The MAC results of FP8 tensors are in high-precision (such as FP32). The result after an operator needs to be intercepted as FP8, so, how are the operators divided in various network models? For example, before adding the shortcut path and the main path in ResNet, will they be both quantized as FP8?

limitations: The scheme in this paper might make the hardware design more complex, because it needs to support different m/e/b, which is mentioned in [3] (see the ref section in "Strengths and Weakness"). I recommend the authors to discuss on the implication for actual hardware implementation.
review_title: null

Meta-review: metareview: This paper had mixed reviews. 

One very positive expert reviewer (8) pointed out this paper used a rigorous approach to showing than FP8 can outperform INT8 in inference, which I agree is very interesting and useful.

 Another reviewer gave borderline acceptance (5), and I did not find any remaining concerns following the authors' response.

One reviewer gave borderline reject (4), but I did not find any remaining coherent major concerns following the authors' rebuttal. Also, this reviewer seemed less experienced, so I down-weighted this reviewer's score. 

Another reviewer gave borderline reject (4) with the following remaining concerns:

(1) "I think this scheme does not show advantages over the existing work. "
But I don't think this is true, since as far as I know previous work did not show such an advantage of FP over INT.

(2) "The practical application of the algorithm in this paper will bring extra overhead. "
I agree the authors should give more details here (especially for flexible), but at least for the flex bias method, the extra overhead seems quite reasonable (as this is similar to the standard method used for INT), so I'm not sure what is the issue.  

(3) "it is a very intuitive view that we should adopt a format with more exponent bits on the data with a large distribution range". 
But the authors' response correctly said this is not true (as a uniform distribution would be better represented using INT, no matter what it's range), and is not what they are saying. 

Therefore, I think the reviewer had some errors in understanding here, and so I down-weighted this reviewer's score.

Also, the following paper seems relevant:
A Block Minifloat Representation for Training Deep Neural Networks, ICLR 2022
confidence: Certain
Decision: Accept