Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a flexible floating-point format that can save storage space by being highly configurable in terms of the bit width of the exponent/fraction field, the exponent bias, and the presence of the sign bit. The experiments in the paper demonstrate that the proposed format achieves a very low accuracy loss of < 0.3% compared to the regular float32 format for several popular image classification models. 

Strengths
---
- The concept of flexible floating-point format makes sense and can potentially result in significant space savings for large models. 
- The loss drop in the models compared to the standard float32 format seems to be minimal
- The authors evaluated the proposed format over multiple models: VGG 16, ResNet-50, ResNet-34, and ResNet-18 

Weaknesses
---
- No full system implementation to demonstrate the performance/memory gains. I that the updates to the form registers will be infrequent and cheap 
- The hardware details are very sketchy. Figure 6 seems to be only showing the high-level components.


Overall, the paper’s ideas are promising but my score reflects the fact that the authors presented end-to-end experiments demonstrating the performance/storage gains.
review_title: Promising idea but no end-to-end experiment 

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a new flexible floating point format (FFP8) on 8 bits, to help alleviate the high memory demand of deep networks inference, while preserving high accuracy. There is a large body of literature on reducing the data format, typically from 32 bits to 16, 8 and even below. There is previous work on using an 8-bit floating point FP(8), usually (1,4,3) or (1,5,2) where 1 bit is used for sign, 5 or 4 bits are used for the exponent and 3 or 2 bits are used for the fraction. 

The new FFP8 proposed in the paper offers more configurable parameters: (1) the bit width of exponent/fraction, (2) the exponent bias (usually this is implicit in FP8 but it can be changed in FFP8), (3) the presence of the sign bit (this can be removed and the bit can be used for exponent or fraction). By allowing flexible control of the dynamic range of the 8-bit FFP8, the accuracy loss is minimized. The authors observe that both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNNs. Therefore, the variable exponent size and exponent bias are better suited. Some activation functions always produce non-negative results (e.g., ReLU), therefore the sign bit can be reclaimed. 

The paper includes experimental evaluation where a best fit format selection shows a minimal accuracy loss for the VGG-16 network. Further optimization can be achieved over each layer, because distributions across different layers are dissimilar for both weights and activations, and the distribution of an individual layer is typically narrower than the one of the entire model.

I advocate for the acceptance of the paper. Although there is a long line of research examining the data format for training and inference in deep networks, the paper provides evidence of the usefulness of some flexibility in the 8-bit format. The proposed would not require hardware changes except for simple translations between FP32 and FFP8.

Minor comments / typos:

Fig. 4 and 5 - the vertical axis might look better with the same max values (range).

pg.2 : Moreover, recent studies proposed several training frameworks that __producing__ weights only in 8-bit floating-point formats

pg.3 : Note that there is actually only one parameter, the bit width of the exponent (y), __that__ can be freely chosen when defining
review_title: New flexible floating point format - review

final_score: rating: 3: Clear rejection
review_content: This paper explores 8-bit floating point formats for the inference of deep neural networks. The quantization is applied on the weight and activation tensors, but computation engine remains in FP32.  To cover the different ranges of weight and activation tensors, the authors propose to use exponent bias. The authors did ablation studies on the impact of numerical formats, e.g. bit-width and exponent biases, on model accuracies. The experiments are performed on vision models on ImageNet, including VGG and ResNet 50/34/18. 

The paper is relatively clear written, and experiments seem sound, however, I have some major concerns on the objective and novelty of this paper. 

1). It is not clear to me the objective of this paper. This paper introduces an 8-bit quantized inference framework. However, it is well-known that, 8-bit precision can be applied to popular DNN models to accelerate inference while maintaining model accuracies and many such systems have already been put into products (e.g. TPU). The state-of-the-art inference quantization work are primarily in the precision of 4 bit or less. But this paper did not compare their work to any of the latest inference works. Instead, the three references this paper compared to, i.e. (Wang & Choi, 2018; Cambier et al., 2020; Sun et al., 2019), are all for the acceleration of DNN TRAINING, where the challenging can be very different, e.g. quantization of gradient tensors.

2). On the same note, this paper claimed advantages of using FP32 computation engine, however, the three training papers, by definition, can also be used in FP32 computations through the same high precision converting described in this paper. In addition, it is not clear to me how much one can benefit from quantization while keeping computation in 32bit? For example, one may save some memories space, but the whole inference will be limited by the FP32 computations in terms of throughput, latency, power and cost. Can the authors provide an application case where only quantizing data is beneficial?

3). The main contribution claimed in the paper are to use exponent bias to cover tensors with different range of distribution. However, this technique has be introduced and discussed in (Sun et at., 2019) paper and has also been investigated in detail in (Cambier et al., 2020) paper. The authors of this paper did not provide any new insight although using in a much simpler case, i.e. only forward paths.

I think this paper lacks clear objective, novelty and insightful analysis, therefore not good enough for ICLR.

review_title: paper lacks clear objective and novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary:

The paper introduces a new floating point format FFP8 that can adaptively choose the exponent bias as well as the existence of the sign bit. FFP8 is more flexible than other 8-bit floating point formats with fixed exponent biases. The authors show how FF8 can be used to cover the dynamic range of NN weights and activations while still achieving higher precision than commonly used FP 8 formats.

## Strengths:

* Experiments on how various settings of FFP8 can be used to represent weights and activations of different layers and comparison with existing FP 8 formats.
* Varying exponent bias is a good insight to shift the dynamic range of FP formats.

## Weaknesses and Questions for the Authors

* Limited details on how hardware support for FFP8 look like.
The authors claim in section 5 that the extra hardware support required for FFP8 would be minimal. However, no data or simulations are there to back this claim. In a possible hardware implementation of FFP8, each floating point instruction should be preceded by an instruction which sets (x,y,b) registers for FFP8 to F32 conversion. The frequency of setting these registers may vary based on the application, but nevertheless is an overhead compared to the current FP 8 implementations. The authors should quantify this overhead with a hardware simulation. Alternatively, the authors can describe a different and an efficient implementation of FP 8. This is an important detail that is missing from the paper.
* Compiler support for FFP8
 The authors have not described how FFP 8 would be emitted, specially when different settings are used for different layers. Do you expect the programmer to hand code this? Or will there be compiler support? If so, what would the compiler support look like? What would the compiler optimization algorithm look like to produce FFP 8 code? These details are not answered by the paper.
* More evaluation on other commonly used NNs
The authors present results for VGG-16. I would like to see more results on different networks. The authors can cut down on some of the graphs or make them smaller to get more space (specially Figure 5). Also, in general I feel like rather than giving out all the results that bring us to the same conclusion, the authors can summarize the results or graphs to make the message / conclusion of the paper more clearer.

review_title: More information about HW and Compiler support needed

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees with the reviewers that the paper has limited novelty as there are already previous studies on setting floating point configurations. Additionally, the particular hardware setting that the authors provide seems to rely on a fp32 FMA, which defeats the purpose of a low bit floating point(where smaller FMA could have been used).Therefore this paper is rejected.

Thank you for submitting the paper to ICLR. 

Decision: Reject