Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Good ideas
Important area
Impressive results. 
Could be very useful for many embedded applications. 

I’m not a hardware expert but can see why this would be Useful.                            

                                                                                                                                                                                                                                                                                                                    
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper focuses on training neural networks using 8-bit floating-point numbers (FP8). The goal is highly motivated: training neural networks faster, with smaller memory footprint and energy consumption.
The proposed approach S2FP8 eliminates the need for loss scaling and does not require keeping some of the layers in FP32 precision as in Mellempudi et al. (2019).
The paper is well written, easy to follow, and provides a detailed background for readers who are not knowledgeable in this field.

On the downside, the first sections give the impression that FP32 is not used at all: "S2FP8 does not require keeping the first and last layer in FP32 precision, which is needed for other approaches (Mellempudi et al., 2019).". However, Section 3.2 says that "Master weights are kept in FP32" and that "Accumulations inside the GEMM kernel are kept in full FP32 precision". I think this should be stated earlier, because otherwise, the introduction is overclaiming.

The evaluation is very convincing - the approach is demonstrated for image classification, Transformer-based translation, and neural collaborative filtering. S2FP8 outperforms previous FP8 approaches and reaches the accuracy of FP32 out-of-the-box. 
It would be interesting to see the Transformer results with S2FP8 on additional datasets rather than the English-Vietnamese dataset only, which is considered a low-resource dataset, and on Transformer-base rather than "tiny". If you have such results on additional NMT datasets, I would be interested to see them, even if the performance of S2FP8 is worse than FP32 (I will not reduce my rating because of lower results there).

Overall, this is a good paper that presents an important contribution. Thus, I recommend accepting this paper.

Minor:
* Section 2, paragraph 3: "(Mellempudi et al. 2019) also demonstrated .." the name of the authors should be outside of the parentheses (\citet). Same in 4th paragraph: "(Zhou et al., 2016) quantized .."
* Equation (2) is a little difficult to read because of the sequence "and max log", in which each word has a different role. It might worth to break it into two lines, add brackets, or use logical "and" instead of the word "and"?
* Equation (3) uses "i-prime" in the argument for "max", but "i-prime" is not used.
* Figure 4 is referenced before Figure 3, this is a little confusing (the reader needs to scroll down for Figure 4, and scroll up for Figure 3).


review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: The paper suggest the method to train neural networks using 8 bit floating point precision values. The method requires a hardware modification to evaluate per tensor statistics, which intuitively capture dynamic range that a given tensor values may fall into. Those per tensor statistics are used to map 8-bit representation into a rational number. Statistics themselves are calculated using 32 bit floating point numbers.

While the method is interesting, I do not think it is practical due to the required hardware modifications. I am by no means not a hardware design expert, but I am not convinced that the gain of using 8 vs 16 bit floating point numbers outweights any extra complexity of hardware implementation. Mapping between representation and actual numerical values is much more complex than when using standard floating point representation (as any given numerical value is defined by a single its 8 bit representation + alpha + beta tensor statistics), integrated circuitry used to execute arithmetic operations would likely be much more complex than when using standard floating point operations. It is plausible that the added complexity (in terms of transistor count) would negate any potential price or energy savings over simply using 16 bit floating point representation. This should definitely be discussed in depth in this paper where the main contribution is an algorithm to be implemented in hardware.

As different layers will have different (alpha,beta) tensor statistics, from the computational perspective they would look like different data types which would be cast into the same common type before doing any arithmetic operations on them. This would almost certainly require extra computational steps to be done potentially negating computational benefits over simply using 16 bit floating point numbers.

The algorithm evaluates per-tensor and not per-tensor-element statistics. This may work well in cases where all entries in a given tensor are of the same dynamic range, but may break down in other cases: for example, diagonal entries in a matrix multiplication layer inside LSTM may have very different statistics comparing to non diagonal entries.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: There has been a great deal of interest and research into reduced numerical precision of weights, activations and gradients of neural networks. If, for example, 16 bit floating point can be used instead of 32 bit floating point, then the memory bandwidth is halved along with significant gains in computational performance.

In this work the authors propose an 8-bit floating point format (denoted S2FP8) for tensors. In general, computing activations and gradients with such low precision at training time, has generally proved challenging without a number of tricks such as scaling the loss of each minibatch to within a reasonable range. Such ``tricks'' can be difficult to tune for each problem.

The key idea here is that for each tensor of 8-bit numbers, two 32 bit floating point statistics are recorded as well. These determine (in log-space) a scale and an offset for the 8-bit numbers (eq 1). This means that in this format tensors of significantly different scales can be well-represented (although larger scales necessarily implies low precision).

Matrix multiplications are done in FP32 precision and then converted in S2FP8 format. This requires an additional step to accumulate the summary statistics of each tensor in order to convert from FP32 to S2FP8 (the mean and the max of the tensor elements in log space).

The weights of the network are stored in FP32 and the gradients and activations are computed in S2FP8 and used to update the weights.

They test this approach in ResNet, a small transformer and a MLP for collaborative filtering. They find it reaches similar performance to FP32 where standard FP8 format has worse performance or results in Nan's.

Improvements in computational efficiency, both at training and inference, are active areas of research and this work contributes a novel approach using summary statistics. However, there are a several ways this work could be improved.

1. There is no comparisons with bfloat16, which is becoming a widely used approach to lower precision and is gaining significant hardware support [1].

2. Discussion and analysis regarding the need to gather summary statistics after each matrix multiplication (or other tensor operation). It is claimed that this brings minimal HW complexity, but this doesn't seem well justified. For a large tensor, this additional reduction to compute statistics may be expensive (in memory bandwidth and computation), particularly since this is done with FP32.

3. Even with the current implementation on a GPU, it should be possible with kernel fusions to gain some significant savings in memory bandwidth (and therefore computational speed), but there is no attempt anywhere to show any runtime benefit on current hardware.

Minor issues:

Some captions are very terse and the figures would benefit from a clearer explanation (e.g. figure 6).

[1] https://en.wikipedia.org/wiki/Bfloat16_floating-point_format
review_title: Official Blind Review #2

Meta-review: Main description:  paper focuses on training neural networks using 8-bit floating-point numbers (FP8). The goal is highly motivated: training neural networks faster, with smaller memory footprint and energy consumption.

Discussions
reviewer 3:  gives a very short review and is not knowledagble in this area (rating is weak accept)
reviewer 4: well written and convincing paper, some minor technical flaws (not very knowledgable)
reviewer 1: interesting paper but argues not very practical (not very knowledgable)
reviewer 2: this is the most thorough and knowledable review, and here the authors like the scope of the paper and its interest to ICLR. 
Recommendation: going mainly by reviewer 2, i vote to accept this as a poster
Decision: Accept (Poster)