Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduced a new representation (Block Minifloat) for training DNNs with low precisions of 8-bit or less. This new representation combines FP8 formats and the shared exponent bias concept to cover the dynamic range of tensors needed for DNN training. Compared to other published FP8 format, this representation has smaller exponents, which allows to use a more efficient Kulisch accumulator. The representation has been verified on a spectrum of deep learning models and datasets.

Overall, the paper is well written. The idea is clearly presented, and experiments are sound. Particularly, the hardware evaluation gives some impressive results. However, for the sake of clarity, I have some questions.

1). The shared-exponent bias itself is not new and use it for FP8 training is also straightforward and has limited novelty. What interesting is that the author uses this method to push for smaller exponent bits which in turn allows a more efficient accumulator. However, as a key contribution of this work, the authors did not give enough information and details on why “exponents up to 4 bits offer distinct advantages” for Kulishch accumulation. Could the authors explain more on the accumulator and provide some evident why smaller exponent is critical?

2). From emulation point of view, the authors use existing CUDA libraries for GEMM which basically uses standard FP32 floating point accumulation. When the authors use certain bit-width for Kulish accumulator elements (for e.g. Table 9 and Table 10), how do you know this accumulator setting won’t impact model convergence? 

3). Hardware overhead for denorm support: the exponent bias is only guard against overflow, so denorm numbers are used to cover the range of smaller numbers. The authors claim that the hardware overhead is minimum since only input multiplicands are needed for the detection of denorm. However, there should be additional complexity than that. For example, how to handle denorm numbers in addend and how to tell the result produced is norm or denorm? Could the authors describe the hardware that needed to convert numbers from an intermediate format or a floating point format to their BM format?

4). For experiment, it would have been good to include some larger networks, such as ResNet50 on ImageNet to compared to SOTA.

review_title: A good submission but need to provide more information

final_score: rating: 7: Good paper, accept
review_content: The authors proposes block-minifloat (BM), a floating-point format for DNN training. BM is a fairly simple extension to block floating-point (BFP), which was proposed in (Drumond 2018 and Yang 2019). In BFP, a block of integer mantissas share a single exponent. In BM, a block of narrow floats share a single exponent bias. The shared exponent bias helps to shorten the exponent field on each individual float element. This is a good contribution, though a bit trivial.

Where the paper make a strong contribution is in the hardware implementation of BM, something which neither Drumond or Yang really got into. The authors propose to use a Kulisch accumulator for minifloat dot products, which basically works by converting the floats to integers with a shift, and accumulating with a very wide register. Kulisch accumulators are normally far too wide to be practical (see Eq. 4), but they've been proposed for posit computation (https://engineering.fb.com/2018/11/08/ai-research/floating-point-math/). This seem like a great idea here since BM can reduce the exponent length to only 2 or 3 bits.

The authors also did a good job evaluating the area and power of the BM hardware circuit. They built a 4x4 systolic array multiplier using BM units in RTL, and synthesized to place and route. The results show that BM6 can be 4x smaller and use 2.3x less power than FP8 while achieving to comparable accuracy. This is a pretty impressive result, and the hardware evaluation methodology is more stringent than most quantization papers at NeurIPS/ICML/ICLR. The only **minor** issue I have here is the area/power numbers are reported for BM8/BM6, but the exact config is not specified. E.g. is BM8 referring to (2e, 5m)?

The accuracy comparison is pretty standard, with CIFAR and ImageNet results using mostly ResNet-18. The authors' simulation framework slows training by 5x, so this is as much as I would expect. One **major** issue is that Tables 1 and 3 shows that for training to succeed, the forward and backwards BM formats must be different. Table 3 has three separate BM formats for each row. Implementing them all in hardware could incur significant overhead, which the paper doesn't discuss. The authors mention that the HFP8 paper does the same - but that paper defends this practice by showing that their two formats (which only differ by 1 e/m bit) can be supported by a single FP unit with minimal overhead. This paper uses (2e,5m), (4e,3m) and (6e,9m) in the same experiment labeled "BM8", which seems both misleading and unjustified. Note that SWALP and S2FP8 (and bfloat16/float16 papers) would use the same format in forwards and backwards pass and avoid this overhead.

A few other insights: (1) subnormal floats are important and can't just be flushed to zero; (2) a square BM block size of 48x48 seems to work fine.

Minor issues:
 - The methodology for hardware area seem solid (Appendix 4), but there isn't much detail on power. Was power obtained through modeling or using an RTL simulator? What kind of test vectors were used?
 - The area/power numbers are given for "BM8", but what's the precise format? I assumed it was (2, 5).
 - The introduction of log-BM seems very sudden, and they're only used for VGG-16? Did regular BM5 not work? I'm not sure what to take away from the comparison in Table 2.
 - Equation 6 was a bit confusing for me. It would be helpful to explain briefly how each term was derived.
 - Training in BM requires you to update the exponent biases in each step (?), which requires computing the dynamic range of each $N \times N$ block. I believe this is probably negligible, but it should be discussed as an additional overhead.

EDIT: the authors have clarified that the hardware area results take into account the need to support multiple formats, which addressed my biggest issue with the paper. I have raised my score to a 7 (accept).
review_title: Simple extension of block floating point, interesting contribution on the hardware aspects

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a family of numerical representations for training neural networks based on minifloats that share a common exponent across blocks. The authors perform a lot of software simulations to explore the design space on many different models and tasks. Hardware designs have also been synthesized and reported.

Pros:
-	The proposed representation is very general and covers a lot of existing designs, while also allowing for new ones. 
-	An exhaustive exploration of the design space, and the discovery of some new low-precision representations that offer high accuracy as well as computational density.
-	A lot of different models and datasets are examined. 

Cons
-	Some of the main contributions require a more careful, detailed explanation in order to fully appreciate and assess the contribution.
-	Results are presented in a confusing way.

Main areas for improvement:

One of the important contributions of the paper seems to be how the minifloat distribution is aligned with the value distribution and how $\beta$ is calculated to avoid underflow. However, this topic is only given a couple of sentences of explanation together with Figure 2, which does not provide much more information. I think the paper would be improved with a precise mathematical explanation of this re-alignment.

The block size for sharing exponents is determined semi-analytically by using Equation 6 to find a balance between the area cost and the dynamic range of the resulting numerical representation. However, this equation is introduced rather abruptly. The paper would be improved by providing more explanation of how Equation 6 was derived and some intuition into what the different terms mean. Furthermore, given the authors have implemented the block minifloat scheme in hardware, is it possible to show that Equation 6 actually matches what is seen when synthesizing the design? 
 
The tables in Section 4, and the corresponding text, need some work. In Table 2, the footnote (2) is defined but seems to be unused. In Table 2, the different columns (w, x, dw, dx, acc) are not defined anywhere and it is also not clear what the footnote (2) means in this context since it applies to the ‘acc’ column across all schemes. Additionally, in the corresponding text the authors discuss the performance of BM7, but this scheme is not found in the table.


Additional comments and questions:

-	In equation (1) the sign bit $s$ is defined before the equation, but exponent $E$ and mantissa $F$ are not defined until the sentence afterwards. 
-	In equation (2) it is not immediately clear what the index $i$ is meant to indicate. Furthermore, $X_i^a$ seems to indicate that the definition of $a$ depends on itself. 
-	Aside from using minifloats, how does this work compare to Flexpoint [1], which takes a similar approach to shared exponents?
-	In Figure 6 – are the accuracy measurements actually taken from the hardware simulation? 

[1] Köster, Urs, et al. "Flexpoint: An adaptive numerical format for efficient training of deep neural networks." Advances in neural information processing systems. 2017.
review_title: New numerical representations for training DNNs with very few bits, with impressive results. Some details need more explanation.

Meta-review: This paper proposes a new approach to training networks with low precision called Block Minifloat. The reviewers found the paper well written and found that the empirical results were sufficient. In particular, they found the hardware implementation was a strong contribution. Furthermore, the rebuttal properly addressed the comments of the reviewer.
Decision: Accept (Poster)