A Block Minifloat Representation for Training Deep Neural Networks
Authors: Sean Fox
Seyedramin Rasoulinezhad
Julian Faraone
david boland
Philip Leong
Venue: ICLR 2021 Poster (2021)

Abstract: Training Deep Neural Networks (DNN) with high efficiency can be difficult to achieve with native floating-point representations and commercially available hardware. Specialized arithmetic with custom acceleration offers perhaps the most promising alternative. Ongoing research is trending towards narrow floating-point representations, called minifloats, that pack more operations for a given silicon area and consume less power. In this paper, we introduce Block Minifloat (BM), a new spectrum of minifloat formats capable of training DNNs end-to-end with only 4-8 bit weight, activation and gradient tensors. While standard floating-point representations have two degrees of freedom, via the exponent and mantissa, BM exposes the exponent bias as an additional field for optimization. Crucially, this enables training with fewer exponent bits, yielding dense integer-like hardware for fused multiply-add (FMA) operations. For ResNet trained on ImageNet, 6-bit BM achieves almost no degradation in floating-point accuracy with FMA units that are $4.1\times(23.9\times)$ smaller and consume $2.3\times(16.1\times)$ less energy than FP8 (FP32). Furthermore, our 8-bit BM format matches floating-point accuracy while delivering a higher computational density and faster expected training times.

Published as a conference paper at ICLR 2021
A BLOCK MINIFLOAT REPRESENTATION FOR TRAIN- ING DEEP NEURAL NETWORKS
Sean Fox, Seyedramin Rasoulinezhad, Julian Faraone, David Boland & Philip Leong School of Electrical and Information Engineering
The University of Sydney
Sydney, NSW 2006, AUS (first). {last) @sydney edu.
ABSTRACT
Training Deep Neural Networks (DNN) with high efficiency can be difficult to achieve with native floating-point representations and commercially available hardware. Specialized arithmetic with custom acceleration offers perhaps the most promising alternative. Ongoing research is trending towards narrow floating-point representations, called minifloats, that pack more operations for a given silicon area and consume less power. In this paper, introduce Block Minifloat (BM), a new spectrum of minifloat formats capable of training DNNs end-to-end with only 4-8 bit weight, activation and gradient tensors. While standard floating-point representations have two degrees of freedom, via the exponent and mantissa, BM exposes the exponent bias as an additional field for optimization. Crucially, this enables training with fewer exponent bits, yielding dense integer-like hardware for fused multiply-add (FMA) operations. For ResNet trained on ImageNet, 6-bit BM achieves almost no degradation in Moating-poin accuracy with FMA units that are 4.1 x (23.9x) smaller and consume 2.3 (16.1x) less energy than FP8 (FP32). Furthermore, our 8-bit BM format matches floating-point accuracy while delivering higher computational density and faster expected training times.
INTRODUCTION
The energy consumption and execution time associated with training Deep Neural Networks (DNNs) is directly related to the precision of the underlying numerical representation. Most commercial accelerators, such as NVIDIA Graphics Processing Units (GPUs), employ conventional floating-point representations due to their standard of use and wide dynamic range. However, double-precision (FP64) and tanlee-prcisson (FP32) formats have relatively high memory bandwidth requirements and incur significant hardware overhead for general matrix multiplication (GEMM). To reduce these costs and deliver training at increased speed and scale, representations have moved to 16-bit formats, with NVIDIA and Google providing FP16 [154--754) (019)) and Bfloat16 (Kalamkar et al.) 2019) respectively. With computational requirements for DNNs likely to increase, further performance gains are necessary in both datacenter and edge devices, where there are stricter physical constraints.
New number representations must be easy to use and lead to high accuracy results. Recent 8-bit floating-point representations have shown particular promise, achieving equivalent FP32 accuracy over different tasks and datasets WWang et al. 2018 Sun et al. 2019). We refer to such representations as minifloats in this paper. Minifloats are ideal candidates for optimization. By varying the number of exponent and mantissa bits, many formats can be explored for different trade-offs of dynamic range and precision. These include logarithmic and fixed point representations which provide substantial gains in speed and hardware density compared to their floating- point counterparts. For instance, 32-bit integer adders are approximately 10x smaller and 4x more energy efficient than comparative FP16 units (Dally 2015).. That said, fixed point representations still lack the dynamic range necessary to represent small gradients for backpropagation, and must be combined with other techniques for training convergence.
Block floating point (BFP) in YYngg et al. 2019; Drumond et al. 2018) share exponents across blocks of 8-bit integer numbers, and provide type of coarse-grained dynamic range for training. This

---

Published as a conference paper at ICLR 2021
approach will typically incur some aceuracy loss more challenging datasets, however all dot- products within the block can be computed with dense fixed point logic. In comparison, HFP8 (Sun) et al. 2019) minifloats require larger floating -ponnt units (expensive FP16 adders in particular) but have at least 5 exponent bits dedicated to each gradient and suffer zero degradation in training accuracy. It would seem that an ideal representation should bridge the gap between each of these approaches. Our work achieves this for 8-bit and sub 8-bit precision schemes, overcoming two key challenges in the process. These are listed below and discussed with related works.
1.1 CHALLENGES AND RELATED WORK
While several works have demonstrated training with fewer than 8 bits of precision, they typically lead to loss of accuracy more complex problems and have performance bottlenecks because parts of the algorithm are left in high precision (Hubara et al. 2017 Zhou al., 2016: Miyashita al., 2016). Therefore, training end-to-end with reduced precision representations that are persistent remains a key challenge. In this regard, 8 bit tensors with 16-bit updates can be trained effectively (Banner al., (2018). Data loss arises when formats do not have enough range to capture variations in tensor distributions during training. BFloat (Kalamkar et al. 2019) adds two extra exponent bits for custom 16-bit representation, and the Apex library used in (Micikevicius et al. 2017; Wang et al. 2018, /Snn et al., 2019) for scaling the loss function into a numerically representable range. Block floating point and other variants apply similar functionality for fixed point numbers. but at finer granularity. WAGE (Wu et al. 2018) uses layer-wise scaling factors, SWALP (Yang et al. 2019) shares exponents across feature maps and convolution channels, and HBFP Drumond et al.
Minimising data loss with fewer bits:
2018) does the same for dot-products, though their implementation
requires caching of intermedian aite activations in FP32 and wide weight storage for better accuracy. S2FP8 (Cambier et al. 2020) replaces loss-scaling in FP8 (Wang et al. (018) with squeeze and shift factors that center 8-bit minifloats over the mean exponent of the value distribution. Shift factors operate similarly to BFP shared exponents. whereas squeeze factors can divert precision away from high value regions leading to errors in dot-product calculations. We provide some empirical evidence of this effect in Section 4.5 Finally, HFPS (Sun al. 2019) defines two minifloat formats that are optimized for range and precision requirements of forward and backward paths separately. In this work, we seek minifloat formats that are also optimized for arithmetic density.
Increasing the performance density of floating-point Most DNN training frameworks are devel- oped with GEMM accumulation in FP32. The authors in (Wang et al. (0188) reduced the accumulation width to FP16 with chunk-based computations and stochastic rounding. However, training minifloats with even denser dot-products has not been demonstrated. For DNN inference, ELMA (Johnson 2018) and posit number systems (Gustafson & Yonemoto (017) describe arithmetic that accumulate minifloat-like numbers as integers. Such work is applicable when the number of exponent bits is small, however training under such regimes can lead to data loss due to limited dynamic range.
1.2 CONTRIBUTIONS
In this paper, we present the Block Minifloat (BM) representation which addresses both of the aforementioned challenges. BM is modification of block floating point that replaces the fixed point values with minifloats, whilst maintaining shared exponents across blocks of numbers. BM formats generalise far wider spectrum of reduced precision representations and produce better outcomes than previous 8-bit regimes. Specific contributions of our work include:
Block Minifloat (BM), more efficient alternative to INT8 and FP8 for end-to-end DNN training. Shared exponent biases provide dynamic range and accuracy, while small exponent encodings provide fine-grained dynamic range and reduce the hardware cost of GEMM accumulation.
A new 8-bit floating foating-point format that uses no more than 4 exponent bits, achieving equiva- lent accuracy to floating- pointt with denser hardware via efficient Kulisch accumulation.
An exploration of the BM design space showing high accuracy DNN training with sub 8-bit representations for all weights, activations and gradients. This includes two techniques for minimising data loss of a practical implementation. namely gradual underflow and cost-aware block designs.
2

---

Published as a conference paper at ICLR 2021
Σmp. bias (B)
[x] minifloat (3,2)
value distribution
Align with max.
exponent
log, |al
(a) Minifloat
(b) BM (Shared exp. hias)
Figure 2: Exponent bias shifts the minifloat distribution align with the maximum expo- nent of the value distribution
Figure I: Minifloat and Block Minifloat (BM) ten-
representations
2 BLOCK MINIFLOAT REPRESENTATION
2.1 MINIFLOAT NUMBER FORMAT
Equation [I) computes the real value of minifloat number, where (e, m) denote the number of exponent and mantissa bits in the representation.
X(e, m) fE [E=1 =
(-1)* 21-8 (-1)* x 2
x F0+Fx22*
(denormal) (normal)
otherwise,
x1 F++Fx27
The decimal expansions of the exponent and mantissa are both unsigned integers, given by and F respectively, refers the sign bit and 8 2 is the exponent bias for the binary- offset encoding scheme. This consistent with IEEE-754 floating-point standards, except that our minifloats are considerably smaller (4-8 bits only), generalise to multiple (e, m) configurations, and do not handle nan/infinity bit patterns. Instead, arithmetic is allowed to saturate at the limits of the representable range [X X X For example, minifloat representation with X (4,3) have exponent and mantissas that range between [0, 15) and [0, 7] respectively. Therefore, the largest mar = 480 and X mán = 2- This corresponds dynamic range of 108 dB shown in Table 7 Appendix A.I
(e)
normal and smallest denormal positive numbers are X+
2.2 SHARED EXPONENT BIAS
The main difference between minifloat and block minifloat (BM) representations are highlighted in Figure [1 Minifloats have one exponent per element, but that exponent must be wide enough to tolerate changes in DNN tensor distributions during training (i.e. 5 bits for gradients in FP8 (Wang ct al. 2018)). In contrast, BM share exponent biases across blocks N minifloat numbers. This provides equivalent dynamic range with respect the block, but with fewer exponent bits than the original minifloat format. Block floating point (BFP) operates similarly, but all numbers within the block integers (Drumond al. 2018). BM can generalise for this case, i.c. when e 0.
The real value of the ith element from BM tensor a is given in Equation ( where X is an unbiased minifloat tensor, represented by (e.m) exponent and mantissa bits, and 3a is the shared exponent bias.
(e, m) x 2-8a
(2)
In this example, a, can only be represented accurately when the shared exponent bias 8n (calculated for the entire tensor) and the distribution of X jointly captures the value distribution of For example, large and small values in could saturate or be lost altogether if 3, is too large or too small. However, some leeway exists when exponents are shared across dot products. This is because dot products are reduce operations, meaning their sum is dominated by the largest values in the inputs. For this reason, we calculate 3, to specifically guard against overflow, and unlike (Cambier et al.) 2020) we don't "on't apply any scaling which could divert precision away from larger value regions. Our method of updating during training illustrated in Figure 2 and formalized in Equation below.
9, = max (llog, [a[]) - (2 -1)
(3)
The first term denotes the maximum exponent for the tensor a, which changes and must be updated during training, while the second term fixed and refers the maximum exponent of X.
3

---

Published as a conference paper at ICLR 2021
In terms of hardware, shared biases ensure that all dot products can be computed with denser minifloat arithmetic. This is shown in Equation 4 for BM tensors a and b, each with N elements.
a.b Σ (xf x (X) x 2 2- = 2- x (x" x*)
(4)
The dot product, X have minifloat formats with smaller exponents, while the cost of calculating, storing and aligning the exponent biases during training amortized over the length of the dot-product. Next we show how minifloat formats with fewer exponent bits lead to faster and more compact hardware.
2.3 KULISCH ACCUMULATION
Table Kulisch accumulator examples
Kulisch accumulator (Kulisch & Miranker 2014) a fixed point accumulator that is wide enough compute an error free sum of scalar floating-pone point products, over the entire range of possible values. Kulisch accumula- tors operate by shifting the mantissa of the floating-pon maiing product into internal register according nent of the product. The sum proceeds as integer addition which is 4- 10x more efficient in terms of area and power compared to FP16 (Dally 2013). The number of bits re- quired for the internal register (i.e. the addend) and shifter, scale the size and complexity of the accumulator and are provided as formulas in Equation (5) for BM operands
Format
Kulisch Acc. kadd
(Ca, ma Ma)(es, ms)
kshift
(8,23)/C (8,)/((88))) 23)
561
512
the expo-
(5.2)/(6,1)
102
96
(4,3)/(5,2)
56
48
(3, (3,4)/44,3)
34
24
(2,3)((3,2)
20
12
INT8
32
(Ca, ma) and b = (es, m²).
kadd = 1 + (2 + ma + 1) + (2 + my +
(5)
kshift 2 + 2
In the above equations, kadd calculates the number of bits required for the largest product of two numbers, plus one extra bit for the addition, and kshift determines the maximum number of bits the mantissa product must be shifted to align with the addend. Crucially, by considering the size of kodd and kshift, BM formats can be designed to trade-off fine fine-grained dynamic range (i.e. exponent bits) for more precision and smaller hardware. In fact, formats with exponents up to bits may yield kadd of approximately the same size as INT8/INT32 arithmetic units, while kadd becomes prohibitively wider and more expensive for larger exponents. This clearly shown via example in Table [ above, but more importantly, it is supported by hardware synthesis results given in Section S and Appendix[A.4 For example, an 8-bit minifloat format having exponent bits achieves a 1.6x area reduction compared to HFP8 (Sun et al., 2019) with 5 exponent bits. Furthermore, through an extensive set of experiments we discover that such representations can also achieve high training accuracy which forms key contribution of our work.
TRAINING WITH BLOCK MINIFLOAT
3.1 MINIMIZING DATA Loss
BM arithmetic will incur data loss when the value distribution is too wide or requires more precision than can be captured by the underlying minifloat representation within a block. Below, we describe steps taken to mitigate this problem without substantially increasing implementation overheads.
Our minifloats support denormal numbers as defined in Equation (1. Denormal numbers have precision close to zero, and ensure that consecutively smaller quantized numbers approach zero gradually. The alternative is flush-to-zero which discards the mantissa bits when = This equates to approximately 12.5% of the exponent encoding when e = 3; this is highly inefficient. Overhead for denormal numbers in hardware is minimal, and only requires detection of E = 0 and a single bit flip in the multiplier. Our experiments show that gradual underflow is crucial for BM formats with less than four exponent bits.
Gradual underflow:
4

---

Published as a conference paper at ICLR 2021
BM
BM+Kul.
FP32
BM
BM+Kul.
FP32
BM
BM
BM+Kul
FP32
BM
X
-/+
H-
-/+
-/+
W
GEMM
1=1
GEMM
e,
GEMM
W
RF
W
RF
X
VW
W
(a) Fwd activation
(b) Bwd activation grad.
(c) Bwd weight grad. and update
Figure 3: End-to-end Training with Block Minifloat (BM). All off-chip memory transfers are low precision BM tensors. BM alignments, weight updates, quantization, batchnormalization and ReLU are executed in on-chip scalar FP32 units. The register file (RF) stores a block of V W..
Matrix multiplication with BM is computed by dividing tensors into N x N blocks that bound the number of exponent biases and reduce data loss, since each block shares one exponent bias. Square blocks are chosen so that biases are contiguous memory regardless of whether the block is operating in the forward path or after transposition in the backward path. As such, BM can be stored with a persistent data structure, that doesn't 't require recasting or extra memory transfers during training. This makes BM easy to use at the software level, but does mean that biases shared across independent dot products. In terms of hardware cost, Equation 7 formalizes the relationship between the size of N and three overheads, where refer to relative area costs for each overhead.
Block Size:
Kulisch
FP
Memory
logg N
8+
cost = an
02
03
kadd
N
N2
For the first term, the width of the Kulisch accumulator must increase by log, N bits prevent overflow in the BM dot-products. In the second term, floating-pon hardware (including quantization and conversion modules) are required to accumulate, align and convert BM partial results, but the cost is amortized over N fixed point operations. Finally, in the last term, additional memory is required to compute and store one 8-bit bias every N x N values. For large block sizes, the extra silicon area from Equation i negligible compared to the GEMM but data loss from sharing biases still be significant. In Section [.5] we used Equation [ and determined that a block size of N = 48 offers a good balance for both objectives and is used for the rest of this paper.
Hybrid representation: Different minifloat representations for forward and backward paths have been shown to produce better accuracy for FP8 training (Sun et al., 2019). We apply the same idea, and find the best balance of precision and range for both paths separately. Full details for all BM formats are provided in Table 7 (in Appendix A.I where forward and backward configurations are given by (e,, r (es, my notation. Our formats cover each precision level between 4 and 8 bits, and are denoted by BM4, BM5, BM6, BM7 and B.8 in our experiments. For example, B.M (2, (3, 2) refers to 6-bit BM training with weight and activation tensors represented by (2, 3) and activation gradient tensors represented by (3, 2) minifloat formats.
3.2 TRAINING DETAILS AND GPU SIMULATION
BM offers an alternative to standard FP32 for the computationally intensive parts of training, which is typically mapped to general matrix multiplication (GEMM). However, specialised hardware is required realise its potential gains in speed and energy efficiency. For the purposes of this paper, we simulate the behaviour of B.M hardware using GPUs and Py PyTorch[] Given that dot products are computed exactly via Kulisch accumulators, existing CUDA libraries for GEMM can be used without modification, and all data loss is attributed to quantization only. Figure provides an illustration of the setup for each GEMM in forward and backward paths. In a practical implementation, BM does not require any costly movement or storage of high precision tensors. This is enabled by scalar processors after the GEMM (for FP32 operations, Kulisch doating-poinn conversion, block minifloat alignments, quantization etc.) and a weight update scheme that can compute and cache high precision gradients on-chip (Sun et al. 2019). Weight, activation and gradient tensors are quantized to BM numbers with stochastic rounding described in (Wang al.) 2018). For the software simulation, quantization is applied before each GEMM in forward and backward paths and contributes significant performance overhead compared to standard PyTorch layers. An approximate
Our implementation is available http://githuh.com/foov 1
5

---

Published as a conference paper at ICLR 2021
5x slow-down is realised most networks and datasets, with support for denormal numbers the main implementation bottleneck. The realisation of the same function comparatively cheap in custom hardware however, and can be fully pipelined for fast training times.
EXPERIMENTS
We evaluated the training accuracy of BM on subset of image, language and object detection modelling tasks. The entire spectrum of representations were explored on ImageNet (Deng et al. 2009) and CIFAR (Krizhevsky al., 2009) image recognition benchmarks, with results compared against well-calibrated INT8, FP8 and FP32 baselines. On other tasks, BM8 is compared with an FP32 baseline.
Table 3: Training Accuracy (%) on CIFAR-10 for VGG16 and log quantization
Table 2: Final Validation Accuracy (%) CIFAR datasets for R8sNet-18
kshift (bits)
Scheme
CIFAR-10
CIFAR-IO0
CIFAR-10
FP32
Log
FP32 (ours)
94.9
77.5
log-5b'
94.1
93.8
-0.3
32
BM6 (2,3)((322)
95.1
77.2
log-BM5 (ours)
93.8
93.4
-0.4
32
BM5 (2,2)(3.1)
94.7
76.1
log-BM4 (ours)
93.8
93.1
-0.7
BM4 (2.1)/(3,0)
94.2
73.7
(Miyashita al.
2016)
2
results achieved with base v/
4.1 CIFAR-10 AND CIFAR-100
We ran CIFAR experiments using SGD with momentum of 0.9 for 200 epochs in batches of 128 images and initial learning rate of 0.1 which is decayed by a factor of at the 60th, 120th and 160th epochs. Table&preeeet results for training ResNet-18 (He et al. 2016) with only small BM6, BM5 and BM4 representations. These offer the highest reduction in memory usage while still reaching very close the FP32 baseline. For example, 6-bit BM training only records 0.3% loss in accuracy compared to FP32 on CIFAR- 100 while theoretically saving 25% of memory read and write overheads compared FP8. We also tested logarithmic BM formats CIFAR-10 10 and VGG16 network. Log representations arise when m = 0, and require only adds and shifts for multiply-add arithmetic. Our results use the same training parameters as before and are shown in Table 3 We compare against the only previously known result for log training, i.c. log-5b (Miyashita et al. 2016) and achieve similar results with respect FP32 for 5-bit and 4-bit. BM representations have exponent biases that shifts tensor distributions dynamically during training, whereas log-5b define offset parameters at each layer that are fixed. Allowing biases to vary during training gives BM an advantage, and results in similar validation accuracy with only 4-bit words. This corresponds to approximately half the cost for multiplication in the linear domain (by exponent add and Kulisch shift).
Table 4: Top-1 accuracy (%) of reduced precision (RP) training on ImageNet for ResNet- 18 models
Numerical representation (c, m)
ResNet- 18
Scheme
W
X dw
dx
acc
FP32
RP
SWALP Yang al.
2019)
81
81
g1
81
321
70.3
65.8
S2FP8 Cambier et al. 2020)
(5,
2)/()))()) 23)
(5,
(5, 2)
(5, 2)
(8, 23)
70.3
69.6
HFP8
Sun al. 2019)
(4,3)
(4,3)
(5, 10)
(5,2)
(5, 10)
69.4
69.4
BM8 (2.3)
(2,5)
(2,5)
(6, ()
(4,3)
31
69.7
69.8
BM7 (2,4)
(2,4)
(2,4)
(6, ()
(4.2)
291
69.7
69.6
BM6 (2,3)/(3,2)
(2,3)
(2,3)
(6.9)
(3,2)
201
69.7
69.0
BM5 (2,2)(3311)
(2
(2,2)
(6,9)
(3,
181
69.7
66.8
Fixed point

---

Published as a conference paper at ICLR 2021
4.2 IMAGENET
The ImageNet dataset has 1000 class labels, and consists of 256x256 images split into training set with 1.28 million images and validation set with 50,000 images. We use ResNet-18 (He al. 2016) and AlexNet (Krizhevsky et al., 2012) architectures from the official PyTTrrhh implementation and train on one GPU with standard settings: SGD with momentum of 0.9, batches of 256 images, and an initial learning rate of 0.1 (0.01 for AlexNet) which is decayed by a factor of 10 at epoch 30 and 60. ResNet-18 18 has been widely tested upon previous work, and offers the most suitable benchmark for exploring the full spectrum of BM representations, especially given the size of the network as well as the cost of BM quantization on training times (approx. 5x slow-down). Results are presented in Table where columns w, dw, dx and acc refer to the numerical representation for weight, activation, weight gradient, activation gradient and on-chip GEMM accumulator. We achieve FP32 equivalent accuracy for BM8 and BM7, slight degradation for BM6, while our BM5 exceeds the reported accuracy for 8-bit SWALP Yang et al. 2019). Compared to S2FP8 (Cambier et al. 2020), BM6 representation reaches similar levels of relative accuracy, but with two fewer bits and without a high precision master copy of the weights. We provide some insights into possible reasons for this in Section 4.5 by considering the possibility of diminishing returns in accuracy from scaling minifloat representations. Compared with HFP8 (Sun et al. 2019), which offers robust 8-bit training results, BM8 produces the same accuracy on ImageNet while improving upon HFP8 in hardware density and performance. BM8 tensors can be represented with fewer exponent bits, and thus perform dot products via Kulisch accumulators that are smaller and faster than FP16 units. Furthermore, BM offers tradeoffs for even denser arithmetic and lower memory usage. In these regimes, BM hardware is more comparable to SWALP (Yang et al. 2019)) which performs the GEMM in fixed point. Proof of BM design efficiencies are provided with RTL synthesis results in Section 5 Figure 6
200
dation perplexdd
180
BMB (2.5)(4.3)
FP32
160
HFP8 (4.3)/(5,2)
Model (Dataset) [Metric]
FP32
BM8
140
BFPa (0,7)/(0,7)
AlexNet (ImageNet)
56.0
56.2
120
EfficientNet-b0 (small ImageNet)
62.6
61.8
100
LSTM (PTB)[Val ppl.]
84.7
87.33
Transformer-bes base (BLLU)
32.3
31.8
SSD-Lite (MbNetV2) (VOC)[MAP)
68.6 68.0
10
epochs
Table 5: Baseline FP32 v BM8 training on Image, Language and Object Detection models
Figure 4: Validation perplexity of LSTM model on Penn Treebank
4.3 LANGUAGE MODELLING WITH LSTM
We compared 8-bit formats for language modeling on the Penn Treebank dataset Marcus et al. (1993). We adapted the 2-layer Long Short Term Memory (LSTM) network from PyTorrh Examples and perform all GEMM operations with BM8 arithmetic. The batch size is 20, initial learning rate 20 with 0.25 decay, the embedding and hidden dimensions are 650 and sequence length is 35. Results in Figure 4 show BM8 with (2, 5)/(4, 3) hybrid configuration achieving better accuracy than BFP8 and HFP8 variants. The proposed BM8 representation has more fine-grained dynamic range and fewer mantissa bits than BFP8, and more precision and fewer exponent bits than HFP8 formats. This design point achieves better outcomes in terms of accuracy and hardware density than either representation separately (see Figure [) Section 5). Validation perplexity of 87.33 is also comparable to 84.70 obtained with full-precision floating-point.
4.4 ADDITIONAL EXPERIMENTS
To demonstrate wider applicability of the BM number representation, we tested BM8 on several additional networks and modelling tasks. Results are summarized in Table with full details of each
Immlemmentttion available at http://sh..aaamyyycmmmmmeeeeeene
Implementation available at http:///wcccooooooooommmmeteeeee mooel
7

---

Published as a conference paper at ICLR 2021
experiment provided in Appendix A.3 A.3 Crucially, every network tested achieved comparable accuracy with baseline FP32. This includes EfficientNet-bO0 (Tan & Le 2019) image classification and SSD- lite (Liu et al. 2016) with MobileNet-V V2 object detection models, both of which represent the type of network and application well suited to resource constrained hardware devices. Furthermore, we also trained small Transformer network for translation on the IWSLT German to English dataset (Cettolo et al. (2014). In future work, we plan to scale our implementation and demonstrate training with BM representations on larger networks and datasets. Network design with BM is another interesting research direction, since the majority of network architectures have been designed and optimized while assuming an FP32 arithmetic scheme.
1.0
40
2.4
area cost
RMSE (3,2)
HMSSE 1d00 pro-pptdductt
flow (3,2)
0.8
BM6 w/o
range
undde flow (85)
B6 w/ FP32
2.0
60
0.6
24
40
0.4
1.2
0.2
10
20
30
50 epochs
40
60
70
80
90
1.0
1.5
2.0
2.5
3.0
3.5
2a
2
2
bose
(a) Validation Accuracy:
Training
(b) HW (left axis) vs Range (right (c) Minifloat scaling by varying the
with denormal numbers on Ima- axis): Selecting the block size exponent base geNet
Figure 5: Experiments for minimising data loss with 6-bit Block Minifloat (BM6)
4.5 EMPIRICAL ANALYSIS
Effect of Denormal Numbers: To study the effect that denormal numbers have on training conver- gence in sub 8-bit networks, we trained ResNet-18 18 on ImageNet for BM6 with denormals (ours) and without denormals, using QPyTorch library (Zhang al. 2019). Results are plotted against floating-poii accuracy in Figure [a Without denormals, small numbers are flushed-to-zere and training stagnates immediately. Although not shown here, 8-bit representations with more than e bits do not suffer similar accuracy degradation without denormals. This investigation confirms the importance of denormal numbers for training BM formats with fewer exponent bits, and differentiates our software and hardware experiments substantially from previous 8-bit regimes.
Selecting the Block Size: We conducted experiments on CIFAR 100 to determine suitable block sizes those which simultancously increase dynamic range and have low hardware overhead. Results are shown in Figure 5bb
different block settings, We took the average of the largest range observed in gradient tensors at the entire duration of training. Estimates of area come from Equation 7 with parameters; a1 = 1, 02 = 10 (relative area of fixed point and floating-pon poinggg--onnttto respectively), kadd 21 and ag O. We saturate the area cost at N 256, which is consistent with the length of dot-products supported by the GEMM architecture in TPU hardware (Jouppi et al. (2017). Finally, N 48 emerged as good selection, corresponding to one floating-point unit every 48 multplyy-ccemmultee operations and one 8-bit exponent bias every 2304 minifloat numbers.
Scaling the Minifloat Representation: In Figure 2 which was discussed previously, minifloats have exponent biases that shift the representation to align with the maximum of the underlying value distribution. Additionally, the minifloat representation could be scaled (or stretched) over a wider or narrower part of the value distribution. We investigate this effect by varying the base of the exponent, and inspecting the underflow and root mean square error (rmse) dot-products after quantization; results are shown in Figure 5c The tensor under test is gradient tensor with maximum exponent -17 and mean exponent of 21. Mean scaling was proposed in S2FP8 (Cambier et al. 2020) for 8-bit training and works by centering the minifloat over the mean of the exponent value distribution.
For the (3,2) format, mean scaling requires a base of 2.52, calculated as b = 2 -17421
This is akin to redirecting precision from high value regions into smaller underflow regions, the result of which observably leads to increased error in the tested 6-bit regime. Better approaches could be designed to detect underflow and use higher precision arithmetic where necessary.
8

---

Published as a conference paper at ICLR 2021
Area (um²)
Power
Component
(μW)
70
FP32
HFP8
BM8
FP32
4782
10051
FP8 (w/ FP16 add)
829
1429
FP16
FP8
BM6
69
INT8 (w/ INT32 add)
417
1269
Top- Accuracy
BM8
391
1141
68
BM6
200
624
BM5
INT8 (4x4 systolic)
7005
20253
67
FP8 (4x4 systolic)
18201
56202
BM8 (4x4 systolic)
6976
18765
INT8 SWALP
Table Logic area and power single- cycle fused multiply-Add (FMA) and 4x4 array multipliers. Synthesized at 750 MHz with Cadence RTL Compiler 14.11 and 28nm cell library
0.000
0.001 0.002
0.003
0.004
0.005
0.006
Computational Density (1/um²)
Figure 6: Computational density ResNet- 18 ac- curacy on ImageNet
HARDWARE EVALUATION
In this section, evaluate the proposed block minifloat representation in hardware and compare against competitive integer and floating-poin arithmetic. Figure summarizes our results with plot of computational density (measured operations per unit silicon area) and ResNet-18 training accuracy on ImageNet. Computational density was obtained from an RTL design of single-cycle fused multiply-add (FMA) units and 4x4 systolic array multipliers. We performed synthesis at 750MHz for 28nm silicon technology and recorded area and power measurements for each number representation. Table 6 provides subset of these results, with coverage of all BM formats supplied in Appendix A.4
In summary, BM8 and BM6 arithmetic units 2.1 X (12.2x) and 4.1 x (23.9x) smaller and consume 1.25 x (8.8x) and 2.3 x x16.1x) less power than competitive FPS/(FP32) representations. Such arithmetic, which has similar hardware complexity to INT8, may especially useful in embedded applications where there are stricter area and power constraints but training still needs to achieve normal levels of accuracy and relatively high performance. With high computational density, BM arithmetic can achieve higher training throughput on compute intensive problems, while sub 8-bit BM formats have lower bandwidth requirements leading faster training times memory bound applications. Finally, overheads related to conversion from Kulisch floating-pol -pontt and BM quantization are expected to contribute little logic area relative to GEMM. This includes modules for leading-one one detection, barrel shifter, maximum exponent calculation, pre-quantization buffering and stochastic rounding, each of which have an efficient implementation. Further support of these claims and other system-level effects the subject of future work.
CONCLUSION
representation called Block Minifloat (BM) was presented for training DNNs effectively with reduced precision. Our representation allows the implicit exponent bias within IEEE-754 floating-point specifications to vary for block of numbers, and can be trained with high accuracy using narrow exponent encodings. We describe how few exponent bits lead to significantly smaller hardware, while smaller representations reduce memory bandwidth requirements, leading faster training than previous 8-bit approaches.

---

Published as a conference paper at ICLR 2021
REFERENCES
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in neural information processing systems, pp. 5145-5153, 2018.
Léopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oguz Elibol, and Hanlin Tang. Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks. arXiv preprint 1eXiv:2001006677 2020.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop Spoken Language Translation, Hanoi, Vietnam, volume 57, 2014.
William J Dally. High performance hardware for machine learning. (orfferaaeee/05////ooorlaaaessse http:///ww/../aaaaaaaaaa/////aaa Accessed: 2020-05-16.
cC/ 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Icee, 2009.
Mario Drumond, LIN Tao, Martin Jaggi, and Babak Falsafi. Training DNNs with hybrid block floating point. In Advances in Neural Information Processing Systems, pp. 453-463, 2018.
John L Gustafson and Isaac T Yonemoto. Beating floating point at its own game: Posit arithmetic. Supercomputing Frontiers and Innovations, 4(2):71 86, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1))6869-8888 2017.
IEEE-754. IEEE 754-2019, Standard for Floating- Poaating-Pointtaoe Arithmetic. June 2019. ISBN 1-5044-5925-3 (print), 1-5044-5924-5 (c-PDF). doi: https://doi.or org/
10. 109/EEEETT 876622.
Jeff Johnson.
Rethinking floating point for deep learning. arXiv preprint 1rXiv:1811.07777 2018.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1-12, 2017.
Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey, A study of BFLOATI6 for deep learning training. CoRR, abs/1905. 12322, 2019. URL
12322
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Ulrich W Kulisch and Willard I. Miranker. Computer arithmetic in theory and practice. Academic press, 2014.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.
10

---

Published as a conference paper at ICLR 2021
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):313-335 June 1993. ISSN 0891-2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint 11X0.v:700003440000 2017.
Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01023 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520. 2018.
Xiao Sun, Jungwook Choi, Chia-Yu Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan, Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. Advances in Neural Information Processing Systems, pp. 1901-4910, 2019.
Mingxing Tan and Quoc V Le. Efficientne:: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 2019.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yo Yu Chen, and Kailash Gopalakrishnan, Training deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems, pp. 7675-7684, 2018.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with with integers in deep neural networks. CORR, abs/ 802.04600. 2018.
64680
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Christo- pher De Sa. SWALP: Stochastic Weight Averaging in Low-Precision Training, arXiv preprint 1rXiv:994.11144 2019.
Tianyi Zhang, Zhiqiu Lin, Guandao Yang, and Christopher De Sa. Qpytorch: A low-precision arithmetic simulation framework. arXiv preprint arXiv:1910.04:3 2019.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arX1v:1600.00166, 2016

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 BLOCK MINIFLOAT NUMBER FORMATS
The full spectrum of Block Minifloat (BM) formats presented in Table 7 BM formats designed with consideration for the hardware cost of the Kulisch accumulator and also dynamic range and precision requirements.
Table 7: Comparison of Block Minifloat number formats
Format (e, m)
Range² (dB)
Precision (c)
Kulisch Ace.
Scheme
kadd
kshift
FP32
(8,23)
1668
2-24
561
512
Bíloat16 (Kalamkar et al.
2019
(8,7)
1529
2-8
529
512
FP16
(5,10)
241
2-11
87
64
FP8 Waag et al.
2018)
(5,2)
185
2-3
64
S2FP8 (Cambier ct al.
2020
(5,2)
185
2 2-3
71
64
HFP8 (Sun ct al.
2019
(4,3)(5,2)
108/185
2 4/2
56
48
SWALP (Yng el
2019
(0,7)
42.1
2 8
INT8 [Wu al.
(0,7)
42.1
2
32
BM8 (ours)
(2,5)(4.3)
48.0/108
2 2-1/2-1
20
BM7 (ours)
(2,4)(4422
41.9/101
2-5/2-3
29
20
BM6 (ours)
(2,3)(3,2)
35.6/53.0
2-1/2-3
20
12
BM5 (ours)
(2,2)(3.11)
28.9/45.7
2-3/2
12
BM5-log (ours)
(4,0)(4.0)
28.9/45.7
2-3/2-2
33
32
BM4 (ours)
(2.1)(3.0)
21.6/42.1
2-3/2-1
16
12
BM4-log (ours)
(3,09(300)
28.9/45.7
2 2-3/2-2
16
hybrid formats, i.e. forward/backward
2 dynamic range in decibels 20 logıo (X (Xxiax/ /x+
min
3 relative round-off error, 2- 2-1
A.2 SOFTWARE IMPLEMENTATION DETAILS
Block Minifloat (BM) arithmetic requires custom hardware achieve gains in speed and energy efficiency. We use QPyTorch (Zhang et al. 2019), an open source framework for low-precision training, to simulate the behaviour of BM hardware with existing Py PyTorch and CUDA libraries. QPyTorch provides a simple interface for applying quantization in the forward path (weight and activation tensors) and backward path (error, gradient and momentum tensors), ensuring that all numbers have low-precision representation, while the actual GEMM and AXPY operations are computed
floating point (FP32). This last point means that QPyTorch can not ordinarily be used to research low-precision accumulation strategies. Our work is different. Kulisch accumulators (as described Section 2 compute exact dot-products, and therefore our GEMMs udequately approximated by FP32 arithmetic (which is close to exact). QPyTorch is avalaible online and supports floating- point (without denormals), fixed point and block floating point number formats. Our code implementation is extension of this package.
"Imllementation available at http::/ggihaab.co///iig/////// Available https:/gituub.com/stee h/tplock_hhiihllllko
12

---

Published as a conference paper at ICLR 2021
A.3
MODEL DETAILS AND ADDITIONAL EXPERIMENTS
A.3.1
BLOCK MINIFLOAT CONVERGENCE CURVES ON IMAGENET
FP32
BM5
FP32
BM5
80
BM8 BM4
BMB
BM4
4.5
log5
70
log5
4.0
BM6
BM6
3.5
3.0
50
40-
1.0
10
20
50
60
70
80
90
200
10
20
30
50
60
70
epochs
spocts
Figure 7: Train loss and top-1 validation accuracy for the full spectrum of Block Minifloat formats trained on ImageNet using a ResNet-18 model
A.3.2 COMPARISON WITH BLOCK FLOATING POINT (BFP) ON IMAGENET
As described in Section 1, block minifloats bridge the gap between narrow floating-point and block floating point (BFP) representations. The main idea that better outcomes in terms of accuracy and hardware efficiency can be achieved by exploring the spectrum between the two representations. While BFP ensures that the majority of computation involves dense integer arithmetic, the lack of fine-gres -ne-grained dynamic range typically leads to accuracy loss on larger models and more complex datasets. In (below), we show that BM recovers accuracy loss for 6-bit and 8-bit formats on ImageNet while maintaining the same advantages in hardware.
Table 8: Comparison of Block Minifloat (BM) and Block Floating Point (BFP) number formats trained on ImageNet with ResNet-18 model.
BFP (ours)
BM (ours)
Scheme
6-bit 8-bit
67.0
69.0
+2.0
69.2
69.8
+0.6
A.3.3 TRANSFORMER MODEL (WMT)
We trained the Transformer Base model from the FairSeq repository on the IWSLT' German to English translation task. We used Adam optimizer and modified the FairSeq implementation with BM8 quantization. We used default training parameters found in the repository and trained for 25 epochs. BLEU scores were calculated using the script from the repository and show similar convergence between BM8 and FP32 models.
Immleementation available http://githhh.ccm//yyrrrffffrgge
13

---

Published as a conference paper at ICLR 2021
35
30
25
15
10
FP32
BM8
5
10
15
20
epochs
Figure 8: Training convergence curves for Transformer on IWSLT' 14 DR-En dataset
A.3.4 SSD-LITE (MOBILENET-VE V2) (VOC)
We adapted a PyTorch implementation of SSD-lite from online repository The base network MobileNet-V2 (Sandler et al. 2018) which was pretrained on ImageNet. The enitre network is trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset. We apply BM8 quantization all weights, activations and gradients before GEMM computations in the forward and backward paths. The network was trained with default parameter settings provided the repository as follows: SGD with momentum 0.9, weight decay factor 0.0005, batches of 32 images, and cosine annealing (Imaz 200) with an initial learning rate of 0.01. After 200 epochs, BM8 achieves a mAP of 68.0 which is sufficiently close to the reported accuracy 68.6.
A.3.5 EFFICIENTNET-B6 (IMAGENET)
We adapted a PyTorch implementation of EfficientNet (Tan & Le 2019) from an online repository We trained the smallest EfficientNet-b0 network on a reduced sized ImageNet dataset, where the images are resized from 256x256 to 128x 128. This choice was made to accelerate the training time, which is slowed down by 5x with our BM8 quantization function. The network trained on one GPU for only 60 epochs using batch size 256 and initial learning rate of 0. which is decayed exponentially with gamma of 0.90387. Figure 9 shows convergence of BM8 with an FP32 baseline.
100
80
val. accuracy
60
40
20
FP32
BM8
0
10
20
30
40
50
60
epochs
Figure Training convergence curves for EfficientNet--OO ImageNet
A.4 HARDWARE SYNTHESIS
Fused multply-ade (FMA) units were designed in RTL for floating pooating-poitt and block minifloat rep- resentations. We modified code from the Deepfloa repository for FP32, FP16 and FPS units. The BM units with Kulisch accumulation were hand written in Verilog following the block design given
"Implementation available at http://itthl.cmmmfaaaaaaaayrrsss Implemennatiio available at http://iihccccmmmammiiimiiliiico Implementation available http://tthhccom/icccctteesseeeet
14

---

Published as a conference paper at ICLR 2021
All designs were synthesised 750Mhz using Cadence RTL compiler 14.11 and commercial 28nm standard cell library. Since GEMM hardware is typically designed from tiles of smaller computational units, also provide synthesis results for small 4x4 systolic array multipliers. Full coverage of our results are shown in Table and Table TO Table i also provided, and shows the component breakdown and sealing of Kulisch related costs in different 8-bit BM regimes. Given that BM relies on hybrid formats, the multiplier operands are both sized for the largest mantissa plus one bit for denormal support. Compared to (4,3) which is the format used in HFP8 (Sun et al. 2019). BM8 (2,5)(443)) is 1.6x smaller. This is because BM8 has narrower exponent encod- ings that reduces the width of the Kulisch accumulator. Finally, the overhead for converting from Kulisch to FP32 is relatively small. We synthesised this to only contribute 264umm in area for 32-bit accumulators, the cost of which is amortized over the length of the dot product.
in Figure 10
A
m
Implic biis
denorm
kahift
KULISCH ADDEND
kadd (+)
Figure 10: Block diagram of block minifloat multiply-add, A + C, where A and B are minifloats and C is an integer
Table 9: Synthesized logic area and power of single-cycle fused multiply-Add (FMA) at 750 MHz on 28mm chip.
Table 10: Synthesized logic area and power of 4x4 systolic array multipliers at 750 MHz on 28nm chip.
Component
Area um?
Power uW
FP32
4782
10051
FP16
1116
2120
Component
Area por²
Power μW
FP8 (w/ FP16 add)
829
1429
INT8 (w/ INT32 add)
417
1269
FP8 (w/ FP16 add)
18201
56202
BM8
391
1141
INT8 (w/ INT32 add)
7005
20253
BM7
280
840
BM8
6976
18765
BM6
200
624
BM6
4083
11959
BM5
171
546
BM5-log
231
801
BM4
115
361
BM4-log
120
426
15

---

Published as a conference paper at ICLR 2021
Table 11: Component breakdown and logic area for different 8-bit BM formats
Format
Details kadd
Area (pm²) kadd
(c, m)/(e, m)
multiply
kshift
comb."
kshift
total
(3,4)/(4,3)
(5b x 5b)
34
24
210
93
74
377
(2,5)/(4,3)
(6b x 66)
31
20
235
79
77
391
(3,4)/(5,2)
(5b 5b)
49
40
253
199
95
547
(4.3)/(5.2
(4b x 4b)
56
48
259
276
104
639
(5,2)/(5.2)
(3b x 36)
64
300
361
113
774
(0, (0,7)(0,,7)
(8b x 86)
32
NA
NA
NA
NA
418
combinational logic includes multiply component
16


Reviews:
[{"review_id": "pYHwUaesTTd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduced a new representation (Block Minifloat) for training DNNs with low precisions of 8-bit or less. This new representation combines FP8 formats and the shared exponent bias concept to cover the dynamic range of tensors needed for DNN training. Compared to other published FP8 format, this representation has smaller exponents, which allows to use a more efficient Kulisch accumulator. The representation has been verified on a spectrum of deep learning models and datasets.\n\nOverall, the paper is well written. The idea is clearly presented, and experiments are sound. Particularly, the hardware evaluation gives some impressive results. However, for the sake of clarity, I have some questions.\n\n1). The shared-exponent bias itself is not new and use it for FP8 training is also straightforward and has limited novelty. What interesting is that the author uses this method to push for smaller exponent bits which in turn allows a more efficient accumulator. However, as a key contribution of this work, the authors did not give enough information and details on why “exponents up to 4 bits offer distinct advantages” for Kulishch accumulation. Could the authors explain more on the accumulator and provide some evident why smaller exponent is critical?\n\n2). From emulation point of view, the authors use existing CUDA libraries for GEMM which basically uses standard FP32 floating point accumulation. When the authors use certain bit-width for Kulish accumulator elements (for e.g. Table 9 and Table 10), how do you know this accumulator setting won’t impact model convergence? \n\n3). Hardware overhead for denorm support: the exponent bias is only guard against overflow, so denorm numbers are used to cover the range of smaller numbers. The authors claim that the hardware overhead is minimum since only input multiplicands are needed for the detection of denorm. However, there should be additional complexity than that. For example, how to handle denorm numbers in addend and how to tell the result produced is norm or denorm? Could the authors describe the hardware that needed to convert numbers from an intermediate format or a floating point format to their BM format?\n\n4). For experiment, it would have been good to include some larger networks, such as ResNet50 on ImageNet to compared to SOTA.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "g4Jir8CZZ6a", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposes block-minifloat (BM), a floating-point format for DNN training. BM is a fairly simple extension to block floating-point (BFP), which was proposed in (Drumond 2018 and Yang 2019). In BFP, a block of integer mantissas share a single exponent. In BM, a block of narrow floats share a single exponent bias. The shared exponent bias helps to shorten the exponent field on each individual float element. This is a good contribution, though a bit trivial.\n\nWhere the paper make a strong contribution is in the hardware implementation of BM, something which neither Drumond or Yang really got into. The authors propose to use a Kulisch accumulator for minifloat dot products, which basically works by converting the floats to integers with a shift, and accumulating with a very wide register. Kulisch accumulators are normally far too wide to be practical (see Eq. 4), but they've been proposed for posit computation (https://engineering.fb.com/2018/11/08/ai-research/floating-point-math/). This seem like a great idea here since BM can reduce the exponent length to only 2 or 3 bits.\n\nThe authors also did a good job evaluating the area and power of the BM hardware circuit. They built a 4x4 systolic array multiplier using BM units in RTL, and synthesized to place and route. The results show that BM6 can be 4x smaller and use 2.3x less power than FP8 while achieving to comparable accuracy. This is a pretty impressive result, and the hardware evaluation methodology is more stringent than most quantization papers at NeurIPS/ICML/ICLR. The only **minor** issue I have here is the area/power numbers are reported for BM8/BM6, but the exact config is not specified. E.g. is BM8 referring to (2e, 5m)?\n\nThe accuracy comparison is pretty standard, with CIFAR and ImageNet results using mostly ResNet-18. The authors' simulation framework slows training by 5x, so this is as much as I would expect. One **major** issue is that Tables 1 and 3 shows that for training to succeed, the forward and backwards BM formats must be different. Table 3 has three separate BM formats for each row. Implementing them all in hardware could incur significant overhead, which the paper doesn't discuss. The authors mention that the HFP8 paper does the same - but that paper defends this practice by showing that their two formats (which only differ by 1 e/m bit) can be supported by a single FP unit with minimal overhead. This paper uses (2e,5m), (4e,3m) and (6e,9m) in the same experiment labeled \"BM8\", which seems both misleading and unjustified. Note that SWALP and S2FP8 (and bfloat16/float16 papers) would use the same format in forwards and backwards pass and avoid this overhead.\n\nA few other insights: (1) subnormal floats are important and can't just be flushed to zero; (2) a square BM block size of 48x48 seems to work fine.\n\nMinor issues:\n - The methodology for hardware area seem solid (Appendix 4), but there isn't much detail on power. Was power obtained through modeling or using an RTL simulator? What kind of test vectors were used?\n - The area/power numbers are given for \"BM8\", but what's the precise format? I assumed it was (2, 5).\n - The introduction of log-BM seems very sudden, and they're only used for VGG-16? Did regular BM5 not work? I'm not sure what to take away from the comparison in Table 2.\n - Equation 6 was a bit confusing for me. It would be helpful to explain briefly how each term was derived.\n - Training in BM requires you to update the exponent biases in each step (?), which requires computing the dynamic range of each $N \\times N$ block. I believe this is probably negligible, but it should be discussed as an additional overhead.\n\nEDIT: the authors have clarified that the hardware area results take into account the need to support multiple formats, which addressed my biggest issue with the paper. I have raised my score to a 7 (accept).", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wgf1zfl6SqZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a family of numerical representations for training neural networks based on minifloats that share a common exponent across blocks. The authors perform a lot of software simulations to explore the design space on many different models and tasks. Hardware designs have also been synthesized and reported.\n\nPros:\n-\tThe proposed representation is very general and covers a lot of existing designs, while also allowing for new ones. \n-\tAn exhaustive exploration of the design space, and the discovery of some new low-precision representations that offer high accuracy as well as computational density.\n-\tA lot of different models and datasets are examined. \n\nCons\n-\tSome of the main contributions require a more careful, detailed explanation in order to fully appreciate and assess the contribution.\n-\tResults are presented in a confusing way.\n\nMain areas for improvement:\n\nOne of the important contributions of the paper seems to be how the minifloat distribution is aligned with the value distribution and how $\\beta$ is calculated to avoid underflow. However, this topic is only given a couple of sentences of explanation together with Figure 2, which does not provide much more information. I think the paper would be improved with a precise mathematical explanation of this re-alignment.\n\nThe block size for sharing exponents is determined semi-analytically by using Equation 6 to find a balance between the area cost and the dynamic range of the resulting numerical representation. However, this equation is introduced rather abruptly. The paper would be improved by providing more explanation of how Equation 6 was derived and some intuition into what the different terms mean. Furthermore, given the authors have implemented the block minifloat scheme in hardware, is it possible to show that Equation 6 actually matches what is seen when synthesizing the design? \n \nThe tables in Section 4, and the corresponding text, need some work. In Table 2, the footnote (2) is defined but seems to be unused. In Table 2, the different columns (w, x, dw, dx, acc) are not defined anywhere and it is also not clear what the footnote (2) means in this context since it applies to the ‘acc’ column across all schemes. Additionally, in the corresponding text the authors discuss the performance of BM7, but this scheme is not found in the table.\n\n\nAdditional comments and questions:\n\n-\tIn equation (1) the sign bit $s$ is defined before the equation, but exponent $E$ and mantissa $F$ are not defined until the sentence afterwards. \n-\tIn equation (2) it is not immediately clear what the index $i$ is meant to indicate. Furthermore, $X_i^a$ seems to indicate that the definition of $a$ depends on itself. \n-\tAside from using minifloats, how does this work compare to Flexpoint [1], which takes a similar approach to shared exponents?\n-\tIn Figure 6 – are the accuracy measurements actually taken from the hardware simulation? \n\n[1] Köster, Urs, et al. \"Flexpoint: An adaptive numerical format for efficient training of deep neural networks.\" Advances in neural information processing systems. 2017.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach to training networks with low precision called Block Minifloat. The reviewers found the paper well written and found that the empirical results were sufficient. In particular, they found the hardware implementation was a strong contribution. Furthermore, the rebuttal properly addressed the comments of the reviewer.
Decision: Accept (Poster)