Accuracy Boosters: Epoch-Driven Mixed-Mantissa Block Floating-Point for DNN Training
Authors: Simla Burcu Harma
Canberk Sönmez
Babak Falsafi
Martin Jaggi
Yunho Oh
Venue: Submitted to ICLR 2023 (2023)
Keywords: DNN training
low-precision training
mixed-precision training
efficient training
number formats
numerical encodings
block floating-point
DNN accelerators

Abstract: The unprecedented growth in DNN model complexity, size and the amount of training data have led to a commensurate increase in demand for computing and a search for minimal encoding. Recent research advocates Hybrid Block Floating-Point (HBFP) as a technique that minimizes silicon provisioning in accelerators by converting the majority of arithmetic operations in training to 8-bit fixed-point. In this paper, we perform a full-scale exploration of the HBFP design space including minimal mantissa encoding, varying block sizes, and mixed mantissa bit-width across layers and epochs. We propose Accuracy Boosters, an epoch-driven mixed-mantissa HBFP that uses 6-bit mantissa only in the last epoch and converts 99.7% of all arithmetic operations in training to 4-bit mantissas. Accuracy Boosters enable reducing silicon provisioning for an HBFP training accelerator by 16.98× as compared to FP32, while preserving or outperforming FP32 accuracy.

Under review as a conference paper at ICLR 2023
ACCURACY BOOSTERS: EPOCH-DRIVEN MIXED-
MANTISSA BLOCK FLOATING-POINT FOR DNN TRAINING
Anonymous authors Paper under double-bling review
ABSTRACT
The unprecedented growth in DNN model complexity, size and the amount of training data have led commensurate increase in demand for computing and a search for minimal encoding. Recent research advocates Hybrid Block Floating- Point (HBFP) as a technique that minimizes silicon provisioning accelerators by converting the majority of arithmetic operations in training to 8-bit fixed-point. In this paper, we perform a full-scale exploration of the HBFP design space includ- ing minimal mantissa encoding, varying block sizes, and mixed mantissa bit- witth across layers and epochs. We propose Accuracy Boosters, an epoch-driven mixed- mantissa HBFP that uses 6-bit mantissa only in the last epoch and converts 99.7% of all arithmetic operations in training to 4-bit mantissas. Accuracy Boosters en- able reducing silicon provisioning for HBFP training accelerator by 16.98x compared to FP32, while preserving or outperforning FP32 accuracy.
INTRODUCTION
Improvements in Deep Neural Network (DNN) algorithms over the past decade have led to an un- precedented growth in model complexity and dataset size and consequently the required computa- tional resources to train DNN models. One of the largest DNN models (GPT-3) (Brown et al.) 2020) has 175 billion parameters and requires 3.14 1.14x 20 FLOPs train. With the slowdown Moore's law, researchers and vendors have begun to search for ways to improve the arithmetic density of the underlying hardware platforms. Narrow bit-width (with lower precision) number formats (Wang & Kanwar 2019;; Micikevicius et al. 22018) Sun et al. [019 Mellempudi et al. [019 Sun et al. 20200 have emerged as a promising approach to increase arithmetic density, as well as, reduce the required operand storage and communication bandwidth while maintaining high accuracy for training.
Many have tried fixed-poing formats, often used in inference, to further reduce silicon logic complex- ity for arithmetic (Courbariaux et al.] 2013a; Hubara al.) 2016 [Rastegari et al.) 20166] Dettmers et al. 20222abb b). Fixed-point formats, unfortunately, dramatically suffer from a limited range in nu- merical representation especially for arithmetic in backward propagation. As such, researchers have tried mixed-ireeeeeee -ixeed-precisoon training to trade off accuracy for efficiency (Zhang et al. 202 Fu et al. 2020
2021
Recently there have been several proposals for block floating point (Köster et al. 2017 Das et al. 2018 Zhang al. 202T), a numerical encoding that groups block of mantissas which rely on only fixed-point arithmetic with single exponent. Block floating point asympototically approaches the arithmetic density fixed point with larger block sizes and naturally lends itself well to mixed- precision hardware where a block with the same number of exponent bits can have fixed-point datapath which bitsliced for various multiples of mantissa bit encodings (e.g., the same today's CPU cores implement SIMD). While block floating point has been promising in for inference (e.g.. Microsoft Floating Point (Darvish Rouhani et al. 20200), most proposals to train with block floating point have either failed reach its full potential by requiring small blocks and/or just fall short of reaching FP32 accuracy.
One specific proposal, Hybrid Block Floating Point (HBFP) (Drumond al. 2018), mixed- precision format where the dominant fraction of training which is the dot products happens in block floating point (e.g., convolutions, matrix multiplications, outer products), and higher precision (e.g..

---

Under review as a conference paper at ICLR 2023
FP32) is used for other less frequent operations requiring larger numerical ranges (e.g., activations, regularizations). HBFP simultaneously offers the high accuracy of floating point and the superior hardware density of fixed point, delivering up to 8.5 higher throughput than FPI6 with 2x more compact models (Drumond 2020). Prior work on HBFP only presented a preliminary analysis of the design space for power-of-two mantissa bit widths (e.g.. 2, 4, 8-bit mantissas).
In this paper, we make the observation that the parameter space for HBFP quite rich presenting several opportunities for further improving efficiency and density in hardware platforms. First, custom accelerators can support non-power-of-twe numerical formats; and minimizing the number of bits improves operand storage and communication linearly, and arithmetic logic quadratically. Second, there is an interplay between the block size and the number of mantissa bits, allowing for an overall denser numerical format with smaller blocks while maintaining high accuracy. Finally, HBFP allows for mixed-mantissa block floating point encodings. Prior work studies training with various HBFP formats isolation; however, the design space of mixed-mantissa HBFP yet to be explored.
To minimize the number of bits in HBFP, we explore the interplay between the block size and the number of mantíssa bits. We show that HBFP with six more mantissa bits has no sensitivity to block size. While HBFP with a smaller number of mantissa bits sensitive to block size, these configurations do not result in enough accuracy even with the smallest blocks and require additional methods to increase accuracy. Accuracy Boosters further minimizes the number of bits and enables training with 4-bit mantissas. Our method improves epoch-wise ducing high precision to the training process only at the last epoch. The main contributions of this paper are follows:
training by intro-
We show that HBFP6 is the smallest HBFP format achieving competitive accuracies.
We enable HBFP5 training for smaller models (e.g., ResNet20) by using small block sizes and for larger models (e.g., DenseNet40) by keeping the first and last layers in FP32.
We improve the silicon area- and meery--ffficiency training without significant accuracy loss by performing large fraction of epochs in low precision for CNN and Transformer models. We show that for a few models, our method even outperforms FP32 training.
2 WHY MINIMIZE HBFP?
HBFP is fixxeddprecision training technique that brings area- and energy-efficient fixed-point arith- metic into DNN training. DNN training traditionally uses floating- point operations because training algorithms require high precision. However, floating-poin hardware occupies larger silicon area and consumes more energy than the fixed-point arithmetic due to exponent management and normal- ization. HBFP alleviates the need for floating-poin operations by employing fixed-point arithmetic for dot products, the most common type of operations in DNN training. Furthermore, it assigns ponents to fixed-point tensors to emulate the dynamic range provided by floating -pontt to maintain the training accuracy. In other words, HBFP achieves fixed-point efficiency in DNN training with Moating-point accuracy.
We argue that employing both smaller mantissa bit widths and larger block sizes are the keys to improving HBFP hardware efficiency. Prior results show that, to the first order, minimizing the number of bits in fixed-point arithmetic reduces operand storage and memory bandwidth linearly and multiplication power and area quadratically (Gholami et al. 202T). As result, HBFP hardware efficiency increases with the reduced number of mantissa bits due the large fraction of fixed-point operations. We also note that the hardware area and energy expenditure of HBFP accelerators is determined by the number of mantissa bits and the block size because the overhead of the exponent bits is negligible. Therefore, we work with 10-bit exponents as in prior work (Drumond et al.) 2018), and explore the HBFP design space by varying the mantissa bit width and the block size. Our experiments with different mantissa bit widths give rise to a re-configurable DNN accelerator for mantissa HBFP. Furthermore, we demonstrate that using smaller block sizes reduces the number of fixed-point operations in dot products causing increase in the area overhead. In line with these observations, we establish decreasing the mantissa bit widths and increasing the block sizes as design guidelines for HBFP-hased accelerators.
2

---

Under review as a conference paper at ICLR 2023
To validate our guidelines, compare the area improvements of HBFP formats of different man- tissa bit widths and block sizes, by synthesizing the arithmetic units using the Synopsys Design Compiler and TSMC 28nm technology. We determine the costs of individual design components and combine them using the mathematical equations that govern the entire accelerator structure. For FP32 dot product units of size N, estimate the hardware cost the FP32 adders, N FP32 multipliers, one FP32 accumulator (adder), and one floating-poin activation unit. For HBFP dot product units, we estimate the hardware cost the sum of the of N fixed-point adders, N fixed-point multipliers, one FP32 accumulator (adder), one floating-pon point ac- tivation unit, and one adder for signed exponents. We also add the costs of conversions between FP32 and fixed-point numbers by synthesizing the converter blocks. In Figure the nomenclature HBFPm refers to an HBFP configuration with mantissa bit width m. We observe that reduction in the mantissa bits alone (when the block size is fixed to 256) results in 10x and 17x reduction in area and power using HBFP6 and HBFP4 respectively relative to FP32.
of the cost of N
Being numerical format originally proposed for DNN training, HBFP is also capable of inference, a task that inherently easier than training. observe that HBFP8 and HBFP6 with a block size of 576 suitable for inference for the most popular CNN models (i.e., ResNet, MobileNet, VGG, ShuffleNet) without any accuracy loss. Furthermore, MSFP, an HBFP-like numerical format that uses signed magnitude arithmetic, was shown to achieve superior inference performance out-of- the-box INT8 quantization (Darvish Rouhani et al. 2020). These findings suggest that HBFP is a versatile numerical format for both training and inference, paving the way unified accelerator capable of efficient, accurate training and high-perffomance inference.
40
Silicon Area Ratio (FP32/MBFPm)
30
25
20
fp32/hbfp2
fp32/hbfp3
fp32/hbfp4
10
fp32/hbfp5
fp32/hbfp6
5
fp32/hbfp7
fp32/hbfp8
100
200
300
400
500
600
Block Size
Figure I: The silicon area ratio between FP32 and HBFP with various block sizes. The dotted lines show the points when the design achieves 95% of the maximum hardware benefit.
The rest of this paper provides step-by-step explanation of how to minimize the number of bits in HBFP with the help of mixed-precision training while maintaining outperforming FP32 accuracy.
MINIMIZING HBFP
In this section, describe the steps to follow minimize HBFP, that is to bring smaller mantissa bit widths and larger block sizes into training. We investigate the interplay between the number of mantissa bits and the block size, and also the first and last layers of CNNs in FP32. We leverage and elaborate on the insight that not all epochs have the same effect on the DNN model accuracy, and introduce the Accuracy Boosters method.
existing technique recover the data loss, keeping
For the rest of this paper, we use the notation HBFP N) for an HBFP format with a block size of N. mantissa bit-width and 10-bit exponent. For a tensor encoded in this format, we have
3

---

Under review as a conference paper at ICLR 2023
mantissa tensor consisting of N-many elements and a shared exponent, both of which using two's complement arithmetic.
3.1 THE SMALLEST HBFP CONFIGURATION
Prior work has shown that HBFP(8, 576) is the smallest HBFP format with power-of-two mantissa bits that can train wide range of models like CNNs, LSTMs, and BERT with FP32 accuracy (Dru- mond] (020). They also show that HBFP8 shows almost no sensitivity to the block size, meaning that arbitrarily large block sizes yield similar accuracy, within 0.5% of the FP32 baseline. However, they did not explore the entire parameter space and focused only on power-of-two configurations (HBFP8 and HBFP4). While power-of-two-bit numbers align naturally with the memory struc- ture and encode matrices in tightly-packer way. non- non-power-of-twee bit mantissas can improve the arithmetic density even further, as studied by Darvish Rouhani et al. (2020).
Considering that non- pon--ower-of-wee format is feasible, we claim that HBFP6 achieves FP32 accu- racy providing around 1.5x silicon area gain over HBFP8. Among 16-bit encodings, FP16 without training does not reach FP32 accuracy (Micikevicius et al. 2018). Prior work shows that 11, which consists of 8 mantissa bits and exponent bits, is capable of training DNN mod- els with FP32 accuracy (Wang & Kanwar 2019). In the light of this, we hypothesize that HBFP6 with 6 mantissa bits and 10 exponent bits can also be used to train DNN models. Our experiments show that HBFP6 with block size of 1 is capable of training models with FP32 accuracy. As matter of fact, HBFP6 can train DNN models with block sizes up to 256.
3.2 FURTHER MINIMIZING HBFP
Block floating point shares a single exponent across a block of values using the exponent of the largest tensor value. Since block floating point format does not apply normalization (2exponent x O. O.mantissa instead of 2emponent x I.antissaa), the precision within block is highly dependent on the largest element in that block, which decides on the exponent value. The interval between two consecutive representable numbers is calculated as in Equation I As the number of elements sharing the same exponent (block size) increases, variance of the values in a block significantly increases, which leads to precision loss for the small elements in the block. As the number of mantissa bits decreases, the model's sensitivity to the block size increases because the interval increases leading to higher quantization errors. Larger mantissa bit widths make the distribution more resilient to the quantization error and to larger block sizes, as each tensor element can be represented more accurately.
2largest exponent
interval 2# of mantissa bits
As prior work (Drumond et al. 2018 Drumond 2020) only investigated power-of-twe mantissa bits and focused mostly the design space of HBFP8, the interplay between the number of mantissa bits and the block size is left unexplored. We investigate the whole design space of HBFP varying both parameters, and claim that reducing the block size will enable reducing the number of mantissa bits, and thus improve hardware efficiency. As we have shown in Section more desirable for hardware efficiency. To minimize HBFP one step further, we investigate existing techniques to recover the data loss due to quantization.
2 larger block sizes are
For CNN models, prior work indicates that the first convolution layer and the last fully connected layer have larger impact on the final model accuracy and keeping these layers in FP32 allows for reducing precision the rest of the layers (Choi al. 2018 Zhou al., 2018 Mellempudi al. 2019 Wang et al. 2018). The first layer takes the input images and filters the images with several convolution kernels and returns feature maps. Thus, is critical for the final model to keep the input information fully accurate and to preserve the data in the initial feature map. The last layer of a CNN returns a probability distribution over the possible outcomes for image classification which makes crucial to retain information better for this layer.
According to our calculations, the first and last layers of the CNN models account for a negligible amount of computation; and thus, keeping them in high precision during HBFP training does not result in a significant increase in the hardware area or energy consumption. For one of the smallest CNN models ResNet20, these layers take up only 1.08% of the total number of FLOPss in the model.
4

---

Under review as a conference paper at ICLR 2023
For larger models, the percentage is even lower (e.g.. 0.39% for ResNet50, 0.27% for ResNet74). Thus keeping these layers in FP32 helps further minimize HBFP while not introducing significant hardware overhead.
3.3 ACCURACY BOOSTERS
In this section, we show how we can leverage mixed-pprecisio training for minimizing HBFP with- out sacrificing aceuracy. Existing mixed-precision techniques include varying the precision between different layers, functions, and epochs. FracTrrain gradually increases the precision of weights, ac-
tivations, and gradients during training (Fu et al.) 2020). It also introduces an algorithm adapting the precision of activations and gradients in different layers depending on the input. Fu et al. (2021) argue that low precision due to quantization affects DNN training similarly to exploration with a high learning rate and introduce CPT, periodic precision schedule for low- low-precision DNN training. In both of these methods and in contrast to our work, the backward pass uses a higher precision than the forward pass (6 8 vs. the forward pass; therefore, CPT spends most of the time in 8- and 6-bit fixed-point computations. Moreover, the silicon area and energy gain that these fixed-point calculations provide over FP32 are 9 and 15x respectively. In comparison, HBFP6 and HBFP4 provide 10x and 17x area and power gain respectively as we have computed in Section 2
bits). The backward pass usually takes around 3x long as
We leverage the insight that each training epoch has a different effect on the final model's 's accuracy. Rahaman al.) (2019) and Xu al. (2019) show that DNNs first learn wow-freueeccy components, where the frequency is defined for the coordinates of the input space. Xu et al. (2019) also empiri- cally show that for CNN models, the high- high-frequency components have higher complexities and are learned in the last epochs. In light of these findings, we hypothesize that freuuerr functional components are more sensitive to quantization errors. Thus, higher precision is required for the last stage of DNN training, where the optimization occurs after appropriate amount of generalization in the network. After reaching a certain loss value in our 4-bit training, switching the tensors back to 6-bit mantissas enable the sensitive fine-tuning performed in the final epochs and help increase the accuracy even more.
For most DNN models, using HBFP6 for the last epoch is sufficient to boost the final model ac- curacy, while the rest of the epochs are trained using HBFP4. We estimate the gain in silicon provisioning and power consumption by using the average of the individual gains of the numerical formats weighted with the fraction of the training time the numerical format is used. When we consider the number of epochs that each hardware used for, the average area and power gain are calculated to be 1.5x more than HBFP6 (16.98x over FP32).
4 EXPERIMENTAL RESULTS
In this section, we will first describe our experimental setup, then we will explain and discuss our experimental results on the state-of oftte--f-the-an models and datasets for various DNN tasks. We trained ResNe/20/50/74 (He et al.] 2016), and DenseNet40 (Huang et al., 2016) on CIFARIO and CIFAR 100 (Krizhevsky Transformer- Base /Vaswani et al. 2017) the WMT16 16 English-German dataset for machine trans- lation. We use FP32 as baseline for both model accuracies and hardware comparisons. To evaluate the impact of method, we tune the models using FP32, and then train the same models from scratch with the same hyperparameters in HBFP. For the image classification experiments, we re- port Topl validation accuracies; for machine translation, we report BLBU scores.
Hinton 2009) datasets for image classification. We also trained a
4.1 MINIMIZING HBFP
Table 1 shows the Top1 validation accuracies for ResNer20. ResNet50, ResNet74, and DenseNet40 on CIFARIO 10 and CIFARIO0 datasets trained with various HBFP configurations. We observe that HBFP6 is the smallest HBFP configuration that gives accuracies within 2% of FP32 accuracy for block sizes up to 256. We also observe that unnecessarily large block sizes like 576 (providing only 0.8% silicon area gain compared to block size of 256) hurt accuracy for some models. Larger blocks will contain a larger variety of values in terms of magnitude (affected e.g. by outliers), so will result in larger approximation errors than smaller blocks and lower accuracy in training.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Top-1 validation accuracies of various CNN models for various HBFP configurations
Models and Datasets
CIFARIO
CIFAR100
Number Format
Block Size /
ResNer20
ResNet74
ResNet50
ResNet74
DenseNer40
Area Gain
FP32
91.72
93.57
74.11
74.55
72.42
HВFP8
576 7.23
91.52
93.36
73.76
74.32
73.73
16 8.85
91.12
93.38
73.11
73.51
72.08
25 9.29
91.09
92.54
72.59
73.20
71.77
36 9.71
91.29
92.61
72.45
72.87
71.83
HBFP6
49 / 10.00
91.33
92.93
72.34
72.40
71.87
64 / 10.19
91.12
92.93
72.12
72.40
71.81
256 / 10.71
91.38
92.79
72.01
72.53
71.50
576 10.81
90.65
92.19
70.80
72.51
71.02
16 10.02
90.18
92.24
67.15
70.52
69.36
25 10.95
90.21
92.04
67.85
69.03
70.00
36 11.55
90.12
91.99
67.12
69.91
69.01
HBFP5
49 / 11.94
89.68
91.96
68.72
68.67
69.33
64 12.21
89.72
91.32
66.81
69.29
68.85
256 12.92
89.13
91.13
63.01
66.57
67.84
576 13.07
88.91
90.98
64.61
64.41
67.57
16 /
90.80
92.56
71.23
72.31
70.90
25 /
90.62
91.79
71.76
72.58
70.52
HBFP5 with
36
90.62
92.73
71.45
72.26
70.01
first and last
49
90.49
92.02
70.51
72.07
70.16
layers in FP32
64
90.31
92.41
69.59
72.09
70.10
256
90.07
92.11
68.76
70.20
69.21
576
90.37
91.06
68.66
69.14
68.78
16 / 12.52
82.59
76.85
63.70
25 14.03
81.82
78.62
64.25
36 / 15.01
80.84
76.64
63.34
HBFP4
49 15.68
79.32
71.19
65.55
64 / 16.15
80.18
74.35
62.37
256 17.43
76.96
60.65
60.02
576 17.69
75.33
66.70
59.77
Total Number of FLOPs
41M
174M
119M
326M
542M
required to train the model
Table [ also shows that for the smallest model ResNet20, decreasing the block size of HBFP5 25 leads 1.5% accuracy degradation but 1.1x hardware gain compared to HBFP6 in return (see Figure D As the block size increases, the accuracy decreases more for HBFP5. Among larger models, ResNet74 trained CIFARIO gives similar results ResNet20 on CIFARIO (1.3% accuracy degradation with HBFP5). ResNet50 and ResNet74 on CIFAR 100 do not perform well with HBFP5 unlike smaller variations, and result in up 7% accuracy degradation. HBFP5 incurs
3 - 4% accuracy degradation average for moderate block sizes (e.g.. 49, 64). We show that almost all DNN models, keeping the first and last layers FP32 enables HBFP5 training. For ResNet20 and ResNet74 on CIFARIO, the accuracy drop is around 1% relative to FP32 accuracy for block size of 64 while bringing 1.22x silicon area and energy gain. ResNet50 and ResNet74 CIFAR 100 accuracies improve significantly compared naive HBFP5 for moderate block sizes (e.g., 49, 64), and the models get more sensitive as the block size increases to 256 and higher.
We also report HBFP4 accuracies to show the limitations of HBFP format. Even for the small models like ResNet20, with block size 16, the accuracy drops more then 9%. As the accuracy drop for ResNet50 and ResNet74 on CIFAR 100 is considerably high even with HBFP((5, 16),

---

Under review as a conference paper at ICLR 2023
did not train these models with HBFP4. We observe that for HBFP4, the sensitivity to the block size increases for all the models because the distortions in the tensor distributions increase. So, block sizes larger than 16 incur even more dramatic accuracy drops like 26.87% for ResNet74, showing the significance of only one bit of mantissa for the accuracy.
4.2 ACCURACY BOOSTERS
Considering the observed accuracy values in Table and our hardware model explained in Section a block size of 49 is within 95% of the maximum hardware gain (the point where the HBFP6 line in Figure I levels off) while achieving accuracies with less than 1% degradation. Thus, we choose block size of 49 as the sweet spot and perform Accuracy Boosters using this block size. We perform the last epoch of the training in HBFP(6, 49) and the rest in (BFP 44) for all the experimental settings. We also trained by keeping the last 10 epochs in HBFP(6, 49) to observe the improvement in accuracy for the CNN models. We can see that for most of the CNN models, our proposed technique outperforms FP32. When we keep the last 10 epochs in HBFP6, we observe that the accuracies slightly increase. We also trained with block size of 256 by using Accuracy Boosters and observe that the accuracy loss is less than 1% for most of the models and DenseNet40 still outperforms FP32 accuracy(see Table
Table 2: Top-1 validation accuracies of various CNN models for Accuracy Boosters
Models and Datasets
CIFARIO
CIFARI00
Epochs using HBFP6
Block Size
ResNer20
ResNet74
ResNer50
ResNet74
DenseNet40
Only last
49
91.45
92.85
74.25
74.39
74.06
Only last
256
90.76
92.40
72.78
74.32
74.77
Last
49
91.69
92.87
74.50
74.45
74.26
FP32
91.72
93.57
74.11
74.55
72.42
For Transformer, Accuracy Boosters achieves BLEU Score of 25.08 when used only for the last epoch and 25.40 when used for the last 10 epochs, while the FP32 score is 26.09. We also observe that for Transformer, while HBFP6 outperforms FP32, HBFP4 does not incur high amounts of accu- racy loss but Accuracy Boosters still increase the accuracy to the level close to FP32 (see Table The total number of calculations required to train Transformer- Base is 3.3 18 FLOPs.
Table 3: BLEU Scores for Transformer Base trained on WMT16 English-Gerne dataset with various training techniques
Training Technique
FP32
HBFP 49)
HBFP(4, 49)
booster (last)
booster (last 5)
BLEU Score
26.09
26.16
24.73
25.08
25.40
The Accuracy Boosters method offers up to 16.98x hardware gain compared to FP32, using only 4 bits for 99.7% of the total training time, while having comparable better accuracy. Table shows the Topl validation accuracies for various models with Accuracy Boosters at last and 10 epochs. Models trained
CIFAR10 trained for 160 epochs, whereas for CIFARI00 the total number of epochs for all models is 300. The transformer is trained for 35 epochs and Accuracy Boosters are applied in the last and epochs. Figure 2 illustratee the training process of various CNN models and datasets trained with FP32, HBFP(6. (6, 49), and Accuracy Boosters.
RELATED WORK
In recent years, there has been a significant amount of research on inference and training Couur- bariaux et al. 2015a Hubara et al. 2016 Rastegari et al. 2016 Zhou et al. 2018 Li & Liu 2016

---

Under review as a conference paper at ICLR 2023
Table 4: Power consumption ratios between FP32 and various Accuracy Booster configurations.
Model
ResNet20 CIFARIO
ResNet74 CIFAR1O
ResNet50 CIFARTO0
ResNer74 CIFARI00
Transformer (last)
Transformer
(last 5)
Ratio
28.2
36.2
35.0
36.5
38.0
30.0
100
100
Ampart
80
Accuracy
Topl Accuracy
60
40
40
fp32
fp32
hbfp6
hbfp6
20
booster
20
booster
last
10
last 10
20
40
60
80
100
120
140
160
50
100
150
200
250
300
Epoch
Epoch
(a) ResNer20 CIFAR10
(h) ResNet50 CIFAR100
100
100
80
80
Top1 Accuracy
Top1 Accuracy
60
60
40
40
fp32
fp32
hbfp6
hbfp6
20
booster
booster
150
250
300
0
100
200
250
300
Epoch
Epoch
(c) ResNe174 on CIFARI00
(d) DenseNet40 on CIFARI00
Figure 2: Topl Accuracies various models with various training techniques.
Lin et al. 2016 Courbariaux et al. 2015b Dettmers et al. 2022a) with narrow numerical repre- sentations. Google Brain has introduced 6 as drop-in replacement for FP32 (the IEEE 754 single-precision floating-point format) in accelerators, using the same number of exponent bits as FP32 with half the number of overall bits (Wang & Kanwar 2019). NVIDIA's mixed-precision training proposes to use FP16 (IEEE 754 half-precision) for training and FP32 for data loss re- covery (e.g., for weight updates) (Micikevicius al.) 2018). Prior work proposed training models with FP8, either using mixed precision training or customizing the number of bits provisioned for the exponent and mantissa (Sun al.] 2019). The provisioning done according to the preci- sion requirement difference in the forward and backward passes. Recent research advocates the use of Block Floating-Point for DNN training (Drumond) 2020) and inference (Darvish Rouhani et al. 2020). Flexpoint (Köster et al. 2017) and Dynamic Fixed-Point (Das al. 2018 propose block- block-floatingg flonting-point formats for training with a 16-bit mantissa and shared exponent. Prior work proposed novel format for training DNNs with BFP, called Hybrid Block Floating- Point (HBFP) (Drumond et al. 2018). In this paper, we argue that reducing the mantissa bit width in HBFP results significant improvement in silicon efficiency while designing hardware for DNN training.
8

---

formats only for Under review as a conference paper at ICLR 2023
Many have proposed techniques to compensate for the data loss introduced by narrower numerical representations. Mellempudi et al. (2019) use mixed precision with enhanced loss scaling to train models accurately with FP8. |Sun et al. (2019) leverage the precision requirement difference in the forward and backward passes and propose a hybrid FP8 format where they vary the number of bits provisioned for the exponent/mantissa. [Sun al. (2020) show that mantissa-free radix-4 FP4 training is possible with adaptive gradient scaling and two-phase rounding.
Mixed-precision training has emerged as a popular technique increase the fraction of leaner arith- metic formats within the training process, motivating us to explore the design space of mixed-
mantissa HBFP. Several techniques vary the precision layer-wise by using higher precision arith-
metic for layers with greater significance (Khoram & [018 Yang & Jin 3021;SSher et al.. 2020).
Specifically, Choi et al. (2018); Zhhou et al.
a (2018): Mellempudi et al. (2019): Wang et al. (2018) use FP32 for the first and last layers. Fuu et al. (2021) employ fixed-point arithmetic with different bit widths epoch-wist over the course of training. Combining the layer-wise and epoch- ap- proaches, Fu et al. (2020); Zhang et al. 2021 Noh et al. (2022) vary the precision adaptively per epoch and layer at the same time using control mechanisms. While all the aforementioned studies employ leaner arithmetic for fraction of the training process, they fail to make the leaner arithmetic the common case of the training process. In particular, |Fu et al. (2021) and Fuu et al.] (2020) use high-precision arithmetic (8-bit or 6-bit) for the backward pass, which more memory-bandwee baaddiddhh intensive and requires up to 3x more computation than the forward pass.
Recent work (Dettmers et al. 2022b suggests that during mixed-precision FP16 training, the op- timizer states can be reduced to 8 bits by using block-wise quantization method. This observa- tion is in line with our work that applies quantization by extracting the largest exponent per block. Similarly, FAST (Zhang et al. (022]) uses a woock-llatingg-oin--baes approach using and 4-bit mantissas. Unlike our work. FAST requires fine-tuning several addi- tional hyperparameters for its training algorithm, making it difficult to apply to other DNN models. Another block floating-pointe
layer-wise mixed precision
work, FlexBlock (Noh et al.) 2022), uses 4 and 8-bit mantis- sas with various block sizes and also needs higher-precision precision flock-flaating-- weight gradient calculations that suffer more from quantization errors.
CONCLUSION
The immense growth in DNN model sizes and complexities, and exploding amount of data have led to commensurate increase in demand for computing, and thus the design of silicon-efficient DNN accelerators. Low-precision training techniques and specialized numerical formats have been introduced to increase the arithmetic density of the DNN accelerators. One of such formats, Hybrid Block Floating-Poii Point (HBFP), that allows for a majority of the DNN's arithmetic operations (i.e., dot products) to be performed using fixed-poin arithmetic and limits floating-poin arithmetic to minimal set of operations, has been shown to achieve FP32 accuracy with 8-bit mantissas. Moreover, lower number of mantissa bits allows for exceptional improvements in hardware (c.g.. up to 17.5x gain over FP32 silicon area). In this paper, we investigate the empirical limits of HBFP for emerging models and datasets. We show that HBFP6 the smallest HBFP format achieving FP32 accuracy. To further minimize the number of bits, we analyze the effect of the block size and show that for small models, lowering the block size to 25 allows for training with HBFP5. Keeping the first and last layers in FP32 enables HBFP5 training with block sizes of 49 and higher for all models. We propose the Accuracy Boosters technique to bring HBFP4 into training, varying the number of mantissa bits between 4 and 6 throughout epochs, leveraging the insight that each epoch has different effect on training. We show that the last stage of training requires more precision than the rest. Our method achieves 16.98x hardware overhead gain over FP32, while outperforming FP32 accuracy.
7 REPRODUCIBILITY STATEMENT
All the experimental results reported in the paper are reproducible using the code submitted as a supplementary material. The models, datasets and hyperparameters used in the training of all the models are explained in Section 4 Appendix A and B Both image classification and machine translation tasks have scripts train the models mentioned in the paper.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tier Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httpttss//prccccc.gs..oot 1457c0d6bfcb4336 L55708661100011110008822422222.2 Abstsaaatttaattttt.ttt..... html
Awst
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. CoRR, abs/1805.06000 805.06085, 2018. URL http:////iiioo.gg//o//o haagg/ab//ine abs/1805..
06085
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Train- ing deep neural networks with binary weights during propagations. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 28: Annual Conference on Neural Infor- mation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 3123-3131, 2015a. URL httpps::/prooccc.nnggg/////////s neuri ips. http/paaer/2015/astts BB256CL1177942225522255555622222 Abstracttbbsrr.t.. html
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Low precision arithmetic for deep learning. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015b. URL /ttta:://axxv.orrr org/ abs/ 1412. 7024
Bita Darvish Rouhani, Daniel Lo. Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky. Sarah Massengill, Lita Yang. Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, XIA SONG, Subhojit Som, Kaustav Das, Saurabh T, Steve Reinhardt, Sitaram Lanka, Eric Chung, and Doug Burger. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systemx, volume 33, pp. 10271-10281. Curran As- sociates, Inc., 2020. URL SBU/sttsss//ppocceiisssss.oee neurps wttss.ccc/eee httpssscccpappeee///ea.. TAne220Dfee/Taaaaascccaaaaaaaa
pdf
Dipankar Das. Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj D. Kalamkar, Sasikanth Avancha, Kunal Banerjee. Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradeep Dubey, Jesús Corbal, Nikita Shustrov, Roman Dubtsov, Evarist Fomenko, and Vadim O. Pirogov. Mixed precision training of convolutional neural networks using integer operations. In 6th International Conference Learning Representatiinss ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. Utttttps:
net. htt.://www..eettommamoaaa131111.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int80: 8-bit matrix mul- tiplication for transformers at scale. CoRR, abs/2208 07339, 2022a. doi: 10.48550/arXiv22208 07339. URL /tct::///Go..oog/1...555/////// http://rxxv..222..0733
Tim Dettmers, Mike Lewis. Sam Shleifer, and Luke Zettlemoye.. 8-bit optimizers via biock-wise quantization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL htttss:/ net/ http:/forummiisshpppppeee wtwpk//xxosspppk..o.
Mario Drumond, Tao Lin, Martin Jaggi, and Babak Falsafi. Training DNNs with Hybrid Block Floating Point. arXiv:1804.0152 00560112666... [cs, stat]. December 2018. URL http: http:///axvvv... org/
abs 1804. 01522
arXiv: 1804.01526.
10

---

Under review as a conference paper at ICLR 2023
Mario Paulo Drumond. Coltrain: Co-located dnn training and inference. pp. 115, 2020. doi: 10.5075/epfl-tee thenst--hesss---e 10265. URL htt.iitpp///nooccieeee.eeeee http://e..pppcccceeeeoro/////0..
Yonggan Fu, Haoran You, Yang Zhao, Yue Wang, Chaojian Li, Kailash Gopalakrishnan, Zhangyang Wang, and Yingyan Lin. Fractrain: Fractionally squeezing bit savings both temporally and spatially for efficient DNN training. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httt:htt::///// /ttpltttts../pprcceed..oo neurips cc/
Yonggan Fu, Han Guo, Meng Li, Xin Yang. Yining Ding, Vikas Chandra, and Yingyan Lin. CPT: efficient deep neural network training via cyclic precision. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
1= 2 a2
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A Survey of Quantization Methods for Efficient Neural Network Inference. urXiv:2103.11 13630 [cs], June 2021. http://tttt///ai..orr/g//////// www.oo/ggbbbbaaaa 03 1.3...300 13630 arXiv: 2103.13630.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In 2015 IEEE International Confer- ence on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026-1034. IEEE Computer Society, 2015. doi: 10.1 109/CCC..0155.2...
htR.htss:: /ooi. http//oo.....c.../// org/10.
1109/ICCV 2015.12 123
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016 URL doi org/ 10. wwwp1111999...22 2016..
Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. CoRR, abs//608.06993,, 2016. httttp::/aaxw...o///aaattae
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10. 2016, Barcelona, Spain, pp. 4107-4115, 2016. URL http:htttsp:ppceeeteeeeeee ngs neurips. cc/
.ttml
Soroosh Khoram and Jing Li. Adaptive quantization of neural networks. pp. 13, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report O, University of Toronto, Toronto, Ontario, 2009.
Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, Luke Horoff, Amir Khosrowshahi, Carey Kloss, Ruby J Pai, and Naveen Rao. Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:/ttt/p:pprrreee nips. cc/ cc//papper/211/haas/
Fengfu Li and Bin Liu. Temary weight networks. CoRR, abs/1605.04411, 2016. URL http:
// 1/arxiv. wttpv//.gggbbbb/bbbabeoe 060./6606..00 04711
Darryl Dexu Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In Pooriaa Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2849-2855 M..R.rrg. 2016. URL wtl.http::// httpp:/pproceddiiii https//n........ss/////////1111/
/tt.a///aa//o/o/ooooooooaaaaooaa html

---

Under review as a conference paper at ICLR 2023
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed Precision Training With 8-bit Floating Point. arXiv:1905.123: [cs, stat], May 2019. URL http:
arXiv: 1905.12334.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. arXiv: [cs, stat], February 2018. (RLlattp://aaaa org/
abs 11101 03740
arXiv: 1710.03740.
Seock-Hwan Noh, Jahyun Koo, Seunghyun Lee, Jongse Park, and Jacha Kung. Flexblock: A flexible dnn training accelerator with multi-mode block floating point support, 2022. URL https:
//rravv.rrr//bb//223...
Myle Ott, Sergey Fdunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019 Demonstrations, 2019.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 2019. URL http:// proceedings..o mww. wtw.mmlr.preses. wrwppprssssss.. vv7 rahamanl9a99 9a. html
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision - ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 525-542. Springer, 2016. doi: 10.
32. URL
https: /ooi. .r.g/rg/1 0. 1007 978-99--- 3-
Janghao Shen, Yue Wang, Pengfei Xu, Yonggan Fu, Zhangyang Wang, and Yingyan Lin. Frac- tional skipping: Towards finer-grained dynamic CNN inference. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational
Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pp. 5700-5708. AAAI Press, 2020. URL https: /ojs aaai . org/ wwwgginoen wnwiinddxxp... php//AAI/ AAAI/
art article/view/Ci 6025
Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vi- jayalakshmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrish- nan. Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Net- works. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL htttpp::/prrceeddiagg neuripss........................ httpppppccccpppee/////aaaaaae html
Xiao Sun, Naigang Wang. Chia-Ya Yu Chen, Jiamin Ni. Ankur Agrawal, Xiaodong Cui, Swa- gath Venkataramani, Kaoutar El Maghraoui. Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. Ultra-Low Precision 4-bit Training of Deep Neural Networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 1796-1807. Curran Asso- wwwp://weeediii.....eeeeeeeeeeee newrips cc pappr//220//ilel
ciates, Inc., 2020. URL [3319482233111ccbeeecccc3333
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gameett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 0008,600800000000 2017. URL http:///ooooocaaaaooo.ooo..o/aaa
hew.:/eeeedddngg............pp.. 3f /
12

---

Under review as a conference paper at ICLR 2023
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yo Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. Proceedings of the 32nd Inter- national Conference Neural Information Processing Systems, NIPS' 18, pp. 7686-7695, Red Hook, NY, USA, 2018. Curran Associates Inc.
Shibo Wang and Pankaj Kanwar. BF The secret to high performance on Cloud TPUs, August 2019.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. CoRR, abs/1901.06522 2019. URLhttps
lrg.i.oorggaa.....a..oooooo....r abs/ / 901. 06523
Linjie Yang and Qing Jin. Fracbits: Mixed precision quantization via fractional bit-widths. In Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Thirty-Third Conference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Edu- cational Advances in Artificial Intelligence EAAI 2021, Virtual Event, February 2-9, 2021, pp. 10612-10620. AAAI Press, 2021. URL hhtpss://ojs aaai. org/ hetg//ndex.... phpppppppa..//ppp AAAI/ article/view/1720
Sai Qian Zhang, Bradley McDanel, and H. T. Kung. FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding. arXiv:21 10.15456 [cs], October 2021. URL
http://aat
org/ abs 21100 15456
arXiv: 2110.15456.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa- Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. 16X10:1606.06160 [cs], February 2018. arXiv: 1606.06160.
13

---

Under review as a conference paper at ICLR 2023
A HYPERPARAMETERS FOR IMAGE CLASSIFICATION EXPERIMENTS
We report the hyperparameters we use for all our experiments in Table S
All convolutional layer
weights are initialized using zero-mean Gaussian distribution whose standard deviation (std) is
/2ny (ny is the number of activations in the output), method introduced in He et al. (2015). All the
Batch Normalization layer weights are initialized and kept FP32 HBFP suggests (Dru-
mond et al. 2018).
Table 5: Hyperparameters for various sizes of ResNet on CIFARIO
and CIFARI00
Resnet20 & ResNet74
Resnet50 & ResNer74 on CIFARIO 100
Hyperparameter
on CIFARI0
Epochs
160
300
Batch size
128
128
Optimizer
SGD
SGD True
Nesterov
True
Learning rate
0.1
0.1 150, 225
Learning rate decay epochs
82, 122
Learning rate warmup
True
True
Weight decay
le-04
le-04
Momentum
0.9
0.9
B HYPERPARAMETERS FOR NEURAL MACHINE TRANSLATION EXPERIMENTS
Our implementation is based on Transformer implementation of Fairseq (Ott et al. 2019). We report the hyperparameters use for all our experiments in Table 6 For the ones we do not specify, use the default hyperparameters.
Table 6: Hyperparameters for Transformer Base on WMT 16 Translation Task English to German
Transformer architecture hyperparameters
Dataset
wmtl6 www.eenddeebb..eee..e bpe3kk
Architecture
maanformer_wmil en..de
Number of layers
6
Number of heads
Number of hidden dimensions
512
Optimizer
Adam
Adam-betas
(0.9.
Learning rate
0.0007
Learning rate scheduler
Inverse Square Root
Warmup updates
4000
Warmup initial learning rate
1e-07
Label smoothing
0.1
Maximum tokens
3584
Criterion
labelsmoothedd.rosss.ntopp
Clip-norm
0.0
Weight decay
0.0
Dropout
0.1
Update frequency
2
Share all embeddings
True
14


Reviews:
[{"review_id": "yANML5fJe8u", "reviewer": "Reviewer_gycw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the HBFP parameter search space to further improve efficiency and density in hardware accelerators.\nThey propose training on HBFP6, after experiments making the trade off: lower block size gives more accuracy, but is less HW efficient.\nThey show accuracy booster method that switch number of mantissa bits in different epochs in the training stage. They evaluate their experiments on CNN and transformers for cifar10/100 and wmt16.", "summary_of_the_review": "Overall the paper is well written and clear to understand.\nThe paper should show additional data or experiments that will make the claims and proposed methods even stronger.", "strength_and_weaknesses": "The paper is well written and clear. The paper shows comparison between various HBFP configurations and presents accuracy booster method\n\n\nFew points that could improve the paper:\n\nIt would be interesting to see trade-off accuracy and hardware efficiency when the variable block size is across different layers.\n\nThe paper should also evaluate on bigger dataset: Imagenet and updated model: ViT\n\nTable 1: should have HBFP8 for comparison \n\nHBFP6 with block size 1, is same as having 16bit for every value, wouldn't FP16 with mixed-precision training be better or equivalent in terms of accuracy? \n\nIn table1, HBFP6 doesnt achieve same accuracy of FP32 model. Thus, the intro's claim should be revisited: \"We show that HBFP6 is the smallest HBFP format achieving FP32 accuracy\".\n\nIn figure2, should add some form to show power savings while using accuracy booster.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_tynKQ-pYYo", "reviewer": "Reviewer_VDoK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors empirically study hybrid block floating-point (HBFP). HBFP uses a single shared exponent with a block of multiple mantissas. The authors run training experiments on CIFAR-10/100 using ResNets and DeseNet40 with HBFP. Mantissa bits of 6, 5, and 4 are tested with block size ranging from 16 to 576. Both forward and backward pass is quantized to HBFP. The authors conclude that HBFP5 with the first and last layers in FP32 is sufficient to match FP32 training accuracy.\n\nThe authors also experiment with FracTrain, where most epochs of training is done in low precision and the final one or few epochs in higher precision. For CNNs, training in HBFP4 with last epoch in HBFP6 recovers FP32 accuracy. However, for Transformer there was still a gap between this setup and FP32.", "summary_of_the_review": "A purely empirical paper that studies the HBFP format for DNN training. The paper does not have much novelty and the experimental section is weak, with data mostly coming from ResNets trained on CIFAR-10/100. The main takeaway is that training with HBFP5 works fine for CNNs. If training with HBFP4, the last few epochs should be done in higher precision like HBFP6.", "strength_and_weaknesses": "Strengths:\n1. Paper was easy to understand\n2. HBFP seems like a interesting data format with potential\n\nWeaknesses:\n1. The paper is entirely empirical. All conclusions are drawn largely from experiments on CIFAR-10/100 using ResNets and DenseNet40. One 1 table had data on transformers. This is a very weak set of benchmarks and models for an empirical paper.\n2. The results are not too exciting. For CNNs, the first and last layers must be kept in FP32 - so HBFP is only used for the \"easy to quantize\" layers. For the Accuracy Booster technique, it doesn't seem to work well on transformer.\n3. The paper combines two existing techniques: HBFP and FracTrain. The techniques are mostly orthogonal and not combined in a novel or insightful way. I didn't get much out of the paper besides \"certain configurations of HBFP4/5/6 can be used to train CNNs\", which isn't all that surprising.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RPdk3ci-Rwh", "reviewer": "Reviewer_FLAx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces \"accuracy boosters\" that aim to train DNNs using HBFP4 for most of the epochs while switching to HBFP6 for the last few epochs. Thy show that mixed precision training is applicable under HBFP setting and achieve similar accuracy with better hardware utilization.\n", "summary_of_the_review": "I enjoyed reading the paper. I am not an expert in different fixed point formats. However, the background section was fairly detailed for me to appreciate the use of HBFP.\n\nThe technique seems fairly intuitive. The authors adapt mixed precision training in the context of HBFP. The results show that by training most of the time with HBFP4 and switching to HBFP6 at the end allows the authors to achieve comparable accuracy at a lower hardware cost.\n\nThat said the technique for finding this schedule was mainly through manual experimentation. Therefore, I am not sure of the novelty of the work. ", "strength_and_weaknesses": "Strengths:\n\n* Good background section that clearly explains the use of HBFP\n* Intuitive explanation of mixed precision training in the context of HBFP with proper experimental backing\n\nWeaknesses:\n\n* It is unclear why training at the very end using HBFP6 allows you to recover the accuracy loss, more explanation would have been better", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HpwN8wFWf14", "reviewer": "Reviewer_L7nb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigated a proper bit-precision for a block floating-point format for deep neural network training and revealed that training with a small number of mantissa bits can be compensated if the training iterations with a large mantissa bit are followed. The authors also investigate the interplay between the block size and the mantissa bits for the block floating-point performance. The authors provided the experimental results to empirically support their claims.", "summary_of_the_review": "In this paper, the authors investigated the characteristics of block floating point format. Although some of the empirical findings in this paper are interesting, many lessons derived from these observations seem to be incremental, limiting the novelty of this paper.", "strength_and_weaknesses": "(Strengths)\n- an interesting observation that the accuracy degradation by aggressively reduced mantissa bits can be recovered by the training with the increased number of mantissa bits.  \n\n(Weaknesses)\n- This paper is mainly about the presentation of empirical findings, and many of them are already well-known (e.g., full precision for the first/last layers.)\n\n- There are no deeper insights into the interplay between the block size and the mantissa bits.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3GGz1pj7Yc", "reviewer": "Reviewer_gLsp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Block floating point format is a HW implementation technique for deep neural networks (DNNs), wherein floating point (FP) matrix multiply (Matmul) operations are simplified to fixed point multiply operations by sharing the exponent bits for a block of input values. Further, the mantissa bits for the fixed-point format are truncated to the minimum required for achieving the same or better accuracy for a given ML task than FP baseline. \n\nThis paper presents a design space exploration over mantissa bit-width and block size for a range of DNNs targeting different applications. Based on this exploration the authors identify that 6-bit mantissa format (HBFP6) achieve little or no loss of accuracy for significant HW savings. \n\nFurther, the authors propose the approach Accuracy Boosters, which uses the insight that the epochs in a model training have varying importance for final accuracy, e.g., last few epochs are critical and suffer the largest accuracy impact due to lower mantissa bit-width. Therefore, by selectively applying higher bit-width mantissa config (e.g., 6) to the last few epochs authors demonstrate a significant accuracy improvement even if majority of the cycles use lower mantissa bit-width (e.g., 4)\n\nAuthors show the accuracy results for a few DNNs targeting different applications and claim ~17x improvement in their figure of merit (FOM) for hardware cost, active Silicon area used. \n", "summary_of_the_review": "The authors provide two new insights, 1) HBFP6 can achieve high accuracy for a range of networks and 2) Accuracy boosters is a reliable way to get higher accuracy while spending lesser energy. However, the figure of merit used by the authors, Si area, is not very useful as even a single usage of a unique data format requires associated hardware which if custom designed will come with its own area. Energy efficiency would have been a more useful metric. Also, the analysis of Accuracy boosters is limited and could use more experiments. ", "strength_and_weaknesses": "Authors provide a good prior art section describing a range of papers that attempted the problem of developing new encoding formats to reduce the hardware cost of training high accuracy DNNs. \n\nThe paper tries to estimate hardware cost by estimating the approximate area of a design (adding the area for components misses glue logic) that trains DNNs for a given mantissa/exponent bit-width and block size. As a first order metric this is helpful to understand the savings achieved by reducing the mantissa bit-width or changing the block size. But a real hardware design would require the author to either choose a fixed HBFP format (e.g., HBFP5 only) or a few formats (e.g., HBFP4 and HBFP6) or to choose a reconfigurable hardware which will keep switching the size of HBFP mantissa bit-width used. Note that, the added cost of using reconfigurable hardware can be a limiting factor in its usage. \n\nIf the authors commit to having HW available to support a few formats, they can then simply compare the energy efficiency of using mixed HBFP format as a figure of merit. \n\nUsing the chosen FOM, approximate hardware area, Figure 1 illustrates the improvements achieved over FP32. To improve the readability of the figure, please add horizontal lines showing when the design achieves 95% of the hardware benefit. Also consider adding lines that link the points with same/similar accuracy for a given benchmark, e.g., 2% accuracy loss point for the different HBFP bit widths.\nAuthors highlight that HBFP6 achieves good accuracy for a range of benchmarks, this is a new empirical result not explored in prior art. \n\nSection 4.1 explores the results of minimizing HBFP bit widths. This is a minor issue, but the authors claim that HBFP6 achieves accuracy within 2% of FP32 for all block sizes but ResNet50 trained for CIFAR-100 with block size 576 fails this condition. Some of the following discussion in this section is confusing, e.g., “for the smallest model ResNet20, decreasing the block size of HBFP5 to 25 leads to 1.5% accuracy degradation but 1.1× hardware gain in return”, it’s not clear what the baseline here is. Decreasing the block size generally helps improve accuracy and reduce the hardware gain. Also, it would be helpful for improving the readability if the authors added the area improvement ratios discussed in this section as a column in Table 1. It’s not easy to confirm these area improvement numbers from Figure 1. \n\nAlso, Table 1 highlights how using FP32 format for first and last layers helps improve the accuracy of training despite using HBFP format on other layers, but this has been done in previous papers. And the requirement of FP32 format even for a single layer means that Si area associated will be required, hence energy saving would be a better metric for comparing the efficacy of this technique. \n\nAccuracy boosters is a useful technique which minimizes the HBFP mantissa bit width for epochs that are least impacted by it. The accuracy improvement achieved using this technique is remarkable. It enables HBFP4 to train ResNet50/74 to higher than FP32 accuracies which wasn’t feasible by default as show in Table 1. It is unclear however, that why the authors didn’t fully explore the benefit of using Accuracy boosters? How would it fare for HBFP3/2/1? They could potentially come with their tradeoff of how many epochs need a bit width boost. \n\nMinor issue, the Conclusion section states, “Keeping the first and last layers in FP32 enables HBFP5 training with block sizes of 49 and higher for all models.”, it’s unclear what the authors mean here, cause for larger block sizes accuracy numbers drop regardless of the technique used. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject