Toward Efficient Low-Precision Training: Data Format Optimization and Hysteresis Quantization
Authors: Sunwoo Lee
Jeongwoo Park
Dongsuk Jeon
Venue: ICLR 2022 Poster (2022)
Keywords: low-precision training
quantized training
logarithmic weight
data format optimization
hysteresis quantization

Abstract: As the complexity and size of deep neural networks continue to increase, low-precision training has been extensively studied in the last few years to reduce hardware overhead. Training performance is largely affected by the numeric formats representing different values in low-precision training, but finding an optimal format typically requires numerous training runs, which is a very time-consuming process. In this paper, we propose a method to efficiently find an optimal format for activations and errors without actual training. We employ this method to determine an 8-bit format suitable for training various models. In addition, we propose hysteresis quantization to suppress undesired fluctuation in quantized weights during training. This scheme enables deeply quantized training using 4-bit weights, exhibiting only 0.2% degradation for ResNet-18 trained on ImageNet.

Published as a conference paper at ICLR 2022
TOWARD EFFICIENT LOW-PRECISION TRAINING: DATA FORMAT OPTIMIZATION AND HYSTERESIS
QUANTIZATION
Sunwoo Lee, Jeongwoo Park, Dongsuk Jeon Graduate School of Convergence Science and Technology Scoul National University, Seoul, Korea {ori915, jeffjw, djeon1 J0su J@snu.ac..... ac.
ABSTRACT
As the complexity and size of deep neural networks continue to increase, low- precision training has been extensively studied in the last few years reduce hardware overhead. Training performance is largely affected by the numeric for- mats representing different values in low-precision training, but finding optimal format typically requires numerous training runs, which is a very time-consuming process, In this paper, we propose method efficiently find an optimal format for activations and errors without actual training. We employ this method to deter- mine an bit format suitable for training various models. In addition, we propose hysteresis quantization to suppress undesired fluctuation in quantized weights dur- ing training. This scheme enables deeply quantized training using 4-bit weights, exhibiting only 0.2% degradation for ResNet- 18 trained on ImageNet.
INTRODUCTION
Deep neural networks have been used in various fields such as vision, audio, natural language pro- cessing, and reinforcement learning. As larger and more complex neural networks are adopted, the energy and time consumed for training have become a critical issue in hardware implementation. Using low-bit representations in training significantly reduces hardware overhead and memory foot- print; hence, neural network training with limited precision has been extensively studied recently. For instance, 16-bit formats are already adopted in commercial devices such as FP16 (EEE 2019) in Nvidia GPUs and 6 (Kalamkar et al. 2019) in Google TPU (Wang et al. 2019). Also, Köster et al. (2017) suggested new data format using a shared exponent suitable for low-precision training. Recently, it has been demonstrated that even 8-bit formats could be adopted in deep neural network training with reasonable accuracy (Sun et al.) 2019 Fox et al., 2020). However, there are various issues in realizing low lowpprecision training in practical applications detailed below.
Optimal data format for low-precision training: Training performance susceptible to the data format we use to represent variables in the network. When a value is represented using floating- point format with a fixed number of bits, there a trade-off between dynamic range and precision. For instance, allocating more bits the exponent part in floating- -pintt format enlarges the dynamic range but lowers precision due to fewer bits in the mantissa part. Recent studies on 8-bit training suggest various ways reduce the dynamic range required for number representation representation precision. Early work on 8-bit training (Wang et al. 2018) adopts a 5-bit exponent represent different variables using single format, but (Snn et al. (019) examine the statistics of each variable and optimize the numeric formats separately. Specifically, the values used in the forward path (weight and activation) have relatively narrow dynamic range, and only 4 bits are allocated to the exponent. FFox et al.) (2020) propose to divide data into smaller blocks and assign shared exponent bias to each block. Since the values in a block tend to exhibit similar statistics, the forward (weight and activation) and backward (error) paths could be represented using only 2-bit and 4-bit exponents, respectively. Note that the shared exponent bias effectively identical to the scaling factor. If variable has value of 2" and shared exponent bias of b, then its actual value is m thibias which is identical to the scaling factor of 2has However, these approaches difficult generalize since we should empirically decide numeric formats for each task, neural
enhance

---

Published as a conference paper at ICLR 2022
Different dataset
Full- praciston model
From- scratch training
Fine-tuning pre-tralined model
Imagenet A Accuracy
Colle
Quart
Linwet
Currs
Concas
Quart
Snalec BossPrddzct Attention
Resnat18
Quarc
Linear
Quant
Different quantization
Different model
position
69.8%
67.7%
68.8%
(a) Various quantized training environments
(h) Performance degradation in quan- tized models
Figure 1: Two Challenges in low-precision training. (a) An optimal format varies with dataset, quantization scheme, and model. (b) There is a performance gap between from-scratch training with quantized weights and fine-tuning of pre-trained full- full-precision model.
network structure, and quantization scheme (Fig. [). Furthermore, analyzing the statistics of each variable not enough determine optimal format. Their distributions often have long tail, and hence the dynamic range of the numeric format should be experimentally selected through many trial-and-errors in actual training.
Performance degradation in from-scratch training: Previous studies on quantized models show that model could achieve comparable accuracy full-precision models even using 1- or 2-bit weights (Choi et al.) Martnnez et al.) (020)) through fine-tuning a pre-trained model. However, low-precision training where neural network trained from scratch using low-precision values and computations, the trained model typically shows noticeable accuracy drop (Flhoushi et al.) 202T]. Fig. [b) shows the Top-1 validation accuracy of ResNet-18 (He et al.J (2016) trained on ImageNet (Deng et al. (209) for difffrrnnt training schemes. The weights are quantized into 4-bit base-2 logarithmic format. Prom-scratch training of the model with quantized weights results in 2.1% accuracy drop, whereas only 1.0% degradation is observed if we fine-tune pre-trane pre-trained model. This suggests that even though a better solution (i.e., a set of parameters) exists for given format, it cannot be reached through from-scratch training.
To formalize the issues above, here we divide quantization in low-precision training into two types: network quantization and data flow quantization. Network quantization refers the quantization of the neural network model. An example of this type of quantization is weight quantization. In net- work quantization, we need reduce the performance difference between from-scratch training and fine-tuning (Yang et al. 2019b). On the other hand, data flow quantization refers to the on-the-fly quantization that occurs when data propagate through the network low- lowpprecision training. Exam- ples include activation, етгоr, and weight gradient quantizations. Additional errors are introduced in weight update computation due this type of quantization, which leads to performance degradation. Hence, we need to find an optimal format to minimize accuracy drop due to computation emrors in data flow guantization.
In this paper, present systematic approach to implementing low-precision training various models and tasks. First, we present a method to efficiently find an optimal format for data flow quantization. In addition, introduce hysteresis quantization technique, quantization method for network quantization that can mitigate the issues of from-scratch training. Our main contributions are:
We present method that predict the training performance of various numeric formats for data flow quantization. This method allows us to determine an appropriate data format for different neural network structures, datasets, and tasks efficiently.
2

---

Published as a conference paper at ICLR 2022
Using the method above, propose optimal 8-bit format suitable for low-precision training of various models, which enables quantization BatchNorm layer input and im- proves hardware efficiency with minimal performance degradation.
quantization scheme that utilizes the hysteresis effect to improve the performance of from-scratch training in network quantization. This scheme enables ultra-low-precisionn training using 4-hit logarithmie weights.
We propose
DATA FLOW QUANTIZATION
2.1 NUMERIC FORMATS
There are many numeric formats that be constructed with bits depending how much dy- namic range is required and how many valid bits are used for representing a value. For example, using 8 bits we could implement 8-bit fixed point integer format, 8-bit floating-point formats such as FP152, FP143, and 2P125 (FPlxy represents sign bit, x exponent bits, and y mantissa bits), 8-bit posit format (Gustafson & Yonemoto 2017), and 8-bit float-fix format (Han al., 2019). Since the diversity of formats that could be formulated using n bits is nearly unlimited, here we assume some constraints limit candidates while still including widely used formats such as fixed-point and floating-poin formats as below:
The MSB (Most Significant Bit) is used sign bit and other bits represent magnitude. Accordingly, only symmetric formats that have identical representable ranges for positive and negative numbers considered. Two's complement representation is slightly asym- metric since it can represent one more negative value, but does not incur a significant difference.
The number of valid bits of a larger value is greater than or equal the number of valid bits of a smaller value. The valid bits stand for significant digits in binary representation.
The ratio between consecutive representable values does not exceed 2. For example, the base-4 logarithmic format is excluded.
We could obtain 166 8-bit formats that meet these constraints. Then, we reduce and valid bits in each format to obtain 7- and 6-bit formats, resulting in 498 formats in total. More information the numeric formats considered in our experiments is provided in Appendix A.I
2.2 ACTIVATION AND ERROR QUANTIZATION
In a neural network consisting of layers, the training process described by
A f((W(,A)
(1)
E1 = gi (W,, FE+1)
(2)
Get hr(Ai, E1+1)
(3)
W = oGGui, Wi)
(4)
where A, E, W, and G. are activation, етror, weight, and weight gradient, respectively. f. g. h. and are forward, backward, gradient, and update functions. / and represent the layer number and time step. We follow the quantized training scheme suggested by Fox et al. (2020), but with the following modifications reduce hardware implementation costs. and E are quantized not only for the GEMM imput but also for the BatchNorm layer input. BatchNorm layer normalizes input using the mean and variance of each channel, but these values are obtained only after observing all the inputs from the previous layer, necessitating that all input values are temporarily stored in memory. Therefore, quantizing the BatchNorm layer's input significantly reduces memory footprint and memory access overhead. Additionally, the scope of sharing exponent bias is extended to layer (A, and Ej) to avoid the overhead of aligning partial sums from different blocks in block-wist exponent sharing. Finally, instead of determining the shared exponent bias by analyzing all values in the layer, we conservatively update it by detecting overflow and underutilization that occurred in the previous mini-batch.
3

---

Published as a conference paper at ICLR 2022
4 G May
Spearman's Cerrelation0..22
5.06
IManl
1:06
Spearmnn's Cerreiation-0.900
Spearman's Cordiations007700
Spearmen's 9
580
2+1" 3015
2-15"
Training Loss
Training Loss
Training Loss
Training Loss.
(a) Error quantization
(b) Activation quantization
Figure 2: Training loss vs. proposed performance indicators. Blue dots represent full-precision training runs, and yellow, green, and red dots represent training runs with 8-, 7-, and 6-bit formats, respectively.
2.3 INDICATORS OF TRAINING PERFORMANCE
Effect of quantized error: Quantizing the error E' the backward path is independent of how the forward path behaves since the loss surface of the model does not change. Therefore, the optimal W that the network needs to reach through training remains the same regardless of the error quantization scheme. However, when the error is quantized, quantization error AE is introduced in E, which incurs a noise NAg in G through the gradient function in Eq. 3 and potentially updates each weight in the wrong direction. While some amount of noise may improve the training performance through regularization, using low- oowpprccision formats already introduces a large noise in the network, incurring performance degradation (see Appendix (A.).. Therefore, we suggest that the weight gradient error NAE could be good indicator of degradation in training performance. One way to implement this predicting performance using the magnitude of Naæ; however, the noise is in the same direction as G.., it would only change the amount of each update and result in less severe effect. Instead, we could measure the misalignment between G.. + NAE and G for performance prediction. The misalignment between two vectors is estimated by
A.B
(AA, cos-
(5)
||
-1
Then, the change in the update direction due to NAE is L(Gw, Gw + NAB). We can expect that the smaller Z(G w. G + NAE). the better the training performance.
Effect of quantized activation: Contrary to eгrоr quantization, activation quantization affects the way the forward path operates, and the loss surface of the model changes. Hence, the global optima of weight parameters shift, where the amount of shift would be proportional to the quantization noise. The displacement of global optima can be indirectly estimated using the direction of the weight gradients G... If the angle Z(Gw,, G + Naa) is small, the deviation of the global optima expected to small as well, suggesting better training performance.
In the discussions above, we assumed that the angles /G G + NAE) and (Gw Gw + NAA) could be used to predict training performance. We experimentally prove this by comparing the training performance of different numeric formats. For 498 numeric formats in 6 8 bits, compare the loss obtained from training with the proposed performance indicators (Fig. 2 Training loss is obtained by training ResNet-18 for 60 epochs. The batch size is 128 images and the initial learning rate is 0.1, which is decayed by cosine scheduler. We average angles from 100 mini-batches after quantizing pre-trained model. Note that we use G of the first layer since it can reflect quantization errors that occur in the activations and errors of all the layers in the network. The weight gradients from the full- iull-irecision network, the network with quantized activations, and the network with quantized eπтоrs are G G + NAA, and Gw + NAE, respectively. Fig. shows that using the misalignment angle results in not only higher Spearman's 's correlation but also more distinct shape for low training losses, making it better metric than the error magnitude. For instance, using the error magnitude would predict the best format for transformer incorrectly (see Fig. e in Appendix A.3) While obtaining the misalignment angle requires additional computations, its overhead is negligible since the part that requires the most time and computation is to obtain G... G + NAg. and G + NAA. which still significantly lower than actual training. Using this method, we could determine the optimal format for specific neural network model, dataset, and task very efficiently we only need to
CIFAR-10 dataset using SGD with momentum of 0.9
4

---

Published as a conference paper at ICLR 2022
Fall
LiG Gw Nax)
LG Gur Naa
Fall
0,8187
L(G Gar
1.307
1,343
1.302
1,202
0.7769
1.197
11 L(Gm Gw Nax)
L(G Gw Nax)
L(Gw. Gw N
ResNet-18 (ImageNet)
(b) ResNet-101 (ImageNet)
(c) MobileNetV2 (ImageNet) (Imaaeeeee
L(Gw C Nan)
LiG Gw Man) x *
L(G Gw Man)
t f R t f 1
* FP152
2,881
FP145
FP134
4,037
3,621
INT8
2005
2.031
LlGm. N
L(Gw. Gw Nas)
L(Gw, Nar)
(d) 2- 2-yeyer LSTM (PTB)
(e) Transformer (IWLST)
(l) MobileNerV2 SSDLite (VOC)
Figure 3: Scatter plots displaying misalignment angles for 166 8-bit formats. The numbers next to four selected data formats (INT8, FP152, FP143, and FP134) represent the loss measured in actual training.
measure the misalignment angle without time-consuming network training. For experiments in Fig. 2. the amount of computation is reduced by 99.6%, and the reduction will be even larger for larger datasets and complex networks that need more epochs for training.
2.4 OPTIMAL FORMAT FOR DATA FLOW QUANTIZATION
Here we show that we could find an optimal format for training with quantized errors and activations using the proposed performance estimation method above. To find format suitable for wide range of models, we select models with different architectures, layer types, and target tasks that are widely used in quantized training research for experiments: ResNet-18. 18, ResNet- 101, MobileNetV2 (Sandler et al. 2018), 2-layer LSTM, small transformer for translation on the IWSLT German to English dataset (Cettolo et al., 2014). and SSD-Lite (Liu al., 2016) with MobileNetV2. We first measure misalignment angles for 166 8-bit formats. To verify the correlation between the training performance and the misalignment angles, select four formats that exhibit low hardware implementation costs (INT8, FP152, FP143, and FP134) and train the networks using each format. While we may use different formats for activation and error, it requires complicated datapath (Sun) et al. 2019) and hence we only consider single format for both variables. The experimental results Fig. 3 demonstrate that the training performance is higher if both misalignment angles are small in all tasks and models, confirming that the proposed indicators could be used to determine the optimal numeric format. Fig. suggests that FP134 and FP143 are the best candidates across all models. For hardware implementation. FP134 is the most optimal format due to its low implementation cost, which is discussed Appendix A.7 in detail. Note that using the error magnitude leads to the same conclusion that FP134 is the best format for the target models. See Appendix A.3 for more details.
NETWORK QUANTIZATION
In quantized neural networks, the weight parameters are generally quantized in a way that minimizes the quantization error (Choi et al. 2019: Martinez et al. 2020). For instance, x quantized round ()), proper value is selected for the scaling factor
into fixed-point format through to minimize the quantization error. However, as the weights continue to change during training, need calculate for every update, which could cause significant overhead. Therefore, prior studies on low precisiioon training suggest constraining the scaling factor to the power of 2 in the shared exponent (Köster et 2017) the shared exponent bias (Fox et al. 2020). In this section,
5

---

Published as a conference paper at ICLR 2022
4~ Small Áuw
Aww
Ow Small Aw
Large Awe
WMW
decreasing peth
WMW
WWW
Ing path
(a) Conventional quantization
(b) Hysteresis quantization
Figure 4: Comparison quantization schemes.
we analyze the issues behind weight quantization and propose new quantization scheme to mitigate
those issues.
3.1 FLUCTUATION OF WEIGHT PARAMETERS
In typical low-precision training, master copy of weight parameters is separately maintained in high precision, and those weights are updated based on the computed weight gradient. This high- precision weight quantized into a low-precision format and used for the forward path computation during training. the scaling factor s is constrained to 2", the quantization threshold remains the same unless is updated due to overflow or underutilization. If the optimal weight is located between two representable values of a data format, the quantized weight would fluctuate alternately between the two values in each update (Fig. ma) even for very small weight update, causing large fluctuations and undermining training performance.
3.2 HYSTERESIS QUANTIZATION
To mitigate the fluctuation issue above, we propose introduce the concept of hysteresis to quan- tization. More specifically, quantize each weight differently in a way that the quantized value tends stay at its current value, effectively minimizing undesired oscillation between two values due to small weight updates. The equation below shows example of the proposed quantization scheme.
if w > Q
=
[ if wt Q
(6)
where is the original value, Q is its quantized value, and 1 is the time step. The proposed hysteresis quantization reduces fluctuation significantly, stabilizing the training process and allowing the network reach global optima more efficiently. In Fig. 4b), if the weight change AW is small, then enough number of those changes should be accumulated to flip Qu. Hence, the update frequency is now proportional to the weight gradient. This helps the network to learn better while suppressing fluctuations for small GT values. Alternatively, we may mitigate weight quantization errors by adopting AdaRound (Nagel et al. (2020), which learns whether each weight should be rounded up or down to produce the same output as high-precision weights. However, whenever full-precision weights are updated, we need to re-train the learnable parameters (i.e., quantization scheme of each weight), incurring a large overhead and undermining the benefit of low-precision training.
3.3
FORMAT FOR NETWORK QUANTIZATION
To verify the effectiveness of the proposed hysteresis quantization, we select 4-bit logarithmic rep- resentation as an ultra-low-precisionn format for weight parameters. This format has the same dy- namic range INT8 which is widely used for weight quantization, and is more hardware-efficient as multiplication is implemented only using simple shift operations. There have been attempts to logarithmic weights in quantized neural networks (Lee et al. 2017; Elhoushi et al.) 2021), but from-scratch training shows a significant performance degradation. In logarithmic data formats, the interval of quantization points is not uniform, making the effect of fluctuation more severe.

---

Published as a conference paper at ICLR 2022
w/o hysteresis hysteresis
wi hysteresis
Training Loss
w/ hysteresis
Qw changes
40:
Layers
Epochs
(a) Average number of Qu each updatc
changes
(b) Learning curve
Figure 5: Experimental results of hysteresis quantization.
Fig. shows experimental results of ResNet-18 training on ImageNet using 4-bit logarithmic weights. Note that we apply channel-wise quantization to convolutional layers to compensate for the insufficient expression range and layer-wise quantization to the other types of layers. Further details on the experimental setup are provided in Appendix [A.3.I] First, we measure how many quantized weights Qu change when the network performs one weight update using mini-batch and average them over the first 100 updates in the 60th epoch. The experimental result displayed in Fig. 55a) clearly shows that using hysteresis significantly reduces weight change frequency and stabilizes the training process. Fig. 5 () compares the training performance of quantization schemes with and without hysteresis. Hysteresis quantization not only speeds up training but also achieves better results at the end of training. Note that hysteresis quantization is applicable to other data formats, and additional experimental results can be found in Appendix A.4
4 EXPERIMENTAL RESULTS
Wellgtt
Erroc.
Weight
Sigmoid
tanh Sigmoid
FPB
BatchMorm
FPS
Activation...
Enon
(a) Forward path computation
(b) Backward path computation
Activation Erronia
FPB
FP32
FPB Weight Gradient
Gradient GEMM
(c) Weight gradient computation
Figure 6: Computation flow of 8-bit low-precision training.
4.1 LOW-PRECISION TRAINING SCHEME
For low-precision training, we need to quantize four variables: activation, error, weight, and weight gradient. In our experiments, we apply the quantized training scheme detailed in 2.2 to all of these variables. as depicted Fig. 6 As in previous studies on 8-bit training, the inputs of GEMM are all quantized into 8 bits. Additional functions applied to GEMM results in the forward and backward paths. ReLU, tanh, and sigmoid functions are performed directly on the input, whereas the input of NatchNorm is re-quantized.

---

Published as a conference paper at ICLR 2022
Table 1: Training Performance of FP134 Data Format
Model (Dataset) [Metric]
Baseline (FP32)
FP134
ResNet-18 (ImageNet)
69.8
69.8
ResNet-110 (ImageNet)
77.6
77.4
MobileNetV2 (ImageNet)
72.2
71.9
2-layer LSTM (PTB) [ppl.]
91.5
92.0
Transformer (IWSLT) [BLEU]
34.8
34.5
MobileNett2 + SSDLite (VOC) [mAP]
68.3
68.2
Table 2: Comparisons of Data Formats for Low-Precision Training of ResNet-18 18 on ImageNet
Formats (Exponent, Mantissa)
Top-1 Accuracy
Quantization Scheme
GEMM Batch
dw dx Input -Norm
Acc.
FP32 Proposed
Input
SWALP
Yang al.
2019a
8a
81
81
81
32
70.3
65.8
S2FP8 Cambier et al.
(0200)
(5,2)(8,23)
(5,2)
(5,2)
(5,2)
(8,23)
70.3
69.6
HFP8
[Sun et al.
2019
(4.3)
(4.3)
(6.9)
(6,9)
(5,2)
(6,9)
69.4
69.4
BM8
Fox et al.
2020)
(2,5)
(2,5)
31a
(6,9)
(4.3) 31
69.7
69.8
FP8-SEB (Park et al.
2021
(4,3)
(4.3)
(4.3)
(4.3)
(4.3) (6,23)
69.7
69.0
FP134 (Ours)
(3,4)
(3,4)
(3,4)
(3,4) (3,4) (6,23)
69.8
69.8
Fixed Point
4.2
8-BIT LOW-PRECISION TRAINING
we found that FP134 is the optimal format for low- low-precision training using the pro- posed performance prediction method. We measure the training performance of this format and compare it against other 8-bit data formats from recent studies by applying those formats to the training of various neural network models. More details on the experimental setup are provided in Appendix A.5 The performance of the proposed data format is summarized in Table | Overall, 8-bit training using FP134 achieves nearly the same performance as the full- full- precision training on all models. Even in MobileNetV2, which is known to be sensitive to quantization due to the small num- ber of parameters, only 0.3% degradation occurred. [Sun et al. (2099) show that HFP8 also exhibits only 0.2% accuracy degradation in MobileNerV2 (71.81% vs. 71.61%) but they quantize Batch- Norm input into 16 bits instead of 8 bits, roughly doubling complexity. Additionally, since the forward and backward paths employ different data formats, HFP8 is actually implemented using 9-bit MAC units in hardware (Agrawal et al. 2021). Table 2 compares the training performance of various data formats for ResNet- 18 training. The columns w, x, dw, dx, and acc refer to weight, activation, weight gradient, error, and GEMM accumulation, re- spectively. Our FP134 format exhibits no accuracy drop compared full-precision training. HFPS (Sun et al. 2019) and BM8 (Fox et al. (2020) demonstrate similar performance, but they both use higher precision represent BatchNorm inputs, and different formats adopted in the forward and backward paths, necessitating complex computation units when implemented in hardware, as decribed above. In addition. BM8 assumes block-wis. sharing of exponent bias, incurring additional overhead in memory access and data alignment. FP8-SEB (Park et al. 2021) addresses this issue by employing layer-wise exponent bias sharing and multi-way MAC units, but it results in 0.7% ac- curacy drop for ResNet-18 training. Contrarily, our data format shows no performance degradation, while deeply quantizing BatchNorm inputs into the same format and allowing for simple datapath by using identical data format in the forward and backward paths.
In Section 2.4
memory access and computational
4.3 ULTRA-LOW- TRAINING WITH 4-BIT LOGARITHMIC WEIGHTS
Elhoushi et al. (2021) recently demonstrated that 4-bit logarithmic weights could be used for net- work quantization. Fine-tuning of pre-trained model only showed 0.2% accuracy degradation, but
8

---

Published as a conference paper at ICLR 2022
Table Comparisons of Training Schemes Using 4-bit Logarithmic Weights for ResNet-18
Formats (Exponent, Mantissa)
Top-1 Accuracy
From Scratch
x, dw, dx
Acc.
FP32
Proposed
DeepShift-Q
Elhoushi et al. 2021)
X
(3,0)
69.8
69.6
(3,0)
69.8
65.3
FP134 4-bii Log W
O
(3,0)
(3,4)
(6,23)
69.8
67.7
+ Log W + Hysteresis
(3,0)
(3,4)
(6,23)
69.8
69.6
X: fine-tuning of pre-trained models, O: from-scratch training
Table 4: 4-bit Logarithmic Weight Training with and without Hysteresis Quantization.
Model (Dataset) [Metric]
FP32
4-bit Log W
4-bit Log W + Hysteresis
ResNet-18 (ImageNet)
69.8
67.7
69.6
+1.9
ResNet-101 (ImageNet)
77.6
76.6
77.3
+0.7
MobileNetV2 (ImageNet)
72.2
67.5
69.6
+2.1
2-layer LSTM (PTB) [ppl.]
91.5
93.3
93.2
-0.1"
Transformer (IWSLT) [BLEU]
34.8
33.5
33.8
+0.3
MobileNetV2 + SSDLite (VOC) [mAP]
68.3
63.3
66.1
+2.8
Lower is better
from-scratch training of the same model resulted a 4.5% accuracy drop in ResNet- training (Ta- 3. Similarly, experiments show 2.1% lower accuracy when training ResNet-18 18 using 4-bit logarithmic weights and FP134 format for other variables. However, using hysteresis quantization greatly improves the training performance and reduces accuracy degradation to 0.2%. This is iden- tical to the training performance achieved through fine-tuning a pre-trained model by EEhoushi et al.) (02T), confirming that hysteresis quantization effectively solves the issue of sub-optimal solutions in from-scratch training. In addition, Table 4 demonstrates that hysteresis quantization improves the training performance in all target models. Note that we quantized all trainable weights except for the BatchNorm parameters into 4 bits in experiments; the training performance could be further improved by using higher precision for serostvenstivv parts such as the first/last layers and residual connections.
5 CONCLUSION
In low-precision training, the dynamic range of tensor is data-dependant, and hence an optimal data format depends on various factors such as model, dataset, and quantization scheme. We showed that the training performance of a specific data format for activation and error could be predicted by observing errors introduced the weight gradients. Based on this observation, we deter- mined an optimal 8-bit format for low-precision training very efficiently without running numerous training runs. The proposed FP134 format achieved similar or better accuracy compared to prior works, while allowing for efficient hardware implementation through quantizing BatchNorm inputs and using a unified data format in both forward and backward paths. In addition, we proposed the hysteresis quantization scheme for network quantization, which improves training performance by suppressing undesired fluctuations and stabilizing the training process. In Sitra-low-precistie training with 4-bit logarithmic weights, hysteresis quantization significantly improves training per- formance by mitigating sub-optimal solutions, closely matching the performance obtained through fine-tuning pre-trained model. We expect that these schemes can complement each other to enable practical low preciiiiio training on various models and tasks.
ACKNOWLEDGMENTTS
This work was supported by the National Research Foundation of Korea (Grant No. NRF. 00221IC11106880) The EDA tool was supported by the IC Design Education Center.

---

Published as a conference paper at ICLR 2022
REFERENCES
Ankur Agrawal, Sae Kyu Lee, Joel Silberman, Matthew Ziegler, Mingu Kang, Swagath Venkatara- mani, Nianzheng Cao. Bruce Fleischer, Michael Guillom, Matthew Cohen, al. A 7nm 4-core ai chip with 25.6 tilops hybrid fp8 training, 102.4 tops int4 inference and workload-awan throttling. In 202/ IEEE International Solid-State Circuits Conference (ISSCC), volume 64, pp. 144- 146. IEEE, 2021.
Léopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oguz H Elibol, and Hanlin Tang. Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks. arXiv preprint 1xXi::000.006777 2020.
Mauro Cettolo, Jan Nichues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign. iwslt 2014. In Proceedings of the International Workshop on Spoken Language T Translation, Hanoi, Vietnam, volume 57, 2014.
Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang. Accurate and efficient 2-bit quantized neural networks. In My 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Henry Tian, and Joey Yiwei Li. Deepshift: To- wards multiplication-les neural networks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp.
2021.
Sean Fox, Seyedramin Rasoulinezhad, Julian Faraone, Philip Leong, et al. A block minifloat repre- sentation for training deep neural networks. In International Conference Learning Represen- tations, 2020.
John Gustafson and Isaac T Yonemoto. Beating floating point at its own game: Posit arithmetic. Supercomputing Frontiers and Innovations, 4(2):71-86, 2017.
Dong Han, Shengyuan Zhou, Tian Zhi, Yibo Wang, and Shaoli Liu. Float-fix: An efficient and hardware-friendly data type for deep neural network. International Journal of Parallel Program- ming, 47(3):345-355 359, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and patlern recognition, pp- 770-778, 2016.
IEEE. Ieee standard for floating-poin arithmetic, ieee std 754-2019 (revision of ieee 754-2008).
Institute of Electrical and Electronics Engineers New York, 2019.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint urXiv::905 12020, 2019.
Urs Köster, Tristan J Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William H Constable, Oguz H Elibol, Scott Gray, Stewart Hall, Luke Hornof, et al. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. arXiv preprint 1 2017.
Edward H Lee, Daisuke Miyashita, Elaina Chai, Boris Murmann, and S Simon Wong. Lognet: Energy-efficient neural networks using logarithmic computation. In 20/7 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5900-5904. IEEE, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pp. 21-37. Springer, 2016.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treehank. 1993.
10

---

Published as a conference paper at ICLR 2022
Brais Martinez, Jing Yang. Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binang convolutions. arXiv preprint arXiv:2003.11535, 2020.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort, Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Jeongwoo Park, Sunwoo Lee, and Dongsuk Jeon. A neural network training processor with 8-bit shared exponent bias floating point and multiple-way fused multiply-add trees. IEEE Journal of Solid-State Circuity, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Xiao Sun, Jungwook Choi, Chia-Y Yu Chen, Naigang Wang, Swagath Venkataramani, Xiaodong Cui, Wei Zhang, Kailash Gopalakrishnan, et al. Hybrid 8-bit floating point (hfp8) training and infer- ence for deep neural networks. 2019.
Thierry Tambe, En- Yu Yang. Zishen Wan, Yuntian Deng, Vijay Janapa Reddi, Alexander Rush, hardware co-design of adaptive floating- peating-pointt en-
David Brooks, and Gu- Yeon Wei. Algorithn-hardware codings for resilient deep learning inference. 2020 57th ACM/IEEE Design Automation Con- ference (DAC), pp. 1-6. IEEE, 2020.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yo Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Systems, pp. 7686-7695, 2018.
Yu Emma Wang, Gu- Yeon Wei, and David Brooks. Benchmarking tpu, gpu, and opu platforms for deep learning. arXiv preprint arXiv: 1907. 10701, 2019.
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. Swalp: Stochastic weight averaging in low precision training. In International Conference Machine Learning, pp. 7015-7024. PMLR, 2019a.
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian- sheng Hua. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7308-7316, 2019b.

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 VARIOUS FORMATS ANALYZED IN SECTION
In this paper, made three assumptions the quantization formats that analyzed. Firstly, 1-bit is allocated as a sign bit, so only symmetric formats are allowed, and secondly, the number of valid bits with large absolute numerical value must be greater than or equal to the number of valid bits with a small absolute numerical value. Lastly, the base does not exceed 2.
Considering the above assumptions, we provide a systematical approach for generating different quantization methods that were used for analysis in Section in order to create quantization meth- ods that have trade-offs in terms of dynamic range and the number of valid bits. The quantization method is expressed with the following items: i) list of decreasing positive real numbers P that contains the interval points (Eq. 7 and ii) non-increasing integer list L. that accompanies the in- terval list, with each item representing the number of valid bits (Eq. 8). Here, is shared exponent bias.
P= 12 2", ..... 2 p--+11 where E
(8) The quantization points Q are generated in each of the intervals that are sliced with 24.-1 evenly distributed datapoints. If the interval is (2*+1 )² the quantization point Q can be expressed by
L 16.,
where 4s E N < +224
Eq.
1
2 24 ) .... 20
2/--1
Q
24 21
21
}}
(9)
2 --
Notice that L for an a-bit quantization must satisfy
K-1 pa-1=14 Σ 2/2-1
(10)
Since the format is symmetric, only half of the data points are assigned to positive numbers, so the exponent in Eq. should be a instead of a. The reason for adding 1 is to include zero value. For example, when shared exponent bias is -1, an 8-bit fixed-point quantization would be expressed follows:
P = (2 2-,21222
2 2--,2--3 2 2 2- 2-2
(11)
L
4,,,,,,,,3,,211
(12)
datapoints, the next interval from 0.5 to 0.25 with 26-1 etc. Various cases shown in Fig. 7 with plotted the x-axis and /. plotted on the y-axis. Since P represents the range of values due to shared exponent bias that is independent of the data format, L can represent all of the various data formats consider in this paper.
The first interval from to 0.5 would be evenly sliced by 27
Numbeer va baits
Our notation
8-bit Fixed-pointt
[7,6,5,4,3,2.1]]
8-bit Logarithmic number
11111111,1111
8-blt FP143
144444444444444433222
8-bit Posit
[11133456664332.1
8-blt Float-Fix
[6.655..44433222...
Log (Power)
Figure 7: Number of valid bits in various 8-bit formats.
When selecting 8-bit formats, we chose the formats so that the intervals with less than 3 valid bits do not appear for more than two digits to reduce the search space, as they have an unnecessarily large dynamic range. Thus, formats such as [7.6.5.4,2,2.2,, were excluded from the search space. Considering all of the generation rules, we selected 166 distinct 8-bit formats with different dynamic range and valid bits from [7,6,5,443,2,1] to [3.3.3....3.2.I After the number of valid bits for an 8-bit format is selected, or is subtracted from each value to create a corresponding 7- and 6-bit for- mats. For example, in the case of (6.5.5.5.5,44444333..1 8-bit format, the 7-bit corresponding format 15.34444.4.3,333,2,,1 and 6-bit corresponding format is [4,3,3,3,3,,,,,2,,]] From the generated 166 8-bit formats, 7-bit and 6-bit formats were also generated using this rule.
12

---

Published as a conference paper at ICLR 2022
A.2 SOFTWARE IMPLEMENTATIO DETAILS
To support quantized training for various formats, custom C++ and CUDA codes to emulate quan- tized data were written. Our custom C++ and CUDA extension code could perform quantization- related functions through utilizing the Python APIs in PyTorch for extendable research while main- taining high performance. We emulate the quantized value using custom code in the part that needs quantization throughout the network, and PyTorch built-in functions are used for computation ker- nels such as convolution and matrix multiplication. We created a package named Iptorch, short for low precision PyTorch, and the code can be found in the supplementary material.
A.3 ANGLE vs. MAGNITUDE TO PREDICT PERFORMANCE
Fai
30
0.8187
Fall
1.343
007776
1,302
1217
0.7769
1:197
15
25
50
125
11
175
E|Nael
E|
E|Macl
(a) ResNet-18 (ImageNet)
(b) ResNet- 101 (ImageNet)
(c) MobileNetV2 (ImageNet)
180000
Fal
Fall
800000
69000
E|Naal
FP152
2,001
*
*
FP134
4,014
Fall
INTS
3,531
4.005
3,527
620
Z|Nacl
ZiN
E|Marl
(d) 2-yayer LSTM (PTB)
(e) Transformer (IWLST)
MobileNetV2 SSDLite (VOC)
Figure 8: Scatter plots showing Naa and |Naal of 166 8-bit quantization formats. The number next to the star represents the loss obtained through actual training.
In addition to the misalignment angles of G (<(G G + NAA) and Z(G e Gu + + NAE)), as defined in Section 2.3 we used the magnitude of noise (lN AA and NAE in order predict the final trained performance, and the results are shown in Fig. g Fig. and Fig. g show that both the error magnitude and the misalignment angle are good metrics for determining optimal data format. For the six target models, both metrics suggest FP134 as the best format. However, the misalignment angle still better captures the training performance. For instance, in Fig. [(e), although FP134 34 shows smaller noise magnitude, the actual training loss is smaller for FP143. Similarly, in Fig. bb), (c) and (f), although INT8 failed and FP152 succeeded in training, the absolute value of noise did not indicate clear superior of the two formats. Based on these observations, conclude that the misalignment angles are more suitable for predicting training performance compared against using the absolute value of noise.
A.4 HYSTERESIS QUANTIZATION WITH INTEGER WEIGHTS
In addition to 4-bit logarithmic weights, we also tested the hysteresis quantization scheme on a low- precision integer format (INT4) that uses uniform quantization. The results are shown Table 5 Experimental results show that using hysteresis improves the performance in most cases. In addition, in MobileNetV2 training with INT4 weights, training initially failed, but using hysteresis enables reliable training, which suggests that hysteresis quantization not only helps the network to reach the optimal point but also prevents divergence in an unwanted direction during the training process.
However, it is interesting for the INT4 format. We suspect that this is due to the weight distribution characteristics of the LSTM model. As shown in Fig. g most of the weights have relatively large magnitude in the
that the hysteresis quantization is less effective the LSTM model
13

---

Published as a conference paper at ICLR 2022
Table 4-bit Uniform Weight Training with and without Hysteresis Quantization.
Model (Dataset) [Metric]
FP32
INT4 W
INT4 W Hysteresis
A
ResNet- 18 (ImageNet)
69.8
61.8
67.0
+5.2
MobileNetV2 (ImageNet)
72.2
Fail
69.7
2-layer LSTM (PTB) [ppl.]
91.5
113.4
118.3
+4,91
Transformer (IWSLT) [BLEU]
34.8
32.3
32,4
+0.1
MobileNetV2 + SSDLite (VOC) [mAP]
68.3
Fail
65.7
Lower is better
0.40
2-layer LSTM
0.35
ResNet-18
0.30
Percent [%)
0.25
0.15
0.05
0.00
Base-2 Logarithm normalized absolute value
Figure 9: Weight distribution of first layer of 2-layer LSTM and ResNet-18.
LSTM model when normalized, contrary to ResNet-18 which the weights are more evenly dis- tributed. In logarithmic formats, the relative amount of quantization error is similar for all values. In contrast, the relative amount of quantization error is smaller for large values in uniform quantization. Therefore, the weight parameters of LSTM are severely affected by fluctuation in logarithmic formats, making our hysteresis quantization scheme more effective in those formats compared to uniform quantization.
A.5 EXPERIMENTAL DETAILS
A.5.1 RESNET-18 (IMAGENET)
70
Training Acc (%)
Validation Acc [96]
40
Baseline
40
Baseline
FP130 30 w/ hysteresis
FP130 w/ hysteresis
FP1 30 w/o hysteresis
30
FP130 w/a hysteresis
FP1 34 w/ Hysteresis
FP13c w/ Hysteresis
FP1 34 w/o Hysteresis
FP1 34 w/o Hysteresis
db Epochs
8b
Epochs
Figure 10: Top-1 Accuracy ImageNet using ResNet- model.
14

---

Published as a conference paper at ICLR 2022
We conducted ImageNet experiments using SGD with momentum 0.9 for 90 epochs with batch size of 256 images and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs. We used the ResNet-18 18 architecture from the official PyTorch implementation Fig. T0 shows Top-1 training & validation accuracy graphs. Observation of the training graph indicates that all of the results are close to the baseline within 0.2% with the exception of FP130 without hysteresis quantization.
A.5.2 RESNET-101 (IMAGENET)
80
70
79-
Training Acc [%]]
Validation Acc [%6]
60
80
40
Baseline
FP130 30 hysteresis
FP130 w/ hysteresis
10
FP1 30 w/o hysteresis
FP130 w/o hysteresis
FP1 34 w/o Hysteresis
10-
FP1 P4134k w/o Hysteresis
40 Epochs
20
80
Epochs
Figure Top-1 Accuracy on ImageNet using a ResNet-101 model.
We trained ResNet-101 by applying the same training method as ResNet-18. We conducted Im- ageNet experiments using SGD with momentum of 0.9 for 90 epochs with batch size of 256 images and an initial learning rate of 0.1 which is decayed by factor of 10 at the 30th and 60th epochs. We used the ResNet-101 architecture from the official PyTorch implementation" Fig. shows Top-1 training & validation accuracy graphs. Observation of the training graph indicates that all of the results are close to the baseline with less than 0.3% performance drop except for FP130 without hysteresis quantization.
A.5.3
MOBILENETV2 (IMAGENET)
70
80
Training Acc [%]]
Validation Acc [%6]
60
40
Baseline
Baseline
FP130 w/ hysteresis
FP130 w/ hysteresis
FP1 30 hysteresis
30
FP130 w/o hysteresis
29
Mysteresis
Hysteresis
FP1 Hysteresis
Hysteresis
30
100
100
200
250
50
100
150
200
250
Epochs
Epochs
Figure 12: Top-1 Accuracy on ImageNet using a MobileNet- V2 model.
We conducted ImageNet experiments using SGD with momentum of 0.9 for 270 epochs with
a batch size of 256 images and cosine annealing with an initial learning rate of 0.05. We used the MobileNetV2 architecture from the official PyTorch implementatior") Fig. 12 shows Top- training & validation accuracy graphs. Observation of the training graph indicates that FP130
15

---

Published as a conference paper at ICLR 2022
without hysteresis leads to very unstable fluctuations throughout the training. On the other hand, in FP130 with hysteresis, training is less susceptible to fluctuations and follows the baseline (FP32) training closely until the learning rate decreases toward the latter part of learning, where both FP130 with hysteresis and FP134 show some degradation from the baseline. This is seen as a limitation due to the low precision of each format.
A.5.4 2-LAYER LSTM (PTB)
480
Baseline
Baseline
FP130 w/ hysteresis
FP130 w/ hysteresis
Training Perplexity
350
FP1 30 w/o hysteresis
FP1 34 w/ Hysteresis
Validation Perplexity
FP134 w/ Hysteresis
FP1 P0030 w/o hysteresis
500
FP1 34 w/o Hysteresis
FP1 34 w/o Hysteresis
250
150
10
15
25
35
20
30
35
40
Epochs
Epochs
Figure 13: Perplexify on PTB using 2-layer LSTM model.
We adopted the 2-layer Long Short Term Memory (LSTM) network from PyTorch Examples" for language modeling the Penn Treebank dataset (Marcus al. 1993). We experiments in batches of 20 sentences with an initial learning rate of 20 which is decayed by a factor of 4 at epoch 11, 16, 26, 31 and 37. The embedding and hidden dimensions are 650 and the sequence length is 35. Fig. shows training & validation perplexity.
A.5.5 TRANSFORMER MODEL (IWLST)
We adopted the Transformer Base model from the FairSed repository the 14 German to English translation task. We used Adam optimizer and default training parameters found in the repository and trained from seratch for 25 epochs. BLEU scores were calculated using the script from the repository.
A.5.6 MOBILENETV2 + SSDLITE (VOC)
The base network MobileNetV2 which was pretrained with each format Appendix AA3.3 The entire network trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset. We used SGD with a momentum 0.9 for 200 epochs in batches of 32 images and cosine annealing with initial learning rate of 0.01. Fig. [4 shows validation loss at every 5 epochs. Even in this experiment, in the case of FP130 without hysteresis the loss fluctuates significantly. whereas in FP130 with hysteresis learning proceeds much more stably. FP134 showed similar results the baseline regardless of hysteresis quantization.
We adopted PyTorch implementation of SSDLite from the online repository
A.6 MODEL QUANTIZATION METHODS
We quantized GEMM input and batchnorm input in all quantized training experiments. Among the six models used in the experiment, the quantization details for three representative structures are shown in the Fig. 15 In each structure of figure, inputs such as h. V. K, and Q are also all quantized in 8 bits.
httprr///hhiiooolyyyyye///meeeee language model htttps//githuh.co//yyoorrr hhapss//gghhhcccmmmggaaaooonnnss
16

---

Published as a conference paper at ICLR 2022
Baseline
6.0
FP130 w/ hysteresis
FP130 w/o hysteresis
FP134 Hysteresis
FP134 w/o Hysteresis
Validation
5.0
4.5
4.0
3.5
50
100
175
200
Epochs
Figure 14: Validation loss VOC using MobileNetV2 SSDLite.
TP
Tanh
Qartt
Cony
Quant
Concel
Quart
Ounf
Quant:
Quant
Quant
Quent
Quart
SFV
Quart Lineer
Quart
Quant
h
Lineer
Linear
K
Residual
(b) LSTM
(c) Multi-head Attention
Block in ResNet
Figure 15: Quantized network structures.
A.7 HARDWARE EVALUATION
For hardware implementation comparisons, implemented conventional MAC unit and multi-way MAC unit with integer-based accumulation (Tambe et al.] 2020 Park et al. 2021) that support data formats presented Section 4.2 For accumulation, we use FP169 with chunk-based accumulation (Wang et al. 2018). Experimental results in Table 6 show that FP134 exhibits lower
Table 6: Area and Power of 2-Stage Pipelined MAC Units Synthesized in 40nm Process.
MAC
Area per MAC [um²]
Power per MAC [μW]
Structure
FP134
FP1431
HFP82
BM8
Flex
FP134
FP1431
HFP82
BM8
1
Conventional
1355
1320
1308
1460
3800
122
116
106
537
Multi-way
2-input
1335
1480
2342
1865
2268
178
178
283
258
371
4-input
888
1034
1615
1296
1885
120
135
205
184
351
8-input
678
836
1343
1074
1672
97
123
194
168
342
16-input
571
698
1065
957
1540
95
114
170
155
329
32-input
511
668
994
898
1485
87
111
170
152
326
64 14-input
509
638
955
856
1450
88
110
172
149
326
Park et al. 2021
Sun et al.
2019
3
Fox et al, (2020)
Köster et al.
2017)
17

---

Published as a conference paper at ICLR 2022
cost than FP143 and other formats in previous studies. Note that HFP8 (Sun et al. 2019) and BM8 (Fox et al. 2020) employ different formats for activation and error. Therefore, they need to be implemented in FP153 and FP145 support all operations with single MAC unit (Agrawal al. 2021]. Since Flex 16+5 (Köster et al. 2017) requires 16-bit multiplication, its cost is significantly higher than other 8-bit formats.
bit
bit
m bit
clesn
.049
sign
exp
subnoraal
a-input
XOR
+
X
adder tree
n+1 bit
2m-+2 bit
FP 169
+2m-2 bit
signed shifted number
2mt bit
FP 169
Figure 16: Multi-way MAC unit with adder tree.
conventional MAC unit consists of a multiplier and an accumulator. In the multiplier, the expo- nents of two input operands are summed while their mantissas multiplied. The multiplication part is more complex, and hence dominates the area of the multiplier. As result, the size of the multiplier is larger when more bits are allocated to mantissa. In the accumulator, a floating-point adder adds the multiplication results to partial sum in FP169. The adder is decomposed into shifter that aligns the mantissa by the exponent difference, an integer adder that sums aligned man- tissas, and a quantization unit that converts the result back to FP169. Since the result is re-quantized into F91 the addition operation of aligned mantissas does not need to lossless. FP169 format has 10-bit mantissa including one hidden bit. We only need to accurately calculate higher 10 bits, which necessitates a 12-bit adder considering rounding. Shifting by more than 12 bits not needed even if the result of the multiplier has larger exponent range. Therefore, the shifter, adder, and quantization unit, which are the components of the accumulator, are not affected by the input for- mat. There minor differences such as adder that calculates the difference between exponents and shifter with a different bit width of the input, but their costs are ignorable.
Contrarily, multi-way MAC consists of multiplier, a shifter for alignment, adder tree, malization unit, and a final accumulator. The multiplier and the final accumulator are identical to those of the conventional MAC. However, since only one normalization unit and one final accumu- lator are shared across multiple inputs, their implementation cost becomes insignificant for a larger number of inputs. The shifter for alignment converts the multiplier output to integer format since the cost of integer addition is lower than that of floating- point addition. Then, the adder tree sums those integer values, and the normalization unit converts the result back to a floating- point format. The cost of the shifter for alignment, adder tree, and normalization unit is all determined by the integer bit width, and the larger the exponent range of the input operands, the larger the required bit width, shown in Fig. [6 In FP134, FP143, and FP152, the minimum integer bit widths 23, 37, and 67 bits, respectively. Since the bit width is sufficiently large, the cost difference of these units exceeds the cost difference of the multiplier. Therefore, the cost of multi-way MAC increases with the number of exponent bits.
When designing neural network training processor, some parts of the hardware (e.g., batch normal- ization, non-linear activation functions such as tanh and sigmoid. and softmax function) are typically implemented with higher precision to avoid performance drop. Hence, need to consider data for-
18

---

Published as a conference paper at ICLR 2022
Table Area and Power of Format Converting Units Synthesized in 40nm Process.
Area [um²]
Power [uW]
Direction
FP134
FP143
HFP8'
BM8 Flex16+5'
FP134
FP143
HFP8'
BM8 Flexl6+5*
To FP32
155
141
145
176
330
28
26
27
30
53
From FP32
139
144
152
162
427
19
20
22
23
55
Sun al. (2019)
Fox et al.
(2020)
Köster et al. (2017)
Table FPGA Implementation Results of 2-Stage Pipelined MAC Units
MAC
FP134
FP143'
HFP82
BM83
Flex
Structure
LUT
FF
LUT
FF
LUT
FF
LUT
FF
LUT
FF
Conventional
284
35
269
33
273
33
304
35
606
69
Multi-way
2-input
499
62
648
76
990
108
755
80
891
70
4-input
719
63
958
77
1589
109
1193
81
1499
71
8-input
1141
64
1503
2681
110
2013
82
2664
16-input
1997
6.5
2603
79
4632
110
3516
83
4955
73
32-input
3526
66
4758
80
8614
112
6591
84
9500
74
64-input
6713
67
9074
81
16289
113
12724
85
18649
75
Park et al. (2021)
Sun et al.
(2019)
3
Fox et al.
(2020)
Köster et al. (2017)
mat conversion overheads when comparing different formats. If we consider various 8-bit data for- mats with different representation methods, as we did in Table and assume that computations other than MAC operations are implemented full precision, the processing architecture (except MAC units) will be identical for all formats. In addition, the on/off-chip memory space, control logics, and on-chip interconnects will remain the same. The only difference would be the low- pww-preciiion MAC units and the data conversion units between full-precising and low-precision formats. However, the cost of conversion between low low-precision and high-precision floating point formats is typically very low and does vary much with the low-precision format. For low-precision high-precision conversion, we only have to add a bias-correction term to the exponent and add after the mantissa. For high-precision
low-precision conversion, we need add bias-correction term to the expo- nent, clamp the overflowed value to the maximum, and round off the mantissa. The cost is very low compared the MAC operation, and the cost difference between different low- now-precision formats is negligible. We have synthesized the conversion units for different formats, and their costs are presented in Table 7 The experimental results confirm that the overhead of data format conversion significantly lower than MAC operations. In addition, all formats except Flexpoint exhibit similar conversion costs.
In addition to the synthesis result for ASIC implementation in Table 6 measured the hardware overhead of MAC units different data formats on FPGA. Table shows the synthesis results on Xilinx Artix-7 FPGA (XC7A100TCSG32411 (BRAMs), and we used a compiler directive to avoid using DSP modules for fair comparisons. Table & shows similar trend to Table 6 the cost of one MAC gradually decreases the number of inputs increases the multi-way MAC. Also, due to integer-based addition in the adder tree, the cost of FP134, which has the smallest dynamic range, exhibits lower costs than the other formats.
Those MAC units do not need block RAMs
A.8 EFFECT OF QUANTIZATION NOISE ON DATA FLOW QUANTIZATION
Table 9 shows the training results when both activation and error are quantized in various data for- mats. If an appropriate amount of noise is introduced in the network during training, it will increase the training loss but reduce the validation loss, suggesting that the model has been improved due to the regularization effect. However, if the noise level continues increase, the model's 's perfor- mance will start to degrade at some point. For instance, when MobileNetV2 quantized in FP134, its performance improved through the regularization effect since the training loss increases while
19

---

Published as a conference paper at ICLR 2022
Table 9: Training Result Comparisons
FP32
FP134
FP143
FP152
Model (Dataset)
train
val
train
val
train
val
train
val
ResNet-18
Loss
1.286
1.220
1.292
1.223
1.302
1.233
1.343
1.261
(ImageNet)
Accuracy
69.50
69.73
69.46
69.67
69.19
69.63
68.27
68.93
MobileNetV2 (ImageNet)
Loss
1.182
1.118
1.197
1.113
1.217
1.133
1.301
1.192
Accuracy
71.47
72.20
71.16
72.25
70.63
71.92
68.70
70.35
2-layer LSTM
Loss
3.994
4.517
4.014
4.525
4.005
4.518
4.037
4.535
(PTB)
ppl.
54.27
91.54
55.39
92.30
54.89
91.68
56.63
93.20
Transformer
Loss
3.498
3.847
3.531
3.885
3.527
3.887
3.621
3,942
(IWSLT)
BLEU
34.75
34.77
34.6
33.79
MobileNetV2
Loss
2.743
2.773
2.801
2.970
+SSDLite(VOC)
mAP
68.34
68.16
68.32
66.56
the validation loss decreases compared FP32. However, both the training and validation losses increase when quantized in most cases, resulting in lower accuracy. This suggests that using very low precision data format already introduces large amount of noise the network, incurring per- formance degradation. Hence, it is necessary to reduce error in the network to improve the training performance low-precision training.
20


Reviews:
[{"review_id": "wcxdfjNpEpK", "reviewer": "Reviewer_mPme", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Contribution:**\n\n1.\tThe authors propose a metric to evaluate the performance of different numeric formats. Using the proposed metric, the authors further find an optimal 8-bit numeric format suitable for various models.\n\n2.\tThe authors find that the performance degradation of 8-bit training is due to the fluctuation issue of quantized weights. To solve this, the authors propose a hysteresis quantization scheme to improve the performance of from-scratch training.\n\n3.\tExperiments on 8-bit and 4-bit training show the promising performance of the proposed method. \n\n**Questions and points needed to be improved:**\n\n1.\tIn Figure 2, the improvement of spearman’s correlation of the proposed metric over the magnitude of the error is marginal (0.9283 vs. 0.9215). It seems that the magnitude of the error is a good metric to measure the performance degradation. What are the advantages of the proposed metric? More explanations and results are required.\n\n2.\tIn Section 3.1, the authors state that the amount of change in the quantized weight due to the fluctuation is not necessarily proportional to the weight gradient. To mitigate the fluctuation issue above, the authors propose hysteresis quantization scheme. However, Figure 5 can not show the effect of hysteresis quantization. The number of $Q_w$ changes is the same in Figure 5(a) and Figure 5(b). More explanations are required.\n\n3.\tThe idea of changing the rounding function in network quantization is similar to AdaRound[1]. It would be better for the authors to add more discussions on the difference between the proposed method and AdaRound.\n\n4.\tIn Table 4, many notations are unclear. What do “X” and “O” denote? What do dw and x denote? More explanations are required. \n\n**Reference:**\n\n[1] Up or Down? Adaptive Rounding for Post-Training Quantization. ICML 2020.\n\n", "summary_of_the_paper": "The authors propose a method to predict the performance of different numeric formats, which allows determining the optimal data format for various neural network architectures, datasets, and tasks efficiently. By comparing 498 formats in total, the authors find an optimal 8-bit format suitable for various models. To improve the performance of from-scratch training, the authors further propose hysteresis quantization to mitigate the ﬂuctuation issue. Experiments on 8-bit and 4-bit training demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Experiments show promising performance. However, there are still some concerns regarding the proposed method. Some notations in the experiments are not clear. More explanations are required.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "izx4xR2vRfP", "reviewer": "Reviewer_AC1G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While there are few nitpicks I have (these should be fixed prior to publishing), the ideas in the paper are well-founded and useful.  I believe there is a meaningful contribution to the field here.\n\nIn the intro, there really needs to be a citation for Koster et al (2017) on Flexpoint. That was one of the first papers discussing these issues and one of first implementations of different numerics in modern hardware.\n\nIn the second paragraph of the introduction, the line \"When a value is represented using a fixed number of bits, there is a trade-off between dynamic range and precision\" is not quite correct.  This is only true for floating-point formats, so simply stating that will clear this up.\n\nIn the \"Performance degradation in from-scratch training\" section: The line \"the neural network is trained from scratch while all the values in the network - not only parameters, but also other variables in the network (e.g., activation, error, and weight gradient) \" is not necessarily true and different components are done differently even in this paper. I'm not sure why there is so much text devoted to differentiating quantized vs from-scratch training, but I think this could removed.  The paper is about from-scratch training so just state that.\n\nUnder Numeric Formats, I'm not sure what a \"symmetric\" format is and why 2's complement is asymmetric.  Please define or remove.\n\nUnder Effect of quantized activation, it says \"...suggesting that the misalignment angle is a better metric to predict the ranking of various formats based on the training performance.\" I think this is true, but more due to the shape of the curve rather than the (small) effect on spearman correlations.  It might be clearer to state this; the angle differences are high for even small losses.\n\nThe hysteresis method has some great results and might really be the key to unlocking lower precision.  Great work!", "summary_of_the_paper": "The authors propose 2 impactful methods to aid in the design of numeric precision for neural network training: a method to quickly determine which formats work for weights and activations using angular deviation of gradients between low precision and FP32, and a hysteresis method for dealing with low precision representations.  ", "summary_of_the_review": "The paper presents 2 very important ideas that I believe are of high utility to low precision efforts. The idea of weight hysteresis could actually spawn many new methods that lead to better hardware going forward.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_2uFXb0fLV", "reviewer": "Reviewer_mUcf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros.\n1. This paper addresses a very relevant topic. Though model compression has been widely discussed in the recent literature, efficient training with quantized networks is an approach worth exploring.\n2. The technique part of this paper is simple and easy to follow. It is clear that the Hysteresis setting contributes to a consistent improvement over the standard approach on both image recognition and NLP tasks.\n\ncons.\n1. There are some related works missing in comparisons such as Flexpoint [1] and HFINT [2]. Especially, HFINT first notices the difference in weights distribution between CNN and NLP models/tasks then presents an adaptive floating-point format with hardware implementations. Compared with the static FP143/FP134 mode used in the current draft, they use an on-the-fly adaptive format. In light of this, it would be better to include a HARDWARE EVALUATION section to verify the effectiveness of the proposed \"data flow quantization\".\n2. Please denote \"weight gradients\" as $G_{W}$ or $G^{W}$. $WG$ commonly refers to matrix multiplication.\n3. Figure 2: How to distinguish 8-,7-,6-bit quantization formats? \n4. It seems that Figure 8 (in A.3) has almost the same tendency as Figure 2. In general, F134 performs the best. Why introduce a more complicated metric to determine the optimal format? Note that the authors conduct all experiments under the setting of FP134.\n4. How to obtain $WG+N_{\\Delta E}$ and $WG+N_{\\Delta A}$?\n5. Table1: Why does \"1-input\" consume a much larger area than conventional MAC? Besides, why conventional MAC consumes a smaller area along with the decrease of mantissa, yet Multi-way MAC behaves in the opposite way? I expect more results on LUTs, DSPs, BRAMs, and Power except for Area.\n6. Is hysteresis quantization compatible with uniform quantization?\n7. Does FP134 still involve a scaling factor $s$ during training?\n\nref.\n* [1] Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks. NIPS2017\n* [2] Algorithm-Hardware Co-Design of Adaptive Floating-Point Encodings for Resilient Deep Learning Inference. DAC2020\n\nsupplementary:\n1. Please include a **README** file in the supplementary material. \n2. How to understand the magic number $555555543210$ used for \"data format for data flow quantization\"?", "summary_of_the_paper": "The submission starts from an interesting point that various quantized training environments may require different formats for training accurate deep neural networks. They present a metric based on the misalignment between $\\frac{\\partial \\ell}{\\partial w}+noise$ and $\\frac{\\partial \\ell}{\\partial w}$ to determine the optimal format. To mitigate the fluctuation issue caused by network quantization, they propose a hysteresis quantization scheme to avoid frequent changes of quantized points. The experiment results fully support the effectiveness of the proposed methods.", "summary_of_the_review": "Overall, the idea of hysteresis quantization seems neat and new. However, the effectiveness of the performance indicator needs to be further clarified. The empirical success of FP130 with hysteresis seems to further weaken the motivation of finding the optimal data format. \n\nThe authors are encouraged to address the weakness. I will consider changing the final rating according to the authors' feedback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OgeU5637J9h", "reviewer": "Reviewer_i9MY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Introduction discusses the difference between quantized models (after fine-tuning a pre-trained model) and from-scratch training for quantization (to gain speed-up for training) clearly and the direction of research to investigate why from-scratch training methods show increased degradation in accuracy is certainly important.\n\nUnfortunately, this reviewer finds the following serious concerns in this paper.\n\n(1) The authors argue that measuring the misalignment angle is better than the magnitude of the error. But in Section 2.3, it is not clear why the noise in the opposite direction is harmful. Similar to stochastic variation, noise on gradients can show some benefits if the amount of error is right such that regularization effects can be obtained. Moreover, as shown in Figure 2, reducing the magnitude of error also tends to present the training loss change. Is it difficult or impossible to find FP134 as an optimal one with magnitude-based error measurement? Supporting data to validate the claim (that the angle is better than the magnitude) needs to be provided.\n\n(2) This paper seems to suggest a design methodology of NPU in the form of ASIC. Then, the authors need to prove that FP134 can be applied to a wide range of models. Such a particular format would show significant accuracy degradation if model size increases. What would be the limit of such a format? Moreover, the authors show the superiority of FP134 using ResNet-18 model while a few additional experimental results using some simple Transformers, such a strong argument to choose FP134 as an optimal one needs to consider what kind of limitations would be provided. For example, schemes in Table 3 might not be good for ResNet-18 but may be good for ResNet-101. This reviewer is not sure whether FP134 is a customized one for small models such as ResNet-18.\n\n(3) A quantization technique using Hysteresis is interesting. But more detailed discussions and theories why Hysteresis is important for 4-bit Log W need to be included. Is Hysteresis generally helpful for other quantization formats and larger models as well?\n\n", "summary_of_the_paper": "This paper proposes a method to find an optimal quantization format based on error angle estimation and hardware overhead. The authors also present an hysteresis-based quantization method to reduce fluctuation of exponent values such that training (from the scratch) using only 4-bit weights can result in negligible amount of accuracy degradation for ResNet-18 on ImageNet. Experimental results are provided for ResNet-18, MobileNetv2, 2-layer LSTM, Transformer, and MobileNetV2+SSDLite. For 8-bit quantization, FP134 is chosen and such quantization is also applied to BatchNorm layers to reduce memory consumption.", "summary_of_the_review": "Overall, even though this reviewer finds some interesting ideas including Hysteresis and experimental results are good for ResNet-18, the followings need to be addressed.\n\n(a) FP134 cannot be found by measuring the magnitude of the error?\n\n(b) Do the authors suggest FP134 as a format to be applied to a wide range of models (especially even with large model size?). If so, please provide supporting data and theories.\n\n(c) Most experimental data are only for ResNet-18.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method to determine which precision to use for the weights, as well as a quantisation method using hysteresis to improve performance with low-precision weights, including 4-bits.
Reviewers tend to agree that the two points presented are useful and can have a large impact on the field.
Generally, reviewers pointed out that motivations, notations and experimental studies could be improved. This has been partly addressed by the authors.
I recommend to accept this paper for ICLR 2022.
Decision: Accept (Poster)