Exploring  the Potential of Low-Bit Training of Convolutional Neural Networks
Authors: Kai Zhong
Xuefei Ning
Tianchen Zhao
Zhenhua Zhu
Shulin Zeng
Guohao Dai
Yu Wang
Huazhong Yang
Venue:  (2021)
Keywords: CNN
training
quantization
low-bit
energy efficiency

Abstract: In this paper, we propose a low-bit training framework for convolutional neural networks. Our framework focuses on reducing the energy and time consumption of convolution kernels, by quantizing all the convolutional operands (activation, weight, and error) to low bit-width.  Specifically, we propose a multi-level scaling (MLS) tensor format,  in which the element-wise bit-width can be largely reduced to simplify floating-point computations to nearly fixed-point. Then, we describe the dynamic quantization and the low-bit tensor convolution arithmetic to efficiently leverage the MLS tensor format. Experiments show that our framework achieves a superior trade-off between the accuracy and the bit-width than previous methods. When training ResNet-20 on CIFAR-10, 
all convolution operands can be quantized to 1-bit mantissa and 2-bit exponent, while retaining the same accuracy as the full-precision training. When training ResNet-18 on ImageNet, with 4-bit mantissa and 2-bit exponent, our framework can achieve an accuracy loss of less than $1\%$. Energy consumption analysis shows that our design can achieve over $6.8\times$ higher energy efficiency than training with floating-point arithmetic.

Under review as a conference paper at ICLR 2021
EXPLORING THE POTENTIAL OF LOW-BIT TRAINING OF CONVOLUTIONAL NEURAL NETWORKS
A Anonymous authors
Paper under double-bling review
ABSTRACT
In this paper, we propose a low-bit training framework for convolutional neural networks. Our framework focuses on reducing the energy and time consumption of convolution kernels, by quantizing all the convolutional operands (activation, weight, and error) to low bit-width. Specifically, we propose multi-level scaling (MLS) tensor format, in which the glement-wisst bit-width can be largely reduced to simplify floating-point computations to nearly fixed-point. Then, we describe the dynamic quantization and the low-bit tensor convolution arithmetic to efficiently leverage the MLS tensor format. Experiments show that our framework achieves a superior trade-off between the accuracy and the bit- width than previous methods. When training ResNet-20 on CIFAR-10, all convolution operands can be quantized 1-bit mantissa and 2-bit exponent, while retaining the same accuracy the full-precision training. When training ResNet-18 on ImageNet, with 4-bit mantissa and 2-bit exponent, our framework can achieve an accuracy loss of less than 1 1%. Energy consumption analysis shows that our design can achieve over 6.8x higher energy efficiency than training with floating point arithmetic.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved state-of-the-ard performance in many computer vision tasks, such as image classification (Krizhevsky et al. (012) and object detection (Redmon al. 2016; Liu et al.) 2016). However, deep CNNs are both computation and storage-intensive. The training process could consume up to hundreds of ExaFI.OPs of computations and tens of GBytes of storage (Simonyan & Zisserman 2014), thus posing a tremendous challenge for training in resource-constraing environments. At present, the most common training method is to use GPUs, but it consumes much energy. The power of a running GPU is about 250W, and it usually takes more than 10 GPU-days to train one CNN model on ImageNet (Deng et al. 2009). It makes Al applications expensive and not environment-free friendll.
Table 1: The number of different operations in the training process (batch size = 1). Abbreviations: "EW-Add": element-wise addition, "Ir": forward pass; "B": backward pass.
Op Name
Op Type
ResNet18 (ImageNet)
ResNer20 (CIFAR-10)
Conv (F)
Mul&Add
2.72E+10
4,05E+07
Conv (B)
Add
5,44E+10
8.11E+07
BN (F)
Mul&Add
3,01E+07
1.88E+05
BN (B)
Mul&
3.01E+07
1.88E+05
EW-Add (F)
Add
1.49E+07
7.37E+04
EW-Add (B)
Add
1.20E+07
7.37E+04
Weight Update (B)
Add
1.12E+07
2.68E+05
Reducing the precision of NNs has drawn great attention since it can reduce both the storage and computational complexity. It is pointed out that the power consumption and circuit area fixed-point multiplication and addition units greatly reduced compared with floating point ones (Horowitz) 2014). Many studies (Jacob et al. 2017a Dong et al. 2019 Banner et al.) 2018b) focus on amending the training process acquire seduced-precissee model with higher inference efficiency.
boating-poont

---

Under review as a conference paper at ICLR 2021
Besides the studies improving inference efficiency, there exist studies that accelerate the training process. Wang et al. (2018) and Sun et al. (2019) reduce the floating- point bit-width to during the training process. Wu et al. (2018) implements full-integer training procedure to reduce the cost but fails to get acceptable performance.
As shown in Tab. Conv in the training process accounts for the majority of the operations. Therefore, this work aims at simplifying convolution to low-bit operations, while retaining a similar performance with the full-precision baseline. The contributions of this paper are:
This paper proposes a low-bit training framework to improve the energy efficiency of CNN training. We design low-bit tensor format with multi-level scaling (MI. format), which can strike a better trade-off between the accuracy and bit-width, while taking the hardware efficiency into consideration. The multi-level scaling technique extracts the common exponent of tensor elements as much as possible to reduce the element-wise bit- width, thus improving the energy efficiency. To leverage the MLS format efficiently, we develop the corresponding dynamic quantization and the MLS tensor convolution arithmetic.
2. Extensive experiments demonstrate the effectiveness of our low-bit training framework. One only needs 1-bit mantissa and 2-bit exponent to train ResNet-20 on CIFAR-I0 10 while retaining the same accuracy the full-precision training. On ImageNet, using 4-bit mantissa and 2-bit exponent is enough for training ResNet-18. with precision loss within 1%. Our method achieves higher energy efficiency using fewer bits than previous floating-poin training methods and better accuracy than previous fixed-point training methods.
3. We estimate the hardware energy that implements the MLS convolution arithmetic. Using our MLS tensor format, the energy efficiency convolution can be improved by over 6.8x, than the full- full-precision training, and over 1.2x than previous low-bit training methods.
2 RELATED WORK
2.1
QUANTIZATION
Earlier quantization methods like Han et al. 2015) focused on the post-training quantization, and quantized the pre-trained full-precision model using the codebook generated by clustering other criteria (e.g., SQNR Lin et al. (015), entropy Park et al. (2017)) [Banner et al. (2018b) selected the quantization width and clipping value for each channel through the analytical investigation. Jacob et al.) (20176) developed an integer arithmetic convolution for efficient inference, but it's hard to be
used training because the scale of the output tensor should be known before calculation. These quantization methods need pretrained models, and cannot accelerate the training process.
2.2 QUANTIZE- AWARE TRAINING
Quantize-awaad aware training considered quantization effects in the training process. Some methods trained
an ultra low-bit network like binary (Rastegari et al., (216)) or ternary (Li et al. 2016 networks, with a layer- wise scaling factor. Despite that the tollow-up studies (Liu al. 2020 Qin et al.) 2019) have been proposing training techniques to improve the performance of binary networks, the extremely low bit-width still causes notable performance degradation. Other methods sought to retain the accuracy with relatively higher precision, such 8-bit (Jacob et al., 2017a). Gysel al. (2018) developed a GaU-based training framework to get dynamic fixed point models. These methods focus on accelerating the inference process and the training process is still using floating-pooi operations.
2.3 LOw-BIT TRAINING
To accelerate the training process, studies have been focusing on design a better floating foating-point data format. Dillon al. (2017) proposed novel 16-bit floating-point format that is more suitable for
CNN training, while Köster et al. (2017) proposed the Flexpoint that contains 16-bit mantissa and
5-bit tensor-shared exponent (scale), which is similar to the dynamic fixed-point format proposed by
Gysel et al. (018). Recently, 8-bit floating-pon point (Wang et al. 2018 (Sun et al. 2019) was used with chunk-based accumulation and hybrid format to solve swamping.
2

---

Under review as a conference paper at ICLR 2021
Tensor Scale
Man,
Group Scale
Group Scale
Group Scale
ExPg
Expg
M
ExPg
M
Expy Many
Expx
Man,
Expy
Man,
Expx Many
S
Expx Group 1
Many
Expy Group N
Many
Group
Figure 1: Illustration of the multi-level scaling (MLS) low-bit tensor data format.
Some studies used fxxedpooin in both the forward and backward processes (Zhou al., 2016 Wu et al. 2018): Yang et al. (2020) implemented full-integer training framework for artthmeeti machines. However, their methods caused notable accuracy degradation. [Banner et al. (201 8a) used 8-bit and 16-bit quantization based integer arithmetic (Jacob et al., 2017b) to achieve comparable accuracy with the full-precision baseline. But it's not very suitable for training as we discussed earlier. These methods reduced both the training and inference costs. In this paper, we seek to strike better trade-off between accuracy and bit-width.
3 MULIT-LEVEL SCALING LOW-BIT TENSOR FORMAT
In this paper, we denote the filters and feature map convolution operation as weight (W) and activation (A), respectively. In the back-propagations the gradients of feature map and weights are denoted as error (E) and gradient (G), respectively.
3.1 MAPPING FORMULA OF THE QUANTIZATION SCHEME
the fixed-point representation. In commonly used scheme (Jacob et al. (017b), the mapping function is floal scale x (Fix+ Bias), in which scale and Bias are shared one tensor. However, since data distribution changes during training, one cannot simplify the Bias calculation as Jacob et al. (2017b) did. Thus, we adopt an unbiased quantization scheme.an extend the scaling factor to three levels for better representation ability. The mapping formula of our quantization scheme is
In quantized CNNs, floating-point values quantized
time
X I S.li.j,k. k. S, x Syli.fl [00xxxii,.e Ai.
where [-] denotes the indexing operation, S, is a sign tensor, S, is tensor-wise scaling factor shared one tensor, and Sy a group-wiss wise scaling factor shared in one group, which is structured subset of the tensor. Our paper considers three grouping dimensions: 1) grouping by the 1-st dimension of tensor, 2) the 2-nd dimension of tensor, or 3) the and the 2-nd dimensions simultaneously. S., Sg. and X use the same format, which we refer to as (E, M), customized floating-point format with E-bit exponent and M-bit mantissa (no sign bit). A value in the format M) is
(1+
Man
float [MFn Fxp)) Frac 2 2-Eep
2 2-p
2M
where Man and Exp are the M-bit mantissa and E-bit exponent, and Frac [1,2) is fraction.
3.2 DETAILS ON THE SCALING FACTORS
The first level tensor-wise scaling factor S. is set to an ordinary floating-point number (EE,, M.) = (8, 23)). which is the same unquantized data in training to retain the precision as much as possible. Considering the actual hardware implementation cost, there are some restric- tions the second level group-wise scaling factor Sg. Since calculation results using different tensor groups need to be aggregated, using S, in an ordinary floating point format will make more ex expensive conversions and operations necessary in the hardware implementation. We proposed two
3

---

Under review as a conference paper at ICLR 2021
hardware-frin -freanddrrr group-wis wise scaling scheme, whose formats can be denoted (Eg. 0), and (Eg. 1). The scaling factor in the (Eg. 0) format is simply a power of two, which can be implemented easily shifting on the hardware. From Eq. 2 S I2F (Mag. Erpy) value in the (Eg. format can be written as
S = - Mang 2
2 Eaps +2 Bxpg- -1 Mong = 2-Epp
x 2 Exps
(3)
Mang = 0
which is of two shifting, and be implemented with small hardware overhead.
The third level scaling factor S₂ = T2F Exp=) = 2 Expx is the element-wise exponent in X = (1 + Mana), ). and we can see that the elements The specific values of Ex and M, determine the type and the cost of the basic multiplication and accumulation (MAC) operation, which will be discussed later in Sec. 5.2
in Eq. are in a (Ez, M2) format.
Forward
[Low]
[High]
[High]
Lowl
Activationn-1
BN
Q
Activation;
ReLU
Weight, Buffer
Weight,
Backward
Error
BN
Error
Mask
Low-bit Cone 4.2)
Dynamic
Gradient
High bit Ops
Gradienty
SGD
Low-blt Data
High-bit Data
Figure 2: Computation flow of the proposed low-bit training.
4 LOW-BIT TRAINING FRAMEWORK CNN
As shown in Fig. the convolution operation (Conv) followed by batch normalization (BN), nonlinear operations (e.g. ReLU, pooling). Since Convs account for the majority computational cost, we apply quantization right before Convs the training process, including three types: Conv(W, A). Conv(W, H), and Conv(A, E). Note that the output data of Convs is in floating-point format, and other operations operate on floating-pott summarized in Appendix Alg. 2, in which the major differences from the vanilla training process are the dynamic quantizaiton procedure DynamicQ Suaiiiicciaatiziiiinn and the low-bit tensor convolution arithmetic LowbitConv.
numbers. An iteration of this low-bit training process is
4.1 DYNAMIC QUANTIZATION
The dynamic quantization converts dlatting-oinn tensors to MLS tensors by calculating the scaling
factors S., S, Sg and the elements X, as shown in Alg. ] Exponent(.) and Fraction (-) are to obtain the exponent (an integer) and fraction (a fraction E [0, 1)) of a floating-point number. While calculating the quantized elements X, we adopt the stochastic rounding (Gupta et al. 2015). and implement it by using a uniformly distributed random tensor r ~ U[-
[x]
with probability x - [x]
StochasticRound(1., r) = Roundzt + +
(4)
with probability [ar] x
Note that Alg. describes how simulate the dynamic quantization process using floating point arithmetic. While the hardware design, the exponent and mantissa are obtained directly, and the clip/quantize operations are done by taking out some bits from machine number.
4

---

Under review as a conference paper at ICLR 2021
Algorithm I: Software-simulated dynamic quantization process
Input: X: float 4-d tensor; Azis: grouping dimension; R: UI 2: 2
11 distributed random tensor;
(Eg. My): format of group-wise scaling factors; (Frr 22): format of each element
Output: Sg: sign tensor; Sy: tensor group-wise scaling factors; Sgi group-wise scaling factors;
X: quantized elements
calculating scaling factors */
S Sign(X X)
Sr (abss (X), axis = Aris)
Group-wise maximum magnitude I/ Tensor-wiss maximun magnitude // Group-wiie scaling factors before quantization <
St Mar(S,,)
Sgf = S
Ex Fracy Exponent(S Fraction(Sgf)
E Clipp Expp, 28,
II Clip Exp, to Eig bits Il Quantize Fracy My bits
Fracy Fracg 2M²) 2
Sg Fracg BBpps
I/ Group-wise scaling factors after quantization
8 calculating elements */
, Abs(() + Sg + St Expx; Fracz Expoment(Xx)) Fraction((X))))
I/ Dividing the scaling factors
1 Quantize Frac, to Mr bits with underflow handling #/ 2
11 Exmin
12 Fracas = Fracæ x MM if not underflow, else Fracæ x 2d,, - Eumin -
13 Fraczint = PTip(StochasticRol
Fraczs: R)). 0, 0,2 - 1)
14 Fracz Fraczint x 2- if not underflow, else Fraczint x 2 M M Eamin
15 Expa = Clip( Exp2; Eamin.
16 X = Fracz x Fmpp
I/ Elements after quantization
17 Return Sx. St, Sg. X
t: The underflow handling follows the IEEE 754 standard (Hough 2019).
4.2 LOW-BIT TENSOR CONVOLUTION ARITHMETIC
In this section, we describe how to do convolution with two low-bit MLS tensors. Denoting the input channel number as and the kernel size as K, the original formula of convolution in training is:
C- C-1K-1K-
Z co, x. -
W(co, ci, i.
+i,
(S)
We take Conv(W, A) as the example to describe the low-bit convolution arithmetic, and the other two types of convolution can be implemented similarly. Using the MLS data format and denoting the corresponding values (scaling factors S, exponents Exp, fractions Frac) of W and A by the superscript (w) and (4), one output element Z [n. c0, x, v) of Cono(W, A) can calculated as:
C-IK-1K-1
Z [, Co, x,
Σ
(s(w) s(")) [co, ci W[co ci, i,), (s S(" (n, ci Ai,, i +i
di=0
1=
K-1K-1
(s(a) ((a)
s(())
((()) (co, ci ( (n, mi)
Σ
W c0,, ci, ii, ci, +i,+
=((() s(p) cf,
ci=0
(6)
Eq. 6 shows that the accumulation consist of intra-group MACs that calculates P(n, co, ci) and enter-group MACs that calculates Z. And the intra-group calculation of P(n, co, ci] is:
K-1
co, con = Σ
(Frac(w) (co, ii Frac(()) (n, ci, i.jl) 2 (app Eap (p.ci..))
0
(7)
5

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison of low-bit training methods CIFAR-10 and ImageNet. Single number in the bit-width stands for Mg, the corresponding Eg is O. "f" indicates that FP numbers are used.
Dataset
Method
Bit- Width (W/A/E/Acc)
Model
Accuracy
Baseline
Banner al. 2018a
112-
ResNet-20
81.5%
90.36%
(Wu et al. 2018
288 32
VGG-like
93.2%
94.1%
Rastegari et 2016
f32 f32
ConvNet
89.83%
91.8%
CIFAR-10
444 16
ResNet-20
92.32%
92.45%
222 16
ResNet-20
90.39%
92.45%
Ours
(1, 2)
(1, 2) (1,2) 16
ResNet-20
91.48%
92.45%
(2,
(2 (2, 1) 16
ResNet-20
91.97%
92.45%
iZoou et al.
2016)
8 32
AlexNet
53.0%
55.9%
Wu et
2018)
2 8 32
AlexNet
48.4%
56.0%
Yang et al.
2020
8 8 32
ResNet-18
64.8%
68.7%
Banner et al.
2018a
8 f32
ResNet-18
66.4%
67.0%
(unn et al.
2019
(5,3)
(5, 3) (5,3) f32
ResNet-18 69.0%
69.3%
ImageNet
8 8 8 32
ResNet-18
68.5%
69.1%
666 16
ResNet-18 67.6%
69.1%
4 444 16
ResNet-18 66.5%
69.1%
Ours
(1.6)
(1.6) (1.6) 16
ResNet-18 68.0%
69.1%
(2 5 (, (2.5) 32
ResNet-18
68.3%
69.1%
22, 4)
(2,4)
(2,
42 32
R8sNet-18 68.2%
69.1%
where Frac, Exp fractions and exponents, whose precision is (M, 1)-bits and Ei-bits, respectively. Thus the intra-group calculation contains (M₂ + 1)-bit multiplication, (2 -ii shifting, and the (2.M, + 2 2)-bit integer results accumulated with enough bit-width to some previous work (Wang al. 2018 Sun et al. 2019), since they use E, 5. As for the inter-group calculation, each element in ((P) is (, 2) number obtained by multiplying two (E, 1) numbers. Omitting the index for simplicity, the calculation can be written as:
get the partial sum P. And the accumulator has to be Koating-poin
C-1
Z 100, 2]
Σ
( |co, P(c,
c1=0
P Pix,cil2-Eaap
(xp(2) -xlp- [oo,
if Man (p) (o,, ei 00
(8)
P(z, cil2 Emp ci
+ P [[r, cil2 Emp Emp(e) (10,
if Man (p) cco, ci 01/10
cim0
[co,ct] + P(r, cil2 Bap if Man (p) c0, cij = 11 Plz, [ju.cc]22-Eapp (ap
Due the special format of ((P), the calculation of the floating-point Z be implemented efficiently on hardware, in which no floating-pone point multiplication is involved.
5 EXPERIMENTS
We train ResNet (He et al. 2016) on CIFAR-10 (Krizhevsky) 2010) and ImageNet (Deng et al. 2009) with our low-bit training framework. We experiment with the MLS tensor formats using different (E,, Mz) configurations. And we adopt the same quantization bit-width for W, A. E. thus that hardware design simple. The training results on CIFAR- 10 and ImageNet are shown in Tab. [ We can see that our method can achieve smaller accuracy degradation using lower bit- bit-width. Previous study (Zhou et al. 2016) found that quantizing E to a low bit-width hurt the performance. However, method can quantize E to M2- 1 or 2 on CIFAR- 10, with small accuracy drop from 92.45% to 91.97%. On ImageNet, the accuracy degradation of our method rather minor under 8-bit quantization (0.6% accuracy drop from 69.1% % to 68.5%). In the cases with lower bit- our method achieves a higher accuracy (66.5% with only 4-bit than Banner et al. (2018a) who uses 8-bit (66.4%). With (2, data format, the accuracy loss is less than 1%. In this case, the bit-widie

---

Under review as a conference paper at ICLR 2021
Table 3: Ablation study (ResNet-20 CIFAR-10). "Div." means that the training failed to converge
#group
Mg
E
M =4
M, 3
M,
2
M,
None
0
90.02
85.68
Div.
Div.
C
0
91.54
88.35
82.29
Div.
0
91.78
89.62
80.71
Div.
nc
0
92.14
91.64
88.97
76.98
nc
1
92.37
91.73
90.39
82.61
1
None
90.02
85.68
Div.
Div.
1
None
91.67
90.11
84.72
70.4
None
92.32
92.34
91.58
90.32
None
3
92.66
92.41
92.47
92.04
1
0
92.37
91.73
90.39
82.61
nc
1
92.52
92.16
91.48
89.97
1
2
92.37
92.65
92.05
91.97
of the intermediate results is 2M, + 1-,++111- conducted using 16-bit integers, instead of Noating-pointe (Mellempudi et al. 2019)
- 14, which means that the accumulation can be
5.1
ABLATION STUDIES
5.1.1
GROUPING DIMENSION
Group- troup-wise scaling is beneficial because the data ranges vary across different groups. We compare average relative quantization error using the three grouping dimensions (Sec. 3.1 with (8, 1) group-wise scaling format and (0, 3) element format. The first row of Fig. shows that the AREs are smaller when each tensor is split to N groups.
Furthermore, we compare these grouping dimensions in the training process. The results in the first section of Tab. 3 show that the reduction of AREs is important to the accuracy low-bit training. And when tensors are split to N x C groups, the low-bit training accuracy is higher. And that Mg = important for the performance, especially with low M, (e.g., when M, = 1).
D4
goup
16
12
10
18
14
0.6
3.6
0.
0.0
3.20
2.4
0.1
0.55
32
10
12
14
16
16
14
16
18
10
Figure 3: Average relative quantization errors (AREs) of W, E, A (left, middle, right) in each layer when training ResNet-20 on 10. X axis: Layer index. Row 1: Different grouping dimensions ((0, 3) formatted X, (8, 1) formatted Sy); Row 2: Different E.e ((F,, 3) formatted X, no group-wise scaling); Row Different E, (EE,, 3) formatted X, (8, 1) formatted Sy, N groups).

---

Under review as a conference paper at ICLR 2021
5.1.2 ELEMENT- WISE EXPONENT
To demonstrate the effectiveness of the clement-wise exponent, compare the AREs of quantization with different Eg without group-wise scaling, and the results are shown in the second row of Fig. 3
We can see that using more exponent bits results in larger dynamic ranges and smaller AREs. And
with larger Eg, the AREs of different layers are closer. Besides the ARE evaluation, Tab. 3 also shows that larger Eg achieves better accuracies, especially when M, is extremely small.
As shown in Fig. Row 3 and Tab. 3 when jointly using the group-wiss scaling and the element-wise exponent, the ARE and aceuracy are further improved. And we can see that the group-wii wise scaling essential for simplifying the floating- ppintt accumulator fixed-point, since one can use small Eg (e.g., 2) by using the group-wise scaling technique.
Operations
Intager Operations
Scale
MUL
FP Acc
Adder Tree
MUL
INT Acc
Scale Unit
Adder Tree
Scale
FP MUL.
FP Acc
FP MUL
INT Acc
Scale
Unit
(a) Original method
(b) Our method
Figure 4: The convolution hardware architecture. (a) Previous studies (Melempudii et al. 2019) developed low-bit floating- point multiplication (FP MUL) (e.g., 8-bit), but FP32 accumulations still needed. (b) We not only makes FP MUL. less than 8-bit, but also simplifies the local accumulator.
5.2 HARDWARE ENERGY ESTIMATION
Fig. 4 shows a typical convolution hardware architecture, which consists of three main components: local multiplication, local accumulation, and addition tree. Our algorithm mainly improves the local MAC. Compared with the full- prrccisionn design, we simplify the FP MUL to use bit-width less than 8 and the local FP Acc to use 16-bit integer. According the data reported by (Yang al. 2020). the energy efficiencies are at least 7x and 20x higher than full-precision design, respectively.
While our method could significantly reduce the cost of the convolution, it also introduces some overhead: 1) Group-wise maximum statistics (Line 2) and scaling factors division (Line 9) in Alg. [] accounts for the main overhead dynamic quantization. The cost of these two operations is comparable with that of batch normalization operation, which is relatively small compared with convolution since the number of operations is fewer by orders of magnitude (Tab. [T. 2) The group- wise scaling factors introduce additional scaling. Fortunately, when using the (Eg 0) (Eg, 1) format, we can implement it efficiently with shifting (see Eq. 3
To summarize, the introduced overhead is small compared with the reduced cost. According to the numbers of different operations in the training process (Tab. [T] and the energy consumption of each operation (Appendix Tab. 3) (Horowitz, 2014), we can estimate that convolution arithmetic is over 6.8x x more energy-efficient than full precision one when training ResNet (see Appendix for the details). Due to the simplified integer accumulator, our energy efficiency least 24% higher than other low-bit foating-poin training algorithms (Mellempudi et al. 2019 Wang et al. 2018).
6 CONCLUSION
This paper proposes low-bit training framework to enable training a CNN with lower bit-width convolution while retaining the accuracy. Specifically, design a multi-level scaling (MLS) tensor format, and develop the corresponding quantization procedure and low-bit convolution arithmetic. Experimental results and the energy analysis demonstrate the effectiveness of our method.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. NeurIPS, 2018a. Sa.
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In NeurlPS, 2018b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Joshua V. Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A. Saurous. TensorFlow Distributions. arXiv e-prints, 1rXiv:1711.10600 November 2017.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with
ArXiv, abs/1905.03660 2019.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015,
Philipp Gysel, Jon Pimentel, Mohammad Motamedi, and Soheil Ghiasi. Ristretto: framework for empirical study of resource-efficient inference in convolutional neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29:5784-5789 2018.
Song Han, Huizi Mao, and William J. Dally, Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.001499 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
M. Horowitz. 1.1 computing's energy problem (and what do about it). 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, 2014.
David G Hough. The ieee standard 754: One for the history books. Computer, 52(12):009-12, 2019.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-rrithmeti-onn inference. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2704-2713, 2017a.
Benoit Jacob et al. gemmlowp: small self-contained low-precision gemm library. 2017b.
Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K. Bansal, William Constable, Oguz Elibol, Stewart Hall, Luke Hornof, Amir Khosrowshahi, Carey Kloss, Ruby J. Pai, and Naveen Rao. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In NIPS, 2017.
Alex Krizhevsky. Convolutional deep belief networks on cifar-10. 2010.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. Burges, L. Bottou, and K. Q. Weinberger (eds.). Advances in Neural Information Processing Systems 25, pp. 1097-1105, Curran Associates, Inc., 2012.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv: 1605.0007 2016.
Darryl Dexu Lin, Sachin Talathi, and V. Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks, ArXiv, abs/1511 0111111116699,.. 066393, 2015.

---

Under review as a conference paper at ICLR 2021
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.
Z. Liu, Zhiqiang Shen, M. Savvides, and K. Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. ArXiv, 0ts/22003.03488. 2020.
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed precision training with 8-bit floating point. ArXiv, abs/1905. 12334, 2019.
Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Shighted-entropy-beed quantization for deep neural networks. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 7197-7205, 2017.
Haotong Qin, Ruihao Gong, Xianglong Liu, Ziran Wei, Fengwei Yu, and Jingkuan Song. Ir-net: Forward and backward information retention for highly accurate binary neural networks. ArXiv, Dtss/1909.10788 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. ECCV, 2016.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Xiao Sun, J. Choi, Chia- Yu Chen, Naigang Wang, Swagath Venkataramani, V. Srinivasan, Xiaodong Cui, W. Zhang, and K. Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NeurlPS, 2019.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Y Yu Chen, and Kailash Gopalakrishnan, Training deep neural networks with 8-bit floating point numbers. In NeurlPS, 2018.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. ArXiv, abs/ 18011000224888 2018
Y. Yang. Shuang Wu, Lei Deng, Tianyi Yan. Yuan Xie, and Guoqi Li. Training high-performance and large-scale deep neural networks with full 8-bit integers. Neural networks: the official journal of the International Neural Network Society, 125:70-82, 2020.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. ArXiv, abs/ 1606666666111 0616 2016.
10

---

Under review as a conference paper at ICLR 2021
A LOW-BIT TRAINING FRAMEWORK
Algorithm 2: The t-th iteration of low-bit training with vanilla SGD
Input: L: number of layers; W.'L. float weights; A inputs; T: label; lr: learning Output: W updated float weights
o forward propagation
for I in do
gW DynamiQQuantizationn W
qA [-1
ConnamicQuantizattinn (-1)
Z' CoubiilCon((qWW . )
Y
BatchNorm(2 )
A
Activation Y
Blass
Criterion(A",
DA
* backward propagation */
for in L do Blass Blass
AAttvvation (Y')
x
Glass Blass
BY
x
gE' Dynamic uanlization Blass 82"
10
G' LowtisCoruuyEE ,'''"'"''""'" -gA'=1)
11
W 1+1 Wt lr G'
if is not then
AaA LowbitCCone(EE
Bloss
Gloss ST E 0lsss 0
0gA--1)
Return WII.
B EXPERIMENTAL SETUP
In all the experiments, the first and the last layer are left unquantized following previous studies (Zhou et al. 2016 Mellempudi et al. 2019 Sun et al. 2019). For both CIFAR-10 10 and ImageNet, SGD with momentum 0.9 and weight decay 5e-4 is used, and the initial learning rate train the models for 90 epochs on ImageNet, and decay the learning rate by 10 every 30 epochs. On CIFAR-10, we train the models for 160 epochs and decay the learning rate by at epoch 80 and 120.
0.1. We
3.5
3.0
14
14
max
hall
2.0
(A] group
group
(6) group number
1d) group
Figure 5: Maximum value of each group of A (left two) and E (right two). (a)(c): Grouped by channel: (b)()): Grouped by sample.
11

---

Under review as a conference paper at ICLR 2021
C GROUP-WISE SCALING
Group-wind wise scaling is beneficial because the data ranges vary across different groups, as shown in Fig. 5 The blue line shows the value in each group when A and E are grouped by channel or sample. If we use the overall maximum value (green lines in Fig. () as the tensor-wise scaling, many small elements will be swamped. And usually, there are over half of the groups, in which all elements are smaller than half of the overall maximum (red line). Thus, to fully exploit the bit-width, it is natural use group-wis wise scaling factors.
6,=
100
E,
95
23.57
52
94.22
12.41
Accuracy
90
9112
00:11
85
70
FP32
3
Mantissa Bits(M,)
Figure 6: Performances with different (E,,, M2) configurations, no group-wise scaling is used.
D ELEMENT-WISE EXPONENT
Fig. shows the performances of training ResNer20 on CIFAR-10 with different (En, M₂) configu- rations. We can see that, when the mantissa bit-width M, is extremely low (e.g.. 1), the element-wise exponent bit-width Eg is essential for achieving acceptable performance.
E ENERGY EFFICIENCY ESTIMATION
Tab. 4 Horowitz 2014) reported that the energy consumption a FP32 multiplication (FP32 MUL) about times that FP32 addition (FP32 ADD). Denoting the energy consumption of FP32 ADD as O' and FP32 MUL as 4C, according to (Yang et al. 2020), we can estimate the cost of FP8 MUL and INT16 ADD 4/7C' and 1/20C, respectively. Then, using the operation statisties in Tab. 1 we can calculate the energy consumption of one training iteration, and estimate the energy efficiency improvement ratio:
4(MUTT) + 1 ACC) 1
Ratio
6.8
+ M/7(#MUL)+ ADD #Sall)) + 1 (T#TreAC
(9)
Table The cost estimation of primitive operations with 45nm process and 0.9V (Horowitz) 2014).
Energy(p.J) Mul
Area(/um?) Mul
Params
Add
Add
8-bit Fix
0.2
0.03
282
36
16-bit Float
1.1
0.40
1640
1360
32-bit Float
3.7
0.90
7700
4184
In order to evaluate energy efficiency advantage more accurately, we have implemented the RTL design of the two MAC units in Fig and used Design Compiler to simulate the and power,
12

---

Under review as a conference paper at ICLR 2021
the power results are shown in 5 We can see that the simulated results of RTL implemented is similar with our estimation above, both showing the energy efficiency of our framework.
Table 5: The power evaluation (mW) results of MAC units with different arithmetic with TSMC 65mm process and 100MHz clock, simulated by Design Compiler.
Framework Full Precision Other Low-bit Training
Ours
MUL
0.532
0.023
0.0192 0.0094
LocalACC
0.140
0.140
13


Reviews:
[{"review_id": "WuTNTYbQA0o", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nEfficient training is becoming a crucial research topic as deep learning models get deeper and more complex to improve model accuracy. The authors propose a low-bit floating point quantization method to reduce energy and time consumption during training. To enhance training efficiency, the authors suggest multi-level scaling (MLS) tensor format that enables low-bit training. MLS is designed to compute complex operations with lower-cost operators (e.g., shift, add) or lower-bit operators that can be hardware-friendly. This reviewer considers the proposed hardware-friendly MLS format as the main contribution of the manuscript.\n\nThis reviewer raises the following serious concerns.\n \n- I have no idea whether this work can help efficient training in practice. Since there is no prior information on how many bits are required for training without serious accuracy degradation, any efficient training method needs to be robust to various learning settings and dataset configurations. This work, however, studies the optimal formats for particular models, such as ResNet models. It would be required to suggest a general data format and demonstrate that such a format can be applied to various DNN models.\n- The authors argue that the proposed method achieves over 6.8x higher energy efficiency than training with floating-point (FP) arithmetic units. Unlike training with FP, the proposed method requires dynamic quantization for every batch. The authors need to present the computational overhead of the dynamic quantization with a sufficiently thorough analysis. If the cost is negligible, the ground should also be provided. Also, it would be better if the authors suggest the expected time reduction and space complexity compared to full-precision training. Even though Section 5.2 briefly addresses such overhead of dynamic quantization (as to be comparable with that of a batch normalization), this reviewer cannot estimate the overall benefit of the proposed method compared with the previous works.\n\nOverall, this reviewer votes for rejection. An efficient training algorithm needs to be supported by detailed experimental results on various types of DNN models while any computational overhead should be reasonably addressed.\n\nMinor comments:\n- Section 2 needs to be focused on training while the comparisons with previous works can be elaborated in the experimental results.\n- Table 1 is not relevant unless the authors want to suggest ResNet-specific accelerator designs.\n- Similarly, Table 2 and 3, and Figure 3 are not necessary if the authors claim that the flow in Figure 2 can be general.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RzxQV8Pnlls", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript describes a new low-bit training framework as well as a new low-bit format and shows promising accuracy-precision trade-offs and better energy efficiency. The proposed method achieves no accuracy drop with 3-bit training on CIFAR dataset and 1% accuracy drop with 6-bit training on ImageNet dataset. Although the results are comparable to state-of-art works like \"Hybrid 8-bit Floating Point\", this paper has several drawbacks.\n\n1. Section 3 provides detailed description of the MLS design. However, it is also very important to show a logic justification. Previous works will usually adopt metrics like mismatch probability across layers to show advantages of their design. Therefore, I would suggest authors to move section 5.1.1 (or Appendix C) to section 3 as a justification for MLS design. \n\n2. Figure 2 looks like a typical quantization-aware training flow and I did not see any potential difference on the figure caused by MLS scheme. Therefore I wonder if it is right or necessary to put Figure 2 in this section.\n\n3. It is not clear to me that how MLS format avoids 32-bit FP multiplication. As the first level scaling factor is still FP32, I suppose every activation still requires FP32 mul?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jcfu75dKVG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new tensor format referred to as multi-level scaling (MLS) for training deep neural networks in low precision. Dynamic quantization and MLS tensor arithmetic are used to enhance the effectiveness of MLS. Hardware energy efficiency gains are listed as 6.8X over full-precision and 1.2X over previous low-precision methods.\n\nComments and concerns are listed below:\n\n1. This paper is a difficult read because of the disconnect between the figures used and the text. For example, the notation in Fig. 1 does not match the notation in the text describing Fig. 1 in 3.1 and 3.2. The symbol 'E' is used for both activation gradient and for the number of exponent bits. Typo ('quantizaiton') in line 7 of section 4. Algorithm 1 is too busy to be useful. Please abstract it out so the flow is clearer. The paper could do with a complete rewrite to tighten it up.\n\n2. The main results in Table 2 shows that the improvement w.r.t. (Sun, 2019) is incremental. This seems consistent with the 1.2X energy efficiency gain mentioned elsewhere. Also, since this paper purports to contribute to low-bit training, why isn't there a comparison with the references (Dillon, Koester, Gysel, and Wang)? Instead Banner is included even though it falls in the category of post-training quantization.\n\n3. It was not clear from Fig. 2 if the high-bit data and computations are in FP-32 or equivalent? Not sure what is meant.\n\nOverall, this was a difficult paper to read and understand. The results seem incremental.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IuJmZv9wSQz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigated the low-bit training problem and proposed a novel method, which can reduce the element-wise bit-width to simplify floating-point computations to nearly fixed-point. The major contribution can be summarized as follows: a multi-level scaling (MLS) tensor format, a dynamic quantization and the low-bit tensor convolution arithmetic. Experiments show the proposed method could strike a good trade-off between the accuracy and bit-width, while taking the hardware efficiency into consideration.\n\n----Strengths:\n- Experiment shows the proposed method can outperform the state-of-the-art methods\n- There is abundant model analysis in this paper.\n----Weaknesses: \n- The paper is written in a very high-level manner. The paper could have benefitted with mode detailed explanations throughout, to aid comprehension.\n- Many details of the method are unclear. For example: The definition of {I, j, k, l} in Formula 1 are unknown. Besides, in figure2, the exhibition of the proposed low-bit training computation flow is also confusing.\n- On the algorithm side, The MLS format appears to be just a new combination of quantitative activation and multiple gain items. it is not clear how or why the proposed model should be more efficiency to reduce the element-wise bit-width compared with other methods? Is it just an observation/interpretation of the results?\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a low-bit floating point quantization method to reduce energy and time consumption for deep learning training. Dynamic quantization and MLS tensor arithmetic are used to enhance the effectiveness of MLS. The motivation is clear and the efficient training is an important problem to address. However, the effectiveness of proposed method is not well justified and experimental results are less convincing.  In addition, the clarify of paper still needs to be further improved. 
Decision: Reject