Logarithmic Unbiased Quantization: Practical 4-bit Training in Deep Learning
Authors: Brian Chmiel
Ron Banner
Elad Hoffer
Hilla Ben Yaacov
Daniel Soudry
Venue: ICLR 2022 Submitted (2022)
Keywords: quantization
efficient training
4 bit training

Abstract: Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only a third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers. 
In this work, we examine the importance of having unbiased quantization in quantized neural network training, where to maintain it, and how. Based on this, we suggest a logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training. For example, in ResNet50 on ImageNet, we achieved a degradation of 1.18 %; we further improve this to degradation of only 0.64 % after a single epoch of high precision fine-tuning combined with a variance reduction method. Finally, we suggest a method that exploits the low precision format by avoiding multiplications during two-thirds of the training process, thus reducing by 5x the area used by the multiplier. A reference implementation is supplied in the supplementary material.

Under review as a conference paper at ICLR 2022
LOGARITHMIC UNBIASED QUANTIZATION: PRACTI- CAL 4-BIT TRAINING IN DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Quantization of the weights and activations is of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers. In this work, we examine the importance of having unbiased quantization in quantized neural network training, where to maintain it, and how. Based on this, suggest a logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phase to 4-bit, achieving state-of-tie ahrthe results in 4-bit training. For example, in ResNet50 on ImageNet, we achieved a degradation of 1.18%; we further improve this to degradation of only 0.64% after single epoch of high precision fine-tuning combined with a variance reduction method. Finally, we suggest a method that exploits the low precision format by avoiding multiplications during two-thirds of the training process, thus reducing by 5x the area used by the multiplier. reference implementation is supplied in the supplementary material.
INTRODUCTION
Deep neural networks (DNNs) are a powerful tool that has shown superior performance in various tasks spanning computer vision, natural language processing, autonomous cars, and more. Unfortu- nately, their vast demand for computational resources, especially during the training process, is one of the main bottlenecks in the evolution of these models.
Training of DNNs consists of three main general- matill-matiii---oe 1a1--mm-matiiitlllllll multiply (GEMM) phases: the forward phase, backward phase, and update phase. Quantization has become one of the main methods to compress DNNs and reduce the GEMM computational resources. There has been a significant advance the quantization of the forward phase. There, it is possible quantize the weights and activations to bits while preserving model accuracy (Banner et al.] [019 Nahshan et al. 2019 [hhlgat et al.) 20200] Choi et al. (0188)). Despite these advances, they only apply a third of the training process while the backward phase and update phase are still computed with higher precision.
al. (2020) was able, for the first time, to train DNN while reducing the numerical precision of most of its parts to 4 bits with some degradation (e.g., 2.49% error in ResNet50). To do so. Sun et al. (2020) suggested non-standard radix-4 floating-point format, combined with double quantization of the neural gradients (called two-phase rounding). This was an impressive step forward the ability to quantize all the training processes. However, since a radix-4 format is not aligned with conventional radix-2, any numerical conversion between the two requires an explicit multiplication to modify both the exponent and mantissa and may require additional hardware (Kupriianova et al.) 2013). Thus, their non-standard quantization requires specific hardware support that can reduce the benefit of quantization to low bits.
Recently,
The main challenge in reducing the numerical precision of the entire training process is quantizing the neural gradients, i.e. the backpropagated error. Specifically, Chmiel et al. (2021) showed the neural gradients have a heavy tailed lognoralll distribution, and therefore they should be logarithmically quantized at low precision levels. For example, for FP4 the optimal format was (sign.exponent.mantiss [1,3,0], i.e. without mantissa bits. In contrast, that weights and activations

---

Under review as a conference paper at ICLR 2022
well approximated with Normal or Laplacian distributions (Banner et al. 2019 Choi et al. and therefore are better approximated using uniform quantization (e.g., INT4).
8a
In this work, in order to reduce the computational resourees bottleneck, we dive deeper into under- standing neural gradients' quantization. Based on the findings offChiill et al. (202I), we focus on the [1,3,0] FP4 format for the neural gradients. We analyze different types of quantization stochas- tic rounding schemes to explain the importance of having an unbiased rounding scheme for the neural gradients with such a logarithmic quantization format. We compare it with forward phase quantization, where the bias is not a critical property.
Building on this analysis, we suggest method called logarithmic unbiased quantization (LUQ) for unbiased quantization of the neural gradients to the standard FP4 format of [1,3,0]. This, together with quantization of the forward phase INT4, achieves state-of-the-an results in full training in bits (e.g.. 1.18% error in ResNer50), with no overhead. Moreover, we suggest two additional methods to further reduce the degradation, with some overhead: the first method reduces the quantization variance of the neural gradients, while the second is simple method of fine-tuning in high precision. Combining LUQ with these two proposed methods we achieve, for the first time, only 0.64% error in 4-bit training of ResNer50. The overhead of our additional methods is no more than the previously presented in Sun et al. (020)).
Finally, exploit the specific FP4 format mantissa used for the neural gradients ([1,3,0) and suggest replacing the multiplication blocks with a proposed multiplication free backpropagation (MF-BPROP) block. This way completely avoid multiplication in the backward phase and the update phase (which constitute two thirds of the training) and reduce by 5x the area which was previously used for the multiplications in these phases.
The main contributions of this paper:
comparison of different rounding schemes.
We suggest simple and hardware friendly logarithmically unbiased quantization for the neural gradients called LUQ.
We demonstrate that two simple methods can further improve the accuracy in 4-bit training: (1) variance reduction using re-sampinng and (2) high precision fine-tuning for one epoch.
We design a modern hardware block that exploits LUQ quantization to avoid multiplication in two thirds of the training process, thus reducing by 5x the multiplier logical area.
2 ROUNDING SCHEMES COMPARISON
In this section, we study the effects of unbiased rounding for the forward and backward passes. We show that rounding seaddingtt--nergeee ne-rest (RDN) should be applied for the forward phase while stochastic rounding (SR) is more suitable for the backward phase. Specifically, We show that although SR is unbiased, generically has worse mean-square-error compared to RDN.
Given that we want to quantize x in bin with lower limit [(x) and an upper limit u(z), stochastic rounding can be stated as follows:
SR(x)= = ((x),
with probability p(z)
x e(x)
u(x) 1(x)
u(x), with probability 1 p(x) u(x) 1(x)
2 2-1(z)
The expected rounding value is given by E[S[(z)] = 1(x) p(=) + u(x) -1-p(z)) x.
where here and below the expectation is over the randomness of SR (i.c., is a deterministic constant). Therefore, stochastic rounding is an unbiased approximation of ær since has zero bias:
Bias{SR(x)] = E*SR(()) x] = E[SR(r)] x = 0.
(3)
However, stochastic rounding has variance, given by
Var[SR(x)] (I(x:) E[SR(*)))² p(x) + (u(x) E[SR(r))) -x(2))
(4)
= (x ((x)) (u(x)= - =x)
2

---

Under review as a conference paper at ICLR 2022
where the last transition follows from substituting the terms E[SR(=)]), and p(z) into Eq. (4)
We turn to consider the round- -oodd-t-noee co-ncarest method (RDN). The bias of RDN is given by
Bias[RDN min (a (z), u(x) x). Since RDN is deterministic method, it is evident that the variance is i.e.
(5)
Var[RDD(<x]] =0
(6)
Finally for every value a and a rounding method R(xx), the mean-square--rror (MSE) can be written as the sum of the rounding variance and the squared rounding bias,
MSE[R(x) = 1 [[[())) x]² = Var[R(x)] + Bias" [R(x]]
Therefore, we have the following MSE distortion when using round-to-nearest and stochastic round- ing:
[min (a 1(x). x)1 (xr 1(x)) (w(x)) - x))
RDN(x) SR(x)
MSE-
(8)
Note that since min(a, b)2 S a for every b, we have that
MSE (x) 2 MSE(RDN(x)), Vo.
In Fig. la we plot the caaun-suure-erre error for [0, 1), [(x) 0, and u(z) 1. While round-to-
nearest has a lower MSE than stochastic rounding, the former is a biased estimator.
g 550
g
MSR
40
0.10
30
Baseline
30
Baseline
0.05
stachastic rounding
20
Fwd
20
Bwd SR
to-neerest
10-
Fwd
10
RDN
0.00
0-
0-
0.4
0.6
0.8
1.0
75 300 125
125 150 Epochs
100
Epochs
(a)
(b)
(c)
Figure 1: Comparison between stochastic rounding (SR) and round-to-nearest (RDN) quantization. In (a) we present the MSE of uniform distributed tensor with the two different rounding schemes. Quantization 4 bits of the forward phase (b) and backward phase (c) of ResNet18 Cifar100 dataset with SR and RDN. Notice that while MSE is important in the forward phase, unbiasness achieved with SR crucial for the backward phase. The bwd and fwd (b) and (c) respectively, are in full precision focus the effect of the rounding scheme only in one pass of the network in each experiment.
2.1 BACKGROUND: UNBIASED GRADIENT ESTIMATES
To prove convergence, textbook analyses of SGD typically assume the expectation of the smini-batch) weight gradients is sufficiently close to the true (full-batch) gradient (e.g., assumption 4.3 in (Bottou et al. 2018)). This assumption is satisfied when the weight gradients are unbiased. Next, we show that this condition is met when the neural gradients quantized without bias.
Denote W, as the weights between layer and I, C' the cost function, and fi as the activation function at layer l. Given an input-output pair (x, y). the loss is:
C (v. ST. (W.,ft-1 (Wt-1 (Wafi (W.x))...)))
(10)
Let 3º be the weighted input (pre-activation) of layer and denote the output (activation) of layer I by ar The derivative of the loss terms of the inputs is given by the chain rule:
dC dar. dzz. dar.-1 dzr-1
doi
dz
81 day dzz dar-1 džı-1 dar- dz
(11)
dai-1
3

---

Under review as a conference paper at ICLR 2022
In its quantized version, Eq. (II)
gets the following form.
dC
dar.
daj
d21
big - Q
Q
Q
Q
(12)
dap
deL
dz1
day-1
Assuming ((x)) is an unbiased stochastic quantizer with [[Q(x]] òig is an unbiased approximation of backpropogation ó1
x, the quantized backpropogation
dC
da
da
dæ
El =
E
Q
Q
Q
Q
dar.
dzr.
dz
day-1
dC
daz
da1
dæ
E
Q
E
Q
E
Q
E
Q
dat
dzz
dz1
day-1
(13)
dC
d
d2L
dal-1
d2L-1
da1
dæ1
dat
dzz
dat-1
d=L-1
dar-2
dzi
das-1
61
In Eq. (3) we used the linearity of pack-propagation to express the expected product as a product of expectations. Finally, since the gradient of the weights in layer | is Vw; 6, ot-1 and in its quantized form it becomes Vw; Ci = ōig at-1, the update Ww, Cy is an unbiased estimator of Cw,C:
E VV Cg] = F. [ōi, a-11 E [ōig]] 0111 & at-1 Vw,C.
(14)
The forward pass is different from the backward pass in that unbiasedness at the tensor level is not necessarily guarantee unbiasedness at the model level since the activation functions and loss function are not linear. Therefore, even after stochastic quantization, the forward phase remains biased.
Conclusions. It was previously proved that unbiased neural gradients quantization leads to an unbi- ased estimate of the weight gradients (e-g., Chen et al. (2020a)), which enables proper convergence of SGD (Bottou et al. 2018). Thus, bias in the gradients can hurt the performance and should be avoided, even at the cost of increasing the MSE. Therefore, neural gradients, following should be quantized using SR, following subsection 2.1 However, the forward phase should be quantized deterninistically (using RDN) since stochastic rounding will not make the loss estimate unbiased (due to the non-linearity of the loss and activation functions) while unnecessarily increasing the MSE (as shown in Eq. (.. There are cases where adding limited noise, such as dropout, increases MSE but improves generalization. However, this is typically not the case, especially if the noise is large. Figs. Ib and show that these theoretical observations are consistent with empirical observations favoring RDN for the forward pass and SR for the backward pass.
3 LUQ LOGARITHMIC UNBIASED QUANTIZER
recent work (Chmiel et al. 2021) showed that the neural gradients can be approximated with the lognormal distribution. This distribution has many values concentrated around the mean but is also heavy-tailed, making the extreme values orders of magnitudes larger than the small values sampled from this distribution. They exploit this fact and showed that the neural gradients can be pruned to high pruning ratio without accuracy degradation (e.g., 85% in ResNet18 ImageNet dataset), using an unbiased pruning method. We build on top of this pruning method, and combine it with unbiased logarithmic quantizer, as described below.
Unbiased stochastic pruning Given an underflow threshold we define stochastic pruning operator, which prunes given value x, as
if | 2
To (x)
sign(z)-o a
Ir] w.p.
w.p.
if |x| < a
(15)
12 if |x| < a. o
4

---

Under review as a conference paper at ICLR 2022
Unbiased FP quantizer Given underflow threshold a, let Qa(X) be FP round-to- nearest b-bits quantizer with bins {a, 2a, ..., 20-22-aa,, Assume, without loss of generality, 2" a < x < a (n {0,1... 1..., b.. 1}). We will use the following unbiased quantizer, which is a special case of SR (Eq. 1
22- w.p.
2
(x) =
(16)
a
w.p. 1 2"a 2n
2-2 2-2n--a
la
2
la
It is unbiased since
2" σ
2n
= a.
a 2n 2"
(17)
2n-la
and as a special case of Eq.
Underflow threshold In order to create unbiased quantizer, the largest quantization value 2 should avoid clipping any values of x, otherwise this will create a bias. Therefore, the underflow threshold a is chosen as the optimal unbiased value, ie
max (x))
a:
22
where for FP4.
Logarithmic rounding Traditionally, stochastic rounding in eq. [6lis implemented by adding a uniform random noise ~ U[- -_--_--,_ 2"-'aa o] to æ and then use round-to-nearest operation. In order to implement round-to-neares. directly the exponent, we need to correct an inherent bias since a 2l1g( [1)) + a- 1 (5)s)))) )
For bin [2n-1,20 ) the midpoint Fm is
2n
2
3
Im
2
(18)
2
4
Therefore, we can apply round-to mund-to-necrest-powee (RDNP) directly on the exponent of any value 2n 2 < 2" follows:
RDN = 2ol 2 21 ($) = 2RDN(=+ (1) 2RDN(o-0.084)
(19)
Logarithmic unbiased quantization (LUQ) In the following, suggest LUQ, 4-bit unbiased estimation of neural gradients that apply stochastic pruning (Eq. (15) to the 4-bit floating point quantizer Qa (x:
X, = Ta (Qa(z))
(20)
Since Ta and Qa are unbiased, it follows by the law of total expectation that X2 is an unbiased estimator for x:
=| = ETT (2)())) E E -
(21)
where the expectation over the randomness Ta and Qa. Fig. 2 we show an illustration of LUQ The first step includes stochastic pruning for the values below the pruning threshold ([æ) < a). The second step includes the logarithmic quantization with format FP4 of the values above the pruning threshold (|z] > a). In Fig. 3a we show an ablation study of the effect of the different parts of LUQ on ResNet50 ImageNet dataset while standard FP4 diverges, adding sochastic-pruniis nearest- power allow to converge with significant degradation. Combining both methods improves the results and finally the suggested LUQ which includes additionally the suggested underflow threshold choice the optimal unbiased value gets the best results.
or round-to-

---

Under review as a conference paper at ICLR 2022
la (x)(()))
xx
x
--------
-
a a
a
a
The effect of LUQQ on the neural gradients histograms for one layer ResNet18 Cifar (10 dataset, with the underflow threshold (red dashed line). The first step (green arrow) represents the effect of stochastic pruning (Eq. (15) on the neural gradient. The second step (grey arrow) represents the logarithmic unbiased quantization (Eq. (16)). that quantize all the values [x] > a.
Figure 2:
3.1 SMP: REDUCING THE VARIANCE WHILE KEEPING UNBIASED
In the previous section, we presented unbiased method for logarithmic quantization of the neural gradients called LUQ. Following the bias-variance decomposition, if the gradients are now unbiased, then the only remaining issue should be their variance. Therefore, we suggest method reduce the quantization variance by repeatedly sampling from the stochastic quantizers in LUQ, and averaging the resulting samples of the final weight gradients. The different samples can be calculated parallel, so the only overhead on the network throughput will be the averaging operation. The power overhead will 1 of the number of additional samples since it affects only in the update GEMM (Eq. (24g). For N different samples, the proposed method will reduce the variance by a factor f. without affecting the bias (Gilli et al. 2019). In Fig. 3b we show the effect of the different number of samples (SMP) on 2-bit quantization of ResNet18 Cifar100 dataset. There, we achieve with 16 samples accuracy similar to full- nell-precision network. This demonstrates that the variance is the only remaining issue in neural gradient quantization using LUQ, and that the proposed averaging method can erase this variance gap, with overhead.
80
70
60
50
8 40
Accurais
Baseline
Baseline
30
FP4
bits:
20
FP4 + RDNP
LUQ 2 bits with 2 samples
LUQ
samples
FP4 RDNP
LUQ bits with 8 samples
LUQ
bits with samples
25
50
100 Epochs
125
150
175
200
Epochs
(a)
Figure 3: (a): ResNet50 top-1 validation accuracy in ImageNet dataset with different quantization schemes for the neural gradients. SP refers to stochastic pruning (Eq. (15)). RDNP refers to round- to-nearest-power (Eq. (19)). Notice that with the suggested LUQ
able to almost close the degradation from baseline. (b): ResNet18 top-1 validation accuracy in CIFAR 100 with quantization
of the neural gradients 2-bit (FP2- [1,1,0] format) using different samples numbers to reduce the variance. Notice that 16 samples completely close the gap to the baseline.
3.2 FNT: FINE-TUNING IIGH PRECISION FOR EPOCII
We suggest running one additional epoch in which we increase all the network parts to full-precision, except the weights which remain in low precision. We notice that with this scheme we get the best accuracy for the fine-tuned network. In inference time the activations and weights quantized to lower precision. In Table we can see the effect of the proposed fine-tuning scheme, improving the accuracy of the models 0.4%.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
In this section, evaluate the proposed LUQ for 4-bit training various DNN models. For all models, we use their default architecture, myper-parameee and optimizers combined with a custom-modifie Pytorch framework that implemented all the low precision schemes. Additional experimental details appear in Appendix A.
Main results Table 1 we show the top-1 accuracy achieved in 4-bit training using LUQ to quantizing the neural gradients to FP4 and combined with previously suggested method, SAWB (Choi et al. [00 Sa), to quantize the weights and activations to INT4. We compare our method with Oltra-low (Sun et 2020) showing better results in all the models, achieving SOTA in 4-bit training. Moreover, we improve the results with the two proposed schemes: neural gradients sampling (SMP Section 3.1) and fine-tune in high precision (FNT Section (.22) achieving for the first time in 4-bit training 0.64% error in ResNet-50 ImageNet dataset, with our simple and hardware friendly methods.
In Table 2 we apply the proposed LUQ on NLP models, achieving less than 0.4% BLUE score degradation in Transformer- base model on the WMT En-De task.
Overhead of SMP and FNT We limit our experiments with the proposed SMP method to only two samples. This is to achieve similar computational overhead Ultra-low, with their suggested wo-phase-rroudding (TPR) which also generates a duplication for the neural gradient quantization. The FNT method is limited to only 1 epoch to reduce the overhead in comparison to Ultra-low, which keeps the 1xl convolutions in 8-bit. Specifically. the throughput of 4-bit training network is 16x in comparison high precision training (Sun al.] (020). This means that doing one additional epoch in high precision reduces the throughput of ResNet-50 training by ~ 16% In comparison, Ultra-low (Sun al. 2020) does full-training with all the Ix1 convolutions in 8-bits, which reduces the throughput by 50%.
Table 1: Comparison of 4-bit training of the proposed method LUQ with Ultra-low (Sun al. 2020
in various DNNs models with ImageNet dataset. FNT refers to fine-tune the trained model one additional epoch with the neural gradients high precision (Section 3.2) and SMP refers to doing two samples of the SR quantization of neural gradients in order to reduce the variance (Section 3.1
Model
Baseline
Ultra-low
LUQ
LUQ + FNT
LUQ SMP
LUQ + SMP FNT
R8sNet-18
69.7 %
68.27%
69.0%
69.39 %
69.1
69.47 %
ResNet-50
76.5%
74.01%
75.32
75.52
75.63 %
75.86
MobileNei-V2
71.9 %
68.85 %
69.69 %
69.87 %
69.9 %
70.13 %
ResNext-50
77.6 %
N/A 76.12 %
76.39 %
76.32 %
76.55 %
Table 2: Comparison of the BLUE score for 4-bit training of the proposed method LUQ with Ultra-low
Sun et al. 2020) in Transfomer base model on the WMT En-De task.
Model
Baseline
Ultra-low
LUQ
Transfomer-base
27.5
25.4
27.17
Forward-backwar ablations In Table 3 we show the top-1 accuracy in ResNet50 with different quantization schemes. The forward phase (activations + weights) is quantized to INT4 with SAWB (Choi et al. 2018a) and the backward phase (ncural gradients) to FP4 with LUQ. As expected, the network is more sensitive to the quantization of the backward phase.
5 MF-BPROP: MULTIPLICATION FREE BACKPROPAGATION
The main problem of using different datatypes for the forward and backward phases is the need to cast them to a common data type before the multiplication during the backward and update phases. In our case, the weights (W) and pre- activations (a) are quantized to INT4 while the neural gradients (&C) to FP4, where C' represent the loss function, ø a non-linear function and v the post-activation. During the backward and update phases, in each layer there are two GEMMs between different
7

---

Under review as a conference paper at ICLR 2022
datatypes:
[Forward] di = Win-1
Ul = p(ai)
(22)
aC
[Backward] duz-1 Diag(d ( (ai-1)
(23)
da
ow: = ac
ac
[Update]
das of
(24)
Regularly, calculate these GEMMs there is need cast both data types common data type (in our case, FP7 [1, 1, then do the GEMM and finally, the results are usually accumulated in a wide accumulator (Fig. Aap. This casting cost is not negligible. For example, casting INT4 to FP7 consumes ~ 15% of the area of an FP7 multiplier.
dealing with a special case where we do GEMM between a number without mantissa (neural gradient) and number without exponent (weights and activations), since INT4 is almost equivalent FP4 with format [1.0,3]. We suggest transforming the standard GEMM block (Fig. 4a) to Multiplication Free BackPROP (MF-BPROP) block which contains only transformation standard FP7 format (see Fig. 4b) and simple XOR operation. More details on this transformation appear in Appendix A.3) In our analysis (Appendix (A.4) we show the MF-BPROP block reduces the area the standard GEMM block by 5x. Since the FP32 accumulator still the most expensive block when training with a few bits, we reduce the total area in our experiments by 8%. However, as previously showed (Wang et al., (018)) 16-bits accumulators work well with 8-bit training, so is reasonable to think, it should work also with 4-bit training. In this case, the analysis (Appendix A.4 shows that the suggested ME-BPROP block reduces the total area by 22%.
In our case,
Standard CEMM block
Summation block
INTA
FP7 (1-4-2)
FP3 (1-4-2)
Casting to FP32
FP32 Actumu
multiplier
Casting to
(a)
MF- BPROPPRO block
Summation block
Transform to
Casting FP32
FP32
Accumalator
FP7 12-4-21
124
Figure 4: (a): Standard MAC block illustration containing the two main blocks one for GEMM and second for accumulator. The GEMM block for hybrid datatype in our case (FP4 and INT4) requires a casting to a common datatype before being inserted into the multiplier. (b): The suggested MAC block, which replace the multiplier with the proposed (FF-BPROP. Instead of doing an expensive casting followed by multiplication, we propose to make only a simple XOR and a transformation (Appendix A.3) reducing the GEMM area by 5x (Appendix) A.4
RELATED WORKS
Neural networks Quantization has been extensively investigated in the last few years. Most of the quantization research has focused on reducing the numerical precision of the weights and activations for inference (e.g., Courbariaux et al. (2016): Rastegari al. (2016 6); Banner et al. (2019); Nahshan et al. 2019); Choi et al. &b); Bhalgat et al. (2020); Choi et al. (2018a) Liang et al. (2021)). bits with
In standard ImageNet models, the best performing methods can achieve quantization
small or no degradation Choi et al. 2201 20) These methods can be used to reduce the computational resources in approximately third of the training. However, without quantizing the neural gradients,
8

---

Under review as a conference paper at ICLR 2022
cannot reduce the computational resources in the remaining two thirds of the training process. An orthogonal approach of quantization is low precision for the gradients of the weights in distributed training (Alistarh et al., 2016; Bernstein et al., 2018) in order to reduce the bandwidth and not the
training computational resources.
Sakr& & Shanbhag 20199) suggest systematic approach to design full training using fixed point quantization which includes mixed-precision quantization. |Banner et al. (2018) first showed that it is possible use INT8 quantization for the weights, activations, and neural gradients, thus reducing the computational footprint of most parts of the training process. Concurrently, Wang et al. (2018) was the first work to achieve full training in FP8 format. Additionally, they suggested a method to reduce the accumulator precision from 32bit to 16 bits, by using chunk-basee accumulation and floating point stochastic rounding. Later, Wiedemann et al. (2020) showed full training in INT8 with improved convergence, by applying stochastic quantization scheme to the neural gradients called non-subraatiive-diherrnn (NSD) that induce sparsity followed by stochastic quantization. Also, [Sun] al. 2019) presented a novel hybrid format for full training in FP8, while the weights and activations are quantized to [1.4.3] format, the neural gradients are quantized to [1,5,2] format to catch a wider dynamic range. Fournarakis & Nagel (202]) suggest method to reduce the data traffic during the
calculation of the quantization range.
While it appears that it is possible to quantize to 8-bits all computational elements in the training process, 4-bits quantization of the neural gradients is still challenging. Chmiel et al. (202T) suggested that this difficulty stems from the heavy-tailed distribution of the neural gradients, which can be approximated with a lognormal distribution. This distribution is more challenging to quantize in comparison to the normal distribution which is usually used to approximate the weights or activations (Banner et al. 2019).
Sun et al. (2020) was the first work that presented a method to reduce the numerical precision to 4-bits for the vast majority of the computations needed during DNNs training. They use known methods to quantize the forward phase to INT4 (SAWB (Choi et al. 2018a) for the weights and PACT (Choi et 2018b) for the activations) and suggested to quantize the neural gradients twice (one for the update and another for the next layer neural gradient) with non-standard radix-4 FP4 format. The use of the radix-4, instead of the commonly used radix-2 format, allows covering wider dynamic range. The main problem of their method is the specific hardware support for their suggested radix-4 datatype, which may limit the practicality of implementing their suggested data type.
Chen et al. (2020b) suggested reducing the variance in neural gradients quantization by dividing them into several blocks and quantizing each to INT4 separately. Their method requires each iteration to sort all the neural gradients and divide them into blocks, a costly operation that will affect the network throughput. Additionally, they suggested another method to quantize each sample separately. The multiple scales per layer in both methods do not allow the use of an efficient GEMM operation.
CONCLUSIONS
In this work, we analyze the difference between two rounding schemes: round-to-neane nearest and stochastic-rounding. We showed that, while the former has lower MSE and works better for the quantization of the forward phase (weights and activations), the latter is an unbiased approximation of the original data and works better for the quantization of the backward phase (specifically, the neural gradients).
Based on these conclusions, we propose logarithmic unbiased quantizer (LUQ) to quantize the neural gradients to format FP4 [1,3,0]. Combined with a known method for quantizing the weights and activations to INT4 we achieved, without overhead, state-of-the-an the- in 4-bit training in all the models we examined, e.g., 1.18 % error in ResNet50 vs 2.49 % for the previous known SOTA (Sun et al. (2020)) Moreover, we suggest two more methods to improve the results, with overhead comparable to (Snn et al. (2020). The first reduces the quantization variance, without affecting the unbiasedness of LUQ, by averaging several samples of stochastic neural gradients quantization. The second is a simple method for fine-tuning in high precision for one epoch. Combining all these methods, were able for the first time to achieve 0.64 % error in 4-bit training of ResNet50 ImageNet dataset.
Finally, we exploit the special formats used for quantization (INT4 in the forward phase and FP4 for- mat [1,3,0] in the backward phase) and suggest a block called MF-BPRO) that avoids multiplication during two thirds of the training, thus reducing by 5x the area previously used by the multiplier.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
A source code of the experiments appears in the supplementary material. Additionally, in Ap- pendix A.I we added the details of the experiments, including the hyper-parameters used.
REFERENCES
Dan Alistarh, Demjan Grubic, Jungshian Li, Ryota Tomioka, and M. Vojnovic. Qsgd: Communication- optimal stochastic gradient descent, with applications to training neural networks. 2016.
R. Banner, Yury Nahshan, and Daniel Soudry, Post training 4-bit quantization of convolutional networks for rapid-deploymen In NeurlPS, 2019.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In NeurIPS, 2018.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: compressed optimisation for non-convex problems. ArXiv, abs/1802.0444/ 2018.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2978-2985, 2020.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.
Jianfei Chen, Yu Gai, Zhewei Yao, Michael W Mahoney, and Joseph E Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. arXiv preprint Xi1::20000.14998, 2020a.
Jianfei Chen, Yujie Gai, 7. Yao, M. Mahoney, and Joseph Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. In NeurIPS, 2020b.
Brian Chmiel, Liad Ben-Uri, Moran Shkolnik, E. Hoffer, Ron Banner, and Daniel Soudry. Neural gradients are lognormally distributed: understanding sparse and quantized training. In ICLR, 2021.
Jungwook Choi, P. Chuang, Zhuo Wang. Swagath Venkataramani, V. Srinivasan, and K. Gopalakrish- nan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). ArXiv, Abs/1807.06964 2018a.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Chuang, V. Srinivasan, and K. Gopalakrish- nan. Pact: Parameterized clipping activation for quantized neural networks. ArXiv, 0bs//805500000 2018b.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 -1. arXiv e-prints, art. 1xXiv:1602.022 February 2016.
Marios Fournarakis and Markus Nagel. In-hindsight quantization range estimation for quantized training. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3057-3064, 2021.
Manfred Gilli, Dietmar Maringer, and Enrico Schumann. Chapter 6 generating random numbers. In Manfred Gilli, Dietmar Maringer, and Enrico Schumann (eds.), Numerical Methods and Optimiza- tion in Finance (Second Edition), pp. 103-132. Academic Press, second edition edition, 2019. ISBN
doi: /ttaa//eooooco..oooooooooooooooa
12-811065511-51 8.00177-0 URL https?
www. www.eceeeccccccc rect, com
Sasan Iman and Massoud Pedram. Logic Synthesis for Low Power VLSI Designs. Kluwer Academic Publishers, USA, 1997. ISBN 0792380762.
10

---

Under review as a conference paper at ICLR 2022
Olga Kupriaanva, Christoph Lauter, and Jean-Michel Muller. Radix conversion for iece754-2008 mixed radix floating-pon point arithmetic. 2013 Asilomar Conference on Signals, Systems and Comput- ers, Nov 2013. doi: 10. 10..1099acsxc2221368880000 ACSSC 2013.
URL CelLhttp:/ dx doi. www.do........111000
681 04401
Tailin Liang, C. John Glossner, Lei Wang, and Shaobo Shi. Pruning and quantization for deep neural network acceleration: A survey. ArXiv, abs/2101.096777 2021.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, and Avi Mendelson. Loss aware post-training quantization. arXiv preprint arXiv:1911.07190 2019. BtRL:ttp://aat wttp:://aaxvvvvoooo org. abs/////////////o///////////aae/ abs 911. C7190
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet
classification using binary convolutional neural networks. In ECCV, 2016.
Charbel Sakr and Naresh R Shanbhag. Per-tensor fixed-point quantization of the back-propagation algorithm. 2019.
Xiao Sun, Jungwook Choi, Chia- Yu Chen, Naigang Wang. Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NeurIPS, 2019.
Xiao Sun, Naigang Wang, Chia- Yu Chen, Jiamin Ni, A. Agrawal, Xiaodong Cui, Swagath Venkatara- mani, K. E. Maghraoui, V. Srinivasan, and K. Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. In NeurlPS, 2020.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Yu Chen, and K. Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In NeurIPS, 2018.
Simon Wiedemann, Temesgen Mchari, Kevin Kepp, and W. Samek. Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network training. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp- 3096-3104, 2020.
A APPENDIX
A.1 EXPERIMENTS DETAILS
In all our experiments we use the most common approach (Banner et al. 2018 Choi al.. (008b) for quantization where high precision of the weights are kept and quantized on-the-fly. The updates are done in full precision.
ResNet ResNext We run the models ResNet- 18, ResNet-50 and ResNext-50 from torchvision. We use the standard pre-prous pre-proeessing of ImageNet ILSVRC2012 dataset. We train for 90 epochs, initial learning rate of 0.1 with 0.1 decay epochs 30,60,80. We use standard SGD with momentum of 0.9 and weight decay of le-4 The minibatch size used is 256. Following the DNNs quantization conventions (Banner et al. 2018 Nahshan et al. 2019 Choi et al. (018b) kept the first and last layer (FC) at higher precision. Additionally, similar to Sun et al. (020 we adopt the full precision at the shortcut which constitutes only small amount of the computations (~ 1%). The "underflow threshold" in LUQ is updated in every bwd pass as part of the quantization of the neural gradients. In all experiments, the BN is calculated in high- precision.
We run Mobilenet V2 model from torchvision. We use the standard pre-processing of ImageNet 2LSV2 dataset. We train for 150 epochs, use an initial learning rate of 0.05 with a cosine learning scheduler. We use standard SGD with momentum of 0.9 and weight decay of 4e-5. The minibatch size used is 256. Following the DNNs quantization conventions (Banner et al. 2018
MobileNet V2
Nahshan et al. 2019 Choi et al. 2018b) we kept the first and last layer (FC) at higher precision. Additionally, similar to Sun et al. (2020) we adopt the full precision at the depthwise layer which constitutes only small amount of the computations (~~ 3%). The "underflow threshold" in LUQ is updated in every bwd pass as part of the quantization of the neural gradients. In all experiments, the BN is calculated in high-precision.
11

---

Under review as a conference paper at ICLR 2022
Transformer We run the Transformer- model based the Fairseq implementation the WMT 14 En-De translation task. We use the standard hyperparameters of Fairseq including Adam optimizer. We implement LUQ over all attention and feed forwards layers.
Table 3: ResNet-50 accuracy with ImageNet dataset while quantization different parts of the network. The forward phase is quantized to INT4 format with SAWB (Choi et al., (01 while the backward phase is quantized with the proposed LUQ. As expected, the quantization of the backward phase makes more degradation to the network accuracy.
Forward
Backward
Accuracy
FP32
FP32
76.5 %
INT4
FP32
76.35 %
FP32
FP4
75.6 %
INT4
FP4
75.32 %
A.2 ADDITIONAL EXPERIMENTS
LUQ requires the measurement of the maximum of the neural gradient in order to get the underflow threshold (Section 3 In Fig. 5a we compare the proposed LUQ with the method proposed in Fournarakis & Nagel 2021 which suggests reducing the data movement overhead that occurs in the calculation on-the-fly of the quantization ranges by using a running average of the previous iterations notice, the limited dynamic range in 4-bit quantization requires exact statisties
statistics. As measurement of the tensor since the proposed approximation induces significant accuracy degradation. The SMP method (Section 3.1 1 has a power overhead of of the number of additional samples since it influences only the update GEMM. In Fig. 5b we compare LUQ with one additional sample which has 33% power overhead with regular LUQ with additional 33% epochs. The scheduler was expanded respectively. We can notice that even both methods have similar overhead the variance reduction achieved with SMP more important for the network accuracy than increasing the training time.
70
60
Accuracy (%6
Accuracy (%))
40
30
Baseline
20
Baseline
LUQ
LUQ bits samples
20
In hihh
10
LUQ bits langer training
6D
100
150
200
Epechs
Epochs
(a)
(b)
Figure 5: (a): Comparison of the top-1 validation accuracy in ResNet-50 ImageNet dataset with the proposed LUQ and with the quantization dynamic range approximation in In-hind Fournarakis & (b): Comparison of ResNet- 18 bit training on Cifar 100 dataset of LUQ with 2 samples with longer training of regular LUQ. Both methods have similar overhead, but the SMP method leads to better accuracy.
Nagel (202)) 1) in 4-bit training.
A.3 TRANSFORM TO STANDARD FP7
We suggest method to avoid the of an expensive GEMM block between the INT4 (activation weights) and FP4 (neural gradient). It includes main elements: The first a simple xor operation between the sign of the numbers and the second is transform block to standard FP7 format. In
12

---

Under review as a conference paper at ICLR 2022
Fig. O we present an illustration of the proposed method. The transformation be explained with a simple example: for simplicity, we avoid the sign which requires only xor operation. The input arguments are 3 (011 bits representation in INT4 format) and 4 (011 bits representation in FP4 1-3-0 format). The concatenation brings to the bits 011 011. Then looking at the table the input column where the M=3 (since the INT4 argument 3) and get the results in FP7 format of 0100 10 E+1 2) which is 12 in FP7 (1-4-2) as the expected multiplication result.
In the next section, we analyze the area of the suggested block in comparison the standard GEMM block, showing 5x area reduction.
MF- BPROP block
INT4
Transform to
standard FP7 (1-4-2)
FP4
INT4
Exp
Mant
Exp
Mant
0
0
FP4
2
E11
E12
0
E12
Figure 6: Illustration of ME-BPROP block which replaces a standard multiplication. It includes: (1) simple xor operation between the sign. (2) A transform standard FP7 format. We present the table to make this transform E and M represent the bits of the FP4 and INT4 respectively without the sign. Exp and Mant the bits of the output exponent (4-bit) and mantissa (2-bit) of the output in FP7 format.
A.4 BACKPROPAGATION WITHOUT MULTIPLCATION ANALYSIS
In this section, we show rough estimation of the logical area of the proposed MF-BPROP block which avoids multiplication and compares with the standard multiplier. In hardware design, the logical area can be good proxy for power consumption (Iman & Pedram 1997). Our estimation doesn't 't include synthesis optimization. In Table 4 we show the estimation of the number of gates of standard multiplier, getting 264 logical gates while the proposed MF- --ROP block has estimation of 49 gates (Table 5) achieving a ~ 5x area reduction. For fair comparison we remark that in the proposed scheme the FP32 accumulator is the most expensive block with estimation of 2453 gates, however we helieve it can be reduced to narrow accumulator such as FP16 (As previously shown in Wang al. (2018) which have estimated area of 731 gates. In that case, reduce the total by 22%.
Table 4: Rough estimation of the number of logical gates for a standard GEMM block which contain two blocks: a casting to FP7 and a FP7 multiplier.
Block
Operation
Gates
Exponent 3:1 mux
12
Casting to FP7
Mantissa 4:1 mux
Mantissa multiplier
99
Exponent adder
37
Sign xor
FP7 [1,4,2) multiplier
Mantissa normalization
48
Rounding adder
12
Fix exponent
37
Total
264
13

---

Under review as a conference paper at ICLR 2022
Table 5: Rough estimation of the number of logical gates for the proposed MF-BPROP block.
Block
Operation
# Gates
Exponent adder
30
MF-BPROP
Mantissa 4:1 mux
18
Sign xor
Total
49
14


Reviews:
[{"review_id": "i3Qud3KPH59", "reviewer": "Reviewer_9cKC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is interesting, but leaves me with too many questions and uncertainties on all three of it's contributions to make a proper judgment. It would be great if the authors could address my questions below for clarity:\n\nI appreciate the rounding scheme analysis w.r.t. the MSE. I haven't seen that specifically before, and it's insightful to understand why stochastic rounding on the forward pass is not necessarily a good idea. There is one small caveat that should be mentioned, and that is that noise sometimes helps for proper convergence and regularization. Dropout adds noise, and increases the MSE, but still sometimes helps. It would be good if the authors rephrased this section slightly to take this into account properly. \n\nThe conclusion on page 3 is much too fast and needs to be worked out significantly more for clarity. Paraphrasing: \"Unbiased gradients are necessary for convergence, therefore gradients should be quantized with stochastic rounding\". I can't find the statements regarding the biased gradients in Bottou 2010, perhaps the authors could point me to where exactly this is mentioned. \nI would also like to understand better what the bias of the gradients actually means, and why this would be so detrimental. If you talk about the bias of the forward pass given the stochastic distribution, I understand. However, it's unclear to me what the effect of this is on the gradients themselves. It would be good if the authors worked this out. What are the gradients biased with respect to? The FP32 gradients? Is that even the correct thing to compare to?\nTo me, figure 1 is insufficient evidence that a 'bias' is necessarily the issue in training. This effect would have to be disentangled from other effects that adding noise to a network would have on training. As it stands, the difference could come from just the addition of noise.\nSimilarly, The argument that RDN should be used in the forward pass goes much too fast for me as well. It would be nice if the authors showed explicitly how the linearity causes a non-biased estimate of the gradients in the forward pass. \n\nSection 3.1 Extra sampling would also linearly increase the cost of the method. So why would this be better than simply training more iterations on different data as opposed to the same data? A discussion of the overhead/complexity of this method already here would be much appreciated. For the results section, I also don't see how the overhead comes into play with the comparison the raw LUQ method itself.\n\nSection 3.2 Wouldn't we generally want the activations in a lower bit-width as well?\n\nMethod - It is unclear to me if the weights in this scheme are kept in a quantized format, or if they are in floating points with the common 'shadow-weight' approach. Please take more care in describing this properly. If the weights are in floating point, that would mean a very significant overhead during training, as the floating points would have to be loaded somewhere, then quantized, then an operation is to be applied to them. In full quantized training, you would want to have the weights quantized as well, but I see no mention of this. Similarly, are the updates to the weights quantized? See e.g. the WAGE paper.\n\nUnderflow Threshold - What are you taking the maximum over? If you are taking a maximum over all the values in the tensor you are quantizing, you will run into problems with quantization as described in the In-Hindsight quantization paper (https://arxiv.org/pdf/2105.04246.pdf). Essentially, for activation quantization, if you need any statistic of a tensor to do the quantization, you have to write significant amounts of data to memory, which is very slow. You might as well not have done activation/gradient quantization at that point. \n\nOverhead of SMP and FNT. This number shouldn be 8x compared to FP16. Compared to FP32 it would be 16x at least. \n\nMain results. It seems you are using at least a different ResNet-18 model compared to the Ultra-low paper, as the baseline accuracy is 69.7% for your model, but Ultra-low reports 69.4. Could some difference not be explained based on diffent models? The delta in performance is the same, roughly 0.7 for ResNet18. Also, how does this method compare to other works before it like WAGE (https://arxiv.org/pdf/1802.04680.pdf) and other papers that are based on this?\n\nYou mention that the shortcut connection in ResNets and the depth-wise layers are kept in full precision. For the depth-wise layer, does that mean the input activations to it, or the output activations? For both, I don't think this is necessarily trivial to do. It's not a given that a fixed point training engine also has a full-precision engine on it's die. Having that would significantly increase the die size. Similarly, although the compute is small of this solution, the data movement is not. Data movement is a very important metric on efficient devices (the ones we would be doing quantized training for). This needs to be addressed in the context of overall efficiency.\n\nMF-BPROP - I definitely appreciate the hardware implementation, as if you'd have just the FP7 calculation MAC array, you might as well just do INT8 computations all the way as opposed to going throug the trouble of quantizing everything to INT4/FP4. But this section does highlight one problem with this method: Dedicated hardware is necessary for this method to work efficiently. This greatly limits the scope of this work in it's practicality. And this brings me back to the above data movement questions that were glossed over. The suggested method is not a general method for quantized training and more of a 'hardware and software go hand-in-hand' kind of method, but many of the practical details of the hardware implementation and costs/overhead are left out. Please comment on this.\n\nType-o's\npage 3 round-to-zero | nearest\npage 4 avoid of clipping | -of\nPage 5 The a different samples | -a\nPage 6 4-bit training in various DNN models | in -> on\n\n", "summary_of_the_paper": "The paper has a short analysis on rounding schemes, comparing nearest-rounding to stochastic rounding for quantized training. It also introduces LUQ, a new quantized training scheme with a FP4 format for the gradients. In order to make this work efficiently in hardware, they introduce a new hardware block that does multiplication-less gradient calculations for the backward pass. ", "summary_of_the_review": "This is an interesting paper with some interesting insights, but the paper tries to do too much in too small a space. This leads me to have too many open questions for a good rating. The paper would likely have been better if it honed in on a single aspect that they present, and did so in a clearer fashion with more bases covered.\nThe conclusions on page 3 are drawn much too fast, and are unclear. The authors have to convince the reader a lot harder to make the claims they make.\nOn the one hand the paper's main method only seemingly work more efficiently with very dedicated hardware implementations, but several important aspects of efficient hardware implementations of training, such as data movement, are seemingly ignored. \n\nI would be willing to increase my rating if the authors addressed my above questions, rewrote section 2 significantly with a more convincing analysis, and added a better complexity analysis of the overhead of their method in terms of data movement.\n\n*** post-rebuttal ***\n\nGiven the author's excellent response to my questions, and the questions of the other authors, I am increasing my review to a 5. For a 6 or higher I definitely need the efficiency conundrum resolved, as for now I just don't see how to resolve the dichotomy between being either a generally applicable method on common-day and general hardware, or a method that is really aimed at hyper-efficient dedicated implementations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fOa8Aab2yxk", "reviewer": "Reviewer_t7UD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While this paper is interesting and presents promising results, I did have many questions listed hereafter:\n\nIntroduction: Radix-4 representation is criticized arguing that conversion to radix-2 requires an explicit multiplication to modify exponent and mantissa. This is untrue, in the paper by Sun et al. that the authors cite, the radix-4 format does not use a mantissa, it only uses 3 exponent bits, and therefore converting to radix-2 simply requires appending the exponent field by a zero.\n\nSection 2: In the calculation of rounding variance and MSE, why is the input x considered deterministic?\n\nIn Section 2, conclusion, the following claim is made:  \"the forward phase should be quantized deterministically (using RDN) since stochastic rounding will not make help making the loss estimate unbiased (due to the non-linearity of the loss and activation functions) while unnecessarily increasing the mean-square-error\". Here the authors are claiming that the use of non-linear activations and loss will negate the fact that SR is unbiased. Why is that the case? Is this a known result from prior arts? If so, the authors should add a reference as was done in this same paragraph regarding the works of Chen et al., and Bottou.\n\nThe LUQ proposed in eq. (11)-(12) is not unbiased when n=b-1, i.e., in the quantization region corresponding to the largest magnitudes. Usually, such boundary cases do not matter, but given that here there are only 16 regions, and that the data is assumed to be heavy-tailed (most of the data would actually fall in this final region), this issue might be significant. Can the authors comment on what happens in the final quantization region?\n\nRelated to the above, this is more of a suggestion. Rather than setting an underflow value as quantization meta-parameter, why isn't the max of the tensor used instead? This would avoid the above issue. Have the authors considered that? Did tuning an underflow hyperparameter yield better results?\n\nIn eq.(14), why is a subtraction of a 1/2 term needed when RDN is defined as per eq.(5)? Also this equation is improperly typeset.\n\nOne final question: The issue of quantization bias occurring in gradients and weight updates has previously been studied in the paper below. Can the authors compare their findings and check if their conclusions are consistent with prior arts?\nSakr et al., Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm, in ICLR 2019.\n\nThe experimental results, and crucially, the implementation details in the Appendix look good.\n\nFinally, there are many typos and grammatical errors throughout the paper. I urge the authors to perform a spell check.", "summary_of_the_paper": "This paper proposes techniques to quantize the gradients in the back-propagation algorithm down to 4 bits. To do so, the authors propose the [1,3,0] radix-2 FP4 format, as well as bias removal and variance reduction techniques. In order to achieve SOTA accuracy, the method requires an additional stage of high precision fine-tuning.", "summary_of_the_review": "The paper is interesting, tackles an important problem, and presents promising results. Therefore, I do not recommend a clear reject. However, there are unfortunately many issues I though were unclear and hence raised in my detailed review. Therefore, I cannot recommend acceptance of the paper if these issues are not addressed. Hence, for now I am rating the paper as a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AF9gPAFVip4", "reviewer": "Reviewer_Sxpj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Strength **\n1. Overall the paper is well written, quite thorough, and well-cited. The theoretical analysis related to stochastic rounding and rounding-to-nearest are well justified.\n2. The paper achieved superior training results in 4-bit training.\n3. The authors not only provided the 4-bit training algorithm but also suggested dedicated hardware blocks.\n** Weaknesses **\n1. Some parts of the proposed method still need high-precision\n2. Some experimental results, MobileNet-V2 and ResNext-50, are not provided because of the time deadline.\n3. The hardware implementation costs are evaluated in terms of logical area, but I think it should also be analyzed in terms of memory area since the multiple sampling (SMP) and underflow threshold computation parts need additional memories.\n\n** minor comments **\n1. In Figure 1 (b) and (c), which rounding methods are adopted for Bwd and Fwd, respectively.\n2. In order to reduce variance, the author performs multiple samplings and averages them, but as the sampling increases, does not it get closer to the RDN?\n3. There is a typo on page 5. (The a different samples -> The different samples)\n4. The authors mentioned that \"we increase all network parts to full-precision, except the weights\" for the high-precision fine-tuning. If activation is also trained with full precision here, after fine-tuning, the activation precision should be lowered back to 4 bits.  Was there any performance degradation at this point?\n  ", "summary_of_the_paper": "The paper proposes some techniques to train a deep neural network in 4-bit and provides a theoretical analysis. The experimental results support the effectiveness of the proposed method. Although some part of the method still needs high precision, the Reviewer thinks that it could open a door to training the deep neural network with ultra-low bits.", "summary_of_the_review": "This paper proposed several techniques for effective 4-bit training. Some parts of the proposed method still require high-precision, but it might be a good starting point for full 4-bit training in the near future.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "37HR_5dg9HO", "reviewer": "Reviewer_na7b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrong points:\n* Theoretically well-motivated choice of quantization scheme for forward and backward path.\n* The logarithmic unbiased quantization method is theoretically sound and well explained.\n* Strong results for 4 bit training of various ImageNet models.\n* Clear ablation studies for the contribution of SMP and FNT in table 1.\n* Most parts of the paper are well written and easy to follow.\n\nWeak points:\n* It is unclear what the contributions of unbiased stochastic pruning and logarithmic unbiased rounding in LUC are. An ablation study would be insightful on this.\n* The SMP method for reducing variance requires further clarification. It is unclear where and when this exactly is applied. For example, are the multiple samples only used to calculate the weight update (similar to the second sample in Sun et al.) or is the average of multiple samples also used in back propagating to he other layers. If the latter, then this seems to likely imply that a higher bit width needs to be used in the matmul of the next layer. How does the multiple samples compare in performance and complexity to using a different bit width?\n* FNT: full-precision fine-tuning makes the comparison to other low-bit training methods unfair. But results are also show without it and show competitive results.\n* The transform to Standard FP7 lacks detail and explanation. It is hard to understand how the input/output table is constructed.\n* While the paper is easy to read and clearly written, many small details are missing, especially on the experimentation part (see questions below).\n\nQuestions:\n* What is the HW impact of rounding to nearest power? This seems significantly more complicated than adding uniform noise as it is the case for uniform quantization.\n* How do they define the pruning threshold? Are they learned/updated? I could not find any details on this in the paper.\n* How do you deal with BN?\n* What quantization approach is applied in the forward pass (weights and activations)? Rounding to nearest is clear, but how are the ranges defined/learned etc.\n\nEditorial notes: \n* Few appreciations introduced are unclear what they stand for. E.g SMP, FNT, RDN. The later I assume comes from rounding-to-nearest, but would than not RTN be the right appreciation?\n* Typo below equation 9 (rounding-to-zero).\n* Page 3: “will not make help making the loss estimate unbiased”", "summary_of_the_paper": "The paper focuses on reducing the computational footprint of training neural networks using quantization. They investigate the importance of having unbiased gradients and show that stochastic rounding is important for the backwards pass. They propose a logarithmic unbiased quantization (LUQ) scheme which achieves SOTA results for 4-bit training on ImageNet. Results are further improved with multiple sampling (SMP) and a final full precision fine-tuning (FNT) at the cost of extra compute overhead. Finally they introduce a HW block which exploits LUQ by avoiding multiplication which can reduce the multiplier area by 5 times.", "summary_of_the_review": "Overall a well written paper that is easy to follow (except a few parts highlighted above). The proposed approach is somewhat novel, it combines stochastic rounding with logarithmic quantization and stochastic pruning. The main strength are that most parts are well motivated and the strong 4 bit training results. The main weaknesses are around SMP and some other parts that are unclear. Overall the contributions slightly outweigh the weaknesses of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for 4-bit quantized training of NNs (forward and backward), obtaining SOTA 4-bit training quantization, motivated by an analysis of rounding schemes (an important aspect) in quantized training. The main concerns from the reviewers were that the approach was not practical (both a general concern, and of specific note here since the word is used in the title and motivation of the work), due to lack of compatibility with (current) general purpose hardware, and lack of suitability of the approach for specialized hardware, so it is unclear what the actual use case is for the approach. The authors argued that (1) this is not a problem on some hardware and (2) that past works have not been held to this standard. I did not find the authors to provide a strong argument during the discussion period to address these concerns.
Decision: Reject