Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
Authors: Brian Chmiel
Ron Banner
Elad Hoffer
Hilla Ben-Yaacov
Daniel Soudry
Venue: ICLR 2023 poster (2023)
Keywords: quantization
4bit
acceleration
compression

Abstract: Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only a third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers. 

Previous works separately showed that accurate 4-bit quantization of the neural gradients needs to (1) be unbiased and (2) have a log scale. However, no previous work aimed to combine both ideas, as we do in this work. Specifically, we examine the importance of having unbiased quantization in quantized neural network training, where to maintain it, and how to combine it with logarithmic quantization. Based on this, we suggest a $\textit{logarithmic unbiased quantization}$ (LUQ) method to quantize all both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training without overhead. For example, in ResNet50 on ImageNet, we achieved a degradation of 1.1 %. We further improve this to degradation of only 0.32 % after three epochs of high precision fine-tunining, combined with a variance reduction method---where both these methods add overhead comparable to previously suggested methods. 
A reference implementation is supplied in the supplementary material.

Published as a conference paper at ICLR 2023
ACCURATE NEURAL TRAINING WITH 4-BIT MATRIX MULTIPLICATIONS AT STANDARD FORMATS
Brian Chmiel
Ron Banner
Elad Hoffer
Hilla Ben Yaacov
Daniel Soudry
An Intel company, Caesarea, Israel, O "Department of Electrical Engineering Technion, Haifa, Israel
HHabana Labs
bochiel
rbanner, ehoffeey hbyaacov @habana. ai
daniel soudry @gmail com
ABSTRACT
Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers.
Previous works separately showed that accurate 4-bit quantization of the neural gra- dients needs (1) be unbiased and (2) have a log scale. However, no previous work aimed to combine both ideas, as we do in this work. Specifically, examine the importance of having unbiased quantization where to maintain it, and how combine it with logarithmic quantization. Based on this, we suggest logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phases 4-bit, achieving state-of-the-arr results in 4-bit training without the overhead. For example, in ResNet50 on ImageNet, we achieved degradation of 1.1%. We further improve this degradation of only 0.32% after three epochs of high precision fine-tuning, combined with a variance reduction method-where both these methods add overhead comparable to previously suggested methods. A reference implementation is supplied in the supplementary material.
quantized neural network training,
INTRODUCTION
Deep neural networks (DNNs) training consists of three main phases: the forward phase, backward phase, and update phase. Quantization has become one of the main methods to compress DNNs and reduce the GEMM computational resources. Previous works showed the weights and activations in the forward pass 4 bits while preserving model accuracy (Banner et al., 20119: Nahshan et al. 2019; Bhalgat et al.) 2020 Choi et al.) (0088)). Despite these advances, they only apply third of the training process, while the backward phase and update phase are still computed with higher precision.
(GEMM)
Recently, Sun al. 2020 was able, for the first time, to train DNN while reducing the numerical precision of most of its parts to 4 bits with some degradation (e.g.. 2.49% error in ResNet50). To do so, Su, et al. (2020) suggested non-standard radix-4 floating-pon quantization of the neural gradients (called two-phase rounding). This was an impressive step forward the ability to quantize all GEMMs training. However, since a radix-4 format is not aligned with conventional radix-2, any numerical conversion between the two requires explicit multiplication to modify both the exponent and mantissa. Thus, their non-standard quantization requires specific hardware support (Kupriianova al. 2013) that significantly reduce the benefit of quantization to low bits (Appendix A.6), and make it less practical.
format, combined with double
The main challenge in reducing the numerical precision of the entire training process is quantizing the neural gradients, i.c. the backpropagated error. Previous works showed separately that, to achieve accurate low precision representation of the neural gradients, it is important use: (1) Logarithmic

---

Published as a conference paper at ICLR 2023
quantization and (2) Unbiased quantization. Specifically, Chmiel et al. (2021) showed the neural gradients have a heavy tailed near-lognormal distribution and found an analytical expression for the optimal floating point format. At low precision levels. the optimal format is logarithmically quantized. For example, for FP4 the optimal format is gonneexponet,,mantiinn
= [1,3,0], i.e. without mantissa bits. In contrast, weights and activations are well approximated with Normal Laplacian distributions (Banner et al., 2209:; Choi et 20.12201a), and therefore are better approximated using uniform quantization (e.g., INT4). However, Chmiel et al. (2021) did not use unbiased quantization (nor did any of the previous works that use logarithmic quantization of the neural gradients (Li et al., 2020, Miyashita et [ Ortiz et al., (018)). Therefore, they were unable to successfully quantize in this FP4 format (their narrowest format was FP5).
Chen et al. (020a) showed that unbiased quantization of the neural gradients is essential to get unbiasedness in the weight gradients, which is required in SGID analysis convergence (Bottou et al. 2018). However, they focused on quantization using integer formats, as did other works that pointed out on the importance of being unbiased (Banner et al. 2018 Zhong et al., 2022). Naive quantization of the neural gradients using the optimal FP4 format (logarihhmic) results in biased estimates of the FP32 weight gradients- and this leads to severe degradation in the test accuracy. For example, major issue is that under aggressive (naive) quantization many neural gradients with magnitudes below the representable range are zeroed, resulting in biased estimates of the FP32 gradients and reduced model accuracy.
Using either logarithmic scale or unbiased rounding alone catastrophically fails at 4bit quantization of the neural gradients (c.g., see Fig. Dielow) Therefore, it is critical to combine them, as we do in this paper in Section 4 To do this, we stochastically quantize gradients below the representable range to either zero or the smallest representable magnitude a to provide unbiased estimates within that "underflow" range. Additionally. in order to represent the maximum magnitude without bias, we dynamically adjust so that the maximum can always be represented with an exponentiated scaling starting at a. Finally, to completely eliminate bias, we devise an efficient way to use stochastic rounding on logarithmic scale, on the values between a and the maximum. Together, this gradient quantization method is called Logarithmic Unbiased Quantization (LUQ), and for 4-bit quantization it uses numerical format with one sign bit, three exponent bits, and zero mantissa bits, along with stochastic mapping (to zero or a) of gradients whose values are below a and stochastic rounding within the representable range.
Main contribution LUQ, for the first time, combines logarithmic quantization with unbiased quan- tization for the neural gradients and does this efficiently using standard format. By additionally quantizing the forward phase to INT4, we enable, for the first time, an efficient scheme for "full 4-bit training", i.e. the weights, activations and neural gradients are quantized to 4-bit in standard formats (see Appendix [) so all GEMMs can be done in 4-bit, and also bandwidth can be reduced. As we show, this method requires little to no overhead while achieving state-of-the-art accuracy results: for example, in ResNet50 we get 1.1% error degradation with standard formats; in comparison, the previous method (Sun et al., 2020) had 2.49% error degradation but required non-standard formats, as well as additional modifications which have additional overhead.
Moreover, in Section we suggest two optional simple methods to further reduce the degradation, with some overhead: the first method reduces the quantization variance of the neural gradients using s-mmllling while the second is fine-tuning in high precision. Combining LUQ with these two proposed methods we achieve, for the first time, only 0.32% error in the 4-bit training of ResNet50. The overhead of our additional methods is no more than similar modifications previously suggested in Sun et al. (2020). Lastly, in Section 7 we discuss how to reduce remaining overheads such as data movement, scaling operations, and GEMM-related operations.
RELATED WORKS
Neural networks Quantization has been extensively investigated in the last few years. Most of the quantization research has focused on reducing the numerical precision of the weights and activations for inference (e.g.. Courbariaux et al. (2016); Rastegari et al. (2016); Banner et al. 2019): Nahshan et al. ) Choi et al. 8b); Bhal gat et al. (2020); Choi et al. (2018a); Liang et al. (2021)). In this case, for standard ImageNet models, the best performing methods can achieve quantization to 4 bits with small no degradation Choi al. (2018a) 8a): Sakr et al. (2022). These methods can
2

---

Published as a conference paper at ICLR 2023
be used to reduce the computational resources in approximately third of the training (Eq. !I) However, without quantizing the neural gradients, we cannot reduce the computational resources in the remaining two thirds of the training process (Eq. 2 and Eq. (3)). An orthogonal approach is low precision for the gradients of the weights in distributed training (Alistarh et al. 2016, Bernstein et al. 2018) in order to reduce the bandwidth and not the training computational resources.
Sakr & Shanbhag (219 suggest a systematic approach to design a full training using fixed point quantization which includes mixed-precision quantization. Banner et al. (2018) first showed that it is possible to use INT8 quantization for the weights, activations, and neural gradients, thus reducing the computational footprint of most parts of the training process. Concurrently, Wang et al.) (2018) was the first work to achieve full training in FP8 format. Additionally, they suggested a method to reduce the accumulator precision from 32bit to 16 bits, by using chunk-based accumulation and floating point stochastic rounding. Later, WWedemann et al. (2020) showed full training in INT8 with improved convergence, by applying a stochastic quantization scheme to the neural gradients called nont-subrrcctiv--iiheetet
(NSD). Also, |Sun et al. (2019) presented novel hybrid format for full training in FP8, while the weights and activations are quantized to [1,4.3] format, the neural gradients are quantized to [1,5,2] format to catch wider dynamic range. |Fournrakiss & Nagel (2021) suggested a method to reduce the data traffic during the calculation of the quantization range, using
the moving average of the tensor's statistics.
While it appears that it is possible to quantize to 8-bits all computational elements in the training process, 4-bits quantization of the neural gradients is still challenging. [Chieel et al. (202T) suggested that this difficulty stems from the heavy-tailed distribution of the neural gradients, which can be approximated with a lognormal distribution. This distribution is more challenging quantize in comparison to the normal distribution which is usually used to approximate the weights or activations (Banner al. 2019.. Different works (Li al., 2020 Miyashita et al.) 2016) tried to use logarithmic quantization for the neural gradients, however, they failed to quantize them unbiasedly.
Sun et al. (2200) was the first work that presented a method to reduce the numerical precision to 4-bits for the vast majority of the computations needed during DNNs training. They use known methods to quantize the forward phase to INT4 and suggested quantizing the neural gradients twice with a non-standard radix-4 FP4 format. The use of the radix-4, instead of the commonly used radix-2 format, allows for covering a wider dynamic range. The main problem with their method is the specific hardware support for their suggested radix-4 datatype, which may limit the practicality of implementing their suggested data type.
Chen al. (2020b) suggested reducing the variance in neural gradients quantization by dividing them into several blocks and quantizing each to INT4 separately. They require expensive sorting. Additionally, their per-sample quantization do not allow the use of an efficient GEMM operation.
3 BACKGROUND: QUANTIZATION FORMATS AND ROUNDING SCHEMES
Which quantization schemes should we use in 4bit training? Previous works (Choi et al. Banner et al. 2019) showed that weights and activations can be quantized INT4 with little to no accuracy degradation. In contrast, for the neural gradients, a recent work (Chmiel et al. showed analytically that the optimal format is logarithmic ([1,3.0). Combining all these schemes, we focus on full b-t training using standard formats, with the following three 4-bit quantized GEMMs:
2018 8a
202
[Forward] 21 QINT (W; (aN((a))):
ai = f(()
[Backward]
-t-1 = QINT (WT))rr((h)
6; (z)) o g
(2)
ac Qrr (6i)((a)))
[Update] ow,
(3)
where C is the loss function, O is a component-wveee wise product and, in each layer I, fi is the activation function, the weights (Wi) and activations (a)) are quantized with INT4 (QINT) while the neural gradients 5, A BC are quantized with logarithmic FP4 (Qrr), 2t are the pre-activations, and gi A ac 0zg
dag
Next, we aim to find which rounding scheme should we use in each quantizer. Thus, we study the effects of unbiased rounding during the three phases of training (Eqs. D and 3h. We show that rounding-ter to neanest (RDN) should be applied for the weights and activations (QINT), while the unbiased method of stochastic rounding (SR) is more suitable for the neural gradients (QFP).
3

---

Published as a conference paper at ICLR 2023
3.1 MEAN SQUARE ERROR COMPARISON
In this section, we show that, although stochastic rounding (SR) is unbiased, it generically has worse mean-square enror (MSE) compared round-to-nearest (RDN). Given that we want quantize x in a bin with a lower limit (x) and an upper limit u(x), stochastic rounding can be stated as follows:
[(x), u(z), w.p. p(x)) [(x) (()
w.p. p(x)
x-1
(R(x)
(4)
(())
The expected rounding value is given by E[SR(z)] = I(x) p(x) + u(x) (1 111ppaa p(x)) = x;
(5)
where here and below the expectation is over the randomness of SR (i.e., is deterministic constant). In Table we present the bias, variance, and MSE of RDN and SR. Full derivatives appear in Appendix A.2
Table 1: Comparison of the bias, variance, and MSE of two different rounding schemes: round-to- nearest (RDN) and stochastic rounding (SR)
Rounding
Bias
Variance
MSE
RDN
min (rr ((x), u(x) x))
O
[min (xr - [(x), u(x) - x²)
SR
(x - I(x)) (u(x)
x))
(x - ((x)) (u(x) x)
From Table I
since min(a, 6)2 S a 6 for every a, b, we have tha J
MSE (x)) 2 MSE[RDDN
Vx.
(6)
In Fig. la plot the mean-square-error for E 1). ((x:) 0, and u(x) 1. However, while round-to-nearest has a lower MSE than SR, the former is a biased estimator.
3.2 WHEN IS IT IMPORTANT TO USE UNBIASED QUANTIZATION?
To prove convergence, textbook analyses of SGD typically assume the expectation of the (mini-batch) weight gradients is sufficiently close to the true (full- bull-batch) gradient (e.g., assumption 4.3 in (Bottou jt al. 2018)). This assumption satisfidd when the weight gradients are unbiased. Next, we explain (as pointed out by previous works, such as Chen et al. (2020a)) that the weight gradients are unbiased when the neural gradients are quantized stochastically without bias.
Recall W, and fi are, respectively, the weights and activation function at layer I and C' is the cost function. Given an input-output pair (ar, y), the loss
C (y, fr. (r. (W..11 ... fa (W2fi (Wiz))....))
Backward and Update Phases Recall 2 and aj are, respectively, the pre- and post- activations of layer I, and 6 e dc Defining oy QPp():), we show in Appendix A.3 that the gradient of the weights in layer I is Vw, _ and its quantized form is Vw. o/ Therefore, the update Vw C. is an unbiased estimator of Vwi C:
E (V C.J] F. [8%
F. OT 8 VW.
Forward phase The forward phase is different from the backward and updates phases in that unbiasedness at the tensor level is not necessarily guarantee of unbiasedness at the model level since the activation functions and loss functions are not linear. For example, suppose we have two weight layers W1, W2, activation f. input x, and an SR quantizer Q. Then, despite that Q is unbiased (i.e., (Q(x) x), we get:
(ff x=) + ([
since f is non-linear. This means there is no point to use SR in the forward path since it will increase the MSE (Eq, (6)) but it will not fix the bias issue.
Note Fq. implies that [((=)MSF[(R(]))) 2 DGJ)M for any distribution p(z).
4

---

Published as a conference paper at ICLR 2023
3.3 CONCLUSIONS: WHEN TO USE EACH ROUNDING SCHEME?
Following the above results, the activation and weights quantization in the forward phase (QINT rT in eq. Eq. (TD) should use RDN. This is because SR will increase the MSE (as shown in Eq. 6 an increase which typically harms the final accuracy but will not help make the loss estimate unbiased, due to the non-linearity of the loss and activation functions (e.g.. Eq. (9). To avoid mismatch (and additional bias), we use RDN in QINT also in the other phases of training.
As we explained in section 3.2 unbiased neural gradients quantization leads an unbiased estimate of the weight gradients, which enables proper convergence of SGD (Bottou et al.) 2018). Thus, bias in the gradients can hurt the performance and should be avoided, even at the cost of increasing the MSE. Therefore, neural gradient quantization (QFP), should be done using SR rounding scheme, following subsection 3.2 In Figs. [b] I we see that these theoretical observations are consistent with empirical observations favoring RDN for the weights and activations (QINT) and SR for the neural gradients (QFP).
g
g
0.10
30
Baseline
30
Baseline
0.05
stachastic rounding Stnn--t--nearest
20
Owr SR
20
Ops SR
10
Owr
10
On RDN
0.00
0
75 100
0
100 125 Epochs
Epochs
(a)
(b)
(c)
Figure 1: Comparison between stochastic rounding (SR) and round-to-nearest (RDN) quantization.
In (a) we present the MSE of uniform distributed tensor with the two different rounding schemes. Quantization to 4 bits of the activations and weights (QINT) (b) and neural gradients (QFP) (c) of ResNet18- Cifar 100 dataset with SR and RDN. While MSE is important in QINT for the weights and activations, unbiasedness achieved with SR is crucial for the neural gradients in Qrp. The neural gradients in (b) and the weights and activations in (c), are in full precision to focus the effect of the rounding scheme only in one part of the network in each experiment.
LUQ: A LOGARITHMIC UNBIASED QUANTIZER
Following the conclusions from the previous section, the neural gradients quantizer (QFP eqs. and should be logarithmic and be completely unbiased. In this section we aim to do this efficiently and create, for the first time, Logarithmic Unbiased Quantizer (LUQ).
Standard radix-2 2 floating point defines a dynamic range. In standard FP, all the values below the minimum FP representation are pruned to 0 and all the values above the maximum FP representation are clipped to the maximum. In order to create fully unbiased quantizer, we need to keep all the following three regions unbiased: below range minimum, inside range, and above range maximum.
1) Below FP minimum: Stochastic underflow Given an underflow threshold a we define a stochastic pruning operator, which prunes given value x, as
if |x| 2 w.p.
Ta (x) = sign(x) a
if |
< a |zl
(10)
w.p. 1 la
if
< <a
2) Above FP maximum: Underflow threshold In order to create an unbiased quantizer, the largest
quantization value 22--1 should avoid clipping any values in tensor æ, otherwise this will create bias. Therefore, the maximal quantization value is chosen as max([=]). the minimal value which will avoid clipping and bias. Accordingly, the underflow threshold is (with b 3 for FP4)
max((x*) (x))
(11)
22
"There are some cases where adding limited noise, such as dropout, locally increases the MSE improves generalization. However, this is typically not the case, especially if the noise is large.
5

---

Published as a conference paper at ICLR 2023
Blased
Biased
Blased
FP
80
2
2
70
Blased
Unbiased
Blased
60
(86)
FP SR
50
20
2
2
40
Baseline
Unbiased
Unbiased
Blased
30
FP4
FP + SR + SU
20
FF4
2
INT4 Unbiased
Unbiased
Unblased
Unblased
0
LUg
LUQ
0 20
Epochs
40
60
80
a2
a2²
022
Figure 2: (Left): ResNer50 top-1 validation accuracy in ImageNet dataset with different quantization schemes for the neural gradients. FP4 refers to standard logarithmic (1-3-0) floating point quantization. SR refers to stochastic rounding, which makes the quantization unbiased inside the FP range. SU refers to stochastic underflow which makes the quantization unbiased below minimum FP. "INT4 unbiased" refers to the combination of INT4 and SR which fully unbiased. Notice that while biased logarithmic quantization ("FP4"), partially biased logarithmic ("FP4+ + SR", "FP4 + SR + SU"), and uniform unbiased ("INT4 unbiased") lead to significant accuracy degradation, the proposed fully unbiased logarithmic quantization ("LUQ") has a minimal degradation. (Right): Illustration of the different logarithmic quantization schemes for FP2 ([1.1,0) format), where 2 and 2 are, respectively, the minimal and maximal FP representations. Two arrows for the same point mean SR thicker lines represent higher probability. Only LUQ able to achieve unbiasedness in all floating point ranges.
The operation in Eq. (1I) can have some overhead, but in Section 7 we present two methods ("Reducing the data movement" and "Reducing the cost of the scaling operation") to decrease this overhead.
3) Inside FP range: Logarithmic SR Given an underflow threshold a, let (x) be FP round-to- nearest b-bits quantizer with bins fa, 2a. .... 2a.... 22"-'a2 aa). Assume, without loss of generality, 2"-la < x < a (n E 10. b- - We will use the following quantizer, which is a special case of SR (Eq. (4 and is unbiased as special case of Eq. 5
2 a w.p. 2
(x(x)
(12)
2 w.p.
2-22-1a 2
The naive implementation of stochastic rounding can be expensive since it cannot use the standard quantizer. Traditionally, in order to use the standard quantizer the implementation includes simply adding uniform random noise e ~ U[-2" 2
2"= 2 2"=²a] to x and then using a round-to-nearest opera- tion. The overhead of such stochastic rounding is typically negligible (Appendix AA.5.I in comparison to other operations neural networks training. Moreover, it is possible to reduce any such over- head with the re-use of the random samples (Appendix A.5.2 In our case, in order to implement logarithmic round-to-nearest we need to correct an inherent bias since a-2 logg(
[m] a )1 o a- 2 !i) )7.
For a bin [2n-1, 2"), the midpoint v. is
2n
2-
3
I
2
(13)
2
4
Therefore, we can apply cound-to-nearrst-powee (RDNP) directly on the exponent of any value
2n-1 < 2 < 2n as follows:
RDN = 2 lo (1 2n) = 21=+1og log (1)
) = (a- (1) -)) x 2.0N(0-0.081)
(14)
Notice that the use of RDNP avoids the need of converting back to linear space in order to implement SR and avoid additional computational overhead.
Logarithmic unbiased quantization (LUQ) LUQ, the quantization method we suggested above, unbiased since it can be thought of as applying logarithmic stochastic rounding (Eq. (12) top of stochastic pruning (Eq. (17h)
X, Q. (Ta()))
(15)

---

Published as a conference paper at ICLR 2023
Since Ta and Qa unbiased, Xg is unbiased estimator for 2; from the law of total expectation,
B[X,] E [Q, (Tx( E
[ [Q. (Ta [T, (x)) E [T. (Tx(a) x;
(16)
where the expectation is over the randomness of Ta and Qa.
In Fig. 2 (Left) we show ablation study of the effect of the different quantization schemes on ResNet50 in ImageNet: while standard (biased) FP4 diverges, adding stochastic rounding or stochastic underflow (which make the process partially unbiased) enables convergence, but with significant degradation. Combining logarithmic quantization with full unbiasedness in LUQ obtained minimal accuracy degradation. Notice also that only unbiasedness without logarithmic quantization ("INT4 unbias") completely diverges. In Fig. 2 (Right) we show an illustration of the different logarithmic quantization schemes, where only LUQ achieved a fully logarithmic unbiasedness FP quantization.
OPTIONAL METHODS
Next, we present two optional methods to improve accuracy some computational cost.
5.1 SMP: REDUCING THE VARIANCE WHILE KEEPING IT UNBIASED
In the previous section, we presented an unbiased method for logarithmic quantization of the neural gradients called LUQ. Following the bias- variance decomposition, if the gradients are now unbiased, then the only remaining issue should be their variance. Therefore, we suggest an optional method to reduce the quantization variance by repeatedly sampling from the stochastic quantizers in LUQ. and averaging the resulting samples of the final weight gradients. The proposed sampling can be implemented serially or in parallel. The serial implementation (re-using the same quantizer) has a power and throughput overhead but does not requires additional hardware (area) support, so it should be used if the chip area is the bottleneck. The parallel implementation avoids almost completely the throughput overhead (except the averaging operation), but it requires additional area for the multiple quantizers, so it should be used when the throughput is the bottleneck. For N different samples, the proposed method will reduce the variance by a factor of +. without affecting the bias (Gilli et al. 2019). In Appendix Fig. 5) we show the effect of the different number of samples (SMP) on 2-bit quantization of ResNet 8 Cifar1 100 dataset. There, with 16 samples, we achieve accuracy similar to full-precision network. This demonstrates that the variance is the only remaining issue in neural gradient quantization using LUQ and that the proposed averaging method can crase this variance gap. with some overhead.
different approach reduce the variance can be increase the bitwidth in the update phase. In order to keep using standard formats, we should increase the update phase to FP8-INT4, which leads to 3.5x degradation in compute density comparison to FP4-INT4 as shown in Sun et al. (2020) (Table s-1). Therefore, using the proposed SMP with two samples (as we shall do) has significant advantage in compute density (1.75x) in comparison to increasing the bitwidth.
5.2 FNT: TUNE-GUNING HIGH PRECISION
After the 4-bit training is finished, we suggest optional method to reduce the gap from the full precision model, by running T additional iterations in which we increase all the network parts to higher precision, except the weights which remain in low precision. We noticed that with this scheme we get the best accuracy for the fine-tuned network. At inference time the activations and weights are quantized to lower precision. During the fine-tune phase, the Learning Rate (LR) is increased linearly T during 7 iterations and then reduced linearly with the same slope:
LRT + (LI LRT)
i < T
LR,-
772
(17)
LRT
(T-s)
else
T/2
where LRy is the final LR of the 4-bit training and LRbase is the maximal LR of the fine-tune phase.
6 EXPERIMENTS
In this section, we evaluate the proposed LUQ for 4-bit training on various DNN models. For all models, we use their default architecture, pyree--as,,etees, and optimizers combined with
7

---

Published as a conference paper at ICLR 2023
custom-modified Pytorch framework that implemented all the low precision schemes. Additional experimental details appear in Appendix A.4
INT4 quantization INT4 quantization methods for the weights and activations (forward pass) were well studied in the past. In this paper, we used SAWB Choi et al. (2018s) 2018a) to quantize the weights and activations. SAWB determines the quantization scaling factor by first finding the optimal (in terms of MSE) scaling factor on six distribution approximations of the true tensor distribution, and then applying linear regression to find the chosen scaling factor.
Training time measurement Notice that, currently, Al accelerators do not support 4-bit formats for training. This means that we can only simulate the quantization process, but are not able to measure training time or memory reduction. This is the common practice in the neural network quantization literature, where the algorithms often appear before the hardware that can support them. For example, though we can find FP8 training publications since 2019 (Sun et al. 2019), only recently did Nvidia announce their first GPU that supports the FP8 format (H100).
Main results In Table 2 we show the Top-1 accuracy achieved in 4-bit training using LUQ to quantize the neural gradients to FP4 and combined with a previously suggested method, SAWB (Choi et al. (201 8a), to quantize the weights and activations to INT4. We compare our method with Ultra-low (Sun et al. (020) showing better results all the models, achieving SOTA in 4-bit training. Moreover, improve the results by using the proposed SMP (Section[5.1] In Table 3 we show the effect of the proposed fine-tuning, reducing or closing completely the gap from full full-precision model. We verified that stochasticity has only negligible effect on the variance of final performance by running a few different seeds. Additional experiments appear in Appendix A.5
Table 2: Comparison of 4-bit training of the proposed method LUQ with Ultra-low (Sun al. 2020
in various vision models with ImageNet dataset, bransformer-base in WMT En-De task dataset and BERT fine-tune in SQUAD dataset. SMP refers doing two samples of the SR quantization of neural gradients in order to reduce the variance (Section 5.1
Model
Baseline
Ultra-low
Sun et al.
2020
LUQ
LUQ SMP
ResNet-18
69.7 %
68.27%
69.09%
69.24 %
Re
76.5%
74.01%
75.42 %
75.63 %
MobileNet-V2
71.9
68.85 85 %
69.55 %
69.7
ResNext-50
77.6
N/A
76.02 %
76. 12 %
Transfomer-base
27.5 (BLEU)
25.4
27.17
27.25
BERT fine-tune
87.03 (FI)
N/A
85.75
85.9
B
76.2
N/A
73.7 %
74.1 %
Table Effect of the proposed FNT method (Section 5.2)
using FP16 format with different epochs.
Model
Baseline
LUQ SMP
+FNT epoch
+FNT epochs
+FNT epochs
ResNet-18
69.7 %
69.24 %
69.7 %
ResNet-50
76.5 %
75. 63%
75,89 %
76 %
76.18 %
MobileNet-V2 V2
71.9 %
69.7 %
70.1 %
70.3 %
70.3 %
ResNext-50
77.6 %
72.12%
76.25 %
76.33 %
76.7 %
Overhead of SMP and FNT We limit our experiments with the proposed SMP method to only two samples. This is to achieve similar computational overhead as Ultra-low(Sun et al.) 2020). with their suggested two-phase-rounding (TPR) which also generates a duplication for the neural gradient quantization. Additional ablation study of the SMP overhead appears in Appendix A.5 The throughput of a 4-bit training network is approximately 8x in comparison to full precision training (Sun throughput by ~ 8%. In comparison. Ultra-low (Sun et al. 2020) does full- -trriiiiing with all the Ixl convolutions in 8bit, which reduces the throughput by 50% in comparison to all 4bit training.
2020). This means that doing one additional epoch in high precision reduces the
Recall Ulw used non-standard radix-4 quantization format, that significantly reduces the benefit of using low bit quantization.
8

---

Published as a conference paper at ICLR 2023
Forward-backwan ablations In Appendix Table we show the Top-1 accuracy in ResNet50 with different quantization schemes. The forward phase (activations + weights) is quantized to INT4 with SAWB (Choi et al.) 2 and the backward phase (neural gradients) to FP4 with LUQ. As expected, the network is more sensitive to the quantization of the backward phase.
7 DISCUSSION
Conclusions In this work, we analyze the difference between two rounding schemes: round-to- nearest and stochastic-rounding. We showed that, while the former has lower MSE and works better for the quantization of the forward phase (weights and activations), the latter is an unbiased approximation of the original data and works better for the quantization of the backward phase (specifically, the neural gradients).
Based on these conclusions and previous works (Chmiel et al. (2021) that showed the optimally of logarithmic quantization, we propose the first method that combined logarithmic quantization with unbiasedness, with the proposed logarithmic unbiased quantizer (LUQ) which quantize the neural gradients to format FP4 [1,3,0]. Combined with a known method for quantizing the weights and activations to INT4 we achieved, without overhead, state-of-the-ar in standard format 4-bit training in all the models we examined, e.g.. 1.1% % error in ResNet50 vs. 2.49% for the previous known SOTA (Sun et al. (020)), which used non-standard format).
Moreover, we suggest two more methods to improve the results, with overhead comparable to Sun et al. 2020). The first reduces the quantization variance, without affecting the unbiasedness of LUQ. by averaging several samples of stochastic neural gradients quantization. The second is a simple method for fine-tuning in high precision for one epoch. Combining all these methods, we were able for the first time achieve 0.32% error in 4-bit training of ResNet50 ImageNet dataset.
Reducing the data movement So far, we focused on resolving the 4-bit GEMM operation bottle- neck in DNNs training. It reduces not only the computational resources for the GEMM operations, but also reduces the required memory in DNNs training. However, LUQ, similarly to previous quantization methods (Sun et al., 2020 Choi et al. 2018b(a) requires a statistical measurement of the tensor to define the quantization dynamic range. Specifically, LUQ requires a measurement of the maximum of the neural gradient tensor. Such measurement increases the data movement from and to memory, making this data movement potential bottleneck in some hardware.
In order to avoid this bottleneck when it is a major issue, we verified that LUQ can be combined with the In-hindsight Fournarkkis & Nagel (2021) statistics estimation method, which uses pre-computed measurement to quantize the current tensor and in parallel extract the current statistics for the next iteration. The maximum estimate in LUQ, mn is calculated as:
m f = (1 - n) max([=' + 17. m 1
(18)
where is the momentum hyperparameter, and æ is tensor of neural gradients. In Appendix A.5.5 we show the effectiveness of this statistic estimation, which eliminates this data movement bottleneck, with negligible change to the accuracy. Though this method can potentially introduce some bias to the quantization process, the bias seems to be negligible (see Appendix Fig. Z
present an optional method to convert the underflow threshold a (Eq. (TT) to a power-of-two, at a small cost to accuracy. This can
Reducing the cost of the scaling operation In Appendix A.7
reduce or eliminate the computational overhead of the multiplication by in eq. Eq. (2). which can
be significant in some cases.
Multiplication free backpropagation In this work, we reduce the GEMM bottleneck by combin- ing two different data-types for the forward (INT4) and backward (FP4) passes. Standard GEMM operations with different formats, require casting the operand to a common datatype before the multiplication. The cost of the casting operation can be significant. We notice, that we are dealing with special case, where one of the operands includes only a mantissa (weights and activations) and the other only an exponent (neural gradients). Our initial analysis (Appendix A.8) shows that this allows, with small hardware changes, reduction of the area of standard GEMM block by 5x.
Accumulation width A different future direction is to reduce the accumulator width, which is usually kept FP32. As explained in Appendix A.8 the FP32 accumulator is the most expensive block when training in low bits. Now, after allowing training with 4-bit it is reasonable to think that the accumulator width can be reduced.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436), Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of Schmidt Career Advancement Chair in AI.
REFERENCES
Greaphcore
ipu.
URL
http:://bos.raaatcceea////ppppge
tensor lww -se--ser---e www.l/ueeeuuuleeee/meee http://wwwr..aaaa//////////////a
Habana URL httpa///a///aa/ooo/////aaa/a///a
Tesla
static/ technology.
dojo
URL.
logy https://essla-cchhhhrrooo com/
http://aw_..aaaaaaa.ooooooooo//r -oeella--djo-techooooo
Dan Alistarh, Demjan Grubic, Jungshian Li, Ryota Tomioka, and M. Vojnovic. Qsgd: Communication- optimal stochastic gradient descent, with applications training neural networks. 2016.
R. Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for repid-depl In NeurlPS, 2019.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In NeurlPS, 2018.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: compressed optimisation for non-convex problems. ArXiv, abs/1 102.044344 2018.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2978-2985, 2020.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.
Jianfei Chen, Yu Gai, Zhewei Yao, Michael W Mahoney, and Joseph E Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. arXiv preprint 17X2 2020a.
Jianfei Chen, Yujie Gai, Z. Yao, M. Mahoney, and Joseph Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. In NeurIPS, 2020b.
Brian Chmiel, Liad Ben-Uri, Moran Shkolnik, E. Hoffer, Ron Banner, and Daniel Soudry. Neural gradients are lognormally distributed: understanding sparse and quantized training. In ICLR, 2021.
Jungwook Choi, P. Chuang, Zhuo Wang, Swagath Venkataramani, V. Srinivasan, and K. Gopalakrish- nan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). ArXiv, Abs/1807.00664 2a.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Chuang, V. Srinivasan, and K. Gopalakrish- nan. Pact: Parameterized clipping activation for quantized neural networks. ArXiv, has//805506008 2018b.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. arXiv e-prints, art arXiv:1602.02887 February 2016.
10

---

Published as a conference paper at ICLR 2023
Marios Fournarakis and Markus Nagel. In-hindsight quantization range estimation for quantized training. 2021 /EEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3057-3064, 2021.
Manfred Gilli, Dietmar Maringer, and Enrico Schumann. Chapter 6 generating random numbers. In Manfred Gilli, Dietmar Maringer, and Enrico Schumann (eds.), Numerical Methods and Optimiza- tion in Finance (Second Edition), pp. 132.-32.000 Academic Press, second edition edition, 2019. ISBN
978-0-12-815065-1 doi: http://d....ooooooooo/////66626. 0001700. URL https
/ www. scissiiiiee encedi rect. com http://cw.....eeeeeeeaaaaaaaaa/a wtt.a//waa....aaa...............
Sasan Iman and Massoud Pedram. Logic Synthesis for Low Power VLSI Designs. Kluwer Academic Publishers, USA, 1997. ISBN 0792380762.
Olga Kupriianova, Christoph Lauter, and Jean-Michel Muller. Radix conversion for ieee754-2008 mixed radix floating-ponn arithmetic. 2013 Asilomar Conference on Signals, Systems and Comput- ers, Nov 2013. doi: 10. 109/3cssc.2013.68 104771 URL hww.:http///// //x doi wrt./..gg1111000 /9/ ACSSC. 2013. 6410
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. In ICLR, 2020.
Tailin Liang, C. John Glossner, Lei Wang, and Shaobo Shi. Pruning and quantization for deep neural network acceleration: A survey. ArXiv, abs/2101.0900 2021.
Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. ArXiv, abs/1603.01025, 2016.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. ArXiv, abs/2 106.0006.9) 2021.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, and Avi Mendelson. Loss aware post-training quantization. arXiv preprint 19Xiv:191/0 1,00190, 2019. URLp:ttp://axxivccm
C7190
Mare Ortiz, Adrián Cristal, Eduard Ayguadé, and Marc Casas. Low-precision floating-point schemes for neural network training. ArXiv, abs/1804.05267, 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Charbel Sakr and Naresh R Shanbhag. Per-tensor fixed-point quantization of the back-propagation algorithm. 2019.
Charbel Sakr, Steve Dai, Rangharajan Venkatesan, Brian Zimmer, William J. Dally, and Brucek Khailany. Optimal clipping and magnitude-aware differentiation for improved quantization-awwa aware training. In ICMI., 2022.
Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang. Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NeurlPS, 2019.
Xiao Sun, Naigang Wang, Chia- Yu Chen, Jiamin Ni, A. Agrawal, Xiaodong Cui. Swagath Venkatara- mani, K. F. Maghraoui, Srinivasan, and K. Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. In NeurlPS, 2020.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Yu Chen, and K. Gopalakrishnan. Training deep neural networks with b-bit floating point numbers. In NeurIPS, 2018
Simon Wiedemann, Temesgen Mchari, Kevin Kepp, and W. Samek. Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network training. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4096-3104 2020.
Kai Zhong, Xuefei Ning, Guohao Dai, Zhenhua Zhu, Tianchen Zhao, Shulin Zeng, Yu Wang, and Huazhong Yang. Exploring the potential of low-bit training of convolutional neural networks. IEEE Transactions Computer-Aidee Design of Integrated Circuits and Systems, 2022.
11


Reviews:
[{"review_id": "P_TjFXF44K", "reviewer": "Reviewer_8T7H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a 4-bit quantization method (LUQ) for matrix multiplication in training deep learning models. This method uses 4-bit integer quantization for weight and activation quantization, and use the proposed logarithmic unbiased quantization (LUQ) for gradient back  propagation. The major contributions are the following:\n\n1. A study showing that at the 4-bit quantization level for matmul, stochastic quantization is preferred for backward pass while the nearest rounding presents better accuracy when applied in forward pass.\n\n2. The first training pipeline with fully 4-bit matmul using standard format. With the LUQ design, this pipeline does not require special techniques like Radix-4 or double pass rounding [Sun et al. 2020]. Yet, it gives the SoTA 4-bit training accuracy with small accuracy degradations to single precision training on several NLP and vision tasks.\n\n3. Two practical techniques to further improve the accuracy of 4-bit training 1) Variance reduction with resampling for stochastic rounding 2) high precision continuous training.\n\n\nReference: \n[Sun et al. 2020] Ultra-Low Precision 4-bit Training of Deep Neural Networks", "summary_of_the_review": "Overall I recommended marginally-above-acceptance. I think the LUQ method provides ways to achieve 4-bit matmul in forward and backward pass using standard hardware and software techniques, which opens the door towards system and hardware practical 4-bit training. The writing and technical discussion could be further improved by resolving clarity comments above. I would highly recommend some centralized discussion about practical way to determining the threshold (\\alpha) in LUQ method. If there is a simple yet effective way to achieve this, the impact will be much stronger.", "strength_and_weaknesses": "Strength\n\n1. The effort and design towards the simplicity is impressive. The key of the practical use of low precision training is the simplicity which could be efficiently implemented and embraced by the HW/SW practitioners. Previous methods needs very special hardware design or algorithmic pieces that could trigger more compute overhead than the savings of going to 4-bit. The LUQ method uses standard Radix-2 and just standard way of stochastic rounding. By choosing the right rounding method for forward and backward pass. This simple method could achieve stronger accuracy than previous method.\n\n2. The two optional methods to improve 4-bit accuracy is practical and potentially lead to impact on future hardware designs to accommodate these considerations.\n\n\nWeakness\n\nSome of the technical and writing aspects need some more clarity to be fully evaluated.\n\n1. Around Equ 13 and 14, the authors discussed a method to correct the inherent bias in directly rounding the exponent. Is the intention to discuss an exact or approximated approach to correct the bias yet still preserve the hardware efficiency of directly rounding the exponents? I ask this question mostly because that converting back to linear space from logarithmic space and then do the stochastic rounding might trigger additional computation overhead. If this discussion intends to tell how rounding on exponent can achieve bias correction, further elaborations are needed to clarify how the bias correction is achieved.\n\n2. The value of \\alpha in Equ 18 determines the exact interval end point values of LUQ. What is the way of determining \\alpha in practice? Seems from the beginning of page 6, \\alpha is determined by the gradient value range. Does this need dataset / model specific values in practice? If yes, a discussion on the method to determination in the main text or appendix would would be useful. This is important for understanding the practical overhead of enabling the 4-bit training using LUQ. \n\n3. Around Equ 15, the authors claimed the quantized gradient is unbiased. It seems this is under the assumption that the activation is unbiased. If this is the case the assumption should be elaborated to avoid conflations with the fact that LUQ suggest biased forward activation quantization.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KwWy55Ds0q", "reviewer": "Reviewer_Qvzo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to train neural networks with 4-bit matrix multiplications. The authors propose logarithm unbiased quantization (LUQ), which combines stochastic rounding and logarithm quantization, both of which are known to be required for extreme low-bit training. The proposed training algorithm involves 4-bit deterministic integer quantization for weight and activation and 4-bit LUQ for gradients. The authors further propose some optional methods to improve the accuracy, including averaging the quantized gradient (SMP) and high-precision fine-tuning (FNT). On various computer vision and natural language processing benchmarks, the proposed approach can train with 4-bit numerical formats, with superior accuracy than previous FP4 training. ", "summary_of_the_review": "Though the novelty is somewhat thin. I still think this is a good paper due to \n1. Its simplicity and effectiveness;\n2. Its reproducibility. At least the official code is provided, which I suspect to be the first open-sourced recipe for 4-bit training.", "strength_and_weaknesses": "Strength:\n- The proposed approach is simple. \n- The proposed approach achieves good accuracy, the result is impressive.\n- Training with low-bit arthemetic is a major problem. The paper can be significant.\n- The experiments cover a wider range of applications than other papers in the field.\n\nWeaknesses:\n- The novelty is somewhat thin: Until the second half of page 5, the paper is mostly presenting existing backgrounds. The novelty mainly falls in Sec. 4. But the LUQ itself is rather straightforward to design, once the goal of designing logarithmic and unbiased quantizer is clear. The approaches in Sec. 5 are also rather standard and to some extent explored in previous literature. I'd say the main contribution of this paper is   showing that such a simple combination of existing techniques is sufficient to achieve (surpringly good) accuracy, rather than proposing novel techniques. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0jc-ioHn7Kz", "reviewer": "Reviewer_8NA6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes techniques to quantize the gradients in the back-propagation algorithm down to 4 bits. To do so, the authors propose the LUQ format, as well as bias removal and variance reduction techniques. In order to achieve SOTA accuracy, the method requires an additional stage of high precision fine-tuning.", "summary_of_the_review": "The paper is interesting, tackles an important problem, and presents promising results. I raised some questions and hope to receive responses from the authors.\n", "strength_and_weaknesses": "While this paper is interesting and presents promising results, I did have a few questions listed hereafter:\nSection 3: In the calculation of rounding variance and MSE, why is the input x considered deterministic?\n\nIn Section 3's conclusion, the authors claim that the quantization MSE has to be minimized in the forward pass, which is a reasonable claim. Some works have shown how to provably do that for INT4 [1]. Can this work be compared to the proposed technique?\n[1] Sakr, Charbel, et al. \"Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training.\" International Conference on Machine Learning. PMLR, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster