Improved Fully Quantized Training via Rectifying Batch Normalization
Authors: Kaixin Xu
Jie Lin
Zhe Wang
Peng Hu
Ziyuan Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Model Compression
Gradient Quantization
Convolution Neural Networks
Batch Normalization

Abstract: Quantization-aware Training (QAT) is able to reduce the training cost by quantizing neural network weights and activations in the forward pass and improve the speed at the inference stage. QAT can be extended to Fully-Quantized Training (FQT), which further accelerates the training by quantizing gradients in the backward pass as back-propagation typically occupies half of the training time. Unfortunately, gradient quantization is challenging as Stochastic Gradient Descent (SGD) based training is sensitive to the precision of the gradient signal. Particularly, the noise introduced by gradient quantization accumulates during backward pass, which causes the exploding gradient problem and results in unstable training and significant accuracy drop. Though Batch Normalization (BatchNorm) is a de-facto resort to stabilize training in regular full-precision scenario, we observe that it fails to prevent the gradient explosion when gradient quantizers are injected in the backward pass. Surprisingly, our theory shows that BatchNorm could amplify the noise accumulation, which in turn hastens the explosion of gradients. A BatchNorm rectification method is derived from our theory to suppress the amplification effect and bridge the performance gap between full-precision training and FQT. Adding this simple rectification loss to baselines generates better results than most prior FQT algorithms on various neural network architectures and datasets, regardless of the gradient bit-widths used (8,4, and 2 bits).

Under review as a conference paper at ICLR 2023
IMPROVED FULLY QUANTIZED TRAINING VIA RECTI- FYING BATCH NORMALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Quantization-awar Training (QAT) is able reduce the training cost by quantiz- ing neural network weights and activations the forward pass and improve the speed the inference stage. QAT can be extended Fully-Quantized Training (FQT), which further accelerates the training by quantizing gradients in the back- ward pass backpropagation typically occupies half of the training time. Un- fortunately, gradient quantization is challenging as Stochastic Gradient Descent (SGD) based training is sensitive to the precision of the gradient signal. Particu- larly, the noise introduced by gradient quantization accumulates during backward pass, which causes the exploding gradient problem and results in unstable train- ing and significant accuracy drop. Though Batch Normalization (BatchNorm) is a de-facto resort to stabilize training in regular full-precision scenario, we observe that it fails to prevent the gradient explosion when gradient quantizers are injected in the backward pass. Surprisingly, our theory shows that BatchNorm could am- plify the noise accumulation, which in turn hastens the explosion of gradients. A BatchNorm rectification method is derived from our theory to suppress the ampli- fication effect and bridge the performance gap between full-precision training and FQT. Adding this simple rectification loss to baselines generates better results than most prior FQT algorithms on various neural network architectures and datasets, regardless of the gradient bit- bit-widths used (8,4, and 2 bits).
INTRODUCTION
Quantization-awae aware Training (QAT) a popular track of research that simulates the neural network quantization (weights and activations) during the course of training to curb the inference-time accu- racy drop of low-bit models (e.g. INT8 quantization). On the other hand, theoretical calculations on the BitOps (Yang & Jin (22T); Guo et al.) (2020) computation costs can easily conclude that backpropagation accounts for half of the computations during training. Empirical data" back- ward pass sometimes even costs more in practice. Decreasing the gradient bit-widthh will apparently reduce computation overheads of backpropagation Horowitz (2014). If variables in backward pass also quantized, adding up the forward quantization in QAT. all the network variables required in training would be fully quantized and the whole training process could be accelerated on dedicated hardware, i.e., Fully-Quantizer Training (FQT), providing huge accessibility of large model training to users with limited computation capability. Recent work Zhuu et al. (2020) has shown that INT8 FQT speeds up the forward pass and the backward pass by 1.63 and 1.94x respectively when training ResNet-50 on ImageNet with NVIDIA Pascal GPU.
Yet gradient quantization under the FQT scheme is vastly underexplored, as it is notoriously more challenging than forward quantization in QAT. It observed that network training is sensitive to the precision of gradients, and low-bit gradient quantization leads to unstable training and significant accuracy drop (see Fig. [ More importantly, the accumulation of gradient quantization noise in backward pass (see Fig. 2} causes the exploding gradient problem during hackpropagation, even resulting in training failure. In contrast to weight/activation quantization, gradient quantization noise produced during backpropagation cannot be automatically corrected by optimizing objective loss.
Unlike prior works optimizing gradient quantizers for quantization noise reduction Zhou et al. 2016); Choi al. 2018); Zhu et al. (2020), this paper reveals the negative effect of Batch Nor-

---

Under review as a conference paper at ICLR 2023
malization (BatchNorm) amplifying the gradient quantization noise accumulation, when training deep Convolutional Neural Networks (CNNs) with low bit gradients. We show that the noise amplifi- cation effect further explodes the gradients during the backward pass. We thus propose a BatchNorm rectification method to suppress the noise amplification effect and alleviate the gradient explosion problem, which in turn enables stabilized training and better accuracy at low-hit gradients.
Our contributions are summarized as follows:
We discover that BatchNorm fails to prevent the exploded low-bit gradients in full- quantized training through theoretic analysis, and may even amplify the accumulated gra- dient quantization noise, which further aggravates the gradient explosion.
According to our theory, we propose simple yet effective BatchNorm variance rectifica- tion algorithm without introducing noticeable overhead, suppress the noise amplification effect, resulting in alleviated gradient explosion.
Extensive experiments on MNIST, CIFAR-10, and ImageNet show that our method achieves improved training and higher accuracy state-of-the-arts with vanilla gradi- ent quantizers, regardless of the gradient bit-widths used (8,4,2 bits).
2 RELATED WORK
Quantization-Awar Training (QAT). DoReFa-Net Zhou al. (2016) proposed to optimize the clipping value and the scaling factor of the uniform quantizers for weights and activations separately. was validated on image classification task under multiple bit-widths, but only with the rather sim- ple AlexNet architecture. PACT et al. (2018) proposed to quantize the activations with learn- able layer-wise clipping value, which not surprisingly achieved better accuracy than DoReFa-Net at 5-bit down to 2-bit. Most QAT works quantize weights and activations simultancously. by opti- mizing the uniform quantization parameters Zhang et al. (2018a) Esser et al. (2020); Bhalgat et al. 2020), layer-wise or channel-wiss mixed- preciiion quantization Jin et al. (2020): Lou ct al. (2020), or leveraging non-uniform quantization such as Logarithmic quantizer Miyashita et al. (2016) and piece-wise linear quantizer Fang al. (2020). Most recent QAT works [hou et al. (2016); Choi ct al. 2018); Zhang et al. (201 8a); Esser et al. (2020): Bhalgat et al. (2020) used "Straight- Throuug Estimator" (STE) Bengio et al. (2013) to estimate the gradient of the non-differentiable quantization function, while other work Gong et al. (219)) softened the linear quantization operation in order to match the true gradient with STE.
qully-quantized Training (FQT). FQT aims to accelerate and quantize the backward pass of net- work training with low-bit error signals and gradients, agnostic to single machine or parallel training- Early attempt Zhou et al.) (2016) adopted a primitive quantizer design based on uniform quantizer for gradients (without scaling and other optimization) and large performance drops are witnessed when training with low-bit gradients. SBM Banner et al. (2018) adopted fixed-point 8-bit gra- dient quantization, but only focused on improving the quantization schemes in the forward pass. WAGEUBN Yang al. (2020) quantized gradients to 8-bit integers, but also showed huge per- formance gap against its pull-precision counterpart. NITI Wang et al. (2020) integrated gradient calculations with parameter update operations reduce the gradient quantization noise with well- designed quantizers. However, can only support shallow CNN architectures and did not explore any deeper networks with BatchNorm. In Zhu al. (2020), the authors considered the sharp and wide distribution of gradients, and proposed to clip the gradients according to the deviation of the gradient distribution before quantization, achieving on-par results with full-precision training. To compensate the quantization loss gradient, AFP Zhang et al. (020)) and CPT Fu et al. (202T) used higher precision data aid low-precision training. DAINTS Zhao al. (202)) adopted be- spoke 8-bit channel-wise gradient quantization to suppress the negative effect of quantization noise during training. Gradient quantization with less than INT8 representations remains largely unex- plored. FP4 (Snn et al.) (2020) managed to train modern CNN architectures using 4-bit gradients without significant accuracy loss, but the gradients were represented as floating-pont numbers.
Batch Normalization in QAT. Most QAT approaches either left BatchNorm in between parame- terized layer (Conv/FC) and activation layer (ReLU) without quantization Zhou et al. (2016); Choi et al. (2018) with quantization Yang al. (2020), absorbed BatchNorm into Conv before weight quantization in the forward pass Jacob et al. (2018), or directly trained BatchNorm free shallow network architecture achieve full 8-bit integer-only arithmetic Wang et al. 2020). To
2

---

Under review as a conference paper at ICLR 2023
40-
Var. of gradients (wirt. w.)
10a
2-bi bits
4-bits
Accuray
104
8-bits
20-
Ey
10-1
10-
10-3
10
Epoch
15
Figure 1: Validation accuracy curve of AlexNet on ImageNet with W2A2dz3dW2 w/ or w/o the pro- posed L.. (dr denotes error signal, dW is gradients of weights, same below).
layer#
Figure 2: Illustration of gradient explosion problem (measured by the variance of gradients across lay- ers) with {2, 4, 8)-bit uniformly quantized gradients. Statistics are computed from ResNet-20 on CIFAR-10 with 8-bit W/A. Y-axis in log scale.
best knowledge, this paper is the first systematically study the effect of BatchNorm on the training stability of quantized networks with low-bit gradients.
Norm-free Networks. There another interesting line of works that gets rid of normalization layers from CNN architectures for good Zhhnn et al. (2018b) while still manages to train the full-precisio networks stably. However, there is still lack of attempts to adapt norm-free networks to QAT or even FQT settings the first place, where potential problems need to be addressed when low-bit quantizers are introduced. Therefore, we decide currently it is not mature enough to discuss this track in this paper which targets at FQT.
Prior efforts reducing variances in backpropagation. Rectifying gradient variances during back- propagation has been sporadically discussed for full-precision training, e.g. in Kaiming Initializa- tion/H He et al. (015)) where it leverages weight distributions in Conv and FC layers and one can opt for backward variance rectification if the gradients observed be chaotic. However, optimal for- ward and back ward rectification still cannot be satisfied simultaneously, especially when backward signals contains significant amount of quantization noises. our attempts, using "fan-out" mode in kaiming initialization alone still cannot avoid training crashes in worse cases (e.g. MobileNet- V2 under W4A4G4). To our knowledge, there is no work studying the impact of normalization layers in CNNs for gradient rectification under the FQT setting.
3 PRELIMINARIES
Key Notations. We denote the variance of a probabilistic variable as D(.). We denote gradient w.r.t weights as Iw and eгrоr signal as 9a- We use the plural term "gradients" generally refer to all backward pass variables including Iw and gæ- In such contexts, subscripts of g are omitted. Similar the additive quantization noise for weight and activation quantization in MMelerr et al. (2019). the quantized gradients ğ at each layer can be decomposed into three parts:
ğ - + c(g) + 8g(g).
where g and e(g) denote the original gradients and the gradient quantization noise at current layer respectively, and (g(g) represents the accumulated gradient quantization noise propagated from all its succeeding layers during backward. E.g., for the (1 1)th layer, 5, (ga.,,) represents the quantization noise accumulated from layer I to the last layer (see Fig.B)
In this manuscript, we format the bit-widths of weights (W), activations (A), backward errors (dr) and gradients of weights (dW) used in experiments as W/A/dz/dW if emor da and gradients dW are assigned different bit-widths, or simplified as W/AG da and dW have the same bit-width.
Batch Normalization. Batch Normalization Ioffe & Szegedy (2015) is widely adopted technique to stabilize the training of deep full-precision networks. The forward pass in a BatchNorm layer consists of operations calculating the mean and variance of each channel over a mini-batch with N input samples { [x,, IN Each channel of the input x,, is first normalized to i. = (x. pe)/ The normalized input & is finally linearly scaled and shifted y yT i + 3. The relationship between backward error on BatchNorm's imput x, and that on o, is: (we omit channel notations here for simplicity)
3

---

Under review as a conference paper at ICLR 2023
Forward pass
Backward pass
1-th layer (Conv/FC)
L-th layer (Conv/FC)
1-th layer (Conw/FC)
1-th layer (Conv/FC)
Qw
Qw
W
Gir
QG
W
W
Fay
S,Cau c(De))
4--2
MAC
RaLL
Qa
MAG
ReLU
Qa
a2
MAC
Bu
BN- RELL
gaL Ag
Figure 3: Illustration of weight/activation quantization in forward pass and gradient quantization in backward pass. MAC denotes the acctulllleee operations. Q(,) denotes the quantizer for weights (W), activations (A), or Gradients (G). In backward pass, the gradient quantization noise (9 accumulated from the 1th layer the last layer is propagated the (1 1)th layer and added to the (1 1)th layer's gradient quantization noise e(ga;=1)
induced by itself.
N
1 fa. No
Nga..
9s..
g Tn
PROBLEM IDENTIFICATION ACCUMULATION OF GRADIENT QUANTIZATION NOISE EXPLODES GRADIENTS
Gradients play crucial role in backpropagation based optimization and make a huge impact on training stability and convergence speed. Intuitively, as we inject quamtizers in between backward pass, the error signal become
and more noisy each time it passes through quantizer. As the bit-width decreases, the quantization noise injected in error signal at each single layer increases ex- ponentially. From a perspective of variance, since quantization introduces additive noise c(g) the original signal g. the variance of quantization noise (() is also added to the error signal (gg), which will be reflected tion. As shown in Fig. 2 under various bit-width of gradients, we observe the variance of quantized error signals expanded during backpropagation. As the bit-width decreases, the variance also flated much more drastically on shallow layers than late layers, implying that quantization noise is culprit of the variance accumulation and explosion. As a result, the quantization impact on error signal and the weight update are severely affected, especially in those early layers. Eventually in worse cases when the accumulated quantization noise overwhelming, the training goes off nature course and crashes. (e.g. under extremely low bit- bit-width Fig.
the propagated errors and will increase over the course of backpropaga-
Fig. 3 provides a glance at the accumulation mechanism of the gradient quantization noise in the backward pass. During backpropagation, the quantization of the error signal on the 1th layer intro- duces the quantization noise denoted as e(ga,) ). c(ga,) is propagated to its predecessor the (1 1)th layer, together with the gradient quantization noise 6, accumulated from the (1 + 1)th layer to the last layer L. Similarly, e(ga,,)) and (g are propagated to the (1 (2-2)th layer, and so on.
As shown in Fig. both forward and backward pass have similar accumulation phenomenon, but why backward pass suffers more from the quantization during training? This is because quantization noise introduced in the forward pass reflected in the computation graph w.r.t. the objective loss, therefore their impact can be partially offset by quantization aware training, while quantization noise introduced during backward does not contribute to the objective loss.
In view of some cases under the setting of distributed gradient compression Alistarh et al. 2011 which only quantizes full- full-precision gradients after backpropagation is done, can stably train CNNs with as low as 4-bit gradients, we attribute the exploding gradient problem in FQT setting mainly to the accumulation of gradient quantization noise introduced by low-bit gradient quantizers.
BATCHNORM AMPLIFIES THE ACCUMULATED NOISE
In this section, develop theoretical framework to understand the role of BatchNorm in gradient quantization, explaining why BatchNorm may worsen the gradient explosion problem in FQT. One refer to Appendix A.2 for the proofs of the theorems.
4

---

Under review as a conference paper at ICLR 2023
Quantifying the impact of BatchNorm Noise Accumulation. Through theoretical studies, we find that BatchNorm may amplify the accumulation of gradient quantization noise. This finding might be counter- inttitrvitiiiivv BatchNorm has been expected to regularize the "variance" and prevent the gradient explosion problem, by scaling the activations forward pass. However, not only vanilla BatchNorm mainly focuses rectifying variances in forward pass, but also does not count in the situation where the gradient signals are noisy. When еитоr signal passes through such scaling layer inside BatchNorm, the error signal is also scaled by the reciprocal of the corresponding scaling factor (σ) when calculating its derivative. When training process is in full-precision, such scaling on error signal is manageable. But when the error signal contains accumulated noise from previous layers, the noise is scaled at the same time, causing unpredictable behavior to the backpropagation. Hence our theoretical focus fundamentally different from previous "variance rectification and reduction" studies.
The following theorem quantifies how specifically BatchNorm affects the accumulation effect.
Assumption 1. (ga.) and 2; are id.dd. and are both zero-mean Zhao al. (2021)
Theorem 1, Given Assumption 7 for a BatchNorm layer in a quantized network, the relationship between the gradient quantization noise w.E.t the BatchNorm' 's input æ, and that of the normalized imput is depends on the a of BatchNorm with batch size N, in the form of
((m))(mm))
D(6 N2 (N2 + 2N).
(3)
Remark 1.1. In Eq. is the ratio of statistical variances between the scaled error signal ōg(9m.) and one before scaling 5y(ga.) (see Fig. 4.
we define the amplification factor of the accumulated noise 1), which
Corollary 1.1. To prevent BatchNorm from introducing more gradient quantization noise (i.e. the statistical variance ((gg(ge)) ))) when propagating the error signal to preceding layers, a desirable
n?" should not be greater than 1. Thus, desirable a of the BatchNorm should be a 2 o²
VI+R 2
N
In short, Theorem studies the amplification factor 1) over the accumulated gradient quantization noise when gradient passes through BatchNorm scaling, and Corollary [.rrvviddes a close form solution of the ideal condition of the BatchNorm variance a to satisify minimum noise accumulation.
Scaling gH
9 gB
gyL
9x
gx
+ MAC'
As the accumulated quantization noise during backward can- not be automatically amortized by the training objective in FQT, ones are left with only options to either (1) minimize the primary source of noise by improving gradient quantizer design, or (2) minimize the accumulation of such noise. With the first choice being obvious, in this paper, we instead aim to raise people's 's awareness about the second choice and the importance of properly scaling the noisy error signal to alleviate noise amplification problem in backward pass and the eventual training sta- bility.
go2
Figure 4: Illustration of gradient propagation inside BatchNorm.
6 OUR METHOD: RECTIFYING BATCHNORM FOR GRADIENT QUANTIZATION
Inspired by our theory in Sec. we develop solution to suppress the noise amplification effect in a principled way, which in turn reduces the gradient explosion for improved training of quantized networks with low bit gradients. Based on Theorem T and Corollary I.] we expect the to be larger than the ideal value a', so that the noise amplification factor n output and the input of BatchNorm is minimized. Therefore, propose method stabilize the training with low bit gradients, by rectifying the variance of BatchNorm computed in forward pass: min. f(w)
((d,(g=,))
between the
((6,(92,))
L
-
MSE min ai (1) ,
s.t.
Lo =
= O,
L
1=1
5

---

Under review as a conference paper at ICLR 2023
where f - R is the regular objective loss (e.g., cross-entropy loss for classification) and S denotes neural network weights. OL = {oi)== 1 is the set of a from all BatchNorm layers in the network with depth L. We use Mean Square Error (MSE) between _ and to enforce that or at each BatchNorm layer approaches to a². The Lagrangian dual approximation form of Eq. is:
f(()) f(w) + XLo.
(5)
where À is an adjustable parameter to balance ((w) and the proposed rectification loss L
Gradient Computation and Computation Overhead. The overhead introduced by our proposed rectification term only has linear time complexity. During backpropagation, the error signal of rectification term Lg when propagated to BatchNorm's 's input on channel i at layer / is:
OC. da
a N C2- - 2 ,) - M)). of < o
(6)
0.
otherwise
where o², N and channel-wise BatchNorm parameters μi, of are all constant scalars during back- propagation, showing the added rectification is very cheap and negligible when training on devices capable of vectorized computation optimization including GPUs. More detailed analysis can be found in Appendix A.4
7 EXPERIMENTS
To evaluate our method, conduct extensive experiments on various neural network architectures and popular datasets for image classification with low-bit gradients.
Experimental Setup. To highlight the impact of BatchNorm, we only use two vanilla quantizers for gradients in all experiments without any optimizations: uniform quantizer and logarithmic quantizer (see Appendix I..I. Our method introduces only one hyper-paramen ally initialized then can be ramped down by the cosine rule during training or stay the same. For each parametrized layer in backpropagation, the gradient 9w and error signal 9ar are quantized separately, and they can be quantized to different bit-widths. To evaluate our rectification method, we ensure all the used CNN architectures have BatchNorm layers, including ShallowNet and AlexNet-BN, the architecture details of which are listed in Appendix A.3 More details are listed in Appendix A.4
À in Eq. (5). which is manu-
7.1
MAIN RESULTS
INT8 comparisons. We compare our method (training with La) to state-of-the-art gradient quan- tization approaches reporting results with 8-bit gradients: UI8 Zhu et al. (2020), FP8 Wang et al.) 2018), AFP et al. (2020), SBM et al. (2018. DAINTS et al. (2021). Using 8-bit Logarithmic quantizer for gradients and the proposed rectifier C., Tab. shows that our method out- performs the oftte-of-thh-are in almost all cases, despite that simple vanilla quantizers on the gradient, while the quantizer designs of the counterparts are heavily engineered, e.g. DAINT8 Zhao al. (2021) adopts vector quantization to process error signal in channel- wiie manner. We found that MobileNet- V2 on ImageNet is harder to train with 8-bit gradients with vanilla quantizer designs even training with our L., ending up with around 1% accuracy drop than SOTA Zhao et al. (202 As an ablation, for ResNet-20 on CIFARIO. 10, our method boosts the accuracy by 1% from baseline training (w/o C.), also for R8 on ImageNet, C. achieves almost 2% improvement.
Comparisons on 4-bits gradients. Since there are very few works on quantized neural networks with less than 8-bit gradients, we compare our method named FP4 Sun et al. (2020) which actually adopt 4 bit floating point representations, and the base- line which without the proposed L.. As shown in Tab. 2 we observe that in most cases our method with INT4 gradient quantization reports higher accuracy than FP4 et al. (2020), despite FP4 [un et al. (2000) adopts floating-point quantization with customized radix and scaling selec- tions. We also observe that our method (w/ L.) performs better than the baseline, further verifies the effectiveness of the proposed L. On the other hand, we observe MobileNet- V2 with INT4 gradi- ents is still unstable and hard to converge why most FQT works did not report results for INT4 gradients on MobileNet- V2). Thus, we'd d like to leave the FQT of MobileNet-V V2 for future work.
4-bit floating point quantization method
with our rectification method deployed (it explained

---

Under review as a conference paper at ICLR 2023
Dataset
Arch
Method
Top-I
Dataset
Arch
Method
Top-1
U18
92.0
FP4
91.5
FP8
92.2
VGG-16
w/o Lo
90.7
ResNet-20
DAINT8
92.8
Ours
92.5
w/o L.r
91.9
CIFAR-10
FP4
92.7
CIFAR-10
Ours
92.9
ResNet-18
w/o L.
93.7
U18
93.4
Ours
94.0
Mobile-V V2
DAINT8
94.4
FP4
56.3
w/o L.
94.5
AlexNet-BN
w/o Lo
57.2
Ours
94.6
Ours
57.3
U18
69.7
FP4
68.3
FP8
67.3
ImageNet
ResNet-18
w/o Lo
69.61
ResNet-18
DAINT8
70.2
Ours
69.71
SBM
69.6
FP4
74.01
w/o C.
69.1
ResNet-50
w/o L.
73.67
Ours
70.9
Ours
73.4
U18
76.3
ImageNet
SBM
76.3
Table 2: Comparison with FP4 gradient quanti- ation with bit-width W4A4G4. Log-INTI quan- tizer is used for gradients.
DAINT8
76.6
ResNet-50
AFP
76.2
w/o Lg
76.4
Ours
76.8
OIS
71.2
0.1
0.25
0.5
DAINT8
71.9
Mobile-Vo V
AFP
70.5
Top- (%)
68.6
68.4
70.1
67.0
w/o C.r
70.74
Ours
70.9
Table 3: Effect of À for training quan-
tized ResNet-18 on CIFAR-10 with bit-width
Table Comparing state-of ofte--of-the-ar methods with bit-width WSA8G8. (Mobile- V2-Moobleeeeee--VV2
W2A2G2.
7.2 EXTREMELY LOW BIT- WIDTHS
Despite scarce exploitation in the wild and the challenges, further attempt to quantize gradients to even lower bit-widths, with different backbone network architectures, bit-widths combinations, and gradient quantizers, to further evaluate the theoretical capability of the proposed rectifier Lg, When gradients are quantized to very low bit-widths, we expect the training becoming extremely unstable the quantization noise and eventually the accumulation effect becoming much severer. Consider- ing the increased training instability, we perform three independent trials for each experiments and report the comprehensive scores as stt).
Simple network architectures. As shown in Tab. we first study the training of a two-layer quantized neural network ShallowNet MNIST. We set the bit-width for weights, activations, and gradients as 2-bit (W2A2G2) and use Logarithmic quantizer for gradients. We observed that bascline training without La crashed twice out of repetitive runs, while the training with the proposed C. stable throughout. In other words, our method outperforms the baseline by a large margin, improving the average accuracy by +54.4% (from 38.8% 93.2%)
On larger dataset ImageNet, we also have the similar observation for AlexNet-Bo -nn where baseline method fails train completely while our method with E, can train stably throughout three trials.
More complex network architectures. We further test the effectiveness of our method in FQT training of networks with more complex structures. We push the boundary of the lowest bit-widths settings we achieve, as lowest as e.g. W2A2G2 on ResNet-18 in Tab. 5 We observed that the FQT on VGG-16 is slightly more sensitive to the quantization than ResNet-18. 18, thus we have to set higher bit-widths for VGG-16. We also notice that VGG-16, emror (dx) requires more bit width than gradients w.r.t. weights (dW). In all, our method performs consistently better than the baseline (w/o L.)) all settings, in particular, the performance gain VGG-16 is significant.
As an additional remark, we notice that compared to other models, ResNets are more robust against more quantization noise throughout experiments suggested in Tab. and Tab. 5 We conjec-

---

Under review as a conference paper at ICLR 2023
90
3.5
17.5
STD
AVG wis C
0.5
STD m/
AVG W/
SID mi
70
12.5
2.5
0.6
2.0
0.4
50
7.5
2.5
1.0
0.5
25
50
75
100
0.2
0..0
25
50
75
160
40
50
AVG WT
AVS We
100
5.0
0.0
50
MyA
100
1.0
200
400
0.0
0.1
200
400
(a)
Training Loss
for
Shal- (b) Training
Loss
for VGG- (c) Training Loss for ResNet-
MNIST.
16@ CIFAR-10.
18 @
SI@CIFAR-10
AVS wi
wio
30
wis & STD Er
50
NVG m/
AVG w/c C,
STD MiD 4 STD w
10
AVG wi
AVIS wit
STD Wi
60
30
20
40
5
10
15
5
20
15
10
20
10
20
200
400
0
200
400
Test Acc. lowNet@MNIST.
(d)
for
Shal- (c)
Test
Acc.
for
VGG- (1)
Test
Acc.
for ResNet-
16@CIFAR-10.
18@CIFAR-ID
Figure 5: Illustration stabilized training of quantized networks with lower than 4-bit gradients w/ foo compared the baseline w/o L.. Our method (w/ L.) shows higher averaged accuracy and lower variance across repeated runs. X-axis in all sub-figures represents epochs.
Bit-widths (W/A/dadW)
Runs
Lr
Avg Top-1 1 (%)
Diff. (%)
#1
#2
#3
ShallowNet on MNIST
w/o
10.0
94.9
11.4
38.8 + 48.7
2/2/2/2
w/
92.0
94.4
93.3
93.2 1 1.1
+54.4
AlexNet-BN on ImageNet
w/o
2/2/3/2
w/
47.6
49.46
46.53
47.86 + 1.48
+47.86
Table Ablations C. under extremely low bit-widths. Scores in red denotes failed train.
ture that it is due to the full-precision shortcuts within, making them naturally more robust against the accumulation effect during backward pass. Theoretical investigation towards this phenomenon would be an interesting future research topic.
7.3 OTHER DISCUSSIONS
Can C. stabilize training? Fig, fillustrates the improved training of quantized networks with less than 4-bit gradients, thanks to the rectification loss C.. Fig. 5a and Fig. [5d]are are ShallowNet trained on MNIST for bit-width configuration W2A2G2 with Logarithmic gradient quantizer. Fig. 5b and Fig. 5e are VGG-16 trained on CIFAR-10 for W4A4124dW2 with Logarithmic gradient quantizer. Fig. 5c and Fig. 5i are ResNet-18 trained CIFAR- 10 for W4A4G4 with uniform gradient quan- tizer. We plot out the average loss/accuracy (AVG) and standard deviation of loss/accuracy (STD) of 3 trails separately. From Fig. Sa Fig. 5b and Fig. |c we observe that the rectification loss L. dominates the training loss for the first few epochs, forcing the optimization adapted to the low-bit gradients. Afterwards, the training process becomes much more stable with training loss decreased and test accuracy increased gradually (see Fig. 5d Fig. 5e and Fig. 5t On the contrary. training without L. not able to suppress the negative effect gradient quantization noise, resulting in training instability even crash.
Can Cg suppress the noise amplification effect? To verify such stabilizing effect indeed comes from the proposed rectification, further studies its impacts on gradient distribution layer by layer. Fig. Gillustrates the distribution of variances of gradients w.r.t. layer output activations when training quantized VGG-16 CIFAR- 10 with bit-width W4A4dz4dW2. By injecting the rectification loss E, defined on BatchNorm in training objective function, one can see the noise amplification effect is largely suppressed, and thus the gradients are less exploded (measured by the variances of gradients).
8

---

Under review as a conference paper at ICLR 2023
bit-widths (W/A/dz/dW)
Gradient
Avg Top-1 (%)
C.
Quantizer
VGG-16
w/o
50.0 8.4
4/4/4/2
Uniform
w/
66.7
= 8.7
w/o
66.8
4.9
4/4/4/2
Logarithm
w/
71.0 + 5.4
ResNet-18
w/o
70.0 1.5
2/2/2/2
Uniform
w/
71.0 + 0.8
w/o
59.9 = 2.1
2/2/2/2
Logarithm
w/
61.6 + 2.0
Table 5: Ablations of C. and quantizers under extremely low bit-widths.
Normalized Variance gradients w.r.t. activations
00
W/o BN
w/ BN
10
10-2
Loss
7
10
W/ Cy.
w/S En Har-2
Iter-
10-2
C., Neer-.44
10-10
W/ Ey. 3.2k
C,,
2
4
6
10
0
200
400
layer
Iter
Figure Distribution of variances of gradients w.r.t. activations when train- ing VGG-16 on CIFAR-IO with bit-width W4A4dz4dW2. Logarithmic quantizer is used for gradients.
Figure 7: Train loss curve of AlexNet on Im- ageNet with W4A4dz4dW2 w/ or w/o Batch- Norm. (de denotes error signal, dW is gradients of weights, same below).
As result, it help prevent crashing in training with low bit gradients (see NaN at the 2.4k iteration for training without fa).
Effect of X. As shown in Tab. study the effect of the hyper-parameee À when training quan- tized ResNet- 18 on CIFAR-10 10 with bit-width W2A2G2. The optimal values X for different ex- periments could be different and we heuristically tune them separately.
Can BatchNorm layer be discarded when training with low-bit gradients? Since BatchNorm amplifies the accumulated quantization noise during backpropagation, one may argue that straight- forward way to prevent the noise amplification effect is removing BatchNorm layer from the net- work architecture. To verify the point, we conducted experiments on training AlexNet with or without BatchNorm under W4A4d #4444244W2 using logarithmic quantizer. As shown in Fig. 7 train- ing AlexNet without BatchNorm quickly collapsed in the early stage in training. This implies that at the moment BatchNorm is still an essential building block to deep CNNs training for its rectify- ing benefits mainly forward pass, while our rectification method complementarily stabilizes the backward pass, at least in the case when quantization is applied especially in FQT.
CONCLUSIONS
In this paper, we study an under-explored factor causing the gradient explosion problem when train- ing deep CNNs with low-bit gradients, from theoretical perspective. Our theory sheds light on the negative effect of BatchNorm in amplifying the accumulated gradient quantization noise during backpropagation, which leads unstable training effective method to stabilize FQT with low-bit gradients, which consistently brings performance gain wide range of CNNs and datasets compared state-of-the-art FQT algorithms.
crash. The theory inspires simple yet

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovie. Qsgd: Communication-elicienn sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems, pp. 1709-1720, 2017.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. arXiv preprint vxXiv:1805. / 1046, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.344. 2013.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 696-697, 2020.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprini 180.006080 2018.
Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang. Accurate and efficient 2-bit quantized neural networks. In MLSys, 2019.
Steven K Esser, Jeffrey I. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Learned step size quantization. In ICLR, 2020.
Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post-training piccewise linear quantization for deep neural networks. In European Conference on Computer Vision, pp. 69-86. Springer, 2020.
Yonggan Fu, Han Guo, Meng Li, Xin Yang. Yining Ding, Vikas Chandra, and Yingyan Lin. Cpt: Efficient deep neural network training via eyclic precision. arXiv preprint 2xii::101.0996 2021.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging frel-precision and low-bit neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4852 1861, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European conference on computer vision, pp. 544-560. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference (ISSCC), 2014.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient pnngerr-rithmmetc-oon inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Qing Jin, Linjie Yang, and Zhenyu Liao. Adabits: Neural network quantization with adaptive bit- widths. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
10

---

Under review as a conference paper at ICLR 2023
Iya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Qian Lou, Feng Guo, Minje Kim, Lantao Liu, and Lei Jiang. Autoq: Automated kernel-wise neural network quantizations. In ICLR, 2020.
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Recovering neural network quantization error through weight factorization. In International Con- ference Machine Learning, pp. 4486-4495. PMLR, 2019.
Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv
2016.
X Sun, N Wang, C Chen, J Ni, A Agrawal, X Cui, S Venka, K HI, V Srini, and K Gopa. Ultra-low precision 4-bit training of deep neural networks. In NeurlPS, 2020.
Maolin Wang, Seyedramin Rasoulinezhad, Philip HW Leong, and Hayden KH So. Niti: Training integer neural networks using integer-only arithmetic. arXiv preprint arXiv:2009.13100 2020.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 31, pp. 7675-7684. Curran Associates, Inc., 2018.
Linjie Yang and Qing Jin. Fracbits: Mixed precision quantization via fractional bit-widths. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10612-10620. 2021.
Yukuan Yang, Lei Deng, Shuang Wu, Tianyi Yan, Yuan Xie, and Guoqi Li. Training high- performance and large-scale deep neural networks with full 8-bit integers. Neural Networks, 125:70-82, 2020.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. Proceedings of the European conference computer vision (ECCV), pp. 365-382, 2018a.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2018b.
Xishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu, Di Huang, Shiyi Zhou, Jiaming Guo, Qi Guo, Zi- dong Du, Tian Zhi, et al. Fixed-point back- proppaaopagttion training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2330-2330000 2338, 2020.
Kang Zhao, Sida Huang, Pan Pan, Yinghan Li, Yingya Zhang, Zhenyu Gu, and Yinghui Xu. Dis- tribution adaptive int8 quantization for training cnns. In Proceedings of the Thirty- Siirt-Fiftth AAAI Conference on Artificial Intelligence, 2021.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06.00 06160, 2016.
Feng Zhu, Ruihao Gong. Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969-1979, 2020.
A APPENDIX
A.1 GRADIENT QUANTIZER CHOICES
For gradient quantization, in each layer, the full precisiec error dæ and gradient dW are quantized, then de-quantized, and the quantized error propagated back the next layer during backward pass
11

---

Under review as a conference paper at ICLR 2023
Zhou al. 2016); Zhu et al. (2020); Yang al. (2020); Wang al. (2018). Uniform quantizer is
the default and common choice gradient quantization, where given the gradients E {dr, dW', the asymmetric uniform quantizer with bit-width B and quantization levels ranging from [a, b] can be formulated as:
94 Quant, (g', ,a,o,aa,,, b, B) round (cliplo'.
b 2B-1 -1)
(7)
where g' g + 5g (g) (see Sec. 3 for details) and clip(æ;, o, b) =
a), b). The quantized
gradients are de-quantized as ğ 94 28-1-T Similarly, the symmetric uniform quantizer can be defined as Quant, Quant, (g. c, e, B) where the e is the clipping value.
b-a
We also explored logarithmic quantizer Miyashita et al. (2016) in this paper for gradient quantization given by:
94 = Quantiag ing (g',c, B) = sign(g') Quant, (loga Joga (c)-2 "Joga (o).B), 0,
g + 0:
(8)
A.2 QUANTIFYING THE AMPLIFICATION EFFECT IN GQNA
Theorem 1. For a BatchNorm layer within a quantized network, the relationship between gradient quantization error of BatchNorm's 's input and that of the normalized input 2; only depends on the batch size and the a of BatchNorm, in the form of
((0,(gm:)) ((6)(
N2 (N² 2N).
(9)
Proof.
First we extend the 2q to its quantized counterpart:
N
N
gn. =
Nga: Σ
ga.
3;
(10)
No
1=0
1=0
and
N
1
e(gx.) =
Ne(ga.) Σ e(go.)
$z
(gg,, )z:
(11)
No
1=0
1=0
Expand Eq. 10p using Eq. I
N
1
gx. de (9x,) e(gxe))
Ngaa, + -
+e(gz.))
Σ
(g.. + dggggt)) + c(gg.)
No
t=0
+
gg. +gggge,) e(ge,))
(12)
Eliminating terms in Eq. 12 using Eq. and Eq.
we have:
N
N Σ
(gm.))
(ga.) ) Σ
(g..)
(g
)a;
(13)
No
7=0
2=0
Calculate the variance D(-) of LHS and RHS of the above we have: (assume o, (g+,) and a; are i.d.d. and are both zero-mean)
((6,(ga.)) N2
D
N8.(ga.) - -
(g
-;
Σ
b (ge, )a:
(14)
1=0
1=0
N
N2 N²D(b,(ga.) +
8g(ge.)
D
(g
(gt; )a:
(15)
2
N
N
N²D(6,(ga.)) + Σ
(66 +
5, (ga.)
(16)
2
i=0
1=00
12

---

Under review as a conference paper at ICLR 2023
N
(6 + (ND(, ((;;) ()
Σ
(17)
==0
(
(18)
N2
[N
(2,(6,
(19)
N2
1
2
+
(()(()))))
(20)
02
No
Therefore,
(mm)( ))
2
1
N (N² + +2N).
(21)
(9a))
02
No²
A.3 ARCHITECTURE DETAILS
ShallowNet.
class ShallowNet (nn. Module):
def ____ii__ns_______________ (self):
init_()
sclf.convl Convdd 20, 5, 1)
self.ba1 nn. an.BatcchNorm2d22000
self.relul nn. Relu ainplaace False)
self.pool1 nn_MaxPoc12d (2, 2)
self.conv2 nn.Conv2d (20, 50, 5, 1)
seli.bn2 (5..atehoorm22(550)
self.relu2 Falee
self.poo12 Mn.Maxxool2dd2, 2)
self.avgpool = AvV stride-1)
self.fc = nn. Lin.iinearr0 10)
def (sel,, x)
x self. (self. relul (self. (self. f.novilooll (x))) self.pool2 (selff.eelrrrr (self bn2 (self f.cov22 (x)))
sclf.avgpool
X x.view(x.size -1)
x (el
return
AlexNet-BN.
class AlexNetEN (nn.ouleee)
def
__iiiinseel. num classes=1000)
super (ALexNetBN, self) aalf)__nnil init ()
self.features nn. Sequential nn. Co.Cond 96, kernel_sizo-12, stride-4), (..eeDDinplace-True)
Co, 256, kernel_sizes: padding-2, groups-2, + bias-False), nn. nn. (256, eps-le-1, momentum=0.99 .)), nn (kermee ReU (rue
stride=2, mode=Troee True),
(25, 384, kernel_size-3, padding-1, bias-False), nn BatehNohoo (384, eps-le-4, momentum=0.99 9),, stride-2, padding-1),
nn, (oxeeooll nn. ReUU inu.ee
nn. Conv2d (384, 384, kernel_sizeri padding=1, groups=2, FalseF
13

---

Under review as a conference paper at ICLR 2023
mm.Batchhoorm22 (384, eps=le-4, momentum=0... (n.ReeLudinpllcco-rree)))
nn Con 256, kernel_size=3, padding=1, groups-2, bias-False),
nn. Batcchohoo (256, eps-le-4, momentum-0.99
nn MaxPool212 (eernel_siieeeeee stride=2), na. na.ReLU (inlaaccee))))r
celf.classifier mm.Sequential
iineir (256 6, 4096, bias-False), nn mn.attchNormld4400 (4096, eps-le-4, momentum=0.99 9), nn. (iL Ln. 4096, bias-False), nn (498 eps-1e--- nn Rn.ReLU (i..accooip)lla nn. nn. (4096, num_classes),
eps=le-4, momentun=0.99 .99,,
def (seaf, x)
x self. features (x)
(xv (0), 256 6 6)
self classsiessiiir(
return
A.4 TRAINING SETTINGS AND
Choi et Following Sun al. 2020), adopt Parameterized Clipping Activation (hoi for activation quantization and Statistics Aware Weight Binning (SAWB) Choi et al. (2019) for weight quantization. We choose the SGD optimizer and train all quantized network models from scratch. The learning rate is adjusted with a cosine scheduler (Loshchilov & Hutter (2017)). For MNIST, we set the learning rate 0.1, weight decay 0.0001, and train for 20 epochs. For CIFAR-10, 10, we set the learning rate as 0.1 for all architectures. weight decay 0.0001 for ResNet-18 and VGG-16, and weight decay 0. 0.0004 for MobileNet-V2. For ImageNet, we the learning rate as 0.0512 and weight decay as 0.0001 for ResNet-18. 18, learning rate 0.01 and weight decay 0.0005 for AlexNet, and learning rate and weight decay 0.00004 for MobileNet-V2 V2. All models on CIFAR-10 and ImageNet are trained for 120 epochs, except for MobileNet-V2 that is trained for 150 epochs. For ImageNet, training images are randomly cropped to 224x224 and then randomly flipped horizontally. For experiments runs by varying the random seed and report the average accuracy with standard deviation. Same as other INT8 works Zhu et al. (2020), left the first and the last weighted layers as well activations in full-precision for all INT8 experiments. For INT4 experiments, we followed the setting in Choi al. 2019) that only left shortcut layers in ResNets in full-precision and quantized all other layers.
al. (2018)
MNIST and CIFAR-10,
repeat the training of each model for
We specify the choices of and its ramping down strategy as below.
Comparing SOTA with 8-bit gradients
Dataset
Arch
X
Ramping down?
ResNet-20
0.5
CIFAR-10
MobileNet-V V2
0.5
ResNet- 18
0.5
ImageNet
ResNet-50
0.5
N
MobileNer-V2
0.5
Table 6: Comparing state-of-the-art with 8-bit gradients.
Comparing SOTA with 4-bit gradients
Experiments with lower-than-4-bit gradients
14

---

Under review as a conference paper at ICLR 2023
Dataset
Arch
Ramping down??
VGG-16
0.5
CIFAR-10
ResNet-18
0.5
Y
AlexNet
0.5
N
ImageNet
ResNet-18 18
0.5
Y
Table 7: Comparing our 4-bit fixed-point gradient quantization to the 4-bit floating-poit gradient quantiation FP4 Sun al. (2020).
bit-widths (W/A/dz/dW)
Gradient
À
Ramp down?
Quantizer
ShallowNet on MNIST
2/2/2/2
Uniform
0.1
N
VGG-16 on CIFAR-10 Uniform
4/4/4/2
0.5
Y
4/4/4/2
Logarithm
0.5
ResNet-18
CIFAR-10
2/2/2/2
Uniform
0.5
Y
2/2/2/2
Logarithm
0.1
Y
Table Less than 4-bit gradients.
A.5 DETAILED ANALYSIS COMPUTATION OVERHEAD
To more directly derive in the linear complexity conclusion, one can simplify Eq. 6 into
dC2 Sa:
C1 @ a C2 where @ denotes the element-wise multiplication, C1, C2 are both constant tensors with the same shape as a₁- The partial derivative 8C, is further aggregated with the error signal da;
from objective function 8f(w] to compute the gradient w.r.t. weights (f'(w)) Here C. introduces no extra computation cost.
Tab. gives an empirical verification of the actual training overhead of our method compared to the baseline.
Dataset
Arch
w/o L.
Ours
Diff.
ResNet-18
43.4
44.6
0.67%
ImageNet
ResNet-50
143.32
152.58
6.46%
MobileNer-V2
162.4
171.2
5.41%
Table 9: Comparison of training time (hours).
A.6 DETAILED ABLATIONS HYPER-PARAMETER (ON HIGHER BIT-WIDTIIS)
To comprehensively evaluate the influence of the choices of byper-parameter X, we further tested the model performance on higher bit-width.
Tab. 10 shows the results under different À values of ResNet-18 on CIFAR-10. The quantizer choice for gradients log quantizer.
0.1
0.25
0.5
Top-1 1 (%)
93.7
93.88
93.57
94.0
93.71
Table 10: Ablation study of À under bit-widths W4A4G4,
15


Reviews:
[{"review_id": "rI3FISVOt-i", "reviewer": "Reviewer_FcHZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider fully quantized training (FQT), a subfield of quantization-aware training (QAT), where even the gradients are quantized during backprop. A common issue in FQT is that training can be very unstable. \n\nThe authors study the role of batch normalization (BN) in causing training instability, and they identify BN as a significant culprit. Hence, the authors propose adding a regularization term in the loss function of models using BN, and they empirically demonstrate their approach's effectiveness.", "summary_of_the_review": "The authors propose a solution to a relevant problem and demonstrate its effectiveness empirically. However, the paper suffers heavily in terms of clarity of writing and mathematical formalism. Most importantly, it contains mathematical statements that are, at best uninterpretable and flat-out wrong at worst. Hence, the paper is unpublishable in its current form.", "strength_and_weaknesses": "## Strengths\nThe authors propose a solution to a relevant problem, stabilizing FQT, which should be of interest to many people in the ML community. The proposed solution is simple to implement and is intuitively clear: the regularization term they propose doesn't allow the normalizing standard deviation term in BN to become \"too small\" for an appropriate notion of \"small\", which prevents instability during training. Finally, they compare their proposed approach on a reasonable selection of benchmarks and verify their method's effectiveness from different angles.\n\n## Weaknesses\n - Simply by virtue of the extremely sloppy mathematical setup, Thm 1 cannot possibly be correct. The theorem makes a statement about the variance of deterministic quantities. Halfway through the proof in the appendix, the authors suddenly assume that these quantities are random and state assumptions on them. I am not claiming that the statement of Thm 1 is \"morally wrong\" because perhaps it can be shown to be correct with a proper setup. However, as the authors present it, it is uninterpretable.\n - Given how well the authors' solution appears to perform on the benchmarks they consider, I think it would be very useful to show the performance gap between models trained with their method or that are trained without using quantization.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nQ0qcmJF7oA", "reviewer": "Reviewer_GegD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author studies the batchnorm's negative effect when training with quantized gradient. To address the problem, the author proposes to add an additional loss to control the variance of batchnorm. Empirical results show improvement when quantizing weight, activation and gradient.", "summary_of_the_review": "The authors make a novel observation that batchnorm may contribute to gradient explosion in FQT. But I think more comprehensive theoretical and empirical results may be needed to make it more convincing. The method proposed seem to be too sensitive to the choice of the new hyper-parameter $\\lambda$, which makes it hard to validate its better performance since the improvements in Table 1 and 2 become negligible compared with the variance in Table 3.", "strength_and_weaknesses": "Strength:\n1. The observation of the batchnorm's negative effect on gradient explosion seems novel.\n2. The empirical results are comprehensive.\n\nWeaknesses:\n1. Does theorem 1 take weight and activation quantization into consideration? Why is the result not affected by the quantization?\n2. Corollary 1.1: should be not greater -> should not be greater\n3. Please provide a formal definition of D in Theorem 1.\n4. Eq. (4): why should $\\sigma_l$ approaches $\\sigma_*$? Theorem 1 seems to suggest that $\\eta$ should be as large as possible.\n5. From Table 4, the performance is extremely sensitive to the choice of $\\lambda$, considering the gap in Table 1 and 2 is not very obvious compared to $\\lambda$. Therefore, repeated experiments should be conducted for fair comparison in Table 1 and 2.\n6. How does it work for training full-precision networks with quantized gradients?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GdVSFXPFsp_", "reviewer": "Reviewer_dU93", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of quantized training where the gradients are quantized as well in addition to quantization aware training methods. The paper builds up on an interesting observation related to the detrimental effect of batch normalisation in quantized training. It is shown that batch norm amplifies the accumulated gradient quantization noise during the backpropagation. Based on this observation, a rectification method is proposed to reduce the negative effect of accumulated gradient quantization noise.", "summary_of_the_review": "Overall the paper makes a good attempt at the problem of quantized training and the proposed idea seems novel as well as technically solid to me. Though the results are not particularly strong. Still I would recommend weak acceptance at this stage.", "strength_and_weaknesses": "Strengths:\n- The paper is easy to follow and well-written.\n- There has been a lot of research focused on quantization aware training methods in the literature but this paper aims to solve the challenging task of quantized training. It is an important problem to increase training efficiency and this paper makes a good attempt at it.\nThe proposed method achieves improvements consistently on multiple datasets.\n\nWeaknesses:\n- The computational overhead of the approach has been shown to be not significant but it would be useful if the authors put some empirical comparisons on training time of the proposed method vs the baselines.\n- In table 1, the results for Mobilenet-V2 are considerably worse than UI8 and DAINT8. Can the authors explain the reason why the proposed method is worse in that setup?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject