GQ-Net: Training Quantization-Friendly Deep Networks
Authors: Rundong Li
Rui Fan
Venue:  (2020)
Keywords: Network quantization
Efficient deep learning

Abstract: Network quantization is a model compression and acceleration technique that has become essential to neural network deployment. Most quantization methods per- form fine-tuning on a pretrained network, but this sometimes results in a large loss in accuracy compared to the original network. We introduce a new technique to train quantization-friendly networks, which can be directly converted to an accurate quantized network without the need for additional fine-tuning. Our technique allows quantizing the weights and activations of all network layers down to 4 bits, achieving high efficiency and facilitating deployment in practical settings. Com- pared to other fully quantized networks operating at 4 bits, we show substantial improvements in accuracy, for example 66.68% top-1 accuracy on ImageNet using ResNet-18, compared to the previous state-of-the-art accuracy of 61.52% Louizos et al. (2019) and a full precision reference accuracy of 69.76%. We performed a thorough set of experiments to test the efficacy of our method and also conducted ablation studies on different aspects of the method and techniques to improve training stability and accuracy. Our codebase and trained models are available on GitHub.

Under review as a conference paper at ICLR 2020
GQ-NET: TRAINING QUANTIZATION-FRIENDDY DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Network quantization is model compression and acceleration technique that has become essential to neural network deployment. Most quantization methods per- form fine-tuning on pretrained network, but this sometimes results in large loss in accuracy compared to the original network. We introduce a new technique train quantizution-frieet frienndly networks, which can be directly converted to an accu- rate quantized network without the need for additional fine-tuning. Our technique allows quantizing the weights and activations of all network layers down to bits, achieving high efficiency and facilitating deployment in practical settings. Com- pared to other fully quantized networks operating at 4 bits, we show substantial improvements in accuracy, for example 66.68% top-1 accuracy ResNet-18, compared to the previous state-of-the-an accuracy of 61. 61.52% Louizos et al. (2019) and a full precision reference accuracy of 69.76%..99 19.76%. We performed a thorough set of experiments to test the efficacy of our method and also conducted ablation studies different aspects of the method and techniques improve training stability and accuracy. Our codebase and trained models are available on GitHub.
ImageNet using
INTRODUCTION
Neural network quantization is a technique to reduce the size of deep networks and to bypass com-
putationally and energetically expensive floating-point arithmetic operations in favor of efficient
integer arithmetic on quantized versions of model weights and activations. Network quantization
has been the focus of intensive research in recent years (Rastegari et al.
2016 Zhou et al. 2016
Jacob et al. 2018 Krishnamoorthi 2018 Jung et al., [ Louizos et al. 2019 Nagel al. 2019
Gong et al. 2019), with most works belonging to one of two categories. The first line of work quan- tizes parts the network while leaving portion of its operations, e.g. computations in the first and last network layers in floating point. While such networks can be highly efficient, using bitwidths down to or 4 bits with minimal loss in network accuracy (Zhang they may be difficult to deploy in certain practical settings, due to the complexity of extra floating point hardware needed to execute the non-quantized portions of the network. Another line of work aims for ease of real world deployment by quantizing the entire network, including all weights and activations in all convolutional and fully connected layers; we term this scheme strict quantization. Maintaining accuracy under strict quantization is considerably more challenging. While nearly loss- less 8-bit strictly quantized networks have been proposed (Jacob et al. 20188), to date state-of-the-art 4 bit networks incur large losses in accuracy compared to full precision reference models. For ex- ample, the strict 4-bit ResNet- 18 model in Louizos et al. (0119) has 61.52% accuracy, compared to 69.76% for the full precision model, while the strict 4-bit MobileNet-v2 model in Krishhamoorthi 20188) has 62.00% accuracy, compared to 71.88% accuracy in full precision.
al. 2018 Jung et al., 2018),
To understand the difficulty of training accurate low-bitwidth strictly quantized networks, consider common training procedure which begins with pre-trained network, quantizes the model, then applies fine-tuning using straight-througs achieves sufficient quantized accuracy. This process faces two problems. First, as the pre-trained model was not initially trained with the task of being subsequently quantized in mind, it may not be That is, the fine-tuning process may need to make substantial changes the initial model in order to transform it to an accurate quantized model. Second, fine-tuning a model, especially low bitwidths, is difficult due the lack of accurate gradient information provided
estimators (STE) for gradient updates until the model
"quantization-friennt

---

Under review as a conference paper at ICLR 2020
30
6
Porward pass
Backward pass
Operatars
Porameter feature maps
Figure 1: Architecture of the proposed GQ-Net. Input Zo follows the top and bottom paths to produce the full precision and quantized outputs ry and L resp. These are combined through loss functions Lj and bg form the overall loss L, which is optimized by backpropagation. For more details please refer to Section 3
by STE. In particular, fine-tuning using STE is done by updating a model represented internally with floating point values using gradients computed at the nearest quantizations of the floating point values. Thus for example, we apply 4 bit quantization to floating point model parameters in the range [0, 1]. random parameter will incur average round-off error of 1/32. which will be incorporated into the error the STE gradient for this parameter, leading to possibly ineffective fine-tuning.
To address these problems, we propose GQ-Net, a guided quantization training algorithm. The main goal of GQ-Net is to produce accurate and quantization-frian friendll full precision model, i.e. model whose quantized version, obtained by simply rounding each full precision value to its near- quantized point, has nearly the same accuracy itself. To do this, design loss function for the model which includes two components, one to minimize error with respect to the training labels, and another component minimize the distributional difference between the model's out- puts and the outputs of the model's quantized version. This loss function has the effect of guiding the optimization process towards a model which both accurate, by virtue of minimizing the first loss component, and which is also similar enough to its quantized version due to minimization of the second component to ensure that quantized model also accurate. In addition, because the first component of the loss function deals only with floating point values, it provides accurate gradient information during optimization, in contrast to STE- optimization which uses biased gradients rounded points, which further improves the accuracy of the quantized model. Since GQ-Net directly produces a quantized model which does not require further fine-tuning, the num- ber of epochs required train GQ-Net is substantially less than the total number of epochs needed to train and fine-tune a model using the traditional quantization approach, leading to significantly reduced wall-clock training time. We note that GQ-Net's technique independent of and can be used in conjunction with other techniques for improving quantization accuracy, as we demonstrate in Section 4.3 Finally, we believe that the guided training technique we propose can also be applied to other neural network structural optimization problems such as network pruning.
We implemented GQ-Net in PyTorch and our codebase and trained models are publicly available We validated GQ-Net on the ImageNet classification task with the widely used ResNet-18 18 and
An anonymous codehase has been submitted to OpenReview. The GitHub repository will be made public after the review process.
2

---

Under review as a conference paper at ICLR 2020
compact MobileNet-vi/Vy models, and also performed thorough set of ablation experiments to study different aspects of our technique. In terms of quantization accuracy loss compared to refer- ence floating point models, GQ-Net strictly quantized using 4-bit weights and activations surpasses existing state-of-the-art strict methods by up to 2.7 x, and also improves upon these methods even when they use higher bitwidths. In particular, 4-bit GQ-Net applied to ResNet-18 achieves 66.68% top-1 accuracy, compared to 61.52% accuracy in Louizos et al. (2019) and a reference floating point accuracy of 69%.6, while on MobileNet-v2 GQ-Net achieves 66. 15% top-1 accuracy compared to 62.00% accuracy in Krishnamoorthi (2018) and reference floating point accuracy of 71.88% Ad- ditionally, GQ-Net achieves these results using layer-wise quantization, as opposed to channel-win quantization in Krishnamoorthi 2018)) which further enhances the efficiency and practicality of the technique.
2 RELATED WORKS
Neural network quantization has been the subject of extensive investigation in recent years. Quan- tization can be applied to different part of neural networks, including weights, activations or gradi- ents. Courbariaux et al. (2015), Hou et al. (2016), Zhou et al. (017)). Hou & Kwok (2018) and other works quantized model weights to binary, ternary or multi-bit integers to reduce model size. |Weil et al. (2018) quantized activations of object detection models for knowledge transfer. AAlistarh et al. (2016). Hou et al. (2019) quantized model gradients to accelerate distributed training. Another line of work quantizes both weights and activations to accelerate model inference by utilizing fix-point or integer arithmetic. These works include Courbariaux et al.] (2016). Rastegari et al. (2016), (Gyse al. (2016), Krishnamoorthi (2018), Choi et al. (2018). Zhang al. (2018), Jung et al. (2018).
A large set of methods have been proposed to improve training or fine-tuning for network quan- tization. Straight through estimators (Bengio al. 2013) (STE) propagate gradients through non-differentiable operations with the identity mapping. Other training methods "soften" non- differentiable operations similar differentiable ones in order for gradients to pass through, then gradually anneal to piecewise continuous functions by applying stronger constraints. This line of works include Loouzes et al. 2019), 9.. Gong et al. a (2019) Bai et al. (2018). Some works regard quanti- zation as stochastic process that produces parameterized discrete distributions, and guides training using gradients with respect to these parameters Soudry et al. (2014), Shayer et al. (2018). Another line of works does not require fine tuning, and instead re-calibrates modifies the original network to recover accuracy using little or even no data He & Cheng (2018). Nagel et al. (2019). Meller et al. 2019).
Several recent works have focused on quantizing all parts of a network, typically in order to support deployment using only integer arithmetic units and avoiding the cost and complexity of additional floating point units. Gysel et al. (016)) proposed performing network inference using dynamic fixedpooint arithmetic, where bitwidths for the integer and mantissa parts are determined based model's weight distribution. Jacob et al. (2018); Krishnamoorthi (2018) proposed the quantization training and deployment algorithm behind the Tensorflow-Lite quantization runtime, which gener- ates strictly quantized networks that can be easily implemented in hardware. Louizos et al. (2019) proposed a training method for strictly quantized models based on annealing a smooth quantization function to piecewise continuous one. There has also been recent work on using parameterized quantizers which are optimized during quantization training. Choi et al. (2018) introduced learnable upper bounds to control the range of quantization. /Zhang et al. (2018) 8) proposed quantizers with a learnable basis which an be executed using fixed- point arithmetic. Jung et al. (2018) proposed to optimize weight scaling and quantization ranges jointly from task losses.
GQ-NET
In this section we describe the architecture of our proposed GQ-Net and then discuss components of the architecture which can be tuned to improve performance.
3.1 GQ-NET ARCHITECTURE
The major components of GQ-Net include the following, and are illustrated in Figure
3

---

Under review as a conference paper at ICLR 2020
1. An L-layer neural network h-) with all computations performed using full precision floating point arithmetic. Here W = {Wi..... W denotes the parameter (weights) of the model, with W,, i E 1.... being the weights in layer i and expressed in floating point.
2. The quantized model (w:.(-)) built from (-). Here Q {Qr...... .................. QE. Qu.... Q7 is a set of quantizers, i.e. mappings from floating point to (scaled) integer values; the quan- tizers may be parameterized, and we describe how to optimize these parameters in Section Q." quantizes weights W, and Q quantizes activations in layer
3.2
Let æo denote an input to hw. To construct output l (20) of the quantized network, we proceed layer by layer. We first quantize the weights in layers i = 1,...,A L as w; (wx)), and also quantize the input by setting 2n = Q (zo). we compute the quantized activations ft in layer i iteratively for i L...., L using é; = Q2 (#+), where ă; 9: (á;; 1;-1), and g.(.) denotes the nonlincarity function in layer i and denotes convolution. Note that since tū; and - are quantized, ft can be computed using integer fixed point arithmetic.
3. Next, we construct a loss function L incorporating both the training loss L, of the full precision model hw and loss Lg capturing the difference between hy and the quantized model hw.q-
[ w + Wigg
Here wy: Wi E R are parameters capturing the relative importance of training loss versus distributional loss. In this paper, we focus on image classification networks, and thus we set Ly to be the cross-entropy loss between outputs from lww and the training labels. In ad- dition, we set L. = DKL (o (hw (lowb. where denotes the softmax function, to be the KL divergence between distributions a (hw) and o (hw.Q) on each input. Hence, minimizing the second term in L corresponds to pushing the floating point and quantized models to behave similarly to each other.
Since the weight parameters W appear in both terms in L, the two terms can give conflict- ing signals for updating W during the optimization of L. causing the optimization to be unstable. We discuss how to deal with this problem in Section 3.2
To train GQ-Net, we successively take mini-batches of training samples and labels and use them to compute L during the forward pass and propagate gradients with respect to W and the parameters of Q during the backward pass in order minimize L. After has converged sufficiently, we take the quantized weights in (- as the quantized model.
3.2 OPTIMIZING GQ-NET
We now deseribe how different components of GQ-Net can be optimized to improve accuracy and training stability.
Weight scheduling for Lj and L Parameters wy and why capture the relative importance of the cross entropy and KI. divergence errors during training. A large wy ignores the similarity be- tween the floating point and quantized models and may result in model that is accurate but not quantization-friee friendll. Conversely, a large w, ignores guidance on accuracy from the floating point model and may result in similar but poorly performing floating point and quantized models.
We tested different schemes for weighting Ly and Ey. and found that using fixed values such as wy Wy 0.5 already yields better results than many current methods, as discussed in Section However, further experimentation showed that scheduling, i.e. dynamically modifying the values of wy, wl during training can produce higher accuracy than using static values. For example, consider schedule as shown in Figure 2a which initially sets wy 1, 1, = O, then alternates between setting wig 1 and wy = several times. Schedules of this sort can be understood as initially favoring model accuracy so that the floating point model is driven to high accuracy region of model space, before increasing the importance of model similarity so that a quantization -riendlly model can be found in the high accuracy region. This is repeated several times, leading to increasingly more accurate full precision models whose accuracy is then transferred to the quantized model. As demonstrated in Section this schedule results in better performance than static ones.
4

---

Under review as a conference paper at ICLR 2020
1.0
7000
0000
5000
0.6
LR
0.4
3000
2000
1000
0.0
20000
40000
60000
(a)
(b)
Figure 2: (a) Schedule for learning rate, wy and Wy. r-axis shows the number of train- ing steps, y-axis shows the value of the learning rate or loss weight. (b) Distribution of layer2. downsample. 0 convolution layer outputs in full precision (blue) and 4-bit quan- tized (orange) 18, showing the necessity of multi-domain BN. Note that outputs are not quantized until non-linear layers, thus the quantized distribution has more than 16 bins.
Reducing interference between L and Eg The loss function includes terms in Ly and Cy. where the former is function of the floating point parameters W, and the latter involves both W and the quantized version W of W parameterized by 0. We discovered that directly optimizing L results in reduced accuracy, which we attribute to conflicting updates to W produced by gradients from the C and Lg terms.
Ideally, we would like update to minimize Cy. while independently updating W to minimize Lg. While this is clearly not possible due to the dependency between W and W, we found that heuristic based on this idea helped improve accuracy. In particular, let æz = hw (Ir) and #y hw.q (zo) be the output of the full precision and quantized networks on input æg. so that C, = compute L3 from FL and derive Va, fy as usual, and use this via the chain rule to update W. However, when computing bg from 3r, and Tt. treat Fy as constant tensor which does not produce any gradients, and only derive Vä, Lyg and use this via the chain rule to update W. We can implement this behavior using the det ach operator in PyTorch or the stop-gradient operator in TensorFlow.
DKL(a (xLL [xx),,
in Section 3.1 During back propagation
T for a discrete set T. Motivated by recent work such PACT (Choi al., 2018), we adopt layer-wise linear quantizers with learnable boundaries in GQ-Net. In particular, for each layer i, we use one weight quantizer function QT (-) for all weights in the layer, and one activation quantizer function Qi (-) for all activations. Here, 8 and @ represent learnable parameters; for expository simplicity we drop a, w and in the following and denote all parameters by 0. 0 consists of k, the quantization bitwidth, and and ub representing the lower and upper quantization boundaries. We use uniform quantization, which is substantially easier to implement in hardware, and set
Parameterized quantizer GQ-Net can use any type of quantizer Q(-)
A ub lb
2
-
Qo(x)= = clamp(e, lb, ub) 1b
A + lb
Where clamp(æ, a, b) max(a, min(æ, b)), and L.1 represents the round operator.
During training, gradients propagate through the nondifferentiable [.] operator using the straight though estimator (STE), i.e. 8e from the loss function L, and thus the quantizers will learn to set the appropriate quantization bound- aries improve accuracy.
0| 1. Parameters 1b, ub are updated by the gradients propagated
5

---

Under review as a conference paper at ICLR 2020
Alternatingly optimizing W and The accuracy of the quantized model depends both the weights W of the floating point model as well as how these are quantized using the quantizers parameterized by 0. We found that jointly optimizing W and 0 in each iteration resulted in unstable training and poor accuracy. Performance was substantially improved by alternatingly optimizing W and 0. That is, in each training epoch update either the values in W or 0 while freezing the values of the other set of parameters, then switch the updated and frozen sets in the next epoch. The reason alternating optimization improved training that both W and affect the values of quantized weights, so that updating both simultaneously may cause suboptimal changes to quantized weights.
Multi-domain batch normalization Batch normalization is a critical component in large model training. However, since GQ-Net trains floating point and quantized model simultaneously, we need to adjust the way batch normalization is performed to achieve good accuracy. In particular, as illustrated in Figure 2 activations from the floating point and quantized models follow differ- ent distributions. Thus, normalizing them with same set of running statistics can hinder training. Instead, we regard activations different numerical precision as being from different domains, sim- ilar to as in multi-domain transfer learning, and normalize them with separate statisties {uy, and {ug: }: = "ن E-Bx, This modification only introduces minor storage overhead, while it significantly improves GQ-Net's accuracy in both full-precision and quantized settings.
oj
dg
EXPERIMENTS
To validate the effectiveness of GQ-Net and assess its different components, we conducted series of comparisons and ablation studies using the ImageNet classification task. We used the ILSVRC 2012 dataset consists of 1.2M training samples and 10K validation samples from 1K categories, and evaluated our system using top-1 I and top-5 validation accuracies.
4.1 IMPLEMENTATION DETAILS
Network settings We used the ResNet-18, MobileNet-vi and MobileNet-v2 architectures in the ImageNet experiments. All MobileNets used channel expansion ratio 1.0. Unlike some recent works, we did not modify the order of the Cony, BN and ReLU layers. We replaced the BatchNorm layers in these models with SyncBatchNorm, i.e. we used mini-batch statistics Hr, and o Og computed from all distributed GPUs during training.
Quantization settings Unless otherwise specified, all of the following experiments were con- ducted with parameterized linear quantizers using bitwidth of 4. The weight and activation quan- tizers each have their own parameters 0 = {lb, ub). which are initialized at the iteration right before the quantization error penalty Cy is enabled. Specifically, for weight quantizers, {1b, ub} are ini- tialized by the minimum and maximum elements in the weight tensors each layer. For activation quantizers, (1b, ub} initialized by the upper and lower 99.9% percentile values, computed from 5 mini-batches sampled from the training set. Weights and activations of all layers were quantized, including the first and last layers.
Training protocol We used the same training protocol for all architectures and quantization set- tings. Training was performed on 32 distributed GPUs, each with a mini-batch size of 64, and stopped after 120 epochs on the training set. Model weights W and guantization parameters were optimized with different optimization settings. Model weights W were randomly initialized using the Kaiming-normal scheme (He et al. 2015) without using pre-trained model, and optimized by SGD with 0.9 Nesterov momentum and 10 weight decay. The learning rate warmed-up from 0.2 to 0.8 in the first epochs, and decayed twice by factor of 0.1 at epochs 60 and 90. Quantization parameters 0 were optimized using Adam without weight decay, with coefficients for first and sec- ond momentums set to 31 = 0.9 and 32 = 0.999, and learning rate fixed to 10 during the entire training process. Following standard practice, training samples to 224 x 224 pixels, followed by random horizontal flipping and normalization. Validation samples were centrally cropped to 224 x 224 pixels, followed by normalization.
resized and randomly cropped

---

Dynamically adjusting the weights W and why during different parts of the training process as de- scribed in Section 3.2 improved quantization accuracy by +0.23%. This suggests that the impor- tance of the accuracy loss Cy and the distributional loss E, may not be equal at different stages of Under review as a conference paper at ICLR 2020
Table 1: Comparison with other strict quantization methods on top-1 and top-5 accuracy for Ima- geNet. W/A indicates the quantization bitwidths for weights and activations, respectively. 32/32 indicates full full-precision models.
ResNct-18
MobileNet-vi
MobileNet-v2
Method
W/A
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Reference
32/32
69.76
89.08
70.60
71.88
90.29
Ours
32/32
69.89
89.27
70.68
89.64
71.09
90.12
White Paper
4/8
65.00
62.00
White Paper
8/4
64.00
58.00
Integer-only
5/5
64.64
86.67
RelaxedQuant
5/5
65.10
86.57
61.38
83.73
RelaxedQuant
4/4
61.52
83.99
GQ-Net (ours)
4/4
66.68
87.46
65.04
86.09
66.15
86,92
4.2 COMPARISON WITH OTHER STRICT QUANTIZATION METHODS
We validated the effectiveness of GQ-Net by comparing it with several other state-of-the-art quan- tization methods. As GQ-Net seeks to fully quantize a network and execute it using only integer operations, perform comparisons with other strict quantization methods. The comparison base- lines include results from (Krishnamoorthi 2018). (Jacob et al., 2018) and (Louizos et al. 2019), indicated White Paper, Integer-only and RelaxedQuant resp. in Table
The first row in the table contains full precision accuracy results evaluated using reference imple- mentations while the second row contains the full precision accuracy for models trained with training protocol. For the ResNet-18 18 model which widely studied in the network compression literature, we significantly outperform the state-of-the-art strict quantization method RelaxedQuant in an equal bitwidth setting 1.6. top-1 accuracy improvement with 4-bit weights and activa- tions). Our 4-bit model even outperforms the comparison methods when they use higher bitwidths (+1.58% compared with 5-bit RelaxedQuant, +2.04% compared with 5-bit Integer-only). For the compact MobileNets family, our method also achieves higher accuracy using lower bitwidths. For example, compared with the White Paper method at 8-bit weights and 4-bit activations, our 4-bit W/A GQ-Net MobileNet-v2 model achieves +8.11 15% top-1 accuracy improvement, and our GQ- Net MobileNet-vl model obtains +1.04% higher top-1 accuracy in the same setting.
4.3 ABLATION STUDIES
We also validated the effectiveness of the different components of GQ-Net discussed in Section 3.2 by progressively adding them to the vanilla quantization-friett these protocols to the ResNet-18 architecture with its weights and activations quantized to 4-bits. For the vanilla setting, W and @ were optimized jointly at each training step, loss weights were set to w/ = Wa = 0.5 through the entire training process, gradients from L, propagated through both (xr) and o (āz), and full precision and quantized activations were normalized using the same set of running statistics. For a more complete comparison, we also evaluated these settings using the full precision GQ-Net model. The numerical results are given in Table
training protocol, and applying
By alternatingly updating model weights W and quantizer parameters 0 between training steps, quantization accuracy improved by +4.24% over the vanilla protocol. This indicates that although gradients V w L and VoL. both guide their respective parameters minimize training loss, combin- ing them in a training step makes the quantized weights W derived from these parameters subopti- mal.
*Implementatioos of R8 and MobileNet-v2 are taken from torchvision package (v0.3.0. httts::/pytorr orgg docs/ stable/torchh inniviiionn / 1 1.hmml) The accuracy of MohileNet-vi cited from the original paper.
7

---

Under review as a conference paper at ICLR 2020
Table 2: Full-precision (FP) and quantized (Q) top-1 accuracy for 4-bit weights and activations in ResNet- 18 on ImageNet. Components of GQ-Net are progressively applied. The first row indicates the vanilla setting. check mark in column "Alt (W.0)" means alternatingly optimizing W and 8. A check for "Schedule wi,wh"" means wi and Wi were dynamically adjusted, as described in (3.2 check for "Detach (xr1)" means that (xr,) is treated as a constant and we do not propagate V Vo(wg) Cy during the backward pass. A check for "Multi-domain BN" means that full-precision and quantized activations are normalized using separate running statistics. The full full-precision reference has top-1 accuracy 69.89.)
Alt (W.0) Schedule w7,4,
Detach (xrr)
Multi-domain BN
Top-I (FP)
Top-1 (Q)
65.21
60.95
67, 14
65.19
66.56
65.42
66.61
66.08
68.30
66.68
the training process. The schedule used, alternating between short periods of w 0 and longer periods of W/ = 1, suggests we should first allow the floating point model settle into a reasonably accurate state before enforcing its similarity to the quantized model.
Blocking the gradient from L,, caused by (xy, further improved quantization accuracy by +0.66% This indicates that although the full precision logits æ and quantized logits 7 are both derived from the same set of parameters W, it is useful to heuristically separate their effects during backpropaga- tion.
Normalizing full precision and quantized activations by different sets of running statistics in the BatchNorm layers improved both full precision accuracy (+1.69%) and quantized accuracy (+0.60%). This highlights the difference in the mean and variance of the full precision and quan- tized activations, despite the similarity of the full precision and quantized models in terms of KL. divergence.
Lastly, we considered the effectiveness of using parameterized quantizers. We tested replacing the parameterized quantizers in GQ-Net with naive linear quantizers using fixed weight and activation quantization ranges set to the 99.9% percentile values derived from 5 initial training mini-hatches. We found that using fixed quantizers significantly lowered accuracy, by 3.59%. We note that Re- laxedQuant also uses learned quantizers, and that replacing these with fixed quantizers may also result in decreased accuracy.
CONCLUSION
In this paper we presented GQ-Net, novel method for training accurate quantized neural networks. GQ-Net uses a loss function balancing full precision accuracy as well as similarity between the full precision and quantized models guide the optimization process. By properly tuning the weights of these two factors, we obtained fully quantized networks whose accuracy significantly exceeds the state of the art. We are currently studying additional ways adjust GQ-Net components further improve accuracy. We are also interested in combining GQ-Net with complementary quantization techniques, and applying similar methodologies other neural network optimization problems.
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Communicatonn-fficiee sgd via gradient quantization and encoding, 2016.
Qsgd:
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.
8

---

Under review as a conference paper at ICLR 2020
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, 2018.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations, 2015,
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or 1, 2016.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks, 2019.
Philipp Gysel, Mohammad Motamedi, and Soheil Ghiasi convolutional neural networks, 2016.
Hardware-oriented approximation of
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing
human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi: 10.1109/icc.22155.223 URL http:// doi.
org/10.1109/I0e 20155.20
Xiangyu He and Jian Cheng. Learning compression from limited unlabeled data. In The European Conference on Computer Vision (ECCV), September 2018.
Lu Hou and James T. Kwok. Loss-aware weight quantization of deep networks, 2018.
Lu Hou, Quanming Yao, and James T. Kwok. Loss-aware binarization deep networks, 2016.
Lu Hou, Ruiliang Zhang, and James Kwok. Analysis of quantized models. In International Conference Learning Reprexentations, 2019. URL http://s/////peeeeee...... forrmm?
id-ryM IoAaYX
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeticoonl inference. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: (t009/ccpp:2018.002868 URL httphhtp///aet http//xx..d/ doi org/ 10. 10.1109/ 101199/vvr... 2018 00286
20080
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon Han, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss, 2018.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper, 2018
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019. /ttpttpps///pperreveee..... http:/iwwwwweeeeeeeeeeeeeeeeeeee hww..nnt//formmmll-.kkkk.
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different recovering neural network quantization error through weight factorization, 2019.
Markus Nagel, Mart Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. Lecture Notes in Computer Science, pp. 525-542, 2016. ISSN 1611-3349. doi: 0000079978--31--449942 dx.doi .rgg 10. 007 - 9- 0-493-00333
URL http://
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local repa- rameterization trick. In International Conference on Learning Representations, 2018. URL
httpps://opnrewview net ttte///ooommlldByRHHtrr

---

Under review as a conference paper at ICLR 2020
Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter- free training of multilayer neural networks with continuous or discrete weights.
In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q- Wein-
berger (eds.), Advances in Neural Information Processing Systems 27, pp. 963-
971. Curran Associates, Inc., 2014. URL http://papers. wwtpppcccpppppeeeeeeeeeco
5269 expectation pdf
multi multilayerr neural -ntrall-etwt
Yi Wei, Xinyu Pan, Hongwei Qin, Wanli Ouyang, and Junjie Yan. Quantization mimic: Towards very tiny enn for object detection. Lecture Notes in Computer Science, pp- 274-290, 2018. ISSN
1611-3349. doi: 11110779778-3-000--123---111 URL http://dx.. http:///...ii.0//111.1..1.....0.
978 3 030 01237-333
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks, 2018.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights, 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients, 2016.
10


Reviews:
[{"review_id": "SkeVhAmjYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces GQ-Net, a novel technique that trains quantization friendly networks that facilitate for 4 bit weights and activations. This is achieved by introducing a loss function that consists of a linear combination of two components: one that aims to minimize the error of the network on the training labels of the dataset and one that aims to minimize the discrepancy of the model output with respect to the output of the model when the weights and activations are quantized. The authors argue that this has the effect of “guiding” the optimization procedure in finding networks that can be quantized without loss of performance. For the discrepancy metric the authors use the KL divergence from the predictive distribution of the floating point model to the one of the quantized model. The authors then propose several extra techniques that boost the performance of their method: 1. scheduling the weighting coefficients of the two loss terms (something which reminisces iterative pruning methods), 2. stopping the gradient of the floating point model w.r.t. the second loss term, 3. learning the parameters of the uniform quantizer, 4. alternating optimization between the weights and the parameters of the quantizers and 5. using separate batch normalization statistics for the floating point and quantized models. The authors then evaluate their method on Imagenet classification using ResNet-18 and Mobilenet v1 / v2, while also performing an ablation study about the extra tricks that they propose.\n\nThis work is well written and in general conveys the main idea in an effective manner. Quantization friendly neural networks in an important subject in order to make deep learning tractable for real world applications. The idea seems on a high level to be interesting and simple; train floating point models that can fit the data well while also encouraging them to be robust to quantization by enforcing the predictive distributions of the fixed and floating point models to be similar in the KL-divergence sense. Nevertheless, I do have some comments that would hopefully help in improving this work:\n\n- It does seem that GQ-Nets need extra tricks in order to perform well, and those tricks come with their own set of hyperparameters that need to be tuned. For example, at section 4.3 you mention that the top-1 accuracy of vanilla GQ-Nets is 60.95, which is lower than the RelaxedQuant baseline (that has 61.52). This raises the question whether the boost in performance is due to the several additional steps employed (which in general can be applied to other quantization techniques as well), and not due to the main idea itself. \n- Do you employ the straight-through estimator (STE) for the weights in the L_q objective? In the second paragraph of the second page you argue that due to the biased gradients of STE the performance is in general reduced, so I was wondering whether STE posed an issue there or whether you used an alternative estimator. \n- How is batch normalization handled? Do you absorb the scale and shifts in the weights / biases before you perform quantization or do you quantize the weights and then apply the BN scale and shift in full precision?\n- How do you ensure and ub > lb when you learn the quantizer? In general learning the quantizer can be also done with alternative techniques (e.g. simply learning the scale and offset) so I was wondering whether you noticed benefits from using the ub, lb parametrization compared to others.\n- Do you show the pre-quantization distributions at Figure 2b? In the caption you mention quantized but the resolution seems to be higher than the 16 values you should get with 4 bits. Furthermore, it should be noted that the discrepancy in BN in quantized models was, as far as I am aware, firstly noticed at [1] (and subsequently at RelaxedQuant) and both of these methods simply re-estimated the moving averages during the inference time.\n\nOverall, I am on the fence about this work and tend to reject. Having said that, I am of course willing to revise my score after the discussions with the authors / other reviewers.\n\n[1] Probabilistic Binary Neural Networks, Jorn W.T. Peters, Max Welling", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hye6gdYCYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a new quantization-friendly network training algorithm called GQ (or DQ) net. I addresses the existing issues in the common paradigm, where a floating-point network is trained first, followed by a second-phase training step for the quantized version. It is a well-written paper. Concepts were clearly explained and easy to follow. Below I present my comments about some details in the paper that were not entirely clear for me. \n\n- The two loss terms conflict each other. If the training algorithm focuses too much on the first term, it will make the network less friendly to the quantization process. On the other hand, the second one is going to enforce too much emphasis on the accuracy from the quantized network. It is natural to involve some hyperparameter search to find the balance between the two blending parameters. The paper suggests a strategy as to how to handle this issue, but it is not comprehensive, and rather controversial. I think the paper will benefit from a more in-depth discussion and analysis on this regularization issue. \n\n- The schedule for the loss term blending parameters looks drastic to me. It’s more like “train the floating point net first, and then train the quantized one, and then revisit the floating point one, and so on.” I know I simplified, because the floating point network never stops getting updated as it’s \\omega_f is always 1. However, it seems to me that this drastic scheduling strategy sounds like very similar to the traditional approach that trains the floating point network first and then finetune the quantized one, except for the fact that this proposed algorithm repeats this process a few times. Hence, I think the authors’ argument about the supremacy of the proposed method to the two-step finetuning approach is not clearly supported. \n\n- The exponentially decaying learning rate scheduling looks like the one from ResNet. I’m wondering if it should be the best, especially with the drastic introduction and omission of the second loss. \n\n- In the ablation studies, it seems that some of the suggested training options are conflicting each other and the clear winner seems to be the multi-domain BN. I cannot conclude anything from this analysis as to which one is more important than the other one, except for the Alt{W,\\theta} case. \n\nSome minor things:\n\n- What’s the name of the proposed network? Is it GQ or DQ?\n ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlVa_8WcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a framework towards 4-bit auantization of CNNs. Specifically, during training, the proposed method contains a full precision branch supervised by classification loss for accurate prediction and representation learning, as well as a parameterized quantization branch to approximate the full precision branch. A quantization loss between the full precision branch and the quantization branch is defined to minimize the difference between activation distributions. The authors proposed a series of improvements, including alternative optimization, dynamic scheduling, detach and batch normalization to help boosting the performance to SOTA under 4-bit quantization.\n\nStrengths:\n+ Well-written paper with good clarity and technical correctness.\n+ Proposed method seems light, sweet and technically correct.\n+ Good experimental performance and result on ImageNet.\n+ Good and clear ablation study.\n\nWeaknesses:\n- Major performance improvement comes from the combination of different incremental improvements.\n- Lack of evaluations with variety of datasets (CIFAR-10/MNIST)/configurations (other bitwidth)\n- Lack of the citation and comparison to many most recent works on binarized networks (except XNOR-Net)\n\nComments:\nI consider this a well-written paper with great clarity and good empirical performance. I enjoyed reading the paper. The proposed framework seems technically correct and effective. \n\nHowever, a major weakness of this work is that most of the performance improvement comes from a combination of add-on improvements, except that the authors put them together into a unified framework and explained elegantly. The vanilla architecture, which is a main contribution and described in Fig. 1, doesn't seem to give that significant improvement. To some extent, the real technical contributions of this work are partly weakened given the add-on combinations and the existence of similar methods. For example, the alternative optimization of W and \\theta is similar to alternative re-training in network pruning, although a unified loss/optimization framework is applicable in this case. Others such as dynamic scheduling and gradient detach are also heuristic-driven.\n\nThe results on ImageNet under 4-bit quantization are strong and convincing, but the paper could benefit from conducting additional experiments on different datasets and bitwidth configurations. A more comprehensive study similar to Louizos et al., 2019 will be great. Citations and comparisons to more recent binarized networks other than XNOR-Net will be appreciated too.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper propose a new quantization-friendly network training algorithm called GQ (or DQ) net. The paper is well-written, and the proposed idea is interesting. Empirical results are also good. However, the major performance improvement comes from the combination of different incremental improvements. Some of these additional steps do seem orthogonal to the proposed idea. Also, it is not clear how robust the method is to the various hyperparameters / schedules. For example, it seems that some of the suggested training options are conflicting each other. More in-depth discussions and analysis on the setting of the regularization parameter and schedule for the loss term blending parameters will be useful.
Decision: Reject