Linear Symmetric Quantization of Neural Networks for Low-precision Integer Hardware
Authors: Xiandong Zhao
Ying Wang
Xuyi Cai
Cheng Liu
Lei Zhang
Venue:  (2020)
Keywords: quantization
integer-arithmetic-only DNN accelerator
acceleration

Abstract: With the proliferation of specialized neural network processors that operate on low-precision integers, the performance of Deep Neural Network inference becomes increasingly dependent on the result of quantization. Despite plenty of prior work on the quantization of weights or activations for neural networks, there is still a wide gap between the software quantizers and the low-precision accelerator implementation, which degrades either the efficiency of networks or that of the hardware for the lack of software and hardware coordination at design-phase. In this paper, we propose a learned linear symmetric quantizer for integer neural network processors, which not only quantizes neural parameters and activations to low-bit integer but also accelerates hardware inference by using batch normalization fusion and low-precision accumulators (e.g., 16-bit) and multipliers (e.g., 4-bit). We use a unified way to quantize weights and activations, and the results outperform many previous approaches for various networks such as AlexNet, ResNet, and lightweight models like MobileNet while keeping friendly to the accelerator architecture. Additional, we also apply the method to object detection models and witness high performance and accuracy in YOLO-v2. Finally, we deploy the quantized models on our specialized integer-arithmetic-only DNN accelerator to show the effectiveness of the proposed quantizer. We show that even with linear symmetric quantization, the results can be better than asymmetric or non-linear methods in 4-bit networks. In evaluation, the proposed quantizer induces less than 0.4\% accuracy drop in ResNet18, ResNet34, and AlexNet when quantizing the whole network as required by the integer processors.

Published as a conference paper at ICLR 2020
LINEAR SYMMETRIC QUANTIZATION OF NEURAL NETWORKS FOR LOW-PRECISION INTEGER HARD- WARE
Xiandong Zhao' Ying Wang Xuyi Cail2, Cheng Liu', Lei Zhang!
Institute of Computing Technology. Chinese Academy of Sciences'
University of Chinese Academy of Sciences?
State Key Laboratory of Computer Architecture fzhaoxiandong, wangying2009, caixuyi18s, liucheng,nnn zlei]eict. ac.
ABSTRACT
With the proliferation of specialized neural network processors that operate on low-precision integers, the performance of Deep Neural Network inference be- comes increasingly dependent on the result of quantization. Despite plenty prior work on the quantization of weights activations for neural networks, there is still a wide gap between the software quantizers and the low-precision accel- erator implementation, which degrades either the efficiency of networks that of the hardware for the lack of software and hardware coordination at design- phase. In this paper, we propose learned linear symmetric quantizer for integer neural network processors, which not only quantizes neural parameters and acti- vations low-bit integer but also accelerates hardware inference by using batch normalization fusion and low- low-precision accumulators (e.g., 16-bit) and multipli- ers (e.g., 4-tit). We use a unified way to quantize weights and activations, and the results outperform many previous approaches for various networks such AlexNet, ResNet, and lightweight models like MobileNet while keeping friendly to the accelerator architecture. Additional, we also apply the method to object de- tection models and witness high performance and accuracy in YOLO-v2. Finally, we deploy the quantized models our specialized integer-arithmeticonnn DNN accelerator to show the effectiveness the proposed quantizer. We show that even with linear symmetric quantization, the results can be better than asymmetric non-linear methods in 4-bit networks. In evaluation, the proposed quantizer in- duces less than 0.4% accuracy drop in ResNet18, ResNet34, and AlexNet when quantizing the whole network as required by the integer processors.
INTRODUCTION
Deep neural networks have shown excellent performance on various computer vision and natural language processing tasks, such as classification (Krizhevsky et al. 2012, Simonyan & Zisserman 2015 He et al. 2016), object detection (Girshick] 2015 Redmon et al. 2016; He et al.) 2017). segmentation Long et al. 2015 et recognition (Nassff et al. 2019). J etc. While the past few years witnessed the success of DNNs cloud and server-end computers, neural networks have been recently pushed to embedded and mobile areas to enable edge intelligence. For these scenarios, the power provision and computational strength on the edge computing devices are limited. As result, it essential to have more efficient network architectures and less expensive inference overhead. Therefore, there is increasing attention from the research community study the compression of modern deep neural networks that are typically over- wer-parameterized and computationally costly.
machine translation (Zhang al. 2018b). speech
Several categories of approaches are proposed decrease the computational overhead of neural net- works, such as lightweight neural network architectures (Howard et al. 2017), neural architecture
search (NAS) (Elsken al. 2018), and network pruning (Han et al.) 2015 2016 Wen et al. 2016
"Corresponding author

---

Published as a conference paper at ICLR 2020
Comparison between different quantizers: All-Layer (AL) denotes quantizing all the pa- rameters of all the operators in networks, including weights, bias, activations, and the scaling factor for low- low-precision networks; BN donates that the BN operation is only invoked in training but merged into weights and induces overhead in integer inference; Linear-Symmetric (LS) denotes linear symmetric quantization; Activation Functions (AF) donates the support Leaky ReLU and activa- tion functions besides ReLU. Structure-Intar (SI) indicates the network structure is unmodified.
Table 1:
Method
AL
BN
LS
A/
ST
Deep Compression (Han et al.
2016
WQ (Park al. 2017
LQ-Nets Zhang et al. 201 8a
Min-Max Linear Quanti
DoReFa (Zhou et al. 2
RQ Louizos cf al.
2019)
WRPN (Mishra et a. 2018
PACT (Choi 2018
LLSQ(
Naive linear quantization, which finds min-max value at runtime. Clustering-hasct approaches quantize weights.
DoReFa falls into linear asymmetric quantizer due to the need for offset.
In Choi (0018), they use PACT to quantize activations, and DoReFa to quantize weights.
DoReFa, RQ WRPN, and PACT are designed for ReLU, but they can be extended to support other activation functions in theory.
Molchanov al. 2017). Besides these techniques, quantizing sigh-precision floating pointt net- works to lower bitwidth representation can also drastically decrease both the static parameters and the intermediate data generated during the network inference, resulting in reduced memory footprint and also computational intensity. And this paper focuses on the quantization of neural networks.
Quantization technique is also closely related to the implementation of specialized hardware that maps the procedure of network inference onto the energy-efficient low-precision integer or fixed- point arithmetic circuits. In the hardware perspective, low-precision integer accelerators or proces- sors are dominating the solutions targeted on neural network inference, especially for mobile and embedded scenarios. Google's 's Tensor Processing Unit 1.0 (TPU) (Jouppi et al.) (017)) Unified Deep Neural Network Accelerator (UNPU) (Lee et al. 2018), Eyeriss (Chen et al. 2018), Stripes (Judd) al. 2017) and many other newly proposed hardware implemen- tations are generally reliant on the effectiveness of the underlying quantization techniques, which are especially crucial for the low-precision integer hardware designed to process binary, ternary, 4-bit or 8-bit networks. In other words, quantization is not only a method to reduce the memory footprint as in traditional work, but also a mandatory step to make the network deployable on integer hardware.
al., 2)1 Tragmatic(Albericoe
Though there is lot of prior work that investigates low- low-precision quantization, they mainly target on reducing the memory overhead caused by floating or high precision data representation in the networks, but not focus on specialized integer hardware for network inference. To enable the neural network processors to work with low-precision integer operands and minimize the accuracy losses, a good network quantizer must satisfy the constraints as enlisted in Table
First, all the parameters, including weights, bias, activations, partial results that eventually accumu- late to an activation, and even the scaling factors, which are indispensable for low-precision networks like binary and ternary representation, must be quantized into low bitwidth integers as required by the underlying specialized hardware. In some prior work (Zhou et al. 2016 Zhu al. 2017 Zhang et al., 201 Mishra et al.) 2018 Choi 2018), they either leave bias and scaling factors unquantized or keep the first and last layer in full high precision. Besides, some designs rely on high-precision internal register or ALUs to support high-precision partial results that are generated during com- putation before the final output of activations or features. For example, Kriihnamoorthi (2018), which quantizes the weights and activations to 8-bit, directly use 32-bit accumulators to cache the intermediate values or partial results to avoid overflows. However, for 4-bit and lower bitwidth, the integer accelerators cannot afford high bitwidth accumulators, which indicates higher silicon area and power cost. For antlger-only-aritnnee forwir method. The value range of these numbers is wide, resulting in overflows of the
we quantize the bias to fixed- fixed-point numbers by using
2

---

Published as a conference paper at ICLR 2020
low bitwidth accumulators. To overcome this problem, quantize the bias to 8-bit and finetune the bias of the model. As shown in Figure the bitwidth of accumulators can be reduced to 16-bit.
Second, the BatchNorm (BN) layer does not necessarily need to be processed during inference for the reduction of computation and memory cost. For most of the convolutional neural networks, BN layers are often after the Conv FC layers. In these situations, BN be merged into the weights and biases of the corresponding Conv or FC layers. However, in (Zouu et al.] (2016); Zhang et al.) (2018)), they use asymmetric or non-linear quantization, causing barriers to BN fusion. There are two ways to overcome this obstacle. One is "BN folded training" (Krishnamoorthi 2018), which adopts BN fusion before weights quantization in every training step: the other is use symmetric linear quantization. However, the first method doubles the training time, while the second one has no additional computational overhead, which will be introduced in Section 3.4
Third, linear quantization is necessary for -arte-rrr- accelerators. There are many non-linear quantization methods which achieve excellent bitwidth reduction efficacy and accuracy tradeoff'.
In these cases, it requires additional transformation to have correct arithmetic results after quantiz- ing the value into non-linear distribution. For example, as Han et al. (2016); Park et al. (2017). it necessitates the operation of table lookup to have correct multiplication between quantized val- ues. However, the linear quantization can make full use of the low-precision arithmetic compo- nents in sffethe-shelff accelerators. Further, linear quantization and asymmetric mode. Asymmetric quantization has one more parameter (e.g., zero-point (Kr-) ishnamoorthi 2018)) than symmetric quantization, and it requires additional subtraction linear- operation before multiplication. As a result, the symmetrical mode is compatible with the main- stream integer accelerator chip design and do not require the redesign of datapath in these hardware.
be divided into symmetric mode
Fourth, different CNNs or applications usually use a variety of activation functions. For instance, the object detection model Redmon et al. (2016) typically uses Leaky ReLU. And the bottleneck of ResNet block does not use any activation function. The quantization methods are expected to be adapted to these situations. However, Zhang et al. (2018a): Park al. (2017) only focus on the quantization of activations after ReLU. In this paper, we demonstrate our method is friendly to different activation methods such as Leaky ReLU.
Some of the previous researches change the network structure for better quantization performance, e.g.. Mishra et al. (2018) double or even triple the convolutional filters to reduce accuracy degrada- tion. For the energy-efficte architecture to hardware and adds to computational and memory access overhead due to the in- creased filters and parameters. As result, keeping the network structure intact is important.
integer neural network chips, it needs to remap the changed network
Concerning all the factors above, in this paper, we present a learned linear symmetric quantiza- tion (LLSQ) method and also evaluate it low-precision neural network accelerator through seddware-softwar co-design. Specifically, our mainly contributions are:
Unlike most of other guantization methods, we quantize the whole network including the first and last layers. We also quantize bias and scaling factors, in support of the low bitwidth integer arithmetic units and accumulators on the accelerator.
We adopt learned linear symmetric quantization schemes which are hardware friendly (such as the convenience of BN fusion implementation) while achieving state-of-the-ard predic- tion accuracy.
We design a specialized low- oow-precision CNN inference accelerator to validate the methodol- ogy, which supports 2/4/8 integer operating and work with high efficiency. We then deploy our quantization model on the accelerator to illustrate the efficacy of the workflow.
MOTIVATION
Edge embedded neural network accelerators generally have three primary design goals- small- footprint, gegh-tthooughuu/owwllteeee and low-power. For different applications and scenarios, the prior researches specialized deep learning processors often falling into different categories: cloud-oriented hardware for warchouse machines, low power mobile processors and ultra-low power accelerators for loT or cyber-physical devices.
3

---

Published as a conference paper at ICLR 2020
For mobile and embedded usage, specialized neural network processors are becoming increasingly popular as an efficient hardware solution of inference. DianNao (Chen et al. 2014) is proposed for fast inference of DNNs and it uses 16-bit fixed-point multipliers for small silicon area and low- energy. Later, Shi (Du et al. 2013) is introduced and it burns extremely low energy con- sumption by putting all weights onto the SRAM to eliminate considerable DRAM accesses. Besides, DeepBurning (Wang ct al. 2016) simplifies the design flow of accelerator for different NN models. Eyeriss (Chen al. 2018 also another representative of low-power accelerators. And it presents a row-stationary (RS) dataflow to minimize data movement energy consumption on a spatial archi- tecture. To further reduce computation overhead, EIE (Han et al.. 2016) exploits the sparsity and low-bit compression of the NNs and achieves better throughput, energy and area efficiency. These typical edge neural network processors are accepting fixed-point data input and using fixed-point processing elements to reduce the power and chip area overhead caused by floating-poit point arithmetic components and memory. For the cloud scenarios, specialized architectures like TPU (Jouppi et al.) 2017) and FPGA-based accelerator cards are also replacing conventional GPGPU and CPU for high- throughput inference tasks. Even for cloud-oriented inference architectures, fixed-point processing architectures like TPU favored because they are able to deliver much higher throughput for the given power budget and silicon area overhead.
However, for the fixed-point or integer hardware targeted neural network acceleration, quantiza- tion is prerequisite to convert the floating-poin network model into the fixed-point format compati- ble with the specialized hardware, and is also a critical step ensure the accuracy of the network after conversion. Many prior quantization methods are intended to reduce the running overhead of networks but ignore the architecture and working mechanism of integer neural network processors, as illustrated in Table 1 and they sometimes face considerable accuracy losses, or performance penalty or even fail to supported on the realistic integer datapath due to the unconsciousness of the underlying hardware. This problem becomes particularly important for the hardware that is de- signed to run low bitwidth networks such as binary, ternary, and 2/4-bit models. For instance, Deep compression and WQ are clustering- baasering-based quantization methods, and they still need high-precision values to represent the weights. bias, and activations. As a result, they are not compatible with the hardware that only supports low-precision computing. LQ-Neess non-linear quantization based on the binary code and basis vector, and it can theoretically calculate the inner products between quantized weights and activations by bitwise operations only. However, it requires intensive modifi- cations to the design of current processors by adding a lot of look-up tables in the datapath. Further, bias and scaling factors are not quantized in PACT and WRPN, resulting in performance penalty when employing additional gigh-precision or float-point ALUs to deal with them. In contrast, our LLSQ is designed ease the model quantization flow for the specialized integer neural network processors by conforming to the constraints specified in Table [ To validate the importance of hawdware-aware quantizer and softwareehardware co-design, we also design a specialized CNN ac- celerator for wearable applications. And the specialized accelerator supports 2/4/8 integer operation and adopts the dataflow of low latency and energy design.
3 NETWORKS WITH LEARNED LINEAR SYMMETRIC QUANTIZATION
In this section, we firstly give the overview of the proposed quantization scheme. Then we detail the scheme including low-precision representation, quantized network training, and the deployment of quantization model on our specialized integer-only CNN accelerator for fast inference.
3.1 OVERVIEW OF LLSQ
Many of the previous researches focus on the quantization-aween aware training in GPU, showing the po-
tential of low-bit quantization on CNNs. Han et al.) (0166); Park et al. (2017); Zhang et al. (2018a) propose non-linear guantization methods but lacks of a detailed description the hardware feasibil- ity. Krishnamoorthi (2018) provides a quantization scheme that quantizes weights and activations into 8-bit integers and poigger-arttmmeiccoonn evident hardware acceleration effects, but does not fully exploit lower prwer-ireiision quantization. Based on the researches, we propose a quantization scheme for sfate-of-ther art specialized accelerators operating on low-precision integers only. Figure shows an overview of the proposed scheme.
implementation on ARM CPUs. The method achieves
4

---

Published as a conference paper at ICLR 2020
Activation Butter
Train W
Nox
Scale Buffer
Bla Bufter
Retranin
24bbit
Mlo tb
pount
Weight Butfer
BN Fusion
Retramiin
Quan of B and
Quantization Schemes
PE Array
Figure 1: An overview LLSQ: using pre-trained weights for fast convergence; Retraining of the network with quantized weights and activations; BN fusion for efficient inference; Quantization of bias and scaling factors; Deployment figure, weights, activations, bias, and scaling factors quantized to low-bit integers. And the bandwidth of accumulator can be set to lower (c.g., 16-bit in our experiments).
the quantized model to our accelerator. As shown this
Compared with prior work, our proposed quantization scheme pays more attention the constraints imposed by real hardware. We use unified learned linear symmetric quantizer to quantize weights and activations. And the quantizer has only one parameter, known as the scaling factor. Linear symmetric quantization consumes little additional resources based on the mainstream integer accel- erator designs while achieving state-of-the-ar aceuracy in various networks. After that, we adopt BN fusion for fast inference hardware. As for bias and scaling factors, also quantize them to low-bitwidth integers. The integer accelerator illustrated in Figure [ an illustrative case of 4-bit quantization and hardware acceleration.
3.2 MAKING FULL USE OF THE PRE- PRE-TRAINED PARAMETERS
In experiments, we find that it is more efficient to start with the pre-trained full-pre cull-precision parameters before guantization. Louizos al. (2019); Zhou al. (2017) pre-trained weights for fast con- vergence and deployment, while Zhang et al. [201 8a); Choi 2018); Cai et al. (2017) train quantized network from scratch show the robustness of the algorithm. However, for some object detection models, the backbone models and pre-trained weights are essential to the detection performance. Redmon et al. (0166) shows that the pre-trained high-resolution classification network gives an in- crease of almost 4% mAP. To have better performance in classification. object detection, and other CNN based tasks, in this paper, we use pre-trained parameters to initialize the networks.
3.3 LOW-PRECISION REPRESENTATION AND QUANTIZATION ALGORITHM
We use channel-wise quantization for Conv layers and layer-wise quantization for FC layers and activations. And we adopt the symmetric linear quantization to quantize weights or activations into bits words(e.g.. 4-bit), which can be defined as
a Quantize |a) x clamp( - - 2
x"
where x" R is one kernel of weights one layer of activations, the variable E R is the quantization parameter, known as the scaling factor, while q E (-2k-1.... ...., 0, 1...., 2 11 is the integer values flowing in the integer accelerator and 2 E (-26-la,- 0, .o..... (2--1_ 1 [a)
the quantized weights or activations. Note that for activations, which are non-negative values the ActFun is ReLU, clamp them [0, 2 1], resulting in q 10, 2 1} and 2 E a..,,..., -2 1)a}, respectively.
As defined above, use a as quantization parameter. And we optimize with:
a' = arg min / p(x" |a0" x²1'
(2)
5

---

Published as a conference paper at ICLR 2020
Table Comparison of SG and EMA.
Method
VGGSmalll444
VGGSmallw2a2
ResNet 8w444
ResNett 8w3a3
EMA
93.95
92.78
69.48
66.80
SG
94.34
93.31
69.84
68.08
VGGSmall is trained on Cifarl0 and ResNet18 is on ImageNet.
where x", ,x are the same factors defined in Equation 1 p(x=") is the probability density distribution of w", and I E {1, 2} is an optional constraint (We use 2 in our experiments). In Figure we present the relationship between quantization error and a. When fixing weights x', we can find the optimal a" by using the brute-force search approach, which induces high computation cost. Besides, the weights are updated during the re-training phase and the optimal value a* changes accordingly. In other words, the optimal value for the factors is not fixed and it takes considerable computational overhead to find the dynamic optimal value.
Inspired by Zhang et al. (2018)), we find through exper- iments that there is no need to find the optimal value a", and it works well enough to find near-optimal value ā'. Generally, quantization can be considered as a regular- ization of the networks, and the quantization parameter a needs only to be adjusted to a near-optimal value to preserve the network capacity.
1.00E
egoch epochhhcchhchc
Quantization Error
1.00E 04
1.002-05
Then the problem becomes how to find ã" in the for- ward pass of the network training phase. At the begin-
1.008-05
ning of training, we assign a an initial value. Then in every training iteration, we explore between 2a and a/2 to find better search direction dbetter E {-1, 1}, and use doetter as the simulated gradient (SG) of a which is detailed in Equation | The gradients of other parameters are still obtained by backpropagation algo- rithm. After that, we update all parameters with the gra- dients or simulated gradients. Another method is updat-
1.00E-07
1/a
Figure 2: L2 distance of quantization. The data is from weights of the first FC layer in AlexNet. As shown in the figure, the optimal a" changes with the updating of weights.
ing by the exponential moving average (EMA). We experiment both of the methods, and the results show that SG is generally better than EMA on various networks (See Table 2). not specifically stated,
the SG method in experiments. The re-training process with weights and activations quantized is summarized Step 1 of Algorithm
3.4 BN LAYER FUSION OF QUANTIZED NETWORKS
As described in Section I merging the BN layers into convolutional layers can reduce the latency of network inference by removing additional computation overhead. The operator of quantized Com and FC layers can be expressed as
Oa Oae
where a, q the while b is the bias and is the output feature vector. Note that aa. Ow and b are full precision values. And the BN layer can be formulated follows:
as Equation 1. aa Ow donate the quantized activations and weights,
(4)
26
where a and a² are EMA statistics, 7 and 3 are learned parameters in BN layers. Obviously, we can merge BN layers and figure out the corrected parameters:
(oww b- (b u)C + +B; (= - +e
(5)
For brevity, only consider the operation of channel.

---

Published as a conference paper at ICLR 2020
3.5 BIAS AND SCALING FACTOR QUANTIZATION FOR LOW-BIT INTEGER ONLY ARITHMETIC
Further, the outputs of layers are quantized according Equation the bias use Og as its scaling factor. And for the multiplier 2,0m we use bit-shift quantization
|
For integer-only-arrthmeeii
do (See Equation T0 so that no multiplication but bit-shift operation needed hardware.
ao = aa Raqadwaa qw +
q. = Caāw
(q + 9b)
(6)
b 1, 12% -1
where clamp(1
Note that Ohn is very small number, resulting in large quantization noise when adopting the clamp operation. addition, the quantization of the scaling factors a can also raise the quantization noise of weights and activations. Parameter re-training summarized in Step 2 of Algorithm [ is
required.
In the re-training phase, we adopt STE (Bengio et al. 2013) to realize the non-differentiable quanti- zation function.
For weights and bias, have
Oy
dy
dy
0y
(7)
Ow
Ow
d6
For activations, we have
8y
By
if S a" s (2 1)a
(8)
da
0
otherwise
4 EXPERIMENTAL RESULTS
In this section, three sets of experiments on Cifar10, ImageNet and Pascal VOC datasets are pre- sented. First, we conduct our proposed learned linear symmetric quantization (LLSQ) on weights and activations, leaving the first and last layers in full precision for fair comparison with Zhang et al. 20) Second, we quantize the whole networks including the first and last layers, which is referred as LLSQF (LLSQ for Full network). Finally, quantize the remaining bias and scaling factors. LLSQ is implemented in PyTorch (Paszke et al. 2017). and most of the baselines it uses in evaluation from PyTorchh Model Zoc
4.1 QUANTIZATION OF WEIGHTS AND ACTIVATIONS
We firstly employ the (GG-Small network on Cifarl0 to verify the LLSQ method. After that, we use AlexNet (Krizhevsky et al. 2012), ResNet18, ResNet34 (He et al., 2016), particularly the light- weight and
network architecture of MobileNet (Howard et al. 2017 | [Sandler et al.) 2018) etc. to conduct more detailed experiments on the ImageNet dataset. Finally, we also quantize YOLOv2 (Redmon & Farhadi 2017) to demonstrate that LLSQ also works well for complicated applications and especially the task adopting the activation functions like Leaky ReLU other than ReLU used in previous work.
VGG-SMALL ON CIFAR10
The VGG-Small architecture is the same with Louizos et al. (2019); Zhang et al. (2018a) consisting of six Comv layers, three MaxPool layers, and one FC layer. We adopt cosine learn rate scheduler to train the VGG-Small reference and the quantized models. Specifically, we train the reference network for 400 epochs using an initial learning rate of 2c-2. And for the training of the quantized network, we use warmup learning rate scheduler in the first ten epochs with initial learning rate of 2e-3. In all quantization experiments, the total training epochs are 100. The VGG-Small quantization results provided in Table 3 With 3-bit weights and 3-bit activations, the accuracy

---

Published as a conference paper at ICLR 2020
Table 3:
Comparison with the state-of-the-ard low-bit quantization methods on CIFAR-IO. The bitwidth for weights(w:), activations(a), bias(b) and scaling factor((a) are given.
Method
Bits wla/bla
Acc(%)
Degradation(
LQ-Nets
Reference
93.8
Zhang et al. 2018a)
3/3
93.8
0.0
2/2
93.5
0.3
RQ
Reference
93.05
Louizos et al.
2019
8/8
93.30
-0.25
4/4
91.57
1,48
2/2
90.92
2.31
LLSQ (ours)
Reference
93.34
4/4
94.34
-1.00
3/3
94.02
-0.68
2/2
93.31
0.03
(LSQ)
4/4
94.30
-0.96
3/3
94.07
-0.73
2/2
93.12
0.22
4/4/8/8
93.84
-0,50
first and last layer in full precision
using our method is better than state-of-the-art method, LQ-Nets. And even when the first and last layers are all quantized in the same way, the loss of accuracy is minimal.
IMAGENET DATASET
We then quantize AlexNet, ResNet18, ResNet34 and MobileNetv2 on ILSVRC2012 dataset with different bitwidth configuration demonstrate the effectiveness of the method. All of the pre- trained float-point weights except MobileNetv2 are downloaded from the PyTorch Model Zoo, and they are trained for 90 epochs with step learning rate scheduler. After loading the praitre weights, we employ a warmup learning scheduler in the first three epochs and the cosine scheduler the remained 57 epochs with an initial learning rate of 2e-2.
As shown in Figure 3a when quantizing both weights and activations, our degradation of accuracy significantly smaller than LQ-Nets, PACT, and RQ. Especially, LLSQ outperforms the baselines when quantizing weights and activations into 4-bit. And it also outperforms other non-linear quan- tization methods with different bitwidth. Figure 3b shows that even with the first and last layers quantized, it can still achieve near baseline performance. In overall, the accuracy drop is less than 0.4% in ResNet18, ResNet34, and AlexNet when quantizing the whole network. We also quantize MobileNetv2, more compact network, and obtain results that significantly better than RQ. Please check T detailed results.
OBJECT DETECTION ON PASCAL VOC
We also apply the proposed LLSQ to YOLOv2. The backbone of YOLOv2 is Darknet-19. and its activation function is Leaky RELU, so that the activations contain negative values. For YOLOv2 on Pascal VOC, we adopt the same quantization configuration (See Section 3.3 of the weights to the activations. Results listed in Table 4 As shown in the table, LLSQ induces minor losses of mAP in different bitwidth presentation. For comparison, we also quantize the activations into signed 5-bit integers using PACT, and consequently face considerable mAP losses (54.8mAP). Please note that we use the open- spen-source Py Torch implementation of YOLOv2 as the baseline. We train the quantized model for 170 epochs (2/3 of baseline) with an initial learning rate of le-4 (1/10 of baseline).
8

---

Published as a conference paper at ICLR 2020
11.00 LLSOF
ST
LO-Nets
75
LLSOF
2.0
Accuracy (%)
1.5
Degratation
1.0
60
50
-0.5
3/3
ResNet18
ResNet84
AlexNet
-1
(b) Comparison LLSQ LLSQE. LLSQ outper- forms the baseline, and LLSQF receives only minor (a) Comparison with other state-of-art methods. performance penalty after quantizing the first and the
414
44
3/3
4/4
2/2
Lower better.
last layers.
Figure 3: Quantization results on different networks.
Table 4: LLSQ on YOLOv2 detector.
bitwidth
mAP
acto
bike
hird
bool
bottle
bus
car
chair
DOW
table
dog
borse
mbike
persun plant
sheep
sola
Irain
IV
76.1
81.2
w3ta5
71.2
75.5
75.9
71.4
60.4
42.4
80.6
8000
83.3
53.5
75.8
68.1
708
82.6
79.5
71.6
45.5
69.9
72.1
84.6
70.4
w468
73.4
74.5
79.1
75.5
60.6
43.8
80.9
80.7
85.8
56.6
80.0
70.9
83.5
84.5
81.0
74.5
47.5
74.8
75.2
84.1
73.6
w4632
74.2
4.2
BN FUSION AND QUANTIZATION BIAS AND THE SCALING FACTOR
We adopt BN fusion in the PostAct (Conv >N ReLUU) networks according the formula in Sec-
tion 3.4 And the scaling factor of bias the product of the corresponding scaling factors belonging to the activations and the weights, respectively. After that, visualize the bias value distribution of VGG-Small. Figure 4 shows b/ as is distributed between a vast range (-1000, 1000), resulting in overflows of low bitwidth accumulators. And the overflow phenomena have significantly harmful impact on the network performance. To deal with this issue, we quantize the bias and the scaling factors 8-bit words, and then fine-tune the networks restore the original performance. Gener- ally, we need fine-tuning for one epoch only. After the quantization of bias and scaling factor, we have fully quantized model and have it deployed onto our integer-only accelerator with 16 bitwidth accumulators. Table andd ( show that the accuracy loss is negligible with 44448 quantization on both VGG-Small and AlexNet.
0,0020
densily
densty
0.0010
0.0005
0.0000
bylos
by/as
(a) bias of convl
(b) bias of conv2
Figure 4: Distribution of the bias/scaling factor. The data is from VGG-Small with w4a4 quantiza- tion.
4.3 DEPLOYMENT ONTO REALISTIC HARDWARE
The introduced linear symmetric quantization intended to deploy the quantized networks to spe- cialized integer-only aritheiieee CNN accelerators other integer-only hardware. Our accelerators adopt the typical 2D systolic array architecture (Chen et al. 2018), but they are featured with 4-bit or 2-bit low-precision operation. As shown in Figure 1 the 8/4/2-bit accelerator has 32x7 array of

---

Published as a conference paper at ICLR 2020
Table 5: Comparison of our low-precision integer Neural Network Processors.
Bitwidth
#MAC Unit
Throughput (GOps/sec)
Silicon Area
Power (mW)
(mm²)
8-bit
224
179.2
4.71
228
4-bit
224
179.2
2.80
93
2-bit
224
179.2
1.84
41
Implemented and synthesized with Synopsys Design Compiler (DC) under the 40nm technology.
processing elements (PE). And the MAC unit in each PE consists of 4-bit multiplier and 16-bit accumulator. For the 4-bit accelerator, we use INT4 representation for weights, UINT4 for activa- tions, INT8 for the bias and scaling factors, respectively. For the 2-bit accelerator, we use INT2 for weights, UINT2 for activations, INT8 for the bias and scaling factors, respectively. Through the quantization process described in the paper, we can have a fully quantized network that works directly on the CNN accelerator. In addition, as we use linear symmetric quantization, we can use operations without introducing shifters or
trriiht-ffowwad way conduct multiply-accumulle lookup tables, which means the quantized models can run on state-of-the-ee integer accelerators and ensures that their output accuracy degradation minimal as presented the above sections. Fi- nally, we implement the 8/4/2-bit integer neural network processors with Synopsys Design Compiler (DC) under the 40nm technology. clocked at 800MHz. Table S shows that the 4/2-bit implemen- tation achieves up to 2.56x lower silicon area and 5.56x lower power compared to that of the 8-bit baseline.
5 CONCLUSIONS
In this paper, we introduced a learned linear symmetric quantization (LLSQ) to quantize the whole network including the bias and sealing factors. We also use BN fusion and low bitwidth accumulators to reduce the network inference overhead and the hardware resources in integer neural accelerators. We show that our proposed method performs well for various networks on Cifar 10, ImageNet, and Pascal VOC datasets. We also show that even the linear symmetric quantizer can obtain better results than asymmetric or non-linear quantization in the case of 4-bit networks. Finally, we deploy the quantized network onto specialized integer-only neural network accelerator. Currently, the bitwidth every layer in a network is all the same. Prior researches empirically find that different layers have different sensitivity bitwidth of quantization. Hence in the future, will explore framework to support more flexible bitwidth for different layers or finer-grained quantization.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 61874124 and Grant 61902375.
REFERENCES
Jorge Albericio, Alberto Delmas, Patrick Judd, Sayeh Sharify, Gerard O' O'Leary, Roman Genov, and Andreas Moshovos. Bit-pragmatic deep neural network computing. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2017, Cambridge, MA, USA, October 14-18, 2017, pp. 382-394 2017. doi: 02.1445/11239393112999 URL
https://doi.com org/10..0gg/..... 1141111511/9 //////12399999 9139999999999229999 3123982
Yoshua Bengio, Nicholas Leonard, and Aaron C Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv: Learning, 2013.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by
half-wave gaussian quantization. In CVPR, 2017.
Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam.
Diannao: small-footprin high-thre thghthhouughput accelerator for ubiquitous machine-leariing archi- fectural support for programming languages and operating systems, 49(4):269-29) 2244, 2014.
10

---

Published as a conference paper at ICLR 2020
Y. Chen, J. Emer, and V. Sze. Eyeriss: A spatial architecture for energy- dataflow for convolutional neural networks. IEEE Micro, pp. 1-1, 2018. ISSN 0272-1732. doi: 10.1109/MM. 2017.265085944.
Jungwook Choi. Pact: Parameterized clipping activation for quantized neural networks. arXiv: Computer Vision and Pattern Recognition, 2018.
Zidong Du, Robert Fasthuber, Tianshi Chen, Paolo lenne, Ling Fei Li, Tao Luo, Xiaobing Feng, Yunji Chen, and Olivier Temam. Shidiannao: shifting vision processing closer to the sensor. international symposium on computer architecture, 43(3):22-104, 2015.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
Journal of Machine Learning Research, 20(55):1-21, 2018.
Ross B Girshick. Fast r-cnn. international conference on computer vision, pp. 1440-1448, 2015.
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. Dally. Eie: Efficient inference engine on compressed deep neural network. 2016 ACM/IEE 43rd Anmual International Sym- posium Computer Architecture (ISCA), pp. 243-254, June 2016. doi: 10.1109/SCA.2211..330
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. neural information processing systems, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. international conference on learning represenlations, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B Girshick. Mask r-cnn. international 2017.
conference on computer vision, pp.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017.
Norman P Jouppi, C S Young, Nishant Patil, David A Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh K Bhatia, Nan Boden, AI Borchers, al. In-datacenter performance analysis of a tensor processing unit. international symposium on computer architecture, 45(2):1-12,2017.
Patrick Judd, Jorge Albericio, Tayler H Hetherington, Tor M Aamodt, and Andreas Moshovos.
Stripes: bit-serial deep neural network computing. international symposium on microarchitctuure, pp. 1-12, 2016.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 1rXiv:1806.000 2018
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. neural information processing systems, 141(5):1097-1105, 2012.
Jinmook Lee, Changhyeon Kim, Sang Hoon Kang, Dongjoo Shin, Sangycob Kim, and Hoijun Yoo. Unpu: A 50.6 50.6tops/w unified deep neural network accelerator with 1b-to-16b fully-variable weight bit-precision. pp. 218 22220, 2018
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. computer vision and pattern recognition, pp. 3431-3440, 2015.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. international conference on learning repre- sentations, 2019.
Asit K Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: Wide reduced- preccssion networks. international conference learning representations, 2018
11

---

Published as a conference paper at ICLR 2020
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. international conference on learning represen- tations, 2017.
Ali Bou Nassif, Ismail Shahin, Imtinan Attili, Mohammad Azzeh, and Khaled Shaalan. Speech recognition using deep neural networks: A systematic review. IEEE Access, 7:19143-19165, 2019.
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for seman- tic segmentation. international conference computer vision, pp. 1520-1528, 2015.
Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted-enttroy--basse quantization for deep neural networks. pp. 7197-7205, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. pp. 6517-6525, 2017.
Joseph Redmon, Santosh Kumar Divvala. Ross B Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. computer vision and pattern recognition, pp. 779-788, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. international conference on learning representations, 2015.
Y. Wang, Xu, Y. Han, Li, and X. Li. Deepburning: Automatic generation of fpga-based learning accelerators for the neural network family. In 2016 6 53nd ACM/EDAC//EEE Design Automation Conference (DAC), pp. 1-6, June 2016. doi: 10.1 2889000..
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. neural information processing systems, pp. 2074-2082. 2016.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. european conference on computer vision, pp. 373-390, 2018a.
Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. Asynchronous bidirectional decoding for neural machine translation. national conference artificial intelli- gence, pp. 5698-5705, 2018b.
Aojun Zhou, Anbang Yao, Yiwen Guo. Lin Xu, and Yurong Chen. Incremental network quanti- zation: Towards lossless cnns with low-issin weights. CoRR, abs/1702.03044, 2017. URL
org/
/ abs 002 03044
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv: Neural and Evolutionary Computing, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained temary quantization. interna- tional conference on learning representations, 2017.
APPENDIX
IMAGENET DETAILED
We conduct experiments AlexNet, ResNet18, ResNer34, and MobileNetv2. For all of the periments, we adopt channel-wise quantization for Com layers and layer-wise quantization for FC layers well the activations. The AlexNet architecture the same as the Pyrooc Model Zoo,
12

---

Published as a conference paper at ICLR 2020
Table 6: Train Time of ResNet18
#Training Process
training time
Train the fp32 network from scratch
1.0x
Quantize w/a to 4/4 according to Stepl of Alg.
0.69x
Quantize w/ a 3/3 according to Stepl of Alg.
0..9x
Quantize b/a a to 8/8 according to Step2 of Alg.
0.01x
and it consists of five Conv layers, three FC layers, three MaxPool layers, and two Dropout layers. To prevent over-fitting, keep the Dropout layers when quantizing AlexNet. As shown in Figure 5d we use the same learning rate scheduler for all experiments on ImageNet. The test curves are also shown in Figure As begin with the pre-trained full-precision weights, the test accuracy is already acceptable after one-epoch training. The final results are listed in Table
Training time. The proposed LLSQ requires about 2/3 training epochs than that of floating-poin point network training. In each training iteration, LLSQ needs extra computation cost to optimize the quantizers. Specifically, the simulated gradients generation of the scaling factors is the major cost. Table 6 shows the total training time comparison of ResNet18 network. The quantization training time is 70% of baseline only.
(a) AlexNet
(b) ResNet18
6080
(c) ResNet34
(d) learning rate.
Figure 5: Test curves for AlexNet, ResNet18, and ResNet34 on ImageNet.
THE LLSQ ALGORITHM
Generate simulate gradients for a:
Fm Σ (xl quantize a))²
E₁ Σ (xc quantizey O 2 ))2
E, Σ (xe quantizes(a)) 2a))²
dbetter arg min([Er Em E,])
AG. -a² a dhetter
13

---

Published as a conference paper at ICLR 2020
where x' is kernel of weights layer of activations. arg min Em: Er]) E {0, selects the index of the smallest number in the array [Et, Em: Er].
The bit-shift quantization can be formulated as:
a
(Q clamppround 22eode a), -2)--2611 2 1)
(10)
29code
where (en(a(()) is the scaling factors be quantized, & E Z is the bitwidth, and goode Z is the parameter of the bit-shift quantizer simply obtained by:
gcode = k ceil(log, + - 10 10
(11)
Algorithm LLSQ
Input: Dataset (æ, y). where is input and y is label; Pre-trained full full-precision parameters (w, b), where w is weights and is bias; Suppose the network consists of I. layers, un represents the ith kernel of weights of the 4eh layer while is the output of the 4n layer.
Output: The quantized scaling factors:
á [[a" (0) ................................ W6
[6"
0 [oa.......
09
The quantized weights w and bias be.
Step Quantize weights and activations and Re-training
// Re-training of quantized networks can converge faster and end with higher accuracy due to the mechanism of BN layers.
repeat
Forward: ag input
for / I.... L do
for w. in wi do // This is accelerated in parallel when implemented.
w (0
Quantize(w) (o, per Eq.
Generate simulated gradients for a un(") per Eq. (9)
end for
w < Concat wi (6)g
Og < (BN(UUBBCCoooll
w., b:)))
a + Quantize(a: (aa)) per Eq.
Generate simulated gradients for aag per Eq. 9
end for
Backward:
Generate AG for weights and bias and AE for activation per Eq. backpropagation algorithm.
8
and
Update uz b, O. Caa
äter +- iter +
len dataset)
until iter 2 itermax // need about 60 epochs, e.g. iter max
60
batcheize
Step 2: Quantize bias and scaling factor after BN fusion and Re-training
iter <-
Φ.. b + BN fusion per Eq. 5
repeat
Forward:
ag < input
for 1 I,... do
6g x- SQ(( per Eq. 10
for w in wy do This is accelerated in parallel when implemented.
(0) Quantize(w;," (6) 6a (n)))) per Eq.
wi
w
14

---

Published as a conference paper at ICLR 2020
0 Q (
0g 6a (2) per Eq.
Generate simulated gradients for à w/ per Eq.
end for
(6)g wo - Concat wi
by + Concat 6/tha
a + ReLU(Conv w/. b)))
0 + SQ per Eq. 110)
a < Quantize(a; oa per Eq.
Generate simulated gradients for Oa, per Eq. (9
end for
Backward:
Generate AG for weights and bias and AE for activation per Eq. backpropagation algorithm.
and
Update w, b, G.w Otg
iter iter +1
(ettostt
until iter 2 iieemaa only need one epoch, e.g. itermar
batchsize
15

---

Published as a conference paper at ICLR 2020
Table 7: Comparison with state-of-the-art quantization methods on ImageNet. Top1, Top5 accu- racy(%) and degradation of Top1 are given.
Method
Model
# Bits
Top1(%) Top5(%)
Degradation (%)
wla/bla
LQ-Nets"
ResNet18
Reference
70.3
89.5
Zhang et al. 2018a)
4/4
69.3
88.8
1.0
3/3
68.2
87.9
2.1
RQ
ResNet18
Reference
69.54
89.19
Louizos
2019
8/8
69.97
89.44
-0.43
4/4
61.52
83.99
8.02
RQ+ST
8/8
69.63
89.33
-0.09
Louizos al.
2019
4/4
62,46
84.78
7.08
LLSQ(ours)
ResNet18
Reference
69.76
89.08
4/4
69.84
89.14
-0.08
3/3
68.08
88.20
1.68
LLSQF(ours)
ResNet18
4/4
69,40
88.72
0.36
3/3
66.67
87.42
3.09
LO-Nets Zhang et al.
ResNet34
Reference
73.80
91,40
2018a)
3/3
71.9
90.2
1.9
LLSQ(our)
ResNet34
Reference
73.30
91.42
4/4
73.60
91.28
-0.30
3/3
72.02
90.66
1.28
L.LSQF(our))
ResNet34
4/4
72.94
91.20
0.36
3/3
70.97
89.95
2.33
LLSQ(ours)
AlexNet
Reference
56.55
79.09
4/4
56.57
79.02
-0.02
4/4/8/8
56.45
80.15
0.10
3/3
55.36
78.20
0.19
LLSQF(ours)
AlexNet
4/4
56.40
78.85
0.15
4/4/8/8
55.58
77.47
0.97
3/3
54.28
77.65
2.27
RQ
Mobilenet(v1)
Reference
70.61
89.47
Louizos al. 2019
8/8
70.43
89.42
0.18
6/6
68.02
88.00
2.59
5/5
61.38
83.73
9.23
RO+ST
Mobilenet(v1)
8/8
70.06
89.52
0.55
Louizos et al.
2019
6/6
67.62
87.78
2.99
5/5
56.85
80.35
13.76
LLSQ(ours)
MobileNet(v22)
Reference
71.80
90.37
6/6
71.20
89.99
0.60
5/5
70.45
89.69
1.35
4/4
67.37
87.99
4.43
First and last layers full precision.
16


Reviews:
[{"review_id": "ryxfBzICtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes to train a linear symmetric quantizing function for integer processors.\n\nThe proposed idea makes sense at a high level, and the empirical results look somewhat compelling, but the write-up is not particularly clear (see clarity-related comments). I found it hard to extract a complete picture of how the proposed approach operates: from the leftmost diagram in Figure 1 I can infer what high-level steps are involved, but I wouldn’t know how to re-implement the approach from the textual description itself.\n\nClarity-related comments:\n\n- The submission never explicitly states what x^r and p(x^r) correspond to in the network. My understanding from the context is that x^r is the scalar value of a model parameter or activation, and that p(x^r) is the empirical distribution over all model parameters and activations. Can the authors clarify?\n- Some kind of pseudo-code for the re-training of the weights and activations (Section 3.3) would help clear up reader confusion. At this point, are only the weights and activations quantized in the network? How is backpropagation handled (and if the straight-through estimator is used, why is it only mentioned in Section 3.5)? How do parameter and alpha updates interleave?\n- How many alpha values are there? Section 3.3 gives the impression that there is a single quantization parameter shared by all parameters and activations, but Equation 3 uses different alpha values for the weights and activations.\n- Section 3.4 is difficult to parse due to bad notation. If the output of the convolution layer is fed into the batch normalization layer, it would be clearer to reuse symbols (i.e. change “x” to “o” in Equation 4).\n\nAdditional comments:\n\n- Organization of the different sections could be improved. Related work discussion is scattered throughout three different sections, namely Introduction, Motivation, and Related Work.\n- Writing quality could be improved  (e.g., “*Except that*, some designs rely [...]”, “Edge or embedded neural network accelerators *are generally having* three primary design goals [...]”) but has a relatively small negative impact on readability.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJezXLYC5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the quantization of ConvNets. This paper proposes a learned linear symmetric quantizer to reduce the precision of weight, bias, and activation. The proposed approach works as the following: for a pre-trained neural network, it computes the new weight and activation as a product of a quantized value with a scaling factor. The quantization is based on a simple linear, symmetric function as in equation (1). The value of the scaling factor is searched by \"simulated gradient\" or exponential moving average during re-training. Next, batch normalization is fused into convolution, and the scaling factor and biases are re-calculated. Last, the scaling factor on the convolution is merged with bias terms, to remove the need for multiplication in hardware implementation. Since bias terms usually have a much larger dynamic range, higher precision is used to represent biases. Experiments show that the method achieves competitive results compared with previous quantization methods, and the quantized models can be deployed on hardware more easily. \n\nThe contribution of the paper, in my opinion, is to show that using simple methods without many bells and whistles, we can achieve competitive quantization performance.  And when performing quantization, it is important to consider the hardware implementation. Details on how to deal with scaling factors, how to deal with biases, and so on, can have significant influences on the overall performance. \n\nHowever, the main concern of the paper is that the methods adopted in the paper are too plain. The paper successfully integrated previous methods but did not propose new ideas that inspire future research. As a result, I would not recommend acceptance for publication. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByxRzN_RKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a linear symmetric quantizer for integer accelerators called LLSQ, which learns the quantization scaling factor using simulated gradient as update policy. Their main contribution is enabling inference on integer-only hardware by covering all parameters of all operators in convolutional networks, including weight, bias, activation and scaling factor. To address the quantization noise issue in bias parameters, they adopt Straight-Through Estimator and fine-tune the parameters after quantization. To improve inference efficiency, they apply BN layer fusion. They conduct experiments on public datasets for image classification and object detection to conclude that LLSQ achieves lower accuracy degradation compared to previous work. Finally, they test the quantized model on a specialized integer accelerator, showing the feasibility of the quantization on real hardware.\n\nIn conclusion, this paper generalizes linear symmetric quantization to all parameters in order to deploy the network on specialized integer neural network processors for efficient inference. In terms of algorithmic contribution, this paper introduces the scaling factor as a learnable parameter of the quantizer, but lacks enough theoretical justification. Therefore, I would consider weakly accepting the paper.\n\nFor the algorithm, the following should be addressed.\n1.\tIn scaling factor updating policy, the simulated gradient performs better compared to EMA. However, the motivation for choosing this policy is unclear, and there is no mathematical derivation of the gradient value.\n2.\tIn this study, pretrained network weights in full precision are used, thus the quantization procedure should take fixed weights as inputs. However, in section 3.3, the quantization scaling factor is updated in the training phase of the network, with weights being updated at the same time This seems to be a contradiction, and the experiment results where quantizer is trained with network weights fixed should also be presented.\n\nFor the experiment, the following should be addressed.\n1.\tThe training time for different bit-width settings should be included in the results.\n2.\tThe reason for leaving the first and last layers in full precision is unclear, and doing so may go against the objective of deploying the quantized model on specialized integer hardware according to the paper.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the question of how to quantize deep neural networks, for processors operating on low-precision integers.  The authors propose a methodology and have evaluated it thoroughly. The reviewers all agree that this question is important in practice, though there was disagreement about how novel a contribution this paper is specifically, and on its clarity. The clarity questions were resolved on rebuttal, so I lean to accepting the paper.
Decision: Accept (Poster)