Linear Symmetric Quantization of Neural Networks for Low-precision Integer Hardware

1 INTRODUCTION . Deep neural networks have shown excellent performance on various computer vision and natural language processing tasks , such as classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ) , object detection ( Girshick , 2015 ; Redmon et al. , 2016 ; He et al. , 2017 ) , segmentation ( Long et al. , 2015 ; Noh et al. , 2015 ) , machine translation ( Zhang et al. , 2018b ) , speech recognition ( Nassif et al. , 2019 ) , etc . While the past few years witnessed the success of DNNs on cloud and server-end computers , neural networks have been recently pushed to embedded and mobile areas to enable edge intelligence . For these scenarios , the power provision and computational strength on the edge computing devices are limited . As a result , it is essential to have more efficient network architectures and less expensive inference overhead . Therefore , there is increasing attention from the research community to study the compression of modern deep neural networks that are typically over-parameterized and computationally costly . Several categories of approaches are proposed to decrease the computational overhead of neural networks , such as lightweight neural network architectures ( Howard et al. , 2017 ) , neural architecture search ( NAS ) ( Elsken et al. , 2018 ) , and network pruning ( Han et al. , 2015 ; 2016 ; Wen et al. , 2016 ; ∗Corresponding author Molchanov et al. , 2017 ) . Besides these techniques , quantizing high-precision floating-point networks to lower bitwidth representation can also drastically decrease both the static parameters and the intermediate data generated during the network inference , resulting in reduced memory footprint and also computational intensity . And this paper focuses on the quantization of neural networks . Quantization technique is also closely related to the implementation of specialized hardware that maps the procedure of network inference onto the energy-efficient low-precision integer or fixedpoint arithmetic circuits . In the hardware perspective , low-precision integer accelerators or processors are dominating the solutions targeted on neural network inference , especially for mobile and embedded scenarios . Google ’ s Tensor Processing Unit 1.0 ( TPU ) ( Jouppi et al. , 2017 ) , Unified Deep Neural Network Accelerator ( UNPU ) ( Lee et al. , 2018 ) , Eyeriss ( Chen et al. , 2018 ) , Stripes ( Judd et al. , 2016 ) , Pragmatic ( Albericio et al. , 2017 ) and many other newly proposed hardware implementations are generally reliant on the effectiveness of the underlying quantization techniques , which are especially crucial for the low-precision integer hardware designed to process binary , ternary , 4-bit or 8-bit networks . In other words , quantization is not only a method to reduce the memory footprint as in traditional work , but also a mandatory step to make the network deployable on integer hardware . Though there is a lot of prior work that investigates low-precision quantization , they mainly target on reducing the memory overhead caused by floating or high precision data representation in the networks , but not focus on specialized integer hardware for network inference . To enable the neural network processors to work with low-precision integer operands and minimize the accuracy losses , a good network quantizer must satisfy the constraints as enlisted in Table 1 . First , all the parameters , including weights , bias , activations , partial results that eventually accumulate to an activation , and even the scaling factors , which are indispensable for low-precision networks like binary and ternary representation , must be quantized into low bitwidth integers as required by the underlying specialized hardware . In some prior work ( Zhou et al. , 2016 ; Zhu et al. , 2017 ; Zhang et al. , 2018a ; Mishra et al. , 2018 ; Choi , 2018 ) , they either leave bias and scaling factors unquantized or keep the first and last layer in full or high precision . Besides , some designs rely on high-precision internal register or ALUs to support high-precision partial results that are generated during computation before the final output of activations or features . For example , Krishnamoorthi ( 2018 ) , which quantizes the weights and activations to 8-bit , directly use 32-bit accumulators to cache the intermediate values or partial results to avoid overflows . However , for 4-bit and lower bitwidth , the integer accelerators can not afford high bitwidth accumulators , which indicates higher silicon area and power cost . For integer-only-arithmetic , we quantize the bias to fixed-point numbers by using a straight-forward method . The value range of these numbers is wide , resulting in overflows of the low bitwidth accumulators . To overcome this problem , we quantize the bias to 8-bit and finetune the bias of the model . As shown in Figure 1 , the bitwidth of accumulators can be reduced to 16-bit . Second , the BatchNorm ( BN ) layer does not necessarily need to be processed during inference for the reduction of computation and memory cost . For most of the convolutional neural networks , BN layers are often after the Conv or FC layers . In these situations , BN can be merged into the weights and biases of the corresponding Conv or FC layers . However , in Zhou et al . ( 2016 ) ; Zhang et al . ( 2018a ) , they use asymmetric or non-linear quantization , causing barriers to BN fusion . There are two ways to overcome this obstacle . One is “ BN folded training ” ( Krishnamoorthi , 2018 ) , which adopts BN fusion before weights quantization in every training step ; the other is to use symmetric linear quantization . However , the first method doubles the training time , while the second one has no additional computational overhead , which will be introduced in Section 3.4 . Third , linear quantization is necessary for state-of-the-art accelerators . There are many non-linear quantization methods which achieve excellent bitwidth reduction efficacy and accuracy tradeoffs . In these cases , it requires additional transformation to have correct arithmetic results after quantizing the value into non-linear distribution . For example , as in Han et al . ( 2016 ) ; Park et al . ( 2017 ) , it necessitates the operation of table lookup to have correct multiplication between quantized values . However , the linear quantization can make full use of the low-precision arithmetic components in off-the-shelf accelerators . Further , linear quantization can be divided into symmetric mode and asymmetric mode . Asymmetric quantization has one more parameter ( e.g. , zero-point ( Krishnamoorthi , 2018 ) ) than symmetric quantization , and it requires additional subtraction or linearoperation before multiplication . As a result , the symmetrical mode is compatible with the mainstream integer accelerator chip design and do not require the redesign of datapath in these hardware . Fourth , different CNNs or applications usually use a variety of activation functions . For instance , the object detection model Redmon et al . ( 2016 ) typically uses Leaky ReLU . And the bottleneck of ResNet block does not use any activation function . The quantization methods are expected to be adapted to these situations . However , Zhang et al . ( 2018a ) ; Park et al . ( 2017 ) only focus on the quantization of activations after ReLU . In this paper , we demonstrate our method is friendly to different activation methods such as Leaky ReLU . Some of the previous researches change the network structure for better quantization performance , e.g. , Mishra et al . ( 2018 ) double or even triple the convolutional filters to reduce accuracy degradation . For the energy-efficient integer neural network chips , it needs to remap the changed network architecture to hardware and adds to computational and memory access overhead due to the increased filters and parameters . As a result , keeping the network structure intact is important . Concerning all the factors above , in this paper , we present a learned linear symmetric quantization ( LLSQ ) method and also evaluate it on a low-precision neural network accelerator through hardware-software co-design . Specifically , our mainly contributions are : • Unlike most of other quantization methods , we quantize the whole network including the first and last layers . We also quantize bias and scaling factors , in support of the low bitwidth integer arithmetic units and accumulators on the accelerator . • We adopt learned linear symmetric quantization schemes which are hardware friendly ( such as the convenience of BN fusion implementation ) while achieving state-of-the-art prediction accuracy . • We design a specialized low-precision CNN inference accelerator to validate the methodology , which supports 2/4/8 integer operating and work with high efficiency . We then deploy our quantization model on the accelerator to illustrate the efficacy of the workflow . 2 MOTIVATION . Edge or embedded neural network accelerators generally have three primary design goals— smallfootprint , high-throughput/low-latency , and low-power . For different applications and scenarios , the prior researches on specialized deep learning processors are often falling into different categories : cloud-oriented hardware for warehouse machines , low power mobile processors and ultra-low power accelerators for IoT or cyber-physical devices . For mobile and embedded usage , specialized neural network processors are becoming increasingly popular as an efficient hardware solution of inference . DianNao ( Chen et al. , 2014 ) is proposed for fast inference of DNNs and it uses 16-bit fixed-point multipliers for small silicon area and lowenergy . Later , ShiDianNao ( Du et al. , 2015 ) is introduced and it burns extremely low energy consumption by putting all weights onto the SRAM to eliminate considerable DRAM accesses . Besides , DeepBurning ( Wang et al. , 2016 ) simplifies the design flow of accelerator for different NN models . Eyeriss ( Chen et al. , 2018 ) is also another representative of low-power accelerators . And it presents a row-stationary ( RS ) dataflow to minimize data movement energy consumption on a spatial architecture . To further reduce computation overhead , EIE ( Han et al. , 2016 ) exploits the sparsity and low-bit compression of the NNs and achieves better throughput , energy and area efficiency . These typical edge neural network processors are accepting fixed-point data input and using fixed-point processing elements to reduce the power and chip area overhead caused by floating-point arithmetic components and memory . For the cloud scenarios , specialized architectures like TPU ( Jouppi et al. , 2017 ) and FPGA-based accelerator cards are also replacing conventional GPGPU and CPU for highthroughput inference tasks . Even for cloud-oriented inference architectures , fixed-point processing architectures like TPU are favored because they are able to deliver much higher throughput for the given power budget and silicon area overhead . However , for the fixed-point or integer hardware targeted on neural network acceleration , quantization is prerequisite to convert the floating-point network model into the fixed-point format compatible with the specialized hardware , and it is also a critical step to ensure the accuracy of the network after conversion . Many prior quantization methods are intended to reduce the running overhead of networks but ignore the architecture and working mechanism of integer neural network processors , as illustrated in Table 1 , and they sometimes face considerable accuracy losses , or performance penalty or even fail to be supported on the realistic integer datapath due to the unconsciousness of the underlying hardware . This problem becomes particularly important for the hardware that is designed to run low bitwidth networks such as binary , ternary , and 2/4-bit models . For instance , Deep compression and WQ are clustering-based quantization methods , and they still need high-precision values to represent the weights , bias , and activations . As a result , they are not compatible with the hardware that only supports low-precision computing . LQ-Nets uses non-linear quantization based on the binary code and basis vector , and it can theoretically calculate the inner products between quantized weights and activations by bitwise operations only . However , it requires intensive modifications to the design of current processors by adding a lot of look-up tables in the datapath . Further , bias and scaling factors are not quantized in PACT and WRPN , resulting in performance penalty when employing additional high-precision or float-point ALUs to deal with them . In contrast , our LLSQ is designed to ease the model quantization flow for the specialized integer neural network processors by conforming to the constraints specified in Table 1 . To validate the importance of hardware-aware quantizer and software/hardware co-design , we also design a specialized CNN accelerator for wearable applications . And the specialized accelerator supports 2/4/8 integer operation and adopts the dataflow of low latency and energy design .

Summary: This paper proposes a linear symmetric quantizer for integer accelerators called LLSQ, which learns the quantization scaling factor using simulated gradient as update policy. Their main contribution is enabling inference on integer-only hardware by covering all parameters of all operators in convolutional networks, including weight, bias, activation and scaling factor. To address the quantization noise issue in bias parameters, they adopt Straight-Through Estimator and fine-tune the parameters after quantization. To improve inference efficiency, they apply BN layer fusion. They conduct experiments on public datasets for image classification and object detection to conclude that LLSQ achieves lower accuracy degradation compared to previous work. Finally, they test the quantized model on a specialized integer accelerator, showing the feasibility of the quantization on real hardware.