HIGHLY EFFICIENT 8-BIT LOW PRECISION INFERENCE OF CONVOLUTIONAL NEURAL NETWORKS

1 INTRODUCTION . While convolutional neural networks ( CNN ) shows state-of-the-art ( SOTA ) accuracy for wide range of computation vision tasks , it still faces challenges during industrial deployment due to its high computational complexity of inference . Low precision is one of the key techniques being actively studied recently to conquer the problem Vanhoucke et al . ( 2011 ) ; Hwang & Sung ( 2014 ) ; Rastegari et al . ( 2016 ) ; Miyashita et al . ( 2016 ) ; Mellempudi et al . ( 2017 ) . With hardware acceleration support , low precision inference can compute more operations per second , reduce the memory access pressure and better utilize the cache , and deliver higher throughput and lower latency . Convolution is the primary operation in CNN models and it is a common practice to enable 8-bit low precision ( INT8 ) inference for convolution in deep learning frameworks ( e.g. , TensorFlow , MXNet , and TensorRT ) . To make it work , convolution utilizes INT8 computation , which requires two scale factors for activation and weight , respectively . It is workable for standard convolution with single group and two groups Krizhevsky et al . ( 2012 ) . However , it does not work well for convolution with large groups , especially for depthwise convolution Howard et al . ( 2017 ) ; Chollet ( 2016 ) . In addition to direct convolution , it is worthwhile to explore INT8 Winograd convolution Lavin & Gray ( 2016 ) for better performance , which is absent in previous research 2 . Although recent work have demonstrated INT8 inference with minimal accuracy loss across various models Vanhoucke et al . ( 2011 ) ; Gysel ( 2016 ) ; Wu et al . ( 2016 ) ; Jacob et al . ( 2017 ) , INT8 inference is limited due to more complex topology primarily introduced by sum operation in residual block He et al . ( 2016 ) and concatenation operation in inception block Szegedy et al . ( 2015 ; 2016 ) . Existing solutions need to convert the convolution output from INT8 to FP32 , and apply the sum or concatenation operation on FP32 . The sacrifice of memory bandwidth and frequent data conversion lead to considerable performance overhead and therefore limit the real deployment . Moreover , there is no systematical study of INT8 inference on various use cases , including image classification Krizhevsky et al . ( 2012 ) ; Simonyan & Zisserman ( 2014 ) ; Szegedy et al . ( 2015 ; 2016 ) ; He et al . ( 2016 ) , object detection Ren 1 % denotes percentage point for simplicity in the context of accuracy loss 2http : //nvdla.org/primer.html et al . ( 2015 ) ; Dai et al . ( 2016 ) ; Liu et al . ( 2016 ) , image segmentation Long et al . ( 2015 ) ; Wei et al . ( 2016 ) , etc . In this paper , we present a general technique towards efficient INT8 inference of CNN models . We experiment the technique on top of a widely-used deep learning framework . To the best of our knowledge , our work is the first attempt to address the above problems . We summarize our contributions below : 1 . We provide a systematical approach to channel-wise quantization of convolution , which is essential to keep the accuracy for depthwise convolution . Top1 accuracy of INT8 inference on MobileNet-V1 and MobileNet-V2 is improved by 1.98 % and 70.6 % , respectively . 2 . We explore the approach of INT8 Winograd convolution and present the calibration details that can not be trivially derived from direct convolution . Our experiment on VGG-16 shows Top1 and Top5 accuracy loss with INT8 Winograd convolution is minimal within 0.30 % and 0.25 % from FP32 baseline , reducing from 5.31 % and 3.38 % , respectively . 3 . We add the support of sum in residual block , concatenation in inception block , and convolution for classification . We also fuse the memory-bound operation convolution with a rectified linear unit ( ReLU ) Nair & Hinton ( 2010 ) and fold the parameters of batch normalization Ioffe & Szegedy ( 2015 ) into convolution kernels . With topology-wise INT8 support , inference speed is greatly improved by data conversion reduction and memory saving . 4 . To our knowledge , this is the first time such a systematic study is applied to and empirical result is reported on many CNN use cases and models . We develop a calibration tool that automatically generates optimized INT8 model from FP32 model without the need of fine-tuning or retraining for easy and repeatable deployment . We perform a comprehensive study on 18 widely-used CNN models and demonstrate the effectiveness of INT8 inference across a wide range of applications , including image classification , object detection , image segmentation , and super resolution . The inference throughput and latency are improved by 1.6X and 1.5X respectively , while the accuracy loss is minimal within 0.6 % to no loss from FP32 baseline . We believe our methodology is general for CNN models and can provide the guide and reference on other frameworks . All the code and models will be publicly available soon . The rest of the paper is organized as follows , Section 2 discusses related work on low-precision inference in deep learning . Section 3 describes INT8 inference quantization approach and recipe for CNN models . Section 4 includes experimental results , comprehensive study , and related discussion . Finally , Section 5 concludes the summary with results and future directions . 2 RELATED WORK . Computer vision tasks win considerable attentions in deep learning field in recent years . Although CNN models provide SOTA accuracy for various computer vision tasks , it still faces challenges during industrial deployment due to its high computational complexity of inference . NVidia have demonstrated minimal accuracy loss of INT8 inference on several CNN models for image classification ( e.g. , ResNet-50 , ResNet-152 , VGG-16 , VGG-19 , GoogleNet , AlexNet ) . With the emerging of CNN models , more topology structures have been proposed to accelerate the performance , e.g. , depthwise convolution and inception block . Therefore , it is worthwhile to study INT8 inference on those new structures and understand whether traditional approaches are suitable . Besides TensorRT INT8 solution , some other open-source deep learning frameworks have started to support INT8 inference . TensorFlow approaches the conversion of floating-point arrays of numbers into INT8 representations as a compression problem and provides a graph transformation tool for model conversion with INT8 quantization operators inserted into the transformed graph . Similar to TensorFlow , MXNet also provides a calibration tool to transform FP32 to INT8 model . However , the throughput and latency is not ideal due to considerable data conversion overhead and lack of computation fusion . In our work , we add INT8 support of sum in residual block , concatenation in inception block , and convolution for classification , together with necessary computation folding on convolution and batch normalization and fusion on convolution and ReLU . On the other hand , Winograd convolution Lavin & Gray ( 2016 ) is a fast algorithm to speed up the convolution performance . Although it has been widely studied and used in FP32 training and inference , INT8 Winograd convolution is still not available in mainstream deep learning frameworks . Our work is the first attempt to explore INT8 Winograd convolution carefully and deliver the workable solution to keep the accuracy . In additional to existing inference tools and frameworks from industry , many researchers have experimented low-precision inference with customized low-bit for activation and weights in deep learning tasks . INT8 activations and weights have been proposed in Vanhoucke et al . ( 2011 ) , while biases and first layer input are kept with FP32 for the task of speech recognition on CPUs . CNN approximation has been presented Gysel ( 2016 ) to perform automatic network quantization and scoring , using different bit-widths for number representation , to find a good balance between compression rate and network accuracy . Baidu researchers 3 have successfully used 8-bits of fixed precision with 1 sign bit , 4-bits for the integer part and 3-bits for the fractional part . Various quantization techniques have been discussed in Sze et al . ( 2017 ) , showing minimal to no loss at reduced precision while keeping FP32 for the first and last layers . Deep compression with pruning , quantization , and Huffman coding has been worked out to reduce the storage requirement of neural networks significantly without affecting the accuracy , thus making easy for deployment on edge device Han et al . ( 2015 ) . Moreover , we focus on the efficient inference on commodity servers while others might require special hardware support like FPGA . Of course , some of our insights like calibrating INT8 Winograd can complement others ’ work as well . 3 RECIPE OF INT8 INFERENCE . In this section , we first formulate quantization and de-quantization mathematically and then present the general recipe of INT8 inference . 3.1 QUANTIZATION AND DE-QUANTIZATION . We define a quantization function Q : Rn×R×N 7→ Zn×R in Equation 1 to turn an n-dimensional rational tensor r into an n-dimensional integer tensor z with the scale factor q and bit-precision p. Here n could be of arbitrary dimensionality . The function Round is a rounding function approximating a rational tensor with an integer tensor . Q ( r , q , p ) = Qp ( r , q ) = Qp , q ( r ) = ( z , q ) , z = max ( min ( Round ( qr ) , 2 p − 1 ) , −2p ) , where r ∈ Rn , q ∈ R , p ∈ N+ , z ∈ Zn , Round : Rn 7→ Zn ( 1 ) We also define a de-quantization function D : Zn × R 7→ Rn that approximates the rational tensor r with its quantized form z in Equation 2 . D ( z , q ) = Dq ( z ) = z q = r′ ≈ r ( 2 ) We then define + and × arithmetics on ( z , q ) in Equation 3 . Here we assume + and × have already been defined for tensor r and z , e.g. , when they are matrices . ( z1 , q1 ) + ( z2 , q2 ) = Qp ( Dq1 ( z1 ) +Dq2 ( z2 ) , min ( q1 , q2 ) ) ( z1 , q1 ) × ( z2 , q2 ) = ( z1 × z2 , q1q2 ) ( 3 ) In practice , we perform sampling for each activation , weight and bias tensor on the given dataset to get a maximum absolute value max from each tensor and set the scale factor of the tensor as 2p−1 max where p is the precision of quantization . p = 8 is used for all non-negative activation tensors which are mostly true for popular CNN models after batch normalization operations are folded with convolution and ReLU with zero negative slope is fused into convolution Xu et al . ( 2015 ) . For potentially negative input tensors such as the one for first convolution , the operation falls back to FP32 since the hardware-accelerated INT8 convolution only supports non-negative activations as input ( more details refer to Rodriguez et al . ( 2018 ) ) . p = 7 is used for weight tensors . Then most activations and weights can be stored with INT8 . We employ round-half-to-even as the Round function for best statistical accuracy . 3https : //cdn.oreillystatic.com/en/assets/1/event/258/Benchmarking deep learning inference Presentation.pptx

Summary: This paper reads more like a technical/hardware white paper than a research paper. No real theory is offered, and the results are not really experiments testing hypotheses, but simply reporting the results of their design choices on a various of models.  Thus, it is not clear that this paper is especially suitable for ICLR research track per se.  Furthermore, the calibration method (to find suitable INT8 weights from fp32 ones without further training) appears to be essentially identical to techniques already reported by Nvidia as used by their TensorRT.  The discussion of INT8 for Winograd is something that could have been new and interesting (i.e. this reader has not seen Nvidia discuss this issue previously), but in the end this paper did not offer anything surprising or especially insightful in the brief Section 3.2.2 explaining their approach.  Furthermore, the experiments such as Table 2 do not include Winograd results anyway because that does not give competitive results using INT8, as the authors admit.