Title: HIGHLY EFFICIENT 8-BIT LOW PRECISION INFERENCE OF CONVOLUTIONAL NEURAL NETWORKS Anonymous

Abstract: High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents a general technique toward 8-bit low precision inference of convolutional neural networks, including 1) channel-wise scale factors of weights, especially for depthwise convolution, 2) Winograd convolution, and 3) topology-wise 8-bit support. We experiment the techniques on top of a widely-used deep learning framework. The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining. We perform a systematical and comprehensive study on 18 widely-used convolutional neural networks and demonstrate the effectiveness of 8-bit low precision inference across a wide range of applications and use cases, including image classification, object detection, image segmentation, and super resolution. We show that the inference throughput and latency are improved by 1.6X and 1.5X respectively with minimal within 0.6% 1 to no loss in accuracy from FP32 baseline. We believe the methodology can provide the guidance and reference design of 8-bit low precision inference for other frameworks. All the code and models will be publicly available soon.

1. INTRODUCTION:

While convolutional neural networks (CNN) shows state-of-the-art (SOTA) accuracy for wide range of computation vision tasks, it still faces challenges during industrial deployment due to its high computational complexity of inference. Low precision is one of the key techniques being actively studied recently to conquer the problem Vanhoucke et al. (2011) ; Hwang & Sung (2014) ; Rastegari et al. (2016) ; Miyashita et al. (2016) ; Mellempudi et al. (2017) . With hardware acceleration support, low precision inference can compute more operations per second, reduce the memory access pressure and better utilize the cache, and deliver higher throughput and lower latency.

Convolution is the primary operation in CNN models and it is a common practice to enable 8-bit low precision (INT8) inference for convolution in deep learning frameworks (e.g., TensorFlow, MXNet, and TensorRT). To make it work, convolution utilizes INT8 computation, which requires two scale factors for activation and weight, respectively. It is workable for standard convolution with single group and two groups Krizhevsky et al. (2012) . However, it does not work well for convolution with large groups, especially for depthwise convolution Howard et al. (2017) ; Chollet (2016) . In addition to direct convolution, it is worthwhile to explore INT8 Winograd convolution Lavin & Gray (2016) for better performance, which is absent in previous research2 . Although recent work have demonstrated INT8 inference with minimal accuracy loss across various models Vanhoucke et al. (2011) ; Gysel (2016) ; Wu et al. (2016) ; Jacob et al. (2017) , INT8 inference is limited due to more complex topology primarily introduced by sum operation in residual block He et al. (2016) and concatenation operation in inception block Szegedy et al. (2015; 2016) . Existing solutions need to convert the convolution output from INT8 to FP32, and apply the sum or concatenation operation on FP32. The sacrifice of memory bandwidth and frequent data conversion lead to considerable performance overhead and therefore limit the real deployment. Moreover, there is no systematical study of INT8 inference on various use cases, including image classification Krizhevsky et al. (2012) ; Simonyan & Zisserman (2014) ; Szegedy et al. (2015; 2016) ; He et al. (2016 ), object detection Ren et al. (2015) ; Dai et al. (2016) ; Liu et al. (2016) , image segmentation Long et al. (2015) ; Wei et al. (2016), etc. In this paper, we present a general technique towards efficient INT8 inference of CNN models. We experiment the technique on top of a widely-used deep learning framework. To the best of our knowledge, our work is the first attempt to address the above problems. We summarize our contributions below:

1. We provide a systematical approach to channel-wise quantization of convolution, which is essential to keep the accuracy for depthwise convolution. Top1 accuracy of INT8 inference on MobileNet-V1 and MobileNet-V2 is improved by 1.98% and 70.6%, respectively. 2. We explore the approach of INT8 Winograd convolution and present the calibration details that cannot be trivially derived from direct convolution. Our experiment on VGG-16 shows Top1 and Top5 accuracy loss with INT8 Winograd convolution is minimal within 0.30% and 0.25% from FP32 baseline, reducing from 5.31% and 3.38%, respectively. 3. We add the support of sum in residual block, concatenation in inception block, and convolution for classification. We also fuse the memory-bound operation convolution with a rectified linear unit (ReLU) Nair & Hinton (2010) and fold the parameters of batch normalization Ioffe & Szegedy (2015) into convolution kernels. With topology-wise INT8 support, inference speed is greatly improved by data conversion reduction and memory saving. 4. To our knowledge, this is the first time such a systematic study is applied to and empirical result is reported on many CNN use cases and models. We develop a calibration tool that automatically generates optimized INT8 model from FP32 model without the need of fine-tuning or retraining for easy and repeatable deployment. We perform a comprehensive study on 18 widely-used CNN models and demonstrate the effectiveness of INT8 inference across a wide range of applications, including image classification, object detection, image segmentation, and super resolution. The inference throughput and latency are improved by 1.6X and 1.5X respectively, while the accuracy loss is minimal within 0.6% to no loss from FP32 baseline.

We believe our methodology is general for CNN models and can provide the guide and reference on other frameworks. All the code and models will be publicly available soon.

The rest of the paper is organized as follows, Section 2 discusses related work on low-precision inference in deep learning. Section 3 describes INT8 inference quantization approach and recipe for CNN models. Section 4 includes experimental results, comprehensive study, and related discussion. Finally, Section 5 concludes the summary with results and future directions. In additional to existing inference tools and frameworks from industry, many researchers have experimented low-precision inference with customized low-bit for activation and weights in deep learning tasks. INT8 activations and weights have been proposed in Vanhoucke et al. (2011) , while biases and first layer input are kept with FP32 for the task of speech recognition on CPUs. CNN approximation has been presented Gysel (2016) to perform automatic network quantization and scoring, using different bit-widths for number representation, to find a good balance between compression rate and network accuracy. Baidu researchers3 have successfully used 8-bits of fixed precision with 1 sign bit, 4-bits for the integer part and 3-bits for the fractional part. Various quantization techniques have been discussed in Sze et al. (2017) , showing minimal to no loss at reduced precision while keeping FP32 for the first and last layers. Deep compression with pruning, quantization, and Huffman coding has been worked out to reduce the storage requirement of neural networks significantly without affecting the accuracy, thus making easy for deployment on edge device Han et al. (2015) . Moreover, we focus on the efficient inference on commodity servers while others might require special hardware support like FPGA. Of course, some of our insights like calibrating INT8 Winograd can complement others' work as well.

2. RELATED WORK:



3. RECIPE OF INT8 INFERENCE:

In this section, we first formulate quantization and de-quantization mathematically and then present the general recipe of INT8 inference.

3.1. QUANTIZATION AND DE-QUANTIZATION:

We define a quantization function 1to turn an n-dimensional rational tensor r into an n-dimensional integer tensor z with the scale factor q and bit-precision p. Here n could be of arbitrary dimensionality. The function Round is a rounding function approximating a rational tensor with an integer tensor.

Q : R n × R × N → Z n × R in Equation

Q(r, q, p) = Q p (r, q) = Q p,q (r) = (z, q), z = max(min(Round(qr), 2 p -1), -2 p ), where r ∈ R n , q ∈ R, p ∈ N + , z ∈ Z n , Round : R n → Z n (1)

We also define a de-quantization function D : Z n × R → R n that approximates the rational tensor r with its quantized form z in Equation 2.

EQUATION

We then define + and × arithmetics on (z, q) in Equation 3. Here we assume + and × have already been defined for tensor r and z, e.g., when they are matrices.

EQUATION

In practice, we perform sampling for each activation, weight and bias tensor on the given dataset to get a maximum absolute value max from each tensor and set the scale factor of the tensor as

2 p -1

max where p is the precision of quantization. p = 8 is used for all non-negative activation tensors which are mostly true for popular CNN models after batch normalization operations are folded with convolution and ReLU with zero negative slope is fused into convolution Xu et al. (2015) . For potentially negative input tensors such as the one for first convolution, the operation falls back to FP32 since the hardware-accelerated INT8 convolution only supports non-negative activations as input (more details refer to Rodriguez et al. (2018) ). p = 7 is used for weight tensors. Then most activations and weights can be stored with INT8. We employ round-half-to-even as the Round function for best statistical accuracy.

3.2. GENERAL RECIPE:

We present the general INT8 recipe for CNN models, including depthwise convolution, Winograd convolution, and topology-wise more INT8 support.

3.2.1. DEPTHWISE CONVOLUTION:

As a common practice, INT8 convolution uses a single scale factor for each tensor, i.e. one for activation and one for weight respectively. It is workable for standard convolution with single group (e.g., VGG-16, GoogleNet-V1, and ResNet-50) and two groups (e.g., AlexNet). However, it does not perform well for convolution with large groups, especially for depthwise convolution (e. Based on the above findings, we propose channel-wise scale factors for weight tensor, similar to Krishnamoorthi (2018) . Each scale factor represents the dynamic data range per each filter. The resulting scale factors are q activation × q weighti , where q activation is the scale factor of activation and q weighti is the scale factor of the i th filter. With channel-wise scaling factors, Top1 accuracy of INT8 inference on MobileNet-V1 and MobileNet-V2 is improved by 1.98% and 70.6%, respectively.

3.2.2. WINOGRAD CONVOLUTION:

Winograd is a fast algorithm for convolution and it has been widely-used in FP32 training and inference Lavin & Gray (2016) . However, the study of INT8 Winograd convolution is not publicly available. Considering the attractive performance gains, it is worthwhile to explore INT8 Winograd convolution. We select standard algorithm F(2, 3) for discussion, which can leverage INT8 computation benefit from integer-based input transformation matrix. To make INT8 Winograd convolution work, the key component is to take the scale factor for activation and weight after transformation.

x a = B T x b B q xa = q x b * max x b /max xa (4)

Equation 4 shows the formula to compute the scale factor after transformation, where B and B T are transformation matrices defined in Lavin & Gray (2016) . Before and after transformation, we have the activation tensor for x b and x a , the scale factor for q x b (for direction convolution by default) and q xa , the maximum absolute value for max x b and max xa , respectively. Similarly, we can compute the scale factor of weight before and after transformation. The scale factor of activation and weight after transformation is set for INT8 Winograd convolution finally. We experiment the idea on VGG-16, a classical model for Winograd convolution. With the scale factor q xa , Top1 and Top5 accuracy loss is minimal within 0.30% and 0.25% from FP32 baseline, while with the scale factor q x b , the accuracy loss is significant with 5.31% and 3.38%, respectively. Note that our approach is general and can be applied to other algorithms besides standard algorithm F(2, 3).

3.2.3. TOPOLOGY-WISE INT8 SUPPORT:

We extend INT8 computation to other computation types besides convolution and also apply constant folding and computation fusion to consecutive computations so that almost all input and output activation tensors use INT8 while accumulators use INT32 or FP32 for best accuracy. In this section, we discuss these topology-wise INT8 opportunities. We also discuss topology patterns in which output tensors should be kept in FP32 for good accuracy. 

EQUATION

4. EXPERIMENTAL RESULTS:

With the general recipe of INT8 inference, we experiment the techniques and develop the calibration tool on top of a widely-used deep learning framework . We next discuss the experimental configurations and perform a systematical study on 18 classical CNN models. We demonstrate the effectiveness of INT8 inference across a wide range of applications and use cases.

4.1. CALIBRATION TOOL FOR INT8 INFERENCE:

We develop the calibration tool that automatically generates the optimized INT8 model from FP32 model without the need of fine-tuning or retraining. The calibration process has two inputs, CNN model with pre-trained FP32 weights and calibration dataset. Besides, the tool provides the additional items to facilitate the calibration process:

Iteration number. It allows user to define the iteration number for sampling on activation.

Scale factor mode. It allows user to define scale factor mode single or multiple (channel-wise).

Calibration strategy. It allows users to define the calibration algorithm (Direct or KL) to compute the scale factor by 2 p -1 max , where p is the quantization precision. Direct selects the maximum absolute value of the tensor as max directly, while KL computes max in terms of the entropy loss of quantization following the work in TensorRT.

Accuracy tuning. It allows users to define the accuracy loss tolerance on INT8 model. Calibration process makes some operations fall back to FP32 to meet the accuracy goal.

4.2. EXPERIMENTAL SETUPS:

We select totally 18 CNN models in our experiments in Table 1 . Basically, we have three rules for model selection: 1) it is classical and representative; 2) it comes from various use cases; and 3) it is publicly available with pre-trained weight or is easy to train with existing hyper-parameters. 

√ VGG-19 IC √ ResNet-50 IC √ ResNet-101 IC √ ResNet-152 IC √ ResNet-50 (FB) IC × MobileNet-V1 IC √ MobileNet-V2 IC √ Inception-V3 IC √ Inception-ResNet-V2 IC √ SqueezeNet-V1 0 IC √ SqueezeNet-V1 1 IC √ SSD (VGG-16) OD √ SSD (MobileNet-V1) OD √ Faster-RCNN (VGG-16) OD × R-FCN (ResNet-101) OD × FCN IS √ FSRCNN SR ×

Topology column shows the selected CNN model. On ResNet-50, we use two versions, default one from He et al. (2016) and variant one from FaceBook (with FB) Goyal et al. (2017) . Use case column shows the model category, IC (image classification), OD (object detection), IS (image segmentation), and SR (super resolution). Weight column shows whether the pre-trained weight is publicly available.

With respect to calibration dataset, we use ImageNet-1k Russakovsky et al. (2015) for image classification, PASCAL VOC Everingham et al. (2015) for object detection and image segmentation, and internal gaming images for super resolution.

4.3. ACCURACY AND PERFORMANCE:

We perform calibration on training dataset with sampling iteration from 1, 2, 5, 10, 20, to 30, scale factor mode single or multiple, and different algorithm Direct or KL. The total calibration cost is affordable since it takes seconds to minutes to complete each calibration. We measure the accuracy on validation dataset independently from calibration dataset. On performance side, we measure the performance of INT8 inference and speedup over FP32 using dummy data, as shown in Table 2 . We can see that the throughput and latency are improved et al. (2018) and therefore the latency improvement might be smaller for those non-computationintensive topologies (e.g., MobileNetV2).

4.4. DISCUSSION:

To align the model with best accuracy, the above performance in et al. (2011) ; Sze et al. (2017) , we also experiment the first convolution using INT8 than FP32, which shows reasonable accuracy within 1% loss.

Our experimental results also demonstrate the impact of calibration process on accuracy with different sampling iteration, different calibration algorithm, or different scale factor mode. We summarize our findings: (1) Channel-wise scaling factors can always deliver better accuracy than single scale factor, especially for depthwise convolution;

(2) Direct algorithm is more effective in most cases than KL, while KL algorithm can deliver better accuracy than FP32 baseline in some cases; and (3) More sampling iterations show more stable dynamic data rage and therefore better accuracy. How to select the optimal calibration strategy is an interesting topic as one of our future directions.

5. CONCLUSION:

In this paper, we propose the general recipe of INT8 inference and experiment the techniques on a widely-used deep learning framework. We develop an automatic calibration tool for optimized INT8 model generation and demonstrate the effectiveness on 18 CNN models across a wide range of use cases. The inference throughput and latency are improved by 1.6X and 1.5X respectively, while the accuracy loss is minimal within 0.6% to no loss from FP32 baseline. We believe our methodology is general for CNN models and can provide the guide and reference on other frameworks.

:

% denotes percentage point for simplicity in the context of accuracy loss

http://nvdla.org/primer.html

https://cdn.oreillystatic.com/en/assets/1/event/258/Benchmarking deep learning inference Presentation.pptx

Figures and Tables:

Figure 1: Distribution of the first 10 filters for standard convolution (a) vs. depthwise convolution conv2 1/dwise in MobileNet V2 (b). Figure (a) shows the stable dynamic data range for the first 10 filters (also for the entire weight tensor). Figure (b) shows the fluctuant dynamic data range cross filters. The 6th filter has 50x bigger dynamic range than the other filters.

Pooling. Both max pooling and average pooling are computed directly with INT8. The scale factors of the input and output tensors are same. We use INT32 accumulator for average pooling to avoid arithmetic overflow.Concatenation. Tensor concatenation is a common operation such as those in Inception blockSzegedy et al. (2015; 2016). Figure2demonstrates the inception block that concatenates convolution output per filter. Our study shows that the dynamic ranges of the input tensors are quite close. So we set the scale factor of INT8 output tensor to the smallest scale factor of INT8 input tensors.

Figure 2: Concatenation in inception block. Dynamic data range is pretty stable in activations for concatenation within the range from 1.6x to 2.0x among the topologies in our work.

Figure 3: Default residual block (a) and fused residual block (b). FP32 Tensor for Classification. Convolution and global average pooling becomes the new operator combination for image classification, which evolves from typical fully-connected operator. Mo-bileNet Howard et al. (2017); Sandler et al. (2018) and SqueezeNet Iandola et al. (2016) are two CNN models with such structure (shown in Figure4). Our practice shows FP32 tensor output for convolution is essential to keep the accuracy, while using INT8 computation inside convolution. It improves INT8 Top1 accuracy by 6.4% and 1.4% for SqueezeNet-V1 0 and SqueezeNet-V1 1, respectively.

used in FP32 training and inference, INT8 Winograd convolution is still not available in mainstream deep learning frameworks. Our work is the first attempt to explore INT8 Winograd convolution carefully and deliver the workable solution to keep the accuracy.

Table content: <table><tr><td>widely studied and</td></tr><tr><td>Computer vision tasks win considerable attentions in deep learning field in recent years. Although</td></tr><tr><td>CNN models provide SOTA accuracy for various computer vision tasks, it still faces challenges</td></tr><tr><td>during industrial deployment due to its high computational complexity of inference.</td></tr><tr><td>NVidia have demonstrated minimal accuracy loss of INT8 inference on several CNN models for</td></tr><tr><td>image classification (e.g., ResNet-50, ResNet-152, VGG-16, VGG-19, GoogleNet, AlexNet). With</td></tr><tr><td>the emerging of CNN models, more topology structures have been proposed to accelerate the perfor-</td></tr><tr><td>mance, e.g., depthwise convolution and inception block. Therefore, it is worthwhile to study INT8</td></tr><tr><td>inference on those new structures and understand whether traditional approaches are suitable. Besides</td></tr><tr><td>TensorRT INT8 solution, some other open-source deep learning frameworks have started to support</td></tr><tr><td>INT8 inference. TensorFlow approaches the conversion of floating-point arrays of numbers into INT8</td></tr><tr><td>representations as a compression problem and provides a graph transformation tool for model conver-</td></tr><tr><td>sion with INT8 quantization operators inserted into the transformed graph. Similar to TensorFlow,</td></tr><tr><td>MXNet also provides a calibration tool to transform FP32 to INT8 model. However, the throughput</td></tr><tr><td>and latency is not ideal due to considerable data conversion overhead and lack of computation fusion.</td></tr><tr><td>In our work, we add INT8 support of sum in residual block, concatenation in inception block, and</td></tr><tr><td>convolution for classification, together with necessary computation folding on convolution and batch</td></tr><tr><td>normalization and fusion on convolution and ReLU. On the other hand, Winograd convolution Lavin</td></tr><tr><td>&amp; Gray (2016) is a fast algorithm to speed up the convolution performance. Although it has been</td></tr></table>

Selected CNN Models

Table content: <table><tr><td>Topology</td><td>Use Case Weight Topology</td><td>Use Case Weight</td></tr><tr><td>VGG-16</td><td>IC</td><td/></tr></table>

Table2shows the best accuracy of CNN models under INT8 inference. Note that we use standard metrics to measure the accuracy, Top1 and Top5 for image classification, mAP (mean Average Precision) for object detection, mean accuracy and IoU (Intersection of Union) for image segmentation, and SSIM (Structural SIMilarity) and PSNR (Peak Signal-to-Noise Ratio) for super resolution. Our experiments demonstrate the effectiveness across a wide range of use cases, keeping the accuracy loss from FP32 baseline, within 0.6% for Top1 and 0.3% for Top5 on image classification, 0.5% for mAP on object detection, 0.2% for mean IoU on image segmentation, and 0.1% for PSNR on super resolution. Moreover, INT8 inference recipe also works well for models ResNet-50/101/152 with sparsity removalRodriguez et al. (2018).On the other hand, we evaluate the errors of 50k images from ImageNet validation set for FP32 and INT8 inference and find that there is no obvious bias at image class based on empirical analysis on incorrectly-predicted images. With further analysis on typical images, we figure out that it is more difficult for INT8 model to distinguish the objects with small differences. As an example, INT8 model can recognize the dog (ILSVRC2012 val 00046423) correctly, but fails to figure out the accurate breed. Moreover, we find that the information loss from FP32 to INT8 model may lead to potential misclassification (e.g., ILSVRC2012 val 00031193). We also compute the entropy of Softmax output for both FP32 and INT8 model. The results show the probability is average for INT8 model, which indicates the entropy increases and Top1 classification capability decreases.

Table content: <table/>

Accuracy and Performance: FP32 VS. INT8 Top1 and Top5 shows the accuracy for image classification with (INT8, INT8 -FP32). The models with †, ‡, and § use different accuracy metrics that are shown in Section 4.3. Throughput and latency shows INT8 inference performance and speedup over FP32. Throughput is measured by images per second on large batch size and latency is measured by milliseconds on single batch size.by 1.6X and 1.5X in average and 2.0X and 2.1X as maximum, respectively. Please note that the convolution improvement on INT8 over FP32 is 1.3X based on HW instructions support Rodriguez

Table content: <table><tr><td>Topology</td><td>Top1</td><td>Top5</td><td colspan="2">Throughput Latency</td></tr><tr><td>VGG-16</td><td colspan="3">68.18% (-0.16%) 88.32% (-0.05%) 156 (1.5X)</td><td>11.43 (1.3X)</td></tr><tr><td>VGG-19</td><td colspan="3">68.28% (-0.17%) 88.27% (-0.11%) 126 (1.5X)</td><td>13.24 (1.3X)</td></tr><tr><td>ResNet-50</td><td colspan="3">72.71% (-0.05%) 90.97% (-0.04%) 671 (1.7X)</td><td>2.50 (1.9X)</td></tr><tr><td>ResNet-101</td><td colspan="2">73.92% (-0.10%) 91.72% (0.00%)</td><td>349 (1.6X)</td><td>6.48 (1.5X)</td></tr><tr><td>ResNet-152</td><td colspan="3">74.48% (-0.27%) 92.00% (-0.11%) 234 (1.6X)</td><td>10.44 (1.5X)</td></tr><tr><td>ResNet-50 (FB)</td><td>76.40% (0.04%)</td><td colspan="2">93.17% (-0.10%) 575 (1.7X)</td><td>2.72 (2.1X)</td></tr><tr><td>MobileNet-V1</td><td colspan="4">69.58% (-0.29%) 89.14% (-0.22%) 2618 (1.9X) 0.81 (1.4X)</td></tr><tr><td>MobileNet-V2</td><td colspan="4">71.07% (-0.53%) 90.00% (-0.24%) 2042 (2.0X) 1.43 (1.1X)</td></tr><tr><td>Inception-V3</td><td colspan="2">77.15% (-0.06%) 93.31% (0.00%)</td><td>393 (1.7X)</td><td>5.33 (1.5X)</td></tr><tr><td>Inception-ResNet-V2</td><td colspan="3">80.11% (-0.49%) 95.14% (-0.06%) 163 (1.6X)</td><td>15.27 (1.3X)</td></tr><tr><td>SqueezeNet-V1 0</td><td colspan="4">57.32% (-0.24%) 80.04% (-0.26%) 1700 (1.9X) 0.82 (1.3X)</td></tr><tr><td>SqueezeNet-V1 1</td><td colspan="4">57.85% (-0.19%) 80.68% (-0.12%) 2651 (1.8X) 0.74 (1.0X)</td></tr><tr><td>SSD (VGG-16)  †</td><td>77.75% (0.03%)</td><td>N/A</td><td>71 (1.5X)</td><td>23.91 (1.6X)</td></tr><tr><td colspan="3">SSD (MobileNet-V1)  † 72.07% (-0.47%) N/A</td><td colspan="2">1038 (1.8X) 3.69 (2.1X)</td></tr><tr><td>Faster-RCNN  †</td><td colspan="2">71.05% (-0.42%) N/A</td><td>14 (1.5X)</td><td>72.72 (1.5X)</td></tr><tr><td>R-FCN  †</td><td colspan="2">78.87% (-0.29%) N/A</td><td>20 (1.6X)</td><td>50.43 (1.6X)</td></tr><tr><td>FCN  ‡</td><td colspan="3">82.00% (-0.21%) 69.73% (-0.11%) 8 (1.3X)</td><td>128.81 (1.3X)</td></tr><tr><td>FSRCNN  §</td><td colspan="2">0.9000 (-0.0066) 29.65 (-0.11)</td><td>115 (1.3X)</td><td>27.76 (1.1X)</td></tr></table>

Table 2 does not include INT8 Winograd convolution. We expect to deliver similar performance improvement of Winograd on INT8 as FP32 Lavin & Gray (2016) during our development. Different from previous work Vanhoucke

Table content: <table/>

Peer review (iclr2019):
 This paper designs a system to automatically quantize the CNN pretrained models. This system contains three main components: 1) different scale factors for channel-wise network; 2) Winograd 8bit quantization; 3) topology wise 8bit operation support. All these three techniques are standard ways to perform model quantization. The work is solid in the sense that 1) as far as I know there is no work actually using all of these quantization schemes, and designs a system to automatically do quantization with additional algorithm support(retrain strategy). 2) Significantly amount of experiment on quantizing 18 existing widely used CNN models for different applications, e.g., image classification, image segmentation, etc. 3) it reports the actually inference speed up comparing INT 32 to INT 8, although most of the speed up is less than 2.0. Several questions: 1) What is the difference between retrain and calibration? As mentioned in the paper, the system does not require retrain, but it seems to me that calibration step, e.g., sampling to find the maximum values, etc, is a form of retrain. I think if that is the case, maybe some quantized models with retrain is worthy comparing with. 2) Many quantization techniques are used in the system, are there any conclusions for what techniques are most important for a particular CNN network/application? 3) Are there any new/novel quantization algorithms in the system? 4) The inference speed up is mostly less than 2 times, with some are achieving 2.1 speed up while some are without any speedup. Any reason for that? Also what is the overhead of the inference time?