Quantization for Rapid Deployment of Deep Neural Networks

1 INTRODUCTION . Deploying state-of-the-art deep neural networks ( DNNs ) to embedded systems is a challenging task due to the inherent nature of huge number of computations and large memory requirements . These impediments are partly caused by the considerable amount of the redundancies found in the network parameters intended for ease of training . Thus , the parameters encompasses abundant opportunities for trimming strategies , namely pruning and quantizing to low precision ( Choi et al. , 2017 ; Courbariaux et al. , 2015b ; Han et al. , 2015 ) . However , running DNN inference on accelerators equipped with fixed-point arithmetic units in an energy efficient manner also requires limiting the precision of the feature maps ( Lin & Annapureddy , 2016 ; Migacz , 2017 ; Mishra et al. , 2017 ) . Previous works ( Courbariaux et al. , 2015b ; Gysel , 2016 ; Lin & Annapureddy , 2016 ; Migacz , 2017 ) have exhibited converting pretrained DNNs to 8-bit precision does not induce any accuracy loss . The feature maps and the network parameters were quantized at the granularity of layers to accommodate for large diversities in the dynamic range across the layers . Even though they showed good results for a few popular DNNs like AlexNet , VGG-Net , or GoogLeNet , it is not clear whether it would still work for many other recent DNNs with compact architectures . From our experiments , we were able to observe that this was not the case for some of the recent stateof-the-art DNNs . For example , applying 8-bit quantization to the individual layers of the MobileNet series as done in the previous works showed large accuracy degradations . The excessive accuracy loss could be mitigated by fine tuning the quantized networks ( Gysel , 2016 ; Lin & Annapureddy , 2016 ) . However , in order to reach a competitive level of accuracy for each network with fine tuning , full-size training and validation datasets were needed to be incorporated along with painstakingly long periods of optimization . As many DNN developers only provide the pretrained networks in full precision without the training or the validation datasets from reasons such as privacy or the outright massiveness of the data size , such obstacles hinder rapid and easy deployment of DNNs in full precision to embedded accelerators designed for low precision . Instead of converting a full precision pretrained network to lower precision suitable for embedded accelerators , it is also possible to train one from scratch by constraining the weights and the activations ( Hubara et al. , 2016 ; Mishra et al. , 2017 ; Zhou et al. , 2016 ; Zhuang et al. , 2017 ) . However , achieving state-of-the-art accuracy on large benchmarks such as ImageNet classification , increase in the network size in terms of connections ( Mishra et al. , 2017 ) or integrating complicated training process ( Zhuang et al. , 2017 ) is required . Nevertheless , these methods have not been proven for various types of network architectures . Considering the fact that GPUs are the most popular devices for training , it is more practical to convert DNNs into lower precision after utilizing the GPUs ’ full precision data path for training . In this paper , we introduce a novel technique in which fine tuning is not necessary for 8-bit linear quantization which quantizes the feature maps and the parameters for individual channels instead of layers to accommodate for the inter-channel variations in the dynamic range . We propose manipulating the kernel weights prior to inference for HW-friendly implementation of channel-wise quantization . Our method significantly reduces the accuracy loss caused by quantizing to lower precision without increasing the inference computation cost . The results show that various state-of-the-art DNNs trained on the ImageNet dataset can readily be converted for 8-bit fixed-point accelerators without fine tuning by using a few training samples for profiling . 2 LOW PRECISION QUANTIZATION . It is common practice to quantize the activations and the network parameters for each layer to account for the differences in the dynamic range across the layers ( Gysel , 2016 ; Lin & Annapureddy , 2016 ; Migacz , 2017 ) . Previous implementations such as Ristretto ( Gysel , 2016 ) , a fixed-point quantization simulator based on Caffe , reserves three placeholders for the fractional lengths ( defined as the number of required bits for the fractional part of a fixed-point number ) per layer , one each for the input and output feature maps ( IFM and OFM respectively ) and for the layer parameters ( weights and biases ) . At every layer , IFM , OFM , and the weights are polled separately for max values and the fractional lengths are calculated accordingly . During run-time , the MSBs and LSBs of the parameters and the activations of each layer are clipped to be containable within the given bit-width and the fractional lengths in order to emulate a generic fixed-point H/W implementation . We will use the term layer-wise quantization hereafter to describe this scheme in contrast with channel-wise quantization proposed in this paper . A major down-side of the layer-wise quantization is that the inter-channel variations of the feature maps and the weights are not fully accounted for . Since the fractional length is usually selected to cover the maximum value in a layer , the layer-wise quantization tends to cause excessive information loss in channels with a smaller dynamic range . Therefore , accuracy may be degraded significantly and sometimes never recovered even after exhaustive retraining . 2.1 CHANNEL-WISE QUANTIZATION . In the channel-wise quantization , the fractional lengths for the feature maps and the weights can be customized for each channel to minimize the impact of low-precision rounding . Each channel of the IFMs and the OFMs has an independent fractional length based on its expected dynamic range while each channel of the kernels has a fractional length which tightly fits its known values . Figure 1 demonstrates how the IFMs and the kernels from different channels having different fractional lengths in the channel-wise quantization scheme are computed through a convolution layer compared to the layer-wise scheme . In this example , the input and the output of the convolution layer and weights are all bound to 8 bits while the partial sums are allowed to be accumulated in 32 bits as to avoid data loss . The traversal in the layer-wise scheme is straight-forward as there aren ’ t any discrepancies while adding up the partial sums . On the other hand , the channel-wise method must cope with adding partial sums of varying fractional lengths . A naive solution would be to place a shifter in front of the partial sum adder to adjust all the partial sums to have the same fractional length . However , this scheme is not practical since too many shift operations are required . We resolve this complication by pre-coordinating the fractional lengths of the weights . The fractional length of a partial sum is determined by adding those of the IFM and the kernel being multiplied together . As the partial sums resulting from different input channels will have different fractional lengths , the smallest fractional length across all partial sums is selected as the reference . The red box in Figure 1 depicts this step . Then , the fractional lengths of the kernels in all the other channels are adjusted during the pre-processing stage to produce this reference fractional length when multiplied with their corresponding IFMs . Limitation was set on the amount adjusted so that the minimum value of the modified fractional lengths of the kernels would not be smaller than the layer-wise quantization . The overall procedure to determining the channel-wise fractional length is summarized in Algorithm 1 . The channel-wise quantization can be applied to a fully-connected ( FC ) layer by considering each unit as a channel . However , for simplicity , we use the layer-wise quantization for the activations of fully-connected ( FC ) layers . Notwithstanding , the weights of FC layers still needs to be adapted to the preceding layer . Figure 2 shows three such scenarios where we fallback to the layer-wise quantization . In scenario ( b ) and ( d ) , the activations quantized channel-wise from the previous convolution layer are multiplied with the channel-wise quantized weights of an FC layer which are pre-adjusted to yield the activations having an identical fractional length , hence , the layer-wise quantization for the activations . For scenario ( c ) where two FCs are stacked consecutively , the layer-wise quantization is utilized throughout the path . 2.2 FRACTIONAL LENGTH DETERMINATION . Determining the fractional length ( placing the dot in between the integer and fractional part within the given bit-width ) is easier said than done . Previous works ( Gysel , 2016 ; Migacz , 2017 ) profiled the target dataset to look for the max value which became the max representable value in the dynamic range . Profiling , running a network in forward path and collecting statistics , provides either an estimation of the dynamic range when executed during pre-processing stage on a subset of the training dataset or an exact fit when performed during run-time on the actual data being processed . Algorithm 1 HW-friendly channel-wise quantization . Profiling dataset is a subset of training data set . fl stands for fractional length ( flker : kernel fl , flifm : input feature map fl , flofm : output feature map fl , flpSumji : fl for partial sum of one kernel and one input , fladderj : adder fl , fl bias j : bias fl , shiftj : layer output bit-wise shift amount ) . Require : network architecture , network parameters , profiling dataset Ensure : flker , flbias , flifm , flofm , shift , quantized network parameters 1 . Profile weights and activations Calculate statistics of weights and activations of each channel on profiling dataset 2 . Calculate channel-wise fractional lengths For each layer , calculate flker from statistics for each channel of kernels calculate flifm , flofm from statistics of activations for each channel flpSumji : = fl ker ji + fl ifm i for all ( i , j ) pairs of input and output channels fladderj : = min i ( flpSumji ) for each j flbiasj : = fl adder j flkerji ← flkerji − ( flpSumji − flbiasj ) shiftj : = fl bias j − fl ofm j 3 . Quantize network parameters with flker , flbias The obvious side effect is that selecting just the right size of the dataset to profile is not trivial . A large dataset will increase the chance of electing an outlier as the max value which will overestimate the dynamic range . This will consequently penalize the fractional part of the fixed-point representation . On the other extreme where insufficient size of the profiling dataset is used , the values overflowing the determined fixed-point notation during inference will cause severe performance degradation . Lin proposed to use the n-th moments of the distribution instead of the max value to identify the optimal fixed-point bit-width and the fractional length ( Lin & Annapureddy , 2016 ) . The dynamic range of the activation is determined so that the signal-to-quantization-noise-ratio ( SQNR ) caused by quantization would be minimized . In this case , two factors contribute to the quantization noise . The first is the quantization error found within the dynamic range and the latter is the overload error where the values beyond the range are clipped to either the upper or the lower bound . When the number of the quantization levels is fixed , there exists an optimum interval between the levels which makes these two errors balanced . This method is much less sensitive to the size of the profiling dataset since the n-th moments of a distribution are more stable measures than the max value . A positive side effect is that optimum interval can be found even with a small dataset as shown in Section 3.1 . Figure 3 illustrates the superpositioned probability density functions ( PDFs ) of the pre-activation values of the individual OFM channels measured on GoogLeNet ( Szegedy et al. , 2014 ) trained with the ImageNet dataset . All PDFs were normalized and shifted to have an unit variance and a mean of zero prior to compositing the functions . As can be seen from the graph , there is a large variation in the distribution of the pre-activation values . In Lin & Annapureddy ( 2016 ) , normal distribution was used to approximate them . However , the mean distribution over all the channels , shown in Figure 3 , suggests Laplace distribution rather than normal distribution . We also found that the fractional length obtained by either normal or Laplace distribution tends to underestimate the dynamic range due to the heavy tails of the actual distribution in many channels . In those cases , truncated super Cauchy distribution , defined as follows , provides smaller quantization-induced noise by appropriately considering the tails . f ( x ) = √ 2 πγ [ 1+ ( x−x0γ ) 4 ] , if − 15 < x− x0 < 15 0 , otherwise ( 1 ) Here , x0 is the location parameter and γ is the scale parameter .

Summary: The paper proposes channel-wise 8-bit quantization rather than layer-wise. It further takes advantage of work using moment analysis instead of just MAX values to avoid susceptibility to outliers. The main take-away seems to be that channel-wise set ups limit the need for outlier removal and the care with which you select your data subset when performing quantization.