Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations

We propose precision gating ( PG ) , an end-to-end trainable dynamic dual-precision quantization technique for deep neural networks . PG computes most features in a low precision and only a small proportion of important features in a higher precision to preserve accuracy . The proposed approach is applicable to a variety of DNN architectures and significantly reduces the computational cost of DNN execution with almost no accuracy loss . Our experiments indicate that PG achieves excellent results on CNNs , including statically compressed mobile-friendly networks such as ShuffleNet . Compared to the state-of-the-art prediction-based quantization schemes , PG achieves the same or higher accuracy with 2.4× less compute on ImageNet . PG furthermore applies to RNNs . Compared to 8-bit uniform quantization , PG obtains a 1.2 % improvement in perplexity per word with 2.7× computational cost reduction on LSTM on the Penn Tree Bank dataset . 1 INTRODUCTION . In recent years , deep neural networks ( DNNs ) have demonstrated excellent performance on many computer vision and language modeling tasks such as image classification , semantic segmentation , face recognition , machine translation , and image captioning ( Krizhevsky et al. , 2012 ; He et al. , 2016a ; Ronneberger et al. , 2015 ; Chen et al. , 2016 ; Zhao et al. , 2018 ; Schroff et al. , 2015 ; Luong et al. , 2015 ; Vaswani et al. , 2017 ) . One evident trend in DNN design is that as researchers strive for better accuracy , both the model size and the number of DNN layers have drastically increased over time ( Xu et al. , 2018 ) . At the same time , there is a growing demand to deploy deep learning technology in edge devices such as mobile phones , VR/AR glasses , and drones ( Wu et al. , 2019 ) . The limited computational , memory , and energy budgets on these devices impose major challenges for the deployment of large DNN models at the edge . DNN quantization is an important technique for improving the hardware efficiency of DNN execution ( Zhao et al. , 2017 ) . Numerous studies have shown that full-precision floating-point computation is not necessary for DNN inference — quantized fixed-point models produce competitive results with a small or zero loss in prediction accuracy ( Lin et al. , 2016 ; He et al. , 2016b ; Zhou et al. , 2016 ; 2017 ) . In some cases , quantization may even improve model generalization by acting as a form of regularization . Existing studies mainly focus on static quantization , in which the precision of each weight and activation is fixed prior to inference ( Hubara et al. , 2017 ; He et al. , 2016b ) . Along this line of work , researchers have explored tuning the bitwidth per layer ( Wu et al. , 2018b ; Wang et al. , 2019 ; Dong et al. , 2019 ) as well as various types of quantization functions ( Wang et al. , 2018 ; Courbariaux et al. , 2016 ; Li et al. , 2016 ; Zhou et al. , 2016 ) . However , static DNN quantization methods can not exploit input-dependent characteristics , where certain features can be computed in a lower precision during inference as they contribute less to the classification result for the given input . For example , in computer vision tasks , the pixels representing the object of interest are typically more important than the background pixels . In this paper , we reduce the inefficiency of a statically quantized DNN via precision gating ( PG ) , which computes most features with low-precision arithmetic operations and only updates few important features to a high precision . More concretely , PG first executes a DNN layer in a low precision and identifies the output features with large magnitude as important features . It then computes a sparse update to increase the precision of those important output features . Intuitively , small values make a small contribution to the DNN ’ s output ; thus approximating them in a low precision is reasonable . Precision gating enables dual-precision DNN execution at the granularity of each individual output feature , and therefore greatly reducing the average bitwidth and computational cost of the DNN . We further introduce a differentiable gating function which makes PG applicable to a rich variety of network models . Experimental results show that PG achieves significant compute reduction and accuracy improvement on both CNNs and LSTMs . Compared to the baseline CNN counterparts , PG obtains 3.5 % and 0.6 % higher classification accuracy with up to 4.5× and 2.4× less computational cost for CIFAR-10 and ImageNet , respectively . On LSTM , compared to 8-bit uniform quantization PG boosts perplexity per word ( PPW ) by 1.2 % with 2.8× less compute on the Penn Tree Bank ( PTB ) dataset . Our contributions are as follows : 1 . We propose precision gating ( PG ) , which to the best of our knowledge is the first end-toend trainable method that enables dual-precision execution of DNNs . PG is applicable to a wide variety of CNN and LSTM models . 2 . PG enables DNN computation with lower average bitwidth than other state-of-the-art quantization methods . By employing a low-cost gating scheme , PG has the potential to reduce DNN execution costs in both commodity and dedicated hardware . 3 . PG achieves the same sparsity during back-propagation as forward propagation , which dramatically reduces the computational cost for both passes . This is in stark contrast to prior dynamic DNN optimization methods that focus only on reducing the inference cost . 2 RELATED WORK . Quantizing activations . Prior studies show that weights can be quantized to low bitwidth without compromising much accuracy ( Zhu et al. , 2017 ) ; however , quantizing activations with a low bitwidth ( e.g. , 4 bits ) typically incurs a nontrivial accuracy degradation ( Mishra et al. , 2018 ; Zhou et al. , 2016 ) . This is partially caused by large activation and weight outliers , which stretch the quantization grid too wide and too sparse under a low precision , thus increasing the error ( Park et al. , 2018 ; Zhao et al. , 2019 ) . To address this problem , Choi et al . ( 2018 ) propose PACT to reduce the dynamic range of the activations through clipping the outliers using a learnable threshold . PACT provides a more effective quantization scheme under a very low bitwidth ( e.g. , 4 bits ) . In this work we incorporate the PACT method in the training flow of PG to handle the large outliers . Prediction-based execution . Prior works have explored predicting ReLU-induced zeros and maxpooling compute redundancy to lower the computational cost of CNNs . For example , Lin et al . ( 2017 ) ; Song et al . ( 2018 ) propose zero-prediction to utilize a few of the most-significant bits in the input activations to predict the sign of the output activation . Zero-prediction removes redundancy by exploiting the fact that negative outputs will be suppressed by the ReLU anyway . Yet , this method only applies to ReLU activations and only when a linear layer is directly followed by ReLU . Hence such method does not apply to RNNs that use sigmoid or tanh as the activation function and many modern CNN networks that use batch normalization ( Ioffe & Szegedy , 2015 ) before ReLU . Hua et al . ( 2019a ; b ) propose channel gating to dynamically turn off a subset of channels that contribute little to the model prediction result . Precision gating is orthogonal to this pruning technique as channel gating executes the whole network at the same precision . More recently , Cao et al . ( 2019 ) propose SeerNet , which also executes a CNN model in dual precision . For each convolutional layer , SeerNet first executes a quantized version of the layer and uses the results to predict the output sparsity induced by the ReLU or the computational sparsity induced by the max-pooling layer . For those activations that are not suppressed according to the prediction , SeerNet computes the original convolution in full-precision ( 32-bit float ) . One key dif- Input Activations = < < + ℎ ference between PG and SeerNet is that PG reuses the result of the low-precision compute as a partial product when it performs the high-precision multiplication in the update phase . In contrast , the full-precision compute in SeerNet does not reuse the output from the quantized layer , which incurs a higher execution cost . Feature-level precision tuning . There is also prior work that uses a different precision to handle the outlier quantization . Park et al . ( 2018 ) propose value-aware quantization where the majority of data are computed at reduced precision while a small number of outliers are handled at high precision . Our approach is significantly different because we allow dual precision for every feature , not only for the outliers . 3 PRECISION GATING ( PG ) . In this section we first describe the basic mechanism of PG . We then discuss how to design the gating scheme to accelerate both forward and backward passes . Finally , we consider incorporating outlier clipping to reduce the quantization error . 3.1 BASIC FORMULATION . We first define a linear layer in a neural network ( either convolutional or fully-connected ) as O = I∗W , where O , I , and W are the output , input , and weights , respectively . Suppose I is represented in a B-bit fixed-point format , which is shown in Figure 1 . PG partitions I into ( 1 ) Ihb , the Bhb most-significant bits ( MSBs ) , and ( 2 ) Ilb , the remaining Blb least-significant bits ( LSBs ) . Here B = Bhb +Blb . More formally , we can write : I = Ihb < < Blb + Ilb ( 1 ) Here < < denotes the left shift operator . We can then reformulate a singleB-bit linear layer into two lower-precision computations as : O = Ohb + Olb = [ W ∗ ( Ihb < < Blb ) ] + [ W ∗ Ilb ] ( 2 ) Ohb is the partial product obtained by using the MSBs of input feature ( Ihb ) whereas Olb represents the remaining partial product computed with Ilb . Precision gating works in two phases . In the prediction phase , PG performs the computation Ohb = W ∗ ( Ihb < < Blb ) . Output features of Ohb greater than a learnable gating threshold ∆ are considered important . In the update phase , PG computes Olb = W ∗ Ilb only for the important features and adds it to Ohb . The overall execution flow of PG is illustrated in Figure 2 , where unimportant output features only take the upper path while important ones are computed as the sum of both paths . More precisely , this can be summarized as follows : O = { Ohb Ohb ≤ ∆ Ohb + Olb Ohb > ∆ ( 3 ) In essence , PG intends to compute the majority of ( unimportant ) features with Bhb bits and only a small set of important features with B bits . The importance of each element in the output O is calculated by comparing the magnitude of its partial sum Ohb to ∆ . Let Sp be the percentage of unimportant activations over all features . PG saves Sp·BlbB fraction of the compute in the original DNN model . PG thus achieves dual-precision execution using a lightweight gating mechanism which only adds a comparison operation . The goal is to minimize the average bitwidth of the multiply-add operations in the DNN execution .

Summary: This paper outlines a new method that allows using a variety of precision in the numerical representation of the network to increase performance (both in terms of accuracy and speed). They learn a threshold value for which all activation values above the threshold are learned at full precision, while all below are learned at reduced precision. This enables substantial performance gains.