Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations

We propose precision gating ( PG ) , an end-to-end trainable dynamic dual-precision quantization technique for deep neural networks . PG computes most features in a low precision and only a small proportion of important features in a higher precision to preserve accuracy . The proposed approach is applicable to a variety of DNN architectures and significantly reduces the computational cost of DNN execution with almost no accuracy loss . Our experiments indicate that PG achieves excellent results on CNNs , including statically compressed mobile-friendly networks such as ShuffleNet . Compared to the state-of-the-art prediction-based quantization schemes , PG achieves the same or higher accuracy with 2.4× less compute on ImageNet . PG furthermore applies to RNNs . Compared to 8-bit uniform quantization , PG obtains a 1.2 % improvement in perplexity per word with 2.7× computational cost reduction on LSTM on the Penn Tree Bank dataset . 1 INTRODUCTION . In recent years , deep neural networks ( DNNs ) have demonstrated excellent performance on many computer vision and language modeling tasks such as image classification , semantic segmentation , face recognition , machine translation , and image captioning ( Krizhevsky et al. , 2012 ; He et al. , 2016a ; Ronneberger et al. , 2015 ; Chen et al. , 2016 ; Zhao et al. , 2018 ; Schroff et al. , 2015 ; Luong et al. , 2015 ; Vaswani et al. , 2017 ) . One evident trend in DNN design is that as researchers strive for better accuracy , both the model size and the number of DNN layers have drastically increased over time ( Xu et al. , 2018 ) . At the same time , there is a growing demand to deploy deep learning technology in edge devices such as mobile phones , VR/AR glasses , and drones ( Wu et al. , 2019 ) . The limited computational , memory , and energy budgets on these devices impose major challenges for the deployment of large DNN models at the edge . DNN quantization is an important technique for improving the hardware efficiency of DNN execution ( Zhao et al. , 2017 ) . Numerous studies have shown that full-precision floating-point computation is not necessary for DNN inference — quantized fixed-point models produce competitive results with a small or zero loss in prediction accuracy ( Lin et al. , 2016 ; He et al. , 2016b ; Zhou et al. , 2016 ; 2017 ) . In some cases , quantization may even improve model generalization by acting as a form of regularization . Existing studies mainly focus on static quantization , in which the precision of each weight and activation is fixed prior to inference ( Hubara et al. , 2017 ; He et al. , 2016b ) . Along this line of work , researchers have explored tuning the bitwidth per layer ( Wu et al. , 2018b ; Wang et al. , 2019 ; Dong et al. , 2019 ) as well as various types of quantization functions ( Wang et al. , 2018 ; Courbariaux et al. , 2016 ; Li et al. , 2016 ; Zhou et al. , 2016 ) . However , static DNN quantization methods can not exploit input-dependent characteristics , where certain features can be computed in a lower precision during inference as they contribute less to the classification result for the given input . For example , in computer vision tasks , the pixels representing the object of interest are typically more important than the background pixels . In this paper , we reduce the inefficiency of a statically quantized DNN via precision gating ( PG ) , which computes most features with low-precision arithmetic operations and only updates few important features to a high precision . More concretely , PG first executes a DNN layer in a low precision and identifies the output features with large magnitude as important features . It then computes a sparse update to increase the precision of those important output features . Intuitively , small values make a small contribution to the DNN ’ s output ; thus approximating them in a low precision is reasonable . Precision gating enables dual-precision DNN execution at the granularity of each individual output feature , and therefore greatly reducing the average bitwidth and computational cost of the DNN . We further introduce a differentiable gating function which makes PG applicable to a rich variety of network models . Experimental results show that PG achieves significant compute reduction and accuracy improvement on both CNNs and LSTMs . Compared to the baseline CNN counterparts , PG obtains 3.5 % and 0.6 % higher classification accuracy with up to 4.5× and 2.4× less computational cost for CIFAR-10 and ImageNet , respectively . On LSTM , compared to 8-bit uniform quantization PG boosts perplexity per word ( PPW ) by 1.2 % with 2.8× less compute on the Penn Tree Bank ( PTB ) dataset . Our contributions are as follows : 1 . We propose precision gating ( PG ) , which to the best of our knowledge is the first end-toend trainable method that enables dual-precision execution of DNNs . PG is applicable to a wide variety of CNN and LSTM models . 2 . PG enables DNN computation with lower average bitwidth than other state-of-the-art quantization methods . By employing a low-cost gating scheme , PG has the potential to reduce DNN execution costs in both commodity and dedicated hardware . 3 . PG achieves the same sparsity during back-propagation as forward propagation , which dramatically reduces the computational cost for both passes . This is in stark contrast to prior dynamic DNN optimization methods that focus only on reducing the inference cost . 2 RELATED WORK . Quantizing activations . Prior studies show that weights can be quantized to low bitwidth without compromising much accuracy ( Zhu et al. , 2017 ) ; however , quantizing activations with a low bitwidth ( e.g. , 4 bits ) typically incurs a nontrivial accuracy degradation ( Mishra et al. , 2018 ; Zhou et al. , 2016 ) . This is partially caused by large activation and weight outliers , which stretch the quantization grid too wide and too sparse under a low precision , thus increasing the error ( Park et al. , 2018 ; Zhao et al. , 2019 ) . To address this problem , Choi et al . ( 2018 ) propose PACT to reduce the dynamic range of the activations through clipping the outliers using a learnable threshold . PACT provides a more effective quantization scheme under a very low bitwidth ( e.g. , 4 bits ) . In this work we incorporate the PACT method in the training flow of PG to handle the large outliers . Prediction-based execution . Prior works have explored predicting ReLU-induced zeros and maxpooling compute redundancy to lower the computational cost of CNNs . For example , Lin et al . ( 2017 ) ; Song et al . ( 2018 ) propose zero-prediction to utilize a few of the most-significant bits in the input activations to predict the sign of the output activation . Zero-prediction removes redundancy by exploiting the fact that negative outputs will be suppressed by the ReLU anyway . Yet , this method only applies to ReLU activations and only when a linear layer is directly followed by ReLU . Hence such method does not apply to RNNs that use sigmoid or tanh as the activation function and many modern CNN networks that use batch normalization ( Ioffe & Szegedy , 2015 ) before ReLU . Hua et al . ( 2019a ; b ) propose channel gating to dynamically turn off a subset of channels that contribute little to the model prediction result . Precision gating is orthogonal to this pruning technique as channel gating executes the whole network at the same precision . More recently , Cao et al . ( 2019 ) propose SeerNet , which also executes a CNN model in dual precision . For each convolutional layer , SeerNet first executes a quantized version of the layer and uses the results to predict the output sparsity induced by the ReLU or the computational sparsity induced by the max-pooling layer . For those activations that are not suppressed according to the prediction , SeerNet computes the original convolution in full-precision ( 32-bit float ) . One key dif- Input Activations = < < + ℎ ference between PG and SeerNet is that PG reuses the result of the low-precision compute as a partial product when it performs the high-precision multiplication in the update phase . In contrast , the full-precision compute in SeerNet does not reuse the output from the quantized layer , which incurs a higher execution cost . Feature-level precision tuning . There is also prior work that uses a different precision to handle the outlier quantization . Park et al . ( 2018 ) propose value-aware quantization where the majority of data are computed at reduced precision while a small number of outliers are handled at high precision . Our approach is significantly different because we allow dual precision for every feature , not only for the outliers . 3 PRECISION GATING ( PG ) . In this section we first describe the basic mechanism of PG . We then discuss how to design the gating scheme to accelerate both forward and backward passes . Finally , we consider incorporating outlier clipping to reduce the quantization error . 3.1 BASIC FORMULATION . We first define a linear layer in a neural network ( either convolutional or fully-connected ) as O = I∗W , where O , I , and W are the output , input , and weights , respectively . Suppose I is represented in a B-bit fixed-point format , which is shown in Figure 1 . PG partitions I into ( 1 ) Ihb , the Bhb most-significant bits ( MSBs ) , and ( 2 ) Ilb , the remaining Blb least-significant bits ( LSBs ) . Here B = Bhb +Blb . More formally , we can write : I = Ihb < < Blb + Ilb ( 1 ) Here < < denotes the left shift operator . We can then reformulate a singleB-bit linear layer into two lower-precision computations as : O = Ohb + Olb = [ W ∗ ( Ihb < < Blb ) ] + [ W ∗ Ilb ] ( 2 ) Ohb is the partial product obtained by using the MSBs of input feature ( Ihb ) whereas Olb represents the remaining partial product computed with Ilb . Precision gating works in two phases . In the prediction phase , PG performs the computation Ohb = W ∗ ( Ihb < < Blb ) . Output features of Ohb greater than a learnable gating threshold ∆ are considered important . In the update phase , PG computes Olb = W ∗ Ilb only for the important features and adds it to Ohb . The overall execution flow of PG is illustrated in Figure 2 , where unimportant output features only take the upper path while important ones are computed as the sum of both paths . More precisely , this can be summarized as follows : O = { Ohb Ohb ≤ ∆ Ohb + Olb Ohb > ∆ ( 3 ) In essence , PG intends to compute the majority of ( unimportant ) features with Bhb bits and only a small set of important features with B bits . The importance of each element in the output O is calculated by comparing the magnitude of its partial sum Ohb to ∆ . Let Sp be the percentage of unimportant activations over all features . PG saves Sp·BlbB fraction of the compute in the original DNN model . PG thus achieves dual-precision execution using a lightweight gating mechanism which only adds a comparison operation . The goal is to minimize the average bitwidth of the multiply-add operations in the DNN execution .

Summary: This paper presents an interesting quantization technique that is, unusually, end-to-end trainable and not just an inference technique. According to the experiments, the method achieves better performance and computational savings as compared to other quantization method baselines. The results are admirably demonstrated on a variety of models, including CNN and RNN-based neural nets, as well as on several datasets in different domains, including ImageNet, CIFAR10, and PTB. We see the method seems to generalize across all of these.