Precision Highway for Ultra Low-precision Quantization

1 INTRODUCTION . Energy-efficient inference of neural networks is becoming increasingly important in both servers and mobile devices ( e.g. , smartphones , AR/VR devices , and drones ) . Recently , there have been active studies on ultra-low-precision inference using 1 to 4 bits ( Rastegari et al. , 2016 ; Hubara et al. , 2016 ; Zhu et al. , 2016 ; Zhou et al. , 2016 ; 2017 ; Zhuang et al. , 2018 ; Choi et al. , 2018b ; Liu et al. , 2018 ; Choi et al. , 2018a ) and their implementations on CPU and GPU ( Tulloch & Jia , 2017 ) , and dedicated hardware ( Park et al. , 2018 ; Sharma et al. , 2017 ) . However , as will be explained in section 5.2 , the existing quantization methods suffer from a problem called accumulated quantization error where large quantization errors get accumulated across layers , making it difficult to enable ultra-low precision in deep neural networks . In order to address this problem , we propose a novel concept called precision highway where an end-to-end path of high-precision information reduces the accumulated quantization error thereby enabling ultra-low-precision computation . Our proposed work is similar to recent studies ( Liu et al. , 2018 ; Choi et al. , 2018b ) which propose utilizing pre-activation residual networks , where skip connections are kept in full precision while the residual path performs low-precision computation . Compared with these works , our proposed method offers a generalized concept of high-precision information flow , namely , precision highway , which can be applied to not only the pre-activation convolutional networks but also both the post-activation convolutional and recurrent neural networks . Our contributions are as follows . • We propose a novel idea of network-level approach to quantization , called precision highway and quantitatively analyze its benefits in terms of the propagation of quantization errors and the difficulty of convergence in training based on the shape of loss surface . • We provide the detailed analysis of the energy and memory overhead of precision highway based on the state-of-the-art hardware accelerator model . According to our experiments , the overhead is negligible while offering significant improvements in accuracy . • We apply precision highway to both convolution and recurrent networks . We report a 3-bit quantization of ResNet-50 without accuracy loss and a 2-bit quantization with a very small accuracy loss . We also provide the sub 4-bit quantization results of long short-term memory ( LSTM ) for language modeling . 2 RELATED WORK . Migacz ( 2017 ) presented an int8 quantization method that selects an activation truncation threshold to minimize the Kullback-Leibler divergence between the distributions of the original and quantized data . Jacob et al . ( 2017 ) proposed a quantization scheme that enables integer-arithmetic only matrix multiplications ( practically , 8-bit quantization for neural networks ) . These methods are implemented on existing CPUs or GPUs ( Jacob et al. , 2015 ; ACL ) . Hubara et al . ( 2016 ) presented a binarization method and demonstrated the performance benefit on a GPU . Rastegari et al . ( 2016 ) proposed a binary network called XNOR-Net in which a weight-binarized AlexNet gives the same accuracy as a full-precision one . Zhou et al . ( 2016 ) presented DoReFa-Net , which applies tanh-based weight quantization and bounded activation . Zhou et al . ( 2017 ) proposed a balanced quantization that attempts to balance the population of values on quantization levels . He et al . ( 2016b ) proposed utilizing full precision for internal cell states in the LSTM because of their wide value distributions . This work is similar to ours in that high-precision data are selectively utilized to improve the quantized network . Our difference is proposing a network-level end-to-end flow of high-precision activation . Recently , Zhuang et al . ( 2018 ) presented 4-bit quantization with ResNet-50 . They adopt Dorefa-net style weight quantization with static bounded activation , and improve accuracy by adopting multi-step quantization and knowledge distillation during fine-tuning . Mishra et al . ( 2017 ) proposed a trade-off between the number of channels and precision . Clustering-based methods have the potential to further reduce the precision ( Han et al. , 2015 ) . However , they require a lookup table and full-precision computation , which makes them less hardware-friendly . Recently , Choi et al . ( 2018b ; a ) and Liu et al . ( 2018 ) proposed utilizing full-precision on the skip connections in pre-activation residual networks1 . Compared with those works , our proposed idea has a salient difference in that it offers a network-level solution and demonstrates that the end-to-end flow of high-precision information is crucial . In addition , our method is not limited to pre-activation residual networks , but general enough to be applied to both post-activation convolutional and recurrent neural networks . 3 PROPOSED METHOD . In precision highway , we build a path from the input to output of a network to enable the end-toend flow of high-precision activation , while performing low-precision computation . Our proposed method was motivated ( 1 ) by a residual network where the signal , i.e. , the activation/gradient in a forward/backward pass , can be directly propagated from one block to another ( He et al. , 2016a ) and ( 2 ) by the LSTM , which provides an uninterrupted gradient flow across time steps via the inter-cell state path ( Gers et al. , 2000 ) . Our proposed method focuses instead on improving the accuracy of quantized network by providing an end-to-end high-precision information flow . In this section , we first describe the precision highway in the cases of residual network ( section 3.1 ) and recurrent neural network ( section 3.2 ) . Then , we discuss practical issues to be addressed before application to other networks in section 3.3 . 3.1 PRECISION HIGHWAY ON RESIDUAL NETWORK . In the case of a residual network , we can form a precision highway by making high-precision skip connections . In this subsection , we explain how high-precision skip connections can be constructed to reduce the accumulated quantization error . 1This work was conducted in parallel with Choi et al . ( 2018b ; a ) ; Liu et al . ( 2018 ) In the conventional residual block shown in Figure 1 ( a ) , quantization ( denoted as Q [ 0,1 ] k , k-bit linear quantization in range from 0 to 1 ) is applied to all of the activations after the activation function . In the figure , thick ( thin ) arrows represent high-precision ( low-precision ) activations . As the figure shows , the input of a residual block is first quantized , and the quantized input ( x+ e in the figure ) , which contains the quantization error e , enters both the skip connection and residual path . The output of a residual block , y , is calculated as follows : y = F ( x+ e ) + x+ e = F ( x ) + x+ er + e , ( 1 ) where F ( ) represents a residual function ( typically , 2 or 3 consecutive convolutional layers ) . For simplicity of explanation , we assume that F ( x+ e ) can be decomposed into F ( x ) + er , where er represents the resulting quantization error of the residual path incurred by the quantization operations on the residual path as well as the quantization error in the input , e. As the equation shows , output y has two quantization error terms , that of residual path , er , and that of the skip connection , e. Figure 1 ( b ) shows our idea of high-precision skip connection . Compared with Figure 1 ( a ) , the difference is the location of the first quantization operation in the residual block . In Figure 1 ( b ) , quantization is applied only to the residual path after the bifurcation to the residual path and skip connection . As shown in the figure , the skip connection now becomes a thick arrow , i.e. , a high-precision path . The proposed idea gives the output of the residual block as follows : y = F ( x+ e ) + x = F ( x ) + x+ er . ( 2 ) As Equation 2 shows , the proposed idea eliminates the quantization error of skip connection e. Thus , only the quantization error of the residual path er remains in the output of the residual block . Note that all of the input activations of the residual path are kept in low precision . It enables us to perform low-precision convolution operations in the residual path . We keep high-precision activation only on the skip connection and utilize it only for the element-wise addition . As will be shown in our experiments , the overhead of computation and memory access cost is small since the element-wise addition is much less expensive than the convolution on the residual path , and the low-precision activation is accessed for the computation on the residual path . As will be shown later , our method gives a smaller quantization error , and the gap between the quantization error of the existing method and that of ours becomes wider across layers . Because of the reduction of the accumulated quantization error , the proposed method offers much better accuracy than the state-of-the-art methods with an ultra-low precision of 2 and 3 bits . Note also that , as shown in figure 1 ( c ) , our idea can be applied to other types of residual blocks , including the full pre-activation residual block ( He et al. , 2016a ) as proposed in some recent works ( Choi et al. , 2018a ; Liu et al. , 2018 ) . However , our idea is general in that it is applicable to recurrent networks as well as post-activation convolutional networks . Especially , our proposed idea is advantageous over the existing ones since hardware accelerators tend to be designed assuming as the input non-negative input activations enabled by ReLU activation functions ( Park et al. , 2018 ; Kim et al. , 2018 ) . Contrary to the existing works ( Choi et al. , 2018a ; Liu et al. , 2018 ) , we provide a detailed analysis of the effect of precision highway . 3.2 PRECISION HIGHWAY ON RECURRENT NEURAL NETWORK . Figure 2 illustrates how the precision highway can be constructed on the LSTM ( Gers et al. , 2000 ) . In time step t , the LSTM cell takes , as an input , new input xt , along with the results of the previous time step , output ht−1 and cell state ct−1 . First , it calculates four intermediate signals : i ( input gate ) , f ( forget gate ) , g ( gate gate ) , and o ( output gate ) . Then , it produces two results , ct and ht , as follows : it = σ ( Wiixt + bii +Whih ( t−1 ) + bhi ) , ( 3a ) ft = σ ( Wifxt + bif +Whfh ( t−1 ) + bhf ) , ( 3b ) gt = tanh ( Wigxt + big +Whgh ( t−1 ) + bhg ) , ( 3c ) ot = σ ( Wioxt + bio +Whoh ( t−1 ) + bho ) , ( 3d ) ct = ft c ( t−1 ) + it gt , ( 3e ) ht = ot tanh ( ct ) , ( 3f ) where σ represents a sigmoid function , the element-wise multiplication , W the weight matrix , and b the bias . In the conventional LSTM operation , as Figure 2 ( a ) shows , the quantization ( gray box denoted by Qk with the output value range as the superscript ) is applied to all of the activations before computation . The results of a time step , ct and ht , are calculated based on such inputs with quantization errors . More specifically , cell state ct is calculated with the quantized , i.e. , low-precision , inputs of ct−1 , f , i , and g. Thus , cell state ct accumulates the quantization errors of those inputs . In addition , output ht also accumulates the quantization errors from its inputs , ct and o . Then , they are propagated to the next time steps . Thus , we have the problem of accumulated quantization error across the time steps . Such an accumulation of quantization error will prevent us from achieving ultra-low precision . Figure 2 ( b ) shows how we can build the precision highway in the LSTM cell . The figure shows that the quantization operation is applied only to the inputs of matrix multiplication ( a circle denoted with × in the figure ) . Thus , all of the other operations and their input activations are in high precision . Specifically , when calculating ct , the inputs are not quantized , which reduces the accumulation of quantization error on ct . The computation of ht can also reduce the accumulation of quantization error by utilizing high-precision inputs . The construction of such a precision highway allows us to propagate high-precision information , i.e. , cell states ct and outputs ht , across time steps . Note that we benefit from low-precision computation by performing low-precision matrix multiplications ( in Equations 3a-3d ) , which dominate the total computation cost . In our proposed method , all of the element-wise multiplications in Equations 3e and 3f are performed in high precision . However , the overhead of this high-precision element-wise multiplications is negligible compared with the matrix multiplication in Equations 3a-3d . In addition , this method can be applied to other types of recurrent neural networks . For instance , the GRU ( Chung et al. , 2014 ) can be equipped with a precision highway , in a way similar to that shown in Figure 2 ( b ) , by keeping high-precision output ht while performing low-precision matrix multiplications and high-precision element-wise multiplications .

Summary: This paper studies methods to improve the performance of quantized neural networks.  The paper is largely centered around the idea of "precision highways" (full-precision residual connections) that run in parallel to fully-quantized convolutions.  However, the paper also throws in a toolbox of other methods like distillation from a teacher network, a quantization method based on the Laplace distribution, and a fine tuning scheme.