OPTIMAL BINARY QUANTIZATION FOR DEEP NEURAL NETWORKS

1 INTRODUCTION . A major challenge in the deployment of Deep Neural Networks ( DNNs ) is their high computational cost . Finding effective methods to improve run-time efficiency is still an area of research . We can group various approaches taken by researchers into the following three categories . Hardware optimization : Specifically designed hardwares are deployed to efficiently perform computations in ML tasks . Compiler optimization : Compression and fusion techniques coupled with efficient hardware-aware implementations , such as dense and sparse matrix-vector multiplication , are used . Model optimization : Run-time performance can also be gained by modifying the model structure and the underlying arithmetic operations . While hardware and compiler optimization are typically lossless ( i.e . incur no loss in model accuracy ) , model optimization trades-off computational cost ( memory , runtime , or power ) for model accuracy . For example , by scaling the width of the network ( Zagoruyko & Komodakis , 2016 ) . The goal of model optimization is to improve the trade-off between computational cost and model accuracy . This work falls into this category . 1.1 ARCHITECTURE OPTIMIZATION . One strategy to construct efficient DNNs is to define a template from which efficient computational blocks can be generated . Multiple instantiations of these blocks are then chained together to form a DNN . SqueezeNet ( Iandola et al. , 2016 ) , MobileNets ( Howard et al. , 2017 ; Sandler et al. , 2018 ) , ShuffleNets ( Zhang et al. , 2018b ; Ma et al. , 2018 ) , and ESPNets ( Mehta et al. , 2018 ; 2019 ) fall into this category . Complementary to these methods , NASNet ( Zoph et al. , 2018 ) and EfficientNet ( Tan & Le , 2019 ) search for an optimal composition of blocks restricted to a computational budget ( e.g. , FLOPS ) by changing the resolution , depth , width , or other parameters of each layer . 1.2 PRUNING AND COMPRESSION . Several methods have been proposed to improve runtime performance by detecting and removing computational redundancy . Methods in this category include low-rank acceleration ( Jaderberg et al. , 2014 ) , the use of depth-wise convolution in Inception ( Szegedy et al. , 2015 ) , sparsification of kernels in deep compression ( Han et al. , 2015 ) , re-training redundant neurons in DSD ( Han et al. , 2016b ) , depth-wise separable convolution in Xception ( Chollet , 2017 ) , pruning redundant filters in PFA ( Suau et al. , 2018 ) , finding an optimal sub-network in lottery ticket hypothesis ( Frankle & Carbin , 2018 ) , and separating channels based on the features resolution in octave convolution ( Chen et al. , 2019 ) . While some of these compression methods can be applied to a trained network , most add training-time constraints to create a computationally efficient model . 1.3 LOW-PRECISION ARITHMETIC AND QUANTIZATION . Another avenue to improve runtime performance ( and the focus of this work ) is to use low-precision arithmetic . The idea is to use fewer bits to represent weights and activations . Some instances of these strategies already exist in AI compilers , where it is common to cast weights of a trained model from 32 bits to 16 or 8 bits . However , in general , post-training quantization reduces the model accuracy . This can be addressed by incorporating lower-precision arithmetic into the training process ( during-training quantization ) , allowing the resulting model to better adapt to the lower precision . For example , in Gupta et al . ( 2015 ) ; Jacob et al . ( 2018 ) the authors use 16 and 8 bits fixed-point representation to train DNNs . Using fewer bits results in dramatic memory savings . This has motivated research into methods that use a single bit to represent a scalar weight : In Courbariaux et al . ( 2015 ) the authors train models with weights quantized to the values in { −1 , 1 } . While this results in a high level of compression , model accuracy can drop significantly . Li et al . ( 2016 ) and Zhu et al . ( 2016 ) reduce the accuracy gap between full precision and quantized models by considering ternary quantization ( using the values in { −1 , 0 , 1 } ) , at the cost of slightly less compression . To further improve the computational efficiency , the intermediate activation tensors ( feature maps ) can also be quantized . When this is the case , an implementation can use high-performance operators that act on quantized inputs , for example a convolutional block depicted in Figure 1 ( left ) . This idea has been explored in ( Courbariaux et al. , 2016 ; Rastegari et al. , 2016 ; Zhou et al. , 2016 ; Hubara et al. , 2017 ; Mishra et al. , 2017 ; Lin et al. , 2017 ; Cai et al. , 2017 ; Ghasemzadeh et al. , 2018 ; Zhang et al. , 2018a ; Choi et al. , 2018 ) , and many other works . We call a mapping from a tensor with full precision entries to a tensor with the same shape but with values in { −1 , 1 } a binary quantization . When both weights and activations of a DNN are quantized using binary quantization , called Binary Neural Network ( BNN ) , fast and power-efficient kernels which use bitwise operations can be implemented . Observe that the inner-product between two vectors with entries in { −1 , 1 } can be written as bitwise XNor operations followed by bitcounting ( Courbariaux et al. , 2016 ) . However , the quantization of both weights and activations further reduces the model accuracy . In this work , we focus on improving the accuracy of the quantized model through improved quantization . The computational cost remains similar to the previous BNNs ( Rastegari et al. , 2016 ; Hubara et al. , 2017 ) . 1.4 MAIN CONTRIBUTIONS . In this work , we analyze the accuracy of binary quantization when applied to both weights and activations of a DNN , and propose methods to improve the quantization accuracy : • We present an analysis of the quantization error and show that scaled binary quantization is a good approximation ( Section 2 ) . • We derive the optimal 1-bit ( Section 3.1.1 ) , 2-bits ( Section 3.2.2 ) , and ternary ( Section 3.2.3 ) scaled binary quantization algorithms . • We propose a greedy k-bits quantization algorithm ( Section 3.2.4 ) . • Experiments on the ImageNet dataset show that the optimal algorithms have reduced quan- tization error , and lead to improved classification accuracy ( Section 5 ) . 2 LOW-RANK BINARY QUANTIZATION . Binary quantization ( that maps entries of a tensor to { −1 , 1 } ) of weights and activation tensors of a neural network can significantly reduce the model accuracy . A remedy to retrieve this accuracy loss is to scale the binarized tensors with few full precision values . For example , Hubara et al . ( 2017 ) learn a scaling for each channel from the parameters of batch-normalization , and Rastegari et al . ( 2016 ) scale the quantized activation tensors using the channel-wise average of pixel values . In this section , using low-rank matrix analysis , we analyze different scaling strategies . We conclude that multiplying the quantized tensor by a single scalar , which is computationally the most efficient option , has approximately the same accuracy as the more expensive alternatives . We introduce the rank-1 binary quantization– an approximation to a matrix X ∈ Rm×n : X 'X1 S , ( 1 ) where X1 ∈ Rm×n is a rank-1 matrix , S ∈ { −1 , 1 } m×n , and is element-wise multiplication ( Hadamard product ) . Note that this approximation is also defined for tensors , after appropriate reshaping . For example , for an image classification task , we can reshape the output of a layer of a DNN with shape h × w × n , where h , w , and n are height , width , and number of channels , respectively , into an m× n matrix with m = hw rows and one column per channel . We define the error of a rank-1 binary quantization as ‖X−X1 S‖F , where ‖ ‖F is the Frobenius norm . Entries of S are in { −1 , 1 } , therefore , the quantization error is equal to ‖X S−X1‖F . Note that ‖X S‖2F ( the total energy ) , which is equal to sum of the squared singular values , is the same for any S. Different choices of S change the distribution of the total energy among components of the Singular Value Decomposition ( SVD ) of X S. The optimal rank-1 binary quantization is achieved when most of the energy of X S is in its first component . In Rastegari et al . ( 2016 ) , the authors proposed to quantize the activations by applying the sign function and scale them by their channel-wise average . We can formulate this scaling strategy as a special rank-1 binary quantization X ' a1 > sign ( X ) , where ai = ∑n j=1 |Xi , j | n for 1 ≤ i ≤ m , sign ( x ) = { −1 if x < 0 1 if x ≥ 0 , ( 2 ) and 1 is an n-dimensional vector with all entries 1 . In Appendix A we show that the optimal rank-1 binary quantization is given by S = sign ( X ) and X1 = truncated1-SVD ( |X| ) , where sign ( X ) is the element-wise sign of X , and truncated1-SVD ( |X| ) = σ1u1v > 1 is the first component of the SVD of X sign ( X ) = |X| . Moreover , we empirically analyze the accuracy of the optimal rank-1 binary quantization for a random matrix X , where its entries are i.i.d . ∼ N ( 0 , 1 ) . This is a relevant example since after the application of Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) activation tensors are expected to have a similar distribution . The first singular value of |X| captures most of the energy σ21 ( |X| ) /‖X‖2F ' 0.64 , and the first left and right singular vectors are almost constant vectors . Therefore , a scalar multiple of sign ( X ) approximates X well : X ' σ1u1v > 1 sign ( X ) ' v 11 > sign ( X ) = vsign ( X ) , where v ∈ R≥0 . We use this computationally efficient approximation called scaled binary quantization . 3 SCALED BINARY QUANTIZATION . In Section 2 we showed that scaled binary quantization is a good approximation to activation and weight tensors of a DNN . Next we show how we can further improve the accuracy of scaled binary quantization using more bits . To simplify the presentation ( 1 ) we flatten matrix X ∈ Rm×n in to a vector x ∈ RN with N = mn , and ( 2 ) we assume the entries of x are different realizations of a random variable x with an underlying probability distribution p ( x ) . In practice , we compute all statistics using their unbiased estimators from vector x ( e.g. , ∑ i xi/N is an unbiased estimator of Ex∼p [ x ] ) . Furthermore , for f : R → R , we denote entrywise application of f to x by f ( x ) . The quantized approximation of x is denoted by xq , and the error ( loss ) of quantization is ‖x − xq‖2 . All optimal solutions are with respect to this error and hold for an arbitrary distribution p ( x ) .

Summary: This paper proposes a new family of quantized neural networks, where the weights are quantized based on fixed precisions. The authors proposed the optimal 1-bit/2-bit/ternary quantization schemes based on minimizing the L_2 loss. The authors proposed a greedy algorithm to approximate the optimal k-bit quantization.  The authors showed through extensive experiments on real DNNs that the proposed optimal quantization (Opt in the tables) can give better generalization as compared to several state-of-the-art alternative methods.