Lattice Quantization

1 INTRODUCTION . Complex tasks such as image recognition on large datasets ( Krizhevsky et al. , 2017 ; He et al. , 2015 ) and speech recognition ( Hinton et al. , 2012 ) are very efficiently solved by deep convolutional neural networks . To improve precision , deep learning models have become continually larger , more memory demanding and computationally heavier . Popular models like ResNet ( He et al. , 2015 ) use millions of parameters ( Resnet-50 uses 25Mio of them , which represents a storage of almost 100MB of data ) . However , for embedded systems applications , storing such quantities of information is often impossible in practice . Some architectures are specifically designed to reduce this memory footprint like MobileNet ( Howard et al. , 2017 ; Sandler et al. , 2018 ) ( Mobilenet-v2 requires just 14MB of memory ) . But compressing existing architectures is easier than inventing new ones for specific needs . Therefore , various methods were designed to improve memory efficiency and reduce computational complexity of these models , in order to be able to use DCNNs for low power and/or low memory applications . There are two major ways of reducing the size of a deep neural network : • Reducing the total number of parameters , through pruning ( Blalock et al. , 2020 ) , weight sharing , low-rank factorization . • Reducing the memory footprint of its parameters and operation complexity , mainly through quantization ( Krishnamoorthi , 2018 ) . Recently , quantization of popular models down to 4 and even 3-bit has been made possible , with little to no degradation of task performance ( Esser et al. , 2020 ; Jin et al. , 2019 ) . These breakthroughs often rely on retraining the model from scratch with stochastic gradient descent , using the full training dataset . This class of methods is known as quantization-aware training , and leads to the best performance in DCNNs quantization . However , using these methods can sometimes be impractical . The first drawback is the need for the full dataset . In some cases we only have access to a trained full precision network , and for confidentiality reasons , it may be impossible to access the full training dataset of this network . A second drawback is their practicality : training a quantized network from scratch can be extremely demanding in computation resources , last generation GPUs , which might not be available for the developer . To overcome these constraints , several methods have been suggested that do not rely on retraining a network from scratch . This framework is called post-training quantization . It is commonly acknowledged that post-training methods do not lead to task performance as good as quantization aware training ( Krishnamoorthi , 2018 ) , especially when dealing with bitwidths 4 or lower , but they do solve the aforementioned problems , and can be critical for rapid deployment . Inspired by Nagel et al . ( 2019 ) , we propose a classification of quantization methods given their level of complexity and data requirements , which extends their own classification . We will use this framework to help the reader compare our method with those of other authors . • Level 1a No data and no finetuning of the network ’ s parameters . “ As simple as an API call ” . The required inputs are only the model definition and its weights . ( Nagel et al. , 2019 ) . • Level 1b No data and no finetuning of the network ’ s parameters . Limited data can be used to calibrate activation quantization thresholds . Per-channel quantization is widely used . ( Banner et al. , 2019 ; Zhao et al. , 2019 ) . • Level 2a Some samples of application data are needed , no finetuning of the network ’ s parameters . Data determines the quantization function ( i.e . step , threshold , channel splitting , mixed precision ) , or updates batch normalization statistics . Per-channel and per-layer quantization coexist in the state of the art . ( Choukroun et al. , 2019 ; Liu et al. , 2021 ) . • Level 2b Some samples of application data are needed , restricted finetuning pipeline for the network ’ s parameters . Data is used for layerwise optimization ( Nagel et al. , 2020 ; Wang et al. , 2020 ) or blockwise optimization ( Li et al. , 2021 ) to improve quantization locally . From this level on , per-layer quantization is standard . • Level 3 Quantization aware training . Requires the full dataset and parameter optimization , needs finetuning and hyperparameter tuning to achieve good performance . In this paper , we introduce a new level 1b post-training quantization technique for deep convolutional neural networks , which achieves level 1b state-of-the-art classification performance on ImageNet down to 4-bit weights on Resnet architectures , with less than 1 % accuracy drop compared to full precision models . We also show good performance for 3-bit quantization with 3 % accuracy drop compared to full precision . Our method relies on a new quantizer that uses linear correlations between the parameters of convolution layers to minimize its error . The goal of this paper is not to show absolute best performance among all post-training quantization methods , but rather to show some improvement over the scalar quantizer is a standard and simple experimental setting . Our contributions . 1 . We introduce a new quantizer , based on the structure of lattices , to allow for more flexible and adaptive quantization of weights , contrary to most other techniques that rely on uniform quantizers . 2 . We explore the impact of combining parameter and activation quantization for our quantizer , to allow for faster inference and low complexity computation . 3 . We provide insights and analysis of how our quantizer works , and its memory overhead . This paper is organized as follows : section 2 presents previous work on post-training quantization as well as quantization-aware training . Section 3 analyzes the parameter correlations inside DCNNs and introduces the intuition behind our approach . In section 4 , we detail our approach and compare it to existing state-of-the art methods . In section 5 , we give additional results of our quantizer without enhancements as an ablation study . Finally , in section 6 , we provide analysis of the characteristics of our quantizer , including its quantization error , its distribution , and its memory overhead . 2 RELATED WORK . The performance of DCNN quantization schemes depends heavily on hypotheses : availability of training data , computation time and hardware capacities . The best results are currently achieved by quantization-aware training , which is the least restrictive : unlimited access to training data , unlimited computation time and resources , heavy hyperparameter finetuning and human expertise . Recently , promising results in this area have been achieved by Esser et al . ( 2020 ) , Jin et al . ( 2019 ) , which both used uniform quantizers for weights and activations . Non uniform quantizers , including vector quantizers relying on clustering techniques , have also been successfully employed ( Han et al. , 2016 ; Stock et al. , 2020 ) . Post-training quantization meets stricter specifications than quantization-aware training . Nagel et al . ( 2019 ) for example assumed that no data is available at all for quantization . Banner et al . ( 2019 ) introduced bias correction and per-channel bit allocation . Choukroun et al . ( 2019 ) specifically designed a quantizer to minimize the MSE loss of the quantization operation . Other approaches that use a few samples of data have been proposed ( Nagel et al. , 2020 ) , which used data in order to learn whether to round the weights up or down . Wang et al . ( 2020 ) proceeded by bit optimization , and Liu et al . ( 2021 ) exploited multipoint quantization to approximate full precision weight vectors with a linear combination of low bit vectors . 3 PRELIMINARY OBSERVATIONS . Most post-training schemes try to optimize task performance by adapting their quantizer to better fit the typically bell-shaped density function of the model weights . Moreover , these scalar methods do not pay attention to potential correlations between these parameters . As can be seen on Figure 1 , correlations may exist between the weight values of trained DCNNs . Figure 2 is the correlation matrix of the 9 distributions of filter parameters in a 3 × 3 convolution layer . The first distribution is the one of weights located in the upper left corner of a filter , the second distribution is the one of weights located in the upper center of a filter , etc . Formally , we plot in line i and column j the points ( wi , wj ) for each filter f = ( w1 , ... , w9 ) in a chosen layer . On the long diagonal , we plot the histogram of wi . We notice that filters tend to have correlated coordinates , with quite high correlation coefficients . The same observation can be made in other 3 × 3 layers . From this observation , we justify the main assumption of our method : a quantizer “ shaped as a parallelogram ” is more data efficient than a uniform quantizer “ shaped as a square ” . 4 METHODS . Many state-of-the-art quantization methods rely on scalar uniform quantizers , which require only a few additional parameters : step size , bitwidth , and/or threshold . Other approaches use vector quantization , which may either be scalar or multidimensional , leading to more flexible quantization sets ( Han et al. , 2016 ; Stock et al. , 2020 ) but are often limited by the need to use codebooks . LatticeQ takes the best of both approaches . It adds a limited number of additional parameters to encode quantization bases , and offers a broader variety of possible quantization sets than scalar uniform quantizers . 4.1 LATTICE BASED WEIGHT QUANTIZER . In order to quantize the weights , LatticeQ uses lattices , which are algebraic structures that discretize the notion of vector space ( see APPENDIX A for more details ) . Each lattice has a basis , meaning that each point of the lattice can be written as an integer linear combination of the vectors of this basis . This integer linear combination is the encoding of our quantization . A lattice has infinite cardinality . In order to use this structure as our quantizer we need a finite number of quantization points . We truncate our lattice in the following fashion : let Λ be a lattice , and B = ( bi ) 1≤i≤n ∈ Rn a basis of Λ . Given b the bitwidth , our quantization set is Q = { q ∈ Rn , q = ∑n i=1 µibi , ∀i ∈ { 1 , ... , n } , −2b−1 ≤ µi ≤ 2b−1 − 1 } . In memory , each quantized block is represented by its coordinates in the quantization basis . Let us suppose B = ( b1 , b2 , b3 ) = ( ( b1,1 , b1,2 , b1,3 ) , ( b2,1 , b2,2 , b2,3 ) , ( b3,1 , b3,2 , b3,3 ) ) is our quantization basis ( with each bi , j a scalar , possibly quantized with a uniform min/max quantizer ) , and : F q = ( f1 f2 f3 f4 f5 f6 f7 f8 f9 ) is a 3×3 quantized filter . It consists in 3 quantized blocks : ( f1 , f2 , f3 ) , ( f4 , f5 , f6 ) and ( f7 , f8 , f9 ) . The convolution represented by F q is the concatenation of three 1× 3 vectors : F = Dequant ( F q ) = ( ( f1b1 + f2b2 + f3b3 ) ( f4b1 + f5b2 + f6b3 ) ( f7b1 + f8b2 + f9b3 ) ) Note that if we choose B as a uniform scaling of an orthonormal basis ( B = λ×In ) , the quantization set is exactly the one of classic scalar quantization ( we call this simplified version “ Cubic LatticeQ ” ) . The quantization process for a 3 × 3 layer is simple : we flatten the weights and group them by blocks of 3 . Then , using the quantization basis , we search for the quantization point nearest to this block . The vector found is the quantization point for this block . In practice , we can use Dequant to calculate the convolution in the quantized network . However , for per-layer quantization , we propose an optimized way of dealing with the inference in a LatticeQ-quantized network in APPENDIX E. Quantizing literally means defining a function from an infinite set to a finite set . In the scalar model , the quantization operation relies on a simple round function . In order to quantize on a lattice , we need to use a more complex algorithm . Actually , the problem of finding the closest lattice point to a real vector is known to be NP-hard ( Micciancio , 1998 ) , and is called Closest Vector Problem ( CVP ) : “ Given x ∈ Rn and Λ a lattice of Rn , find λ ∈ Λ such that d ( x , λ ) = min { d ( x , l ) , l ∈ Λ } . ” Algorithm 1 Nearest plane algorithm 1 : function BABAI ( Basis B , Vector t ) 2 : B∗ = GramSchmidt ( B ) 3 : b← t 4 : for j ∈ { n , ... , 1 } do 5 : uj ← < b , b∗j > < b∗j , b ∗ j > 6 : b← b− bujebj 7 : end for 8 : return x = ∑n j=1bujebj = t− b 9 : end function Algorithm 2 Gram-Schmidt algorithm 1 : function GRAMSCHMIDT ( Basis B ) 2 : b∗1 ← b1 3 : for j ∈ { 2 , ... , n } do 4 : b∗j ← bj − ∑j−1 i=1 < bj , b ∗ i > b ∗ i 5 : end for 6 : return B∗ = ( b∗j ) 1≤j≤n 7 : end function In order to solve the closest vector problem , we use the nearest plane algorithm ( Babai , 1986 ) . It is fairly easy to understand and implement , computationally light , and still provides good approximations in low dimension . We detail the implementation in algorithms 1 and 2 . Now that we have an algorithm to quantize on a lattice , we want to find a good lattice . For the data free approach , we opt for a simple random search with restart as described in algorithm 3 . Restarts simply consist in running the algorithm several times in a row and keeping the best result of all the runs . We look for a lattice that reduces the mean cube error loss ( MCE ) between the full precision weights of the layer ( or channel ) , and their quantized version . We chose this loss rather than MSE because it performed slightly better ( Table 1 ) . We assume this is because the MCE loss gives more importance to larger weights that are far away from any quantization point compared to the MSE . Algorithm 3 FindBasis 1 : function FINDBASIS ( Weight tensor W , Basis dimension dim , Bitwidth bits , Temperatures T ) 2 : B ← Idim 3 : for 0 ≤ s ≤ |T | do 4 : B′ ← B′ + Sample ( G ( 0 , σ = T ( s ) , |B| ) ) 5 : if MCELoss ( W , W qΛB′ ) < MCELoss ( W , W q ΛB ) then 6 : B ← B′ 7 : end if 8 : end for 9 : return B 10 : end function The random transformation we use is a multivariate gaussian noise addition with standard deviation T ( s ) , where T is a sequence of temperature steps , that can be adjusted as a hyperparameter , see APPENDIX B . In order to perform low bit computations , we may want to quantize the basis itself to integer values , which we did in all our experiments , by adding a simple min/max quantizer for B′ between line 4 and line 5 in algorithm 3 . Since the approximation factor of the nearest plane algorithm grows in square root in the basis dimension n ( Babai , 1986 ) , there is no interest in choosing high dimensional bases . Moreover , it increases the storage requirements , since an n-dimensional basis requires n2 entries in memory . Experimentally , we notice that the most accurate dimension for the quantization basis is 3 for 3×3 layers ( Table 2 ) , and 2 for 1×1 layers .

Summary: LATTICE QUANTIZATION provides a novel way to quantize weights.  This paper relies on weight correlation inside convolution filters so we can select better basis to represent weights (so called "Lattice quantization").  Firstly authors quantize numbers to lattice and then find an good lattice.  Due to weight correlation, a good lattice quantization can be more efficient than conventional uniform number representation (Cubic LatticeQ).    In the same bit precision, Lattice quantization achieved less quantization error in convolution layers.  Less quantization in Lattice quantization also translates to better Neural Network accuracy (compared to conventional scalar uniform quantization).