Lattice Quantization

1 INTRODUCTION . Complex tasks such as image recognition on large datasets ( Krizhevsky et al. , 2017 ; He et al. , 2015 ) and speech recognition ( Hinton et al. , 2012 ) are very efficiently solved by deep convolutional neural networks . To improve precision , deep learning models have become continually larger , more memory demanding and computationally heavier . Popular models like ResNet ( He et al. , 2015 ) use millions of parameters ( Resnet-50 uses 25Mio of them , which represents a storage of almost 100MB of data ) . However , for embedded systems applications , storing such quantities of information is often impossible in practice . Some architectures are specifically designed to reduce this memory footprint like MobileNet ( Howard et al. , 2017 ; Sandler et al. , 2018 ) ( Mobilenet-v2 requires just 14MB of memory ) . But compressing existing architectures is easier than inventing new ones for specific needs . Therefore , various methods were designed to improve memory efficiency and reduce computational complexity of these models , in order to be able to use DCNNs for low power and/or low memory applications . There are two major ways of reducing the size of a deep neural network : • Reducing the total number of parameters , through pruning ( Blalock et al. , 2020 ) , weight sharing , low-rank factorization . • Reducing the memory footprint of its parameters and operation complexity , mainly through quantization ( Krishnamoorthi , 2018 ) . Recently , quantization of popular models down to 4 and even 3-bit has been made possible , with little to no degradation of task performance ( Esser et al. , 2020 ; Jin et al. , 2019 ) . These breakthroughs often rely on retraining the model from scratch with stochastic gradient descent , using the full training dataset . This class of methods is known as quantization-aware training , and leads to the best performance in DCNNs quantization . However , using these methods can sometimes be impractical . The first drawback is the need for the full dataset . In some cases we only have access to a trained full precision network , and for confidentiality reasons , it may be impossible to access the full training dataset of this network . A second drawback is their practicality : training a quantized network from scratch can be extremely demanding in computation resources , last generation GPUs , which might not be available for the developer . To overcome these constraints , several methods have been suggested that do not rely on retraining a network from scratch . This framework is called post-training quantization . It is commonly acknowledged that post-training methods do not lead to task performance as good as quantization aware training ( Krishnamoorthi , 2018 ) , especially when dealing with bitwidths 4 or lower , but they do solve the aforementioned problems , and can be critical for rapid deployment . Inspired by Nagel et al . ( 2019 ) , we propose a classification of quantization methods given their level of complexity and data requirements , which extends their own classification . We will use this framework to help the reader compare our method with those of other authors . • Level 1a No data and no finetuning of the network ’ s parameters . “ As simple as an API call ” . The required inputs are only the model definition and its weights . ( Nagel et al. , 2019 ) . • Level 1b No data and no finetuning of the network ’ s parameters . Limited data can be used to calibrate activation quantization thresholds . Per-channel quantization is widely used . ( Banner et al. , 2019 ; Zhao et al. , 2019 ) . • Level 2a Some samples of application data are needed , no finetuning of the network ’ s parameters . Data determines the quantization function ( i.e . step , threshold , channel splitting , mixed precision ) , or updates batch normalization statistics . Per-channel and per-layer quantization coexist in the state of the art . ( Choukroun et al. , 2019 ; Liu et al. , 2021 ) . • Level 2b Some samples of application data are needed , restricted finetuning pipeline for the network ’ s parameters . Data is used for layerwise optimization ( Nagel et al. , 2020 ; Wang et al. , 2020 ) or blockwise optimization ( Li et al. , 2021 ) to improve quantization locally . From this level on , per-layer quantization is standard . • Level 3 Quantization aware training . Requires the full dataset and parameter optimization , needs finetuning and hyperparameter tuning to achieve good performance . In this paper , we introduce a new level 1b post-training quantization technique for deep convolutional neural networks , which achieves level 1b state-of-the-art classification performance on ImageNet down to 4-bit weights on Resnet architectures , with less than 1 % accuracy drop compared to full precision models . We also show good performance for 3-bit quantization with 3 % accuracy drop compared to full precision . Our method relies on a new quantizer that uses linear correlations between the parameters of convolution layers to minimize its error . The goal of this paper is not to show absolute best performance among all post-training quantization methods , but rather to show some improvement over the scalar quantizer is a standard and simple experimental setting . Our contributions . 1 . We introduce a new quantizer , based on the structure of lattices , to allow for more flexible and adaptive quantization of weights , contrary to most other techniques that rely on uniform quantizers . 2 . We explore the impact of combining parameter and activation quantization for our quantizer , to allow for faster inference and low complexity computation . 3 . We provide insights and analysis of how our quantizer works , and its memory overhead . This paper is organized as follows : section 2 presents previous work on post-training quantization as well as quantization-aware training . Section 3 analyzes the parameter correlations inside DCNNs and introduces the intuition behind our approach . In section 4 , we detail our approach and compare it to existing state-of-the art methods . In section 5 , we give additional results of our quantizer without enhancements as an ablation study . Finally , in section 6 , we provide analysis of the characteristics of our quantizer , including its quantization error , its distribution , and its memory overhead . 2 RELATED WORK . The performance of DCNN quantization schemes depends heavily on hypotheses : availability of training data , computation time and hardware capacities . The best results are currently achieved by quantization-aware training , which is the least restrictive : unlimited access to training data , unlimited computation time and resources , heavy hyperparameter finetuning and human expertise . Recently , promising results in this area have been achieved by Esser et al . ( 2020 ) , Jin et al . ( 2019 ) , which both used uniform quantizers for weights and activations . Non uniform quantizers , including vector quantizers relying on clustering techniques , have also been successfully employed ( Han et al. , 2016 ; Stock et al. , 2020 ) . Post-training quantization meets stricter specifications than quantization-aware training . Nagel et al . ( 2019 ) for example assumed that no data is available at all for quantization . Banner et al . ( 2019 ) introduced bias correction and per-channel bit allocation . Choukroun et al . ( 2019 ) specifically designed a quantizer to minimize the MSE loss of the quantization operation . Other approaches that use a few samples of data have been proposed ( Nagel et al. , 2020 ) , which used data in order to learn whether to round the weights up or down . Wang et al . ( 2020 ) proceeded by bit optimization , and Liu et al . ( 2021 ) exploited multipoint quantization to approximate full precision weight vectors with a linear combination of low bit vectors . 3 PRELIMINARY OBSERVATIONS . Most post-training schemes try to optimize task performance by adapting their quantizer to better fit the typically bell-shaped density function of the model weights . Moreover , these scalar methods do not pay attention to potential correlations between these parameters . As can be seen on Figure 1 , correlations may exist between the weight values of trained DCNNs . Figure 2 is the correlation matrix of the 9 distributions of filter parameters in a 3 × 3 convolution layer . The first distribution is the one of weights located in the upper left corner of a filter , the second distribution is the one of weights located in the upper center of a filter , etc . Formally , we plot in line i and column j the points ( wi , wj ) for each filter f = ( w1 , ... , w9 ) in a chosen layer . On the long diagonal , we plot the histogram of wi . We notice that filters tend to have correlated coordinates , with quite high correlation coefficients . The same observation can be made in other 3 × 3 layers . From this observation , we justify the main assumption of our method : a quantizer “ shaped as a parallelogram ” is more data efficient than a uniform quantizer “ shaped as a square ” . 4 METHODS . Many state-of-the-art quantization methods rely on scalar uniform quantizers , which require only a few additional parameters : step size , bitwidth , and/or threshold . Other approaches use vector quantization , which may either be scalar or multidimensional , leading to more flexible quantization sets ( Han et al. , 2016 ; Stock et al. , 2020 ) but are often limited by the need to use codebooks . LatticeQ takes the best of both approaches . It adds a limited number of additional parameters to encode quantization bases , and offers a broader variety of possible quantization sets than scalar uniform quantizers . 4.1 LATTICE BASED WEIGHT QUANTIZER . In order to quantize the weights , LatticeQ uses lattices , which are algebraic structures that discretize the notion of vector space ( see APPENDIX A for more details ) . Each lattice has a basis , meaning that each point of the lattice can be written as an integer linear combination of the vectors of this basis . This integer linear combination is the encoding of our quantization . A lattice has infinite cardinality . In order to use this structure as our quantizer we need a finite number of quantization points . We truncate our lattice in the following fashion : let Λ be a lattice , and B = ( bi ) 1≤i≤n ∈ Rn a basis of Λ . Given b the bitwidth , our quantization set is Q = { q ∈ Rn , q = ∑n i=1 µibi , ∀i ∈ { 1 , ... , n } , −2b−1 ≤ µi ≤ 2b−1 − 1 } . In memory , each quantized block is represented by its coordinates in the quantization basis . Let us suppose B = ( b1 , b2 , b3 ) = ( ( b1,1 , b1,2 , b1,3 ) , ( b2,1 , b2,2 , b2,3 ) , ( b3,1 , b3,2 , b3,3 ) ) is our quantization basis ( with each bi , j a scalar , possibly quantized with a uniform min/max quantizer ) , and : F q = ( f1 f2 f3 f4 f5 f6 f7 f8 f9 ) is a 3×3 quantized filter . It consists in 3 quantized blocks : ( f1 , f2 , f3 ) , ( f4 , f5 , f6 ) and ( f7 , f8 , f9 ) . The convolution represented by F q is the concatenation of three 1× 3 vectors : F = Dequant ( F q ) = ( ( f1b1 + f2b2 + f3b3 ) ( f4b1 + f5b2 + f6b3 ) ( f7b1 + f8b2 + f9b3 ) ) Note that if we choose B as a uniform scaling of an orthonormal basis ( B = λ×In ) , the quantization set is exactly the one of classic scalar quantization ( we call this simplified version “ Cubic LatticeQ ” ) . The quantization process for a 3 × 3 layer is simple : we flatten the weights and group them by blocks of 3 . Then , using the quantization basis , we search for the quantization point nearest to this block . The vector found is the quantization point for this block . In practice , we can use Dequant to calculate the convolution in the quantized network . However , for per-layer quantization , we propose an optimized way of dealing with the inference in a LatticeQ-quantized network in APPENDIX E. Quantizing literally means defining a function from an infinite set to a finite set . In the scalar model , the quantization operation relies on a simple round function . In order to quantize on a lattice , we need to use a more complex algorithm . Actually , the problem of finding the closest lattice point to a real vector is known to be NP-hard ( Micciancio , 1998 ) , and is called Closest Vector Problem ( CVP ) : “ Given x ∈ Rn and Λ a lattice of Rn , find λ ∈ Λ such that d ( x , λ ) = min { d ( x , l ) , l ∈ Λ } . ” Algorithm 1 Nearest plane algorithm 1 : function BABAI ( Basis B , Vector t ) 2 : B∗ = GramSchmidt ( B ) 3 : b← t 4 : for j ∈ { n , ... , 1 } do 5 : uj ← < b , b∗j > < b∗j , b ∗ j > 6 : b← b− bujebj 7 : end for 8 : return x = ∑n j=1bujebj = t− b 9 : end function Algorithm 2 Gram-Schmidt algorithm 1 : function GRAMSCHMIDT ( Basis B ) 2 : b∗1 ← b1 3 : for j ∈ { 2 , ... , n } do 4 : b∗j ← bj − ∑j−1 i=1 < bj , b ∗ i > b ∗ i 5 : end for 6 : return B∗ = ( b∗j ) 1≤j≤n 7 : end function In order to solve the closest vector problem , we use the nearest plane algorithm ( Babai , 1986 ) . It is fairly easy to understand and implement , computationally light , and still provides good approximations in low dimension . We detail the implementation in algorithms 1 and 2 . Now that we have an algorithm to quantize on a lattice , we want to find a good lattice . For the data free approach , we opt for a simple random search with restart as described in algorithm 3 . Restarts simply consist in running the algorithm several times in a row and keeping the best result of all the runs . We look for a lattice that reduces the mean cube error loss ( MCE ) between the full precision weights of the layer ( or channel ) , and their quantized version . We chose this loss rather than MSE because it performed slightly better ( Table 1 ) . We assume this is because the MCE loss gives more importance to larger weights that are far away from any quantization point compared to the MSE . Algorithm 3 FindBasis 1 : function FINDBASIS ( Weight tensor W , Basis dimension dim , Bitwidth bits , Temperatures T ) 2 : B ← Idim 3 : for 0 ≤ s ≤ |T | do 4 : B′ ← B′ + Sample ( G ( 0 , σ = T ( s ) , |B| ) ) 5 : if MCELoss ( W , W qΛB′ ) < MCELoss ( W , W q ΛB ) then 6 : B ← B′ 7 : end if 8 : end for 9 : return B 10 : end function The random transformation we use is a multivariate gaussian noise addition with standard deviation T ( s ) , where T is a sequence of temperature steps , that can be adjusted as a hyperparameter , see APPENDIX B . In order to perform low bit computations , we may want to quantize the basis itself to integer values , which we did in all our experiments , by adding a simple min/max quantizer for B′ between line 4 and line 5 in algorithm 3 . Since the approximation factor of the nearest plane algorithm grows in square root in the basis dimension n ( Babai , 1986 ) , there is no interest in choosing high dimensional bases . Moreover , it increases the storage requirements , since an n-dimensional basis requires n2 entries in memory . Experimentally , we notice that the most accurate dimension for the quantization basis is 3 for 3×3 layers ( Table 2 ) , and 2 for 1×1 layers .

Summary: The paper explores the post-training inference quantization. Observing that the weight distributions between filters are often highly correlated, the authors propose a new quantizer based on algorithmic theory of lattices for the quantization of weight tensors. The method is verified on popular vision models with precision down to 4/3 bits.