Post-Training Weighted Quantization of Neural Networks for Language Models

1 INTRODUCTION . Training techniques for deep neural networks ( DNNs ) have been developed in ways to incur a lot of parameter redundancy to expedite seeking local minima ( Denil et al. , 2013 ; Jonathan Frankle , 2019 ) . As a result , various model compression techniques including parameter pruning ( Han et al. , 2015 ; He et al. , 2017 ) , quantization ( Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ) , low-rank approximation ( N. Sainath et al. , 2013 ; Prabhavalkar et al. , 2016 ) , and knowledge distillation ( Hinton et al. , 2015 ; Polino et al. , 2018 ) are proposed to lower storage requirements and improve inference performance . Several compression techniques can be combined in a synergistic way to enhance compression ratio ( Han et al. , 2016 ; Zhu et al. , 2017 ) . In this work , we consider parameter quantization that maintains structured model formats and presents high compression ratio . Note that due to limited hardware resources , quantization is an essential method for any inference systems . In general , quantization is classified into uniform quantization based on fixed-point parameter representations ( Jacob et al. , 2018 ; Han et al. , 2016 ) and non-uniform quantization associated with the binary codes ( Zhou et al. , 2017 ; Rastegari et al. , 2016 ) or codebooks ( Choi et al. , 2017 ; Stock et al. , 2020 ) . Most DNN quantization methods are performed based on the principle of minimizing the mean squared error ( MSE ) of quantized parameters ( Rastegari et al. , 2016 ; Xu et al. , 2018 ; Zhou et al. , 2017 ) . Optimizing the MSE is also an underlying principle of low-rank approximation techniques such as the singular value decomposition ( SVD ) ( Prabhavalkar et al. , 2016 ; N. Sainath et al. , 2013 ) . Note that , however , minimizing the MSE implies that each parameter is equally important ( i.e. , squared errors from parameters are accumulated without considering importance of each weight ) . In practice , the impact of each parameter perturbation from quantization on training loss can be vastly different and such impact needs to be analyzed through a sensitivity study of each parameter toward a change in training loss value . In other words , minimizing the MSE ( or the Euclidean distance between original parameters and quantized parameters ) may not correspond to minimizing training loss function after quantization . Robustness to quantization error of each parameter can be expressed as sensitivity . Sensitivity of i-th parameter wi is the amount of change in the loss function when wi is perturbed . A parameter associated with high sensitivity would require relatively smaller quantization error if quantization is performed in a group manner . Several previous works acknowledge distinct sensitivity of each parameter to improve quantization quality . Note that because exact sensitivity estimation of each parameter toward loss function is highly complicated , various heuristic techniques have been introduced . For example , Hessian-weighted k-means clustering is used for codebook-based implementations ( Choi et al. , 2017 ) or Taylor series expansion to bound loss function difference is conducted to decide the optimal quantization bits of each weight ( Khoram & Li , 2018 ) . The Hessian matrix can be used to assign different numbers of quantization bits for each layer ( Dong et al. , 2019 ; Shen et al. , 2019 ) . Minimizing the reconstruction error on the output activations after each layer quantization is performed in ( Stock et al. , 2020 ) . In this paper , we propose a weighted quantization framework where quantized parameters follow the structure of the binary codes so as to achieve high compression ratio and high computational efficiency ( Rastegari et al. , 2016 ; Jeon et al. , 2020 ) . Specifically , given that an importance of each parameter is represented as a real number between 0 and 1 , we extract an optimal quantization solution modified from the previous binary-coding-based quantization methods that employ equal parameter importance . Similar to previous attempts , we also find that calculating accurate importance of each parameter is challenging . As a successful approximation of importance , we suggest that magnitude-based importance estimation is especially effective for post-training non-uniform quantization . 2 POST-TRAINING PARAMETER QUANTIZATION FOR LANGUAGE MODELS . The number of parameters for language models is dramatically increasing ( e.g. , GPT-3 ( Brown et al. , 2020 ) requires 175 billion parameters ) . Correspondingly , model compression for language models is becoming a mandatory process to reduce response time and inference energy . We devise a compression method considering the followings : • Recent language models are usually memory-bound because of small batch size and lacking layers of high reuse ( e.g. , conv layers ) . Thus , reducing memory footprint is critical . • Compression algorithms should be supported by dedicated kernels , designed specifically for language models if possible . • Compression-aware training is challenging and expensive if hyper-parameters are added to already huge language models ( hence , we choose a post-training method . ) Fixed-point inference using uniform quantization is not desirable for language models because of noticeable accuracy degradation ( Shen et al. , 2019 ; Jeon et al. , 2020 ) while the advantage of small computational units ( e.g. , INT8 MAC ) is insignificant for memory-bound applications . Thus , we adopt float-based parameter quantization ( i.e. , expected values of quantized parameters remains to be of full precision ) that induce a lot smaller number of quantization bits compared to fixed-point quantization ( Xu et al. , 2018 ; Stock et al. , 2020 ) . Recently , a kernel library , called BiQGEMM ( Jeon et al. , 2020 ) , was introduced to support binarycoding-based quantization techniques to accelerate quantized neural networks . Using lookup tables , BiQGEMM enables byte-level memory accesses and achieves 8.3× run-time memory footprints and 3.5× speed up with a mobile CPU for Transformer ( Chung et al. , 2020 ) . As a result , binary-codingbased quantization has become a practical approach to quantizing language models . As such , we restrict our interests to binary-coding-based quantization technique in this paper . Quantization-aware training is an active research area to improve model accuracy ( Courbariaux et al. , 2015 ; Lee et al. , 2018 ) . We note that in the case of language models , however , there are numerous occasions when retraining for quantization is not available . For example , quantizationaware training requires in-depth knowledge on model compression while model designers may not have such expertise . On the other hand , the original training code or the entire training data may not be shared with model compression engineers . Also , modifying the original DNN models to be aware of quantization would increase model design efforts and training time significantly . Since language models already demand significant training time and cost , adding additional training complexity by quantization-aware training would not be a practical option . As such , post-training quantization without retraining is gaining increasing attention ( Zhao et al. , 2019 ; Nagel et al. , 2019 ) . 3 WEIGHTED QUANTIZATION BASED ON THE BINARY CODES . As discussed , we choose post-training binary-coding-based quantization as our strategy to compress language DNN models efficiently . Following the Binary-Weight-Networks ( Rastegari et al. , 2016 ) introducing the binary codes as a quantization format , a weight vector w is approximated to be αb by using a scaling factor α ∈ R and a binary vector b ( ∈ { −1 , +1 } n ) , where n is the vector size . A real number scaling factor is shared by multiple weights such that binary vector b occupies most of the weight storage requirements . Binary codes eliminate the need for dequantization for inference , leading to reduced on-chip memory size for weights . In this section , we study general weighted quantization methodologies when quantization follows the format of the binary codes and and quantization error recognizes sensitivity information . 3.1 GREEDY METHOD AND ALTERNATING METHOD WITHOUT IMPORTANCE CONSIDERATIONS . In general , non-uniform weight quantization methods ( in the form of the binary codes ) strive to minimize ‖w − αb‖2 . In the case of 1-bit quantization , we obtain the following analytical solution : b∗ = sign ( w ) , α∗ = w > b∗ n . ( 1 ) On the other hand , in the case of multi-bit quantization , there is no analytical solution ( Rastegari et al. , 2016 ; Xu et al. , 2018 ) . As a result , various approximated methods exist for multi-bit quantization . Greedy Method As a computationally simple method , 1-bit quantization shown in Eq . ( 1 ) can be extended to multi-bit ( q-bit ) quantization ( Guo et al. , 2017 ) . Specifically , ith-bit ( i > 1 ) quantization is performed by minimizing the residue of ( i− 1 ) th-bit quantization as following : min αi , bi ‖ri−1 − αibi‖2 , where ri−1 = w − i−1∑ j=1 αjbj , 1 < i ≤ q . ( 2 ) The optimal solution of Eq . ( 2 ) is then given as b∗i = sign ( ri−1 ) , α ∗ i = r > i−1b ∗ i n . ( 3 ) Alternating Method Greedy method described above is non-iterative . In order to reduce ‖w −∑q i=1 αibi‖2 further than Greedy method , iterative methods would be necessary while increasing the number of iterations tends to lower quantization error . Once initial α and b values are calculated by Greedy method , one can notice that { αi } qi=1 can be refined ( Guo et al. , 2017 ) as [ α1 , ... , αq ] = ( ( B > q Bq ) −1 B > q w ) > , when Bq = [ b1 , ... , bq ] ∈ { −1 , +1 } n×q . ( 4 ) Then , Bq can be refined as well by binary search given a new refined { αi } qi=1 . As a result , { αi } q i=1 and Bq are refined alternatively . Alternating refinements of { αi } qi=1 and Bk are repeated until there is no noticeable improvement in quantization error . Such iterative quantization procedure is introduced as Alternating multi-bit method ( Xu et al. , 2018 ) . 3.2 IMPORTANCE-AWARE WEIGHTED QUANTIZATION . Let us assume that the importance of the i-th parameter is normalized and given as mi ( 0 ≤ mi ≤ 1 ) . Then , we minimize the weighted quantization loss ∑n i=1 ( mi ( wi− ŵi ) 2 ) where wi is quantized to be ŵi = ∑q j=1 ( αjbj ) . Before studying how to estimate importance values , we are interested in finding modified versions of Greedy method and Alternating method when importance values are given . For 1-bit quantization , weighted quantization also has the following analytical solution : b∗ = sign ( w ) , α∗ = ∑n i=1 ( mi|wi| ) ∑n i=1mi . ( 5 ) Note that if all importance values are equal ( e.g. , mi = 1 for all i ) , then Eq . ( 5 ) becomes the same as Eq . ( 1 ) . Correspondingly , Eq . ( 1 ) can be regarded as a special case of Eq . ( 5 ) . Compared to the conventional Greedy method , our proposed importance-aware Greedy method demands modifications to αi calculations as α∗i = ∑n i=1 ( mi|ri−1| ) / ∑n i=1mi . For the importance-aware Alternating method , we first conduct the importance-aware Greedy method . Then , Eq . ( 4 ) is to be transformed to employ importance . Let us define an n-by-n diagonal matrix M = diag ( m1 , ... , mn ) , where each diagonal element is an importance value mi . By solving linear least squares , α values are refined as [ α1 , ... , αq ] = ( ( B > q MBq ) −1 B > q Mw ) > , when Bq = [ b1 , ... , bq ] ∈ { −1 , +1 } n×q , ( 6 ) while refining Bq is still performed by binary search using refined scaling factors . Accordingly , Eq . ( 4 ) is a particular case of Eq . ( 6 ) when M is an identity matrix . Overall , our proposed importance-aware quantization scheme is comprehensive to include previous methods as a subset . In the rest of this paper , we investigate simple and efficient schemes to estimate importance metrics applicable to post-training non-uniform binary-coding-based quantization .

Summary: Based on a previous classic binary coding scheme, this paper proposed to introduce a modification $m_i$ on the binarization scaling factor $\alpha$, by considering the weight magnitude. It further use 3 hyperparamters to refine $m_i$ by constraining its upper/lower bound and exponent. Besides,  this work spent lengthy content to describe how to determine the hyperparamters.