Post-Training Weighted Quantization of Neural Networks for Language Models

1 INTRODUCTION . Training techniques for deep neural networks ( DNNs ) have been developed in ways to incur a lot of parameter redundancy to expedite seeking local minima ( Denil et al. , 2013 ; Jonathan Frankle , 2019 ) . As a result , various model compression techniques including parameter pruning ( Han et al. , 2015 ; He et al. , 2017 ) , quantization ( Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ) , low-rank approximation ( N. Sainath et al. , 2013 ; Prabhavalkar et al. , 2016 ) , and knowledge distillation ( Hinton et al. , 2015 ; Polino et al. , 2018 ) are proposed to lower storage requirements and improve inference performance . Several compression techniques can be combined in a synergistic way to enhance compression ratio ( Han et al. , 2016 ; Zhu et al. , 2017 ) . In this work , we consider parameter quantization that maintains structured model formats and presents high compression ratio . Note that due to limited hardware resources , quantization is an essential method for any inference systems . In general , quantization is classified into uniform quantization based on fixed-point parameter representations ( Jacob et al. , 2018 ; Han et al. , 2016 ) and non-uniform quantization associated with the binary codes ( Zhou et al. , 2017 ; Rastegari et al. , 2016 ) or codebooks ( Choi et al. , 2017 ; Stock et al. , 2020 ) . Most DNN quantization methods are performed based on the principle of minimizing the mean squared error ( MSE ) of quantized parameters ( Rastegari et al. , 2016 ; Xu et al. , 2018 ; Zhou et al. , 2017 ) . Optimizing the MSE is also an underlying principle of low-rank approximation techniques such as the singular value decomposition ( SVD ) ( Prabhavalkar et al. , 2016 ; N. Sainath et al. , 2013 ) . Note that , however , minimizing the MSE implies that each parameter is equally important ( i.e. , squared errors from parameters are accumulated without considering importance of each weight ) . In practice , the impact of each parameter perturbation from quantization on training loss can be vastly different and such impact needs to be analyzed through a sensitivity study of each parameter toward a change in training loss value . In other words , minimizing the MSE ( or the Euclidean distance between original parameters and quantized parameters ) may not correspond to minimizing training loss function after quantization . Robustness to quantization error of each parameter can be expressed as sensitivity . Sensitivity of i-th parameter wi is the amount of change in the loss function when wi is perturbed . A parameter associated with high sensitivity would require relatively smaller quantization error if quantization is performed in a group manner . Several previous works acknowledge distinct sensitivity of each parameter to improve quantization quality . Note that because exact sensitivity estimation of each parameter toward loss function is highly complicated , various heuristic techniques have been introduced . For example , Hessian-weighted k-means clustering is used for codebook-based implementations ( Choi et al. , 2017 ) or Taylor series expansion to bound loss function difference is conducted to decide the optimal quantization bits of each weight ( Khoram & Li , 2018 ) . The Hessian matrix can be used to assign different numbers of quantization bits for each layer ( Dong et al. , 2019 ; Shen et al. , 2019 ) . Minimizing the reconstruction error on the output activations after each layer quantization is performed in ( Stock et al. , 2020 ) . In this paper , we propose a weighted quantization framework where quantized parameters follow the structure of the binary codes so as to achieve high compression ratio and high computational efficiency ( Rastegari et al. , 2016 ; Jeon et al. , 2020 ) . Specifically , given that an importance of each parameter is represented as a real number between 0 and 1 , we extract an optimal quantization solution modified from the previous binary-coding-based quantization methods that employ equal parameter importance . Similar to previous attempts , we also find that calculating accurate importance of each parameter is challenging . As a successful approximation of importance , we suggest that magnitude-based importance estimation is especially effective for post-training non-uniform quantization . 2 POST-TRAINING PARAMETER QUANTIZATION FOR LANGUAGE MODELS . The number of parameters for language models is dramatically increasing ( e.g. , GPT-3 ( Brown et al. , 2020 ) requires 175 billion parameters ) . Correspondingly , model compression for language models is becoming a mandatory process to reduce response time and inference energy . We devise a compression method considering the followings : • Recent language models are usually memory-bound because of small batch size and lacking layers of high reuse ( e.g. , conv layers ) . Thus , reducing memory footprint is critical . • Compression algorithms should be supported by dedicated kernels , designed specifically for language models if possible . • Compression-aware training is challenging and expensive if hyper-parameters are added to already huge language models ( hence , we choose a post-training method . ) Fixed-point inference using uniform quantization is not desirable for language models because of noticeable accuracy degradation ( Shen et al. , 2019 ; Jeon et al. , 2020 ) while the advantage of small computational units ( e.g. , INT8 MAC ) is insignificant for memory-bound applications . Thus , we adopt float-based parameter quantization ( i.e. , expected values of quantized parameters remains to be of full precision ) that induce a lot smaller number of quantization bits compared to fixed-point quantization ( Xu et al. , 2018 ; Stock et al. , 2020 ) . Recently , a kernel library , called BiQGEMM ( Jeon et al. , 2020 ) , was introduced to support binarycoding-based quantization techniques to accelerate quantized neural networks . Using lookup tables , BiQGEMM enables byte-level memory accesses and achieves 8.3× run-time memory footprints and 3.5× speed up with a mobile CPU for Transformer ( Chung et al. , 2020 ) . As a result , binary-codingbased quantization has become a practical approach to quantizing language models . As such , we restrict our interests to binary-coding-based quantization technique in this paper . Quantization-aware training is an active research area to improve model accuracy ( Courbariaux et al. , 2015 ; Lee et al. , 2018 ) . We note that in the case of language models , however , there are numerous occasions when retraining for quantization is not available . For example , quantizationaware training requires in-depth knowledge on model compression while model designers may not have such expertise . On the other hand , the original training code or the entire training data may not be shared with model compression engineers . Also , modifying the original DNN models to be aware of quantization would increase model design efforts and training time significantly . Since language models already demand significant training time and cost , adding additional training complexity by quantization-aware training would not be a practical option . As such , post-training quantization without retraining is gaining increasing attention ( Zhao et al. , 2019 ; Nagel et al. , 2019 ) . 3 WEIGHTED QUANTIZATION BASED ON THE BINARY CODES . As discussed , we choose post-training binary-coding-based quantization as our strategy to compress language DNN models efficiently . Following the Binary-Weight-Networks ( Rastegari et al. , 2016 ) introducing the binary codes as a quantization format , a weight vector w is approximated to be αb by using a scaling factor α ∈ R and a binary vector b ( ∈ { −1 , +1 } n ) , where n is the vector size . A real number scaling factor is shared by multiple weights such that binary vector b occupies most of the weight storage requirements . Binary codes eliminate the need for dequantization for inference , leading to reduced on-chip memory size for weights . In this section , we study general weighted quantization methodologies when quantization follows the format of the binary codes and and quantization error recognizes sensitivity information . 3.1 GREEDY METHOD AND ALTERNATING METHOD WITHOUT IMPORTANCE CONSIDERATIONS . In general , non-uniform weight quantization methods ( in the form of the binary codes ) strive to minimize ‖w − αb‖2 . In the case of 1-bit quantization , we obtain the following analytical solution : b∗ = sign ( w ) , α∗ = w > b∗ n . ( 1 ) On the other hand , in the case of multi-bit quantization , there is no analytical solution ( Rastegari et al. , 2016 ; Xu et al. , 2018 ) . As a result , various approximated methods exist for multi-bit quantization . Greedy Method As a computationally simple method , 1-bit quantization shown in Eq . ( 1 ) can be extended to multi-bit ( q-bit ) quantization ( Guo et al. , 2017 ) . Specifically , ith-bit ( i > 1 ) quantization is performed by minimizing the residue of ( i− 1 ) th-bit quantization as following : min αi , bi ‖ri−1 − αibi‖2 , where ri−1 = w − i−1∑ j=1 αjbj , 1 < i ≤ q . ( 2 ) The optimal solution of Eq . ( 2 ) is then given as b∗i = sign ( ri−1 ) , α ∗ i = r > i−1b ∗ i n . ( 3 ) Alternating Method Greedy method described above is non-iterative . In order to reduce ‖w −∑q i=1 αibi‖2 further than Greedy method , iterative methods would be necessary while increasing the number of iterations tends to lower quantization error . Once initial α and b values are calculated by Greedy method , one can notice that { αi } qi=1 can be refined ( Guo et al. , 2017 ) as [ α1 , ... , αq ] = ( ( B > q Bq ) −1 B > q w ) > , when Bq = [ b1 , ... , bq ] ∈ { −1 , +1 } n×q . ( 4 ) Then , Bq can be refined as well by binary search given a new refined { αi } qi=1 . As a result , { αi } q i=1 and Bq are refined alternatively . Alternating refinements of { αi } qi=1 and Bk are repeated until there is no noticeable improvement in quantization error . Such iterative quantization procedure is introduced as Alternating multi-bit method ( Xu et al. , 2018 ) . 3.2 IMPORTANCE-AWARE WEIGHTED QUANTIZATION . Let us assume that the importance of the i-th parameter is normalized and given as mi ( 0 ≤ mi ≤ 1 ) . Then , we minimize the weighted quantization loss ∑n i=1 ( mi ( wi− ŵi ) 2 ) where wi is quantized to be ŵi = ∑q j=1 ( αjbj ) . Before studying how to estimate importance values , we are interested in finding modified versions of Greedy method and Alternating method when importance values are given . For 1-bit quantization , weighted quantization also has the following analytical solution : b∗ = sign ( w ) , α∗ = ∑n i=1 ( mi|wi| ) ∑n i=1mi . ( 5 ) Note that if all importance values are equal ( e.g. , mi = 1 for all i ) , then Eq . ( 5 ) becomes the same as Eq . ( 1 ) . Correspondingly , Eq . ( 1 ) can be regarded as a special case of Eq . ( 5 ) . Compared to the conventional Greedy method , our proposed importance-aware Greedy method demands modifications to αi calculations as α∗i = ∑n i=1 ( mi|ri−1| ) / ∑n i=1mi . For the importance-aware Alternating method , we first conduct the importance-aware Greedy method . Then , Eq . ( 4 ) is to be transformed to employ importance . Let us define an n-by-n diagonal matrix M = diag ( m1 , ... , mn ) , where each diagonal element is an importance value mi . By solving linear least squares , α values are refined as [ α1 , ... , αq ] = ( ( B > q MBq ) −1 B > q Mw ) > , when Bq = [ b1 , ... , bq ] ∈ { −1 , +1 } n×q , ( 6 ) while refining Bq is still performed by binary search using refined scaling factors . Accordingly , Eq . ( 4 ) is a particular case of Eq . ( 6 ) when M is an identity matrix . Overall , our proposed importance-aware quantization scheme is comprehensive to include previous methods as a subset . In the rest of this paper , we investigate simple and efficient schemes to estimate importance metrics applicable to post-training non-uniform binary-coding-based quantization .

Summary: The paper employs the binary-coding-based post-training quantization (without retraining) for language modeling. The key contribution is that weight importance is considered while determining binary code (a, B). Two methods, Greedy and Alternating, are also modified to use the importance. The algorithm uses a novel normalized importance, which directly uses weight magnitude and some hyper-parameters. Because the performance is sensitive to hyperparameters, Bayesian optimization is used to find task- and model-specific settings.