Analysis of Quantized Models

1 INTRODUCTION . Deep neural networks are usually huge . The high demand in time and space can significantly limit deployment on low-end devices . To alleviate this problem , many approaches have been recently proposed to compress deep networks . One direction is network quantization , which represents each network weight with a small number of bits . Besides significantly reducing the model size , it also accelerates network training and inference . Many weight quantization methods aim at approximating the full-precision weights in each iteration ( Courbariaux et al. , 2015 ; Lin et al. , 2016 ; Rastegari et al. , 2016 ; Li & Liu , 2016 ; Lin et al. , 2017 ; Guo et al. , 2017 ) . Recently , loss-aware quantization minimizes the loss directly w.r.t . the quantized weights ( Hou et al. , 2017 ; Hou & Kwok , 2018 ; Leng et al. , 2018 ) , and often achieves better performance than approximation-based methods . Distributed learning can further speed up training of weight-quantized networks ( Dean et al. , 2012 ) . A key challenge is on reducing the expensive communication cost incurred during synchronization of the gradients and model parameters ( Li et al. , 2014a ; b ) . Recently , algorithms that sparsify ( Aji & Heafield , 2017 ; Wangni et al. , 2017 ) or quantize the gradients ( Seide et al. , 2014 ; Wen et al. , 2017 ; Alistarh et al. , 2017 ; Bernstein et al. , 2018 ) have been proposed . In this paper , we consider quantization of both the weights and gradients in a distributed environment . Quantizing both weights and gradients has been explored in the DoReFa-Net ( Zhou et al. , 2016 ) , QNN ( Hubara et al. , 2017 ) , WAGE ( Wu et al. , 2018 ) and ZipML ( Zhang et al. , 2017 ) . We differ from them in two aspects . First , existing methods mainly consider learning on a single machine , and gradient quantization is used to reduce the computations in backpropagation . On the other hand , we consider a distributed environment , and use gradient quantization to reduce communication cost and accelerate distributed learning of weight-quantized networks . Second , while DoReFa-Net , QNN and WAGE show impressive empirical results on the quantized network , theoretical guarantees are not provided . ZipML provides convergence analysis , but is limited to stochastic weight quantization , square loss with the linear model , and requires the stochastic gradients to be unbiased . This can be restrictive as most state-of-the-art weight quantization methods ( Rastegari et al. , 2016 ; Lin et al. , 2016 ; Li & Liu , 2016 ; Guo et al. , 2017 ; Hou et al. , 2017 ; Hou & Kwok , 2018 ) are deterministic , and the resultant stochastic gradients are biased . In this paper , we relax the restrictions on the loss function , and study in an online learning setting how the gradient precision affects convergence of weight-quantized networks in a distributed environment . The main findings are : 1 . With either full-precision or quantized gradients , the average regret of loss-aware weight quantization does not converge to zero , but to an error related to the weight quantization resolution ∆w and dimension d. The smaller the ∆w or d , the smaller is the error ( Theorems 1 and 2 ) . 2 . With either full-precision or quantized gradients , the average regret converges with a O ( 1/ √ T ) rate to the error , where T is the number of iterations . However , gradient quantization slows convergence ( relative to using full-precision gradients ) by a factor related to gradient quantization resolution ∆g and d. The larger the ∆g or d , the slower is the convergence ( Theorems 1 and 2 ) . This can be problematic when ( i ) the weight quantized model has a large d ( e.g. , deep networks ) ; and ( ii ) the communication cost is a bottleneck in the distributed setting , which favors a small number of bits for the gradients , and thus a large ∆g . 3 . For gradients following the normal distribution , gradient clipping renders the speed degradation mentioned above dimension-free . However , an additional error is incurred . The convergence speedup and error are related to how aggressive clipping is performed . More aggressive clipping results in faster convergence , but a larger error ( Theorem 3 ) . 4 . Empirical results show that quantizing gradients significantly reduce communication cost , and gradient clipping makes speed degradation caused by gradient quantization negligible . With quantized clipped gradients , distributed training of weight-quantized networks is much faster , while comparable accuracy with the use of full-precision gradients is maintained ( Section 4 ) . Notations . For a vector x , √ x is the element-wise square root , x2 is the element-wise square , Diag ( x ) returns a diagonal matrix with x on the diagonal , and x y is the element-wise multiplication of vectors x and y . For a matrix Q , ‖x‖2Q = x > Qx . For a matrix X , √ X is the element-wise square root , and diag ( X ) returns a vector extracted from the diagonal elements of X . 2 PRELIMINARIES . 2.1 ONLINE LEARNING . Online learning continually adapts the model with a sequence of observations . It has been commonly used in the analysis of deep learning optimizers ( Duchi et al. , 2011 ; Kingma & Ba , 2015 ; Reddi et al. , 2018 ) . At time t , the algorithm picks a model with parameter wt ∈ S , where S is a convex compact set . The algorithm then incurs a loss ft ( wt ) . After T rounds , the performance is usually evaluated by the regret R ( T ) = ∑T t=1 ft ( wt ) − ft ( w∗ ) and average regret R ( T ) /T , where w∗ = arg minw∈S ∑T t=1 ft ( w ) is the best model parameter in hindsight . 2.2 WEIGHT QUANTIZATION . In BinaryConnect ( Courbariaux et al. , 2015 ) , each weight is binarized using the sign function either deterministically or stochastically . In ternary-connect ( Lin et al. , 2016 ) , each weight is stochastically quantized to { −1 , 0 , 1 } . Stochastic weight quantization often suffers severe accuracy degradation , while deterministic weight quantization ( as in the binary-weight-network ( BWN ) ( Rastegari et al. , 2016 ) and ternary weight network ( TWN ) ( Li & Liu , 2016 ) ) achieves much better performance . In this paper , we will focus on loss-aware weight quantization , which further improves performance by considering the effect of weight quantization on the loss . Examples include loss-aware binarization ( LAB ) ( Hou et al. , 2017 ) and loss-aware quantization ( LAQ ) ( Hou & Kwok , 2018 ) . Let the full-precision weights from all L layers in the deep network be w. The corresponding quantized weight is denoted Qw ( w ) = ŵ , where Qw ( · ) is the weight quantization function . At the ( t + 1 ) th iteration , the second-order Taylor expansion of ft ( ŵ ) , i.e. , ft ( ŵt ) +∇ft ( ŵt ) > ( ŵ− ŵt ) + 12 ( ŵ− ŵt ) > Ht ( ŵ − ŵt ) is minimized w.r.t . ŵ , where Ht is the Hessian at ŵt . A direct computation of Ht is expensive . In practice , this is approximated by Diag ( √ v̂t ) , where v̂t is the moving average : v̂t = βv̂t−1 + ( 1− β ) ĝ2t = ∑t j=1 ( 1− β ) βt−j ĝ2j , ( 1 ) with gt the stochastic gradient , β ' 1 , and is readily available in popular deep network optimizers such as RMSProp and Adam . Diag ( √ v̂t ) is also an estimate of Diag ( √ diag ( H2t ) ) ( Dauphin et al. , 2015 ) . Computationally , the quantized weight is obtained by first performing a preconditioned gradient descent wt+1 = wt − ηtDiag ( √ v̂t ) −1ĝt , followed by quantization via solving the following problem : ŵt+1 = Qw ( wt+1 ) = arg min ŵ ‖wt+1 − ŵ‖2Diag ( √v̂t ) s.t . ŵ = αb , α > 0 , b ∈ ( Sw ) d. ( 2 ) For simplicity of notations , we assume that the same scaling parameter α is used for all layers . Extension to layer-wise scaling is straightforward . For binarization , Sw = { −1 , +1 } , the weight quantization resolution is ∆w = 1 , and a simple closed-form solution is obtained in ( Hou et al. , 2017 ) . For m-bit linear quantization , Sw = { −Mk , . . . , −M1 , M0 , M1 , . . . , Mk } , where k = 2m−1 − 1 , 0 = M0 < · · · < Mk are uniformly spaced , with weight quantization resolution ∆w = Mr+1 −Mr . An efficient approximate solution of ( 2 ) is obtained in ( Hou & Kwok , 2018 ) . 2.3 GRADIENT QUANTIZATION . In a distributed learning environment with data parallelism , the main bottleneck is often on the communication cost due to gradient synchronization . By quantizing the gradients before synchronization ( Seide et al. , 2014 ; Wen et al. , 2017 ; Alistarh et al. , 2017 ) , this cost can be significantly reduced . For example , assuming that the full-precision gradient is 32-bit , the communication cost can be reduced 32/m times when gradients are quantized to m bits . Most recent gradient quantization methods ( Wen et al. , 2017 ; Alistarh et al. , 2017 ; Zhang et al. , 2017 ) require the quantized gradient to be unbiased , and thus use stochastically quantized gradients . On the other hand , deterministic gradient quantization makes the quantized gradient biased , and the resultant analysis more complex . In this paper , we consider the more general m-bit stochastic linear quantization ( Alistarh et al. , 2017 ) : Qg ( gt ) = st · sign ( gt ) qt , ( 3 ) where st = ‖gt‖∞ , qt ∈ ( Sg ) d , and Sg = { −Bk , . . . , −B1 , B0 , B1 , . . . , Bk } , with k = 2m−1− 1 , 0 = B0 < B1 < · · · < Bk are uniformly spaced . The gradient quantization resolution is defined as ∆g = Br+1 − Br . The ith element qt , i in qt is equal to Br+1 with probability ( |gt , i|/st −Br ) / ( Br+1 − Br ) , and Br otherwise . Here , r is an index satisfying Br ≤ |gt , i|/st < Br+1 . Note that Qg ( gt ) is an unbiased estimator of gt . 3 PROPERTIES OF A QUANTIZED MODEL . In this section , we consider quantization of both weights and gradients in a distributed environment with N workers using data parallelism . For easy illustration , we use the parameter server model ( Li et al. , 2014b ) in Figure 1 , though it also holds for other configurations such as the AllReduce model ( Rabenseifner , 2004 ) . At the tth iteration , worker n ∈ { 1 , 2 , . . . , N } computes the full-precision gradient ĝ ( n ) t w.r.t . the quantized weight and quantizes ĝ ( n ) t to g̃ ( n ) t = Qg ( ĝ ( n ) t ) . The quantized gradients are then synchronized and averaged at the parameter server as : g̃t = 1 N ∑N n=1 g̃ ( n ) t . The server updates the second moment ṽt based on g̃t , and also the full-precision weight as wt+1 = wt − ηtDiag ( √ ṽt ) −1g̃t . The weight is quantized using loss-aware weight quantization to produce ŵt+1 = Qw ( wt+1 ) , which is then sent back to all the workers .

Summary: This paper studies the convergence properties of loss-aware weight quantization with different gradient precisions in the distributed environment, in which servers keeps the full-precision weights and workers keeps quantized weights. The authors provided convergence analysis for weight quantization with full-precision, quantized and quantized clipped gradients. Specifically, they find that: 1) the regret of loss-aware weight quantization with full-precision gradient converge to an error related to the weight quantization resolution and dimension d. 2) gradient quantization slows the convergence by a factor related to gradient quantization resolution and dimension d. 3) gradient clipping renders the speed degradation dimension-free.