DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

1 INTRODUCTION . Recently , with the booming of Artificial Intelligence ( AI ) , 5G wireless communications , and CyberPhysical Systems ( CPS ) , distributed learning plays an increasingly important role in improving the efficiency and accuracy of learning , scaling to a large input data size , and bridging different wireless computing resources ( Dean et al. , 2012 ; Bekkerman et al. , 2011 ; Chilimbi et al. , 2014 ; Chaturapruek et al. , 2015 ; Zhu et al. , 2020 ; Mills et al. , 2019 ) . Distributed Stochastic Gradient Descent ( SGD ) is the core in a vast majority of distributed learning algorithms ( e.g. , various distributed deep neural networks ) , where distributed nodes calculate local gradients and an aggregated gradient is achieved via communication among distributed nodes and/or a parameter server . However , due to limited bandwidth in practical networks , communication overhead for transferring gradients often becomes the performance bottleneck . Several approaches towards communicationefficient distributed learning have been proposed , including compressing gradients ( Stich et al. , 2018 ; Alistarh et al. , 2017 ) or updating local models less frequently ( McMahan et al. , 2017 ) . Gradient quantization reduces the communication overhead by using few bits to approximate the original real value , which is considered to be one of the most effective approaches to reduce communication overhead ( Seide et al. , 2014 ; Alistarh et al. , 2017 ; Bernstein et al. , 2018 ; Wu et al. , 2018 ; Suresh et al. , 2017 ) . The lossy quantization inevitably brings in gradient noise , which will affect the convergence of the model . Hence , a key question is how to effectively select the number of quantization bits to balance the trade-off between the communication cost and its convergence performance . Existing algorithms often quantize parameters into a fixed number of bits , which is shown to be inefficient in balancing the communication-convergence trade-off ( Seide et al. , 2014 ; Alistarh et al. , 2017 ; Bernstein et al. , 2018 ) . An efficient scheme should be able to dynamically adjust the number of quantized bits according to the state of current learning model in each gradient descent step to balance the communication overhead and model accuracy . Several studies try to construct adaptive quantization schemes through design heuristics and/or empirical evidence . However , they do not come up with a solid theoretical analysis ( Guo et al. , 2020 ; Cui et al. , 2018 ; Oland & Raj , 2015 ) , which even results in contradicted conclusions . More specifically , MQGrad ( Cui et al. , 2018 ) and AdaQS ( Guo et al. , 2020 ) suggest using few quantization bits in early epochs and gradually increase the number of bits in later epochs ; while the scheme proposed by Anders ( Oland & Raj , 2015 ) states that more quantization bits should be used for the gradient with larger root-mean-squared ( RMS ) value , choosing to use more bits in the early training stage and fewer bits in the later stage . One of this paper ’ s key contributions is to develop a theoretical framework to crystallize the design tradeoff in dynamic gradient quantization and settle this contradiction . In this paper , we propose a novel dynamic quantized SGD ( DQSGD ) framework for minimizing communication overhead in distributed learning while maintaining the desired learning accuracy . We study this dynamic quantization problem in both the strongly convex and the non-convex optimization frameworks . In the strongly convex optimization framework , we first derive an upper bound on the difference ( that we term the strongly convex convergence error ) between the loss after N iterations and the optimal loss to characterize the strongly convex convergence error caused by sampling , limited iteration steps , and quantization . In addition , we find some particular cases and prove the tightness for this upper bound on part of the convergence error caused by quantization . In the non-convex optimization framework , we derive an upper bound on the mean square of gradient norms at every iteration step , which is termed the non-convex convergence error . Based on the above theoretical analysis , we design a dynamic quantization algorithm by minimizing the strongly convex/non-convex convergence error bound under communication cost constraints . Our dynamic quantization algorithm is able to adjust the number of quantization bits adaptively by taking into account the norm of gradients , the communication budget , and the remaining number of iterations . We validate our theoretical analysis through extensive experiments on large-scale Computer Vision ( CV ) and Natural Language Processing ( NLP ) tasks , including image classification tasks on CIFAR-10 and CIFAR-100 and text classification tasks on AG-News . Numerical results show that our proposed DQSGD significantly outperforms the baseline quantization methods . To summarize , our key contributions are as follows : • We propose a novel framework to characterize the trade-off between communication cost and modeling error by dynamically quantizing gradients in the distributed learning . •We derive an upper bound on the convergence error for strongly convex objectives and non-convex objectives . The upper bound is shown to be optimal in particular cases . • We develop a dynamic quantization SGD strategy , which is shown to achieve a smaller convergence error upper bound compared with fixed-bit quantization methods . •We validate the proposed DQSGD on a variety of real world datasets and machine learning models , demonstrating that our proposed DQSGD significantly outperforms state-of-the-art gradient quantization methods in terms of mitigating communication costs . 2 RELATED WORK . To solve large scale machine learning problems , distributed SGD methods have attracted a wide attention ( Dean et al. , 2012 ; Bekkerman et al. , 2011 ; Chilimbi et al. , 2014 ; Chaturapruek et al. , 2015 ) . To mitigate the communication bottleneck in distributed SGD , gradient quantization has been investigated . 1BitSGD uses 1 bit to quantize each dimension of the gradients and achieves the desired goal in speech recognition applications ( Seide et al. , 2014 ) . TernGrad quantizes gradients to ternary levels { −1 , 0 , 1 } to reduce the communication overhead ( Wen et al. , 2017 ) . Furthermore , QSGD is considered in a family of compression schemes that use a fixed number of bits to quantize gradients , allowing the user to smoothly trade-off communication and convergence time ( Alistarh et al. , 2017 ) . However , these fixed-bit quantization methods may not be efficient in communication . To further reduce the communication overhead , some empirical studies began to dynamically adjust the quantization bits according to current model parameters in the training process , such as the gradient ’ s mean to standard deviation ratio ( Guo et al. , 2020 ) , the training loss ( Cui et al. , 2018 ) , gradient ’ s root-mean-squared value ( Oland & Raj , 2015 ) . Though these empirical heuristics of adaptive quan- tization methods show good performance in some certain tasks , their imprecise conjectures and the lack of theoretical guidelines in the conjecture framework have limited their generalization to a broad range of machine learning models/tasks . 3 PROBLEM FORMULATION . We consider to minimize the objective function F : Rd → R with parameter x minx∈Rd F ( x ) = Eξ∼D [ l ( x ; ξ ) ] , ( 1 ) where the data point ξ is generated from an unknown distribution D , and a loss function l ( x ; ξ ) measures the loss of the model x at data point ξ . Vanilla gradient descent ( GD ) will solve this problem by updating model parameters via iterations x ( n+1 ) = x ( n ) − η∇F ( x ( n ) ) , where x ( n ) is the model parameter at iteration n ; η is the learning rate ; ∇F ( x ( n ) ) is the gradient of F ( x ( n ) ) . A modification to the GD scheme , minibatch SGD , uses mini-batches of random samples with size K , AK = { ξ0 , ... , ξK−1 } , to calculate the stochastic gradient g ( x ) = 1/K ∑K−1 i=0 ∇l ( x ; ξi ) . In distributed learning , to reduce the communication overhead , we consider to quantize the minibach stochastic gradients : x ( n+1 ) = x ( n ) − ηQsn [ g ( x ( n ) ) ] , ( 2 ) where Qsn [ · ] is the quantization operation that works on each dimension of g ( x ( n ) ) . The i-th component of the stochastic gradient vector g is quantized as Qs ( gi ) = ‖g‖p · sgn ( gi ) · ζ ( gi , s ) , ( 3 ) where ‖g‖p is the lp norm of g ; sgn ( gi ) = { +1 , −1 } is the sign of gi ; s is the quantization level ; and ζ ( gi , s ) is an unbiased stochastic function that maps scalar |gi|/‖g‖p to one of the values in set { 0 , 1/s , 2/s , . . . , s/s } : if |gi|/‖g‖p ∈ [ l/s , ( l + 1 ) /s ] , we have ζ ( gi , s ) = l/s , with probability 1− p , ( l + 1 ) /s , with probability p = s |gi| ‖g‖p − l. ( 4 ) Note that , the quantization level is roughly exponential to the number of quantized bits . If we use B bits to quantize gi , we will use one bit to represent its sign and the other B − 1 bits to represent ζ ( gi , s ) , thus resulting in a quantization level s = 2B−1 − 1 . In total , we use Bpre + dB bits for the gradient quantization at each iteration : a certain number of Bpre bits of precision to construct ‖g‖p and dB bits to express the d components of g. Given a total number of training iterations N and the overall communication budget C to upload all stochastic gradients , we would like to design a gradient quantization scheme to maximize the learning performance . To measure the learning performance under gradient quantization , we follow the commonly adopted convex/non-convex-convergence error δ ( F , N , C ) ( Alistarh et al. , 2017 ) : δ ( F , N , C ) = F ( x ( N ) , C ) − F ( x∗ , C ) , for strongly convex F , 1 N ∑N−1 n=0 ‖∇F ( x ( n ) ) ‖22 , for non-convex F , ( 5 ) where x∗ is the optimal point to minimize F . In general , this error δ ( F , N , C ) is hard to determine ; instead , we aim to lower and upper bound this error and design corresponding quantization schemes . 4 DYNAMIC QUANTIZED SGD . In this part , we derive upper bounds on the strongly convex/non-convex convergence error δ ( F , N , C ) and lower bounds on the strongly convex-convergence error . By minimizing the upper bound on this convergence error , we propose the dynamic quantized SGD strategies for strongly convex and non-convex objective functions . 4.1 PRELIMINARIES . We first state some assumptions as follows . Assumption 1 ( Smoothness ) . The objective function F ( x ) is L-smooth , if ∀x , y ∈ Rd , ‖∇F ( x ) − ∇F ( y ) ‖2 6 L‖x− y‖2 . It implies that ∀x , y ∈ Rd , we have F ( y ) ≤ F ( x ) +∇F ( x ) T ( y − x ) + L 2 ‖y − x‖22 ( 6 ) ‖∇F ( x ) ‖22 ≤ 2L [ F ( x ) − F ( x∗ ) ] ( 7 ) Assumption 2 ( Strongly convexity ) . The objective function F ( x ) is µ-strongly convex , if ∃µ > 0 , F ( x ) − µ 2 xTx is a convex function . From Assumption 2 , we have : ∀x , y ∈ Rd , F ( y ) ≥ F ( x ) +∇F ( x ) T ( y − x ) + µ 2 ‖y − x‖22 ( 8 ) Assumption 3 ( Variance bound ) . The stochastic gradient oracle gives us an independent unbiased estimate ∇l ( x ; ξ ) with a bounded variance : Eξ∼D [ ∇l ( x ; ξ ) ] = ∇F ( x ) , ( 9 ) Eξ∼D [ ‖∇l ( x ; ξ ) −∇F ( x ) ‖22 ] ≤ σ2 . ( 10 ) From Assumption 3 , for the minibatch stochastic gradient g ( x ) = [ ∑K−1 i=0 ∇l ( x ; ξi ) ] /K , we have Eξ∼D [ g ( x ) ] = ∇F ( x ) ( 11 ) Eξ∼D [ ‖g ( x ; ξ ) ‖2 ] ≤ ‖∇F ( x ) ‖22 + σ2/K . ( 12 ) We have the relationship of gradients before and after quantization : Qs [ g ( x ) ] = g ( x ) + ̂ , where ̂ represents the quantization noise , following the probability distribution that can be shown in Proposition 1 . The proof of Proposition 1 is given in Appendix A . Proposition 1 ( Quantization Noise magnitude ) . For the stochastic gradient vector g , if the quantization level is s , then the i-th component of quantization noise follows as : p ( ̂i ) = s ‖g‖p − s2 ‖g‖2p ̂i , 0 < ̂i ≤ ‖g‖p s , s ‖g‖p + s2 ‖g‖2p ̂i , − ‖g‖p s ≤ ̂i ≤ 0 . ( 13 ) Following Proposition 1 , we can get Êi [ Qs [ g ] ] = g and Êi [ ‖Qs [ g ] − g‖22 ] = d 6s2 ‖g‖2p . This indicates that the quantization operation is unbiased , and the variance bound of Qs [ g ] is directly proportional to ‖g‖2p and inversely proportional to s2 , which means that gradients with a larger norm should be quantized using more bits to keep E [ ‖Qs [ g ] − g‖22 ] below a given noise level . Therefore , we have the following lemma to characterize the quantization noise Qs [ g ] . Lemma 1 . For the quantized gradient vector Qs [ g ] , we have E [ Qs [ g ] ] = ∇F ( x ) ( 14 ) E [ ‖Qs [ g ] ‖22 ] ≤ ‖∇F ( x ) ‖22 + σ2 K + d 6s2 ‖g‖2p ( 15 ) We can see that the noise various of Qs [ g ] contains two parts : the first part is the sampling noise σ2 K , the second part is the quantization noise d 6s2 ‖g‖2p .

Summary: 1.	The authors considered uniform upper bound of the stochastic gradients g_i. The authors may argue that "The classical theoretical analysis of SGD assumes that the stochastic gradients are uniformly bounded". But one can even strongly argue that this bound is actually $\infty$. Moreover, an even stronger argument can be made that the above assumption is in contrast with strong convexity. Please see ["SGD and Hogwild! Convergence Without the Bounded Gradients Assumption" by Nguyen et al.] as one of the instances. Please understand there are relaxed assumptions such as Strong growth condition on a stochastic gradient as in Assumption 4 of [1].