Nested Dithered Quantization for Communication Reduction in Distributed Training

In distributed training , the communication cost due to the transmission of gradients or the parameters of the deep model is a major bottleneck in scaling up the number of processing nodes . To address this issue , we propose dithered quantization for the transmission of the stochastic gradients and show that training with Dithered Quantized Stochastic Gradients ( DQSG ) is similar to the training with unquantized SGs perturbed by an independent bounded uniform noise , in contrast to the other quantization methods where the perturbation depends on the gradients and hence , complicating the convergence analysis . We study the convergence of training algorithms using DQSG and the trade off between the number of quantization levels and the training time . Next , we observe that there is a correlation among the SGs computed by workers that can be utilized to further reduce the communication overhead without any performance loss . Hence , we develop a simple yet effective quantization scheme , nested dithered quantized SG ( NDQSG ) , that can reduce the communication significantly without requiring the workers communicating extra information to each other . We prove that although NDQSG requires significantly less bits , it can achieve the same quantization variance bound as DQSG . Our simulation results confirm the effectiveness of training using DQSG and NDQSG in reducing the communication bits or the convergence time compared to the existing methods without sacrificing the accuracy of the trained model . 1 INTRODUCTION . In recent years , the size of deep learning problems has increased significantly both in terms of the number of available training samples as well as the complexity of the model . Hence , training deep models on a single processing node is unappealing or nearly impossible . As such , large-scale distributed machine learning in which the training samples are distributed among different repository or processing units ( referred to as workers ) has started to be a viable approach for tackling the memory , storage and computational constraints . The requirement to exchange the gradients or the parameters of the model incurs significant communication overhead which is a major bottleneck in distributed training algorithms . In recent years , there has been a great amount of effort on reducing the communication overhead . The majority of existing methods can be categorized into two groups : The first group mitigates the communication bottleneck by reducing the overall transmission rate via sparsification , quantization and/or compression of the gradients . For example , Seide et al . ( 2014 ) reduces the communication overhead significantly by one-bit quantization of the stochastic gradients ( SG ) . However , the reduced accuracy of gradient may impair the convergence rate . Using different quantization levels or adaptive quantizers , one can alleviate such issues by decreasing the error in the quantized gradients in the expense of increased communication bits Dryden et al . ( 2016 ) . Moreover , applying entropy coding algorithms such as Huffman coding on the quantized values can further reduce the communication bit-rate Øland & Raj ( 2015 ) ; Strom ( 2015 ) . Alistarh et al . ( 2017 ) introduced QSGD which uses probabilistic ( stochastic ) quantization of SGs instead of ordinary fixed ( deterministic ) quantization methods . They investigated its convergence guarantee and the trade-off between the quantization precision and variance of QSG . Terngrad Wen et al . ( 2017 ) probabilistically quantizes the gradients into { −1 , 0 , +1 } and it is shown that the convergence rate can be improved by layer-wise quantization and gradient clipping . The second group of works attempts to attenuate the communication bottleneck by relaxing the synchronization between workers . Each worker may continue its own computations while some others are still communicating and exchanging parameters . Carefully scheduling and managing the asynchronous parameter exchange can lead to a better utilization of both the communication bandwidth and the computational power of the distributed system . Examples of such approaches include DownpourSGD Dean et al . ( 2012 ) , Hogwild ! Niu et al . ( 2011 ) , Hogwild++ Zhang et al . ( 2016 ) and Stale Synchronous Parallel model of computation Ho et al . ( 2013 ) . Our Contributions . Our work in this paper falls within the first line of research , i.e . reducing the communication overhead by quantizing and compressing the gradients . We first introduce using dithered quantization in the distributed computations of the stochastic gradient and show that stochastic quantizer of Alistarh et al . ( 2017 ) and ternarization of Wen et al . ( 2017 ) can be considered as special cases of our proposed method , although the reconstruction algorithms are slightly different . The convergence of dithered quantized stochastic gradient descent algorithm is analyzed and its convergence speed w.r.t . the number of workers and quantization precision is investigated . Next , we observe that in a typical distributed system , the stochastic gradients computed by the workers are correlated . However , the existing communication methods ignore that correlation . We tap into the question of how that correlation can be exploited to further reduce the communication without sacrificing the precision or convergence of the learning algorithm . We model the correlation between the stochastic gradients computed by each worker and propose a nested quantization scheme to reduce the communication bits without increasing the variance of the quantization error or reducing the convergence speed of the distributed training algorithm . 1.1 NOTATIONS . Throughout the paper , bold lowercase letters represent vectors and the i-th element of the vector x is denoted as xi . Matrices are denoted by bold capital letters such asX , with the ( i , j ) -th element represented by Xi , j or [ X ] i , j . Given a real number x ∈ R , bxe is the nearest integer to x . For a random variable u , u ∼ U [ a , b ] if its probability distribution is uniform over interval [ a , b ] and u ∼ N ( µ , σ2 ) if it follows a Gaussian distribution with mean µ and variance σ2 . 2 PRELIMINARIES . 2.1 DITHERED QUANTIZATION . It is well-known that the error in ordinary quantization especially when the number of quantization levels is low , depends on the input signal and is not necessarily uniformly distributed . In Dithered Quantization , a ( pseudo- ) random signal called dither is added to the input signal prior to quantization . Adding this controlled perturbation can cause the statistical behavior of the quantization error to be more desirable Schuchman ( 1964 ) ; Gray & Stockham ( 1993 ) ; Gray & Neuhoff ( 1998 ) . Let Q ( · ) be an M-level uniform quantizer with quantization step size of ∆ , i.e. , Q ( v ) = ∆ bv/∆e where bαe is the nearest integer to α . The dithered quantizer is defined as follows ; 1 Definition ( Dithered Quantization ) . For an input signal x , let u be a dither signal , independent of x . The dithered quantization of x is defined as x̃ = Q ( x+ u ) − u . Remark 1 . To transmit the dithered quantization of x , it is sufficient to send the index of the quantization bin that x + u resides in , i.e. , b ( x+ u ) /∆e . The receiver reproduces the ( pseudo ) random sequence u using the same random number generator algorithm and seed number as the sender . It is then subtracted from Q ( x+ u ) to form the dithered quantized value , x̃ . Theorem 1 ( Schuchman ( 1964 ) ) . If 1 ) the quantizer does not overload , i.e. , |x + u| ≤ M∆2 for all input signals x and dither u , and 2 ) The characteristic function of the dither signal , defined as Mu ( jν ) = Eu [ ejνu ] , satisfies Mu ( j 2πl∆ ) = 0 for all l 6= 0 , then the quantization error e = x− x̃ is uniform over [ −∆/2 , ∆/2 ] and it is independent of the signal x . It is common to consider U [ ∆/2 , ∆/2 ] as the distribution of the random dither signal . It can be easily verified that this choice of the dither signal satisfies the conditions of Thm . 1 , and it does not increase 1Throughout the paper , we assume that all quantizers are centered around 0 . This is the case also for ternary Wen et al . ( 2017 ) and stochastic quantizations Alistarh et al . ( 2017 ) . the bound of the quantization error , i.e , |x̃− x| ≤ ∆/2 which is the same as the traditional uniform quantization with the same step size . In some cases , the receiver may not be able to reproduce the dither signal to subtract from Q ( x+ u ) . Hence , quantization is simply defined as as x̃h = Q ( x + u ) . We refer to this approach as the half-dithered quantization as the dither signal is applied only to the quantizer , not the reconstruction of x . In this case , the quantization error is not necessarily independent of the signal , however by an appropriate choice of the dither signal , the moments of the quantization error will be independent Gray & Stockham ( 1993 ) . For example , if the dither signal u is the sum of k independent random variables , each having uniform distribution U [ −∆/2 , ∆/2 ] , then the k-th moment of the quantization error , = x− x̃h , would be independent of the signal , given by E [ k|x ] = E [ k ] = ( k + 1 ) ∆ 2 12 . 2.1.1 RELATIONSHIP WITH TERNARY AND STOCHASTIC QUANTIZATIONS . Here , we examine the relation between the dithered quantization , Ternary quantization of Wen et al . ( 2017 ) and the stochastic quantization in Alistarh et al . ( 2017 ) . Without loss of generality , assume that the vector x is normalized such that |xi| ≤ 1 . Although the reconstruction of quantized values in our method is different from those in TernGrad and QSGD , we show that these quantizers can be considered as a special case of the half-dithered quantizer . M -level Stochastic Quantization in Alistarh et al . ( 2017 ) is defined as Q ( s ) ( xi ) = { sign ( xi ) l/M with probability l + 1−M |xi| sign ( xi ) ( l + 1 ) /M with probability M |xi| − l , ( 1 ) where |xi| ∈ [ l/M , ( l + 1 ) /M ] . The ternary quantizer of Wen et al . ( 2017 ) can be considered as a special case of stochastic quantizer with M = 1 . Lemma 2 . Stochastic quantization is the same as ( 2M + 1 ) -level half-dithered quantizer with step-size ∆ = 1M and uniform dither u ∼ U [ − 1 2M , 1 2M ] . In other words , stochastic quantizer adds a uniformly distributed dither to the input signal before quantization , but at the receiver , it does not subtract the dither from the quantized value . Therefore , the quantization error is not independent of the signal Gray & Stockham ( 1993 ) . It can be easily verified that although the quantization is unbiased , E [ x−Q ( s ) ( x ) ] = 0 , its variance depends on the value of the input signal : E [ ( [ Q ( s ) ( x ) − x ] i ) 2 ] = ( |xi| − l/M ) ( ( l + 1 ) /M − |xi| ) , if |xi| ∈ [ l/M , ( l + 1 ) /M ] . It can be easily verified that the variance of the quantization error varies in the interval [ 0 , 14M2 ] depending on the value of x . If x is uniformly distributed over [ −1 , 1 ] , the average quantization variance would be 16M2 , twice the variance of the dithered quantization .

Summary: Authors establish a connection between communication reduction in distributed optimization and dithered quantization. This allows us to understand prior approaches in a new perspective, and also motivates authors to develop two new distributed training algorithms which communication overhead is significantly reduced. The first algorithm, DQSG, uses dithered quantization to reduce the communication bits. The second algorithm, NDQSG, uses nested dithered quantization to further reduce the amount of needed communication. The usefulness of these algorithms are empirically validated by computing the raw communication bits and average entropy of them. Therefore, dithered communication seems to provide both theory and algorithm which are useful.