Compressing gradients in distributed SGD by exploiting their temporal correlation

1 INTRODUCTION . Distributed optimization has become the norm for training machine learning models on large datasets . With the need to train bigger models on ever-growing datasets , scalability of distributed optimization has become a key focus in the research community . While an obvious solution to growing dataset size is to increase the number of workers , the communication among workers has proven to be a bottleneck . For popular benchmark models such as AlexNet , ResNet and BERT the communication can account for a significant portion of the overall training time ( Alistarh et al. , 2017 ; Seide et al. , 2014 ; Lin et al. , 2018 ) . The BERT ( “ Bidirectional Encoder Representing from Transformers ” ) architecture for language models ( Devlin et al. , 2018 ) comprises about 340 million parameters . If 32-bit floating-point representation is used one gradient update from a worker amounts to communicating around 1.3GB ( 340×106 parameters× 32 bits per parameter× 2−33 gigabytes per bit≈ 1.3GB ) . Frequently communicating such large payloads can easily overwhelm the network resulting in prolonged training times . In addition , large payloads may increase other forms of costs in distributed optimization . Novel approaches such as federated learning employ mobile devices as worker nodes . Exchanging information with mobile devices is heavily constrained due to communication bandwidth and budget limitations . Therefore , communication remains an important bottleneck in distributed optimization and reducing communication is of utmost importance . Gradient compression alleviates the communication bottleneck . The idea is to apply a compression scheme on gradients before sending them over the network . There has been an increasing amount of literature on gradient compression within the last few years ( Seide et al. , 2014 ; Aji & Heafield , 2017 ; Alistarh et al. , 2017 ; Wen et al. , 2017 ; Wangni et al. , 2018 ; Wu et al. , 2018 ; Lin et al. , 2018 ; Wang et al. , 2018 ) . Such compression schemes have been demonstrated to work well with distributed stochastic gradient descent ( SGD ) and its variants . However , SGD with arbitrary compres- sion schemes may not converge . Karimireddy et al . ( 2019 ) give one example of non-convergence . The recently proposed error-feedback based algorithms ( Stich et al. , 2018 ; Karimireddy et al. , 2019 ) circumvent the convergence issue . Error-feedback methods accumulate the compression error and feed it back to the input of the compression scheme so that the error gets transmitted over subsequent iterations . The dist-EF-SGD algorithm proposed by Zheng et al . ( 2019 ) applies error-feedback to two-way compression , in which both worker-to-master and master-to-worker communications are compressed . Theoretical guarantees provided by dist-EF-SGD are valid for all compression schemes that fall under the definition of ‘ δ-approximate compressors ’ , also referred to as δ-compressors . The authors prove that error-feedback with two-way compression asymptotically achieves theO ( 1/ √ T ) convergence rate of SGD . However , the analysis by Zheng et al . ( 2019 ) suggests that dist-EF-SGD converges slower than SGD by a constant factor . Our contributions in this paper are as follows . We propose SignXOR , a novel compression scheme that exploits temporal correlation of gradients . We prove that SignXOR is a δ-compressor , and we provide convergence guarantees for SignXOR by employing dist-EF-SGD . We strengthen the convergence bound by Zheng et al . ( 2019 ) to show that dist-EF-SGD asymptotically converges at the same O ( 1/ √ T ) rate as SGD . Consequently , we show that the proposed method asymptotically achieves the SGD convergence rate . We empirically validate the proposed method on CIFAR100 and ImageNet datasets and demonstrate that the ratio between total communication budgets of SignXOR and Scaled-sign is less than 50 % . Notation : For x ∈ Rd , x [ j ] denotes the jth entry of x , ‖x‖1 denotes the ` 1-norm , and ‖x‖ denotes the ` 2-norm . For vector inputs sgn ( · ) function outputs the sign of the input element-wise . The index set { 1 , . . . , n } is denoted by [ n ] , and denotes elementwise multiplication . 2 RELATED WORK . The most common gradient compression schemes can be categorized into those based on sparsification and those based on quantization . The methods based on sparsification such as Top-k , Rand-k ( Stich et al. , 2018 ; Lin et al. , 2018 ) and Spectral-ATOMO ( Wang et al. , 2018 ) preserve only the most significant gradient elements , effectively reducing the quantity of information carrying gradient components . On the other hand , the methods based on quantization such as QSGD ( Alistarh et al. , 2017 ) , TernGrad ( Wen et al. , 2017 ) and SignSGD ( Bernstein et al. , 2018 ) reduce the overall floating-point precision of the gradient . Therefore , these two classes of methods can be respectively thought of as approaches that reduce the quantity versus the quality of the gradient . One can think of this in analogy to image compression . For example , JPEG image compression that is based on discrete cosine transform determines both which transform coefficients to store ( the quantity ) and at what level of resolution to store those coefficients ( the quality ) . Sign-based compression schemes such as Scaled-sign , SignSGD and Signum ( Bernstein et al. , 2018 ) sit at the far end of quantization-based algorithms . Such schemes quantize real values to only two levels , +1 and −1 . For example , the compressing function of Scaled-sign takes in a vector x ∈ Rd , and the decompressing function outputs the vector ( ‖x‖1/d ) sgn ( x ) . This means that the compressed representation needs to store only the sign of each entry x [ j ] , along with the scaling constant ‖x‖1/d . In practice one can avoid the ‘ zero ’ output of sgn by mapping it to +1 or −1 . This allows the two outcomes +1 and −1 to be represented using one bit per entry , making the size of the compressed representation d + 32 bits in total ( assuming 32-bit single-precision representation of the scaling constant ) . As per Shannon ’ s source coding theorem ( MacKay , 2003 , p. 81 ) , the sequence of +1 and −1 can be further compressed without any information loss if the probability of encountering +1 is different from that for −1 . However , in our experiments on Scaled-sign compression we observe that both outputs are equally likely across all iterations . Any lossy gradient compression scheme introduces noise , also known as distortion , in addition to the measurement noise that is already present in the stochastic gradients computed by the workers . It is reasonable to expect the additional compression error hurts the convergence rate of the algorithm . However , it has been empirically observed that significant compression ratios can be achieved before observing any impact on convergence ( Seide et al. , 2014 ; Alistarh et al. , 2017 ) . One can achieve even greater compression while keeping the convergence rate nearly the same by employing errorfeedback ( Stich et al. , 2018 ; Karimireddy et al. , 2019 ; Zheng et al. , 2019 ) . Algorithms based on error-feedback accumulate the compression error in past iterations and add it to the input of the compressing function . This allows all of the gradient information to get transmitted over a sequence of iterations , albeit with a delay . For smooth functions the gradient does not change considerably , therefore , the delay does not significantly impact the rate of convergence . The dist-EF-SGD algorithm proposed by Zheng et al . ( 2019 ) is based on error-feedback and offers two-way compression under the master-worker topology . The dist-EF-SGD algorithm provides convergence guarantees for δ-compressors . An operator C : Rd → Rd is a δ-compressor for all x ∈ Rd if ‖C ( x ) − x‖2 ≤ ( 1− δ ) ‖x‖2 for some δ ∈ ( 0 , 1 ] . The δ is a measure of the distortion due to applying C. A good compressor will have δ close to 1 and a large compression ratio . One can show that Scaled-sign is a 1d -approximate compressor . The Scaled-sign compression scheme with dist-EF-SGD offers convergence guarantees under standard assumptions even though Scaled-sign without error-feedback may not converge ( Karimireddy et al. , 2019 ) . Zheng et al . ( 2019 ) generalize the dist-EF-SGD algorithm to include blockwise compression , in which the gradient is partitioned into blocks that are compressed separately . A natural partitioning method is to consider as blocks the elements of a deep neural network such as tensors , matrices and vectors . Blockwise compression allows better exploitation of redundancy that may be present only within a block . Zheng et al . ( 2019 ) empirically demonstrate that Scaled-sign with dist-EF-SGD achieves almost the same performance of SGD with respect to training loss and test accuracy . This indicates that we can allow more distortion in the compression before observing a significant impact on training performance . The proposed SignXOR compression is based on allowing additional distortion in the interest of achieving a higher compression ratio . 3 PROPOSED ALGORITHM : SIGNXOR . Our proposal is motivated by the concept of delta encoding . Delta encoding refers to techniques that store data as the difference between successive samples , rather than directly storing the samples themselves ( Smith et al. , 1997 ) . One often encounters delta encoding in applications such as integer compression and video compression . For example , it is more space efficient to store the first-order differences of a digitized audio signal than to store the values of the original signal . First-order differences have smaller magnitudes compared to the original sequence and , therefore , the differences can be represented by a comparatively smaller number of bits . A similar approach is used in the inter-picture prediction method in high efficiency video coding ( HEVC ) ( Sze et al. , 2014 ) . Interpicture prediction makes use of temporal correlations across video frames encoding the differences between frames . This requires less storage compared to storing each video frame . Jiang et al . ( 2018 ) employ delta encoding in a more related application to distributed optimization . At the core of their algorithm , delta encoding is employed to compress an increasing sequence of integers . Our SignXOR algorithm applies delta encoding to represent temporal changes in the gradient . In essence , SignXOR maintains a binary vector that indicates whether the sign of a gradient entry is equal ( or not equal ) to the sign of the corresponding entry in the previous gradient . The equality ( or non-equality ) can be represented by a binary 1 ( or 0 ) . This procedure resembles the binary XOR operation , hence the name SignXOR . We employ a generalized version of original dist-EF-SGD by Zheng et al . ( 2019 ) ( the original dist-EF-SGD is specified in Algorithm 2 therein ) to provide convergence guarantees for the proposed compression scheme . The generalization is to make dist-EF-SGD compatible with SignXOR . We outline the proposed method in Algorithm 1 and Algorithm 2 . Generalized dist-EF-SGD : Algorithm 1 presents generalized dist-EF-SGD for setup with a master and n workers . The three main differences between the generalized and original dist-EF-SGD versions are as follows . First , Algorithm 1 delegates compression and decompression tasks to two separate functions encode and decode . Second , Algorithm 1 maintains a vector ḡk at all nodes including the master . The vector ḡk is the average gradient all workers used to update the parameter vector wk in the last iteration . Third , the encode and decode functions each take the last gradient ḡk as the second argument . This is in contrast to the original dist-EF-SGD algorithm in which compression is based only on the gradient in the current iteration . Since there are no differences related to compression performance between the original and generalized dist-EF-SGD algorithms , they encompass the same theoretical guarantees . The compressing function encode takes in two inputs and outputs Gik . This output is the actual payload sent to master over the communication channel . In addition to Gik , the ith worker also Algorithm 1 : Generalized dist-EF-SGD compatible with SignXOR input : initial parameter vector w0 ; step sizes { η0 , . . . , ηT−1 } initialize : let 0 be the all-zeros vector of dimension d ; let ḡ0 ∈ Rd be a vector with entries sampled uniformly from [ −1 , 1 ] ; on ith worker store ḡ0 ; set ei0 to 0 ; set parameter vector to w0 ; on master store ḡ0 ; set e0 to 0 ; 1 for k ∈ { 0 , . . . , T − 1 } do 2 on ith worker 3 compute ĝik = g i k + 1 ηk eik where g i k is the stochastic gradient at wk ; 4 compute Gik = encode ( ĝik , ḡk ) and ḡik = decode ( Gik , ḡk ) ; 5 send Gik to master and update error eik+1 = ηk ( ĝik − ḡik ) ; 6 receive Gk from master and compute ḡk+1 = decode ( Gk , ḡk ) ; 7 update parameter vector wk+1 = wk − ηkḡk+1 ; 8 on master 9 receive Gik and compute ḡik = decode ( Gik , ḡk ) for all i ∈ [ n ] ; 10 compute ĝk = 1 n ∑ i∈ [ n ] ḡ i k + 1 ηk ek ; 11 compute Gk = encode ( ĝk , ḡk ) and ḡk+1 = decode ( Gk , ḡk ) ; 12 broadcast Gk to all workers and update ek+1 = ηk ( ĝk − ḡk+1 ) ; computes ḡik which is what the master will obtain by decompressing Gik . The vector ḡik is used to update eik+1 , the compression error fed back in the next iteration . The master collects Gik from all workers and decompresses each to obtain ḡik . All workers receive the master broadcast Gk , and input it , along with ḡk , to the decode function to decompress and obtain ḡk+1 . Note that in the kth iteration all nodes use the same ḡk vector as the second argument in the encode and decode functions . The encode and decode functions corresponding to SignXOR compression scheme are specified in Algorithm 2 . For the ease of explanation we consider the specific case when the master compresses ĝk to obtain Gk , and the workers decompress Gk to obtain ḡk+1 as an approximation to ĝk . Algorithm 2 : SignXOR compression and decompression input : hyperparameter 0 ≤ α < 14 ( 1− √ 1− 1d ) 2 < 1 ; 1 function encode ( x ∈ Rd , y ∈ Rd ) : 2 compute r , the fraction of +1 ’ s in sgn ( x ) ; 3 compute q , the fraction of elements in x such that sgn ( x [ j ] ) = sgn ( y [ j ] ) ; 4 initialize binary vector b ∈ { 0 , 1 } d to all-zeros ; 5 for all j ∈ [ d ] , if sgn ( x [ j ] ) = sgn ( y [ j ] ) set b [ j ] = 1 with probability 1− α ; 6 compute p , the fraction of 1 ’ s in b ; 7 compress b with a lossless scheme and compute scalar a = ‖x‖1/d ; 8 output G = { a , compressed representation of b } ; 9 function decode ( G = { a ∈ R , compressed representation of a vector b ∈ { 0 , 1 } d } , y ∈ Rd ) : 10 expand and decompress G to obtain a and b ; 11 output a sgn ( y ) ( 2b− 1 ) ; Compressing function : We consider the case when master calls encode with arguments x = ĝk and y = ḡk . Note that the scalars r , q and p are not used anywhere in Algorithm 2 . These scalars help us describe the algorithm and also become useful in our theoretical analysis . The output of encode function is random when α 6= 0 , and deterministic when α = 0 . Let us first consider α = 0 , in which case b [ j ] = 1 if and only if sgn ( ĝk [ j ] ) = sgn ( ḡk [ j ] ) . This implies that p is the fraction of entries in ĝk and ḡk that have the same signs . This is a measure of the ( positive or negative ) correlation between the two vectors . The core idea in the proposed compression scheme is to compress the binary vector b using a lossless compression scheme . Shannon ’ s source coding theorem ( MacKay , 2003 , p. 81 ) states that d i.i.d . random variables each with entropy H ( p ) = −p log2 p− ( 1− p ) log2 ( 1− p ) can be compressed into approximately dH ( p ) bits with negligible risk of information loss , as d → ∞ . In our case , while the length of the parameter vector d is well over a million for models of practical interest , the entries of b are not necessarily i.i.d .. However , we demonstrate in Section 5 that there exist readily available lossless compression algorithms that can compress b to very close to dH ( p ) bits , the Shannon limit . Note that binary entropy H ( p ) is symmetric around p = 0.5 , and satisfies 0 ≤ H ( p ) ≤ 1 with H ( 0.5 ) = 1 . When p ≈ 0.5 the size of the compressed representation gets close to d , which is same as that offered by Scaled-sign . Further compression of b is only possible if p is away from 0.5 . In our experiments presented in Section 5.2 we observe that when α = 0 , p remain close to but slightly lower than 0.5 . This implies a low correlation between ĝk and ḡk . We remedy this issue by making α > 0 . Next we explain how making α > 0 yields compression gains and , at the same time , induces correlation between ĝk and ḡk . First , note that increasing α introduces distortion to b , driving p away from 0.5 and towards 0 . A lower p decreases the entropy of b , yielding a higher compression ratio realized with the lossless compressor . Since we start with a p slightly less than 0.5 , if we were to drive p towards 1 we would first increase H ( p ) before starting to incur compression gains . For this reason , we design the encoder to push p towards zero . In summary , increasing α offers rate savings in the current iteration by adding distortion to the current gradient . Second , we explain how added distortion induces correlation between ĝk and ḡk . Note that q is a measure of correlation between the two vectors sgn ( ĝk ) and sgn ( ḡk ) measured prior to adding distortion using α . We emphasize that q does not depend on the errors introduced in the current iteration . Rather , q depends on α only through the past iterations . In the experimental results presented in Section 5.2 we observe that increasing α also decreases q . This means that in a given iteration the inputs to the encoder are already correlated . Therefore , the b vector that encodes equality ( or nonequality ) of sgn ( ĝk ) and sgn ( ḡk ) can be compressed even without adding distortion in the current iteration . The underlying mechanism that causes this temporal correlation in our encoder is errorfeedback . Recall that the idea behind error-feedback is to keep track of the compression error in the last iteration and add it back to the input of the encoder in the current iteration ( with correction for the step size ) . Specifically , ĝk includes the compression error incurred in ḡk . This feedback system induces the temporal relation between the two vectors that we see through q . In summary , our compression mechanism interacts with the error-feedback mechanism to increase temporal correlation which we then exploit to realize further compressive gains . The suggested upper bound for α ensures theoretical convergence of the SignXOR algorithm . We show in Section 5 that in practice α can be increased considerably more than the suggested upper bound before seeing an impact on the training performance . Decompressing function : Let us now consider the case when a worker calls decode with arguments G = Gk and y = ḡk . The first argument Gk contains the scalar a and the compressed representation of binary vector b , which can be recovered exactly as compression of b is lossless . Noting that b is computed with ḡk as the second argument to the encode function , the output of the decoder is obtained by inverting the sign of ḡk [ j ] whenever b [ j ] is 0 , and by scaling the result by a . One can compactly express the output of this operation as a sgn ( ḡk ) ( 2b− 1 ) . Remarks : Note that we can recover the Scaled-sign compression scheme by setting α = 0 in Algorithm 2 . In comparison to SignXOR , the compressed representation of Scaled-sign stores the sign of each entry in ĝk , and the scaling constant a = ‖ĝk‖1/d . We demonstrate in Section 5.2 that r , the fraction of +1 ’ s in sgn ( ĝk ) , is approximately 0.5 across all iterations . This means that we can not compress further the sequence of signs , and the encoded representation of Scaled-sign requires at least d bits . Algorithm 1 can be easily extended to accommodate blockwise compression as was explained by Zheng et al . ( 2019 ) . In blockwise compression the gradient ĝk is partitioned into blocks , and the blocks are processed separately using encode and decode functions . In our numerical experiments we employ the blockwise extension of Algorithm 1 . 4 THEORETICAL GUARANTEES In this section we summarize our theoretical results on Algorithm 1 and Algorithm 2 . First , we prove that the SignXOR compression scheme presented in Algorithm 2 is a δ-compressor . Second , we show that for any δ-compressor the generalized dist-EF-SGD scheme in Algorithm 1 converges at the same O ( 1/ √ T ) rate as SGD . Putting these two together yields the desired result . SignXOR is a δ-compressor : We consider the general form of encode and decode functions , i.e. , with inputs ( x , y ) for encode , and with inputs ( G , y ) for decode . Let us define the operator Cyα : Rd → Rd where Cyα ( x ) = a sgn ( y ) ( 2b − 1 ) with a = ‖x‖1 d and the binary vector b is such that b = 1 with probability 1 − α if sgn ( x ) = sgn ( y ) , and b = 0 otherwise . Note that a and b have the same meaning as in Algorithm 2 . The operator Cyα ( x ) is representative of decode ( encode ( x , y ) , y ) , therefore , Cyα ( x ) is the SignXOR compressor . Since C y α is a randomized operator , we show that it is a δ-compressor in expectation as stated in Theorem 1 . Theorem 1 . There exist a δ ∈ ( 0 , 1 ] such that E [ ‖Cyα ( x ) − x‖2 ] ≤ ( 1 − δ ) ‖x‖2 for all y ∈ Rd if α < 14 ( 1− √ 1− 1d ) 2 . The proof is provided in Appendix A.1 . Although the suggested upper bound for α is extremely small for large d , we demonstrate in Section 5 that in practice α can be set quite close to 1 . Next we discuss the convergence rate of dist-EF-SGD for an arbitrary δ-compressor . Convergence rate of dist-EF-SGD : Zheng et al . ( 2019 ) compare the convergence rates of dist-EFSGD and vanilla SGD in their Corollary 1 . As authors note , although the convergence rates of the two algorithms are same in O notation , the former is slower by a constant factor . The differences between the bounds for dist-EF-SGD and vanilla SGD in Corollary 1 are as follows . In both cases , for large T the dominant terms are those with √ T in denominator . The ratio between the dominant terms corresponding to dist-EF-SGD and vanilla SGD is 1.5 , suggesting that the former is always slower than the latter by a factor of 1.5 . In our Theorem 2 we strengthen the bound for dist-EFSGD so that the dominant terms in the two algorithms are the same . This means that dist-EFSGD achieves same convergence rate as vanilla SGD in the limit as T → ∞ . Our proof is for an arbitrary δ-compressor that is not necessarily SignXOR . To this end we consider Algorithm 1 and define C ( x ) = decode ( encode ( x , y ) , y ) . We assume that C is a δ-compressor for all y ∈ Rd , i.e. , E [ ‖C ( x ) − x‖2 ] ≤ ( 1− δ ) ‖x‖2 for some δ ∈ ( 0 , 1 ] . We setup the optimization problem next . Let F : Rd → R be a function that is lower bounded by F∗ and has a gradient ∇F . We consider a distributed optimization setup with a master and n workers . For some wk ∈ Rd the ith worker calculates the stochastic gradient gik at wk . We assume that g i k is an unbiased estimate of ∇F ( wk ) , and that gik has bounded variance . Specifically , we assume that gik satisfies the two properties E [ gik|wk ] = ∇F ( wk ) and E [ ‖gik −∇F ( wk ) ‖2 ] ≤ σ2 . We also assume that F is L-Lipschitz smooth , and that the gradient of F is bounded . The latter implies that E [ ‖gik‖2 ] ≤ G2 for some scalar G. The master and workers generate a sequence { w0 , . . . , wT } as per Algorithm 1 . Convergence results of this system is summarized in Theorem 2 . Theorem 2 . For a given w0 and a step size schedule ηk = 1L√T ( 1− 1 2T 1/4 ) , the convergence of the system outlined in Algorithm 1 after T iterations is given by E [ min k=0 , ... , T−1 ‖∇F ( wk ) ‖ 2 ] < 2L ( F ( w0 ) − F∗ ) + σ 2 n 2 √ T − 1 + 2L ( F ( w0 ) − F∗ ) + 8 ( 1−δ ) G2 δ2 ( 1 + 1δ2 ) 2T 3/4 − T 1/4 +O ( 1 T ) . ( 1 ) We defer the proof of Theorem 2 to Section A.2 in the Appendix . In comparison , the bound for SGD has only the first term in ( 1 ) . Note that the last two terms in ( 1 ) converge to zero faster than the first term . This means that dist-EF-SGD asymptotically achieves the same convergence rate as SGD . Since SignXOR is a δ-compressor we conclude that the proposed algorithm converges asymptotically at the same rate as SGD .

Summary: The authors present a new scheme for compressing gradients for use in distributed training. In addition to the previously proposed techniques of sending the sign of the gradient components along with the scale, and the use of error feedback (each sender tracks the error introduced by quantization, and adjusts future gradient updates using it), the authors also propose to exploit the temporal correlation of gradient values (i.e., over successive steps). They do so by computing the delta between two steps, and then use a hyperparameter $\alpha$ to keep only a fraction of the deltas and that is sent losslessly.