Rethinking gradient sparsification as total error minimization

1 Introduction . With the emergence of huge DNNs consisting of hundreds of millions to billions of parameters [ 12 , 50 ] , distributed data-parallel training [ 66 ] is an increasingly important workload . As the training process typically spans several compute nodes ( or workers ) that periodically exchange the local gradient vectors at each iteration of the optimizer ( e.g. , SGD ) , communication among nodes remains in many cases the main performance bottleneck [ 32 , 40 , 46 ] . Lossy gradient compression techniques are becoming a common approach to rein in communication efficiency [ 62 ] . In particular , sparsification , which sends only a subset of gradient coordinates ( e.g. , Top-k [ 4 , 8 ] sends the k largest gradient coordinates by magnitude in each iteration ) , may significantly reduce data volumes and thus speed up training . However , due to its lossy nature , compression raises a complex trade-off between training performance and accuracy . For instance , Agarwal et al . [ 3 ] note that training ResNet-18 on CIFAR-100 using sparsification speeds up training significantly ( 3.6× ) , but it also degrades final accuracy by 1.5 % . On the other hand , Lin et al . [ 37 ] reports a 500× data 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reduction via sparsification under deep gradient compression ( DGC ) for ResNet-50 on ImageNet while preserving the same final accuracy when adopting a carefully-tuned warmup phase . The vast literature on gradient compression largely considers a fixed communication budget per iteration while leaving it up to practitioners to grapple with specifying an additional hyper-parameter that determines the degree of compression before training begins . Meanwhile , recent adaptive Top-k sparsifiers [ 3 , 65 ] empirically demonstrate that tuning the degree of compression in different phases of DNN training yields a more communication-efficient scheme than a fixed compression scheme ( e.g. , a static k for Top-k ) . However , these works lack a theoretical framework proving that adaptive compression enjoys better convergence guarantees than the fixed compression scheme . This raises a fundamental question : Given a fixed communication budget , is there a provably better communication scheme than fixed per-iteration compressed communication ? We first observe that Top-k is the communication-optimal sparsifier for a fixed per-iteration communication budget ( §4.3 ) . Then , our insight is that by adopting a different perspective that accounts for the effect of sparsification throughout training , a more efficient communication scheme is possible under a revised notion of optimality that considers an overall communication budget ( instead of a per-iteration budget ) . We consider sparsification by using the error-feedback ( EF ) mechanism [ 8 , 53 ] , a delayed gradient component update strategy that is instrumental for the convergence of the state-of-the-art sparsifiers [ 10 ] . Let et denote the error arising due to sparsification at iteration t. In EF , this error is added to the gradient update at iteration t+ 1 . We identify that the term affecting the non-convex convergence in EF-SGD is the total-error : ∑ t‖et‖2 [ 33 , 54 ] . Directly minimizing the total-error is not possible ; thus , Top-k minimizes ‖et‖2 at each iteration . We argue that it is possible to focus on the sum of ‖et‖2 and devise a communication scheme that achieves a smaller total-error than any fixed communication sparsifier . We demonstrate that to achieve this change of perspective ; it is sufficient to consider a practical yet straightforward mechanism that is a natural counterpart of Top-k : the hard-threshold sparsifier , which communicates the gradient coordinates with magnitude greater than or equal to a fixed given threshold , λ ≥ 0 , in each iteration . Although the two sparsifiers are in an equivalence relation ( a given λ corresponds to a k ) , under the total-error minimization perspective , we adopt a fixed threshold , λ , which implies a variable k at every iteration . To illustrate intuitively why this change of perspective yields benefits , consider the following example . Figure 1 shows an experiment in the distributed setting where 20 workers train a 6,000-parameter logistic regression model on the gisette LIBSVM dataset [ 14 ] by using the Top-k and hardthreshold sparsifiers , configured to send the same data volume.1 The loss function is strongly convex and has a unique minimizer , x ? , therefore , a unique optimum , f ( x ? ) . We see that hard-threshold converges at the same speed as SGD while communicating ∼ 600× less data , whereas Top-k has a significantly slower convergence speed . We attribute this to the fact that Top-k has a large error accumulation in the initial 500 iterations , while the error magnitude for hard-threshold is less than 0.04 throughout training ( cf . Figure 1c ) . Our results with DNN training also reflect this insight ( §6 ) . 1We train for 10 epochs and set k = 0.17 % for Top-k , and λ = 4.2 for hard-threshold . Moreover , the hard-threshold sparsifier has computational benefits over Top-k sparsifier , as hardthreshold ’ s underlying filtering operation requires d comparisons in each iteration , where d is the number of parameters . In contrast , Top-k is a compute-intensive sparsifier ( e.g. , on CPU , the computational complexity is O ( d log2 k ) [ 48 ] ) . For GPUs , several optimized implementations are proposed but they rely on the data distribution and are efficient only for a small k [ 48 ] . For instance , PyTorch uses Radix select algorithm [ 5 ] which has a computational complexity ofO ( db/re d ) where b is the number of bits to represent gradient values and r is the radix size [ 42 ] . Finally , while the hard-threshold sparsifier already exists in the literature [ 20 , 55 ] , we are the first to formally study it and theoretically demonstrate its benefits as an adaptive counterpart of Top-k . Moreover , our argument in favor of hard-threshold precisely falsifies the claim by Dryden et al . [ 18 ] that stopped its widespread adoption — a hard-threshold may lead to a degenerate situation when the EF in gradient compression builds up . This paper makes the following contributions : Communication complexity model ( §4 ) . We propose a communication complexity model that captures the effects of compression in the entire optimization process . We allow for variable communication in each iteration by only imposing a total communication budget . We show that the hard-threshold sparsifier is the communication-optimal sparsifier in this model . Absolute compressors ( §5 ) . We identify that the hard-threshold sparsifier , along with other existing compressors [ 16 , 46 ] , belongs to the class of absolute compressors , which have an absolute bound on the error . Absolute compressors have not been formally studied before with EF . We show that absolute compressors with EF converge for both strongly convex and non-convex loss functions . In both cases , similar to the δ-contraction operators [ 33 ] , absolute compressors enjoy the same asymptotic convergence with linear speedup ( with respect to the number of workers ) as no-compression SGD . However , δ-contraction operators have a worse dependence on δ in the distributed setting with heterogeneous data , while absolute compressors do not have such an anomaly . Experiments ( §6 ) . We conduct diverse experiments on both strongly convex and non-convex ( for DNNs ) loss functions to substantiate our claims . Our DNN experiments include computer vision , language modeling , and recommendation tasks , and our strongly convex experiment is on logistic regression . We find that the hard-threshold sparsifier is consistently more communication-efficient than the Top-k sparsifier given the same communication budget . 2 Related work . Gradient compression techniques can be broadly classified into quantization [ 7 , 16 , 33 , 47 , 60 ] , sparsification [ 4 , 37 , 59 ] , hybrid compressors [ 9 , 18 , 55 ] , and low-rank methods [ 57 , 58 ] . The stateof-the-art compressors are biased δ-contraction operators [ 37 , 57 ] , see §4.5 . We refer to [ 62 ] for a recent survey and quantitative evaluation of these techniques . Error-feedback ( EF ) or memory was first empirically used in [ 47 , 55 ] . However , [ 33 , 53 , 54 ] were the first to give a convergence analysis of the EF framework , which was extended to the distributed setup in [ 10 , 64 ] . Recently , [ 61 ] proposed error-reset , a different form of EF , while [ 29 ] introduced another alternative by communicating a compressed version of the error . EF has also been combined with variance-reduction [ 22 , 43 ] and acceleration [ 44 ] . Communication-optimal compression . [ 6 , 15 , 21 , 45 ] devised a communication-optimal compressor by minimizing the worst-case compression factor2 under a per-vector communication budget . Adaptive compression . [ 59 ] designed an adaptive sparsifier that minimizes expected sparsity of the compressed vector under a given variance budget . While AdaQS [ 23 ] periodically doubles the quantization states in QSGD [ 7 ] to reduce the compression factor , DQSGD [ 63 ] sets the number of quantization states proportional to the gradient norm . ACCORDION [ 3 ] chooses a low compression ratio if training is in a critical regime [ 2 ] , and a high compression ratio otherwise . 2For a vector x and a possibly randomized compression operator C , we denote the compression error as EC [ ‖x− C ( x ) ‖2 ] , and compression factor as EC [ ‖x− C ( x ) ‖2 ] /‖x‖2 . Efficient Top-k estimation is a focus of many recent works . While [ 37 ] estimates the Top-k threshold on a randomly sampled subset , [ 1 , 49 ] estimate it by fitting parametric statistical distributions . [ 25 ] estimates the threshold every 1 , 000 iterations . These works determine a threshold to approximately determine a fixed Top-k set , while we use a hard-threshold to determine the k in each iteration . 3 Background on Error-Feedback ( EF ) SGD . Consider the following distributed optimization problem with n workers : minx∈Rd f ( x ) : = 1 n ∑ i∈ [ n ] fi ( x ) , ( 1 ) where fi ( x ) = Ezi∼Di l ( x ; zi ) denotes the loss function evaluated on input zi sampled from Di , the data-distribution at the ith worker . Let gi , t denote the stochastic gradient computed at ith worker at iteration t such that gi , t = ∇fi ( xt ) + ξi , t with E [ ξi , t|xt ] = 0 . Algorithm 1 gives the pseudo-code of compressed EF SGD [ 8 , 33 , 53 ] to solve ( 1 ) . Let ei , t denote the locally accumulated error at ith worker due to compression from previous steps . Adding this error to the current gradient , γtgi , t provides the corrected update , pi , t , where γt > 0 is the step-size . This corrected update is further compressed and exchanged with other machines , and the local error is updated for the next step . Algorithm 1 : Distributed EF SGD Input : C-compressor Initialize x0 ∈ Rd ; for worker i ∈ [ n ] in parallel do Initialize error ei,0 = 0 ; for iteration t = 0 , . . . , T − 1 do pi , t ← ei , t + γtgi , t / * Incorporate EF into update * / ; ∆i , t ← γtC ( pi , tγt ) ; ei , t+1 ← pi , t −∆i , t / * Update error * / ; ∆̄t = 1 n ∑ i∈ [ n ] ∆i , t / * Exchange and average ∆i , t among workers * / ; xt+1 ← xt − ∆̄t ; Remark 1 . In Algorithm 1 , we have ∆i , t ← γtC ( pi , tγt ) , while [ 8 , 33 , 53 ] consider ∆i , t ← C ( pi , t ) . We do this to extend the EF framework for absolute compressors . We note that Algorithm 1 is more general as γtC ( pi , tγt ) is equivalent to C ( pi , t ) for all known δ-contraction operators .

Summary: In this paper, the authors demonstrate that, in the context of distributed optimization problems with $n$ workers, the hard-threshold sparsifier is the optimal sparsifier for a proposed communication complexity model which where the goal is to minimize the total error for a sequence of responses. This allows the authors to compare the sum of compression errors for various algorithms and to demonstrate that while for the per-iteration $k$-element budget the Top-$k$ sparsifier is optimal, when it comes to total error the hard threshold sparsifier is better.   The authors also compare the convergence rates of the top-$k$ sparsifier vs the hard threshold sparsifer for image classification, language modeling, and recommendation tasks.