Error Compensated Distributed SGD Can Be Accelerated

1 Introduction . When training very large scale supervised machine learning problems , such as those arising in the context of federated learning [ Konečný et al. , 2016b , McMahan et al. , 2017 , Konečný et al. , 2016a ] ( see also recent surveys [ Li et al. , 2019 , Kairouz , 2019 ] ) , distributed algorithms are indispensable . In such settings , communication is generally much slower than ( local ) computation , which makes it the key bottleneck in the design of efficient distributed systems . There are several ways to tackle this issue , including reliance on large mini-batches [ Goyal et al. , 2017 , You et al. , 2017 ] , asynchronous learning [ Agarwal and Duchi , 2011 , Lian et al. , 2015 , Recht et al. , 2011 ] , local updates [ Ma et al. , 2017 , Stich , 2020 , Khaled et al. , 2020 , Hanzely and Richtárik , 2020 , Woodworth et al. , 2020 ] and communication compression ( e.g. , quantization and sparsification ) [ Alistarh et al. , 2017 , Bernstein et al. , 2018 , Mishchenko et al. , 2019 , Seide et al. , 2014 , Wen et al. , 2017 ] . Communication compression . In this work , we focus on the last of these techniques : communication compression . The key idea here is to apply a lossy compression transformation/operator to the messages before they are communicated so as to save on communication time . While compression reduces the communicated bits in each communication round , it introduces errors , and this generally leads to an increase in the number of communication rounds needed to find a solution of any predefined accuracy . Still , compression has been found useful in practice , as the trade-off often seems to prefer compression to no compression . ∗King Abdullah University of Science and Technology , Thuwal , Saudi Arabia †King Abdullah University of Science and Technology , Thuwal , Saudi Arabia ; Moscow Institute of Physics and Technology , Dolgoprudny , Russia ‡Hong Kong University of Science and Technology , Hong Kong 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contractive and unbiased compressors . There are two large families of such compression operators : contraction and unbiased compressors [ Beznosikov et al. , 2020 ] . A ( possibly ) randomized map Q : Rd → Rd is called a contraction compressor if there exists a constant 0 < δ ≤ 1 such that E [ ∥x−Q ( x ) ∥2 ] ≤ ( 1− δ ) ∥x∥2 , ∀x ∈ Rd . ( 1 ) Further , we say that a randomized map Q̃ : Rd → Rd is an unbiased compressor if there exists a constant ω ≥ 0 such that E [ Q̃ ( x ) ] = x and E [ ∥Q̃ ( x ) ∥2 ] ≤ ( ω + 1 ) ∥x∥2 , ∀x ∈ Rd . ( 2 ) It is well known that ( see , e.g. , [ Beznosikov et al. , 2020 ] ) after appropriate scaling , any unbiased compressor satisfying ( 2 ) becomes a contraction compressor . Indeed , it is easy to verify that for any Q̃ satisfying ( 2 ) , 1ω+1 Q̃ is a contraction compressor satisfying ( 1 ) with δ = 1/ ( ω+1 ) . Thus we can construct corresponding contraction compressor from any unbiased compressor . However , there are empirically very powerful contractive compressors , such as TopK [ Alistarh et al. , 2018 ] ( described below ) , which do not arise this way , and because of this , the class of contractive compressors is important in practice . For illustration purposes , we now define two canonical examples of contraction compressors . Let 1 ≤ K ≤ d. The TopK compressor is defined as ( TopK ( x ) ) π ( i ) = { ( x ) π ( i ) if i ≤ K , 0 otherwise , where π is a permutation of { 1 , 2 , ... , d } such that ( |x| ) π ( i ) ≥ ( |x| ) π ( i+1 ) for i = 1 , ... , d − 1 . The RandK compressor is defined as ( RandK ( x ) ) i = { ( x ) i if i ∈ S , 0 otherwise , where S is chosen uniformly from the set of all K element subsets of { 1 , 2 , ... , d } . For TopK and RandK compressors , we have δ ≥ K/d [ Stich et al. , 2018 ] . Some frequently used unbiased compressors include random dithering [ Alistarh et al. , 2017 ] , random sparsification [ Stich et al. , 2018 ] , and natural compression [ Horváth et al. , 2019b ] . For more examples of contraction and unbiased compressors , we refer the reader to [ Beznosikov et al. , 2020 ] . Related Work . If we assume that the accumulated error is bounded , and in the case of unbiased compressors , the convergence rate of error compensated SGD was shown to be the same as that of vanilla SGD [ Tang et al. , 2018 ] . However , if we only assume bounded second moment of the stochastic gradients , in order to guarantee the boundedness of the accumulated quantization error , some decaying factors need to be involved in general , and error compensated SGD is proved to have some advantage over QSGD [ Alistarh et al. , 2017 ] in some perspective for convex quadratic problems [ Wu et al. , 2018 ] . On the other hand , for contraction compressors , error compensated SGD actually has the same convergence rate as vanilla SGD [ Stich et al. , 2018 , Stich and Karimireddy , 2019 , Tang et al. , 2019 ] . Since SGD only has a sublinear convergence rate , these error compensated methods could not get linear convergence rate . If f is non-smooth and ψ = 0 , error compensated SGD was studied in [ Karimireddy et al. , 2019 ] in the single node case , and the convergence rate is of order O ( 1/ √ δk ) . For variance-reduced methods , QSVRG [ Alistarh et al. , 2017 ] handles the smooth case ( ψ ≡ 0 ) , and VR-DIANA [ Horváth et al. , 2019a ] , ADIANA [ Li et al. , 2020 ] handle the composite case ( general ψ ) . However , the compressors of these algorithms need to be unbiased . Error compensation in VR-DIANA and ADIANA does not need to be used since their method successfully employs variance reduction ( of the variance introduced by the compressor ) instead . Recently , an error compensated method called EC-LSVRG-DIANA which can achieve nonaccelerated linear convergence for the strongly convex and smooth case was proposed in [ Gorbunov et al. , 2020 ] , but besides the contraction compressor , the unbiased compressor is also needed in the algorithm . Error compensation . While contractive compressors can be more powerful in practice , they are not unbiased , and this leads to severe difficulties in the algorithm design space , and poses challenges for theoreticians trying to understand the convergence behavior of distributed algorithms using contractive compressors . Indeed , while there is no issue with the use of contractive compressors in the non-distributed setting ( which is of course not interesting from a practical point of view ) in combination with standard gradient-type methods , serious convergence issues arise in the distributed setting . Indeed , classical methods such gradient descent can diverge ( even exponentially fast ) when the TopK compressor is used to compress uplink gradient information [ Beznosikov et al. , 2020 ] . Fortunately , this issue can be resolved using the error compensation ( EC ) 4 mechanism , which can be traced back to at least Seide et al . [ 2014 ] . However , this mechanism remained a heuristic until recently . To highlight the difficulty of analyzing EC , we remark that the first theoretical breakthroughs applied to the single node regime only [ Stich et al. , 2018 , Stich and Karimireddy , 2019 ] . More recently , theoretical progress was made in the crucially important distributed setting as well [ Beznosikov et al. , 2020 , Gorbunov et al. , 2020 ] . However , we are still very far in our understanding of optimization algorithms using contractive compressors . One of the key open problems in this area—the problem we address in this paper in the affirmative— is whether it is possible to design provably accelerated gradienttype methods that work with contractive compressors . In this paper , we give a confirmed answer by studying error compensation in conjunction with the acceleration mechanism employed in loopless Katyusha ( L-Katyusha ) [ Kovalev et al. , 2020 , Qian et al. , 2019 ] , which is a loopless variant of Katyusha [ Allen-Zhu , 2017 ] . The acceleration in Katyusha or L-Katyusha is an extension of Nesterov ’ s acceleration [ Nesterov , 1983 , 2004 ] in the stochastic regime . Moreover , Katyusha and L-Katyusha can achieve the optimal iteration complexity among randomized methods for solving finite-sum strongly convex optimization problems [ Lan and Zhou , 2018 ] . 2 Problem Description and Summary of Contributions . We formally describe the distributed optimization problem we solve in Section 2.1 , and then summarize our key contributions in Section 2.2 . 2.1 Problem Description . We consider the composite finite-sum optimization problem min x∈Rd P ( x ) : = { 1 n n∑ τ=1 f ( τ ) ( x ) + ψ ( x ) } , ( 3 ) where f ( x ) : = 1n ∑ τ f ( τ ) ( x ) is an average of n smooth convex functions , distributed over n compute nodes , and ψ : Rd → R∪ { +∞ } is a proper closed convex function representing a possibly nonsmooth regularizer . On each node τ , f ( τ ) ( x ) is an average of m smooth convex functions , i.e. , f ( τ ) ( x ) = 1m m∑ i=1 f ( τ ) i ( x ) , representing the average loss over the training data stored on this node . We denote the smoothness constants of functions f , f ( τ ) and f ( τ ) i using symbols Lf , L̄ and L , respectively.5 These constants are in general related as follows : Lf ≤ L̄ ≤ nLf , L̄ ≤ L ≤ mL̄ . ( 4 ) While we specifically focus on the case when m = 1 , our results are also new in the m = 1 case , and hence this regime is relevant as well . We assume throughout that problem ( 3 ) has at least one optimal solution x∗ . We will optionally use the following additional assumption for the contraction compressor . The assumption is not necessary , but when used , it will lead to better complexity . Assumption 2.1 . E [ Q ( x ) ] = δx and E [ Q1 ( x ) ] = δ1x for all x ∈ Rd , where δ and δ1 are compression parameters of Q and Q1 , respectively . It is easy to verify that RandK compressor satisfies Assumption 2.1 with δ = K/d , and 1ω+1 Q̃ , where Q̃ is any unbiased compressor , also satisfies Assumption 2.1 with δ = 1/ ( ω+1 ) . 4Error compensation is also known under the name error feedback . 5Function f : Rd → R is smooth if it is differentiable , and has L-Lipschitz gradient : ∥∇f ( x ) −∇f ( y ) ∥ ≤ L∥x− y∥ for any x , y ∈ Rd . L is called the smoothness constant of f .

Summary: The authors consider accelerated training algorithms using gradient compression with error compensation. For this purpose, they incorporated gradient compression and error compensation in the Katyusha algorithm and developed ECKL. The algorithm quantizes both the SG and VR term and keeps track of the quantization error for error compensation in the next iteration.  The paper analyzed the convergence rate and iteration complexity of ECKL for the smooth convex functions and compared it with variants of DIANA both theoretically and via some simple simulations on logistic regression.