Communication-efficient SGD: From Local SGD to One-Shot Averaging

√ T ) communications for T local gradient steps in order for the error to scale proportionately to 1/ ( NT ) , this has been successively improved in a string of papers , with the state of the art requiring Ω ( N ( poly ( log T ) ) ) communications . In this paper , we suggest a Local SGD scheme that communicates less overall by communicating less frequently as the number of iterations grows . Our analysis shows that this can achieve an error that scales as 1/ ( NT ) with a number of communications that is completely independent of T . In particular , we show that Ω ( N ) communications are sufficient . Empirical evidence suggests this bound is close to tight as we further show that √ N or N3/4 communications fail to achieve linear speed-up in simulations . Moreover , we show that under mild assumptions , the main of which is twice differentiability on any neighborhood of the optimal solution , one-shot averaging which only uses a single round of communication can also achieve the optimal convergence rate asymptotically . 1 Introduction . Stochastic Gradient Descent ( SGD ) is a widely used algorithm to minimize convex functions f in which model parameters are updated iteratively as xt+1 = xt − ηtĝt , where ĝt is a stochastic gradient of f at the point xt and ηt is the learning rate . This algorithm can be naively parallelized by adding more workers independently to compute a gradient and then average 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . them at each step to reduce the variance in estimation of the true gradient∇f ( xt ) ( Dekel et al. , 2012 ) . This method requires each worker to share their computed gradients with each other at every iteration . We will refer to this method as `` synchronized parallel SGD . '' However , it is widely acknowledged that communication is a major bottleneck of this method for large scale optimization applications ( McMahan et al. , 2017 ; Konečnỳ et al. , 2016 ; Lin et al. , 2018b ) . Often , mini-batch parallel SGD is suggested to address this issue by increasing the computation to communication ratio . Nonetheless , too large mini-batch size might degrade performance ( Lin et al. , 2018a ) . Along the same lines of increasing the computation over communication effort , local SGD has been proposed to reduce communications ( McMahan et al. , 2017 ; Dieuleveut , Patel , 2019 ) . In this method , workers compute ( stochastic ) gradients and update their parameters locally , and communicate only once in a while to obtain the average of their parameters . Local SGD improves the communication efficiency not only by reducing the number of communication rounds , but also alleviates the synchronization delay caused by waiting for slow workers and evens out the variations in workers ’ computing time ( Wang , Joshi , 2018b ) . On the other hand , since individual gradients of each worker are calculated at different points , this method introduces residual error as opposed to fully synchronized SGD . Therefore , there is a tradeoff between having fewer communication rounds and introducing additional errors to the gradient estimates . The idea of making local updates is not new and has been used in practice for a while ( Mangasarian , 1995 ; Konečnỳ et al. , 2016 ) . However , until recently , there have been few successful efforts to analyze Local SGD theoretically and therefore it is not fully understood yet . Zhang et al . ( 2016 ) show that for quadratic functions , when the variance of the noise is higher far from the optimum , frequent averaging leads to faster convergence . The first question we try to answer in this work is : how many communication rounds are needed for Local SGD to have the similar convergence rate of a synchronized parallel SGD while achieving performance that linearly improves in the number of workers ? Stich ( 2019 ) was among the first who sought to answer this question for general strongly convex and smooth functions and showed that the communication rounds can be reduced up to a factor of H = O ( √ T/N ) , without affecting the asymptotic convergence rate ( up to constant factors ) , where T is the total number of iterations and N is number of parallel workers . Focusing on smooth and possibly non-convex functions which satisfy a Polyak-Lojasiewicz condition , Haddadpour et al . ( 2019 ) demonstrate that only R = Ω ( ( TN ) 1/3 ) communication rounds are sufficient to achieve asymptotic performance that scales proportionately to 1/N . Recently , Khaled et al . ( 2020 ) and Stich , Karimireddy ( 2019 ) improve upon the previous works by showing linear speed-up for Local SGD with only Ω ( N poly log ( T ) ) communication rounds when data is identically distributed among workers and f is strongly convex . Their works also consider the cases when f is not necessarily strongly convex as well as the case of data being heterogeneously distributed among workers . More recently , Yuan , Ma ( 2020 ) proposed a new accelerated method that requires only Ω ( N1/3 poly log ( T ) ) communication rounds for linear speed-up . While their results improve upon the earlier work , the communication requirements remain dependent on the total iterations T . One-Shot Averaging ( OSA ) , a method that takes an extreme approach to reducing communication , involves workers performing local updates until the very end when they average their parameters ( Mcdonald et al. , 2009 ; Zinkevich et al. , 2010 ; Zhang et al. , 2013c ; Rosenblatt , Nadler , 2016 ; Godichon-Baggioni , Saadane , 2020 ) . This method can be seen as an extreme case of Local SGD with R = 1 and H = T local steps . Dieuleveut , Patel ( 2019 ) ; Godichon-Baggioni , Saadane ( 2020 ) provide an analysis of OSA and show that asymptotically , linear speed-up in the number of workers is achieved for a weighted average of iterates . However , both of these works make restrictive assumptions such as uniformly three-times continuously differentiability and bounded second and third derivatives or twice differentiability almost everywhere with bounded Hessian , respectively . The second question we attempt to answer in this work , is whether these assumptions can be relaxed and OSA can achieve linear speed-up in more general scenarios . In this work , we focus on smooth and strongly convex functions with a general noise model . Our contributions are three-fold : We notice that FedAC ( Yuan , Ma , 2020 ) has a better dependence on the number of workers N , in expense of ( poly logarithmic ) dependence on T . With that in mind , we still believe our communication strategy is of independent interest , particularly in the framework of non-accelerated methods . We have performed extensive numerical experiments and comparisons between the two methods and highlighted the regimes where each method outperforms the other . It is worth mentioning that although the the communication complexity by Woodworth et al . ( 2020 ) depends on T , their bound has a lower dependence on condition number κ . Hence , their results are stronger than ours only when κ = Ω ( log T ) . The rest of this paper is organized as follows . In the following subsection we outline the related literature and ongoing works . In Section 2 we define the main problem and state our assumptions . We present our theoretical findings in Section 3 followed by numerical experiments in Section 4 and conclusion remarks in Section 5 . 1.1 Related work . There has been a lot of effort in the recent research to take into account the communication delays and training time in designing faster algorithms ( McDonald et al. , 2010 ; Zhang et al. , 2015 ; Bijral et al. , 2016 ; Kairouz et al. , 2019 ) . See ( Tang et al. , 2020 ) for a comprehensive survey of commu- nication efficient distributed training algorithms considering both system-level and algorithm-level optimizations . Many works study the communication complexity of distributed methods for convex optimization ( Arjevani , Shamir , 2015 ; Woodworth et al. , 2020 ) and statistical estimation ( Zhang et al. , 2013b ) . Woodworth et al . ( 2020 ) present a rigorous comparison of Local SGD with H local steps and minibatch SGD with H times larger mini-batch size and the same number of communication rounds ( we will refer to such a method as large mini-batch SGD ) and show regimes in which each algorithm performs better : they show that Local SGD is strictly better than large mini-batch SGD when the functions are quadratic . Moreover , they prove a lower bound on the worst case of Local SGD that is higher than the worst-case error of large mini-batch SGD in a certain regime . Zhang et al . ( 2013b ) study the minimum amount of communication required to achieve centralized minimax-optimal rates by establishing lower bounds on minimax risks for distributed statistical estimation under a communication budget . A parallel line of work studies the convergence of Local SGD with non-convex functions Zhou , Cong ( 2018 ) . Yu et al . ( 2019 ) was among the first works to present provable guarantees of Local SGD with linear speed-up . Wang , Joshi ( 2018b ) and Koloskova et al . ( 2020 ) present unified frameworks for analyzing decentralized SGD with local updates , elastic averaging or changing topology . The follow-up work of Wang , Joshi ( 2018a ) presents ADACOMM , an adaptive communication strategy that starts with infrequent averaging and then increases the communication frequency in order to achieve a low error floor . They analyze the error-runtime trade-off of Local SGD with nonconvex functions and propose communication times to achieve faster runtime . Another line of work reduces the communication by compressing the gradients and hence limiting the number of bits transmitted in every message between workers ( Lin et al. , 2018b ; Alistarh et al. , 2017 ; Wangni et al. , 2018 ; Stich et al. , 2018 ; Stich , Karimireddy , 2019 ) . Asynchronous methods have been studied widely due to their advantages over synchronized methods which suffer from synchronization delays due to the slower workers ( Spiridonoff et al. , 2020 ) . Wang et al . ( 2019 ) study the error-runtime trade-off in decentralized optimization and proposes MATCHA , an algorithm which parallelizes inter-node communication by decomposing the topology into matchings . However , these methods are relatively more involved and they often require full knowledge of the network , solving a semi-definite program and/or calculating communication probabilities ( schedules ) as in Hendrikx et al . ( 2019 ) . The homogeneous data assumption . In this work , we focus on the case when the data distribution is the same across workers . A number of previous works ( Khaled et al. , 2020 ; Haddadpour et al. , 2019 ; Stich , 2019 ; Dieuleveut , Patel , 2019 ) studied local SGD under this assumption . The assumption is valid when the same data set is either shared across multiple workers in the same cluster , or the assignment of data points to workers is random so that any distributional differences are small . Sharing the data set across multiple workers in this way is a popular strategy to speed up training . For example , such data sharing is implemented in ( Chen et al. , 2012 ; Yadan et al. , 2013 ; Zhang et al. , 2013a ) to speed up training of deep neural networks with multiple GPUs within a single server . While there are many widely used mechanisms such as Horovod ( Sergeev , Del Balso , 2018 ) for synchronous data-parallel distributed training , they share a major communication bottleneck of broadcasting gradients to all workers ( Grubic et al. , 2018 ) . Local SGD improves on these methods by reducing the communication of model parameters from every iteration to a smaller number of rounds during the entire optimization process . Our approach further reduces the communication overhead by communicating less as the number of iterations grows .

Summary: ## Update after rebuttall  I am satisfied with the authors' response and elevate my score to 6.  ##  This paper presents several new theoretical results on the convergence of Local SGD as well as numerical experiments to check the tightness of the theory. In particular, the authors try and remove logarithmic terms from the bounds of Local SGD by making the communication less frequent as the iteration counter increases. The main results of the paper are:  1. Theorem 1 gives convergence for strongly convex functions under uniform-with-strong-growth noise. I argue in my detailed review that this result looks suboptimal.  2. Theorem 2 gives a result for PL functions with sub-Gaussian noise for one-shot averaging. This result is interesting as it promises a linear speed-up, although under somewhat restrictive assumptions.   3. The experiments show that linear speed-up is lost when the number of communication rounds is chosen smaller than predicted by the theory of Theorem 1.  I tend to vote for rejection of this work, even though I appreciate some of the results obtained here. My main concerns are 1) applicability of the message to neural network training, which is highly relevant to FL; 2) the theoretical improvements appear to be small; 3) some small flaws in the proofs.