Communication-efficient SGD: From Local SGD to One-Shot Averaging

√ T ) communications for T local gradient steps in order for the error to scale proportionately to 1/ ( NT ) , this has been successively improved in a string of papers , with the state of the art requiring Ω ( N ( poly ( log T ) ) ) communications . In this paper , we suggest a Local SGD scheme that communicates less overall by communicating less frequently as the number of iterations grows . Our analysis shows that this can achieve an error that scales as 1/ ( NT ) with a number of communications that is completely independent of T . In particular , we show that Ω ( N ) communications are sufficient . Empirical evidence suggests this bound is close to tight as we further show that √ N or N3/4 communications fail to achieve linear speed-up in simulations . Moreover , we show that under mild assumptions , the main of which is twice differentiability on any neighborhood of the optimal solution , one-shot averaging which only uses a single round of communication can also achieve the optimal convergence rate asymptotically . 1 Introduction . Stochastic Gradient Descent ( SGD ) is a widely used algorithm to minimize convex functions f in which model parameters are updated iteratively as xt+1 = xt − ηtĝt , where ĝt is a stochastic gradient of f at the point xt and ηt is the learning rate . This algorithm can be naively parallelized by adding more workers independently to compute a gradient and then average 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . them at each step to reduce the variance in estimation of the true gradient∇f ( xt ) ( Dekel et al. , 2012 ) . This method requires each worker to share their computed gradients with each other at every iteration . We will refer to this method as `` synchronized parallel SGD . '' However , it is widely acknowledged that communication is a major bottleneck of this method for large scale optimization applications ( McMahan et al. , 2017 ; Konečnỳ et al. , 2016 ; Lin et al. , 2018b ) . Often , mini-batch parallel SGD is suggested to address this issue by increasing the computation to communication ratio . Nonetheless , too large mini-batch size might degrade performance ( Lin et al. , 2018a ) . Along the same lines of increasing the computation over communication effort , local SGD has been proposed to reduce communications ( McMahan et al. , 2017 ; Dieuleveut , Patel , 2019 ) . In this method , workers compute ( stochastic ) gradients and update their parameters locally , and communicate only once in a while to obtain the average of their parameters . Local SGD improves the communication efficiency not only by reducing the number of communication rounds , but also alleviates the synchronization delay caused by waiting for slow workers and evens out the variations in workers ’ computing time ( Wang , Joshi , 2018b ) . On the other hand , since individual gradients of each worker are calculated at different points , this method introduces residual error as opposed to fully synchronized SGD . Therefore , there is a tradeoff between having fewer communication rounds and introducing additional errors to the gradient estimates . The idea of making local updates is not new and has been used in practice for a while ( Mangasarian , 1995 ; Konečnỳ et al. , 2016 ) . However , until recently , there have been few successful efforts to analyze Local SGD theoretically and therefore it is not fully understood yet . Zhang et al . ( 2016 ) show that for quadratic functions , when the variance of the noise is higher far from the optimum , frequent averaging leads to faster convergence . The first question we try to answer in this work is : how many communication rounds are needed for Local SGD to have the similar convergence rate of a synchronized parallel SGD while achieving performance that linearly improves in the number of workers ? Stich ( 2019 ) was among the first who sought to answer this question for general strongly convex and smooth functions and showed that the communication rounds can be reduced up to a factor of H = O ( √ T/N ) , without affecting the asymptotic convergence rate ( up to constant factors ) , where T is the total number of iterations and N is number of parallel workers . Focusing on smooth and possibly non-convex functions which satisfy a Polyak-Lojasiewicz condition , Haddadpour et al . ( 2019 ) demonstrate that only R = Ω ( ( TN ) 1/3 ) communication rounds are sufficient to achieve asymptotic performance that scales proportionately to 1/N . Recently , Khaled et al . ( 2020 ) and Stich , Karimireddy ( 2019 ) improve upon the previous works by showing linear speed-up for Local SGD with only Ω ( N poly log ( T ) ) communication rounds when data is identically distributed among workers and f is strongly convex . Their works also consider the cases when f is not necessarily strongly convex as well as the case of data being heterogeneously distributed among workers . More recently , Yuan , Ma ( 2020 ) proposed a new accelerated method that requires only Ω ( N1/3 poly log ( T ) ) communication rounds for linear speed-up . While their results improve upon the earlier work , the communication requirements remain dependent on the total iterations T . One-Shot Averaging ( OSA ) , a method that takes an extreme approach to reducing communication , involves workers performing local updates until the very end when they average their parameters ( Mcdonald et al. , 2009 ; Zinkevich et al. , 2010 ; Zhang et al. , 2013c ; Rosenblatt , Nadler , 2016 ; Godichon-Baggioni , Saadane , 2020 ) . This method can be seen as an extreme case of Local SGD with R = 1 and H = T local steps . Dieuleveut , Patel ( 2019 ) ; Godichon-Baggioni , Saadane ( 2020 ) provide an analysis of OSA and show that asymptotically , linear speed-up in the number of workers is achieved for a weighted average of iterates . However , both of these works make restrictive assumptions such as uniformly three-times continuously differentiability and bounded second and third derivatives or twice differentiability almost everywhere with bounded Hessian , respectively . The second question we attempt to answer in this work , is whether these assumptions can be relaxed and OSA can achieve linear speed-up in more general scenarios . In this work , we focus on smooth and strongly convex functions with a general noise model . Our contributions are three-fold : We notice that FedAC ( Yuan , Ma , 2020 ) has a better dependence on the number of workers N , in expense of ( poly logarithmic ) dependence on T . With that in mind , we still believe our communication strategy is of independent interest , particularly in the framework of non-accelerated methods . We have performed extensive numerical experiments and comparisons between the two methods and highlighted the regimes where each method outperforms the other . It is worth mentioning that although the the communication complexity by Woodworth et al . ( 2020 ) depends on T , their bound has a lower dependence on condition number κ . Hence , their results are stronger than ours only when κ = Ω ( log T ) . The rest of this paper is organized as follows . In the following subsection we outline the related literature and ongoing works . In Section 2 we define the main problem and state our assumptions . We present our theoretical findings in Section 3 followed by numerical experiments in Section 4 and conclusion remarks in Section 5 . 1.1 Related work . There has been a lot of effort in the recent research to take into account the communication delays and training time in designing faster algorithms ( McDonald et al. , 2010 ; Zhang et al. , 2015 ; Bijral et al. , 2016 ; Kairouz et al. , 2019 ) . See ( Tang et al. , 2020 ) for a comprehensive survey of commu- nication efficient distributed training algorithms considering both system-level and algorithm-level optimizations . Many works study the communication complexity of distributed methods for convex optimization ( Arjevani , Shamir , 2015 ; Woodworth et al. , 2020 ) and statistical estimation ( Zhang et al. , 2013b ) . Woodworth et al . ( 2020 ) present a rigorous comparison of Local SGD with H local steps and minibatch SGD with H times larger mini-batch size and the same number of communication rounds ( we will refer to such a method as large mini-batch SGD ) and show regimes in which each algorithm performs better : they show that Local SGD is strictly better than large mini-batch SGD when the functions are quadratic . Moreover , they prove a lower bound on the worst case of Local SGD that is higher than the worst-case error of large mini-batch SGD in a certain regime . Zhang et al . ( 2013b ) study the minimum amount of communication required to achieve centralized minimax-optimal rates by establishing lower bounds on minimax risks for distributed statistical estimation under a communication budget . A parallel line of work studies the convergence of Local SGD with non-convex functions Zhou , Cong ( 2018 ) . Yu et al . ( 2019 ) was among the first works to present provable guarantees of Local SGD with linear speed-up . Wang , Joshi ( 2018b ) and Koloskova et al . ( 2020 ) present unified frameworks for analyzing decentralized SGD with local updates , elastic averaging or changing topology . The follow-up work of Wang , Joshi ( 2018a ) presents ADACOMM , an adaptive communication strategy that starts with infrequent averaging and then increases the communication frequency in order to achieve a low error floor . They analyze the error-runtime trade-off of Local SGD with nonconvex functions and propose communication times to achieve faster runtime . Another line of work reduces the communication by compressing the gradients and hence limiting the number of bits transmitted in every message between workers ( Lin et al. , 2018b ; Alistarh et al. , 2017 ; Wangni et al. , 2018 ; Stich et al. , 2018 ; Stich , Karimireddy , 2019 ) . Asynchronous methods have been studied widely due to their advantages over synchronized methods which suffer from synchronization delays due to the slower workers ( Spiridonoff et al. , 2020 ) . Wang et al . ( 2019 ) study the error-runtime trade-off in decentralized optimization and proposes MATCHA , an algorithm which parallelizes inter-node communication by decomposing the topology into matchings . However , these methods are relatively more involved and they often require full knowledge of the network , solving a semi-definite program and/or calculating communication probabilities ( schedules ) as in Hendrikx et al . ( 2019 ) . The homogeneous data assumption . In this work , we focus on the case when the data distribution is the same across workers . A number of previous works ( Khaled et al. , 2020 ; Haddadpour et al. , 2019 ; Stich , 2019 ; Dieuleveut , Patel , 2019 ) studied local SGD under this assumption . The assumption is valid when the same data set is either shared across multiple workers in the same cluster , or the assignment of data points to workers is random so that any distributional differences are small . Sharing the data set across multiple workers in this way is a popular strategy to speed up training . For example , such data sharing is implemented in ( Chen et al. , 2012 ; Yadan et al. , 2013 ; Zhang et al. , 2013a ) to speed up training of deep neural networks with multiple GPUs within a single server . While there are many widely used mechanisms such as Horovod ( Sergeev , Del Balso , 2018 ) for synchronous data-parallel distributed training , they share a major communication bottleneck of broadcasting gradients to all workers ( Grubic et al. , 2018 ) . Local SGD improves on these methods by reducing the communication of model parameters from every iteration to a smaller number of rounds during the entire optimization process . Our approach further reduces the communication overhead by communicating less as the number of iterations grows .

Summary: In this paper, the authors study the theoretical speedup of local SGD and one-shot averaging. A synchronization scheme that communicates more frequently at the beginning is also suggested. Some simple numerical experiments are provided to verify the theoretical results.