Local SGD Converges Fast and Communicates Little

1 INTRODUCTION . Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) consists of iterations of the form xt+1 : = xt − ηtgt , ( 1 ) for iterates ( weights ) xt , xt+1 ∈ Rd , stepsize ( learning rate ) ηt > 0 , and stochastic gradient gt ∈ Rd with the property Egt = ∇f ( xt ) , for a loss function f : Rd → R. This scheme can easily be parallelized by replacing gt in ( 1 ) by an average of stochastic gradients that are independently computed in parallel on separate workers ( parallel SGD ) . This simple scheme has a major drawback : in each iteration the results of the computations on the workers have to be shared with the other workers to compute the next iterate xt+1 . Communication has been reported to be a major bottleneck for many large scale deep learning applications , see e.g . ( Seide et al. , 2014 ; Alistarh et al. , 2017 ; Zhang et al. , 2017 ; Lin et al. , 2018b ) . Mini-batch parallel SGD addresses this issue by increasing the compute to communication ratio . Each worker computes a mini-batch of size b ≥ 1 before communication . This scheme is implemented in state-of-the-art distributed deep learning frameworks ( Abadi et al. , 2016 ; Paszke et al. , 2017 ; Seide & Agarwal , 2016 ) . Recent work in ( You et al. , 2017 ; Goyal et al. , 2017 ) explores various limitations of this approach , as in general it is reported that performance degrades for too large mini-batch sizes ( Keskar et al. , 2016 ; Ma et al. , 2018 ; Yin et al. , 2018 ) . In this work we follow an orthogonal approach , still with the goal to increase the compute to communication ratio : Instead of increasing the mini-batch size , we reduce the communication frequency . Rather than keeping the sequences on different machines in sync , we allow them to evolve locally on each machine , independent from each other , and only average the sequences once in a while ( local SGD ) . Such strategies have been explored widely in the literature , under various names . An extreme instance of this concept is one-shot SGD ( McDonald et al. , 2009 ; Zinkevich et al. , 2010 ) where the local sequences are only exchanged once , after the local runs have converged . Zhang et al . ( 2013 ) show statistical convergence ( see also ( Shamir & Srebro , 2014 ; Godichon-Baggioni & Saadane , 2017 ; Jain et al. , 2018 ) ) , but the analysis restricts the algorithm to at most one pass over the data , which is in general not enough for the training error to converge . More practical are schemes that perform more frequent averaging of the parallel sequences , as e.g . ( McDonald et al. , 2010 ) for perceptron training ( iterative parameter mixing ) , see also ( Coppola , 2015 ) , ( Zhang et al. , 2014 ; Bijral et al. , 2016 ; Zhang et al. , 2016 ) for the training of deep neural networks ( model averaging ) or in federated learning ( McMahan et al. , 2017 ) . The question of how often communication rounds need to be initiated has eluded a concise theoretical answer so far . Whilst there is practical evidence , the theory does not even resolve the question whether averaging helps when optimizing convex functions . Concretely , whether running local SGD on K workers is K times faster than running just a single instance of SGD on one worker.1 We fill this gap in the literature and provide a concise convergence analysis of local SGD . We show that averaging helps . Frequent synchronization of K local sequences increases the convergence rate by a factor of K , i.e . a linear speedup can be attained . Thus , local SGD is as efficient as parallel mini-batch SGD in terms of computation , but the communication cost can be drastically reduced . 1.1 CONTRIBUTIONS . We consider finite-sum convex optimization problems f : Rd → R of the form f ( x ) = 1 n n∑ i=1 fi ( x ) , x ∗ : = arg minx∈Rd f ( x ) , f ? : = f ( x ? ) , ( 2 ) where f is L-smooth2 and µ-strongly convex3 . We consider K parallel mini-batch SGD sequences with mini-batch size b that are synchronized ( by averaging ) after at most every H iterations . For appropriate chosen stepsizes and an averaged iterate x̂T after T steps ( for T sufficiently large , see Section 3 below for the precise statement of the convergence result with bias and variance terms ) and synchronization delay H = O ( √ T/ ( Kb ) ) we show convergence E f ( x̂T ) − f ? = O ( G2 µbKT ) , ( 3 ) with second moment bound G2 ≥ E ‖∇fi ( x ) ‖2 . Thus , we see that compared to parallel minibatch SGD the communication rounds can be reduced by a factor H = O ( √ T/ ( Kb ) ) without hampering the asymptotic convergence . Equation ( 3 ) shows perfect linear speedup in terms of computation , but with much less communication that mini-batch SGD . The resulting speedup when taking communication cost into account is illustrated in Figure 1 ( see also Section D below ) . Under the assumption that ( 3 ) is tight , one has thus now two strategies to improve the compute to communication ratio ( denoted by ρ ) : ( i ) either to increase the mini-batch size b or ( ii ) to increase the communication interval H . Both strategies give the same improvement when b and H are small ( linear speedup ) . Like mini-batch SGD that faces some limitations for b 1 ( as discussed in e.g . ( Dekel et al. , 2012 ; Ma et al. , 2018 ; Yin et al. , 2018 ) ) , the parameter H can not be chosen too large in local SGD . We give some pratical guidelines in Section 4 . Our proof is simple and straightforward , and we imagine that—with slight modifications of the proof—the technique can also be used to analyze other variants of SGD that evolve sequences on 1On convex functions , the average of the K local solutions can of course only decrease the objective value , but convexity does not imply that the averaged point is K times better . 2f ( y ) ≤ f ( x ) + 〈∇f ( x ) , y − x〉+ L 2 ‖y − x‖2 , ∀x , y ∈ Rd . 3f ( y ) ≥ f ( x ) + 〈∇f ( x ) , y − x〉+ µ 2 ‖y − x‖2 , ∀x , y ∈ Rd . different worker that are not perfectly synchronized . Although we do not yet provide convergence guarantees for the non-convex setting , we feel that the positive results presented here will spark further investigation of local SGD for this important application ( see e.g . ( Yu et al. , 2018 ) ) . 1.2 RELATED WORK . A parallel line of work reduces the communication cost by compressing the stochastic gradients before communication . For instance , by limiting the number of bits in the floating point representation ( Gupta et al. , 2015 ; Na et al. , 2017 ; Sa et al. , 2015 ) , or random quantization ( Alistarh et al. , 2017 ; Wen et al. , 2017 ) . The ZipML framework applies this technique also to the data ( Zhang et al. , 2017 ) . Sparsification methods reduce the number of non-zero entries in the stochastic gradient ( Alistarh et al. , 2017 ; Wangni et al. , 2017 ) . A very aggressive—and promising—sparsification method is to keep only very few coordinates of the stochastic gradient by considering only the coordinates with the largest magnitudes ( Seide et al. , 2014 ; Strom , 2015 ; Dryden et al. , 2016 ; Aji & Heafield , 2017 ; Sun et al. , 2017 ; Lin et al. , 2018b ; Stich et al. , 2018 ) . Allowing asynchronous updates provides an alternative solution to disguise the communication overhead to a certain amount ( Niu et al. , 2011 ; Sa et al. , 2015 ; Lian et al. , 2015 ) , though alternative strategies might be better when high accuracy is desired ( Chen et al. , 2016 ) . The analysis of Agarwal & Duchi ( 2011 ) shows that asynchronous SGD on convex functions can tolerated delays up to O ( √ T/K ) , which is identical to the maximal length of the local sequences in local SGD . Asynchronous SGD converges also for larger delays ( see also ( Zhou et al. , 2018 ) ) but without linear speedup , a similar statement holds for local SGD ( see discussion in Section 3 ) . The current frameworks for the analysis of asynchronous SGD do not cover local SGD . A fundamental difference is that asynchronous SGD maintains a ( almost ) synchronized sequence and gradients are computed with respect this unique sequence ( but just applied with delays ) , whereas each worker in local SGD evolves a different sequence and computes gradient with respect those iterates . For the training of deep neural networks , Bijral et al . ( 2016 ) discuss a stochastic averaging schedule whereas Zhang et al . ( 2016 ) study local SGD with more frequent communication at the beginning of the optimization process . The elastic averaging technique ( Zhang et al. , 2015 ) is different to local SGD , as it uses the average of the iterates only to guide the local sequences but does not perform a hard reset after averaging . Among the first theoretical studies of local SGD in the non-convex setting are ( Coppola , 2015 ; Zhou & Cong , 2018 ) that did not establish a speedup , in contrast to two more recent analyses ( Yu et al. , 2018 ; Wang & Joshi , 2018 ) . Yu et al . ( 2018 ) show linear speedup of local SGD on non-convex functions for H = O ( T 1/4K−3/4 ) , which is more restrictive than the constraint on H in the convex setting . Lin et al . ( 2018a ) study empirically hierarchical variants of local SGD . Local SGD with averaging in every step , i.e . H = 1 , is identical to mini-batch SGD . Dekel et al . ( 2012 ) show that batch sizes b = T δ , for δ ∈ ( 0 , 12 ) are asymptotically optimal for mini-batch SGD , however they also note that this asymptotic bound might be crude for practical purposes . Similar considerations might also apply to the asymptotic upper bounds on the communication frequency H derived here . Local SGD with averaging only at the end , i.e . H = T , is identical to one-shot SGD . Jain et al . ( 2018 ) give concise speedup results in terms of bias and variance for one-shot SGD with constant stepsizes for the optimization of quadratic least squares problems . In contrast , our upper bounds become loose when H → T and our results do not cover one-shot SGD . Recently , Woodworth et al . ( 2018 ) provided a lower bound for parallel stochastic optimization ( in the convex setting , and not for strongly convex functions as considered here ) . The bound is not known to be tight for local SGD .

Summary: The authors of this paper analyze a well known technique for parallel training, where each compute node locally trains a model with SGD, and once in a while the K compute nodes average their models. Local SGD, although not as widely used as mini-batch SGD, can provide some gains in terms of the cost of communication. This can be achieved by decreasing the frequency of synchronization, while locally also increasing the minibatch.