PopSGD: Decentralized Stochastic Gradient Descent in the Population Model

1 INTRODUCTION . Distributed machine learning has become commonplace , and it is not unusual to encounter systems which distribute model training among tens or even hundreds of nodes . In this paper , we take this trend to the extreme , and ask : would it be possible to distribute basic optimization procedures such as stochastic gradient descent ( SGD ) to thousands of agents ? How could the dynamics be implemented in such a large-scale setting , and what would be with the resulting convergence and speedup behavior ? To get some intuition , let us consider the classical data-parallel distribution strategy for SGD Bottou ( 2010 ) . We are in the classical empirical risk minimization setting , where we have a set of samples S from a distribution , and wish to minimize the function f : Rd → R , which is the average of losses over samples from S by finding x ? = argmin x ∑ s∈S fs ( x ) /|S| . Assume that we have P compute nodes which can process samples in parallel . Data-parallel SGD consists of parallel iterations , in which each node computes the gradient for one sample , followed by a gradient exchange . Globally , this leads to the iteration : xt+1 = xt − ηt P∑ i=1 g̃it ( xt ) , where ηt is the learning rate , xt is the value of the global parameter , initially 0d , and g̃it ( xt ) is the stochastic gradient with respect to the parameter obtained by node i at time t. When extending this strategy to high node counts , two major bottlenecks are communication and synchronization . In particular , to maintain a consistent view of the parameter xt , the nodes would need to broadcast and receive all gradients , and would need to synchronize with all other nodes , at the end of every iteration . Recently , a tremendous amount of work has been dedicated to address these two barriers . In particular , there has been significant progress on communication-reduced variants of SGD ( e.g . Seide et al . ( 2014 ) ; Strom ( 2015 ) ; Alistarh et al . ( 2017b ) ; Wen et al . ( 2017 ) ; Aji and Heafield ( 2017 ) ; Dryden et al . ( 2016 ) ; Grubic et al . ( 2018 ) ) , asynchronous variants ( e.g . Recht et al . ( 2011 ) ; Sa et al . ( 2015 ) ; Duchi et al . ( 2015 ) ; Alistarh et al . ( 2018b ) ) , as well as large-batch or periodic model averaging methods , which aim to reduce the frequency of communication ( e.g . Goyal et al . ( 2017 ) ; You et al . ( 2017 ) and Chen and Huo ( 2016 ) ; Stich ( 2018 ) ) , or even decentralized synchronous variants ( e.g . Lian et al . ( 2017a ) ; Tang et al . ( 2018 ) ; Koloskova et al . ( 2019 ) ) . Using such techniques , it is possible to scale SGD to hundreds of nodes , even for complex objectives such as the training of deep neural networks . However , in systems with node counts in the thousands or larger , some of the communication and synchronization requirements of these algorithms become infeasible . In this paper , we consider the classic population model of distributed computing Angluin et al . ( 2006 ) , defined as follows . We are given a population of n compute agents , each with its own input , which cooperate to perform some globally meaningful computation with respect to their inputs . Interactions occur pairwise , where the two interaction partners are randomly chosen in every step . Thus , algorithms are specified in terms of the agents ’ state transitions upon an interaction . The basic unit of time is a single pairwise interaction between two nodes , whereas global ( parallel ) time is measured as the total number of interactions divided by n , the number of nodes . Parallel time corresponds intuitively to the average number of interactions per node to reach convergence . Population protocols have a rich history in distributed computing ( e.g . Angluin et al . ( 2006 ; 2007 ; 2008a ; b ; c ) ; Alistarh et al . ( 2017a ; 2018a ) ) , and are standard in modelling distributed systems with millions or billions of nodes , such as Chemical Reaction Networks ( CRNs ) Bower and Bolouri ( 2004 ) ; Chen et al . ( 2017 ) and synthetic DNA strand displacement cascades Chen et al . ( 2013 ) . The key difference between population protocols and the synchronous gossip models ( e.g . Xiao and Boyd ( 2004 ) ; Lian et al . ( 2017a ) ; Koloskova et al . ( 2019 ) ) previously used to analyze decentralized SGD is that nodes are not synchronized : since interactions occur randomly at arbitrary times , there are no global rounds , and nodes lack a common notion of time . While the population model is a theoretical construct , we show that it can be efficiently mapped to large-scale super-computing scenarios , with large numbers of compute nodes connected by a fast point-to-point interconnect , where we can avoid the high costs of global synchronization . An immediate instantiation of SGD in the population model would be to initially assign one sample si from the distribution to each node i , and have each node maintain its own parameter estimate xi . Whenever two nodes interact , they exchange samples , and each performs a gradient update with respect to the other ’ s sample . If we assume interaction pairs are uniform random ( with replacement ) , each node would obtain a stochastic gradient upon each interaction , and therefore each model would converge locally . However , this instance would not have any parallel speedup , since the SGD instances at each node are essentially independent . In this context , a natural change to the above procedure is to have nodes i and j first perform a gradient step , and then also average their resulting models upon every interaction . Effectively , if node i interacts with node j , node i ’ s updated model becomes xi ← x i + xj 2 − ηi g̃ i ( xi ) + g̃j ( xj ) 2 , ( 1.1 ) where j is the interaction partner , and the stochastic gradients g̃i and g̃j are taken with respect to each other ’ s samples . The update for node j is symmetric . In this paper , we analyze a variant of the above protocol , which we call PopSGD , in the population protocol model . We show that , perhaps surprisingly , this simple decentralized SGD averaging dynamic provides strong convergence guarantees for both convex and non-convex objectives . First , we prove that , under standard convexity and smoothness assumptions , PopSGD has convergence speedup that linear in the number of nodes n. Second , we show that , if the objective is non-convex but satisfies the Polyak-Łojasiewicz ( PL ) assumption , PopSGD can still ensure linear convergence in the number of nodes . Third , we show a Θ ( √ n ) speedup in the non-convex case in the absence of this assumption , matching or slightly improving results from previous work which considered similar models ( Lian et al. , 2017b ) . On the practical side , we provide convergence and speedup results using an efficient implementation of PopSGD using Pytorch/MPI applied to regression tasks , but also to the standard CIFAR/ImageNet classification tasks for deployments on a multi-GPU nodes , and on the Piz Daint supercomputer ( Piz ) . Experiments confirm the scalability of PopSGD . We also observe an improvement in convergence versus number of SGD iterations per model at higher node counts , in both convex and non-convex settings . In particular , using PopSGD , we are able to train the ResNet18 and ResNet50 He et al . ( 2016 ) models to full accuracy using only 1/8 the number of SGD updates per model , compared to the sequential baseline , resulting in fast convergence with nearly linear scalability . This suggests that , even though interactions occur only pairwise , uniformly at random , and in an uncoordinated manner , as long as the convergence time is large enough to amortize the information propagation , the protocol enjoys the full parallel speedup of mini-batch SGD with a batch size proportional to the number of nodes . While similar speedup behaviour has been observed in various synchronous models for the convex case– e.g . Stich ( 2018 ) ; Koloskova et al . ( 2019 ) , or for complex accelerated algorithms Hendrikx et al . ( 2018 ) –we are the first to show that SGD does not require the existence of globally synchronized rounds or global communication . Central to our analytic approach is a new technical connection between averaging decentralized SGD and the line of research studying load-balancing processes in computer science ( e.g . Azar et al . ( 1999 ) ; Mitzenmacher ( 2000 ) ; Talwar and Wieder ( 2007 ) ; Peres et al . ( 2015a ) ; Boyd et al . ( 2006 ) ) . Intuitively , we show PopSGD can be viewed as a composition between a set of instances of SGD–each corresponding to one of the local parameters xi–which are loosely coupled via pairwise averaging , whose role is to “ balance ” the models by keeping them well concentrated around their mean , despite the random nature of the pairwise interactions . Our analysis characterizes this concentration , showing that , in essence , the averaging process propagates enough information to globally “ simulate ” SGD with a batch of size Θ ( n ) , even though communication is only performed pairwise . We emphasize that the convexity of the objective function in isolation would not be sufficient to prove this fact : simply averaging n independent SGD models at the end of training would not lead to speedup in the objective ( please see e.g . Stich ( 2018 ) for a detailed discussion ) . Along the way , we overcome non-trivial technical difficulties , such as the lack of a common notion of time among nodes , or the fact that , due to the structure of SGD , this novel load-balancing process exhibits non-trivial correlations within the same round . Related Work . The study of decentralized optimization algorithms dates back to Tsitsiklis ( 1984 ) , and is related to the study of gossip algorithms for information dissemination Kempe et al . ( 2003 ) ; Xiao and Boyd ( 2004 ) ; Boyd et al . ( 2006 ) . Gossip is usually studied in one of two models Boyd et al . ( 2006 ) : synchronous , structured in global rounds , where each node interacts with a randomly chosen neighbor , and asynchronous , where each node wakes up at times given by a local Poisson clock , and picks a random neighbor to interact with . The population model is functionally equivalent to the asynchronous gossip model , since the interaction times in the latter model can be “ discretized ” to lead to pairwise uniform interactions . The key difference between our work and averaging in the gossip model , e.g . Boyd et al . ( 2006 ) , is that their input model is static ( node inputs are fixed , and node estimates must converge to the true mean ) , whereas we study the a dynamic setting , where the models are updated in each round by SGD , and should remain concentrated around the parameter mean as it converges towards the optimum . Several optimization algorithms have been analyzed in this setting Nedic and Ozdaglar ( 2009 ) ; Johansson et al . ( 2009 ) ; Shamir and Srebro ( 2014 ) . Tang et al . ( 2018 ) ; Koloskova et al . ( 2019 ) analyze quantization in the synchronous gossip model . Lian et al . ( 2017a ; b ) ; Assran et al . ( 2018 ) consider SGD-type algorithms in the gossip model . Specifically , they analyze the same SGD averaging dynamic in the non-convex setting . Table 2 in the appendix summarizes their assumptions , results , and rates . Their results are phrased in the synchronous gossip model , in which nodes interact in a sequence of perfect matchings , for which they provide O ( 1/ √ Tn ) convergence rates ( under analytical assumptions ) . Further , they extend their results to a variant of the gossip model where updates can be performed based on stale information . Upon careful examination , we find that their results can be extended to the population protocol/asynchronous gossip model , although at the cost of slower convergence relative to the synchronous case . For Lian et al . ( 2017b ) , the convergence rate bound requires that the total number of iterations is Ω ( n6 ) , while in our case only O ( n4 ) iterations are needed . The difference comes from the fact that our bound on the potential Γ is tighter . For Assran et al . ( 2018 ) , speedup with respect to the number of nodes depends on the parameter C. Which in turn , depends on the dimension of the objective function , number iterations for the graph given by edge sets of all matrices used in averaging to be connected and the diameter of aforementioned graph . Unfortunately , in the population model parameter C will not be a constant and this will eliminate the speedup . Further , these references present scalability results for training neural networks using both synchronous and asynchronous variants of their algorithms , thereby slightly relaxing their analytic assumptions . Relative to this prior work , our contributions are as follows . First , we consider both convex and non-convex objectives . We are the first to show linear speedup in the objective for convex objectives in this decentralized asynchronous model . Our bounds in the non-convex case match or slightly improve those presented above , under similar assumptions . Furthermore , under the PL condition , we are the first to show linear convergence speedup in the non-convex case . Our analysis technique relies on a fine-grained analysis of individual interactions , which is different than that of previous work . From the implementation perspective , the performance of our algorithm is competitive with that of previous methods , notably DA-PSGD Lian et al . ( 2017b ) and SGP Assran et al . ( 2018 ) . Other instances in the literature which consider dynamic interaction models are Nedic et al . Nedic et al . ( 2017 ) , who present a gradient tracking algorithm in a different dynamic graph model , and Hendrickx et al . Hendrikx et al . ( 2018 ) , who achieve exponential convergence rates in a gossip model where transmissions are synchronized across edges . The algorithm they consider is a more complex instance of accelerated coordinate descent , and is therefore quite different from the simple dynamics we consider . Neither reference considers large-scale deployments for non-convex objectives .

Summary: The paper considers scaling distributed stochastic gradient descent to large number of nodes.  Paper proposes novel asynchronous variant to decentralized SGD, called PopSGD. It models asynchrony with the population model. Paper theoretically analyzes the proposed method and shows that in the convex case PopSGD has a linear speedup in the number of nodes compared to the sequential training on one node. However, in the non-convex setting, the PopSGD rate doesn’t have a linear speedup. The paper also provides experimental evaluation of PopSGD where they scale PopSGD up to 1000 of nodes in the convex optimization; and also apply PopSGD to neural network training on ImageNet.