Decentralized SGD with Asynchronous, Local and Quantized Updates

1 INTRODUCTION . Several techniques have been recently explored for scaling the distributed training of machine learning models , such as communication-reduction , asynchronous updates , or decentralized execution . For background , consider the classical data-parallel distribution strategy for SGD ( Bottou , 2010 ) , with the goal of solving a standard empirical risk minimization problem . Specifically , we have a set of samples S , and wish to minimize the d-dimensional function f : Rd → R , which is the average of losses over samples from S , by finding x ? = argmin x ∑ s∈S ` s ( x ) /|S| . We have n compute nodes which can process samples in parallel . In data-parallel SGD , each node computes the gradient for one sample , followed by a gradient exchange . Globally , this leads to the iteration : xt+1 = xt − ηt n∑ i=1 g̃it ( xt ) , where xt is the value of the global parameter , initially 0d , ηt is the learning rate , and g̃it ( xt ) is the stochastic gradient with respect to the parameter xt , computed by node i at time t. When executing this procedure at large scale , two major bottlenecks are communication , that is , the number of bits transmitted by each node , and synchronization , i.e. , the fact that nodes need to wait for each other in order to progress to the next iteration . Specifically , to maintain a consistent view of the parameter xt above , the nodes need to broadcast and receive all gradients , and need to synchronize globally at the end of every iteration . Significant work has been dedicated to removing these two barriers . In particular , there has been progress on communication-reduced variants of SGD , which propose various gradient compression schemes ( Seide et al. , 2014 ; Strom , 2015 ; Alistarh et al. , 2017 ; Wen et al. , 2017 ; Aji and Heafield , 2017 ; Dryden et al. , 2016 ; Grubic et al. , 2018 ; Davies et al. , 2020 ) , asynchronous variants , which relax the strict iteration-by-iteration synchronization ( Recht et al. , 2011 ; Sa et al. , 2015 ; Duchi et al. , 2015 ) , as well as large-batch or periodic model averaging methods , which aim to reduce the frequency of communication ( Goyal et al. , 2017 ; You et al. , 2017 ) and ( Chen and Huo , 2016 ; Stich , 2018 ) , or decentralized variants , which allow each node to maintain its own , possibly inconsistent , model variant ( Lian et al. , 2017 ; Tang et al. , 2018 ; Koloskova et al. , 2019 ) . ( We refer the reader to the recent surveys of ( Ben-Nun and Hoefler , 2019 ; Liu and Zhang , 2020 ) for a detailed discussion . ) Using such techniques , it is possible to scale SGD , even for complex objectives such as the training of deep neural networks . However , for modern large-scale models , the communication and synchronization requirements of these parallel variants of SGD can still be burdensome . Contribution . In this paper , we take a further step towards removing these scalability barriers , showing that all the previous scaling techniques—decentralization , quantization , asynchrony , and local steps—can in fact be used in conjunction . We consider a highly decoupled setting with n compute agents , located at vertices of a connected communication graph , each of which can execute sequential SGD on its own local model , based on a fraction of the data . Periodically , after some number of local optimization steps , a node can initiate a pairwise interaction with a uniform random neighbor . Our main finding is that this procedure can converge even though the nodes can take several local steps between interactions , may perform asynchronous communication , reading stale versions of each others ’ models , and may compress data transmission through quantization . However , both in theory and practice , we observe trade-offs between convergence rate and degree of synchronization , in that the algorithm may need to perform additional gradient steps in order to attain a good solution , relative to the sequential baseline . Our algorithm , called SWARMSGD , is decentralized in sense that each node maintains local version of the model , and two interacting nodes only see each others ’ models . We further allow that the data distribution at the nodes may not be i.i.d . Specifically , each node i is assigned a set of samples Si , and maintains its own parameter estimate xi . Each node i performs local SGD steps on its model xi based on its local data , and then picks a neighbor uniformly at random to share information with , by averaging of the two models . ( To streamline the exposition , we ignore quantization and model staleness unless otherwise specified . ) Effectively , if node i interacts with node j , node i ’ s updated model becomes xit+1 ← xit , Hi + x j t , Hj 2 , ( 1 ) where t is the total number of interactions performed by all nodes up to this point , j is the interaction partner of i at step t + 1 , and the input models xit , Hi and x j t , Hj have been obtained by iterating the SGD step Hi and Hj times , respectively , locally from the previous interaction of either node . We assume that Hi and Hj are random variables with mean H , that is , each node performs H local steps in expectation between two communication steps . The update for node j is symmetric , so that the two models match after the averaging step . In this paper , we analyze variants of the above SwarmSGD protocol . The main intuition behind the algorithm is that the independent SGD steps will allow nodes to explore local improvements to the objective function on their subset of the data , while the averaging steps provide a decentralized way for the models to converge jointly , albeit in a loosely coupled way . We show that , as long as the maximum number of local steps is bounded , this procedure still converges , in the sense that gradients calculated at the average over all models are vanishing as we increase the number of interactions . Specifically , assuming that the n nodes each take a constant number of local SGD steps on average before communicating , we show that SwarmSGD has Θ ( √ n ) speedup to convergence in the nonconvex case . This matches results from previous work which considered decentralized dynamics but which synchronized upon every SGD step , e.g . ( Lian et al. , 2017 ; 2018 ) . Our analysis also extends to arbitrary regular graph topologies , non-blocking ( delayed ) averaging of iterates , and quantization . Generally , we show that the impact of decentralization , asynchrony , quantization , and local updates can be asymptotically negligible in reasonable parameter regimes . On the practical side , we show that this algorithm can be mapped to a distributed system setting , where agents correspond to compute nodes , connected by a dense communication topology . Specifically , we apply SwarmSGD to train deep neural networks on image classification and machine translation ( NMT ) tasks , deployed on the Piz Daint supercomputer ( Piz , 2019 ) . Experiments confirm the intuition that the average synchronization cost of SwarmSGD per iteration is low : it stays around 10 % or less of the batch computation time , and remains constant as we increase the number of nodes . For example , using SwarmSGD deployed on 16 nodes , we are able to train a TransformerXL ( Vaswani et al. , 2017 ) model on WMT17 ( En-Ge ) 1.5× faster than a highly-optimized largebatch SGD baseline , and to slightly higher accuracy , without additional hyper-parameter tuning . At the same time , our method appears to be faster and more accurate than the previous practical decentralized methods , e.g . ( Lian et al. , 2017 ; 2018 ; Assran et al. , 2018 ) , in the same setting . Importantly , we also note a negative result : in less overparametrized settings such as training residual CNNs ( He et al. , 2016 ) on ImageNet ( Russakovsky et al. , 2015 ) , nodes do need to perform more iterations over the dataset relative to the baseline in order to recover full accuracy . This is predicted by the analysis , and confirms similar findings in previous work ( Assran et al. , 2018 ) . Overall , however , our family of methods should be well-suited to training very large modern models in large-scale settings , where global synchronization among all nodes is prohibitively expensive . Related Work . The study of decentralized optimization algorithms dates back to Tsitsiklis ( 1984 ) , and is related to the study of gossip algorithms for information dissemination ( Kempe et al. , 2003 ; Xiao and Boyd , 2004 ; Boyd et al. , 2006 ) . Gossip is usually studied in one of two models ( Boyd et al. , 2006 ) : synchronous , structured in global rounds , where each node interacts with a randomly chosen neighbor , and asynchronous , where each node wakes up at times given by a local Poisson clock , and picks a random neighbor to interact with . The model we consider can be seen as equivalent to the asynchronous gossip model . The key differences between our work and averaging in the gossip model , e.g . Boyd et al . ( 2006 ) , are that that 1 ) we consider local SGD steps , which would not make sense in the case of averaging fixed initial values ; and 2 ) the gossip input model is static ( node inputs are fixed , and node estimates must converge to the true mean ) , whereas we study a dynamic setting , where models are continually updated via SGD . Several optimization algorithms have been analyzed in this setting ( Nedic and Ozdaglar , 2009 ; Johansson et al. , 2009 ; Shamir and Srebro , 2014 ) , while Tang et al . ( 2018 ) ; Koloskova et al . ( 2019 ) analyze quantization in the synchronous gossip model . Lian et al . ( 2017 ; 2018 ) and Assran et al . ( 2018 ) considered SGD-type algorithms in gossip-like models . Specifically , they analyze the SGD averaging dynamic in the non-convex setting but do not allow nodes to perform local updates or quantize . In particular , nodes perform pairwise averaging upon every SGD step . Table 2 in the Appendix provides a thorough comparison of assumptions , results , and rates . Their results are phrased in the synchronous gossip model , in which nodes interact in a sequence of perfect matchings , for which they provide O ( 1/ √ Tn ) convergence rates under analytical assumptions . Lian et al . ( 2018 ) extends these results to a variant of the gossip model where updates can be performed based on stale information , similarly to our non-blocking extension . Upon careful examination , one can find that their results can be extended to the asynchronous gossip setting we consider , as long as nodes are not allowed to perform local SGD updates to their models ( corresponding toH = 1 ) or to quantize communication . Extending the analysis of distributed SGD to allow for local steps is challenging even in centralized models , see for instance Stich ( 2018 ) . If we assume H = 1 , our technique yields similar or better bounds relative to previous work in the decentralized model , as our potential analysis is specifically-tailored to this dynamic interaction model . For instance , for Assran et al . ( 2018 ) , the speedup with respect to the number of nodes depends on a parameter C , which in turn , depends on 1 ) the dimension d of the objective function , 2 ) the number of iterations for the graph given by edge sets of all matrices used in averaging to be connected , and the 3 ) diameter of the aforementioned connected graph . In the dynamic interaction model we consider , the parameter C will be at least linear in the number of nodes n , which will eliminate any speedup . We present a systematic comparison in Appendix B . In sum , relative to prior work on decentralized algorithms , our contributions are as follows . We are the first to consider the impact of local updates , asynchrony , and quantization in conjunction with decentralized SGD . We show that the cost for the linear reduction in communication in H given by local steps is at worst a squared variance increase in the parameter H . Our analysis technique relies on a fine-grained analysis of individual interactions , which is different than that of previous work , and can yield improved bounds even in the case where H = 1 . By leveraging the lattice-based quantization scheme of Davies et al . ( 2020 ) , we also allow for communication-compression . From the implementation perspective , the performance of our algorithm is superior to that of previous methods , notably D-PSGD ( Lian et al. , 2017 ) , AD-PSGD ( Lian et al. , 2018 ) and SGP ( Assran et al. , 2018 ) , mainly due to the ability to take local steps . Wang and Joshi ( 2018 ) and Koloskova et al . ( 2020 ) provide analysis frameworks for the synchronous version of decentralized SGD with local updates , and possibly changing topologies . This is a different setting from ours , since it requires each agent to take an equal number of gradient steps before every interaction round , and therefore does not allow for agents to progress at different speeds ( asynchrony ) . Further , we support quantization , and validate our analysis at scale .

Summary: The paper proposes and analyses a distributed learning algorithm for training with Stochastic Gradient Descent a global model on a regular graph, that allows for local and asynchronous gradient updates. Nodes continuously update their local models $X^i$ by gradient descent, while they communicate with their peers (a peer at a time) and update their local model with the pair model average $\frac{X^i + X^j}{2}$. Three extensions of the algorithm are also proposed to relax different constraints, while maintaining the convergence guarantees: