Exponential Graph is Provably Efficient for Decentralized Deep Training

1 Introduction . Efficient distributed training methods across multiple computing nodes are critical for large-scale modern deep learning tasks . Parallel stochastic gradient descent ( SGD ) is a widely-used approach , which , at each iteration , computes a globally averaged gradient either using Parameter-Server [ 28 ] or All-Reduce [ 47 ] . Such global coordination across all nodes in parallel SGD results in either significant bandwidth cost or high latency , which can notably hamper the training scalability . Decentralized SGD [ 45 , 11 , 30 , 3 ] based on partial averaging has been one of the promising alternatives to parallel SGD in distributed deep training . Partial averaging , as opposed to the global averaging exploited in parallel SGD , only requires each node to compute the locally averaged model within its neighborhood . Decentralized SGD does not involve any global operations , so it has much lower communication overhead per iteration . The fewer neighbors each node needs to communicate , the more efficient the per-iteration communication is in decentralized SGD . ∗Equal Contribution . Corresponding Author : Wotao Yin 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The reduced communication in decentralized SGD comes with a cost : slower convergence . While it can asymptotically achieve the same convergence linear speedup as parallel SGD [ 30 , 3 , 25 , 64 ] , i.e. , the training speed increases proportionally to the number of computing nodes ( see the definition in Sec . 2 ) , decentralized SGD requires more iterations to reach that stage due to the ineffectiveness to aggregate information using partial averaging . We refer those iterations before decentralized SGD reaches its linear speedup stage as transient iterations ( see the definition in Sec . 2 ) , which is an important metric to measure the influence of partial-averaging [ 48 , 65 ] on convergence rate of decentralized SGD . The less effective the partial averaging is , the more transient iterations decentralized SGD needs to take . Fig . 1 illustrates the transient iterations of decentralized SGD for the logistic regression problem . It is observed that decentralized SGD can asymptotically converge as fast as parallel SGD , but it requires more iterations ( i.e. , transient iterations ) to reach that stage . Per-iteration communication and transient iterations in decentralized SGD are determined by the network topology ( we also use graph interchangeably with topology ) . The maximum degree of the graph decides the communication cost while the connectivity influences the transient iteration complexity . Generally speaking , a sparsely-connected topology communicates cheaply but endows decentralized SGD with more transient iterations due to the less effective information aggregation . A skillful choice of network topology , which is critical to achieve balance between periteration communication and transient iteration complexity , is under-studied in literature . This work studies exponential graphs which are empirically successful [ 3 , 61 , 27 , 14 , 67 ] but less theoretically understood in deep training . Exponential graphs have two variants . In a static exponential graph , each node communicates to dlog2 ( n ) e neighbors ( see Sec . 3 and Fig . 2 ) . In one-peer exponential graph , however , each node cycles through all its neighbors , communicating , only , to a single neighbor per iteration ( see Sec . 4 and Fig . 2 ) . This paper will first clarify the connectivity and averaging effectiveness of these exponential graphs , and then apply them to decentralized momentum SGD to obtain the state-of-the-art balance between per-iteration communication and transient iteration complexity among all commonly-used topologies . Our main results ( as well as our contributions ) are : • We prove that the spectral gap , which is used to measure the connectivity of the graph ( see the definition in Sec . 2 ) , of the static exponential graph is upper bounded by O ( 1/ log2 ( n ) ) . Before us , many literatures ( e.g . [ 27 ] ) claimed its upper bound to be O ( 1 ) incorrectly . • Since one-peer exponential graphs are time-varying , it is difficult to derive their spectral gaps . However , we establish that any log2 ( n ) consecutive sequence of one-peer exponential graphs can together achieve exact averaging when n is a power of 2 . • With the above results , we establish that one-peer exponential graph , though much sparser than its static counterpart , surprisingly endows decentralized momentum SGD with the same convergence rate as static exponential graph in terms of the best-known bounds . • We derive that exponential graphs achieve Ω̃ ( 1 ) 2 per-iteration communication and Ω̃ ( n3 ) transient iterations , both of which are nearly the best among other known topologies , see Table 1 . The one-peer exponential graph is particularly recommended for decentralized deep training . • We conduct extensive industry-level experiments across different tasks and models with various decentralized methods , graphs , and network size to validate our theoretical results . 2Notation Ω̃ ( · ) hides all logarithm factors . 2 Revisit Decentralized Momentum SGD and Related Works . This section reviews basic concepts and existing results on decentralized momentum SGD . Problem . Suppose n computing nodes cooperate to solve the distributed optimization problem : min x∈Rd f ( x ) = 1 n n∑ i=1 fi ( x ) where fi ( x ) : = Eξi∼DiF ( x ; ξi ) . ( 1 ) Function fi ( x ) is local to node i , and random variable ξi denotes the local data that follows distribution Di . We do not assume each distribution Di is the same across all nodes . Network topology and weights . Decentralized methods are based on partial averaging within neighborhood that is defined by the network topology ( see the figure 2 as an example of six nodes ) . We assume all computing nodes are connected by a ( directed or undirected ) network topology . We define wij , the weight to scale information flowing from node j to node i , as follows : wij { > 0 if node j is connected to i , or i = j ; = 0 otherwise . ( 2 ) Ni : = { j|wij > 0 } is defined as the set of neighbors of node i which also includes node i itself and the weight matrix W : = [ wij ] ni , j=1 ∈ Rn×n are denoted as a matrix that stacks the weights of all nodes . This matrix W characterizes the sparsity and connectivity of the underlying network topology . Algorithm 1 DmSGD Initialize γ , x ( 0 ) i ; let m ( 0 ) i = 0 , β ∈ ( 0 , 1 ) For k = 0 , 1 , 2 , ... , T − 1 , every node i do Sample weight matrix W ( k ) ; Update gradient g ( k ) i = ∇F ( x ( k ) i ; ξ ( k ) i ) ; m ( k+1 ) i = ∑ j∈Ni w ( k ) ij ( βm ( k ) j + g ( k ) j ) ; x ( k+1 ) i = ∑ j∈Ni w ( k ) ij ( x ( k ) j − γm ( k ) j ) ; Decentralized momentum SGD ( DmSGD ) . There are many variants of decentralized momentum SGD [ 3 , 20 , 32 , 67 ] . This paper will focus on the one proposed by [ 64 ] ( listed in Algorithm 1 ) , which imposes an additional partialaveraging over the momentum to achieve further speed up . The topology is allowed to change with iterations . When W k ≡W , topology and weight matrix will remain static . Assumptions . We introduce several standard assumptions to facilitate future analysis : A.1 [ SMOOTHNESS ] Each fi ( x ) is L-smooth , i.e. , ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x−y‖ for any x , y ∈ Rd . A.2 [ GRADIENT NOISE ] The random sample ξ ( k ) i is independent of each other for any k and i . We also assume E [ ∇F ( x ; ξi ) ] = ∇fi ( x ) and E‖∇F ( x ; ξi ) −∇fi ( x ) ‖2 ≤ σ2 . A.3 [ DATA HETEROGENEITY ] It holds that 1n ∑n i=1 ‖∇fi ( x ) −∇f ( x ) ‖2 ≤ b2 for any x ∈ Rd . A.4 [ WEIGHT MATRIX ] The weight matrix W ( k ) is doubly-stochastic , i.e . W ( k ) 1 = 1 and 1TW ( k ) = 1T . If W ( k ) ≡ W , we assume ρ ( W ) : = maxλi ( W ) 6=1 { |λi ( W ) | } ∈ ( 0 , 1 ) , where λi ( W ) is the i−th eigenvalue of the matrix W .3 The quantity 1− ρ , which is also referred to as the spectral gap of the weight matrix W , measures how well the topology is connected [ 53 ] . In the large and sparse topology which is most valuable to deep training , it typically holds that 1− ρ→ 0 . 3When there is no ambiguity , we simply use ρ instead of ρ ( W ) . Throughout the paper we do NOT use ρ as spectral radius . Instead , it is the second largest eigenvalue in magnitude . Note we can not directly sort the eigenvalues since W is not necessarily symmetric and the eigenvalue can be a complex number . Communication overhead . According to [ 5 ] , global averaging across n nodes either incurs Ω ( n ) bandwidth cost via Parameter-Server , or Ω ( n ) latency via Ring-Allreduce . In either way , it takes Ω ( n ) per-iteration communication time , which is proportional to the network size n. As to decentralized methods , we will similarly assume the per-iteration communication time to be Ω ( maximum degree ) . Convergence . Under Assumptions A.1–A.4 , DmSGD with static topology will converge at [ 64 , 25 ] : 1 T T∑ k=1 E ‖∇f ( x̄ ( k ) ) ‖2 = O ( σ2√ nT + nσ2 T ( 1− ρ ) + nb2 T ( 1− ρ ) 2 ) ( 3 ) in which x̄ ( k ) = 1n ∑n i=1 x ( k ) i . It is worth noting that no analysis in literature , to our knowledge , exists for DmSGD over time-varying topologies with non-convex costs . Linear speedup . When T is sufficiently large , the first term 1/ √ nT dominates ( 3 ) . This also applies to parallel SGD . Decentralized and parall SGDs all require T = Ω ( 1/ ( n 2 ) ) iterations to reach a desired accuracy , which is inversely proportional to n. Therefore , an algorithm is in its linear-speedup stage at T th iteration if , for this T , the term involving nT is dominating the rate . Transient iterations . Transient iterations are referred to those iterations before an algorithm reaches linear-speedup stage , that is when T is relatively small so non-nT terms still dominate the rate ( see illustration in Appendix C ) . To reach linear speedup , T has to satisfy ( derivation in Appendix C ) Homogeneous data : T = Ω ( n3 ( 1− ρ ) 2 ) Heterogeneous data : T = Ω ( n3 ( 1− ρ ) 4 ) ( 4 ) which corresponds to the transient iteration complexity in the homo/hetero-geneous data scenarios .

Summary: The work proposes efficient communication topologies for Decentralized momentum Stochastic Gradient Descent (DmSGD)  algorithms used for training deep learning models. In particular, they focus on exponential graphs, where each nodes is connected to O(log(n)) other nodes. They analyze the spectral gap, which shows the graph connectivity, for exponential graphs and establish a logarithmic bound on it and claim that it was reported incorrectly in the literature to be constant.  Moreover, they propose to communicate over one-peer exponential graphs, in which each node communicates to a single neighbor. They theoretically show that a sequence of O(log(n))  one-peer exponential graphs achieve the same convergence rates, despite being far more efficient in terms of communication. They further confirm the effectiveness of their proposed method by running experiments on real datasets and applications and comparing them to other topologies and methods.