Exponential Graph is Provably Efficient for Decentralized Deep Training

1 Introduction . Efficient distributed training methods across multiple computing nodes are critical for large-scale modern deep learning tasks . Parallel stochastic gradient descent ( SGD ) is a widely-used approach , which , at each iteration , computes a globally averaged gradient either using Parameter-Server [ 28 ] or All-Reduce [ 47 ] . Such global coordination across all nodes in parallel SGD results in either significant bandwidth cost or high latency , which can notably hamper the training scalability . Decentralized SGD [ 45 , 11 , 30 , 3 ] based on partial averaging has been one of the promising alternatives to parallel SGD in distributed deep training . Partial averaging , as opposed to the global averaging exploited in parallel SGD , only requires each node to compute the locally averaged model within its neighborhood . Decentralized SGD does not involve any global operations , so it has much lower communication overhead per iteration . The fewer neighbors each node needs to communicate , the more efficient the per-iteration communication is in decentralized SGD . ∗Equal Contribution . Corresponding Author : Wotao Yin 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The reduced communication in decentralized SGD comes with a cost : slower convergence . While it can asymptotically achieve the same convergence linear speedup as parallel SGD [ 30 , 3 , 25 , 64 ] , i.e. , the training speed increases proportionally to the number of computing nodes ( see the definition in Sec . 2 ) , decentralized SGD requires more iterations to reach that stage due to the ineffectiveness to aggregate information using partial averaging . We refer those iterations before decentralized SGD reaches its linear speedup stage as transient iterations ( see the definition in Sec . 2 ) , which is an important metric to measure the influence of partial-averaging [ 48 , 65 ] on convergence rate of decentralized SGD . The less effective the partial averaging is , the more transient iterations decentralized SGD needs to take . Fig . 1 illustrates the transient iterations of decentralized SGD for the logistic regression problem . It is observed that decentralized SGD can asymptotically converge as fast as parallel SGD , but it requires more iterations ( i.e. , transient iterations ) to reach that stage . Per-iteration communication and transient iterations in decentralized SGD are determined by the network topology ( we also use graph interchangeably with topology ) . The maximum degree of the graph decides the communication cost while the connectivity influences the transient iteration complexity . Generally speaking , a sparsely-connected topology communicates cheaply but endows decentralized SGD with more transient iterations due to the less effective information aggregation . A skillful choice of network topology , which is critical to achieve balance between periteration communication and transient iteration complexity , is under-studied in literature . This work studies exponential graphs which are empirically successful [ 3 , 61 , 27 , 14 , 67 ] but less theoretically understood in deep training . Exponential graphs have two variants . In a static exponential graph , each node communicates to dlog2 ( n ) e neighbors ( see Sec . 3 and Fig . 2 ) . In one-peer exponential graph , however , each node cycles through all its neighbors , communicating , only , to a single neighbor per iteration ( see Sec . 4 and Fig . 2 ) . This paper will first clarify the connectivity and averaging effectiveness of these exponential graphs , and then apply them to decentralized momentum SGD to obtain the state-of-the-art balance between per-iteration communication and transient iteration complexity among all commonly-used topologies . Our main results ( as well as our contributions ) are : • We prove that the spectral gap , which is used to measure the connectivity of the graph ( see the definition in Sec . 2 ) , of the static exponential graph is upper bounded by O ( 1/ log2 ( n ) ) . Before us , many literatures ( e.g . [ 27 ] ) claimed its upper bound to be O ( 1 ) incorrectly . • Since one-peer exponential graphs are time-varying , it is difficult to derive their spectral gaps . However , we establish that any log2 ( n ) consecutive sequence of one-peer exponential graphs can together achieve exact averaging when n is a power of 2 . • With the above results , we establish that one-peer exponential graph , though much sparser than its static counterpart , surprisingly endows decentralized momentum SGD with the same convergence rate as static exponential graph in terms of the best-known bounds . • We derive that exponential graphs achieve Ω̃ ( 1 ) 2 per-iteration communication and Ω̃ ( n3 ) transient iterations , both of which are nearly the best among other known topologies , see Table 1 . The one-peer exponential graph is particularly recommended for decentralized deep training . • We conduct extensive industry-level experiments across different tasks and models with various decentralized methods , graphs , and network size to validate our theoretical results . 2Notation Ω̃ ( · ) hides all logarithm factors . 2 Revisit Decentralized Momentum SGD and Related Works . This section reviews basic concepts and existing results on decentralized momentum SGD . Problem . Suppose n computing nodes cooperate to solve the distributed optimization problem : min x∈Rd f ( x ) = 1 n n∑ i=1 fi ( x ) where fi ( x ) : = Eξi∼DiF ( x ; ξi ) . ( 1 ) Function fi ( x ) is local to node i , and random variable ξi denotes the local data that follows distribution Di . We do not assume each distribution Di is the same across all nodes . Network topology and weights . Decentralized methods are based on partial averaging within neighborhood that is defined by the network topology ( see the figure 2 as an example of six nodes ) . We assume all computing nodes are connected by a ( directed or undirected ) network topology . We define wij , the weight to scale information flowing from node j to node i , as follows : wij { > 0 if node j is connected to i , or i = j ; = 0 otherwise . ( 2 ) Ni : = { j|wij > 0 } is defined as the set of neighbors of node i which also includes node i itself and the weight matrix W : = [ wij ] ni , j=1 ∈ Rn×n are denoted as a matrix that stacks the weights of all nodes . This matrix W characterizes the sparsity and connectivity of the underlying network topology . Algorithm 1 DmSGD Initialize γ , x ( 0 ) i ; let m ( 0 ) i = 0 , β ∈ ( 0 , 1 ) For k = 0 , 1 , 2 , ... , T − 1 , every node i do Sample weight matrix W ( k ) ; Update gradient g ( k ) i = ∇F ( x ( k ) i ; ξ ( k ) i ) ; m ( k+1 ) i = ∑ j∈Ni w ( k ) ij ( βm ( k ) j + g ( k ) j ) ; x ( k+1 ) i = ∑ j∈Ni w ( k ) ij ( x ( k ) j − γm ( k ) j ) ; Decentralized momentum SGD ( DmSGD ) . There are many variants of decentralized momentum SGD [ 3 , 20 , 32 , 67 ] . This paper will focus on the one proposed by [ 64 ] ( listed in Algorithm 1 ) , which imposes an additional partialaveraging over the momentum to achieve further speed up . The topology is allowed to change with iterations . When W k ≡W , topology and weight matrix will remain static . Assumptions . We introduce several standard assumptions to facilitate future analysis : A.1 [ SMOOTHNESS ] Each fi ( x ) is L-smooth , i.e. , ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x−y‖ for any x , y ∈ Rd . A.2 [ GRADIENT NOISE ] The random sample ξ ( k ) i is independent of each other for any k and i . We also assume E [ ∇F ( x ; ξi ) ] = ∇fi ( x ) and E‖∇F ( x ; ξi ) −∇fi ( x ) ‖2 ≤ σ2 . A.3 [ DATA HETEROGENEITY ] It holds that 1n ∑n i=1 ‖∇fi ( x ) −∇f ( x ) ‖2 ≤ b2 for any x ∈ Rd . A.4 [ WEIGHT MATRIX ] The weight matrix W ( k ) is doubly-stochastic , i.e . W ( k ) 1 = 1 and 1TW ( k ) = 1T . If W ( k ) ≡ W , we assume ρ ( W ) : = maxλi ( W ) 6=1 { |λi ( W ) | } ∈ ( 0 , 1 ) , where λi ( W ) is the i−th eigenvalue of the matrix W .3 The quantity 1− ρ , which is also referred to as the spectral gap of the weight matrix W , measures how well the topology is connected [ 53 ] . In the large and sparse topology which is most valuable to deep training , it typically holds that 1− ρ→ 0 . 3When there is no ambiguity , we simply use ρ instead of ρ ( W ) . Throughout the paper we do NOT use ρ as spectral radius . Instead , it is the second largest eigenvalue in magnitude . Note we can not directly sort the eigenvalues since W is not necessarily symmetric and the eigenvalue can be a complex number . Communication overhead . According to [ 5 ] , global averaging across n nodes either incurs Ω ( n ) bandwidth cost via Parameter-Server , or Ω ( n ) latency via Ring-Allreduce . In either way , it takes Ω ( n ) per-iteration communication time , which is proportional to the network size n. As to decentralized methods , we will similarly assume the per-iteration communication time to be Ω ( maximum degree ) . Convergence . Under Assumptions A.1–A.4 , DmSGD with static topology will converge at [ 64 , 25 ] : 1 T T∑ k=1 E ‖∇f ( x̄ ( k ) ) ‖2 = O ( σ2√ nT + nσ2 T ( 1− ρ ) + nb2 T ( 1− ρ ) 2 ) ( 3 ) in which x̄ ( k ) = 1n ∑n i=1 x ( k ) i . It is worth noting that no analysis in literature , to our knowledge , exists for DmSGD over time-varying topologies with non-convex costs . Linear speedup . When T is sufficiently large , the first term 1/ √ nT dominates ( 3 ) . This also applies to parallel SGD . Decentralized and parall SGDs all require T = Ω ( 1/ ( n 2 ) ) iterations to reach a desired accuracy , which is inversely proportional to n. Therefore , an algorithm is in its linear-speedup stage at T th iteration if , for this T , the term involving nT is dominating the rate . Transient iterations . Transient iterations are referred to those iterations before an algorithm reaches linear-speedup stage , that is when T is relatively small so non-nT terms still dominate the rate ( see illustration in Appendix C ) . To reach linear speedup , T has to satisfy ( derivation in Appendix C ) Homogeneous data : T = Ω ( n3 ( 1− ρ ) 2 ) Heterogeneous data : T = Ω ( n3 ( 1− ρ ) 4 ) ( 4 ) which corresponds to the transient iteration complexity in the homo/hetero-geneous data scenarios .

Summary: This paper studies the exponential graph used for decentralized training in theory, and shows it effectively reduces the convergence bound and speeds up the deep learning tasks. A novel analysis for the spectral gap bound on exponential graph is given, which is tighter than the one shown in previous papers. Various experiments on large-scale vision tasks are conducted, the results seem to be aligned with the theory.