On the Effect of Consensus in Decentralized Deep Learning

1 INTRODUCTION . Highly over-parametrized deep neural networks show impressive results in machine learning tasks , which also lead to a dramatic increase in the size , complexity , and computational power of the training systems . In response to these challenges , distributed training algorithms ( i.e . data-parallel large mini-batch SGD ) have been developed for use in data-center ( Goyal et al. , 2017 ; You et al. , 2018 ; Shallue et al. , 2018 ) . These SOTA training systems in general use the All-Reduce communication primitive to perform exact averaging on the local mini-batch gradients computed on different subsets of the data , for the later synchronized model update . However , exact averaging with All-Reduce is sensitive to the communication hardware of the training systems , causing the bottleneck in efficient deep learning training . To this end , decentralized training has become an indispensable training paradigm for efficient large scale training in the data-center , alongside its orthogonal benefits on preserving users ’ privacy for edge AI ( Bellet et al. , 2018 ; Kairouz et al. , 2019 ) . Several very recent papers aim to address the communication overhead in data-center training by compressing the gradients ( Koloskova et al. , 2020a ; Vogels et al. , 2020 ) or designing better a communication topology ( Assran et al. , 2019 ) . However , decentralized training for deep learning models still often results in a severe loss in generalization performance , even after hyper-parameter fine-tuning ( see our Table 1 as well as Tables 1 , 2 , 3 in Assran et al. , 2019 ) . This phenomenon is poorly understood even in relatively straightforward i.i.d . data distribution scenarios ( i.e . the data-center case ) , to which very few works are dedicated ( but none of them provide insights into the generalization gap ) . In this work , we theoretically identify the consensus distance , i.e . the average discrepancy between the nodes , as the key parameter that captures the joint effect of decentralization . We show that there exists a critical consensus distance : when the consensus distance is lower than this critical value , the optimization is almost unhindered . With the insight derived from optimization convergence , we further question the existence of the critical consensus distance for deep learning in terms of generalization , and identify the training phase when the critical consensus distance matters . We believe that the answers to these questions are valuable in practice , as they offer the possibility to design training strategies that strike appropriate trade-off between targeted generalization performance and affordable communication resources . • We show that tracking the consensus distance over the training phases can explain the generalization gap between centralized and decentralized training . • We show theoretically that when the consensus distance does not exceed a critical value , then decentralization exerts negligible impact to the optimization . We argue how this critical value depends crucially on the learning rate and the mixing matrix . • Through the lens of consensus distance , we empirically investigate what is the desirable level of consensus distance during different phases of training , in order to ensure high generalization performance . Our extensive experiments on Computer Vision ( CV ) tasks ( CIFAR-10 and ImageNet-32 ) as well as Natural Language Processing ( NLP ) tasks ( transformer models for machine translation ) do confirm that a critical consensus distance indeed exists , and that consensus distances higher than this critical value heavily penalize the final generalization performance . Surprisingly , we find that large consensus distance might be beneficial in later training phases after optimization has plateaued , leading to improved generalization , which is consistent with the observations in centralized Post-Local SGD ( Lin et al. , 2020b ) . • Based on our findings , we propose practical guidelines on how to achieve favorable generalization performance with low communication expenses , on arbitrary communication networks . While our numerical study mainly focuses on the data-center setting with homogeneous nodes , our findings also apply to decentralized training over time-varying topologies and the more difficult heterogeneous setting alike . Our findings not only explain previously proposed ad-hoc solutions ( e.g . using All-Reduce in the first training phase as in Assran et al. , 2019 ) but our insights will allow the principled design of better decentralized training systems in future work . 2 RELATED WORK . Gossip averaging ( Kempe et al. , 2003 ; Xiao & Boyd , 2004 ; Boyd et al. , 2006 ) forms the backbone of many decentralized learning algorithms . The convergence rate of gossip averaging towards consensus among the nodes can be expressed in terms of the ( expected ) spectral gap of the mixing matrix . Lian et al . ( 2017 ) combine SGD with gossip averaging for deep learning and show that the leading term in the convergence rate O ( 1 nε2 ) is consistent with the convergence of the centralized mini-batch SGD ( Dekel et al. , 2012 ) and the spectral gap only affects the asymptotically smaller terms . Similar results have been observed very recently for related schemes ( Scaman et al. , 2017 ; 2018 ; Koloskova et al. , 2019 ; 2020a ; b ) . As the communication topology also impacts the cost per round ( number of peer-to-peer communications ) , sparse topologies have been proposed and studied recently ( Assran et al. , 2019 ; Wang et al. , 2019 ; Nadiradze et al. , 2020 ) . Whilst a few recent works focus on the impact of the topology on the optimization performance ( Luo et al. , 2019 ; Neglia et al. , 2020 ) , we here identify the consensus distance as a more canonical parameter that can characterize the overall effect of decentralized learning , beyond only the topology , through which we are able to provide deeper understanding on the more fine-grained impact of the evolution of the actual consensus distance , on the generalization performance of deep learning training . Prior work identified the consensus distance as an important parameter that can affect optimization performance and convergence , and provide approaches to increase consensus : for instance Scaman et al . ( 2017 ) ; Sharma et al . ( 2019 ) perform multiple consensus steps per round , and Tsitsiklis ( 1984 ) ; Nedić & Ozdaglar ( 2009 ) ; Duchi et al . ( 2012 ) ; Yuan et al . ( 2016 ) choose carefully tuned learning rates . However , these work do not provide insights into how consensus distance at different training phases impacts the decentralized training , which is the main target of this work . 3 THEORETICAL UNDERSTANDING . In this section we consider decentralized training with stochastic gradient descent ( D-SGD ) without momentum , but we are using the momentum version in all our DL experiments . 3.1 NOTATION AND SETTING . The agents are tasked to solve a sum-structured optimization problem f : Rd → R of the form f ? : = minx∈Rd [ f ( x ) : = 1n ∑n i=1 fi ( x ) ] , ( 1 ) where the components fi : Rd → R are distributed among the n nodes and are given in stochastic form : fi ( x ) : = Eξ∼Di [ Fi ( x , ξ ) ] , where Di denotes the local data distribution on node i ∈ [ n ] . For data-center settings , where data is re-shuffled periodically among nodes , these distributions are identical , but in other scenarios there can be differences between nodes . In D-SGD , each agent i ∈ [ n ] maintains local parameters x ( t ) i ∈ Rd , and updates them as : x ( t+1 ) i = ∑n j=1 wij ( x ( t ) j − η∇Fj ( x ( t ) j , ξ ( t ) j ) ) , ( D-SGD ) that is , by a stochastic gradient step based on a sample ξ ( i ) i ∼ Di , followed by gossip averaging with neighboring nodes in the network encoded by the mixing weightswij . As parameters can differ across nodes , we define x̄ : = 1n ∑n i=1 xi and X : = [ x1 , . . . , xn ] ∈ Rd×n , and X̄ : = [ x̄ , . . . , x̄ ] ≡ X 1 n11 > . Assumption 1 ( Mixing matrix ) . Every sample of the ( possibly randomized ) mixing matrix W = { wij } ∈ Rn×n is doubly stochastic and there exists a parameter p > 0 such that EW ∥∥XW − X̄∥∥2 F ≤ ( 1− p ) ∥∥X− X̄∥∥2 F , ∀X ∈ Rd×n . ( 2 ) This assumption covers a broad variety of settings ( see e.g . Koloskova et al. , 2020b ) , such as D-SGD with fixed ( constant ) mixing matrix with spectral gap ρ , with parameter p = 1− ( 1− ρ ) 2 = Θ ( ρ ) , but also for randomly chosen mixing matrices , for instance random matchings . Assumption 2 ( L-smoothness ) . Each function fi ( x ) : Rd → R , i ∈ [ n ] is differentiable and there exists a constant L ≥ 0 such that for each x , y ∈ Rd : ‖∇fi ( y ) −∇fi ( x ) ‖ ≤ L ‖x− y‖ . Assumption 3 ( Bounded noise σ and diversity ζ ) . There exists constants σ2 , ζ2 s.t . ∀x1 , . . .xn ∈ Rd 1 n ∑n i=1 Eξi ‖∇Fi ( xi , ξi ) −∇fi ( xi ) ‖ 2 2 ≤ σ2 , 1 n ∑n i=1 ‖∇fi ( xi ) −∇f ( xi ) ‖ 2 2 ≤ ζ2 . ( 3 ) 3.2 DECENTRALIZED CONSENSUS OPTIMIZATION . Under the above assumptions , which are standard in decentralized optimization , the convergence rate of ( D-SGD ) has been shown to be : Theorem 1 ( Koloskova et al . ( 2020b ) ) . Let fi be L-smooth and stepsize γ ≤ γmax = O ( p L ) . Then there exists an optimal stepsize γ ≤ γmax such that 1T ∑T−1 t=0 E ∥∥∇f ( x̄ ( t ) ) ∥∥2 2 ≤ ε for T = O ( σ2 nε2 + √ pσ + ζ pε3/2 + p ε ) · L ( f ( x0 ) − f ? ) . In comparison , for centralized mini-batch SGD ( C-SGD ) we are allowed to choose a potentially much larger stepsize γ ≤ O ( 1 L ) , and can bound the number of iterations by O ( σ2 nε2 + 1 ε ) . While asymptotically both these rates are equivalent , they differ in low accuracy setting when ε is not too small . That is , especially in the first phase of optimization where the lower order terms matter . To measure differences between agents , we use the consensus distance Ξ2t : = 1 n ∑n i=1 ∥∥x̄ ( t ) −x ( t ) i ∥∥2 . Remark 2 ( Critical Consensus Distance ( CCD ) ) . If the consensus distance is bounded by Ξ2t ≤ ( 1 Ln γσ2 + 1 8L2 ∥∥∥∇f ( x̄ ( t ) ) ∥∥∥2 = : Γ2t ) ( 4 ) for all t , then in D-SGD we may choose larger stepsizes γ ≤ γ′max = O ( 1 L ) and recover the convergence rate of C-SGD , that is O ( σ2 nε2 + 1 ε ) ( Dekel et al. , 2012 ; Bottou et al. , 2018 ) . We denote Γ2t as critical consensus distance ( CCD ) . The proof can be found in the Appendix A.1 . Note that the CCD does not depend on the graph topology and that Γ2t > 0 , which means that we do not need perfect consensus between agents to recover the C-SGD rate , but we allow consensus distance Ξ2t ≥ 0 ( i.e . the Ξ2t = 0 ∀t , as we have for centralized optimization is sufficient , but not necessary ) . We now estimate the magnitude of the consensus distance in D-SGD and compare it to CCD . Proposition 3 ( Typical consensus distance ) . Let φ2t : = 1n ∑n i=1 ∥∥∇fi ( x ( t ) i ) ∥∥2 . Then under the assumption that γ , p are constant , and the φt do not change too fast between iterations , i.e . not decreasing faster than exponentially : φ2t ≤ ( 1+p/4 ) φ2t+1 , the consensus distance in D-SGD satisfies Ξ2t = ( 1− p ) γ2 · O ( φ2t p2 + σ2 p ) . ( 5 ) We give the proof in the Appendix A.2 . While these assumptions do not hold in epochs with learning rate decay , we observed in practice that when the learning rate is constant indeed the gradients do not change too fast and found it to be a reasonable approximation to capture the practical behavior ( see Figure 5 ( b ) ) .

Summary: This paper studies decentralized gradient methods for training deep networks. It focuses on the so-called "critical consensus distance" and how disagreement during different stages of training ultimately effects optimization (training loss) and learning (generalization error). Theory is provided for the case of synchronous symmetric averaging methods, and the paper is complemented with detailed experiments on CIFAR and tiny-ImageNet.