Why Does Decentralized Training Outperform Synchronous Training In The Large Batch Setting?

1 INTRODUCTION . Deep Learning ( DL ) has revolutionized AI training across application domains : Computer Vision ( CV ) ( Krizhevsky et al. , 2012 ; He et al. , 2015 ) , Natural Language Processing ( NLP ) ( Vaswani et al. , 2017 ) , and Automatic Speech Recognition ( ASR ) ( Hinton et al. , 2012 ) . Stochastic Gradient Descent ( SGD ) is the fundamental optimization method used in DL training . Due to massive computational requirements , Distributed Deep Learning ( DDL ) is the preferred mechanism to train large scale Deep Learning ( DL ) tasks . In the early days , Parameter Server ( PS ) based Asynchronous SGD ( ASGD ) training was the preferred DDL approach ( Dean et al. , 2012 ; Li et al. , 2014 ) as it did not require strict system-wide synchronization . Recently , ASGD has lost popularity due to its unpredictability and often inferior convergence behavior ( Zhang et al. , 2016b ) . Practitioners now favor deploying Synchronous SGD ( SSGD ) on homogeneous High Performance Computing ( HPC ) systems . The degree of parallelism in a DDL system is dictated by batch size : the larger the batch size , the more parallelism and higher speedup can be expected . However , large batches require a larger learning rate and overall they may negatively affect model accuracy because 1 ) large batch training usually converges to sharp minima which do not generalize well ( Keskar et al. , 2016 ) and 2 ) large learning rates may violate the conditions ( i.e. , the smoothness parameter ) required for convergence in nonconvex optimization theory ( Ghadimi & Lan , 2013 ) . Although training longer with large batches could lead to better generalization ( Hoffer et al. , 2017 ) , doing so gives up some or all of the speedup we seek . Through meticulous hyper-parameter design ( e.g. , learning rate ) tailored to each specific task , SSGD-based DDL systems have enabled large batch training and shortened training time for some challenging CV tasks ( Goyal et al. , 2017 ; You et al. , 2017 ) and NLP tasks ( You et al. , 2019 ) from weeks to hours or less . However , it is observed that SSGD with large batch size leads to large training loss and inferior model quality for ASR tasks ( Zhang et al. , 2019b ) , as illustrated in Figure 1a ( red curve ) . In this paper we found for other types of tasks ( e.g . CV ) and DL models , large batch SSGD has the same problem ( Figure 1b and Figure 1c ) . The cause of this problem could be that training gets trapped at saddle points since large batches reduce the magnitude of noise in the stochastic gradient and prevent the algorithm from exploring the whole parameter space . To solve this problem , one may add isotropic noise ( e.g. , spherical Gaussian ) to help SSGD escape from saddle points ( Ge et al. , 2015 ) . However , this is not a good solution for high-dimensional DL training as shown in the blue curves of Figure 1 . One possible reason is that the complexity of escaping a saddle point by adding isotropic noise has a polynomial dependency on the dimension of the parameter space , so adding such noise in a high dimensional space ( such as deep learning ) does not bring significant benefits . In this paper , we have found that Decentralized Parallel SGD ( DPSGD ) ( Lian et al. , 2017b ) greatly improves large batch training performance , as illustrated in the green curves in Figure 1 . Unlike SSGD , where each learner updates its weights by taking a global average of all learners ’ weights , DPSGD updates each learner ’ s weights by taking a partial average ( i.e. , across a subset of neighboring learners ) . Therefore , in DPSGD , each learner ’ s weights differ from the weights of other learners.1 The key difference among SSGD , SSGD with Gaussian noise 2 and DPSGD is the source of noise during the update , and this noise directly affects performance in deep learning . This naturally motivates us to study Why decentralized training outperform synchronous training in the large batch setting ? More specifically , we try to understand whether their performance difference is caused by their different noise . We answer these questions from both theoretical and empirical perspectives . Our contributions are : • We analyze the dynamics of DDL algorithms , including both SSGD and DPSGD . We show , both theoretically and empirically , that the intrinsic noise in DPSGD can 1 ) reduce the effective learning rate when the gradient is large to help convergence ; 2 ) enhance the search in weight space for flat minima with better generalization . • We conduct extensive empirical studies of 12 CV and ASR tasks with state-of-the-art CNN and LSTM models . Our experimental results demonstrate that DPSGD consistently outperforms SSGD , across application domains and Neural Network ( NN ) architectures in the large batch setting , without any hyper-parameter tuning . To the best of our knowledge , we are unaware of any generic algorithm that can improve SSGD large batch training on this many models/tasks . The remainder of this paper is organized as follows . Section 2 details the problem formulation and learning dynamics analysis of SSGD , SSGD+Gaussian , and DPSGD ; Section 3 and Section 4 detail the empirical results ; and Section 5 concludes the paper . 2 ANALYSIS OF STOCHASTIC LEARNING DYNAMICS AND EFFECTS OF LANDSCAPE-DEPENDENT NOISE . We first formulate the dynamics of an SGD based learning algorithm with multiple ( n > 1 ) learners indexed by j = 1 , 2 , 3 , ... n following the same theoretical framework established for a single learner ( Chaudhari & Soatto , 2018 ) . At each given time ( iteration ) t , each learner has its own weight vector ~wj ( t ) , and the average weight vector ~wa ( t ) is defined as : ~wa ( t ) ⌘ n 1 Pn j=1 ~wj ( t ) . 1The detailed DPSGD algorithm and its learning dynamics are described in Section 2 . 2We use the terms ” SSGD with Gaussian noise ” and `` SSGD⇤ '' interchangeably in this paper . Each learner j updates its weight vector according to the cross-entropy loss function Lµj ( t ) ( ~w ) for minibatch µj ( t ) that is assigned to it at time t. The size of the local minibatch is B , and the overall batch size for all learners is nB . Two multi-learner algorithms are described below . ( 1 ) Synchronous Stochastic Gradient Descent ( SSGD ) : In the synchronous algorithm , the learner j 2 [ 1 , n ] starts from the average weight vector ~wa and moves along the gradient of its local loss function Lµj ( t ) evaluated at the average weight ~wa : ~wj ( t+ 1 ) = ~wa ( t ) ↵rLµj ( t ) ( ~wa ( t ) ) , ( 1 ) where ↵ is the learning rate . ( 2 ) Decentralized Parallel SGD ( DPSGD ) : In the DPSGD algorithm ( Lian et al. , 2017a ) , learner j computes the gradient at its own local weight ~wj ( t ) . The learning dynamics follows : ~wj ( t+ 1 ) = ~ws , j ( t ) ↵rLµj ( t ) ( ~wj ( t ) ) . ( 2 ) where ~ws , j ( t ) is the starting weight set to be the average weight of a subset of “ neighboring '' learners of learner-j , which corresponds to the non-zero entries in the mixing matrix defined in ( Lian et al. , 2017a ) ( note that ~ws , j = ~wa if all learners are included as neighbors ) . By averaging over all learners , the learning dynamics for the average weight ~wa for both SSGD and DPSGD can be written formally the same way as : ~wa ( t + 1 ) = ~wa ( t ) ↵~ga , where ~ga = n 1 Pn j=1 ~gj is the average gradient and ~gj is the gradient from learner-j . The difference between SSGD and DPSGD is the weight at which ~gj is computed : ~gj ⌘ rLµj ( t ) ( ~wa ( t ) ) is computed at ~wa for SSGD ; ~gj ⌘ rLµj ( t ) ( ~wj ( t ) ) is computed at ~wj for DPSGD . By projecting the weight displacement vector ~wa ⌘ ↵~ga onto the direction of the gradient ~g ⌘ rL ( ~wa ) of the overall loss function L at ~wa , we can write the learning dynamics as : ~wa ( t+ 1 ) = ~wa ( t ) ↵e~g + ~⌘ , ( 3 ) where ↵e ⌘ ↵~ga · ~g/||~g||2 is an effective learning rate and ~⌘ = ↵~ga + ↵e~g is the noise term that describes the stochastic weight dynamics in directions orthogonal to ~g . The noise term has zero mean h~⌘iµ = 0 and its strength is characterized by its variance ( t ) ⌘ ||~⌘||2 . and ↵e are related by the equality : ↵2e||~g||2 + = ↵2||~ga||2 , which indicates that a higher noise strength leads to a lower effective learning rate ↵e . The noise strength ( and hence ↵e ) is different in SSGD and DPSGD . The DPSGD noise DP is larger than the SSGD noise S by an additional noise ( 2 ) ( > 0 ) that originates from the difference of local weights ( ~wj ) from their mean ( ~wa ) : DP = S + ( 2 ) , see Appendix B for details . By expanding ( 2 ) w.r.t . ~wj ⌘ ~wj ~wa , we obtain the average ( 2 ) over minibatch ensemble { µ } : h ( 2 ) iµ ⌘ ↵2h||n 1 nX j=1 [ rLµj ( ~wj ) rLµj ( ~wa ) ] ||2iµ ⇡ ↵2 X k , l , l0 HklHkl0Cll0 , ( 4 ) where Hkl = r2klL is the Hessian matrix of the loss function and Cll0 = n 2 Pn j=1 wj , l wj , l0 is the weight covariance matrix . It is clear that ( 2 ) depends on the loss landscape – it is larger in rough landscapes and smaller in flat landscapes . It is important to stress that the noise ~⌘ in Eq.3 is not an artificially added noise . It is intrinsic to the use of minibatches ( random subsampling ) in SGD-based algorithms and is enhanced by the difference among different learners in DPSGD . The noise strength varies in weight space via its dependence on the loss landscape , as explicitly shown in Eq.4 . However , besides its landscape dependence , SGD noise also depends inversely on the minibatch size B ( Chaudhari & Soatto , 2018 ) . With n synchronized learners , the noise in SSGD scales as 1/ ( nB ) , which is too small to be effective for a large batch size nB . A main finding of our paper is that the additional landscape-dependent noise ( 2 ) in DPSGD can make up for the small SSGD noise when nB is large and help enhance convergence and generalization in the large batch setting . In the following , we investigate the effects of this landscape-dependent noise for SSGD and DPSGD using the MNIST dataset where each learner is a fully connected network with two hidden layers ( 50 units per layer ) . We focus on the large batch setting using nB = 2000 in the experiments .

Summary: This paper claims that decentralized parallel SGD (DPSGD) performs better than synchronous SGD (SSGD) and noisy version of synchronous SGD (SSGD*) in large batch setting. Theoretically, it shows that the noise in DPSGD is landscape-dependent, which may help generalization. Experimental results on CV and ASR tasks show that DPSGD can outperform baselines when batch size is very large. Meanwhile, DPSGD is observed to adaptively adjust the effective learning rate and converge to flatter minima.