Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging

√ K ) , the same as SGD . The performance evaluation demonstrates it en- ables a linear performance scale-up with the cluster size . 1 INTRODUCTION . Training deep learning models using data parallelism on a large-scale distributed cluster has become an effective method for deep learning model training . The enormous training data set allows a huge batch of training tasks on different data samples running in parallel . The pinnacle of this method reduces the training time of the benchmark ResNet-50 from days to a couple of minutes ( Goyal et al. , 2017 ; You et al. , 2017b ; Akiba et al. , 2017 ; You et al. , 2017a ; Ying et al. , 2018 ; Goodfellow et al. , 2016 ) . During Mini-batch stochastic gradient descent ( SGD ) , these workers have to halt , wait for the computed gradients aggregated from all of the workers and receive a weight update before starting the next batch . The wait time tends to worsen when the number of workers increases . As the workloads are spread over a larger cluster , the computation time is greatly shortened and the communication overheads take a larger portion of the overall cost . System designers address this concern by improving inter-chip connects with higher throughput and lower latency and refining network topology ( Li et al. , 2019 ) , such as NVIDIA DGX-1 ( NVI , 2017 ) and NVIDIA DGX-2 ( NVS , 2018 ) . Additional care has been given to reduce the intermediate steps that would increase communication latency . These methods effectively reduce the wait time during Mini-batch SGD on a large-scale distributed system ( Gau , 2019 ) . Communication efficient SGD algorithms ( Lin et al. , 2017 ; Wangni et al. , 2018 ; Alistarh et al. , 2018 ; Dean et al. , 2012 ; Recht et al. , 2011 ; Zhang et al. , 2015 ; Wang & Joshi , 2018 ; Wang & Joshi , 2018 ; Lin et al. , 2018 ) are proposed to reduce the communication requirements . A successfully modified SGD algorithm shows their convergence rates comparable to Mini-batch SGD through both theoretical analysis and experimental results . Another challenge is to demonstrate good performance evaluation results based on the common large distributed training system setups . A modern data center design prefers cost-efficient hardware blocks and a balanced configuration for the typical workloads ( Barroso et al. , 2018 ) . Under these workloads , hardware resources are utilized in a balanced fashion . A distributed training system works in the opposite manner . During forward/back propagations , the computing resources are fully used while the system inter-connects are completely idle . During SGD , the computing resources are mostly idle while the system interconnects are throttled at the peak throughputs . The performance of distributed training systems may be improved in addition to better hardware . That is , training workloads may be re-structured for balanced utilization of the hardware resources . Inspired by the modern system design practices , we propose DaSGD , a new SGD method , enabling SGD running parallelly with forward/back propagation and balanced utilization of the hardware resources . It replaces a Mini-batch SGD with Local SGD iterations . In a Local SGD ( Lin et al. , 2018 ; Wang & Joshi , 2018 ) , each worker evolves a local model by performing sequential SGD updates , before synchronizing by averaging models among the workers . DaSGD uses Local SGD to add weight synchronization points and allows weights to be updated between Local SGD iterations . Based on the network throughput and the model size , it schedules delayed model averaging for a defined number of Local SGD iterations , which allows workers to compute the next batch while the weights are transferred over a large distributed cluster . The theoretical analysis shows its convergence rate is O ( 1/ √ K ) , where K is the iteration step , same as Mini-batch SGD . The main contributions of this paper are the following . • We present DaSGD as an algorithm-system co-design method for a large-scale distributed training system . It enables designing a more balanced and better-utilized system , more than a new variant of the SGD algorithm . The discussions and analyses in this paper are organized around its equivalency to the traditional SGD and its benefit from the system design perspective . • We provide the theoretical analysis of its convergence rate . It shows the convergence rate at O ( 1/ √ K ) , the same as Mini-batch SGD . • Our experiments focus on the training progresses ( in terms of loss and accuracy ) at the epoch level . It shows DaSGD allows the training converges at the same rate of SGD epoch by epoch , which is a good indicator of statistical efficiency and the eventual time-toconvergence . The experiments also explore the proper ranges of these parameters . • A performance evaluation of real-life systems measures many performance issues in a system , such as the reduction algorithm , GPU interconnect topology and throughputs , network throughputs , which are out of the scope of our discussion . Instead , we abstract an analytical model from the key performance parameters of the system configuration and the training setup . The analytical model demonstrates DaSGD introduces a linear performance scale-up with the cluster size . 2 BACKGROUND . 2.1 STOCHASTIC GRADIENT DESCENT . Stochastic Gradient Descent ( SGD ) is the backbone of numerous deep learning algorithms ( Ghadimi & Lan , 2013 ) . Supervised deep learning demands massive training datasets . Training a deep learning model needs many epochs for training to converge . A variant of classic SGD , synchronous minibatch SGD ( Bottou , 2010 ) , has become the mainstream due to a faster convergence rate. , supported by prevalent machine learning frameworks , such as Tensorflow ( Abadi et al. , 2016 ) , Pytorch ( Paszke et al. , 2019 ) , MxNet ( Chen et al. , 2015 ) . Mini-batch SGD as a weight update process is shown in Eq . 1. xk+1 = xk − η B B∑ j=1 ∇F ( xk , s ( j ) k ) ( 1 ) where x ∈ Rd is the weight of a model , η is the learning rate , B is the batch size , S is the training dataset , s ( j ) k ⊂ S is a random sample , ∇F ( xk , s ( j ) k ) is the stochastic gradient given the sample s ( j ) k . From a system perspective , a distributed training system may compute a batch of gradients on several workers . At the end of a batch , a reduction operation is performed on the gradients on a worker first and a worker sends out only a copy of local averaged gradients . Further reductions are performed on gradients from different workers until a final copy of the averaging gradients is obtained . The above equation may be rewritten as xk+1 = xk − η M B∑ j=1 g ( xk , s ( j ) k ) ( 2 ) where M is the number of workers , g ( xk , s ( j ) k ) is the stochastic gradient that worker j aggregates locally for that batch . g ( xk , s ( j ) k ) = M B B M∑ i=1 ∇ ( xk , s ( i ) k ) ( 3 ) 2.2 COMMUNICATION EFFICIENT SGD ALGORITHMS . 2.2.1 ASYNCHRONOUS SGD . There are a few asynchronous training methods , such as Downpour SGD ( Dean et al. , 2012 ) , Hogwild ( Recht et al. , 2011 ) , Elastic Averaging SGD ( Zhang et al. , 2015 ) . In these models , every worker has its own copy of weights . A worker performs forward propagation and back propagation on its partition of samples , and then sends the calculated gradients asynchronously to a pool of parameter servers that manage a central copy of weights . The parameter servers update the central copy and then send the new weights asynchronously to each worker . While each worker communicates gradients at a different time and avoids congestions at worker inter-connects , the parameter servers might be a performance bottleneck . For non-convex problems , ASGD requires that the staleness of gradients is bounded ( Lian et al. , 2015 ) to match the convergence rate O ( 1/ √ K ) of synchronous SGD , where K denotes the total Iteration steps . 2.2.2 LOCAL SGD . Another set of methods targets at reducing the frequency of inter-worker communication and is called periodic averaging or Local SGD ( Wang & Joshi , 2018 ; Wang & Joshi , 2018 ; Lin et al. , 2018 ) . A worker performs SGD on its local copy of weights for τ times , where τ denotes the local iteration steps . After τ local updates , local copies are synchronously averaged across all workers globally . Several works suggested that Local SGD incurs the same convergence rate O ( 1/ √ K ) as SGD . The total number of steps to train a model remains similar but the total amount of inter-worker communication is reduced by τ times . This has a similar effect as training with a large batch size , where the number of synchronizations decreases with an increase of batch size . With Local SGD , SGD and forward/back propagations are still blocking while system resources are unbalanced . 3 DASGD . In this paper , we propose a new algorithm , called Local SGD with Delayed Averaging , DaSGD for short . It aggregates gradients and updates weights in a relaxed manner , which helps parallelize the computation of forward/backward propagation with two other execution components : the execution of global weight averaging and inter-worker data communication . Our algorithm was initially inspired by the Local SGD algorithm ( discussed in Section 2 ) . Although Local SGD was designed to reduce communication and synchronization overhead , it still involves a significant amount of communication overhead . To further decrease communication overhead , even to zero , the proposed algorithm exploits a delayed averaging approach that makes two novel improvements based on Local SGD . First , in order to merge remote weights by other workers with local in a deterministic way , DaSGD serializes forward propagations and back propagations for different samples . Second , workers start with local computations for the next local batch while waiting for the aggregation and synchronization of global weights . In this way , the global communication and synchronization overhead are hidden or overlapped by local computations at the cost of a delayed update of local weights . However , theoretically we will prove that the convergence rate is the same as Mini-batch SGD . Furthermore , DaSGD parameterizes the overlapping degree so that when a large training cluster requires a longer time to synchronize , a worker may perform more iterations of local computations . Fig . 1 illustrates the proposed algorithm by showing a wall-clock time diagram of 6 training iterations . There are 2 workers , dividing a batch into 2 local batches . Each worker computes 6 local batches . Each local batch contains Bl samples . Each worker maintains a local copy of model . According to Local SGD , for a local batch , each worker operates Bl forward/backward propagations and then updates the weights of its local model . After 3 local updates , a worker synchronizes local weights with the other workers , resulting in an AllReduce operation being generated to average the model weights . In Fig . 1 , all workers wait for , at local iteration 3 , the global synchronization to be finished and then start to operate on the next local batch each , in the scenario of Local SGD . DaSGD implements a key feature by imposing delay update on Local SGD . As shown in Fig . 1 , a worker , at local step 3 , broadcasts its local weights to the wild and then immediately starts to compute on the next local batch , without waiting for the global synchronization to be finished . Later , at local iteration 4 , the worker receives all the other workers ’ weights and then updates its local weights . This design very efficiently overlaps the communication of weights and forward/backward propagations of the next local batch . In DaSGD , we use τ to denote the number of local batches between two consecutive global synchronizations . Therefore , τ is a controlling parameter that quantifies the number of propagations between weight averaging globally . During the delay update , both local computation and the global communication of weights are executed in parallel . As long as communication time is no more than the computation time of d local iterations , the communication time can be hidden in the overall model training time . Careful tuning of d and τ can realize full parallelism of global averaging and local computations . Unlike Local SGD , τ does not have to be large , as it is not only used to reduce inter-worker communication overhead ( Lin et al. , 2017 ) . In the following part of this section , in order to compare the proposed algorithm and traditional SGDs , we start with the update framework of each algorithm , and then qualitatively analyze execution time . Finally , we discussed the updated rules and the convergence rate in detail .

Summary: This paper proposed DaSGD, an algorithm for large-scale large-batch training of deep neural networks. The algorithm combines Local SGD with delayed averaging steps to hide the communication overhead. However, workers still synchronize their forward/backward passes in each iteration. A convergence rate of O(1/sqrt(K)) to a stationary is provided for smooth non-convex objectives, with respect to the average parameter set across workers. Numerical experiments provided on CIFAR10 and some brief results on ImageNet-1k with a ResNet18 and a batch-size of 256.