RedSync : Reducing Synchronization Traffic for Distributed Deep Learning

1 INTRODUCTION . For training large-scale deep neural networks ( DNNs ) on multiple computing nodes , data parallelism has emerged as the most popular choice due to its simplicity and effectiveness ( Dean et al . ( 2012 ) ; Recht et al . ( 2011 ) ) . However , the communication bandwidth of network fabric has become the bottleneck limiting data parallel performance . On one hand , models of DNNs , which already contain tens to hundreds of layers and totaling 10-20 million parameters today , continue to grow bigger . Therefore , the requirement of communicating model parameter updates among all computing nodes poses a higher challenge to network bandwidth . On the other hand , the development of DNN training accelerators has shifted the bottleneck of training towards communication across models . As the evolution of the inter-connected network bandwidth is not as fast as computing hardware , synchronization overhead has become the bottleneck of data parallelism on distributed systems using new computing hardware . Many recent studies focused on reducing the communication cost between nodes by reducing the size of the gradients to be transmitted . One line of work ( Seide et al . ( 2014 ) ; Alistarh et al . ( 2017 ) ; Wen et al . ( 2017 ) ) propose to quantize the gradients to low-precision values . Considering compression ratio ( ratio of compressed gradients size to their original size ) achieved by quantization is limited , another line of research orthogonal to quantization is to sparsify communication gradients and restrict weight-updates to a small subset of parameters . Residual Gradient Compression ( RGC ) method ( Strom ( 2015 ) ; Aji & Heafield ( 2017 ) ; Chen et al . ( 2017 ) ; Lin et al . ( 2017 ) ; Sattler et al . ( 2018 ) ) is currently the most promising pruning method to achieve good compression ratio while ensuring no loss of training accuracy . It transmits only a small subset of gradients and maintains the remaining gradients locally as residuals to be added to gradients of the next iteration . The first RGC implementation is proposed by Strom ( 2015 ) and uses a threshold-based method to only send gradients larger than a predefined constant threshold for fully-connected layers . Considering a predefined threshold is hard to be chosen appropriately , Aji & Heafield ( 2017 ) improve the robustness of RGC by selecting top 1 % gradients to communicate according to their magnitude . Because these two implementations are tuned for some specific network structures , applying them to other DNNs will lead to accuracy loss as indicated in Chen et al . ( 2017 ) . Based on their work , the latest RGC variants , such as ( Sattler et al . ( 2018 ) ; Chen et al . ( 2017 ) ; Lin et al . ( 2017 ) ) , are able to achieve a 0.1 % compression ratio on local gradients while ensuring almost no loss of model accuracy on a variety of DNN structures after introducing some key modifications . Despite of good model accuracy achieved with simulation experiments , no recent studies have discussed the potential performance gain after integrating the latest RCG methods to real distributed training system , especially to the multi-GPU systems equipped with high-quality network infrastructures . The challenges of applying RGC to distributed GPU systems come from two aspects . First , there is no efficient compression algorithm proposed for RGC method . According to our experimental results , selecting top-0.1 % elements with the state-of-the-art GPU-based top-k algorithm are so expensive that the overhead of compression is much higher than the benefits of network bandwidth reduction . Second , synchronization of sparse data structures is nontrivial to be supported with existing efficient communication libraries , such as Message Passing Interface ( MPI ) , which are designed for dense data structures . Targeting multi-GPU systems , a highly-efficient RGC implementation called RedSync is proposed . Our contributions are listed as follows : • We combined pruning and quantization techniques together to compress transmitting gradients . A set of parallel-friendly top-0.1 % selection methods are designed to support pruning operations inside GPU device memory , which are orders of magnitude faster than the stateof-the-art GPU-based top-k selection method . • Considering the distribution characteristics of communication data , we apply allgather operation using MPI for a sparse synchronization scheme . A cost model is derived to analyze both communication cost and calculation overhead . Based on it , we pointed out potential performance gain and the bottleneck of our implementation . • RedSync is able to ensure almost no accuracy loss to train a set of DNNs after integrating with the latest algorithm improvements . This is the first work , as far as we known , to evaluate the performance of RGC method on the scale of 128 GPUs . RedSync provides significant performance improvements for communication-intensive networks , like VGG , AlexNet and some LSTMs . 2 DESIGN AND IMPLEMENTATION OF REDSYNC . Algorithm 1 Residual Gradient Compression Input : node id k , the number of node N Input : dataset χ Input : mini batch size b per node Input : initial model w = w [ 0 ] , ... , w [ # layer ] Input : compression ratio D V k ← 0 for t = 0 , 1 , ... max iter do sample b elements as χtk Gk ← ∇ f ( χtk ; w ) by forward and backward propagation for j = # layer , # layer − 1 , ... , 0 do V kj + = G k j Masks← select ( V kj , D ) Gkj ← Allreduce ( compress ( V kj ·Masks ) ) V kj ← V kj ( 1 - Masks ) end for w← SGD ( w , decompress ( Gk ) ) end for We first give an overview of a simple RGC workflow used in RedSync ( see more details in Algorithm 1 ) . We denote a DNN model as f ( w ) , where w is the vector of parameters . We assume a system has N workers . Each worker , say the k-th worker , holds a local dataset χtk at iteration t with size b and a local copy of the global weight w. Synchronous SGD method is adopted in RedSync . At each iteration , node k computes the gradient Gk using local data , where Gkj indicates gradients of layer j . Each node also maintains a residual V k , which is initialized as 0 and used to accumulate untransmitted gradient from previous iterations . After added with latest gradient , a subset of residuals is selected as the communication-set , and is compressed into sparse data structures . The select operation in Algorithm 1 chooses more important elements based on magnitude . Those selected elements ( denoted ask Masks ) are synchronized among all the nodes using allreduce operations , which is able to take advantage of the highly-optimized allreduce operation on HPC systems ( Thakur et al . ( 2005 ) ) . Synchronous SGD 10 2 10 1 100 101 102 lo g ( tim e ) ( s ec ) RadixSelct top 0.1 % trim top 0.1 % thd-BS top 0.1 % sampled thd-BS top 0.1 % ( interval=5 ) Comm . ( peak 3.5 GBps ) implemented with allreduce has been widely adopted in state-of-the-art large-scale CNN training tasks ( Goyal et al . ( 2017 ) and You et al . ( 2017 ) ) . Remaining elements outside the communicationset are assigned as new residuals of the next iteration . The workflow of this algorithm is the same as an RGC variant called Deep Gradient Compression Method mentioned Lin et al . ( 2017 ) . In the following , we details our contribution in implementations of select , Allreduce and decompress to make this workflow efficient in practice . 2.1 PARALLEL-FRIENDLY COMPRESSION . The efficiency of communication-set selection method is critical for the RGC system ’ s overall performance . Since a predefined threshold is difficult to determine , recent work ( Lin et al . ( 2017 ) ; Sattler et al . ( 2018 ) ) suggest to select top 0.1 % elements from residuals of each layer as the communication-set . However , the top-0.1 % selection is nontrivial to be implemented on GPU . One of the most efficient top-k selection methods designed for GPU can be implemented based on radixSelect algorithm ( Alabi et al . ( 2012 ) ) , which determines each bit of the k-th largest element by scan and scatter . Serial scan ( Sengupta et al . ( 2007 ) ) and scatter operations are extremely timeconsuming . As shown in Figure 1 , the computation time for top-0.1 % with radixSelect on a Titan X GPU sometimes is even slightly higher than the time for synchronizing these parameters through a 3.5 GBps network . To avoid performing a top-0.1 % operation on a large number of parameters , we propose two communication-set selection algorithms called trimmed top-k selection and threshold binary search selection , which are more efficient on GPUs . Trimmed top-k selection . Observing that the distribution of residuals is usually similar to a normal distribution , we can use statistical features to remove most of the smaller elements and limit radixSelect operation on a relatively small subset . As shown in Algorithm 2 , we first calculate the mean and maximum of residuals ’ absolute values of this layer . A relative large threshold value is chosen according to mean and maximum value , for example , 0.8 × ( max − mean ) + mean . Operation count nonzero gets the number of elements whose absolute values are greater than the threshold . If the number is smaller than k ( the number of top-0.1 % elements ) , we dynamically decrease the threshold until we find the number of parameters whose absolute value above the threshold is larger than k. Then we trim all elements that are less than the threshold and perform a top-k selection operation using radixSelect on the remaining elements . Operation mean , max and count nonzero can all be efficiently implemented with a single reduction operation . nonzero indices is a typical stream compaction problem , which uses just one scan operation as its backbone ( Sengupta et al . ( 2006 ) ) . Threshold binary search selection . For some layers with very large numbers of parameter elements , even conducting radixSelect on a small subset of elements will still be a very time-consuming operation . In order to completely avoid using radixSelect operation on GPU , we propose a method to select approximate top-0.1 % elements as communication-set . Instead of identifying the kth ( top 0.1 % th ) largest element , we search for a threshold to make it between the kth to 2kth largest element , and then select elements larger than the threshold as communication-set . In this case , at least 0.1 % largest elements are included in the communication-set . As shown in Algorithm 3 , we use a binary search algorithm to find such a threshold . To avoid excessive searching , it will always be terminated when the difference of left bound and right bound is less than a small value . Algorithm 2 trimmed top-k Selection Input : tensor to be compressedX Input : number of elements remained k Output : < indice , values > 1 : mean← mean ( abs ( X ) ) 2 : max← max ( abs ( X ) ) 3 : ← 0.2 4 : ratio← ( 1− ) 5 : nnz = count nonzero ( abs ( X ) > threshold ) 6 : while nnz > k do 7 : threshold← mean+ ratio× ( max−mean ) 8 : nnz = count nonzero ( abs ( X ) > threshold ) 9 : ratio = ratio− 10 : end while 11 : indice← nonzero indices ( abs ( X ) > threshold ) ) 12 : values← X [ indice ] Algorithm 3 Top-k selection with threshold binary search selection Input : tensor to be compressedX Input : number of elements remained k Input : Termination condition parameter Output : < indice , values > 1 : mean← mean ( abs ( X ) ) ; max← max ( abs ( X ) ) 2 : l← 0.0 ; r ← 1.0 ; threshold = 0.0 3 : while r − l > do 4 : ratio = l + ( r − l ) /2 5 : threshold← mean+ ratio× ( max−mean ) 6 : nnz = count nonzero ( abs ( X ) > threshold ) 7 : if nnz > k and 2k > nnz then 8 : break 9 : else if nnz < k/2 then 10 : r = threshold 11 : else 12 : l = threshold 13 : end if 14 : end while 15 : indice← nonzero indices ( abs ( X ) > threshold ) ) 16 : values← X [ indice ] For layers with large sizes , such as the first fully-connected layer in VGG16 and softmax layer in LSTM , the time for count nonzero operation is still not negligible . We further improve the efficiency of the selection algorithm by reducing the number of count nonzero operations . We recommend that , after a threshold binary search for this layer , the threshold element can be reused in the next few iterations . The interval of search is empirically set to 5 , and the selection algorithm introduces only one nonzero count overhead on average . In Figure 1 , we compared the time cost of different selection approaches on parameter lists of different sizes . Compared with directly performing radixSelect , both proposed methods significantly reduce the selection time for large sizes . For top-0.1 % selection on 64MB elements , trimmed top-k and sampled threshold binary search selection are 38.13 and 16.17 × faster than radixSelect . In practice , we dynamically choose compression strategies : For smaller parameter sets such as biases and batch norm layers , we do not compress residuals or directly use radixSelect to select top-0.1 % significant elements . Trimmed top-k selection is suitable for parameters of middle size layers , like convolutional layers , because it can ensure the compression ratio to be exactly 0.1 % and introduce no extra communication bandwidth requirements . Threshold binary search based selection is suitable for large size layers , like hidden layers and softmax layers in LSTMs , for which the compression cost is more critical to be optimized than the communication cost .

Summary: Paper focuses on Residual Gradient Compression (RGC) as a promising approach to reducing the synchronization cost of gradients in a distributed settings. Prior approaches focus on the theoretical value of good compression rates without looking into the overall cost of the changes. This paper introduces RedSync that builds on the existing approaches by picking the most appropriate ones that reduce the overall cost for gradient reduction without unduly focusing on the compression rate.