Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication

1 INTRODUCTION . Distributed Stochastic Gradient Descent ( DSGD ) is a training setting , in which a number of clients jointly trains a deep learning model using stochastic gradient descent ( Dean et al. , 2012 ; Recht et al. , 2011 ; Moritz et al. , 2015 ) . Every client holds an individual subset of the training data , used to improve the current master model . The improvement is obtained by investing computational resources to perform iterations of stochastic gradient descent ( SGD ) . This local training produces a weight update ∆W in every participating client , which in regular or irregular intervals ( `` communication rounds '' ) is exchanged to produce a new master model . This exchange of weight updates can be performed indirectly via a centralized server or directly in an all-reduce operation . In both cases , all clients share the same master model after every communication round ( see figure 1 ) . In vanilla DSGD the clients have to communicate a full gradient update during every iteration . Every such update is of the same size as the full model , which can be in the range of gigabytes for modern architectures with millions of parameters ( He et al. , 2016 ; Huang et al. , 2017 ) . Over the course of multiple hundred thousands of training iterations on big datasets the total communication for every client can easily grow to more than a petabyte . Consequently , if communication bandwidth is limited , or communication is costly , distributed deep learning can become unproductive or even unfeasible . DSGD is a very popular training setting with many applications . On one end of the spectrum , DSGD can be used to greatly reduce the training time of large-scale deep learning models by introducing device-level data parallelism ( Chilimbi et al. , 2014 ; Zinkevich et al. , 2010 ; Xing et al. , 2015 ; Li et al. , 2014 ) , making use of the fact that the computation of a mini-batch gradient is perfectly parallelizable . In this setting , the clients are usually embodied by hardwired high-performance computation units ( i.e . GPUs in a cluster ) and every client performs one iteration of SGD per communication round . Since communication is high-frequent in this setting , bandwidth can be a significant bottleneck . On the other end of the spectrum DSGD can also be used to enable privacy-preserving deep learning ( Shokri & Shmatikov , 2015 ; McMahan et al. , 2016 ) . Since the clients only ever share weight updates , DSGD makes it possible to train a model from the combined data of all clients without any individual client having to reveal their local training data to a centralized server . In this setting the clients typically are embedded or mobile devices with low network bandwidth , intermittent network connections , and an expensive mobile data plan . In both scenarios , the communication cost between the individual training nodes is a limiting factor for the performance of the whole learning system . For the synchronous distributed training scheme described above , the total amount of bits communicated by every client during training is given by btotal ∈ O ( Niter × f︸ ︷︷ ︸ # communication rounds × |∆W6=0| × ( b̄pos + b̄val ) ︸ ︷︷ ︸ # bits per communication × K︸︷︷︸ # receiving nodes ) ( 1 ) whereNiter is the total number of training iterations ( forward-backward passes ) every client performs , f is the communication frequency , |W6=0| is the sparsity of the weight update , b̄pos , b̄val are the average number of bits required to communicate the position and the value of the non-zero elements respectively and K is the number of receiving nodes ( ifW is dense , the positions of all weights are predetermined and no position bits are required ) . Substantial research has gone into the effort of reducing the amount of communication necessary between the clients via lossy compression schemes . Using the systematic of equation 1 , we can organize prior approaches into three different groups : Sparsification methods restrict weight updates to modifying only a small subset of the parameters , thus reducing |∆W6=0| . Strom ( 2015 ) presents an approach ( later modified by Tsuzuku et al . ( 2018 ) ) in which only gradients with a magnitude greater than a certain predefined threshold are sent to the server . All other gradients are aggregated into a residual . This method achieves compression rates of up to 3 orders of magnitude on an acoustic modeling task . In practice however , it is hard to choose appropriate values for the threshold , as it may vary a lot for different architectures and even different layers . Instead of using a fixed threshold to decide what gradient entries to send , Aji & Heafield ( 2017 ) use a fixed sparsity rate . They only communicate the fraction p entries of the gradient with the biggest magnitude , while also collecting all other gradients in a residual . At a sparsity rate of p = 0.001 their method slightly degrades the convergence speed and final accuracy of the trained model . Lin et al . ( 2017 ) present modifications to the work of Aji et al . which close this performance gap . These modifications include using a curriculum to slowly increase the amount of sparsity in the first couple communication rounds and applying momentum factor masking to overcome the problem of gradient staleness . Their method achieves compression rates ranging from ×270 to ×600 on different architectures , without slowdown in convergence speed . Communication delay methods try to reduce the communication frequency f . McMahan et al . ( 2016 ) propose Federated Averaging to reduce the cumulative communication . In Federated Averaging , instead of communicating after every iteration , every client performs multiple iterations of SGD to compute a weight update . The authors observe that this delay of communication does not significantly harm the convergence speed in terms of local iterations and report a reduction in the number of necessary communication rounds by a factor of ×10 - ×100 on different convolutional and recurrent neural network architectures . In a follow-up work Konečnỳ et al . ( 2016 ) combine this communication delay with random sparsification and probabilistic quantization . They restrict the clients to learn random sparse weight updates or force random sparsity on them afterwards ( `` structured '' vs `` sketched '' updates ) and combine this sparsification with probabilistic quantization . While their method also combines communication delay with ( random ) sparsification and quantization , and achieves good compression gains for one particular CNN and LSTM model , it also causes a major drop in convergence speed and final accuracy . Dense quantization methods try to reduce the amount of value bits b̄val . Different quantization methods have been proposed that reduce the bit-width of the gradients to ternary ( Wen et al. , 2017 ) , binary ( Seide et al. , 2014 ; Bernstein et al. , 2018 ) or arbitrary ( Alistarh et al. , 2017 ) bitwidths . While these are theoretically well-founded and come with strong convergence guarantees , they are also limited to a maximum compression rate of ×32 , compared to the regular 32-bit encoding . 2 ON THE ACCUMULATION OF GRADIENT INFORMATION . Communication delay and sparsification methods as described above already achieve impressive compression rates , however the phenomenon underlying their successes is still only poorly understood . We present a new information-theoretic perspective that is based on the observation that both of these approaches achieve compression by accumulating gradient information locally before sending it to the server . In the case of communication delay all gradients are accumulated uniformly for a fixed amount of iterations , while in the case of sparsification methods they are accumulated non-uniformly until they exceed some fixed or adaptive threshold . In both cases the rate of compression is proportional to the number of steps that the updates are being delayed on average . Consider now the optimization path ∆W1 , .. , ∆WT taken by SGD on the loss-surface between some initialization pointW0 and the modelWT =W0 + ∑T t=1 ∆Wt trained for T iterations . Following this path , we can model the changes occurring to any individual weight in the network w as a noisy stochastic process via ∆wt = st + nt , t = 1 , .. , T ( 2 ) where st denotes the deterministic signal ( i.e . the true direction of the minimum ) , while nt denotes the noise , induced by mini-batch sampling in SGD ( `` batch noise '' ) and the stochasticity of the learning process itself ( `` optimization noise '' , see figure 2 for an illustration ) . For the sake of simplicity , and motivated by the central limit theorem we can assume ( a ) that this noise nt is normally distributed at every time-step nt ∼ N ( 0 , σ2 ) with the variance being constant in time V ( nt ) = σ2 for all t = 1 , .. , T . Since the optimization process has the tendency to damp noise as investigated for instance in LeCun et al . ( 2012 ) it is also reasonable to assume ( b ) that the noise is ( negatively ) self-correlated . The noise process is then given by n1 = N1 , nt = αnt−1 +N t , with N t normally distributed and all N t uncorrelated , α ∈ ( −1 , 0 ) . Given these assumptions we can bound the variance of the accumulated parameter updates . Theorem 2.1 . Under assumptions ( a ) and ( b ) , the variance of the accumulated noise can be bounded by V ( T∑ t=1 nt ) ≤ σ2 ( T ( 1 + α ) + 1 ) . ( 3 ) The proof can be found in the supplement . Theorem 2.1 directly leads us to a lower bound on the signal-to-noise ratio of the accumulated weight-updates : Corollary 2.1.1 . Under assumptions ( a ) and ( b ) , accumulation increases the signal-to-noise ratio from s̄/σ to SNR ( T∑ t=1 ∆wt ) = E [ ∑T t=1 s t + nt ] √ V [ ∑T t=1 s t + nt ] ≥ ∑T t=1 s t√ σ2 ( T ( 1 + α ) + 1 ) ≈ √ T√ 1 + α s̄ σ ( 4 ) with s̄ = 1T ∑T t=1 s t being the signal-average over time . This means that a weight-update will be more informative the longer the accumulation period and the stronger the noise correlates temporally . Convergence speed will not be compromised for as long as the information content of the accumulated update is equal to the cumulative information content of the individual updates ( c.f . fig . 2 ( c ) ) . This line of reasoning helps to shed light on both the successes of communication delay and gradient sparsification . In fact , it implies that both of these approaches are actually very similar in the way they affect the information flow from client to server on the individual weight level . We find that this intuition is also verified empirically . Figure 3 shows validation errors for ResNet32 model trained on CIFAR for 60000 iterations at different levels of communication delay and gradient sparsity . We observe multiple things : 1 . ) The validation error remains more or less constant along the off-diagonals of the matrix where the total sparsity ( i.e . the product of communication delay and gradient sparsity ) is constant . 2 . ) The existing methods of Federated Averaging ( McMahan et al. , 2016 ) ( purple ) and Gradient Dropping/ DGC ( Aji & Heafield , 2017 ; Lin et al. , 2017 ) ( yellow ) are just lines in the two-dimensional space of possible compression methods . 3 . ) There exists a roughly triangular area of approximately constant error , optimal compression methods lie along the hypotenuse of this triangle . We find this behavior consistently across different model architectures , more examples can be found in the supplement . These results indicate , that communication delay and sparsification affect the convergence in a roughly multiplicative way and that there seems to exist a fixed information budged in DSGD , necessary to maintain unhindered convergence . In the following we present a framework that allows us to smoothly trade of these two types of gradient accumulation against one another . By doing so our proposed framework can adapt to the requirements of the distributed learning environment and achieve state-of-the-art compression results by reaping the benefits from both approaches .

Summary: Lowering costs for communicating weights between workers is an important intermediate goal for distributed optimization, since presumably it can limit the parallelization achievable once available bandwidth is saturated.  This work reports reasonable approaches to try to overcome this through a mix of techniques, though none in particular seem especially novel or surprising.  For example, the abstract claims a novel binarization method, but what is described in the paper does not seem especially novel (e.g. zero the negative weights and replace positives with their mean, if negative's mean < positive's mean, else vice versa); but more importantly, the experiments don't explore/support why this approach is any better (or when worse) than other schemes.