signSGD with Majority Vote is Communication Efficient and Fault Tolerant

1 INTRODUCTION . The most powerful supercomputer in the world is currently a cluster of over 27,000 GPUs at Oak Ridge National Labs ( TOP500 , 2018 ) . Distributed algorithms designed for such large-scale systems typically involve both computation and communication : worker nodes compute intermediate results locally , before sharing them with their peers . When devising new machine learning algorithms for distribution over networks of thousands of workers , we posit the following desiderata : D1 fast algorithmic convergence ; D2 good generalisation performance ; D3 communication efficiency ; D4 robustness to network faults . When seeking an algorithm that satisfies all four desiderata D1–4 , inevitably some tradeoff must be made . Stochastic gradient descent ( SGD ) naturally satisfies D1–2 , and this has buoyed recent advances in deep learning . Yet when it comes to large neural network models with hundreds of millions of parameters , distributed SGD can suffer large communication overheads . To make matters worse , any faulty SGD worker can corrupt the entire model at any time by sending an infinite gradient , meaning that SGD without modification is not robust . A simple algorithm with aspirations towards all desiderata D1–4 is as follows : workers send the sign of their gradient up to the parameter server , which aggregates the signs and sends back only the majority decision . We refer to this algorithm as SIGNSGD with majority vote . All communication ∗JB was primary contributor for theory . JZ was primary contributor for large-scale experiments . to and from the parameter server is compressed to one bit , so the algorithm certainly gives us D3 . What ’ s more , in deep learning folklore sign based methods are known to perform well , indeed inspiring the popular RMSPROP and ADAM optimisers ( Balles & Hennig , 2018 ) , giving hope for D1 . As far as robustness goes , aggregating gradients by a majority vote denies any individual worker too much power , suggesting it may be a natural way to achieve D4 . In this work , we make the above aspirations rigorous . Whilst D3 is immmediate , we provide the first convergence guarantees for SIGNSGD in the mini-batch setting , providing theoretical grounds for D1 . We show how theoretically the behaviour of SIGNSGD changes as gradients move from high to low signal-to-noise ratio . We also extend the theory of majority vote to show that it achieves a notion of Byzantine fault tolerance . A distributed algorithm is Byzantine fault tolerant ( Blanchard et al. , 2017 ) if its convergence is robust when up to 50 % of workers behave adversarially . The class of adversaries we consider contains interesting special cases , such as robustness to a corrupted worker sending random bits , or a worker that inverts their gradient estimate . Though our adversarial model is not the most general , it is interesting as a model of network faults , and so gives us D4 . Next , we embark on a large-scale empirical validation of our theory . We implement majority vote in the Pytorch deep learning framework , using CUDA kernels to bit pack sign tensors down to one bit . Our results provide experimental evidence for D1–D4 . Comparing our framework to NCCL ( the state of the art communications library ) , we were able to speed up Imagenet training by 25 % when distributing over 7 to 15 AWS p3.2xlarge machines , albeit at a slight loss in generalisation . Finally , in an interesting twist , the theoretical tools we develop may be brought to bear on a seemingly unrelated problem in the machine learning literature . Reddi et al . ( 2018 ) proved that the extremely popular ADAM optimiser in general does not converge in the mini-batch setting . This result belies the success of the algorithm in a wide variety of practical applications . SIGNSGD is equivalent to a special case of ADAM , and we establish the convergence rate of mini-batch SIGNSGD for a large class of practically realistic objectives . Therefore , we expect that these tools should carry over to help understand the success modes of ADAM . Our insight is that gradient noise distributions in practical problems are often unimodal and symmetric because of the Central Limit Theorem , yet Reddi et al . ( 2018 ) ’ s construction relies on bimodal noise distributions . 2 RELATED WORK . For decades , neural network researchers have adapted biologically inspired algorithms for efficient hardware implementation . Hopfield ( 1982 ) , for example , considered taking the sign of the synaptic weights of his memory network for readier adaptation into integrated circuits . This past decade , neural network research has focused on training feedforward networks by gradient descent ( LeCun et al. , 2015 ) . It is natural to ask what practical efficiency may accompany simply taking the sign of the backpropagated gradient . In this section , we explore related work pertaining to this question . Deep learning : whilst stochastic gradient descent ( SGD ) is the workhorse of machine learning ( Robbins & Monro , 1951 ) , algorithms like RMSPROP ( Tieleman & Hinton , 2012 ) and ADAM ( Kingma & Ba , 2015 ) are also extremely popular neural net optimisers . These algorithms have their Algorithm 1 SIGNUM with majority vote , the proposed algorithm for distributed optimisation . Good default settings for the tested machine learning problems are η = 0.0001 and β = 0.9 , though tuning is recommended . All operations on vectors are element-wise . Setting β = 0 yields SIGNSGD . Require : learning rate η > 0 , momentum constant β ∈ [ 0 , 1 ) , weight decay λ ≥ 0 , mini-batch size n , initial point x held by each of M workers , initial momentum vm ← 0 on mth worker repeat on mth worker g̃m ← 1n ∑n i=1 stochasticGradient ( x ) . mini-batch gradient vm ← ( 1− β ) g̃m + βvm . update momentum push sign ( vm ) to server . send sign momentum on server V ← ∑M m=1 sign ( vm ) . aggregate sign momenta push sign ( V ) to each worker . broadcast majority vote on every worker x← x− η ( sign ( V ) + λx ) . update parameters until convergence roots in the RPROP optimiser ( Riedmiller & Braun , 1993 ) , which is a sign-based method similar to SIGNSGD except for a component-wise adaptive learning rate . Non-convex optimisation : parallel to ( and oftentimes in isolation from ) advances in deep learning practice , a sophisticated optimisation literature has developed . Nesterov & Polyak ( 2006 ) proposed cubic regularisation as an algorithm that can escape saddle points and provide guaranteed convergence to local minima of non-convex functions . This has been followed up by more recent works such as NATASHA ( Allen-Zhu , 2017 ) that use other theoretical tricks to escape saddle points . It is still unclear how relevant these works are to deep learning , since it is not clear to what extent saddle points are an obstacle in practical problems . We avoid this issue altogether and satisfy ourselves with establishing convergence to critical points . Gradient compression : prior work on gradient compression generally falls into two camps . In the first camp , algorithms like QSGD ( Alistarh et al. , 2017 ) , TERNGRAD ( Wen et al. , 2017 ) and ATOMO ( Wang et al. , 2018 ) use stochastic quantisation schemes to ensure that the compressed stochastic gradient remains an unbiased approximation to the true gradient . These works are therefore able to bootstrap existing SGD convergence theory . In the second camp , more heuristic algorithms like 1BITSGD ( Seide et al. , 2014 ) and deep gradient compression ( Lin et al. , 2018 ) pay less attention to theoretical guarantees and focus more on practical performance . These algorithms track quantisation errors and feed them back into subsequent updates . The commonality between the two camps is an effort to , one way or another , correct for bias in the compression . SIGNSGD with majority vote takes a different approach to these two existing camps . In directly employing the sign of the stochastic gradient , the algorithm unabashedly uses a biased approximation of the stochastic gradient . Carlson et al . ( 2016 ) and Bernstein et al . ( 2018 ) provide theoretical and empirical evidence that signed gradient schemes can converge well in spite of their biased nature . Their theory only applies in the large batch setting , meaning the theoretical results are less relevant to deep learning practice . Still Bernstein et al . ( 2018 ) showed promising experimental results in the mini-batch setting . An appealing feature of majority vote is that it naturally leads to compression in both directions of communication between workers and parameter server . As far as we are aware , all existing gradient compression schemes lose compression before scattering results back to workers . Byzantine fault tolerant optimisation : the problem of modifying SGD to make it Byzantine fault tolerant has recently attracted interest in the literature ( Yin et al. , 2018 ) . For example , Blanchard et al . ( 2017 ) proposed KRUM , which operates by detecting and excluding outliers in the gradient aggregation . Alistarh et al . ( 2018 ) propose BYZANTINESGD which instead focuses on detecting and eliminating adversaries . Clearly both these strategies incur overheads , and eliminating adversaries precludes the possibility that they might reform . El Mhamdi et al . ( 2018 ) point out that powerful adversaries may steer convergence to bad local minimisers . We see majority vote as a natural way to protect against less malign faults such as network errors , and thus satisfy ourselves with convergence guarantees to critical points without placing guarantees on their quality . 3 THEORY . 3.1 ASSUMPTIONS . We aim to develop an optimisation theory that is relevant for real problems in deep learning . For this reason , we are careful about the assumptions we make . For example , we do not assume convexity because neural network loss functions are typically not convex . Though we allow our objective function to be non-convex , we insist on a lower bound to enable meaningful convergence results . Assumption 1 ( Lower bound ) . For all x and some constant f∗ , we have objective value f ( x ) ≥ f∗ . Our next two assumptions of Lipschitz smoothness and bounded variance are standard in the stochastic optimisation literature ( Allen-Zhu , 2017 ) . That said , we give them in a component-wise form . This allows our convergence results to encode information not just about the total noise level and overall smoothness , but also about how these quantities are distributed across dimension . Assumption 2 ( Smooth ) . Let g ( x ) denote the gradient of the objective f ( . ) evaluated at point x . Then ∀x , y we require that for some non-negative constant ~L : = [ L1 , ... , Ld ] ∣∣∣f ( y ) − [ f ( x ) + g ( x ) T ( y − x ) ] ∣∣∣ ≤ 1 2 ∑ i Li ( yi − xi ) 2 . Assumption 3 ( Variance bound ) . Upon receiving query x ∈ Rd , the stochastic gradient oracle gives us an independent , unbiased estimate g̃ that has coordinate bounded variance : E [ g̃ ( x ) ] = g ( x ) , E [ ( g̃ ( x ) i − g ( x ) i ) 2 ] ≤ σ2i for a vector of non-negative constants ~σ : = [ σ1 , .. , σd ] . Our final assumption is non-standard . We assume that the gradient noise is unimodal and symmetric . Clearly , Gaussian noise is a special case . Note that even for a moderate mini-batch size , we expect the central limit theorem to kick in rendering typical gradient noise distributions close to Gaussian . See Figure 2 for noise distributions measured whilst training resnet18 on Cifar-10 . Assumption 4 ( Unimodal , symmetric gradient noise ) . At any given point x , each component of the stochastic gradient vector g̃ ( x ) has a unimodal distribution that is also symmetric about the mean . Showing how to work with this assumption is a key theoretical contribution of this work . Combining Assumption 4 with an old tail bound of Gauss ( 1823 ) yields Lemma 1 , which will be crucial for guaranteeing mini-batch convergence of SIGNSGD . As will be explained in Section 3.3 , this result also constitutes a convergence proof for a parameter regime of ADAM . This suggests that Assumption 4 may more generally be a theoretical fix for Reddi et al . ( 2018 ) ’ s non-convergence proof of mini-batch ADAM , a fix which does not involve modifying the ADAM algorithm itself .

Summary: The authors present a distributed implementation of signSGD with majority vote as aggregation. The result is a  communication efficient and byzantine robust distributed training method. This is an interesting and relevant problem. There are two parts in this paper: first the authors prove a convergence guarantee for signSGD, and then they prove that under a weak adversary attack signSGD will be robust to a constant fraction of adversarial nodes. The authors conclude with some limited experiments.