Title: SIGNSGD WITH MAJORITY VOTE IS COMMUNICATION EFFICIENT AND FAULT TOLERANT

Abstract: Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines. As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable. The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults. We explore a particularly simple algorithm for robust, communication-efficient learning-SIGNSGD. Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote. This algorithm uses 32× less communication per iteration than full-precision, distributed SGD. Under natural conditions verified by experiment, we prove that SIGNSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of ADAM as a byproduct. Aggregating sign gradients by majority vote means that no individual worker has too much power. We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially. The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate. On the practical side, we built our distributed training system in Pytorch. Benchmarking against the state of the art collective communications library (NCCL), our framework-with the parameter server housed entirely on one machine-led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines. * JB was primary contributor for theory. JZ was primary contributor for large-scale experiments.

1. INTRODUCTION:

The most powerful supercomputer in the world is currently a cluster of over 27,000 GPUs at Oak Ridge National Labs (TOP500, 2018). Distributed algorithms designed for such large-scale systems typically involve both computation and communication: worker nodes compute intermediate results locally, before sharing them with their peers. When devising new machine learning algorithms for distribution over networks of thousands of workers, we posit the following desiderata: D1 fast algorithmic convergence; D2 good generalisation performance;

D3 communication efficiency;:

D4 robustness to network faults.

When seeking an algorithm that satisfies all four desiderata D1-4, inevitably some tradeoff must be made. Stochastic gradient descent (SGD) naturally satisfies D1-2, and this has buoyed recent advances in deep learning. Yet when it comes to large neural network models with hundreds of millions of parameters, distributed SGD can suffer large communication overheads. To make matters worse, any faulty SGD worker can corrupt the entire model at any time by sending an infinite gradient, meaning that SGD without modification is not robust.

A simple algorithm with aspirations towards all desiderata D1-4 is as follows: workers send the sign of their gradient up to the parameter server, which aggregates the signs and sends back only the majority decision. We refer to this algorithm as SIGNSGD with majority vote. All communication Figure 1 : Toy experiments. SIGNSGD with majority vote is run on a 1000-dimensional quadratic with N (0, 1) noise added to each gradient component. Adversarial experiments are run with 27 total workers. These plots may be reproduced in a web browser by running this Jupyter notebook.

to and from the parameter server is compressed to one bit, so the algorithm certainly gives us D3. What's more, in deep learning folklore sign based methods are known to perform well, indeed inspiring the popular RMSPROP and ADAM optimisers (Balles & Hennig, 2018) , giving hope for D1. As far as robustness goes, aggregating gradients by a majority vote denies any individual worker too much power, suggesting it may be a natural way to achieve D4.

In this work, we make the above aspirations rigorous. Whilst D3 is immmediate, we provide the first convergence guarantees for SIGNSGD in the mini-batch setting, providing theoretical grounds for D1. We show how theoretically the behaviour of SIGNSGD changes as gradients move from high to low signal-to-noise ratio. We also extend the theory of majority vote to show that it achieves a notion of Byzantine fault tolerance. A distributed algorithm is Byzantine fault tolerant (Blanchard et al., 2017) if its convergence is robust when up to 50% of workers behave adversarially. The class of adversaries we consider contains interesting special cases, such as robustness to a corrupted worker sending random bits, or a worker that inverts their gradient estimate. Though our adversarial model is not the most general, it is interesting as a model of network faults, and so gives us D4.

Next, we embark on a large-scale empirical validation of our theory. We implement majority vote in the Pytorch deep learning framework, using CUDA kernels to bit pack sign tensors down to one bit. Our results provide experimental evidence for D1-D4. Comparing our framework to NCCL (the state of the art communications library), we were able to speed up Imagenet training by 25% when distributing over 7 to 15 AWS p3.2xlarge machines, albeit at a slight loss in generalisation.

Finally, in an interesting twist, the theoretical tools we develop may be brought to bear on a seemingly unrelated problem in the machine learning literature. Reddi et al. (2018) proved that the extremely popular ADAM optimiser in general does not converge in the mini-batch setting. This result belies the success of the algorithm in a wide variety of practical applications. SIGNSGD is equivalent to a special case of ADAM, and we establish the convergence rate of mini-batch SIGNSGD for a large class of practically realistic objectives. Therefore, we expect that these tools should carry over to help understand the success modes of ADAM. Our insight is that gradient noise distributions in practical problems are often unimodal and symmetric because of the Central Limit Theorem, yet Reddi et al. ( 2018)'s construction relies on bimodal noise distributions.

2. RELATED WORK:

For decades, neural network researchers have adapted biologically inspired algorithms for efficient hardware implementation. Hopfield (1982) , for example, considered taking the sign of the synaptic weights of his memory network for readier adaptation into integrated circuits. This past decade, neural network research has focused on training feedforward networks by gradient descent (LeCun et al., 2015) . It is natural to ask what practical efficiency may accompany simply taking the sign of the backpropagated gradient. In this section, we explore related work pertaining to this question.

Deep learning: whilst stochastic gradient descent (SGD) is the workhorse of machine learning (Robbins & Monro, 1951) , algorithms like RMSPROP (Tieleman & Hinton, 2012) and ADAM (Kingma & Ba, 2015) are also extremely popular neural net optimisers. These algorithms have their Algorithm 1 SIGNUM with majority vote, the proposed algorithm for distributed optimisation. Good default settings for the tested machine learning problems are η = 0.0001 and β = 0.9, though tuning is recommended. All operations on vectors are element-wise. Setting β = 0 yields SIGNSGD. Require: learning rate η > 0, momentum constant β ∈ [0, 1), weight decay λ ≥ 0, mini-batch size n, initial point x held by each of M workers, initial momentum v m ← 0 on m th worker repeat on m th worker gm ← 1

n n i=1 stochasticGradient(x) mini-batch gradient v m ← (1 -β)g m + βv m update momentum push sign(v m ) to server send sign momentum on server V ← M m=1 sign(v m )

aggregate sign momenta push sign(V ) to each worker broadcast majority vote on every worker

x ← x -η(sign(V ) + λx) update parameters until convergence roots in the RPROP optimiser (Riedmiller & Braun, 1993) , which is a sign-based method similar to SIGNSGD except for a component-wise adaptive learning rate.

Non-convex optimisation: parallel to (and oftentimes in isolation from) advances in deep learning practice, a sophisticated optimisation literature has developed. Nesterov & Polyak (2006) proposed cubic regularisation as an algorithm that can escape saddle points and provide guaranteed convergence to local minima of non-convex functions. This has been followed up by more recent works such as NATASHA (Allen-Zhu, 2017) that use other theoretical tricks to escape saddle points. It is still unclear how relevant these works are to deep learning, since it is not clear to what extent saddle points are an obstacle in practical problems. We avoid this issue altogether and satisfy ourselves with establishing convergence to critical points.

Gradient compression: prior work on gradient compression generally falls into two camps. In the first camp, algorithms like QSGD (Alistarh et al., 2017) , TERNGRAD (Wen et al., 2017) and ATOMO (Wang et al., 2018) use stochastic quantisation schemes to ensure that the compressed stochastic gradient remains an unbiased approximation to the true gradient. These works are therefore able to bootstrap existing SGD convergence theory. In the second camp, more heuristic algorithms like 1BITSGD (Seide et al., 2014) and deep gradient compression (Lin et al., 2018) pay less attention to theoretical guarantees and focus more on practical performance. These algorithms track quantisation errors and feed them back into subsequent updates. The commonality between the two camps is an effort to, one way or another, correct for bias in the compression.

SIGNSGD with majority vote takes a different approach to these two existing camps. In directly employing the sign of the stochastic gradient, the algorithm unabashedly uses a biased approximation of the stochastic gradient. Carlson et al. (2016) and Bernstein et al. (2018) provide theoretical and empirical evidence that signed gradient schemes can converge well in spite of their biased nature. Their theory only applies in the large batch setting, meaning the theoretical results are less relevant to deep learning practice. Still Bernstein et al. (2018) showed promising experimental results in the mini-batch setting. An appealing feature of majority vote is that it naturally leads to compression in both directions of communication between workers and parameter server. As far as we are aware, all existing gradient compression schemes lose compression before scattering results back to workers.

Byzantine fault tolerant optimisation: the problem of modifying SGD to make it Byzantine fault tolerant has recently attracted interest in the literature (Yin et al., 2018) . For example, Blanchard et al. (2017) proposed KRUM, which operates by detecting and excluding outliers in the gradient aggregation. Alistarh et al. (2018) propose BYZANTINESGD which instead focuses on detecting and eliminating adversaries. Clearly both these strategies incur overheads, and eliminating adversaries precludes the possibility that they might reform. El Mhamdi et al. (2018) point out that powerful adversaries may steer convergence to bad local minimisers. We see majority vote as a natural way to protect against less malign faults such as network errors, and thus satisfy ourselves with convergence guarantees to critical points without placing guarantees on their quality. Figure 2 : Gradient distributions for resnet18 on Cifar-10 at mini-batch size 128. At the start of epochs 0, 1 and 5, we do a full pass over the data and collect the gradients for three randomly chosen weights (left, middle, right) . In all cases the distribution is close to unimodal and symmetric.

3. THEORY:



3.1. ASSUMPTIONS:

We aim to develop an optimisation theory that is relevant for real problems in deep learning. For this reason, we are careful about the assumptions we make. For example, we do not assume convexity because neural network loss functions are typically not convex. Though we allow our objective function to be non-convex, we insist on a lower bound to enable meaningful convergence results. Assumption 1 (Lower bound). For all x and some constant f * , we have objective value f (x) ≥ f * .

Our next two assumptions of Lipschitz smoothness and bounded variance are standard in the stochastic optimisation literature (Allen-Zhu, 2017). That said, we give them in a component-wise form. This allows our convergence results to encode information not just about the total noise level and overall smoothness, but also about how these quantities are distributed across dimension.

Assumption 2 (Smooth). Let g(x) denote the gradient of the objective f (.) evaluated at point x.

Then ∀x, y we require that for some non-negative constant

L := [L 1 , ..., L d ] f (y) -f (x) + g(x) T (y -x) ≤ 1 2 i L i (y i -x i ) 2 .

Assumption 3 (Variance bound). Upon receiving query x ∈ R d , the stochastic gradient oracle gives us an independent, unbiased estimate g that has coordinate bounded variance:

E[g(x)] = g(x), E (g(x) i -g(x) i ) 2 ≤ σ 2 i for a vector of non-negative constants σ := [σ 1 , .., σ d ].

Our final assumption is non-standard. We assume that the gradient noise is unimodal and symmetric.

Clearly, Gaussian noise is a special case. Note that even for a moderate mini-batch size, we expect the central limit theorem to kick in rendering typical gradient noise distributions close to Gaussian. See Figure 2 for noise distributions measured whilst training resnet18 on Cifar-10. Assumption 4 (Unimodal, symmetric gradient noise). At any given point x, each component of the stochastic gradient vector g(x) has a unimodal distribution that is also symmetric about the mean.

Showing how to work with this assumption is a key theoretical contribution of this work. Combining Assumption 4 with an old tail bound of Gauss (1823) yields Lemma 1, which will be crucial for guaranteeing mini-batch convergence of SIGNSGD. As will be explained in Section 3.3, this result also constitutes a convergence proof for a parameter regime of ADAM. This suggests that Assumption 4 may more generally be a theoretical fix for Reddi et al. ( 2018)'s non-convergence proof of mini-batch ADAM, a fix which does not involve modifying the ADAM algorithm itself. 

3.2. MINI-BATCH CONVERGENCE OF SIGNSGD:

With our assumptions in place, we move on to presenting our theoretical results, which are all proved in Appendix C. Our first result establishes the mini-batch convergence behaviour of SIGNSGD. We will first state the result and make some remarks. We provide intuition for the proof in Section 3.3.

Theorem 1 (Non-convex convergence rate of mini-batch SIGNSGD). Run the following algorithm for K iterations under Assumptions 1 to 4:

x k+1 = x k -η sign(g k ).

Set the learning rate, η, and mini-batch size, n, as

η = f 0 -f * L 1 K , n = 1.

Let H k be the set of gradient components at step k with large signal-to-noise ratio S i :=

|g k,i | σi , i.e. H k := i S i > 2 √ 3 .

We refer to 2 √ 3 as the 'critical SNR'. Then we have

1 K K-1 k=0 E   i∈H k |g k,i | + i ∈H k g 2 k,i σ i   ≤ 3 L 1 (f 0 -f * ) N .

where N = K is the total number of stochastic gradient calls up to step K.

Theorem 1 provides a bound on the average gradient norm. The right hand side of the bound decays like O 1 √ N , establishing convergence to points of the objective where the gradient vanishes.

Remark 1: mini-batch SIGNSGD attains the same O 1 √ N non-convex convergence rate as SGD. Remark 2: the gradient appears as a mixed norm: an 1 norm for high SNR components, and a weighted 2 norm for low SNR compoenents.

Remark 3: we wish to understand the dimension dependence of our bound. We may simplify matters by assuming that, during the entire course of optimisation, every gradient component lies in the low SNR regime. Figure 3 shows that this is almost true when training a resnet18 model. In this limit, the bound becomes:

1 K K-1 k=0 E d i=1 g 2 k,i σ i ≤ 3 L 1 (f 0 -f * ) N .

Further assume that we are in a well-conditioned setting, meaning that the variance is distributed uniformly across dimension (σ 2 i = σ 2 d ), and every weight has the same smoothness constant (L i = L). σ 2 is the total variance bound, and L is the conventional Lipschitz smoothness. These are the Published as a conference paper at ICLR 2019 quantities which appear in the standard analysis of SGD. Then we get

1 K K-1 k=0 E g k 2 2 ≤ 3σ L(f 0 -f * ) N .

The factors of dimension d have conveniently cancelled. This illustrates that there are problem geometries where mini-batch SIGNSGD does not pick up an unfavourable dimension dependence.

3.3. THE SUBTLETIES OF MINI-BATCH CONVERGENCE:

Intuitively, the convergence analysis of SIGNSGD depends on the probability that a given bit of the sign stochastic gradient vector is incorrect, or

P[sign(g i ) = sign(g i )]

. Lemma 1 provides a bound on this quantity under Assumption 4 (unimodal symmetric gradient noise).

Lemma 1 (Bernstein et al. (2018) ). Let gi be an unbiased stochastic approximation to gradient component g i , with variance bounded by σ 2 i . Further assume that the noise distribution is unimodal and symmetric. Define signal-to-noise ratio S i := |gi| σi . Then we have that

P[sign(g i ) = sign(g i )] ≤ 2 9 1 S 2 i if S i > 2 √ 3 , 1 2 -Si 2 √ 3

otherwise which is in all cases less than or equal to 1 2 .

The bound characterises how the failure probability of a sign bit depends on the signal-to-noise ratio (SNR) of that gradient component. Intuitively as the SNR decreases, the quality of the sign estimate should degrade. The bound is important since it tells us that, under conditions of unimodal symmetric gradient noise, even at extremely low SNR we still have that P[sign(g i ) = sign(g i )] ≤ 1 2 . This means that even when the gradient is very small compared to the noise, the sign stochastic gradient still tells us, on average, useful information about the true gradient direction, allowing us to guarantee convergence as in Theorem 1.

Without Assumption 4, the mini-batch algorithm may not converge. This is best appreciated with a simple example. Consider a stochastic gradient component g with bimodal noise: g = 50 with probability 0.1; -1 with probability 0.9.

The true gradient g = E[g] = 4.1 is positive. But the sign gradient sign(g) is negative with probability 0.9. Therefore SIGNSGD will tend to move in the wrong direction for this noise distribution.

Note that SIGNSGD is a special case of the ADAM algorithm (Balles & Hennig, 2018) . To see this, set β 1 = β 2 = = 0 in ADAM, and the ADAM update becomes: We conclude this section by noting that without Assumption 4, SIGNSGD can still be guaranteed to converge. The trick is to use a "large" batch size that grows with the number of iterations. This will ensure that the algorithm stays in the high SNR regime where the failure probability of the sign bit is low. This is the approach taken by both Carlson et al. (2016) and Bernstein et al. (2018) .

- g g2 = - g |g| = -sign(g)

3.4. ROBUSTNESS OF CONVERGENCE:

We will now study SIGNSGD's robustness when distributed by majority vote. We model adversaries as machines that may manipulate their stochastic gradient as follows.

Definition 1 (Blind multiplicative adversary). A blind multiplicative adversary may manipulate their stochastic gradient estimate gt at iteration t by element-wise multiplying gt with any vector v t of their choice. The vector v t must be chosen before observing gt , so the adversary is 'blind'. Some interesting members of this class are:

(i) adversaries that arbitrarily rescale their stochastic gradient estimate;

(ii) adversaries that randomise the sign of each coordinate of the stochastic gradient;

(iii) adversaries that invert their stochastic gradient estimate.

SGD is certainly not robust to rescaling since an adversary could set the gradient to infinity and corrupt the entire model. Our algorithm, on the other hand, is robust to all adversaries in this class.

For ease of analysis, here we derive large batch results. We make sure to give results in terms of sample complexity N (and not iteration number K) to enable fair comparison with other algorithms.

Theorem 2 (Non-convex convergence rate of majority vote with adversarial workers). Run algorithm 1 for K iterations under Assumptions 1 to 4. Switch off momentum and weight decay (β = λ = 0). Set the learning rate, η, and mini-batch size, n, for each worker as

η = f 0 -f * L 1 K , n = K.

Assume that a fraction α < 1 2 of the M workers behave adversarially according to Definition 1. Then majority vote converges at rate:

1 K K-1 k=0 E g k 1 2 ≤ 4 √ N 1 1 -2α σ 1 √ M + L 1 (f 0 -f * ) 2

where N = K 2 is the total number of stochastic gradient calls per worker up to step K.

The result is intuitive: provided there are more machines sending honest gradients than adversarial gradients, we expect that the majority vote should come out correct on average.

Remark 1: if we switch off adversaries by setting the proportion of adversaries α = 0, this result reduces to Theorem 2 in (Bernstein et al., 2018) . In this case, we note the 1 √ M variance reduction that majority vote obtains by distributing over M machines, similar to distributed SGD.

Remark 2: the convergence rate degrades as we ramp up α from 0 to 1 2 . Remark 3: from an optimisation theory perspective, the large batch size is an advantage. This is because when using a large batch size, fewer iterations and rounds of communication are theoretically needed to reach a desired accuracy, since only

√:

N iterations are needed to reach N samples. But from a practical perspective, workers may be unable to handle such a large batch size in a timely manner. It should be possible to extend the result to the mini-batch setting by combining the techniques of Theorems 1 and 2, but we leave this for future work.

4. EXPERIMENTS:

For our experiments, we distributed SIGNUM (Algorithm 1) by majority vote. SIGNUM is the momentum counterpart of SIGNSGD, where each worker maintains a momentum and transmits the sign momentum to the parameter server at each step. The addition of momentum to SIGNSGD is proposed and studied in (Balles & Hennig, 2018; Bernstein et al., 2018) .

We built SIGNUM with majority vote in the Pytorch deep learning framework (Paszke et al., 2017) using the Gloo (2018) communication library. Unfortunately Pytorch and Gloo do not natively support 1-bit tensors, therefore we wrote our own compression code to bit-pack a sign tensor down to an efficient 1-bit representation. We obtained a performance boost by fusing together smaller tensors, which saved on compression and communication costs. Figure 7 : Left: comparing convergence of majority vote to QSGD (Alistarh et al., 2017) . resnet18 is trained on Cifar-10 across M = 3 machines, each at bach size 128. 1-bit QSGD stochastically snaps gradient components to {0, ±1}. 2-way refers to the compression function Q(.) being applied in both directions of communication: machine i sends Q( gi ) to the server and gets Alistarh et al. (2017) develop a theory for L2 QSGD, but experimentally benchmark max QSGD which has much larger communication costs. For this experiment, 1-bit max QSGD gives roughly 5× more compression than the 32× compression of majority vote, but this further gain turns out to be small relative to the cost of backpropagation. See Appendix A for QSGD experiments at higher bit-precision. Right: the table gives a theoretical comparison of the compression cost of each algorithm-see Appendix B for derivations. all experiments are run at identical hyperparameter settings, with weight decay switched off for simplicity. The network still learns even at 43% adversarial. Right: at 43% adversarial, learning became slightly unstable. We decreased the learning rate for this setting, and learning stabilised. (Blanchard et al., 2017) . We train resnet18 on Cifar-10 across 7 workers, each at batch size 64. Momentum and weight decay are switched off for simplicity, and for majority vote we divide the learning rate by 10 at epoch 100. Negative adversaries multiply their stochastic gradient estimate by -10. Random adversaries multiply their stochastic gradient estimate by 10 and then randomise the sign of each coordinate. For MULTI-KRUM, we use the maximum allowed security level of f = 2. Notice that MULTI-KRUM fails catastrophically once the number of adversaries exceeds the security level, whereas majority vote fails more gracefully.

Q( M i=1 Q(g i )) sent back.

We test against SGD distributed using the state of the art NCCL (2018) communication library. NCCL provides an efficient implementation of allreduce. Our framework is often 4× faster in communication (including the cost of compression) than NCCL, as can be seen in Figure 4 . Further code optimisation should bring the speedup closer to the ideal 32×.

4.1. COMMUNICATION EFFICIENCY:

We first benchmark majority vote on the Imagenet dataset. We train a resnet50 model and disitribute learning over 7 to 15 AWS p3.2xlarge machines. These machines each contain one Nvidia Tesla V100 GPU, and AWS lists the connection speed between machines as "up to 10 Gbps".

Results are plotted in Figure 5 . Per epoch, distributing by majority vote is able to attain a similar speedup to distributed SGD. But per hour majority vote is able to process more epochs than NCCL, meaning it can complete the 80 epoch training job roughly 25% faster. In terms of overall generalisation, majority vote reaches a slightly degraded test set accuracy. We hypothesise that this may be fixed by inventing a better regularisation scheme or tuning momentum, which we did not do.

In Figure 6 

4.2. ROBUSTNESS:

In this section we test the robustness of SIGNUM with majority vote to Byzantine faults. Again we run tests on the Imagenet dataset, training resnet50 across 7 AWS p3.2xlarge machines. Our adversarial workers take the sign of their stochastic gradient calculation, but send the negation to the parameter server. Our results are plotted in Figure 8 . In the left hand plot, all experiments were carried out using hyperparameters tuned for the 0% adversarial case. Weight decay was not used in these experiments to simplify matters. We see that learning is tolerant of up to 43% (3 out of 7) machines behaving adversarially. The 43% adversarial case was slightly unstable (Figure 8 , left), but re-tuning the learning rate for this specific case stabilised learning (Figure 8 , right).

In Figure 9 we compare majority vote to MULTI-KRUM (Blanchard et al., 2017) with a security level of f = 2. When the number of adversaries exceeds f , MULTI-KRUM fails catastrophically in our experiments, whereas SIGNSGD fails more gracefully. Note that MULTI-KRUM requires 2f + 2 < M , therefore f = 2 is the maximum possible security level for these experiments with M = 7 workers.

5. DISCUSSION AND CONCLUSION:

We have analysed the theoretical and empirical properties of a very simple algorithm for distributed, stochastic optimisation. We have shown that SIGNSGD with majority vote aggregation is robust and communication efficient, whilst its per-iteration convergence rate is competitive with SGD for training large-scale convolutional neural nets on image datasets. We believe that it is important to understand this simple algorithm before going on to devise more complex learning algorithms.

An important takeaway from our theory is that mini-batch SIGNSGD should converge if the gradient noise is Gaussian. This means that the performance of SIGNSGD may be improved by increasing the per-worker mini-batch size, since this should make the noise 'more Gaussian' according to the Central Limit Theorem.

We will now give some possible directions for future work. Our implementation of majority vote may be further optimised by breaking up the parameter server and distributing it across machines. This would prevent a single machine from becoming a communication bottleneck as in our experiments. B BITS SENT PER ITERATION: SIGNSGD VS. QSGD

In this section, we perform theoretical calculations of the number of bits sent per iteration in distributed training. We compare SIGNSGD using majority vote aggregation to two forms of QSGD (Alistarh et al., 2017) . These calculations give the numbers in the table in Figure 7 .

The communication cost of SIGNSGD with majority vote is trivially 2M d bits per iterations, since at each iteration M machines send d-dimensional sign vectors up to the server, and the server sends back one d-dimensional sign vector to all M machines.

There are two variants of QSGD given in (Alistarh et al., 2017) . The first we refer to as L2 QSGD which is the version developed in the theory section of (Alistarh et al., 2017) . The second we refer to as max QSGD which is the version actually used in their experiments. For each version we compute the number of bits sent for the highest compression version of the algorithm, which is a ternary quantisation (snapping gradient components into {0, ±1}). We refer to this as 1-bit QSGD.

The higher precision versions of QSGD will send more bits per iteration.

1-bit L2 QSGD takes a gradient vector g and snaps i th coordinate g i to sign(g i ) with probability |gi| g 2 and sets it to zero otherwise. Therefore the expected number of bits set to ±1 is bounded by

E[#bits] = i=1 |g i | g 2 = g 1 g 2 ≤ √ d.

To send a vector compressed in this way, for each non-zero component 1 bit is needed to send the sign and log d bits are to send the index. Therefore sending a vector compressed by 1-bit L2 QSGD requires at most

√ d(1 + log d) bits.

In the experiments in Figure 7 we see that the '2-way' version of 1-bit L2 QSGD (which recompresses the aggregated compressed gradients) converges very poorly. Therefore it makes sense to use the 1-way version where the aggregated compressed gradient is not recompressed. A sensible way to enact this is to have each of the M workers broadcast their compressed gradient vector to all other workers. This has a cost of (M -1) √ d(1 + log d) bits for each of the M workers, and from this we get the total cost of O(M 2 √ d log d) for 1-bit L2 QSGD.

The final algorithm to characterise is 1-bit max QSGD. 1-bit max QSGD takes a gradient vector g and snaps i th coordinate g i to sign(g i ) with probability |gi| g ∞ and sets it to zero otherwise. As noted in (Alistarh et al., 2017) , there are no sparsity guarantees for this algorithm, so compression will generally be much lower than for 1-bit L2 QSGD.

It is easy to see that 1-bit max QSGD requires no more than O(d) bits to compress a d-dimensional vector, since 2d bits can always store d numbers in {0, ±1}. To see that we can't generally do better than O(d) bits, notice that 1-bit max QSGD leaves sign vectors invariant, and thus the compressed form of a sign vector requires exactly d bits. The natural way to enact 1-bit max QSGD is with a two-way compression where the M workers each send an O(d)-bit compressed gradient up to the server, and the server sends back an O(d)-bit compressed aggregated result back to the M workers. This gives a number of bits sent per iteration of O(M d).

For very sparse vectors 1-bit max QSGD will compress much better than indicated above. For a vector g with a single non-zero entry, 1-bit max QSGD will set this entry to 1 and keep the rest zero, thus requiring only log d bits to send the index of the non-zero entry. But it is not clear whether these extremely sparse vectors appear in deep learning problems. In the experiments in Figure 7 , 1-bit max QSGD led to compressed vectors that were 5× more compressed than SIGNSGD-in our experimental setting this additional improvement turned out to be small relative to the time cost of backpropagation.

C PROOFS C.1 ACCURACY OF THE SIGN STOCHASTIC GRADIENT:

Lemma 1 (Bernstein et al. (2018) ). Let gi be an unbiased stochastic approximation to gradient component g i , with variance bounded by σ 2 i . Further assume that the noise distribution is unimodal and symmetric. Define signal-to-noise ratio S i := |gi| σi . Then we have that

P[sign(g i ) = sign(g i )] ≤ 2 9 1 S 2 i if S i > 2 √ 3 , 1 2 -Si 2 √ 3

otherwise which is in all cases less than or equal to 1 2 .

Recall Gauss' inequality for unimodal random variable X with mode ν and expected squared deviation from the mode τ 2 (Gauss, 1823; Pukelsheim, 1994) :

P[|X -ν| > k] ≤ 4 9 τ 2 k 2 if k τ > 2 √ 3 , 1 -k √ 3τ

otherwise By the symmetry assumption, the mode is equal to the mean, so we replace mean µ = ν and variance

σ 2 = τ 2 . P[|X -µ| > k] ≤ 4 9 σ 2 k 2 if k σ > 2 √ 3 , 1 -k √ 3σ

otherwise Without loss of generality assume that g i is negative. Then applying symmetry followed by Gauss, the failure probability for the sign bit satisfies:

P[sign(g i ) = sign(g i )] = P[g i -g i ≥ |g i |] = 1 2 P[|g i -g i | ≥ |g i |] ≤ 2 9 σ 2 i g 2 i if |gi| σ > 2 √ 3 , 1 2 -|gi| 2 √ 3σi otherwise = 2 9 1 S 2 i if S i > 2 √ 3 , 1 2 -Si 2 √ 3 otherwise C.2 MINI-BATCH CONVERGENCE GUARANTEES

Theorem 1 (Non-convex convergence rate of mini-batch SIGNSGD). Run the following algorithm for K iterations under Assumptions 1 to 4:

x k+1 = x k -η sign(g k ).

Set the learning rate, η, and mini-batch size, n, as

η = f 0 -f * L 1 K , n = 1.

Let H k be the set of gradient components at step k with large signal-to-noise ratio S i :=

|g k,i | σi , i.e. H k := i S i > 2 √ 3 .

We refer to 2 √ 3 as the 'critical SNR'. Then we have

1 K K-1 k=0 E   i∈H k |g k,i | + i ∈H k g 2 k,i σ i   ≤ 3 L 1 (f 0 -f * ) N .

where N = K is the total number of stochastic gradient calls up to step K.

Proof. First let's bound the improvement of the objective during a single step of the algorithm for one instantiation of the noise. I[.] is the indicator function, g k,i denotes the i th component of the true gradient g(x k ) and gk is a stochastic sample obeying Assumption 3.

First take Assumption 2, plug in the algorithmic step, and decompose the improvement to expose the stochasticity-induced error:

f k+1 -f k ≤ g T k (x k+1 -x k ) + d i=1 L i 2 (x k+1 -x k ) 2 i = -ηg T k sign(g k ) + η 2 d i=1 L i 2 = -η g k 1 + η 2 2 L 1 + 2η d i=1 |g k,i | I[sign(g k,i ) = sign(g k,i )]

Next we find the expected improvement at time k + 1 conditioned on the previous iterate.

E[f k+1 -f k |x k ] ≤ -η g k 1 + η 2 2 L 1 + 2η d i=1 |g k,i | P[sign(g k,i ) = sign(g k,i )]

By Assumption 4 and Lemma 1 we have the following bound on the failure probability of the sign:

P[sign(g i ) = sign(g i )] ≤ 2 9 1 S 2 i if S i > 2 √ 3 , 1 2 -Si 2 √ 3 otherwise ≤ 1 6 if S i > 2 √ 3 , 1 2 -Si 2 √ 3

otherwise Substituting this in, we get that

E[f k+1 -f k |x k ] ≤ -η g k 1 + η 2 2 L 1 + 2η i∈H k |g k,i | 6 + 2η i ∈H k |g k,i | 1 2 - |g k,i | 2 √ 3σ i = -η d i=1 |g k,i | + η 2 2 L 1 + η i∈H k |g k,i | 3 + η i ∈H k |g k,i | -η i ∈H k g 2 k,i √ 3σ i = - 2η 3 i∈H k |g k,i | -η i ∈H k g 2 k,i √ 3σ i + η 2 2 L 1

Interestingly a mixture between an 1 and a variance weighted 2 norm has appeared. Now substitute in the learning rate schedule, and we get:

E[f k+1 -f k |x k ] ≤ - f 0 -f * L 1 K   2 3 i∈H k |g k,i | + 1 √ 3 i ∈H k g 2 k,i σ i   + f 0 -f * 2K ≤ - f 0 -f * 3 L 1 K   i∈H k |g k,i | + i ∈H k g 2 k,i σ i   + f 0 -f * 2K

Now extend the expectation over the randomness in the trajectory and telescope over the iterations:

EQUATION

Finally, rearrange and substitute in N = K to yield the bound

1 K K-1 k=0 E   i∈H k |g k,i | + i ∈H k g 2 k,i σ i   ≤ 3 √ 3 2 L 1 (f 0 -f * ) N ≤ 3 L 1 (f 0 -f * ) N .

C.3 ROBUSTNESS OF MAJORITY VOTE:

Theorem 2 (Non-convex convergence rate of majority vote with adversarial workers). Run algorithm 1 for K iterations under Assumptions 1 to 4. Switch off momentum and weight decay (β = λ = 0). Set the learning rate, η, and mini-batch size, n, for each worker as

η = f 0 -f * L 1 K , n = K.

Assume that a fraction α < 1 2 of the M workers behave adversarially according to Definition 1. Then majority vote converges at rate:

1 K K-1 k=0 E g k 1 2 ≤ 4 √ N 1 1 -2α σ 1 √ M + L 1 (f 0 -f * ) 2

where N = K 2 is the total number of stochastic gradient calls per worker up to step K.

Proof. We need to bound the failure probability of the vote. We can then use this bound to derive a convergence rate. We will begin by showing this bound is worst when the adversary inverts the signs of the sign stochastic gradient.

Given an adversary from the class of blind multiplicative adversaries (Definition 1), the adversary may manipulate their stochastic gradient estimate gt into the form v t ⊗ gt . Here v t is a vector of the adversary's choosing, and ⊗ denotes element-wise multiplication. The sign of this quantity obeys:

sign(v t ⊗ gt ) = sign(v t ) ⊗ sign(g t ).

Therefore, the only thing that matters is the sign of v t , and rescaling attacks are immediately nullified. For each component of the stochastic gradient, the adversary must decide (without observing g t , since the adversary is blind) whether or not they would like to invert the sign of that component.

We will now show that the failure probability of the vote is always larger when the adversary decides to invert (by setting every component of sign(v t ) to -1). Our analysis will then proceed under this worst case.

For a gradient component with true value g, let random variable Z ∈ [0, M ] denote the number of correct sign bits received by the parameter server. For a given adversary, we may decompose Z into the contribution from that adversary and a residual term X from the remaining workers (both regular and adversarial):

Z(sign(v)) = X + I[sign(v)sign(g) = sign(g)],

where g is the adversary's stochastic gradient estimate for that component, v is the adversary's chosen scalar for that component, and I is the 0-1 indicator function. We are considering Z to be a function of sign(v).

But by Assumption 4 and Lemma 1, we see that I[+1 × sign(g) = sign(g)] is a Bernoulli random variable with success probability p ≥ 1 2 . On the other hand, I[-1×sign(g) = sign(g)] is a Bernoulli random variable with success probability q = 1 -p ≤ 1 2 .

The essential quantity in our analysis is the probability that more than half the workers provide the correct sign bit. But from the preceding discussion, this clearly obeys:

P Z(+1) ≤ M 2 = P X + I[+1 × sign(g) = sign(g)] ≤ M 2 ≤ P X + I[-1 × sign(g) = sign(g)] ≤ M 2 = P Z(-1) ≤ M 2 .

As we will see below, the implication of this is that our bounds always worse under the setting v = -1, and so we will adopt v = -1 hereon. It is worth remarking that blindness in the definition of blind multiplicative adversaries is important to ensure that I[sign(v)sign(g) = sign(g)] is indeed a random variable as described above. Were the adversary not blind, then the adversary could in effect deterministically set sign(v)sign(g). Cooperative adversaries, for example, could use this power to control the vote.

Now we restrict to our worst case blind multiplicative adversaries, that always choose to invert their sign stochastic gradient estimate. So, we have (1 -α)M good machines and αM adversaries. The good workers each compute a stochastic gradient estimate, take its sign and transmit this to the server. The bad workers follow an identical procedure except they negate their sign bits prior to transmission to the server. It is intuitive that because the proportion of adversaries α < 1 2 , the good workers will win the vote on average. To make this rigorous, we will need Lemma 1 and Cantelli's inequality. Cantelli (1928) tells us that for a random variable X with mean µ and variance σ 2 :

P[µ -X ≥ |λ|] ≤ 1 1 + λ 2 σ 2 (1)

For a given gradient component, again let random variable Z ∈ [0, M ] denote the number of correct sign bits received by the parameter server. Let random variables G and B denote the number of good and bad workers (respectively) who (possibly inadvertently) sent the correct sign bit. Then, letting p be the probability that a good worker computed the correct sign bit, q := 1 -p and := p -1 2 we can decompose Z as follows:

Z = G + B G ∼ binomial[(1 -α)M, p] B ∼ binomial[αM, q] E[Z] = (1 -α)M p + αM q = M 2 + (1 -2α)M Var[Z] = (1 -α)M pq + αM pq = M 1 4 -2 .

The vote only fails if Z < M 2 which happens with probability < 4 S 2 by the condition on S.

P Z ≤ M 2 = P E[Z] -Z ≥ E[Z] - M 2 ≤ 1 1 + (E[Z]-M 2 ) 2 Var[Z] by Cantelli's inequality ≤ 1 2 Var[Z] (E[Z] -M 2 ) 2 since 1 + x 2 ≥ 2x = 1 2 M 1 4 -2 (1 -2α) 2 M 2 2 = 1 2 1 4 2 -1 (1 -2α) √ M

We have now completed the first part of the proof by showing the key statement that for the i th gradient component with signal to noise ratio S i := |gi| σi , the failure probability of the majority vote is bounded by

P[vote fails for i th coordinate] = P Z i ≤ M 2 ≤ 1 (1 -2α) √ M S i ( )

The second stage of the proof will proceed by straightforwardly substituting this bound into the convergence analysis of SIGNSGD from Bernstein et al. (2018) .

First let's bound the improvement of the objective during a single step of the algorithm for one instantiation of the noise. I[.] is the indicator function, g k,i denotes the i th component of the true gradient g(x k ) and sign(V k ) is the outcome of the vote at the k th iteration.

First take Assumption 2, plug in the step from Algorithm 1, and decompose the improvement to expose the error induced by stochasticity and adversarial workers:

f k+1 -f k ≤ g T k (x k+1 -x k ) + d i=1 L i 2 (x k+1 -x k ) 2 i = -ηg T k sign(V k ) + η 2 d i=1 L i 2 = -η g k 1 + η 2 2 L 1 + 2η d i=1 |g k,i | I[sign(V k,i ) = sign(g k,i )]

Next we find the expected improvement at time k + 1 conditioned on the previous iterate.

E[f k+1 -f k |x k ] ≤ -η g k 1 + η 2 2 L 1 + 2η d i=1 |g k,i | P[sign(V k,i ) = sign(g k,i )]

From ( ), we have that the probability of the vote failing for the i th coordinate is bounded by

P[sign(V k,i ) = sign(g k,i )] ≤ σ k,i (1 -2α) √ M |g k,i |

where σ k,i refers to the variance of the k th stochastic gradient estimate, computed over a mini-batch of size n. Therefore, by Assumption 3, we have that σ k,i ≤ σ i / √ n.

We now substitute these results and our learning rate and mini-batch settings into the expected improvement:

E[f k+1 -f k |x k ] ≤ -η g k 1 + 2η √ n σ 1 (1 -2α) √ M + η 2 2 L 1 = - f 0 -f * L 1 K g k 1 + 2 f 0 -f * L 1 K 2 σ 1 (1 -2α) √ M + f 0 -f * 2K

Now extend the expectation over randomness in the trajectory, and perform a telescoping sum over the iterations:

f 0 -f * ≥ f 0 -E[f K ] = K-1 k=0 E[f k -f k+1 ] ≥ K-1 k=0 E f 0 -f * L 1 K g k 1 -2 f 0 -f * L 1 K 2 σ 1 (1 -2α) √ M - f 0 -f * 2K = K(f 0 -f * ) L 1 E 1 K K-1 k=0 g k 1 -2 f 0 -f * L 1 σ 1 (1 -2α) √ M - f 0 -f * 2

We can rearrange this inequality to yield the rate:

1 K K-1 k=0 E g k 1 ≤ 1 √ K 2 σ 1 (1 -2α) √ M + 3 2 L 1 (f 0 -f * ) E 1 K K-1 k=0 g k 1 ≤ 1 √ K 3 2 L 1 (f 0 -f * ) + 2 σ 1

Since we are growing our mini-batch size, it will take N = O(K 2 ) gradient calls to reach step K. Substitute this in on the right hand side, square the result, use that 3 2 < 2, and we are done:

1 K K-1 k=0 E g k 1 2 ≤ 4 √ N σ 1 (1 -2α) √ M + L 1 (f 0 -f * ) 2

:



Figures and Tables:

Figure 3: Signal-to-noise ratio (SNR) whilst training resnet18 on Cifar-10 at batch size 128. At the start of each epoch we compute the SNR for every gradient component. We plot summary statistics like the mean over weights and the max. By roughly epoch 40, all gradient components have passed below the critical line (see Theorem 1) and remain there for the rest of training.

Figure 4: Timing breakdown for distributing on the cloud. Left: comparing communication (including compression) for training resnet50. Right: comparing communication (including compression) and computation. resnet50 results use 7 p3.2xlarge machines for training Imagenet, each at batch size 128. alexnet uses 7 p3.2xlarge machines for Imagenet, each at batch size 64. QRNN uses 3 p3.16xlarge machines for training WikiText-103, each at batch size 240.

For d weights, M machines. Derivations in Appendix B.

Figure8: Imagenet robustness experiments. We used majority vote to train resnet50 distributed across 7 AWS p3.2xlarge machines. Adversaries invert their sign stochastic gradient. Left: all experiments are run at identical hyperparameter settings, with weight decay switched off for simplicity. The network still learns even at 43% adversarial. Right: at 43% adversarial, learning became slightly unstable. We decreased the learning rate for this setting, and learning stabilised.

Figure9: Comparing the robustness of majority vote to MULTI-KRUM(Blanchard et al., 2017). We train resnet18 on Cifar-10 across 7 workers, each at batch size 64. Momentum and weight decay are switched off for simplicity, and for majority vote we divide the learning rate by 10 at epoch 100. Negative adversaries multiply their stochastic gradient estimate by -10. Random adversaries multiply their stochastic gradient estimate by 10 and then randomise the sign of each coordinate. For MULTI-KRUM, we use the maximum allowed security level of f = 2. Notice that MULTI-KRUM fails catastrophically once the number of adversaries exceeds the security level, whereas majority vote fails more gracefully.

Figure 10: QSGD at varying levels of precision. Top row: L2 QSGD. Bottom row: max QSGD.resnet18 is trained on Cifar-10 across M = 3 machines, each at bach size 128. The one-way version of QSGD is used, meaning that the compression function is not re-applied after aggregation. The comparison is given in terms of number of epochs. A comparison in terms of wall-clock time will depend on details of the systems implementation.

This correspondence suggests that Assumption 4 should be useful for obtaining mini-batch convergence guarantees for ADAM. Note that when Reddi et al. (2018) construct toy divergence examples for ADAM, they rely on bimodal noise distributions which violate Assumption 4.

Table content: <table/>

we compare majority vote to ADAM (distributed by NCCL) for training QRNN(Bradbury et al., 2017) on WikiText-103. Majority vote completes an epoch roughly 3 times faster than ADAM, but it reaches a degraded accuracy so that the overall test perplexity after 2 hours ends up being similar. In Figure7we show that majority vote has superior convergence to the 'theory' version of QSGD thatAlistarh et al. (2017) develop. Convergence is similar for the 'max' version thatAlistarh et al. (2017) use in their experiments. Additional results are given in Appendix A.

Table content: <table/>

Peer review (iclr2019):
 The authors present a distributed implementation of signSGD with majority vote as aggregation. The result is a communication efficient and byzantine robust distributed training method. This is an interesting and relevant problem. There are two parts in this paper: first the authors prove a convergence guarantee for signSGD, and then they prove that under a weak adversary attack signSGD will be robust to a constant fraction of adversarial nodes. The authors conclude with some limited experiments. Overall, the idea of combining low-communication methods with byzantine resilience is quite interesting. That is, by limiting the domain of the gradients one expects that the power of an adversary would be limited too. The application of the majority vote on the gradients is an intuitive technique that can resolve weak adversarial attacks. Overall, I found the premise quite interesting. There are several issues that if fixed this could be a great paper, however I am not sure if there is enough time between rebuttals to achieve this for this round of submissions. I will summarize these key issues below. 1) Although the authors claim that this is a communication efficient technique, signSGD (on its communication merit) is not compared with any state of the art communication efficient training algorithm, for example: - 1Bit SGD [1] - QSD [2] - TernGrad [3] - Deep Gradient compression [4] I think it is important to include at least one of those algorithms in a comparison. Due to the lack of comparisons with state of the art it is hard to argue on the relative performance of signSGD. 2) Although the authors claim byzantine resilience, this is against a very weak type of adversary, eg one that only sends back the opposite sign of the local stochastic gradient. An omniscient adversary can craft attacks that are significantly more sophisticated, for which a simple majority vote would not work. Please see the results in [b1]. 3) The authors although reference some limited literature on byzantine ML, they do not compare with other byzantine tolerant ML methods. For example check [eg, b1-b4] below. Again, due to the lack of comparisons with state of the art it is hard to argue on the relative performance of signSGD. Overall, although the presented ideas are promising, a substantial revision is needed before this paper is accepted for publication. I think it is extremely important that an extensive comparison is carried out with respect to both communication efficient algorithms, and/or byzantine tolerant algorithms, since signSGD aims to be competitive with both of these lines of work. This is a paper that has potential, but is currently limited by its lack of appropriate comparisons. [1] https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/IS140694.pdf [2] https://papers.nips.cc/paper/6768-qsgd-communication-efficient-sgd-via-gradient-quantization-and-encoding.pdf [3] https://papers.nips.cc/paper/6749-terngrad-ternary-gradients-to-reduce-communication-in-distributed-deep-learning.pdf [4] https://arxiv.org/pdf/1712.01887.pdf [b1] https://arxiv.org/pdf/1802.07927.pdf [b2] https://arxiv.org/pdf/1803.01498.pdf [b3] https://dl.acm.org/citation.cfm?id=2933105 [b4] https://arxiv.org/pdf/1804.10140.pdf [b5] https://arxiv.org/pdf/1802.10116.pdf ######################## I would like to commend the authors for making a significant effort in revising their manuscript. Specifically, I think adding the experiments for QSGD and Krum are an important addition. However, I still have a few major that in my opinion are significant: - The experiments for QSGD are only carried for the 1-bit version of the algorithm. It has been well observed that this is by far the least well performing variant of QSGD. That is, 4 or 8 bit QSGD seems to be significantly more accurate for a given time budget. I think the goal of the experiments should not be to compare against other 1-bit algorithms (though to be precise, 1-bit QSGD is a ternary algorithm) , but against the fastest low-communication algorithm. As such, although the authors made an effort in adding more experiments, I am still not convinced that signSGD will be faster than 4 or 8 bit QSGD. I want to also acknowledge in this comment the fact that these experiments do take time, and are not easy to run, so I commend them again for this effort. - My second comment relates to comparisons with state of the art algorithms in byzantine ML. The authors indeed did compare against Krum, however, as noted in my original review there are many works following Blanchard et al. For example as I noted https://arxiv.org/pdf/1802.07927.pdf (the Bulyan algorithm) shows that there exist significantly stronger defense mechanisms for byzantine attacks. I think it would have been a much stronger comparison to compare with Bulyan. Overall, I think the paper has good content, and the authors significantly revised their paper according to the reviews. However, several more experiments are needed for convincing a potential reader of the main claims of the paper, i.e., that signSGD is a state of the art communication efficient and byzantine tolerant algorithm. I will increase my score from 5 to 6, and I will not oppose the paper being rejected or accepted. My personal opinion is that a resubmission for a future venue would yield a much stronger and more convincing paper assuming more extensive and thorough comparisons are added.