Title: Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Authors: Yujun Lin
Song Han
Huizi Mao
Yu Wang
Bill Dally
Venue: ICLR 2018
Keywords: distributed training

Abstract: Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

TL;DR: we find 99.9% of the gradient exchange in distributed SGD is redundant; we reduce the communication bandwidth by two orders of magnitude without losing accuracy. 

Review discussion:
I think this is a good work that I am sure will have some influence in the near future. I think it should be accepted and my comments are mostly suggestions for improvement or requests for additional information that would be interesting to have.
Generally, my feeling is that this work is a little bit too dense, and would like to encourage the authors in this case to make use of the non-strict ICLR page limit, or move some details to appendix and focus more on more thorough explanations. With increased clarity, I think my rating (7) would be higher.
Several Figures and Tables are never referenced in the text, making it a little harder to properly follow text. Pointing to them from appropriate places would improve clarity I think.
Algorithm 1 line 14: You never seem to explain what is sparse(G). Sec 3.1: What is it exactly that gets communicated? How do you later calculate the Compression Ratio? This should surely be explained somewhere.
Sec 3.2 you mention 1% loss of accuracy. A pointer here would be good, at that point it is not clear if it is in your work later, or in another paper. The efficient momentum correction is great!
As I was reading the paper, I got to the experiments and realized I still don't understand what is it that you refer to as "deep gradient compression". Pointer to Table 1 at the end of Sec 3 would probably be ideal along with some summary comments.
I feel the presentation of experimental results is somewhat disorganized. It is not clear what is immediately clear what is the baseline, that should be somewhere stressed. I find it really confusing why you sometimes compare against Gradient Dropping, sometimes against TernGrad, sometimes against neither, sometimes include Gradient Sparsification with momentum correction (not clear again what is the difference from DGC). I recommend reorganizing this and make it more consistent for sake of clarity. Perhaps show here only some highlights, and point to more in the Appendix.
Sec 5: Here I feel would be good to comment on several other things not mentioned earlier.
Why do you only work with 99.9% sparsity? Does 99% with 64 training nodes lead to almost dense total updates, making it inefficient in your communication model? If yes, does that suggest a scaling limit in terms of number of training nodes? If not, how important is the 99.9% sparsity if you care about communication cost dominating the total runtime? I would really like to better understand how does this change and what is the point beyond which more sparsity is not practically useful.
Put differently, is DGC with 600x size reduction in total runtime any better than DGC with 60x reduction?
Finally, a side remark:
Under eq. (2) you point to something that I think could be more discussed.
When you say what you do has the effect of increasing stepsize, why don't you just increase the stepsize?
There has recently been this works on training ImageNet in 1 hour, then in 24 minutes, latest in 15 minutes... You cite the former, but highlight different part of their work. Broader idea is that this is trend that potentially makes this kind of work less relevant.
While I don't think that makes your work bad or misplaced, I think mentioning this would be useful as an alternative approach to the problems you mention in the introduction and use to motivate your contribution.
...what would be your reason for using DGC as opposed to just increasing the batch size?
We thank the reviewer for the comments.
-    Several Figures and Tables are never referenced in the text, making it a little harder to properly follow text. Pointing to them from appropriate places would improve clarity I think.
We revised our paper. All the figures and tables are referenced properly in the text.
-    Algorithm 1 line 14: You never seem to explain what is sparse(G). Sec 3.1: What is it exactly that gets communicated? How do you later calculate the Compression Ratio?
We have change the name of function to encode(G). The encode() function packs 32-bit nonzero gradient values and 16-bit run lengths of zeros in the flattened gradients. The encoded sparse gradients get communicated. These are described in the Sec 3.1 now.
The compression ratio is calculated as follows:
The Gradient Compression Ratio = Size[ encode( sparse( G_k ) ) ] / Size [G_k]
It is defined in the Sec 4.1 now.
-    Sec 3.2 you mention 1% loss of accuracy. A pointer here would be good, at that point it is not clear if it is in your work later, or in another paper.
We pointed to the Figure 3(a) in the updated draft, and also cite the paper AdaComp [1].
-    Pointer to Table 1 at the end of Sec 3 would probably be ideal along with some summary comments.
We make a summary at the end of Sec 3 and add Appendix D to show the overall algorithm of DGC in the updated draft.
-    I find it really confusing why you sometimes compare against Gradient Dropping, sometimes against TernGrad, sometimes against neither, sometimes include Gradient Sparsification with momentum correction (not clear again what is the difference from DGC).
Because related work didn’t cover them all.  Gradient Dropping [2] only performed experiments on 2-layer LSTM for NMT, and 3-layer DNN for MNIST; TernGrad [3] only performed experiments on AlexNet, GoogleNet and VGGNet.
Therefore, we compared our AlexNet result with TernGrad.
DGC contains not only momentum correction but also momentum factor masking and warm-up training. Momentum correction and Local gradient clipping are proposed to improve local gradient accumulation. Momentum factor masking and warm-up training are proposed to overcome the staleness effect. Comparison between Gradient Sparsification with momentum correction and DGC shows their impact on training respectively.
-    Why do you only work with 99.9% sparsity? Does 99% with 64 training nodes lead to almost dense total updates, making it inefficient in your communication model? If yes, does that suggest a scaling limit in terms of number of training nodes? If not, how important is the 99.9% sparsity if you care about communication cost dominating the total runtime?
Yes, 99% with 128 training nodes lead to almost dense total updates, making it inefficient in communication. The scaling limit N in terms of number of training nodes depends on the gradient sparsity s: N ≈1/(1-s).
When the gradient sparsity is 99.9%, the scaling limit is 1024 training nodes.
-    When you say what you do has the effect of increasing stepsize, why don't you just increase the stepsize? What would be your reason for using DGC as opposed to just increasing the batch size?
Since the memory on GPU is limited, the way to increase the stepsize is to increase training nodes. Previous work in increasing the stepsize focus on how to deal with very large mini-batch training, while our work focus on how to reduce the communication consumption among increased nodes under poor network bandwidth. DGC can be considered as increasing the stepsize temporally on top of increasing the actual stepsize spatially.
References:
[1] Chen, Chia-Yu, et al. "AdaComp: Adaptive Residual Gradient Compression for Data-Parallel Distributed Training." arXiv preprint arXiv:1712.02679 (2017).
[2] Aji, Alham Fikri, and Kenneth Heafield. Sparse Communication for Distributed Gradient Descent. In Empirical Methods in Natural Language Processing (EMNLP), 2017.
[3] Wen, Wei, et al. TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning. In Advances in Neural Information Processing Systems, 2017.
Thanks for the response, I feel confident this contribution should be accepted.
This paper proposes additional improvement over gradient dropping(Aji & Heafield) to improve communication efficiency.
- First of all, the experimental results are thorough and seem to suggest the advantage of the proposed techniques.
- The result for gradient dropping(Aji & Heafield) should be included in the ImageNet experiment.
- I am having a hard time understanding the intuition behind v_t introduced in the momentum correction. The authors should provide some form of justifications.
- For example, provide an equivalence provide to the original update rule or some error analysis would be great
- Did you keep a running sum of v_t overall history? Such sum without damping(the m term in momentum update) is likely lead to the growing dominance of noise and divergence.
- The momentum masking technique seems to correspond to stop momentum when a gradient is synchronized. A discussion about the relation to asynchronous update is helpful.
- Do you do non-sparse global synchronization of momentum term? It seems that local update of momentum is likely going to diverge,  and the momentum masking somehow reset that.
- In the experiment, did you perform local aggregations of gradients between GPU cards before send out to do all0reduce in a network? since doing so will reduce bandwidth requirement.
In general, this is a paper shows good empirical results.
But requires more work to justify the proposed correction techniques.
I have read the authors updates and changed my score accordingly(see series of discussions)
I take a look at the other reviews after they get online.
While most of them give accept to the paper given the strong empirical results provided by the paper.
However, the problem is mentioned by all the reviewers(it is unclear why momentum correction is needed and the intuition behind this). I would like to emphasize that the current rule seems will is likely lead to the growing dominance of noise and divergence (because of no damping).
I strongly encourage the author to clarify this. We cannot simply accept a paper for great empirical result without justification of why the rule works (all the reviewers are confused by this point)
We thank the reviewer for the comments. We have revised our paper.
-    Did you keep a running sum of v_t overall history? Such sum without damping(the m term in momentum update) is likely lead to the growing dominance of noise and divergence.  Do you do non-sparse global synchronization of momentum term? It seems that local update of momentum is likely going to diverge, and the momentum masking somehow reset that.
We already revised our paper, and described the momentum correction more precisely in Section 3.2.
Basically, the momentum correction performs the momentum SGD without update locally, and accumulates the velocity u_t locally. The optimization performs SGD with v_t instead of momentum SGD with G_t after momentum correction.  We add figure 2 to illustrate the difference.
Therefore, we do not keep a running sum of v_t overall history, but keep a running sum of u_t. v_t is the running sum result and will be cleared after update (with or without momentum factor masking).
For example, at iteration t-1,
u_{t-1}  = m^{t-2} g_{1}+ … + m g_{t-2} + g_{t-1},
v_{t-1}  = (1+…+m^{t-2}) g_{1} + … + (1+m) g_{t-2} + g_{t-1}.
Update, w_{t} = w_{1} – lr x v_{t-1}
After update, v_{t-1}  = 0.
Next iteration,
u_{t} = m^{t-1} g_{1} + … + m g_{t-1} + g_{t},
v_{t}  = m^{t-1} g_{1} + … + m g_{t-1} + g_{t}.
Update, w_{t+1} = w_{t} – lr x v_{t}
= w_{1} – lr x (v_{t-1} + v_{t} )
= w_{1} - lr x [ (1+…+m^{t-1}) g_{1} + … + (1+m) g_{t-1} + g_{t} ]
Which is the same as the dense momentum SGD.
-    Did you perform local aggregations of gradients between GPU cards before send out to do all0reduce in a network?
Yes.
This is the comment after author's updated a revised version of the paper.
I now understand the proposed momentum correction method(with some effort).
However, the presentation can be further improved.  I list my suggestions here:
Clarify the approach:
- The author should emphasize that the sparsification + thresholding have async nature, as the update only trigger when sparse condition applies and causes the mismatch problem.
- It is a bit hard to understand the vanilla baseline(I might not do it in that way). It can be explained as local gradient aggregation and only apply momentum update at the trigger point. The update rule is no longer equivalent to SGD with momentum.
- The paper did not explicitly mention that the value of v_t gets reset after triggering a communication, it should be explicitly mentioned in the update equation.
Justify the correctness:
Rough intuition gives most of the current justification. We can know that one is better than another, because of the equivalence.
The author should try to do more formal justification, at least in some special cases, instead of leaving it as a debt for yourself or the potential readers
- For example, the author should be able to prove that, under only one machine and one weight value. The weight value after K updates using the vanilla approach vs. the new approach.
-A fundamental question that needs to be answered, is that why thresholding trigger method(which is async) works as good as sync SGD. A proof sketch to show the loss change after K step update would shed light on this and may give insights on what a good update rule is(possibly borrow some analysis from aync update and stale gradient)
- Therefore, we do not keep a running sum of v_t overall history, but keep a running sum of u_t. v_t is the running sum result and will be cleared after update (with or without momentum factor masking).
For example, at iteration t-1,
u_{t-1}  = m^{t-2} g_{1}+ … + m g_{t-2} + g_{t-1},
v_{t-1}  = (1+…+m^{t-2}) g_{1} + … + (1+m) g_{t-2} + g_{t-1}.
Update, w_{t} = w_{1} – lr x v_{t-1}
After update, v_{t-1}  = 0.
Next iteration,
u_{t} = m^{t-1} g_{1} + … + m g_{t-1} + g_{t},
v_{t}  = m^{t-1} g_{1} + … + m g_{t-1} + g_{t}.
Update, w_{t+1} = w_{t} – lr x v_{t}
= w_{1} – lr x (v_{t-1} + v_{t} )
= w_{1} - lr x [ (1+…+m^{t-1}) g_{1} + … + (1+m) g_{t-1} + g_{t} ]
Which is the same as the dense momentum SGD.
In the momentum factor masking section, both v_t and u_t get reset after trigerring a communication. Why only v_t gets reset in your example?
Thank you for your question.
This example is to show how the momentum correction works, and therefore we do not consider the staleness effect in this example.
Suppose the last update is at iteration t-1, the next update at iteration t+T-1, and we only consider the gradients  { g_{t}, g_{t+1}, ..., g_{t+T-1} }
- Dense Update
w_{t+T} = w_{t} - lr x [ ... + (1 + m +  ... + m^{T-1}) x g_{t} +  (1 + m +  ... + m^{T-2}) x g_{t+1} + ... + 1 x g_{t+T-1}]
w_{t+\tau} = w_{t} - lr x [ ... + (1 + m +  ... + m^{\tau-1}) x g_{t} +  (1 + m +  ... + m^{\tau-2}) x g_{t+1} + ... + (1 + m +  ... + m^{\tau-T}) x g_{t+T-1} + ...], where \tau > T
- Only local gradient accumulation
the coefficients of  { g_{t}, g_{t+1}, ..., g_{t+T-1} } are always the same.
w_{t+T} = w_{t} - lr x [ ... + 1 x g_{t} +  1 x g_{t+1} + ... +  1 x g_{t+T-1}]
w_{t+\tau} = w_{t} - lr x [ ... + (1 + m + m^2 + ... + m^{\tau-T}) x (g_{t} + g_{t+1} + ... + g_{t+T-1}) + ...]
- With the momentum correction,
the coefficients of  { g_{t}, g_{t+1}, ..., g_{t+T-1} } are always the same as the dense update.
w_{t+T} = w_{t} - lr x [ ... + (1 + m +  ... + m^{T-1}) x g_{t} +  (1 + m +  ... + m^{T-2}) x g_{t+1} + ... + 1 x g_{t+T-1}]
w_{t+\tau} = w_{t} - lr x [ ... + (1 + m +  ... + m^{\tau-1}) x g_{t} +  (1 + m +  ... + m^{\tau-2}) x g_{t+1} + ... + (1 + m +  ... + m^{\tau-T}) x g_{t+T-1} + ...], where \tau > T
- With the momentum correction and momentum factor masking
we clear the local u_{t} to prevent the delayed gradients from misleading the optimization after they are used for the update.
w_{t+T} = w_{t} - lr x [ ... + (1 + m +  ... + m^{T-1}) x g_{t} +  (1 + m +  ... + m^{T-2}) x g_{t+1} + ... + 1 x g_{t+T-1}]
w_{t+\tau} = w_{t} - lr x [ ... + (1 + m +  ... + m^{T-1}) x g_{t} +  (1 + m +  ... + m^{T-2}) x g_{t+1} + ... + 1 x g_{t+T-1} + ...], where \tau > T
We thank the reviewer for the suggestions. We have revised our paper.
- The author should emphasize that the sparsification + thresholding have async nature, as the update only trigger when sparse condition applies and causes the mismatch problem.
We talked about the asynchrony nature in another way in the Sec 3.3: “Because we delay the update of small gradients, when these updates do occur, they are outdated or \emph{stale}.”
- It is a bit hard to understand the vanilla baseline(I might not do it in that way). It can be explained as local gradient aggregation and only apply momentum update at the trigger point. The update rule is no longer equivalent to SGD with momentum.
We revised our paper to describe the vanilla sparse SGD + momentum baseline in Sec 3.2: "If SGD with the momentum is directly applied to the sparse gradient scenario (line 15 in Algorithm \ref{alg:ssgd}), the update rule is no longer equivalent to Equation \ref{eq:msgd}, which becomes:".
- The paper did not explicitly mention that the value of v_t gets reset after triggering a communication, it should be explicitly mentioned in the update equation. Justify the correctness:
We revised our paper to explicitly mention resetting the value of v_t by the mask in Sec 3.2: "Similarly to the line 12 in Algorithm \ref{alg:ssgd}, the accumulation result $v_{k,t}$ gets cleared by the mask in the $sparse\left( \right)$ function."
Rough intuition gives most of the current justification. We can know that one is better than another, because of the equivalence. The author should try to do more formal justification, at least in some special cases, instead of leaving it as a debt for yourself or the potential readers
- For example, the author should be able to prove that, under only one machine and one weight value. The weight value after K updates using the vanilla approach vs. the new approach.
We have shown that the sparsification + local gradient accumulation can be considered as “increasing the batch size over time” in Sec 3.1.
-A fundamental question that needs to be answered, is that why thresholding trigger method(which is async) works as good as sync SGD. A proof sketch to show the loss change after K step update would shed light on this and may give insights on what a good update rule is(possibly borrow some analysis from aync update and stale gradient)
[1] has shown that running stochastic gradient descent (SGD) in an asynchronous manner can be viewed as adding a momentum-like term to the SGD iteration and a smaller momentum coefficient can work as good as sync SGD. We introduced the momentum factor masking to dynamically reduce the momentum term in Sec 3.3.
Meanwhile, the asynchrony nature in the sparsification + local gradient accumulation can be considered as “increasing the batch size over time” in Sec 3.1, and there’s numerous work showing that increasing the batch size is feasible.
References
[1] Mitliagkas, I., Zhang, C., Hadjis, S., & Re, C. (2017). Asynchrony begets momentum, with an application to deep learning. In 54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016.
The paper is thorough and on the whole clearly presented.
However, I think it could be improved by giving the reader more of a road map w.r.t. the guiding principle. The methods proposed are heuristic in nature, and it's not clear what the guiding principle is. E.g., "momentum correction". What exactly is the problem without this correction? The authors describe it qualitatively, "When the gradient sparsity is high, the interval dramatically increases, and thus the significant momentum effect will harm the model performance". Can the issue be described more precisely?
Similarly for gradient clipping, "The method proposed by Pascanu et al. (2013) rescales the gradients whenever the sum of their L2-norms exceeds a threshold. This step is conventionally executed after gradient aggregation from all nodes. Because we accumulate gradients over iterations on each node independently, we perform the gradient clipping locally before adding the current gradient... " What exactly is the issue here? It reads like a story of what the authors did, but it's not really clear why they did it.
The experiments seem quite thorough, with several methods being compared. What is the expected performance of the 1-bit SGD method proposed by Seide et al.?
re. page 2: What exactly is "layer normalization"?
re. page 4: What are "drastic gradients"?
We thank the reviewer for the comments.
-    What exactly is the problem without this correction? Can the issue be described more precisely?
We already revised our paper, and described the momentum correction more precisely in Section 3.2. Basically, the momentum correction performs the momentum SGD without update locally and accumulates the velocity u_t locally.
-    What exactly is the issue of Gradient clipping?
When training RNN, people usually use Gradient Clipping to avoid the exploding gradient problem. The hyper-parameter for Gradient Clipping is the threshold thr_G of the gradients L2-norm. The gradients for optimization is scaled by a coefficient depending on their L2-norm.
Because we accumulate gradients over iterations on each node independently, we need to scale the gradients before adding them to the previous accumulation, in order to scale the gradients by the correct coefficient. The threshold for local gradient clipping thr_Gk should be set to N^{-1/2} x thr_G. We add Appendix C to explain how N^{-1/2} comes.
-    What is the expected performance of the 1-bit SGD method proposed by Seide et al.?
1-bit SGD [1] encodes the gradients as 0 or 1, so the data volume is reduced by 32x.
Meanwhile, since 1-bit SGD quantizes the gradients column-wise, a floating-point scaler per column is required, and thus it cannot yield much speed benefit on convolutional neural networks.
-    What exactly is "layer normalization"
“Layer Normalization” is similar to batch normalization but computes the mean and variance from the summed inputs in a layer on a single training case. [2]
-	 What are "drastic gradients"?
It means the period when the network weight changes dramatically.
References:
[1] Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
[2] J. Lei Ba, J. R. Kiros, and G.E.Hinton, Layer Normalization. ArXiv e-prints, July 2016

Meta-review:
role: PC
data: comment: This work proposes a hybrid system for large-scale distributed and federated training of commonly used deep networks. This problem is of broad interest and these methods have the potential to be significantly impactful, as is attested by the active and interesting discussion on this work. At first there were questions about the originality of this study, but it seems that the authors have now added extra references and comparisons.

Reviewers were split about the clarity of the paper itself. One notes that "on the whole clearly presented", but another finds it too dense, disorganized and needing of more clear explanation. Reviewers were also concerned that methods were a bit heuristic and could benefit from more details. There were also many questions about these details in the discussion forum, these should make it into the next version.  The main stellar aspect of the work were the experimental results, and reviewers call them "thorough" and note they are convincing. 

Decision: Accept (Poster)