Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a novel way of compressing gradient updates for distributed SGD, in order to speed up overall execution. While the technique is novel as far as I know (eq. (1) in particular), many details in the paper are poorly explained (I am unable to understand) and experimental results do not demonstrate that the problem targeted is actually alleviated.

More detailed remarks:
1: Motivating with ImageNet taking over a week to train seems misplaced when we have papers claiming to train ImageNet in 1 hour, 24 mins, 15 mins...
4.1: Lemma 4.1 seems like you want B > 1, or clarify definition of V_B.
4.2: This section is not fully comprehensible to me.
- It seems you are confusingly overloading the term gradient and words derived (also in other parts or the paper). What is "maximum value of gradients in a matrix"? Make sure to use something else, when talking about individual elements of a vector (which is constructed as an average of gradients), etc.
- Rounding: do you use deterministic or random rounding? Do you then again store the inaccuracy?
- I don't understand definition of d. It seems you subtract logarithm of a gradient from a scalar.
- In total, I really don't know what is the object that actually gets communicated, and consequently when you remark that this can be combined with QSGD and the more below it, I don't understand it. This section has to be thoroughly explained, perhaps with some illustrative examples.
4.3: allgatherv remark: does that mean that this approach would not scale well to higher number of workers?
4.4: Remarks about quantization and mantissa manipulation are not clear to me again, or what is the point in doing so. Possible because the problems above.
5: I think this section is not too useful unless you can accompany it with actual efficient implementation and contrast the practical performance. 
6: Given that I don't understand how you compress the information being communicated, it is hard to believe the utility of the method. The objective was to speed up training time because communication is bottleneck. If you provide 12,000x compression, is it any more practically useful than providing 120x compression? What would be the difference in runtime? Such questions are never discussed. Further, if in the implementation you discuss masking mantissa, I have serious concern about whether the compression protocol is feasible to implement efficiently, without writing some extremely low-level code. I think the soundness of work addressing this particular problem is damaged if not implemented properly (compared to other kinds of works in current ML related research). Therefore I highly recommend including proper time comparison with a baseline in the future.
Further, I don't understand 2 things about the Tables. a) how do you combine the proposed method with Momentum in SGD? This is not discussed as far as I can see. b) What is "QSGD, 2bit" If I remember QSGD protocol correctly, there's no natural mapping of 2bit to its parameters.
review_title: Ok but not good enough

final_score: rating: 7: Good paper, accept
review_content: The authors propose a new gradient compression method for efficient distributed training of neural networks. The authors propose a novel way of measuring ambiguity based on the variance of the gradients. In the experiment, the proposed method shows no or slight degradation of accuracy with big savings in communication cost. The proposed method can easily be combined with other existing method, i.e., Storm (2015), based on the absolute value of the gradient and shows further efficiency. 

The paper is well written: clear and easy to understand. The proposed method is simple yet powerful. Particularly, I found it interesting to re-evaluate the variance with (virtually) increasing larger batch size. The performance shown in the experiments is also impressive. 

I found it would have also been interesting and helpful to define and show a new metric that incorporates both accuracy and compression rate into a single metric, e.g., how much accuracy is lost (or gained) per compression rate relatively to the baseline of no compression. With this metric, the comparison would be easier and more intuitive. 

review_title: Simple yet efficient new algorithm for gradient compression with good performance.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a variance-based gradient compression method to reduce the communication overhead of distributed deep learning. Experiments on real datasets are used for evaluation. 

The idea to adopt approximated variances of gradients to reduce communication cost seems to be interesting. However, there also exist several major issues in the paper.

Firstly, the authors propose to combine two components to reduce communication cost, one being variance-based gradient compression and the other being quantization and parameter encoding. But the contributions of these two components are not separately analyzed or empirically verified. 

Secondly, the experimental results are unconvincing. The accuracy of Momentum SGD for ‘Strom, \tau=0.01’ on CIFAR-10 is only 10.6%. Obviously, the learning procedure is not convergent. It is highly possible that the authors do not choose a good hyper-parameter. Furthermore, the proposed method (not the hybrid) is not necessarily better than Strom except for the case of Momentum SGD on CIFAR-10. Please note that the case of Momentum SGD on CIFAR-10 may have a problematic experimental setting for Strom. In addition, it is weird that the experiment on ImageNet does not adopt the same setting as that on CIFAR-10 to evaluate both Adam and Momentum SGD. 

review_title: The idea to adopt approximated variances of gradients to reduce communication cost seems to be interesting. However, there also exist several major issues in the paper.

Meta-review: The reviewers find the gradient compression approach novel and interesting, but they find the empirical evaluation not fully satisfactory. Some aspects of the paper have improved with the feedback from the reviewers, but because of the domain of the paper, experimental evaluation is very important. I recommend improving the experiments by incorporating the reviewers' comments.
Decision: Invite to Workshop Track