Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
In this paper, the authors provide a series of new statistical algorithms which are particular suitable for distributed learning. The key difference lies between the proposed algorithm and the traditional SGD is that the update only relies on the sign of the stochastic gradient. Due to such update rule, the algorithm only requires 1-bit information about the gradient estimation, therefore, the communication cost will be largely reduced. The authors also provide the convergence guarantee of the algorithm, as well as the empirical comparison on the ResNet-20 on CIFAR-10, where the proposed algorithm achieves better results.

I vote for acceptance of this paper. My only concern is the explanation of the algorithm and the convergence theorem. Specifically, the intuition why only the sign of the gradient estimation is enough for the convergence of the algorithm is not clear. I understand the details of the technical proof of the convergence are omitted due to the space limitation. However, to convey a fully reasonable idea, at least the intuitation and the interpretation of the theorem should be added. 

Secondly, as one of the major benefits of such algorithm is its efficiency in terms of the communication cost, it will be great if the empirical communication cost is compared with SGD and its variants, e.g., RMSprop and ADAM.
review_title: A solid contribution, but needs more explanation

final_score: rating: 7: Good paper, accept
review_content: This paper discusses SignGD, an alternation of GD that uses only the sign of the gradient components. Under several assumptions, the authors provide guarantees that ensure convergence to a stationary point. They also compare their method to SGD for a deep learning task.

Theoretical part:
The theoretical part is nice and illustrates the convergence rate of SignGD to a stationary point.
Unfortunately, it seems like in theory SignGD  is slow compared to GD. Nevertheless, SignGD has a benefit in the distributed setting since we only have to communicate signs rather then full gradients.  


Experimental part:
-The experiments are interesting and show that SignGD has comparable convergence to Adam and GD.


I recommend to accept the paper to ICLR workshop.

review_title: Nice paper

Meta-review: Congratulations, your paper was accepted to the ICLR workshop.
Decision: Accept