Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper studies how asynchrony affects model training by investigating dynamic stability of minimum points that A-SGD can access. They point out that not all local minimum points are accessible, and asynchrony can affect which minimum points can be accessed, and thus helps to explain why models trained by A-SGD have higher generalization gap. The authors also propose shifted-momentum that utilize momentum for asynchronous training.

Overall, this paper provides nice insights and thorough theoretical analysis. Experiments are carefully designed to validate their results. I think this paper is well written and its novelty is significant.

Strength:
- Theoretical formulation and analysis in this paper is nice and elegant.
- Provide theoretical insights of A-SGD with momentum, which is important.
- Experiments of minima selection are carefully designed. I like the idea to observe trajectories ``leaving minimum''.

Some quick questions:
- In Fig. 3, we can clearly see a threshold of \eta. I notice that when \tau=16 the fluctuation is more significant than other three cases. Can you explain why this appears?
- In Sec. 3.1, do you consider any kind of learning rate scheduling to change learning rate over epochs, like you did in Sec. 3.2?
- It would be great to evaluate on more tasks, as it has been shown that some may be more robust than others (Dai et al., 2019).

Wei Dai, Yi Zhou, Nanqing Dong, Hao Zhang, and Eric Xing. Toward Understanding the Impact of Staleness in Distributed Machine Learning. In Proc. International Conference on Learning Representations (ICLR), 2019.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The authors introduce a theoretical model for delayed gradients in asynchronous training. It is a very nice model and solving the corresponding differential equation allows to study its stability. Authors derive stability bounds for pure SGD (learning rate needs to decrease with delay) and for SGD with momentum, where they introduce a nice momentum formulation that improves stability. These are nice insights and good results and they are validated by experiments. More experiments and practical analysis would be welcome though. Some example questions: would introducing some sychronization help? Is the lower learning rate hurting training speed when measures as wall-clock time to accuracy?

I am very grateful for the authors' response. It would still be good to see more experiments, but I hope this paper gets accepted.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The authors model A-SGD as a dynamical system, where parameters are updated with delayed gradients. The authors analyze the stability of this system, and they first derive that the learning rate must scale linearly with the inverse of the delay around a minimum to remain stable. Using a similar analysis they show that the standard way of incorporating momentum into A-SGD requires small learning rates for high momentum values, and they propose "shifted momentum," which allows for stability under higher momentum values. Experimentally, the authors show that around minima the learning rate needed to retain stability scales linearly with the inverse of the delay, that there appears to be an analogous threshold when training models from scratch, that shifted momentum allows for higher momentum values, and finally that on several datasets A-SGD with an appropriate learning rate is able to generalize at least as well as large batch synchronous training.

This is a nice paper with a large number of interesting theoretical and experimental results, and I believe it should be accepted. I think there are some largely presentational issues that should be addressed, however:

- I think the authors should attempt to make a stronger case for the practical implications of their analysis: in particular, in the most practical setting (where we don't have a minimum obtained from synchronous training), what does the provided analysis allow us to do? Part of this might involve being more explicit about the results in Table 1: what exactly was the procedure for selecting the learning rates? Is it meaningfully different than just lowering the learning rate?

- Equation (3) is rather obscure without the Appendix, especially since unbolded x hasn't been introduced anywhere. I think the authors should try to convey more of what's going on in this equation in the main text.

Minor: 'looses' should be 'loses' throughout, and it might be good to include a conclusion section. 
review_title: Official Blind Review #3

Meta-review: The paper considers the problem of training neural networks asynchronously, and the gap in generalization due to different local minima being accessible with different delays. The authors derive a theoretical model for the delayed gradients, which provide prescriptions for setting the learning rate and momentum.

All reviewers agreed that this a nice paper with valuable theoretical and empirical contributions.


Decision: Accept (Spotlight)