Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents a simple momentum scheme which can be applied to distributed and decentralized SGD schemes. The scheme proposes to do a sequence inner/local steps of any optimizer without any momentum, but then only apply momentum on the outer level, after each global synchronization round.

The paper is clearly written and experiments are well set up.

In a more general view, I'd encourage the authors to discuss in the paper that momentum can be applied both in the inner and the outer loop. For example both [K19] and [Assran et al. (2019)] have already applied momentum in a similar setting but focusing on the inner level (which would be harder to analyze in theory, but might be a good method in practice).

The provided theoretical convergence result is valuable as a contribution, even if it does not show benefits of momentum over SGD, but at least it shows a slowdown which is bounded, as in single-machine SGD. The authors have done non-trivial work to extend it to the case of communication and local updates, but I didn't have time to check the entire long proof in the appendix.

The experimental results are convincing in comparison to the baselines without momentum.
I would have hoped the authors also add a more clear picture of how the proposed 'outer' momentum would compare to practical 'inner' momentum schemes such as used by [Assran et al. (2019)] and [K19].

The main question for me is on the significance of the contribution. The benefits of momentum are well known in practice in the single-machine case, and theoretically not well understood. The paper here translates this type of results also to the decentralized case. As momentum is common it is not very surprising that it is also beneficial here. The paper could be strengthened by adding comparisons of different inner/outer momentum variants which are unique to the distributed setting. 

Minor comments:
- At the end of Section 3, it is said that "Local SGD, ... perform averaging on the model parameters rather than on gradients.".
While not totally wrong, I think one should say that local SGD can easily do averaging of the model changes/deltas, instead of the models themselves.

- Start of Page 3 and of Section 5: potentially clarify notion of 'descent direction', as with SGD those are not technically descent on the original (local or global) objective, but only on the currently sampled stochastic f_i .

UPDATE AFTER REBUTTAL
The discussion and other reviews were helpful, for instance that the paper should still clarify better the fact that the proposed method is just a very minor generalization of BMUF, to some more SGD variants (including decentralized). Nevertheless, reviewers seem to agree that the paper is well-presented in giving a more clear review of such methods, and valuable in improving the theoretical understanding of such distributed momentum methods. I keep the current score.

References:
[K19] Koloskova, Anastasia, et al. "Decentralized Deep Learning with Arbitrary Communication Compression." arXiv preprint arXiv:1907.09356 (2019).
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Paper summary: 
The paper proposes a general framework to improve the optimization and generalization performance of several communication efficient algorithms, including local SGD, SGP. A convergence analysis is provided for smooth non-convex losses. 

Score: Accept. 

Detailed comments: 
Pros: 
* The paper is written in a clear and well-organized form. The experimental setup description, as well as the ablation study, provide a clear guideline to use this framework in practice. 
* The extensive empirical experiments in this paper justify the effectiveness of the proposed methods. 

Cons: 
* Providing the convergence analysis is encouraged but more understanding is required. The analysis is quite standard and the convergence rate with extra effect (increasing the upper bound) in Eq.5 cannot explain why it convergences faster and generalizes better (e.g. in Figure 2) than AR-SGD. 
* As one main contribution of this paper is in terms of the theoretical convergence guarantees, the related work should precisely mention the recent progress in this area and (maybe) point out the difference compared to the prior work. 

Minor comments: 
1. It is confusing to talk about the \tau in the main paper and the same \tau notation in Algorithm 3 (default parameter in OSGP). Are these two factors the same?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors verify the effect of BMUF[1], which is called slow momentum in this paper, on computer vision and natural language processing tasks with different kinds of local optimizers.  They also provided the theoretical convergence guarantee of BMUF.

The literature survey of this paper is quite good and the experimental results are convincing. However, they should modify their claim that "BMUF is a special case of SlowMomentum".
In classical block momentum version of BMUF, the  update formula is:
u_{t+1} = \beta u_{t} + \alpha (x_{t,0}-x_{t,\tau})
x_{t+1,0} = x_{t,0} - u_{t+1}
\beta is called block momentum and \alpha is block learning rate

in this paper, the update formula becomes:
u_{t+1} = \beta u_{t} +  (x_{t,0}-x_{t,\tau})/\gamma_{t}
x_{t+1,0} = x_{t,0} - \alpha\gamma_{t}u_{t+1}

Obviously this two formula are equivalent. BMUF is a general framework, which can work with different kinds of local optimizer. The author should not narrow down the definition of BMUF as BMUF with SGD as local optimizer and \alpha=1. Actually, \alpha=1 is used in all experiments of this paper.

In conclusion, the organization and writing of this paper is satisfactory, the experiments and theoretical proof is valuable for respective researchers. They should clarify that SlowMomentum is same with BMUF with classical block momentum. I will give a weak accept to this paper.

 [1] Kai Chen and Qiang Huo. Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5880–5884, 2016.
review_title: Official Blind Review #2

Meta-review: This paper presents a new approach, SlowMo, to improve communication-efficient distribution training with SGD. The main method is based on the BMUF approach and relies on workers to periodically synchronize and perform a momentum update. This works well in practice as shown in the empirical results. 

Reviewers had a couple of concerns regarding the significance of the contributions. After the rebuttal period some of their doubts were clarified. Even though they find that the solutions of the paper are an incremental extension of existing work, they believe this is a useful extension. For this reason, I recommend to accept this paper.
Decision: Accept (Poster)