Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new distributed computation technique for SGD training of deep neural networks. The proposed method is a modification of the local SGD which updates models distributed to several workers in a parallel way and synchronize the model parameters at every few epochs. The local SGD shows a nice performance but it is not robust against large mini-batch size. The proposed method is called post-local SGD that starts the local SGD after some epochs of standard mini-batch SGD. This modification makes the local SGD robust against the large mini-batch size. The authors conducted thorough experiments to investigate the performance of the proposed method. The experiments reveal that the proposed method gives better performances than the mini-batch SGD and the vanilla local SGD.

Pros:
- As far as I checked, the numerical experiments are strong. They checked several points of view. Several settings of mini-batch sizes and number of workers are compared. For more detailed points, they compared the proposed method for not only the vanilla SGD but also other optimizers such as momentum SGD. Moreover, different choices of timing of starting local SGD (t') are compared. 
- The proposed method is simple and easy to implement.

Cons:
- The local SGD itself is not new and has been already proposed (indeed, the authors are also explaining this point in the paper), and theoretical investigation of local SGD has also been well exploited. This method is just a combination of normal mini-batch SGD and local SGD. In that sense, there are not a so much eye-opening idea in the proposed method.
- SGLD interpretation is instructive, but it does not explain why the "post"-local SGD outperforms the local SGD. We can intuitively guess why post-local SGD is better but it is not a rigorous theory. 
- The post local SGD algorithm has more freedom than existing method, which results in more tuning parameters. There are parameters such as the batch-size before and after switching, the number of workers, the timing of switching and so on. It could be disadvantage of the proposed method.

Minor comment:
- Please give the precise meaning of the dominant eigenvalue.
- I think the setting that the data are distributed to all workers is acceptable. It is also an interesting setting from the HPC point of view (on the other hand, it would be less interesting in terms of federated learning).
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a variant of local SGD, post-local SGD, for distributed training of deep neural networks. It targets to mitigate the generalization gap caused by large batch training. The idea is straightforward and easy to understand-- start the training with standard mini-batch SGD and later switch to local SGD. The rationale behind this scheme is that switching to local SGD helps the training converge to flatter minima compared to using large-batch SGD, which correlates with sharper minima, and that helps close the generation gap. Switching to local SGD at the second phase also helps improve communication efficiency by reducing the amortized communication volume. The authors perform empirical studies using ResNet and DenseNet to conclude that post-local SGD outperforms large-batch SGD in terms of generalization performance while also with improved communication efficiency. 

Strengths:
+ The post-local SGD technique is simple yet seems to be useful in practice.
+ Provide a thorough evaluation of the communication efficiency and generalization performance of local SGD. 
+ Introduce a hierarchical version of post-local SGD to better adapt to the topology of the GPU cluster, which often consists of heterogeneous interconnects.

Weaknesses:
- The design and experiments are largely empirical without theoretical derivation.
- It is less clear about the benefit of post-local SGD when applied to ADAM, which is widely used for distributed training of NLP tasks. 
- Scalability improvements over mini-batch SGD are largely done by ignoring other optimization techniques that also reduce the communication volume, such as gradient compression[1], Terngrad[2].

Overall, the post-local SGD proposed by the paper seems to be a promising technique for large-scale distributed training. The motivation and explanation of the work are clear. My major concern is about the generalizability of this work. ResNet50 is not that interesting from a distributed training perspective. It is less clear whether the performance gains are consistent across tasks. The authors are encouraged to report experimental results on distributed training of large LM models.

[1]"Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training", Lin et. al., ICLR 2018
[2]"Terngrad: Ternary gradients to reduce communication in distributed deep learning", Wen et. al., NeurIPS 2017
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a variant of local SGD: post-local SGD, which improves the generalization performance compared to large-batch SGD. This paper also empirically studies the trade-off between communication efficiency and performance. Additionally, this paper proposes hierarchical local SGD. The paper is well-written, the experiments show good performance.

However, there are several weakness in this paper:

1. The post-local SGD is a simple extension of local SGD. Roughly speaking, post-local SGD uses fully synchronous SGD to warm up local SGD. The novelty of this algorithm is limited.

2. The main statement that post-local SGD improves the generalization performance, is only supported by empirical results. No theoretical analysis is provided. Thus, the contribution of this paper is limited.

3. In this paper, it is reported that in some experiments, local SGD significantly outperforms mini-batch SGD. As shown in Figure 3, when the number of workers is large enough, mini-batch SGD has extremely bad performance. However, such bad result is potentially caused by a bad choice of learning rates. In this paper, the authors use "linearly scaling the learning rate w.r.t. the global mini-batch size". However, some recent papers also suggest using square root scaling instead of linear scaling [1]. I think the mini-batch SGD fails simply because the learning rates are too large. However, the authors claim that the learning rates for mini-batch SGD are fine-tuned (in Figure 3), which makes the empirical results questionable.


-----------
References

[1] You, Yang, et al. "Large batch optimization for deep learning: Training bert in 76 minutes." arXiv preprint arXiv:1904.00962 (2019).

===================
Update after author's feedback:

I do not have strong reasons to doubt the empirical results after reading the author's feedback, so I increase the score.
review_title: Official Blind Review #1

Meta-review: The authors propose a simple modification of local SGD for parallel training, starting with standard SGD and then switching to local SGD. The resulting method provides good results and makes a practical contribution. Please carefully account for reviewer comments in future revisions.
Decision: Accept (Poster)