Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In the paper, the authors propose a novel three-stage training strategy for deep learning models, training using large batch, training using small-batch locally and then aggregate models. Experimental results show that the proposed method converges faster than compared methods.  I have the following concerns:

1) In figure 1, It looks like that the local models in worker 1, 2, 3 can reach the same testing accuracy without average. What is the meaning of computing average in this case?

2) In the paper, authors mentioned that “Note that there exist training schemes in the literature that train on even larger batch sizes such as 32k (You et al., 2017; Jia et al., 2018), but these methods require a lot of hyperparameter tuning specific to the dataset.” As far as I know, they just need to tune warmup steps and peak learning rate, which is also required in the paper. In the proposed method, it is required to tune the switch point between phase 1 and phase 2.

3) In the experiment, it also uses warmup for small batch size, is it necessary?

4) Does the large batch training use layer-wise learning rate scaling? From my point of view, it is better to use it in the large-batch training. 

5) A guideline about how to select the switch point between phase 1 and phase 2 should be given if it takes time to tune it. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a 2-stage SGD variant that improves the generalization. The experiments show good performance.
However, there are some weakness in this paper:

1. (Minor issue) The Update() function in Algorithm 1 seems to be something very general. However, it seems that Update() is simply SGD or SGD with (Nesterov) momentum, according to Section 5.1. Furthermore, the authors never explicitly explain what exact Update() function is used, which is very unfriendly to the readers.

2. The major issue is that the proposed algorithm and the contribution (improvement of generalization) is not novel. Phase 2 of Algorithm 2 is called local SGD, proposed in [1,2]. Local SGD also has variants with Polyak momentum and Nesterov momentum [3]. Furthermore, [4] has already proposed an algorithm, post-local SGD, which is basically the same as Algorithm 1 in this paper (run fully synchronous SGD first and then local SGD). Note that [4] also shows that post-local SGD converges to flatter minima, and results in better generalization. Please correct me if I'm wrong, and explain the difference between Algorithm 1 and (post-)local SGD in details.


----------
Reference

[1] Stich, Sebastian U.. “Local SGD Converges Fast and Communicates Little.” ArXiv abs/1805.09767 (2018).
[2] Yu, Hao et al. “Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning.” AAAI (2018).
[3] Yu, Hao et al. “On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization.” ICML (2019).
[4] Lin, Tao et al. “Don't Use Large Mini-Batches, Use Local SGD.” ArXiv abs/1808.07217 (2018).

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a parallel version of the stochastic weight averaging method. It utilizes to phases to train the DNN. The first phase consists of distributed large-batch training, where the learning rate is scaled linearly with respect to the scale of the batch size. The second phase consists of using small batches with SWA to obtain the final model. Experiments verify that this method is able to achieve similar generalization performance as small-batch methods in less training time. A comparison against small-batch SWA, large-batch SWA, etc. 

Strengths:

The proposed algorithm is a natural extension to SWA, and appears to give good generalization performance. It is able to utilize large-batch training effectively, which is perhaps surprising given the amount of tuning necessary in Shallue, et al. (2018) in order to achieve good performance. The experiments are well-detailed, and some interesting visualizations, graphs, and empirical analyses are provided.

Weaknesses:

I think that this paper is fairly complete; the only question is whether or not it contains enough novelty, as it is a natural extension of SWA to the parallelized setting. No theoretical analysis of the algorithm is given.

Some questions and comments:

- How sensitive is the algorithm to the choice of the transition point? How was the transition point tuned?
- How large of a batch size can one use before this approach breaks down or is no longer efficient?
- In Figure 2, LB is shown to obtain worse training error than SGD. What is the reason for this? This seems contrary to theory (assuming one has already converged to a neighborhood of the solution).
- The authors comment in the conclusion that "it is possible that regions with bad and good generalization performance are connected through regions of low training loss". Can one check if this is related to the invariances described in Dinh, et al. (2017)? 
- What is the relationship between SWAP and methods on local SGD?

This paper is missing some classical optimization references on increasing batch sizes, which has been well-studied within that literature:

[1] Byrd, Richard H., et al. "Sample size selection in optimization methods for machine learning." Mathematical programming 134.1 (2012): 127-155.
[2] Bollapragada, Raghu, Richard Byrd, and Jorge Nocedal. "Adaptive sampling strategies for stochastic optimization." SIAM Journal on Optimization 28.4 (2018): 3312-3343.
[3] Bollapragada, Raghu, et al. "A progressive batching L-BFGS method for machine learning." arXiv preprint arXiv:1802.05374 (2018). 
[4] Friedlander, Michael P., and Mark Schmidt. "Hybrid deterministic-stochastic methods for data fitting." SIAM Journal on Scientific Computing 34.3 (2012): A1380-A1405.

Although developing some theory for this algorithm would be beneficial, this paper performs a comprehensive set of experiments and is well-written. For these reasons, I'm inclined to accept the paper.
review_title: Official Blind Review #1

Meta-review: The authors proposed a simple and effective approach to parallel training based on stochastic weight averaging. Moreover, the authors have carefully addressed the reviewer comments in the discussion period, particularly the relation to local SGD, to the satisfaction of reviewers. Local SGD mimics sequential SGD with noise induced by lack of synchronization, whereas SWAP averages multiple samples from a stationary distribution, and synchronizes at the end. Please clarify these points and carefully account for reviewer comments in the final version. Overall, the proposed approach will make an excellent addition to the program, both elegant and practically useful.
Decision: Accept (Poster)