Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper is an empirical contribution regarding SGD arguing that it presents two different behaviors which the authors name a noise dominated regimen, and a curvature dominated regime. They observe that the behaviors seem to arise in different batch sizes 

The authors derive empirical conclusions and perform experiments in different settings. The paper is well-written and the experimental setup seems to be carefully carried out. 

I find the observations interesting, but the contribution is empirical and not entirely new. It would be nice if there were some theoretical results to back up the observations. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the properties of SGD as a function of batch size and learning rate. Authors argue that SGD has two regimes:  a noise dominated regime (small batch size) and curvature dominated regime (large batch size). Authors conduct through numerical experiments highlighting how learning rate changes as a function of batch size (initially linear growth and then saturates). The critical contribution of this work appears to be the observation that large batch size can be worse than small under same number of steps demonstrating implicit regularization of small batch size.

The two regime claim of the paper is not really novel. These regimes are fairly well covered by previous works (e.g. Belkin et al as well as others). When it comes to experiments, constant epoch budget is also fairly well understood and the behavior in Figure 1 is not really surprising (as the eventual training performance gets worse with large batches).

The interesting part in my opinion is the experiments on constant steps. Authors verify large batch size reduces test accuracy while improving train. I believe these experiments are novel and the results are interesting. Besides CIFAR 10, authors test this hypothesis in two other datasets while tuning the learning rate. On the other hand, contribution is somewhat incremental given observations made by related literature (Keskar et al and others).

Some remarks:
1) In Table 1, batch size 16k has effective LR of 32. However in Figure 1c SGD with momentum at batch size 8k uses an effective LR of 4. Can you explain this inconsistency i.e. why is there such a huge jump from 4 to 32 (in reality we expect the effective LR to stay constant in the curvature regime). I also understand that one is constant epoch and other is constant step. However 4 to 32 seems a bit inconsistent.

2) Does momentum help in constant step budget (with sufficiently large steps so that training loss is small)?

3) Readability: Consider explaining what is meant by "warm-up", "epoch budget", "step budget" clearly and upfront.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper attempts to clarify the debate on large-batch neural network training, particularly on the relationship between learning rate, batch sizes and test performance. The authors claim two contributions towards understanding how the hyper-parameters of SGD affect final training and test performance: (1) SGD exhibits two regimes with different behaviours and (2) large-batch training leads to degradation of test performance even with same step budgets.

Overall, the authors did a comprehensive study on large-batch training with the support of extensive experiments. But I'm concerned with the novelty and contributions of this paper. I tend to reject this paper because (1) the first contribution of the paper is not new as it has already been recognized by a few paper that SGD exhibits two different regimes; (2) this paper makes the debate of large-batch training even muddier.

Main argument:
The paper does not do a great job in clarify the debate. Particularly, the authors mixed their observations up with the results of published works, making it hard to identify the contributions of this paper. For example, the two regimes mentioned in the paper has been identified by a few other works and the contribution of this paper is just to verify them again. Also, I find the experiments done in section 3 and 4 are similar to previous works and even the conclusions are similar. The only new observation I'm aware of in these two sections is that the training loss and test accuracy are independent of batch size in the noise dominated regime.

Back to introduction section, the goal of this paper (as claimed in the beginning of second paragraph) is to clarify the debate. But does this paper really achieves this goal? In terms of learning rate scaling, this paper gets similar conclusions as Shallue et al. (2018). In terms of the difference between vanilla SGD and SGD with momentum, Zhang et al. (2019) already argued that the difference depends on specific batch sizes and SGD with momentum only outperforms SGD in the curvature dominated regime. 

I think the authors should instead focus on the discussion of generalization performance and the observation that training loss and test accuracy are independent of batch size in noise dominated regime. To my knowledge, this part is novel and interesting. 

In summary, I'm inclined to reject this paper given the current version. However, I think the paper is still worth reading if the authors can reorganize the paper and I might increase my score if my concerns get resolved.

review_title: Official Blind Review #1

Meta-review: Authors provide an empirical evaluation of batch size and learning rate selection and its effect on training and generalization performance. As the authors and reviewers note, this is an active area of research with many closely related results to the contributions of this paper already existing in the literature. In light of this work, reviewers felt that this paper did not clearly place itself in the appropriate context to make its contributions clear. Following the rebuttal, reviewers minds remained unchanged. 
Decision: Reject