Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The blog post is structured as follows: 
(1) Motivation role of mini-batch size in training deep neural networks by contextualizing the pros and cons SGD (close to small batch learning) vs GD (close to large batch learning)
(2) Review of related work, especially, methods that try to achieve the best of both small batch and large batch learning
(3) Summary of the SWAP approach in the ICLR Paper: https://arxiv.org/pdf/2001.02312.pdf

In part (3), the blog post pretty much summarizes the main parts of the paper without significantly changing the language or content. The end of the post has two additional bullet points on extending the scope of the paper, but these points are not expanded further and I am not sure what exactly they mean. 

Thus, the main contribution of the blog post is in (1) and (2). Here however, the treatment of the subject is not very crisp and precise. 
 
- In (1) there have been lot of work and discussions on the subject of SGD vs GD. I found the treatment in the blog post to be needlessly verbose at the same time imprecise. 
(a) To begin with, imo, the table in the middle is not the best way to represent the pro's and cons. Firsly, mini-batch SGD is not precisely defined without specification of batchsize and in particular its pros and cons will depend on batch size. Rather, we can think of as MB-SGD as the umbrella algorithm of which GD and 1-SGD are extreme special cases. Secondly, some of the pro's and cons of SGD vs GD as stated are not always true. e.g., "May get stuck in local minima" is stated as a con for "GD", while that is the problem with non-convexity in general and also holds for 1-SGD. Similarly, faster wall clock time of SGD, while true in many cases, is not universally true. (b) More broadly, when comparing GD vs -1-SGD, there are some tradeoffs mainly computational in terms of parallelizability, cost/iteration which are universal and well understood; and other tradeoffs that are observed in practice but not well understood -- this includes low wall clock time of SGD convergence, better generalization under SGD training. These distinctions do not come out very clearly from the blog post. In fact, I felt the the first two paragraphs of the original paper provides a cleaner comparison. 
(c) In the second paragraph of blog post, it is claimed that (S)GD is inherently motivated by the non-convexity of optimization objective. From the references provided therein, it is not clear to me why this is true (even if we restrict to DNN training objective) or if it is even widely accepted. 
(d) On the topic of sharp vs flat minima, the blog post mainly summarizes the argument in seminal work or Keskar et al. [4], while there have been a lot of follow up work, both challenging and improving upon the hypothesis by Keskar et al. which make more nuanced arguments on what helps and hurts (see Dinh et al., 2017,Li et al., 2018,McCandlish et al., 2018,Hoffer et al., 2017 in the original ICLR paper) --- the blog post misses some of the essential nuances or even references to them which could be misleading. 

- In (2) summarizing related work: The blog post covers many related work which is good, but the closest work are SWA and local-SGD, which together have the main ideas of the original ICLR paper. In fact, based on the original paper, SWA and SWAP are very similar in spirit and also have similar performance. However, SWA is just stated as yet another method for local update and local-SGD is not even mentioned. The details on the similarities and differences, which is important for contextualizing the paper are not highlighted in the blog post. I also believe the Fig 3 is not quite accurate. This should in fact be a 2D plot as the more communication axis is not always the same as less local computation. 

Overall, I believe the blog post is too verbose without adding significant new insight or content to the original ICLR paper. 
review_title: The blog post is too verbose and does not add significantly to the readability or content of the original ICLR paper 

Decision: Reject