Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper demonstrates empirically that the gradient noises of SGD with ResNet and Adam with Bert are different: one is well-concentrated, while the other one is heavy-tailed. The paper claims that this difference costs the failure of SGD on training Bert. Furthermore, the authors proposes gradient clipped SGD and its adaptive version ACClip. Experiments show that ACClip outperforms Adam on training Bert.

In general, the paper is well-written and has addressed an important practical and theoretical problem of why SGD fails to train Bert and how to fix this problem. The theory appears to be solid. My only concern is how generalizable ACClip is. Experiments show that it outperforms Adam on training Bert. How about the other architectures where Adam is usually applied? Is ACClip competitive to Adam in those applications? What’s the performance of ACClip on DL applications where SGD + momentum works well, such as ResNet on the ImageNet dataset?

What is exactly \delta f(x)? Is this the full batch gradient over all training examples? 

Some typos: 
1.	Page 1: thereby providing a explanation
2.	Page 4: at most af factor of 2 and Adam

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposed a very interesting claim: When training a neural network, if the (stochastic) gradient noise is Gaussian-like, then SGD performs better than Adam; On the other hand if the gradient noise is Heavy tailed, then Adam perform better than SGD.

The paper supported this argument with experiments showing that ResNet50 on ImageNet, the noise is more like Gaussian while BERT on language learning tasks the noise is more heavy-tailed. The paper also gave a theoretical result showing that Adam converges in the regime of heavy-tailed noise. 


The experiment finding is quite surprising to me, since many papers (see e.g. 
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
)  claim that the SGD noise is heavy-tailed for image recognization tasks such as CIFAR-10, CIFAR-100. The referred paper used rigorous statistical testing for the tail-index of the SGD noise, while this paper simply drew some image. 


Moreover, the theoretical result in this paper also worries me quite a bit, since from the bounds it seems that Adam is the dominating algorithm (both in the heavy-tail case and in Gaussian tail case). Moreover, SGD also converges in the heavy-tail noise case (by showing that the norm of x_t is not too large during the training process using martingale-based argument). Hence, the upper bound of the theoretical result is convincing enough to claim that Adam is better than SGD in certain regime. 


After Rebuttal: I have read the authors' responses and acknowledge the sensibility of the statement. I have higher my score: In particular, if the noise is indeed Gaussian as opposite to the "known results", this paper should be accepted. 



review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper gives theoretical and empirical results for a gradient clipping variant of Adam they call ACClip.  While the theoretical analysis is rather  sophisticated and nontrivial, I personally do not believe that analyses of this form are of any value in guiding practice.  But that is a long discussion that is not specific to this paper.  The bottom line is that for me it is mainly the experimental results that matter.

The experimental results are not compelling.  It is now clear that careful hyperparameter search is critical to drawing experimental conclusions about optimizers.  This paper simply states the hyperparameters used with no discussion of hyperparameter search. I strongly believe that any claim about optimizers needs to be backed up by experiments with very careful hyper-parameter optimization.

Postscript:  I have modified this review in response to the authors.  I remain unconvinced that the theory is providing anything more than an intuitive hypothesis that Adam is importance when the variance is large.  Since Adam and RMSprop are explicitly damping variance in the gradients, this intuition is reasonable even before we prove any theorems.  I still believe the theorems do not add really add anything to the intuition and it is the experiments that matter.
review_title: Official Blind Review #1

Meta-review: This paper tries to explain why Adam is better than sgd for training attention model. In specific, it first provides some empirical and theoretical evidence that a heavy-tailed distribution of the noise in stochastic gradients is the cause of SGD's worse performance. Then the authors studied a clipped variant of SGD that circumvents this issue, and revisited Adam through the lens of clipping. Overall, this paper conveys some interesting ideas. On the other hand, the theorems proved in this paper do not provide additional insight besides the intuition and the experiments are weak (hyperparameters are not carefully tuned). So even after author response, it still does not gather sufficient support from the reviewers. This is a borderline paper, and due to a rather limited number of papers the conference can accept, I encourage the authors to improve this paper and resubmit it to future conference.

Decision: Reject