Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this work, authors show that the bad performance of Adam is from the large variance of adaptive learning rate at the beginning of the training.
Pros:
1.	Authors demonstrate that the variance of the first few stages is large, which may interpret the degradation in the performance of Adam.
2.	The empirical study supports the claim about the large variance.

Cons:
1.	Theoretically, authors didn’t illustrate why the large variance can result in the bad performance in terms of, e.g., convergence rate, generalization error, etc.
2.	The performance of the proposed algorithm is still worse than SGD and it makes the analysis less attractive.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: I haven't worked in this area before, and my knowledge to the topic of designing optimizers for improving stochastic gradient descent is limited to the level of advanced ML courses at graduate school. Nevertheless, below I try my best to evaluate the technicality of this paper

====================
In this work the authors studied the variance issue of the adaptive learning rate and aim to justify the warm-start heuristic. They also demonstrate that  the convergence issue of many of the stochastic gradient descent algorithms is due to large variance induced by the adaptive learning rate in the early stage of training. To tackle this issue, they proposed a variant of ADAM, which is known as rectified ADAM, whose learning rate not only takes the momentum into the account, but it also adapts to the variance of the previous gradient updates.  

In the first part of the paper, the authors analyzed the variance issue exists in the existing ADAM algorithm, such that with limited samples in the early stage of training, the variance of the adaptive learning rate becomes rather large and it induces high variance to the gradient update to ADAM. In general I found this theoretical justification on the observation of variance issue in ADAM sound, and quite intuitive. In the second part, they proposed the algorithm, namely rectified ADAM, where the difference here to take the second moment  of the gradient into account when updating the adaptive learning rate. They showed that the variance of the adaptive learning rate with such rectification is more numerically stable (especially when variance is intractable in vanilla ADAM), and under some regularity assumption it decreases in the order or O(1/\rho_t).

In extensive numerical studies of supervised learning, the authors showed that RADAM achieves a better accuracy than ADAM (although in Table 1, I am a bit puzzled why the best accuracy is indeed from SGD, if so, what's the point of all adaptive learning rates, is that because SGD requires extensive lr tuning?) Because the superiority in accuracy they also showed that RADAM manages to have more stable training and achieves lower training loss, which is quite interesting. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Authors propose a way to rectify the variance of the adaptive learning rate (RAdam) and apply the optimizer to applications in image classification, language modeling and neural machine translation. The experiments demonstrate not only a strong results over baseline Adam with warmup learning rate but the robustness of the optimizer. The authors additionally demonstrate the theoretical justification behind their optimizer, however I am not very qualified to make the judgement on the theory. Overall judging from the authors description of approach and experimental results, I recommend acceptance.

review_title: Official Blind Review #3

Meta-review: The paper considers an important topic of the warmup in deep learning, and investigates the problem of the adaptive learning rate. While the paper is somewhat borderline, the reviewers agree that it might be useful to present it to the  ICLR community.
Decision: Accept (Poster)