Venue: NeurIPS 2020 Workshop DL-IG

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper is about a proposed modification of the Adam optimization which is claimed to have fast convergence, comparable generalization of the trained models to those obtained using SGD, and good training stability. The basic idea behind this improvement is to adapt the stepsize according to the so-called "belief" in the current gradient direction (i.e., if the gradients largely deviate from the exponential moving average, then take small steps, otherwise continue taking larger steps). The method is shown to work efficiently in multiple benchmark problems. While I recommend the authors to test the method on wider types of landscapes (e.g., benchmark optimization problems in 2 and 3 dimensions), I recommend to accept the paper for the workshop.
review_title: Review on "AdaBelief Optimizer: Adapting Stepsizes by theBelief in Observed Gradients"

final_score: rating: 7: Good paper, accept
review_content: This paper presents an algorithm for training deep networks called AdaBelief. The algorithm is a modification of Adam and changes the how the run-time average of the second moment of the gradient is calculated. The motivation for this modification comes through some simple yet revealing examples in Section 2.2; AdaBelief can be heuristically connected to second-order updates like those in Newton’s method and while Adam is essentially sign-SGD for low-variance gradients, AdaBelief considers both the sign and the magnitude of the gradient.

Empirical  results are provided on a large number of tasks from classification, language modeling and generative modeling. The results on ImageNet are impressive.
review_title: Review

Decision: Accept (Poster)