Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new algorithm which brings closer SGD and Adam while also incorporating new corrections to improve behavior of the optimizer in contexts where there is very small or very large eigen values.

Decision

I vote for weak rejection because the core modification proposed to Adam is minor and is mostly supported by intuition and preliminary experiments.

Justification

There is many modifications proposed, but most are secondary corrections for stability, such as the warm-up schedule with the redefinition of beta_2. These modifications could as well be incorporated in Adam without the core modification that is the smoothing presented in section 3. These additional modifications also make it difficult to measure the importance of the core contribution. Without getting rid of them, an ablation study on toy problems (even synthetic data) would be necessary for a better understanding.

In section 3.1, the temporal definition of beta_2t is integrating a warm-up. While the reason for doing so is supported in introduction of section 3, the effect of this modification should be weighted against no warm-up, and also compared with its effect on Adam.

There is an error in algorithm 1. The last element of the last line (Perform the update) should be \alpha (1 - \eta) m_t/d_t. The code in Appendix corroborates this correction. Minor related note, the use of d_t to define the denominator of what d_i represents in section 3 is very confusing. I would suggest to use the ratio notation of d_i from the equations in the algorithm for coherence.

If we get pass the warm-up scheduling, by massaging the equation we get that the algorithm is different from Adam on 2 points, 1) the bias are not corrected and 2) the denominator sqrt(v) + epsilon is replaced by sqrt(v) + sqrt(mean(v) + epsilon^2). I have difficulty convincing myself that smoothing by the average is solving the issues raised in the paper and there is no experiments to study its effect directly.

The experiments are on 3 datasets, but only the computer vision ones are run on multiple architectures. Caption of figure 1 explains that each model is trained 3 times, but the source of variation between each run is not described. Are the models initialized differently? In any case, there is an overlap for 3 of the 5 models between SGD and SoftAdam which makes the comparison rather unconvincing. There is no standard deviation for Adam, and none on Penn Treebank dataset and IWSLT. For a better comparison, all hyper-parameters of the algorithms should be optimized for each run. I understand that SoftAdam is meant to be close to both SGD and Adam, but using the same hyper-parameters may induces misleading results by favoring some (model, optimizer) combination nevertheless.

Minor comments

In section 2, second paragraph, the term 'mini-batch' should be used instead of 'batch'.
In section 2, last sentence, the betas should have no t.
In section 2.1, fourth equation (unnumbered), the eigen vector xi_i is presented as a vector and then used as a scalar. Notation should be uniformed.
In Section 2.1 around equation (2), the use of i and j is incoherent.
In Section 3:
- Overall, this understanding *of* has
- we consider the *an* update
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary:

This work proposed a new algorithm called softAdam to unify the advantages of both Adam and SGD. The authors also showed experiments to backup their theoretical results. 

Pros:

The authors provided analysis of different algorithms including Adam and SGD on simple quadratic function, then proposed a new algorithm called softAdam which outperforms both Adam and SGD. Experiment results backup their theory. 

Cons:

- The novelty of this work is limited. The main contribution of this work is to provide a new adaptive gradient method called softAdam, which changes the update rules for some parameters including \beta. However, neither intuition or theoretical guarantees are provided in this paper. I recommend the authors to add some explanation about why softAdam outperforms other existing algorithms. Besides, the difference between softAdam and original Adam method is little. 
- The theoretical analysis about existing adaptive methods provides nothing new. The authors showed some analysis on quadratic model, which is a very simple model and hence can not reflect the true model we face in the practice. I suggest the authors provide some analysis on more general model, including convex functions and non-convex functions. 
- The settings of experiments are limited. The authors should at least compare softAdam with other baseline algorithms on some modern deep learning tasks including Imagenet.  

Minor comments:

- Page 4, section 3, 'this understanding of has'... lacks object.
- This paper lacks some references in this area. 

J. Chen and Q. Gu. Closing the generalization gap of adaptive gradient methods in training
deep neural networks. arXiv preprint arXiv:1806.06763, 2018.
Ward, R., Wu, X. and Bottou, L. (2018). Adagrad stepsizes: Sharp convergence over nonconvex
landscapes, from any initialization. arXiv preprint arXiv:1806.01811 .
Li, X. and Orabona, F. (2018). On the convergence of stochastic gradient descent with adaptive
stepsizes. arXiv preprint arXiv:1805.08114 .


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper tends to explain how the tradeoffs between convergence speed and convergence performance are made by different optimization methods. Moreover, the paper modifies Adam-like updating rules and proposes a novel optimization methods, SoftAdam. Finally, the paper performs numerical experiments on traditional image classification tasks as well as language modeling tasks.

Pros
The paper involves the language modeling tasks in empirical results besides traditional image classification tasks, which helps to explain the convergence performance of optimization methods in a wider range of applications.

Cons
1. The writing of this paper is not well organised. Section 1 lacks detailed description of the main idea and the proposed optimization methods, which actually confuses the reader. Section 2 describes too much details of SGD and Adam, and lacks a clear "intuition" which readers exactly expect.
2. The notation in the paper is little confusing. In the update rule of z_{t+1} in Section 2, what is meaning of z? In Section 3, n_t and n_\infty are used before a proper defination, and what is relationship between \alpha and \alpha_t in the implementation of the SoftAdam?
3. The motivation of the proposed method is weak. Such a weak motivation is mainly because of the insufficient "intuition" in Section 2. The author mentions "the Hessians in deep learning problems are not diagonal", but does not provide further explanation on why more importance should be lay on serving both max and min eigenvalues.
4. There are also several minor problems on the numerical results. Firstly, why the colors of "softAdam" and "sgd" are switched several times in Figure 1? Secondly, the figural result in Figure 1 and he numerical results on language processing models both lack a display of the confidence range.
     
review_title: Official Blind Review #1

Meta-review: The reviewers all agreed that the proposed modification was minor. I encourage the authors to pursue in this direction, as they mentioned in their rebuttal, before resubmitting to another conference.
Decision: Reject