On the Convergence of Adam and Beyond

Abstract:  Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with ``long-term memory'' of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.

Reviews and author responses:

The paper presents three contributions: 1) it shows that the proof of convergence Adam is wrong; 2) it presents adversarial and stochastic examples on which Adam converges to the worst possible solution (i.e. there is no hope to just fix Adam's proof); 3) it proposes a variant of Adam called AMSGrad that fixes the problems in the original proof and seems to have good empirical properties.

The contribution of this paper is very relevant to ICLR and, as far as I know, novel.
The result is clearly very important for the deep learning community.
I also checked most of the proofs and they look correct to me: The arguments are quite standard, even if the proofs are very long.

One note on the generality of the results: the papers states that some of the results could apply to RMSProp too. However, it has been proved that RMSProp with a certain settings of its parameters is nothing else than AdaGrad (see Section 4 in  Mukkamala and Hein, ICML'17). Hence, at least for a certain setting of its parameters, RMSProp will converge. Of course, the proof in the ICML paper could be wrong, I did not check that...

A general note on the learning rate: The fact that most of these algorithms are used with a fixed learning rate while the analysis assume a decaying learning rate should hint to the fact that we are not using the right analysis. Indeed, all these variants of AdaGrad did not really improve the AdaGrad's regret bound. In this view, none of these algorithms contributed in any meaningful way to our understanding of the optimization of deep networks *nor* they advanced in any way the state-of-the-art for optimizing convex Lipschitz functions.
On the other hand, analysis of SGD-like algorithms with constant step sizes are known. See, for example, Zhang, ICML'04 where linear convergence is proved in a neighbourhood of the optimal solution for strongly convex problems.
So, even if I understand this is not the main objective of this paper, it would be nice to see a discussion on this point and the limitations of regret analysis to analyse SGD algorithms.

Overall, I strongly suggest to accept this paper.


Suggestions/minor things:
- To facilitate the reader, I would state from the beginning what are the common settings of beta_1 and beta_2 in Adam. This makes easier to see that, for example, the condition of Theorem 2 is verified.
- \hat{v}_{0} is undefined in Algorithm 2.
- The graphs in figure 2 would gain in readability if the setting of each one of them would be added as their titles.
- McMahan and Streeter (2010) is missing the title. (Also, kudos for citing both the independent works on AdaGrad)
- page 11, last equation, 2C-4=2C-4. Same on page 13.
- Lemma 4 contains x_1,x_2,z_1, and z_2: are x_1 and z_1 the same? also x_2 and z_2?

This work identifies a mistake in the existing proof of convergence of
Adam, which is among the most popular optimization methods in deep
learning. Moreover, it gives a simple 1-dimensional counterexample with
linear losses on which Adam does not converge. The same issue also
affects RMSprop, which may be viewed as a special case of Adam without
momentum. The problem with Adam is that the "learning rate" matrices
V_t^{1/2}/alpha_t are not monotonically decreasing. A new method, called
AMSGrad is therefore proposed, which modifies Adam by forcing these
matrices to be decreasing. It is then shown that AMSGrad does satisfy
essentially the same convergence bound as the one previously claimed for
Adam. Experiments and simulations are provided that support the
theoretical analysis.

Apart from some issues with the technical presentation (see below), the
paper is well-written.

Given the popularity of Adam, I consider this paper to make a very
interesting observation. I further believe all issues with the technical
presentation can be readily addressed.



Issues with Technical Presentation:

- All theorems should explicitly state the conditions they require
  instead of referring to "all the conditions in (Kingma & Ba, 2015)".
- Theorem 2 is a repetition of Theorem 1 (except for additional
  conditions).
- The proof of Theorem 3 assumes there are no projections, so this
  should be stated as part of its conditions. (The claim in footnote 2
  that they can be handled seems highly plausible, but you should be up
  front about the limitations of your results.)
- The regret bound Theorem 4 establishes convergence of the optimization
  method, so it plays the role of a sanity check. However, it is
  strictly worse than the regret bound O(sqrt{T}) for online gradient
  descent [Zinkevich,2003], so it cannot explain why the proposed
  AMSgrad method might be adaptive. (The method may indeed be adaptive
  in some sense; I am just saying the *bound* does not express that.
  This is also not a criticism of the current paper; the same remark
  also applies to the previously claimed regret bound for Adam.)
- The discussion following Corollary 1 suggests that sum_i
  hat{v}_{T,i}^{1/2} might be much smaller than d G_infty. This is true,
  but we should always expect it to be at least a constant, because
  hat{v}_{t,i} is monotonically increasing by definition of the
  algorithm, so the bound does not get better than O(sqrt(T)).
  It is also suggested that sum_i ||g_{1:T,i}|| = sqrt{sum_{t=1}^T
  g_{t,i}^2} might be much smaller than dG_infty, but this is very
  unlikely, because this term will typically grow like O(sqrt{T}),
  unless the data are extremely sparse, so we should at least expect
  some dependence on T.
- In the proof of Theorem 1, the initial point is taken to be x_1 = 1,
  which is perfectly fine, but it is not "without loss of generality",
  as claimed. This should be stated in the statement of the Theorem.
- The proof of Theorem 6 in appendix B only covers epsilon=1. If it is
  "easy to show" that the same construction also works for other
  epsilon, as claimed, then please provide the proof for general
  epsilon.


Other remarks:

- Theoretically, nonconvergence of Adam seems a severe problem. Can you
  speculate on why this issue has not prevented its widespread adoption?
  Which factors might mitigate the issue in practice?
- Please define g_t \circ g_t and g_{1:T,i}
- I would recommend sticking with standard linear algebra notation for
  the sqrt and the inverse of a matrix and simply using A^{-1} and
  A^{1/2} instead of 1/A and sqrt{A}.
- In theorems 1,2,3, I would recommend stating the dimension (d=1) of
  your counterexamples, which makes them very nice!

Minor issues:

- Check accent on Nicol\`o Cesa-Bianchi in bibliography.
- Near the end of the proof of Theorem 6: I believe you mean Adam
  suffers a "regret" instead of a "loss" of at least 2C-4.
  Also 2C-4=2C-4 is trivial in the second but last display.


This paper examines the very popular and useful ADAM optimization algorithm, and locates a mistake in its proof of convergence (for convex problems). Not only that, the authors also show a specific toy convex problem on which ADAM fails to converge. Once the problem was identified to be the decrease in v_t (and increase in learning rate), they modified the algorithm to solve that problem. They then show the modified algorithm does indeed converge and show some experimental results comparing it to ADAM.

The paper is well written, interesting  and very important given the popularity of ADAM. 

Remarks:
- The fact that your algorithm cannot increase the learning rate seems like a possible problem in practice. A large gradient at the first steps due to bad initialization can slow the rest of training. The experimental part is limited, as you state "preliminary", which is a unfortunate for a work with possibly an important practical implication. Considering how easy it is to run experiments with standard networks using open-source software, this can easily improve the paper. That being said, I understand that the focus of this work is theoretical and well deserves to be accepted based on the theoretical work.

- On page 14 the fourth inequality not is clear to me.

- On page 6 you talk about an alternative algorithm using smoothed gradients which you do not mention anywhere else and this isn't that clear (more then one way to smooth). A simple pseudo-code in the appendix would be welcome.

Minor remarks:
- After the proof of theorem 1 you jump to the proof of theorem 6 (which isn't in the paper) and then continue with theorem 2. It is a bit confusing.
- Page 16 at the bottom v_t= ... sum beta^{t-1-i}g_i should be g_i^2
- Page 19 second line, you switch between j&t and it is confusing. Better notation would help.
- The cifarnet uses LRN layer that isn't used anymore.

We thank the reviewer for very helpful and constructive feedback. 

About Mukkamala and Hein 2017 [MH17]: Thanks for pointing this paper. As the anonymous reviewer rightly points out, the [MH17] does not look at the standard version of RMSProp but rather a modification and thus, there is no contradiction with our paper. We will make this point clear in the final version of the paper.

Regarding note about learning rate: While it is true that none of these new rates improve upon Adagrad rates, in fact, in the worst case one cannot improve the regret of standard online gradient descent in general convex setting. Adagrad improves this in the special case of sparse gradients (see for instance, Section 1.3 of Duchi et al. 2011). However, these algorithms, which are designed for specific convex settings, appear to perform reasonably well in the nonconvex settings too (especially in deep networks). Exponential moving average (EMA) variants seem to further improve the performance in the (dense) nonconvex setting. Understanding the cause for good performance in nonconvex settings is an interesting open problem. Our aim was to take an initial step to develop more principled EMA approaches. We will add a description in the final version of the paper.

Lemma 4: Thanks for pointing it out and sorry for the confusion. Indeed, x1 = z1 and x2 = z2. We have corrected this typo.

We have also revised the paper to address the minor typos mentioned in the review.

We deeply appreciate the reviewer for a thorough and constructive feedback. 

- Theorem 2 & 3 are much more involved and hence the aim of Theorem 1 was to provide a simplified counter-example for a restrictive setting, thereby providing the key ideas of the paper.
- We will emphasize your point about projections in the final version of the paper.
- We agree that the role of Theorem 4 right now is to provide a sanity check. Indeed, it is not possible to improve upon the of online gradient descent in the worst case convex settings. Algorithms such as Adagrad exploit structure in the problem such as sparsity to provide improved regret bounds. Theorem 4 provides some adaptivity to sparsity of gradients (but note that these are upper bounds and it is not clear if they are tight). Adaptive methods seem to perform well in few non-sparse and nonconvex settings too. It remains open to understand it in the nonconvex settings of our interest. 
- Indeed, there is a typo; we expect ||g{1:T,i}|| to grow like sqrt(T). The main benefit in adaptive methods comes in terms of sparsity (and dimension dependence). For example see Section 1.3 in Duchi et al. 2011). We have revised the paper to incorporate these changes.
- We can indeed assume that x_1 = 1 (without loss of generality) because for any choice of initial point, we can always translate the function so that x_1 = 1 is the initial point in the new coordinate system. We will add a discussion about this in the final version of the paper.
- The last part of Theorem 6 explains the reduction with respect to general epsilon. We will further highlight this in the final version of the paper.

Other remarks:

Regarding widespread adoption of Adam: It is possible that in certain applications the issues we raised in this work are not that severe (although they can still lead to degradation in generalization performance). On the contrary, there exist a large number of real-world applications, for instance training models with large output spaces, which suffer from the issues we have highlighted and non-convergence has been observed to occur more frequently. Often, this non-convergence is attributed to nonconvexity but our paper shows one of the causes that applies even to convex settings. 
As stated in the paper, using a problem specific large beta2 seems to help in some applications. Researchers have developed many tricks (such as gradient clipping) which might also play a role in mitigating these issues. We propose two different approaches to fix this issue and it will be interesting to investigate these approaches in various applications.

We have addressed all other minor concerns directly in the revision of the paper.


Thanks David, for your interest in this paper and helpful comments (and pointers). We have addressed your concerns regarding typos in the latest revision of the paper.


Thanks for your interest in our paper and for your feedback. We believe that beta1 is not an issue for convergence of Adam (although our theoretical analysis assumes a decreasing beta1). For example, in stochastic convex optimization, momentum based methods have been shown to converge even for constant beta1. That said, it is indeed interesting to develop better understanding of the effect of momentum in convergence of these algorithms (especially in the nonconvex setting).

As the paper shows, for any constant beta2, there exists a counter-example for non-convergence of Adam (both in online as well as stochastic setting, Theorem 2 & Theorem 3). Using a large beta2 can partially mitigate this issue in practice but it is not clear how high beta2 should be and it is indeed an interesting research question. Our paper proposes a couple of approaches (AMSGrad & AdamNC) for addressing these issues. AMSGrad allows us to use a fixed beta2 by changing the structure of the algorithm (and also allows us to use a much slow decaying learning rate than Adagrad). AdamNC looks at an approach where beta2 changes with t, ultimately converging to 1, hopefully allowing us to retain the benefits of Adam but at the same time circumventing its non-convergence.

The aim of synthetic experiments was to demonstrate the effect of non-convergence. We can modify it to demonstrate similar problem for any constant beta2.


(1) Thanks for the interest in our paper and looking into the analysis carefully. We believe there is a misunderstanding regarding the proof. The third inequality follows from the lower bound v_{t+i-1} \ge (1-\beta)\beta^{i-1}_2 C^2. The fourth inequality actually follows from the upper bound on v_{t+i-1}  (which implicitly uses \beta^{i'-1}_2 C^2 \le 1). We revised the paper to provide the detailed derivation, including specifying precise constants that were previously omitted.

(2) Actually, an easy observation from our analysis is that we can bound the regret of AMSGrad by O(G_infty sqrt(T)) as well. This can be easily seen from the proof of Lemma 2 where in the analysis the term \sum_{t=1}^T |g_{t,i}/\sqrt{t} can be bounded by O(G_infty sqrt(T)) instead of O(\sqrt{\log(T) ||g_{1:T}||_2). Thus, the regret of AMSGrad is upper bounded by minimum of O(G_infty sqrt(T)) and the bound presented in Theorem 4, and thus the worst case dependence on T is \sqrt{T} rather than \sqrt{T \log(T)}. We will make this point in the final version of the paper.

We thank the reviewer for the helpful and supportive feedback. The focus of the paper is to provide a principled understanding for the exponential moving average (EMA) adaptive optimization methods, which are now used as building blocks of many modern deep learning applications. The counter-example for non-convergence we show is very natural and is observed to arise in extremely sparse real-world problems (e.g., pertaining to problems with large output spaces). We provided two general directions to address the convergence issues in these algorithms (by either changing the structure of the algorithm or by gradually increasing beta2 as algorithm proceeds). We have provided preliminary experiments on a few commonly used networks & datasets but we do agree that a thorough empirical study will be very useful and is part of our future plan. 

- Fourth inequality on Page 14: We revised the paper to explain it further.
- We will be happy to elaborate our comment about smoothed gradients in the final version of the paper.
- We also addressed other minor suggestions.


Dear authors,

It's a very good paper, but I have some questions as follows:

(1) In the last paragraph on Page 14, it says the fourth inequality is from $\beta^{i'-1}_2 C^2 \le 1$, but I couldn't go through from the third inequality to the fourth inequality on Page 14. It seems that you applied the lower bound of $v_{t+i-1}$ (i.e. $v_{t+i-1} \ge (1-\beta)\beta^{i-1}_2 C^2$ which is not desired) instead of its upper bound (which is truly required)? 

(2) In Corollary 1, from my understanding, the L2 norm of $g_{1:T,1}$ should be upper bounded by $\sqrt(T)G_{\inf}$, so the regret be $O(\sqrt(T \logT))$ instead of $O(\sqrt(T))$ as stated in the remark of Corollary 1. 

Correct me if I'm wrong. Thanks!

Thanks for the inspiring paper. The observations are interesting and important!

It is easy to capture that exponential moving average might not able to capture the long-term memory of the gradients. 

The paper is mainly focused on the beta2 that involving the averaging of second moment. It makes me wonder whether the beta1 on the averaging of the first moment gradient also suffer the similar problem. 

It seems a direct solution would be using a large beta1 and beta2. (Always keep the maximum of the entire history seems is not the best solution and an average over a recent history might be a better alternative.) 

I did not carefully check the detail of the paper. But generally, one would have a similar concern I think. Could you explain the benefits of the proposed algorithm? 

The synthetic experiments seem to use a relatively insufficient large beta2 regarding the large gradient gap, which makes it not able to capture the necessary long-term dependency. 

The RMSProp used in Section 4 in Mukkamala and Hein, ICML'17 is not the standard RMSProp but a modification in which the parameter used for computing the geometrical averages of the gradient entries squared changes with time. So there is no contradiction with this paper, that shows counterexamples for the standard algorithm in which that parameter is constant. 


Congratulations for this paper, I really enjoyed it. It is a well written paper that contains an exhaustive set of counterexamples. I had also noticed that the proof of Adam was wrong and included it in my Master Thesis  (https://damaru2.github.io/convergence_analysis_hypergradient_descent/dissertation_hypergradients.pdf Section 2.4) and I enjoyed reading through the paper and finding that indeed it was not just that the proof was wrong but that the method does not converge in general, not even in the stochastic case.

I noticed some typos / minor things that seem that need to be fixed:

+ In the penultimate line of page 16 there is this equality v_{t-1} = .... g_i. This g_i should be squared.

+ In the following line, there is another square missing in a C, it should be (1-\beta_{t-1}_2)(C^2 p + (1-p)) and there is a pair of parenthesis missing in the next term, it should be  (1-\beta_2^{t-1})((1+\delta)C-\delta)

+ The fact that in Theorems 2 and 3 \beta_2 is allowed to be 1 is confusing, since the method is not well defined if \beta_2 is 1 (and you don't use an \epsilon in the denominator. If you use an \epsilon then with \beta_1 = 0 the method is equivalent to SGD so it converges for a choice of alpha). In particular, in the proof of theorem 3 \sqrt{1-\beta_2}  appears in some denominators and so does \sqrt{\beta_2} but there is no comment about what happens when this quantities are 0. There should be a quick comment on this or the \beta_2 \leq 1 should be removed from the theorems.

Best wishes


We thank you for your interest in our paper and for pointing out this missing detail. We use a decrease step size of alpha/sqrt(t) (as suggested by our theoretical analysis) for the stochastic optimization experiment. The use of decreasing step size leads to a more stable convergence to the optimal solution (especially in scenarios where the variance is reasonably high). We did not use epsilon in this particular experiment since the gradients are reasonably large (in other words, using a small epsilon like 1e-8 should produce more or less identical results). We will add these details in the next revision of our paper.

Hello,

I tried implementing AMSGrad (here: https://colab.research.google.com/notebook#fileId=1xXFAuHM2Ae-OmF5M8Cn9ypGCa_HHBgfG) for the experiment on the stochastic optimization setting and obtain that x_t approaches -1 faster that on the paper but convergence seems less stable, so I was wondering about the specific values for other hyperparameters like the learning rate and epsilon which weren't mentioned, in my case I chose a learning of 1e-3 and an epsilon of 1e-8 which seems to be the standard value on most frameworks.

Meta-review: This paper analyzes a problem with the convergence of Adam, and presents a solution. It identifies an error in the convergence proof of Adam (which also applies to related methods such as RMSProp) and gives a simple example where it fails to converge. The paper then repairs the algorithm in a way that guarantees convergence without introducing much computational or memory overhead. There ought to be a lot of interest in this paper: Adam is a widely used algorithm, but sometimes underperforms SGD on certain problems, and this could be part of the explanation. The fix is both principled and practical. Overall, this is a strong paper, and I recommend acceptance.


Decision: accepted-oral-papers