Title: RMSprop converges with proper hyper-parameter
Authors: Naichen Shi
Dawei Li
Mingyi Hong
Ruoyu Sun
Venue: ICLR 2021
Keywords: RMSprop
convergence
hyperparameter

Abstract: Despite the existence of divergence examples, RMSprop remains 
one of the most popular algorithms in machine learning. Towards closing the gap between theory and practice, we prove that RMSprop converges with proper choice of hyper-parameters under certain conditions. More specifically, we prove that when the hyper-parameter $\beta_2$ is close enough to $1$, RMSprop and its random shuffling version converge to a bounded region in general, and to critical points in the interpolation regime. It is worth mentioning that our results do not depend on  ``bounded gradient"  assumption, which is often the key assumption utilized by existing theoretical work for Adam-type adaptive gradient method. Removing this assumption allows us to establish a phase transition from divergence to non-divergence for RMSprop. 

Finally, based on our theory, we conjecture that in practice there is a critical threshold $\sf{\beta_2^*}$, such that RMSprop generates reasonably good results only if $1>\beta_2\ge \sf{\beta_2^*}$. We provide empirical evidence for such a phase transition in our numerical experiments.


Review discussion:
Summary:
The paper studies one of the most popular algorithms in machine learning: RMSprop. More specifically, it investigates the relation between the hyper-parameters and the convergence of the algorithm. By proving the convergence without using "bounded gradient" assumption, the authors establish a phase transition from divergence to non-divergence for RMSProp.
Pros:
1. The paper concerns one of the most important algorithms in machine learning. In my opinion, the problem is practical and of interest in machine learning community.
2. The results of the paper provide explicit conditions for the hyper-parameters of RMSprop/Adam that ensure the convergence of the algorithms. These results provide basic guidelines for tuning hyper-parameters of the algorithms in practice.
Cons:
Apart from the strong points, I still have some concerns about the clarity of the paper. I hope the authors can address my concerns to improve the quality of the paper.
1. The parameter $\beta_2$ is the most important subject of the paper. Until algorithm 1, the paper discusses $\beta_2$ without defining it clearly. It would be more clear if $\beta_2$ is mentioned from the beginning of the paper that it comes form algorithm 1.
2. The authors divide the problems into 2 sub-classes to investigate: realizable and non-realizable, which are not clearly defined. It would be better if the authors can define these 2 sub-classes more formally.
3. The experiments supporting the theoretical results are comprehensible.
However, I would suggest the authors provide a figure with x-axis to be epochs and y-axis to be accuracy so that the readers can have better idea upon how SGD and RMSProp behave during training.
Thank you for your comments! Below we provide the responses to specific comments.
1. Definition of $\beta_2$:
That’s a good suggestion! We now modified the structure as follows: we added an explanation that β2 is from Algorithm 1 in the beginning of the introduction, as suggested by the reviewer.
2. Formal definition of realizable and non-realizable problems
Thanks for the suggestion. We moved the formal definitions from the section on the main result to the end of the introduction section. As stated there, we say a problem is realizable if it satisfies condition (3) with $D_0=0$, and non-realizable if it satisfies condition (3) with $D_0>0$.
3. Figure illustrating performance of algorithms during training
Thank you for the suggestion. We followed the suggestion and now included a figure in Appendix A.1; see Figure A1 in the modified version of the paper.
The paper starts off from the recent realization that there exists divergent examples for any set of hyperparameters for algorithms in the Adam family, such as RMSProp. It sets out to study the effect of the beta2 parameter on convergence for a fixed specific problem. The analysis shows that there exists a beta2 < 1 that leads to convergence for realizable problems, and to convergence to a bounded region of interest for non-realizable problems, without requiring a bounded-gradient assumption. Experiments confirm this new theory.
Overall, the paper is well-written, clear and easy to read. One of its strongest points is how well the analysis and the relevance of the results is motived.
For instance, the importance of removing the assumption on the bounds on the gradient because it effectively removes one of the convergence/divergence regimes is well executed.
There is also significant efforts on providing clear simplified examples from rather complex theorems, which is very appreciated (e.g. Corollary 4.1).
Further, there is a real effort to contrast the results with the previous work, and to explain how it complements them, resolving clearly what initially appears as direct contradictions.
The results are relevant, both from the point of view of the theory, where it adds to a body of work explaining how and why the Adam family of algorithm performs well on modern machine learning taskloads, and from the point of view of the practitioner, outlining what hyperparameter tuning is necessary to achieve convergence. They are also original, in the sense that they provide novel insights, while removing problematic assumptions that permeate most of the related work.
A couple of things could be improved:
- as pointed out in the paper, if beta2 = 1, the algorithm degenerates to SGD.
While there is a remark explaining why as long as beta2 < 1 the two algorithms differ, it would be informative to compare the convergence regimes with high beta2 to SGD directly, to validate that there exists a set of hyperparameters that not only provide convergence, but improved convergence properties compared to SGD (otherwise the results are a lot less relevant), as well as give an order of magnitude of what value is typically necessary for beta2.
- condition (4) in theorem 4.3 is quite difficult to apprehend, with a slightly worrying beta2^n term. More exegesis would be beneficial for reader comprehension.
Overall, this is a nice, well-written and relevant paper that clears the bar for publication in its current version.
Thank you for the detailed and constructive comments! Below we provide the responses to specific comments.
1. RMSProp and SGD
Thanks for the suggestion. Our convergence rate in high $\beta_2$ regime is $O\left(\frac{\log T}{\sqrt{T}}\right)$, and under the nonconvex setting, the best proved rate of SGD is also $O\left(\frac{\log T}{\sqrt{T}}\right)$ (Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming).
Thus, the two rates are theoretically comparable.
We agree that proving “adaptive gradient methods have improved convergence properties compared to SGD” is a very interesting problem.  In our paper and other previous works such as Mnih et al., 2016, Seward et al., 2020; Yazıcı et al., 2019,  the advantage of RMSprop over SGD is mainly demonstrated through experiments. The intuition of the advantage of adaptive gradient methods is that by using adaptive learning rate, we can better utilize the local structure of the problem and thus achieve faster convergence, which is similar to the idea of Quasi-Newton method.
Nevertheless, even for convex problems, the theoretical advantage of the Quasi-Newton method was not well understood (a possible benefit is asymptotic superlinear convergence, but such a benefit is not easy to prove for stochastic versions).
Thus it seems that providing a strong theoretical justification for the advantage of Adam-type methods is challenging and requires much more work. We hope to explore this problem in the future.
2. Exigesis on condition (4):
Thanks for the comment. The estimate of $\beta_2$ by condition (4) in theorem 4.3 is $1-\beta_2\le O\left(\frac{1}{n^{3.5}}\right)$. We did not try to optimize this bound before as we focus on proving the existence of such a threshold. We agree that explaining the bound will help readers understand RMSProp better, so we add some discussions on $\beta_2$ below and in the revised version. The bound indicates that the transition point is roughly $\beta_2 = 1 - 1/poly(n)$.  In our experiments on Resnet18 with batchsize 8 and n $\approx$ 5000, the transition point of $\beta_2$ is between $0.95$ and $0.99$; for n $\approx 2500$ (batch size 16), the transition point lies in [0.9,0.95];
for $n\approx 1200$, the transition point lies in [0.8,0.9]. This indicates that the transition point is decreasing over $n$, which qualitatively matches our bound of $\beta_2$. In this experiment, the practical size of the $\beta_2$-threshold
seems to be somewhere between $1 - O(1/\sqrt{n})$ and $1 - O(1/n)$. Note that we do not have extensively studied the order of the transition point, so this is a preliminary empirical estimate. If the empirical estimate is reasonable, then there is a gap between our theoretical bound and this empirical estimate.
In an effort to reduce the gap, we revisited our proof, and find that our proof only requires $1-\beta_2\le O\left(\frac{1}{n\rho_1\rho_2\rho_3}\right)$, where $\rho_1$, $\rho_2$, and $\rho_3$ are three quantities that characterize the distribution of the gradients; see equations (14)-(16), equation (37) and the remark after it in the appendix of the revised version. The upper-bounds of $\rho_1$, $\rho_2$, and $\rho_3$ are $\sqrt{n}$, $n$, and $n$ respectively, thus in the worst case the bound is $1-\beta_2\le O\left(\frac{1}{n^{3.5}}\right)$. This is why we obtain this bound in the original condition (4). The benefit of introducing the quantities $\rho_1$, $\rho_2$,  and $\rho_3$ is that the threshold of $\beta_2$ may be smaller than the worst-case estimate if $\rho_1$, $\rho_2$,  and $\rho_3$ are small in practice.
For instance, if $\rho_i = O(1)$, then  $1 - \beta_2 \approx O(1/n). $ We checked the size of $\rho_i$'s in a MNIST experiment, and find that $\rho_1 \rho_2 \rho_3$ roughly lies in $[ 1, n ]$, in which case the bound of $\beta_2$ is somewhere between $1- O\left(\frac{1}{n^{1}}\right)$ and $1- O\left(\frac{1}{n^{2}}\right)$. This is closer to the emprical transition point we observe. Anyhow, it remains a question to provide a more precise estimate of $\beta_2$.
We add a remark (Remark 2 after Theorem 4.3) in the revised version to explain the size of $\beta_2$ and the stronger bound based on $\rho_i$'s; we also provide a preliminary empirical estimate of $\rho_i's$ in Appendix A.5.
This work revisits a famous counterexample on the convergence of Adam (originally presented in Reddi 2018). The authors show that, if the EMA parameter beta2 in RMSprop and Adam is chosen high enough, then both methods converge to a bounded region in the stochastic setting.
In addition, the authors provide some results for the full-batch case.
Crucially, and differently from many other papers on the topic, the gradients are not assumed to be bounded and the beta2 hyperparameter is not chosen to increase to 1.
The paper is well written and the logic of it is convincing. I like the introduction and Figure 1 (this nicely illustrates the relevance of this paper). It is also very well organized. Unfortunately, I did not have the time I wish I had to dig into the proofs (just had a quick check), but the methodology of the authors and the results are convincing.
This is overall a very nice paper, with clean and easy to read results, that clarifies an important point: it is misleading to claim that “Adam does not converge” (which was pointed out in Reddi 2018 to introduce AMSgrad). I have heard this (wrong) claim many times in the optimization community – hence I think this paper deserves attention (therefore my clear accept). This work truly does merge the gap between theory and practice in non-convex stochastic optimization.
Just a few suggestions: I think the authors should cite and discuss the results in Defossez et al. 2019 (On the convergence of Adam and Adagrad).
Also, I think Figure 1 deserves better quality. It's done in matlab so in the xlabel command you can put 'interpreter','latex' and 'fontsize',20.
Finally, I spotted 1 typo: in Remark2 “cases of non-divergence cases”.
Thank you for your supportive and encouraging comments! Below we provide the responses to specific suggestions.
1. The work Defossez et al. 2019 On the convergence of Adam and Adagrad：
Thank you for the suggestion.
That is a representative paper and we now added the discussion of it in the related work section: “The work (Defossez et al. 2019) establishes a clean convergence result and also provides some insights on the momentum mechanisms by improving the dependence of $1-\beta_1$ in the convergence rate.
However the work still assumes bounded gradient and non-zero $\epsilon$.”
2. x-label of figure 1：
Thanks a lot for your suggestion! We have replotted Figure 1 according to your suggestion.
3. Typo：
Thanks for pointing it out. We now changed it to “cases of non-divergence”.
We correct two typos in our proof.
- On page 20 of the appendix, the two summation inequalities should be
\sum_{t=t_{init}}^T\frac{1}{t}\le \log\frac{T}{t_{init}-1},
and
\sum_{t=t_{init}}^T\frac{1}{\sqrt{t}}\ge 2\left(\sqrt{T+1}-\sqrt{t_{init}}\right)
Therefore, by setting $t_{init}=2$, we can derive the convergence result at the end of Appendix D as,
\min_{t\in[2,T]} \lVert\nabla f_t\rVert_1\le \frac{1}{\sqrt{T+1}-\sqrt{2}}\left( Q_{1,1}+Q_{2,1}\log T\right)
- The results on page 22, page 37, and page 42 should be adjusted accordingly.

Meta-review:
role: PC
data: comment: The paper shows convergence results for RMSprop in certain regimes. The reviews are uniformly positive about this paper and I recommend acceptance.

Decision: Accept (Spotlight)