SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison with AdaGrad. AdaGrad [12] is another popular adaptive optimizer. Under Assumptions 1 and 2, the state-of-art iteration complexity of AdaGrad is  $\mathcal{O}\left(\frac{\log 1 / \varepsilon}{\varepsilon^4}\right)$  [13], which is worse than Adam by a logarithmic factor. Here we show that such a gap may be not due to the limitation of analysis, and can be explained by analogizing AdaGrad to Adam without momentum as SGD with diminishing learning rate to SGD with constant learning rate. To start with, the update rule of AdaGrad is given as
$$
\boldsymbol {\nu} _ {t} = \boldsymbol {\nu} _ {t - 1} + \boldsymbol {g} _ {t} ^ {\odot 2}, \boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \frac {1}{\sqrt {\boldsymbol {\nu} _ {t}}} \odot \boldsymbol {g} _ {t}. \tag {8}
$$
We first show that in Algorithm 1, if we allow the hyperparameters to be dynamical, i.e.,
$$
\boldsymbol {\nu} _ {t} = \beta_ {2, t} \boldsymbol {\nu} _ {t - 1} + (1 - \beta_ {2, t}) \boldsymbol {g} _ {t} ^ {\odot 2}, \boldsymbol {m} _ {t} = \beta_ {1, t} \boldsymbol {m} _ {t - 1} + (1 - \beta_ {1, t}) \boldsymbol {g} _ {t}, \boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta_ {t} \frac {1}{\sqrt {\boldsymbol {\nu} _ {t}}} \odot \boldsymbol {m} _ {t}, \tag {9}
$$
then Adam is equivalent to AdaGrad by setting  $\eta_t = \frac{\eta}{\sqrt{t}}$ ,  $\beta_{1,t} = 0$ , and  $\beta_{2,t} = 1 - \frac{1}{t}$ . Specifically, by setting  $\pmb{\mu}_t = t\pmb{\nu}_t$  in Eq. (9), we have Eq. (9) is equivalent to with Eq. (8) (by replacing  $\pmb{\nu}_t$  by  $\pmb{\mu}_t$  in Eq. (8)). Comparing the above hyperparameter setting with that in Theorem 2, we see that the above hyperparameter setting can be obtained by changing  $T$  to  $t$  and setting  $c = 0$  in Theorem 2. This is similar to the relationship between SGD with diminishing learning rate  $\Theta(1/\sqrt{t})$  and SGD with diminishing learning rate  $\Theta(1/\sqrt{T})$ . Recall that the iteration complexity of SGD with diminishing learning rate  $\Theta(1/\sqrt{t})$  also has an additional logarithmic factor than SGD with constant learning rate, which may explain the gap between AdaGrad and Adam.
Despite that our work provides the first result closing the upper bound and lower bound of the iteration complexity of Adam, there are several limitations listed as follows:
Dependence over the dimension  $d$ . The bounds in Theorem 1 and Theorem 2 is monotonously increasing with respect to  $d$ . This is undesired since the upper bound of iteration complexity of SGD is invariant with respect to  $d$ . Nevertheless, removing such an dependence over  $d$  is technically hard since we need to deal with every coordinate separately due to coordinate-wise learning rate, while the descent lemma does not hold for a single coordinate but combines all coordinates together. To our best knowledge, all existing works on the convergene of Adam also suffers from the same problem. We leave removing the dependence over  $d$  as an important future work.
No better result with momentum. It can be observed that in Theorem 1 and Theorem 2, the tightest bound is achieved when  $\beta_{1} = 0$  (i.e., no momentum is applied). This contradicts with the common wisdom that momentum helps to accelerate. Although the benefit of momentum is not very clear even for simple optimizer SGD with momentum, we view this as a limitation of our work and defer proving the benefit of momentum in Adam as a future work. Also, our result does not imply that setting  $\beta_{1}$  is not as critical as setting  $\beta_{2}$ . The primary objective of this paper is to characterize the dependence on  $\varepsilon$ , and the importance of setting  $\beta_{1}$  might be justified in other ways or characterizations. To help readers gain a deeper understanding of this issue, we include experiments to illustrate the dependence of performance on  $\beta_{1}$  in Appendix E.
This work was founded by the CAS Project for Young Scientists in Basic Research under Grant No. YSBR-034 and the Innovation Funding of ICT, CAS under Grant No.E000000.
[21] H. Li, A. Jadbabaie, and A. Rakhlin. Convergence of Adam under relaxed assumptions. arXiv preprint arXiv:2304.13972, 2023.
[22] Y. Liu, Y. Gao, and W. Yin. An improved analysis of stochastic gradient descent with momentum. Advances in Neural Information Processing Systems, 33:18261-18271, 2020.
[23] L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.09843, 2019.
[24] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
[25] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. volume 32, 2019.
[26] S. J. Reddi, S. Kale, and S. Kumar. On the convergence of Adam and beyond. arXiv preprint arXiv:1904.09237, 2019.
[27] N. Shi, D. Li, M. Hong, and R. Sun. RMSprop converges with proper hyper-parameter. In International Conference on Learning Representations, 2021.
[28] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[30] B. Wang, Y. Zhang, H. Zhang, Q. Meng, Z.-M. Ma, T.-Y. Liu, and W. Chen. Provable adaptivity in Adam. arXiv preprint arXiv:2208.09900, 2022.
[31] B. Wang, H. Zhang, Z. Ma, and W. Chen. Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. In The Thirty Sixth Annual Conference on Learning Theory, pages 161–190. PMLR, 2023.
[32] X. Xie, P. Zhou, H. Li, Z. Lin, and S. Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. arXiv preprint arXiv:2208.06677, 2022.
[33] M. Zaheer, S. Reddi, D. Sachan, S. Kale, and S. Kumar. Adaptive methods for nonconvex optimization. Advances in neural information processing systems, 31, 2018.
[34] Y. Zhang, C. Chen, N. Shi, R. Sun, and Z.-Q. Luo. Adam can converge without any modification on update rules. arXiv preprint arXiv:2208.09632, 2022.
[35] Z. Zhou, Q. Zhang, G. Lu, H. Wang, W. Zhang, and Y. Yu. Adashift: Decorrelation and convergence of adaptive learning rate methods. arXiv preprint arXiv:1810.00143, 2018.
[36] F. Zou, L. Shen, Z. Jie, W. Zhang, and W. Liu. A sufficient condition for convergences of Adam and RMSProp. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11127-11135, 2019.
A Other Related works
Section 3 has provided a detailed discussion over existing convergence analysis of Adam. In this section, we briefly review other related works. Adam is proposed with a convergence analysis in online optimization [20]. The proof, however, is latter shown to be flawed in Reddi et al. [26] as it requires the adaptive learning rate of Adam to be non-increasing. This motivates a line of works modifying Adam to ensure convergence. The modifications include enforcing the adaptive learning rate to be non-increasing [26, 5], imposing upper bound and lower bound of the adaptive learning rate [23], and using different approach to estimate second-order momentum [35, 7]. Recently, Chen et al. [6] discover a new optimizer Lion through Symbolic Discovery, which uses sign operation to replace the adaptive learning rate in Adam, achieving comparable performance of Adam with less memory costs.
The following two lemmas are useful when bounding the second-order term.
Lemma 4. Assume we have  $0 < \beta_{2} < 1$  and a sequence of real numbers  $(a_{n})_{n = 1}^{\infty}$ . Let  $b_{0} > 0$  and  $b_{n} = \beta_{2}b_{n - 1} + (1 - \beta_{2})a_{n}^{2}$ . Then, we have
$$
\sum_ {n = 1} ^ {T} \frac {a _ {n} ^ {2}}{b _ {n}} \leq \frac {1}{1 - \beta_ {2}} \left(\ln \left(\frac {b _ {T}}{b _ {0}}\right) - T \ln \beta_ {2}\right).
$$
Lemma 5. Assume we have  $0 < \beta_{1}^{2} < \beta_{2} < 1$  and a sequence of real numbers  $(a_{n})_{n = 1}^{\infty}$ . Let  $b_{0} > 0$ ,  $b_{n} = \beta_{2}b_{n - 1} + (1 - \beta_{2})a_{n}^{2}$ ,  $c_{0} = 0$ , and  $c_{n} = \beta_{1}c_{n - 1} + (1 - \beta_{1})a_{n}$ . Then, we have
$$
\sum_ {n = 1} ^ {T} \frac {| c _ {n} | ^ {2}}{b _ {n}} \leq \frac {(1 - \beta_ {1}) ^ {2}}{(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}) ^ {2} (1 - \beta_ {2})} \left(\ln \left(\frac {b _ {T}}{b _ {0}}\right) - T \ln \beta_ {2}\right).
$$
Proof. To begin with,
$$
\frac {| c _ {n} |}{\sqrt {b _ {n}}} \leq (1 - \beta_ {1}) \sum_ {i = 1} ^ {n} \frac {\beta_ {1} ^ {n - i} | a _ {i} |}{\sqrt {b _ {n}}} \leq (1 - \beta_ {1}) \sum_ {i = 1} ^ {n} \frac {\beta_ {1} ^ {n - i} | a _ {i} |}{\sqrt {b _ {n}}} \leq (1 - \beta_ {1}) \sum_ {i = 1} ^ {n} \left(\frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {n - i} \frac {| a _ {i} |}{\sqrt {b _ {i}}}.
$$
Applying Cauchy's inequality, we obtain
$$
\begin{array}{l} \frac {\left| c _ {n} \right| ^ {2}}{b _ {n}} \leq (1 - \beta_ {1}) ^ {2} \left(\sum_ {i = 1} ^ {n} \left(\frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {n - i} \frac {\left| a _ {i} \right|}{\sqrt {b _ {i}}}\right) ^ {2} \\ \leq (1 - \beta_ {1}) ^ {2} \left(\sum_ {i = 1} ^ {n} \left(\frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {n - i}\right) \left(\sum_ {i = 1} ^ {n} \left(\frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {n - i} \frac {| a _ {i} | ^ {2}}{b _ {i}}\right) \leq \frac {(1 - \beta_ {1}) ^ {2}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \left(\sum_ {i = 1} ^ {n} \left(\frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {n - i} \frac {| a _ {i} | ^ {2}}{b _ {i}}\right). \\ \end{array}
$$
Summing the above inequality over  $n$  from 1 to  $T$  then leads to
$$
\begin{array}{l} \sum_ {n = 1} ^ {T} \frac {| c _ {n} | ^ {2}}{b _ {n}} \leq \frac {(1 - \beta_ {1}) ^ {2}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \sum_ {n = 1} ^ {T} \left(\sum_ {i = 1} ^ {n} \left(\frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {n - i} \frac {| a _ {i} | ^ {2}}{b _ {i}}\right) = \frac {(1 - \beta_ {1}) ^ {2}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \sum_ {n = 1} ^ {T} \frac {| a _ {n} | ^ {2}}{b _ {n}} \left(\sum_ {i = 0} ^ {T - n} \left(\frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {i}\right) \\ \leq \frac {(1 - \beta_ {1}) ^ {2}}{(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}) ^ {2}} \sum_ {n = 1} ^ {T} \frac {| a _ {n} | ^ {2}}{b _ {n}} \leq \frac {(1 - \beta_ {1}) ^ {2}}{(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}) ^ {2} (1 - \beta_ {2})} \left(\ln \left(\frac {b _ {T}}{b _ {0}}\right) - T \ln \beta_ {2}\right). \\ \end{array}
$$
The proof is completed.
The following lemma bound the update norm of Adam.
Lemma 6. We have  $\forall t\geq 1$ $|\pmb{w}_{t + 1,i} - \pmb{w}_{t,i}|\leq \eta \frac{1 - \beta_1}{\sqrt{1 - \beta_2}\sqrt{1 - \frac{\beta_1^2}{\beta_2}}}\leq \eta \frac{1 - \beta_1}{\sqrt{1 - \beta_2}\sqrt{1 - \frac{\beta_1}{\sqrt{\beta_2}}}}.$
Proof. We have that
$$
\begin{array}{l} \left| \boldsymbol {w} _ {t + 1, i} - \boldsymbol {w} _ {t, i} \right| = \eta \left| \frac {\boldsymbol {m} _ {t , i}}{\sqrt {\boldsymbol {\nu} _ {t , i}}} \right| \leq \eta \frac {\sum_ {i = 0} ^ {t - 1} (1 - \beta_ {1}) \beta_ {1} ^ {i} | \boldsymbol {g} _ {t - i , l} |}{\sqrt {\sum_ {i = 0} ^ {t - 1} (1 - \beta_ {2}) \beta_ {2} ^ {i} | \boldsymbol {g} _ {t - i , l} | ^ {2} + \beta_ {2} ^ {t} \boldsymbol {\nu} _ {0 , i}}} \\ \leq \eta \frac {1 - \beta_ {1}}{\sqrt {1 - \beta_ {2}}} \frac {\sqrt {\sum_ {i = 0} ^ {t - 1} \beta_ {2} ^ {i} | \boldsymbol {g} _ {t - i , l} | ^ {2}} \sqrt {\sum_ {i = 0} ^ {t - 1} \frac {\beta_ {1} ^ {2 i}}{\beta_ {2} ^ {i}}}}{\sqrt {\sum_ {i = 0} ^ {t - 1} \beta_ {2} ^ {i} | \boldsymbol {g} _ {t - i , l} | ^ {2}}} \leq \eta \frac {1 - \beta_ {1}}{\sqrt {1 - \beta_ {2}} \sqrt {1 - \frac {\beta_ {1} ^ {2}}{\beta_ {2}}}}. \\ \end{array}
$$
Here the second inequality is due to Cauchy's inequality. The proof is completed.
C Proof of Theorem 1
This section collects the proof of Theorem 1. As a part of the proof, we first provide formal descriptions of Lemma 1, Lemma 2, and Lemma 3, and their corresponding proofs. We then proceed to prove Theorem 1 leveraging these lemmas.
C.1 Formal description of Lemma 1, Lemma 2, and Lemma 3 and their proof
Lemma 7 (Formal version of Lemma 1). Let all conditions in Theorem 1 hold. Then, we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\langle \boldsymbol {G} _ {t}, - \frac {\eta}{1 - \frac {\beta_ {1}}{\sqrt {\boldsymbol {\nu} _ {t}}}} \left(\frac {1}{\sqrt {\boldsymbol {\nu} _ {t}}} - \frac {1}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t}}}\right) \odot \boldsymbol {m} _ {t} \right\rangle \right] \leq \frac {5}{8} \sum_ {i = 1} ^ {d} \eta \frac {1 - \beta_ {1}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \mathbb {E} \frac {| \boldsymbol {G} _ {t , i} | ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} + \frac {2 \eta \sqrt {1 - \beta_ {2}} \sigma_ {0}}{\left(1 - \frac {\beta_ {1} ^ {2}}{\beta_ {2}}\right) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{\boldsymbol {\nu} _ {t , i}} \\ + \eta \frac {4 (1 - \beta_ {1})}{(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}) ^ {2} \sqrt {\beta_ {2}}} \sigma_ {1} ^ {2} \sum_ {i = 1} ^ {d} \mathbb {E} \left(\frac {\boldsymbol {G} _ {t - 1 , i} ^ {2}}{\sqrt {\beta_ {2} \tilde {\boldsymbol {\nu}} _ {t , i}}} - \frac {\boldsymbol {G} _ {t , i} ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t + 1 , i}}}\right) + \sum_ {i = 1} ^ {d} \frac {2 \eta \sqrt {1 - \beta_ {2}} \sigma_ {0}}{(1 - \beta_ {1}) (1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}})} \mathbb {E} \left[ \left(\frac {| \boldsymbol {m} _ {t , i} | ^ {2}}{\boldsymbol {\nu} _ {t , i}}\right) \right] \\ + \frac {6 4 (1 + \sigma_ {1} ^ {2}) \sigma_ {1} ^ {2} L ^ {2} \eta^ {3} d}{\beta_ {2} ^ {2} (1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}) ^ {3} (1 - \beta_ {1}) \sigma_ {0} \sqrt {1 - \beta_ {2}}} \mathbb {E} \left\| \frac {1}{\sqrt {\boldsymbol {\nu} _ {t - 1}}} \odot \boldsymbol {m} _ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
Proof. To start with,
$$
\begin{array}{l} \mathbb {E} ^ {| \mathcal {F} _ {t}} \left[ \left\langle \boldsymbol {G} _ {t}, - \frac {\eta}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \left(\frac {1}{\sqrt {\boldsymbol {\nu} _ {t}}} - \frac {1}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t}}}\right) \odot \boldsymbol {m} _ {t} \right\rangle \right] \\ = \mathbb {E} ^ {| \mathcal {F} _ {t}} \left[ \left\langle \boldsymbol {G} _ {t}, - \frac {\eta}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \left(\frac {(1 - \beta_ {2}) (\sigma_ {0} ^ {2} \mathbf {1} _ {d} - \boldsymbol {g} _ {t} ^ {\odot 2})}{\sqrt {\boldsymbol {\nu} _ {t}} \sqrt {\tilde {\boldsymbol {\nu}} _ {t}} (\sqrt {\boldsymbol {\nu} _ {t}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t}})}\right) \odot \boldsymbol {m} _ {t} \right\rangle \right] \\ \leq \sum_ {i = 1} ^ {d} \frac {\eta}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \mathbb {E} ^ {| \mathcal {F} _ {t}} \left[ | \boldsymbol {G} _ {t, i} | \left(\frac {(1 - \beta_ {2}) (\sigma_ {0} ^ {2} + \boldsymbol {g} _ {t , i} ^ {2})}{\sqrt {\boldsymbol {\nu} _ {t , i}} \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}} (\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}})}\right) | \boldsymbol {m} _ {t, i} | \right] \\ = \underbrace {\sum_ {i = 1} ^ {d} \frac {\eta}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \mathbb {E} ^ {| \mathcal {F} _ {t}}} _ {\text {I . 1 . 1}} \left[ | \boldsymbol {G} _ {t, i} | \left(\frac {(1 - \beta_ {2}) \boldsymbol {g} _ {t , i} ^ {2}}{\sqrt {\boldsymbol {\nu} _ {t , i}} \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}} (\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}})}\right) | \boldsymbol {m} _ {t, i} | \right] \\ + \underbrace {\sum_ {i = 1} ^ {d} \frac {\eta}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \mathbb {E} ^ {| \mathcal {F} _ {t}}} _ {\text {I . 1 . 2}} \left[ | \boldsymbol {G} _ {t, i} | \left(\frac {(1 - \beta_ {2}) \sigma_ {0} ^ {2}}{\sqrt {\boldsymbol {\nu} _ {t , i}} \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}} (\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}})}\right) | \boldsymbol {m} _ {t, i} | \right]. \\ \end{array}
$$
As for I.1.1, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {d} \frac {\eta}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \mathbb {E} ^ {| \mathcal {F} _ {t}} \left[ | \boldsymbol {G} _ {t, i} | \left(\frac {(1 - \beta_ {2}) \boldsymbol {g} _ {t , i} ^ {2}}{\sqrt {\boldsymbol {\nu} _ {t , i}} \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}} (\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}})}\right) | \boldsymbol {m} _ {t, i} | \right] \\ \stackrel {(*)} {\leq} \sum_ {i = 1} ^ {d} \frac {\eta (1 - \beta_ {1})}{\left(\sqrt {1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}}\right) ^ {3}} \mathbb {E} ^ {| \mathcal {F} _ {t}} \left[ | \boldsymbol {G} _ {t, i} | \left(\frac {\sqrt {1 - \beta_ {2}} \boldsymbol {g} _ {t , i} ^ {2}}{\sqrt {\tilde {\nu} _ {t , i}} (\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}})}\right) \right] \\ \stackrel {(\circ)} {\leq} \sum_ {i = 1} ^ {d} \frac {\eta (1 - \beta_ {1})}{\left(\sqrt {1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}}\right) ^ {3}} \frac {| \boldsymbol {G} _ {t , i} |}{\sqrt {\tilde {\nu} _ {t , i}}} \sqrt {\mathbb {E} ^ {| \mathcal {F} _ {t}} \boldsymbol {g} _ {t , i} ^ {2}} \sqrt {\mathbb {E} ^ {| \mathcal {F} _ {t}} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{(\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}) ^ {2}}} \\ \stackrel {(\bullet)} {\leq} \sum_ {i = 1} ^ {d} \frac {\eta (1 - \beta_ {1}) \sqrt {1 - \beta_ {2}}}{\left(\sqrt {1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}}\right) ^ {3}} \frac {| \boldsymbol {G} _ {t , i} |}{\sqrt {\tilde {\nu} _ {t , i}}} \sqrt {\sigma_ {0} ^ {2} + \sigma_ {1} ^ {2} \boldsymbol {G} _ {t , i} ^ {2}} \sqrt {\mathbb {E} ^ {| \mathcal {F} _ {t}} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{(\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}) ^ {2}}} \\ \leq \sum_ {i = 1} ^ {d} \frac {\eta (1 - \beta_ {1}) \sqrt {1 - \beta_ {2}}}{\left(\sqrt {1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}}\right) ^ {3}} \frac {| \boldsymbol {G} _ {t , i} |}{\sqrt {\tilde {\nu} _ {t , i}}} (\sigma_ {0} + \sigma_ {1} | \boldsymbol {G} _ {t, i} |) \sqrt {\mathbb {E} | \mathcal {F} _ {t} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{(\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}) ^ {2}}}, \\ \end{array}
$$
where inequality  $(\ast)$  uses Lemma 6, inequality  $(\circ)$  is due to Holder's inequality, and inequality  $(\bullet)$  is due to Assumption 3. Applying mean-value inequality respectively to  $\sum_{i=1}^{d} \frac{\eta(1-\beta_1)\sqrt{1-\beta_2}}{\left(\sqrt{1-\frac{\beta_1}{\sqrt{\beta_2}}}\right)^3} \mathbb{E}^{|F_t|} \frac{|G_{t,i}|}{\sqrt{\tilde{\nu}_{t,i}}} \sigma_0 \sqrt{\mathbb{E}^{|F_t|} \frac{g_{t,i}^2}{(\sqrt{\nu_{t,i}}+\sqrt{\tilde{\nu}_{t,i}})^2}}$  and  $\sum_{i=1}^{d} \frac{\eta(1-\beta_1)\sqrt{1-\beta_2}}{\left(\sqrt{1-\frac{\beta_1}{\sqrt{\beta_2}}}\right)^3} \mathbb{E}^{|F_t|} \frac{|G_{t,i}|}{\sqrt{\tilde{\nu}_{t,i}}} \sigma_1 |G_{t,i}| \sqrt{\mathbb{E}^{|F_t|} \frac{g_{t,i}^2}{(\sqrt{\nu_{t,i}}+\sqrt{\tilde{\nu}_{t,i}})^2}}$ , we obtain that the right-hand-side of the above inequality can be bounded by
$$
\begin{array}{l} \frac {1}{8} \sum_ {i = 1} ^ {d} \eta \frac {1 - \beta_ {1}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \sqrt {1 - \beta_ {2}} \sigma_ {0} \frac {| \boldsymbol {G} _ {t , i} | ^ {2}}{\tilde {\boldsymbol {\nu}} _ {t , i}} + \frac {2 \eta \sqrt {1 - \beta_ {2}} \sigma_ {0}}{\left(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} ^ {| \mathcal {F} _ {t}} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{(\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}) ^ {2}} \\ + \frac {1}{8} \sum_ {i = 1} ^ {d} \eta \frac {1 - \beta_ {1}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \frac {| \boldsymbol {G} _ {t , i} | ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} + 2 \eta \frac {(1 - \beta_ {2}) (1 - \beta_ {1})}{(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}) ^ {2}} \sigma_ {1} ^ {2} \frac {| \boldsymbol {G} _ {t , i} | ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} \mathbb {E} ^ {| \mathcal {F} _ {t}} \sum_ {i = 1} ^ {d} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{(\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}) ^ {2}} \\ \leq \frac {1}{8} \sum_ {i = 1} ^ {d} \eta \frac {1 - \beta_ {1}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \frac {\left| \boldsymbol {G} _ {t , i} \right| ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} + \frac {2 \eta \sqrt {1 - \beta_ {2}} \sigma_ {0}}{\left(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} ^ {| \mathcal {F} _ {t}} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{\boldsymbol {\nu} _ {t , i}} \\ + \frac {1}{8} \sum_ {i = 1} ^ {d} \eta \frac {1 - \beta_ {1}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \frac {\left| \boldsymbol {G} _ {t , i} \right| ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} + 2 \eta \frac {(1 - \beta_ {2}) (1 - \beta_ {1})}{(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}) ^ {2}} \sigma_ {1} ^ {2} \frac {\left| \boldsymbol {G} _ {t , i} \right| ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} \mathbb {E} ^ {| \mathcal {F} _ {t}} \sum_ {i = 1} ^ {d} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{(\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}) ^ {2}}. \tag {10} \\ \end{array}
$$
Here the inequality is due to  $\widetilde{\pmb{\nu}}_{t,i} = (1 - \beta_2)\sigma_0^2 +\beta_2\pmb{\nu}_{t - 1,i}\geq (1 - \beta_2)\sigma_0^2$  . Meanwhile, we have
$$
\begin{array}{l} \left(\frac {1}{\sqrt {\beta_ {2} \widetilde {\boldsymbol {\nu}} _ {t , i}}} - \frac {1}{\sqrt {\widetilde {\boldsymbol {\nu}} _ {t + 1 , i}}}\right) \boldsymbol {G} _ {t, i} ^ {2} \\ = \frac {\pmb {G} _ {t , i} ^ {2} ((1 - \beta_ {2}) ^ {2} \sigma_ {0} ^ {2} + \beta_ {2} (1 - \beta_ {2}) \pmb {g} _ {t , i} ^ {2})}{\sqrt {\beta_ {2} \tilde {\pmb {\nu}} _ {t , i}} \sqrt {\tilde {\pmb {\nu}} _ {t + 1 , i}} (\sqrt {\beta_ {2} \tilde {\pmb {\nu}} _ {t , i}} + \sqrt {\tilde {\pmb {\nu}} _ {t + 1 , i}})} \geq \frac {\pmb {G} _ {t , i} ^ {2} \beta_ {2} (1 - \beta_ {2}) \pmb {g} _ {t , i} ^ {2}}{\sqrt {\beta_ {2} \tilde {\pmb {\nu}} _ {t , i}} \sqrt {\tilde {\pmb {\nu}} _ {t + 1 , i}} (\sqrt {\beta_ {2} \tilde {\pmb {\nu}} _ {t , i}} + \sqrt {\tilde {\pmb {\nu}} _ {t + 1 , i}})} \\ \geq \frac {\sqrt {\beta_ {2}}}{2} \frac {\pmb {G} _ {t , i} ^ {2} (1 - \beta_ {2}) \pmb {g} _ {t , i} ^ {2}}{\sqrt {\widetilde {\pmb {\nu}} _ {t , i}} (\sqrt {\pmb {\nu} _ {t , i}} + \sqrt {\widetilde {\pmb {\nu}} _ {t , i}}) ^ {2}}. \\ \end{array}
$$
Applying the above inequality back to Eq. (10), we obtain that
$$
\begin{array}{l} \sum_ {i = 1} ^ {d} \frac {\eta}{1 - \beta_ {1}} \mathbb {E} ^ {| \mathcal {F} _ {t}} \left[ | \boldsymbol {G} _ {t, i} | \left(\frac {(1 - \beta_ {2}) \boldsymbol {g} _ {t , i} ^ {2}}{\sqrt {\boldsymbol {\nu} _ {t , i}} \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}} (\sqrt {\boldsymbol {\nu} _ {t , i}} + \sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}})}\right) | \boldsymbol {m} _ {t, i} | \right] \\ \leq \frac {1}{4} \sum_ {i = 1} ^ {d} \eta \frac {1 - \beta_ {1}}{1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}} \frac {| \boldsymbol {G} _ {t , i} | ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} + \frac {2 \eta \sqrt {1 - \beta_ {2}} \sigma_ {0}}{\left(1 - \frac {\beta_ {1} ^ {2}}{\beta_ {2}}\right) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} ^ {| \mathcal {F} _ {t}} \frac {\boldsymbol {g} _ {t , i} ^ {2}}{\boldsymbol {\nu} _ {t , i}} \\ + \eta \frac {4 \left(1 - \beta_ {1}\right)}{\left(1 - \frac {\beta_ {1}}{\sqrt {\beta_ {2}}}\right) ^ {2} \sqrt {\beta_ {2}}} \sigma_ {1} ^ {2} \sum_ {i = 1} ^ {d} \mathbb {E} ^ {| \mathcal {F} _ {t}} \left(\frac {1}{\sqrt {\beta_ {2} \tilde {\nu} _ {t , i}}} - \frac {1}{\sqrt {\tilde {\nu} _ {t + 1 , i}}}\right) \boldsymbol {G} _ {t, i} ^ {2}. \tag {11} \\ \end{array}
$$
Furthermore, due to Assumption 1, we have (we define  $G_0 \triangleq G_1$ )