SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[52] Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences, 64, 2021.
[53] Dongruo Zhou, Jinghui Chen, Yuan Cao, Yiqi Tang, Ziyan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671, 2018.
[54] Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang, Weinan Zhang, and Yong Yu. Adashift: Decorrelation and convergence of adaptive learning rate methods. ArXiv, abs/1810.00143, 2018.
[55] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251, 2017.
[56] Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for convergences of adam and rmsprop. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11119-11127, 2018.
In this section, we discuss additional related work on variants of Adam and variance reduction methods.
Variants of Adam. After Reddi et al. [37] pointed out the non-convergence issue with Adam, various variants of Adam that can be proved to converge were proposed [56; 18; 6; 5; 31; 54]. For example, AMSGrad [37] and AdaFom [6] modify the second order momentum so that it is non-decreasing. AdaBound [31] explicitly imposes upper and lower bounds on the second order momentum so that the stepsize is also bounded. AdaShift [54] uses a new estimate of the second order momentum to correct the bias. There are also some works [53; 18; 20] that provide convergence guarantees of these variants. One closely related work to ours is [47], which considers a variance-reduced version of Adam by combining Adam and SVRG [22]. However, they assume bounded gradients and can only get an asymptotic convergence in the non-convex setting.
Variance reduction methods. The technique of variance reduction was introduced to accelerate convex optimization in the finite-sum setting [39; 22; 41; 32; 11]. Later, many works studied variance-reduced methods in the non-convex setting and obtained improved convergence rates for standard smooth functions. For example, SVRG and SCSG improve the  $\mathcal{O}(\epsilon^{-4})$  gradient complexity of stochastic gradient descent (SGD) to  $\mathcal{O}(\epsilon^{-10/3})$  [2; 36; 25]. Many new variance reduction methods [15; 43; 29; 27; 9; 30] were later proposed to further improve the complexity to  $\mathcal{O}(\epsilon^{-3})$ , which is optimal and matches the lower bound in [3]. Recently, [38; 7] obtained the  $\mathcal{O}(\epsilon^{-3})$  complexity for the more general  $(L_0, L_1)$  smooth functions. Our variance-reduced Adam is motivated by the STORM algorithm proposed by [9], where an additional term is added in the momentum update to correct the bias and reduce the variance.
B Proof sketches of informal lemmas in Section 5.3
In this section, we provide the proof sketches of the informal lemmas in Section 5.3. We focus on illustrating the ideas rather than rigorous proof details. Please see Appendix E for more rigorous and detailed proofs of Adam in the stochastic setting.
Proof Sketch of Lemma 5.4. By the definition of  $\tau$ , for all  $t < \tau$ , we have  $f(x_{t}) - f^{*} \leq F$  which implies  $\| \nabla f(x_{t}) \| \leq G$ . Then from the update rule (18) in Proposition E.1 provided later in Appendix E, it is easy to verify  $\hat{v}_{t} \preceq G^{2}$  since  $\hat{v}_{t}$  is a convex combination of  $\{(\nabla f(x_{s}))^{2}\}_{s \leq t}$ . Let  $h_{t} := \eta / (\sqrt{\hat{v}_{t}} + \lambda)$  be the stepsize vector and denote  $H_{t} := \mathrm{diag}(h_{t})$ . We know
$$
\frac {\eta}{2 G} I \preceq \frac {\eta}{G + \lambda} I \preceq H _ {t} \preceq \frac {\eta}{\lambda} I. \tag {7}
$$
As discussed in Section 5.2, when  $\eta$  is small enough, we can apply Corollary 5.2 to obtain
$$
\begin{array}{l} f \left(x _ {t + 1}\right) - f \left(x _ {t}\right) \lesssim \left\langle \nabla f \left(x _ {t}\right), x _ {t + 1} - x _ {t} \right\rangle \\ = - \left\| \nabla f (x _ {t}) \right\| _ {H _ {t}} ^ {2} - \nabla f (x _ {t}) ^ {\top} H _ {t} \epsilon_ {t} \\ \leq - \frac {1}{2} \| \nabla f (x _ {t}) \| _ {H _ {t}} ^ {2} + \frac {1}{2} \| \epsilon_ {t} \| _ {H _ {t}} ^ {2} \\ \leq - \frac {\eta}{4 G} \left\| \nabla f (x _ {t}) \right\| ^ {2} + \frac {\eta}{2 \lambda} \left\| \epsilon_ {t} \right\| ^ {2}, \\ \end{array}
$$
where in the first (approximate) inequality we ignore the second order term  $\frac{1}{2} L\| x_{t + 1} - x_t\|^2\propto \eta^2$  in Corollary 5.2 for small enough  $\eta$ ; the equality applies the update rule  $x_{t + 1} - x_t = -H_t\hat{m}_t = -H_t(\nabla f(x_t) + \epsilon_t)$ ; in the second inequality we use  $2a^{\top}Ab\leq \| a\| _A^2 +\| b\| _A^2$  for any PSD matrix  $A$  and vectors  $a$  and  $b$ ; and the last inequality is due to (7).
Proof Sketch of Lemma 5.5. By the update rule (18) in Proposition E.1, we have
$$
\epsilon_ {t + 1} = \left(1 - \alpha_ {t + 1}\right) \left(\epsilon_ {t} + \nabla f (x _ {t}) - \nabla f \left(x _ {t + 1}\right)\right). \tag {8}
$$
For small enough  $\eta$ , we can apply Corollary 5.2 to get
$$
\left\| \nabla f \left(x _ {t + 1}\right) - \nabla f \left(x _ {t}\right) \right\| ^ {2} \leq L ^ {2} \left\| x _ {t + 1} - x _ {t} \right\| ^ {2} \leq \mathcal {O} \left(\eta^ {2} G ^ {2 \rho}\right) \| \hat {m} _ {t} \| ^ {2} \leq \mathcal {O} \left(\eta^ {2} G ^ {2 \rho}\right) \left(\left\| \nabla f \left(x _ {t}\right) \right\| ^ {2} + \left\| \epsilon_ {t} \right\| ^ {2}\right), \tag {9}
$$
where the second inequality is due to  $L = \mathcal{O}(G^{\rho})$  and  $\| x_{t + 1} - x_t\| = \mathcal{O}(\eta)\| \hat{m}_t\|$ ; and the last inequality uses  $\hat{m}_t = \nabla f(x_t) + \epsilon_t$  and Young's inequality  $\| a + b\|^2\leq 2\| a\|^2 +2\| b\|^2$ . Therefore,
$$
\begin{array}{l} \left\| \epsilon_ {t + 1} \right\| ^ {2} \leq (1 - \alpha_ {t + 1}) \left(1 + \alpha_ {t + 1} / 2\right) \left\| \epsilon_ {t} \right\| ^ {2} + \left(1 + 2 / \alpha_ {t + 1}\right) \left\| \nabla f \left(x _ {t + 1}\right) - \nabla f \left(x _ {t}\right) \right\| ^ {2} \\ \leq \left(1 - \alpha_ {t + 1} / 2\right) \| \epsilon_ {t} \| ^ {2} + \mathcal {O} \left(\eta^ {2} G ^ {2 \rho} / \alpha_ {t + 1}\right) \left(\| \nabla f (x _ {t}) \| ^ {2} + \| \epsilon_ {t} \| ^ {2}\right) \\ \leq (1 - \beta / 4) \| \epsilon_ {t} \| ^ {2} + \frac {\lambda \beta}{1 6 G} \| \nabla f (x _ {t}) \| ^ {2}, \\ \end{array}
$$
where the first inequality uses (8) and Young's inequality  $\| a + b\|^2 \leq (1 + u)\| a\|^2 + (1 + 1 / u)\| b\|^2$  for any  $u > 0$ ; the second inequality uses  $(1 - \alpha_{t+1})(1 + \alpha_{t+1}/2) \leq 1 - \alpha_{t+1}/2$  and (9); and in the last inequality we use  $\beta \leq \alpha_{t+1}$  and choose  $\beta = \Theta(\eta G^{\rho+1/2})$  which implies  $\mathcal{O}(\eta^2 G^{2\rho} / \alpha_{t+1}) \leq \frac{\lambda\beta}{16G} \leq \beta/4$ .
C Probabilistic lemmas
In this section, we state several well-known and useful probabilistic lemmas without proof.
Lemma C.1 (Azuma-Hoeffding inequality). Let  $\{Z_t\}_{t \geq 1}$  be a martingale with respect to a filtration  $\{\mathcal{F}_t\}_{t \geq 0}$ . Assume that  $|Z_t - Z_{t-1}| \leq c_t$  almost surely for all  $t \geq 0$ . Then for any fixed  $T$ , with probability at least  $1 - \delta$ ,
$$
Z _ {T} - Z _ {0} \leq \sqrt {2 \sum_ {t = 1} ^ {T} c _ {t} ^ {2} \log (1 / \delta)}.
$$
Lemma C.2 (Optional Stopping Theorem). Let  $\{Z_t\}_{t \geq 1}$  be a martingale with respect to a filtration  $\{\mathcal{F}_t\}_{t \geq 0}$ . Let  $\tau$  be a bounded stopping time with respect to the same filtration. Then we have  $\mathbb{E}[Z_{\tau}] = \mathbb{E}[Z_0]$ .
D Proofs related to  $(\rho, L_0, L_\rho)$  smoothness
In this section, we provide proofs related to  $(\rho, L_0, L_\rho)$  smoothness. In what follows, we first provide a formal proposition in Appendix D.1 showing that univariate rational functions and double exponential functions are  $(\rho, L_0, L_\rho)$  smooth with  $\rho < 2$ , as we claimed in Section 3.2.1, and then provide the proofs of Lemma 3.4, Lemma 5.1, and Corollary 5.2 in Appendix D.2, D.3 and D.4 respectively.
D.1 Examples
Proposition D.1. Any univariate rational function  $P(x) / Q(x)$ , where  $P, Q$  are two polynomials, and any double exponential function  $a^{(b^x)}$ , where  $a, b > 1$ , are  $(\rho, L_0, L_\rho)$  smooth with  $1 < \rho < 2$ . However, they are not necessarily  $(L_0, L_1)$  smooth.
Proof of Proposition D.1. We prove the proposition in the following four parts:
1. Univariate rational functions are  $(\rho, L_0, L_\rho)$  smooth with  $1 < \rho < 2$ . Let  $f(x) = P(x) / Q(x)$  where  $P$  and  $Q$  are two polynomials. Then the partial fractional decomposition of  $f(x)$  is given by
$$
f (x) = w (x) + \sum_ {i = 1} ^ {m} \sum_ {r = 1} ^ {j _ {i}} \frac {A _ {i r}}{(x - a _ {i}) ^ {r}} + \sum_ {i = 1} ^ {n} \sum_ {r = 1} ^ {k _ {i}} \frac {B _ {i r} x + C _ {i r}}{(x ^ {2} + b _ {i} x + c _ {i}) ^ {r}},
$$
where  $w(x)$  is a polynomial,  $A_{ir}, B_{ir}, C_{ir}, a_i, b_i, c_i$  are all real constants satisfying  $b_i^2 - 4c_i < 0$  for each  $1 \leq i \leq n$  which implies  $x^2 + b_i x + c_i > 0$  for all  $x \in \mathbb{R}$ . Assume  $A_{ij_i} \neq 0$  without loss of
generality. Then we know  $f$  has only finite singular points  $\{a_i\}_{1 \leq i \leq m}$  and has continuous first and second order derivatives at all other points. To simplify notation, denote
$$
p _ {i r} (x) := \frac {A _ {i r}}{(x - a _ {i}) ^ {r}}, \quad q _ {i r} (x) := \frac {B _ {i r} x + C _ {i r}}{(x ^ {2} + b _ {i} x + c _ {i}) ^ {r}}.
$$
Then we have  $f(x) = w(x) + \sum_{i=1}^{m} \sum_{r=1}^{j_i} p_{ir}(x) + \sum_{i=1}^{n} \sum_{r=1}^{k_i} q_{ir}(x)$ . For any  $3/2 < \rho < 2$ , we know that  $\rho > \frac{r+2}{r+1}$  for any  $r \geq 1$ . Then we can show that
$$
\lim  _ {x \rightarrow a _ {i}} \frac {\left| f ^ {\prime} (x) \right| ^ {\rho}}{\left| f ^ {\prime \prime} (x) \right|} = \lim  _ {x \rightarrow a _ {i}} \frac {\left| p _ {i j _ {i}} ^ {\prime} (x) \right| ^ {\rho}}{\left| p _ {i j _ {i}} ^ {\prime \prime} (x) \right|} = \infty , \tag {10}
$$
where the first equality is because one can easily verify that the first and second order derivatives of  $p_{ij_i}$  dominate those of all other terms when  $x$  goes to  $a_i$ , and the second equality is because  $\left|p_{ij_i}'(x)\right|^{\rho} = \mathcal{O}\left((x - a_i)^{-\rho(j_i + 1)}\right), \left|p_{ij_i}''(x)\right| = \mathcal{O}\left((x - a_i)^{-(j_i + 2)}\right)$ , and  $\rho(j_i + 1) > j_i + 2$  (here we assume  $j_i \geq 1$  since otherwise there is no need to prove (10) for  $i$ ). Note that (10) implies that, for any  $L_\rho > 0$ , there exists  $\delta_i > 0$  such that
$$
\left| f ^ {\prime \prime} (x) \right| \leq L _ {\rho} \left| f ^ {\prime} (x) \right| ^ {\rho}, \quad \text {i f} | x - a _ {i} | <   \delta_ {i}. \tag {11}
$$
Similarly, one can show  $\lim_{x\to \infty}\frac{\left|f'(x)\right|^{\rho}}{\left|f''(x)\right|} = \infty$ , which implies there exists  $M > 0$  such that
$$
\left| f ^ {\prime \prime} (x) \right| \leq L _ {\rho} \left| f ^ {\prime} (x) \right| ^ {\rho}, \quad \text {i f} | x | > M. \tag {12}
$$
$$
\mathcal {B} := \left\{x \in \mathbb {R} \mid | x | \leq M \text {a n d} | x - a _ {i} | \geq \delta_ {i}, \forall i \right\}.
$$
We know  $\mathcal{B}$  is a compact set and therefore the continuous function  $f^{\prime \prime}$  is bounded within  $\mathcal{B}$ , i.e., there exists some constant  $L_0 > 0$  such that
$$
\left| f ^ {\prime \prime} (x) \right| \leq L _ {0}, \quad \text {i f} x \in \mathcal {B}. \tag {13}
$$
Combining (11), (12), and (13), we have shown
$$
\left| f ^ {\prime \prime} (x) \right| \leq L _ {0} + L _ {\rho} \left| f ^ {\prime} (x) \right| ^ {\rho}, \quad \forall x \in \operatorname {d o m} (f),
$$
which completes the proof of the first part.
2. Rational functions are not necessarily  $(L_0, L_1)$  smooth. Consider the ration function  $f(x) = 1/x$ . Then we know that  $f'(x) = -1/x^2$  and  $f''(x) = 2/x^3$ . Note that for any  $0 < x \leq \min \{(L_0 + 1)^{-1/3}, (L_1 + 1)^{-1}\}$ , we have
$$
\left| f ^ {\prime \prime} (x) \right| = \frac {1}{x ^ {3}} + \frac {1}{x} \cdot \left| f ^ {\prime} (x) \right| > L _ {0} + L _ {1} \left| f ^ {\prime} (x) \right|,
$$
which shows  $f$  is not  $(L_0, L_1)$  smooth for any  $L_0, L_1 \geq 0$ .
3. Double exponential functions are  $(\rho, L_0, L_\rho)$  smooth with  $1 < \rho < 2$ . Let  $f(x) = a^{(b^x)}$ , where  $a, b > 1$ , be a double exponential function. Then we know that
$$
f ^ {\prime} (x) = \log (a) \log (b) b ^ {x} a ^ {(b ^ {x})}, \quad f ^ {\prime \prime} (x) = \log (b) (\log (a) b ^ {x} + 1) \cdot f ^ {\prime} (x).
$$
For any  $\rho > 1$ , we have
$$
\lim  _ {x \rightarrow + \infty} \frac {| f ^ {\prime} (x) | ^ {\rho}}{| f ^ {\prime \prime} (x) |} = \lim  _ {x \rightarrow + \infty} \frac {| f ^ {\prime} (x) | ^ {\rho - 1}}{\log (b) (\log (a) b ^ {x} + 1)} = \lim  _ {y \rightarrow + \infty} \frac {(\log (a) \log (b) y) ^ {\rho - 1} a ^ {(\rho - 1) y}}{\log (b) (\log (a) y + 1)} = \infty ,
$$
where the first equality is a direct calculation; the second equality uses change of variable  $y = b^{x}$ ; and the last equality is because exponential function grows faster than linear function. Then we complete the proof following a similar argument to that in Part 1.
4. Double exponential functions are not necessarily  $(L_0, L_1)$  smooth. Consider the double exponential function  $f(x) = e^{(e^x)}$ . Then we have
$$
f ^ {\prime} (x) = e ^ {x} e ^ {(e ^ {x})}, \quad f ^ {\prime \prime} (x) = (e ^ {x} + 1) \cdot f ^ {\prime} (x).
$$
For any  $x\geq \max \left\{\log (L_0 + 1),\log (L_1 + 1)\right\}$  , we can show that
$$
\left| f ^ {\prime \prime} (x) \right| > \left(L _ {1} + 1\right) f ^ {\prime} (x) > L _ {0} + L _ {1} \left| f ^ {\prime} (x) \right|,
$$
D.2 Proof of Lemma 3.4
Before proving Lemma 3.4, we need the following lemma that generalizes (a special case of) Gronwall's inequality.
Lemma D.2. Let  $u:[a,b] \to [0,\infty)$  and  $\ell :[0,\infty) \to (0,\infty)$  be two continuous functions. Suppose  $u^{\prime}(t) \leq \ell(u(t))$  for all  $t \in (a,b)$ . Denote function  $\phi(w) \coloneqq \int \frac{1}{\ell(w)} \, dw$ . We have for all  $t \in [a,b]$ ,
$$
\phi (u (t)) \leq \phi (u (a)) - a + t.
$$
Proof of Lemma D.2. First, by definition, we know that  $\phi$  is increasing since  $\phi' = \frac{1}{\ell} > 0$ . Let function  $v$  be the solution of the following differential equation
$$
v ^ {\prime} (t) = \ell (v (t)) \forall t \in (a, b), \quad v (a) = u (a). \tag {14}
$$
It is straightforward to verify that the solution to (14) satisfies
$$
\phi (v (t)) - t = \phi (u (a)) - a.
$$
Then it suffices to show  $\phi(u(t)) \leq \phi(v(t)) \forall t \in [a, b]$ . Note that
$$
(\phi (u (t)) - \phi (v (t))) ^ {\prime} = \phi^ {\prime} (u (t)) u ^ {\prime} (t) - \phi^ {\prime} (v (t)) v ^ {\prime} (t) = \frac {u ^ {\prime} (t)}{\ell (u (t))} - \frac {v ^ {\prime} (t)}{\ell (v (t))} \leq 0,
$$
where the inequality is because  $u'(t) \leq \ell(u(t))$  by the assumption of this lemma and  $v'(t) = \ell(v(t))$  by (14). Since  $\phi(u(a)) - \phi(v(a)) = 0$ , we know for all  $t \in [a, b]$ ,  $\phi(u(t)) \leq \phi(v(t))$ .
With Lemma D.2, one can bound the gradient norm within a small enough neighborhood of a given point as in the following lemma.
Lemma D.3. Suppose  $f$  is  $(\rho, L_0, L_\rho)$  smooth for some  $\rho, \rho, L_0, L_\rho \geq 0$ . For any  $a > 0$  and points  $x, y \in \mathrm{dom}(f)$  satisfying  $\| y - x \| \leq \frac{a}{L_0 + L_\rho(\| \nabla f(x) \| + a)^\rho}$ , we have
$$
\| \nabla f (y) \| \leq \| \nabla f (x) \| + a.
$$
Proof of Lemma D.3. Denote functions  $z(t) \coloneqq (1 - t)x + ty$  and  $u(t) \coloneqq \| \nabla f(z(t)) \|$  for  $0 \leq t \leq 1$ . Note that for any  $0 \leq t \leq s \leq 1$ , by triangle inequality,
$$
u (s) - u (t) \leq \| \nabla f (z (s)) - \nabla f (z (t)) \|.
$$
We know that  $u(t) = \| \nabla f(z(t)) \|$  is differentiable since  $f$  is second order differentiable<sup>2</sup>. Then we have
$$
\begin{array}{l} u ^ {\prime} (t) = \lim _ {s \downarrow t} \frac {u (s) - u (t)}{s - t} \leq \lim _ {s \downarrow t} \frac {\| \nabla f (z (s)) - \nabla f (z (t)) \|}{s - t} = \left\| \lim _ {s \downarrow t} \frac {\nabla f (z (s)) - \nabla f (z (t))}{s - t} \right\| \\ \leq \left\| \nabla^ {2} f (z (t)) \right\| \| y - x \| \leq \left(L _ {0} + L _ {\rho} u (t) ^ {\rho}\right) \| y - x \|. \\ \end{array}
$$
Let  $\phi (w)\coloneqq \int_0^w\frac{1}{(L_0 + L_\rho v^\rho)\|y - x\|} dv$  . By Lemma D.2, we know that
$$
\phi \left(\| \nabla f (y) \|\right) = \phi (u (1)) \leq \phi (u (0)) + 1 = \phi \left(\| \nabla f (x) \|\right) + 1.
$$
Denote  $\psi (w)\coloneqq \int_0^w\frac{1}{(L_0 + L_\rho v^\rho)} dv = \phi (w)\cdot \| y - x\|$ . We have
$$
\begin{array}{l} \psi \left(\| \nabla f (y) \|\right) \leq \psi \left(\| \nabla f (x) \|\right) + \| y - x \| \\ \leq \psi (\| \nabla f (x) \|) + \frac {a}{L _ {0} + L _ {\rho} (\| \nabla f (x) \| + a) ^ {\rho}} \\ \leq \int_ {0} ^ {\| \nabla f (x) \|} \frac {1}{\left(L _ {0} + L _ {\rho} v ^ {\rho}\right)} d v + \int_ {\| \nabla f (x) \|} ^ {\| \nabla f (x) \| + a} \frac {1}{\left(L _ {0} + L _ {\rho} v ^ {\rho}\right)} d v \\ = \psi (\| \nabla f (x) \| + a) \\ \end{array}
$$
Since  $\psi$  is increasing, we have  $\| \nabla f(y)\| \leq \| \nabla f(x)\| +a$ .
Here we assume  $u(t) > 0$  for  $0 < t < 1$ . Otherwise, we can define  $t_m = \sup \{0 < t < 1 \mid u(t) = 0\}$  and consider the interval  $[t_m, 1]$  instead.
With Lemma D.3, we are ready to prove Lemma 3.4.
Proof of Lemma 3.4. Denote  $z(t) \coloneqq (1 - t)x + ty$  for some  $y \in \mathbb{R}^d$  satisfying  $\| y - x \| \leq \frac{a}{L_0 + L_\rho(\|\nabla f(x)\| + a)^\rho}$ . We first show  $y \in \operatorname{dom}(f)$  by contradiction. Suppose  $y \notin \operatorname{dom}(f)$ , let us define  $t_{\mathrm{b}} \coloneqq \inf \{0 \leq t \leq 1 \mid z(t) \notin \mathcal{X}\}$  and  $z_{\mathrm{b}} \coloneqq z(t_{\mathrm{b}})$ . Then we know  $z_{\mathrm{b}}$  is a boundary point of  $\mathcal{X}$ . Since  $f$  is a closed function with an open domain, we have
$$
\lim  _ {t \uparrow t _ {\mathrm {b}}} f (z (t)) = \infty . \tag {15}
$$
On the other hand, by the definition of  $t_{\mathrm{b}}$ , we know  $z(t) \in \mathcal{X}$  for every  $0 \leq t < t_{\mathrm{b}}$ . Then by Lemma D.3, for all  $0 \leq t < t_{\mathrm{b}}$ , we have  $\| \nabla f(z(t)) \| \leq \| \nabla f(x) \| + a$ . Therefore for all  $0 \leq t < t_{\mathrm{b}}$
$$
\begin{array}{l} f (z (t)) \leq f (x) + \int_ {0} ^ {t} \left\langle \nabla f (z (s)), y - x \right\rangle d s \\ \leq f (x) + \left(\| \nabla f (x) \| + a\right) \cdot \| y - x \| \\ <   \infty , \\ \end{array}
$$
which contradicts with (15). Therefore we have shown  $y \in \operatorname{dom}(f)$ . We have
$$
\begin{array}{l} \| \nabla f (y) - \nabla f (x) \| = \left\| \int_ {0} ^ {1} \nabla^ {2} f (z (t)) \cdot (y - x) d t \right\| \\ \leq \| y - x \| \cdot \int_ {0} ^ {1} \left(L _ {0} + L _ {\rho} \| \nabla f (z (t)) \| ^ {\rho}\right) d t \\ \leq \| y - x \| \cdot \left(L _ {0} + L _ {\rho} \cdot \left(\| \nabla f (x) \| + a\right) ^ {\rho}\right) \\ \end{array}
$$
where the last inequality is due to Lemma D.3.
D.3 Proof of Lemma 5.1
Proof of Lemma 5.1. Denote  $G \coloneqq \| \nabla f(x) \|$  and  $L \coloneqq 3L_0 + 4L_\rho G^\rho$ . Let  $y \coloneqq x - \frac{1}{2L} \nabla f(x)$ . Then we have
$$
\| y - x \| = \frac {G}{2 L} = \frac {G}{6 L _ {0} + 8 L _ {\rho} G ^ {\rho}} \leq \min \left\{\frac {1}{5 L _ {\rho} G ^ {\rho - 1}}, \frac {1}{5 (L _ {0} ^ {\rho - 1} L _ {\rho}) ^ {1 / \rho}} \right\} =: r,
$$
where the inequality can be easily verified considering both cases of  $G \leq (L_0 / L_\rho)^{1 / \rho}$  and  $G \geq (L_0 / L_\rho)^{1 / \rho}$ . Then based on Corollary 5.2, we have  $y \in \mathrm{dom}(f)$  and
$$
f ^ {*} - f (x) \leq f (y) - f (x) \leq \left\langle \nabla f (x), y - x \right\rangle + \frac {L}{2} \| y - x \| ^ {2} = \frac {3 L G ^ {2}}{8} \leq \frac {L G ^ {2}}{3},
$$
D.4 Proof of Corollary 5.2
Proof of Corollary 5.2. First, Lemma 3.4 states that for any  $a > 0$
$$
\| y - x \| \leq \frac {a}{L _ {0} + L _ {\rho} \cdot (\| \nabla f (x) \| + a) ^ {\rho}} \Rightarrow \| \nabla f (y) - \nabla f (x) \| \leq \left(L _ {0} + L _ {\rho} \cdot \left(\| \nabla f (x) \| + a\right) ^ {\rho}\right) \| y - x \|.
$$
If  $\| \nabla f(x) \| \leq G$ , we choose  $a = \max \{G, (L_0 / L_\rho)^{1 / \rho}\}$ . Then it is straightforward to verify that
$$
\frac {a}{L _ {0} + L _ {\rho} \cdot (\| \nabla f (x) \| + a) ^ {\rho}} \geq \min  \left\{\frac {1}{5 L _ {\rho} G ^ {\rho - 1}}, \frac {1}{5 \left(L _ {0} ^ {\rho - 1} L _ {\rho}\right) ^ {1 / \rho}} \right\} =: r,
$$
$$
L _ {0} + L _ {\rho} \cdot \left(\| \nabla f (x) \| + a\right) ^ {\rho} \leq 3 L _ {0} + 4 L _ {\rho} G ^ {\rho} =: L.
$$
Therefore we have shown for any  $x, y$  satisfying  $\|y - x\| \leq r$ ,
$$
\left\| \nabla f (y) - \nabla f (x) \right\| \leq L \| y - x \|. \tag {16}
$$
Next, let  $z(t) \coloneqq (1 - t)x + ty$  for  $0 \leq t \leq 1$ . We know
$$
\begin{array}{l} f (y) - f (x) = \int_ {0} ^ {1} \left\langle \nabla f (z (t), y - x \right\rangle d t \\ = \int_ {0} ^ {1} \left\langle \nabla f (x), y - x \right\rangle + \left\langle \nabla f (z (t)) - \nabla f (x), y - x \right\rangle d t \\ \leq \left\langle \nabla f (x), y - x \right\rangle + \int_ {0} ^ {1} L \| z (t) - x \| \| y - x \| d t \\ = \left\langle \nabla f (x), y - x \right\rangle + L \| y - x \| ^ {2} \int_ {0} ^ {1} t d t \\ = \left\langle \nabla f (x), y - x \right\rangle + \frac {1}{2} L \left\| y - x \right\| ^ {2}, \\ \end{array}
$$
where the inequality is due to (16).
E Convergence analysis of Adam
In this section, we provide detailed convergence analysis of Adam. We will focus on proving Theorem 4.1 under the bounded noise assumption (Assumption 3) in most parts of this section except Appendix E.6 where we will show how to generalize the results to noise with sub-Gaussian norm (Assumption 4) and provide the proof of Theorem 4.2.
For completeness, we repeat some important technical definitions here. First, we define
$$
\epsilon_ {t} := \hat {m} _ {t} - \nabla f (x _ {t}) \tag {17}
$$
as the deviation of the re-scaled momentum from the actual gradient. Given a large enough constant  $G$  defined in Theorem 4.1, denoting  $F = \frac{G^2}{3(3L_0 + 4L_\rho G^\rho)}$ , we formally define the stopping time  $\tau$  as
$$
\tau := \min  \left\{t \mid f \left(x _ {t}\right) - f ^ {*} > F \right\} \wedge (T + 1),
$$
i.e.,  $\tau$  is the first time when the sub-optimality gap is strictly greater than  $F$ , truncated at  $T + 1$  to make sure it is bounded in order to apply Lemma C.2. Based on Lemma 5.1 and the discussions below it, we know that if  $t < \tau$ , we have both  $f(x_{t}) - f^{*} \leq F$  and  $\| \nabla f(x_{t}) \| \leq G$ . It is clear to see that  $\tau$  is a stopping time<sup>3</sup> with respect to  $\{\xi_{t}\}_{t \geq 1}$  because the event  $\{\tau \geq t\}$  is a function of  $\{\xi_{s}\}_{s < t}$  and independent of  $\{\xi_{s}\}_{s \geq t}$ . Next, let
$$
h _ {t} := \frac {\eta}{\sqrt {\hat {v} _ {t}} + \lambda}
$$
be the stepsize vector and  $H_{t} \coloneqq \mathrm{diag}(h_{t})$  be the diagonal stepsize matrix. Then the update rule can be written as
$$
x _ {t + 1} = x _ {t} - h _ {t} \odot \hat {m} _ {t} = x _ {t} - H _ {t} \hat {m} _ {t}.
$$
Finally, as in Corollary 5.2 and Lemma 5.3, we define the following constants.
$$
r := \min  \left\{\frac {1}{5 L _ {\rho} G ^ {\rho - 1}}, \frac {1}{5 (L _ {0} ^ {\rho - 1} L _ {\rho}) ^ {1 / \rho}} \right\},
$$
$$
L := 3 L _ {0} + 4 L _ {\rho} G ^ {\rho},
$$
$$
D := 2 G / \lambda .
$$
E.1 Equivalent update rule of Adam
The bias correction steps in Lines 7-8 make Algorithm 1 a bit complicated. In the following proposition, we provide an equivalent yet simpler update rule of Adam.
Indeed,  $\tau - 1$  is also a stopping time because  $\nabla f(x_{t})$  only depends on  $\{\xi_s\}_{s < t}$ , but that is unnecessary for our analysis.