SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 4.3 gives a linear convergence rate and the  $\mathcal{O}((\eta \mu)^{-1}\log (1 / \epsilon))$  gradient complexity to achieve an  $\epsilon$ -sub-optimal point. Note that for  $\ell$ -smooth functions, we have  $\frac{r(G)}{G} = \frac{1}{L}$  (see Remark 3.4), which means we can choose  $\eta = \frac{1}{2L}$ . Then we obtain the  $\mathcal{O}(\kappa \log (1 / \epsilon))$  rate, where  $\kappa \coloneqq L / \mu$  is the local condition number around the initial point  $x_0$ . For standard Lipschitz smooth functions, it reduces to the classical rate of gradient descent.
4.2 Nesterov's accelerated gradient method
In the case of convex and standard Lipschitz smooth functions, it is well known that Nesterov's accelerated gradient method (NAG) achieves the optimal  $\mathcal{O}(1 / T^2)$  rate. In this section, we show that
Algorithm 1: Nesterov's Accelerated Gradient Method (NAG)
input A convex and  $\ell$  -smooth function  $f$  , stepsize  $\eta$  , initial point  $x_0$  1: Initialize  $z_{0} = x_{0}$ $B_{0} = 0$  , and  $A_0 = 1 / \eta$    
2: for  $t = 0,\ldots$  do   
3:  $B_{t + 1} = B_t + \frac{1}{2}\left(1 + \sqrt{4B_t + 1}\right)$    
4:  $A_{t + 1} = B_{t + 1} + 1 / \eta$    
5:  $y_{t} = x_{t} + (1 - A_{t} / A_{t + 1})(z_{t} - x_{t})$    
6:  $x_{t + 1} = y_{t} - \eta \nabla f(y_{t})$    
7:  $z_{t + 1} = z_{t} - \eta (A_{t + 1} - A_{t})\nabla f(y_{t})$    
8: end for
under the  $\ell$ -smoothness condition with a sub-quadratic  $\ell$ , the optimal  $\mathcal{O}(1 / T^2)$  rate can be achieved by a slightly modified version of NAG shown in Algorithm 1, the only difference between which and the classical NAG is that the latter directly sets  $A_{t + 1} = B_{t + 1}$  in Line 4. Formally, we have the following theorem, whose proof is deferred to Appendix C.
Theorem 4.4. Suppose  $f$  is convex and  $\ell$ -smooth where  $\ell$  is sub-quadratic. Then there always exists a constant  $G$  satisfying  $G \geq \max \left\{8\sqrt{\ell(2G)((f(x_0) - f^*) + \|x_0 - x^*\|^2)},\| \nabla f(x_0)\| \right\}$ . Denote  $L \coloneqq \ell (2G)$  and choose  $\eta \leq \min \left\{\frac{1}{16L^2},\frac{1}{2L}\right\}$ . The iterates generated by Algorithm 1 satisfy
$$
f (x _ {T}) - f ^ {*} \leq \frac {4 (f (x _ {0}) - f ^ {*}) + 4 \| x _ {0} - x ^ {*} \| ^ {2}}{\eta T ^ {2} + 4}.
$$
It is easy to note that Theorem 4.4 achieves the accelerated  $\mathcal{O}(1 / T^2)$  convergence rate, or equivalently the  $\mathcal{O}(1 / \sqrt{\epsilon})$  gradient complexity to find an  $\epsilon$ -sub-optimal point, which is optimal among gradient-based methods [Nesterov, 2003].
In order to prove Theorem 4.4, we also use induction to show the gradients along the trajectory of Algorithm 1 are bounded by  $G$ . However, unlike gradient descent, the gradient norm is no longer a potential function or monotonically non-increasing, which makes the induction analysis more challenging. Suppose that we have shown  $\| \nabla f(y_s) \| \leq G$  for  $s < t$ . To complete the induction, it suffices to prove  $\| \nabla f(y_t) \| \leq G$ . Since  $x_t = y_{t-1} - \eta \nabla f(y_{t-1})$  is a gradient descent step by Line 6 of Algorithm 1, Lemma 4.1 directly shows  $\| \nabla f(x_t) \| \leq G$ . In order to also bound  $\| \nabla f(y_t) \|$ , we try to control  $\| y_t - x_t \|$ , which is the most challenging part of our proof. Since  $y_t - x_t$  can be expressed as a linear combination of past gradients  $\{\nabla f(y_s)\}_{s < t}$ , it might grow linearly with  $t$  if we simply apply  $\| \nabla f(y_s) \| \leq G$  for  $s < t$ . Fortunately, Lemma 3.5 allows us to control the gradient norm with the function value. Thus if the function value is decreasing sufficiently fast, which can be shown by following the standard Lyapunov analysis of NAG, we are able to obtain a good enough bound on  $\| \nabla f(y_s) \|$  for  $s < t$ , which allows us to control  $\| y_t - x_t \|$ . We defer the detailed proof to Appendix C.
Note that Theorem 4.4 requires a smaller stepsize  $\eta = \mathcal{O}(1 / L^2)$ , compared to the classical  $\mathcal{O}(1 / L)$  stepsize for standard Lipschitz smooth functions. The reason is we require a small enough stepsize to get a good enough bound on  $\| y_t - x_t\|$ . However, if the function is further assumed to be  $\ell$ -smooth with a sub-linear  $\ell$ , the requirement of stepsize can be relaxed to  $\eta = \mathcal{O}(1 / L)$ , similar to the classical requirement. See Appendix C for the details.
In the strongly convex setting, we can also prove convergence of NAG with different  $\{A_t\}_{t \geq 0}$  parameters when  $f$  is  $\ell$ -smooth with a sub-quadratic  $\ell$ , or  $(\rho, L_0, L_\rho)$ -smooth with  $\rho < 2$ . The rate can be further improved when  $\rho$  becomes smaller. However, since the constants  $G$  and  $L$  are different for GD and NAG, it is not clear whether the rate of NAG is faster than that of GD in the strongly convex setting. We will present the detailed result and analysis in Appendix D.
5 Non-convex setting
In this section, we present convergence results of gradient descent (GD) and stochastic gradient descent (SGD) in the non-convex setting.
5.1 Gradient descent
Similar to the convex setting, we still want to bound the gradients along the trajectory. However, in the non-convex setting, the gradient norm is not necessarily non-increasing. Fortunately, similar to the classical analyses, the function value is still non-increasing and thus a potential function, as formally shown in the following lemma, whose proof is deferred to Appendix E.
Lemma 5.1. Suppose  $f$  is  $\ell$ -smooth where  $\ell$  is sub-quadratic. For any given  $F \geq 0$ , let  $G \coloneqq \sup \left\{u \geq 0 \mid u^2 \leq 2\ell(2u) \cdot F\right\}$  and  $L \coloneqq \ell(2G)$ . For any  $x \in \mathcal{X}$  satisfying  $f(x) - f^* \leq F$ , define  $x^+ \coloneqq x - \eta\nabla f(x)$  where  $\eta \leq 2/L$ , we have  $x^+ \in \mathcal{X}$  and  $f(x^+) \leq f(x)$ .
Then using a standard induction argument, we can show  $f(x_{t}) \leq f(x_{0})$  for all  $t \geq 0$ . According to Corollary 3.6, it implies bounded gradients along the trajectory. Therefore, we can show convergence of gradient descent as in the following theorem, whose proof is deferred to Appendix E.
Theorem 5.2. Suppose  $f$  is  $\ell$ -smooth where  $\ell$  is sub-quadratic. Let  $G := \sup \left\{u \geq 0 \mid u^2 \leq 2\ell(2u) \cdot (f(x_0) - f^*)\right\}$  and  $L := \ell(2G)$ . If  $\eta \leq 1/L$ , the iterates generated by (3) satisfy  $\| \nabla f(x_t) \| \leq G$  for all  $t \geq 0$  and
$$
\frac {1}{T} \sum_ {t <   T} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 (f (x _ {0}) - f ^ {*})}{\eta T}.
$$
It is clear that Theorem 5.2 gives the classical  $\mathcal{O}(1 / \epsilon^2)$  gradient complexity to achieve an  $\epsilon$ -stationary point, which is optimal as it matches the lower bound in [Carmon et al., 2017].
5.2 Stochastic gradient descent
In this part, we present the convergence result for stochastic gradient descent defined as follows.
$$
x _ {t + 1} = x _ {t} - \eta g _ {t}, \tag {4}
$$
where  $g_{t}$  is an estimate of the gradient  $\nabla f(x_{t})$ . We consider the following standard assumption on the gradient noise  $\epsilon_t\coloneqq g_t - \nabla f(x_t)$
Assumption 4.  $\mathbb{E}_{t - 1}[\epsilon_t] = 0$  and  $\mathbb{E}_{t - 1}\left[\| \epsilon_t\| ^2\right]\leq \sigma^2$  for some  $\sigma \geq 0$ , where  $\mathbb{E}_{t - 1}$  denotes the expectation conditioned on  $\{g_s\}_{s < t}$ .
Under Assumption 4, we can obtain the following theorem.
Theorem 5.3. Suppose  $f$  is  $\ell$ -smooth where  $\ell$  is sub-quadratic. For any  $0 < \delta < 1$ , we denote  $F := 8(f(x_0) - f^* + \sigma) / \delta$  and  $G := \sup \{u \geq 0 \mid u^2 \leq 2\ell(2u) \cdot F\} < \infty$ . Denote  $L := \ell(2G)$  and choose  $\eta \leq \min \left\{\frac{1}{2L}, \frac{1}{4G\sqrt{T}}\right\}$  and  $T \geq \frac{F}{\eta\epsilon^2}$  for any  $\epsilon > 0$ . Then with probability at least  $1 - \delta$ , the iterates generated by (4) satisfy  $\| \nabla f(x_t) \| \leq G$  for all  $t < T$  and
$$
\frac {1}{T} \sum_ {t <   T} \| \nabla f (x _ {t}) \| ^ {2} \leq \epsilon^ {2}.
$$
As we choose  $\eta = \mathcal{O}(1 / \sqrt{T})$ , Theorem 5.3 gives the classical  $\mathcal{O}(1 / \epsilon^4)$  gradient complexity, where we ignore non-leading terms. This rate is optimal as it matches the lower bound in [Arjevani et al., 2019]. The key to its proof is again to bound the gradients along the trajectory. However, bounding gradients in the stochastic setting is much more challenging than in the deterministic setting, especially with the heavy-tailed noise in Assumption 4. We briefly discuss some of the challenges as well as our approach below and defer the detailed proof of Theorem 5.3 to Appendix F.
First, due to the existence of heavy-tailed gradient noise as considered in Assumption 4, neither the gradient nor the function values is non-increasing. The induction analyses we have used in the deterministic setting hardly work. In addition, to apply Lemma 3.3, we need to control the update at each step and make sure  $\| x_{t + 1} - x_t\| = \eta \| g_t\| \leq G / L$ . However,  $g_{t}$  might be unbounded due to the potentially unbounded gradient noise.
To overcome these challenges, we define the following random variable  $\tau$ .
$$
\begin{array}{l} \tau_ {1} := \min  \left\{t \mid f \left(x _ {t + 1}\right) - f ^ {*} > F \right\} \wedge T, \\ \tau_ {2} := \min  \left\{t \left| \| \epsilon_ {t} \| > \frac {G}{5 \eta L} \right. \right\} \wedge T, \tag {5} \\ \tau := \min  \left\{\tau_ {1}, \tau_ {2} \right\}, \\ \end{array}
$$
where we use  $a \wedge b$  to denote  $\min \{a, b\}$  for any  $a, b \in \mathbb{R}$ . Then at least before time  $\tau$ , we know that the function value and gradient noise are bounded, where the former also implies bounded gradients according to Corollary 3.6. Therefore, it suffices to show the probability of  $\tau < T$  is small, which means with a high probability,  $\tau = T$  and thus gradients are always bounded before  $T$ .
Since both the gradient and noise are bounded for  $t < \tau$ , it is straightforward to bound the update  $\| x_{t + 1} - x_t\|$ , which allows us to use Lemma 3.3 and other useful properties. However, it is still non-trivial to upper bound  $\mathbb{E}[f(x_{\tau}) - f^{*}]$  as  $\tau$  is a random variable instead of a fixed time step. Fortunately,  $\tau$  is a stopping time with nice properties. That is because both  $f(x_{t + 1})$  and  $\epsilon_t = g_t - \nabla f(x_t)$  only depend on  $\{g_s\}_{s\leq t}$ , i.e., the stochastic gradients up to  $t$ . Therefore, for any fixed  $t$ , the events  $\{\tau >t\}$  only depend on  $\{g_s\}_{s\leq t}$ , which show  $\tau$  is a stopping time. Then with a careful analysis, we are still able to obtain an upper bound on  $\mathbb{E}[f(x_{\tau}) - f^{*}] = \mathcal{O}(1)$ .
On the other hand,  $\tau < T$  means either  $\tau = \tau_{1} < T$  or  $\tau = \tau_{2} < T$ . If  $\tau = \tau_{1} < T$ , by its definition, we know  $f(x_{\tau + 1}) - f^{*} > F$ . Roughly speaking, it also suggests  $f(x_{\tau}) - f^{*} > F / 2$ . If we choose  $F$  such that it is much larger than the upper bound on  $\mathbb{E}[f(x_{\tau}) - f^{*}]$  we just obtained, by Markov's inequality, we can show the probability of  $\tau = \tau_{1} < T$  is small. In addition, by union bound and Chebyshev's inequality, the probability of  $\tau_{2} < T$  can also be bounded by a small constant. Therefore, we have shown  $\tau < T$ . Then the rest of the analysis is not too hard following the classical analysis.
5.3 Reconciliation with existing lower bounds
In this section, we reconcile our convergence results for constant-stepsize GD/SGD in the non-convex setting with existing lower bounds in [Zhang et al., 2019] and [Wang et al., 2022], based on which the authors claim that adaptive methods such as GD/SGD with clipping and Adam are provably faster than non-adaptive GD/SGD. This may seem to contradict our convergence results. In fact, we show that any gain in adaptive methods is at most by constant factors, as GD and SGD already achieve the optimal rates in the non-convex setting.
[Zhang et al., 2019] provides both upper and lower complexity bounds for constant-stepsize GD for  $(L_0, L_1)$ -smooth functions, and shows that its complexity is  $\mathcal{O}(M\epsilon^{-2})$ , where
$$
M := \sup  \left\{\| \nabla f (x) \| \mid f (x) \leq f \left(x _ {0}\right) \right\}
$$
is the supremum gradient norm below the level set of the initial function value. If  $M$  is very large, then the  $\mathcal{O}(M\epsilon^{-2})$  complexity can be viewed as a negative result, and as evidence that constant-stepsize GD can be slower than GD with gradient clipping, since in the latter case, they obtain the  $\mathcal{O}(\epsilon^{-2})$  complexity without  $M$ . However, based on our Corollary 3.6, their  $M$  can be actually bounded by our  $G$ , which is a constant. Therefore, the gain in adaptive methods is at most by constant factors.
[Wang et al., 2022] further provides a lower bound which shows non-adaptive GD may diverge for some examples. However, their counter-example does not allow the stepsize to depend on the initial sub-optimality gap. In contrast, our stepsize  $\eta$  depends on the effective smoothness constant  $L$ , which depends on the initial sub-optimality gap through  $G$ . Therefore, there is no contradiction here either. We should point out that in the practice of training neural networks, the stepsize is usually tuned after fixing the loss function and initialization, so it does depend on the problem instance and initialization.
5.4 Lower bound
For  $(\rho, L_0, L_\rho)$ -smooth functions with  $\rho < 2$ , it is easy to verify that the constant  $G$  in both Theorem 5.2 and Theorem 5.3 is a polynomial function of problem-dependent parameters like  $L_0, L_\rho, f(x_0) - f^*, \sigma$ , etc. In other words, GD and SGD are provably efficient methods in the non-convex setting for  $\rho < 2$ . In this section, we show that the requirement of  $\rho < 2$  is necessary in the non-convex setting with the lower bound for GD in the following Theorem 5.4, whose proof is deferred in Appendix G. Since SGD reduces to GD when there is no gradient noise, it is also a lower bound for SGD.
Theorem 5.4. Given  $L_0, L_2, G_0, \Delta_0 > 0$  satisfying  $L_2\Delta_0 \geq 10$ , for any  $\eta \geq 0$ , there exists a  $(2, L_0, L_2)$ -smooth function  $f$  that satisfies Assumptions 1 and 2, and initial point  $x_0$  that satisfies  $\| \nabla f(x_0) \| \leq G_0$  and  $f(x_0) - f^* \leq \Delta_0$ , such that gradient descent with stepsize  $\eta$  (3) either cannot reach a 1-stationary point or takes at least  $\exp(L_2\Delta_0/8)/6$  steps to reach a 1-stationary point.
In this paper, we generalize the standard Lipschitz smoothness as well as the  $(L_0,L_1)$ -smoothness [Zhang et al., 2020] conditions to the  $\ell$ -smoothness condition, and develop a new approach for analyzing the convergence under this condition. The approach uses different techniques for several methods and settings to bound the gradient along the optimization trajectory, which allows us to obtain stronger results for both convex and non-convex problems. We obtain the classical rates for GD/SGD/NAG methods in the convex and/or non-convex setting. Our results challenge the folklore belief on the necessity of adaptive methods for generalized smooth functions.
There are several interesting future directions following this work. First, the  $\ell$ -smoothness can perhaps be further generalized by allowing  $\ell$  to also depend on potential functions in each setting, besides the gradient norm. In addition, it would also be interesting to see if the techniques of bounding gradients along the trajectory that we have developed in this and the concurrent work [Li et al., 2023] can be further generalized to other methods and problems and to see whether more efficient algorithms can be obtained. Finally, although we justified the necessity of the requirement of  $\ell$ -smoothness with a sub-quadratic  $\ell$  in the non-convex setting, it is not clear whether it is also necessary for NAG in the convex setting, another interesting open problem.
This work was supported, in part, by the MIT-IBM Watson AI Lab and ONR Grants N00014-20-1-2394 and N00014-23-1-2299. We also acknowledge support from DOE under grant DE-SC0022199, and NSF through awards DMS-2031883, DMS-1953181, and DMS-2022448 (TRIPODS program).
Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake E. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199:165 - 214, 2019.
Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Yair Carmon, John C. Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming, pages 1-50, 2017.
Ziyi Chen, Yi Zhou, Yingbin Liang, and Zhaosong Lu. Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. arXiv preprint arXiv:2303.02854, 2023.
Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in Neural Information Processing Systems, 35:9955-9968, 2022.
Alexandre d'Aspremont, Damien Scieur, and Adrian Taylor. Acceleration methods. Foundations and Trends® in Optimization, 5(1-2):1-245, 2021. ISSN 2167-3888. doi: 10.1561/2400000036. URL http://dx.doi.org/10.1561/2400000036.
Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive sgd. ArXiv, abs/2302.06570, 2023.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. Convergence of adam under relaxed assumptions. arXiv preprint arXiv:2304.13972, 2023.
Arkadj Semenovič Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. Wiley-Interscience, 1983.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
Jiang Qian, Yuren Wu, Bojin Zhuang, Shaojun Wang, and Jing Xiao. Understanding gradient clipping in incremental gradient methods. In International Conference on Artificial Intelligence and Statistics, 2021.
Amirhossein Reisizadeh, Haochuan Li, Subhro Das, and Ali Jabbabaie. Variance-reduced clipping for non-convex optimization. ArXiv, abs/2303.00883, 2023.
Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, and Wei Chen. Provable adaptivity in adam. arXiv preprint arXiv:2208.09900, 2022.
Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. ArXiv, abs/2010.02519, 2020.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jabbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:1905.11881, 2019.
Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences, 64, 2021.
A Proofs related to generalized smoothness
In this section, we provide the proofs of propositions and lemmas related to the generalized smoothness condition in Definition 1 or 2. First, in Appendix A.1, we justify the examples we discussed in Section 3. Next, we provide the detailed proof of Proposition 3.2 in Appendix A.2. Finally, we provide the proofs of the useful properties of generalized smoothness in Appendix A.3, including Lemma 3.3, Lemma 3.5, and Corollary 3.6 stated in Section 3.1.2.
A.1 Justification of examples in Section 3
In this section, we justify the univariate examples of  $(\rho, L_0, L_\rho)$ -smooth functions listed in Table 2 and also provide the proof of Propositions 3.7.
First, it is well-known that all quadratic functions have bounded Hessian and are Lipschitz smooth, corresponding to  $\rho = 0$ . Next, [Zhang et al., 2019, Lemma 2] shows that any univariate polynomial is  $(L_0, L_1)$ -smooth, corresponding to  $\rho = 1$ . Then, regarding the exponential function  $f(x) = a^x$  where  $a > 1$ , we have  $f'(x) = \log(a)a^x$  and  $f''(x) = \log(a)^2a^x = \log(a)f'(x)$ , which implies  $f$  is  $(1,0,\log(a))$ -smooth. Similarly, by standard calculations, it is straightforward to verify that logarithmic functions and  $x^p, p \neq 1$  are also  $(\rho, L_0, L_\rho)$ -smooth with  $\rho = 2$  and  $\rho = \frac{p - 2}{p - 1}$  respectively. So far we have justified all the examples in Table 2 except double exponential functions  $a^{(b^x)}$  and rational functions, which will be justified rigorously by the two propositions below.
First, for double exponential functions in the form of  $f(x) = a^{(b^x)}$  where  $a, b > 1$ , we have the following proposition, which shows  $f$  is  $(\rho, L_0, L_\rho)$ -smooth for any  $\rho > 1$ .
Proposition A.1. For any  $\rho > 1$ , the double exponential function  $f(x) = a^{(b^x)}$ , where  $a, b > 1$ , is  $(\rho, L_0, L_\rho)$ -smooth for some  $L_0, L_\rho \geq 0$ . However, it is not necessarily  $(L_0, L_1)$ -smooth for any  $L_0, L_1 \geq 0$ .
Proof of Proposition A.1. By standard calculations, we can obtain
$$
f ^ {\prime} (x) = \log (a) \log (b) b ^ {x} a ^ {(b ^ {x})}, \quad f ^ {\prime \prime} (x) = \log (b) (\log (a) b ^ {x} + 1) \cdot f ^ {\prime} (x). \tag {6}
$$
Note that if  $\rho > 1$ ,
$$
\lim _ {x \to + \infty} \frac {| f ^ {\prime} (x) | ^ {\rho}}{| f ^ {\prime \prime} (x) |} = \lim _ {x \to + \infty} \frac {| f ^ {\prime} (x) | ^ {\rho - 1}}{\log (b) (\log (a) b ^ {x} + 1)} = \lim _ {y \to + \infty} \frac {(\log (a) \log (b) y) ^ {\rho - 1} a ^ {(\rho - 1) y}}{\log (b) (\log (a) y + 1)} = \infty ,
$$
where the first equality is a direct calculation based on (6); the second equality uses change of variable  $y = b^{x}$ ; and the last equality is because exponential functions grow faster than affine functions. Therefore, for any  $L_{\rho} > 0$ , there exists  $x_0 \in \mathbb{R}$  such that  $|f''(x)| \leq L_{\rho} |f'(x)|^\rho$  if  $x > x_0$ . Next, note that  $\lim_{x \to -\infty} f''(x) = 0$ . Then for any  $\lambda_1 > 0$ , there exists  $x_1 \in \mathbb{R}$  such that  $|f''(x)| \leq \lambda_1$  if  $x < x_1$ . Also, since  $f''$  is continuous, by Weierstrass's Theorem, we have  $|f''(x)| \leq \lambda_2$  if  $x_1 \leq x \leq x_0$  for some  $\lambda_2 > 0$ . Then denoting  $L_0 = \max \{\lambda_1, \lambda_2\}$ , we know  $f$  is  $(\rho, L_0, L_\rho)$ -smooth.
Next, to show  $f$  is not necessarily  $(L_0, L_1)$ -smooth, consider the specific double exponential function  $f(x) = e^{(e^x)}$ . Then we have
For any  $x \geq \max \left\{\log (L_0 + 1), \log (L_1 + 1)\right\}$ , we can show that
In the next proposition, we show that any univariate rational function  $f(x) = P(x) / Q(x)$ , where  $P$  and  $Q$  are two polynomials, is  $(\rho, L_0, L_\rho)$ -smooth with  $\rho = 1.5$ .
Proposition A.2. The rational function  $f(x) = P(x) / Q(x)$ , where  $P$  and  $Q$  are two polynomials, is  $(1.5, L_0, L_{1.5})$ -smooth for some  $L_0, L_{1.5} \geq 0$ . However, it is not necessarily  $(\rho, L_0, L_\rho)$ -smooth for any  $\rho < 1.5$  and  $L_0, L_\rho \geq 0$ .
Proof of Proposition A.2. Let  $f(x) = P(x) / Q(x)$  where  $P$  and  $Q$  are two polynomials. Then the partial fractional decomposition of  $f(x)$  is given by
where  $w(x)$  is a polynomial,  $A_{ir}, B_{ir}, C_{ir}, a_i, b_i, c_i$  are all real constants satisfying  $b_i^2 - 4c_i < 0$  for each  $1 \leq i \leq n$  which implies  $x^2 + b_i x + c_i > 0$  for all  $x \in \mathbb{R}$ . Assume  $j_i \geq 1$  and  $A_{ij_i} \neq 0$  without loss of generality. Then we know  $f$  has only finite singular points  $\{a_i\}_{1 \leq i \leq m}$  and has continuous first and second order derivatives at all other points. To simplify notation, denote