SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haochuan Li*
Jian Qian*
jianqian@mit.edu
Yi Tian
yitian@mit.edu
Classical analysis of convex and non-convex optimization methods often requires the Lipschitz continuity of the gradient, which limits the analysis to functions bounded by quadratics. Recent work relaxed this requirement to a non-uniform smoothness condition with the Hessian norm bounded by an affine function of the gradient norm, and proved convergence in the non-convex setting via gradient clipping, assuming bounded noise. In this paper, we further generalize this non-uniform smoothness condition and develop a simple, yet powerful analysis technique that bounds the gradients along the trajectory, thereby leading to stronger results for both convex and non-convex optimization problems. In particular, we obtain the classical convergence rates for (stochastic) gradient descent and Nesterov's accelerated gradient method in the convex and/or non-convex setting under this general smoothness condition. The new analysis approach does not require gradient clipping and allows heavy-tailed noise with bounded variance in the stochastic setting.
In this paper, we study the following unconstrained optimization problem
$$
\min  _ {x \in \mathcal {X}} f (x), \tag {1}
$$
where  $\mathcal{X} \subseteq \mathbb{R}^d$  is the domain of  $f$ . Classical textbook analyses [Nemirovskij and Yudin, 1983, Nesterov, 2003] of (1) often require the Lipschitz smoothness condition, which assumes  $\left\| \nabla^2 f(x) \right\| \leq L$  almost everywhere for some  $L \geq 0$  called the smoothness constant. This condition, however, is rather restrictive and only satisfied by functions that are both upper and lower bounded by quadratic functions.
Recently, Zhang et al. [2019] proposed the more general  $(L_0,L_1)$ -smoothness condition, which assumes  $\left\| \nabla^2 f(x)\right\| \leq L_0 + L_1\left\| \nabla f(x)\right\|$  for some constants  $L_0,L_1\geq 0$  , motivated by their extensive language model experiments. This notion generalizes the standard Lipschitz smoothness condition and also contains e.g. univariate polynomial and exponential functions. For non-convex and  $(L_0,L_1)$ -smooth functions, they prove convergence of gradient descent (GD) and stochastic gradient descent (SGD) with gradient clipping and also provide a complexity lower bound for constant-stepsize GD/SGD without clipping. Based on these results, they claim gradient clipping or other forms of adaptivity provably accelerate the convergence for  $(L_0,L_1)$ -smooth functions. Perhaps due to the
lower bound, all the follow-up works under this condition that we are aware of limit their analyses to adaptive methods. Most of these focus on non-convex functions. See Section 2 for more discussions of related works.
In this paper, we significantly generalize the  $(L_0,L_1)$ -smoothness condition to the  $\ell$ -smoothness condition which assumes  $\| \nabla^2 f(x)\| \leq \ell (\| \nabla f(x)\|)$  for some non-decreasing continuous function  $\ell$ . We develop a simple, yet powerful approach, which allows us to obtain stronger results for both convex and non-convex optimization problems when  $\ell$  is sub-quadratic (i.e.,  $\lim_{u\to \infty}\ell (u) / u^2 = 0$ ) or even more general. The  $\ell$ -smooth function class with a sub-quadratic  $\ell$  also contains e.g. univariate rational and double exponential functions. In particular, we prove the convergence of constant-stepsize GD/SGD and Nesterov's accelerated gradient method (NAG) in the convex or non-convex settings. For each method and setting, we obtain the classical convergence rate, under a certain requirement of  $\ell$ . In addition, we relax the assumption of bounded noise to the weaker one of bounded variance with the simple SGD method. See Table 1 for a summary of our results and assumptions for each method and setting. At first glance, our results "contradict" the lower bounds on constant-stepsize GD/SGD in [Zhang et al., 2019, Wang et al., 2022]; this will be reconciled in Section 5.3.
Our approach analyzes boundedness of gradients along the optimization trajectory. The idea behind it can be informally illustrated by the following "circular" reasoning. On the one hand, if gradients along the trajectory are bounded by a constant  $G$ , then the Hessian norms are bounded by the constant  $\ell(G)$ . Informally speaking, we essentially have the standard Lipschitz smoothness condition<sup>2</sup> and can apply classical textbook analyses to prove convergence, which implies that gradients converge to zero. On the other hand, if gradients converge, they must be bounded, since any convergent sequence is bounded. In other words, the bounded gradient condition implies convergence, and convergence also implies the condition back, which forms a circular argument. If we can break this circularity of reasoning in a rigorous way, both the bounded gradient condition and convergence are proved. In this paper, we will show how to break the circularity using induction or contradiction arguments for different methods and settings in Sections 4 and 5. We note that the idea of bounding gradients can be applied to the analysis of other optimization methods, e.g., the concurrent work [Li et al., 2023] by subset of the authors, which uses a similar idea to obtain a rigorous and improved analysis of the Adam method [Kingma and Ba, 2014].
Contributions. In light of the above discussions, we summarize our main contributions as follows.
- We generalize the standard Lipschitz smoothness and also the  $(L_0, L_1)$ -smoothness condition to the  $\ell$ -smoothness condition, and develop a new approach for analyzing convergence under this condition by bounding the gradients along the optimization trajectory.
- We prove the convergence of constant-stepsize GD/SGD/NAG in the convex and non-convex settings, and obtain the classical rates for all of them, as summarized in Table 1.
Besides the generalized smoothness condition and the new approach, our results are also novel in the following aspects.
- The convergence results of constant-stepsize methods challenge the folklore belief on the necessity of adaptive stepsize for generalized smooth functions.
- We obtain new convergence results for GD and NAG in the convex setting under the generalized smoothness condition.
- We relax the assumption of bounded noise to the weaker one of bounded variance of noise in the stochastic setting with the simple SGD method.
Gradient-based optimization. The classical gradient-based optimization problems for the standard Lipschitz smooth functions have been well studied for both convex [Nemirovskij and Yudin, 1983,
2This statement is informal because we can only bound Hessian norms along the trajectory, rather than almost everywhere within a convex set as in the standard Lipschitz smoothness condition. For example, even if the Hessian norm is bounded at both  $x_{t}$  and  $x_{t + 1}$ , it does not directly mean the Hessian norm is also bounded over the line segment between them, which is required in classical analysis. A more formal statement will need Lemma 3.3 presented later in the paper.
Table 1: Summary of the results.  $\epsilon$  denotes the sub-optimality gap of the function value in convex settings, and the gradient norm in non-convex settings. “*” denotes optimal rates.
Method	Convexity	l-smoothness	Gradient complexity
GD	Strongly convex Convex	No requirement	O(log(1/ε)) (Theorem 4.3) O(1/ε) (Theorem 4.2 )
Non-convex	Sub-quadratic l	O(1/ε2)* (Theorem 5.2)
Quadratic l	Ω(exp. in cond #) (Theorem 5.4 )
NAG	Convex	Sub-quadratic l	O(1/√ε)* (Theorem 4.4 )
SGD	Non-convex	Sub-quadratic l	O(1/ε4)* (Theorem 5.3)
Nesterov, 2003, d'Aspremont et al., 2021] and non-convex functions. In the convex setting, the goal is to reach an  $\epsilon$ -sub-optimal point  $x$  satisfying  $f(x) - \inf_{x} f(x) \leq \epsilon$ . It is well known that GD achieves the  $\mathcal{O}(1/\epsilon)$  gradient complexity and NAG achieves the accelerated  $\mathcal{O}(1/\sqrt{\epsilon})$  complexity which is optimal among all gradient-based methods. For strongly convex functions, GD and NAG achieve the  $\mathcal{O}(\kappa \log(1/\epsilon))$  and  $\mathcal{O}(\sqrt{\kappa} \log(1/\epsilon))$  complexity respectively, where  $\kappa$  is the condition number and the latter is again optimal. In the non-convex setting, the goal is to find an  $\epsilon$ -stationary point  $x$  satisfying  $\|\nabla f(x)\| \leq \epsilon$ , since finding a global minimum is NP-hard in general. It is well known that GD achieves the optimal  $\mathcal{O}(1/\epsilon^2)$  complexity which matches the lower bound in [Carmon et al., 2017]. In the stochastic setting for unbiased stochastic gradient with bounded variance, SGD achieves the optimal  $\mathcal{O}(1/\epsilon^4)$  complexity [Ghadimi and Lan, 2013], matching the lower bound in [Arjevani et al., 2019]. In this paper, we obtain the classical rates in terms of  $\epsilon$  for all the above-mentioned methods and settings, under a far more general smoothness condition.
Generalized smoothness. The  $(L_0, L_1)$ -smoothness condition proposed by Zhang et al. [2019] was studied by many follow-up works. Under the same condition, [Zhang et al., 2020] considers momentum in the updates and improves the constant dependency of the convergence rate for SGD with clipping derived in [Zhang et al., 2019]. [Qian et al., 2021] studies gradient clipping in incremental gradient methods, [Zhao et al., 2021] studies stochastic normalized gradient descent, and [Crawshaw et al., 2022] studies a generalized SignSGD method, under the  $(L_0, L_1)$ -smoothness condition. [Reisizadeh et al., 2023] studies variance reduction for  $(L_0, L_1)$ -smooth functions. [Chen et al., 2023] proposes a new notion of  $\alpha$ -symmetric generalized smoothness, which is roughly as general as  $(L_0, L_1)$ -smoothness. [Wang et al., 2022] analyzes convergence of Adam and provides a lower bound which shows non-adaptive SGD may diverge. In the stochastic setting, the above-mentioned works either consider the strong assumption of bounded gradient noise or require a very large batch size that depends on  $\epsilon$ , which essentially reduces the analysis to the deterministic setting. [Faw et al., 2023] proposes an AdaGrad-type algorithm in order to relax the bounded noise assumption. Perhaps due to the lower bounds in [Zhang et al., 2019, Wang et al., 2022], all the above works study methods with an adaptive stepsize. In this and our concurrent work [Li et al., 2023], we further generalize the smoothness condition and analyze various methods under this condition through bounding the gradients along the trajectory.
3 Function class
In this section, we discuss the function class of interest where the objective function  $f$  lies. We start with the following two standard assumptions in the literature of unconstrained optimization, which will be assumed throughout Sections 4 and 5 unless explicitly stated.
Assumption 1. The objective function  $f$  is differentiable and closed within its open domain  $\mathcal{X}$ .
Assumption 2. The objective function  $f$  is bounded from below, i.e.,  $f^{*} \coloneqq \inf_{x \in \mathcal{X}} f(x) > -\infty$ .
A function  $f$  is said to be closed if its sub-level set  $\{x \in \operatorname{dom}(f) \mid f(x) \leq a\}$  is closed for each  $a \in \mathbb{R}$ . A continuous function  $f$  with an open domain is closed if and only if  $f(x)$  tends to positive infinity when  $x$  approaches the boundary of its domain [Boyd and Vandenberghe, 2004]. Assumption 1 is necessary for our analysis to ensure that the iterates of a method with a reasonably small stepsize stays within the domain  $\mathcal{X}$ . Note that for  $\mathcal{X} = \mathbb{R}^d$  considered in most unconstrained
optimization papers, the assumption is trivially satisfied as all continuous functions over  $\mathbb{R}^d$  are closed. We consider a more general domain which may not be the whole space because that is the case for some interesting examples in our function class of interest (see Section 3.1.3). However, it actually brings us some additional technical difficulties especially in the stochastic setting, as we need to make sure the iterates do not go outside of the domain.
3.1 Generalized smoothness
In this section, we formally define the generalized smoothness condition, and present its properties and examples.
3.1.1 Definitions
Definitions 1 and 2 below are two equivalent ways of stating the definition, where we use  $\mathcal{B}(x,R)$  to denote the Euclidean ball with radius  $R$  centered at  $x$ .
Definition 1 ( $\ell$ -smoothness). A real-valued differentiable function  $f: \mathcal{X} \to \mathbb{R}$  is  $\ell$ -smooth for some non-decreasing continuous function  $\ell: [0, +\infty) \to (0, +\infty)$  if  $\|\nabla^2 f(x)\| \leq \ell(\|\nabla f(x)\|)$  almost everywhere (with respect to the Lebesgue measure) in  $\mathcal{X}$ .
Remark 3.1. Definition 1 reduces to the classical  $L$ -smoothness when  $\ell \equiv L$  is a constant function. It reduces to the  $(L_0, L_1)$ -smoothness proposed in [Zhang et al., 2019] when  $\ell(u) = L_0 + L_1u$  is an affine function.
Definition 2  $((r,\ell)$ -smoothness). A real-valued differentiable function  $f:\mathcal{X}\to \mathbb{R}$  is  $(r,\ell)$ -smooth for continuous functions  $r,\ell :[0, + \infty)\rightarrow (0, + \infty)$  where  $\ell$  is non-decreasing and  $r$  is non-increasing, if it satisfies 1) for any  $x\in \mathcal{X}$ ,  $\mathcal{B}(x,r(\| \nabla f(x)\|))\subseteq \mathcal{X}$ , and 2) for any  $x_{1},x_{2}\in \mathcal{B}(x,r(\| \nabla f(x)\|))$ ,  $\| \nabla f(x_1) - \nabla f(x_2)\| \leq \ell (\| \nabla f(x)\|)\cdot \| x_1 - x_2\|$ .
The requirements that  $\ell$  is non-decreasing and  $r$  is non-increasing do not cause much loss in generality. If these conditions are not satisfied, one can replace  $\ell$  and  $r$  with the non-increasing function  $\tilde{r}(u) \coloneqq \inf_{0 \leq v \leq u} r(v) \leq r(u)$  and non-decreasing function  $\tilde{\ell}(u) \coloneqq \sup_{0 \leq v \leq u} \ell(v) \geq \ell(u)$  in Definitions 1 and 2. Then the only requirement is  $\tilde{r} > 0$  and  $\tilde{\ell} < \infty$ .
Next, we prove that the above two definitions are equivalent in the following proposition, whose proof is involved and deferred to Appendix A.2.
Proposition 3.2. An  $(r,\ell)$ -smooth function is  $\ell$ -smooth; and an  $\ell$ -smooth function satisfying Assumption 1 is  $(r,m)$ -smooth where  $m(u) \coloneqq \ell(u + a)$  and  $r(u) \coloneqq a / m(u)$  for any  $a > 0$ .
The condition in Definition 1 is simple and one can easily check whether it is satisfied for a given example function. On the other hand, Definition 2 is a local Lipschitz condition on the gradient that is harder to verify. However, it is useful for deriving several useful properties in the next section.
3.1.2 Properties
First, we provide the following lemma which is very useful in our analyses of all the methods considered in this paper. Its proof is deferred to Appendix A.3.
Lemma 3.3. If  $f$  is  $(r,\ell)$ -smooth, for any  $x \in \mathcal{X}$  satisfying  $\| \nabla f(x) \| \leq G$ , we have 1)  $\mathcal{B}(x,r(G)) \subseteq \mathcal{X}$ , and 2) for any  $x_1, x_2 \in \mathcal{B}(x,r(G))$ ,
$$
\left\| \nabla f \left(x _ {1}\right) - \nabla f \left(x _ {2}\right) \right\| \leq L \| x _ {1} - x _ {2} \|, \quad f \left(x _ {1}\right) \leq f \left(x _ {2}\right) + \left\langle \nabla f \left(x _ {2}\right), x _ {1} - x _ {2} \right\rangle + \frac {L}{2} \| x _ {1} - x _ {2} \| ^ {2}, \tag {2}
$$
where  $L\coloneqq \ell (G)$  is the effective smoothness constant.
Remark 3.4. Since we have shown the equivalence between  $\ell$ -smoothness and  $(r,\ell)$ -smoothness, Lemma 3.3 also applies to  $\ell$ -smooth functions, for which we have  $L = \ell(2G)$  and  $r(G) = G / L$  if choosing  $a = G$  in Proposition 3.2.
Lemma 3.3 states that, if the gradient at  $x$  is bounded by some constant  $G$ , then within its neighborhood with a constant radius, we can obtain (2), the same inequalities that were derived in the textbook analysis [Nesterov, 2003] under the standard Lipschitz smoothness condition. With (2), the analysis for generalized smoothness is not much harder than that for standard smoothness. Since we
Table 2: Examples of univariate  $(\rho, L_0, L_\rho)$  smooth functions for different  $\rho$ s. The parameters  $a, b, p$  are real numbers (not necessarily integers) satisfying  $a, b > 1$  and  $p < 1$  or  $p \geq 2$ . We use  $1^+$  to denote any real number slightly larger than 1.
ρ	0	1	1	1+	1.5	2	p-2/p-1
Example Functions	Quadratic	Polynomial	ax	a(bx)	Rational	Logarithmic	xp
mostly choose  $x = x_{2} = x_{t}$  and  $x_{1} = x_{t + 1}$  in the analysis, in order to apply Lemma 3.3, we need two conditions:  $\| \nabla f(x_{t})\| \leq G$  and  $\| x_{t + 1} - x_t\| \leq r(G)$  for some constant  $G$ . The latter is usually directly implied by the former for most deterministic methods with a small enough stepsize, and the former can be obtained with our new approach that bounds the gradients along the trajectory.
With Lemma 3.3, we can derive the following useful lemma which is the reverse direction of a generalized Polyak-Lojasiewicz (PL) inequality, whose proof is deferred to Appendix A.3.
Lemma 3.5. If  $f$  is  $\ell$ -smooth, then  $\| \nabla f(x) \|^{2} \leq 2\ell(2\| \nabla f(x) \|) \cdot (f(x) - f^{*})$  for any  $x \in \mathcal{X}$ .
Lemma 3.5 provides an inequality involving the gradient norm and the sub-optimality gap. For example, when  $\ell(u) = u^{\rho}$  for some  $0 \leq \rho < 2$ , this lemma suggests  $\| \nabla f(x) \| \leq \mathcal{O}((f(x) - f^{*})^{1/(2-\rho)})$ , which means the gradient norm is bounded whenever the function value is bounded. The following corollary provides a more formal statement for general sub-quadratic  $\ell$  (i.e.,  $\lim_{u \to \infty} \ell(u)/u^2 = 0$ ), and we defer its proof to Appendix A.3.
Corollary 3.6. Suppose  $f$  is  $\ell$ -smooth where  $\ell$  is sub-quadratic. If  $f(x) - f^{*} \leq F$  for some  $x \in \mathcal{X}$  and  $F \geq 0$ , denoting  $G := \sup \{u \geq 0 \mid u^2 \leq 2\ell(2u) \cdot F\}$ , then they satisfy  $G^2 = 2\ell(2G) \cdot F$  and we have  $\| \nabla f(x) \| \leq G < \infty$ .
Therefore, in order to bound the gradients along the trajectory as we discussed below Lemma 3.3, it suffices to bound the function values, which is usually easier.
3.1.3 Examples
The most important subset of  $\ell$ -smooth (or  $(r,\ell)$ -smooth) functions are those with a polynomial  $\ell$ , and can be characterized by the  $(\rho, L_0, L_\rho)$ -smooth function class defined below.
Definition 3  $((\rho, L_0, L_\rho)$ -smoothness). A real-valued differentiable function  $f$  is  $(\rho, L_0, L_\rho)$ -smooth for constants  $\rho, L_0, L_\rho \geq 0$  if it is  $\ell$ -smooth with  $\ell(u) = L_0 + L_\rho u^\rho$ .
Definition 3 reduces to the standard Lipschitz smoothness condition when  $\rho = 0$  or  $L_{\rho} = 0$  and to the  $(L_0,L_1)$ -smoothness proposed in [Zhang et al., 2019] when  $\rho = 1$ . We list several univariate examples of  $(\rho ,L_0,L_\rho)$ -smooth functions for different  $\rho$ s in Table 2 with their rigorous justifications in Appendix A.1. Note that when  $x$  goes to infinity, polynomial and exponential functions corresponding to  $\rho = 1$  grow much faster than quadratic functions corresponding to  $\rho = 0$ . Rational and logarithmic functions for  $\rho >1$  grow even faster as they can blow up to infinity near finite points. Note that the domains of such functions are not  $\mathbb{R}^d$ , which is why we consider the more general Assumption 1 instead of simply assuming  $\mathcal{X} = \mathbb{R}^d$ .
Aside from logarithmic functions, the  $(2,L_0,L_2)$ -smooth function class also includes other univariate self-concordant functions. This is an important function class in the analysis of Interior Point Methods and coordinate-free analysis of the Newton method [Nesterov, 2003]. More specifically, a convex function  $h:\mathbb{R}\to \mathbb{R}$  is self-concordant if  $|h^{\prime \prime}(x)|\leq 2h^{\prime \prime}(x)^{3 / 2}$  for all  $x\in \mathbb{R}$ . Formally, we have the following proposition whose proof is deferred to Appendix A.1.
Proposition 3.7. If  $h: \mathbb{R} \to \mathbb{R}$  is a self-concordant function satisfying  $h''(x) > 0$  over the interval  $(a, b)$ , then  $h$  restricted on  $(a, b)$  is  $(2, L_0, 2)$ -smooth for some  $L_0 > 0$ .
4 Convex setting
In this section, we present the convergence results of gradient descent (GD) and Nesterov's accelerated gradient method (NAG) in the convex setting. Formally, we define convexity as follows.
Definition 4. A real-valued differentiable function  $f: \mathcal{X} \to \mathbb{R}$  is  $\mu$ -strongly-convex for  $\mu \geq 0$  if  $\mathcal{X}$  is a convex set and  $f(y) - f(x) \geq \left\langle \nabla f(x), y - x \right\rangle + \frac{\mu}{2} \|y - x\|^2$  for any  $x, y \in \mathcal{X}$ . A function is convex if it is  $\mu$ -strongly-convex with  $\mu = 0$ .
We assume the existence of a global optimal point  $x^{*}$  throughout this section, as in the following assumption. However, we want to note that, for gradient descent, this assumption is just for simplicity rather than necessary.
Assumption 3. There exists a point  $x^{*}\in \mathcal{X}$  such that  $f(x^{*}) = f^{*} = \inf_{x\in \mathcal{X}}f(x)$
4.1 Gradient descent
The gradient descent method with a constant stepsize  $\eta$  is defined via the following update rule
$$
x _ {t + 1} = x _ {t} - \eta \nabla f (x _ {t}). \tag {3}
$$
As discussed below Lemma 3.3, the key in the convergence analysis is to show  $\| \nabla f(x_{t})\| \leq G$  for all  $t\geq 0$  and some constant  $G$ . We will prove it by induction relying on the following lemma whose proof is deferred to Appendix B.
Lemma 4.1. For any  $x \in \mathcal{X}$  satisfying  $\| \nabla f(x) \| \leq G$ , define  $x^{+} \coloneqq x - \eta \nabla f(x)$ . If  $f$  is convex and  $(r, \ell)$ -smooth, and  $\eta \leq \min \left\{\frac{2}{\ell(G)}, \frac{r(G)}{2G}\right\}$ , we have  $x^{+} \in \mathcal{X}$  and  $\| \nabla f(x^{+}) \| \leq \| \nabla f(x) \| \leq G$ .
Lemma 4.1 suggests that for gradient descent (3) with a small enough stepsize, if the gradient norm at  $x_{t}$  is bounded by  $G$ , then we have  $\| \nabla f(x_{t + 1})\| \leq \| \nabla f(x_t)\| \leq G$ , i.e., the gradient norm is also bounded by  $G$  at  $t + 1$ . In other words, the gradient norm is indeed a non-increasing potential function for gradient descent in the convex setting. With a standard induction argument, we can show that  $\| \nabla f(x_{t})\| \leq \| \nabla f(x_{0})\|$  for all  $t\geq 0$ . As discussed below Lemma 3.3, then we can basically apply the classical analysis to obtain the convergence guarantee in the convex setting as in the following theorem, whose proof is deferred to Appendix B.
Theorem 4.2. Suppose  $f$  is convex and  $(r, \ell)$ -smooth. Denote  $G := \| \nabla f(x_0) \|$  and  $L := \ell(G)$ , then the iterates generated by (3) with  $\eta \leq \min \left\{ \frac{1}{L}, \frac{r(G)}{2G} \right\}$  satisfy  $\| \nabla f(x_t) \| \leq G$  for all  $t \geq 0$  and
$$
f (x _ {T}) - f ^ {*} \leq \frac {\| x _ {0} - x ^ {*} \| ^ {2}}{2 \eta T}.
$$
Since  $\eta$  is a constant independent of  $\epsilon$  or  $T$ , Theorem 4.2 achieves the classical  $\mathcal{O}(1 / T)$  rate, or  $\mathcal{O}(1 / \epsilon)$  gradient complexity to achieve an  $\epsilon$ -sub-optimal point, under the generalized smoothness condition. Since strongly convex functions are a subset of convex functions, Lemma 4.1 still holds for them. Then we immediately obtain the following result in the strongly convex setting, whose proof is deferred to Appendix B.
Theorem 4.3. Suppose  $f$  is  $\mu$ -strongly-convex and  $(r, \ell)$ -smooth. Denote  $G := \| \nabla f(x_0) \|$  and  $L := \ell(G)$ , then the iterates generated by (3) with  $\eta \leq \min \left\{ \frac{1}{L}, \frac{r(G)}{2G} \right\}$  satisfy  $\| \nabla f(x_t) \| \leq G$  for all  $t \geq 0$  and
$$
f (x _ {T}) - f ^ {*} \leq \frac {\mu (1 - \eta \mu) ^ {T}}{2 (1 - (1 - \eta \mu) ^ {T})} \| x _ {0} - x ^ {*} \| ^ {2}.
$$