Title: Convex and Non-convex Optimization Under Generalized Smoothness
Venue: NeurIPS 2023
Accepted: yes
Keywords: Optimization

Convergence

Generalized smoothness
Abstract: Classical analysis of convex and non-convex optimization methods often requires the Lipschitz continuity of the gradient, which limits the analysis to functions bounded by quadratics. Recent work relaxed this requirement to a non-uniform smoothness condition with the Hessian norm  bounded by an affine function of the gradient norm, and proved convergence in the non-convex setting via gradient clipping, assuming bounded noise. In this paper, we further generalize this non-uniform smoothness condition and develop a simple, yet powerful analysis technique that bounds the gradients along the trajectory, thereby leading to  stronger results for both convex and non-convex optimization problems. In particular, we obtain the classical convergence rates for (stochastic) gradient descent and Nesterov's accelerated gradient method in the convex and/or non-convex setting under this general smoothness condition. The new analysis approach does not require gradient clipping and allows heavy-tailed noise with bounded variance in the stochastic setting.

Convex and Non-convex Optimization Under Generalized Smoothness

Abstract

Classical analysis of convex and non-convex optimization methods often requires the Lipschitz continuity of the gradient, which limits the analysis to functions bounded by quadratics. Recent work relaxed this requirement to a non-uniform smoothness condition with the Hessian norm bounded by an affine function of the gradient norm, and proved convergence in the non-convex setting via gradient clipping, assuming bounded noise. In this paper, we further generalize this non-uniform smoothness condition and develop a simple, yet powerful analysis technique that bounds the gradients along the trajectory, thereby leading to stronger results for both convex and non-convex optimization problems. In particular, we obtain the classical convergence rates for (stochastic) gradient descent and Nesterov's accelerated gradient method in the convex and/or non-convex setting under this general smoothness condition. The new analysis approach does not require gradient clipping and allows heavy-tailed noise with bounded variance in the stochastic setting.

1 Introduction

In this paper, we study the following unconstrained optimization problem

where X ⊆ R d is the domain of f . Classical textbook analyses [Nemirovskij and Yudin, 1983, Nesterov, 2003] of (1) often require the Lipschitz smoothness condition, which assumes ∥ ∥ ∇ 2 f ( x ) ∥ ∥ ≤ L almost everywhere for some L ≥ 0 called the smoothness constant. This condition, however, is rather restrictive and only satisfied by functions that are both upper and lower bounded by quadratic functions.

Recently, Zhang et al. [2019] proposed the more general ( L 0 , L 1 ) -smoothness condition, which assumes ∥ ∥ ∇ 2 f ( x ) ∥ ∥ ≤ L 0 + L 1 ∥∇ f ( x ) ∥ for some constants L 0 , L 1 ≥ 0 , motivated by their extensive language model experiments. This notion generalizes the standard Lipschitz smoothness condition and also contains e.g. univariate polynomial and exponential functions. For non-convex and ( L 0 , L 1 ) -smooth functions, they prove convergence of gradient descent (GD) and stochastic gradient descent (SGD) with gradient clipping and also provide a complexity lower bound for constant-stepsize GD/SGD without clipping. Based on these results, they claim gradient clipping or other forms of adaptivity provably accelerate the convergence for ( L 0 , L 1 ) -smooth functions. Perhaps due to the

lower bound, all the follow-up works under this condition that we are aware of limit their analyses to adaptive methods. Most of these focus on non-convex functions. See Section 2 for more discussions of related works.

In this paper, we significantly generalize the ( L 0 , L 1 ) -smoothness condition to the ℓ -smoothness condition which assumes ∥ ∥ ∇ 2 f ( x ) ∥ ∥ ≤ ℓ ( ∥∇ f ( x ) ∥ ) for some non-decreasing continuous function ℓ . We develop a simple, yet powerful approach, which allows us to obtain stronger results for both convex and non-convex optimization problems when ℓ is sub-quadratic (i.e., lim u →∞ ℓ ( u ) /u 2 = 0 ) or even more general. The ℓ -smooth function class with a sub-quadratic ℓ also contains e.g. univariate rational and double exponential functions. In particular, we prove the convergence of constant-stepsize GD/SGD and Nesterov's accelerated gradient method (NAG) in the convex or non-convex settings. For each method and setting, we obtain the classical convergence rate, under a certain requirement of ℓ . In addition, we relax the assumption of bounded noise to the weaker one of bounded variance with the simple SGD method. See Table 1 for a summary of our results and assumptions for each method and setting. At first glance, our results 'contradict' the lower bounds on constant-stepsize GD/SGD in [Zhang et al., 2019, Wang et al., 2022]; this will be reconciled in Section 5.3.

Our approach analyzes boundedness of gradients along the optimization trajectory. The idea behind it can be informally illustrated by the following 'circular' reasoning. On the one hand, if gradients along the trajectory are bounded by a constant G , then the Hessian norms are bounded by the constant ℓ ( G ) . Informally speaking, we essentially have the standard Lipschitz smoothness condition 2 and can apply classical textbook analyses to prove convergence, which implies that gradients converge to zero. On the other hand, if gradients converge, they must be bounded, since any convergent sequence is bounded. In other words, the bounded gradient condition implies convergence, and convergence also implies the condition back, which forms a circular argument. If we can break this circularity of reasoning in a rigorous way, both the bounded gradient condition and convergence are proved. In this paper, we will show how to break the circularity using induction or contradiction arguments for different methods and settings in Sections 4 and 5. We note that the idea of bounding gradients can be applied to the analysis of other optimization methods, e.g., the concurrent work [Li et al., 2023] by subset of the authors, which uses a similar idea to obtain a rigorous and improved analysis of the Adam method [Kingma and Ba, 2014].

Contributions. In light of the above discussions, we summarize our main contributions as follows.

We generalize the standard Lipschitz smoothness and also the ( L 0 , L 1 ) -smoothness condition to the ℓ -smoothness condition, and develop a new approach for analyzing convergence under this condition by bounding the gradients along the optimization trajectory.

We prove the convergence of constant-stepsize GD/SGD/NAG in the convex and non-convex settings, and obtain the classical rates for all of them, as summarized in Table 1.

Besides the generalized smoothness condition and the new approach, our results are also novel in the following aspects.

The convergence results of constant-stepsize methods challenge the folklore belief on the necessity of adaptive stepsize for generalized smooth functions.

We obtain new convergence results for GD and NAG in the convex setting under the generalized smoothness condition.

We relax the assumption of bounded noise to the weaker one of bounded variance of noise in the stochastic setting with the simple SGD method.

2 Related work

Gradient-based optimizaiton. The classical gradient-based optimization problems for the standard Lipschitz smooth functions have been well studied for both convex [Nemirovskij and Yudin, 1983,

Nesterov, 2003, d'Aspremont et al., 2021] and non-convex functions. In the convex setting, the goal is to reach an ϵ -sub-optimal point x satisfying f ( x ) -inf x f ( x ) ≤ ϵ . It is well known that GD achieves the O (1 /ϵ ) gradient complexity and NAG achieves the accelerated O (1 / √ ϵ ) complexity which is optimal among all gradient-based methods. For strongly convex functions, GD and NAG achieve the O ( κ log(1 /ϵ )) and O ( √ κ log(1 /ϵ )) complexity respectively, where κ is the condition number and the latter is again optimal. In the non-convex setting, the goal is to find an ϵ -stationary point x satisfying ∥∇ f ( x ) ∥ ≤ ϵ , since finding a global minimum is NP-hard in general. It is well known that GD achieves the optimal O (1 /ϵ 2 ) complexity which matches the lower bound in [Carmon et al., 2017]. In the stochastic setting for unbiased stochastic gradient with bounded variance, SGD achieves the optimal O (1 /ϵ 4 ) complexity [Ghadimi and Lan, 2013], matching the lower bound in [Arjevani et al., 2019]. In this paper, we obtain the classical rates in terms of ϵ for all the above-mentioned methods and settings, under a far more general smoothness condition.

Generalized smoothness. The ( L 0 , L 1 ) -smoothness condition proposed by Zhang et al. [2019] was studied by many follow-up works. Under the same condition, [Zhang et al., 2020] considers momentum in the updates and improves the constant dependency of the convergence rate for SGD with clipping derived in [Zhang et al., 2019]. [Qian et al., 2021] studies gradient clipping in incremental gradient methods, [Zhao et al., 2021] studies stochastic normalized gradient descent, and [Crawshaw et al., 2022] studies a generalized SignSGD method, under the ( L 0 , L 1 ) -smoothess condition. [Reisizadeh et al., 2023] studies variance reduction for ( L 0 , L 1 ) -smooth functions. [Chen et al., 2023] proposes a new notion of α -symmetric generalized smoothness, which is roughly as general as ( L 0 , L 1 ) -smoothness. [Wang et al., 2022] analyzes convergence of Adam and provides a lower bound which shows non-adaptive SGD may diverge. In the stochastic setting, the above-mentioned works either consider the strong assumption of bounded gradient noise or require a very large batch size that depends on ϵ , which essentially reduces the analysis to the deterministic setting. [Faw et al., 2023] proposes an AdaGrad-type algorithm in order to relax the bounded noise assumption. Perhaps due to the lower bounds in [Zhang et al., 2019, Wang et al., 2022], all the above works study methods with an adaptive stepsize. In this and our concurrent work [Li et al., 2023], we further generalize the smoothness condition and analyze various methods under this condition through bounding the gradients along the trajectory.

3 Function class

In this section, we discuss the function class of interest where the objective function f lies. We start with the following two standard assumptions in the literature of unconstrained optimization, which will be assumed throughout Sections 4 and 5 unless explicitly stated.

Assumption 1. The objective function f is differentiable and closed within its open domain X .

Assumption 2. The objective function f is bounded from below, i.e., f ∗ := inf x ∈X f ( x ) > -∞ .

A function f is said to be closed if its sub-level set { x ∈ dom( f ) | f ( x ) ≤ a } is closed for each a ∈ R . A continuous function f with an open domain is closed if and only f ( x ) tends to positive infinity when x approaches the boundary of its domain [Boyd and Vandenberghe, 2004]. Assumption 1 is necessary for our analysis to ensure that the iterates of a method with a reasonably small stepsize stays within the domain X . Note that for X = R d considered in most unconstrained

optimization papers, the assumption is trivially satisfied as all continuous functions over R d are closed. We consider a more general domain which may not be the whole space because that is the case for some interesting examples in our function class of interest (see Section 3.1.3). However, it actually brings us some additional technical difficulties especially in the stochastic setting, as we need to make sure the iterates do not go outside of the domain.

3.1 Generalized smoothness

In this section, we formally define the generalized smoothness condition, and present its properties and examples.

3.1.1 Definitions

Definitions 1 and 2 below are two equivalent ways of stating the definition, where we use B ( x, R ) to denote the Euclidean ball with radius R centered at x .

Definition 1 ( ℓ -smoothness) . A real-valued differentiable function f : X → R is ℓ -smooth for some non-decreasing continuous function ℓ : [0 , + ∞ ) → (0 , + ∞ ) if ∥ ∥ ∇ 2 f ( x ) ∥ ∥ ≤ ℓ ( ∥∇ f ( x ) ∥ ) almost everywhere (with respect to the Lebesgue measure) in X .

Remark 3.1 . Definition 1 reduces to the classical L -smoothness when ℓ ≡ L is a constant function. It reduces to the ( L 0 , L 1 ) -smoothness proposed in [Zhang et al., 2019] when ℓ ( u ) = L 0 + L 1 u is an affine function.

Definition 2 ( ( r, ℓ ) -smoothness) . A real-valued differentiable function f : X → R is ( r, ℓ ) -smooth for continuous functions r, ℓ : [0 , + ∞ ) → (0 , + ∞ ) where ℓ is non-decreasing and r is non-increasing, if it satisfies 1) for any x ∈ X , B ( x, r ( ∥∇ f ( x ) ∥ )) ⊆ X , and 2) for any x 1 , x 2 ∈ B ( x, r ( ∥∇ f ( x ) ∥ )) , ∥∇ f ( x 1 ) -∇ f ( x 2 ) ∥ ≤ ℓ ( ∥∇ f ( x ) ∥ ) · ∥ x 1 -x 2 ∥ .

The requirements that ℓ is non-decreasing and r is non-increasing do not cause much loss in generality. If these conditions are not satisfied, one can replace ℓ and r with the non-increasing function ˜ r ( u ) := inf 0 ≤ v ≤ u r ( v ) ≤ r ( u ) and non-decreasing function ˜ ℓ ( u ) := sup 0 ≤ v ≤ u ℓ ( v ) ≥ ℓ ( u ) in Definitions 1 and 2. Then the only requirement is ˜ r > 0 and ˜ ℓ < ∞ .

Next, we prove that the above two definitions are equivalent in the following proposition, whose proof is involved and deferred to Appendix A.2.

Proposition 3.2. An ( r, ℓ ) -smooth function is ℓ -smooth; and an ℓ -smooth function satisfying Assumption 1 is ( r, m ) -smooth where m ( u ) := ℓ ( u + a ) and r ( u ) := a/m ( u ) for any a > 0 .

The condition in Definition 1 is simple and one can easily check whether it is satisfied for a given example function. On the other hand, Definition 2 is a local Lipschitz condition on the gradient that is harder to verify. However, it is useful for deriving several useful properties in the next section.

3.1.2 Properties

First, we provide the following lemma which is very useful in our analyses of all the methods considered in this paper. Its proof is deferred to Appendix A.3.

Lemma 3.3. If f is ( r, ℓ ) -smooth, for any x ∈ X satisfying ∥∇ f ( x ) ∥ ≤ G , we have 1) B ( x, r ( G )) ⊆ X , and 2) for any x 1 , x 2 ∈ B ( x, r ( G )) ,

where L := ℓ ( G ) is the effective smoothness constant .

Remark 3.4 . Since we have shown the equivalence between ℓ -smoothness and ( r, ℓ ) -smoothness, Lemma 3.3 also applies to ℓ -smooth functions, for which we have L = ℓ (2 G ) and r ( G ) = G/L if choosing a = G in Proposition 3.2.

Lemma 3.3 states that, if the gradient at x is bounded by some constant G , then within its neighborhood with a constant radius, we can obtain (2), the same inequalities that were derived in the textbook analysis [Nesterov, 2003] under the standard Lipschitz smoothness condition. With (2), the analysis for generalized smoothness is not much harder than that for standard smoothness. Since we

mostly choose x = x 2 = x t and x 1 = x t +1 in the analysis, in order to apply Lemma 3.3, we need two conditions: ∥∇ f ( x t ) ∥ ≤ G and ∥ x t +1 -x t ∥ ≤ r ( G ) for some constant G . The latter is usually directly implied by the former for most deterministic methods with a small enough stepsize, and the former can be obtained with our new approach that bounds the gradients along the trajectory.

With Lemma 3.3, we can derive the following useful lemma which is the reverse direction of a generalized Polyak-Lojasiewicz (PL) inequality, whose proof is deferred to Appendix A.3.

Lemma 3.5 provides an inequality involving the gradient norm and the sub-optimality gap. For example, when ℓ ( u ) = u ρ for some 0 ≤ ρ < 2 , this lemma suggests ∥∇ f ( x ) ∥ ≤ O ( ( f ( x ) -f ∗ ) 1 / (2 -ρ ) ) , which means the gradient norm is bounded whenever the function value is bounded. The following corollary provides a more formal statement for general sub-quadratic ℓ (i.e., lim u →∞ ℓ ( u ) /u 2 = 0 ), and we defer its proof to Appendix A.3.

Corollary 3.6. Suppose f is ℓ -smooth where ℓ is sub-quadratic. If f ( x ) -f ∗ ≤ F for some x ∈ X and F ≥ 0 , denoting G := sup { u ≥ 0 | u 2 ≤ 2 ℓ (2 u ) · F } , then they satisfy G 2 = 2 ℓ (2 G ) · F and we have ∥∇ f ( x ) ∥ ≤ G < ∞ .

Therefore, in order to bound the gradients along the trajectory as we discussed below Lemma 3.3, it suffices to bound the function values, which is usually easier.

3.1.3 Examples

The most important subset of ℓ -smooth (or ( r, ℓ ) -smooth) functions are those with a polynomial ℓ , and can be characterized by the ( ρ, L 0 , L ρ ) -smooth function class defined below.

Definition 3 ( ( ρ, L 0 , L ρ ) -smoothness) . Areal-valued differentiable function f is ( ρ, L 0 , L ρ ) -smooth for constants ρ, L 0 , L ρ ≥ 0 if it is ℓ -smooth with ℓ ( u ) = L 0 + L ρ u ρ .

Definition 3 reduces to the standard Lipschitz smoothness condition when ρ = 0 or L ρ = 0 and to the ( L 0 , L 1 ) -smoothness proposed in [Zhang et al., 2019] when ρ = 1 . We list several univariate examples of ( ρ, L 0 , L ρ ) -smooth functions for different ρ s in Table 2 with their rigorous justifications in Appendix A.1. Note that when x goes to infinity, polynomial and exponential functions corresponding to ρ = 1 grow much faster than quadratic functions corresponding to ρ = 0 . Rational and logarithmic functions for ρ > 1 grow even faster as they can blow up to infinity near finite points. Note that the domains of such functions are not R d , which is why we consider the more general Assumption 1 instead of simply assuming X = R d .

Aside from logarithmic functions, the (2 , L 0 , L 2 ) -smooth function class also includes other univariate self-concordant functions. This is an important function class in the analysis of Interior Point Methods and coordinate-free analysis of the Newton method [Nesterov, 2003]. More specifically, a convex function h : R → R is self-concordant if | h ′′′ ( x ) | ≤ 2 h ′′ ( x ) 3 / 2 for all x ∈ R . Formally, we have the following proposition whose proof is deferred to Appendix A.1.

Proposition 3.7. If h : R → R is a self-concordant function satisfying h ′′ ( x ) > 0 over the interval ( a, b ) , then h restricted on ( a, b ) is (2 , L 0 , 2) -smooth for some L 0 > 0 .

4 Convex setting

In this section, we present the convergence results of gradient descent (GD) and Nesterov's accelerated gradient method (NAG) in the convex setting. Formally, we define convexity as follows.

Definition 4. A real-valued differentiable function f : X → R is µ -strongly-convex for µ ≥ 0 if X is a convex set and f ( y ) -f ( x ) ≥ 〈 ∇ f ( x ) , y -x 〉 + µ 2 ∥ y -x ∥ 2 for any x, y ∈ X . A function is convex if it is µ -strongly-convex with µ = 0 .

We assume the existence of a global optimal point x ∗ throughout this section, as in the following assumption. However, we want to note that, for gradient descent, this assumption is just for simplicity rather than necessary.

Assumption 3. There exists a point x ∗ ∈ X such that f ( x ∗ ) = f ∗ = inf x ∈X f ( x ) .

4.1 Gradient descent

The gradient descent method with a constant stepsize η is defined via the following update rule

As discussed below Lemma 3.3, the key in the convergence analysis is to show ∥∇ f ( x t ) ∥ ≤ G for all t ≥ 0 and some constant G . We will prove it by induction relying on the following lemma whose proof is deferred to Appendix B.

Lemma 4.1. For any x ∈ X satisfying ∥∇ f ( x ) ∥ ≤ G , define x + := x -η ∇ f ( x ) . If f is convex and ( r, ℓ ) -smooth, and η ≤ min { 2 ℓ ( G ) , r ( G ) 2 G } , we have x + ∈ X and ∥∇ f ( x + ) ∥ ≤ ∥∇ f ( x ) ∥ ≤ G .

Lemma 4.1 suggests that for gradient descent (3) with a small enough stepsize, if the gradient norm at x t is bounded by G , then we have ∥∇ f ( x t +1 ) ∥ ≤ ∥∇ f ( x t ) ∥ ≤ G , i.e., the gradient norm is also bounded by G at t +1 . In other words, the gradient norm is indeed a non-increasing potential function for gradient descent in the convex setting. With a standard induction argument, we can show that ∥∇ f ( x t ) ∥ ≤ ∥∇ f ( x 0 ) ∥ for all t ≥ 0 . As discussed below Lemma 3.3, then we can basically apply the classical analysis to obtain the convergence guarantee in the convex setting as in the following theorem, whose proof is deferred to Appendix B.

Theorem 4.2. Suppose f is convex and ( r, ℓ ) -smooth. Denote G := ∥∇ f ( x 0 ) ∥ and L := ℓ ( G ) , then the iterates generated by (3) with η ≤ min { 1 L , r ( G ) 2 G } satisfy ∥∇ f ( x t ) ∥ ≤ G for all t ≥ 0 and

Since η is a constant independent of ϵ or T , Theorem 4.2 achieves the classical O (1 /T ) rate, or O (1 /ϵ ) gradient complexity to achieve an ϵ -sub-optimal point, under the generalized smoothness condition. Since strongly convex functions are a subset of convex functions, Lemma 4.1 still holds for them. Then we immediately obtain the following result in the strongly convex setting, whose proof is deferred to Appendix B.

Theorem 4.3. Suppose f is µ -strongly-convex and ( r, ℓ ) -smooth. Denote G := ∥∇ f ( x 0 ) ∥ and L := ℓ ( G ) , then the iterates generated by (3) with η ≤ min { 1 L , r ( G ) 2 G } satisfy ∥∇ f ( x t ) ∥ ≤ G for all t ≥ 0 and

Theorem 4.3 gives a linear convergence rate and the O (( ηµ ) -1 log(1 /ϵ )) gradient complexity to achieve an ϵ -sub-optimal point. Note that for ℓ -smooth functions, we have r ( G ) G = 1 L (see Remark 3.4), which means we can choose η = 1 2 L . Then we obtain the O ( κ log(1 /ϵ )) rate, where κ := L/µ is the local condition number around the initial point x 0 . For standard Lipschitz smooth functions, it reduces to the classical rate of gradient descent.

4.2 Nesterov's accelerated gradient method

In the case of convex and standard Lipschitz smooth functions, it is well known that Nesterov's accelerated gradient method (NAG) achieves the optimal O (1 /T 2 ) rate. In this section, we show that

input A convex and ℓ -smooth function f , stepsize η , initial point x 0

1: Initialize z 0 = x 0 , B 0 = 0 , and A 0 = 1 /η .

2: for t = 0 , ... do

3: B t +1 = B t + 1 2 ( 1 + √ 4 B t +1 )

4: A t +1 = B t +1 +1 /η

5: y t = x t +(1 -A t /A t +1 )( z t -x t )

6: x t +1 = y t -η ∇ f ( y t )

7: z t +1 = z t -η ( A t +1 -A t ) ∇ f ( y t )

8: end for

under the ℓ -smoothness condition with a sub-quadratic ℓ , the optimal O (1 /T 2 ) rate can be achieved by a slightly modified version of NAG shown in Algorithm 1, the only difference between which and the classical NAG is that the latter directly sets A t +1 = B t +1 in Line 4. Formally, we have the following theorem, whose proof is deferred to Appendix C.

Theorem 4.4. Suppose f is convex and ℓ -smooth where ℓ is sub-quadratic. Then there always exists a constant G satisfying G ≥ max { 8 √ ℓ (2 G )(( f ( x 0 ) -f ∗ ) + ∥ x 0 -x ∗ ∥ 2 ) , ∥∇ f ( x 0 ) ∥ } . Denote { 1 1 }

L := ℓ (2 G ) and choose η ≤ min 2 , . The iterates generated by Algorithm 1 satisfy

It is easy to note that Theorem 4.4 achieves the accelerated O (1 /T 2 ) convergence rate, or equivalently the O (1 / √ ϵ ) gradient complexity to find an ϵ -sub-optimal point, which is optimal among gradient-based methods [Nesterov, 2003].

In order to prove Theorem 4.4, we also use induction to show the gradients along the trajectory of Algorithm 1 are bounded by G . However, unlike gradient descent, the gradient norm is no longer a potential function or monotonically non-increasing, which makes the induction analysis more challenging. Suppose that we have shown ∥∇ f ( y s ) ∥ ≤ G for s < t . To complete the induction, it suffices to prove ∥∇ f ( y t ) ∥ ≤ G . Since x t = y t -1 -η ∇ f ( y t -1 ) is a gradient descent step by Line 6 of Algorithm 1, Lemma 4.1 directly shows ∥∇ f ( x t ) ∥ ≤ G . In order to also bound ∥∇ f ( y t ) ∥ , we try to control ∥ y t -x t ∥ , which is the most challenging part of our proof. Since y t -x t can be expressed as a linear combination of past gradients {∇ f ( y s ) } s<t , it might grow linearly with t if we simply apply ∥∇ f ( y s ) ∥ ≤ G for s < t . Fortunately, Lemma 3.5 allows us to control the gradient norm with the function value. Thus if the function value is decreasing sufficiently fast, which can be shown by following the standard Lyapunov analysis of NAG, we are able to obtain a good enough bound on ∥∇ f ( y s ) ∥ for s < t , which allows us to control ∥ y t -x t ∥ . We defer the detailed proof to Appendix C.

Note that Theorem 4.4 requires a smaller stepsize η = O (1 /L 2 ) , compared to the classical O (1 /L ) stepsize for standard Lipschitz smooth functions. The reason is we require a small enough stepsize to get a good enough bound on ∥ y t -x t ∥ . However, if the function is further assumed to be ℓ -smooth with a sub-linear ℓ , the requirement of stepsize can be relaxed to η = O (1 /L ) , similar to the classical requirement. See Appendix C for the details.

In the strongly convex setting, we can also prove convergence of NAG with different { A t } t ≥ 0 parameters when f is ℓ -smooth with a sub-quadratic ℓ , or ( ρ, L 0 , L ρ ) -smooth with ρ < 2 . The rate can be further improved when ρ becomes smaller. However, since the constants G and L are different for GD and NAG, it is not clear whether the rate of NAG is faster than that of GD in the strongly convex setting. We will present the detailed result and analysis in Appendix D.

5 Non-convex setting

In this section, we present convergence results of gradient descent (GD) and stochastic gradient descent (SGD) in the non-convex setting.

5.1 Gradient descent

Similar to the convex setting, we still want to bound the gradients along the trajectory. However, in the non-convex setting, the gradient norm is not necessarily non-increasing. Fortunately, similar to the classical analyses, the function value is still non-increasing and thus a potential function, as formally shown in the following lemma, whose proof is deferred to Appendix E.

Lemma 5.1. Suppose f is ℓ -smooth where ℓ is sub-quadratic. For any given F ≥ 0 , let G := sup { u ≥ 0 | u 2 ≤ 2 ℓ (2 u ) · F } and L := ℓ (2 G ) . For any x ∈ X satisfying f ( x ) -f ∗ ≤ F , define x + := x -η ∇ f ( x ) where η ≤ 2 /L , we have x + ∈ X and f ( x + ) ≤ f ( x ) .

Then using a standard induction argument, we can show f ( x t ) ≤ f ( x 0 ) for all t ≥ 0 . According to Corollary 3.6, it implies bounded gradients along the trajectory. Therefore, we can show convergence of gradient descent as in the following theorem, whose proof is deferred to Appendix E.

Theorem 5.2. Suppose f is ℓ -smooth where ℓ is sub-quadratic. Let G := sup { u ≥ 0 | u 2 ≤ 2 ℓ (2 u ) · ( f ( x 0 ) -f ∗ ) } and L := ℓ (2 G ) . If η ≤ 1 /L , the iterates generated by (3) satisfy ∥∇ f ( x t ) ∥ ≤ G for all t ≥ 0 and

It is clear that Theorem 5.2 gives the classical O (1 /ϵ 2 ) gradient complexity to achieve an ϵ -stationary point, which is optimal as it matches the lower bound in [Carmon et al., 2017].

5.2 Stochastic gradient descent

In this part, we present the convergence result for stochastic gradient descent defined as follows.

where g t is an estimate of the gradient ∇ f ( x t ) . We consider the following standard assumption on the gradient noise ϵ t := g t -∇ f ( x t ) .

Assumption 4. E t -1 [ ϵ t ] = 0 and E t -1 [ ∥ ϵ t ∥ 2 ] ≤ σ 2 for some σ ≥ 0 , where E t -1 denotes the expectation conditioned on { g s } s<t .

Under Assumption 4, we can obtain the following theorem.

Theorem 5.3. Suppose f is ℓ -smooth where ℓ is sub-quadratic. For any 0 < δ < 1 , we denote F := 8( f ( x 0 ) -f ∗ + σ ) /δ and G := sup { u ≥ 0 | u 2 ≤ 2 ℓ (2 u ) · F } < ∞ . Denote L := ℓ (2 G ) and choose η ≤ min { 1 2 L , 1 4 G √ T } and T ≥ F ηϵ 2 for any ϵ > 0 . Then with probability at least 1 -δ , the iterates generated by (4) satisfy ∥∇ f ( x t ) ∥ ≤ G for all t < T and

As we choose η = O (1 / √ T ) , Theorem 5.3 gives the classical O (1 /ϵ 4 ) gradient complexity, where we ignore non-leading terms. This rate is optimal as it matches the lower bound in [Arjevani et al., 2019]. The key to its proof is again to bound the gradients along the trajectory. However, bounding gradients in the stochastic setting is much more challenging than in the deterministic setting, especially with the heavy-tailed noise in Assumption 4. We briefly discuss some of the challenges as well as our approach below and defer the detailed proof of Theorem 5.3 to Appendix F.

First, due to the existence of heavy-tailed gradient noise as considered in Assumption 4, neither the gradient nor the function values is non-increasing. The induction analyses we have used in the deterministic setting hardly work. In addition, to apply Lemma 3.3, we need to control the update at each step and make sure ∥ x t +1 -x t ∥ = η ∥ g t ∥ ≤ G/L . However, g t might be unbounded due to the potentially unbounded gradient noise.

To overcome these challenges, we define the following random variable τ .

where we use a ∧ b to denote min { a, b } for any a, b ∈ R . Then at least before time τ , we know that the function value and gradient noise are bounded, where the former also implies bounded gradients according to Corollary 3.6. Therefore, it suffices to show the probability of τ < T is small, which means with a high probability, τ = T and thus gradients are always bounded before T .

Since both the gradient and noise are bounded for t < τ , it is straightforward to bound the update ∥ x t +1 -x t ∥ , which allows us to use Lemma 3.3 and other useful properties. However, it is still non-trivial to upper bound E [ f ( x τ ) -f ∗ ] as τ is a random variable instead of a fixed time step. Fortunately, τ is a stopping time with nice properties. That is because both f ( x t +1 ) and ϵ t = g t -∇ f ( x t ) only depend on { g s } s ≤ t , i.e., the stochastic gradients up to t . Therefore, for any fixed t , the events { τ > t } only depend on { g s } s ≤ t , which show τ is a stopping time. Then with a careful analysis, we are still able to obtain an upper bound on E [ f ( x τ ) -f ∗ ] = O (1) .

On the other hand, τ < T means either τ = τ 1 < T or τ = τ 2 < T . If τ = τ 1 < T , by its definition, we know f ( x τ +1 ) -f ∗ > F . Roughly speaking, it also suggests f ( x τ ) -f ∗ > F/ 2 . If we choose F such that it is much larger than the upper bound on E [ f ( x τ ) -f ∗ ] we just obtained, by Markov's inequality, we can show the probability of τ = τ 1 < T is small. In addition, by union bound and Chebyshev's inequality, the probability of τ 2 < T can also be bounded by a small constant. Therefore, we have shown τ < T . Then the rest of the analysis is not too hard following the classical analysis.

5.3 Reconciliation with existing lower bounds

In this section, we reconcile our convergence results for constant-stepsize GD/SGD in the non-convex setting with existing lower bounds in [Zhang et al., 2019] and [Wang et al., 2022], based on which the authors claim that adaptive methods such as GD/SGD with clipping and Adam are provably faster than non-adaptive GD/SGD. This may seem to contradict our convergence results. In fact, we show that any gain in adaptive methods is at most by constant factors, as GD and SGD already achieve the optimal rates in the non-convex setting.

[Zhang et al., 2019] provides both upper and lower complexity bounds for constant-stepsize GD for ( L 0 , L 1 ) -smooth functions, and shows that its complexity is O ( Mϵ -2 ) , where

is the supremum gradient norm below the level set of the initial function value. If M is very large, then the O ( Mϵ -2 ) complexity can be viewed as a negative result, and as evidence that constant-stepsize GD can be slower than GD with gradient clipping, since in the latter case, they obtain the O ( ϵ -2 ) complexity without M . However, based on our Corollary 3.6, their M can be actually bounded by our G , which is a constant. Therefore, the gain in adaptive methods is at most by constant factors.

[Wang et al., 2022] further provides a lower bound which shows non-adaptive GD may diverge for some examples. However, their counter-example does not allow the stepsize to depend on the initial sub-optimality gap. In contrast, our stepsize η depends on the effective smoothness constant L , which depends on the initial sub-optimality gap through G . Therefore, there is no contradiction here either. We should point out that in the practice of training neural networks, the stepsize is usually tuned after fixing the loss function and initialization, so it does depend on the problem instance and initialization.

5.4 Lower bound

For ( ρ, L 0 , L ρ ) -smooth functions with ρ < 2 , it is easy to verify that the constant G in both Theorem 5.2 and Theorem 5.3 is a polynomial function of problem-dependent parameters like L 0 , L ρ , f ( x 0 ) -f ∗ , σ , etc. In other words, GD and SGD are provably efficient methods in the non-convex setting for ρ < 2 . In this section, we show that the requirement of ρ < 2 is necessary in the non-convex setting with the lower bound for GD in the following Theorem 5.4, whose proof is deferred in Appendix G. Since SGD reduces to GD when there is no gradient noise, it is also a lower bound for SGD.

Theorem 5.4. Given L 0 , L 2 , G 0 , ∆ 0 > 0 satisfying L 2 ∆ 0 ≥ 10 , for any η ≥ 0 , there exists a (2 , L 0 , L 2 ) -smooth function f that satisfies Assumptions 1 and 2, and initial point x 0 that satisfies ∥∇ f ( x 0 ) ∥ ≤ G 0 and f ( x 0 ) -f ∗ ≤ ∆ 0 , such that gradient descent with stepsize η (3) either cannot reach a 1 -stationary point or takes at least exp( L 2 ∆ 0 / 8) / 6 steps to reach a 1 -stationary point.

6 Conclusion

In this paper, we generalize the standard Lipschitz smoothness as well as the ( L 0 , L 1 ) -smoothness [Zhang et al., 2020] conditions to the ℓ -smoothness condition, and develop a new approach for analyzing the convergence under this condition. The approach uses different techniques for several methods and settings to bound the gradient along the optimization trajectory, which allows us to obtain stronger results for both convex and non-convex problems. We obtain the classical rates for GD/SGD/NAG methods in the convex and/or non-convex setting. Our results challenge the folklore belief on the necessity of adaptive methods for generalized smooth functions.

There are several interesting future directions following this work. First, the ℓ -smoothness can perhaps be further generalized by allowing ℓ to also depend on potential functions in each setting, besides the gradient norm. In addition, it would also be interesting to see if the techniques of bounding gradients along the trajectory that we have developed in this and the concurrent work [Li et al., 2023] can be further generalized to other methods and problems and to see whether more efficient algorithms can be obtained. Finally, although we justified the necessity of the requirement of ℓ -smoothness with a sub-quadratic ℓ in the non-convex setting, it is not clear whether it is also necessary for NAG in the convex setting, another interesting open problem.

Acknowledgments

This work was supported, in part, by the MIT-IBM Watson AI Lab and ONR Grants N00014-20-1-2394 and N00014-23-1-2299. We also acknowledge support from DOE under grant DE-SC0022199, and NSF through awards DMS-2031883, DMS-1953181, and DMS-2022448 (TRIPODS program).

References

Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake E. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199:165 214, 2019.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Yair Carmon, John C. Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming , pages 1-50, 2017.

Ziyi Chen, Yi Zhou, Yingbin Liang, and Zhaosong Lu. Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. arXiv preprint arXiv:2303.02854 , 2023.

Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in Neural Information Processing Systems , 35:9955-9968, 2022.

Alexandre d'Aspremont, Damien Scieur, and Adrien Taylor. Acceleration methods. Foundations and Trends® in Optimization , 5(1-2):1-245, 2021. ISSN 2167-3888. doi: 10.1561/2400000036. URL http://dx.doi.org/10.1561/2400000036 .

Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive sgd. ArXiv , abs/2302.06570, 2023.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. Convergence of adam under relaxed assumptions. arXiv preprint arXiv:2304.13972 , 2023.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization . Wiley-Interscience, 1983.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003.

Jiang Qian, Yuren Wu, Bojin Zhuang, Shaojun Wang, and Jing Xiao. Understanding gradient clipping in incremental gradient methods. In International Conference on Artificial Intelligence and Statistics , 2021.

Amirhossein Reisizadeh, Haochuan Li, Subhro Das, and Ali Jadbabaie. Variance-reduced clipping for non-convex optimization. ArXiv , abs/2303.00883, 2023.

Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, and Wei Chen. Provable adaptivity in adam. arXiv preprint arXiv:2208.09900 , 2022.

Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. ArXiv , abs/2010.02519, 2020.

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:1905.11881 , 2019.

Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences , 64, 2021.

A Proofs related to generalized smoothness

In this section, we provide the proofs of propositions and lemmas related to the generalized smoothness condition in Definition 1 or 2. First, in Appendix A.1, we justify the examples we discussed in Section 3. Next, we provide the detailed proof of Proposition 3.2 in Appendix A.2. Finally, we provide the proofs of the useful properties of generalized smoothness in Appendix A.3, including Lemma 3.3, Lemma 3.5, and Corollary 3.6 stated in Section 3.1.2.

A.1 Justification of examples in Section 3

In this section, we justify the univariate examples of ( ρ, L 0 , L ρ ) -smooth functions listed in Table 2 and also provide the proof of Propositions 3.7.

̸

First, it is well-known that all quadratic functions have bounded Hessian and are Lipschitz smooth, corresponding to ρ = 0 . Next, [Zhang et al., 2019, Lemma 2] shows that any univariate polynomial is ( L 0 , L 1 ) -smooth, corresponding to ρ = 1 . Then, regarding the exponential function f ( x ) = a x where a > 1 , we have f ′ ( x ) = log( a ) a x and f ′′ ( x ) = log( a ) 2 a x = log( a ) f ′ ( x ) , which implies f is (1 , 0 , log( a )) -smooth. Similarly, by standard calculations, it is straight forward to verify that logarithmic functions and x p , p = 1 are also ( ρ, L 0 , L ρ ) -smooth with ρ = 2 and ρ = p -2 p -1 respectively. So far we have justified all the examples in Table 2 except double exponential functions a ( b x ) and rational functions, which will be justified rigorously by the two propositions below.

First, for double exponential functions in the form of f ( x ) = a ( b x ) where a, b > 1 , we have the following proposition, which shows f is ( ρ, L 0 , L ρ ) -smooth for any ρ > 1 .

Proposition A.1. For any ρ > 1 , the double exponential function f ( x ) = a ( b x ) , where a, b > 1 , is ( ρ, L 0 , L ρ ) -smooth for some L 0 , L ρ ≥ 0 . However, it is not necessarily ( L 0 , L 1 ) -smooth for any L 0 , L 1 ≥ 0 .

Proof of Proposition A.1. By standard calculations, we can obtain

Note that if ρ > 1 ,

where the first equality is a direct calculation based on (6); the second equality uses change of variable y = b x ; and the last equality is because exponential functions grow faster than affine functions. Therefore, for any L ρ > 0 , there exists x 0 ∈ R such that | f ′′ ( x ) | ≤ L ρ | f ′ ( x ) | ρ if x > x 0 . Next, note that lim x →-∞ f ′′ ( x ) = 0 . Then for any λ 1 > 0 , there exists x 1 ∈ R such that | f ′′ ( x ) | ≤ λ 1 if x < x 1 . Also, since f ′′ is continuous, by Weierstrass's Theorem, we have | f ′′ ( x ) | ≤ λ 2 if x 1 ≤ x ≤ x 0 for some λ 2 > 0 . Then denoting L 0 = max { λ 1 , λ 2 } , we know f is ( ρ, L 0 , L ρ ) -smooth.

Next, to show f is not necessarily ( L 0 , L 1 ) -smooth, consider the specific double exponential function f ( x ) = e ( e x ) . Then we have

For any x ≥ max { log( L 0 +1) , log( L 1 +1) } , we can show that

which shows f is not ( L 0 , L 1 ) smooth for any L 0 , L 1 ≥ 0

.

In the next proposition, we show that any univariate rational function f ( x ) = P ( x ) /Q ( x ) , where P and Q are two polynomials, is ( ρ, L 0 , L ρ ) -smooth with ρ = 1 . 5 .

Proposition A.2. The rational function f ( x ) = P ( x ) /Q ( x ) , where P and Q are two polynomials, is (1 . 5 , L 0 , L 1 . 5 ) -smooth for some L 0 , L 1 . 5 ≥ 0 . However, it is not necessarily ( ρ, L 0 , L ρ ) -smooth for any ρ < 1 . 5 and L 0 , L ρ ≥ 0 .

Proof of Proposition A.2. Let f ( x ) = P ( x ) /Q ( x ) where P and Q are two polynomials. Then the partial fractional decomposition of f ( x ) is given by

̸

where w ( x ) is a polynomial, A ir , B ir , C ir , a i , b i , c i are all real constants satisfying b 2 i -4 c i < 0 for each 1 ≤ i ≤ n which implies x 2 + b i x + c i > 0 for all x ∈ R . Assume j i ≥ 1 and A ij i = 0 without loss of generality. Then we know f has only finite singular points { a i } 1 ≤ i ≤ m and has continuous first and second order derivatives at all other points. To simplify notation, denote

Then we have f ( x ) = w ( x ) + ∑ m i =1 ∑ j i r =1 p ir ( x ) + ∑ n i =1 ∑ k i r =1 q ir ( x ) . We know that r +2 r +1 ≤ 1 . 5 for any r ≥ 1 . Then we can show that

where the first equality is because one can easily verify that the first and second order derivatives of p ij i dominate those of all other terms when x goes to a i , and the second equality is by standard calculations noting that j i +2 j i +1 ≤ 1 . 5 . Note that (7) implies that, for any L ρ > j i +1 , there exists δ i > 0 such that

Similarly, one can show lim x →∞ | f ′ ( x ) | 1 . 5 | f ′′ ( x ) | = ∞ , which implies there exists x 0 > 0 such that

We know B is a compact set and therefore the continuous function f ′′ is bounded within B , i.e., there exists some constant L 0 > 0 such that

Combining (8), (9), and (10), we have shown

which completes the proof of the first part.

For the second part, consider the ration function f ( x ) = 1 /x . Then we know that f ′ ( x ) = -1 /x 2 and f ′′ ( x ) = 2 /x 3 . Note that for any ρ < 1 . 5 and 0 < x ≤ min { ( L 0 +1) -1 / 3 , ( L ρ +1) -1 / (3 -2 ρ ) } , we have

which shows f is not ( ρ, L 0 , L ρ ) smooth for any ρ < 1 . 5 and L 0 , L ρ ≥ 0 .

Finally, we complete this section with the proof of Proposition 3.7, which shows self-concordant functions are (2 , L 0 , L 2 ) -smooth for some L 0 , L ρ ≥ 0 .

Proof of Proposition 3.7. Let h : R → R be a self-concordant function. We have h ′′′ ( x ) ≤ 2 h ′′ ( x ) 3 / 2 . Then, for x ∈ ( a, b ) , we can obtain

Integrating both sides from x 0 to y for x 0 , y ∈ ( a, b ) , we have

Define

Therefore,

Since h ′′ ( y ) > 0 , we have | h ′′ ( y ) | = h ′′ ( y ) . Therefore, the above inequality shows that h is (2 , L 0 , L 2 ) -smooth with L 0 = 2( h ′′ ( x 0 ) 1 / 2 -h ′ ( x 0 )) 2 and L 2 = 2 .

A.2 Proof of Proposition 3.2

In order to prove Proposition 3.2, we need the following several lemmas. First, the lemma below partially generalizes Grönwall's inequality.

Lemma A.3. Let α : [ a, b ] → [0 , ∞ ) and β : [0 , ∞ ) → (0 , ∞ ) be two continuous functions. Suppose α ′ ( t ) ≤ β ( α ( t )) almost everywhere over ( a, b ) . Denote function ϕ ( u ) := ∫ 1 β ( u ) du . We have for all t ∈ [ a, b ] ,

Proof of Lemma A.3. First, by definition, we know that ϕ is increasing since ϕ ′ = 1 β > 0 . Let function γ : [ a, b ] → R be the solution of the following differential equation

Integrating both sides, noting that γ ( a ) = α ( a ) by (11), we obtain

Then it suffices to show ϕ ( α ( t )) ≤ ϕ ( γ ( t )) , ∀ t ∈ [ a, b ] . Note that the following inequality holds almost everywhere.

where the inequality is because α ′ ( t ) ≤ β ( α ( t )) by the assumption of this lemma and γ ′ ( t ) = β ( γ ( t )) by (11). Since ϕ ( α ( a )) -ϕ ( γ ( a )) = 0 , we know for all t ∈ [ a, b ] , ϕ ( α ( t )) ≤ ϕ ( γ ( t )) , which completes the proof.

With Lemma A.3, one can bound the gradient norm within a small enough neighborhood of a given point as in the following lemma.

Lemma A.4. If the objective function f is ℓ -smooth, for any two points x, y ∈ R d such that the closed line segment between x and y is contained in X , if ∥ y -x ∥ ≤ a ℓ ( ∥∇ f ( x ) ∥ + a ) for any a > 0 , we have

Proof of Lemma A.4. Denote z ( t ) := (1 -t ) x + ty for 0 ≤ t ≤ 1 . Then we know z ( t ) ∈ X for all 0 ≤ t ≤ 1 by the assumption made in this lemma. Then we can also define α ( t ) := ∥∇ f ( z ( t )) ∥ for 0 ≤ t ≤ 1 . Note that for any 0 ≤ t ≤ s ≤ 1 , by triangle inequality,

̸

We know that α ( t ) = ∥∇ f ( z ( t )) ∥ is differentiable almost everywhere since f is second order differentiable almost everywhere (Here we assume α ( t ) = 0 for 0 < t < 1 without loss of generality. Otherwise, one can define t m = sup { 0 < t < 1 | α ( t ) = 0 } and consider the interval [ t m , 1] instead). Then the following equality holds almost everywhere

where the first inequality is due to (12) and the last inequality is by Definition 1. Let β ( u ) := ℓ ( u ) · ∥ y -x ∥ and ϕ ( u ) := ∫ u 0 1 β ( v ) dv . By Lemma A.3, we know that

Then we have

Denote ψ ( u ) := ∫ u 0 1 ℓ ( v ) dv = ϕ ( u ) · ∥ y -x ∥ . We have ψ ( ∥∇ f ( y ) ∥ ) ≤ ψ ( ∥∇ f ( x ) ∥ ) + ∥ y -x ∥ ≤ ψ ( ∥∇ f ( x ) ∥ ) + a ℓ ( ∥∇ f ( x ) ∥ + a ) ≤ ∫ ∥∇ f ( x ) ∥ 0 1 ℓ ( v ) dv + ∫ ∥∇ f ( x ) ∥ + a ∥∇ f ( x ) ∥ 1 ℓ ( v ) dv = ψ ( ∥∇ f ( x ) ∥ + a ) . Since ψ is increasing, we have ∥∇ f ( y ) ∥ ≤ ∥∇ f ( x ) ∥ + a .

With Lemma A.4, we are ready to prove Proposition 3.2.

Proof of Proposition 3.2. We prove the two directions in this proposition separately.

1. An ( r, ℓ ) -smooth function is ℓ -smooth.

For each fixed x ∈ X where ∇ 2 f ( x ) exists and any unit-norm vector w , by Definition 2, we know that for any t ≤ r ( ∥∇ f ( x ) ∥ ) ,

Then we know that

which implies ∥ ∥ ∇ 2 f ( x ) ∥ ∥ ≤ ℓ ( ∥∇ f ( x ) ∥ ) for any point x if ∇ 2 f ( x ) exists.

Then it suffices to show that ∇ 2 f ( x ) exists almost everywhere. Note that for each x ∈ X , Definition 2 states that the gradient function is ℓ ( ∥∇ f ( x ) ∥ ) Lipschitz within the ball B ( x, r ( ∥∇ f ( x ) ∥ )) . Then by Rademacher's Theorem, f is twice differentiable almost everywhere within this ball. Then we can show it is also twice differentiable almost everywhere within the entire domain X as long as we can cover X with countably many such balls. Define S n := { x ∈ X | n ≤ ∥∇ f ( x ) ∥ ≤ n +1 } for integer n ≥ 0 . We have X = ∪ n ≥ 0 S n . One can easily find an internal covering of S n with balls of size r ( n +1) 3 , i.e., there exist { x n,i } i ≥ 0 , where x n,i ∈ S n , such that S n ⊆ ∪ i ≥ 0 B ( x n,i , r ( n + 1)) ⊆ ∪ i ≥ 0 B ( x n,i , r ( ∥∇ f ( x n,i ) ∥ )) . Therefore we have X ⊆ ∪ n,i ≥ 0 B ( x n,i , r ( ∥∇ f ( x n,i ) ∥ )) which completes the proof.

2. An ℓ -smooth function satisfying Assumption 1 is ( r, m ) -smooth where m ( u ) := ℓ ( u + a ) and r ( u ) := a/m ( u ) for any a > 0 .

For any y ∈ R d satisfying ∥ y -x ∥ ≤ r ( ∥∇ f ( x ) ∥ ) = a ℓ ( ∥∇ f ( x ) ∥ + a ) , denote z ( t ) := (1 -t ) x + ty for 0 ≤ t ≤ 1 . We first show y ∈ X by contradiction. Suppose y / ∈ X , let us define t b := inf { 0 ≤ t ≤ 1 | z ( t ) / ∈ X} and z b := z ( t b ) . Then we know z b is a boundary point of X . Since f is a closed function with an open domain, we have

On the other hand, by the definition of t b, we know z ( t ) ∈ X for every 0 ≤ t < t b. Then by Lemma A.4, for all 0 ≤ t < t b, we have ∥∇ f ( z ( t )) ∥ ≤ ∥∇ f ( x ) ∥ + a . Therefore for all 0 ≤ t < t b,

which contradicts (13). Therefore we have shown y ∈ X . Since y is chosen arbitrarily with the ball B ( x, r ( ∥∇ f ( x ) ∥ )) , we have B ( x, r ( ∥∇ f ( x ) ∥ )) ⊆ X . Then for any x 1 , x 2 ∈ B ( x, r ( ∥∇ f ( x ) ∥ )) , we denote w ( t ) := tx 1 +(1 -t ) x 2 . Then we know w ( t ) ∈ B ( x, r ( ∥∇ f ( x ) ∥ )) for all 0 ≤ t ≤ 1 and can obtain

where the last inequality is due to Lemma A.4.

A.3 Proofs of lemmas implied by generalized smoothness

In this part, we provide the proofs of the useful properties stated in Section 3.1.2, including Lemma 3.3, Lemma 3.5, and Corollary 3.6.

Proof of Lemma 3.3. First, note that since ℓ is non-decreasing and r is non-increasing, we have ℓ ( ∥∇ f ( x ) ∥ ) ≤ ℓ ( G ) = L and r ( G ) ≤ r ( ∥∇ f ( x ) ∥ ) . Then by Definition 2, we directly have that B ( x, r ( G )) ⊆ B ( x, r ( ∥∇ f ( x ) ∥ )) ⊆ X , and that for any x 1 , x 2 ∈ B ( x, r ( G )) , we have

Next, for the second inequality in (2), define z ( t ) := (1 -t ) x 2 + tx 1 for 0 ≤ t ≤ 1 . We know z ( t ) ∈ B ( x, r ( G )) . Note that we have shown

Then we have

where the inequality is due to (14).

Proof of Lemma 3.5. If f is ℓ -smooth, by Proposition 3.2, f is also ( r, m ) -smooth where m ( u ) = ℓ (2 u ) and r ( u ) = u/ℓ (2 u ) . Then by Lemma 3.3 where we choose G = ∥∇ f ( x ) ∥ , we have that B ( x, ∥∇ f ( x ) ∥ ℓ (2 ∥∇ f ( x ) ∥ ) ) ⊆ X , and that for any x 1 , x 2 ∈ B ( x, ∥∇ f ( x ) ∥ ℓ (2 ∥∇ f ( x ) ∥ ) ) , we have

Choosing x 2 = x and x 1 = x -∇ f ( x ) ℓ (2 ∥∇ f ( x ) ∥ ) , it is easy to verify that x 1 , x 2 ∈ B ( x, ∥∇ f ( x ) ∥ ℓ (2 ∥∇ f ( x ) ∥ ) ) . Therefore, we have

which completes the proof.

Proof of Corollary 3.6. We first show G < ∞ . Note that since ℓ is sub-quadratic, we know lim u →∞ 2 ℓ (2 u ) /u 2 = 0 . Therefore, for any F > 0 , there exists some M > 0 such that 2 ℓ (2 u ) /u 2 < 1 /F for every u > M . In other words, for any u satisfying u 2 ≤ 2 ℓ (2 u ) · F , we must have u ≤ M . Therefore, by definition of G , we have G ≤ M < ∞ if F > 0 . If F = 0 , we trivially get G = 0 < ∞ . Also, since the set { u ≥ 0 | u 2 ≤ 2 ℓ (2 u ) · F } is closed and bounded, we know its supremum G is in this set and it is also straightforward to show G 2 = 2 ℓ (2 G ) · F .

Then based on the definition of G , we have ∥∇ f ( x ) ∥ ≤ G .

B Analysis of GD for convex functions

In this section, we provide the detailed convergence analysis of gradient descent in the convex setting, including the proofs of Lemma 4.1 and Theorem 4.2, for which the following lemma will be helpful.

Lemma B.1 (Co-coercivity) . If f is convex and ( r, ℓ ) -smooth, for any x ∈ X and y ∈ B ( x, r ( ∥∇ f ( x ) ∥ ) / 2) , we have y ∈ X and

where L = ℓ ( ∥∇ f ( x ) ∥ ) .

Proof of Lemma B.1. Define the Bregman divergences ϕ x ( w ) := f ( w ) -⟨∇ f ( x ) , w ⟩ and ϕ y ( w ) := f ( w ) - ⟨∇ f ( y ) , w ⟩ , which are both convex functions. Since ∇ ϕ x ( w ) = ∇ f ( w ) - ∇ f ( x ) , we have ∇ ϕ x ( x ) = 0 which implies min w ϕ x ( w ) = ϕ x ( x ) as ϕ x is convex. Similarly we have min w ϕ y ( w ) = ϕ y ( y ) .

Denote r x := r ( ∥∇ f ( x ) ∥ ) . Since f is ( r, ℓ ) -smooth, we know its gradient ∇ f is L -Lipschitz locally in B ( x, r x ) . Since ∇ ϕ x ( w ) - ∇ f ( w ) = ∇ f ( x ) is a constant, we know ∇ ϕ x is also L -Lipschitz locally in B ( x, r x ) . Then similar to the proof of Lemma 3.3, one can easily show that for any x 1 , x 2 ∈ B ( x, r x ) , we have

Note that for any y ∈ B ( x, r ( ∥∇ f ( x ) ∥ ) / 2) as in the lemma statement,

where the first inequality uses triangle inequality and ∇ ϕ x ( y ) = ∇ f ( y ) -∇ f ( x ) ; and the second inequality uses Definition 2. It implies that y -1 L ∇ ϕ x ( y ) ∈ B ( x, r x ) . Then we can obtain

where the last inequality uses (15) where we choose x 1 = y -1 L ∇ ϕ x ( y ) and x 2 = y . By the definition of ϕ x , the above inequality is equivalent to

Similar argument can be made for ϕ y ( · ) to obtain

Summing up the two inequalities, we can obtain the desired result.

With Lemma B.1, we prove Lemma 4.1 as follows.

Proof of Lemma 4.1. Let L = ℓ ( G ) . We first verify that x + ∈ B ( x, r ( G ) / 2) . Note that

where we choose η ≤ r ( G ) / (2 G ) . Thus by Lemma B.1, we have

where the first inequality uses Lemma B.1 and the last inequality chooses η ≤ 2 /L .

With Lemma 4.1, we are ready to prove both Theorem 4.2 and Theorem 4.3.

Proof of Theorem 4.2. Denote G := ∥∇ f ( x 0 ) ∥ . Then we trivially have ∥∇ f ( x 0 ) ∥ ≤ G . Lemma 4.1 states that if ∥∇ f ( x t ) ∥ ≤ G for any t ≥ 0 , then we also have ∥∇ f ( x t +1 ) ∥ ≤ ∥∇ f ( x t ) ∥ ≤ G . By induction, we can show that ∥∇ f ( x t ) ∥ ≤ G for all t ≥ 0 . Then the rest of the proof basically follows the standard textbook analysis. We still provide the detailed proof below for completeness.

Note that ∥ x t +1 -x t ∥ = η ∥∇ f ( x t ) ∥ ≤ ηG ≤ r ( G ) , where we choose η ≤ r ( G ) / (2 G ) . Thus we can apply Lemma 3.3 to obtain

where the last inequality chooses η ≤ 1 /L . Meanwhile, by convexity between x t and x ∗ , we have

Note that ( t +1) × (16) + (17) gives

0

f

(

x

t

)

-

f

∗

(

)

+(1 +

t

+

f

(

⟨∇

x

t

f

+1

(

)

x

t

-

)

, x

f

(

∗

x

t

-

)

x

t

⟩

-⟨∇

f

(

x

t

)

, x

t

+1

-

x

t

⟩ -

1

2

η

∥

x

t

+1

-

x

t

∥

Then reorganizing the terms of the above inequality, noting that

we can obtain

The above inequality implies t ( f ( x t ) -f ∗ ) + 1 2 η ∥ x t -x ∗ ∥ 2 is a non-increasing potential function, which directly implies the desired result.

Proof of Theorem 4.3. Since strongly convex functions are also convex, by the same argument as in the proof of Theorem 4.2, we have ∥∇ f ( x t ) ∥ ≤ G := ∥∇ f ( x 0 ) ∥ for all t ≥ 0 . Moreover, (16) still holds. For µ -strongly-convex function, we can obtain a tighter version of (17) as follows.

2

)

.

Let A = 0 and A = (1 + A ) / (1 -ηµ ) for all t ≥ 0 . Combining (16) and (18), we have

Then reorganizing the terms of the above inequality, noting that

we can obtain

The above inequality means A t ( f ( x t ) -f ∗ ) + 1+ ηµA t 2 η ∥ x t -x ∗ ∥ 2 is a non-increasing potential function. Thus by telescoping we have

C Analysis of NAG for convex functions

In this section, we provide the detailed analysis of Nesterov's accelerated gradient method in the convex setting. As we discussed in Section 4.2, the stepsize size choice in Theorem 4.4 is smaller than the classical one. Therefore, we provide a more fine-grained version of the theorem, which allows the stepsize to depend on the degree of ℓ .

Theorem C.1. Suppose f is convex and ℓ -smooth. For α ∈ (0 , 2] , if ℓ ( u ) = o ( u α ) , i.e., lim u →∞ ℓ ( u ) /u α = 0 , then there must exist a constant G such that for L := ℓ (2 G ) , we have

Choose η ≤ min { 1 16 L 3 -2 /α , 1 2 L } . Then the iterates of Algorithm 1 satisfy

Note that when α = 2 , i.e., ℓ is sub-quadratic, Theorem C.1 reduces to Theorem 4.4 which chooses η ≤ min { 1 16 L 2 , 1 2 L } . When α = 1 , i.e., ℓ is sub-linear, the above theorem chooses η ≤ 1 16 L as in the classical textbook analysis up to a numerical constant factor.

Throughout this section, we will assume f is convex and ℓ -smooth, and consider the parameter choices in Theorem C.1, unless explicitly stated. Note that since f is ℓ -smooth, it is also ( r, m ) -smooth with m ( u ) = ℓ ( u + G ) and r ( u ) = G ℓ ( u + G ) by Proposition 3.2. Note that m ( G ) = ℓ (2 G ) = L and r ( G ) = G/L . Then the stepsize satisfies η ≤ 1 / (2 L ) ≤ min { 2 , r ( G ) } .

Before proving Theorem C.1, we first present several additional useful lemmas. To start with, we provide two lemmas regarding the weights { A t } t ≥ 0 and { B t } t ≥ 0 used in Algorithm 1. The lemma below states that B t = Θ( t 2 ) .

Lemma C.2. The weights { B t } t ≥ 0 in Algorithm 1 satisfy 1 4 t 2 ≤ B t ≤ t 2 for all t ≥ 0 .

Proof of Lemma C.2. We prove this lemma by induction. First note that the inequality obviously holds for B 0 = 0 . Suppose its holds up to t . Then we have

Similarly, we have

Lemma C.2 implies the following useful lemma.

Lemma C.3. The weights { A t } t ≥ 0 in Algorithm 1 satisfy that

Proof of Lemma C.3. First, note that it is easy to verify that A s +1 -A s -1 = B s +1 -B s -1 ≥ 0 , which implies each term in the LHS of the above inequality is non-negative. Then we have

The following lemma summarizes the results in the classical potential function analysis of NAG in [d'Aspremont et al., 2021]. In order to not deal with the generalized smoothness condition for now, we directly assume the inequality (20) holds in the lemma, which will be proved later under the generalized smoothness condition.

Lemma C.4. For any t ≥ 0 , if the following inequality holds,

then we can obtain

Proof of Lemma C.4. These derivations below can be found in [d'Aspremont et al., 2021]. We present them here for completeness.

First, since f is convex, the convexity between x ∗ and y t gives

Similarly the convexity between x t and y t gives

Combining the above two inequalities as well as (20) assumed in this lemma, we have

Furthermore, note that

Meanwhile, we have

Thus we have

Plugging back in (23), we obtain

Thus

So we can reorganize (22) to obtain

Then we complete the proof noting that it is easy to verify

In the next lemma, we show that if ∥∇ f ( y t ) ∥ ≤ G , then the condition (20) assumed in Lemma C.4 is satisfied at time t .

Lemma C.5. For any t ≥ 0 , if ∥∇ f ( y t ) ∥ ≤ G , then we have ∥∇ f ( x t +1 ) ∥ ≤ G , and furthermore,

.

Proof of Lemma C.5. As disccued below Theorem C.1, the stepsize satisfies η ≤ 1 / (2 L ) ≤ min { 2 m ( G ) , r ( G ) 2 G } . Therefore we can apply Lemma 4.1 to show ∥∇ f ( x t +1 ) ∥ ≤ ∥∇ f ( y t ) ∥ ≤ G . For the second part, note that ∥ x t +1 -y t ∥ = η ∥∇ f ( y t ) ∥ ≤ G 2 L ≤ r ( G ) , we can apply Lemma 3.3 to show

With Lemma C.4 and Lemma C.5, we can show that ∥∇ f ( y t ) ∥ ≤ G for all t ≥ 0 , as in the lemma below.

Lemma C.6. For all t ≥ 0 , ∥∇ f ( y t ) ∥ ≤ G .

Proof of Lemma C.6. We will prove this lemma by induction. First, by Lemma 3.5 and the choice of G , it is easy to verify that ∥∇ f ( x 0 ) ∥ ≤ G . Then for any fixed t ≥ 0 , suppose that ∥∇ f ( x s ) ∥ ≤ G for all s < t . Then by Lemma C.4 and Lemma C.5, we know that ∥∇ f ( x s ) ∥ ≤ G for all 0 ≤ s ≤ t , and that for all s < t ,

By telescoping (24), we have for all 0 ≤ s < t ,

For 0 ≤ s ≤ t , since ∥∇ f ( x s ) ∥ ≤ G , then Lemma 3.5 implies

Note that by Algorithm 1, we have

Thus we can obtain

Therefore

Thus we have

Since ∥∇ f ( y s ) ∥ ≤ G and ∥ x s +1 -y s ∥ = ∥ η ∇ f ( y s ) ∥ ≤ r ( G ) for s < t , by Lemma 3.3, we have

Thus

Then we complete the induction as well as the proof.

With the three lemmas above, it is straight forward to prove Theorem C.1.

Proof of Theorem C.1. Combining Lemmas C.4, C.5, and C.6, we know the following inequality holds for all t ≥ 0 .

Then by telescoping, we directly complete the proof.

D Analysis of NAG for strongly convex functions

In this section, we provide the convergence analysis of the modified version of Nesterov's accelerated gradient method for µ -strongly-convex functions defined in Algorithm 2.

The convergence results is formally presented in the following theorem.

Theorem D.1. Suppose f is µ -strongly-convex and ℓ -smooth. For α ∈ (0 , 2] , if ℓ ( u ) = o ( u α ) , i.e., lim u →∞ ℓ ( u ) /u α = 0 , then there must exist a constant G such that for L := ℓ (2 G ) , we have

If we choose

Algorithm 2: NAG for µ -strongly-convex functions

input A µ -strongly-convex and ℓ -smooth function f , stepsize η , initial point x 0

1: Initialize z 0 = x 0 , B 0 = 0 , and A 0 = 1 / ( ηµ ) .

2: for t = 0 , ... do

3: B t +1 = 2 B t +1+ √ 4 B t +4 ηµB 2 t +1 2(1 -ηµ )

4: A t +1 = B t +1 + 1 ηµ

5: τ t = ( A t +1 -A t )(1+ ηµA t ) A t +1 +2 ηµA t A t +1 -ηµA 2 t and δ t = A t +1 -A t 1+ ηµA t +1

6: y t = x t + τ t ( z t -x t )

7: x t +1 = y t -η ∇ f ( y t )

8: z t +1 = (1 -ηµδ t ) z t + ηµδ t y t -ηδ t ∇ f ( y t )

9: end for

The iterates generated by Algorithm 2 satisfy

The above theorem gives a gradient complexity of O ( 1 √ ηµ log(1 /ϵ ) ) . Note that Theorem 4.2 shows the complexity of GD is O ( 1 ηµ log(1 /ϵ ) ) . It seems NAG gives a better rate at first glance. However, note that the choices of G,L,η in these two theorems are different, it is less clear whether NAG accelerates the optimization in this setting. Below, we informally show that, if ℓ ( u ) = o ( √ u ) , the rate we obtain for NAG is faster than that for GD.

For simplicity, we informally assume ℓ ( u ) ≍ x ρ with ρ ∈ (0 , 1) . Let G 0 = ∥∇ f ( x 0 ) ∥ . Then for GD, by Theorem 4.2, we have η gd µ ≍ µ/ℓ ( G 0 ) ≍ µ/G ρ 0 . For NAG, since ℓ is sub-linear we can choose α = 1 in the theorem statement. Since f is µ -strongly-convex, by standard results, we can show that f ( x 0 ) -f ∗ ≤ 1 µ G 2 0 and ∥ z 0 -x ∗ ∥ ≤ 1 µ G 0 . Thus the requirement of G in (27) can be simplified as G ≳ ℓ ( G ) · G 0 /µ , which is satisfied if choosing G ≍ ( G 0 /µ ) 1 / (1 -ρ ) . Then we also have η nag ≍ 1 ℓ ( G ) ≍ ( µ/G 0 ) ρ/ (1 -ρ ) . Thus √ η nag µ ≍ ( µ/G ρ 0 ) 1 / (2 -2 ρ ) . This means whenever 1 / (2 -2 ρ ) < 1 , i.e., 0 ≤ ρ < 1 / 2 , we have √ η nag µ ≳ η gd µ , which implies the rate we obtain for NAG is faster than that for GD.

In what follows, we will provide the proof of Theorem D.1. We will always use the parameter choices in the theorem throughout this section.

D.1 Useful lemmas

In this part, we provide several useful lemmas for proving Theorem D.1. To start with, the following two lemmas provide two useful inequalities.

Lemma D.3. For all 0 < p ≤ 1 and t ≥ 0 , we have

It is obvious that f ( t ) ≥ 0 for t ≤ 2 √ p log( e + 1 p ) . For t > 2 √ p log( e + 1 p ) , we have

Thus f is non-decreasing and which implies τ t · δ s ≤ 1 .

Proof of Lemma D.4. By Algorithm 2, it is easy to verify

This implies

Thus

where in the inequality, we use the fact that B s is non-decreasing with s . Therefore

Thus we have

Lemma D.5. If 0 < ηµ < 1 , then for any t ≥ 1 , we have

Thus

In the next four lemmas, we provide several useful inequalities regarding the weights { A t } t ≥ 0 and { B t } t ≥ 0 used in Algorithm 2.

Lemma D.4. For all s ≤ t , we have

Proof of Lemma D.5. For t ≥ 1 , we have B t ≥ 1 thus

On the other hand, we have

Thus

Lemma D.6. For 0 < ηµ < 1 and t ≥ 1 , we have

Combined with Lemma D.5, we have the desired result.

Lemma D.7. For t ≥ 1 , we have

Proof of Lemma D.7. By Lemma D.5, we have

t -1 ∑ s =0 √ A s +1 A t = t -1 ∑ s =0 √ B s +1 +1 / ( ηµ ) A t ≤ t -1 ∑ s =0 √ B s +1 A t + t √ ηµA t ≤ 3 + 1 √ ηµA t · 2 √ ηµ log( e + 1 ηµ )( ηµ (1 + √ ηµ ) t +1) (by Lemma D.6 and Lemma D.3) ≤ 3 + 4 log( e + 1 ηµ ) . (by Inequality (29))

Proof of Lemma D.6.

Thus, we have

D.2 Proof of Theorem D.1

With all the useful lemmas in the previous section, we proceed to prove Theorem D.1, for which we need several additional lemmas. First, similar to Lemma C.4, the following lemma summarizes the results in the classical potential function analysis of NAG for strongly convex functions in [d'Aspremont et al., 2021].

Lemma D.8. For any t ≥ 0 , if the following inequality holds

then we can obtain

Proof of Lemma D.8. These derivations can be found in d'Aspremont et al. [2021]. We present it here for completeness.

The strong convexity between x ∗ and y t gives

The convexity between x t and y t gives

Combining the above two inequalities and the one assumed in this lemma, we have

Reorganizing we can obtain

Then we complete the proof noting that

Next, note that Lemma C.5 still holds in the strongly convex setting. We repeat it below for completeness.

Lemma D.9. For any t ≥ 0 , if ∥∇ f ( y t ) ∥ ≤ G , then we have ∥∇ f ( x t +1 ) ∥ ≤ G , and furthermore,

With Lemma D.8 and Lemma D.9, we will show that ∥∇ f ( y t ) ∥ ≤ G for all t ≥ 0 by induction in the following lemma.

Lemma D.10. For all t ≥ 0 , we have ∥∇ f ( y t ) ∥ ≤ G .

Proof of Lemma D.10. We will prove this lemma by induction. First, by Lemma 3.5 and the choice of G , it is easy to verify that ∥∇ f ( x 0 ) ∥ ≤ G . Then for any fixed t ≥ 0 , suppose that ∥∇ f ( x s ) ∥ ≤ G for all s < t . Then by Lemma D.8 and Lemma D.9, we know that ∥∇ f ( x s ) ∥ ≤ G for all 0 ≤ s ≤ t , and that for all s < t ,

By telescoping (30), we have for all 0 ≤ s < t ,

For 0 ≤ s ≤ t , since ∥∇ f ( x s ) ∥ ≤ G , then Lemma 3.5 implies

Thus

Therefore

Moreover and

Thus we have

where the second inequality follows from Lemma D.4. We further control term I by

Thus we have

Since ∥∇ f ( x t ) ∥ ≤ G and we just showed ∥ x t -y t ∥ ≤ r ( G ) , by Lemma 3.3, we have

Then we complete the induction as well as the proof.

Proof of Theorem D.1. Combining Lemmas D.8, D.9, and D.10, we know the following inequality holds for all t ≥ 0 .

Then by telescoping, we get

Finally, applying Lemma D.5, we have A t = B t + 1 / ( ηµ ) ≥ 1 / (1 - √ ηµ ) t -1 + 1 / ( ηµ ) . Thus completes the proof.

E Analysis of GD for non-convex functions

In this section, we provide the proofs related to analysis of gradient descent for non-convex function, including those of Lemma 5.1 and Theorem 5.2.

Proof of Lemma 5.1. First, based on Corollary 3.6, we know ∥∇ f ( x ) ∥ ≤ G < ∞ . Also note that

Then by Lemma 3.3 and Remark 3.4, we have x + ∈ X and

Proof of Theorem 5.2. By Lemma 5.1, using induction, we directly obtain f ( x t ) ≤ f ( x 0 ) for all t ≥ 0 . Then by Corollary 3.6, we have ∥∇ f ( x t ) ∥ ≤ G for all t ≥ 0 . Following the proof of Lemma 5.1, we can similarly show

Taking a summation over t < T and rearanging terms, we have

F Analysis of SGD for non-convex functions

In this section, we provide the detailed convergence analysis of stochastic gradient descent for ℓ -smooth and non-convex functions where ℓ is sub-quadratic.

We first present some useful inequalities related to the parameter choices in Theorem 5.3.

Lemma F.1. Under the parameters choices in Theorem 5.3, the following inequalities hold.

Proof of Lemma F.1. First note that by Corollary 3.6, we know

i.e., σL ≤ G 2 δ/ 16 . Then since we choose η ≤ 1 4 G √ T , we have

Next, we show the useful lemma which bounds E [ f ( x τ ) -f ∗ ] and E [ ∑ t<τ ∥∇ f ( x t ) ∥ 2 ] simultaneously.

Lemma F.2. Under the parameters choices in Theorem 5.3, the following inequality holds

Proof of Lemma F.2. If t < τ , by the definition of τ , we know f ( x t ) -f ∗ ≤ F and ∥ ϵ t ∥ ≤ G 5 ηL , and the former also implies ∥∇ f ( x t ) ∥ ≤ G by Corollary 3.6. Then we can bound

where we use the choice of η ≤ 1 2 L . Then based on Lemma 3.3 and Remark 3.4, for any t < τ , we have

where the equality is due to (4); the second inequality uses g t = ϵ t + ∇ f ( x t ) and Young's inequality ∥ y + z ∥ 2 ≤ 2 ∥ y ∥ 2 +2 ∥ z ∥ 2 for any vectors y, z ; and the last inequality chooses η ≤ 1 / (2 L ) . Taking a summation over t < τ and rearanging terms, we have

Now we bound the last two terms on th RHS. First, for the last term, we have

where the first inequality uses τ ≤ T by its defnition; and in the last inequality we use Assumption 4. For the cross term, note that E t -1 [〈 ∇ f ( x t ) , ϵ t 〉] = 0 by Assumption 4. So this term is a sum of a martingale difference sequence. Since τ is a stopping time, we can apply the optional stopping theorem to obtain

Then we have

where the equality is due to (34); the first inequality uses ∥∇ f ( x τ ) ∥ ≤ G by the definition of τ in (5) and Corollary 3.6; the fourth inequality uses E [ X ] 2 ≤ E [ X 2 ] for any random variable X ; and the last inequality uses Assumption 4.

Combining all the bounds above, we get

where the last inequality is due to Lemma F.1.

With Lemma F.2, we are ready to prove Theorem 5.3.

Proof of Theorem 5.3. We want to show the probability of { τ < T } is small, as its complement { τ = T } means f ( x t ) -f ∗ ≤ F for all t ≤ T which implies ∥∇ f ( x t ) ∥ ≤ G for all t ≤ T . Note that

Therefore we only need to bound the probability of each of these two events on the RHS. We first bound P ( τ 2 < T ) . Note that

where the first inequality uses union bound; the second inequality applies Chebyshev's inequality and E [ ∥ ϵ t ∥ 2 ] = E [ E t -1 [ ∥ ϵ t ∥ 2 ]] ≤ σ 2 for each fixed t by Assumption 4; the last inequality uses Lemma F.1.

Next, we will bound P ( τ 1 < T, τ 2 = T ) . Note that under the event { τ 1 < T, τ 2 = T } , we know that 1) τ = τ 1 < T which implies f ( x τ +1 ) -f ∗ > F ; and 2) τ < T = τ 2 which implies ∥ ϵ τ ∥ ≤ G 5 ηL by the definition in (5). Also note that we always have f ( x τ ) -f ∗ ≤ F which implies ∥∇ f ( x τ ) ∥ ≤ G by Corollary 3.6. Then we can show

where we choose η ≤ 1 2 L . Then based on Lemma 3.3 and Remark 3.4, we have

where the first inequality is obtained following the same derivation as in (33); the last equality is due to Corollary 3.6. Therefore we can show that under the event { τ 1 < T, τ 2 = T } ,

Hence,

where the second inequality uses Markov's inequality; the third inequality uses Lemma F.2; and in the last inequality we choose F = 8( f ( x 0 ) -f ∗ + σ ) /δ .

Therefore we can show

Then we also know P ( τ = T ) ≥ 1 -δ/ 2 ≥ 1 / 2 . Therefore, by Lemma F.2,

Then we have

where the last inequality uses the choice of T . Let E := { 1 T ∑ t<T ∥∇ f ( x t ) ∥ 2 > ϵ 2 } denote the event of not converging to an ϵ -stationary point. By Markov's inequality, we have P ( E ) ≤ δ/ 2 . Therefore we have P ( { τ < T } ∪ E ) ≤ δ , which completes the proof.

G Lower bound

In this section, we provide the proof of Theorem 5.4.

Proof of Theorem 5.4. Let c, η 0 > 0 satisfy η 0 ≤ c 2 / 2 . Consider

where c > 0 is a constant and y = ( c + √ c 2 +2 η 0 ) / 2 > 0 is the fixed point of the iteration

and k , b are chosen in such a way that f ( x ) and f ′ ( x ) are continuous. Specifically, choose k = c -1 f ′ ( c/ 2) and b = f ( c/ 2) -cf ′ ( c/ 2) / 4 . Since f ( -x ) = f ( x ) , f ( x ) is symmetric about the line x = 0 . In a small neighborhood, f ( x ) is symmetric about ( y, f ( y )) , so f ′ ( x ) is continuous at y .

Let us first consider the smoothness of f . By symmetry, it suffices to consider x > 0 . Then,

Hence, f ( x ) is (2 , 2 k, 1) -smooth.

Note that f ( x ) has a stationary point 0 . For stepsize η f satisfying η 0 ≤ η f ≤ c 2 / 4 , there exists z = ( c + √ c 2 +2 η f ) ≥ y such that -z = z -η f ( y -c ) -1 and by symmetry, once x τ = z , x t = ± z for all t ≥ τ , making the GD iterations stuck. Now we choose a proper x 0 such that f ′ ( x 0 ) and f ( x 0 ) -f (0) are bounded.

We consider arriving at y from above. That is, x 0 ≥ x 1 ≥ . . . x τ = z > c > 0 . Since in each update where x t +1 = x t -η f ( x t -c ) -1 > c ,

Hence, we can choose τ in such a way that 3 c/ 2 ≤ x 0 < 3 c/ 2 + √ η f . Then,

By definition, y -c = η 0 ( c + √ c 2 +2 η 0 ) -1 . Hence,

Its Hessian is given by

Then,

For stepsize η f < η 0 , reaching below 4 c/ 3 takes at least

steps to reach 4 c/ 3 , where f ′ (4 c/ 3) = log( c/ 3) .

Now we set c and η 0 and scale function f ( x ) to satisfy the parameter specifications L 0 , L 2 , G 0 , ∆ 0 . Define g ( x ) = L -1 2 f ( x ) . Then, g ( x ) is (2 , 2 kL -1 2 , L 2 ) -smooth. Since the gradient of g ( x ) is L -1 2 times f ( x ) , the above analysis for f ( x ) applies to g ( x ) by replacing η 0 with η 1 = L 2 η 0 and η f with η = L 2 η f . To ensure that

it suffices to take c ≥ 2 / √ L 0 L 2 . To ensure that

it suffices to take c ≥ 2 / ( L 2 G 0 ) . To ensure that

it suffices to take

Since we require η 1 ≤ c 2 / 2 , parameters L 2 and ∆ 0 need to satisfy

that is, L 2 ∆ 0 ≥ 5 . 5 log 2 + 0 . 5 , which holds because L 2 ∆ 0 ≥ 10 . Take c = max { 2 / √ L 0 L 2 , 2 / ( L 2 G 0 ) , √ 8 /L 0 } . Then, as long as η ≤ 2 /L 0 , the requirement that η ≤ c 2 / 4 is satisfied. Therefore, on g ( x ) with initial point x 0 , gradient descent with a constant stepsize either gets stuck, or takes at least

steps to reach a 1 -stationary point.

On the other hand, if η > 2 /L 0 , consider the function f ( x ) = L 0 2 x 2 . For any x t = 0 , we always have | x t +1 | / | x t | = | 1 -ηL 0 | > 1 , which means the iterates diverge to infinity.

̸

Peer reviews:
comments: SUMMARY:
This submission claims to relax the Lipschitz assumption on the gradient of the objective function in nonconvex optimisation, and obtains convergence bounds similar to textbook convergence results.

STRENGTHS:
The presentation of the material is clear and the narrative flows well.

WEAKNESSES:
The result presented in this manuscript is not of importance. When we take the gradient information about the initialisation point into consideration, we can correspondingly restrict the feasibility set to the neighbourhood around the initialisation point, and then the $(r,l)$-smoothness defined in this paper can be replaced by the classical Lipschitz condition.

QUESTIONS:
In equation (5), why is there a $-1$ in the definition of $S_{\textrm{rect}}$?

LIMITATIONS:
As there is no novel discovery in this submission, the limitation discussion is not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRelaxed smoothness conditions have been introduced to study the gradient clipping algorithm, and show that clipping in particular allows fixed step-size convergence without smoothness under this relaxed assumption. This paper further generalizes the relaxed smoothness notion used for clipping by bounding the Hessian by any non-decreasing function of the gradient norm. \n\nThen, fixed step-size convergence is shown under the relaxed smoothness assumption in a variety of settings. The key technical part is to show that assuming a large enough initial bound on the gradients and relaxed smoothness, the gradients remain bounded throughout the trajectory regardless of the setting (convex, strongly convex, non-convex, and even stochastic with some necessary caveats). Then, boundedness of the gradients along the trajectory implies the regular smoothness condition, and thus standard analyses can be used to show convergence of gradient descent.\n\nSTRENGTHS:\n- Shows convergence of GD, NAG and SGD for relaxed smoothness conditions, which were not known without gradient clipping. This allows to prove convergence of GD for general classes of functions. \n- Show boundedness of the gradients along trajectories under relaxed smoothness conditions. This is a nice-to-have technical result, especially for NAG.\n\nWEAKNESSES:\n- Fixed step-size rates seem quite pessimistic in this case, since step-sizes all along the trajectory depend on the initial bound on smoothness, which might be very large.\n\nQUESTIONS:\n1) The main argument is that global bounds hold on the gradients. Although it is good to have, this potentially leads to very small step-sizes even in regions for which gradients are small. \nWould it be possible to improve those bounds, and show that the gradients actually decrease along the trajectories (e.g., in the strongly convex setting), in order to allow for increasingly large step-sizes? If so, what would the decrease rate be?\n\n2) How do results in the paper compare with simple clipped (S)GD with clipping threshold set to the gradient norm at $x_0$ (basically, enforce boundedness instead of showing it) ? Convergence rates seem equivalent but additional terms seem different in each case. Could you clarify that?\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper generalizes the recently introduced (L0,L1)-smoothness which itself extends the L-smoothness which is key in analyzing rates of convergence of optimization algorithms. The authors introduce the concept of $\\ell$-smoothness where $\\ell$ is a function of the gradient of the function to minimize (e.g.: a polynomial).\nThe key idea is to show that under the $\\ell$-smoothness assumption, the gradient remains bounded along the iterates of a given algorithm.\nThe authors focus on smooth optimization and tackle both convex and non-convex settings, in which they (roughly) recover the bounds already known for L-smooth functions and show whether they hold or not with generalized smoothness depending on the function $\\ell$.\n\nSTRENGTHS:\nThe paper is well written, and the presentation is clear. The mathematical statements are rigorous and I did not spot mistakes. The paper builds on existing works on (L0-L1)-smoothness but significantly extends them and covers many important settings (convex/non-convex, etc.). Additionally, I found the reasoning interesting and the proof techniques used depart from the classical ones (especially in the non-convex setting). \nOverall, I think that this paper brings a significant contribution to the important question of the convergence of optimization algorithms. My general feeling is very positive.\n\nWEAKNESSES:\nOn the theoretical side I think that there are some minor bugs (see questions below), but nothing really problematic.\nHowever, while the paper has extensive theoretical results, it does not provide any numerical experiments, which is important for machine learning and optimization papers. It would have been, for example, very informative to see how the theory allows making GD and NAG converging beyond Lipschitz assumption by using the author's step-sizes conditions.\nThe related work section seems too short to me given that gradient-based optimization is a very active topic of research. For example the authors call many results as \"well known\" where instead credit could have been given.\n\nQUESTIONS:\n1) Line 106 on the fact that f has to tend to infinity. First it might be good to specify +infinity (instead of infinity) to remove any ambiguity. Then I think that the statement is not true in its current form since R^d is an open set, yet there, the epigraph of f is closed without coercivity assumptions (as the authors discuss after). I think that the discussion is valid for *bounded* open domains. Could you clarify/correct this?\n\n2) Line 157. Following my previous question, the authors motivate the use of a domain X so as to include functions like logarithms and rational functions. However it seems that these functions are not closed (eg: log(x) when x -> 0), so Assumption 2 does not hold. Could you comment on that?\n\n3) Line 76-78. In the paper, and notably at these lines, there is a confusion between NAG which is indeed optimal for convex functions, and Heavy Ball with Friction (HBF). Indeed, NAG while being similar to HBF is not optimal in the strongly convex setting but HBF is. This should be clarified. (l76-78)\n\n4) On Assumption 3 (existence of a minimizer). While being reasonable this makes the studied framework more restrictive than the classical one (where the optimal rate is also valid for functions whose minimizer is at infinity). Therefore the results are not valid for the whole class of convex functions (but a very large subset of it), this should be a little more emphasized while discussing the contributions so as not to mislead the reader in the introduction. \n\n5) It would be better to define rigorously \"sub-quadratic\", in particular, does it include quadratic functions or is it strictly sub-quadratic? \n\n6) Do the authors have any idea on whether their modified NAG (Alg. 1) might improve NAG even when used on L-smooth functions? (even though NAG is already theoretically optimal). If not, is it worse than the vanilla version or does it behave similarly? Numerical experiments would be insightful here too.\n\n7) Typos: l76: strong convex -> strongly convex\nl207: line 6 of the algorithm -> Is it not rather line 4 of the algorithm?\n\nLIMITATIONS:\nSome limitations are discussed, in particular those related to NAG in the non-convex setting. The limitations about existence of a minimizer might be further discussed (see questions). \nSocietal impact is not really applicable here.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new assumption generalizing classical smoothness, and named $\\ell$-smoothness, motivates it by giving providing examples of non-smooth functions belonging to this class, and studies classical algorithms under this assumption.\n\nSTRENGTHS:\n- The paper is clear and fairly compared to related work.\n- The new class is well defined and motivated, and authors show that we can obtain results under it.\n\nWEAKNESSES:\n$\\underline{\\text{General remarks}}$:\n\n- First, I would like to make a somewhat subjective statement about the class: in my humble opinion, it seems a bit flawed as it does not respect fundamental homogeneity properties. Let me explain:\n\nLet $f\\in\\mathcal{F}_{\\ell}$ the class of $\\ell$-smooth functions. For $\\alpha, \\lambda > 0$, define $g$ as $g(x)=\\frac{\\alpha}{\\lambda^2}f(\\lambda x)$ (assume wlog that their minimum is in 0, otherwise translate $f$, then create $g$).\nWe verify $\\nabla g(x)=\\frac{\\alpha}{\\lambda}\\nabla f(\\lambda x)$ and $\\nabla^2 g(x)=\\alpha\\nabla^2 f(\\lambda x)$,\nhence $\\|\\nabla^2 g(x)\\| = \\|\\alpha\\nabla^2 f(\\lambda x)\\| \\leq \\alpha \\ell( \\|\\nabla f(\\lambda x)\\|) = \\alpha \\ell( \\frac{\\lambda}{\\alpha}\\|\\nabla g(x)\\|)$.\n\nFinally, $g\\in\\mathcal{F}_{\\alpha \\ell(\\frac{\\lambda}{\\alpha} .)}$.\n\nLet us assume that after some analysis of an algorithm like GD (this reasoning also applies to momentum), we find out that the step-size that achieves the best worst-case guarantee on the class $\\mathcal{F}_{\\ell}$ is $\\gamma_{\\ell}$.\n\nThen, if I need to optimize $f$, I will use $x_{t+1} = x_t - \\gamma_{\\ell} \\nabla f (x_t)$.\n\nNow, instead I minimize $g$, I will use $x_{t+1} = x_t - \\gamma_{\\alpha \\ell(\\frac{\\lambda}{\\alpha} .)} \\nabla g (x_t) = x_t - \\gamma_{\\alpha \\ell(\\frac{\\lambda}{\\alpha} .)} \\frac{\\alpha}{\\lambda}\\nabla f(\\lambda x_t)$.\n\nNote that $f$ and $g$ have the same minimum and if we introduce the iterates $y_t = \\lambda x_t$, we have\n\n$x_{t+1} = x_t - \\gamma_{\\ell} \\nabla f (x_t)$ when minimizing $f$,\n\nand\n\n$y_{t+1} = y_t - \\gamma_{\\alpha \\ell(\\frac{\\lambda}{\\alpha} .)} \\alpha\\nabla f(y_t)$ when minimizing $g$.\n\nIn short, we need to have $\\gamma_{\\ell} = \\alpha \\gamma_{\\alpha \\ell(\\frac{\\lambda}{\\alpha} .)}$, or again $ \\gamma_{\\alpha \\ell(\\frac{\\lambda}{\\alpha} .)} = \\frac{\\gamma_{\\ell}}{\\alpha}$.\nIndeed, they both optimize their dynamics.\n\nThis shows that $\\lambda$ has no impact on the optimal way to tune an algorithm. And we can stretch the function $\\ell$ as much as we want and notice that probably only $\\ell(0)$ matters, i.e. the smoothness constant in the optimum.\nIn particular, applied to the case where $\\ell(x) = L_0 + L_1x$, it is clear that the optimal parameter cannot depends on $L_1$.\n\nThis observation is explained by the lack of homogeneity in the formula: when scaling a function, only $L_0$ is scaled, not $L_1.$\nIn $L$-smooth class, the worst-case function generally do not belong to the $L-\\varepsilon$-smooth class. Hence there is a hierarchy of the classes, but here, it seems that by scaling a function, the worst-case dynamics does not depend on some part of the class definition, leading to useless specification.\n\nThis point can be further discussed. This is just some thoughts I had based on the class definition which I never used and I am aware from the related works section that some people are working with it. Thus I would be happy if authors could discuss this point.\n\nHowever, based on this remark, my first guess was that the analysis would basically lead as the same analysis as when $\\ell$ is constant, or $L_1=0$, which leads to my next point.\n\n- Second, we indeed recover classical analyses everywhere in this paper. There indeed is an additional argument, that is the sequence of iterate stays in a compact and by regularity of $f$, we can bound the gradients, hence the hessians and conclude with all the classical analyses.\nIndeed:\n    - Th4.2 and 4.3 use classical Lyap analyses.\n    - Lemma 4.1: Using cocoercivity, authors prove $\\|\\nabla f(x_t)\\|$ is decreasing.  Using the same proof, one could have that $\\|x_t-x_\\star\\|$ is decreasing as well. And using descent lemma, we have that $f(x_t)$ is also decreasing.\nFinally, we can prove the descent lemma without assumption of $\\ell$-smoothness by just assuming the continuity of the hessian, and taking G as upper bound of $\\lbrace \\|\\nabla^2 f(x)\\| | x\\in\\mathcal{X} \\text{ and } f(x)\\leq f(x_0) \\rbrace$ which is assumed to be a compact since authors assume that the function tends to infinity on the border of $\\mathcal{X}$. We have the revisited descent lemma: $f(x_{t+1}) \\leq f(x_t) - \\eta\\|\\nabla f(x_t)\\|^2 + \\frac{1}{2}\\eta^2 G \\|\\nabla f(x_t)\\|^2$, and by taking $\\eta$ sufficiently small, we insure in 1 calculus that\n    - 1) $f(x_t)$ is decreasing and that all the hessians keep being smaller than $G$.\n    - 2) the squared gradients are sommable, hence the classical complexity $O(1/\\varepsilon^2)$ that is as in thm5.1, obtained in a much simpler way.  Of course, here we only assumed the hessian to be bounded, so $\\ell$-smoothness + bounded gradients do the job.\n\nIn conclusion, in my opinion, most of the results are almost straightforward from what is known in the literature.\n\n- Third, in my opinion, the stochastic assumption A4 is too strong. I am aware authors claim that some works in the literature assume even stronger assumption, but A4 is way too strong: no multiplicative noise, only additive. This paper claims generalizing smoothness, but linear regressions with MSE losses are not even covered by the section 5.2.  Plus, bounded variance are too easy to handle in general and leads to an analysis close the deterministic case.  Instead, authors should consider expected smoothness (or its equivalent in $\\ell$-smoothness). Actually, under the assumption that there is a finite number of functions under consideration, I would guess we could generalize the same arguments as in the deterministic case to ensure all the hessians to be bounded on the optimization iterates and basically use classical SGD proof in the smooth case.\n\n$~$\n\n$\\underline{\\text{Minor}}$:\n\n- Clarity:\n\n    - Assumption 1: First recall definition of « closed function »\n    - Also define « sub-quadratic »\n    - l.468: It took me a while to find where this proof was. Please do as for other propositions: state it right before the proof. In a general way, please state all theorems right before their proofs if reported in appendix. Use the « restatable » latex package too avoid renumbering.\n    - Proof of lemma B1: From convexity and local smoothness, authors apply the $\\underline{\\text{descent lemma}}$ on the $\\underline{\\text{Bregman divergence}}$ of the objective function to obtain local $\\underline{\\text{cocoercivity}}$. The proof is the same as for global smoothness. Yet, for completness and care of the neighborhood needed to obtain the cocoercivity, I understand that the proof is provided. However, It needs a reference to this classical result in the global smooth case and mention of the 3 underlined terms when used.\n        - l.550: Cocoercivity\n        - l.553: Bregman divergences\n        - l.562: Descent Lemma\n    - l.560: please introduce y: « let y [as in the lemma statement] »\n    - l.248: « Theorem 5.1 gives the classical $O(1/T)$ rate, or $O(1/\\varepsilon^2)$ gradient complexity » -> $O(1/\\sqrt{T})$. Authors do not clearly state if they are talking here about gradient norm or its square, but they need to be consistent when talking about rate and complexity.\n\n- Typos:\n    - l.2: \"Lipshitzness\" -> Lipschitz continuity.\n    - l.76: \"strong convex\" -> strongly convex\n    - l.79: $\\nabla$ is missing\n    - l.144: « $x = x_2 = x_t$ » ? I guess \"$x$\" needs to be removed\n    - l.169: « accelearted » -> accelerated\n    - l.570: 1/L -> 2/L\n\n\n- Misc:\n    - Table 1: missing hline + be precise on the meaning of « - ».\n    - Table 1: for GD non convex: « Inf or $\\Omega$ … » could be summed up as «  $\\Omega$ … ».\n    - l.121-122: Put 2) under 1). It should not exceed the current number of lines.\n\nQUESTIONS:\nCan authors discuss my first point?\n\nAlso, can they try to study SGD under expected smoothness assumption?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper generalizes the classic Lipschtiz smooth gradient condition, as well as a recent improvement. The proposed condition is essentially saying the Hessian norm is bounded by a non-decreasing function of the norm of gradient. Using such conditions, the authors proved convergence rates of gradient descent under various settings (convex, strongly convex, non-convex, stochastic). Classic optimal rates are recovered.\n\nSTRENGTHS:\nThe paper is well presented. The technical contribution looks original and significant, as it is a generalization of the classic smoothness. Intuitions are shared for proving the convergence.\n\nWEAKNESSES:\nOverall, I do not have major concerns. \n\n1. The proof of Theorem 5.1, in particular the split of two time steps in (4) seems delicate and interesting, but also a bit out of blue to me. Is this splitting novel, or did part of it show up in the literature? What motivates such a splitting? \n2. Do you have examples of objective functions that show up in machine learning or other applications, such that they satisfy the generalized smoothness, but not the conventional smoothness? This would add much significance to the paper.\n3. It was not clear to me when we assume the function to be C^2 and when not. In the introduction, It appears at first sight that only twice differentiable functions are considered. Then I saw definition 2, so it is not true. Then I got confused again at Proposition 3.3: it seems that for l-smooth to hold you need f to be C^2. Did you assume it somewhere?\n4. Line 27: “provide a lower bound”. Do you mean on the number of iterations or on the error, or both?\n5. Table 1: Why is it called gradient complexity? Is it the number of times the gradient oracle needs to be queried? I thought a more common name is iteration complexity, but maybe I am wrong. \n6. Lines 46-48: “if gradients along the trajectory are bounded by a constant G, then the Hessian norms are bounded by the constant ℓ(G).” Why is this “if-then” true? Say we look at a univariate objective function f: R -> R. Its gradient g is also univariate. Are you saying that if g is bounded by a constant G, then |nabla g| is also bounded? This can not be true, since nabla g can be arbitrarily large.\n7. Line 77: “condition number” of what? \n8. Proposition 3.6: “If” -> “Suppose”\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}