Title: Directional Smoothness and Gradient Methods: Convergence and Adaptivity
Venue: NeurIPS 2024
Accepted: yes
Keywords: directional smoothness

gradient descent

exponential search

polyak stepsize

normalized gradient descent
Abstract: We develop new sub-optimality bounds for gradient descent (GD) that depend on the conditioning of the objective along the path of optimization, rather than on global, worst-case constants.  Key to our proofs is directional smoothness, a measure of gradient variation that we use to develop upper-bounds on the objective.  Minimizing these upper-bounds requires solving implicit equations to obtain a sequence of strongly adapted step-sizes; we show that these equations are straightforward to solve for convex quadratics and lead to new guarantees for two classical step-sizes.  For general functions, we prove that the Polyak step-size and normalized GD obtain fast, path-dependent rates despite using no knowledge of the directional smoothness.  Experiments on logistic regression show our convergence guarantees are tighter than the classical theory based on $L$-smoothness.

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

Abstract

We develop new sub-optimality bounds for gradient descent (GD) that depend on the conditioning of the objective along the path of optimization rather than on global, worst-case constants. Key to our proofs is directional smoothness, a measure of gradient variation that we use to develop upper-bounds on the objective. Minimizing these upper-bounds requires solving implicit equations to obtain a sequence of strongly adapted step-sizes; we show that these equations are straightforward to solve for convex quadratics and lead to new guarantees for two classical step-sizes. For general functions, we prove that the Polyak step-size and normalized GD obtain fast, path-dependent rates despite using no knowledge of the directional smoothness. Experiments on logistic regression show our convergence guarantees are tighter than the classical theory based on L -smoothness.

1 Introduction

Gradient methods for differentiable functions are typically analyzed under the assumption that f is L -smooth, meaning ∇ f is L -Lipschitz continuous. This condition implies f is upper-bounded by a quadratic and guarantees that gradient descent (GD) with step-size η < 2 /L decreases the optimality gap at each iteration (Bertsekas, 1997). However, experience shows that GD can still decrease the objective when f is not L -smooth, particularly for deep neural networks (Bengio, 2012; Z. Li et al., 2020; J. Cohen et al., 2021). Even for functions verifying smoothness, convergence rates are often pessimistic and fail to predict optimization speed in practice (Paquette et al., 2023).

One alternative to global smoothness is local Lipschitz continuity of the gradient ('local smoothness'). Local smoothness assumes different Lipschitz constants hold for different neighbourhoods, which avoids global assumptions and improves rates. However, such analyses typically rely on boundedness of the iterates and then use local smoothness to obtain L -smoothness over a compact set (Malitsky and Mishchenko, 2020). Boundedness is guaranteed in several ways: Junyu Zhang and Hong (2020) break optimization into stages, Patel and Berahas (2022) use stopping-times, and Lu and S. Mei (2023) employ a line-search. Unfortunately, these approaches modify the underlying optimization algorithm, require local smoothness oracles (Park et al., 2021), or rely on highly complex arguments.

In contrast, we prove simple rates for GD without global smoothness by deriving bounds of the form,

50

100

150

200

Iterations

10

-

2

10

0

10

2

Optimality Gap

ionosphere

10

20

30

40

Iterations

10

-

10

10

-

6

10

-

2

mammographic

1

/M

(

xk

+1

, x

k

)

Polyak

Bound (

L

-Smooth)

Bound (

1

/M

(

xk

+1

, x

k

)

)

Bound (Polyak)

where the directional smoothness function M ( x k +1 , x k ) depends only on properties of f along the chord between x k and x k +1 . Our sub-optimality bounds provide a path-dependent perspective on GD and are tighter than conventional analyses when the step-size sequence is adapted to the directional smoothness, meaning η k < 2 /M ( x k +1 , x k ) . See Figure 1 for two real-data examples highlighting our improvement over classical rates. We summarize all our contributions as follows.

Directional Smoothness. We introduce three constructive directional smoothness functions M ( x, y ) . The first, point-wise smoothness, depends only on the end-points x, y and is easily computed, while the second, path-wise smoothness, yields a tighter bound, but depends on the chord C = { αx +(1 -α ) y : α ∈ [0 , 1] } . The last function, which we call the optimal point-wise smoothness, is both easy-to-evaluate and provides the tightest possible quadratic upper bound.

Sub-optimality bounds. We leverage directional smoothness functions to prove new sub-optimality bounds for GD on convex functions. Our bounds are localized to the GD trajectory, hold for any step-size sequence, and are tighter than the classic analysis using L -smoothness. They are also more general since we do not need to assume that f is globally L -smooth to show progress; all we require is a sequence of step-sizes adapted to the directional smoothness function. Furthermore, our approach extends naturally to acceleration, allowing us to prove optimal rates for (strongly)-convex functions.

Adaptive Step-Sizes in the Quadratic Case. In the general setting, computing step-sizes which are adapted to the directional smoothness requires solving a challenging non-linear root-finding problem. For quadratic problems, we show that the ideal step-size that satisfies η k = 1 /M ( x k +1 , x k ) is the Rayleigh quotient and is connected to the hedging algorithm (Altschuler and Parrilo, 2023).

Exponential Search. Moving beyond quadratics, we prove that the equation η k = 1 /M ( x k +1 , x k ) admits a solution under mild conditions, meaning ideal step-sizes can be computed using Newton's method. Since computing these step-sizes is typically impractical, we adapt exponential search (Carmon and Hinder, 2022) to obtain similar path-dependent complexities up to a log-log penalty.

Polyak and Normalized GD. More importantly, we show that the Polyak step-size (Polyak, 1987) and normalized GD achieve fast, path-dependent rates without knowledge of the directional smoothness. Our analysis reveals that the Polyak step-size adapts to any directional smoothness to obtain the tightest possible convergence rate. This property is not shared by constant step-size GD and may explain the superiority of the Polyak step-size in many practical settings.

1.1 Additional Related Work

Directional smoothness is a relaxation of non-uniform smoothness (J. Mei et al., 2021), which restricts the smoothness function M to depend only on x , the origin point. J. Mei et al. (2021) leverage non-uniform smoothness and a non-uniform Łojasiewicz inequality to break lower-bounds for first-order optimization. Similarly, Berahas et al. (2023) show that a weak local smoothness oracle can break lower bounds for gradient methods. A major advantage of our work over such oracle-based approaches is that we construct explicit directional smoothness functions that are easy to evaluate.

Similar to non-uniform smoothness, Grimmer (2019) and Orabona (2023) consider Hölder-type growth conditions with constants that depend on a neighbourhood of x . Since directional smoothness is stronger than and implies these Hölder error bounds, our M functions can be leveraged to make their results fully explicit (the Hölder bounds are non-constructive). Finally, while they also analyze normalized GD, our rates are anytime and do not use online-to-batch reductions like Orabona (2023).

Directional smoothness is also related to ( L 0 , L 1 ) -smoothness (Jingzhao Zhang et al., 2020; B. Zhang et al., 2020), which can be interpreted as a directional smoothness function with exponential dependence on the distance between x and y . The extension of ( L 0 , L 1 ) -smoothness to ( r, l ) -smoothness by H. Li et al. (2023) shows how to bound sequences of such directional smoothness functions, even for accelerated methods. These approaches are complementary to ours and showcase a setting where directional smoothness leads to concrete convergence rates.

Our work is most closely connected to that by Malitsky and Mishchenko (2020), who use a smoothed version of M ( x, y ) to set the step-size. Vladarean et al. (2021) apply a similar smoothed step-size scheme to primal-dual hybrid gradient methods, while Zhao and Huang (2024) relate directional smoothness to Barzilai-Borwein updates (Barzilai and Borwein, 1988) and Vainsencher et al. (2015) use local smoothness over neighbourhoods of the global minimizer to set the step-size for SVRG.

Finally, we note that adaptivity to directional smoothness is different from adaptivity to the sequence of observed gradients obtained by methods such as Adagrad (Duchi et al., 2010; Streeter and McMahan, 2010). Adagrad and its variants are most useful when the gradients are bounded, such as in Lipschitz optimization, although they can also be used to obtain rates for smooth functions (Levy, 2017). We do not address adaptivity to gradients in this work.

2 Directional Smoothness

We say that a convex function f is L -smooth if for all x, y ∈ R d ,

Minimizing this quadratic upper bound in y gives the classical GD update with step-size η k = 1 /L . However, this viewpoint leads to rates which depend on the global, worst-case growth of f . This is both counter-intuitive and undesirable because the iterates of GD, x k +1 = x k -η k ∇ f ( x k ) , depend only on local properties of f . Ideally, the analysis should also depend only on the local conditioning along the path { x 1 , x 2 , . . . } . Towards this end, we generalize the smoothness upper-bound as follows.

Definition 2.1. We call M : R d,d → R + a directional smoothness function for f if for all x, y ∈ R d ,

If a function is L -smooth, then M ( x, y ) = L is a trivial choice of directional smoothness function. In the rest of this section, we construct different M functions that provide tighter bounds on f while still being possible to evaluate. The first is the point-wise directional smoothness ,

Point-wise smoothness is a directional estimate of L and satisfies D ( x, y ) ≤ 2 L . Indeed, L can be equivalently defined as the supremum of D ( x, y ) / 2 over the domain of f (Beck, 2017). If f is convex and differentiable, then D ( x, y ) is a directional smoothness function according to Definition 2.1.

Lemma 2.2. If f is convex and differentiable, then the point-wise directional smoothness satisfies,

See Appendix A (we defer all proofs to the relevant appendices). In the worst-case, the point-wise directional smoothness D is weaker than the standard upper-bound M ( x, y ) = L by a factor of two. This is not an artifact of the analysis and is generally unavoidable, as the next proposition shows.

Proposition 2.3. There exists a convex, differentiable f and x, y ∈ R d such that if t < 2 , then

x

k

x

k

+1

x

∗

f

(

x

)

Actual Progress

L

-Smooth

M

k

-Smooth

While the point-wise smoothness is easy to compute, this additional factor of two can make Equation (5) looser than L -smoothness - on isotropic quadratics, for example. As an alternative, we define the path-wise directional smoothness ,

and show it verifies the quadratic upper-bound and satisfies Definition 2.1 even without convexity.

Lemma 2.4. For any differentiable function f , the path-wise smoothness (7) satisfies

Path smoothness is tighter than point-wise smoothness since A ( x, y ) ≤ D ( x, y ) , but hard to compute because it depends on the chord between x and y . That is, it depends on the properties of f on the line { tx +(1 -t ) y : t ∈ [0 , 1] } rather than solely on the points x and y like the point-wise smoothness.

Point-wise and path-wise smoothness are constructive, but they may not yield the tightest bounds in all situations. The tightest directional smoothness function, which we call the optimal point-wise smoothness , is the smallest number for which the quadratic upper bound holds,

By definition, H is the tightest possible directional smoothness function; it lower bounds any constant C that satisfies the quadratic bound (2). Thus, H ( x, y ) ≤ M ( x, y ) for any smoothness function M .

The directional smoothness functions introduced in this section represent different trade-offs between computability and tightness. The optimal point-wise smoothness H ( x, y ) requires access to both the function and gradient values, whereas the point-wise directional-smoothness D ( x, y ) requires only access to the gradients and convexity. In contrast, the path-wise direction smoothness A ( x, y ) satisfies Lemma 2.4 with or without convexity, but may be hard to evaluate.

3 Path-Dependent Sub-Optimality Bounds

Using directional smoothness, we obtain a descent lemma which depends only on local geometry,

See Lemma A.1. If η k < 2 /M ( x k , x k +1 ) , then GD is guaranteed to decrease the function value and we call η k adapted to M ( x k , x k +1 ) . However, computing adapted step-sizes is not always straightforward. For instance, finding η k = 1 /M ( x k , x k +1 ( η k )) requires solving a non-linear equation.

The rest of this section leverages directional smoothness to derive new guarantees for GD with arbitrary step-sizes. We emphasize that these results are sub-optimality bounds , rather than convergence rates; a sequence of adapted step-sizes is required to convert our propositions into a convergence theory. As a trade-off, our bounds reflect the locality of GD, rather than treating it as a global method.

We start with the case when f has lower curvature. Instead of using strong convexity or the PLcondition (Karimi et al., 2016), we propose the directional strong convexity constant,

If f is convex, then µ ( x, y ) ≥ 0 and it verifies the standard lower-bound from strong convexity,

Moreover, we have µ ( x, y ) ≥ µ when f is µ -strongly convex. We prove two bounds for convex functions using directional strong convexity. For brevity, we denote M i := M ( x i , x i +1 ) , µ i := µ i ( x i , x ∗ ) , δ i = f ( x i ) -f ( x ∗ ) , and ∆ i = ∥ x i -x ∗ ∥ 2 2 , where x ∗ is a minimizer of f .

Proposition 3.1. If f is convex and differentiable, then GD with step-size sequence { η k } satisfies,

The analysis splits iterations into good steps G , where η k is adapted to the directional smoothness, and bad steps B , where the step-size is too large and GD may increase the optimality gap. When f is L -smooth and µ -strongly convex, using the step-size sequence η k = 1 /L gives

where µ i (2 -M i /L ) ≥ µ . Thus, Equation (13) gives at least as tight a rate as standard assumptions by localizing to the convergence path using any directional smoothness M . When M i < L , the gap in constants yields a strictly improved rate (see Figure 2). We also prove a more elegant bound.

Proposition 3.2. If f is convex and differentiable, then GD with step-size sequence { η k } satisfies,

Unlike Proposition 3.1, this analysis shows linear progress at each iteration and does not divide k into good steps and bad steps. In exchange, the second term in Equation (15) reflects how much convergence is degraded when η k is not adapted to the directional smoothness function M . We conclude this section with a bound for when there is no lower curvature, meaning µ i = 0 .

Proposition 3.3. Let x k = ∑ k i =0 η i x i +1 / ∑ k i =0 η i . If f is convex and differentiable, then GD satisfies,

Eq. (16) is faster than standard analyses whenever M i < L ; it will be a key tool in the next sections.

3.1 Path-Dependent Acceleration

Now we show that directional smoothness can also be used to derive path-dependent sub-optimality bounds for accelerated algorithms - that is, methods obtaining optimal rates for smooth, convex optimization. In particular, we study Nesterov's accelerated gradient descent (AGD) (Nesterov, 1983) and prove that directional smoothness leads to tighter rates given adapted step-sizes. Throughout this section we assume that f is µ -strongly convex with µ = 0 when f is merely convex.

Although our analysis uses estimating sequences (Nesterov et al., 2018), we state AGD in the following 'momentum' formulation, where y k is the momentum and α k the momentum parameter,

If η k ≤ 1 /M ( x k , x k +1 ) , then Equation (10) combined with 1 -η k M ( x k , x k +1 ) / 2 ≥ 1 / 2 implies,

Our analysis leverages the fact that this descent condition for x k +1 is the only connection between the smoothness of f and the convergence rate of AGD. Since Equation (18) depends only on the step-size η k , we can replace L within the analysis of AGD with a sequence of adapted step-sizes. The following theorem controls the effect of these step-sizes to obtain path-dependent bounds.

Theorem 3.4. Suppose f is differentiable, µ -strongly convex and AGD is run with adapted step-sizes η k ≤ 1 /M k . If µ > 0 and α 0 = √ η 0 µ , then AGD obtains the following accelerated rate:

Let η min = min i ∈ [ k ] η i . If µ ≥ 0 and α 0 ∈ ( √ µη 0 , c ) , where c is the maximum value of α 0 for which γ 0 = α 2 0 -η 0 α 0 µ η 0 (1 -α 0 ) satisfies γ 0 < 3 /η min + µ , then AGD obtains the following rate:

If η k = 1 /M k > 1 /L , then these rates are strictly faster than those obtained under L -smoothness and Theorem 3.4 shows that AGD provably benefits from taking the largest possible steps given the local geometry of f . However, obtaining accelerated rates when µ = 0 requires prior knowledge of the minimum step-size; while this is straightforward for L -smooth functions, it is not clear how to extend such result to non-strongly convex acceleration with locally Lipschitz gradients. For example, while H. Li et al. (2023) show that the ( r, l ) -smoothness (a valid directional smoothness function) is bounded over the iterate trajectory, their rate does not adapt to the optimization path.

4 Adaptive Learning Rates

Converting our sub-optimality bounds into convergence rates requires adapted step-sizes satisfying η k < 2 /M ( x k , x k +1 ) . Given an adapted step-size, the directional descent lemma (Equation (10)) implies GD decreases f and we can obtain fast rates if the step-sizes are bounded below. However, x k +1 is itself a function of η k , meaning adapted step-sizes are not straightforward to compute.

For L -smooth f , the different directional smoothness functions M introduced in Section 2 satisfy M ( x k , x k +1 ) ≤ 2 L . This implies η k < 1 L is trivially adapted. As such step-sizes don't capture local properties of f , we introduce the notion of strongly adapted step-sizes , which satisfy

Equation (10) implies GD with a strongly adapted step-size makes guaranteed progress as,

This progress is greater than that guaranteed by L -smoothness when M ( x k , x k +1 ) < L and holds even when f is not L -smooth. However, it is not clear a priori if (i) strongly adapted step-sizes exist or if (ii) any iterative method achieves the progress in Eq. (21). Surprisingly, we provide a positive answer to both questions. Strongly adapted η k are computable and we also prove GD with the Polyak step-size adapts to any choice of directional smoothness, including the optimal point-wise smoothness. Before presenting this strong result, we consider the illustrative case of quadratic minimization.

4.1 Adaptivity in Quadratics

Now we show that step-sizes adapted to both the point-wise smoothness M and the path-wise smoothness A exist when f is quadratic. Let f ( x ) = x ⊤ Bx/ 2 -c ⊤ x, where B is positive semidefinite. Assuming { η k } is strongly adapted to the directional smoothness, Equation (16) implies

10 1

10 3

Iteration

10 4

10 5

10 6

Optimality Gap

10 1

10 3

Iteration

10 2

10 7

Point-Wise Smoothness

10 1

10 3

Iteration

10

3

2×10 3

3×10 3

4×10 3

Adapted Step-Sizes

1/

L

1/

Dk

1/

Ak

2/

L

where we used η i M i = 1 as well as Jensen's inequality. This guarantee depends solely on the average directional smoothness along the optimization trajectory { x 0 , x 1 , . . . } . When f is quadratic, we can exactly compute these smoothness constants. In particular, the point-wise directional smoothness is,

Notably, D ( x i , x i +1 ) has no dependence on x i +1 and the corresponding strongly adapted step-size is given by η i = ∥∇ f ( x i ) ∥ 2 / (2 ∥ B ∇ f ( x i ) ∥ 2 ) -see Lemma C.1. Remarkably, this expression recovers the step-size proposed by Dai and Yang (2006), who show it approximates the Cauchy step-size and converges to the 'edge-of-stability' (J. Cohen et al., 2021) at 2 /L as k →∞ . Combining this simple expression with Equation (23) gives a fast, non-asymptotic convergence rate for GD and new theoretical justification for their work.

We can also compute the path-wise directional smoothness in closed form. As Lemma C.2 shows,

and η i = ∇ f ( x i ) ⊤ ∇ f ( x i ) / [ ∇ f ( x i ) ⊤ B ∇ f ( x i )] is the well-known Cauchy step-size. Path-wise directional smoothness thus provides another interpretation (and convergence guarantee) for the Cauchy step-size, which is traditionally derived by minimizing f ( x -η ∇ f ( x )) in η .

4.2 Adaptivity for Convex Functions

In the last subsection, we proved that strongly adapted step-sizes for the point-wise and path-wise directional smoothness functions have closed-form expressions when f is quadratic. Moreover, these step-sizes recover two classical schemes from the optimization literature, giving them new justification and fast convergence rates. Now we consider the existence of strongly adapted step-sizes for general convex functions. Our first result gives simple conditions for Equation (21) to have at least one solution when M is the point-wise directional smoothness.

Proposition 4.1. If f is convex and continuously differentiable, then either (i) f is minimized along the ray x ( η ) = x -η ∇ f ( x ) or (ii) there exists η > 0 satisfying η = 1 /D ( x, x -η ∇ f ( x )) .

The next proposition uses a similar argument with slightly stronger conditions to show existence of strongly adapted step-sizes for the path-wise smoothness.

Proposition 4.2. If f is convex and twice continuously differentiable, then either (i) f is minimized along the ray x ( η ) = x -η ∇ f ( x ) or (ii) there exists η > 0 satisfying η = 1 /A ( x, x -η ∇ f ( x )) .

Propositions 4.1 and 4.2 do not assume the global smoothness; although neither proof is constructive, it is possible to compute strongly adapted step-sizes for the point-wise directional smoothness using root-finding methods. We show in Section 5 that if f is twice differentiable, then strongly adapted step-sizes can be found via Newton's method using only Hessian-vector products, ∇ 2 f ( x ) ∇ f ( x ) .

4.2.1 Exponential Search

Now we show that the exponential search algorithm developed by Carmon and Hinder (2022) can be used to find step-sizes that adapt on average to the directional smoothness. Consider a fixed

optimization horizon k and denote by x i ( η ) the sequence of iterates obtained by running GD from x 0 using a fixed step-size η . Define the criterion function,

and suppose that we have a step-size η that satisfies ψ ( η ) / 2 ≤ η ≤ ψ ( η ) . Using these bounds in Proposition 3.3 yields the following convergence rate,

While η does not adapt to each directional smoothness M ( x i , x i +1 ) along the path, it adapts to a weighted average of the directional smoothness constants, where the weights are the observed squared gradient norms. This is always smaller than the maximum directional smoothness along the trajectory and can be much smaller than the global smoothness. Furthermore, we have reduced our problem to finding η ∈ [ ψ ( η ) / 2 , ψ ( η )] , which is similar to the problem Carmon and Hinder (2022) solve with exponential search. We adopt their approach as Algorithm 1 and give a convergence guarantee.

Theorem 4.3. Assume f is convex and L -smooth. Then Algorithm 1 with η 0 > 0 requires at most 2 K (log log(2 η 0 /L ) ∨ 1) iterations of GD and in the last run it outputs a step-size η and point x K = 1 K ∑ K -1 i =0 x i ( η ) such that exactly one of the following holds:

̸

where M i def = M ( x ′ i , x ′ i +1 ) and x ′ i are the iterates generated by GD with step-size η ′ ∈ [ η, 2 η ] .

Theorem 4.3 requires f to be L -smooth, but has only a log log dependence on the global smoothness constant. Moreover, the rate scales with the weighted average of smoothness constants along a very close trajectory { x ′ 1 , x ′ 2 , . . . } . In the next section, we give convergence bounds that depend on the unweighted average of the directional smoothness constants along the actual optimization trajectory.

4.2.2 Polyak's Step-Size Rule

Our theory so-far suggests using strongly adapted step-sizes, but neither root-finding nor exponential search are practical methods for large-scale optimization. Thus, we now consider other step-size selection rules which may leverage directional smoothness. In particular, the Polyak step-size sets,

for some γ > 0 , which is optimal for smooth and non-smooth optimization (Hazan and Kakade, 2019) given knowledge of f ( w ∗ ) . Surprisingly, we show that GD with the Polyak step-size also achieves the same guarantee as strongly adapted step-sizes without knowledge of the directional smoothness.

Theorem 4.4. Suppose that f is convex and differentiable and let M be any directional smoothness function for f . Let ∆ 0 := ∥ x 0 -x ∗ ∥ 2 2 . Then GD with the Polyak step-size and γ ∈ (1 , 2) satisfies

Theorem 4.4 measures sub-optimality at an average iterate obtained using the directional smoothness. However, it also holds for the best iterate, ˆ x k = arg min i ∈ [ k ] f ( x i ) , meaning no knowledge of the directional smoothness is required to obtain the guarantee. We prove an alternative guarantee for the Polyak step-size in Theorem D.2, where the progress depends on the sum of step-sizes rather than on the average directional smoothness. This shows that the step-size in Equation (26) can itself be viewed as a measure of local smoothness, albeit without formal justification.

Compared with the standard guarantee for the Polyak step-size under L -smoothness, f ( x k ) -f ( x ∗ ) ≤ 2 L ∆ 0 /k (Hazan and Kakade, 2019), our analysis in Theorem 4.4 with the choice γ = 1 . 5 yields

where the second bound follows from Jensen's inequality and shows that the convergence depends on the average directional smoothness along the trajectory, rather than on L . If f is L -smooth, then M ( x k , x k +1 ) ≤ L immediately recovers the classic rate for Polyak's method up to a 3 / 2 constant factor. If f is not L -smooth, but M ( x k , x k +1 ) is bounded, then Equation (27) generalizes the O (1 /k ) rate proved concurrently by Takezawa et al. (2024), but for any choice of directional smoothness (of which ( L 0 , L 1 ) -smoothness (Jingzhao Zhang et al., 2020) is but one).

Comparison with strongly adapted step-sizes. As we saw for quadratics, strongly adapted step-sizes for any directional smoothness function allow us to obtain the following convergence rate,

This is matches the guarantee given by Equation (27) up to constant factors. As a result, we give a positive answer to the question posed earlier in this section: GD with the Polyak step-size achieves the same convergence for any smoothness function M as GD with step-sizes strongly adapted to M .

Application to the optimal directional smoothness. Theorem 4.4 holds for every directional smoothness function M . Therefore we can specialize Equation (27) with the optimal point-wise directional smoothness H (as defined in Equation (4)) and γ = 1 . 5 to get the guarantee,

This rate requires computing the iterate with the minimum function value, but that is easy to track during optimization. Unlike our previous results, Equation (28) requires no access to the optimal point-wise smoothness, yet obtains a dependence on the tightest constant possible.

4.3 Normalized Gradient Descent

Now we change directions slightly and study normalized GD, whose convergence also depends on the directional smoothness. Normalized GD uses step-sizes which are divided by the gradient magnitude,

Our next theorem shows that normalized GD obtains a guarantee which depends solely on the average of the point-wise directional smoothness D k := D ( x k , x k +1 ) despite no explicit knowledge of D k .

Theorem 4.5. Suppose that f is convex and differentiable. Let D be the point-wise directional smoothness defined by Equation (4) and ∆ 0 := ∥ x 0 -x ∗ ∥ 2 2 . Then normalized GD with a sequence of non-increasing step-sizes η k satisfies

where ˆ x k = arg min i ∈ [ k -1] f ( x i ) . If max i ∈ [ k -1] M ( x i , x i +1 ) is bounded for all k (i.e. f is L -smooth), then for η i = 1 / √ i we have f (ˆ x k ) -f ( x ∗ ) ∈ O (1 /k ) and for η i = 1 / √ i we get the anytime result f (ˆ x k ) -f ( x ∗ ) ∈ O (log( k ) /k ) .

Theorem 4.5 gives a rate for normalized GD which is valid for any convex f without any dependence on global smoothness. However, does not adapt to any smoothness function like the Polyak step-size.

5 Experiments

We evaluate the practical improvement of our convergence rates over those using L -smoothness on two logistic regression problems taken from the UCI repository (Asuncion and Newman, 2007).

0

100

200

Iterations

10

-

3

10

-

2

10

-

1

Optimality Gap

ionosphere

0

100

200

Iterations

10

-

14

10

-

11

10

-

8

10

-

5

10

-

2

horse-colic

0

100

200

Iterations

10

-

2

10

-

1

ozone

1

/L

)

1

/Dk

)

Polyak

Norm. GD

AdGD

Figure 1 compares GD with strongly adapted step-sizes η = 1 /M k , where M k is the point-wise smoothness, against GD with the Polyak step-size. We also plot the exact convergence rates for each method, Equation (16) and Equation (27), respectively, and compare against the classical guarantee for both methods. Our convergence rates are an order of magnitude tighter on the ionosphere dataset and display a remarkable ability to adapt to the path of optimization on mammographic .

Figure 3 compares the performance of GD with strongly adapted step-sizes and with the fixed stepsize η k = 1 /L for a synthetic quadratic with Hessian skew (R. Pan et al., 2022). Results are averaged over twenty random problems. We find that strongly adapted step-sizes lead to significantly faster convergence. Since A k , D k ≪ L , the adapted step-sizes are larger than 2 /L , especially at the start of training; they eventually converge to 2 /L , indicating these methods operate at the edge-of-stability (J. Cohen et al., 2021; J. M. Cohen et al., 2022). This is consistent with Ahn et al. (2022) and Y . Pan and Y. Li (2023), who show local smoothness is correlated with edge-of-stability behavior.

We conclude with a comparison of empirical convergence rates on three additional logistic regression problems from the UCI repository. We compare GD with η k = 1 /L , GD with step-sizes strongly adapted to the point-wise smoothness ( η k = 1 /D k ), GD with the Polyak step-size (Polyak), and normalized GD (Norm. GD) against the AdGD method (Malitsky and Mishchenko, 2020). The Polyak step-size performs best on every dataset but ozone , where GD with η k = 1 /D k solves the problem to high accuracy in just a few iterations. Thus, although Polyak step-sizes have the optimal dependence on directional smoothness, computing strongly adapted step-sizes can still be advantageous.

6 Conclusion

We present new sub-optimality bounds for GD under novel measures of local gradient variation which we call directional smoothness functions. Our results hold for any step-sizes, improve over standard analyses when η k is adapted to the choice of directional smoothness, and depend only on properties of f local to the optimization path. For convex quadratics, we show that computing step-sizes strongly adapted to directional smoothness functions is straightforward and recovers two well-known step-size schemes, including the Cauchy step-size. In the general case, we prove that an algorithm based on exponential search gives a weighted-version of the path-dependent convergence rate with no need for adapted step-sizes. We also show that GD with the Polyak step-size and normalized GD both obtain fast rates with no dependence on the global smoothness parameter. Crucially, the Polyak step-size adapts to any choice of directional smoothness, including the tightest possible parameter.

References

Ahn, Kwangjun, Jingzhao Zhang, and Suvrit Sra (2022). 'Understanding the unstable convergence of gradient descent'. In: International Conference on Machine Learning, ICML 2022, 17-23 July 2022,

Baltimore, Maryland, USA

. Vol. 162. Proceedings of Machine Learning Research, pp. 247-257.

Altschuler, Jason M. and Pablo A. Parrilo (2023). 'Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedule'. In: CoRR abs/2309.07879.

Asuncion, Arthur and David Newman (2007). UCI machine learning repository .

Barzilai, Jonathan and Jonathan M Borwein (1988). 'Two-point step size gradient methods'. In: IMA journal of numerical analysis 8.1, pp. 141-148.

Beck, Amir (2017). First-order methods in optimization . MOS-SIAM series on optimization. Philadelphia : Philadelphia: Society for Industrial and Applied Mathematics ; Mathematical Optimization Society. ISBN: 978-161-197-4-9-9-7.

Bengio, Yoshua (2012). 'Practical Recommendations for Gradient-Based Training of Deep Architectures'. In: Neural Networks: Tricks of the Trade - Second Edition . Vol. 7700. Lecture Notes in Computer Science, pp. 437-478.

Berahas, Albert S., Lindon Roberts, and Fred Roosta (2023). 'Non-Uniform Smoothness for Gradient Descent'. In: arXiv preprint arXiv:2311.08615 abs/2311.08615.

Bertsekas, Dimitri P (1997). 'Nonlinear programming'. In: Journal of the Operational Research Society 48.3, pp. 334-334.

Bubeck, Sébastien et al. (2015). 'Convex optimization: Algorithms and complexity'. In: Foundations and Trends® in Machine Learning 8.3-4, pp. 231-357.

Carmon, Yair and Oliver Hinder (2022). 'Making SGD Parameter-Free'. In: Conference on Learning Theory, 2-5 July 2022, London, UK . Vol. 178. Proceedings of Machine Learning Research, pp. 2360-2389.

Cohen, Jeremy et al. (2021). 'Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability'. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 .

Cohen, Jeremy M. et al. (2022). 'Adaptive Gradient Methods At the Edge of Stability'. In: arXiv preprint arXiv:2207.14484 abs/2207.14484.

Dai, Y. H. and X. Q. Yang (2006). 'A New Gradient Method with an Optimal Stepsize Property'. In: Computational Optimization and Applications 33.1, pp. 73-88.

Duchi, John C., Elad Hazan, and Yoram Singer (2010). 'Adaptive Subgradient Methods for Online Learning and Stochastic Optimization'. In: COLT 2010 - The 23rd Conference on Learning Theory, Haifa, Israel, June 27-29, 2010 , pp. 257-269.

Fernández-Delgado, Manuel et al. (2014). 'Do we need hundreds of classifiers to solve real world classification problems?' In: The journal of machine learning research 15.1, pp. 3133-3181.

Grimmer, Benjamin (2019). 'Convergence Rates for Deterministic and Stochastic Subgradient Methods without Lipschitz Continuity'. In: SIAM J. Optim. 29.2, pp. 1350-1365.

Hazan, Elad and Sham Kakade (2019). 'Revisiting the Polyak step size'. In: arXiv preprint arXiv:1905.00313 .

He, Kaiming et al. (2015). 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification'. In: Proceedings of the IEEE international conference on computer vision , pp. 1026-1034.

Hogan, William W (1973). 'Point-to-set maps in mathematical programming'. In: SIAM review 15.3, pp. 591-603.

Karimi, Hamed, Julie Nutini, and Mark Schmidt (2016). 'Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition'. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16 . Springer, pp. 795-811.

Levy, Kfir Y. (2017). 'Online to Offline Conversions, Universality and Adaptive Minibatch Sizes'. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pp. 1613-1622.

Li, Haochuan et al. (2023). 'Convex and Non-convex Optimization Under Generalized Smoothness'. In: Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 .

Li, Zhiyuan, Kaifeng Lyu, and Sanjeev Arora (2020). 'Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate'. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .

Liu, Dong C and Jorge Nocedal (1989). 'On the limited memory BFGS method for large scale optimization'. In: Mathematical programming 45.1-3, pp. 503-528.

Lu, Zhaosong and Sanyou Mei (2023). 'Accelerated first-order methods for convex optimization with locally Lipschitz continuous gradient'. In: SIAM Journal on Optimization 33.3, pp. 2275-2310.

Malitsky, Yura and Konstantin Mishchenko (2020). 'Adaptive Gradient Descent without Descent'. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event . Vol. 119. Proceedings of Machine Learning Research, pp. 6702-6712.

Mei, Jincheng et al. (2021). 'Leveraging Non-uniformity in First-order Non-convex Optimization'. In: Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event . Vol. 139. Proceedings of Machine Learning Research, pp. 7555-7564.

Mishkin, Aaron, Mert Pilanci, and Mark Schmidt (2024). 'Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation'. In: arXiv preprint arXiv:2404.02378 .

Nesterov, Yurii (1983). 'A method for solving the convex programming problem with convergence rate O (1/k2)'. In: Dokl akad nauk Sssr . Vol. 269, p. 543.

Nesterov, Yurii et al. (2018). Lectures on convex optimization . Vol. 137. Springer.

Orabona, Francesco (2023). 'Normalized Gradients for All'. In: arXiv preprint arXiv:2308.05621 abs/2308.05621.

Pan, Rui, Haishan Ye, and Tong Zhang (2022). 'Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums'. In: ICLR .

Pan, Yan and Yuanzhi Li (2023). 'Toward understanding why adam converges faster than sgd for transformers'. In: arXiv preprint arXiv:2306.00204 .

Paquette, Courtney et al. (2023). 'Halting time is predictable for large models: A universality property and average-case analysis'. In: Foundations of Computational Mathematics 23.2, pp. 597-673.

Park, Jea-Hyun, Abner J Salgado, and Steven M Wise (2021). 'Preconditioned accelerated gradient descent methods for locally Lipschitz smooth objectives with applications to the solution of nonlinear PDEs'. In: Journal of Scientific Computing 89.1, p. 17.

Paszke, Adam et al. (2019). 'Pytorch: An imperative style, high-performance deep learning library'. In: Advances in neural information processing systems 32.

Patel, Vivak and Albert S. Berahas (2022). 'Gradient descent in the absence of global Lipschitz continuity of the gradients: Convergence, divergence and limitations of its continuous approximation'. In: arXiv preprint arXiv:2210.02418 .

Polyak, Boris T (1987). 'Introduction to optimization'. In.

Streeter, Matthew and H. Brendan McMahan (2010). 'Less Regret Via Online Conditioning'. In: arXiv preprint arXiv:1002.4862 .

Takezawa, Yuki et al. (2024). 'Polyak Meets Parameter-free Clipped Gradient Descent'. In: CoRR abs/2405.15010. DOI: 10.48550/ARXIV.2405.15010 . arXiv: 2405.15010 . URL: https: //doi.org/10.48550/arXiv.2405.15010 .

Vainsencher, Daniel, Han Liu, and Tong Zhang (2015). 'Local Smoothness in Variance Reduced Optimization'. In: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pp. 2179-2187.

Virtanen, Pauli et al. (2020). 'SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python'. In: Nature Methods 17, pp. 261-272.

Vladarean, Maria-Luiza, Yura Malitsky, and Volkan Cevher (2021). 'A first-order primal-dual method with adaptivity to local smoothness'. In: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pp. 6171-6182.

Zhang, Bohang et al. (2020). 'Improved Analysis of Clipping Algorithms for Non-convex Optimization'. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .

Zhang, Jingzhao et al. (2020). 'Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity'. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 .

Zhang, Junyu and Mingyi Hong (2020). 'First-order algorithms without Lipschitz gradient: A sequential local optimization approach'. In: arXiv preprint arXiv:2010.03194 .

Zhao, Weijing and He Huang (2024). 'Adaptive stepsize estimation based accelerated gradient descent algorithm for fully complex-valued neural networks'. In: Expert Systems with Applications 236, p. 121166.

A Proofs for Section 2

Lemma 2.2. If f is convex and differentiable, then the point-wise directional smoothness satisfies,

Proof. By the convexity of f we have

Rearranging and then using Cauchy-Schwarz we get

Lemma 2.4. For any differentiable function f , the path-wise smoothness (7) satisfies

Proof. Starting from the fundamental theorem of calculus,

which completes the proof.

Proposition 2.3. There exists a convex, differentiable f and x, y ∈ R d such that if t < 2 , then

Proof. Let H f denote the optimal pointwise directional smoothness associated with some convex and differentiable function f : R d → R (as defined in Equation (4)), and D f denote the pointwise directional smoothness associated with f (as defined in Equation (4)). For any t , the statement of (6) is equivalent to saying H f > t ∥∇ f ( x ) -∇ f ( y ) ∥ ∥ x -y ∥ for all x, y ∈ R d and convex, differentiable f . Observe that Lemma 2.2 already shows that for all convex and differentiable functions f : R d → R

for all x, y ∈ R d . In order to show that this is tight, we suppose by the way of contradiction that there exists some 2 > t ≥ 0 such that for all convex and differentiable functions f : R d → R

for all x, y ∈ R . We shall show that no such t exists by showing for each such t there exists a function f t such that Equation (31) does not hold.

Consider f ϵ ( x ) = √ x 2 + ϵ 2 for ϵ ≤ 1 . The function f is differentiable. Moreover

Therefore f is convex. Let g ( x ) = | x | . Fix x = 1 and y = 0 , we have

Now observe that

Therefore

By definition we have 1 2 ∥ x -y ∥ 2 = 1 2 , therefore

But by our starting assumption we have that there exists some t < 2 such that H f ( x, y ) ≤ t ∥ f ′ ( x ) -f ′ ( y ) ∥ | x -y | for all differentiable and convex functions f . Applying this to f = f ϵ we get

Combining Equations (33) and (34) we have

Rearranging we get

Choosing ϵ = 2 -t 8 > 0 we get a contradiction. It follows that the minimal t such that H ( x, y ) ≤ t | f ′ ( x ) -f ′ ( y ) | | x -y | for all convex and differentiable f is t = 2 .

Lemma A.1. One step of gradient descent with step-size η k > 0 makes progress as

Proof. Starting from Equation (4), we have

B Proofs for Section 3

Lemma B.1. If f is convex, then for any x, y ∈ R d ,

If f is µ strongly convex, then µ ( x, y ) ≥ µ .

Proof. The fundamental theorem of calculus implies

Note that we have implicitly used convexity to verify the inequality in the second line in the case where µ ( x, y ) = 0 . Now assume that f is µ strongly convex. As a standard consequence of strong-convexity, we obtain:

Proposition 3.1. If f is convex and differentiable, then GD with step-size sequence { η k } satisfies,

where λ i = η i M i -2 , G = { i : η i < 2 /M i } , and B = [ k ] \G .

Proof. First note that λ i < 0 for i ∈ G and λ i ≥ 0 for i ∈ B . We start from Equation (10),

Subtracting f ( x ) from both sides and then recursively applying the inequality gives the result. Proposition 3.2. If f is convex and differentiable, then GD with step-size sequence { η k } satisfies,

∗

Proof. Let ∆ k = ∥ x k -x ∗ ∥ 2 2 and observe

Using this expansion in ∆ k +1 -∆ k , we obtain

Now we control the inner-products with directional strong convexity and directional smoothness.

where the last inequality follows from µ k +1 strong convexity between x k +1 and x ∗ . Re-arranging this expression allows us to deduce a rate with error terms depending on the local smoothness,

Proposition 3.3. Let x k = ∑ k i =0 η i x i +1 / ∑ k i =0 η i . If f is convex and differentiable, then GD satisfies,

Proof. Let ∆ k = ∥ x k -x ∗ ∥ 2 2 and observe

∆ k = ∥ x k -x k +1 + x k +1 -x ∗ ∥ 2 2 = ∆ k +1 + ∥ x k -x k +1 ∥ 2 2 +2 ⟨ x k -x k +1 , x k +1 -x ∗ ⟩ . Using this expansion in ∆ k +1 -∆ k , we obtain

Now we use convexity and directional smoothness to control the two inner-products as follows:

Re-arranging this equation and summing over iterations implies the following sub-optimality bound:

Convexity of f and Jensen's inequality now imply the final result,

B.1 Path-Dependent Acceleration: Proofs

This section proves Theorem 3.4 using estimating sequences. Throughout this section, we assume µ ≥ 0 is the global strong convexity parameter, where µ = 0 covers the non-strongly convex case. We start from the estimating sequences version of Equation (17), which is given as follows:

The algorithm is initialized with x 0 = v 0 and some γ 0 > 0 . Note that y 0 = x 0 = v 0 since y k is a convex combination of x k and v k . First we prove that this scheme is equivalent to the one given in Equation (17).

Lemma B.2. Equation (36) and Equation (17) lead to equivalent updates for the y k , x k , and α k sequences. Moreover, given initialization γ 0 > 0 , the corresponding initialization for α 0 is,

Proof. The proof follows Nesterov et al. (2018, Theorem 2.2.3). Expanding the definition of v k +1 , we obtain

Plugging this back into the expression for y k +1 ,

Note that this update is consistent with Equation (17). Since γ k = α 2 k /η k , we can write,

which is also consistent with Equation (17). Finally, the initialization for α 0 is determined by γ 0 in Equation (36) as,

The quadratic formula now implies,

This completes the proof.

As mentioned, our proof uses the concept of estimating sequences.

Definition B.3. Two sequences λ k , ϕ k are estimating sequences for f if λ l ≥ 0 for all k ∈ N , lim k →∞ λ k = 0 , and, for all x ∈ R d .

We use the same estimating sequences as developed by Nesterov et al. (2018). Let λ 0 = 1 , ϕ 0 ( x ) = f ( x 0 ) + γ 0 2 ∥ x -x 0 ∥ 2 2 , and define the updates,

where µ ≥ 0 is the strong convexity parameter, with µ = 0 when f is merely convex. It is straightforward to differentiate ϕ k +1 to see that v k +1 of Equation (36) is the minimizer. Indeed, Nesterov et al. (2018, Lemma 2.2.3) shows that this choice for the estimating sequences obeys the following canonical form:

where γ k +1 and v k +1 are given by Equation (36) and the minimum value is,

Before we can prove our main theorem, we must show that these choices for λ k and ϕ k yield a valid estimating sequence. The following proofs build on (Nesterov et al., 2018) and (Mishkin et al., 2024).

Lemma B.4. Assume α k ∈ (0 , 1] for all k ∈ N . If µ > 0 and γ 0 = µ , then

If µ ≥ 0 and γ 0 ∈ ( µ, µ +3 /η min ) , then,

Proof. Assume γ 0 = µ > 0 . Then γ k = µ for all k and,

As a consequence, as claimed.

Now assume γ 0 ∈ ( µ, 3 L + µ ) . Modifying the proof by Nesterov et al. (2018, Lemma 2.2.4), we compute as follows:

Recursing on this equality implies

If α k = 1 or λ k = 0 , then using λ k +1 = (1 -α k ) λ k implies λ k +1 = 0 and the result trivially holds. Otherwise, recall α 2 k /γ k +1 = η k to obtain,

Finally, this implies

Moreover, this bound holds uniformly for all k ∈ N . We have now exactly reached Eq. 2.2.11 of Nesterov et al. (2018, Lemma 2.2.4) with L replaced by 1 /η min. Applying that Lemma with this modification, we obtain which completes the proof.

Lemma B.5. If f is strongly convex with parameter µ ≥ 0 and η k ≤ 1 /µ for all k ∈ N , then λ k and ϕ k are estimating sequences.

Proof. Using the quadratic formula, we find

Thus, is sufficient for α k > 0 . This holds if µ ≥ γ k . Otherwise, we require,

which holds if and only if η k , γ k > 0 . On the other hand, we also need α k ≤ 1 , which is satisfied when,

as claimed.

Recall λ 0 = 1 and λ k +1 = (1 -α k ) λ k . Since α k ∈ (0 , 1] , λ k ≥ 0 holds by induction. It remains to show that λ k tends to zero, which holds by Lemma B.4 since we have shown α k ∈ (0 , 1] for all k .

Now we establish the last piece,

But this follows immediately by Nesterov et al. (2018, Lemma 2.2.2).

Now we can prove the last major lemma before our convergence result.

Lemma B.6. Suppose f is strongly convex with parameter µ ≥ 0 and η k is a sequence of adapted step-sizes, meaning η k ≤ 1 /M ( x k , x k +1 ) . Then for every k ∈ N ,

Proof. We use an inductive proof again. The inductive assumption is

It is easy to see this holds at k = 0 since,

implies min z ϕ 0 ( z ) = f ( x 0 ) . Using Equation (41), we obtain

where the inequality holds by the inductive assumption. Using convexity of f and recalling α 2 k γ k +1 = η k from the definition of the update (Equation (36)),

Using the fact that the step-sizes are adapted and invoking the directional descent lemma (i.e. Equation (18)) now implies

The remainder of the proof is largely unchanged from the analysis in Nesterov et al. (2018). The definition of y k gives x k -y k = α k γ k γ k +1 ( y k -v k ) , which we use to obtain

since µα k (1 -α k ) γ k 2 γ 0

k +1 ≥ . We conclude the desired result by induction.

The main accelerated result now follows almost immediately.

Theorem 3.4. Suppose f is differentiable, µ -strongly convex and AGD is run with adapted step-sizes η k ≤ 1 /M k . If µ > 0 and α 0 = √ η 0 µ , then AGD obtains the following accelerated rate:

Let η min = min i ∈ [ k ] η i . If µ ≥ 0 and α 0 ∈ ( √ µη 0 , c ) , where c is the maximum value of α 0 for which γ 0 = α 2 0 -η 0 α 0 µ η 0 (1 -α 0 ) satisfies γ 0 < 3 /η min + µ , then AGD obtains the following rate:

Proof. We analyze the equivalent formulation given in Equation (36). See Lemma B.2 for a formal proof that these two schemes produce the same x k , y k , and α k iterates. Note that our proof follows Nesterov et al. (2018) and Mishkin et al. (2024) closely; while their results are very similar, we are not aware of pre-existing works which adapt them to our specific setting.

First, observe that M ( x k , x k +1 ) ≥ µ for all k ∈ N . Since the step-sizes η k are assumed to satisfy η k ≤ 1 /M ( x k , x k +1 ) , we also have that η k ≤ 1 /µ for every k .

Thus, Lemma B.4 and Lemma B.5 apply. Using the definition of an estimating sequence and Lemma B.6, we obtain,

Re-arranging this equation and expanding the definition ϕ 0 (Equation (39)), we deduce the following:

We see that the rate of convergence of AGD is entirely controlled by the convergence of the sequence λ k . If µ > 0 and γ 0 = µ , then Lemma B.4 implies

By Lemma B.2, this initialization is equivalent to choosing α 0 = √ η 0 µ , which is the setting claimed in the theorem.

Alternatively, if µ ≥ 0 and γ 0 ∈ ( µ, µ +3 /η min ) , then,

where the equality, holds by Lemma B.2. Since α 0 ≤ 1 for η 0 ≤ 1 /µ , η 0 is an increasing function of γ 0 . Thus, an upper-bound c on α 0 can be deduced from that on γ 0 using the quadratic formula:

C Proofs for Section 4.1

Lemma C.1. Let B be a positive semi-definite matrix and suppose that

Let x i +1 = x i -η ∇ f ( x i ) . Then for any η > 0 , the pointwise directional smoothness between the gradient descent iterates x i , x i +1 is given by

Proof. We have by straightforward algebra,

Lemma C.2. Let B be a positive semi-definite matrix and suppose that

Let x i +1 = x i -η ∇ f ( x i ) . Then for any η > 0 , the path-wise directional smoothness between the gradient descent iterates x i , x i +1 is given by by

Proof. Let A t ( x, y ) = ⟨∇ f ( x + t ( y -x )) -∇ f ( x ) ,y -x ⟩ t ∥ x -y ∥ 2 2 . We have

The path-wise directional smoothness A is therefore

D Proofs for Section 4.2

Proposition 4.1. If f is convex and continuously differentiable, then either (i) f is minimized along the ray x ( η ) = x -η ∇ f ( x ) or (ii) there exists η > 0 satisfying η = 1 /D ( x, x -η ∇ f ( x )) .

Proof. Let I = { η : ∇ f ( x -η ∇ f ( x )) = ∇ f ( x ) } . For every η ∈ I , it holds that

However, since f is convex, the directional derivative

is monotone non-decreasing in η ′ . We deduce that I must be an interval of form [0 , η ] . If η is not bounded, then f is linear along -∇ f ( x ) and is minimized by taking η →∞ . Therefore, we may assume η is finite.

Let η > η . Then we have the following:

from which we deduce

is sufficient for the implicit equation to hold. Squaring both sides and multiplying by 1 / 2 , we obtain the following alternative root-finding problem:

Because f is C 1 , h is a continuous function and it suffices to show that there exists an interval in which h crosses 0 . From the display above, we see

Continuity now implies ∃ η ′ > η such that h ( η ′ ) < 0 . Now, suppose h ( η ) ≤ 0 for all η ≥ η ′ . Working backwards, we see that this can only occur when

for all η ≥ η ′ . The directional descent lemma (Equation (10)) now implies

Taking limits on both sides as η →∞ implies f ( x -η ∇ f ( x )) is minimized along the ray x ( η ) = x -η ∇ f ( x ) . Thus, we deduce that either there exists η ′′ > η ′ such that h ( η ′′ ) > 0 exists, or f is minimized along the gradient direction as claimed.

Proposition 4.2. If f is convex and twice continuously differentiable, then either (i) f is minimized along the ray x ( η ) = x -η ∇ f ( x ) or (ii) there exists η > 0 satisfying η = 1 /A ( x, x -η ∇ f ( x )) .

Proof. Let

Since f is convex, the directional derivative

is monotone non-decreasing in η ′ . We deduce that J must be an interval of form [0 , η ] . If η is not bounded, then convexity implies

meaning f is minimized along -∇ f ( x ) . Therefore, we may assume η is finite.

We have

Thus, for η > η , the equation we must solve reduces to

Since f is C 2 , h is continuous (see, e.g. Hogan (1973, Theorem 7)) and it suffices to show that there exists an interval over which h crosses 0 .

Using Taylor's theorem, we can re-write this expression as

where for some α ( tη ) ∈ [0 , tη ] . Examining the denominator, we find that,

which, since f is convex, implies

for every α ∈ [0 , η ] . By continuity of the Hessian, for every ϵ > 0 , there exists δ > 0 such that η ′ ∈ [ η, η + δ ] guarantees,

Substituting this into our expression for h ,

for ϵ, δ sufficiently small. Thus, there exists η ′ > η for which h ( η ′ ) < 0

Now let us show that h ( η ) > 0 for some η . For convenience, define

. ′′ ′′

Algorithm 1 Gradient Descent with Exponential Search

1: Procedure ExponentialSearch( x, η 0 )

2: for k = 1 , 2 , 3 , . . . do

3: η out ← RootFindingBisection ( x, 2 -2 k η 0 , η 0 ) .

5: Return η out

4: if η out < ∞ then

6: end if

7: end for

8: End Procedure

9: Procedure RootFindingBisection( x, η lo , η hi )

10: Define ϕ ( η ) = η -ψ ( η ) where ψ ( η ) is given in (24) \\ One access to ϕ requires T descent steps.

11: if ϕ ( η hi ) ≤ 0 then

12: Return η hi

13: end if

14: if ϕ ( η lo ) > 0 then

15: Return ∞

16: end if

17: while η hi > 2 η lo do

18: η mid = √ η lo η hi

19: if ϕ ( η mid ) > 0 then

20: η hi = η mid

21: else

22: η lo = η mid

23: end if

\\ Invariant: ϕ ( η hi ) > 0 , and ϕ ( η lo ) ≤ 0 .

24: end while

25: Return η lo

26: End Procedure

which is a continuous and monotone non-increasing function. Take η →∞ and let

where the limit exists, but may be -∞ . Indeed, it must hold that c < ∞ since,

If c < 0 , then taking η ′′ large enough that g ( η ′′ ) ≤ 0 suffices. Alternatively, if c ≥ 0 , then there exists ˜ η such that g ( η ) ≤ c + ϵ for every η ≥ ˜ η . Choosing η ′′ > max { ˜ η, c } + ϵ yields

This completes the proof.

Theorem 4.3. Assume f is convex and L -smooth. Then Algorithm 1 with η 0 > 0 requires at most 2 K (log log(2 η 0 /L ) ∨ 1) iterations of GD and in the last run it outputs a step-size η and point x K = 1 K ∑ K -1 i =0 x i ( η ) such that exactly one of the following holds:

̸

where M i def = M ( x ′ i , x ′ i +1 ) and x ′ i are the iterates generated by GD with step-size η ′ ∈ [ η, 2 η ] .

Proof of Theorem 4.3. This analysis follows (Carmon and Hinder, 2022). First, instantiate Equation (16) from Proposition 3.3 with η i = η for all i to obtain

Now, observe that if we get a 'Lucky strike' and ϕ ( η hi ) = ϕ ( η 0 ) ≤ 0 , then specializing Equation (45) for η = η 0 we get

This covers the first case of Theorem 4.3.

With the first case out of the way, we may assume that ϕ ( η hi ) > 0 . This implies that η hi > 1 L , since if η ≤ 1 L we have ϕ ( η ) ≤ 0 . Now observe that when η lo = 2 2 -k η 0 ≤ 1 L , we have that ϕ ( η lo ) ≤ 0 , therefore it takes at most k = ⌈ log log η 0 L -1 ⌉ to find such an η lo . From here on, we suppose that ϕ ( η hi ) > 0 and ϕ ( η lo ) ≤ 0 . Now observe that the algorithm's main loop always maintains the invariant ϕ ( η hi ) > 0 and ϕ ( η lo ) ≤ 0 , and every iteration of the loop halves log η hi η lo , therefore we make at most ⌈ log log η 0 L ⌉ loop iterations. The output step-size η lo satisfies η hi 2 ≤ η lo ≤ η hi and ϕ ( η lo ) ≤ 0 . Specializing Equation (45) for η = η 0 and using that ϕ ( η lo ) ≤ 0 we get

By the loop invariant ϕ ( η hi ) > 0 we have

By the loop termination condition we have η lo ≥ η hi 2 , combining this with the last equation we get

Plugging this into Equation (46) we obtain

It remains to notice that η hi ∈ [ η lo , 2 η lo ] .

Theorem 4.4. Suppose that f is convex and differentiable and let M be any directional smoothness function for f . Let ∆ 0 := ∥ x 0 -x ∗ ∥ 2 2 . Then GD with the Polyak step-size and γ ∈ (1 , 2) satisfies

For the proof of this theorem, we will need the following proposition:

Proposition D.1. Let x ∈ R d . Define η x = γ f ( x ) -f ( x ∗ ) ∥∇ f ( x ) ∥ 2 for some γ ∈ (1 , 2) and let ˜ x = x -η x ∇ f ( x ) . Then,

Proof. Observe

By smoothness we have

Plugging back into Equation (47) we get

Let us now use the definition of η x = γ f ( x ) -f ( x ∗ ) ∥∇ f ( x ) ∥ 2 2 to get

Assuming that f ( x ) = f ( x ∗ ) then we get by cancellation

̸

Using the definition of η x again

If f ( x ) = f ( x ∗ ) then ∥∇ f ( x ) ∥ 2 2 = 0 , both sides are identically zero and the statement holds trivially.

Now we can prove our theorem on the convergence of GD with Polyak step-sizes:

Proof of Theorem 4.4. We start by considering the distance to the optimum and expanding the square

Let δ k = f ( x k ) -f ( x ∗ ) . By convexity we have f ( x ∗ ) ≥ f ( x k ) + ⟨∇ f ( x k ) , x ∗ -x k ⟩ . Therefore we can upper bound Equation (48) as

Rearranging we get

where in the second line we used the definition of η k . By Proposition D.1 we have

Using this in Equation (49) gives

Rearranging we get

Summing up and telescoping we get

Let x k = 1 ∑ k -1 i =0 M ( x i ,x i +1 ) -1 ∑ k -1 i =0 M ( x i , x i +1 ) -1 x i , then by the convexity of f and Jensen's inequality we have

Theorem D.2. If f is convex and differentiable, then GD with the Polyak step-size and γ < 2 satisfies,

where x k = ∑ k -1 i =0 η i x i / ( ∑ k -1 i =0 η i )

Proof. The proof begins in the same manner as that for Theorem 4.4,

Re-arranging, summing from i = 0 to k -1 , and dividing by ∑ k -1 i =0 η i ,

which completes the proof.

Lemma D.3. Normalized GD with step-sizes η k satisfies

Proof. By convexity we have

Now note that

where we used Cauchy-Schwarz. Recalling the definition of directional smoothness

in Equation (55) gives

Theorem 4.5. Suppose that f is convex and differentiable. Let D be the point-wise directional smoothness defined by Equation (4) and ∆ 0 := ∥ x 0 -x ∗ ∥ 2 2 . Then normalized GD with a sequence of non-increasing step-sizes η k satisfies

where ˆ x k = arg min i ∈ [ k -1] f ( x i ) . If max i ∈ [ k -1] M ( x i , x i +1 ) is bounded for all k (i.e. f is L -smooth), then for η i = 1 / √ i we have f (ˆ x k ) -f ( x ∗ ) ∈ O (1 /k ) and for η i = 1 / √ i we get the anytime result f (ˆ x k ) -f ( x ∗ ) ∈ O (log( k ) /k ) .

Proof. Here we will first establish that for any non-increasing sequence of step-sizes η k > 0 we have that

The specialized results follow by assuming that ∑ k -1 i =0 M ( x i ,x i +1 ) k is bounded, which it is the case of L -Lipschitz gradients. In particular the min i ∈ [ k -1] f ( x i ) -f ( x ∗ ) ∈ O (1 /T ) result follows by plugging in η i = 1 / √ k and using that

Alternatively we get min i ∈ [ k -1] f ( x i ) -f ( x ∗ ) ∈ O (log( T ) /T ) by plugging in η i = 1 / √ i +1 and using that

With this in mind, let us prove Equation (56).

By convexity,

Re-arranging, dividing through by η 2 k , and then summing over i = 0 , · · · , k -1 gives

where we used that η i -1 ≤ η i = ⇒ 1 η 2 i -1 η 2 i -1 ≤ 0 . Using Jensen's inequality over the map a ↦→ 1 /a , which is convex for a positive, gives

Meanwhile, recall our notation ∆ i = ∥ x i -x ∗ ∥ 2 2 . Expanding the squares and using that f ( x ) is convex, we have that

As before, we use δ i := f ( x i ) -f ( x ∗ ) . Re-arranging, summing both sides of the above over i = 0 , . . . , k -1 and using telescopic cancellation gives

Using the above along with (58) gives,

E Experimental Details

In this section we provide additional details necessary to reproduce our experiments. We run our logistic regression experiments using PyTorch (Paszke et al., 2019). For the UCI datasets, we use the pre-processed version of the data provided by Fernández-Delgado et al. (2014), although we do not use their evaluation procedure as it is known have test-set leakage. Instead, we randomly perform an 80-20 train-test split and use the test set for validation. Unless otherwise stated, all methods are initialized using the Kaiming initialization (He et al., 2015), which is standard in PyTorch.

In order to compute the strongly adapted step-sizes, we run the SciPy (Virtanen et al., 2020) implementation of Newton method on Equation (44). In general, we find this procedure is surprisingly robust, although it can be slow.

Figure 1 : We pick two datasets from the UCI repository to showcase different behaviors of the upper-bounds. We compute a tight-upper bound on L as follows. Recall that for logistic regression problems the Hessian is given by

where A is the data matrix and σ ( z ) = 1 1+exp( z ) is the sigmoid function. A short calculation shows that the diagonal matrix

which is tight when x = 0 . As a result, L = λ max ( A ⊤ A ) / 4 . We compute this manually. We also compute the optimal value for the logistic regression problem using the SciPy implementation of BFGS (Liu and Nocedal, 1989). We use this value for f ( x ∗ ) to compute the Polyak step-size and when plotting sub-optimality. It turns out that the upper-bound based on L -smoothness for both GD with the Polyak step-size (Hazan and Kakade, 2019) and standard GD (Bubeck et al., 2015) is

Figure 3 : We run these experiments using vanilla NumPy. As mentioned in the text, we generate a quadratic optimization problem

where the eigenvalues of A were generated to follow power law distribution with parameter α = 3 . We scaled the eigenvalues to ensure L = 1000 . The dimension of the problem we create is d = 300 . We repeat the experiment for 20 random trials and plot the mean and standard deviations.

Figure 4 : We pick three different datasets from the UCI repository to showcase the possible convergence behavior of the optimization methods. We compute L and f ( w ∗ ) as described above for Figure 1. For normalized GD, we use the step-size schedule η k = η 0 / √ k as suggested by our theory. To pick η 0 , we run a grid search on the grid generated by np.logspace(-8, 1, 20) . We implement AdGD from scratch and use a starting step-size of η 0 = 10 -3 . We use the same procedure to compute the strongly adapted step-sizes as described above.

F Computational Details

The experiments in Figure 3 were run on a MacBook Pro (16 inch, 2019) with a 2.6 GHz 6-Core Intel i7 CPU and 16GB of memory. All other experiments were run on a Slurm cluster with several different node configurations. Our experiments on the cluster were run with nodes using (i) Nvidia A100 GPUs (80GB or 40GB memory) or Nvidia H100-80GB GPUs with Icelake CPUs, or (ii) Nvidia V100-32GB or V100-16GB GPUs with Skylake CPUs. All jobs were allocated a single GPU and 24GB of RAM.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims in the abstract and introduction are justified with rigorous proofs and supported by experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our theoretical results, including necessary assumptions, are addressed in the text.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our theoretical results are accompanied by their necessary assumptions and rigorous proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental procedures and hyper-parameter settings are provided in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the code to reproduce our experiments upon acceptance of the paper. All non-synthetic data used in this paper is open source and freely accessible online. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details necessary to interpret our experimental results are provided in Section 5, while additional details necessary to reproduce the experiments are given in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our experiments consider only deterministic optimization methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We detail the compute resources used in our paper in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All authors are familiar with the code of ethics and conform to its principles. Moreover, our research is primarily theoretical and is of minor ethical concern.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our research is primarily theoretical and has no societal impact beyond the impact of general advances in the fields of machine learning and optimization.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No new data or models are released as part of this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All libraries, models, and data sources are appropriately referenced.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are introduced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not use any crowdsourcing for this work

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: IRB approval was not required for this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work propose different directional smoothness function and use them to establish sub-optimality bounds that are adaptive to the optimization trajectory. This approach can be explicitly used in quadratics or with an exponential search technique for convex objectives to use adaptive stepsizes that enjoy better guarantees compared to constant stepsize GD tuned according to the global smoothness. Furthermore, the paper shows that Polyak and normalized GD obtain convergence guarantees that depend on the directional smoothness without modifying the algorithms.\nThe theoretical claims are empirically evaluated using two logistic regression problems.\n\nSTRENGTHS:\n1. The technique naturally yields results for several existing methods.\n2. Tighter, adaptive guarantees are appealing due to the observed success beyond the theoretical convergence bounds.\n\nWEAKNESSES:\n3. The paper discusses only deterministic optimization, while modern large-scale optimization is largely stochastic. It begs the question, which is not discussed in the paper, whether such approaches are even valid for stochastic optimization (e.g. [1]).\n\n4. The novelty of the approach is unclear for some of the observed results. Consider especially Theorem 4.4 with the Polyak stepsizes, which achieve the best empirical performance in the experiments. It is straightforward to translate the decrease mentioned in [2] (page 2, end of left column), $\\lVert x_{t+1} - x^* \\rVert^2 - \\lVert x_{t} - x^* \\rVert^2 \\leq - \\frac{(f(x_t)-f(x^*))^2}{\\lVert \\nabla f(x_t) \\rVert^2}$ to a bound of the weighted $\\overline{x}$ of $f(\\overline{x_t})-f(x^*) \\leq \\frac{\\lVert x_0 - x^*\\rVert^2}{\\sum_{k=0}^{t} \\eta_t}$. Theorem 4.4 can be considered as replacing $\\eta_t$ with a possibly worse $M(x_{t+1},x_t)$. In this case, the simple adaptive result is only framed for convenience in the directional smoothness framework.\n\n[1] N. Loizou, S. Vaswani, I. H. Laradji, and S. Lacoste-Julien. Stochastic Polyak step-size for SGD: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics, pages 1306–1314, 2021.\n\n[2] Hazan, E. and Kakade, S., 2019. Revisiting the Polyak step size. arXiv preprint arXiv:1905.00313.\n\nQUESTIONS:\n5. Can the framework be of use for stochastic optimization?\n\n6. Can the authors elaborate on the technical novelty of the results? In particular, it is arguably known that the standard analysis technique of GD exploits smoothness only between consecutive points. What is the main difficulty when we replace global smoothness with an adaptive variant?\n\nOverall, the paper presents an appealing framework for adaptive bounds which encompasses a long list of stepsize strategies, while the novelty and deterministic setting are somewhat in question.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new type of non-uniform smoothness, which the authors label _directional smoothness_. Directional smoothness replaces $L$ in the typical $L$-smoothness inequality by a function $M(x,y)$, which describes the smoothness along the line between $x$ and $y$. The authors proof multiple basic properties of this notion as well as sub-optimality bounds for GD with general stepsizes in the deterministic convex setting. Additionally, they provide sub-optimality bounds for three specific stepsizes in the same setting.\n\nSTRENGTHS:\n- The proposed smoothness notion – directional smoothness – is generalising many previous notions of smoothness while still allowing for a meaningful analysis of algorithms.\n- Convergence analysis with directional smoothness can potentially give some intuition which algorithms automatically adapt to (potentially beneficial) local smoothness properties.\n- The authors provide a wide range of properties for directional smoothness and different ways to compute upper bounds on the directional smoothness.\n- The sub-optimality gap for Polyak’s step-size, showing it automatically adapts to the local smoothness in a meaningful way, could potentially explain empirical success of the step-size schedule in practise.\n\nWEAKNESSES:\n- Many of the results until Section 4.2 are direct consequences of the (directional) descent inequality. In particular, it was to be expected that the global smoothness inequality can be replaced with a localized version, and that results along the lines of Proposition 3.1 – 3.3 should hold. That being said, these results can be seen as preliminary results for the propositions / theorems in the later portion of the paper, which provide new insights.\n- ~~There might be some questions regarding the soundness of Proposition 3.2, see Question 1. However, this result is not used in the consequent results and does hence not have any further implications.~~\n- Given this submission introduces a new way to analyse algorithms, seeing its application in different settings and for different algorithms would be highly beneficial to showcase the benefit of the proposed approach. All results in the work are however limited to the deterministic convex setting, and only Gradient Descent with different stepsizes is considered. Showcasing the approach for the stochastic setting or for different algorithms (such as Nesterov AGD) could give a more wholistic picture of the approach.\n\n---\n\n**Edit after Author Rebuttal**\n\nThe authors did fix the proof of Proposition 3.2.\n\nQUESTIONS:\n# Questions\n1. In the proof of Proposition 3.2, why does the last inequality (above line 438) hold? More specifically, in the case $\\eta_k > \\mu_k^{-1}$, the recursive argument cannot be applied as the sign is flipped. How do the authors deal with this case?\n2. Theorem 4.3 is very unclear to me. What is one “iteration” of GD? In order to reach a point $\\hat{x}_T$ with $f(\\hat{x}_T) - f(x^*) \\leq \\varepsilon$, do you require $\\widetilde{O}(\\varepsilon^{-1})$ or $\\widetilde{O}(\\varepsilon^{-2})$ gradient evaluations? Maybe these unclear points could be addressed by including the full algorithm, producing the final output $\\hat{x}_T$, in Algorithm 1.\n3. The authors first derive suboptimality bounds for general stepsizes, before proving results for specific stepsizes. However, the proofs for Theorems 4.4 and 4.5 are not based on the general results, rather they are standalone proofs. Why did the authors not plug in these stepsizes into the general results? This would be a great opportunity to showcase the application of Propositions 3.1 – 3.3.\n4. Given directional smoothness generalises other non-uniform smoothness notions, did the authors examine whether their sub-optimality bounds reconstruct / improve previous results for notions of smoothness that can be modelled by directional smoothness? For example, can results from $(L_0, L_1)$-smoothness [1] be reconstructed? The corresponding $M(x,y)$ is specified in [2, Lemma A.3] / [3, Lemma 8]. \n5. Is there hope for any lower bound results that show strongly adapted stepsizes are indeed the best (non-accelerated) stepsize one can choose in your generalized setting? Given such result, the Polyak-stepsize results would get even stronger.\n6. Are there any difficulties extending the notion of directional smoothness to the typical nonconvex setting and results, i.e. $|| \\nabla F(x) - \\nabla F(y) || \\leq M(x,y) || x-y||$ and the corresponding GD guarantees?\n\n[1]: Li et al., Convex and Non-convex Optimization Under Generalized Smoothness, NeurIPS 2023\n\n[2]: Zhang et al., Improved Analysis of Clipping Algorithms for Non-convex Optimization, NeurIPS 2020\n\n[3]: Hübler et al., Parameter-Agnostic Optimization under Relaxed Smoothness, AISTATS 2024\n\n# Remarks\n- There are multiple potential inconsistencies throughout the work. None of them have an impact on the soundness of the work, but it would be great if the authors could address them. \n\na) Most results use $k$, some use $T$ while others use $t$ as the last iteration index. \n\nb) The averaged iterate is sometimes denoted using hats while other times using widebars. \n\nc) There are inconsistencies whether differentiability assumptions are explicitly stated or not. In sections 2 and 4 they are consistently mentioned, section 3 and its proofs in the appendix do not mention it even-though it is required.\n\nd) Usage of $f(x^*)$ vs. $f^*$.\n\ne) Appendix C denotes the Euclidian norm by $|| \\cdot ||$, while all other sections use $|| \\cdot ||_2$.\n\nf) All results in the appendix besides the three Theorems are numbered relative to the appendix, the theorems relative to the main section.\n- Lemma A.4 might better fit in Appendix B. \n- Proposition 3.2 could potentially be improved by using $2 \\eta_k [f(x^*) – f(x_{k+1})] \\leq -\\eta_k \\mu_k \\Delta_{k+1}$ instead of removing the former term in the last inequality after line 435. This divides each term in the product by $(1+\\eta_k\\mu_k)$. \n- In it’s current version, Equation (41) does not provide additional information. The authors could either remove it or modify Equation (40) to include this step.\n- As mentioned in Line 562, the convergence results in Theorem 4.5 require some smoothness assumption such as $L$-smoothness. This assumption is missing in the statement of Theorem 4.5.\n\nBelow some potential typos.\n- Proposition 3.1: $\\delta_k$ -> $\\delta_{k+1}$\n- Proposition 3.2: $\\Delta_k$ -> $\\Delta_{k+1}$\n- Theorem 3.4, Case 1: Missing parenthesis\n- Equation (23): $\\bar x$ -> $\\bar x_t$\n- Last inequality in equation after Line 232: $M(x_{k+1},x_{k})^{-1}$ -> $M(x_{k+1},x_{k})$\n- Equation after Line 393: $f(x)$ -> $f(y) - f(x)$\n- Second inequality in equation after Line 435: $|| \\nabla f(x) ||$ -> $|| \\nabla f(x) ||^2$\n- Equation after Line 468: $\\Rightarrow$ -> $\\Leftarrow$ (or $\\Leftrightarrow$)\n- Equations after line 475: $M$ -> $D$\n- Algorithm 1, Line 1: $L_0$ -> $\\eta_0$\n- Algorithm 1, Line 3: Wrong parentheses\n- Line 527: follow -> following\n\nLIMITATIONS:\nThe introduction could more clearly mention that the paper (besides Lemma 2.4) exclusively focuses on the deterministic convex setting.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops refined sub-optimality bounds for gradient descent in the convex setting. The authors consider directional smoothness, a local and path dependent smoothness condition, instead of assuming globally bounded smoothness constants in classical analyses. They discussed several interesting examples of directional smoothness functions, and derived sub-optimality bounds with them. For convex quadratic problems, they show that the stepsizes that minimize the upper bounds can be easily computed, which leads to new guarantees for two classical stepsizes. For general convex functions, they show that ideal stepsizes exist and can be computed using Newton's method or by exponential search. They also show that Polyak stepsize and normalized GD can achieve fast and path-dependent rates without knowing the directional smoothness.\n\nSTRENGTHS:\n1. The sub-optimality bounds in terms of directional smoothness and the stepsizes and convergence results following the upper bounds look novel. The perspective is quite interesting.\n2. The proofs look correct and the results look reasonable. The authors have thoroughly studied relevant aspects and details of their theory.\n3. The authors did a good job connecting their results with existing methods and theories, which is quite insightful.\n3. The writing and presentation are also very good. I enjoyed reading the paper.\n\nWEAKNESSES:\n1. The final convergence rates are path-dependent. Although they are more refined and should be better than classical rates, it is not clear how large the gap is. I wonder if there are any interesting applications for which their rates can be more explicit and show clear improvement over classical ones.\n\nQUESTIONS:\n1. Have the authors considered non-convex settings? In fact, since the authors mentioned neural networks in the first paragraph of the introduction, I wonder if this directional smoothness can be used to explain certain phenomenons in deep neural network training, like edge of stability or benefits of adaptivity.\n\nSome potential typos:\na. Equation (3): missing \"+\"\nb. Line 194: it should be the inverse of Hessian?\nc. Theorem 4.3. Case 1: missing \")\"\n\nLIMITATIONS:\nThe authors have adequately addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}