Title: Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods
Venue: NeurIPS 2023
Accepted: yes
Keywords: Nonconvex optimization

Stochastic Gradient Descent

Adaptive methods
Abstract: The classical analysis of Stochastic Gradient Descent (SGD) with polynomially decaying stepsize $\eta_t = \eta/\sqrt{t}$ relies on well-tuned $\eta$ depending on  problem parameters such as Lipschitz smoothness constant, which is often unknown in practice. In this work, we prove that SGD with arbitrary $\eta > 0$, referred to as untuned SGD, still attains an order-optimal convergence rate $\widetilde{\mathcal{O}}(T^{-1/4})$ in terms of gradient norm for minimizing smooth objectives. Unfortunately, it comes at the expense of a catastrophic exponential dependence on the smoothness constant, which we show is unavoidable for this scheme even in the noiseless setting. We then examine three families of adaptive methods — Normalized SGD (NSGD), AMSGrad, and AdaGrad — unveiling their power in preventing such exponential dependency in  the absence of information about the smoothness parameter and boundedness of stochastic gradients. Our results provide  theoretical justification for the advantage of adaptive methods over untuned SGD in alleviating the issue with large gradients.

Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods

Abstract

The classical analysis of Stochastic Gradient Descent (SGD) with polynomially decaying stepsize η t = η/ √ t relies on well-tuned η depending on problem parameters such as Lipschitz smoothness constant, which is often unknown in practice. In this work, we prove that SGD with arbitrary η > 0 , referred to as untuned SGD , still attains an order-optimal convergence rate ˜ O ( T -1 / 4 ) in terms of gradient norm for minimizing smooth objectives. Unfortunately, it comes at the expense of a catastrophic exponential dependence on the smoothness constant, which we show is unavoidable for this scheme even in the noiseless setting. We then examine three families of adaptive methods - Normalized SGD (NSGD), AMSGrad, and AdaGrad - unveiling their power in preventing such exponential dependency in the absence of information about the smoothness parameter and boundedness of stochastic gradients. Our results provide theoretical justification for the advantage of adaptive methods over untuned SGD in alleviating the issue with large gradients.

1 Introduction

In this work, we study the stochastic optimization problem of the form:

where P is an unknown probability distribution, and f : R d → R is an ℓ -Lipschitz smooth function and can be non-convex. In the context of machine learning, ξ typically represent an individual training sample drawn from the data distribution P , and x denotes the weights of the model.

Stochastic Gradient Descent (SGD), originating from the seminal work [58], performs the following update iteratively:

where η t > 0 presents a positive stepsize, and ∇ F ( x t ; ξ t ) is an unbiased stochastic gradient. SGD has shown remarkable empirical success in many modern machine learning applications, e.g., [7, 62]. Its efficiency is usually attributed to its cheap per iteration cost and the ability to operate in an

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Xiang Li ∗

Niao He

online fashion, making it suitable for large-scale problems. However, empirical evidence also reveals undesirable behaviors of SGD, often linked to challenges in selecting appropriate stepsizes. In particular, a number of works report the gradient explosion effect [6, 55, 24] during the initial phase of training, which may eventually lead to divergence or prohibitively slow convergence. The phenomenon is also observed in our experiments (see Figure 1(f)) when the stepsize is poorly chosen. Unfortunately, this phenomenon has not been well understood from a theoretical point of view. The classical analysis of SGD in the smooth non-convex case [22], prescribes to select a non-increasing sequence of stepsizes { η t } t ≥ 1 with η 1 < 2 /ℓ . In particular, the choice η t = 1 / ( ℓ √ t ) , guarantees 2 to find a point x with E [ ∥∇ f ( x ) ∥ ] ≤ ϵ after O ( ϵ -4 ) stochastic gradient calls, which is also known to be unimprovable in the smooth non-convex setting unless additional assumptions are made [3, 17].

However, the bound on the smoothness parameter ℓ is usually not readily available for practitioners, and the limited computing power usually prevents them from exhaustive tuning to find the best stepsize. It is therefore important to provide a theoretical understanding for SGD with an arbitrary stepsize (which we refer to as untuned SGD ) that is agnostic to the problem parameter. The following intriguing question remains elusive in the stochastic optimization literature:

How does untuned SGD with a decaying stepsize η t = η/ √ t perform when η is independent of the smoothness parameter? How can we explain the undesirably large gradients encountered in training with SGD?

Recently, there has been a surge of interest in adaptive gradient methods such as Adam [35], RMSProp [28], AdaDelta [69], AMSGrad [57], AdaGrad [18], Normalized SGD [26], and many others. These methods automatically adjust their stepsizes based on past stochastic gradients, rather than using pre-defined iteration-based schedules. Empirically, they are observed to converge faster than SGD and mitigate the issue of gradient explosion across a range of problems, even without explicit knowledge of problem-specific parameters [35, 46, 55]. Figure 1 provides a basic illustration of performance differences between SGD with η t = 1 / √ t stepsizes and adaptive schemes such as AdaGrad and Normalized SGD with momentum (NSGD-M) [14]. Notably, when the initial stepsize is too large (relative to 1 /ℓ value), SGD reaches the region with large gradients , while adaptive methods do not

0

50

100

150

#gradient calls

10

-

1

10

1

10

3

10

5

10

7

10

9

‖∇

f

(

xt

)

‖

SGD (

/lscript

= 1)

AdaGrad (

/lscript

= 1)

NSGD-M (

/lscript

= 1)

(a) quadratic function (

ℓ

= 1

)

0

50

100

150

#gradient calls

10

-

1

10

1

10

3

10

5

10

7

10

9

‖∇

f

(

xt

)

‖

SGD (

/lscript

= 5)

AdaGrad (

/lscript

= 5)

NSGD-M (

/lscript

= 5)

(b) quadratic function (

ℓ

= 5

)

0

50

100

150

#gradient calls

10

-

1

10

1

10

3

10

5

10

7

10

9

‖∇

f

(

xt

)

‖

SGD (

/lscript

= 10)

AdaGrad (

/lscript

= 10)

NSGD-M (

/lscript

= 10)

(c) quadratic function (

ℓ

= 10

)

0

10

20

30

40

50

#gradient calls

10

-

1

10 1

10 3

10 5

10 7

‖

g

(

xt, ξ

t

)

‖

SGD (

η

= 3)

AdaGrad (

η

= 3)

NSGD-M (

η

= 3)

(d) ResNet (

η

= 3

)

0

10

20

30

40

50

#gradient calls

10

-

1

10 1

10 3

10 5

10 7

10 9

10 11

‖

g

(

xt, ξ

t

)

‖

SGD (

η

= 5)

AdaGrad (

η

= 5)

NSGD-M (

η

= 5)

(e) ResNet (

η

= 5

)

0

10

20

30

40

50

#gradient calls

10 1

10 6

10 11

10 16

10 21

10 26

10 31

‖

g

(

xt, ξ

t

)

‖

SGD (

η

= 7)

AdaGrad (

η

= 7)

NSGD-M (

η

= 7)

(f) ResNet (

η

= 7

)

suffer from such effect. However, the theoretical benefits of adaptive methods over SGD remain unclear. A large number of existing analyses of adaptive methods assume bounded gradients, or even stochastic gradients, precluding not only a fair comparison with SGD whose convergence does not need bounded gradient but also the possibility to explain their benefit when facing gradient explosions. While recent developments show that AdaGrad-type methods [20, 68] can attain a ˜ O ( ϵ -4 ) sample complexity under the same standard assumptions as for SGD analysis, there is still a lack a good explanation for the substantial performance gap observed in practice despite SGD with well-tuned stepsizes theoretically achieving the lower complexity bound. We will address the following open question:

Can we justify the theoretical benefits of adaptive methods over untuned SGD for smooth non-convex problems without assuming bounded gradients?

Consequently, this work is grounded on the premise that gradients may be not bounded and that hyper-parameters are independent of the problem parameters. . The main contributions are as follows:

We show that untuned SGD with diminishing stepsizes η t = η/ √ t finds an ϵ -stationary point of an ℓ -smooth function within ˜ O (( ℓ 2 + σ 4 η 4 ℓ 4 )(4 e ) 2 η 2 ℓ 2 ϵ -4 ) iterations for any η > 0 . Here, σ 2 corresponds to the variance of the stochastic gradient. Although this classical algorithm converges and has the optimal dependence on ϵ , we further show that the disastrous exponential term in η 2 ℓ 2 is unavoidable even when the algorithm has access to exact gradients. This explains its proneness to gradient explosion when the problem parameter is unknown. Previous analyses have failed to capture this exponential term, since they assume η is well-tuned to be Θ(1 /ℓ ) .

AMSGrad, which was proposed to fix the nonconvergence of Adam, is not yet fully understood, with previous analyses depending on bounded stochastic gradients . We show that AMSGrad (norm version) is free from exponential constants in the deterministic setting without tuning, in stark contrast with SGD. Surprisingly, in the stochastic setting when the stochastic gradients are

unbounded, we show that AMSGrad may converge at an arbitrarily slow polynomial rate. To the best of our knowledge, these are the first results for AMSGrad without assuming bounded gradients.

To further illuminate the advantages of adaptive methods, we re-examine the results for Normalized Gradient Descent (NGD), Normalized SGD with momentum (NSGD-M) from [14] and AdaGradnorm from [68], considering stepsize independent of the problem parameters similar to untuned SGD. They all achieve near-optimal complexities while shredding off the exponential factor. As a side result, we provide a strong non-convergence result of NSGD without momentum under any bounded stepsizes, which might be of independent interest.

Our findings contribute a fresh understanding of the performance gap between SGD and adaptive methods. Albeit with a near-optimal rate, untuned SGD is vulnerable to gradient explosion and slow convergence due to a large exponential constant in its complexity, which can be circumvented by several adaptive methods. To the best of our knowledge, this substantial difference is unaddressed in the previous literature, because the majority of analyses for SGD and adaptive methods turn to either well-tuned stepsize based on problem parameters or the assumption of bounded gradients. Part of our results are summarized in Table 1, and full results for a broader range of stepsizes can be found in Table 2 in the appendix.

1.1 Related Work

SGD in nonconvex optimization. Stochastic approximation methods and SGD in particular have a long history of development [58, 34, 8, 13, 51, 56]. When the objective is ℓ -smooth and the gradient noise has bounded variance σ 2 , Ghadimi and Lan [22] and Bottou et al. [9] prove that if stepsize η t = η/ √ T , where η = η ( ℓ, σ 2 ) and T is the total iteration budget, then SGD can find an ϵ -stationary point within O ( ℓσ 2 ϵ -4 ) iterations. Similar complexity (up to a logarithmic term) can also be achieved by decaying stepsizes η/ √ t [22, 17, 65]. This result was later shown to be optimal for first-order methods under these assumptions [3]. Several works consider various relaxations of the stochastic oracle model with bounded variance, for instance, biased oracle [2] and relaxed growth condition [9]. However, these results also heavily rely on sufficiently small η , e.g., η ≤ 1 /ℓ , and the convergence behavior in the large η regime is rarely discussed. Remarkably, Lei et al. [39] characterize the convergence of SGD under individual smoothness and unbiased function values. They consider Robbins-Monro stepsize schemes, which includes η/t α when α > 1 / 2 , and derive O ( ϵ 2 α -1 ) sample complexity including an exponential dependence on the smoothness parameter. Unlike [39], we focus on the standard assumptions and derive better dependency in smoothness constant when α > 1 / 2 . Importantly, we further justify that the exponential constants are unavoidable with a lower bound.

Adaptive methods. We focus on methods directly using gradients to adjust stepsize, rather than other strategies like backtracking line search [4]. Normalized Gradient Descent (NGD) was introduced by [53] for quasi-convex functions. Hazan et al. [26] apply NGD and NSGD with minibatch to the class of locally-quasi-convex functions. Later, Cutkosky and Mehta [14] and Zhao et al. [74] prove NSGD with momentum or minibatch can find an ϵ -stationary point in smooth nonconvex optimization with sample complexity O ( ϵ -4 ) . AdaGrad was introduced in the online convex optimization [18, 48]. In nonconvex optimization, AdaGrad and its scalar version, AdaGrad-norm [61], achieve competitive convergence rates with SGD [66, 43, 31, 44]. RMSProp [28] and Adam [35] use the moving average of past gradients, but may suffer from divergence without hyper-parameter tuning [57]. Recently, it was shown in the finite-sum setting that they converge to a neighborhood, whose size shrinks to 0 by tuning hyper-parameters [60, 72]. However, most of the results on AdaGrad and Adam-type algorithms assume both Lipschitz and bounded gradients [75, 12, 16, 66, 76]. Very recently, Faw et al. [20] and Yang et al. [68] independently show that AdaGrad-norm converges without assuming bounded gradients and without the need for tuning, attaining a sample complexity of O ( ϵ -4 ) .

˜ SGD v.s. adaptive methods. Despite similar complexities, adaptive methods typically converge faster than SGD in practice [10, 47] and are widely used to prevent large gradients [55, 23]. Various attempts have been made to theoretically explain these differences. Some suggest that the advantage of adaptive algorithms is their ability to achieve order-optimal rates without knowledge of problem parameters such as smoothness and noise variance [66, 40, 30]. Other studies investigate the faster

escape from saddle points by adaptive methods [41, 50, 67]. The importance of coordinate-wise normalization in Adam has also been highlighted [5, 37]. Furthermore, the influence of heavy-tail noise on the performance of adaptive methods is studied [71]. However, most previous works do not provide an explanation for the faster convergence of adaptive methods in terms of sample complexity. Notably, Zhang et al. [70] and Wang et al. [64] explain the benefits of gradient clipping and Adam under a relaxed smoothness assumption, a setting where SGD with non-adaptive stepsizes may not converge. In contrast, we analyze SGD and several adaptive methods under standard smoothness and noise assumptions, distinguishing it from the recent work of Wang et al. [64] that focuses on one variant of Adam for finite-sum problems with individual relaxed smoothness and random shuffling.

2 Problem Setting

Throughout this work, we focus on minimizing an ℓ -smooth function f : R d → R . We have access to a stochastic gradient oracle that returns g ( x ; ξ ) at any point x , and we make the following standard assumptions in nonconvex optimization.

Assumption 1 (smoothness) . Function f ( x ) is ℓ -smooth with ℓ > 0 , that is, ∥∇ f ( x 1 ) -∇ f ( x 2 ) ∥ ≤ ℓ ∥ x 1 -x 2 ∥ for any x 1 and x 2 ∈ R d .

Assumption 2 (stochastic gradients) . The stochastic gradient g ( x ; ξ ) is unbiased and has a bounded variance, that is, E ξ [ g ( x ; ξ )] = ∇ f ( x ) and E ξ [ ∥ g ( x ; ξ ) -∇ f ( x ) ∥ 2 ] ≤ σ 2 for any x ∈ R d . We present the general scheme of SGD with initial point x 0 and a stepsize sequence { η t } ∞ t =0 :

Some commonly used stepsizes include polynomially and geometrically decaying stepsize, constant stepsize, cosine stepsize, etc. When the stepsize depends on the instantaneous or past gradients, i.e., { g ( x ; ξ k ) } k ≤ t , we call it adaptive stepsize, namely Normalized SGD [26], AdaGrad [18], Adam [35], AMSGrad [57], etc. In some adaptive methods, momentum is also considered, replacing g ( x t ; ξ t ) in (1) with a moving average m t +1 of the past stochastic gradients (see Section 4 for more details). To set the stage for our analysis, we assume that f ( x 0 ) -min x ∈ R d f ( x ) ≤ ∆ , where ∆ represents the initial gap. Given that the function class of interest is nonconvex, we aim to find an ϵ -stationary point x with E [ ∥∇ f ( x ) ∥ ] ≤ ϵ .

3 Convergence of Untuned SGD

In this section, we focus on SGD with the decaying stepsize:

where η > 0 is the initial stepsize. Most convergent analysis requires η < 2 /ℓ [22, 9] so that there is 'sufficient decrease' in function value after each update, and if η is carefully chosen, it can achieve the near-optimal complexity of ˜ O ( ℓϵ -4 σ 2 ) [3]. Nevertheless, as the smoothness parameter is usually unknown, providing guarantees with optimal η or assuming η to be problem-dependent does not give enough insights into practical training with SGD. Hence we are interested in its convergence behavior in both small and large initial stepsize regimes, i.e., η ≤ 1 /ℓ and η > 1 /ℓ .

Theorem 1. Under Assumptions 1 and 2, if we run SGD with stepsize η t = η √ t +1 , where η > 0 ,

This theorem implies that when the initial stepsize η > 1 /ℓ , SGD still converges with a sample complexity of ˜ O (( ℓ 2 + σ 4 η 4 ℓ 4 )(4 e ) 2 η 2 ℓ 2 ϵ -4 ) . Although the dependency in the target accuracy ϵ is where τ = ⌈ η 2 ℓ 2 -1 ⌉ and A = ( ∆+ ℓσ 2 η 2 2 (1 + log T ) ) .

x

f

(

x

)

Seg. 1

Seg. 2

Seg. 2

Seg. 3

Seg. 4

x

0

x

t

0

x

t

0

+1

∆

˜

∆ exponentially

depends on

η

2

/lscript

2

w

Segment 1:

f

(

x

) =

/lscript

2

x

2

Segment 4:

f

(

x

) =

(

x

-

x

t

0 +1

+

√

{

1

//lscript,

∑

T

t

=

t

0 +1

η

t

}

)

2

{

1

//lscript,

∑

T

t

=

t

0 +1

η

t

}

+

w

-

˜ ∆

4

trajectory

near-optimal, it includes a disastrous exponential term in η 2 ℓ 2 . This is due to polynomially decaying stepsizes: in the first stage before τ = ⌈ ( ηℓ ) 2 -1 ⌉ iterations, the function value and gradients may keep increasing in expectation until reaching an exponential term in η 2 ℓ 2 , which is in stark contrast with adaptive methods that we will see in Section 4; in the second stage after t ≥ τ , the stepsize is small enough to decrease the function value in expectation at a rate of 1 / √ T up to a small term in σ .

If we arbitrarily select η = Θ(1) , untuned SGD may induce large gradients growing exponentially in ℓ in the first stage, as observed in Figure 1. Moreover, deriving the dependence in hyper-parameter η is crucial for assessing the effort required in its tuning: SGD with an η that is c > 1 times larger than the optimally tuned value can yield a gradient norm that is exp( poly ( c )) times larger in the convergence guarantee.

To the best of our knowledge, there is limited study for non-asymptotic analysis of untuned SGD under the same assumptions. Moulines and Bach [49] study untuned SGD under individual smoothness and convexity assumptions, i.e., g ( x ; ξ ) is Lipschitz continuous and F ( x ; ξ ) is convex almost surely. They showed a convergence rate of O (1 /T 1 / 3 ) for the last iterate and a rate arbitrarily close to O (1 /T 1 / 2 ) for the averaged iterate. Later, Fontaine et al. [21] provide ˜ O (1 /T 1 / 2 ) convergence rate for untuned SGD in the convex setting, albeit without an explicit dependency in ℓ and η . In the nonconvex setting, Khaled and Richt´ arik [33] showed, using SGD with constant stepsize η > 0 , the squared gradient min t<T E ∥∇ f ( x t ) ∥ 2 converges at a rate of O ( ℓση +(1 + ℓ 2 η 2 ) T ∆ / ( ηT ) ) . When the total number of iterations T is predetermined, selecting a constant stepsize η = η 0 / √ T results in the rate O ( ℓση 0 √ T + (1+ ℓ 2 η 2 0 /T ) T ∆ η 0 √ T ) ≤ O ( ℓση 0 √ T + e ℓ 2 η 2 0 ∆ η 0 √ T ) , which exhibits the same dependence on T as Theorem 1 and also includes an exponential constant. However, our focus is on settings where T is not known in advance.

Remark 1. We consider stepsize in the order of 1 / √ t , because it is known for SGD to achieve the best dependency in ϵ for nonconvex optimization [17] and easier to compare with adaptive stepsizes. We also present the convergence results for more general polynomially decaying stepsizes, i.e., η t = η ( t +1) α with 0 < α < 1 , in Theorem 6 of the appendix. There exists a trade-off between convergence speed O (1 /T 1 -α 2 ) and the exponential term in ( ηℓ ) 1 /α for α ∈ [1 / 2 , 1) . Intuitively, larger α leads to a shorter time in adapting to 1 /ℓ stepsize but a slower convergence rate. We do not consider constant stepsize, i.e., α = 0 , because it is well known to diverge even in the deterministic setting if the stepsize is agnostic to the problem parameter [52, 1].

The question arises as to whether the exponential term is necessary. In the following, we provide a lower bound for SGD under this choice of stepsize.

Theorem 2. Fixing T ≥ 1 , η > 0 , ℓ > 0 and ∆ > 0 that ηℓ ≥ 5 , there exists a ℓ -smooth function f : R → R and an initial point x 0 with f ( x 0 ) -f ∗ ≤ ∆ such that if we run Gradient Descent with stepsize η t = η √ t +1 , then for t ≤ t 0 = ⌊ η 2 ℓ 2 / 16 -1 ⌋ ,

if T > t 0 , then for t 0 < t ≤ T ,

This theorem suggests that Gradient Descent (GD) with decaying stepsize η/ √ t +1 needs at least Ω( η -4 ℓ -2 (8 e ) η 2 ℓ 2 / 8 ϵ -4 ) iterations to find an ϵ -stationary point in the large initial stepsize regime. Therefore, it validates that an exponential term in η 2 ℓ 2 multiplied by ϵ -4 is not avoidable even in the deterministic setting. It is crucial to note that our result is limited to untuned (S)GD with this particular stepsize scheme.

It is worth pointing out that the existing lower bounds for first-order methods [3] and SGD [17] do not contain any exponential terms. On the other hand, Vaswani et al. [63] established a lower bound in the strongly convex setting for GD with exponentially decreasing stepsizes (1 /T ) t/T η , where T denotes the total number of iterations and η > 0 . With a quadratic function, they showed that if η > 3 /ℓ , the distance to the optimal solution ∥ x t -x ∗ ∥ exhibits exponential growth in t during the initial ˜ Θ( T/ log T ) iterations. However, this exponential growth is rapidly mitigated afterward, resulting in no exponential term at the final iterate T . This behavior starkly contrasts with our nonconvex setting.

We illustrate our hard instance for Theorem 2 in Figure 2, which is one-dimensional. The algorithm starts from a valley of the function f ( x ) = ℓx 2 / 2 , i.e., Segment 1. Because of the large initial stepsize and steep slope, in the first t 0 iterations, Gradient Descent increases the function value as large as ˜ ∆ = Ω ( (8 e ) η 2 ℓ 2 / 16 ∆ ) . Then the iterate x t 0 +1 jumps to the top of a very flat valley, i.e., Segment 4, so that Gradient Descent decreases the gradient as slowly as Ω( T -1 / 4 ) .

Why do not we assume gradients to be bounded? The assumption on bounded gradients is not satisfied even for the simple function f ( x ) = ℓx 2 / 2 . When training neural networks, gradient explosion is often observed [55, 59], which directly suggests that this assumption is not satisfied or only satisfied with a numerically large constant. In Proposition 4 in the appendix, we also provide a simple proof for the convergence under the additional assumption of bounded gradient, i.e., ∥∇ f ( x ) ∥ ≤ G for all x , attaining a sample complexity of ˜ O ( η 2 ℓ 2 G 4 σ 2 ϵ -4 ) without any information about problem parameters. However, compared with Theorem 1 and 2, constant G hides the exponential term. In Figure 1, we observe that the gradient bound along the trajectory of non-adaptive stepsize can be much larger than that of adaptive stepsize even if starting from the same initial point, so assuming bounded gradient will obscure the difference between them.

4 Power of Adaptive Methods

In this section, we focus on the convergence behaviors of adaptive methods, which adjust their stepsizes based on the observed gradients. In particular, when arriving at a point with a large gradient, adaptive methods automatically decrease their stepsizes to counter the effect of possible gradient increase; to list a few, Normalized SGD [26], AdaGrad [18], Adam [35]. Since the analysis for adaptive methods is usually on a case-by-case basis, we will examine three examples - Normalized SGD, AMSGrad-norm, and AdaGrad-norm - to establish a universal observation that they avoid exponential dependency in ℓ without tuning. Although many existing analyses rely on bounded gradients (and function values) or information on problem parameters, we will abandon such assumptions as noted in the previous section. We focus on the norm instead of the coordinate-wise version of adaptive methods, which means each coordinate adopts the same stepsize, because the norm version is usually dimension-independent in the complexity, and is also widely used in both theory and practice [73, 45, 43, 38, 54, 32].

4.1 Family of Normalized SGD

Normalized (Stochastic) Gradient Descent [53, 26], referred to as NGD and NSGD, is one of the simplest adaptive methods. It takes the stepsize in (1) to be normalized by the norm of the current (stochastic) gradient:

where { γ t } t ≥ 0 is a sequence of positive learning rate. Cutkosky and Mehta [14] and Zhao et al. [74] show that NSGD with γ t = γ/ √ T can find an O (1 / √ T + σ ) -stationary point. In order to compare fairly with untuned SGD with decaying stepsize, we present a modification with decaying γ t = γ/ √ t +1 in NSGD.

Proposition 1. Under Assumption 1 and 2, if we run NSGD with γ t = γ √ t +1 , then for any γ > 0 ,

NGD. In the deterministic setting, by Proposition 1, NGD converges to an ϵ -stationary point with a complexity of ˜ O (( γ -2 + γ 2 ℓ 2 ) ϵ -2 ) for any γ > 0 , which importantly does not include any exponential term. Thus, even if the initial stepsize is not small enough, it does not result in a catastrophic gradient explosion.

NSGD. In the stochastic setting, Proposition 1 implies that NSGD can find an ϵ -stationary point only when the noise variance is small enough, i.e., σ ≤ O ( ϵ ) . This is not the consequence of a loose analysis. Hazan et al. [26] show that NSGD with constant γ t ≡ γ does not converge when the mini-batch size is smaller than Θ( ϵ -1 ) for a non-smooth convex function. Here we provide a nonconvergence result in the gradient norm with a smooth objective for all uniformly bounded stepsizes. The intuition behind this is illustrated in Figure 3 in the appendix, where E ξ g ( x ; ξ ) / ∥ g ( x ; ξ ) ∥ can easily vanish or be in the opposite direction of ∇ f ( x ) under certain noises.

Theorem 3. Fixing ℓ > 0 , σ > 0 , ϵ > 0 , ∆ > 0 and stepsize sequence { γ t } ∞ t =0 with γ t ≤ γ max that ϵ 2 < min { σ 2 , 2 ℓ ∆ , 2∆( σ -ϵ ) /γ max } , there exists an ℓ -smooth convex function f , initial point x 0 with f ( x 0 ) -min x f ( x ) ≤ ∆ and zero-mean noises with σ 2 variance such that the output from NSGD satisfies E ∥∇ f ( x t ) ∥ ≥ ϵ for all t .

This theorem implies that, given a fixed function class ( ℓ, ∆ , σ ) and any sequence { γ t } t uniformly upper bounded by γ max , NSGD cannot converge to an arbitrarily small ϵ . Specifically, the expected gradient norm will always stay larger than min { σ, √ 2 ℓ ∆ , γ -1 max ( -∆+ √ ∆ 2 +2∆ γ max σ ) } . Most { γ t } t used in practice is upper bounded, e.g., constant or decreasing sequences. The condition ϵ 2 < 2 ℓ ∆ is necessary by noting that ∥∇ f ( x 0 ) ∥ 2 ≤ 2 ℓ [ f ( x 0 ) -min x f ( x )] ≤ 2 ℓ ∆ . Considering γ t = 1 / √ t +1 , when ∆ ≥ σ and √ 2 ℓ ∆ ≥ σ , it matches with Proposition 1, where NSGD can only converge to a Θ( σ ) -stationary point. Since Sign-SGD and NSGD coincide for one-dimensional objectives, our non-convergent example also applies to Sign-SGD. This sheds light on why increasing batch size improves Normalized and Sign-SGD [74, 37]. However, these methods are generally different in higher dimensions, as Karimireddy et al. [29] show that sign-SGD may not converge even with a full batch.

NSGD with momentum. While NSGD may not always converge, Cutkosky and Mehta [14] introduced NSGD with momentum (NSGD-M) presented in Algorithm 1 with constant γ t ≡ γ . We provide the following modification with diminishing γ t that eliminates the need to specify the total number of runs beforehand.

Proposition 2. Under Assumptions 1 and 2, if we run NSGD-M with α t = √ 2 √ t +2 and γ t = γ ( t +1) 3 / 4 , then for any γ > 0 ,

where C > 0 is a numerical constant.

It implies that NSGD-M attains a complexity of ˜ O (( γ -4 + γ 4 ℓ 4 ) ϵ -4 ) for any γ > 0 . Compared with Theorem 1 and 2, NSGD-M not only achieves near-optimal dependency in the target accuracy ϵ , but also shreds the exponential term when the hyper-parameter is agnostic to smoothness constant.

Algorithm 1 NSGD-M

1: Input: initial point x 0 , stepsize sequence { γ t } , momentum sequence { α t } , and initial momentum g 0 .

2: for t = 0 , 1 , 2 , ... do

3: x t +1 = x t -γ t ∥ g t ∥ g t

4: sample ξ t +1

5: g t +1 = (1 -α t ) g t + α t g ( x t +1 ; ξ t +1 )

6: end for

4.2 AMSGrad-norm

AMSGrad was introduced by Reddi et al. [57] to fix the possible non-convergence issue of Adam. Notably, current analyses of AMSGrad in the stochastic setting show a convergence rate of ˜ O (1 /T 1 / 4 ) , but they rely on the assumption of bounded stochastic gradients [12, 75], which is much stronger than assumptions used for SGD analysis. Here, we examine the simpler norm version of AMSGrad, presented in Algorithm 2. We prove that without assuming bounded stochastic gradients, AMSGradnorm with default γ t = γ/ √ t +1 may converge at an arbitrarily slow polynomial rate. In fact, this holds even if the true gradients are bounded. We believe this result is of independent interest.

Theorem 4. For any ℓ > 0 , ∆ > 0 , σ > 0 and T > 1 , there exists a ℓ -smooth function f : R 2 → R 2 , x 0 with f ( x 0 ) -inf x f ( x ) ≤ ∆ and noise distribution P with variance upper bounded by σ 2 , such that if we run AMSGrad-norm with 0 ≤ β 1 ≤ 1 , 0 ≤ β 2 < 1 and γ t = γ √ t +1 , we have with probability 1 2 , it holds that

for any 1 2 < ζ < 1 , where Γ( · ) denotes the Gamma function.

The intuition behind this theroem is that since AMSGrad utilizes the maximum norm of past stochastic gradients with momentum in the denominator of stepsizes, some noise distributions enable this maximum norm to increase polynomially, making the stepsizes too small. However, we can still explore its benefit in the deterministic setting. Whether it converges without assuming bounded gradients, to the best of our knowledge, is unknown. Here, for simplicity, we consider AMSGrad-norm without momentum, i.e., β 1 = β 2 = 0 .

Theorem 5. Under Assumption 1, if we run AMSGrad-norm with γ t = γ √ t +1 , v 0 > 0 and β 1 = β 2 = 0 in the deterministic setting, then for any γ > 0 and 0 < α < 1 ,

The theorem implies that AMSGrad-norm achieves a complexity of ˜ O (( ℓ 4 γ 4 + ℓ 2 + ℓ 3 γ 2 + ℓγ -2 ) ϵ -4 ) with the default γ t = Θ( t -1 / 2 ) [57, 12, 25]. Compared with untuned Gradient Descent, it gets rid of the exponential dependency. In the proof, we show that before the first iteration τ when stepsize η t

Algorithm 2 AMSGrad-norm

1: Input: initial point x 0 , momentum parameters 0 ≤ β 1 < 1 and 0 ≤ β 2 ≤ 1 , stepsize sequence { γ t } and initial momentum m 0 and v 0 > 0 .

2: ˆ v 0 = v 0

3: for t = 0 , 1 , 2 , ... do

4: sample ξ t

5: m t +1 = β 1 m t +(1 -β 1 ) g ( x t ; ξ t )

7: ˆ v 2 t +1 = max { ˆ v 2 t , v 2 t +1 }

6: v 2 t +1 = β 2 v 2 t +(1 -β 2 ) ∥ g ( x t ; ξ t ) ∥ 2

8: x t +1 = x t -γ t √ ˆ v 2 t +1 m t +1

9: end for

reduces to 1 /ℓ , the accumulated gradient norms ∑ τ -1 t =0 ∥∇ f ( x t ) ∥ 2 are upper bounded polynomially, which is in striking contrast with SGD in Theorem 2. We further provide theoretical guarantees for more general schemes γ ( t +1) α with 0 < α < 1 in Theorem 7 in the appendix. We also derive matching lower bounds in Theorem 8 for any 0 < α < 1 , and justify that AMSGrad may fail to converge with constant γ t ≡ γ (i.e., α = 0 ) if the problem parameter is unknown.

4.3 AdaGrad-norm

AdaGrad chooses its stepsize to be inversely proportional to the element-wise accumulated past gradients [18, 48]. Its norm-version, AdaGrad-norm [61, 66], picks stepsize in (1) to be

where v 0 > 0 . Very recently, AdaGrad is proven to converge in nonconvex optimization without the assumption on bounded gradients or tuning η [20, 68]. Although the result in [68] are presented for minimax optimization problems, a similar result follows immediately for minimization problems. We present the following result for the completeness of the paper and to further illustrate the benefits of adaptive methods over SGD.

Proposition 3. Under Assumptions 1 and 2, if we run AdaGrad-norm, then for any η > 0 and v 0 > 0 ,

The above result implies a complexity of ˜ O ( ( η -2 + σ 2 + η 2 ℓ 2 ) ( ϵ -2 + σ 2 ϵ -4 )) . Notably, if η can be chosen to be 1 / √ ℓ , it achieves the optimal complexity in both ℓ and ϵ up to logarithmic terms like well-tuned SGD [3]. Even if η is agnostic to ℓ , AdaGrad-norm does not suffer from the exponential term present in untuned SGD. One of the intuitions in the deterministic setting, similar to the AMSGrad-norm, is that the accumulated squared gradient norm before the first iteration with stepsize smaller than 1 /ℓ will be upper bounded by a polynomial term (see Theorem 3.2 in [42]). Another benefit of AdaGrad over other methods is to achieve optimal convergence rates simultaneously in deterministic and stochastic settings with the same hyper-parameters. This is sometimes referred to as 'noise adaptivity', which is out of the scope of this paper.

5 Conclusion and Future Directions

We convey a crucial message: SGD with a polynomially decaying stepsize can converge at the order-optimal rate, while remaining agnostic to the problem-specific parameter - a notion that may challenge common belief. However, it is subject to an exponential term, which can be avoided by utilizing an appropriate adaptive scheme. We further reveal the similarity and differences between several adaptive schemes; a detailed exposition of these results can be found in Appendix A. This work opens up several avenues for future research. Firstly, it would be valuable to investigate whether the benefits of adaptive methods extend to high-probability convergence guarantees and apply to a broader range of adaptive optimizers. Secondly, removing the assumption of bounded gradients is crucial for a comprehensive analysis of adaptive algorithms, as it can reveal the true dependence on the smoothness parameter ℓ and highlight their advantage over SGD. Lastly, examining the impact of adaptive algorithms on optimizing non-smooth nonconvex objectives, which are prevalent in training modern machine learning models, presents an interesting research direction.

References

Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. arXiv preprint arXiv:2204.01050 , 2022. (Cited on page 6.)

Ahmad Ajalloeian and Sebastian U. Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008.00051 , 2021. (Cited on page 4.)

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , pages 1-50, 2022. (Cited on pages 2, 4, 5, 7, 10, and 16.)

Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of mathematics , 16(1):1-3, 1966. (Cited on page 4.)

Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning , pages 404-413. PMLR, 2018. (Cited on page 5.)

Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5(2):157-166, 1994. (Cited on page 2.)

Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning , 2(1):1-127, 2009. (Cited on page 1.)

Julius R Blum. Multidimensional stochastic approximation methods. The Annals of Mathematical Statistics , pages 737-744, 1954. (Cited on page 4.)

L´ eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review , 60(2):223-311, 2018. (Cited on pages 4 and 5.)

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. (Cited on page 4.)

Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming , 184(1-2):71-120, 2020. (Cited on pages 2 and 16.)

Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. In International Conference on Learning Representations . International Conference on Learning Representations, ICLR, 2019. (Cited on pages 4 and 9.)

Kai Lai Chung. On a stochastic approximation method. The Annals of Mathematical Statistics , pages 463-483, 1954. (Cited on page 4.)

Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning , pages 2260-2268. PMLR, 2020. (Cited on pages 2, 4, 8, 16, 23, and 24.)

Felipe RS De Gusmao, Edwin MM Ortega, and Gauss M Cordeiro. The generalized inverse weibull distribution. Statistical Papers , 52:591-619, 2011. (Cited on page 30.)

Alexandre D´ efossez, L´ eon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395 , 2020. (Cited on page 4.)

Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. In International Conference on Machine Learning , pages 2658-2667. PMLR, 2020. (Cited on pages 2, 4, 6, 7, 19, 21, and 28.)

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011. (Cited on pages 2, 4, 5, 7, and 10.)

Ilyas Fatkhullin, Anas Barakat, Anastasia Kireeva, and Niao He. Stochastic policy gradient methods: Improved sample complexity for fisher-non-degenerate policies. arXiv preprint arXiv:2302.01734 , 2023. (Cited on page 25.)

Matthew Faw, Isidoros Tziotis, Constantine Caramanis, Aryan Mokhtari, Sanjay Shakkottai, and Rachel Ward. The power of adaptivity in sgd: Self-tuning step sizes with unbounded gradients and affine variance. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 313-355. PMLR, Jul 2022. (Cited on pages 3, 4, and 10.)

Xavier Fontaine, Valentin De Bortoli, and Alain Durmus. Convergence rates and approximation results for sgd and its continuous-time counterpart. In Conference on Learning Theory , pages 1965-2058. PMLR, 2021. (Cited on page 6.)

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013. (Cited on pages 2, 4, and 5.)

Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, Yang Zhang, and Jonathan M Cohen. Stochastic gradient methods with layer-wise adaptive moments for training of deep networks. arXiv preprint arXiv:1905.11286 , 2019. (Cited on page 4.)

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . The MIT Press, 2016. (Cited on page 2.)

Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. A novel convergence analysis for algorithms of the adam family. arXiv preprint arXiv:2112.03459 , 2021. (Cited on page 9.)

Elad Hazan, Kfir Levy, and Shai Shalev-Shwartz. Beyond convexity: Stochastic quasi-convex optimization. Advances in neural information processing systems , 28, 2015. (Cited on pages 2, 4, 5, 7, and 8.)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. (Cited on page 3.)

Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on , 14(8):2, 2012. (Cited on pages 2 and 4.)

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning , pages 3252-3261. PMLR, 2019. (Cited on page 8.)

Ali Kavis, Kfir Y Levy, Francis Bach, and Volkan Cevher. Unixgrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization. NeurIPS , 32, 2019. (Cited on page 4.)

Ali Kavis, Kfir Levy, and Volkan Cevher. High probability bounds for a class of nonconvex algorithms with adagrad stepsize. In ICLR , 2022. (Cited on page 4.)

Ali Kavis, Stratis Skoulakis, Kimon Antonakopoulos, Leello Tadesse Dadi, and Volkan Cevher. Adaptive stochastic variance reduction for non-convex finite-sum minimization. arXiv preprint arXiv:2211.01851 , 2022. (Cited on page 7.)

Ahmed Khaled and Peter Richt´ arik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329 , 2020. (Cited on page 6.)

J. Kiefer and J. Wolfowitz. Stochastic Estimation of the Maximum of a Regression Function. The Annals of Mathematical Statistics , 23(3):462 - 466, 1952. (Cited on page 4.)

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015. (Cited on pages 2, 4, 5, and 7.)

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. (Cited on page 3.)

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Heavy-tailed noise does not explain the gap between SGD and Adam, but sign descent might. In International Conference on Learning Representations , 2023. (Cited on pages 5 and 8.)

Joffrey L Leevy and Taghi M Khoshgoftaar. A survey and analysis of intrusion detection models based on cse-cic-ids2018 big data. Journal of Big Data , 7(1):1-19, 2020. (Cited on page 7.)

Yunwen Lei, Ting Hu, Guiying Li, and Ke Tang. Stochastic gradient descent for nonconvex learning without bounded gradient assumptions. IEEE transactions on neural networks and learning systems , 31(10):4394-4400, 2019. (Cited on page 4.)

Kfir Levy, Ali Kavis, and Volkan Cevher. Storm+: Fully adaptive sgd with recursive momentum for nonconvex optimization. NeurIPS , 34:20571-20582, 2021. (Cited on page 4.)

Kfir Y Levy. The power of normalization: Faster evasion of saddle points. arXiv preprint arXiv:1611.04831 , 2016. (Cited on page 5.)

Xian Li. Adaptive methods for parameter-agnostic nonconvex minimax optimization. Master's thesis, ETH Zurich, Department of Computer Science, 2022. (Cited on page 10.)

Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference on artificial intelligence and statistics , pages 983-992. PMLR, 2019. (Cited on pages 4 and 7.)

Xiaoyu Li and Francesco Orabona. A high probability analysis of adaptive sgd with momentum. arXiv preprint arXiv:2007.14294 , 2020. (Cited on page 4.)

Selena Ling, Nicholas Sharp, and Alec Jacobson. Vectoradam for rotation equivariant geometry optimization. arXiv preprint arXiv:2205.13599 , 2022. (Cited on page 7.)

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations , 2020. (Cited on page 2.)

Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249 , 2020. (Cited on page 4.)

H Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908 , 2010. (Cited on pages 4 and 10.)

Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems , 24, 2011. (Cited on page 6.)

Ryan Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Transactions on Automatic Control , 64(11):4818-4824, 2019. (Cited on page 5.)

A. S. Nemirovski and D. B. Yudin. Problem Complexity and Method Efficiency in Optimization. John Wiley UK/USA, 1983. (Cited on page 4.)

Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003. (Cited on page 6.)

Yurii E Nesterov. Minimization methods for nonsmooth convex and quasiconvex functions. Matekon , 29(3):519-531, 1984. (Cited on pages 2, 4, and 7.)

Werner Palfinger. Continuous remeshing for inverse rendering. Computer Animation and Virtual Worlds , 33(5):e2101, 2022. (Cited on page 7.)

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310-1318. PMLR, 2013. (Cited on pages 2, 4, and 7.)

Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838-855, 1992. (Cited on page 4.)

Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICLR , 2018. (Cited on pages 2, 4, 5, and 9.)

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951. (Cited on pages 1 and 4.)

J¨ urgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks , 61: 85-117, 2015. (Cited on page 7.)

Naichen Shi and Dawei Li. Rmsprop converges with proper hyperparameter. In International conference on learning representation , 2021. (Cited on page 4.)

Matthew Streeter and H Brendan McMahan. Less regret via online conditioning. arXiv preprint arXiv:1002.4862 , 2010. (Cited on pages 2, 4, and 10.)

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018. (Cited on page 1.)

Sharan Vaswani, Benjamin Dubois-Taine, and Reza Babanezhad. Towards noise-adaptive, problem-adaptive (accelerated) stochastic gradient descent. In International Conference on Machine Learning , pages 22015-22059. PMLR, 2022. (Cited on page 7.)

Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, and Wei Chen. Provable adaptivity in adam. arXiv preprint arXiv:2208.09900 , 2022. (Cited on page 5.)

Xiaoyu Wang, Sindri Magn´ usson, and Mikael Johansson. On the convergence of step decay step-size for stochastic optimization. Advances in Neural Information Processing Systems , 34: 14226-14238, 2021. (Cited on page 4.)

Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research , 21(1):9047-9076, 2020. (Cited on pages 4 and 10.)

Zeke Xie, Xinrui Wang, Huishuai Zhang, Issei Sato, and Masashi Sugiyama. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In International Conference on Machine Learning , pages 24430-24459. PMLR, 2022. (Cited on page 5.)

Junchi Yang, Xiang Li, and Niao He. Nest your adaptive algorithm for parameter-agnostic nonconvex minimax optimization. arXiv preprint arXiv:2206.00743 , 2022. (Cited on pages 2, 3, 4, 10, 16, and 32.)

D. Matthew Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701v1 , 2012. (Cited on page 2.)

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations , 2019. (Cited on page 5.)

Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems , 33:15383-15393, 2020. (Cited on page 5.)

Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. Advances in neural information processing systems , 2022. (Cited on page 4.)

Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS) , pages 1-2. Ieee, 2018. (Cited on page 7.)

Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences , 64(3):1-13, 2021. (Cited on pages 4 and 8.)

Dongruo Zhou, Jinghui Chen, Yuan Cao, Yiqi Tang, Ziyan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671 , 2018. (Cited on pages 4 and 9.)

Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for convergences of adam and rmsprop. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11127-11135, 2019. (Cited on page 4.)

A Results Summary

In this work, we study stochastic gradient methods for minimizing smooth functions in the parameteragnostic regime. Firstly, we show SGD with polynomially decaying stepsize 1 / √ t is able to converge with the order-optimal rate, with and without bounded gradients (Proposition 4 and Theorem 1). Its limitation lies in an unavoidable exponential term in ℓ 2 when we do not assume bounded gradients (Theorem 2). We demonstrate that several existing adaptive methods do not suffer from the exponential dependency, such as NGD, AdaGrad, AMSGrad-norm in the deterministic setting (Proposition 1 and Theorem 5), and NSGD-M, AdaGrad in the stochastic setting (Proposition 2 and Proposition 3). However, it does not mean adaptive methods are always better than SGD. We provide a non-convergence result for NSGD (Theorem 3) and a slow convergence result for AMSGrad-norm (Thoerem 4) in the stochastic case. We believe our results shed light on explaining commonly observed large gradients during training and provide a better theoretical understanding of the convergence behaviors of adaptive methods in the regime with unbounded stochastic gradients.

B Proofs for SGD in Section 3

B.1 Upper Bounds for SGD

We provide an extended theorem of Theorem 1 and include more general decaying stepsizes η t = η/ ( t +1) α with 0 < α < 1 .

Theorem 6. Under Assumptions 1 and 2, if we run SGD with stepsize η t = η/ ( t +1) α where η > 0 and 1 / 2 ≤ α < 1 , then with η ≤ 1 /ℓ ,

with η > 1 /ℓ ,

where τ = ⌈ ( ηℓ ) 1 /α -1 ⌉ .

Proof. By ℓ -smoothness of f ( · ) ,

Taking expectation,

We note that η t -ℓη 2 t 2 ≥ η t 2 when η t ≤ 1 ℓ , i.e., t ≥ ( ηℓ ) 1 /α -1 . Define τ = ⌈ ( ηℓ ) 1 /α -1 ⌉ . Therefore, for all t < τ ,

For all t ≥ τ , we have

Summing from t = τ to T -1 , we have

Now we want to bound E f ( x τ ) -f ( x T ) ≤ E f ( x τ ) -f ∗ , where f ∗ ≜ min x ∈ R d f ( x ) . From (3),

where in the second inequality we use ∥∇ f ( x ) ∥ 2 ≤ 2 ℓ [ f ( x ) -f ∗ ] . When τ = 0 , f ( x τ ) -f ( x T ) ≤ ∆ ; when τ ≥ 1 , recursing the inequality above, for j ≤ τ ,

Also, with ∥∇ f ( x ) ∥ 2 ≤ 2 ℓ [ f ( x ) -f ∗ ] , if τ ≥ 1 ,

where in the second inequality we use (6) . Combining with (5) and (6), if τ ≥ 1

We note that

where in the third inequality we use Stirling's approximation. Therefore,

Plugging in η t = η/ ( t +1) α , when α = 1 / 2 ,

when 1 / 2 < α < 1 ,

when 0 < α < 1 / 2 ,

If τ = 0 , from (5),

Plugging in η t , when α = 1 / 2 ,

Remark 2. When we run SGD with stepsize η t = η/ ( t + 1) α , where 1 / 2 < α < 1 , Theorem 6 implies a complexity of O ( (4 e 2 α ) ( ηℓ ) 1 /α 1 -α ( ηℓ ) 1 α (1 -α ) · ϵ -2 1 -α ) in the large initial stepsize regime η > 1 /ℓ . Compared with the case α = 1 / 2 , when α is larger, the convergence rate in T is slower, but it also comes with a smaller exponent, i.e., ( ηℓ ) 1 /α . This is because α = 1 / 2 leads to the best convergence rate in T [17], while the faster decaying stepsize α > 1 / 2 will reach the desirable stepsize 1 /ℓ earlier so that it accumulates less gradient norms before τ . For 0 < α < 1 / 2 , however, it comes with both a larger exponent and a slower convergence rate.

Proposition 4 (with bounded gradient) . Under Assumption 1, 2 and additionally assuming that the gradient norm is upper bounded by G , i.e., ∥∇ f ( x ) ∥ ≤ G for all x ∈ R d , if we run SGD with stepsize η t = η/ √ t +1 with η > 0 , then

Proof. By the smoothness of f ( · ) , we have

Taking expectation and summing from t = 0 to T -1 ,

when 1 / 2 < α < 1 , when 0 < α < 1 / 2 ,

B.2 Lower Bound for SGD

Proof for Theorem 2

Proof. We construct the hard instance with 4 segments of quadratic functions. The function is symmetric about x = 0 , and we will define it on x ≤ 0 as below. We illustrate it in Figure 2.

Segment 1. We define f ( x ) = ℓ 2 x 2 . We pick x 0 such that f ( x 0 ) -f ∗ = ∆ , i.e., x 0 = √ 2∆ ℓ . We define t 0 to be the first iteration that η t 0 = η √ t 0 +1 ≥ 4 ℓ , i.e., t 0 = ⌊ η 2 ℓ 2 16 -1 ⌋ . With the update rule

where in the inequality we use ( ηℓ √ k -1 ) 2 ≥ η 2 ℓ 2 2 k with k ≤ t 0 , in the second inequality we use Stirling's approximation, and in the last inequality we use t ≤ η 2 ℓ 2 / 16 . We note that

Without loss of generality, we assume x t 0 > 0 . Segment 1 is define on the domain { x : | x | ≤ x t 0 } .

Segment 2. This segment is the mirror of Segment 1. On domain { x : x t 0 ≤ x ≤ 2 x t 0 } , we define f ( x ) = -ℓ 2 ( x -2 x t 0 ) 2 + ℓx 2 t 0 .

Segment 3. We note that

where the inequality is from the definition of t 0 , and

We construct a quadratic function such that: it passes ( -2 x t 0 , ℓx 2 t 0 ) with gradient 0; the gradient at x = x t 0 +1 is √ ˜ ∆ 2 √ max { 1 /ℓ, ∑ T -1 t = t 0 +1 η t } . This quadratic function is uniquely defined to be

It can be verified that this function is ℓ -smooth: as x t 0 +1 ≤ -3 x t 0 ,

The function is defined on the domain { x : x t 0 +1 ≤ x ≤ -2 x t 0 } .

Segment 4. For convenience, we define w = f ( x t 0 +1 ) . We can verify that w ≥ ˜ ∆ : as 1 √ t 0 < 4 ηℓ ,

So we conclude w ≥ ˜ ∆ . Now we construct a quadratic function similar to that in Proposition 1 of [17]: it passes ( x t 0 +1 , w ) with gradient √ ˜ ∆ 2 √ max { 1 /ℓ, ∑ T -1 t = t 0 +1 η t } ; the minimum is at x = x t 0 +1 -

√ ˜ ∆max { 1 /ℓ, ∑ T -1 t = t 0 +1 η t } . This quadratic function is defined to be on the domain { x : x ≤ x t 0 +1 } . It is obvious that f ( x ) ≥ 0 and is ℓ -smooth. Following the same reasoning of Proposition 1 in [17], also presented as Lemma 1 in the appendix for completeness, we can conclude for all t : t 0 +1 ≤ t ≤ T ,

where in the second inequality we use T -1 t = t 0 +1 η t = T -1 t = t 0 +1 η √ t +1 2 ηT 1 / 2

∑ ∑ ≤ .

The following lemma is used in the proof of Theorem 2. It is a straightforward modification of Proposition 1 in [17]. We present it here for completeness.

Lemma 1. Under the same setting and notations as the proof of Theorem 2, if we run gradient descent with stepsize { η t } T -1 t = t 0 +1 starting from point x t 0 +1 on function then for all t : t 0 +1 ≤ t ≤ T ,

Proof. From the update of gradient descent, we have which leads to

Recursing this, for j ≤ T

where in the second inequality, we use 1 -z/ 2 ≥ exp(log 1 2 · z ) for 0 ≤ z ≤ 1 . This directly implies what we want to prove by computing ∇ f ( x j ) .

C Proofs for NSGD Family in Section 4

Algorithm 3 Normalized Stochastic Gradient Descent (NSGD)

1: Input: initial point x 0

2: for t = 0 , 1 , 2 , ... do

3: sample ξ t and set learning rate γ t

5: end for

Proof for Theorem 3

Proof. Let us pick f ( x ) = L 2 x 2 with ϵ 2 2∆ < L ≤ ℓ and L < σ -ϵ γ max . Then we pick x 0 such that ϵ L < x 0 < √ 2∆ L , which implies that ∥∇ f ( x 0 ) ∥ > ϵ and f ( x 0 ) -min x f ≤ ∆ . Now we define D = { x : -w ≤ x ≤ w } with ϵ L + γ max < w < σ L . For x ∈ D , we have ∥∇ f ( x ) ∥ ≤ σ and we

normalization

(a) example 1

normalization

(b) example 2

construct the noisy gradients: with δ > 1

It is obvious that ∇ f ( x ) = E [ g ( x ; ξ )] and the variance at this point E ∥∇ f ( x ) -g ( x ; ξ ) ∥ 2 = δ 2 ∥∇ f ( x ) ∥ 2 ≤ σ 2 with δ sufficiently close to 1. With the update rule, we note that x t +1 = x t -γ t w.p. 1/2 and x t +1 = x t + γ t w.p. 1/2, and therefore

For x / ∈ D , we have ∥ x ∥ > ϵ/L + γ max , and we assume there is no noise in the gradients. Therefore, if x t / ∈ D , we know that after one step of update ∥ x t +1 ∥ > ϵ/L , which implies ∥∇ f ( x t +1 ) ∥ > ϵ . Combining two cases that x t ∈ D and x t / ∈ D , we know that E ∥∇ f ( x t ) ∥ > ϵ for all t .

Proof of Proposition 1

Proof. Denote e t = g ( x t ; ξ t ) -∇ f ( x t ) . By Lemma 2 in [14],

Telescoping from t = 0 to T -1 ,

Taking expectation, rearranging and using E [ ∥ e t ∥ ] ≤ ( E [ ∥ e t ∥ 2 ]) 1 / 2 ≤ σ , we derive

Proof for Proposition 2

Proof. We define ˆ e t = g t -∇ f ( x t ) . By Lemma 2 in [14], for any γ t > 0

Telescoping from t = 0 to T -1 ,

By taking expectation on both sides, rearranging and controlling the variance term using Lemma 2, we derive

Lemma 2. Under the setting of Theorem 2, there exist numerical constants C 1 , C 2 > 0 such that for all t ≥ 1 , where ˆ e t = g t -∇ f ( x t ) .

Proof. Define e t = g ( x t ; ξ t ) -∇ f ( x t ) , S t = ∇ f ( x t ) -∇ f ( x t +1 ) . Then

Unrolling the recursion from t = T -1 to t = 0 , we have

Define the σ -field F t := σ ( { x 0 , ξ 0 , . . . , ξ t -1 } ) . Notice that for any t 2 > t 1 ≥ 0 we have

Then taking norm, applying expectation on both sides of (8) and using E [ ∥ ˆ e 0 ∥ ] ≤ σ , we have

where the first inequality holds by Jensen's inequality applied to x ↦→ x 2 , the second inequality follows by (9) and the bound ∥ S t ∥ ≤ ℓ ∥ x t +1 -x t ∥ = ℓγ t . The last step is due to bounded variance E [ ∥ ˆ ϵ 0 ∥ ] ≤ σ and α t ≤ 1 .

By the choice of momentum sequence, we have α 0 = 1 and the first term is zero. By Lemma 3, there exist numerical constants C 1 , C 2 > 0 such that

Therefore, for all T ≥ 1 , we have

Lemma 3 (Lemma 15 in [19]) . Let q ∈ [0 , 1) , p ≥ 0 , γ 0 > 0 and let η t = ( 2 t +2 ) q , γ t = γ 0 ( 1 t +1 ) p for every integer t . Then for any integers t and T ≥ 1 , it holds

where C := 2 p -q (1 -q ) -1 t 0 exp ( 2 q (1 -q ) t 1 -q 0 ) + 2 2 p +1 -q (1 -q ) -2 and t 0 := max { ( p (1 -q )2 q ) 1 1 -q , 2 ( p -q (1 -q ) 2 ) } 1 1 -q .

D Proofs for Scalar AMSGrad and AdaGrad in Section 4

The following is an extended version of Theorem 5 including γ t = γ ( t +1) α with 0 < α < 1 .

Theorem 7. Under Assumption 1, if we run AMSGrad-norm with γ t = γ ( t +1) α , v 0 > 0 and β 1 = β 2 = 0 in the deterministic setting, then for any γ > 0 and 0 < α < 1 , if v 0 < γℓ

if v 0 ≥ γℓ

where

Proof. The effective stepsize of AMSGrad-norm contains a maximum over all gradient norms in the denominator. As it is desirable to find a lower bound for the effective stepsize, we begin by bounding the gradient norms.

Let τ be the first iteration where the effective stepsize is less or equal to 1 /ℓ , i.e., η τ -1 > 1 /ℓ and η τ ≤ 1 /ℓ . First, we assume τ ≥ 1 , i.e., v 0 < γℓ . The time stamp τ itself is naturally bounded by

We have

By ℓ -smoothness of f ( · ) ,

Therefore,

We denote the right hand side as M . Also from (11) and definition of τ , we know that f ( x t ) ≤ f ( x τ ) for t ≥ τ and therefore, for all t ≥ τ ,

which implies

Therefore, we can bound for all t ≥ 0 ,

For t ≥ τ , by (11)

By telescoping from t = τ to T -1 , we get

Then we have

Combining with (10), we obtain

When τ = 0 , we have

which implies

Remark 3. For any 0 < α < 1 , if we compare simplified AMSGrad with γ t = γ ( t +1) α to SGD with η t = η ( t +1) α in the deterministic case (setting σ = 0 in Theorem 6), we observe that they achieve the same convergence rate. However, the complexity of simplified AMSGrad only includes polynomial term in γ and ℓ , while that of SGD includes an exponential term in ( ηℓ ) 1 /α .

In the following, we will first provide the lower bounds for scalar version of AMSGrad (referred to as AMSGrad-norm) with each α ∈ (0 , 1) and discuss why it may fail with α = 0 when problem parameters are unknown, which means that it can not achieve the optimal complexity O ( ϵ -2 ) in the deterministic setting. Second, we show that it also fails to achieve the optimal convergence rate in the stochastic setting when stochastic gradients are unbounded. To make the results more general, we consider the standard scalar AMSGrad with momentum hyper-parameters β 1 and β 2 , which is presented in Algorithm 2.

Before proceeding to our results, we present a lemma which is handy for conducing lower bounds for SGD-like algorithms with momentum (see Algorithm 4). As long as an upper bound is known for stepsize η t , we can derive a lower bound similar to Proposition 1 in [17].

Algorithm 4 General SGD with Momentum

1: Input: initial point x 0 , momentum parameters 0 ≤ β 1 < 1 and initial moment m 0 . 2: for t = 0 , 1 , 2 , ... do 3: sample ξ t 4: m t +1 = β 1 m t +(1 -β 1 ) g ( x t ; ξ t ) 5: obtain stepsize η t > 0 6: x t +1 = x t -η t m t +1 7: end for

Lemma 4. For any ℓ > 0 , ∆ > 0 and T > 1 , there exists a ℓ -smooth function f : R → R , and x 0 with f ( x 0 ) -inf x f ( x ) ≤ ∆ , such that if we run Algorithm 4 with deterministic gradients and η t ≤ ˜ η t for t = 0 , 1 , 2 , ..., T -1 , then we have

Proof. We construct a quadratic function similar to Proposition 1 in [17]. The following function is considered:

Without loss of generality, we assume the initial moment m 0 is non-positive, and we set the initial point x 0 as

Otherwise if the initial moment is set to be positive, then we let x 0 be negative and follow the same reasoning.

Since x 0 is positive, the first gradient direction would be positive, i.e., ∇ f ( x 0 ) > 0 . Let τ be the first iteration such that m τ > 0 . By the update rule and definition of τ , it is obvious that x t ≥ x 0 for t ≤ τ -1 . If T ≤ τ , it trivially holds that ∇ f ( x t ) ≥ ∇ f ( x 0 ) for all 0 ≤ t ≤ T -1 . Otherwise,

we have m τ = β 1 m τ -1 +(1 -β 1 ) ∇ f ( x τ -1 ) ≤ (1 -β 1 ) ∇ f ( x τ -1 ) . That is to say, the gradient estimation m τ used in the τ -th step has the correct direction but its magnitude is no larger than the actual gradient. Starting from the τ -th iteration, x t will monotonically move left towards the solution. Note that since our stepsize is small enough, i.e.,

the updates will remain positive, i.e., x t > 0 for t ≥ τ . By the update rule, we note that x t +1 ≤ x t for t ≥ τ , and therefore ∇ f ( x t +1 ) < ∇ f ( x t ) . We can conclude that for any t ≥ τ , we have m t ≤ ∇ f ( x τ -1 ) . Then for t ≥ τ -1 we have

Then we conclude by

Now we proceed to provide the lower bound for deterministic case.

Theorem 8. For any ℓ > 0 , ∆ > 0 and T > 1 , there exists a ℓ -smooth function f : R → R and x 0 with f ( x 0 ) -inf x f ( x ) ≤ ∆ , such that if we run Algorithm 2 with deterministic gradients, 0 < v 0 ≤ ℓγ 2 , and γ t = γ ( t +1) α with γ ≤ 4∆ v 0 , we have (1) if 0 < α < 1 , for any 0 ≤ β 1 < 1 and 0 ≤ β 2 ≤ 1 , we have

and (2) if α = 0 , for β 1 = 0 and any 0 ≤ β 2 ≤ 1 , we have

Remark 4. From the theorem, we can conclude that the optimal convergence rate 1 √ T for ∥∇ f ( x t ) ∥ is infeasible for AMSGrad with polynomially decreasing stepsize. When α = 0 , a similar result can be obtained for the case β 1 ≥ 0 , β 2 = 0 and small enough v 0 .

Proof. For α > 0 , we have

Let ˜ η t = γ ( t +1) α v 0 and then we have

Applying Lemma 4 directly gives us the desired result.

For α = 0 , we consider function

Note that since v 0 ≤ ℓγ 2 , the function is ℓ -smooth. Let

which satisfies the condition that f ( x 0 ) ≤ ∆ . Then after one update

If we continue this calculation, we find that the iterates will oscillate between γ 2 and -γ 2 forever, which finishes the proof.

Proof for Theorem 4

Proof. We consider a two-dimensional function f : R 2 → R 2 , for x = ( x 1 , x 2 ) ⊤ ∈ R 2 ,

where its function value only depends on the first dimension and we will define F : R → R later. The gradient at x is ∇ f ( x ) = ( ∇ F ( x 1 ) , 0) ⊤ . We add the noise only to the second dimension, i.e., g ( x ; ξ ) = ( ∇ F ( x 1 ) , ξ ) . For any t ≥ 0 , the probability density function of the noise as where s = σ √ Γ ( 1 -ζ 2 ) . Note that the distribution is symmetric and E [ ξ t ] = 0 . Also, we note that | ξ t | follows the Fr´ echet distribution [15] with cumulative distribution function

where we used the exact second moment for Fr´ echet distribution.

and

Next, we will show that ˜ ξ t := max 0 ≤ k ≤ t {| ξ k |} ≥ Ω ( 1 ( t +1) ζ -1 / 2 ) with probability 1 2 . We know that ˜ ξ t also follows Fr´ echet distribution with CDF

Then for constant C > 0 ,

where the last inequality is by selecting C = s ( e ( 1 ζ -1 )) ζ 2 √ 2 and using exp ( -2 m +1 em · t m ) ≤ 1 4 t 2 for any t > 0 and 0 < m < 1 . Then using union bound, we have

Now we have shown that with some probability, the noise is large enough. We can use this property to provide an upper bound ˜ η t for the stepsize as follow

This implies

We observe that the update with AMSGrad-norm in function f corresponds to applying general SGD with momentum (Algorithm 4) to function F with stepsize η t . Therefore, we can pick a hard instance F according to Lemma 4, and by noting that ∥∇ f ( x ) ∥ = |∇ F ( x 1 ) | we reach our conclusion.

Remark 5. As we see above, the function F in the proof is constructed by Lemma 4. We note that even assuming the gradients of f to be bounded, i.e., ∥∇ f ( x ) ∥ ≤ K for all x , will not prevent the slow convergence in Theorem 4. This is because in the proof of Lemma 4 all iterates stay between [0 , x τ -1 ] (e.g., τ = 1 if m 0 = 0 ), so we can construct any Lipschitz function outside of this segment.

Algorithm 5 AdaGrad-norm

1: Input: initial point x 0 , v 0 > 0 and η > 0 2: for t = 0 , 1 , 2 , ... do 3: sample ξ t 4: v 2 t +1 = v 2 t + ∥ g ( x k ; ξ k ) ∥ 2 5: x t +1 = x t -η √ v 2 t +1 g ( x t ; ξ t )

6: end for

Proof for Proposition 3

Proof. Define a function ˜ f : R d × R → R such that ˜ f ( x, y ) = f ( x ) -ℓ 2 y 2 . Since the ˜ f is ℓ -smooth and ℓ -strongly concave about y , the condition number is defined to be κ = 1 . Applying AdaGradnorm to f is equivalent to applying NeAda-AdaGrad (Algorithm 3 in [68]) to ˜ f with y t ≡ 0 . For every x , we know y ∗ ( x ) ≜ arg max y ˜ f ( x, y ) = 0 . Then E ≜ ∑ T -1 t =0 ℓ 2 ∥ y t -y ∗ ( x t ) ∥ 2 2 v 0 = 0 . Plugging in κ = 1 , E = 0 and batchsize M = 1 to Theorem 3.1 in [68], we reach the conclusion.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows mainly two things:\n(a) SGD suffers from an exponential dependence on the initial stepsize if it is not tuned to be smaller than the learning rate. This exponential dependence is unavoidable.\n(b) Methods with gradient normalization and running gradient sum normalization, such as Normalized SGD, AMSGrad, and AdaGrad, suffer no such exponential dependence on the smoothness constant.\n\nThe paper also presents a novel analysis of AMSGrad that removes the bounded gradients assumption.\n\nSTRENGTHS:\n1. The result on AMSGrad is new and a welcome addition to the literature.\n2. The paper's emphasis on the benefits of normalization even in the deterministic setting is good, since this is a point quite overlooked in the community.\n\nWEAKNESSES:\n1. The result on the exponential dependence on the smoothness constant is a known consequence of another result in the literature. Under the assumption on the stochastic gradients $\\mathbb{E} \\|g(x)\\|^2 <= 2 A (f(x)-f_*) + B \\|\\nabla f(x\\||^2 + C$. Note that bounded stochastic gradient variance corresponds to $A=L$, $B=0$ and $C=\\sigma$ (since bounded variance implies $E||g(x)||^2 <= ||\\nabla f(x)||^2 + C <= 2 L (f(x)-f_*) + C$). The result of Theorem 2 in [1] gives for this choice ($A=L$, $B=0$, $C=\\sigma$) a rate of $\\frac{(1+\\gamma^2 L^2)^K}{\\gamma K} \\delta_0 + L \\gamma C$ where $\\delta_0$ is the initial suboptimality. The lower bound is also known, see [2, Theorem 5].\n\n[1] Khaled and Richtárik. Better Theory for SGD in the Nonconvex World. arXiv:2002.03329\n[2] Vaswani, Benjamin Dubois-Taine, and Babanezhad. Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent. arXiv:2110.11442.\n\nQUESTIONS:\n1. Please address the difference between your results and the results I've mentioned in the weaknesses section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
The authors investigate the behavior of untuned SGD in the smooth nonconvex setting and show a new result on the convergence rate of SGD w.r.t. to gradient norm, where there is an exponential dependence on the smoothness constant. They further argue that the exponential dependence is unavoidable through a constructed class of 1-dimensional nonconvex functions. The paper then examines NSGD, AMSGrad and AdaGrad and shows that the exponential dependence can be avoided through adaptiveness, albeit without any information about the problem parameters.

STRENGTHS:
This paper offers an interesting theoretical perspective on the explosive gradient problem and the nonconvergence properties of SGD. The authors complement their theoretical results and ideas with numerical illustrations, and the paper is easy to follow. Their results seem well-justified, but I did not check their proofs in the appendix. I believe this work is of interest to the NeurIPS community.

WEAKNESSES:
1) The current state of numerical experiments seem preliminarily and is only done on one dataset MNIST on a small-network. I would like to see a more comprehensive investigation into larger practical networks, perhaps from [6, 24, 54].

QUESTIONS:
1) In figure 1, what is the size of each layer in the 3-layer neural network? What effect does over-parameterization have on the untuned SGD behavior?

LIMITATIONS:
Yes, the authors have addressed their limitations through the checklist.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the complexity of finding an $\\epsilon$-stationary point for untuned SGD and compares that with three families of adaptive methods - NSGD, AMSGrad and AdaGrad. Compared to previous convergence analysis results for tuned SGD and Adaptive methods: this work gets rid of several assumptions that hides the true convergence behaviour of these algorithms. Specifically, the authors do not assume the step-size for SGD to be dependent on the smoothness parameter (hence untuned) and do not assume bounded gradients for the adaptive methods. These leads to an interesting comparison for convergence of these algorithms.\n\nSTRENGTHS:\n1) The authors show that untuned SGD converges to an $\\epsilon$ stationary point in $O(e^{\\eta^2 l^2}\\epsilon^{-4})$ iterations. Although this algorithm does have optimal dependece on $\\epsilon$ , it has a disastrous exponential term wrt the smoothness parameter $\\eta^2 l^2$, Hence, the assumption on bounded gradients or chosing the $\\eta$ to depend on $l$ is problematic, because we may not have prior knowledge of $l$. They show that even for a smooth 1D-function, the assumption of bounded gradient is problematic. This is indeed true and the experiment in figure-1 supports this claim. \n\n2) Adaptive gradient methods adjust their step-size based on observed gradients and hence can decrease the stepe-size when encountered with a large stepe-size preventing blow-up. This work does not assume bounded gradeint assumption for these methods and show that the convergence rate does not exponenetially depend on the smoothness parameter making it more stable than untuned SGD. \n\nThe core strength of the paper lies in removing assumption on bounded gradients revealing true dependency of these algorithms on smoothness parameter $l$ which highlights the advantage of adaptive methods over untuned SGD.\n\nWEAKNESSES:\nDisclaimer: I am unfamilair with recent developments in this specific diretion. But still i really enjoyed reading this work and I feel it improves over existence convergence results. I have a few questions that i encountered but I don't list them as major weaknesses:\n\n1) the constructed function $f(x) $ in Figure-2 does not look so. Is there an extension of the function outside the segment-1. If so, the author should mention that. Although the equation for sregment-4 and segment-1 are still provided, segment-2 and 3 are still missing. \n\n2) From theorem-1, the main reason untuned SGD may blow up is that some initial $\\eta \\geq \\frac{1}{l}$. But in practice SGD also converges with a small constant learning rate which does fall into this regime and in this regime it does not blow up. However, it is mostly observed that initial large learning rate SGD performs better in terms of generalization [1] . If practitioneers use a small enough (but contant) leanring rate $\\eta \\leq \\frac{1}{l}$, then the whole issue of gradient blow-up can be avoided. Even in such practical cases, no prior knowledge of $l$ is required to select step-size $\\eta$. So is it true that achieving generalization is the bottleneck in chosing step-size as large as possible ?\n\n\n\n[1] Li, Yuanzhi, Colin Wei, and Tengyu Ma. \"Towards explaining the regularization effect of initial large learning rate in training neural networks.\" Advances in Neural Information Processing Systems 32 (2019).\n\nQUESTIONS:\nSee the points above for the questions.\n\nLIMITATIONS:\ni believe the whole problem of gradient blow-up can be avoided by just using small enough practical $\\eta$ according to Thoerem-1. But that would hurt generalization in overparameterized networks. Hence, I believe comparison of constant step SGD $\\eta$ and other adaptive gradient methods should also be done in terms of generalization and not only convergence. This would give us the full picture.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyses the rate of SGD and other adaptive SGD methods in reducing $\\\\mathbb{E}\\\\|\\\\nabla f(x_t)\\\\|$, where $f$ is non-convex $L$-smooth, and shows that SGD has an exponential dependence on $L$ when the stepsize is not properly tuned, while other adaptive methods do not incur this exponential dependence.\n\nSTRENGTHS:\nThe analysis presented in this work is fresh and interesting. The authors make a compelling case that removing the bounded gradient assumption is perhaps essential in revealing a crucial advantage of adaptive SGD methods over standard SGD.\n\nAs I explain in the weakness section, I think the qualitative idea behind the described phenomenon is known. However, the quantitative results are, as far as I know, new and, in my opinion, very interesting. I think the qualitative understanding gained by the quantitative analysis is valuable, so I vote for the paper to be accepted.\n\nWEAKNESSES:\nAt a high level, one can argue that the result was \"known\" in the following sense.\n\nWhen a LR schedule of $\\\\eta_k=\\\\eta/\\\\sqrt{k}$ is used, $\\\\eta_k$ will be large for small $k$. It is known that SGD exhibits divergent behavior when the stepsize exceeds $2/L$, so SGD will initially diverge. The worst-case amount of divergence should be exponential in $L$. Once the $\\\\eta_k$ diminishes to a level below the divergent threshold, then the algorithm needs to recover from the initial exponential divergence, hence the rate.\n\nOn the other hand, adaptive methods should not exhibit such divergent behavior (at least not to an exponential extent). Therefore, there is no initial exponential divergence, so there is no catch-up to play at the later stage of the algorithm.\n\nI think it would be worthwhile for the authors to discuss the view that the exponential constant corresponds to the amount of catch-up SGD needs to make.\n\nQUESTIONS:\nIs the line of reasoning the authors lay out specific to non-convex SGD? It seems to me that a similar type of exponential dependence could be shown for the smooth-convex setup.\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article under review presents results to show that untuned SGD may be less adapted than normalized versions of it when solving smooth non-convex problems.\nIn order to prove this, the authors present several results, from upper to lower bounds on smooth non-convex problems to find a critical point, without the knowledge of the smoothness parameter. They show that while SGD may suffer from the non-adaptivity of the step-size, its normalized version get rid of this problem.\n\nSTRENGTHS:\n- The paper's description of the problem it wants to tackle is good, and the questions addressed are well introduced.\n- While I find it a weakness too (see below), it is remarkable that the authors present both lower bounds and upper bounds on a lot of different settings.  \n- The figures summarize well the main idea of the negative result on untuned SGD and the principle of the lower bound \n- The table helps to navigate in this hairy paper\n\nWEAKNESSES:\n*Main comments*\n\nWhile the problem asked in the introduction on the difference between SGD and Adam (and its many variants) is an important problem where almost nothing is known, I find the answer of this article non-really convincing:\n- The main phenomenon pinpointed by the authors is the presence of the constant $e^{\\eta^2 \\ell^2}$ in the bound to find a critical point: this factor is due to the fact that, initially, the step size is too big compared to the local curvature of the function while after some time, the step-size being a decreasing function $\\eta_t = \\eta/\\sqrt{t}$, the step-size becomes well-conditioned. While some similar phenomenon may take place for some learning problems, I am not sure that such an analysis is the crux of the problem for the comparison between Adam and SGD. Maybe some experiments on non-toyish problem might help convince the reader (or at least myself): do we really see this $e^{\\eta^2 \\ell^2}$ popping out and eventually really slows down the convergence?\n- The article claims to address, for practical purposes, and improve the theory from *bounded stochastic gradients* to *bounded variance* of the stochastic gradient: surely, theoretically it is a nice contribution, but it does not really serve to give an answer to the claimed question. Furthermore, even in this case, I think that the set-up is still not valid for the simplest least square case… so it does not seem to me as an incredible update.\n- Finally, as stated in the strengths paragraph, the article addresses a lot of different setups, with different algorithms, sometimes stochastic, sometimes deterministic and it is very difficult to understand the true contribution of the article if the authors do not pinpoint them. Sometimes the reader, or at least I, was completely lost in what was known, both it terms of technique and/or result.  \n\n*Minor comments*\n\n- Theorem 1: $\\Delta$ is not defined. I am surprised that there is no problem when $\\eta$ is too big (no upper bound on $\\eta$, and it does not diverge!), but this may be an artifact on the bounded variance stochastic gradient assumption.\n- Theorem 2: Good lower bound. I like it together with the illustration, as we understand well the phenomenon. However, why not proving it for SGD? This is only a technical limitation I guess.\n\n*Final precaution.* \n\nOverall, I have to say that I do not come from the community that analyses the general convergence of SGD for non-convex problems and its many normalized variants. Hence, it is hard to see what is the reel contribution of the authors.\n\nQUESTIONS:\nAlready said in the paragraph above\n\nLIMITATIONS:\nAlready said in the paragraph above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}