Title: On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
Venue: NeurIPS 2024
Accepted: yes
Keywords: Adam

optimal rate

high probability convergence

relaxed assumptions
Abstract: In this paper, we study Adam in non-convex smooth scenarios with potential unbounded gradients and affine variance noise. We consider a general noise model which governs affine variance noise, bounded noise, and sub-Gaussian noise. We show that Adam with a specific hyper-parameter setup can find a stationary point with a $\mathcal{O}(\text{poly}(\log T)/\sqrt{T})$ rate in high probability under this general noise model where $T$ denotes total number iterations, matching the lower rate of stochastic first-order algorithms up to logarithm factors. We also provide a probabilistic convergence result for Adam under a generalized smooth condition which allows unbounded smoothness parameters and has been illustrated empirically to capture the smooth property of many practical objective functions more accurately.

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions

Abstract

In this paper, we study Adam in non-convex smooth scenarios with potential unbounded gradients and affine variance noise. We consider a general noise model which governs affine variance noise, bounded noise, and sub-Gaussian noise. We show that Adam with a specific hyper-parameter setup can find a stationary point with a ˜ O (1 / √ T ) rate in high probability under this general noise model where T denotes total number iterations, matching the lower rate of stochastic first-order algorithms up to logarithm factors. Moreover, we show that under the same setup, Adam without corrective terms and RMSProp can find a stationary point with a ˜ O (1 /T + σ 0 / √ T ) rate which is adaptive to the noise level σ 0 . We also provide a probabilistic convergence result for Adam under a generalized smooth condition which allows unbounded smoothness parameters and has been illustrated empirically to capture the smooth property of many practical objective functions more accurately.

1 Introduction

Since its introduction by [34], the Stochastic Gradient Descent (SGD): x t +1 = x t -η t g t has achieved significant success in solving the unconstrained stochastic optimization problems:

where ξ is a random variable, g t is the stochastic gradients and η t is the step-size. From then on, numerous literature focused on the convergence behavior of SGD in various scenarios. Several studies focused on the non-convex smooth scenario where the stochastic gradient g ( x ) is unbiased with affine variance noise, i.e., for some constants σ 0 , σ 1 ≥ 0 and all x ∈ R d ,

Under the noise assumption (2), [3] provided an almost-sure convergence bound for SGD. [4] proved that SGD could reach a stationary point with a O (1 / √ T ) rate when step-sizes are tuned by problemparameters such as the smooth parameter L . The theoretical result also revealed that the analysis of SGD under (2) is not essentially different from the bounded noise case [17].

In the popular field of deep learning, a range of variants based on SGD, known as adaptive gradient methods have emerged. These methods employ the past gradients to adaptively tune their stepsizes and are preferred to SGD for minimizing various objective functions due to their efficiency. Amongthese methods, Adam [23] has been one of the most effective methods empirically. Generally

Junhong Lin ∗

1 [35] requires T ( β 1 , β 2 ) = O ( β 1 β n 2 ( 1 -β 1 1 -β n 1 +1 )) → 0 , which seems could only achieve when β 1 = 0 .

2 Though not explicitly stated, the results in (Zhang et al., 2022) could imply convergence to the stationary point when with some calculations.

3 'FCT" refers to 'full corrective terms". The 'Conv. rate" column presents the convergence rate omitting logarithm factors.

speaking, Adam absorbs some key ideas from previous adaptive methods such as AdaGrad [12, 37] and RMSProp [38] while adding more unique structures. It combines the exponential moving average mechanism from RMSProp and meanwhile adds the heavy-ball style momentum [30] and two unique corrective terms. This unique structure leads to a huge success for Adam in practical applications but at the same time brings more challenges to the theoretical analysis.

Considering the significance of affine variance noise and Adam in both theoretical and empirical fields, it's natural to question whether Adam can find a stationary point at a rate comparable to SGD under the same smooth condition and (2). Earlier researches [14, 41, 2] have shown that AdaGradNorm, a scalar version of AdaGrad, can find a stationary point at the same rate as SGD, not tuning step-sizes based on problem-parameters. Moreover, they addressed an essential challenge brought by the correlation of adaptive step-sizes and noise from (2) which does not appear in SGD's cases. However, since AdaGrad-Norm applies a cumulative step-sizes mechanism which is rather different from the exponential moving average step-sizes in Adam, the analysis for AdaGrad-Norm could not be trivially extended to Adam. Furthermore, the coordinate-wise step-size architecture of Adam, rather than the unified step-size for all coordinates in AdaGrad-Norm, brings more challenge when considering (2). In affine variance noise landscape, existing literature could only ensure the Adam's convergence with random-reshuffling scheme under certain parameter restrictions [53, 42], or deduce the convergence at the expense of requiring bounded gradient assumption and using problemparameters to tune the step-sizes [18], both of which ignored the corrective terms. Some other works proved convergence to a stationary point by altering the original Adam algorithm such as removing certain corrective terms and modifying (2) to a stronger coordinate-wise variant [40, 20].

To the best of our knowledge, existing research has not yet fully confirmed the convergence of Adam under affine variance noise. To address this gap, we conduct an in-depth analysis and prove that Adam with the right parameter can find a stationary point in high probability. We assume a milder noise model (detailed in Assumption (A3)), covering almost surely affine variance noise, the bounded noise, and sub-Gaussian noise. We show that the convergence rate can reach at O ( poly (log T ) / √ T ) matching the lower rate in [1] up to logarithm factors. Our proof employs the descent lemma over the introduced proxy iterative sequence and adopts techniques related to the new proxy step-sizes and error decomposition. Based on this, we are able to handle the correlation between stochastic gradients and adaptive step-sizes and transform the first-order term from the descent lemma into the gradient norm.

Finally, we apply the analysis to the ( L 0 , L q ) -smooth condition [52]. Several researchers have found empirical evidence of objective functions satisfying ( L 0 , L q ) -smoothness but out of L -smoothness range, especially in large-scale language models [50, 39, 11, 8]. Theoretical analysis of adaptive methods under this relaxed condition is more complicated and needs further nontrivial proof techniques. Also, prior knowledge of problem-parameters to tune step-sizes is needed, as indicated by the counter-examples from [41] for the AdaGrad. Existing works [13, 41] obtained a convergence bound for AdaGrad-Norm with (2), and [25] considered Adam with sub-Gaussian noise. In this

Algorithm 1 Adam

paper, we provide a probabilistic convergence result for Adam with the affine variance noise and the generalized smoothness condition.

We also refer readers to see the main contributions of our works and comparisons with the existing works in Table 1.

Notations Weuse [ T ] to denote the set { 1 , 2 , · · · , T } for any positive integer T , ∥·∥ , ∥·∥ 1 and ∥·∥ ∞ to denote l 2 -norm, l 1 -norm and l ∞ -norm respectively. a ∼ O ( b ) and a ≤ O ( b ) denote a = C 1 b and a ≤ C 2 b for some positive universal constants C 1 , C 2 , and a ≤ ˜ O ( b ) denotes a ≤ O ( b ) poly (log b ) . a ≲ b denotes a ≤ O ( b ) . For any vector x ∈ R d , x 2 and √ x denote coordinate-wise square and square root respectively. x i denotes the i -th coordinate of x . For any two vectors x , y ∈ R d , we use x ⊙ y and x / y to denote the coordinate-wise product and quotient respectively. 0 d and 1 d represent zero and one d -dimensional vectors respectively.

2 Problem set up and algorithm

Weconsider unconstrained stochastic optimization (1) over R d with l 2 -norm. The objective function f : R d → R is differentiable. Given x ∈ R d , we assume a gradient oracle that returns a random vector g ( x , z ) ∈ R d dependent by the random sample z . The true gradient of f at x is denoted by ∇ f ( x ) ∈ R d .

Assumptions. We make the following assumptions throughout the paper.

( A1 ) Bounded below: There exists f ∗ > -∞ such that f ( x ) ≥ f ∗ , ∀ x ∈ R d ;

( A2 ) Unbiased estimator: The gradient oracle provides an unbiased estimator of ∇ f ( x ) , i.e., E z [ g ( x , z )] = ∇ f ( x ) , ∀ x ∈ R d ;

( A3 ) Generalized affine variance noise: The gradient oracle satisfies that there are some constants σ 0 , σ 1 > 0 , p ∈ [0 , 4) , E z [ exp ( ∥ g ( x , z ) -∇ f ( x ) ∥ 2 σ 2 0 + σ 2 1 ∥∇ f ( x ) ∥ p )] ≤ exp(1) , ∀ x ∈ R d .

The first two assumptions are standard in the stochastic optimization. The third assumption provides a mild noise model that covers the almost surely bounded noise and sub-Gaussian noise. Moreover, it's more general than almost surely affine variance noise as follows

and enlarge the range of p to [0 , 4) . Assumption ( A3 ) with p = 2 and (3) are also utilized in [2] to establish high probability results for AdaGrad-Norm. It represents a stronger condition than the expected version of (2) that is commonly employed for deriving the expected convergence of algorithms. However, almost surely assumption enables the derivation of stronger high-probability convergence guarantees for algorithms, while still ensuring expected convergence.

The affine noise variance assumption is important for machine learning applications with feature noise (including missing features) [15, 22], in robust linear regression [46], and generally whenever the model parameters are multiplicatively perturbed by noise (e.g., a multilayer network, where noise from a previous layer multiplies the parameters in subsequent layers). We refer interested readers to see e.g., [3, 46, 4, 14, 41, 2] for more discussions about the affine variance noise.

Adam. For the stochastic optimization problem, we study Algorithm 1, which is an equivalent form of Adam [23] with the two corrective terms for m s and v s included into η s for notation simplicity.

The iterative relationship in Algorithm 1 can be also written as for any s ∈ [ T ] ,

where we let x 0 = x 1 and η 0 = η . (4) plays a key role in the convergence analysis, showing that Adam incorporates a heavy-ball style momentum and dynamically adjusts its momentum through β 1 and β 2 , along with adaptive step-sizes. This inspires us to learn from some classical analysis methods for algorithms with momentum and provides some new estimations to fit in with the adaptive property.

3 Convergence of Adam with smooth objective functions

In this section, we assume that the objective function f is L -smooth satisfying that for any x , y ∈ R d ,

We then show that Adam has the following high probability results.

Theorem 3.1. Let T ≥ 1 and { x s } s ∈ [ T ] be the sequence generated by Algorithm 1. If Assumptions (A1)-(A3) hold, and the hyper-parameters satisfy that

for some constants c, C 0 > 0 and ϵ 0 > 0 , then for any given δ ∈ (0 , 1 / 2) , it holds that with probability at least 1 -2 δ ,

where G 2 is defined by the following order with respect to T, ϵ 0 , δ : 2

Theorem 3.1 provides the nearly optimal convergence rate O ( poly (log T ) / √ T ) to find a stationary point when setting the parameter probably: β 2 = 1 - O (1 /T ) . It's worth noting that the setting requires β 2 to be closed enough to 1 when T is sufficiently large, which roughly aligns with the typical setting in [23, 58, 10, 40]. For a more detailed comparison of our results to existing works, including assumptions, convergence rate, and dependency, we refer readers to Table 1.

Adam without corrective terms and RMSProp. We also consider a simplified version of Adam that drops two corrective terms as shown in Algorithm 2 in Appendix. Algorithm 2 with β 1 = 0 can be directly reduced to RMSProp [38]. More importantly, the following result shows that the convergence rate of Algorithm 2 is adaptive to the noise level.

Theorem 3.2 (informal version of Theorem C.2) . Let T ≥ 1 and { x s } s ∈ [ T ] be generated by Algorithm 2 covering RMSProp. Following the assumptions and setup in Theorem 3.1, with probability at least 1 -2 δ , ∑ T s =1 ∥∇ f ( x s ) ∥ 2 /T ≲ ˜ O (1 /T + σ 0 / √ T ) .

4 Convergence of Adam with generalized smooth objective functions

In this section, we study the convergence behavior of Adam in the generalized smooth case. We first provide some necessary introduction to the generalized smooth condition.

4.1 Generalized smoothness

We consider the following ( L 0 , L q ) -smoothness condition: there exist constants q ∈ [0 , 2) and L 0 , L q > 0 , satisfying that for any x , y ∈ R d with ∥ x -y ∥ ≤ 1 /L q ,

The generalized smooth condition was originally put forward by [52] for any twice differentiable function f satisfying that

It has been proved that a lot of objective functions in experimental areas satisfy (9) but out of L -smoothness range, especially in training large language models, see e.g., Figure 1 in [52] and [8].

To better understand the theoretical significance of the generalized smoothness, [50] provided an alternative form in (8) with q = 1 , only requiring f to be differentiable. They showed that (8) is sufficient to elucidate the convergence of gradient-clipping algorithms.

There are three key reasons for opting for (8). Firstly, considering our access is limited to first-order stochastic gradients, it's logical to only assume that f is differentiable. Second, as pointed out by Lemma A.2 in [50] and Proposition 1 in [13], (8) and (9) are equivalent up to constant factors when f is twice differentiable considering q = 1 . Thus, (8) covers a broader range of functions than (9). Finally, it's easy to verify that (8) is strictly weaker than L -smoothness. A concrete example is that the simple function f ( x ) = x 4 , x ∈ R does not satisfy any global L -smoothness but (8). Moreover, the expanded range of q to [0 , 2) is necessary as all univariate rational functions P ( x ) /Q ( x ) , where P, Q are polynomials and double exponential functions a ( b x ) with a, b > 1 are ( L 0 , L q ) -smooth with 1 < q < 2 (see [25, Proposition 3.4]). We refer interested readers to see [52, 50, 13, 25] for more discussions of concrete examples of generalized smoothness.

4.2 Convergence result

We then provide the high probability convergence result of Adam with ( L 0 , L q ) -smoothness condition as follows.

Theorem 4.1. Let T ≥ 1 and δ ∈ (0 , 1 / 2) . Suppose that { x s } s ∈ [ T ] is a sequence generated by Algorithm 1, f is ( L 0 , L q ) -smooth satisfying (8) , Assumptions (A1)-(A3) hold, and the parameters satisfy

where c, ϵ 0 , E 0 , ˜ C 0 > 0 are constants, ˆ H is controlled by O ( log ( T ϵ 0 δ )) 3 , and H, H , L are defined as

Then it holds that with probability at least 1 -2 δ ,

Note that in the above theorem, the order of log T in ˆ H and the final convergence bound is better than the one in Theorem 3.1 under the same noise assumption. This better dependency comes from the expense of using problem parameters to tune step-size ˜ C 0 . Since ˆ H is logarithm order

of T , H, H , L are both polynomial logarithm order of T and the final convergence rate in (12) is O ( poly (log T ) / √ T ) order. Note that ˜ C 0 ≤ O (1 / poly (log T )) from (10) when T ≫ d . Hence, when T is large enough, a possible optimal setting is that η = c 1 / ( √ T poly (log T )) for some constant c 1 > 0 , which roughly matches the typical setting as mentioned before.

Similarly, we also obtain a convergence bound that is adaptive to the noise level for Adam without corrective terms and RMSProp under generalized smoothness.

Theorem 4.2 (informal version of Theorem E.1) . Let T ≥ 1 and { x s } s ∈ [ T ] be generated by Algorithm 2 covering RMSProp. Following the assumptions and the same order of hyper-parameters in Theorem 4.1, with probability at least 1 -2 δ , ∑ T s =1 ∥∇ f ( x s ) ∥ 2 /T ≲ ˜ O (1 /T + σ 0 / √ T ) .

5 Related works

There is a large amount of works on stochastic approximations (or online learning algorithms) and adaptive variants, e.g., [5, 36, 48, 29, 12, 4, 6, 27, 55] and the references therein. In this section, we will discuss the most related works and make a comparison with our main results.

5.1 Convergence with affine variance noise and its variants

We mainly list previous literature considering (2) over non-convex smooth scenario. [3] provided an asymptotic convergence result for SGD with (2). In terms of non-asymptotic results, [4] proved the convergence of SGD, illustrating that the analysis was non-essentially different from the bounded noise case from [17].

In the adaptive methods field, [14] studied convergence of AdaGrad-Norm with (2), pointing out that the analysis is more challenging than the bounded noise and bounded gradient case in [44]. They provided a convergence rate of ˜ O (1 / √ T ) without knowledge of problem parameters, and further improved the bound adapting to the noise level: when σ 1 ∼ O (1 / √ T ) ,

(13) matches exactly with SGD's case [4], showing a fast rate of ˜ O (1 /T ) when σ 0 is sufficiently low. Later, [41] proposed a deep analysis framework obtaining (13) with a tighter dependency to T and not requiring any restriction over σ 1 . They further obtained the same rate for AdaGrad under a stronger coordinate-wise version of (2): for all i ∈ [ d ] ,

[2] obtained a probabilistic convergence rate for AdaGrad-Norm with (3) using a novel induction argument to estimate the function value gap without any requirement over σ 1 as well.

In the analysis of Adam, a line of works [35, 53, 42] considered Adam for finite-sum objective functions under different regimes while possibly incorporating natural random shuffling technique. They could ensure that this variant converged to a bounded region where

under the affine growth condition which is equivalent to (2). Though not explicitly concluded, when setting β 2 = 1 -O (1 /T ) , [53]'s work can also ensure a convergence rate of order ˜ O (1 / √ T ) under certain settings. Besides, both [21] and [18] provided convergence bounds allowing for large heavyball momentum parameter that aligns more closely with practical settings. However, they relied on the assumption for step-sizes where C l ≤ ∥ 1 √ v t + ϵ t ∥ ∞ ≤ C u , ∀ t ∈ [ T ] . [40] and [20] used distinct methods to derive convergence bounds in expectation and high probability respectively, without relying on bounded gradients. Both studies achieved a convergence rate of the form in (13) for Adam ignoring the corrective terms. [20] further achieved a ˜ O (1 / √ T ) rate for Adam. However, the two works only studied coordinate-wise affine variance noise.

In this paper, we derive a stronger high probability convergence rate for Adam with original corrective terms, relying on an almost surely noise assumption. The noise model is general enough to

cover bounded noise, sub-Gaussian noise, and (coordinate-wise) affine variance noise. Although we consider a stronger almost surely assumption, our probabilistic convergence result is also stronger than the expected convergence.

5.2 Convergence with generalized smoothness

The generalized smooth condition was first proposed for twice differentiable functions by [52] (see (9)) to explain the acceleration mechanism of gradient-clipping. This assumption was extensively confirmed in experiments of large-scale language models [52]. Later, [50] further relaxed it to a more general form in (8) allowing for first-order differentiable functions. Subsequently, a series of works [31, 54, 33] studied different algorithms' convergence under this condition.

In the field of adaptive methods, [13] provided a convergence bound for AdaGrad-Norm assuming (2) and (8) with q = 1 , albeit requiring σ 1 < 1 . Based on the same conditions, [41] improved the convergence rate to the form in (13) without restriction on σ 1 . [42] explored how Adam without corrective terms behaves under generalized smoothness with q = 1 and (2). However, they could only assert convergence to a bounded region as shown in (15). [8] showed that an Adam-type algorithm converges to a stationary point under a stronger coordinate-wise generalized smooth condition. Recently, [25] provided a novel framework to derive high probability convergence bound for Adam under the generalized smooth and sub-Gaussian noise case.

In this paper, we consider a more general noise setup and investigate Adam's convergence under the generalized smooth landscape. We prove that Adam is powerful enough to find a stationary point with properly tuned step-sizes even under these relaxed assumptions. Moreover, the convergence rate is not harmed by the relaxation of noise and smoothness, matching the optimal O (1 / √ T ) rate up to logarithm factors.

5.3 Convergence of Adam

Adam was first proposed by [23] with empirical studies and theoretical results on online convex learning. The original proof of convergence in [23] was later shown by [32] to contain gaps. [32] and the subsequent work [43] also showed that for a range of momentum parameters chosen independently with the problem instance, Adam does not necessarily converge even for convex objectives. Many works have focused on its convergence behavior in non-convex smooth fields. A series of works studied Adam ignoring corrective terms, all requiring a uniform bound for gradients' norm. Among these works, [49] demonstrated that Adam can converge within a specific region if step-sizes and decay parameters are determined properly by the smooth parameter. [9] proposed a convergence result to a stationary point and required all stochastic gradients must keep the same sign. To circumvent this requirement, [58] introduced a convergence bound only requiring hyper-parameters to satisfy specific conditions. [10] conducted a simple proof and further improved the dependency on the heavy-ball momentum parameter. Recently, [56] introduced Nesterov-like acceleration into Adam and AdamW [28] indicating their superiority in convergence over the non-accelerated versions. For Adam-related works under (2) or generalized smoothness, we refer readers to Sections 5.1 and 5.2.

Wealso want to highlight that a series of works [24, 45, 51] investigated the geometry of Adam from an l ∞ -norm perspective. [24] and [45] studied the geometry of Adam by regarding it as a variant of SignSGD and [51] showed that full-batch Adam converges towards a linear classifier that achieves the maximum l ∞ -margin when the training data are linearly separable.

6 Proof sketch under the smooth case

In this section, we provide a proof sketch of Theorem 3.1 with some insights and proof novelty. Our proof borrows some ideas from [44, 10, 14, 2, 40, 20]. The detailed proof is in Appendix B.

Preliminary. To start with, we let the stochastic gradient g s = ( g s,i ) i , the true gradient ∇ f ( x s ) = ¯ g s = (¯ g s,i ) i and ξ s = ( ξ s,i ) i = g s -¯ g s . We also let ϵ s = ϵ √ 1 -β s 2 and thus ϵ s = ϵ s 1 d . For any positive integer T and δ ∈ (0 , 1) , we define M T = √ log (e T/δ ) . We denote the adaptive part of the step-size as

We define two auxiliary sequences { p s } s ≥ 1 and { y s } s ≥ 1 ,

We follow from [16, 47] which was used to prove the convergence of SGD with momentum and later applied to handle many variants of momentum-based algorithms. Recalling the iteration of x s in (4), we reveal that y s satisfies

In addition, given T ≥ 1 , we define, ∀ s ∈ [ T ] ,

where G is as in Theorem 3.1. Both G s and G T ( s ) will serve as upper bounds for gradients' norm before time s . We will verify their importance in the later argument.

Starting from the descent lemma. We fix the horizon T and start from the standard descent lemma of L -smoothness. Then, for any given t ∈ [ T ] , combining with (18) and summing over s ∈ [ t ] ,

where we let ∆ s = η s b s -1 η s -1 b s -1 d and use y 1 = x 1 from (17). In what follows, we will estimate A , B , and C respectively.

Probabilistic estimations. To proceed with the analysis, we next introduce two probabilistic estimations showing that the norm of the noises and a related summation of martingale difference sequence could be well controlled with high probability. We show that with probability at least 1 -2 δ , the following two inequalities hold simultaneously for all t ∈ [ T ] :

where D 1 is a constant defined in Lemma B.7 and a s will be introduced later. In what follows, we always assume that (21) and (22) hold for all t ∈ [ T ] and carry out our subsequent analysis with some deterministic estimations.

Estimating A. We first decompose A as

Due to the correlation of the stochastic gradient g s and the step-size η s / b s , the estimating of A.1 is challenging, as also noted in the analysis for other adaptive gradient methods, e.g., [44, 10, 14, 2, 40, 20]. To break this correlation, the so-called proxy step-size technique is introduced and variants of proxy step-size have been introduced in the related literature. However, to our best knowledge, none

of these proxy step-sizes could be used in our analysis for Adam considering potential unbounded gradients under the noise model in Assumption (A3). In this paper, we construct a proxy step-size η s / a s , with a s relying on G T ( s ) in (19), defined as for any s ∈ [ T ] ,

With the so-called proxy step-size technique over η s / a s and ξ s = g s -¯ g s , we decompose A.1 as

In the above decomposition, the first term serves as a descent term. A.1.1 is now a summation of a martingale difference sequence which could be estimated by (22). A.1.2 is regarded as an error term when introducing a s . However, due to the delicate construction of a s , the definition of local gradients' bound G T ( t ) , and using some basic inequalities, we show that

The first RHS term can be eliminated with the descent term while the summation of the last term can be bounded by

due to the step-size's adaptivity, the iterative relationship of the algorithm, the smoothness of the objective function, as well as (21). Here, ˆ m s = m s 1 -β s 1 .

Estimating B and C. The key to estimate B is to decompose B as

To estimate B.1 , we use the updated rule and further decompose ∆ s ⊙ ( x s -x s -1 ) as the following terms

and upper bound the three related inner products. Using some basic inequalities, the smoothness, (24), and some delicate computations, one can estimate the three related inner products, B.2 and C , and thus get that

where b 1 and b 2 are positive constants determined by β 1 , β 2 , d, L, η .

Bounding gradients through induction. The last challenge comes from the potential unbounded gradients' norm. Plugging the above estimations into (20), we obtain that

where c 1 , c 2 , c 3 are constants determined by β 1 , β 2 , d, L, η . Then, we will first show that G 1 ≤ G and suppose that for some t ∈ [ T ] ,

It's then clear to reveal from (25) and the induction assumption that f ( y t +1 ) is restricted by the first-order of G T . Moreover, f ( y t +1 ) -f ∗ could be served as the upper bound of ∥ ¯ g t +1 ∥ 2 since

where we use a standard result ∥∇ f ( x ) ∥ 2 ≤ 2 L ( f ( x ) -f ∗ ) in smooth-based optimization. We also use the smoothness to control ∥ ¯ g t +1 -∇ f ( y t +1 ) ∥ 2 and combine with (26) and (27) to derive that

where ˜ d 1 , ˜ d 2 are constants that are also determined by hyper-parameters and restricted by O (log T -T log β 2 ) with respect to T . Then, using Young's inequality,

Thus, combining with a proper construction G 2 (detailed in (53)), we could prove that

which leads to ∥ ¯ g t +1 ∥ 2 ≤ G 2 . Combining with the induction argument, we deduce that ∥ ¯ g t ∥ 2 ≤ G 2 , ∀ t ∈ [ T +1] .

Final estimation. Following the induction step for upper bounding the gradients' norm, we also prove the following result in high probability:

We could rely on G T to prove that ∥ a s ∥ ∞ ≤ G T √ 1 -β s 2 + ϵ s , ∀ s ∈ [ T ] , and then combine with η s in Algorithm 1 to further deduce the desired guarantee for ∑ T s =1 ∥ ¯ g s ∥ 2 /T .

7 Conclusion

In this paper, we investigate the convergence of the Adam optimization algorithm on non-convex smooth problems under certain relaxed conditions. We begin by considering a mild noise assumption that encompasses several noise types, particularly the almost surely affine variance noise. Under this noise condition, we demonstrate that Adam can find a stationary point at a rate of O ( poly (log T ) / √ T ) with high probability. Within our framework, we introduce a novel proxy stepsize to manage the entanglement of stochastic gradients and adaptive step-sizes, and we employ a new decomposition method to estimate the errors introduced by the proxy step-size, the momentum, and the corrective terms in Adam.

We also extend our analysis to the convergence of Adam when the objective function is generalized smooth. This relaxed assumption is empirically validated to be more realistic in practical applications. Our results indicate that, with appropriate hyper-parameter tuning, Adam can find a stationary point at the same order of convergence rate as in the smooth case.

Limitations. Our study has several limitations that warrant further exploration. First, it would be advantageous to provide experimental results to validate the hyper-parameter settings in our results. Second, the convergence bound is not strictly tight compared to the lower bound, leaving a gap involving logarithmic factors, which may be improved in future work.

References

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199(1-2):165-214, 2023.

Amit Attia and Tomer Koren. SGD with AdaGrad stepsizes: full adaptivity with high probability to unknown parameters, unbounded gradients and affine variance. In International Conference on Machine Learning , 2023.

Dimitri P Bertsekas and John N Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization , 10(3):627-642, 2000.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018.

Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of online learning algorithms. IEEE Transactions on Information Theory , 50(9):2050-2057, 2004.

Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.06763 , 2018.

Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of Adam-type algorithms for non-convex optimization. In International Conference on Learning Representations , 2019.

Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signSGD. In Advances in Neural Information Processing Systems , 2022.

Soham De, Anirbit Mukherjee, and Enayat Ullah. Convergence guarantees for RMSProp and Adamin non-convex optimization and an empirical comparison to Nesterov acceleration. arXiv preprint arXiv:1807.06766 , 2018.

Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of Adam and Adagrad. Transactions on Machine Learning Research , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(7):2121-2159, 2011.

Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: a stopped analysis of adaptive SGD. In Conference on Learning Theory , 2023.

Matthew Faw, Isidoros Tziotis, Constantine Caramanis, Aryan Mokhtari, Sanjay Shakkottai, and Rachel Ward. The power of adaptivity in SGD: self-tuning step sizes with unbounded gradients and affine variance. In Conference on Learning Theory , 2022.

Wayne A Fuller. Measurement error models . John Wiley & Sons, 2009.

Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global convergence of the heavy-ball method for convex optimization. In European Control Conference , 2015.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. A novel convergence analysis for algorithms of the Adam family. In Annual Workshop on Optimization for Machine Learning , 2021.

Yusu Hong and Junhong Lin. Revisiting convergence of adagrad with relaxed assumptions. In The 40th Conference on Uncertainty in Artificial Intelligence .

Yusu Hong and Junhong Lin. High probability convergence of Adam under unbounded gradients and affine variance noise. arXiv preprint arXiv:2311.02000 , 2023.

Feihu Huang, Junyi Li, and Heng Huang. Super-Adam: faster and universal framework of adaptive gradients. In Advances in Neural Information Processing Systems , 2021.

Fereshte Khani and Percy Liang. Feature noise induces loss discrepancy across groups. In International Conference on Machine Learning , pages 5209-5219. PMLR, 2020.

Diederik P Kingma and Jimmy Ba. Adam: a method for stochastic optimization. In International Conference on Learning Representations , 2015.

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between SGD and Adam on Transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations , 2023.

Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. Convergence of Adam under relaxed assumptions. In Advances in Neural Information Processing Systems , 2023.

Xiaoyu Li and Francesco Orabona. A high probability analysis of adaptive SGD with momentum. In Workshop on International Conference on Machine Learning , 2020.

Mingrui Liu, Youssef Mroueh, Jerret Ross, Wei Zhang, Xiaodong Cui, Payel Das, and Tianbao Yang. Towards better understanding of adaptive gradient algorithms in generative adversarial nets. arXiv preprint arXiv:1912.11940 , 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization , 19(4):1574-1609, 2009.

Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964.

Jiang Qian, Yuren Wu, Bojin Zhuang, Shaojun Wang, and Jing Xiao. Understanding gradient clipping in incremental gradient methods. In International Conference on Artificial Intelligence and Statistics , 2021.

Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations , 2018.

Amirhossein Reisizadeh, Haochuan Li, Subhro Das, and Ali Jadbabaie. Variance-reduced clipping for non-convex optimization. arXiv preprint arXiv:2303.00883 , 2023.

Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics , pages 400-407, 1951.

Naichen Shi, Dawei Li, Mingyi Hong, and Ruoyu Sun. RMSProp converges with proper hyper-parameter. In International Conference on Learning Representations , 2020.

Steve Smale and Yuan Yao. Online learning algorithms. Foundations of Computational Mathematics , 6:145-170, 2006.

Matthew Streeter and H Brendan McMahan. Less regret via online conditioning. arXiv preprint arXiv:1002.4862 , 2010.

Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning , 2012.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

Bohan Wang, Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and lower bound of Adam's iteration complexity. In Advances in Neural Information Processing Systems , 2023.

Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of AdaGrad for non-convex objectives: simple proofs and relaxed assumptions. In Conference on Learning Theory , 2023.

Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, and Wei Chen. Provable adaptivity in Adam. arXiv preprint arXiv:2208.09900 , 2022.

Ruiqi Wang and Diego Klabjan. Divergence results and convergence of a variance reduced version of adam. arXiv preprint arXiv:2210.05607 , 2022.

Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: sharp convergence over nonconvex landscapes. Journal of Machine Learning Research , 21(1):9047-9076, 2020.

Shuo Xie and Zhiyuan Li. Implicit bias of adamw: l ∞ -norm constrained optimization. In International Conference on Machine Learning , 2024.

Huan Xu, Constantine Caramanis, and Shie Mannor. Robust regression and Lasso. In Advances in Neural Information Processing Systems , 2008.

Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic momentum methods for deep learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence , 2018.

Yiming Ying and D-X Zhou. Online regularized classification algorithms. IEEE Transactions on Information Theory , 52(11):4775-4788, 2006.

Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In Advances in Neural Information Processing Systems , 2018.

Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. In Advances in Neural Information Processing Systems , 2020.

Chenyang Zhang, Difan Zou, and Yuan Cao. The implicit bias of Adam on separable data. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning , 2024.

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: a theoretical justification for adaptivity. In International Conference on Learning Representations , 2020.

Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. In Advances in Neural Information Processing Systems , 2022.

Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences , 64:1-13, 2021.

Dongruo Zhou, Jinghui Chen, Yuan Cao, Yiqi Tang, Ziyan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. In Annual Workshop on Optimization for Machine Learning , 2020.

Pan Zhou, Xingyu Xie, and Shuicheng Yan. Win: weight-decay-integrated Nesterov acceleration for adaptive gradient algorithms. In International Conference on Learning Representations , 2023.

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. arXiv preprint arXiv:2108.11371 , 2021.

Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for convergences of Adam and RMSProp. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019.

The appendix is organized as follows. The next section presents some necessary technical lemmas, some of which have appeared in the previous literature. In Appendix B and Appendix D, the detailed proofs for Theorem 3.1 and Theorem 4.1 are presented respectively. In Appendix C and Appendix E, we provide detailed results and proofs for Theorem 3.2 and Theorem 4.2, respectively. Finally, Appendix F and Appendix G provide all the omitted proofs in previous sections.

A Complementary lemmas

We first provide some necessary technical lemmas as follows.

Lemma A.1. Suppose that { α s } s ≥ 1 is a non-negative sequence. Given β 2 ∈ (0 , 1] and ε > 0 , we define θ s = ∑ s j =1 β s -j 2 α j . Then, for any t ≥ 1 ,

Proof. See the proof of Lemma 5.2 in [10].

Lemma A.2. Suppose that { α s } s ≥ 1 is a real number sequence. Given 0 ≤ β 1 < β 2 ≤ 1 and ε > 0 , we define ζ s = ∑ s j =1 β s -j 1 α j , γ s = 1 1 -β s 1 ∑ s j =1 β s -j 1 α j and θ s = ∑ s j =1 β s -j 2 α 2 j , then

Proof. The proof for the first inequality can be found in the proof of Lemma A.2 [10]. For the second result, let ˆ M = ∑ s j =1 β s -j 1 . Then using Jensen's inequality, we have

Hence, we further have

Recalling the definition of θ s , we have ε + θ s ≥ ε + β s -j 2 θ j ≥ β s -j 2 ( ε + θ j ) . Hence, combining with 1 -β 1 ≤ 1 -β s 1 ,

Summing up both sides over s ∈ [ t ] , and noting that β 1 < β 2 ,

Finally applying Lemma A.1, we obtain the desired result.

Then, we introduce a standard concentration inequality for the martingale difference sequence that is useful for achieving the high probability bounds, see [26] for a proof.

Lemma A.3. Suppose { Z s } s ∈ [ T ] is a martingale difference sequence with respect to ζ 1 , · · · , ζ T . Assume that for each s ∈ [ T ] , σ s is a random variable only dependent by ζ 1 , · · · , ζ s -1 and satisfies that

then for any λ > 0 , and for any δ ∈ (0 , 1) , it holds that

B Proof of Theorem 3.1

The detailed proof of Theorem 3.1 corresponds to the proof sketch in Section 6.

B.1 Preliminary

To start with, we introduce the following two notations,

which include two corrective terms for m s and v s . It is easy to see that η s satisfies

We follow all the notations in Section 6, which we present here for the convenience of reading,

The following lemmas provide some estimations for the algorithm-dependent terms, which play vital roles in the proof of Theorem 3.1. The detailed proofs could be found in Appendix F.1.

Lemma B.1. Let η s , b s be given in Algorithm 1 and (16) , then

The following lemma could be found similarly in [57, Lemma A.2].

Lemma B.2. Let m s , b s be given in Algorithm 1 and (16) with 0 ≤ β 1 < β 2 < 1 , respectively. Then,

Consequently, if f is L -smooth and we set η = C 0 √ 1 -β 2 for some constant C 0 > 0 , then

The following lemma is necessary for deriving (24) in the proof sketch.

Lemma B.3. Let g s , m s be given in Algorithm 1 and ˆ m s , b s be defined in (29) and (16) . If 0 ≤ β 1 < β 2 < 1 and F i ( t ) = 1 + 1 ϵ 2 ∑ t s =1 g 2 s,i , then for any t ≥ 1 ,

The following lemmas are based on the smooth condition.

Lemma B.4. Suppose that f is L -smooth and Assumption (A1) holds, then for any x ∈ R d ,

Lemma B.5. Let x s be given in Algorithm 1 and y s be defined in (17) . If f is L -smooth, η = C 0 √ 1 -β 2 and 0 ≤ β 1 < β 2 < 1 , then

B.2 Start point and decomposition

Specifically, we fix the horizon T and start from the descent lemma of L -smoothness,

For any given t ∈ [ T ] , combining with (18) and (32) and then summing over s ∈ [ t ] , we obtain the same inequality in (20),

t

∑

s

=1

︸

∥

∥

η

∥

∥

g

s

b

,

s

(∆

∇

f

(

y

︷︷

A

β

1

-

β

1

︷︷

x

1

L

2

︸

⟨

∆

∥

))

2

∥

s

,

∥

∥

︸

C

where we use ∆ s in (20) and y 1 = x 1 . We then further make a decomposition by introducing ¯ g s into A and B

and

) +

t

∑

s

=1

s

〉

β

1

-

β

-

1

x

s

t

∑

s

=1

-

1

f

(

y

t

+1

)

f

(

+

s

-

η

·

s

g

s

b

s

-

1

s

)

+

︸

⊙

1

︸

(

x

s

⊙

(

x

s

︷︷

B

-

x

s

-

1

)

,

∇

f

(

y

s

)

⟩

︸

(33)

B.3 Probabilistic estimations

We will provide two probabilistic inequalities with the detailed proofs given in Appendix F.2. The first one establishes an upper bound for the noise norm, which we have already informally presented in (21).

Lemma B.6. Given T ≥ 1 , suppose that for any s ∈ [ T ] , ξ s = g s -¯ g s satisfies Assumption (A3). Then for any given δ ∈ (0 , 1) , it holds that with probability at least 1 -δ ,

We next provide a probabilistic upper bound as shown in (22) for a summation of the inner product, where we rely on the property of the martingale difference sequence and the proxy step-size a s in (23).

Lemma B.7. Given T ≥ 1 and δ ∈ (0 , 1) . If Assumptions (A2) and (A3) hold, then for any λ > 0 , with probability at least 1 -δ ,

As a consequence, when setting λ = (1 -β 1 ) √ 1 -β 2 / (3 η G T ) , it holds that with probability at least 1 -δ ,

where D 1 = 3 η (1 -β 1 ) √ 1 -β 2 log ( T δ ) .

B.4 Deterministic estimations

In this section, we shall assume that (36) or/and (38) hold whenever the related estimation is needed. Then we obtain the following key lemmas with the detailed proofs given in Appendix F.3.

Lemma B.8. Given T ≥ 1 . If (36) holds, then we have

Lemma B.9. Given T ≥ 1 . If b s = ( b s,i ) i and a s = ( a s,i ) i follow the definitions in (16) and (23) respectively, and (36) holds, then for all s ∈ [ T ] , i ∈ [ d ] ,

Lemma B.10. Given T ≥ 1 . Under the conditions in Lemma B.3 and Lemma B.5, if (36) holds, then the following inequality holds,

where ˆ M = M (1 -β 1 ) and M follows the definition in Lemma B.5, F ( T ) is define by

We move to estimate all the related terms in Appendix B.2. First, the estimation for A.1 relies on both the two probabilistic estimations in Appendix B.3.

Lemma B.11. Given T ≥ 1 , suppose that (36) and (38) hold. Then for all t ∈ [ T ] ,

where D 1 is given as in Lemma B.7 and D 2 = η √ 1 -β 2 1 -β 1 .

We also obtain the following lemma to estimate the adaptive momentum part B.1 .

Lemma B.12. Given T ≥ 1 , if (36) holds, then for all t ∈ [ T ] ,

where

Proposition B.13. Given T ≥ 1 . If f is L -smooth, then the following inequality holds,

where Σ max is as in Lemma B.1 and

Proof. Recalling the decomposition in Appendix B.2. We first estimate A.2 . Using the smoothness of f and (17), we have

Hence, applying Young's inequality, (44) and (30),

Recalling the updated rule in Algorithm 1 and applying (29) as well as (30),

Therefore, applying (45), (46) and β 1 ∈ [0 , 1) , and then summing over s ∈ [ t ]

Applying Cauchy-Schwarz inequality, Lemma B.1, and combining with (44), (46), Σ max ≥ 1 , and β 1 ∈ [0 , 1)

Finally, applying the basic inequality, Lemma B.1 and (46),

Recalling the decomposition in (34) and (35), then plugging (47), (48) and (49) into (33), we obtain the desired result.

B.5 Bounding gradients

Based on all the results in Appendix B.3 and Appendix B.4, we are now ready to provide a global upper bound for gradients' norm along the optimization trajectory.

Proposition B.14. Under the same conditions in Theorem 3.1, for any given δ ∈ (0 , 1 / 2) , it holds that with probability at least 1 -2 δ ,

and

where G 2 is as in Theorem 3.1 and G t , G, G T are given by (19) .

Proof. Applying Lemma B.6 and Lemma B.7, we know that (36) or (38) hold with probability at least 1 -δ . With these two inequalities, we could deduce the desired inequalities (50) and (51). Therefore, (50) and (51) hold with probability at least 1 -2 δ . We first plug (40) and (41) into the result in Proposition B.13, which leads to that for all t ∈ [ T ] ,

Next, we will introduce the induction argument based on (52). We first provide the specific definition of G 2 as follows which is a constant determined by the horizon T and other hyper-parameters but not relying on t , 4

The induction then begins by noting that G 2 1 = ∥ ¯ g 1 ∥ 2 ≤ 2 L ( f ( x 1 ) -f ∗ ) ≤ G 2 from Lemma B.4 and (53). Then we assume that for some t ∈ [ T ] ,

Using this induction assumption over (52) and subtracting with f ∗ on both sides,

Further, we combine with Lemma B.3 and Lemma B.10 to estimate the four summations defined in Lemma B.3, and then use G ≤ G T ≤ 2 M T ( σ 0 + σ 1 G p/ 2 + G ) to control the RHS of (55),

where H ( G ) = σ 1 G p/ 2 + G and ˜ D 1 , ˜ D 2 , ˜ D 3 are defined as

Applying Lemma B.5 and Lemma B.4,

Then combining (56) with (57),

Applying two Young's inequalities where ab ≤ a 2 2 + b 2 2 and ab p 2 ≤ 4 -p 4 · a 4 4 -p + p 4 · b 2 , ∀ a, b ≥ 0 ,

Recalling the definitions of D i , i ∈ [7] in (38), (40), (42), and (43). With a simple calculation relying on η = C 0 √ 1 -β 2 , ϵ ≤ ϵ 0 , Σ max ≤ 1 / √ β 2 and 0 ≤ β 1 < β 2 < 1 , we could deduce that G 2 given in (53) satisfies

Based on (58) and (59), we then deduce that ∥ ¯ g t +1 ∥ 2 ≤ G 2 . Further combining with G t +1 in (19) and the induction assumption in (54),

Hence, the induction is complete and we obtain the desired result in (50). Furthermore, as a consequence of (58), we also prove that (51) holds.

B.6 Proof of the main result

Now we are ready to prove the main convergence result.

Proof of Theorem 3.1. We set t = T in (51) to obtain that with probability at least 1 -2 δ ,

Then, in what follows, we will assume that both (50) and (60) hold. Based on these two inequalities, we could derive the final convergence bound. Since (50) and (60) hold with probability at least 1 -2 δ , the final convergence bound also holds with probability at least 1 -2 δ . Applying a s in (23) and (50), we have

Then combining with the setting η s and ϵ s in (6), we have for any s ∈ [ T ] ,

We therefore combine with (60) to obtain that with probability at least 1 -2 δ ,

Since β 2 ∈ (0 , 1) , we have

where we apply log(1 /a ) ≤ (1 -a ) /a, ∀ a ∈ (0 , 1) . With both sides multiplying T , we obtain that log ( 1 /β T 2 ) ≤ c/β 2 . Then, we further have that when β 2 = 1 -c/T ,

Since 0 ≤ β 1 < β 2 < 1 , there exists some constants ε 1 , ε 2 > 0 such that

Therefore combining (63), (64) and (53), we could verify that G 2 ∼ O (poly(log T )) with respect to T . Finally, using the convergence result in (62), we obtain the desired result.

C Convergence of Adam without corrective terms and RMSProp under smoothness

In this section, we will prove the convergence bounds in Theorem 3.2 for Adam without corrective terms and RMSProp, which are adaptive to the noise level σ 0 . We first present the algorithm.

Algorithm 2 Adam without corrective terms

Input: Horizon T , x 1 ∈ R d , β 1 , β 2 ∈ [0 , 1) , m 0 = v 0 = 0 d , η, ϵ > 0 , ϵ = ϵ 1 d for s = 1 , · · · , T do Draw a new sample z s and generate g s = g ( x s , z s ) ; m s = β 1 m s -1 +(1 -β 1 ) g s ; v s = β 2 v s -1 +(1 -β 2 ) g 2 s ; x s +1 = x s -η · m s / ( √ v s + ϵ ) ;

end for

Then, we will state the formal version for Theorem 3.2.

Theorem C.2 (Adam without corrective terms/RMSProp) . Let T ≥ 1 and { x s } s ∈ [ T ] be the sequence generated by Algorithm 2. If Assumptions (A1)-(A3) hold, and the hyper-parameters are given in (6) , then for any given δ ∈ (0 , 1 / 2) , it holds that with probability at least 1 -2 δ ,

where ˜ G 2 is defined by the following order with respect to T, ϵ 0 , δ : 5

Remark C.3 . (1). Setting β 1 = 0 , Theorem C.2 then shows that RMSProp with the hyper-parameter setup in (6) can also find a stationary point with the convergence rate given in (65).

(2). The convergence rate in Theorem C.2 is of order O 1 /T + σ 0 / T , which can be accelerated to ˜ O (1 /T ) rate when σ 0 is sufficiently low. This form matches the ones for SGD [4] and AdaGrad [2, 19], as well as the lower bound for first-order gradient method [1] on non-convex smooth optimization with affine variance noise up to logarithm factors.

C.1 Proof details

To start with, we follow the same notations of M T , G s , G T ( s ) in (31) and define

We also follow the definition of y s in (17) and obtain from Algorithm 2 that

Then, we have the following claims.

Proposition C.4. Setting η s = η and η = C 0 √ 1 -β 2 , the results in Lemma B.1, Lemma B.2, Lemma B.3, Lemma B.5, Lemma B.6, Lemma B.8 and Lemma B.10 remain unchanged.

Proof. Using η s = η and following the proof for the above lemmas, it's easy to verify that these lemmas still hold.

The result in Lemma B.9 can be improved to the following one.

Lemma C.5. Given T ≥ 1 . If ˜ b s = ( ˜ b s,i ) i and ˜ a s = (˜ a s,i ) i are defined in (67) , and (36) holds, then for all s ∈ [ T ] , i ∈ [ d ] ,

Based on these results, we are able to show that the gradient norm is bounded along the training process generated by Algorithm 2.

Proposition C.6. Under the same conditions in Theorem C.2, for any given δ ∈ (0 , 1 / 2) , it holds that with probability at least 1 -2 δ ,

and

where ˜ G 2 , ˜ G T are as in (88) .

Proof. Let ˜ ∆ s = ˜ b s -1 ˜ b s -1 d . We also start from the descent lemma and and use (68),

The following estimation is based on the probability event in Lemma B.6.

˜ ( √ )

Bounding ˜ A . We first have

Introducing ˜ a s defined in (67) into ˜ A . 1

Following the proof for Lemma B.7 with η s = η , we obtain that with probability at least 1 -δ ,

Setting λ = √ 1 -β 2 / (3 η ˜ G T ) , we obtain that with probability at least 1 -δ ,

Using Lemma C.5 and following the similar deduction for bounding (A.1.2) in Lemma B.11, we have

Plugging (75) and (76) into (73), we obtain that

The estimation for ˜ A . 2 is similar to (45) and (47). Using (44) and Young's inequality,

Thereby,

Bounding ˜ B , ˜ C . Following the same decomposition in (35) with ∆ s replaced by ˜ ∆ s ,

We define ˜ Σ := β 1 1 -β 1 〈 ˜ ∆ s ⊙ ( x s -x s -1 ) , ¯ g s 〉 . Since η s = η , we have the following decomposition

The additional Σ 3 in (159) vanishes. Then, using Lemma C.5 and Young's inequality

Similarly, using Lemma C.5 and Young's inequality

Then, summing up (81) and (82) over s ∈ [ t ] and combining with (80),

Following the similar deduction in (48) and using Lemma B.1, we have

Using Young's inequality and Lemma B.1,

Putting together. Noting that the above results are established based on probability events in Lemma B.6 and (75). Hence, plugging (77), (78), (83), (84) and (85) into (71) and using β 2 < 1 , it holds that with probability at least 1 -2 δ , for all t ∈ [ T ] ,

Then, combining with Lemma B.3 and Lemma B.10 and using η = C 0 1 -β 2 , we obtain that with probability at least 1 -2 δ , for all t ∈ [ T ] ,

where the coefficients are defined as

An induction argument. Noting that (86) holds with probability at least 1 -2 δ for all t ∈ [ T ] , then we will provide an induction to prove the desired result based on (86), which also holds with probability at least 1 -2 δ . We first provide the detailed expression of ˜ G 2 and ˜ G T as follows:

where E 1 , E 2 , E 3 are defined in (87) and M is as in Lemma B.5. The induction then begins by noting that G 2 1 = ∥ ¯ g 1 ∥ 2 ≤ 2 L ( f ( x 1 ) -f ∗ ) ≤ ˜ G 2 from Lemma B.4. Then, we assume that for some t ∈ [ T ] ,

Using this induction assumption over (86) and ˜ G T ≤ 2 M T ( σ 0 + σ 1 ˜ G p/ 2 + ˜ G ) ,

Following the same result in (57), we have

Combining with (89) and (90),

Using two Young's inequalities where ab ≤ a 2 2 + b 2 2 and ab p 2 ≤ 4 -p 4 · a 4 4 -p + p 4 · b 2 , ∀ a, b ≥ 0 ,

where the last inequality comes from the definition of ˜ G in (88). Hence, the induction is complete and we obtain the desired result in (69). Furthermore, as a consequence of (92), we also prove that (70) holds.

C.2 Proof of the main result

Now we are ready to prove the main convergence result.

Proof of Theorem C.2. In what follows, we will assume that Proposition C.6 holds. Hence, the final convergence bound also holds with probability at least 1 -2 δ . We set t = T in (70) to obtain that with probability at least 1 -2 δ ,

Using Lemma B.6, we derive that ∀ s ∈ [ T ] ,

Applying ˜ a s in (67), Proposition C.6, (94) and η = C 0 √ 1 -β 2 , ϵ = ϵ 0 √ 1 -β 2 , we have for any s ∈ [ T ] ,

We therefore combine with (93) to obtain that

which leads to

Following the result in (63) and (64), we show that when β 2 = 1 -c/T , ˜ G 2 , ˜ G T ∼ O (poly(log T )) with respect to T . Finally, using the convergence result in (96), we obtain the desired result.

D Proof of Theorem 4.1

In this section, we shall follow all the notations defined in Section 6. Further, we will add two non-decreasing sequences {L ( x ) s } s ≥ 1 and {L ( y ) s } s ≥ 1 as follows

D.1 Preliminary

We first mention that Lemma B.1, Lemma B.2, and Lemma B.3 in Appendix B.1 remain unchanged since they are independent of the smooth condition. Then the first essential challenge is that we need to properly tune η to restrict the distance between x s +1 and x s , y s +1 and y s within 1 /L q for all s ≥ 1 . The following two lemmas then ensure this point. The detailed proofs could be found in Appendix G.

Lemma D.1. Let x s , y s be defined in Algorithm 1 and (17) . If 0 ≤ β 1 < β 2 < 1 , then for any s ≥ 1 ,

As a consequence, when

then for any s ≥ 1 , all the three gaps in (98) are smaller than 1 /L q .

Lemma D.2. Let η ≤ 1 / ( L q F ) where F is as in Lemma D.1. If f is ( L 0 , L q ) -smooth, then for any s ≥ 1 ,

As a consequence, for any s ≥ 1 ,

In the generalized smooth case, Lemma B.4 does not hold. In contrast, we provide a generalized smooth version of [50, Lemma A.5], which establishes a different relationship between the gradient's norm and the function value gap. Noting that when q = 1 , Lemma D.3 reduces to [50, Lemma A.5].

Lemma D.3. Suppose that f is ( L 0 , L q ) -smooth and Assumption (A1) holds. Then for any x ∈ R d ,

D.2 Probabilistic estimations

The probabilistic inequalities in (36) and (37) remain unchanged since they do not rely on any smooth-related conditions. However, we shall rely on a different setting of λ in (37) as follows.

Lemma D.4. Given T ≥ 1 and δ ∈ (0 , 1) . Under the same conditions of Lemma B.7, if we set λ = (1 -β 1 ) √ 1 -β 2 / (3 η H ) where H is as in (11) , then with probability at least 1 -δ ,

where D 1 is given in Lemma B.7.

The bounds of the four summations in Lemma B.3 also remain unchanged. However, the upper bound for F i ( t ) should be revised by the following lemma. The detailed proof could be found in Appendix G.

Lemma D.5. Given T ≥ 1 . Under the conditions and notations of Lemma B.3, if f is ( L 0 , L q ) -smooth, η = ˜ C 0 √ 1 -β 2 , (36) and (99) hold, then the following inequalities hold,

where J ( t ) is defined as

It's worth noting that J ( t ) is still random relying on the random variable L ( x ) t .

D.3 Deterministic estimations

Note that (41) in Appendix B.4 remains unchanged since it's independent from any smooth-related condition. In terms of A.1 , the only difference is using H to replace G in (40) as we choose a different λ in (37), leading to

We also establish the following proposition which is a generalized smooth version of Proposition B.13.

Proposition D.6. Given T ≥ 1 . If f is ( L 0 , L q ) -smooth and (99) holds, then

where Σ max is as in Lemma B.1 and D 6 ( s ) , D 7 ( s ) are defined as, 6

Proof. The proof follows some same parts in proving Proposition B.14. We start from the descent lemma (101) in Lemma D.2 and sum over s ∈ [ t ] to obtain that

where A and B follow the same definitions in (33). We also follow the decompositions in (34) and (35). We could also rely on the same analysis for the smooth case in (47) but the smooth parameter is replaced by L ( x ) s . Hence, we obtain that

Similarly,

Noting that C' differs from C with L replaced by L ( y ) s . Hence, relying on a similar analysis in (49), we obtain that

Combining (107) with (108), (109) and (110), and noting that L ( x ) s ≤ L ( y ) s from (97), we thereby obtain the desired result.

D.4 Bounding gradients

Based on the unchanged parts in Appendix B.3 and Appendix B.4 and the new estimations in (105) and (106), we are now ready to provide the uniform gradients' bound in the following proposition.

Proposition D.7. Under the same conditions in Theorem 4.1, for any given δ ∈ (0 , 1 / 2) , it holds that with probability at least 1 -2 δ ,

and

where H, H , L are given in (11) and ˆ H is given in (115) .

Proof. Based on the two inequalities (36) and (102), we could deduce the final results in (111) and (112). Since (36) and (102) hold with probability at least 1 -2 δ , we thereby deduce the desired result holding with probability at least 1 -2 δ . To start with, we shall verify that (99) always holds. Recalling η in (10) and F in Lemma D.1,

Hence, we make sure that the distance requirement in (8) always holds according to Lemma D.1. Second, plugging (105) and (41) into the result in (106),

We still rely on an induction argument to deduce the result. First, we provide the detail expressions of ˆ H,H as follows which is determined by hyper-parameters β 1 , β 2 and constants E 0 , d, T, δ, M T ,

where E 0 > 0 is a constant and ˜ J ( T ) is a polynomial of T given as

and ˜ M := E 0 √ d 1 -β 1 /β 2 . The induction then begins by noting that from Lemma D.3 and H in (115),

Consequently, recalling G T ( s ) in (19), L ( x ) s , L ( y ) s in (97) and H , L in (11),

We thus apply (118) to (113),

Further recalling the setting of ˜ C 0 in (10), with a simple calculation it holds that,

Therefore, combining with (118), (120) and ˜ M t in (104), we could use the deterministic polynomial ˜ J ( t ) to further control J ( t ) in (104),

Then, we could use ˜ J ( T ) to control the four summations in Lemma B.3 which emerge in (119). In addition, we rely on η = ˜ C 0 √ 1 -β 2 and the induction assumptions of (117) and (118) to further upper bound the RHS of (119), leading to

Then, combining with (120) and the definition of ˆ H in (114), we obtain that

Then, further using Lemma D.2, Lemma D.3 and H in (115),

Wethen deduce that G t +1 = max { G t , ∥ ¯ g t +1 ∥} ≤ H . The induction is then complete and we obtain the desired result in (111). Finally, as an intermediate result of the proof, we obtain that (112) holds as well.

D.5 Proof of the main result

Proof of Theorem 4.1. The proof for the final convergence rate follows a similar idea and some same estimations in the proof of Theorem 3.1. Setting t = T in (112), it holds that with probability at least 1 -2 δ ,

Then, in what follows, we would assume that (111) and (123) always hold. Relying on the two inequalities, we thereby deduce the final convergence result. Furthermore, since (111) and (123) hold with probability at least 1 -2 δ , the final convergence result also holds with probability at least 1 -2 δ . Using (111) and following the same analysis in (61),

Combining with the parameter setting in (10),

We then combine with (123) and H in (11) to obtain that with probability at least 1 -2 δ ,

Finally, following the same deduction in (63) and (64), we could derive that ˆ H ∼ O ( log 2 ( T ϵ 0 δ )) from (114) and the desired results in Theorem 4.1.

E Convergence of Adam without corrective terms and RMSProp under generalized smoothness

In this section, we will provide the detailed result and proof for the convergence of Algorithm 2 under generalized smoothness. First, we will provide the formal version of Theorem 4.2.

Theorem E.1. Let T ≥ 1 and δ ∈ (0 , 1 / 2) . Suppose that { x s } s ∈ [ T ] is a sequence generated by Algorithm 2, f is ( L 0 , L q ) -smooth satisfying (8) , Assumptions (A1)-(A3) hold, and

where c, ϵ 0 , E 0 , ˜ C 0 > 0 are constants, ¯ H is controlled by O ( log ( T ϵ 0 δ )) 7 , and ˜ H, ˜ H , ˜ L are defined as

Then, it holds that with probability at least 1 -2 δ ,

Remark E.2 . (1). The convergence rate in Theorem C.2 is of order ˜ O ( 1 /T + σ 0 / √ T ) , which can be accelerated to ˜ O (1 /T ) rate when σ 0 is sufficiently low.

(2). Setting β 1 = 0 , Theorem E.1 then shows that RMSProp with the hyper-parameter setup in (124) can also find a stationary point with the convergence rate of ˜ O ( 1 /T + σ 0 / √ T ) order.

E.1 Proof detail

To start with, we follow all the notations defined in Appendix C and Appendix D, particularly G s , G T ( s ) in (31) and L ( x ) s , L ( y ) s in (97). Then, we have the following claims.

Proposition E.3. Setting η s = η and η = ˜ C 0 √ 1 -β 2 , the results in Lemma D.1, Lemma D.2 and Lemma D.3 remain unchanged.

Proof. Using η s = η and following the proof for the above lemmas, it's easy to verify that these lemmas still hold.

Based on these results, we are able to show that the gradient norm is bounded along the training process generated by Algorithm 2 under generalized smoothness.

Proposition E.4. Under the same conditions in Theorem C.2, for any given δ ∈ (0 , 1 / 2) , it holds that with probability at least 1 -2 δ ,

and

Proof. To start with, we shall verify that (99) always holds. Recalling η in (124) and F in Lemma D.1,

Hence, we make sure that the distance requirement in (8) always holds according to Lemma D.1. Then, using the descent lemma in (101) and (68), it leads to for any t ≥ 1 ,

The following estimation is based on the probability event in Lemma B.6.

Bounding ˜ A . Wefirst have the same decomposition as in (72). Then, setting λ = √ 1 -β 2 / (3 η ˜ H ) in (74), we obtain that with probability at least 1 -δ ,

Using Lemma C.5 and following the similar deduction for bounding ( ˜ A . 1 . 2 ) in (76), we have

Combining with (130) and (131), we obtain that

The estimation for ˜ A . 2 is similar to (108) and (78). With L replaced by L ( x ) s in (78),

Bounding ˜ B , ˜ C ′ . Following the same decomposition in (79) and (80), we first note that the estimation for ˜ B . 1 in (83) remains unchangedL

Following the similar deduction in (84) and using Lemma B.1, we can replace L with L ( x ) s to obtain that

Following the similar deduction in (110), we indeed replace L with L ( y ) s in (85)

Putting together. Noting that the above results are established based on probability events in Lemma B.6 and (130). Hence, plugging (132), (133), (134), (135) and (136) into (129) and using β 2 < 1 and L ( x ) s ≤ L ( y ) s from (97), it holds that with probability at least 1 -2 δ , for all t ∈ [ T ] ,

An induction argument. We first provide the detailed expression of ¯ H :

The induction then begins by noting that from Lemma D.3 and ˜ H in (125),

Suppose that for some t ∈ [ T ] ,

Consequently, recalling G T ( s ) in (19), L ( x ) s , L ( y ) s in (97) and H , L in (11),

Further recalling the setting of ˜ C 0 in (10), with a simple calculation it holds that,

Using (140), (141), J ( t ) , ˜ M ′ t defined in Lemma D.5 and I ( t ) , ˜ M ′ defined in (138),

Using Lemma B.3, Lemma B.10 and Lemma D.5, we have

Then, plugging (140) and (142) into (137), and using η = ˜ C 0 √ 1 -β 2 and (141),

where we use ¯ H defined in (138). Further using Lemma D.2, Lemma D.3 and ˜ H in (125),

Wethen deduce that G t +1 = max { G t , ∥ ¯ g t +1 ∥} ≤ ˜ H . The induction is then complete and we obtain the desired result in (127). Finally, as an intermediate result of the proof, we obtain that (143) holds as well.

E.2 Proof of the main result

Now we are ready to prove the main convergence result.

Proof of Theorem C.2. In what follows, we will assume that Proposition E.4 holds. Hence, the final convergence bound also holds with probability at least 1 -2 δ . We set t = T in (127),

Following the similar deduction in (95), with η = ˜ C 0 √ 1 -β 2 , ϵ = ϵ 0 √ 1 -β 2 , we have for any s ∈ [ T ] ,

We therefore combine with (144) and (145) to obtain that

which leads to

Following the result in (63) and (64), we show that when β 2 = 1 -c/T , ¯ H, ˜ H ∼ O (poly(log T )) with respect to T . Finally, using the convergence result in (147), we obtain the desired result.

F Omitted proof in Appendix B

F.1 Omitted proof in Appendix B.1

Proof of Lemma B.1. We fix arbitrary i ∈ [ d ] and have the following two cases. When η s b s -1 ,i η s -1 b s,i < 1 , we have

When η s b s -1 ,i η s -1 b s,i ≥ 1 , let r = β s -1 2 . Since 0 < 1 -β s -1 1 < 1 -β s 1 , ∀ s ≥ 2 , then we have

Since h ( r ) = r/ (1 -r ) is increasing as r grows and r takes the maximum value when s = 2 . Hence, it holds that

Then, since ϵ s -1 ≤ ϵ s , we further have

Combining with (148) and (149), we have

Combining the two cases and noting that the bound holds for any i ∈ [ d ] , we then obtain the desired result.

Proof of Lemma B.2. Denoting ˜ M = ∑ s -1 j =1 β s -1 -j 1 and applying (28) with ˆ M and α j replaced by ˜ M and g j,i respectively,

Hence, combining with the definition of b s,i in (16), we further have for any i ∈ [ d ] and s ≥ 2 ,

where the last inequality applies β 1 < β 2 . We thus prove the first result. To prove the second result, from the smoothness of f ,

Combining with (30) and η = C 0 1 -β 2 ,

Using ∥ x s -x s -1 ∥ ≤ d ∥ x s -x s -1 ∥ ∞ and (151),

Proof of Lemma B.3. Recalling the updated rule and the definition of b s,i in (16), using ϵ 2 s = ϵ 2 (1 -β s 2 ) ≥ ϵ 2 (1 -β 2 ) ,

Proof for the first summation. Using (153), for any i ∈ [ d ] ,

Applying Lemma A.1 and recalling the definition of F i ( t ) ,

Summing over i ∈ [ d ] , we obtain the first desired result.

Proof for the second summation. Following from (153),

Applying Lemma A.2 and β 2 ≤ 1 ,

Summing over i ∈ [ d ] , we obtain the second desired result.

Proof for the third summation. Following from (153),

Applying Lemma A.2, and using β 2 ≤ 1 ,

Summing over i ∈ [ d ] , we obtain the third desired result.

Proof for the fourth summation. Following the definition of ˆ m s,i from (29), and combining with (153),

Applying Lemma A.2 and using β 2 ≤ 1 ,

Summing over i ∈ [ d ] , we obtain the fourth desired result.

Proof of Lemma B.4. Let ˆ x = x -1 L ∇ f ( x ) . Then using the descent lemma of smoothness,

Re-arranging the order, and noting that f (ˆ x ) ≥ f ∗ ,

Proof of Lemma B.5. Applying the norm inequality and the smoothness of f ,

Combining with the definition of y s in (17) and (152), and using β 1 ∈ [0 , 1) , we obtain the desired result that

F.2 Omitted proof in Appendix B.3

Proof of Lemma B.6. Let us denote γ s = ∥ ξ s ∥ 2 σ 2 0 + σ 2 1 ∥ ¯ g s ∥ p , ∀ s ∈ [ T ] . Then from Assumption (A3), we first have E z s [exp ( γ s )] ≤ exp(1) . Taking full expectation,

By Markov's inequality, for any A ∈ R ,

which leads to that with probability at least 1 -δ ,

Proof of Lemma B.7. Recalling the definitions of a s in (23) and ϵ s in Algorithm 1, we have for any s ∈ [ T ] , i ∈ [ d ] ,

Then given any i ∈ [ d ] , we set

Noting that ¯ g s , a s and η s are random variables dependent by z 1 , · · · , z s -1 and ξ s is only dependent on z s . We then verify that X s is a martingale difference sequence since

Noting that ω s is a random variable only dependent by z 1 , · · · , z s -1 and applying Assumption (A3) and Cauchy-Schwarz inequality, we have

Applying Lemma A.3 and (154), we have that for any λ > 0 , with probability at least 1 -δ ,

Note that for any t ∈ [ T ] , (155) holds with probability at least 1 -δ . Then for any fixed λ > 0 , we could re-scale δ to obtain that with probability at least 1 -δ , for all t ∈ [ T ] ,

Using √ σ 2 0 + σ 2 1 ∥ ¯ g s ∥ p ≤ G T ( t ) , s ≤ t from (19), together with (30), we have that with probability at least 1 -δ ,

Finally setting λ = (1 -β 1 ) 1 -β 2 / (3 η G T ) , we then have the desired result in (38).

F.3 Omitted proof of Appendix B.4

Proof of Lemma B.8. First directly applying (36) and G s in (19), for any j ∈ [ s ] ,

Applying the basic inequality, (36) and M T ≥ 1 , for any j ∈ [ s ] ,

Finally, we would use an induction argument to prove the last result. Given any i ∈ [ d ] , noting that v 1 ,i = (1 -β 2 ) g 2 1 ,i ≤ ( G T ( s )) 2 . Suppose that for some s ′ ∈ [ s ] , v j,i ≤ ( G T ( s )) 2 , ∀ j ∈ [ s ′ ] ,

We then obtain that v j,i ≤ ( G T ( s )) 2 , ∀ j ∈ [ s ] . Noting that the above inequality holds for all i ∈ [ d ] , we therefore obtain the desired result.

Proof of Lemma B.9. Recalling the definition of b s,i in (16) and letting a s,i = √ ˜ v s,i + ϵ s in (23),

where we apply g 2 s,i ≤ ∥ g s ∥ 2 ≤ ( G T ( s )) 2 from Lemma B.8 in the first inequality since (36) holds. The second result also follows from the same analysis. We first combine with ϵ s = ϵ √ 1 -β s 2 to obtain that

where we apply √ a -√ b ≤ √ a -b, ∀ 0 ≤ b ≤ a . Applying the definition of b s -1 ,i and a s,i ,

∣

∣

∣

∣

1

-

1

1

a

s,i

∣

∣

∣

∣

∣

∣ √

b

s

-

b

|

ϵ

s

-

b

∣

+

,i

1

∣

)

-

ϵ

-

ϵ

s

T

1

-

s

v

- √

b

s

-

(1

·

-

,i

1

,i

+(

a

s,i

β

∣

∣

2

)

√

(1

(

G

˜

v

-

√

2

∣

-

1

∣

v

s

))

(

+

s,i

β

˜

β

a

s,i

s

ϵ

1

s

-

-

,i

(

a

G

1

s,i

T

(

s

) +

b

s

-

|

ϵ

)

1

,i

√

a

1

s,i

s

-

s,i

v

˜

1

1

,i

1

1

,i

a

a

s,i s,i

v

s,i

2

)(

G

+

T

v

s

√

s

2

v

(

s

s

))

-

1

1

-

-

2

,i

s

-

,i

1

+

ϵ

√

b

s

-

1

,i where the second inequality applies v s -1 ,i ≤ ( G T ( s )) 2 in Lemma B.8 and the last inequality comes from √ 1 -β 2 G T ( s ) ≤ ˜ v s,i .

Proof of Lemma B.10. Applying the basic inequality and (36), for all t ∈ [ T ] , i ∈ [ d ] ,

Combining with Lemma B.2, we have

Further applying the definition of F i ( t ) in Lemma B.3, it leads to F i ( t ) ≤ F ( t ) , ∀ i ∈ [ d ] . Finally, since F ( t ) is increasing with t , we obtain the desired result.

√

b

s

,i

-

=

-

β

2

.

Proof of Lemma B.11. First, we have the following decomposition,

Since (36) holds, we could apply Cauchy-Schwarz inequality, Lemma B.9, and G T ( s ) ≤ G T ( t ) , ∀ s ≤ t from (19) to obtain that for all t ∈ [ T ] ,

Finally, combining with (38) for estimating A.1.1 , we deduce the desired result in (40).

Proof of Lemma B.12. Let us denote Σ := β 1 1 -β 1 ⟨ ∆ s ⊙ ( x s -x s -1 ) , ¯ g s ⟩ where ∆ s is defined in (20). We have

Since (36) holds, we could apply Lemma B.9 and Young's inequality and then use (154), (30), β 1 ∈ [0 , 1) and G T ( s ) ≤ G T ( t ) ≤ G T ( t ) + ϵ, ∀ s ≤ t ,

Using the similar analysis for Σ 1 , we also have

Then we move to bound the summation of Σ 3 over s ∈ { 2 , · · · , t } since m 0 = 0 . Recalling η s in (30), we have the following decomposition,

Noting that ∥ ¯ g s ∥ ≤ G s ≤ G t , ∀ s ≤ t . Then further applying Cauchy-Schwarz inequality and Lemma B.2,

Hence, summing Σ 3 . 1 up over s ∈ [ t ] , applying β 1 ∈ (0 , 1) and noting that Σ 3 . 1 vanishes when s = 1 ,

Similarly, using ∥ ¯ g s ∥ ≤ G s ≤ G t , ∀ s ≤ t and 1 -β s -1 1 ≥ 1 -β 1 ,

Hence, summing Σ 3 . 2 up over s ∈ [ t ] and still applying β 1 ∈ [0 , 1) ,

Combining with (162), (163) and (164), we obtain an upper bound for ∑ t s =1 Σ 3 . Summing (159), (160) and (161) up over s ∈ [ t ] , and combining with the estimation for ∑ t s =1 Σ 3 , we obtain the desired inequality in (41).

G Omitted proof in Appendix D

Proof of Lemma D.1. Recalling in (152), we have already shown that

Applying the definition of y s in (17), an intermediate result in (165) and β 1 ∈ [0 , 1) , 8

Recalling the iteration of y s in (18) and then using Young's inequality

Noting that g s,i /b s,i ≤ 1 / √ 1 -β 2 from (16), we then combine with (30) to have

Applying Lemma B.1 where Σ 2 max ≤ 1 /β 2 and (165),

Summing up two estimations and using 0 ≤ β 1 < β 2 < 1 , we finally have

Combining with (165), (166) and (167), and using 0 ≤ β 1 < β 2 < 1 , we then deduce a uniform bound for all the three gaps.

Proof of Lemma D.2. Under the same conditions in Lemma D.1, we have

Then, using the generalized smoothness in (8),

We could use a similar argument to deduce the bound for ∥∇ f ( x s ) ∥ . Further, combining with L ( x ) s and L ( y ) s in (97), we could bound the generalized smooth parameters as

We could then deduce the first two inequalities in (100). Finally, (101) could be deduced by using the same argument in the proof of [50, Lemma A.3].

Proof of Lemma D.3. Given any x ∈ R d , we let

From the definition of τ , we could easily verify that ∥ ˆ x -x ∥ = τ ∥∇ f ( x ) ∥ ≤ 1 /L q . Since f is ( L 0 , L q ) -smooth, we could thereby use the descent lemma in [50, Lemma A.3] such that

Since f (ˆ x ) ≥ f ∗ , when ∥∇ f ( x ) ∥ = 0 , the desired result is trivial. Let us suppose ∥∇ f ( x ) ∥ > 0 .

Case 1 ∥∇ f ( x ) ∥ q > ∥∇ f ( x ) ∥

When ∥∇ f ( x ) ∥ q < L 0 /L q , it leads to

When ∥∇ f ( x ) ∥ q ≥ L 0 /L q , it leads to

We then deduce that

Case 2 ∥∇ f ( x ) ∥ q ≤ ∥∇ f ( x ) ∥ We could rely on the similar analysis to obtain that 9

Combining (169) and (170), we then deduce the desired result.

Proof of Lemma D.5. Recalling (152), we then obtained that when η = ˜ C 0 √ 1 -β 2 ,

Noting that when (99) holds, we have

Using L ( x ) j ≤ L ( x ) t , ∀ j ≤ t , we have

Similarly, we also have

Further combining with F i ( t ) in Lemma B.3 and J ( t ) in (104),

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides two probabilistic convergence rates for Adam with generalized affine variance noise under smoothness and generalized smooth condition, respectively, which achieves comparable results to many prior results.\n\nSTRENGTHS:\nPlease see the above Summary.\n\nWEAKNESSES:\n1. I suggest that authors should provide detailed formulas for some notations including $\\textbf{g}_t, g(\\textbf{x}), \\nabla f(\\textbf{x})$. Is $\\nabla f(\\textbf{x})$ the gradient with the form of expectation.\n\n2. In line 118, the reference [10] is cited repeatedly.\n\n3. Section 5 is used to discuss the most related works and make comparisons with the main results in this paper. However, authors only discuss the most related works without any comparison with their main results. \n\n4. As mentioned in 1., if $\\nabla f(\\textbf{x})$ is the gradient with the form of expectation, the two results (Theorems 3.1 and 4.1) in this paper are not fully high probability since $\\frac{1}{T} \\sum_{t=1}^T ||\\nabla f(x_t)||^2$ is equivalent to $\\frac{1}{T} \\sum_{t=1}^T ||E_{z_i}[\\nabla f(x_t,z_i)]||^2$ ($z_i$ is the training data defined by me) which is smaller than $\\frac{1}{T} \\sum_{t=1}^T E_{z_i}[||\\nabla f(x_t,z_i)||^2]$. In other words, the results with the form $\\frac{1}{T} \\sum_{t=1}^T E_{z_i}[||\\nabla f(x_t,z_i)||^2]$ can directly derive the corresponding results with the form $\\frac{1}{T} \\sum_{t=1}^T ||E_{z_i}[\\nabla f(x_t,z_i)]||^2$, to say nothing of the one with additional high probability. Therefore, from my perspective, high probability is not an advantage for this paper, but rather weakens $\\frac{1}{T} \\sum_{t=1}^T ||E_{z_i}[\\nabla f(x_t,z_i)]||^2$.\n\nQUESTIONS:\n1. What is the symbol $\\xi$ in Equation (1)? It should be explained in the main text.\n\n2. What are the meanings of “right parameter” mentioned in line 4 (Abstract) and “problem parameter” in line 19?\n\n3. Although, in line 139, authors denote “it’s easy to verify that (8) is strictly weaker than L-smoothness” and provide a concrete example $f(x)=x^4$, can detailed proof be given to verify this argument?\n\n4. In Table 1, authors make comparisons with some prior work. However, the forms of some results (such as [33, 40, 49]) are not in the same form as the average form $\\frac{1}{T} \\sum_{t=1}^T ||\\nabla f(x_t)||^2$ in this paper. Therefore, are these comparisons appropriate?\n\nLIMITATIONS:\nThe contribution of this paper is a bit weak. Although authors weaken some assumptions, such as noise and smoothness, the form of their results is also weakened. Therefore, I don’t ensure the weakened assumptions are their advantages.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors analyze the convergence of Adam under milder noise conditions (affline variance) and milder smoothness conditions (both $L$-smoothness and $(L_0,L_q)$-smoothness) and propose a $O(\\text{polylog}(T)/\\sqrt T)$ convergence rate.\n\nSTRENGTHS:\nThis paper analyses the convergence of Adam under milder smoothness conditions compared to the previous work. The result is relatively solid and convincing. The writing structure is also relatively clear.\n\nWEAKNESSES:\n1. As the author claimed in their paper, they did not provide numerical experiments in this paper. While this paper is a theoretical paper focusing on the convergence analysis of Adam, some simple numerical experiments aligning with the results will make it more convincing.\n2. This paper exhibits a slight lack of novelty. Since after checking out the proof details, I found that the crucial techniques were almost proposed by the previous related works.  However, this weakness is trivial, especially for a theoretical paper, and as I claimed in the Strength part, the result of this paper is solid.\n\nQUESTIONS:\n1. I suggest the authors could recall the readers of the definitions in the proof part, since numerous variables are introduced for proof, like $\\mathcal{G}_t$, $M$, $\\hat M$, $a_t$, $b_t$ and so on. It's a little inconvenient to check the definition in the previous pages each time.\n2. Since coordinate-wise calculations are commonly used in the proof, I suggest the authors could also consider demonstrating their results based on the $L_\\infty$ smoothness condition, as discussed in [1]. Also, I wonder about the difference of $(L_0,L_q)$-smoothness and local smoothness. \n3. For lemma B.2, I happened also to use this result before and I suggest the author cite [2], as I found this result in lemma A.2 of [2].\n4. How do the authors use the Cauchy-Schwarz inequality in the third inequality of line 740? Is this simply derived from $ab \\leq 1/4a^2 + b^2$ ? (Here $a, b$ are both scalars).\n5. In formula (58), line 557, where the last $\\sqrt{\\log}$ term comes from?\n6. What's the meaning of formulas (59) and (60) since $G^2 \\sim O(\\text{polylog}T)$ has been claimed in formula (7)\n\n[1] Balles, Lukas, Fabian Pedregosa, and Nicolas Le Roux. \"The geometry of sign gradient descent.\" arXiv preprint arXiv:2002.08056 (2020).\n\n[2] Zou, Difan, et al. \"Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization.\" The Eleventh International Conference on Learning Representations.\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper studies the high-probability convergence of Adam in the non-convex setting under relaxed assumptions. The authors consider a general noise condition that governs affine, sub-Gaussian, and bounded noise conditions. They also consider a generalized smoothness condition motivated by language model experiments. Under these assumptions, they obtain a convergence rate of $\text{poly}(\log T/\delta)/T$, where $T$ is the number of iterations and $\delta$ is the confidence level.

STRENGTHS:
1. Their result look novel and significant. They have shown the high-probability convergence of Adam under relaxed conditions than all previous papers.
2. The proofs look correct. 
3. The paper is well-written and results are clearly presented.

WEAKNESSES:
1. One major concern is, by choosing $\beta_2=1-1/T$, does the author essentially reduce Adam to SGD with momentum, as this makes $v_t$ almost a constant? Btw, I think for [18] and [23] in Table 1, $\beta_1$ should be $1-1/\sqrt{T}$. Please also check other rows more carefully.

I will increase the score if this concern is addressed.

QUESTIONS:
1. The term “affine variance noise" is confusing to me. I think it should only refer to Equation (2), not Equation (3), as "variance" is defined as an expectation. If I understand correctly, the term in Line 3 refers to (3), which means the condition (A3) is actually stronger than (2), right?
2. Why in Table 1 you did not include [19] which you discussed in Section 5.1?
3. The rate in Theorem 4.1 is dimension dependent, whereas the rate in Theorem 3.1 is dimension free. Do you think it is something fundamental in the relaxed smoothness condition?

LIMITATIONS:
See weaknesses and questions above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13