Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization
Authors: Yufeng Yang
Erin E. Tripp
Yifan Sun
Shaofeng Zou
Yi Zhou
Venue: Accepted by TMLR ()

Abstract: Recent studies have shown that many nonconvex machine learning problems satisfy a generalized-smooth condition that extends beyond traditional smooth nonconvex optimization. However, the existing algorithms are not fully adapted to such generalized-smooth nonconvex geometry and encounter significant technical limitations on their convergence analysis. In this work, we first analyze the convergence of adaptively normalized gradient descent under function geometries characterized by generalized-smoothness and the generalized PL condition, revealing the advantage of adaptive gradient normalization. Our results provide theoretical insights into adaptive normalization across various scenarios. For stochastic generalized-smooth nonconvex optimization, we propose the Independent-Adaptively Normalized Stochastic Gradient Descent algorithm, which leverages adaptive gradient normalization, independent sampling, and gradient clipping to achieve an $\mathcal{O}(\epsilon^{-4})$ sample complexity under relaxed noise assumptions. Experiments on large-scale nonconvex generalized-smooth problems demonstrate the fast convergence of our algorithm.

Published in Transactions on Machine Learning Research (07 (07/2025)
Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization
Yufeng Yang
yufeng- Boonnggnnnggnnoo Caamm.ttta....m edu
Department of Computer Science and Engineering
Texas AB M University
College Station, TX 77848, USA
Erin E. Tripp
ctripp@hamilton.. edu
Mathemalics and Stalislics Deparbment
Hamilton College Clinton, NY 13323, USA
Yifan Sun
Soun@ec.stonybrook edu
Department Compuler Science
Stony Brook University Stony Brook, NY 11794, USA
Shaofeng Zou
zouaasu. edu
School of Electrical, Computer and Energy Engineering Arizona State University Tempe, AZ 85287, USA
Yi Zhou
zi.zhou wi.zhou@tama. edu
Department of Computer Science and Engineering Tezas A&M University College Station, TX 77843, USA
Reviewed OpenReview:
https: // openreview. net/ ftr/formm? id= KXSQONLEfU
Abstract
Recent studies have shown that many nonconvex machine learning problems satisfy generalized-smooth condition that extends beyond traditional smooth nonconvex optimiza- tion. However, the existing algorithms are not fully adapted to such generalized-smool nonconvex geometry and encounter significant technical limitations on their convergence analysis. In this work, we first analyze the convergence of adaptively normalized gradi- ent descent under function geometries characterized by generalizel-smoottness and gen- cralized PE condition, revealing the advantage of adaptive gradicnt normalization. Our results provide theoretical insights into adaplive normalization across various scenarios. For stochastic peterralizdd-mmootto nonconvex optimization, we propose Independent- Normalized Stochastic Gradient Descent algorithm, which leverages adaptive gradient normalization, independent- sampling, and gradient clipping achieve an O(e sam- ple complexity under relaxed noise assumptions. Experiments[] on large-scale nonconvex generalized smooth problems demonstrate the fast convergence of our algorithm.
1 Introduction
In modern machine learning, the convergence of gradient-based optimization algorithms has been well studied in the standard smooth nonconvex selling. However, it has been shown recently that the slandard L-
Code available agghhullom//ooommg/////ooottttnn

---

Published in Transactions on Machine Learning Research (07/2025)
smoothness fails to hold in many nonconvex machine learning problems, inchuding distributionaall--rbuss optinization(RRO) (Levy et al. 2020; Jin et al. 2021), meta-learning (Nichol et al. 2018 Chayti & Jaggi 2024) and language models (Liu et al. 2023 Zhang et al. 2019h. Instead, these problems were shown to
satisfy a so-called generalized-smooth condition (Zhang et al. 2019). in which the smoothness parameter can
scale with the gradient norm in the optimization process.
In the existing literature, various works have proposed different algorithms for solving generalized-smmothh nonconvex optimization problems. Specifically, Li et al. (2024); Zhang et al. (2019); Chen et al.) (2028)
Gorbunov et al. )024)); Vankov et al. (2024b); Reisizadeh et al. (0023) have demonstrated that deterministic first-order algorithms such as gradient descent, normalized gradient descent and clipped gradient descent can achieve 3) iteration complexity under mild assumptions. These complexity results match the lower bound obtained by classical first-order methods (Arjevani et al.) 2023). In particular, Chen et al.] (2023) empirically demonstrated that proper usage of adaptive gradient normalization can substantially accolerate convergence. However, the formal thcoretical justification and understanding of adaptive gradient normal- ization is st.ill lacking for first-order algorithms in generalized--mmooll optimization.
On the other hand, some other works, including Li et al. 2024 Zhang et al. (2019 2020a studied first-order stochastic algorithms in generalized-smoott nonconvex optimization. Specifically, one line of works (Li et al. 2024) focused on the classic stochastic gradient descent, (SGD) algorithm (Ghadimi & Lan 2013). However, in generalized-smootl setting, the convergence analysis of SGD) either relies on adopting very large batch size or involves large constants (Li et al. 2024). These theoretical bottlenecks may be a key factor limiting the practical performance of SGD. Empirical observations from Chen et al. (2023) indicate that SGD often converges slowly due to ill-conditioned smoothness parameters when the gradient norm is large. Another line 2023 focused clipped SGD, which leverages gradient normalization and clipping to handle the generalized-smoode geometry. Although clipped SGD has demonstrated superior performance in solving large-scale problems, the existing theoretical analysis has several limitations. First, to establish convergence guarantecs in the stochastic sctting, existing studies often rely on strong assumptions, such as the stochastic approximation error being almost surely bounded or the use of extremely large batch sizes. Second, the existing designs of clipped SGD adopt the standard stochastic gradient normalization scheme, which is not fully adapted to the function geometry characterized by the seneralized-smooth condition.
of works (Zhang et al.) 2019 2020a Reisizadeh et al. 2023 Koloskova et al.
Having observed the algorithmic and theoretical limitations discussed above, we aim to advance the algorithm design and analysis for generalized-smooth optimization through investigating the following two fundamental and complementary questions.
Q1: In deterministic generalized-smme smooth optimization, how does adoptive gradient normalization affect the convergence rate of first-order algorithm, e.g., under Lojasiewj type conditions?
Q2: In stochastic generalized- optimization, can we design a novel algorithm that guarantees con- vergence under relaxed noise assumptions while relying on only small number of samples for computing each slochaslic gradient?
In this work, we provide comprehensive answers to both questions by developing new algorithms and conver- gence analysis in goneralized-smooth nonconvex optimization. We summarize our contributions as follows.
1.1 Our Contributions
To understand the advantage of using adaptive gradient normalization, we first study the convergence rate of adaptive normalized gradient descent (AN-GD) in deterministic generalized-smooth optimization under the generalized Polyak-Lojasiewicz (PE) condition over a broad spectrum of gradient normalization parameters. Our results reveal the interplay among learning rate, gradient normalization parameter and function geometry parameter, and charactorize their impact on the type of convergence rate. In particular, our results reveal the advantage of using adaptive gradient normalization and provide theoretical guidance on choosing proper gradient normalization parameter to improve convergence rate.
2

---

Published in Transactions on Machine Learning Research (07/2025)
We further propose novel Independent-Adaaptinn Adaptively Normalized Stochastic Gradient Descent (IAN-SGD) al- gorithm tailored for stochastic generalized-smood nonconvex optimization. Specifically, TAN-SGD leverages normalized gradient updates with independent sampling and gradient clipping to reduce bias and enhance algorithm stability. Consequently, we are able to establish convergence of IAN-SGD with O(e sample complexity under a relaxed assumption on the approximation error of stochastic gradient and constant-level batch size. This makes the algorithm well-suited for solving large-scale problems.
We compare the numerical performance of our IAN-SGD algorithm with other state-of-the-art stochastic algorithms in applications of nonconvex phase retrieval, distribattionally-obbor optimization and training deep neural networks, all of which are generalized-smooth nonconvex problems. Our results demonstrate the efficiency of LAN-SGD in solving generalized-smooll nonconvex problems.
2 Related Work
Seneralized-SSoothneee The concept of generalizedl was first introduced by Zhang et al. (2019) with the (Tog I-)-shooth condition, which allows a function to either have an affine-bounedd Hessian norm or be locally L-smooth within a specific region. This definition was extended by Chen ct al. (2023). who proposed the Cāsym (a) and Leem (a) conditions, controlling gradient changes globally with both a constant term and gradient-dependent term associated with power a, thus applying more broadly. Later, Li et al.) (2024) introduced Ê-smoothness, which use a non-decreasing sub-quadratic polynomial to control gradient differences. Mishkin et al. (2024) proposed directional smoothness, which preserves L-smoothness along specific directions.
Algorithms for Generalized-Smootii Optimization. Motivated by achieving comparable lower bounds presented in Arjevani et al. (2023) under standard assumptions, algorithms for solving generalized-smoolt problems can be categorized into two main series. The first series focuses on gradient descent methods with constant learning rate. Li et al. (0024) proved that GD, SGD converge with ((- and O(e complexity under generalized-smoottnnnee
To ensure convergence, Li ct al. (2024) adopted assumptions such as gradient upper-bound and bounded variance assumption.
Another series of work focuses on adaptive methods, In the nonconvex deterministic settings, Zhang et al. 2019 2020a); Gorbunov et al. (2024) : Vankov et al. (2024b) showed that clipped GD can achieve an iteration complexity of O) under mild assumptions. Later, Chen al. (2023) proposed 3-GD that achieves O(--2) iteration complexity. Specifically, in the convex setting. Vankov et al. (024b) analyzed clipped and normalized gradient descent under mild assumptions. Goorbunvv et al. (2024] studied smoothed gradient clipping, gradient descent with Polyak step-size rule, and triangle method by varying the learning rates. Both works achieve the best-known O(E-1) convergence rate for generalized-smmoohh convex optimization. In the nonconvex stochastic setting, Zhang et al. (2020b) proved that clipped SGD achieves an O(c=t) sample complexity under the standard L-smoothness and bounded variance assumptions. Also, Zhang el al. (2019) established the same convergence rate for clipped SGD under generalized smoothness and almost surely bounded noise assumption. Later, |Reisizadch et al. (2023); Koloskova et al. (2023) analyzed the convergence of clipped SGD under the bounded variance assumption. Notably, Koloskova et al. (2023) pointed out that, with bounded variance assumption, clipped SGD) converges to c-stationary point with sample complexity O(e-5) when using f(1) samples per iteration. To improve convergence sample complexity, Reisizadeh et al. (2023) employed a large batch size O(e to suppress the impact of the variance term and achieve O(E complexity. We summarize these results under mild assumptions in Table [
A number of works also investigated the performance of diverse stochastic algorithms extending beyond SGD and clipped SGD under the generalized-smmothh condition. [Wang et al. (2023): Faw et al. 2023 Hong & Lin (0224) studied AdaGrad (Duchi et al. 2011b) under generalized-smmooh and affine variance assump- tion with different learning rate schemes. They all attain ((1/VT) convergence rate under mild conditions.
Xie el al. (2024a) studied trust-region hods convergence under generalizeel-mmootnnen
For stochastic acceleration methods under the generalized seneralized-smootnnness condition, Zhang et al. (2020a) proposed a general clipping framework by leveraging momentum clipping and they achieve Ö(c sample complexity under almost sure bounded noise assumption; Jin et al. (2021) studied normalized SGD with momentum under parameter-dependent learning rates schemes, which achieves O(e-1) sample complexity under generalized-
3

---

Published in Transactions on Machine Learning Research (07/2025)
smooth and affine variance assumption; Hübler et al. (2024) studied normalized SGD with momentum (Cutkosky & Mehta 2020) under parameter-agnostie learning rates schemes, which establishes ((6 con- vergence rate under bounded variance assumption. CCenn et al. (2023) Reisizadeh et al. (2023) demonstrated that the SPIDER algorithm (Fang et al.) 2018) can reach the optimal O(c"²) sample complexity under affine variance assumption for sum-lype functions by employing a large batch size. Furthermore, [Zhang el al. (2024b); Wang et al. (2024a b): Li al. (2023) explored the convergence of RMSprop (Hinton et al., 2012) and Adam (Kingma| 2014] under different noise assumptions, including coordinate- affine variance, ex- pected affine variance, bounded variance and sub nub-gaussian distribution. The acceleration of sign-SGD was explored by Jiang et al. (2024); (Snn et al. (2023b). and its generalized-smooth variants were subsequently analyzed by Crawshaw et al. (2022); Sun et al. (2023a), where they both obtain O(E under almost-sure bound and bounded variance assumplions, respectively.
Method
Smoothness
Noise assumption
Algorithm"
Batch Size*
Convergence?
Complexity
Zhang et al.
2019
GS
a.s. bound
Clip-SGD
2(1)
e-stationary
O(e
Zhang et al.
2020b
LS
var bound
Clip-SGD
0(1)
e-stationary
O(c
Zhang et 2020a
GS
a.s. bound
General-Clip
2(1)
e-stationary
O(e
Li al. 2021
GS
var bound
SGD
9(1)
e-stationary
O(e-t)
Reisizadeh ct al.
2023
GS
var bound
Clip-SGD
(((-²)
c-stationary
O(c-9)
Koloskova al.
2023
GS
bound
Clip-SGD
2(1)
e-stationary
O(e
Our Method
GS
8.5. affine
IAN-SGD
2(1)
esstaionary
O(e-))
Table 1: Comparison of existing clipping- baspeng-based algorithms. Denote the stochastic gradient and clipping threshold as Vfe(w:) and respectively. Explanation of abbreviations: (GS) refers to the generalized- smoothness condition, and (LS) refers to the standard L-smoothness condition. (a.s. bound) means that for every sample, the stochastic gradient bias is almost surely bounded, i.e., [[ (we) - VF(w))) S 72 (var bound) denotes the bounded variance condition, i.e, fe (w;) - (w))) S -2i (a.s. affine) implics that for every sample, the stochastic gradicnt bias satisfies the affine bound fe(w))) VFF())) S Ti [w:)] almost surely, for some 0 S ri < and 72 > 0. The term Clip-SGD refers to algorithms that use stochastic gradient clipping of the form W1+11 W, - mim [" ff SGD refers to stochastic gradient descent; General- Clip denotes algorithms that incorporate momentum during clipping and updates W = Wi (v min [ + (1 - v) minfc, [V (We)) where m441 (1 8)m + VV (wx) The label c-stationary indicates that the algorithm can converge to an arbitrary target error. For Koloskova et al. (2023), we report their convergence result under learning rate 7 = / (T + eIn)) with c = T2/C, where their analysis guarantees convergence to an e-stationary point.
Machine Learning Applications. Generalized- ing frameworks. Levy et al. y Jin et al. (2021) studied the dual formulation of regularized DRO prob- lems, where the loss function objective satisfies generallieed-smooolneee
has been studied under various machine learn-
Chayti Jaggi (2024) identified
their meta-learning objective's smoothness constant increases with the norm of the meta-gradient Gong
et al. (2024b); Hao et al. (2024): Gong et al. (2024a); Liu al. (2022b) explored algorithms for bi-level optimization and federated learning within the context of smmorraliizddssmooenees... Zhang et al. (2024a) developed algorithms for multi-task learning problem where the objective is generalized-smoon
Xie et al.
(2024b) studied online mirror descent when the objective is generalized-smooth. Xian et al. (2024) studied min-max optimization algorithms' convergence behavior under generalized-smoone condition. There is a concurrent work (Vankov et al., 2024a) using independent sampling with clipped SGD framework to solve variational incquality problem (SVI). Based on this idea, they also propose stochastic Korpelevich method for clipped SGD. Under generalized- condition, they proved almost-sure convergence in terms of distance to solution set tailored for solving stochastic SVI problems.
3 Notations
Throughout the work, we denote D as a set of training examples, & as an example sampled from the training dataset D, f() as the objective function, fe(+) as the objective function associated with data sample f. F(w) (fe=(Je(())) the expected loss over all samples following distribution P, and || Eg-orrm over
4

---

Published in Transactions on Machine Learning Research (07 (075))25)
Euclidean space. Let E Rd denote the parameters of function f (i.e., the weights for linear model and neural networks), S denote the infimum of f. To. L1 and are positive constants used to characterize the generalized--mmoothness condition (see Assumption D p. are positive constant used to describe generalized Pl-condition (see Assumption
T1. T2 characterize the stochastic gradient noise (see Assumption 4 7; 3 denote the learning rale and normalization power used adaptive normalization, respectively. In the Distributionally Robust Optimization (DRO) experiments (sce Section 7 we define I(w,,) with parameter w and dual variable 7), as the dual objective derived from regularized @-divergence primal DRO problem.
4 Generalized- nonconvex Optimization
We first introduce generalized-smooe optimization problems. Consider the following optimization problem.
min f(w),
(1)
wER
where f Rd + R denotes nonconvex and differentiable function and w corresponds to the model parameters. We assume that function f satisfies the following generalized-smoothh condition.
Assumption 1 (Gencralized-smoott) The objective function satisfies the following conditions.
1. f is differentiable and bounded below, i.e., f == inf f(w) > -o;;
There erists constants Lo, L1 > and a E [0, 1] such that for any w, w' E Rdd, it holds that
[[ ( f((w))) s + L =. w'll
(2)
The generalized-smoon condition in Assumption is a generalization of the standard smooth condition, which corresponds to the special case of a = 0. It allows the smoothness paramcter scale with the gradient norm polynomially, and therefore Assumption can model functions with highly irregular geometry. Moreover, following the standard proof, it easy to show that generalized-smooth functions satisfy the following descent lemma. See Proof[] in Appendix B
Lemma 1 Under Assumption
1 function f satisfies, for any w, w E Rd,
f(w) f(w') + (Vf(w'), w') + 24o + L VIll) w'll
(3)
We note that there are several variants of generalized-smooth conditions proposed by the previous works (Zhang et al. 2019 Jin et al. 2021 Chen et al. 2023). Below, briefly discuss the relationship between the generalized-snent -mooohhh condition in Assumption and these existing notions.
Remark 1 (LLo, L1) generalized-smmoot condition) The descent lemma of (Lo, L1) generllied--smooth condition proposed in Zhang et al. (2019) is given
1 f(w) < f(w') + (Vf(w'), w - w') 2 ( 5L.llV((')Illl -''l
w'l
which is the same as Lemma with a up to differences in the constant coefficients.
Remark 2 (Symmetric generalized-smooth condition) Chen et al. (2023) introduced a symmetric version of generalized-smooli condition, by replacing [[V a in equation
2 with max (wgg)l,, where
Wa Ow' (1- 9)w. We notice that the asymmetric generalized-smoole condition adopted in Assumption can also imply the following similar symmelric generulized smooth condilion,
| [V
[[('''
Vf(w) vf(w]]] (Lu + Ln(
})[[ wwwwl
w'll
(4)
2

---

Published in Transactions on Machine Learning Research (07/2025)
Jin et al. (2021), it has been shown that certain regularized nonconvex distributionallyy robust optimization
(DRO) problems satisfy the generalized-smooth condition in equation with Lo - L, L - 2M( a = 1.
Moreover, the classic nonconvex phase retrieval problem (Drenth 2007
Miao et al. 1999) can be shown to
satisly the above symmetric generalized-smoote condition in equation (see Proof 2 in Appendix C
In the following sections, we first consider deterministic generalized-smmooll optimization and study the impact of adaptive gradient normalization on the convergence rate of gradient methods. Then, consider stochastic generalized-smooth optimization and propose novel independent sampling scheme for improving the convergence guarantee of stochastic gradient methods under relaxed noisc assumption.
5 Adaptive Gradient Normalization for Deterministic Generalized-Smoodt Optimization
In deterministic generalized- optimization, several previous works have empirically demonstrated the faster convergence of normalized gradient descent-type algorithms (e.g., clipped GD) over the standard gradient descent algorithm in various machine learning applications (Zhang et al. 2019 Chen et al., 2023) On the other hand, theoretically, these algorithms were only shown to achieve the same iteration complexity 3) the gradient descent algorithm in generalized-smootl optimization. In this section, further advance the theoretical understanding and explain the inconsistency between theory and practice, we explore the advantage of adapting gradient normalization to the special generalized-smoott optimization. We aim to show that gradient normalization, when properly adapted to the underlying PL geometry, can help accelerate the convergence rate in generalized-smootl optimization.
(PL) geometry in
Specifically, we consider the class of generalized-smonl problems that satisfy the following generalized PL geometry.
Assumption 2 (Generalized Polyak-Eojasiewiez Geometry) There crists constants μ E RR and 0 < S 2 such that f(-) satisfies, for all w E R [V[(w)] 2 2p(s(w) -*))
(5)
The above generalized PE condition is inspired by the Hurryyka-.ojasieviit (KL.)-xxponent condition pro- posed in Li & Pong (2018). When p > 1, equation 3 reduces to the KL-cxponent condition. When p == 2, equation reduces to the standard PF. condition. Moreover, some recent works have shown that PF-type geometries widely exist in the loss landscape of phase retrieval (Zhou et al., (2016) and over-parametrized deep neural networks (Liu et al. 2022a Scaman et al. 2022). , and we hope that our analysis based on Assumption
2 will allow researchers to rethink the relationship between adaptive nornalization and loss landscape geometry.
Here, we consider the adaptively normalized gradient descent (AN-GD) algorithm proposed by Chen et al. (023 for generalized-smnn smooth nonconvex optimization. The algorithm normalizes the gradient update as follows
Vf(w) (AN-GD) M4+1 = w/: (w:)))
(6)
where y > 0 denotes the learning rate and /3 is a normalization scaling parameter that allows to adapt the normalization scale of the gradient norm to the underlying function geometry. Intuitively, when the gradient norm is large, smaller 8 would make the normalized gradient update more aggressive; when the gradient norm is small, normalization can slow down gradient vanishing and improve numerical stability.
Chen et al. 2023 studied AN-GD in siteeralzzed-mmoodd nonconvex optimization, showing that achieves the standard ( 2) iteration complexity lower bound. In the following theorem, we obtain the convergence rate of AN-GD under the generalized PL condition. See Proof in Appendix E
Thcorem 1 (Convergence of AN-GD) Let Assumptions 1 and 2 hold. Denote A := f(w)) - f" as the 12ne)//0 < 2 for
function value gap. Choose target error 0 < e S min(1, 2 and define learning rate = 8(oo+1a4+T
some 3 E (a, 1]. Then, the following statements hold,
6

---

Published in Transactions on Machine Learning Research (07/2025)
1f < P, in order to achieve AT e, the total number of iterations must satisfy
B2- -2-BB-)(2u)B11)) (( (212-8-)1(2-)) 1)40 ( 10000-2111020001 212-8822---1111110 (2-8-01/01 = 0(( =(
8p(8(Lo + L1)
1
T2 max
(7)
1 = p, -,, in order to achieve AT S e, the lotal number of iterations must satisfy
21-B/1 8lP. (8(Lo + +1)+11 log( 20) = =s0( llog(
Au =
T p. (uE))//
(8)
11128> 2- ps and for some To salisfying ATo S (0((m((2-9)(6t9>)) ) then the total numbbr of iterations after To must satisfy
T T20 log (12 ( + log loog (32(Lo + L1) + 4)0/()/3-///)00 (0( 2
(2n)2/(0+B-2)
= 2 2llo (())
(9)
Theorem characterizes the convergence rates of AN-GD under different choices of and B. In particular, and p = 2, Theorem | reduces to the linear convergence rate achieved by gradient descent
when a = = 3= under the standard PL and L-smooth condition. Theorem | alsoo guides the choice of gradient normalization parameter B under different geometric conditions. For example, if there exists [a, 1] such that p p 3 2, AN-GD can boost its convergence rate from polynomial local linear convergence. also indicates when P is very small such that p 3 < 2, the effects of can be marginal. However, for p + < 2, the iteration complexities depend more on the specific values of p and B. For example, when p and consider two different choices of gradient normalization parameters 31 = 3.32 = 1, AN-GD achieves the iteration complexities O) and Ö(c-"), respectively. This result matches the empirical observation made in (Chen et al. 2023 that choosing smaller 3 [ax, 1] can improve the convergence rate.
We note that several concurrent works also studied normalized gradient descent methods in deterministic convex settings. In Gorbunov et al. (2024). the authors consider various update rules such as Wt+1 = [ww))/(w">) V fw:). For generalized smooth convex optimization,
T.o+-1l/ W/ To I [(m))] and Wh+11 we - - [V/[ they obtain a two-phase convergence result, similar the case when 8 > in Theorem | where the second phase achieves complexity bound O(c-1). They further employ the triangle method to derive global ()--)) complexity bound. In Vankov et al. (004bb), the authors propose related framework by choosing 7 = f(m,)( which aligns with our AN-GD when B = 1. They also obtain O(e-1) complexity for generalized smoothness convex optimization.
Vf(w.
5.1 Comparison between AN-GD and GD.
In Li et al. 2024), it has been shown that GD achieves O)), O(e-)), O(e-2) complexity bounds for generalized-aoe -mmoohhh strongly convex, convex and non-convex optimization, respectively. Compared with Chen et al. (2023), both AN-GD and GD achieve O(e-2) complexity bound for generalized non-convex optimization, and hence are optimal deterministic methods. On the other hand, the following Proposition further obtains the convergence rate of gradient descent (GD) under the same Assumptions 1 and 2 See
Proof 5 in Appendix F
Proposition 1 (Convergence of GD) Let Assumptions and hold. Assume there erists positive 1 and setting y I mint Lg 2L,eh, b gradient descent converges to an c-stationary point within T T T202 9)) iterations.
constant G such that [=)()l))l < Vt E T, When p
From Theorem [ under the setting a = and with 3 = 1, AN-GD converges to a e-stationary point in Ō(E iterations. However, the convergence guarantee for GD requires an additional assumption that the gradient norm upper bounded by a constant G. This constant typically depends on the function value gap Ao and |zg - I',, which can be large in generalized smooth optimization settings. Such large value can be the key reason for the slower empirical convergence of GD observed in Chen al. 2023
7

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
VF FIV
E
fIl
Vfz.
f
Figure 1: Comparison between normalized full gradient (blue) and expected normalized stochastic gradient (red). Here, & and &2 are sampled uniformly at random.
6 Independentyy-mnd-Adaatievyy Normalized SGD for Stochastic Senoralized--mmothh Optimization
In this section, we study stochastic generalized-smmooh optimization problems, where denote fe as the loss function associated with the data sample and we assume that the following expected loss function F()) satisfies the reneralized-smoood condition in Assumption
min F(w) = BE-P [w]] WER
(10)
Having discussed the superior theoretical performance of AN-GD in the previous section, we aim to leverage adaptive gradient normalization to further develop an adaptively normalized algorithm tailored for stochastic goncralizedd-mmoohh optimization.
6.1 Normalized SGD and Its Limitations
To solve the stochastic gonoralized-smooth problem in equation 10 one straightforward approach to replace the full batch gradient in the AN-GD update rule, equation f with the stochastic gradient V (w:), resulting in the following adaptively normalized SGD (AN-SGD) algorithm.
Ve(())))
(AN-SGD) W++1 = Wi - fe
(11)
AN-SGD-type algorithms have attracted lot of attention recently for solving stochastic generalized-smooth problems (Zhang et al. 2019 2020al Liu et al. [022b) Jin et al. (021].. In particular, previous works have shown that, when choosing 3 and using gradient clipping or momentum acceleration, AN-SGD's
variations (e.g., NSGD with momentum (Cutkosky & Mehla 2020 Jin el. al.) 2021), clipped SGD (Zhang jet al. 2019 2020a) ) can achieve a sample complexity of O(c*). This result matches the convergence order of the standard SGD (Ghadimi & Lan 2013 for solving classic stochastic smooth problems. However, AN-SGD has several limitations summarized below.
Biased gradient estimator: The normalized stochastic gradient used in equation 11 is biased, i.e., El [.e( [e( * [VF(W.)] This is due to the dependence between (w,) and (w,))l In par- ticular, the bias can be huge if the stochastic gradients are diverse, as illustrated in Figure
(((w))
Strong assumptions: To control the estimation bias and establish theoretical convergence guarantee for ANSGD-type algorithms in generalized-smoodt nonconvex optimization, the existing studies need to adopt strong assumptions. For example, Zhang et al. (2019 2020a) and Liu et al. (2022b) assume that the stochastic approximation error [VV [Fw]]] is bounded by constant almost surely. In practical applications, this constant can be a large numerical number certain sample & happen to be an outlier. On the other hand, Chen al. (2023): Reisizadeh al. (2023) require AN-SGD type of algorithms such SPIDER (Fang al.) (2018) to have 2) batch size to ensure fast convergence. It essentially makes it full-batch algorithm and is impractical in practice.
8

---

Published in Transactions on Machine Learning Research (07 (07/2025)
6.2 Independently-arr and-Adaptivel Normalized SGD
To overcome the aforementioned limitations, we propose the lollowing independeenlyamm/daddpiivvy nor- malized stochastic gradient (LAN-SG) estimator
Vfe(w) | fer (1
(IAN-SG estimator)
(12)
where & and e' are samples drawn independently from the underlying data distribution. Intuitively, the independence between & and g decorrelates the denominator from the numerator, making an unbiased stochastic gradient estimator (up to scaling factor). Specifically, we have that
Vee(w) (e
E V [e()]] & VF(w).
Eg.e'
(13)
Ve (w)]l
Morcover, as we show later under mild assumptions, the scaling factor E fe can be roughly bounded by the full gradient norm and hence resembling the full-batch AN-GD update. Based on this idea, we formally propose the following independently wtwplddeetll--dddddadpiieeeeeeee aundaadaptively normalized SGD (LAN-SGD) algorithm (see oquation 14), where A, r, are positive constants, V.f(w)), fwr, corresponds to the stochastic gradient associated with two independent samples ç, 5' sampled at time
(e( h
(IAN-SGD): W++1 = WA
=m= where h4 = max {1, (AI]VIe-(w))) + })}
(14)
The above IAN-SGD algorithm adopts clipping strategy for the normalization term ht. Intuitively, when gradient converges in the later optimization phase, the term h4 = 1 and LAN-SGD reduces to SGD, This is consistent with the fact that the generalized-smooth condition reduces to the standard smooth condition when gradient approaches to 0. Moreover, imposing constant lower bound 6 on ht in equation [4 helps develop the theoretical convergence analysis and avoid numerical instability in practice. As we show in the ablation study presented in the appendix, the convergence speed of LAN-SGD insensitive to the choice of 8. We also note that IAN-SGD roquires querying two batches of samples in every itcration. However, we show later in the Theorem 2 and experiments that these batch sizes can be chosen at constant level in both theory and practice.
6.3 Convergence Analysis of IAN-SGD
We adopt the following bias assumption on the stochastic gradient fl((w)..
Assumption 3 (Unbiased stochastic gradient) The stochastic gradient (e is unbiased, i.e., Eş- (V] = F(w)w) for all Rd.
Assumption (Affine approximation Error) There erists ST < <1, such that for any w E Rd
[V VF(w)] s n VF(w)]] + a.s. Ag ~g~D P.P
(15)
Assumption allows the approximation error of stochastic gradient scaling with the full gradient norm on a per-sample basis, while only requiring the error to remain bounded at stationary points. Compared to other noise assumptions, Assumption 4 is much weaker than the almost sure error bound (i.e., тı 0) adopted
in Zhang et al.] 22019 2020a); Liu et al. (2022b) but stronger than expected error bound (i.c., bounded
variance assumption Es [V(w) VF(w) 2 < T adopted in Koloskova el. al. (2023); ; Reisizadeh et. al.
(223). Our motivation for studying the convergence of LAN-SGD under this noise assumption is to strike a balance between achieving standard convergence rates (i.c., O(c-)) under relaxed noise conditions, while keeping the batch size at the le1ellevel. As later we show Theorem IAN-SGD achieves O(e=A) sample
complexity under Assumption 4

---

Published in Transactions on Machine Learning Research (07/2025)
Remark 3 Existing work [Koloskova et al. 2023) has shown that when adopting Q((1) sample per iteration, clipped SGD allains ())-5) sample complexities under generalized- smoool ond bounded variance assumplions. In order to achieve the stendard O)e sample complerity under such assumption, Reisizodeh et al. (2023) demonstrated that clipped SGD requires M) samples per iteration. Through this way, the constant arising from the variance of the upprozination error can be controlled at the (levv-leve.. This, in turn, relures the burden on éteration complerity, requiring only (c=²) iterations for averaging during convergence proof. We leave technicol discussions on oplimizing the batch size while maintaining (EE total sample complexity of IAN-SGD for future work.
Based on these assumptions, we first obtain the following upper bounds regarding [| See Proof 0 in Appendix G
(VVe(wa)))
Lemma 2 Let Assumptions and hold. The gradient [(w)ll and stochastic gradient Vfe(we) satisfy
[V - 1- [ee(()))) + T2 and
1-71 V. < m 1) +
(16)
Lemma 2 indicates that the full gradient norm can be upper bounded by the stochastic gradient plus a constant. This result suggests that for h2 in equation 14 one can choose A = = and -- and is very useful in our convergence analysis to effectively bound the stochastic gradient norm in ht- We obtain the following convergence result of LAN-SGD, See Proof in Appendix [
and 4 hold. For the IAN-SGD algo- -1) }. and A T-m 8 = - r (4L17(27) + 1))² Denote A == F(wo) - F + 5 + L1) + 4T2)3. Then, with prob- ability at least 5, IAN-SGD produces a sequence satisfying minesT [[])()))] S if the total number of iteration T satisfies
Thcorem (Convergence of IAN-SGD) Let Assumptions
rithm, choose learning rale 7 min +1)(27 (2,(27+1)) VT (27 (1)2m (1)))>>
T 2 A Aax {
256A 64L, (27g + +1 ) (1-1)) 2-28 .2T? +1) 64(Lo + L1) + 1 (2))
(17)
2²
Remark 4 Existing work Zhhng et al. 2019) studied the iteration complerity of clipped SGD (t.e., 8 =1 and E,, without independence sampling) the special bounded noise setting Tι = 0, and obtained the complexity
4A 128L, S0L. + 5122172 T> A max
(18)
22
where A =F(cx))F=++ (5Lo + 22222)72 9 As comparison, the above Theorem 2 establishes comparable complerity resull under the more general noise model (Ti 2 0) in Assumplion 4 by leveraging the adaptive gradient normalization and independent sampling of IAN-SGD. In particular, in the special setting TI TI = 3 == 1, these two complexity results match up to absolute constant coefficients.
Another existing work Zhang el al. (020aa) analyzed a generalized clipping frumework that com- bines momentum acceleration and gradient clipping, namely W1+1 = We (u {c, T | }mm + (1
v) min (c, ( (e((w.)) under bounded noise setting (i.e., =1 They establish complexity of ((F((wo))))))))) (r)3o///))))) when 7/c- = O(22), where m. denoles the momentum lerm and e is the clipping threshold. This framework enables them to construct a recursion based directly on [ whereas our analysis is built around recursive bounds involving [V=(())] or [V((2)) at each step. Consequently, while both approaches yield the some order of convergence in terms of e, the complexity expressions slightly differ in the numerators due to different recursions.
Compared to the existing studies (Zhang et al. 2019 2020b Reisizadch et al. 2023 Zhang et al. 2020a
Koloskova et al. 2023) on gradient clipping algorithms, Theorem 2 jestablihhes convergence guarantees with- out the bounded approximation error assumption and the use of extremely large batch sizes that depend on
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
the target error Through numerical experiments in Section 7 and ablation study in A.2 later, we show that it suffices to query small number of independent samples for TAN-SGD in practice.
6.3.1 Proof outline and novelty
The independent sampling strategy adopted by IAN-SGD naturally decouples stochastic gradient from gradi- ent norm normalization, making it easier to achieve the desired optimization progress in generalized-smoolh optimization under relaxed conditions. By the descent lemma, we have that
E5, [((ue+1) - F(w)))/wr) (L + L1 (F 19)
(1) <Bg'
7| (F
2h
(1))
Lu + L L1llv (.llVP(us [=I1V/(w)))(")) )) + 2T (Lo + 41 L/ [woe). (19)
(-
S Be
h
23
where (i) follows from the descent lemma (conditioned w,) and the independence between & and E': (ii) leverages Assumption 4 to bound the second moment E [e (w; (wx) by (4Tî + F) + 472 Then, for the first term in equation
19 we leverage the clipping structure of he to bound the coefficient (2r) + 1) + La L [(W)l)))) by 2- For the second term in equation 19 we again leverage the clipping
structure of he and consider two complementary cases: when [|V((w))] S V1+ 4 + +), this term can be upper bounded by {Lo 41) +473)²: when [[VF(())]| > VT+ 47 (2 + 1), this term can upper bounded by .i /| Summing them up gives the desired bound. We refer Lemma 5 in the appendix for more details. Substituting these bounds into equation 19 and rearranging the terms yields that
Eel 4hg
[F(w))]
<Eg, [F(w.) - F(w++1)) + 2 (Lo + (1)
Furthermore, by leveraging the clipping structure of h. and Assumption 4 the left hand side of above incqual- ity can be lower bounded as 71 V((w)) 2 (in()/\VF(w>))1 (( (2x }. Finally, telescoping
[[
the above inequalities over f and taking expectation leads to the desired bound in equation 17
As a comparison, in the previous works on clipped SGD ZZang et al. 2019 2020a). their stochastic gradient and normalization term h depend on the same mini-batch of samples, and therefore cannot be treated separately in the analysis. For example, their analysis proposed the following decomposition.
[e Fe (IVF(w:)))+++ (V VF(w:) + Vlg(w:) V /n [n?s]
Egg
1225
Hence their analysis nceds to assume constant upper bound for the approximation error [[](() (F(w))] in order to obtain comparable bound to ours.
6.3.2 IAN-SGD under generalized Pt condition
Under Assumption 2 we obtain the following recursion for TAN-SGD. See Appendix for Proof details.
Lemma 3 (Descent incquality of IAN-SGD under generalized PE condition) Let Assumptions
2 and 1 hold. For the IAN-SGD algorithmm, choose largel error 0 < 5 m1, 1/2p} and learning rale
(2ue) mint [To(2?] [1,((27; (2)0 (T,(21] + +1]2=/][1110001----0000 (1)))) To((To L, +1)
(7 b
and A 8 = i I (4L +1) Then, we have the following descent inequality
2// (L) + (2) 3/2 ( 12)) Eusya [A++1] S Ew,, [A
(20)
8
We notice that equation 20 is almost the same as the recursion of AN-GD obtained in Appendix D The main difference ween equation 20 and equation 35 is the usage of o instead of to characterize the recursion.
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
As result, the convergence rate of IAN-SGD is expected to be much slower than AN-GD, i.e. IAN-SGD 32 converge with rate O(E *) when p B - 2; LAN-SGD converge with rate O(e _))))) when p 3 < 2. We
omit the discussions of proof as its convergence analysis exactly follows the proof presented in Appendix
7
Experiments
We conduct numerical experiments to compare LAN-SGD with other state-of-the-are stochastic algorithms, including the standard SGD (Ghadimi & Lan 2013). normalized SGD (NSGD), clipped SGD (Zhang et al.
2019),
SPIDER (Fang et al.
2018), normalized SGD with momentum (Cutkosky & Mehta 2020 (NSGDm
ctc. Compared with previous works that study clipped SGD (Zhang et al. 2019: 2020a b Koloskova et al.
2023 Reisizadeh et al. 2023), the main change of IAN-SGD lies in its usage of independent sampling
and adaptive normalization. To empirically validate the effect of adaptive normalization in accordance with
Theorem T our experiments focus on nonconvex phase retrieval, distributionally robust optimization (DRO),
and deep neural network training. Prior works (Zhou et al., 2016 Jin al.) 2021; Liu al., 2022a Scaman
et aal.. 2022) have shown that. phase retrieval and deep neural networks satisfy generalized PE conditions under mild assumptions. Accordingly, we evaluate the convergence of all algorithms in terms of total sample complexity (for phase retrieval and DRO) and number of epochs (for deep neural networks). Furthermore, to align the DRO formulation with our theoretical assumptions, we focus on regression tasks and adopt the >ddiveegencce to model distributional uncertainty. All experiments were conducted on a PC computer equipped with a 24-core CPU, 32GB of RAM, and a single NVIDIA RTX4090 GPU, running Python 3.8.
7.1 Nonconvex Phase Retrieval
The phase retrieval problem arises in optics, signal processing, and quantum mechanics (Drenth 2007). The goal is to recover signal from measurements where only the intensity is known, and the phase information is missing or difficult to mcasure. Specially, denote the underlying object as w E Rd. Suppose we take m intensity measurements yr = oa 1² 2 + ny for r = .. where Or denoles the measurement vector and n. is the additive noise. We aim to reconstruct w by solving the following regression problem,
1 m
min f(w) #E
2m Σ (yr
ao w 3
(21)
Such nonconvex function satisfies generalized-smoootl with parameter a = 2 In this experiment, generate the initialization wp ~ N111 6) and the underlying truc object w" ~ 0.5, with dimension d = 100. We take m = 3k measurements with a, (V 0.5) and n, ~ (N 16). We implement all of the stochastic algorithms in original form as described in previous literature. We set batch size |B] = 64, and for IAN- SGD, we choose a small independent batch size |B" = 4. We fine-tuned learning rate for all algorithms, i.e., y = 5e for SGD, ? 0.2 for NSGD and NSGD with momentum, y = 0.6 for clipped SGD, and y = 0.25 for SPIDER and TAN-SGD. We set the maximal gradient clipping constant as 20, 8 = 1 for both clipped SGD and IAN-SGD. And we set normalization parameter & = 3. Figure 2 (left) shows the comparison of objective value versus sample complexity. It can be observed that IAN-SGD consistently converges faster than other baseline algorithms.
7.2 Distributtionaly-RRobust Optimization
Distributtionall--robsst optimization (DRO) is a popular approach enhance robustness against data dis- tribution shift. We consider the regularized DRO problem min fw) supq (Eę~a[4ç(w)) AI( } where Q IP represents the underlying distribution and the nominal distribution respectively. X denotes regularization parameer and á denotes d-divergence metric. Under mild technical assumptions, Jin ct al. (221)) showed that such a problem has the following equivalent dual formulation
min I(w,n) AE (f((w) A BER
") n,
(22)
12

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Tuining las curves for phase retrival
Traiting loss curves for DRO
SGD
SGD
NSGD
NSGD
17
NSGDm
NSGDm
10
Clip SGD
Clip SGD
SPIDER
SPIDER
121
IAN-SGD
IAN-SGD
2.0
0.0
0.2
0.4
0.6
1.2
Sample Complexity
Figure 2: Experimental Results on Phase Retrieval and DRO
where o* denotes the convex conjugate function of ø. In particular, such dual objective function satisfies the generalizeed-mmoohh condition with parameter a (Jin et al., 2021 Chen et al., 2023). In this experiment,
the Life Expectancy data (Arshil 2017) for regression task. For objective equation 22 which is the convex conjugate of x²-divergence. We adopt the regularized loss (<(w) 1 (ye - -e w)² + 0.1 Σ In(1 + [w(l1)), where Tig; Ve represent input and output data, respectively. And we optimize the primal and dual variable of L(w, 17) in parallel following conclusion drawn from Jin et al. (2021)
we set X = 0.01, select o (t) = [t + 2)
For the stochastic, momentum moving average parameter used for acceleration methods, set it as 0.1 and 0.25 for NSGD with momentum and SPIDER respectively. For stochastic algorithms without usage of multiple mini-batches, i.., SGD, NSGD, NSGD with momentum and clipped SGD, we set their batch sizes as [B] = 128. For SPIDER, we set [B] = 128 and |B'| = 2313, where the algorithm will conduct a full- gradient computation after every 15 iterations. For IAN-SGD, we set the batch size for two batch samples [B] 128 and [B| 8. We fine-tuned the learning rate for each algorithm, i.e., Y 4e-5 for SGD, y 5e for NSGD, NSGD with momentum and SPIDER, y 0.11 for clipped SGD and IAN-SGD, We set the 5 = 1 maximal gradient clipping constant as 30, 25 for clipped SGD and IAN-SGD respectively. And we set normalization parameter B = 3. Figure (right) shows the comparison of objective value versus sample complexity. can be observed that objective value optimized by IAN-SGD consistently converges faster than other baselines algorithms.
7.3 Deep Neural Networks
Training Loss Curves ResNati8
Training Loss Curves for Reshiet50
8
8
SGD
SGD
7-
7-
Adam
Adam
6-
Adagrad
6
Adagrad
NSGD
5-
5
NSGDm
NSGDm
Clip_SGD
Clip SGD
3-
IANSGD
3-
IANSGD
2
2
1
1
0-
0.
40
Epochs
Epochs
Figure 3: Experimental Result on training ResNet18, ResNet50.
13

---

Published in Transactions on Machine Learning Research (07/2025)
According to Zhang et al. (2019) generalized-smootf condition 2 has been observed to hold in deep neural networks. To further demonstrate the elfectiveness of TAN-SGD algorithm, we conduct experiments for training deep neural networks. Specially, we train ResNet18 and ResNet50 (He et al. 2016) on CIFARIO Dataset (Krizhevsky] 2009) from scratch. We resize images as 32 x 32 and normalize images with standard derivation equals to 0.5 on each dimension. At the beginning of each algorithm, we fix the random seed and initialize model parameters using the Kaiming initialization (He et al. 2015). We compare our algorithm with several baseline methods, including SGD (Robbins & Monro 1951), Adam (Kingma 2014). Adagrad (Duchi et al. 2011a), NSGD, NSGD with momentum (Cutkosky & Mehta 2020 and clipped SGD (Zhang
ct al. 2019).
For SGD; Adam and Adagrad, we utilize PyTorch built-in optimizer (Paszke et al. 2019 to implement training pipelines. We implement training pipelines for NSGD, NSGD with momentum, clipped SGD and TAN-SGD. The normalization constant computed through all model parameters at each iteration. The detailed algorithm scttings are as follows. For batch size, all algorithms use B 128, and B' 32 for TAN-SGD. For stochastice momentum moving average parameter, we set it 0.9, 0.99 for Adam, and 0.25 for normalized SGD with momentum. For clipping threshold used in clipped SGD and LAN-SGD, we set them as 2 and 8 = 1e_1_ The normalization power used for IAN-SGD is 3 = 2 We finc-tuned lcarning rate for all algorithms, i.., = 1e-$ for SGD, Adam and Adagrad, y 1e-1 for NSGD and NSGD with momentum, 7 = 2 for clipped SGD and TAN-SGD. We trained ResNet18, ResNet50 on CIFAR10 dataset for 60, 80 epochs and plot the training loss in Figure 3 Figure 3 (left) shows the training loss ResNet18, 3 (right) shows the training loss of ResNet50. As we can see from these figures, the (pink) loss curve optimized by TAN-SGD indicates fast convergence rate comparable with several baselines, including SGD, NSGD NSGDm, clipped SGD, which demonstrate the effectiveness of IAN-SGD framework.
Based on our experiments on non-convex phase retrieval, distributionally robust optimization (DRO), deep neural networks, and an ablation study A. we found that IAN-SGD performs best when the adaptive normalization parameter is set to 3 = 3. When 33 is too small, i.c., $ S 1 IAN-SGD struggles to converge; when 33 is close to 1, its performance resembles that of clipped SGD. To achieve fast convergence, the learning rate for IAN-SGD should lie between 1e and 2e 2e-1, Users should carefully fine-tune this parameter, overly large learning rates may lead to divergence.
Regarding the numerical stabilization term 8, our ablation study (see Appendix A.3) shows that IAN-SGD is empirically insensitive to its value; small value such as le is sufficient to ensure convergence. Finally, to balance stable performance and fast convergence, ablation results (see Appendix A.2) suggest that user should set batch size B 2 16 and the independent sample batch size |B'| should be at least one-quarter of batch size B
8 Conclusion
In this work, we provide theoretical insights on how normalization interplays with function geometry, and their overall effects on convergence. We then propose independent normalized stochastic gradient descent for stochastic setting, achieving same sample complexity under relaxed assumptions. Our results extend the existing boundary of first-order nonconvex optimization and may inspire developments in this direction. In the future, it is interesting lo explore if the popular acceleration method such as stochastic momentun and variance reduction can be combined with independent sampling and normalization to improve the sample complexity.
9 Acknowledgements
We are grateful to the anonymous reviewers and Dr. Ziyi Chen for insightful comments that led to several improvements in the presentation and technical details. This research was supported in part by the Air Force Rescarch Laboratory Information Directorate (AFRI/RT), through AFRL/RI Information Instituto0, contract number FA8750-20-3-1006 The work of Yufeng Yang and Yi Zhou was supported by the National Science Foundation under grants DMS-2134223, ECCS-2237830. The work of Shaofeng Zou was supported
14

---

Published in Transactions on Machine Learning Research (07/2025)
in part by National Science Foundation under Grants CCF-2438429 and ECCS-228832/CAREEE)) The work of Yifan Sun was supported in part by the ONR grant W91 29911092221-0229.
References
Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199(1):165-22 214, 2023.
Kumar Ajay Arshi. Life expectancy data. Kaggle, 2017. URL sttps://wwi http:/twwwwgg.c.cc/o com/datsset// www..//.ggggll..oa/aaaasssts/ee/
kumaraj www.aef-eeeecccccccyoooooeeeeeee
El Mahdi Chayti and Martin Jaggi. A new first-order meta-learning algorithm with convergence guarantees.
arXiv preprint Xrx:222009 03682, 2024.
Ziyi Chen, Yi Zhou, Yingbin Liang, and Zhaosong Lu. Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang, Robustness to unbounded smoothness of generalized signsgd. Advances in neural information processing systems, 35: 9955 9968, 2022.
Ashok Cutkosky and Harsh Mchta. Momentum improves normalized SGD. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2260 2268. PMLR, 13-18 Jul 2020.
Jan Drenth. Principles of protein X-ray crystallography. Springer Science & Business Media, 2007.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121-2133 2011a.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011b.
Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex optimization via stochastic path integrated differential estimator. In Advances in newral information processing systems, 2018.
Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive sgd. In The Thirty Sirth Annual Conference on Learning Theory, 2023.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. STAM journal on optimization, 23(4):2341 126226 2013.
Xiaochuan Gong, Jie Hao, and Mingrui Liu. An accelerated algorithm for stochastic bilevel optimization under unbounded smoothness. In Advances in Neural Information Processing Systems 37, 2024a.
Xiaochuan Gong, Jie Hao, and Mingrui Liu. nearly optimal single loop algorithm for stochastic bilevel op- timization under unbounded smoothness. Proceedings of the 41st International Conference on Machine Learning, 2024b.
Eduard Gorbunov, Nazarii Tupitsa, Sayantan Choudhury, Alen Aliev, Peter Richtárik, Samuel Horváth, and Martin Takáč. Methods for convex (lo, -mmoooh optimization: Clipping, acceleration, and adaptivity. arXiv preprint Xrx::22409. 14989, 2024.
Jie Hao, Xiaochuan Gong, and Mingrui Liu. Bilevel optimization under unbounded smoothness: A new algorithm and convergence analysis. In 12th International Conference on Learning Representations, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 1026-1034, 2015.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern vecognilion, pp- 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Yusu Hong and Junhong Lin. Revisiting convergence of adagrad with relaxed assumptions. In The 40th Conference Uncerteinty in Artificial Intelligence, 2024.
Florian Hübler, Junchi Yang, Xiang Li, and Niao He. Parameter-agnostic optimization under relaxed smooth- ness. In Proceedings of The 27th International Conference on Artificial Intelligence and Stalistics, 2024.
Wei Jiang, Sifan Yang, Wenhao Yang, and Lijun Zhang. Efficient sign-based optimization: Accelerating convergence via variance reduction. arXiv preprint arXic:2406. 00489,, 2024.
Jikai Jin, Bohang Zhang, Haiyang Wang, and Liwei Wang. Non-convex distributionally robust optimization: Non-asymptotic analysis. In Advances in Neural Information Processing Systems, 2021.
Diederik P Kingma. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.69000 2014.
Anastasia Koloskova, Hadrien Hendrikx, and Sebastian Stich. Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In International Conference on Machine Learning, pp- 17343- 173633 PMLR, 2023.
Alex Krizhevsky, Learning multiple layers of features from tiny images, 2009.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advonces in Neural Information Processing Systems, 2020.
Guoyin Li and Ting Kei Pong. Calculus of the exponent of Sardykka-lojsiewiice incquality and its applications to linear convergence of first-order methods. Foundalions of compulational mathemalics, 18(5): (8(5):1199-1233) 1232, 2018.
Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. In
Advances in Neural Information Processing Systems, 2023.
Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and non-convex optimiza- tion under generalized smoothness. In Advances in Neural Information Processing Systems, 2024.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over parrametereeizee non- lincar systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-116, 2022a.
Mingrui Liu, Zhenxun Zhuang, Yunwen Lei, and Chunyang Liao. A communicttion-cffiiennn distributed gra- dient clipping algorithm for training deep neural networks. In Advances in Neural Information Processing Systems, 2022b.
Pengfei Liu, Xiang Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting mothods in natural language processing. ACM Computing Surveys (CSUR), 55(9):1 35, 2023.
Jianwei Miao, Pambos Charalambous, Janos Kirz, and David Sayre. Extending the methodology of ray orystallography to allow imaging of micrometre-sized non-orystalline specimens. Nature, 400(6742): 342 344, 1999.
Aaron Mishkinn Ahmed Khaled, Yuanhao Wang, Aaron Defazio, and Robert M. Gower. Directional smooth- ness and gradient mothods: Convergence and adaptivity. arXie preprint arXiv:2408.04088 2024.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, orXiv preprint 150::080300999 2018.
16

---

Published in Transactions on Machine Learning Research (07/2025)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, higl-performance deep learning library. Advances in Neural Information Processing Systems, 32:8026 8037, 2019. URL https://pytorca: wwtg/oog///ooooooooooooooo htt////s/saabl//ppp.aae html
Amirhossein Reisizadeh, Haochuan Li, Subhro Das, and Ali Jadbabaie. Variance-reduced clipping for non- convex optimization. arXiv preprint trXXe:2333.00883 2023.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400 407, 1951.
Kevin Scaman, Cedrie Malherbe, and Ludovic Dos Santos. Convergence rales of non-convex stochastic gradient descent under generic lojasiewiez condition and local smoothness. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 17 23 Jul 2022.
Tao Sun, Congliang Chen, Peng Qiao, Li Shen, Xinwang Liu, and Dongsheng Li. Rethinking sign training: Provable nonconvex acceleration without first- and second-order gradient lipschitz, 2023a.
Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of SIGNSGD under weaker assumptions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 33077-33099. PMLR, 23-29 Jul 2023b.
Daniil Vankov, Angelia Nedich, and Lalitha Sankar. Generalized smooth stochastic variational inequalities: Almost sure convergence and convergence rates, 2024a.
Daniil Vankov, Anton Rodomanov, Angelia Nedich, Lalitha Sankar, and Sebastian U. Stich. Optimizing (sooth)smooth functions by gradient methods, 2024b.
Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of adagrad for non-convex ob- jectives: Simple proofs and relaxed assumptions. In The Thirty Siath Annual Conference on Learning Theory, 2023.
Bohan Wang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, and Wei Chen. On the convergence of adam under non-uniform smoothness: Separability from sgdm and beyond, 2024a.
Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi- Ming Ma, Tie- Yan Liu, Zhi QuanQuan Luo, and Wei Chen. Provable adaptivity of adam under non-uniform smoothness. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024b.
Wenhan Xian, Ziyi Chen, and Heng Huang. Delving into the convergence of generalized smooth minimization optimization. In 12th International Conference on Learning Representations, 2024.
Chenghan Xie, Chenxi Li, Chuwen Zhang, Qi Deng, Dongdong Ge, and Yinyu Ye. Trust region methods for nonconvex stochastic optimization beyond lipschitz smoothness. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024a.
Yan-Feng Xie, Peng Zhao, and Zhi-Hua Zhou. Gradient-variation online learning under generalized smooth- ness. anXiv preprint arXiv:2408.00 09077, 2024b.
Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non- convex optimization. In Advances in Neural Information Processing Systems, 2020a.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2019.
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Jingzhao Zhang, Sai Pranceth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383 15393, 2020b.
Qi Zhang, Peiyao Xiao, Kaiyi Ji, and Shaofeng Zou. On the convergence of multi-objective optimization under generalized smoothness. arXiv preprint wxXi:2445.19444/ 2024a.
Qi Zhang, Yi Zhou, and Shaofeng Zou. Convergence guarantees for rmsprop and adam in generalized-smood non-convex optimization with affine noise variance. arXiv preprint arXiv:2404.04456 2024b.
Yi Zhou, Huishuai Zhang, and Yingbin Liang Geometrical properties and accelerated gradient solvers of non-convex phase retrieval. In 2016 54th Annual Allerton Conference on Commumication Control, and Computing (Allerton), pp. 331-335, 2016.
18

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Appendix Table of Contents
Introduction
1
2 Related Work
3
Notations
4
Genoralizad-Smmothh nonconvex Optimization
5
Adaptive Gradient Normalization for Deterministic Generalized-Smmoot) Optimization
6
6 Independenty-aand-kaeptvvvi Normalized SGD for Stochastic Generalizea--Smoohh Op-
timization
8
Experiments
12
Conclusion
14
Acknowledgements
14
A Ablation Study
20
B Proof of Descent Lemma
25
C Proof for Nonconvex Phase Retrieval
25
D Proof of Descent Lemma under Generalized PL condition
27
E Proof of Theorem
28
F Proof of Proposition
32
G Proof of Lemma 2
33
Lemma 5 and Proof
33
Proof of Theorem 2
35
IAN-GGD under generalized PL condition
38
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
A
Ablation Study
In order to have comprehensive understanding of the performance of IAN-SGD in practical problems, conduct ablation studies regarding important components of IAN-SGD, including adaptive normalization 9, batch size [B| of independent samples and numerical stabilizer 6.
A.1 Effects of B
To justify the advantage of using adaptive normalization, we conduct the following two experiments.
In the first experiment, we choose unified normalization parameter 2 for all the normalized methods, i.e, NSGD, NSGD) with momentum, clipped SGD, SPIDER and IAN-SGD. To guarantee convergence, we fine-tuned the learning rate accordingly, i.c., Y 0.03 for NSGD and NSGD with momentum, 7 0.05 for SPIDER, 0.17 for both clipped SGD and TAN-SGD. To make fair comparison, we keep other parameters unchanged. Figure (Ieft) shows the comparison of objective value versus sample complexity for the Phase Retrieval problem. It can be observed that, by adjusting 3 = 3. the objective value optimized by all algorithms decreases much faster compared with Figure? 2 This indicates that adaptive normalization can accelerate convergence. Moreover, compared with other normalization methods, although TAN-SGD requires additional sampling at each iteration, the training loss still decreases faster than NSGD SGD with momentum and SPIDER.
Similarly, for DRO, we unify the normalization parameter of all the normalized methods, i.c., NSGD, NSGD with momentum, clipped SGD, SPIDER and IAN-SGD with 3 = To guarantee algorithm convergence, we fine-tuned learning rate accordingly. For NSGD, NSGDm and SPIDER, we keep learning rate unchanged; For clipped SGD, and IAN-SGD, we set = 0.08. To make a fair comparison, we keep other parameters unchanged. Figure (right) shows the comparison of objective value versus sample complexity. It can be observed that, by setting |3 3 the objective value optimized by all normalization methods decreases faster than Figure 2 This verifies the effectiveness of adaptive normalization. Although LAN-SGD requires additional sampling, it converges faster than other normalized methods.
In summary, the results in Figure indicate that independent sampling and adaptive normalization doesn't 't increase the sample complexity to find a stationary point, which justifies TAN-SGD framework's effectiveness when dealing with nonconvex geometry characterized by generalized-smoolll condition.
10
NSGD
NSGD
NSGDm
6.0
NSGDm
10a
Clip SGD
Clip SGD SPIDER
SPIDER
5.5
IAN-SGD
IAN-SGD
R22)
10a
5.0
4.5
10a
4.0
10a
0.0
0.5
1.0
1.5
2.0
2.5
0.2
0.4
0.6
0.8
1.0
1.2
Sample Complexity
le4
Sample Complexity
1e5
Figure 4: Advantage of using adaptive normalization normalized first-order algorithms
In the second experiment, we vary 3 for IAN-SGD and keep other parameters unchanged. Figure shows the convergence result with different for Phase Retrieval and DRO. It can be observed that decreasing B accelerates convergence in general. However, small can make convergence unstable. In the DRO experiment
20

---

Published in Transactions on Machine Learning Research (07 (075)
shown in Figure (right), observed that when 3. the objective value curve fluctuates lot. If B smaller than 8. the algorithm fails to converge. This phenomenon matches the implications of Theorem where 3 must satisfy 3 E [a: 1] to guarantee convergence.
10a
IAN-SGD with B = 2
7.0
IAN-SGD with P
104
IAN-SGD with
6.5
IAN-SGD with B 1
IAN-SGD with 10
6.0
IAN-SGD with B
IAN-SGD with = 13 20
IAN-SGD with B I0
10a
5.5
IAN-SGD with
IAN-SGD with B
5.0
102
4.5
4.0
104
3.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Sample Complexity
1e4
Sample Complexity
1e5
Figure 5: Effocts of adaptive normalization on convergence of IAN-SGD
A.2 Effects of Independent Samples' Batch Size
In this section, study the effects of batch size for independent samples used in IAN-SGD. For Phase retrieval, we keep [B] = 64 and other paramcters same as section A.1 and we vary independent batch sizes to be [B = (4. 8, 16, 32, 64). Similarly, for DRO, we keep [B| 128 and other parameters same as section A.1 and we vary batch size of independent. samples BB| = {16, 32, 64, 128).
The following Figure 6 shows the convergence of IAN-SGD under different batch size choices for Phase Retrieval (left) and DRO (right). can be observed that for Phase retrieval and DRO problem, small batch
10
IAN-SGD with B' 4
IAN-SGD with B' =
IAN-SGD with B' 8
6.0
IAN-SGD with B' = 16
10a
IAN-SGE SGD with B' 16
IAN-SGD with B' = 32
IAN-SGD with B' 32
5.5
IAN-SGD with B = 64
IAN-SGD with B 64
ANNSGG with B' = 128
10a
5.0
4.5
10a
4.0
3.5
0.0
0.5
1.0
1.5
2.0
2.5
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Sample Complexity
le4
Sample Complexity
1e5
Figure 6: Effects of independent samples' batch size convergence
sizes such as [B'| {4,8) are sufficient guarantee convergence, whereas large batch sizes of B'| do not significantly increase the sample complexity, which verifies the effectiveness of independent sampling.
21

---

Published in Transactions on Machine Learning Research (07 (07/2025)
A.3 Effects of
In Theorem
we set 8 I-
12_ based on Assumption which assumes an affine bound for approximation [- This assumption relaxes the strong assumption used in Zhang al. (2019); Liu
error fe W
et al. 2022b Zhang et al. (2020a), where they assume that the approximation error [[V]{(w) V
is upper bounded by constant. The major weakness is that if certain samples have outlier gradients, such constant upper bound can be very large. Thus, to verify the effectiveness of Assumption convergence of IAN-SGD under a wide range of 6, especially for small
we expect
To study the effect of 8, we keep other parameters the same as mentioned in Section A.1 and only vary For Phase Retrieval, vary {le-h -,e- 1e_1_________ vergence result. For DRO, vary 8 fle 1e-3, 1._e,,_ 10} and Figure (right) shows the convergence result. We observe that TAN-SGD convergence is robust to the choice of 6. But 5 le-3; le-1 demonst faster convergence than others for Phase retrieval and DRO experiments. This result indicates that small is sufficient to guarantee convergence, which verifies the effectiveness of assumption and IAN-SGD when dealing with stochastic nonconvex generalized-smooe geometry.
10) and Figure (left) shows the corresponding con-
10a
IAN-SGD with 5 le
IAN-SGD with 6 le
IAN-SGD with 5 le
6.0
IAN-SGD with 6 le
104
IAN-SGD with 5 1e 1
IAN-SGD with 6 1e
IAN-SGD with 5
5.5
IAN-SGD with 6 1
IAN-SGD with 5 10
IAN-SGD with 6 10
103
5.0
4.5
102
4.0
10
3.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4 1e5
Sample Complexity
1e4
Sample Complexity
Figure Effects of on convergence
A.3.1 Effects of Tı, 72 on convergence
In Theorem we A - 72. To further investigate the effect of the gradient noise assumption, we perform experiments that allow flexible choices of тΙ and T2. Specifically,
1 I-7 and ò - making the sample complexity dependent on both т₁ and
control the stochastic gradient noise by computing gradients over mini-batch samples with B = B' = 16 at each iteration. We then evaluate TAN-SGD under various noise settings with E {0, 0.5, 0.8} and 72 E {10-8 10-3 1,
10). As a result, the scaling factor becomes A E (,, and we define = A. {10-5 10 3 10-1, 10] accordingly. For phase retrieval, we use a unified learning rate 7 11- and for DRO, we set 7 -3 Other parameters, such as the clipping threshold and 3, remain unchanged. Figure plots the results, where the first row shows TAN-SGD's convergence for phase retrieval, and the second row shows IAN-SGD's convergence for DRO.
from these figures, LAN-SGD converges slightly slower as тi increases for both the phase retricval and DRO problems. This observation aligns with Thcorem 2 which shows that when > 0, the resulting complexity slightly worse than in the case nI = 0. Regarding 72, although the final convergence intervals remain approximately the same, we find that smaller values lead to less stable convergence for phase retricval. For DRO, the variation in T2 does not significantly affect convergence when learning rate is unified. This might be because for the DRO experiment, the model initialization and data processing have reduced the noise of the stochastic gradients, enabling convergence with flexible choices of 72.
As
22

---

Published in Transactions on Machine Learning Research (07 2025)
107
10a
10
---.5.1--3 1
T-00. 11-8
050.5
0.0.8
10
11 -0 1e-1
10
0. 1e-1
10"
0.0. 1e-1
Ti 0,
7j 0.5.
T1 - G.B,
201
10"
101
10
0.0
0.7
04
066
0.
1.0
0.0
02
0.4
08
0.8
10
0.0
0.
0.4
0.6
on
1.0
Sampln Complesity
Sample Complechy
Sample Complesty
165
Y 11-88
0.8,8
0. -1-
6 0.
le-3
0.0.
0,
Fi D.5,T)
G.B.
0.
0.5. 10
m 0.0.
3.0
Sample Complexity
165
Sample Complexiy
165
Sample Complexity
165
Figure 8: Effects of 72 for TAN-SGD convergence
A.4 Ablation Study on Extreme Small Batch Size B"|
In this subsection, we study the effects of batch size in IAN-SGD. Specially, we want to justify if IAN-SGD works with B, B' (1)1 practice. For phase retrieval, we set. B B' and keep other parameters the same as in section A.1 For comparison, we vary batch sizes to be {1, 8, 16, 32). To guarantee algorithm converging to same level of magnitude, fine-tune learning rates accordingly, i.c., for cach batch size, set 2e- 3, 4 10100, 0.10]. respectively. Similarly, for DRO, we set [B] = |B'| and keep other
y = 1, parameters the same as section A.1 For comparison, we vary batch size to be {1, 2, 4, 8, 16, 32), and we fine- tuned learning rates accordingly, i.e., for each batch size, we set { 1e- le 5e-3 5e-3, 1e="), respectively. Figure D
plots the convergence of IAN-SGD under different batch size choices for Phase Retrieval (left)) and DRO (right). We conclude TAN-SGD converges under constant-level batch size B, B' 2(1).
10a
9.5
IAN-SGD B, B' 1
IAN-SGD with B'
IAN-SGD 2
IAN-SGD with B' 2
9.0
104
IAN-SGD B, 4
IAN-SGD with B'
IAN-SGD B, 8
8.5
IAN-SGD with B' 8
IAN-SGD B, 16
IAN-SGD with B' 16
10a
IAN-SGD B, 32
8.0
IAN-SGD with B, B' 32
102
7.5
7.0
101
0.0
0.2
0.4
0.6
0.8
1.0
1
2
4
5
6
Sample Complexity
le4
Sample Complexity
1e3
Figure 9: Effectt of batch size on convergence for IAN-SGD
Next, we compare TAN-SGD with other baseline mothods under small batch size. To make trado-off between convergence and stability, we set B 2 for SGD, NSGD with momentum, Clip-SGD and SPIDER; For SPIDER am (Fang al. 2018 set epoch size to be 20. For phase retrieval, we fine-tuned learning
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
rate for all algorithms, i.., 7 1e-5 for SGD, y= 1e-2 for NSGD, NSGD with momentum, 7 5e for clipped SGD and IAN-SGD, 7 5e for SPIDER. Similarly, for DRO, set [ 2 for SGD, NSGD with momentum, clipped SGD and SPIDER. We fine-tuned learning rate for all algorithrms, i.e., SGD, 7 = le -3 for NSGD, NSGD with momentum, clipped SGD and IAN-SGD, y = 5e -3 for SPIDER. We keep other parameters the same as Section A.1 The first row of Figure 10 plots the comparison results of Phase Retrieval (Ieft) and DRO (right). We find in both problems, normalized methods such as NSGD, SPIDER, clipped SGD are slow to converge under small batch size.
le for
To further justify the effectiveness of IAN-SGD under small batch size, also test LAN-SGD and other bascline methods using small batch size to train RosNet (He et al. 2016) over CIFARI0 Krizhevsky 2009 For all algorithms, we unify the batch size B = 32. We fine-tuned learning for all algorithmss, i.e., 7 = le for SGD, 7 le for Adam, = 1e- for Adagrad, 7 5e 2 for NSGD and NSGD with momentum, 7 = 2 for clipped SGD and TAN-SGD, the second row of Figure T0 plots the corresponding results for training ResNet18 (left) and ResNet50 (right). As a result, we found IAN-SGD still outperforms than most baselines. This verifies the effectiveness of using IAN-SGD with small batch size in practice.
105
SGD
9.0
SGD
NSGD
NSGD
104
NSGDm
8.5
NSGDm
Clip SGD
Clip SGD
103
SPIDER
w(x)
8.0
SPIDER
IAN-SGD
JAN-SGD
7.5
10a
7.0
10
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Sample Complexity
1e4
Sample Complexity
1e4
40
40
SGD
35
35
Adam
30
Adagrad
30
25
NSGD
25
NSGD
Loss
NSGDm
Loss
NSGDm
20
20
Clip_SGD
SGD
15
IANSGD
15
Adam
Adagrad
10
10
Clip_SGD
5
IANSGD
0
0
10
15
10
15
20
Epochs
Epochs
Figure 10: Comparison of LAN-SGD with other baselines under small batch size
24

---

Published in Transactions on Machine Learning Research (07 (075)
B
Proof of Descent Lemma 1
Lemma 1 Under Assumption
function f satisfies, for any w, w/ Rd
f(w) f(w') + (Vf(w'), w') + 24 Li VillV w'll
(3)
Proof 1 Denote we = Ow' (1 O).. By the fundamental theorem of coleutus we have f(w') f(w) So (VS(wa), w' w)d0, which implies that
f(w') f(w) (Vf(w), - w) S (VS(wa), w' -w)d0 -w) L. (VS(w), - wwdd0,
Subslituling the generalized-smool condition in Assumplion into the above equation, we have
f(w') - f(w) - (VS(w), w/ w)
(Vf(wo), w' - w)d0 [.' (V.(w), w' w)d0
(Vf(wo) vf(w), w/ w)d0
0
[V/(w)) Vf (w) w' w||do
(6)
B(L + 1 V[(w) ||) w/ wl|'d0
(Lo [I|\ w||
(23)
where (i) applies Caucky-schwaaz schwarz inequality on (V)(ws) - Vf(w), wy w)) and (ti) is due to Assumption I ( I (Lo + L1 [V))] - w|l = A(Lo + [ w|). Re-organizing the above inequllity gives the desired resull.
C Proof for Nonconvex Phase Retrieval
The proof of generalized-smooll property for nonconvex Phase retrieval problem is similar as proof in Chen et al. 2023) with minor changes. We present the proof details here for completeness.
Proof 2 The objective function of phase retrieval problem can be expressed in the following form
2 F(w) Σ Se(w)
1
Σ (uE 1o w/ w
(24)
2m
EED
SED
where fe(w) = 1 [ue - laaTul 2)
We prove that for every sample &, (w)) - (ve la w/ ( (açaE T)w satisfies equation with thus F(w) satisfies finâte-sum versions of equation namely
3/. And
2
[e(( - fe(w') S w w'll (Lo +
[fe(w)] + [VFf(('))l
(25)
m7
2
SED
CED
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
We construct lower bound of || [uw)]l] given and & following
V [ee(w)]]
= 11(1 _ w ME) (aeug) T )u]l
[1( 0g w
2 ae
ye w) ag)
@
| w
ue as w ||a||
(10
(a T w/ 2 |oe wl
606
(1a1) 2 (1a} wl 2 - |Bs [aa]|
(26)
where (1) extracts scalar |a] wl - ve lag w/ out; (ti uses inequality Ja - 61 2 la|3 - 1b// holding for
any a E R (sub-addilive properly of function ((x) x2/3);; (bii applies young 's inequality live $ 1ug w S
3 allag w|l + +P3++blll. Bluel.
Then, for any w. w' E ERA, we have
[Ve' - V
= =I(1af w' -- uz) (aea)) )o' (1aFwu) wl2 yz) (aça )ul
2 |12 w/( yz) (aea )u' 2(125 w 2² - ut) (acae ww= + (1a3 T wd12- ve) (aça)) )w- (1a) wl2 - we) (aga )w'
1
- -(1a w¹ - uz) (aea) )w + (la) T w/ 2 - vz) (acae) Jw'll
2 |ll((E wd12 - ve) (acal) wu' ( | w'( | - -we) (aça)) ) w + (1a) T w/ - (aça) )w' - (1 wl - ye) (aea T )w
+ (1ag w/ 1-- - uE) (aeat) )u' + (1ag w' 2 --==- ux) (aeaş) ) w - (1a5 w 2 - ue) (ae Juo (1aF w - we (aea wull
(1( T 2 + |aद w| 2ye) (a)a)) )) (w' w) + (1aF wo1 2
|aل w/ (3) (aEa5) (w' + w)ll
(
(1 a1 w012
+ +1ae ul2 2 -2ee) (a
T) ) (w0' wIll + al w/1 of w/ (aEaE)
v ) (w' + +w)l
2
11 101 I1
+ +1 w/2 200l-2200 22ye) (aeag) ) (w/ wi)l + 2 |1( w/1
- |a wll ag (a w/ + ag w)
(0)
2 11(155 1012 + Iop w/? - -2uc) (aca)) = (no - -w)ll +
2 [1 w/12 - -ooTw] ( Jae (103 wo1l + |a ||
(4)1
| (1a5 w/( | + la wol + 2/vxl) |w' - w|ll + 2 ||<|| (1a5 w'l + w/(w)) 1 || w|ll
S 2
(e)1
||w0 w |||| [el (3/ag w/1 2 + 3/ w1 vel
2
||w0 w [al
||a|||
(3 ag w0 2
+ 3 0 w/ 1 - 3//ys| - -3/ucl + 8lvel)
WV w w|ll
(2
mmaax [V (wx")
4 F
9
V V
38 + 4yymax (aaxx))?xx)) (max))
(27)
where (i) uses triangular inequality; (ti) uses the fact aFw lag w|; (ki)) uses the fact Iaaaa ||
||a=||
and equation 28 (iv) uses (la w'1 + +aa wl) & 2/a w22 + 2 a wl?; (v) uses equation 26 and denotes
Umax max livel and Omax max |aell-
The inequalily used in (iti) is constructed as following
||aا wd12 - -1o =(1a wi| + | w/) ( ao w/ - oo w/)
5(1a} w/| + |a w/ ))||a (w' - w)
$|| [1(1aT will + 1o w wu w|
(28)
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Thus, fe(w) = (3: - w2 satisfies equation with a = 3 Lo M and L1 3 2
amax. Thus,
F(w) satisfies equation 25 with = 3. Lo Symax mmxx and L1 90
D Proof of Descent Lemma under Generalized Pk condition
Lemma in Chen et al. (2023). For any x 2 0, Ce [0, > 0, and S w S w' such that A 2 w/ w,
we have the following inequality hold
Cx S x + C
(29)
The proof details for this lemma can be found Chen al. 2023 Lemma at Appendix.
Lemma 4 (Descent Lemma under Generalized PL condition) Let Assumption and 2 hold. Apply
AN-GD with 3 E (ax, 1]. Choose targel error < 6 S min 1/2u]. Let step size 7 8(1++1)++1 A, f(w)) f*, then we have
(2pe) 3re
Denote
2
A S S..
7(2p)
2 A
2-8
+ (2 200, Vl. 4
(30)
Proof 3 We consider two complementary cases a > and a a=
When a > O, we have 3 a > 0, and by the descent lemmal we have that
- f
(
S V (w:)) (1i+) - -wi) + 2 (To + [] 12) /+++1 - -wel?
>/Vf((wx) 2-- ? (2Lo7 V || + 2L17 [V 2+ 20)
(8))
K > 2- + 7 (2 Vf (w:))) || 2- + (2LoM) 3 + +2Lr)) 3 3-1)
4
(4)
S 27 V
1|2 + 4
(2L0 + 221) 8-1
2
(2
( 1|=
(88LL + L1) + 1)
(8(Lo + L1) + 1)²
(m)) ? (1x)))
+ (2ue)² (2uE)
8(Lo + In) +
? (2 20
(31)
where (i) follows from the descent lemma 7 (ii) follows from the AN-GD updule rule by replacing Wi+1 w.
with -7 /VV a (iii) follows from the technical lemma above proved in Chen et al. (2025) by letting and substituting equation 29 to 2Lo (2:)) and 2L VV (we))] 2 (Let = [|V](wa) || and C = 2LoY 2L C [0. because < 1/24);; (iv) follows from the fact that a" +b" < (a+ b)7 with T = 2/3 - 2 and a, h 2 0, (w) follows from the step size rule 7 (2ue) ////////////o////////////////a + Tn) +1);
V f(w))
w' 2 8, A
(vi) follows from the facts that 2Lg + 2L, S 8(Lo + L,)+1 and (2Lg + 2 (8(Lo + L1) +1) when
3 > 0.
27

---

Published in Transactions on Machine Learning Research (07 (07/2025)
When 0, the same argument applics if > 0. Next, consider the cose BB 0, and we have We obtain that
8l6+b1)4I
f(w(+1) - f(wr)
/ (e)) (we+1 -wi) - 2 Lo + wi)||w[+1 -well
(60 V ( )1² + 7 (2L0r [v](w=)]] 2217 [V(
4
= VJ (w) 112 + 72
(2Io + 21) V
4
1
1| [/m + 8(Lo +1) + (8(Lo + L1) + /)
(10)
|/V (w)) 112 (w,))(
// (wx) + [w)))) + 2 (2
2 3
(32)
where (i) follows from the descent lemma 1 with a = 0; (it) follows from the updote rule of AN-GD by replacing W++1 we with ((w) (Normalization term vanishes due B = 0); (tii) follows the fact that 2Lo + 2L, I 8(Lo + L1) + 1; (6v) follows from the fact 3 S and (v) holds by adding additional positive number (2pe))..
Thus, in conclusion, for any 8 E [a, 1], we always have the following descent lemma 2
f(w(41) - f(w)) S (1 #4 (2
(33)
(1p
under the step size rule 7 B.)++
Moreover, by the generalized Pt-condition in Definition
we have
Vf( 2 (f(w) -/))²
which implies that
| (wx) (wx) || 1|==0 (2pz) (f(wz) ) =(241
2-d
A,
(34)
Substituting equation
34 into equation 33 use have
((w++1) - [(w)) S 2 (2 2-8 (J(w)) - (³) 2-4
= + 2 (2 -
7
4
Subtracting f on both sides, we have
f(w fr f(w)) - - (2 2-8 (f(w) 8r) ) + 2 (2pc) 2
Denote A, = f(w)) - 8", the above inequality can be writtn as
A411 S A.
7(2p)
2-8
A
2-2
+ 2 (2
2,, Vt.
(35)
2
E
Proof of Theorem i
Thcorem 1 (Convergence of AN-GD) Let Assumptions 1 and 2 hold. Denote A := f(w() f as the
1/ function value gap. Choose target error 0 < S min(1, J. and define learning rate 8(oo+13)+F
< 2 for
some 3 E (a, 1]. Then, the following statements hold,
28

---

Published in Transactions on Machine Learning Research (07 (07/2025)
If < P, in order to achieve DT e, the total number of iterations must satisfy
T2 2 max {
(2- (2-00/1 (21--8-))(2-0) 1) (210((2-8-0)1)) -
8p(8(Lo + +1)
1
=00
(7)
1f P. order to achieve Ar S c, the total number of iterations must satisfy
21 8/p (8(LO + +1)+1) log( 2 = =0(())) log( An T>
))
(8)
1 (16))80
1f128>20 2- ps and for some To satisfying ATo S iterations after To must satisfy
221 ) then the total number of
+ log [og (32 L1) 4)p/ (22 ))) = 22 log ( (1))=T=))
(2n)2/(0t-22)
T T20 log
(9)
Proof 4 Given largel ertor < S min 1/2u), define slopping time as the first time that A. salisfies A c, E, i.e., T= A S e}. In the following proof, we provide on estimate of such stopping time T.
For any t satisfies A. > e, we must have A 1/-3)/0 (2pe) (2p)
since 2-8 > 0. This implies that (2-8)/0
p
(2- (2288//8//00 (2-8)/12 a Thus, equation 35
further reduces the following descent inequality
2-8
7(2p)
2-
A
(36)
For above equation 30 we provide convergence rate based on the value of p and B.
Case I: 3 < 2 p This is equivalent to inequalities
2-2 1. For simplicity of notation, denote 0 = p
2-2,
When 0 > 1, we have following
A4+1 SA.
A SA
A 8 24.0
(37)
1 where its derivative is (' -t We divide the last inequality of equation 37 into two different cases for analysis. One is the case when A, s A S 24, another is the case when A 2 2
Now define an auriliary function ((t)
8-1
When 0 A7H < 2 we have
A D'(1)dt=
A:
Φ (A++1) - ((A,))
A
A.11
(A. A441) 0
(1)
A 1+1
(A,
2
(8)
(2
ATH1
A
4
2
(ee) 724))
A
722))
0
4
1+
2
8
where (i) applies mean value theorem such that ((A - ((A,) = (E) and 4 2 (1(1) Z
A-" holds for any [A41, Ad); (ti) uses the fact A 24, " (tit) utilizes recursion A. A
r(2n)" A9; (ix) uses the fact that A > Ait1 for all 0 > 0.
29

---

Published in Transactions on Machine Learning Research (07 (025))
When A,41 2A,", it holds that A (A,+1)
-
- ! 00 Then, we have
1
(A (A,)
A 1-0 A - )
(1)
((2) - 1 1-
0
(4)
(12) 1ي - 1)
where (i) is due to the recursion 4 = (4
2 1 (ūi is due to the fact the sequence {A, km 2-8 in and denote
generated by equation 36 is non-increasing. Substitute
(2(2 2-8
P
2-リーp 2-8
C'=
2
-1)
2
B
We conclude for all t, we have
( (A,) $(Ao)
-
-
$(A,) 2 Ct.
==0
Substituting the expression ((A,) = 2-1 A into the above incquality, we have
P
2-P-8
A <
(38)
(2 3 p). Cct
In order to oblain AT Se, when C
2-3-p 1210-121222) (112-8-)/(2-0)) - -1)
p
(Ste-2)/2 T should satisfy
1
T
(39)
(21--3-01/2-0) 1) 10+ 100+--2)1 2)/0 (2-Dp(1-)
When C= r(2x)² a/e
3/ 2/0
T should satisfy
8p(8(Lo + L1)
(40)
2- 3- 322)
This concludes when p ++ 3 2, T should satisfy
T2 max (2- (2 (2 (2(2- (12-8-01(2--1) 1) 10+B 1-+0--11022 (1202222-2---11111 =0) (41)
8p(8(Lo (1)+ 1)
Case II: 2-p
This is equivalent to 2-8 = 1, equation 36 reduces to
A 5 A, 7/4
7/ A.,
A, (1
which leads to To obtain AT S we have
A S (1
7p
Do =0(11 ()
2
2
A, 51
7He
Do - <exp(=22
7ut
Ao
(42)
2
2
30

---

Published in Transactions on Machine Learning Research (07 (0/5))25)
To obtain A e, iteration complezity should satisfy
2 T2 7/
21-B/p (8(Lo + +1)+1) log( A 2((=) logg -
logl
μ.. (HE))8p
Case III: 12 8 2 p This is equivalent to 2-3 1.
For simplicity of notation, denote w I-z and C= (2n) Then, equation 36 can be rewritten as
A S S.. - C:
(43)
When De is small enough, 1/2 will dominate D++1 order-wisely since 1/w 1. This fact leads to
(10) C i1 S A + CyAit1 S A + CYAF S A.
(
(0)
where (i) because A++1 2 0; (ii) is because D4t1 S Du: (ii) because re- re-organization of equation 36
There exists a time To inf(t E N[Ar < <
such that for any To S t S T, we have
A < (C>) =(C)) w-
To
=(Ch) wll ol1-ut-20
To
=(Ch)wu/w ((C,)//--
To
wo
(44)
To simplify analysis, denote all porameters associated with Ô
(s(to+ wo/w-1
C(2))
We have
(Cr))/u-1= = - < C. This enables to further reduce the recursion to
A+t1 ((C))alo-11 ((07)a/0011)
To
<0((0))=iT
A
-o
Aro
(45)
To oblain AT e, above recursion should satisfy
((CC))= -at
Aut To
Taking logarithm on both sides and extracting e ( from (Cn) A we have
log(C/e) = T- log ((Cr)= w' log (IC/ATo) : S TTo -
Taking logarithm again on both sides of the above inequality, we have
B p+ 2 log ( log log (To + In] + 4 40/0009-2222
(2p)2/10+8B2)
To log 3-3
log
(32 Lo L1) + 4)p/(0+8-2) AT
+ log
(24))/(0+332))
20 log
8
log (t) + log [log (32(Lo L1) + 4)0/(ots
(2n)2/(0+8-2)
p+ 8+ 9-2
=52(log log
((2 )
31

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
F Proof of Proposition I
Proposition 1 (Convergence of GD) Let Assumptions and 2 hold. Assume there erists positive constant. G such that [VV((=))] & G, E ET, When a 1 and selting S min( Lg 2hi gradient descent converges to an c-stationary point within 2 ()) iterations.
Proof 5 When a = 1, pulling the updale rule of gradient descent W++1 Wt - (W)((w)) into descent lemma equation yields
f(w(+1)
2 ff(w)) [(l)) + 2
(Lo (lv
=f(w)) (7 - L 2
+ 2
[V[()))]]
f(()) 3 1 f((m))]]
L17
\f(
2
(00 f((w) 2 f[
(46)
where (i) is due to descent lemma equalion 7 (wi) and (iii) are due to the learning rale design y S minf to 24, ,, which ensures -(7 La>" ) S - and L f[wx]] I 2
When applying Assumption with p = 1 and denote f(w,) f as D we have
A+11 A. 2
(47)
The rest of proof is exactly the same proof for Theorem 7 Case I, omit discussion here. As result, one can show equation 47 converges to a e-stationary point after ( iterations.
32

---

Published in Transactions on Machine Learning Research (07 (025)
G Proof of Lemma 2
Lemma 2 Let Assumptions and 4 hold. The gradient
|| || [W+)ll and stochastic gradient Vfg(w)) satisfy
VF - (u)e(u)))l + 1-
72 and
s & (1+ (V+ +
(16)
Proof 6 Based on Assumption 4
we have
[VF(w) - (ee(l)) < T2,
which, by triangle inequality, further implies that
I (VF(w)) (uee(l)))l [+/]][([;)]]++++ + T2.
Rearranging the above inequalily yields that
[VF(())) S 1- (Vie + 1-
For the upper bound of (e: il direclly follows from Assumplion 4
H Lemma 5 and Proof
Lemma 5 For 2 (hot [F(()) (")
472, we have the following upper bound
228
(To + Ln [F(w:))(
4T I 2 2Lo + [1) + 472)2 + ? [F(Ww)]]
2 2?
(48)
4h
Proof 7 When VF & + (2T) +1), we have /F(w)))l - (1+ 47g + +1) for any
a >> O. Then, we have
22
(Lo + L1
VF(we)) 4T
2
47
1 (Lo L1)
21
472
27 +1
+1
h
2Lo Tn)(1 + 81 47
2
1
2Lo L1)(1 + +
(49)
where the first inequality uses /VF a < 4 + 1)) and (1 + 47 / (2T + 1))a/22 > 1; the
second inequality uses the fact that to < 1 (so does
I 지.
and upper bound 47 by 1 + 472; the last inequality
uses the fact that thus (1 + 14 < (1 472)2
33

---

Published in Transactions on Machine Learning Research (07 2205)
When [F(w:)) > v1+ 4Tp (2 + 1), we have
7.
L1 (F(w))))))) |
472
2
72
L V
.472
2h
h2
(4)
7
L1 VF 1
472
2hg
4 + 1)(
(w)(2)) + 12 )
I-n
72
T,1 V F (wrz)
473
2hd
4L17(
(1
V Ser (w=)
||
+ 1-7, 1-, (2T) + 1)
(16)
72
L1
V
F(w+))
2h
(4I17)
fee (wx)
+1=
12
-1
(11)
7²
L
/F
2h
(4L (F))))
72
1
[F(w))
2hg 4T,17|
VF (w:)
(6))
72
In
/F((u)]]
2h
4L17
[F(w))]]
(50)
8h
where (i) uses the fact 1 S 4l. (w (ti) uses the fact that now [VF(w;)]
1.2 + (2 + 1) 47 (2T + 1); (Mii uses the fact /F S IV 1 (fv) uses the fact
fe (we (wx))
[VF(w)) (w)) > v 7 4r / (27 + 1) N 1 and thuus [VF(we)] 1F-0
F-a < 1.
Similarly, when
VF(w:)]| >
+ 1), for 21?3 22 4T2, we have
1
2 Lo 472
2h23
7
Lo)) 4T2
2h
< (o) .
2h
472
2h 3 4 (277 + 1)
? 473
8h
3 27 +1
< (1+ 4T
8h
2T +
? |VF
(w
(51)
8h
where the first inequality uses the fact that the third inequality uses the fact 2,:
< 1; the second inequality uses the fact y S 4To((77++1))
4r3
4r3 25 the fourth inequality uses the fact that [[VF(w))]
<
$1+
+
473/ (2T + 1)- Combining equation 49
equation
50 equation 51 gives us the desired resull.
34

---

Published in Transactions on Machine Learning Research (07 2025)
Proof of Theorem 2
hold. For the LAN-SGD algo- rithm, choose learning rate Y = min (Lo(2)]+1) AL. (2r]+1)' vT (27 (r I, and A 1-7
Theorem (Convergence of IAN-SGD) Let Assumptions
3 and
= r (2 1)) Denole A == F(we) FM (To + [1 + 4T Then, with prob- ability at least 3. IAN-SGD produces a sequence satisfying min [w:)))) - if the total number of iteration T satisfies
256A 64L, (2T] + +) 2T (27 +1) 64(Lo + Li) + (2m
T 2 A max
{
(17)
(1- 2
e
Proof 8 By the descent lemma in equation 3 and the update rule of IAN-SGD in equation 14 we have
F 1( - -F(w))
< <VF(w)) (wi+1 - -w))
1 [[o (Lo + 11 (FF()) [= w woel
VF(w)) (Ve((ur) 7 + L, [=(:)(l)))
| fc (u)(())))
(52)
h
Taking expectation over E,e' and we, and by the independence between & and E', we have
En [[e.ee [F(we+1) (((x))/wx))
7
[F(w))]
1
Ee V (e (wx)) 100
< <Ewr
2 2 [[(w)))))
h
h
From Assumption 4
we have
Eg [[]
=Bg V fe((w) VF(w)) + VF Fw)))) [w]]
2E fe
VF(w;)) [ + 2/(:/(
(8)
3 2( (F(T://F(())) (1)) + T2)² + v
(188) (47 2) 24
4
(53)
where inequalities (i) and (ii)) are direct applications of b)² < 2a +b? and (ti) follows from Assumption
4 Substituting equation 53 into the above descent lemma, we have
E [Eg [F(w++1) (F))))F((())) [w:]]]w:]]
VF(w:)]1
Ve [( Ee we
(Tn + T/
hi
h
VF(w))1
1 7 + (
(47 + 2) V Fw:) !1? + +
<Ewie
h2
h
Lo + T, [ )) [VFF(w)))
1 1 To + In [VF + 2
-Evrr [(7g me (
(-
4r3]. (54)
ne
h 29
Next, we provide an upper bound of the term 7(27? 1) Lo [[] in the above equation. By the clipping structure and step size rule, we have 1 h
min {1, 4T.7
[ } s and
< (10(27+11)) which imply that
IL0 < (27 +
(2 + 1).
20 S
(55)
h
35

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
On the other hand, we have
(6)
7 2/6 1/
(47T2(227 +1) 1- TI
((e:(w:)) +
1- a
(11)
(4712 (2T) + +1))
1-7 1-) [v=e(w)))l
1
1- )
72
(10)
2 2 + vF(w)))
(56)
where (1) utilizes the fact h, 2 1 and B 2 a; (ti) utilizes the fact that h ( + 1)( Vfe (w,) +
P:;;; (iü)) utilizes the fact that 7 S (T.(27/+T)) thus (2 S (47 (2T) +1))a/8 since B E [a, 1];
(in)) follows from Lemma 2 Combining equation further reduces to
55 and equation 56 we conclude that 7(27 + 1). Loth [F(())]]
S } and equation 54
Eug
Fg,s (((1+1)) - F [ww))
Lo + L1 (F/(VF(())
Eurg,
? (F(e1))
+ 72
473
2
1120
(5) Fewee
7
|FF (w:)
12 + 1 - + T.T)(1 + 4T2)2
7
V
F(w))))
21
4nd
Euiis
7 (Lo + L1) + 41)(1+773)2)
(57)
4h
where (i) follows from equation 48 in the auriliary Lemma
Rearranging the above inequality yields that
E
7 VF(w;)
12
< Ew. Eg,e [F(w;) - F(w 1(we+))/uu)]
+
1 ² (Lo + (1) 473)². We [T] (58)
4h
2
36

---

Published in Transactions on Machine Learning Research (07/2025)
Next, we further lower bound [V((w)] by [[v)()))] to eliminote the randomness in h4 We have
VF(we) 1
y min {1,
V F (1x)
(41((TT + 1111111111
1- [
+ 2 1
(0)
7 min {
(217(2TT + +1(
71 +1
|F(ww)]]
+
2rz
1-
(10)
min {1,
1
(2T + |
42 + (2
(w) min {7,
(2T + 1)(20)) |[
/|
(2 + (2 VF Fw)(w)
min
{7, (27 + (2C V2 (www))))
3 }|
[VF ||²3
-min (oinn>//F((wx)) 4(2C1) (2T) 1
(59)
where (t) expands the expression of n. (ii utilizes the second inequality of Lemma 2 to upper bound (wx))) by (7 + F) + r2; (ii) utilizes the fact la+by= 2 mint (20)a: (26)s by setting a netl (W:)))) b 2 2 0; (ir) substätute & inside the monimurn operator. Moreover, (v) follows
min
from the slep size rule 7 = + 7))" S 11d2 Substituting equalion 59 into
equation 58 and summing over t from 0 to T P-11 1, we obtain that
(v+))))) 11 2-8
E
min 7
((()))
164, (2C,) (2T +
7 VF
Bui.s'
4h
T-1
<
Ewi
Eg.5' [(w))
F(w1+1))u) wi]
+ T 2 + L1)(1 + +473)².
1=0
By step size rule 7 <
/ the above inequality becomes
1
T
min (2 V (1))
(20, )a8 (2T? 11 s F(wo) - F" + (Lo + [1)
Exe,
t=
Denote K = {tlt E [T] such that >/ /[(x))]]
4T.(2 (2 }. then the above inequality implies that
[FF(W)
< F(w)) - F* + + + L1)(1 + +47))²
Ew,
(60)
YER
and
|v:))))
Σ EVg
(2C,) 1) < F(wo) - F" + ( + L1)(1 + 4r3)².
(61)
TeAe
37

---

Published in Transactions on Machine Learning Research (07/2025)
Now denote RHS as A == F(wo)
F + (Lo + L1)(1 + 4n,)², then we have
min
VF(w>)]]
tET
min
Σ
|
wi ||}]
K
1 [Ko]
EEK
LEK
(1)
E
min
VF(W)))1
(w))))) 2- x' [
(12
K
|K
EEK
SEK
(8)
max
VF
Buce
VF(w=)] 2-00) x
K
K
EEK
tEK
(181)
max
En
(1x)))
Evy,
( I
K
Koa
EEK
LEK*
(ie)
max {y (4A)
4(Lo + L17 + 1) vT + 8L, (2T + 1)(((22)/ -1))
32L, (2T + 1)( 2)/1 1 51) - (A
x
T
where (i) uses the concavity (b).) = y² and (b)) = y='d and Jensen's inequality 40( E. E. Waa) on |VF(())]]: (ti) uses the fact B[min B)] < E[A] < max{E[[]] E[B}; (iii) applies Jensen's inequality over concave function (r())= = y and (-) y= (to) is due to re-organize equation 60 equation 61 and set K = T, k (Since we require either K or K should be larger than 2 to guarantee that K + K T).
n applied
Thas, in order to find point satisfies
Pr(min [FF(w)] c) S 2 [E[T]
We must have Eu, (mine [[ S
2.
This indicates that the RHS of the above incquality should be smaller than 2, this implies T satisfies
T Z Amax 256A 64L, (27 1)(2 27 (27 + 1). 64(Lo + L1) (282
(62)
(1 - Ti )32--8
J IAN-SGD under generalized Pt condition
Lemma 3 (Descent incquality of IAN-SGD under generalized PL condition) Let Assumptions
0 and 4 hold. For the IAN-SGD algorithm, choose largel error < < min 1/2p] and learning rale y
(2rf+
(2ue) min{ sTo(2r +1j (1, (0127 (1,(27 11)-0/1-11)
and A = 8 = 1 r (4L + 1)) Then, we have the following descent inequality
(T, + (1)))((24))200001
Ewyys [1+1]] < Er, [A.
(20)
4
8
Proof 9 When chosen target error <
min (1, 1 and learning rate satisfy
2-20
1
1
S (2iue)
4T
(277
+
4lg 1)
7 - (2ue) 4-20
1
(2T +
+
4L, (2 +1)
6-28
? 3 (24t)
(63)
82 + (1)
(2T 1))(222 (1
38

---

Published in Transactions on Machine Learning Research (07/2025)
detailed dericalion here. the derivations from equation 52
to equation 58 stated in Proof G are still valid, omit discussions of
Next, we lower bound F
by [ to eliminate the rendomness in he For any 3 > a > 0, we
have
VF(w))) 1
lig
y min
(LAL((27? + ()x 1
V F (wx))l
[u:
12 ) 1-m
1-71
(10) y min {1.
V [F(W))]
(2r + +1)
+
V Fw:(w))l
+ 1-
2T2
1
C1
(06) y min {I,
1
} [F(ww))1
44 42 + (2 [/Fm:)))
(2 + (2(2))
(w) min {;:
1)(202) |[VF((m))]
4L7 +
(2T ()( VF(
min
{::
||/F((u))]
427 + || / VF(ut)))
[V 2-0
mmi (min(///V((=a)] L 1)
=
|V 2-0 |
(v4)
(m (2 +1), 4(2Cg) (2T +1)
VF 2-8
(mi))
2 min [|,
1), - - ((2C)) L1 (2 + 1)
(oi)
/F ||=
2 "))
4((C1) ( - 2 L +1)
+1)
(ix)
2
(2-30 1, (2T7 1)
(64)
where (i)-(u) follows the same arguments stated in Proof
G (mi) applics equation 29 by setting C (2,
178/2) (since [2?(2 1) < 1), x [
3, w w'= 2,w 2 3; (vii) follows from (2Tg +
1))8/2 (Li7(2TT + 1))2/2 as (L1>(27) + 1)) 1; (viti) sollows from fact for any 2 0, minfa b, c)
min(a,c} - (b;x)) follows from fact S (L. -2)0(=0)))+00++11 (111201)) Thus, LiT(2TT + +1)
and (T 1)) S S ((c,)77++11 always holds; (for) substitutes assumption to replace
[[F(()))] by 12
When 3 0, above argument still applies, next, consider the case when B = 0, we have
a Vl 2)
+
2 + -1) (2 +1)
-/3 + + +1),
(65)
where the first inequality is due to 5 [1(77711) and the last inequality substitutes assumption to replace [[:)(l by 120
39

---

Published in Transactions on Machine Learning Research (07/2025)
Combining two cases, conclude that for any B 2 we always have
VF(w))
2 (L, 1)) +
(66)
h
Substituting equotion
60
into equation
58
and re-arranging terms, we have
3/2 (Li (L) (27
12 (2,2,,))22-3
4
22 +
Bun,
[F(we) - F(w++1)]
+ (Lg + (1)7(144437)
2
-Bwi
[F(w]) F(w++1))
1 " L.)(1 + 472)²
L2(27) +
(67)
2
Substituting F both sides and re-arranging terms yields that
(L + (2),
Busgy [A+11] S Fan, [..
2 > + (T)++T))(1 + 4T3)²
+ ( +122))]
(2r (1
Since 7 L6 111771/313/ the above inequality further implies
(L2 (2)) (L2 + (2pe))))))))))
Ewsys [Ar+1] S Ew, [
(68)
which gives the desired resull.
40


Reviews:
[{"review_id": "IM6BOw1RzZ", "reviewer": "Reviewer_ijCU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strong points__\n\n0) See Summary Of Contributions\n\n1) The paper is coherent, easy to follow and well-written (aside from some typos noted below).\n\n2) It addresses an important problem of optimizing generalized-smooth functions which has broad practical applications.\n\n3) I found no major mistakes in the proofs.\n\n\n__Major issues__\n\n1) __Could the authors compare their results from Theorem 1 for $\\rho = 2$ with existing results for strongly convex functions, e.g., from Gorbunov et al. (2024), Li et al. (2024)?__ The authors also mention in Related Works results for strongly convex functions from Vankov et al. (2024b), however, I could not locate these. \n\nIn addition, how do the rates behave if $L_0 \\equiv 0?$\n    \n2) Throughout the paper, the authors say that they use weaker  almost surely scaling assumption 4 in comparison with (Zhang et al., 2019; 2020; Liu et al., 2022b). However, they do not mention modern works dedicated to __in expectation__ assumptions. For instance, there are Reisizadeh et al. (2023), Hübler et al. (2024), Jiang et al. (2024) for bounded errors or Chen et al. (2023), Zhang et al. (2024b) for scaling errors. \n\nI think that almost surely assumption 4 is very restrictive, since even additive normal noise does not fit it. Moreover, modern large neural networks tend to have rather heavy-tailed errors [1], making assumption 4 even less applicable. \n\n__Is there any way to prove the theorems under in expectation scaling assumption?__\n\n[1] Zhang, Jingzhao, et al. \"Why are adaptive methods good for attention models?.\" Advances in Neural Information Processing Systems 33 (2020): 15383-15393.\n\n\n3) In Theorem 2, the authors prove that their IAN-SGD can work with small batchsizes $B,B'$. In ablation study and experiments, they verify it for small $B'.$ However, they never take small $B$ and run all methods in experiments with the same large batchsize. \n\n__Could the authors test and compare IAN-SGD and competitors with small $B$?__\n  \n4) In Ablation A.1.2, the authors want to show that IAN-SGD is robust to gradient outliers in assumption 4. However, they do not alter noise parameters $\\tau_1, \\tau_2$, but vary hyperparameter $\\delta$ for the same problems. I do not think that it is effective. Firstly, the outliners remain the same for all $\\delta$. Secondly, in sum $2||\\nabla f_{\\xi_{B'}}(w_t)|| + \\delta$, gradient norm may dominate small $\\delta,$ resulting in the same IAN-SGD steps and final results.\n\n__I would suggest to conduct experiments with various noise parameters $\\tau_1, \\tau_2$  and  IAN-SGD with $\\delta = \\tau_2/\\tau_1.$__\n\n\n__Minor issues__\n\n1) I think it is worth mentioning  the related work dedicated to convergence of SignSGD with momentum under generalized smoothness:  \n\nCrawshaw, M., Liu, M., Orabona, F., Zhang, W., \\& Zhuang, Z. (2022). Robustness to unbounded smoothness of generalized signsgd.\n\n2) In Related works, the authors say \"establishes convergence rate $O(\\varepsilon^{-4})$ and corresponding lower bound\" and right after \"SPIDER algorithm (Fang et al., 2018) can reach the optimal $O(\\varepsilon^{-3})$\". It is better to mention that SPIDER is designed for sum-type functions.\n    \n3) The discussion after Theorem 2 should be written more carefully. For example, RMSProp, NSGD and SignSGD with momentum do not require large batchsizes too, and RMSProp even works under scaling assumption. \n\n4) In \\textbf{Algorithms for Generalized-Smooth Optimization} on 3rd page, reference Cutkosky \\& Mehta (2020) is misused. \n\n5) After \\textbf{Machine Learning Applications} on 1st page, there must be G instead of g.\n    \n6) 16th page, \"Figure 4: advantage of using\" typo.\n\n7) 18th page, \"But When\" typo.\n    \n\n__More questions__\n\n1) Do results still hold under weaker symmetric generalized smoothness Assumption from Chen et al. (2023)?\n\n2) Could the authors provide examples of Generalized Polyak-Łojasiewicz functions for $\\rho < 2?$", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I thank the authors for the submission. I am pleased to review this interesting paper. \n\nThe authors work with Adaptive Normalized Gradient Descent (ANGD) which achieve faster convergence by adapting to problem geometry. Firstly, the authors propose and theoretically justify a new fine-grained approach to adjust deterministic ANGD to generalized Polyak-Łojasiewicz functions (for instance, they extend ANGD theory to strongly convex functions). Secondly, for stochastic optimization, the authors develop a new modification of ANGD (IAN-SGD) to deal with bias of normalized stochastic gradient without large batches. It is based on independent sampling of norm to divide by. Beside optimal theoretical convergence, the authors also demonstrate superior performance of IAN-SGD in relevant real-life setups.  The experiments and ablation study are thorough and representative. They consist of generalized-smooth  Phase Retrieval problem, Distributionally-Robust Optimization and ResNet training on CIFAR10.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "__Verdict__\n\nThe paper proposes interesting and novel ideas for theoretical analysis and practical use of ANGD. These ideas are also supported by superior performance in relevant experiments. However, there are serious issues that undermine my confidence (see Strengths And Weaknesses). \n\n1) From a practical point of view, the authors do not check how IAN-SGD copes with various scaling errors and small batchsizes (for which it was designed).\n\n2) From a theoretical point of view, the proofs are given under almost surely error assumption 4 and asymmetric generalized smoothness, which are quite restrictive and have been relaxed in other papers.\n\n3) For better writing, I would recommend to fix typos from Minor Issues, mention all related works and add more details in discussions after Theorems 1 and 2.\n\nIf the authors address these concerns and polish the writing (fixing typos, expanding discussions, and citing relevant work), I would gladly recommend acceptance.", "broader_impact_concerns": "No ethical concerns - theoretical paper", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uLJZGmNelz", "reviewer": "Reviewer_4DJr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n- Studying AN-GD under the generalized PL condition is nice, I am not sure if it has been done before, but it is nice.\n\n- The noise assumption 4, although it needs to be satisfied almost surely, is interesting and has been considered in lots of prior work for SGD with noise bounded in expectation, it is indeed an interesting relaxation in this setting. The problem of doing this analysis with noise bounded in expectation was already done in prior work like [Koloskova et al. (2023)](https://openreview.net/pdf?id=C3DXiFTrve), other works such as [Zhang et al. (2020)](https://jkjin.com/publication/example/example.pdf) (you already cite this one) also claim that they did not do it only for simplicity.\n\n- The writing is good, ignore minor typos and grammatical errors (the authors should go through the paper and correct the obvious typos).\n\n**Weaknesses**\n\n- The novelty of the paper, in my humble opinion (based on familiarity with the setting of the paper), is very limited.\n\n- The overall organization (not the of the paper) is not good. I did not like how the PL condition was just ignored in the stochastic case. Also, theorem 1 works with both the precision $\\varepsilon$ and the number of iterations $t$, the authors should either present their results in terms of the complexity of the algorithm as a function of the precision or the error in terms of the number of iterations, mixing both makes it slightly difficult to understand the result.\n\n- The dominating term in Th2 is $\\mathcal{O}(\\frac{\\Delta + L_0 + L_1}{\\varepsilon^4})$, might (the assumption are slightly different) be worse than what was obtained in previous works such as [Zhang et al. (2020)](https://jkjin.com/publication/example/example.pdf)  for which this term does not depend on $L_1$, anyways, this needs to be discussed.\n\n- Proof 7 is wrong (it is the first one I checked). You claim that the convexity of squared $L_2$ norm gives something like $\\|x + y\\|^2 \\leq \\|x\\|^2 + \\|y\\|^2$ is wrong, you should use the expectation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies normalized/clipped SGD under the generalized smoothness assumption where smoothness can grow with the gradient norm. The main novelty of the paper is proposing to use two independent batches, one for the gradient and the other one for its norm, doing this simplifies the analysis of normalized/clipped SGD.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Change Th1 to only work with the precision $\\varepsilon$ to stay consistent with Th2.\n\n- Discuss why you did not consider PL in the stochastic case. Again, you should be consistent throughout the paper.\n\n- Include more prior work using the generalized smoothness assumption (I already provided two) and most importantly discuss what do you bring new compared to what is in the previous works (especially the ones I cited concerning the noise assumption and the dependence on $L_1$).\n\n- Go through the proofs and make sure everything is okay.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "afd6hwDctT", "reviewer": "Reviewer_qBxj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- S1. The analysis under generalized Lipschitz smoothness is well motivated. Remarks 1 and 2 clearly show the relationship between other related definitions of generalized smoothness. Propositions 1 and 2 provide clear examples of machine learning problems satisfying the generalized smoothness condition.\n- S2. This work provides a fine-grained convergence analysis of AN-GD under generalized smoothness & generalized PŁ condition.\n- S3. The idea behind the proposed algorithm (IAN-SGD) is interesting. Also, the algorithm has a convergence guarantee under generalized smoothness.\n\n**Weaknesses**\n\n- **W1. Invalid Comparison Between AN-GD and vanilla GD**\n    - Proposition 3 is used to claim the superiority of AN-GD over GD, thereby arguing the benefits of adaptive normalization. However, the proposition only guarantees the convergence upper bound of GD. Suppose the authors wanted to claim that AN-GD is a better algorithm than GD under generalized smoothness + generalized PL condition. In that case, they should have compared the **lower** convergence bound for GD with their convergence upper bound for AN-GD in Theorem 1.\n    - Even if the comparison between Theorem 1 and Proposition 3 were valid, the authors’ claim that GD requires a very small learning rate might not be true. The main reason why they claim that GD is worse than AN-GD is that the constant $G$, an upper bound of gradient norms along the optimization trajectory, can be large. Then when is it a problem? Recall that, when $\\rho=\\alpha=\\beta=1$, the sample complexity until $\\epsilon$-convergence for AN-GD (GD, resp.) is $\\mathcal{O}(\\tfrac1\\epsilon\\log(\\tfrac1\\epsilon))$ ($\\mathcal{O}(\\tfrac{G}\\epsilon)$, resp.). Then GD converges slower than AN-GD when $G\\gtrsim \\log(\\tfrac1\\epsilon)$. Suppose our target error is $\\epsilon=e^{-r}$ for some $r>1$. Let’s say $G=\\Omega(r)$. Then in Proposition 3, the learning rate of GD is $\\gamma  = \\mathcal{O} (r^{-1})$. On the other hand, to guarantee the AN-GD’s convergence rate in Theorem 1, we need to set $\\gamma = \\mathcal{O}(\\epsilon^{\\beta/\\rho})=\\mathcal{O}(\\epsilon)=\\mathcal{O}(e^{-r})$: the AN-GD uses an exponentially small learning rate, which can be much smaller than the learning rate choice of GD.\n    - By the way, I cannot understand why the generalized PŁ condition is introduced only in Section 4. Does this condition have any relationship with generalized smoothness? Also, the range $\\rho\\le 2$ should be specified in Assumption 2, or not?\n- **W2. Issues in Theorem 1.**\n    - I read the whole proof of Theorem 1 and I found some errors and major mistakes. I omit minor typos from here, though there are many.\n    - In almost the last lines of the proof of Case I ($\\rho < 2-\\beta$), I cannot understand why we get a complexity $T=\\mathcal{O}\\left(\\left(\\tfrac1\\epsilon\\right)^{\\tfrac{2-\\rho-\\beta}{\\rho}} + \\left(\\tfrac1\\epsilon\\right)^{\\tfrac{\\beta}{\\rho}}\\right)$. Isn’t the “taking logarithm we have …” part just giving us $T=\\mathcal{O}\\left(\\left(\\tfrac1\\epsilon\\right)^{\\tfrac{2-\\rho}{\\rho}}\\right)$, which is a strictly worse rate than that described in the theorem?\n    - In order to take advantage of Lemma 4, Theorem 1 must state that the target error (NOT a target “accuracy!”) should satisfy $0<\\epsilon \\le \\min\\\\{1, 1/(2\\mu)\\\\}$.\n    - The Case III ($1\\ge \\beta > 2-\\rho$) in the main theorem statement should contain the fact that $T_0$ is chosen so that $\\Delta_{T_0} < \\gamma^{\\frac{\\rho}{\\rho+\\beta-2}}$; otherwise the expression in Equation (9) may not converge to zero.\n    - The proof of the Case III begins with the fact that $\\{\\Delta_t\\}$ converges to zero. But why does the proof consider the convergence in terms of $\\epsilon\\downarrow 0$, instead of $t\\to\\infty$? I guess the latter (in terms of $t$) is correct.\n- **W3. Regarding the Implementation Detail of IAN-SGD (Equation (14))**\n    - Why is the normalization factor $h_t$ defined as a form of $\\max\\\\{1,…\\\\}$? According to this expression, $h_t=1$ when $\\lVert \\nabla f_{\\xi_{B’}}(w_t) \\rVert$ is small enough, which can happen in the later phase of the training (near-convergence). In this case, doesn’t the later behavior of IAN-SGD become nearly identical to vanilla SGD? If so, the $\\mathcal{O}(\\epsilon^{-4})$ rate in Theorem 2 is not very surprising.\n    - There is no description about $\\delta$. Is a term for maintaining numerical stability? What does it mean by “…IAN-SGD requires estimating the value of $\\delta$ …”?\n- **W4. Regarding Assumption 4 and the batch size choices in Theorem 2**\n    - The authors claim that Assumption 4 is “strictly weaker” than the bounded gradient noise assumption ($\\tau_1=0$). However, the assumption only allows $\\tau_1, \\tau_2>0$. Moreover, Lemma 2, a key lemma to prove Theorem 2, considers $\\tau_1 > 0$. Then, is the claim valid?\n    - Even in Section 1.1, the authors claim that they require only a “constant-level” batch size. In my opinion, however, this is misleading and can make the readers misunderstand the result. In fact, the batch size must satisfy $\\Omega(\\tau_1^2)$, which is proportional to the variance of the stochastic gradients. Thus, the proved convergence guarantee requires the batch size that depends on the problem instance, which cannot be claimed as “constant-level”. I have checked the proof of Theorem 2 and Lemma 2; I found they break down unless $B=\\Omega(\\tau_1^2)$.\n    - Lastly, the sentence “By normalizing during data pre-processing, the value of $\\tau_1$ can be approximately controlled as $\\mathcal{O}(1)$ in practice” is not very straightforward to me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work analyzes the convergence rates of optimization algorithms that leverage ‘adaptive gradient normalization’ under an objective function satisfying generalized Lipschitz smoothness condition.\n\n- With deterministic gradient oracles, it studies the (sub-)linear convergence of ‘adaptively normalized gradient descent (AN-GD)’ under an additional assumption of generalized PŁ condition. The authors claim that their analysis provides an intuition about the impact of the normalization parameter $\\beta$ to the convergence rate.\n- Under stochastic (unbiased) gradient oracles, it proposes a novel algorithm called ‘independent-adaptively normalized stochastic gradient descent (IAN-SGD)’ and studies its $\\mathcal{O}(\\epsilon^{-4})$ convergence rate under an additional assumption about moderate gradient noises.\n\nThe authors test the efficacy of their proposed algorithm numerically.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Required Changes**\n\n- Every point I listed above in the weaknesses should be resolved.\n\n**Recommended Changes & Some Questions expected to be answered**\n\n- In the abstract, there seems to be a couple of nearly duplicate sentences: “We analyze the overall affects of adaptive normalization and function geometry on convergence rate. Our results provide a comprehensive understanding of the interplay between adaptive gradient normalization and function geometry.” Can these be more concise?\n- In the second sentence of Section 1, the words “$L$-smoothness fails to characterize the global geometry” are not very clear to me. What does “to fail to characterize the geometry of functions” mean?\n- The “notation” paragraph/section will strengthen the rigor of the paper. It never specifies which type of norm is used in it explicitly, which seems to be a standard Euclidean $\\ell_2$ norm.\n- The motivation of introducing generalized PŁ geometry is weak. Why does it resolve the theory-practice gap, as the authors mentioned? Do the examples provided in the experiments satisfy the generalized PŁ geometry?\n- The main text should contain the pointers to the proof for each theorem, proposition, and lemma.\n- The symbol $\\nabla f_{\\xi_B}(w_t)$ is a bit weird. How about $\\nabla f_{B_t} (w_t)$, where $B_t = \\\\{\\xi \\text{ sampled at time }t\\\\}$?\n- Discussion about choosing a good normalization parameter $\\beta$ will strengthen the paper.\n- It seems nice to compare proof techniques with those of Zhang et al. (2019). But isn’t there any difference in the convergence rate? The comparison regarding the convergence rate will definitely strengthen the paper.\n- Typos:\n    - Put a hyphen: “Kuradyka-Łojasiewicz”.\n    - In Theorem 1, be consistent with $t$ or $T$. recommend to write like “in order to achieve $\\Delta_T\\le \\epsilon$, the total number of iteration satisfies $T=\\Omega(…)$”. Also, For the case III, the expression for $\\Delta_t$ needs an additional parantheses: $\\mathcal{O}\\left(\\left(…\\right)^{\\left(\\frac{\\rho}{2-\\beta}\\right)^{t-T_0}}\\right)$.\n    - Section D, Lemma 3: the original lemma is actually Lemma E.1 of the arXiv version of Chen et al. (2023), which is Lemma 5 of the ICML version of the same work.\n    - Section D, Lemma 4: in the long series of (in)equalities at the beginning of the proof, a plus symbol (’+’) is missing in the third line.\n    - Section F, Proposition 3, Equation (38): I guess it must be corrected as $\\Delta_{t+1} \\le \\Delta_t - \\gamma \\mu^2 \\Delta_t^2$, or no?\n    - Section G: below the equation (45), there are a couple of duplicate words: “equation equation 53”.\n    - Section H, Remark 3: a missing square. “… $\\le \\frac1B(2\\tau_1^2\\lVert \\nabla F(w) \\rVert^2 + 2\\tau_2^2)$”.", "broader_impact_concerns": "Since this work focuses on the theoretical aspects of optimization algorithms, the Broader Impact Statement section seems unnecessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)