The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms
Authors: Elizabeth Collins-Woodfin
Inbar Seroussi
Begoña García Malaxechebarría
Andrew Mackenzie
Elliot Paquette
Courtney Paquette
Venue: NeurIPS 2024 poster (2024)
Keywords: Stochastic optimization
high-dimensional probability/statistics
adaptive stepsizes
Primary area: optimization

Abstract: We develop a framework for analyzing the training and learning rate dynamics on a large class of high-dimensional optimization problems, which we call the high line, trained using one-pass stochastic gradient descent (SGD) with adaptive learning rates. We give exact expressions for the risk and learning rate curves in terms of a deterministic solution to a system of ODEs. We then investigate in detail two adaptive learning rates  -- an idealized exact line search and AdaGrad-Norm -- on the least squares problem. When the data covariance matrix has strictly positive eigenvalues, this idealized exact line search strategy can exhibit arbitrarily slower convergence when compared to the optimal fixed learning rate with SGD. Moreover we exactly characterize the limiting learning rate (as time goes to infinity) for line search in the setting where the data covariance has only two distinct eigenvalues. For noiseless targets, we further demonstrate that the AdaGrad-Norm learning rate converges to a deterministic constant inversely proportional to the average eigenvalue of the data covariance matrix, and identify a phase transition when the covariance density of eigenvalues follows a power law distribution. We provide
our code for evaluation at https://github.com/amackenzie1/highline2024.

The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms
Elizabeth Follins-Woodin
Inbar Seroussi
McGill University
Tel-Aviv University
elizabeth, -ol mmg.: mwg.ll..cmmmcc....
inbarser@tauex tau. ac.
Begoña García Malaxechebarría
Andrew W. Mackenzie
University of Washington
McGill University
begogar9tux edu
. maackenzieemmii.mooill.....
Elliot Paquette
Courtney Paquette"
McGill University elliot. paiiit.pauuetteemggglll... ca
McGill University & Google DeepMind
courtney. wwquetteemggiill........pt ca
Abstract
We develop a framework for analyzing the training and learning rate dynamics on a large class of high-dimensinn dish-siiensional optimization problems, which we call the high line, trained using one-pass stochastic gradient descent (SGD) with adaptive learning rates. We give exact expressions for the risk and learning rate curves in terms a deterministic solution to a system of ODEs. We then investigate in detail two adaptive learning rates an idealized exact line search and AdaGrad-Norm on the least squares problem. When the data covariance matrix has strictly positive eigenvalues, this idealized exact line search strategy can exhibit arbitrarily slower convergence when compared to the optimal fixed learning rate with SGD. Moreover we exactly characterize the limiting learning rate (as time goes to infinity) for line search in the setting where the data covariance has only two distinct eigenvalues. For noiseless targets, we further demonstrate that the AdaGrad-Norm learning rate converges deterministic constant inversely proportional to the average eigenvalue of the data covariance matrix, and identify phase transition when the covariance density of eigenvalues follows power law distribution. We provide 1/igghline20222
our code for evaluation at
Introduction
In deterministic optimization, adaptive stepsize strategies, such as line search (see [40], therein), AdaGrad-Norm [59], Polyak stepsize [48], and others were developed to provide stability and improve efficiency and adaptivity to unknown parameters. While the practical benefits for deterministic optimization problems are well-documented, much of our understanding of adaptive learning rate strategies for stochastic algorithms are still in their infancy.
There are many adaptive learning rate strategies used in machine learning with many design goals. Some are known to adapt to stochastic gradient descent (SGD) gradient noise while others are robust to hyper- parameters (c.g., [4, 63]). Theoretical results for adaptive algorithms tend to focus on guaranteeing minimax-optimal rates, but this theory is not engineered provide realistic performance
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

AdaGrad- Norm Least Squares
109
AdaGrad- Norm Logidtic Regression
100
9.9
10-4
9.8x10
100
9.7 x
10c
Learning
1.0
256
d= 16
d 1024
d= 32
9.5 x0
64
163004
10-3
128
9.4 10
Theary, learn rate
Theory, Ieam. rate
10
Theory,
Theory,
1
10
10
10
10
10
10a
x0-11
SGD Iterationaed
SGG iterationsdd
Figure 1: Concentration of learning rate and risk for AdaGrad-Norm on least squares with label noise w 1 (left) and logistic regression with no noise (right), As dimension increases, both risk and learning rate concentrate around a deterministic limit (red) described by our ODE in Theorem 2.1. The initial risk increase (left) suggests the learning rate started too high, but AdaGrad-Nome adapts. Our ODEs predict this behavior. See Sec. H for simulation details.
comparisons; indeed many adaptive algorithms are minimax-optimal, and so more precise statements needed to distinguish them. For instance, the exact learning rates (or rate schedules) which these strategies converge are unknown, nor their dependence on the geometry of the problem. Moreover, we often do not know how these adaptive stepsizes compare with well-tuned constant decaying fixed learning rate SGD, which can be viewed as a cost associated with selecting the adaptive strategy in comparison tuning by hand.
In this work, we develop a framework for analyzing the exact dynamics the risk and adaptive learning rate strategies for wide class of optimization problems that we call dighesiimesi linear (high line) composite functions. In this class, the objective function takes the form of an expected
risk R Rd -> R over high-dimensional data (a, e) D Rd IR of function f R TR. composed with the linear functions (X,a), (X*, a)) That is, we seek to solve
min { (((X) E Ea.elf((a, X), (a, x)) for (a.e)~,,X' E XER
(I)
We suppose (N0,, K) where K Rddd is the covariance matrix. We train (1) using (one-pass) stochastic gradient descent with adaptive learning rates, 0& (SGD+AL), Our main goal is to give framework for better? performance analysis of these adaptive methods. We then illustrate this framework by considering two adaptive learning rate algorithms on the least squares problem', the results of which appear in Table I: exact line-search (idealistic) (Sec. 3) and AdaGrad-Norm (See. 4). We expect other losses and adaptive learning rates can be studied using this approach.
Main contributions. Performance analysis framework We provide equivalence of R(xx) and learning rate 0k under SGD+AL to deterministic functions (t) and " via solving a deterministic system of ODEs (see Section 2), which we then analyze to show how the covariance spectrum influences the optimization. See Figure 1. As the dimension dl of the problem grows, the learning curves of R (Xx)) become closer to (l) and the curves concentrate around 2 (l) with probability better than any inverse power of (See Theorem 2.1).
Greed can be arbitrarily bad in the presence of strong anisotropy (that is. Tr(K)/d < /d)))/d). Our analysis reveals that exact line search, which is to say optimally decreasing the risk at each step, arbitrarily slower than the best fixed learning rate for SGD least squares problem when def
Amin Amin (K) C > O. The best fixed stepsize (least squares problem) is (Tr(K)/d)- or the inverse of the average eigenvalue, see Polyak stepsize [48]. Line search, on the other hand, converges to fixed stepsize of order Amin/(Tri (Tr(((T))))))))))))))) //d). It can be that Amin/(Tr( ((TT(K22//) < (Tr(K)/d)- making exact line search substantially underperform Polyak stepsize. We further explore this and, in the case where d-eigenvalues of K take only two values )1 > X2 > 0, we give an exact expression as function of \ and X2 for the limiting behavior of 7/ as + (See Fig. 5).
2 More realistic, in that deals with nigh-dimensional anisotropic loss geometries and more precise, in that can distinguish minimax optimal algorithms as more-or-less performant. We extend some results the general strongly convex setting.

---

Table I: Summary of adaptive learning rates results on the least squares problem. We summarize our results for line search and AdaGrad-Norm under various assumptions on the covariance matrix K. We denote Amin the smallest non-zero eigenvalue of K and [[(K] the average eigenvalue. Power law(6,88 3) assumes the eigenvalues of K, {{; follow a power law distribution, that is, for 0 < 3 < ), (1 1) (0.1) for all 1 < i S d and (Xo X ,x,)² X where {wi] are eigenvectors of K (see Prop 4.4). For (see Prop. 4.2), requires a good initialization on b, 17.
d
Learning rate
K assumption
Limiting Yoo
Convergence rate
AdaGrad- Norm(b, m)) (see Sec. 4)
Amin > C
x # t TT(R))
log( Aminn
X2 x11
AdaGrad-Norm(b. 1)
3 8 <1
Ytt X6,8
(1) 36,8 10+6-
Power law (see Sec. 4)
8 8+8=1 =1
88,8 log(1+1)
((t X6,0 Tog(L++)
1< 8=
Ve X 843
(t) Xa, t FoR +1
Exact line search, idealized (see Sec. 3)
Amin > C
Amin x
log(3f) x - toot
kydd
Polyak stepsize (sec Sec.
Amin > C
( x - -Amin Yoot
kkkyaa
AdaGrad-Norm selects the optimal step-size, provided it has a warm start. In the absence of label noise and when the smallest eigenvalue of K satisfies Amin C >>> O, the learning rate converges to a deterministic constant that depends on the average condition number (like in Polyak) and scales
inversely with TK(2
d. [ - 12. Therefore it attains automatically the optimal fixed stepsize in terms of the covariance without knowledge of Tr (K), but pays a penalty in the constant, namely I|Xo- x² If one knew I|x,, X1 then by tuning the parameters of AdaGrad-Noo Norm one might achieve performance consistent with Polyak; this also motivates more sophisticated adaptive algorithms such as DoG [29] and D-Adaptation [18], which adaptively compensate and/or estimate -1
AdaGrad-Norm can use overly pessimistic decaying schedules on hard problems. Consider power law behavior for the spectrum of and the signal X This a natural setting as power law distributions have been observed in many datasets [60]. Here the learning rate and asymptotic convergence of K undergo phase transition. For power laws corresponding to easier optimization problems, the learning rate goes to constant and the risk decays at t- For harder problems, the learning rate decays like t and the risk decays at a different sublinear rate t-22, See Table and Sec. 4 for details.
Notation. Define R, [0, oo). We say an event holds with overwhelming probability, w.O.p., if there is a function w : N + IR with w(d/ log d - 3o so that the event holds with probability at least 1 - e (dd) We let 1 (z) be the indicator function of the set A where it is if x E A and 0 otherwise. For a matrix A E Rm we use ||A]] to denote the Frobenius norm and ||A to denote the operator-2 norm. If unspecified. vector spaces A, B with norms || IIa and || |ls. respectively, and for a 2 O, we say a function F : A - B is pespedo--ipsccheee with constant L if for any A, A E A, we have
assume that the norm is the Frobenius norm. For normed
[[(AA) - F(A)I S LIIA (1 + || + (|
We write f(t) x gt) if there exist absolute constants C, c > 0 such that e g(t) $ f(t) < CC- g(t) for all t. If the constants depend on parameters, e.g.. a, then write x
Related work. Some notable adaptive learning rates in the literature are AdaGrad- Norm [32, 59, 61]. RMSprop [28], stochastic line search, stochastic Polyak stepsize [35], and more recently DoG [29] and D- D-Adaptation [18]. In this work, we introduce a framework for analyzing these algorithms, and we strongly believe it can be used to analyze many more of these adaptive algorithms. We highlight below nonexhaustive list of related work.
3

---

AdaGrad-Norm. AdaGrad, introduced by [19, 36], updates the learning rate at each iteration using the stochastic gradient information. The single stepsize version [32, 59, 61], that depends on the norm of the gradient, (see Table 2 for the updates), has been shown to be robust to input parameters [34]. Several works have shown worst-case convergence guarantees [21, 33, 57, 59]. linear rate of O (eexp(- -x))) is possible for μ-strongly convex, L-smooth functions (k is the condition number /// L). In [62] (similar idea in [.]]]. the authors show for strongly convex, smooth stochastic objectives (with additional assumptions) that the AdaGrad- learning rate exhibits two stage behavior - a burn in phase and then when it reaches the smoothness constant it self-stablizes.
Stochastic line search and Polyak stepsizes. Recently there has been renewed interest in studying stochastic line search [20, 42, 54] and stochastic Polyak stepsize (and their variants) [7. 26, 27, 30, 35, 39, 41. 49]. Much of this research focuses on worst-case convergence guarantees for strongly convex and smooth functions (see e.g., [35]) and designing practical algorithms. In [53], the authors provide a bound on the learning rate for Armijo line search in the finite sum setting with rate of /max//vg. μ where avg. l is the avg. strong convexity and Lamax the max. Lipschitz constant of the individual functions. In this work, we consider slightly different problem. We work with the population loss and we note that the analogue to Lmaa for us would require that the samples a satisfy ||aa" lop S Lmax for all this fails to hold for a No0, K). Moreover, Lmax could be much worse than E [||aa=1log)
Deterministic dynamics of stochastic algorithms in high-dimensions. The literature on deter- ministic dynamics for isotropic Gaussian data has long history [9, 10, 50, 51]. These results have been rigorously proven and extended to other models under the isotropic Gaussian assump- tion [1, 2, 6, 16, 17, 23, 58]. Extensions to multi-pass pass SGD with small mini-batches [46] as well as momentum [31] have also been studied. Other high-dimensional limits leading to different class of dynamics also exist [11-13, 22, 37J. Recently, significant contributions have been made in understanding the effects of a non-identity data covariance matrix on the training dynamics [5, 14, 15, 24, 25, 64]. The non-identity covariance modifies the optimization landscape and affects convergence properties, as discussed in [15]. This work extends the findings of [15] to stochastic adaptive algorithms, exploring the effect of non-identity covariance within these algorithms. Notably, Theorem 1.1 from [15] is restricted to deterministic learning rate schedules, limiting its applicability in many practical scenarios. In contrast, our Theorem 2.1 accommodates stochastic adaptive learning rates, aligning with widely used algorithms in practice.
1.1 Model Set-up
We suppose that a sequence of independent samples {ax yb)) drawn from a distribution D C Rd x R is provided where yı is the target. The target yk is function of some random label noise G₁ E R and the input feature ag dotted with a ground truth signal X E R. (a,, X*). Therefore, the distribution of the data is only determined by the input feature and the noise, i.e., the pair (a, e). In particular, assume (a,e) follows a distributional assumption.
Assumption 1 (Data and label noise). The samples (a.c) ~ DD are normally distributed: e ~ NO, w²) where w E R, and a ~ (O, K), with a covariance matrix K E &dxdd that is bounded in operator norm independent of d; i.e., || lop I C. Furthermore, a and e are independent.
For a, X, X* E R2d, E RR and a function f - IR, we seek to minimize an expected risk function R. : Rdd + R, which we refer to as the high-dimensional linear composite", of the form
R(X) E a, ( eX) for (a,E) ~ D. and (X; = f((a, e)). (2)
In what follows, we use the matrix W = [X[X*] E Rdx2 that concatenates X and X*, and we shall let B B(W)- W KW be the covariance matrix of the Gaussian vector ((a, X), (a, X*)). Assumption 2 (Pseudo-lipschitz f). The function R R is o-pseudo--iipschitz with a S 1.
By assumption, R(X) involves an expectation over the correlated Gaussians (a, X) and (a, Xt). We def
can express this as R(X) h( ()) for some well-behaved function h 2 + IR.
'Note that d need not be large to define this, but the structure allows us to consider d as tunable parameter. Moreover, we increase d, the analysis we do will be more meaningful.

---

Table 2:
Two adaptive learning rates considered in detail. The stochastic adaptive learning rate, gk- is the learning rate directly used in the update for SGD whereas the deterministic, nr is the deterministic equivalent of Ok after scaling.
Algorithm
General update
Least squares
b = b2 0 d x
(V
AdaGrad- Norm(b, v)) bu = x d
9h
same
7t
a 1 ((86(6)) de
6 21x K)
Jo (2)
Exact line search
(xV))(()) |R[(X
[VR(
201K R(Xx)
(idealized)
Σ X7 2N(K))0((1
arg min (t)
Assumption 3 (Risk representation). There exists a function IR2x2 R such that h(B) R(X) is differentiable and satisfies
VxR(X)= En, V (X;;
Furthermore, is continuously differentiable and its derivative Vh a--pseudo-lipschitt.o some 0 1, with constant L(Vh).
The final assumption is the well-behavior of the Fisher information matrix of the gradients. The first coordinate of f is special, as the optimizer must be able to differentiate Thus, we treat fax x², e) a function of single variable with two parameters: f(r, x², E) f(z;=*,e) e) and denote the (almost everywhere) derivative with respect to the first variable as f'.
Assumption 4 (Fisher matrix). Define I(B) def Ea,el(f' X); (a, x)), where the function 1 R2x2_ + R. Furthermore, I is a-psendo-Lippshhen with constant L(T) for some S 1.
A large class of natural regression problems fit within this framework, such as logistic regression and least squares (see [15, Appendix BJ). We also note that Assumptions 3 and 4 are nearly satisfied for L-smooth objectives (see Lemma B.1), and a version of the main theorem holds under just this assumption (albeit with a weaker conclusion).
1.2 Algorithmic set-up
We apply one-pass or streaming SGD with an adaptive learning rate Ok (SGD+AL) to solve minxer R(X), (2). Let Xo Rdd be an initial vector (random non-random). Then SGD+AL iterates by selecting a new data point (ak+1, (6+1) such that ax+1 ~ (1, K) and 6x+1 ~ w0),w²) and makes the update
Bk
Ok ((aa+11, Xa); (ax+1, X*), (x+1) (3) 0241, = X /
X = Xx=
d
where gl > O is learning rate (see assumptions below)S. To perform our analysis, we place the following assumption the initialization Xo and signal X*.
Assumption 5 (Initialization and signal). The initialization point Xo and the signal X* are bounded independent of d, that is, max(]]Xol]] [| & C for some C independent of d.
Adaptive learning rate. Our analysis requires some mild assumptions on the learning rate. To this end, we define a learning rate function 7 R,, x D(10, oo)) x D(O0, oo)) x D((0, oc) - RR, by
D 7k, Neld x -), Go(d x )) Qx x )), for E N, where for any 2 0, (N=(t), (t),), (t) [ock) (Iw)) WW..TW. W., J V (x11 R(X.)).
(4)
Note that cases where Trck" = o(d) can lead to dynamics that converge to full-batch gradient flow. While our theorem specifically addresses scenario where the intrinsic dimension, Dim(K) def TT(KK/l K|sp. satisfies Dim(K) (K) = ((d), other cases, such as Dim K) = o(d, may require different learning rate scalings. "D00, oc)) is the càdlàg function class [0, oc).
5

---

In this definition, for functions taking integer arguments, we extend them to real-valued inputs by first taking the floor function of its argument. Note that the adaptive learning rates can depend on the whole history of stochastic iterates (N&),, gradients (Gk)), and risk (Qx) via this definition.
We also define conditional expectation version of Gg where the filtration F& oxx,, .o...., :k))
(t) def ((<o)))) Blll V 04,
for t 2 0.
With this, we impose the following learning rate condition.
Assumption 6 (Learning rate). The learning rate function ry BR+ D((0, xc)) D(10, oo)) x
D( oo)) + R is ppsseud--Lipcctee D(00, co)) x D(OO, co)). Moreover, for some constant C C(r) > 0 independent of d and 8 > 0,
with constant L(y) (independent of d) in D(0, co)) x
E (k, Guld x)),9) (k, (x x.), [ik Cd (1 ll + +1al@))
w.O.p. (5)
Finally, "y is bounded, i.e, there exists a constant C = ((7) > 0 independens of d so that
r(k, g.gg) q) S C(1 |l + || + ll.ll%)
(6)
The inequality (5) ensures that the learning rate concentrates around the mean behavior of the stochastic gradients. Many well-known adaptive stepsizes satisfy (4) and Assumption 6 including AdaGrad-Norm, DoG, D-Adaptation, and RMSProp (see Table 2, Sec. A, and Sec. C.3).
2 Deterministic dynamics for SGD with adaptive learning rates
Intuition for deriving dynamics: The risk ((X) and Fisher matrix can be evaluated solely in terms of the covariance matrix B. Thus, to know the evolution of the risk over time, would suffice to know the evolution of B. Alas, except in the isotropic case where K is a multiple of the identity, the evolution of B is not autonomous (i.e., its time evolution depends on other unknown variables). However, if we let (Aç, wi) be the eigenvalues and corresponding orthonormal eigenvectors of K, we can consider projections V((Xx) = d W waws? WR. and it turns out that these behave autonomously.
Example: Least Squares. One canonical example of (2) is least squares, where we aim to recover the target X given noisy observations (a, X*) + c. this case, the least squares problem is
(R(X)= J [Ma, X X*) - -) = f=²² + 2 xx)x
min XER
(7)
The pair of functions h (Assumption 3) and I (Assumption 4) can be evaluated simply:
h(B(W)) = ( (B()W) = (x -
-xx)xXXX(XX - x*)+ + tw².
The deterministic dynamics for the risk (t) in this case can be simplified to:
2K Sö 7o da (Xo - X*) + 1/2² + So 2 Tr -2 S 7+ dr) (()) ds. (t) (X) x-""X"""" Ke-K
This is convolution Volterra equation with convergence threshold of %
<
2d Trk
[14, 44, 46, 47].
In the noiseless label case (i.c., 0), the risk is given by (t) 22 Gi=!
A (t)) Using the
ODEs in (9), we get the following deterministic equivalent ODE for the 9 's:
a (t) 27, (t).
(8)
We will perform a deep analysis of the dynamics of the learning rate on least squares (7), which will generalize to settings where the outer function / strongly convex (see D.1).
Deterministic dynamics. To derive deterministic dynamics, we make the following change to continuous time by setting
A iterations of SGD = [td], where t E R is the continuous time parameter.
This time change is necessary, as when we scale the size of the problem, more time needed to solve the underlying problem. This scaling law scales SGD so all training dynamics live on the same

---

space. One can solve a smaller d problem and scale it to recover the training dynamics of the larger problem."
We now introduce coupled system of differential equations, which will allow us to model the behaviour of our learning algorithms. For the ith (4) aieneaeeeegeaaleeeeeeeeeeooooo of K, set
(t(()) def
(t) (22,(() ((22;(1)]
(t),((t]
and averaging over i, B(t) def
(t()..
The Vi(() and (t)) are deterministic continuous analogues of Vi(Xia) and B(Xaa) respectively. Define the following continuous analogues
(t)) def H1. H2,
((t) def Σ M(),, 21 del h()())), f(t) def (()(1))
H2
Ha.
and finally (t,, 1 (-2t)- M (-), n(K) (15 I (), (111))))))))
We now introduce system of coupled ODEs for each (As, wi) eigaaaaliiggnaalleeeeeeeeeoor pair of K
+ (t) + H2, + A d7)2,((t) 22m (e. H2, (2,,(2,,((t)
d7)1,((t) 22 (t),, = H1,,771,,
(9)
with the initializationn of (0()) given by Vx(X)). We finally state the deterministic dynamics for the risk and learning rate.
Theorem 2.1. Under Assumptions 1, 2, 3,4, 5, 6, then for any E (0, 2 and any T > 0
(R(X(a)) (R(()) -
sup
||
<d= w.o.p.
(10)
:d]
The same statements hold comparing Wa to (1) and WWK to I1).
In fact, we can derive deterministic dynamics for large class of statistics which are linear combina- tions of (1) and functions thereof (See Theorem I. I, and Corollary B.1).
One important corollary is deterministic limit for the distance to optimality, (²x,)) = - which is a quadratic form of W Wi and hence covered by Thm. 2.1. The equivalent deterministic dynamics are
d Σ
1 Σ 2
(t)
- + (t)),
(11)
1=1
where (t(t) corresponds (Xx, d ((X, X w
3 Idealized Exact Line Search and Polyak Stepsize
In this section, we consider two classical idealized algorithms - exact line search and Polyak stepsize. In deterministic optimization, these learning rate strategies chosen so that the function value (exact line search) or distance to optimality (Polyak) produces the largest decrease in function value (resp. distance optimality) the next iteration. For stochastic algorithms, hold for the deterministic equivalent to the risk (t) (resp. distance to optimality. (t)) since we know that SGD is close to these deterministic equivalents. Thus, the question is: what choice of learning rate decreases the (t) (exact line search) and/or (t) (Polyak stepsize)? We will restrict to least squares in this section see Appendix F.1 and F.2 for general functions as well as proofs for least squares. These are idealized algorithms because we can not implement them as they require distributional knowledge of a or X Despite this, they provide basis for more practical algorithms.
ask this to
"Note that, holding time fixed, we perform O(d) gradient updates for problem dimension For the problems considered here, this scaling leads to consistent dynamics, but there do exist related problems where a different scaling is more appropriate. For example, under random initialization. to capture the escape of phase retrieval from the digh-dimessional saddle, O(d log d) iterations needed; see for example [56].
7

---

SGD (line soarch
10-
thoory (Eno search)
quolkan 7t S000
4 10
2P18)
10--
SGD (Polyak)
10"
thoory (Polyak)
2 10"
(TY(K)) 2.25
10-31
(IY(A")) 1.77
(TY(A")) 1:27
10"
40
80
20
40
tune (1)
(Tr(K3) RJ 1.12
tinic
Figure 2: Comparison for Exact Line Search and Polyak Stepsize on noiseless least squares problem. The left plot illustrates the convergence of the risk function, while the right plot depicts Amin(K) for Polyak stepsize and exact line search. Both plots highlight the implication of equation (13) in high-dimensional settings, where a broader spectrum indicating slower risk convergence and poorer performance
the convergence of the quotient 7t/ 3 TT(KK)
of results in Amin(() (K)
Tr(??)
TT(())
of exact line search (unmarked) as it deviates from the Polyak stepsize (circle markers). The gray shaded region demonstrates that equation (13) is satisfied. See Appendix H for simulation details.
Polyak Stepsize. A natural threshold to consider is the largest learning rate such that d < 0, which we denote by n: Using the least squares ODE (8), this is precisely
(2:8(()-4))
and
0k TI( R(Xx)
(2R(Xx)-w")
(12)
KF(K
Without label noise, (12) simplifies to 72 0 least squares.
Tkka the exact threshold for convergence of
2
greedy stepsize strategy would maximize the decrease in the distance to optimality at each iteration, denoted by us as Polyak stepsize, 7:
Polyak E arg min, (t) .). In the case of least squares, this is
Polyak it
7t
D
and
Polyak 2 9k
The latter yields the optimal fixed learning rate (up to absolute constant factors) for noiseless target on a least squares problem [35, 43)).
Exact Line Search. In the context of risk, using (8) and noting that (())
2d Σd X, (()),
can find line E arg min d (I); i.e., the greedy learning rate that decreases the risk the most in the next iteration. We call this exact line search. Expressions for the learning rates are given in Table 2, (c.f. Appendix F.1 for general losses). Because these come from ODEs, we can use ODE theory to give exact limiting values for the deterministic equivalent of glime.
Proposition 3.1. [Limiting learning rate; line search on noiseless least squares] Consider the noiseless (w 0) least squares problem (7). Then the learning rate is always lower bounded by
)ma(K) à TT(K2)
Jine for all t t2 0.
Moreover, suppose K has only two distinct eigenvalues X1 >2 > O, i.e., K has d/2 eigenvalues equal Xy eigenvalues and d/2 eigenvalues equal to xz. Then
Amnn (K)
< Iim line < 2kmmn(K) t-500
(13)
Te(K9)
TY(K2)
For proof and explicit formula for limy-+00 Nime see Section F.2. Hence, being greedy for the risk in sufficiently anisotropic setting will badly underperform Polyak stepsize (see Fig. 2).
"The Polyak stepsize analyze in this paper differs slightly from the "classic" stepsize in the literature, that
is, (Xx)) (xx,)= (x(**) Rather than using this form, we skip approximation step in the derivation [27] and use the [X,
exactly optimal form. Both variations of the Polyak stepsize can be analyzed under our assumptions; the choice admittedly somewhat arbitrary. (Note that in the of least squares, the stepsizes coincide.)

---

Lod TouresApOraa
500. -X1)5-4
500. P4 We-Xir-266
Tx30
leaming rMee
leaming
2xpg.
12x k0
6430
Proc
5x
940
35
360 terations
SG0 Iterationn
Figure 3: Quantities effecting AdaGrad-Norme learning rate. (left): Effect of noise (w 1.0) risk (left axis) and learning rate (right axis). Depicted is learning rale asymptotic
it approaches 1. (Center, right): Noiseless least squares (w = 0). As predicted in Prop. 4.2, lim "t depends on avg. eig. of K
(Tr(K)/d) and I| 1|2 but not K Amax/ Aminn See Appendix H for simulation details.
4 AdaGrad-Norm analysis
In this section, we analyze the behavior of AdaGrad-Norm learning rate in the least squares setting (see Sec. D for general strongly convex functions). In the presence of additive noise, the AdaGrad- Norm learning rate decays like t 1/22 regardless of the data covariance K. In contrast, the model with noise exhibits learning rate that depends the spectrum of K, as illustrated in Figure 3. The learning rate is bounded below by a constant when Amin (K)) > O is fixed as -> co, and quantify this lower bound. If the limiting spectral measure of K has unbounded density near O (e.g. power law spectrum), then the learning rate can approach zero and we quantify the rate of this convergence in the least squares setting as a function of spectral parameters.
For least squares with additive noise, the learning rate asymptotic "tt xn/(b2 2 TT(K))) is the fastest decay that AdaGrad- can exhibit. In contrast, the propositions below concern the noiseless case where, for various covariance examples, the decay rate % changes. This is tightly connected to whether the risk is integrable or not. In the simple case of identity covariance, we obtain a closed formula for the trajectory the integral of the risk and therefore also the learning rate.
Proposition 4.1. In the case of identity covariance (K- Ia), the risk solves the differential equation
(t)
(()(()) b242JJ (3) 6242 (s)
(n)(())
(14)
The solution f (s) lower bound to which Ti will converge. Generalizing this broader class of covariance matrices, we get the next proposition, which captures the dependence of % on Tr(K).
(ssdd ds approaches (from below) a positive constant which yields computable
Proposition 4.2. Suppose 1/2 Tr(K)) < b/n, and that Jo (s) ds < co with 7s as in Table AAdaGrad-Noma Norad-Nooffooo for least squares), then Ni ++ 4 T+(K))(00)))
1
An analog of Proposition 4.2 for the strongly convex setting appears in Sec. D (see Prop. D.I). We now consider two cases in which, as d + oc, there are eigenvalues of K arbitrarily close to O.
Proposition 4.3. Assume that, for some C > O, the number of eigenvalues of K below C o(d),
and that (X+, wa) = (dd-1/2) for all (i.e. X* Then, with the initialization Xo = 0, there exists some 5) >> 0 such that "it >> 5 for all t > 0.
concentrated in any eigenvector direction).
Proposition 4.4. Let K have a spectrum that converges as d - oo to the power law measure p()) (1-8)) -0 1 for some 12, and suppose that (7( X; .or 8 2 0. Then:
For 1 3 + 8, there exists & such that "/ 2 J, and R (1) X8,9 1 for all 1
. For 1 < B + 8 < 2, 5.8 68,8 and R() X5,3 t Pez for all 1.
"Our result can be compared to existing findings for SGD under power-law distributions in [8, 52, 55J. While these works explore similar assumptions regarding the covariance matrix spectrum, they do not address the high-dimensional regime with diverging Tr(K), K. focusing primarily on B > 1.

---

t"ibeto + data 2)
constant
duta beta 0.2
deta beta 0.2
dotta
0.4
deta
deta bets
betn 1.0
data + bett
dots bob 1.2
dela + beta 1.4
Celle ieta 1.4
duta beta
duta beta 1.5
delta bets
10-2
dets beta
23"
deta
22 time
Libeta+detal)
10 127
Figure Power law covariance in AdaGrad Norm on least squares problem. Ran exact predictions (ODE) for the risk and learning rate (solid lines). Dashed lines give the predictions from Prop. 4.4 which match experimental results exactly. Phase transition as d + 3 varies. When 8 + 3 < 1 (green), the learning rate (right) is constant as t + co. In contrast, when 2 > 8 + 3 1 (purple), the learning rate decreases at a rate t- ++1/(++)) with 5 + 3 = 1 (white) where the change occurs. Same phase transition occurs in the sublinear rate of the risk decay (left) (see Prop. 4.4).
For 1= + -5,0 log([+1] and (t) X6,3 (1o((47))) 1 for all, 2 1.
This proposition shows non-trivial decay of the learning rate is dictated by the residuals (distance to optimality at initialization) and the spectrum of K. We note that 5 = 0 corresponds to uniform contribution of each mode (e.g. Xo normally distributed). As the eigenmodes of the residuals become more localized, the decay of the learning rate closer to the behaviour in the presence of additive noise. Furthermore, the scaling behaviour of the loss is affected by the structure of the AdaGrad-Nor algorithm (see Fig. 4). Lastly, constant stepsize SGD yields (t) x f8+6-3 with no transition occurring at 3 8 1. Proofs of the above propositions, in a slightly more general setting, are deferred to Sec. D.
Conclusions and Limitations
This work studies stochastic adaptive optimization algorithms when data size and parameter size large, allowing for nonconvex and nonlinear risk functions, as well as data with general covariance structure. The theory shows concentration of the risk, the learning rate and other key functions to deterministic limit, which described by a set of ODEs. The theory is then used to derive the asymptotic behavior of the AdaGrad-Nore and idealized exact line search on strongly convex and least square problems, revealing the influence of the covariance matrix structure on the optimization. potential extension of this work would be study other adaptive algorithms such D-adaptation, DOG, and RMSprop which are covered by the theory. Studying the asymptotic behavior of the risk and the learning rate may improve our understanding of the performance and scalability of these algorithms on more realistic data. Another important application of the theory would be to analyze the ODEs presented here on nonconvex problems.
The current form of the theory is limited to Gaussian data, though many parts of the proof can be extended easily beyond Gaussian data. The main ODE comparison theorem is also only tuned for analyzing problem setups where the trace of the covariance is on the order of the ambient dimension; when the trace of the covariance is much smaller than ambient dimension, other stepsize scalings of SGD are needed. In addition, the analysis is limited to the streaming stochastic adaptive methods. We conjecture that similar deterministic equivalent holds also for multi-pass algorithms at least for convex problems. This has already been shown in the least square problem for SGD with a fixed deterministic learning rate [43, 45]. Lastly, numerical simulations on real datasets (e.g., CIFAR-5m) suggests that the predicted risk derived by our theory matches the empirical risk of multipass SGD beyond Gaussian data (see for example Figure 6).
10

---

Acknowledgments and Disclosure of Funding
E. Bollins-Wooddhe was supported by Fonds de recherche du Québec Nature technologies (FRQNT) postdoctoral training scholarship and Centre de recherches mathématiques (CRM) Applied math postdoctoral fellowship. Research of B. Gareía Malaxechebarría was in part funded by NSF DMS 2023166 (NSF TRIPODS II). Research by E. Paquette was supported by a Discovery Grant from the Natural Science and Engineering Council (NSERC). C. Paquette a Canadian Institute for Advanced Research (CIFAR) AI chair, Quebec AT Institute (MILA) and a Sloan Research Fellow in Computer Science (2024). C. Paquette was supported by Discovery Grant from the Natural Science and Engineering Research Council (NSERC) of Canada, NSERC CREATE grant Interdisciplinary Math and Artificial Intelligence Program (INTER-MATH-AD) MATH-AI), Google research grant, and Fonds de recherche du Québec Nature et technologies (FRQNT) New University Researcher's s Start-Up Program. Additional revenues related to this work: C. Paquette has 20% part-time employment at Google DeepMind.
References
[1] Luca Arnaboldi, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Escaping medi- ocrity: how two-layer networks learn hard single-index models with SGD. arXiv preprint arXiv:2305. 2023.
[2] Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high- dimensional and mean-field dynamics to dimensionless ODEs: unifying approach SGD in two-layers networks. arXiv preprint arXiv:2302.05582 2023.
[3] Krishna B Athreya, Peter E Ney, and PE Ney. Branching processes. Courier Corporation, 2004.
[4] Amit Attia and Tomer Koren. SGD with adagrad stepsizes: full adaptivity with high probability unknown parameters, unbounded gradients and affine variance. In International Conference on Machine Learning, pages I 117-11171. PMLR, 2023.
[5] Krishnakumar Balasubramanian, Promit Ghosal, and Ye He. High-dimensional scaling lim- its and fluctuations of online least-squares SGD with smooth covariance. arXiv preprint 2XXv:2304.00707, 2023.
[6] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. figh-dimmnssonal limit theorems for SGD: Effective dynamies and critical scaling. In Advances in Neural Information Processing Systems, volume 35, pages 25349-25362, New York, 2022. Curran Associates, Inc.
[7] Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Training neural networks for and by interpolation. In International conference machine learning, pages 199-809. PMLR, 2020.
[8] R. Berthier, F. Bach, and P. Gaillard. Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[9] Michael Bichl and Peter Riegler. On-line learning with a perceptron. Europhysics Letters, 28 (7):525, 1994.
[10] Michael Biehl and Holm Schwarze. Learning by on-line gradient descent. Journal of Physics A: Mathematical and general, 28(3):643, 1995.
[11] B. Bordelon and C. Pehlevan. Learning Curves for SGD on Structured Features. In International Conference Learning Representations (ICLR), 2022.
[12) Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics of first order methods with random data. arXiy preprint 1rXiv:211..0757,, 2021.
[13] Kabir Aladin Chandrasekher, Ashwin Pananjady, and Christos Thrampoulidis. Sharp global convergence guarantees for iterative nonconvex optimization with random data. Ann. Statist,, 51(1):179-210, 2023. ISSN 0090-5364,2168-89999 doi: 10. 10.1214/2-a0s22446 URL https: //doi. 1214//2-11211 a0s2246.
11

---

[14] Elizabeth Collins- Collins-Woodin and Elliot Paquette. High-dimensional limit of one- preeppassss SGD on least squares. Electronic Communications in Probability. 29:1-15, 2024, doi: 10.1214/3-ECP5571
[15] Elizabeth Gollins-Woodfing Courtney Paquette, Elliot Paquette, and Inbar Seroussi. Hitting the high-dimensional notes: An ODE for SGD learning dynamics on GLMs and multi-index models. arXiv preprint arKiv:2308.088 08977, 2023.
[16] Alex Damian, Eshaan Nichani, Rong Ge, and Jason D Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index mod- els. In Advances in Neural Information Processing Systems, volume 36, pages 752- 784, 2023. URL pestps::/proceeeddeeei 777766775761111911116656666o.oo.
neurips cc/paper_ra. ppappr_files/ppppppppaaaeeeeeeep filee//papor2222///111 Paper- Cereer-Confeeence pdf.
[17] Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborová, and Florent Krzakala. The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. arXiv preprint arXiv:2402.05220, 2024.
[18] Aaron Defazio and Konstantin Mishchenko. Learning-rate rate-free learning by d-adaptation. In International Conference on Machine Learning, pages 7449-7479, PMLR, 2023.
[19] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[20] Darina Dvinskikh, Aleksandr Ogaltsov, Alexander Gasnikov, Pavel Dvurechensky, Alexander Tyurin, and Vladimir Spokoiny. Adaptive gradient descent for convex and non-convex stochastic optimization. arXiv preprint arXiv: /9/ 008300, 2019.
[21] Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive SGD. In The Thirty Sixth Anmual Conference on Learning Theory, pages 89- 160 PMLR, 2023.
[22] Cédric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zde- borová. Rigorous dynamical mean-field theory for stochastic gradient descent methods. SIAM Journal on Mathematics of Data Science, 6(2):400-427, 2024. doi: N0.1137/23M1594438
URL attts:://oii. org/ 10. 1137/23M1594388.
[23] Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019.
[24] Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning neural networks: The hidden manifold model. Physical Review X, 10(4):041044, 2020.
[25] Sebastian Goldr, Bruno Loureiro, Galen Reeves, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning, pages 426-471, New York, New York, USA, 2022. PMLR.
[26] Robert M Gower, Aaron Defazio, and Michael Rabbat. Stochastic polyak stepsize with a moving target. arXiv preprint arXiv:2 /851, 2021.
[27] Elad Hazan and Sham Kakade, Revisiting the polyak step size. arXiv preprinl 2019.
[28] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
[29] Maor Ivgi, Oliver Hinder, and Yair Carmon. DoG is SGD's best friend: A parameter-free dynamic step size schedule. arXiv preprint arXiv:2302. /2022, 2023.
[30] Xiaowen Jiang and Sebastian U Stich. Adaptive SGD with polyak stepsize and line-search: Robust convergence and variance reduction. Advances Neural Information Processing Systems, 36, 2024.
12

---

[31] Kiwon Lee, Andrew N. Cheng, Courtney Paquette, and Elliot Paquette. Trajectory of Mini- Batch Momentum: Batch Size Saturation and Convergence in High Dimensions. To Appear in NeurlPS 2022, art. arXiv:2206.01029, June 2022.
[32] Kfir Levy. Online to offline conversions, universality and adaptive minibatch sizes. Advances in Neural Information Processing Systems, 30, 2017.
[33] Kfir Y Levy, Alp Yurtsever, and Volkan Cevher. Online adaptive methods, universality and acceleration. Advances in neural information processing systems, 31, 2018.
[34] Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. Proceedings of the Eventy-Secone International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 983-992, 2019.
[35] Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julics Stochastic polyak step-size for SGD: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics, pages 1306-1314. PMLR, 2021.
[36] H Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. arXiv preprint 1002
2010.
[37] F. Mignacco, F. Krzakala, P. Urbani, and L. Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. In Advances in Neural Information Processing Systems, volume 33, pages 9540-9550, 2020.
[38] P. Nakkiran, B. Neyshabur, and H. Sedghi. The Deep Bootstrap Framework: Good Online Learn- ers are Good Offline Generalizers. In International Conference on Learning Representations (ICLR), 2021.
[39] Angelia Nedić and Dimitri Bertsekas. Convergence rate of incremental subgradient algorithms. Stochastic optimization: algorithms and applications, pages 223-264, 2001.
[40] Nocedal and S. J. Wright. Numerical Optimization. Springer New York, 2 edition, 2006.
[41] Antonio Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics SGD with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. Advances in Neural Information Processing Systems, 35:26943-26954, 2022.
[42] C. Paquette and K. Scheinberg. A stochastic line search method with expected complexity analysis. SIAM J. Optim., 30(1):349-376, 2020. ISSN 1052-6234. doi: 10.1137/18M1216550 URL https: : /toii org/ 10. 137/18M1216250.
[43] C. Paguette. K. Lee, F. Pedregosa. and E. Paquette. SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality. In Proceedings of Thirty Fourth Conference on Learning Theory (COL77), volume 134, pages
2021.
[44] Courtney Paquette and Elliot Paquette. Dynamies of stochastic momentum methods on large- scale, quadratic models. In Advances in Neural Information Processing Systems, volume 34, pages 9229-92400 2021. URL httta::://pooceeding newrips httpp//c/pppppp/////..... (wwee66644cfcbbff44444466662---0
Paper .ddf.
[45] Courtney Paquette and Elliot Paquette. High-dimensional optimization. SIAM Views and News,
20:16pp. December 2022.
[46] Courtney Paquette, Elliot Paquette, Ben Adlam, and Jeffrey Pennington. Homogenization of SGD in high-dimensions:: Exact dynamics and generalization properties. arXiv e-prints, art. arXiv:2205.07069, May 2022.
[47] Courtney Paquette, Elliot Paquette, Ben Adlam, and Jeffrey Pennington. Implicit regularization or implicit conditioning? exact risk trajectories of sgd in high dimensions. In Advances in Neural Information Processing Systems, volume 35, pages 35984-35999, New York, 2022. Curran Associates, Inc.
13

---

[48] Boris T Polyak. Introduction to optimization. 1987.
[49] Michal Rolinek and Georg Martius. L4: Practical loss-based stepsize adaptation for deep learning, Advances in neural information processing systems, 31, 2018.
[50] David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. In Advances in Neural Information Processing Systems, volume 8. MIT Press, 1995.
[51] David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21), 1995.
[52] A. Varre, L. Pillaud-Vivien, and N. Flammarion. Last iterate convergence of SGD for Least- Squares in the Interpolation regime. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[53] S. Vaswani, A. Mishkin, I. Laradji, M. Schmidt, G. Gidel, and S. Lacoste-Julien. Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates. Advances in Neural Information Processing Systems (NeurIPS), volume 32, pages 3732-37455 2019.
[54] Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien. Adaptive gradient methods converge faster with over paraamttttttinnnnoonnnn (but you should do line-search). arXiv preprint arXiv:2006.0663 06835, 2020.
[55] M. Velikanov, D. Kuznedelev, and D. Yarotsky. view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta. In International Conference on Learning Representations (ICLR), 2023.
[56] R. Vershynin. High-dimensional probabiiity: An introduction with applications data science. Cambridge University Press, Cambridge, UK, 2018. doi: 10.1017/9781108231333 URL
https://doi ttt//...00/////.o1..... 1111777777111881155666
http:////oooooooo.o..oo 1017
[57] Bohan Wang. Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. In The Thirty Sixth Annual Conference on Learning Theory, pages 161-190. PMLR, 2023.
(58) Chuang Wang, Hong Hu, and Yue Lu. A solvable high-dimensional model of GAN. In Advances in Neural Information Processing Systems, volume 32, New York, 2019. Curran Associates, Inc.
[59] Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research, 21(1):9047-9076 2020.
[60] Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning, pages 23549-23588. PMLR, 2022.
[61] Xiaoxia Wu, Rachel Ward, and Léon Bottou. Wngrad: Learn the learning rate in gradient descent. arXiv preprint arXiv: /08.80833...... 2018.
[62] Yuege Xie, Xiaoxia Wu, and Rachel Ward. Linear convergence of adaptive stochastic gradient descent. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1475-1485, 2020.
[63] Junchi Yang, Xiang Li, Ilyas Fatkhullin, and Niao He. Two sides of one coin: the limits of untuned SGD and the power of adaptive methods. Advances in Neural Information Processing Systems, 36, 2024.
[64] Yuki Yoshida and Masato Okada. Data-dependence of plateau phenomenon in learning with neural network- statistical Systems, volume 32, New York, 2019. Curran Associates, Inc.
-tttook-statisticalll mechanical analysis. In Advances in Neural Information Processing
14

---

The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms
Supplementary material
Broader Impact Statement. The work presented in this paper is foundational research and it is not tied any particular application. The set-up is simple well-studied high-dimensional linear composites (e.g., least squares, logistic regression, phase retrieval) with synthetic data and solved using known algorithms, e.g., AdaGrad-Norm. We present deterministic dynamics for the training loss and adaptive stepsizes. The results are theoretical and we do not anticipate any direct ethical and societal issues. We believe the results will be used by machine learning practitioners and encourage them to use it to build a more just, prosperous world.
A SGD adaptive learning rate algorithms and stepsizes
In this section, write down the explicit update rules for 2 different adaptive stochastic gradient descent algorithms.
Example: AdaGrad-Norm. the recursion, we have that
We begin with AdaGrad (see Algorithm 1). Note by unraveling
17
Ok
(15)
12 ,
with the deterministic equivalent (see Section and also C.3) for this learning rate being
(16)
V02 T(() S6 T( ds
In the case of the least squares problem, the quantity [()(tt) is explicit and
7t
(17)
122
2T R)
f (s) ds
Algorithm 1 AdaGrad-Norm
Require: Initialize > O, Xo E Rd, b E R and set 6o b x d for ..., do
Generate sample a N), Ex ~((,w?)) b% < 67 VX ax e2
0k-1 = x [o
updating learning rate updating step with stochastic gradient
X X Ok-1 Vx V
end for
Example: RMSprop-Norm We consider the "normed" version of RMSprop, that is, where there is only one learning rate parameter.
We consider Algorithm 2 where we put factor of the learning into the exponential moving average for RMSprop. The deterministic equivalent for g& for Alg. 2 (see Section 2) is
(18)
Vipe-at++ T((K)
d J ((((s)))((((s)) ds
In the case of the least squares problem, the quantity (()(t)) explicit and
(19)
Ve + 2T((K) f e --a(tt aat- R)))s ds
15

---

Algorithm 2 RMSprop-Norm, a Exponential Moving Average
Require: Initialize 7) > 0, Xy E R b and set by = x b, a 0 exponential moving avg. g-1 x 2: for k .... do Generate new sample dg ~ (O c ~N(0,0))
b/ by + V> ck)
9k-1 = d x lby
updating learning rate
X - X 0--1 (X
> updating step with stochastic gradient
end for
B The Dynamical nexus
In this section, we prove the main theorem on concentration of the risk curves and learning rates. We shall set some notation. In what follows, we again use W [X]]*] pax2 We also use W+ = [ [W] = [Xo]
We shall also use the shorthand (a, W), and (a, X) so that f((a, X), (a, X*)e))
f((a, e):e) = <r;
We shall let B = B(x) WWWW be the covariance matrix of the Gaussian vector r. We also write I' for the 0.
B.1 Discussion of the assumptions on
In this section we show how the assumptions we put on and are almost satisfied for L-smooth f. We say that f is L-smooth if:
( f (2) S LV r21l2 ||c c2ll?),
which we note implies ff is a-pseudo Lipschitz with =. Lemma B.1. differentiable and satisfies
There exists a function h R2x2 R such that h(B(X)) R(X) is
VxR(X)= V
Furthermore, h is continuously differentiable on {B det B + 0} and its derivative Vh. satisfies an estimate
[V)(B,)) Vh(Ba))l (v2++)))() B, 11p, | (pp)) [|B - B2ll
2. The function I(B) = E..el (T(B:) T(B2)) < (B)) ((B)) min B llops 135" (lop) |[B B2llr-
satisfies an estimate
Proof. To derive the existence of h, note that
R(X) [[E(f((a, (a,),,))))
is an expectation of a Gaussian vector = X),,X) X²)). This vector can be expressed as an image of an iid Gaussian vector 2 by representing r = / B2, and hence we have
h( B del (VBB/V(VBBB. ()))))
As the function f is absolutely continuous with Lipschitz gradient, integral sign and conclude
differentiate under the
VxR(X) vx E [(a, X), (a, X*), E 5 [( X), (a, Xx),)),
For the differentiability of h, suppose for the moment that f is C2 with bounded second derivatives. 10 Setting Q = /B the positive semi-definite square root of B, we have
((Q²) = B(E S( ((Q2,)))e e)))).
19hss condition can be removed in a standard way: one creates an f. which is an approximation to formed by convolving with an isotropic Gaussian of variance This is C and has bounded second derivatives (as f smooth). One then takes the limit 0.
16

---

Then using the chain rule, and setting 0 to be the i-th partial derivative of f.
0a., where we have applied Stein's Lemma. We conclude when det Q o 0 by the implicit function theorem that h is differentiable and have
= E( [(E))),),((22,,)))))
(
h(Q²) = Σ duchdo (QP) E(anh) + [(0rgh)Qss
As matrix equation, this be written (Dh)Q Q(Dh) = JQ where JM = J(==0(((0000)((030)))) This is linear equation in Dh. When Q x O. we can define
A (JQ)e dt,
and note
AQ + QA =
dt (e (JQ)e 19) dt = JQ.
IQ dt defines two-sided inverse for M + MQ+QM, and so Dh = A. Note that by symmetry of J, Q, and Dh
Moreover, the mapping M + Jo e-1Q Me-
JQ =Qh)= + Q(Dh) = QJ.
and therefore
(Dh)Q + Q(Dh) _ [JQ + QJ),
1
2
and taking inverses on both sides, Dh = J.
Undoing Stein's s Lemma, we have Q(Dh) (Dh)Q M, where My [(B(e,0.f(Q==.)))) From L-smoothness of
(Qt) ((2) < L [B Q2I1)) S v2LIIQ1 Q2l1F
Hence
[ Db( 1 (Q2)l
< 11 1lp (((1) Q1 2 M(22)l
$ l|Q 1lop (I)M((11) - M(2))l + 11(Q2 - 21) M(2))l)
Note ((2) (DD) is bounded by L(J), and so we arrive at
[IDN(QP) - D - (v2 + 1)L(TIIQT 1lp Q2llF
(V2
3 Q
We note the bound is symmetric in Qı and Q2, and by density of C2 in space of C1,bi,, this holds for L-smooth f. This concludes the estimates for the derivative of h.
For the Fisher matrix, I [B), from L-smoothness, we have again with Q = VB, I(Q2) ((E(((.)((2::))))))
Then
(T(O)) =(Q2) - ((E((O)((112))) (0./()222))))))))0)) Applying Sauchy-Sehwarz.n. and using the L-smoothness of f.
!T(0)) T(O3)) v T(QR)) + T(Q3) 1( Q2llr-
This lemma shows that an L-smooth function nearly satisfies Assumption and 4 provided that ||B-|| is bounded. Therefore, our concentration result Theorem B. and its Corollaries will hold provided we add a stopping time. Fix M > 0 and let
ha (B) def Then the concentration of the risk under SGD to deterministic function, Theorem B. holds with t replaced with A has (B) has ()). The corollaries of Theorem B. also follow under this added stopping time.
inf 0 : |B8 M).
In the next section, prove this concentration theorem, Theorem B.I.
17

---

B.2 Integro-differential equation for (t,
goal of this paper is to show that quadratic statistics 9: : Rd - applied to SGD converge to deterministic function. This argument hinges on understanding the deterministic dynamics of one important statistic, defined as
S(W,) = W R(2; K)W,
applied to Witd) (SGD updates). Here resolvent of the matrix K. The statistic S(W, =) valuable because encodes many other important quantities including W ( for all polynomials q. We show that ([))))).) is close
[X] and R(2;K) = ( =iaa for the
deterministic function (t, 2) (t, z) which satisfies integro-differential equation.
To introduce the integro-differentia equation, recall by Assumptions and 4
B(W) with B(W) = W KW, and 2-pseudo-Lipschitz functions h R2x2 + R differentiable and I I - R. It will be useful, throughout the remaining paper, to express Vh explicitly as 2 matrix, that is,
R(X) ho B(W) and La, w)
Vhu
Vh12
Vh
Vv2
V
hz2
With these recollections, the integro-differential equation is defined below.
Integro-Differential Equation for ((,, 2). For any contour Ω C C enclosing the eigenval- ues of K, we have an expression for the derivative of P:
(t)(t,)) = F(z, (t. dt
(20)
((=1) ff gtt, 2) dz 11
where J(z
27
2Ti R
of (t, de)
H" (28(t)
2τi Jn Tr TT(KR(2;K))]((
0
(21)
(2. (1))) (2=11 (0())
-1 $ (1, dz, H 4h2i (20 0
Vhn (
0
Here B(1)
2Ti 5
7 is defined in (9), and the initialization is g 2) W. R(2; K)W--
(22)
The functions h 22x2 R and 1 R2x2 + are defined in Assumption and Assumption 4, respectively.
We first note that there an actual solution the antegro-differenti equation. This solution the same as the ODEs defined in the introduction (see (9)) and proved in [15, Lemma 4.1J.
Lemma B.2 (Equivalence coupled ODEs.). The unique solution of (21) with initial condition (22) is given by
(tt,2)
Σ ==1
(t()).
A
In this section, we will be working with approximate solutions to the integro-differential equa- tion (20) (see below for specifics). For working with these solutions, we introduce some no- tation. We shall always work on fixed contour Z surrounding the spectrum of K, given by def {=: |a| = max (1, | [llpp))) We note that this contour is always distance at least 2 from the spectrum of K. We define norm, || lle. continuous function A C R
|| max |A(2)]]-
(23)
zei2
18

---

Definition B.1 ((E, -MMPr))pppiiiiaaa solution to the entegro-different equation), For constants
M.T.E we call a continuous function & [0. oc) x C - 22x2 an (E, M, (T.p,iT)approiiiie solution of (20) if with
def inf t t20 18(t, )1l > M }.
(5)
then
(s(t.)) -S(, S. F(. 8(, ds S
sup
(g
and 8(0, = W R(, W.. where Wo = [Xo|X*] is the initialization of SGD.
We suppress the in the notation for ē.. that is, PM FM (8), when the function S is clear from the context.
We are now ready state and prove one of our main results.
Theorem B.1 (Concentration of SGD and deterministic function (t, 2)). Suppose the risk function R(X) (2) satisfies Assumptions 3, and 4. Suppose the learning rate satisfies Assumption 6, and the initialization Xo and hidden parameters X satisfy Assumption 5. Moreover the data a NO, K) and label noise e satisfy Assumption Let {Waar} be generated from the iterates of SGD. Then there is an O so that for any T, M 0 and d sufficiently large, with overwhelming probability
sup
|[S(W (uj (t, Sd
(24)
OSISTAR (STTAAAASS((WW)))) (S
where the deterministic function (t, 2) solves the integro-differential equation (20).
Proof. By Proposition C.I, for any M and T. we can find à > 0 such that the function S(Wia s) an (d= M, a-ppprxppi solution. (For the deterministic function H, an (0, M, T)) approximate solution by definition.) We now apply the stability result, [15, Prop. 4.1], conclude that there exists a > such that
sup OSTSTARN
|t, z) S(Wa, 2)Ila S d w.O.P.
(25)
where FM is shorthand for fM fM (S(W.-)) FM (I). The result immediately follows.
Corollary B.1. Suppose the assumptions of Theorem B.1 hold. Let f be an a-pseudo-Lipschite function with a S. 1 and let be polynomial. Set
(X) def f(W* ((K)W), @(t) del o(t)=
( (())s dz), where (t, solves (20).
2Ti Jn
Then there is an > 0 such that for d sufficiently large, with overwhelming probability,
sup
lip (Xea) - <(t) s d
SEET
Proof. This is basically equivalent [15, Corollary 4.2]. The only difference is that [15, Corollary 4.2) requires the boundedness of N: however, since our function f is e-pseudo-Lipschie with a S 1, this boundedness follows from [15, Proposition 1.2], and the rest of the proof is identical to the one in [15].
Remark B.1. The learning rate 8k. technically, is not function of W g(K)W. However, As- sumption 6 ensures that the learning rate concentrates around a function W T ((KW. Therefore, Corollary B.1 applies to the learning rate.
C SGD-AL is an approximate solution
We introduce a rescaling of time to relate the k-th iteration of SGD to the continuous time parameter t in the differential equation through the relationship [td]. Thus, when / = 1, SGD has done exactly d updates. Since the parameter t is continuous and the iteration counter À (integer) discrete,
19

---

to simplify the discussion below, we extend k to continuous values through the floor operation,
Xx def [aj. Using the continuous parameter f, the iterates are related by X = X [tdj-
The paper [15] provides net argument showing that we do not need to work with every on the contour defining the interooddifferentil equation, but only polynomially many d. Recall that
Ω = {= | max (2 K llop }} For fixed Ç > 0, we say that Ωg is a d-S-mesh St if 9g C Q and for every с f there exists E E Ωg such that |z ž < d=c. We can achieve this with S2g having cardinality, [ = C(
Lemma C.1 (Net argument, [15], Lemma 5.1). Fix T, M > 0 and let & > 0. Suppose Ωg is a d mesh of S2 with [Ωg] = -C.d ds and positive C > 0. Let the function S(t, = S(Wid, z) satisfy
sup |s(t, S(,,-) (SAS(FMATT
g(, )) ds|la; E
(26)
with Pas inf{t 0 [|s Ilo M). Then S is (e C(M, (olpp M,, approximate solution to the integro-difeeenniall equation, that is,
|[((,,)) (((,,)))((0,))) S: ((, dello S S sup (FM
where C C(M, || op, I(T), [())) a positive constant.
(We prove in Section C.1 that S(t, 2) does indeed satisfy inequality (26).) We also cite the following lemma, which relates two stopping times used throughout this paper.
Lemma C.2 (Stopping time, [15], Lemma 4.2). For a constant C depending on |lops we have
C |S(W All
||all 22
Remark C.I. Fix M >> and define the stopping time on Well. 0 UMs by
1 Dm def inf {t 20: || > M). Due to the previous lemma, any stopping time FM defined on [1, )lls2 corresponds to stopping time 1 on [Wzall, that is, for c C-1, TM S 9.M.
C.1 SGD-AL is an approximated solution
Proposition C.1 (SGD-AL is an approximate solution). Fix a T, M > and 0 < 5/8, where is defined in Assumption 6. Then S(Wia, 2) is a (d= ; M,, MMMM)aaapraaaeee apprroxiiate solution w.o.p., that is,
sup |[S(W id, 2) S(Wo, =) =) 1 F(z, S(W.,,()) ds|la S d OSEE(TATH)
(27)
Again, the proof is very similar to [15, Prop. 5.2]. The one difference is that the martingales and error terms are slightly more involved, because of the non-deterministic stepsize we are using. The remainder of this section, along with section C.2, fills the details of bounding these lower-order terms, so that the proof can proceed as in [15].
C.1.1 Shorthand notation
In the following sections, we will be using various versions of the stepsize 7. In order simplify notation, we set
(Gx)) =
Qudxx )));
(9x) = ). :)),
(Bx) = =>(k, No (X) Qx
Further, setting As del (r( define
(k) def AJ (Xx) (kk def
1a(k) de (Xx))) D
The normalization here (dividing by d) is chosen so that the 7 terms are 0(1); this is formally shown in Lemma C.5.
20

---

C.1.2 SGD-AL under the statistic
We follow the approach in [15, Section 5.3] to rewrite the SGD adaptive learning rate update rule as an integral equation. Considering a quadratic function yp Rd E and performing Taylor expansion, we obtain
(Xx+1)=+(XX) 7( V
7G²)²² 4
V (Xx))
(28)
2d²
We will now relate this equation to its expectation by performing a Doob decomposition, involving the following martingale increments and error terms:
AM
grad
(4))
(->(G+)/<()) E [+GG)
(29)
d
AMR
Hess
(4))
def (r(G))² E [x(G)) [Fa)) [x)),,
(30)
2d
Hess ((p) Fi
del
(E [T(G,) (b) [i]] -Ba) (B.))
(31)
2d
E T (() Fil def
(-E [>(G)] [i] + VR(Xx))
(32)
d
We can then write
((X++1) (xxx)) BBx)
VX(X)) (Ba)²
2d Tr T (Xx)) Fa)
+ AMM prad k (p) + AM Hess (p) + E [ (4) Ful + E ( (g) Fi..
Extending X₁ into continuous time by defining X; [r). we sum up (integrate). For this, we introduce the forward difference
(Ap) _ -((x))))((XX)
giving us
[td -1
(X,) der ((Xo) S(" d. (Xx))) ds + Etds
((Xua) = (x(Xo)
Σ
where [Eea] /
d. A (X,d) ds
max osii[ta]
(Ap((1)))) With this, obtain the Doob
(led)-1)//
decomposition for SGD-AL:
((Xia) x(Xo) - -
VR(Xad) ds
(33)
1 f/' (( [ss) ds
+
2d
[d] -1
+
Egll (sp)
==
with
(v) AM grad (x) + AM Hess (xp
(34)
EE (6) Fyl + E Iaar (4) [F)]
+ E.(())
From here, we can proceed as [15, Section 5.3] to show that SGD-AL is an (c, M, approximated solution.
C.1.3 S(Wea: ed. 2) is an approximate solution
Proof of Proposition C.I. The appropriate stepsize, as function of Wi.
Yt ytd, Noa, (B.)) Qra).
21

---

(Note that N, I and Q can all be found as functions of S(Wid. .) using contour integration.) It is shown in the proof of [15, Proposition 5.2] that given the analogue of (33) for deterministic stepsize, S ((Wad, satisfies
[td] -1
S 2) = S (W). + J. F2, S (Wad: 2)) ds +
Σ
5y (S).
The only terms of (33) that differ in our case are the martingale and error terms. Thus to show that S(Wa., :) is an approximate solution of the integro-differential equation (20) all we need is to bound the martingales and enror terms contained in Eill. Let f {z |2| max [(oopp)]: as previously. We thus have that for all 2 E 2
S(Wia: 2) - S(Wo, 2) - L.¹ F(2, 2)) ds OSISTATM
|[ (S(; (35)
sup
sup OSISTAPM
Qg such that 2 S d-c and the cardinality of S2g, [gg]] Cd where C 0 can depend on || For all Ω2 we note that fM I HeM (see Lemma C.2). Consequently, we evaluate the error with the stopped process 1 def Wa(ene) instead of using F..- By Proposition C.2, the proof of which have deferred to Section C.2, we have, for any 8 > 0
Next, fix constant 5 0. Let S2g C S such that there exists
sup =E USISTADEM
sup
S d 6/44+ w.o.p.
(36)
We deduce that
|S( S(Wo, 2) S F(2, 2)) ds|la, S 2d0-8/4 w.o.p.
sup DSICTARM
An application of the net argument, Lemma C.I, finishes the proof after setting å = 8/8 and
f 8/8.
C.2 Error bounds
All the martingale and enror terms (34) go 0 as d grows. Formally, Propesition C.2. Let the function f be defined in Assumption 2. Let the statistic S (0, x C x be defined as
S(t, = [tdj R(2; K))Waa
[ta].
(37)
where W = [X] x Then, for any E 9 and T, M, >> O, with overwhelming probability,
[S (S(:, S d-8/445
sup oSiSTAB
where to suppress notation we use 1 shorthand for DeM. and e is the constant from Lemma C.2.
Proof. This follows from combining Propositions C.3, C.4, C.5, C.6, and C.7.
The remainder of this subsection devoted to proving these supporting propositions; throughout these proofs will work with the stopping time 9 defined in the proposition above.
C.2.1 Bounds on the lower order terms in the gradient and hessian
Proposition C.3 (Hessian error term). Let f and S be defined in Assumption 2 and (37). Then, and Ç > 0, with overwhelming probability,
for any z E Ω T
[dd]
|
gfter (S(, 2)) Fil]l < d 6/4+5,
sup OS1STAR
22

---

Proof. For arbitrary 2 E Ω and & - A 1,t
set - S (W, to be the ij-th entry of the
matrix 2), Then
2d Ed
=
+ (((())))n(BB)))))))0 | (B&)) I2(k) Fi
+ &3..
We look at |& first.
|1 E [(T(G()² (96))) [F]]
S E [[O()))2 (9h)))) 2 | E (1) Fx]]
< E [n(GG) + ((6)1 (G() - ( |=]] E [k))()) 1 Fxl]
< E [n((G) + (9 |F
-E [[7(Ga) (G))) |r,1* E (1i) 1r1]
For the first term, we use (6). We have
E [n(Ga)+ + (9w)) Fa] S (()) E [2 2 + 211 + II + 1x Fi
All the terms inside the expectation, apart from || fl are deterministic with respect to Fg and bounded by a constant independent of (see Lemma C.6). Since we know from Lemma C.6 that for any 0, all moments of ||& are bounded by dF w.o.p., we conclude
E [>((G)) (Gx)) [F
S df w.O.p.
For the second term, use (5). Again, since || so and ||Qu|lco are bounded due to stopping time, we have
E - ((()) FF < d 5-4
[F + is also bounded by constant (see Lemma C.5), and all together, The last term, E (A))²
we find that |E,] S d=-5/1 with overwhelming probability.
Now let us consider [
|E21 = (((())) ( ((())) = (9() + +B(aa) -1())))())))))----1))))))))))))) - ~ (() ( Fi]ll-
The first term bounded by (6), since Gr and (B())((B:)dd are bounded independent d: the second term is bounded Cd-1 by Lemma C.9, and the last term is bounded by constant by Lemma C.5.
Finally, consider |Es]:
Egg = rBb)2 EE (4) 12(k) FA]l
By (6), the first term is bounded by (())² ||N& I + 11Q11& + 1| (():(()(( these terms are bounded by a constant independent of d (because of the stopping time.) The second term satisfies the assumptions of Lemma C.8 with H (X)(X,,, and is thus bounded by Cd-1 All together,
All of
2d Ed (4) F < 5/4+5_
Summing up to ls = T' and dividing through by 2d, we obtain the desired bound.
Proposition C.4 (Gradient error term). Let ff and S be defined in Assumption 2 and (37). Then, for any E Ω Ç > 0 and T > 0, with overwhelming probability,
gaad ((.. 2)) Fill < d 6/4+5
sup OSESTAR
23

---

Proof. We have
dl (E
FR. -1 VR(X+))
-(0))1)))) (r(94) (Bxx) (1a(k) Fi
= =2
We then have
|& E [T(( (99h) |Fs E [Ta(A)] |
S [n(GE)++ + (9 | +
E [[((G)) 7(9)) |Fa) E [1rs(t)] IFi]²
Just as in the Hessian argument, (6) lets us bound [T()) + (9 Fi by dF w.o.p..
(5) lets us bound [l>(GR) - >9*)] |F&} by d-5/1 w.o.p., and Lemma C.5 lets us bound Fi by a constant, giving an overall bound of |E1] - d d/4/4
[k)]
By the same argument as in the Hessian case, |E2] bounded by Cd=1; in conclusion, FR. S 6
dE
Summing and dividing through by d, we obtain the desired result with Ç =
Proposition C.5 (Gradient martingale). Let I and S be defined as in Assumption 2 and (37). Then, for any E Ω Ç 0 and T O, with overwhelming probability,
Maraa [dt] (S(:, =)) 1//1/
sup OSESTAB
Proof. For notational convenience, set AM AMend
d(k/ano) and F5 (GG))(((()) so that
AM = F5 E[F& Fzl.
Set k = (Fk), that is, ensuring F3 stays in [-B, Then F E [x is in [-23, 2,3].
and so for the martingale Mi with increments AM F E F
FR], Azuma's inequality tells
us that
6
P (IM)) 21) S 2eep
< 2exp
2 (233)
2T
Set d-1/1 and t d 11/22;;; this becomes
(MM 2 1/2t)) < 2exp
-ds
8T
However, M₂ fl is not quite the martingale we started with: there is still an error term,
IM -M
Σ
(Fk E[F
(F)
E[FE [Fll)
h
F
8 k
+ E(
F Fx
We bound this term in overwhelming probability. We have
Pb F + 0)
P ([fkl > B)
= -P -P((G))))(())/) d112//2)
<P(T(GW) 2 (8(1) +P +P((T(())) 2 0814)
24

---

The second term is superpolynomially small by Lemma C.S; the first term is superpolynomially small by (6) and (C.6).
E Fx Ful
| F2) 11)111)))3)) Fiel
$ [Fx 1 3 [ (1 [Fr]
- F FF. B (ド Fi.1
4d (G+) | - | E |Fx]
As before, the first and second expectations are bounded by constants, and the last expectation is just the probability that [Fk] > B, which we have already shown superpolynomially small. So with overwhelming probability, we have
MM - - = Σ E(Fe - (F -E[F (R))
I 1/24
1=0
(any power of would have worked). Combining the error term and the projected martingale, we find that, with overwhelming probability,
[M&l < 1112+55
We can now take the maximum over is from 0 to Td using union bound; this does not affect the overwhelming probability statement.
L
Proposition C.6 (Hessian martingale). Let f and S be defined as in Assumption 2 and (37). Then, for any E Ω2 S > 0 and T >> O, with overwhelming probability,
(S(, M (5(: & d 1/2+5
sup 0SiSTAR
Proof. The proof here is basically identical to the previous one. Again, set F3 = (k)) and Fx Proj,(FFs) with their associated martingales being M₁ Fx E[Fx Fx) and M¹
r - FR]. As before, Azuma's 's inequality, with 3 = d 1+ gives
dc
P(M 2 d 1/2-5-5) < 2 exxp
87
The error term is also quite similar:
Mr
M
<
Fk
F
E[Fs - F Full. +
1=0
We have
P - p 0) < <P(((G))) & ds/4) (())(()) S 1F
both of which are superpolynomially small by (6) and Lemma C.5. For the expectation, we have
F | <
Ed F
(A) [b (1FA))))))) |Fi]
this product is superpolynomially small by (6), Lemma C.6, and Lemma C.5. Overall, we have, with overwhelming probability,
[M&ll S dd-1/2
Taking the supremum, we obtain the desired result.
Proposition C.7 (Integral error term). Let and S be defined as in Assumption and (37). Then, for 2 E S
LEn (S.( = 1/12
Proof. We have, as above,
KEal = =1
d. Ae((Xa) ds
(Ltd)- 11//
< max [5)
(X))(
which is bounded by 1/2 w.o.p. by the boundedness of 11, 12, 1a. and 7(x)).
25

---

C.2.2 General bounds
In this section, we make use of the subgaussian norm || llls of random variable (see [56] for details.) When it exists, this norm is defined as
| x inf {v > 0:00 0, P([X] 1) 2e (
(38)
In partieular, Gaussian random variables have well-defined subgaussian norm. Lemma C.3 ([15), Lemma 5.3). There exist constants C >> O such that d|[W]| Σ [[S(W, 2)lla & WC|,, VVx S C'[WW],, and S(W I C. Lemma C.4 (Preliminary bounds). With f and D. defined as above, for > and 0, we have de w.O.p. and
(1(x)) S | d S de
(rw)) Fil S C(A);
(39)
|
w.O.p. and
EE
| S C(A)-
(40)
Proof of (39) in Lemma C.4. By [15, Lemma 3.4), if function is o-pseudo-LLisschiee with Lipschitz constant L(f) (as in (2)) and the noise is independent of a, then
1f'(r) C(a)(L()) |r| + |e] ma)(m)))..))
Then
[f((r)) (()) |r + |l|) m=))) I C(())L(())( + X ax+1| |c] ml)xmxl..))
(41) write ax+1 VKv,, for standard normal Up. Then
Now, since ax+1 is Gaussian, that X 0g+1 X KKy langle-variable Gaussian, with variance [X]] KX& ||X]||?]|]] (bounded independently of d because of the stopping time on X2). Similarly, e is Gaussian and independent of ax+1, so the expression (41) is bounded w.o.p. by d", and
E (C(a)(1()) + [X] ax+1 (c))max(1.0)) | S(())
for some constant C(A).
Proof of (40) Lemma C.4. We can write a&+1 v Kun, where ty is a standard d-dimensional normal vector. Then, by Hanson- Wright, we have
PP (Ilaa+aa 1 B[[|a++1|| 2| Fill d) TP (10g Ke
Elu
Ke
Fill
2 d)
cd2
s 2 exp
||| + ||
lopel
cdf
< 2exp
d([KK] + |
< exp (-Cd).
Now, note that Elu Kvu Fi] = Tr(K)
S dl llpp- Together, we get that ||a S dlte with
overwhelming probability. Then
A 1
|ax+ 1 (1x))(r2)²
d
which is bounded by de w.o.p. Now for the expectation:
b/2
|
A
| K
2A
E
Fx
< E
Fr
EE (
d
d
[Kllppp
1/02 1|2
<
E
Fr
[
E
42 |Fr
(42)
d
26

---

For the first term, we have
|| lop
|uull
21
11 11
22
FF
=
2x
FF
d
d-1
K
12
E
(11 24 |F
(Jensen's inequality)
K Top
E
[
(i.i.d. assumption)
where we are using the notation oi to refer to the ith component of the vector vh. Now, since of is just a standard Gaussian, all of its moments are bounded. The second term (42) is bounded by constant by (39), as desired.
Lemma C.5 (Gradient and Hessian bounds). Setting
(k) def
Ax/d,
12(k) def (4
T2(k) (x((x)) AR.
for any & > 0 and 2 0, we have
[7( S d=
w.O.p. and
E [[][(]]]] S CC(A,
(43)
[k)( < C,
(44)
[a())] S df
w.o.p. and E [[s(K)) FR] S C(A).
(45)
Proof of (43) in Lemma C.5. Using the fact that | ( Mlop - |IS lla
A DTV2p(Xe)A)
||S(W |12//
d
d 1
(Lemma C.3)
Now, ||Wll is bounded by the stopping time. From Lemma C.4, || is bounded by df w.o.p., and every moment of this expression is bounded independent of d, as desired.
Proof (44) in Lemma C.5. We have
Tr (X
Fa]
d V (Xx) Op (T)) Fa
< (> Ilop || lop LE (T Fx]
$ CM E[f [FR]-
(Lemma C.3)
From Lemma C.4, [ [i] is bounded by constant independent of d, as desired.
Proof of (45) in Lemma C.5. We have
[Vp(X)) S (Vx( a (1"())))
By Lemma C.3, [Xv)( S C| S CM (since we are working under stopping time), and Vx((X,) T "a+i is subgaussian (and thus bounded by d' w.o.p.). By (39), f'(rx) is bounded by de w.o.p., and so their product is bounded by des w.o.p., as desired. Now for the expectation:
E [Vx))
FA] < E V (Xx) @ktil (rx)
(Xxx ax+1 Ful [
The first term is bounded by constant independent of d, since subgaussian moments are bounded. The second term is bounded by Lemma C.4, completing the proof.
D
27

---

Lemma C.6 (Infinity norm bounds). For Gk. No. Qs as defined in 1.2, we have, for any À >> 0, there exists C' such that,
|Gall>> S de |N S C, 11Q11 C,
w.o.p. and
Bll|G Ao Fi 2
w.O.p.
(46)
|| llee C.
(47)
Proof. The first line, (46), follows from (40). For the first inequality, ||G maxosjsk |1 which are all bounded by df with overwhelming probability. A union bound tells us that the maximum also bounded by dº w.o.p.. For the second inequality,
A 12
11,,122 A
| 120
FR) SE
E
Fk
max 0SISk-1
Fk
d
||
112
A
||,11 A
E
FR
max OSYSk-1
d
d=
(w.o.p.)
as desired. The second line more straightforward:
|| = max (1(W)) W
USiSk
Now, || and | are bounded independent of d, and ||X; |l is bounded by cM (because of the stopping time we are using.) Thus the maximum over j of their inner products are bounded by constant. The same thing holds for [|Qu]loo:
1|Qx
max R(X,)
OSiSk
max h(, KW,). 05i5k
Since the derivative of arguments. And indeed, the argument to h is bounded:
is continuous, and thus bounded for bounded
W KW, || - [[5 | |||
lop.
both of which are bounded independent of d. Finally, similar argument applies to Gu:
| 11
|| loc max E OSiSk
F3
max C=
C
d
OSJSk
by Lemma C.4.
We now prove concentration result that closely follows [15, Proposition 5.6).
Lemma C.7 ([15), Lemma 5.2). Suppose v Rd is distributed N ( Ia) and U e Rdx2 has orthonormal columns. Then
v V (Ue~r-U(TT) LUTT.
(48)
U- u-U (UT ~ N U
and U N (0,UUT) with v-U (o) independent
Lemma C.S. For a matrix H Hy with bounded operator norm, or ||Hllop < and BiH Fi] =
HAs set ((a) = T Ha. Then
(f) - T( | S C(H).
Note that the H used here is not the same as the matrix used in the integro-differente equation.
Proof. Many of the computations in this proof are taken directly from [15], but we repeat them here for completeness. We have Fs = o(WW))&)) set F afWW) fr)) A simple calculation shows that
Elg(ar)1) F [[g[ax+1 - Eia++1 Fxl) [f
+ q(Eaau+1 (xl)) E. (rx))
(49)
28

---

To compute the conditional mean E[ax+1| Fi] and covariance (aR+1 - Elax+1 FRl)(at+1 Blax+1| [x))T, use Lemma C.7. By Assumption I, we can write ax+1 v Kuk, for 2k (0,
Now perform QR-dccomposition vKW2 Q&RR where Qs R with or- thonormal columns and Rx E R2x2 is upper triangular (and invertible). Set 11 def Q&Q In
distribution.
a*+1 4 W & VKrb R
As Rg is invertible, by Lemma C.7,
0k+1 0x+ W Vnn Q Up (u(ux-leva) + KIlLEu
(50)
We note that (La U1 ~ (0,, La 11,) and Ug ~ (O, 11:) with (La 11. Ju independent of Il 2g. From this, we have that
E [ak+1 Fill = VKIl.%. where 2 ~ (N
(51)
Moreover the conditional covariance of ax+1 precisely
(E [(ax-1 - E [ax+1 Faa)(a++1 - E [ax+1 ((ee) [Fa))
(52)
-V
where II, Q&
Next, using that H FR) Hg. we expand (49) to get the leading order behavior
E ((((a))))f(>)) {+]= = Tr(HK) (ef (rx))
THHV V K) (r()
(53)
+ (V(K)l E.I (rx)))
Taking the expectation with respect to Fs. we obtain
E (q
Fa]
| E[ Fa).
(54)
where the error E is defined
Ex (IHHVIII K) (r)
(55)
+ ([VKKL+ vR) E. Lf (r.))].
(56)
The proof now turns to bounding the expectation this error quantity.
Fbll THHHKKI / K)) (rx())))) [Fr
|| || op [lop Tr( BIf (rw)² Fin] S || H op Klppp rank Blf (rx)² [
< 2|| [pp |K] || (rr) [F))
By (39), the expectation is bounded by constant, so this term is overall bounded by a constant. We move on to the next term in the error:
II Uj frx)) q( v (TI)))))(( < || K
Taking expectations and using Cauchy Schwarz, we obtain
Ela( So(VKTIN (Tr
[Fa] < H opl
K
lpp
Fi
V (
The first expectation is E[ Teux We thus conclude that E Fx is bounded by constant depending on ||ppp completing the proof.
Fi) = | | 8, and the second is bounded by (39) as before.
Lemma C.9. There is constant C such that
1r(G)) xBBB)l S Cdd1
29

---

Proof. Using the Lipschitz condition on the stepsize, we have
1r( (B&)l
- 19 [B)/d<( x (1+
11 + | [T())(()))]] 21 (Lemma C.6)
I Cl19 - (B:)dd
$ Cdd max
(.)
OSigk
< Cd
(Lemma C.8)
as desired.
C.3 Specific learning rates
In this section, we confirm that AdaGrad-Norm satisfies Assumption In the notation of Assump- tion 6, we have, for AdaGrad-Norm,
(ld,S,9950 S99.9) = 12 So g(x) ds
Note that this reduces the discrete stepsize if we plug in g Gy:
([d, x ),9)
12 + -o G+(ds) ds
+ 10 Eh
a4 (4r (u..)))) ds
7
62 + 1o0 (LuSk) Ei
( du
V18 + de Σk Vx VX ++1))
which is exactly the discrete version of the AdaGrad-Nor Norm stepsize.
Proposition C.8 (Lipschitz). For functions S,9.9 q such that [ds) = g(ds) g(ds) = =0 s 1 the AdaGrad stepsize is Lipschitz. That is,
(td,,, f(d x g(d x (,d x )) - r(ld, f(d ), ), )) S ((4 gl
Remark C.2. This is a stronger condition than the a-pseudo Lipschitz one in Assumption 6.
Proof. To show this, we look at the derivative of the AdaGrad stepsize function. Setting F(x)
Vi242 have
F'(x)] 2(12 + 3//2 < 17
263
30

---

for x E [0, oo). We thus have
(td,, f(d x ) g(d x) x)x (( ((td, f(d x )),((d x -) 1. x ))]
7
7
62 So g(ds) ds
/62+ So g(ds) ds
g(ds) ds F (So ((ds) ds
7
g(ds) ds Lo ((ds) ds
213
7
g(ss)
((ds) ds
213
"
(t |1g 91(x0)
213
npt [1g 011x0
263
where we were able to replace the oc with a / because g(ds) = for >> L. We have thus obtained Lipschitz constant at depending only on t.
25
Next show that the AdaGrad-Norm is bounded.
Proposition C.9 (Boundedness). Suppose is AdaGrad-Norm. Then (6), as part of Assumption holds.
Proof. This is immediate:
f(9.)
1
V12 + g(5) ds
It remains show that AdaGrad-Noma satisfies (5) in Assumption 6.
Proposition C.10 (Concentration). Suppose is AdaGrad-Normm, with GA and Gn being defined as before. Then Equation (5), as part of Assumption holds:
([)) - (9 Fx] & Cd + I1 + [1alls))
Proof. Looking to remove the square roots, have
hn(G)) 79t)) h(2)) (9
For AdaGrad-Norm, we have
1/2
1/2
Σラ= |
12 B
y
k
<
Σ LE |||
| -
(57)
d2b
We now bound the sum above. Set F. = || || /d, F.e = (F)) AM, = F;- F[F [F]. and
AMC- = FE Fi). Then AM E [-28, 28), so Azuma's inequality gives us
-t2
I (IM@) 2 2) s 2exp
2 (
d2+2
P (IMM) 2 d212te) < 2exp
2Td(2d:/2/3
exp
8T
31

---

where we set 3 = d=/2. This is close to the bound we want: the error is
MM M - Σ Fi F.
E
F.f
Fill.
We have
|
P(F;
F. 0) P(F)) 3) =
d=12
d
which superpolynomially small by (40). The expectation is similar:
E
P.
Fill
E F;
P. (1 Fill
< F; F. P EI (1)|))) [
< FE| | E (I |
The first expectation is bounded by constant independent of d by (40), and the second expectation superpolynomially small by the same argument above. We then have
M M d//2+=
with overwhelming probability (note that this would be true for any power of d, by the definition of superpolynomially small.) We thus conclude that
M& 1/2+z
with overwhelming probability. Multiplying by d, we find that
113) s 3/2+ w.o.p.
Plugging this back into (57), we find that
1/2 83/2t
Pr(G))2-(9H1) S
d b4
S Cd 1/2+c
with overwhelming probability, and so, taking the square root,
hn(G,) - (Gn)) S Cd 1/41/42//2
w.O.p,
which less than d' 1/4++ as d grows (we replaced the constant with an extra factor of d²/2.) Controlling the expectation via the boundedness of y, we find that with d = 1/8,
B[]+(G+) - r(Gx) Fil S d w.O.p..
as desired.
D Proofs for AdaGrad-Norm analysis
In this section provide proofs of the propositions related to AdaGrad-Nome in the least squares setting as well as the more general strongly convex setting. Statements of the propositions for least squares examples are found in Section 4.
D.1 Strongly convex setting
In order to derive the limiting learning rate in this case, we need the following assumption and some standard definitions of strong convexity.
Assumption (Risk and loss minimizer). Suppose that
X E arg min (R(X)= = E E,,,[[((X, a), (x*, a)),
exists and has norm bounded independent of d. Then one has, for almost surely N0, K) and e.
Xx" a) arg min, (f(a, (X* a)),
32

---

While at first, this assumption seems quite strong, in fact, in a typical student-teacher setup when label noise is 0 (i.e., 0). where the targets have the same model as the outputs, the assumption is satisfied. Our goal here is not to be exhaustive, but simply to illustrate that our framework admits nontrivial and useful analysis and which gives nontrivial conclusions for the optimization theory of these problems.
Definition D.1 (L-smoothness of outer function f). A function E R. that is Ch-smooth (in the first variable) is called smfooth if the following quadratic upper bound holds for any I, R. y; 2 E R
f(2, v, 2) S [x, v, (f' u. 2), x) L( i x/?
(58)
Note that if f" [a; y, 2) is [(/)-Lipschizz i.e., [s" y: 2) - (0; y/, =)1 S L(ʃ)= - -i, then the inequality (58) holds with constant L. Suppose x² arg min [[ =)} exists. An immediate consequence of (58) that
i(f)
(x f(x,8,2) f(x", (,x",,,2) 22
5
|- x=112.
(59)
2f(
2
Definition D.2 (Restricted Secant Inequality). A function f R + R. that is C smoooh (in the first variable) satisfies the (u, resstriedd secant inequality (RSI) if. for any E R and x² E arg min (z)),
Ju|z 2013, if max{[= 1, x² S0.
(x-x" x= x==x)(((x)) 2
0.
otherwise.
If f satisfies the above for 0 = cc. then say f satisfies the p-RSI. Proposition D.1. Let the outer function f R + lR be a i(J]-smooth function satisfying the
RSI condition with jAJ) with respect to x E RR Suppose X* E argminxx {R(()) exists bounded,
independent of and Assumption 7 holds and that "(o b ( TY(K)* foror some Ç E (0, 1),
20(l)
and that Jo (8)sss ds < oo with Ys as Table (AdaGrad-Nome) general formula), then
2017
700
1 5 1- 22
Proof. Given the Eq. (87) for the distance to optimality, with (Ir V),
(t) 2% Ex,e - x", f' x² M + d Tr(K) (r.)) (f
dt
By the RSI (with constant ji(s)) condition on f. we have that f (xxxxx)) x²)) x²))| 2 j(f)E [(e x =
E (x x,, f(
(60)
where x = (X, a) and x= = (x*,a) a) and we note that x has t-dependence due to the t-dependence in D. By (())-mmoothness
1 (F(x))
in (x =2
2i(l)
This implies that
1
((-()) B ar (1,
- E [ x-)=]==---- (1). 2
(61)
Thus by (60) and (61), we have that
d
o (t) S 7o 2((C)) @(11)
di
Which then yield:
( TT()))) (8))).
33

---

Changing variables u r(t) So Yo ds, we have that S0¹ 2(t) So r(u) du
|||| Rearranging the term in the above equation and taking t + co. We obtain: ||rl S 7o. Using Lemma D.I, with i(v)
(20())((((1)) n=(K)o)' given that ( Tr(
(0)
2p(A)
> 70.
((T(T=()))) E a,c (ff'
instead of the risk
112
Yoc
1 Tr [ Jo i(e) de
TT())(()) So r(v)
(62)
2d
n
x?
Tr(K)
(L()))
D(KIUUI -)()((
22(0)
(2A()- Tr(
where the first inequality by Eq. 61, and the last transition by taking the initial learning rate be Yo = 20(
S, for Ç E (0, 1).
Lemma D.I. Given as in Table 2 (AdaGrad-Norm)) defining g(u) >-((T(())) with I(t) n²
So its ds, then g(u) 3 (xK) Je s( with i(v) = 1 (x))
Proof. Taking the square of both sides of the n equation in Table 2 (AdaGrad-Norm)) changing variables to u = (tt) and rearranging the terms:
i(U)
nf
+
TK(K
du
(63)
g(0)
g g(
such that =(v)= = (())() Taking derivative with respect to u, rearranging terms and integrat- ing leads to the desired result.
D.2 Least squares setting
To study the effect of the structured covariance matrix and cases in which the problem is not strongly convex, we will focus on the linear least square problem. this setting, the continuum limit of the risk for the AdaGrad-Nome Norm algorithm has the form of a convolutional integral Volterra equation,
(1 F(T(t) S (2(((()) (())) ds
(64)
where [(1) = fo 7s ds with.
F(x) def 1
X (0)e 2A:5
(65)
2d
d
K(x)
def
X2e
(66)
d 1=1
In the following consider three cases, strongly convex risk in which the spectrum of the eigenvalues bounded from below (section D.2.1). case in which the spectrum is not bounded from below d + oo, but the number of eigenvalues below some fixed threshold is o(d) (section D.2.2). Finally, power law spectrum supported on [0, 1] with d - xo (section D.2.3).
D.2.1 Proofs for case of fixed d!
Proof of Proposition 4.2. Define the composite functions r(u) = R(T-)(u)), and g(u) (u)). Integrating the formula for the risk:
34

---

J r(u) du
[- F(u) duuddu- [. ( ds du
(w))())
l F(u) du = - S(uu-xr((jj) dxd
/ F(u) du + 7o I r((x) I K(u - x) du dr
-
Taking t + we get
||||| S || + rol/ ||||||||1-
Using ||K||] = So K(x) dr < % and noting that by Eq. (66), and Eq. (65), we have that
[[]|| = (0), and ||i||1 = 22 Tr(K),
||
20)
|r|| -
7o [/]l] 1- 70 Tr(K)
On the hand following Lemma D.3, (0) Tr(K)) ||rll Therefore, ||||| (0).
Next, rewriting the "/ equation in Table (AdaGrad-Norm for least squares) in terms of g(u) (Lemma D.I), we obtain
g(u) =
(67)
Tr(K) So r())
Taking u + oo, and using ||||| x (0))
n
v/
Yoo = g(0) =
(68)
Tr [r)]l
# + T2(K)
This then completes the proof.
Remark D.1. We note that, on the Least square problem i(n = pCf) = 1, therefore, the bound in Proposition D.I yields
Proof of Proposition 4.1. Using the equation for the distance optimality (Eq. 8), we can derive an
equation for the integral of the risk (with no target noise) which we denote by g(l) fo (s) ds:
Tr(K2),
g0(1) = - Σ (1) + + d
(()).
(69)
For K = Ia. this equation simplifies.
Tr(K2) g"(1) = -2g ( + +
d gt))).
(70)
Plugging in the equation for the AdaGrad-None learning rate (Table leads the desired result. We note that by using the equation for the learning rate, one can also derive a close equation for the learning rate itself.
D.2.2 Vanishingly few eigenvalues near 0 as d + oc
We now consider the case where, as d + oc, there are eigenvalues of K arbitrarily close to 0. In Proposition 4.2 we saw constant lower bound on % when d fixed (and thus there are finitely many eigenvalues within any fixed distance of 0). This can be extended to the case where we have some C > O such that the number of eigenvalues of K below C is o(d) (see Proposition 4.3).
Proof of Proposition 4.3. Following the structure of the loss, after some time the risk starts de-
crease, and therefore (1) < Ro for and 0. Using these observations, we obtain a preliminary lower bound of x > 11/21/2 (for 1. > 0), which enables us to deduce that (1) is integrable and finally obtain a constant lower bound for ?r The details of this are below.
35

---

Fort t 2 0 and some C1 > O,
7t 62 + á Ir(K) So ((s)ds
7
n
11-
(71)
V + 3 Ro(
Next, to show that the risk is integrable, we divide the matrix K into two parts K+, and K_, such that the eigenvalues of K are greater than some at > 0 and the eigenvalues of K are smaller than 0g where 0g is decreasing function of s to determined later. We then have that, following Eq.
(8), and the definition of the risk (t)
Dd Ei X (t),
' 7. 1 / 21 Tx(3 (()dss
=()=
X
(72)
i=1
S (0) [-¹ V/6 (2a, - Ir(K3)) 2 (s)ds + 2 Yo (s) ds
with R2(s) = 2d Ensso, (s)) Next, choosing a, = 7/ Tr(3), we show that the last term is of order (1). By Lemma D.2 vi, (1) max (7r, (41), (()) co where the bound co comes from the assumption (X* (x+,w;) 1/2) and the initialization Xo 0. Therefore,
S To (2) ds S Tr(K2)oo S[ No
1 d²
(73)
where N, = Σ.= 1 A Ti(Ke)- This implies that, if N.No o(d), then 2 So B2 ds od(1), provided that d is taken be large before
We then have that up to (1(1) constant,
1 Tr(K3) - ( 2())ss 2 S ( -
(74)
Using Gronwall's inequality,
(t) < (0)e - de S (0)e IKK
(75)
where in the last transition used the lower bound the learning rate derived in Eq. (71). Thus, the risk is integrable, i.e. there is some C3 such that
J. (s)
O0
T
for all t >>> 0. Finally, we plug this into the formula for and conclude that, for all f >> 0,
n
tt 2
(76)
1 TO
62
TKKKJCT
Lemma D.2. Assume that the risk is bounded and attains its maximun time tj. Then, for each i, we have (t) < max((%,4 (t1), (0) for all 2 O.
Proof. Case 1: Suppose that 27 < 2o (0). Then, by equation (8). d (0( 2 0. However, di
since (t) are continuous, this equation implies that (?( S (t) for all and thus 2?(()) S 7,,90 (1,) for all t.
Case 2: Suppose that ()( (0). Then, by equation (8), बे o (0) 0. If 4 27(1) < 0 for
all then 2?(())= (0) for all some point de g?(() > 0. this implies 9?(() S (t)((t and we are in Case 1.
In the next section, we consider cases in which the risk is not integrable, an example of such case when the spectrum of K is supported on the interval [0, or has power-law behavior near 0.
36

---

D.2.3 Power law behavior at d +
Non-asymptotic bound for the Convolutional Volterra In this section, Volterra structure of the risk (Eq. (64)) to derive non-asymptotic bounds on the risk, which will be useful in Section D.2.3 to derive the asymptotic behavior of the risk and the learning rate under
the convolutional
power law assumption on the spectrum of the covariance matrix and the discrepancy from the target at initialization.
Lemma D.3. Let [()) == So Yo ds and lei
( F(T(O)) J.' ((((((()) - (()))
where nt, K. are monotonically decreasing, with ||K]|| < co. Then all t,
(t) 2 F(T(t) + [[. -xx(((())) r(s)) -( ds
If in addition, there exist > 0 and >> such that, for all t > T.
f." K())K(( - ds s 2(1 + {)]l{])((( and 21] + 6) <
then for all t
(((0))+(/// +C +c
211 s P(T(I)
for
K((0) C= (K(T)(2++1) +2 (100////](((
Proof. The lower bound holds trivially, using R(s) 2 (s)( For the upper bound, we start with the following change of variables:
- F(T()) + f u)) du,
r(t)
with g(u) "r (()) Let define the convolution map
(()(T) K. (+N(T) = ( K(T <(T-w)/(w) du.
'.''
Next show that this map is contracting and in particular,
/ K(tt (s)
(2(()= = -(G((())
(77)
1 K(t 3) 1 K(ss ()g(u)f(u) dug(s) ds
1'¹ (J"
-()(( ()((s))((s w)g(s) ds (()) du
/ (t((-))g(u)) ( du
where the third transition is since < The last transition is by change of variables and the assumption that " a monotone decreasing function. Consecutive application of the convolution map will then yield by induction,
gi(((t) & ((00))(x<00pp)) (t=-((t-)g(uu))) ()g((u))(u/e f())()((()aaae du.
37

---

Therefore, expanding the loss and using the above upper bound, and denote by q- = 2(1 + + such that q 1.
( F(t) + Σ (F)(()
(78)
< FF()) + Σ
12 (t))(t()( F(u) du
u)g(u>}
j=1
Σ (2] + +)}> -1
- -1 Ci L. ()))))
< F()) +
$ F(t) + 1-q Ci(K * (t)))(t)
4
(79)
K(O) where the third transition is by Lemma D.4, with C x(T)2++1] + +1, which then completes the proof.
Lemma D.4 (Lemma IV.4.7 in [3]. Suppose K is monotonically decreasing, with |/[|| oo, and that there exists T > 0 such that Vt 2 T, and 2 0.
J K(s)K((L- ds 2 x]]
(80)
Then,
k (1)
K(0) (2T + 1) +1)
sup 120
(1))))
(81)
K(tt)
Proof. Define an sup,20 ((0(21)]]))) trivially a1 1. Consider the + 1 convolution,
(()
((+)(111)" K((t) L. (ts) s) ds (21)
(t)
(82)
By the assumption of the Lemma, we know that there exists some T > O such that for 1I 2 T
((s))((tt 3) ds S (1)))(((
(83)
||
Therefore, if 2 T, we have
k(tt) / ((s))(-(1- 8) ds
(84)
(211/(11)"
% 211/T11 K((- s)(2 n ds < aa
((())(((t -) 8)
kt - 8)
On the other hand, if < T.
1 ((s) 8)
['"
XC(O) (t)n(t))l <
KC(0)
ds K(t)& (2)
(85)
K(T) (2)" K(T)2"
Taking supremum in Eq. (82), and combining the results of Eq. (85), and Eq. (84), obtain that, ant1 S K((0) K(T)22
cc
Solving the above recursion equation,
KC(0) n-2 2 K(T)
IC(0) (2(1 + d))=
On <
2 - (1 + e) (1+6)==-1 KC(T)2n 2(1 + + (1 (1-
0
< (1+6)
(2() 11) +1).
K((0)
rearranging the terms arrived at the required result.
38

---

Asymptotic analysis of the risk Here, we consider family of models with d + oo, for which the following power law asymptoties assumption is satisfied:
Assumption 8. ((r) x- and K((a) = x= for x 2 with Ki 0, 82 >
Corollary D.I apply Lemma D.3 in the setting for which F, and K. has a power law behavior asymptotically. It shows that the risk will then be dominated by F only. Corollary D.2 shows the behavior of the learning rate in this setting. Finally, Lemma D.5 shows that Assumption 8 is consequence of a power law spectrum near zero on the eigenvalues of the covariance matrix and power law assumption on the projected discrepancy at initialization.
Corollary D.1. Suppose Assumption 8 is satisfied, then (() = ((T(())-
Proof. Define g(u) (w) and r(u) (u)) and observe that g(u) is decreasing function. Then, from the upper bound in Lemma D.3, we have
r(u)= & F(u) + I.
( x)) du f (()))(u-
1//2
F(u) +C
u/2
(86)
(()) for d(o)
11 -N1
1 K(u v) du
F(u) C19(0)
2
Ju/2
I F(u) + C2(u -kt1--k + u (((]) =F)(.)).
Combining this upper bound with the lower bound from Lemma D.3 and that K2 > 1, we conclude that r(t) x (()) and (1) x ((T())).
Next, we derive the asymptotics of "/- There are three different cases, depending on whether the risk integrable, which translates threshold with respect the parameter K1.
Corollary D.2. Suppose Assumption 8 then the following asymptotics for the learning rate hold:
For K1 > 1, there exists 7 such that 7/6 2 5 and (t) "1 for all t 2 O.
For K1 < 1, Tt x (1-x))((2212111 m1) and (8) - - for all t 2 1.
For K1 = 1, it x Tog(r+1) and (t) x (oo((t1)) ) 1 for all 2 1.
Proof. Using the notations g()) and r(u) defined above along with the change of variable u = [()), we get Jn (()s ds Jo œ(v)
r(v) du. Combining this with Corollary D.1 and the formula for % we get
s(u)= x
"
1 (T((K) fo
(1=w) #1
du
g(w)
Let T(u) = b2 + TT(( So (1+e)-"1 de and observe that s(u) x ((n)
1
and F((u)
Viai
(T((K) (1+4)-** Thus, (u) satisfies ('()) g(u)
u1) "s so we have
Vi(a)
/ (1 (1+ dv.
VI[u)
In the case of K1 > 1, this implies VI(u) < (0) + C [(1+0) dv. This upper bound on / (u)
gives corresponding lower bound on g(u) and thus lower bound on 7/-
In the case of kn <
have / VTu)
T(O)
+ - so, for u sufficiently large, g(u) x
(1 + u))1-1, To recover the asymptotic for 7. we observe that duc r- (u) g(u) x (1 + u)
Integrating both sides and changing back to t variables, get x (1 + r(t))2-x= (or equivalently
39

---

1 T+t(t) x (1)() Finally, plugging this into the formula for "/ and applying Corollary D.1, we get
"
(1 +1 (1+
7 62 + (T((K fo F(T(s)) ds
In the case of Ki we follow similar procedure as for K1 to show that x ((t) (t)
for sufficiently large t. This implies [t(t) x t/ ltg(t) which gives the desired result after integration. The decay rate of the risk is then immediate using Corollary D.1.
Lemma D.5. Let K have spectrum that converges d + oo the power law measure p())
11 (0,AAns)) with C-1 X--8
for some B <, 1, and Amax > 0, and suppose that P2(0) X,
then Ftt) x =t= and K((t) X2, with K1 satisfies Eq. (80).
8, and fig 3. In addition, K(t) x t t2,,
Proof. Following the definition in Eq. (66), and Eq. (65)
1-3 Amua
F(x)= = 2 J
A 11-8-5e-20 -2Ar
2h Yd
1-
1-
7(2 B 3 8, 2m
12-8-6e-0 dy
2Amax (2x) 8-
21-823-8--6
2- 8-6
Similarly for K.
1 3
Amux
73 - B, 2dmax 1-
K(x) =
2Az dl
1-88 J
21-823-3
x3-3 B
with (8, S0 --- da is the incomplete gamma function. For large z, 7 2) T(s), the complete gamma function. We therefore obtain 11 = 2 - 8 - 8, and Ae = 3 - B. Next, we show that K(x)) x <a satisfies Eq. (80).
/ ((s))((t- - s) s)
1TT K(())C(( (())K(t-s) ss ' K(t)(((t- s)
1/2
J1/2
k((t/2) (f018 K(s) + S" K(tt ss
J1/2
S 2(((/2))
by the power-law assumption for / T, K(1/2) K(() which then complete the proof.
Proof of Proposition 4.4. The proof is an immediate application of Corollary D.2 with, K1 = 3 implied by Lemma D.5.
Remark D.2. This includes the case 3 0, which is the uniform measure [0, Amax].
E Polyak Stepsize
The distance to optimality of SGD is measured say by D² - Let us consider the deterministic equivalent for the distance to optimality g²( in (11). Fixing T > 0 and any & E (0, 1 1/2), we have by Theorem 2. (see also corollary B.1 which show concentration for large class of statistics) that suPoSrST [||X X o (t) < e= w.o.p. In this way, if we want guarantee that the distance to optimality of SGD decreases, we need (t) < with the maximum decrease being min,, ?t).
As turns out, the evolution of 92 is particular simple, as it solves the differential equation (derived from the ODE in (9))
A( = Ea,e(xx - xx, f("
d (2( = -2)A(B(1))) + of (())()
[( = B., (x()))** where (87)
dt
d
(a @ (xxx*) x²) ~ N(0,8))
40

---

101
SGD
111
Ueery limit
10
110k af
1/21
1/2
Ap 1/2
10
Ap 1/21
(9)
Ag 1/2
Line
Gime
Figure 5: Convergence in Exact Line Search on a noiseless least squares problem. The plot on the left illustrates the convergence of the risk function, while the center and right plots depict the convergence of the quotient Sx, (8) and the learning rate Yt respectively. Further details and formulas
(x)(t)
for the limiting behavior can be found in the Appendix F.2. See Appendix H for simulation details.
The distance to optimality threshold, 5% occurs precisely when d 0. This choice of makes the ODE for the distance to optimality stable. By translating the relevant deterministic quantities in 72 back SGD quantities, we get
2(Xx X VR((x)) Tr( d E ( a): (X*, e)
(.((
@ 9k
with the deterministic equiv.
Tr(K) 1 (2)
(88)
greedy learning rate that maximizes the decrease at each iteration is simply given by g arg min (t). This has closed form and we call this Polyak stepsize"l Again translating this back to SGD, we have
Polyak E
Polyak learning rate
Polyak
70% and deterministic equivalent
Tt
Polyak
2
(89)
In this context, the Polyak learning rate impractical because do not known X*. In spite of this, we can learn some things about this learning rate as it is the natural extension of Polyak learning rate SGD.
The quantities A ( and ( in (88) and (89) only depend on the low-dimensional function f and thus do carry any covariance or d dependence. Moreover, under additional assumptions on the function such as (strong) convexity, we can bound from below A(2)/ ((()) Thus, in terms covariance K and d, the Polyak stepsize 9g.
Polyak
TT(K)/(a)
avg. eig
In the case of least squares (see (7)), we get
(2(Xx)) w and on noiseless least squares,
Polyak
Polyak 9%
9
2Tx() R(Xx) R((E)
Tr(K)
d
The latter gives the best fixed learning rate a noiseless target on LS problem (as noted in [35, 43)).
F
Line Search
F.1 General Line Search
Naturally, one can ask similar question in Polyak in the context of line search (i.e., decreasing risk at each iteration of SGD). First, by the structure of the risk (Assumption and 4),
V m and TH(VR(()K)) v(K).
(90)
Therefore using (9), we have that the deterministic equivalent for ||V((X) is Ml 2d 1-1 m²))(t)))). In this case, the deterministic equivalent for the risk A satisties the following ÖDE
d (t)-)-(((ttd ((K)(((()).
(91)
1'Tis is the idea of Polyak stepsize when the problem is deterministic.
41

---

o From this, we get an immediate learning rate (stability) threshold for the risk, that is. 9k is the largest learning rate for which SGD is guaranteed to decrease each iteration, i.e., when the deterministic equivalent of a satisfies d. < or equivalently after translating relevant terms into SGD quantities
|R((x)]1
M (t)
risk threshold g
and deterministic equiv I(WTKW))
T((V'R((N))
x(K) (t))
d
(92)
The greediest approach, which we call exact line search, would choose the learning rate such that line E arg min, d.R. In this case, we get 1t
line Ok
and deterministic equiv Tt
line
7
1 2
F.2 Line Search on least squares
In this section, provide proof of Proposition 3.1, but, we show more than this including the exact limiting value for 7/-
Proposition F.1. Consider the noiseless (w = 0) least squares problem (7). Then the learning rate is always lower bounded by
Nain (K) Tr(K3)
S line for all t Z O.
Yt
Moreover, suppose K has only two distinct eigenvalues X1 > A2 > O, ie., K has d/2 eigenvalues equal to Xn eigenvalues and d/2 eigenvalues equal to X2. In this context, the exact limiting value of line is given by It
2 2 + Xīx)
lim k+000
Tt
(93)
(A1 + X2 (Af +
where x is the positive real root of the second-degree polynomial
P(z) A2de + 1)(22x \1) + (22 \1) 12.
(94)
This leads to
Amin (K) lim gt < 2mmin (K) Tr(k2)
(95)
80000 d Tr(K2)
Proof. We establish the inequality
Amin (K)
I line for all 2 0
Tr(K2)
by observing
X (t) 2 2Amin (K) 1 d
1
Σ (t = 2Amiu (K)))(t).
d
2d 1=1
i=1
Now let us consider K ~ 111 + 1/2 for \1
A2 0.
We define (1)
-,, (t). Utilizing the ODEs in (9), we derive
d 2x(t) = -2:19 x dt
|{A WHEL x (1)
for each distinct eigenvalue À of K. Here I(A = > is the number of eigenvalues of K that Thus, we
are equal to A. It immediately follows by our construction of K that {{x > d establish the following system of ODEs
2.
d 24, (1) = -2 (1)
di
(96)
d
(t, = 22 + +t)
di
(0)
where 2(1) z (AI (1) + X₂ 22, (1)) and
line
7
(0+,,,
42

---

Since (t) 2 0 and \ > X > O, we infer that ((t) 2d \2 2
1 Bx, (t) 2 0. The structure of the exact line search algorithm ensures lim p( 0.
hence lim Dx, (t) = O. Similarly, we deduce lim Dx =
By applying H' rule and substituting the expressions for Mmee and (1) in terms of 2x (1)
and 2x₂ (1), we derive
lim (1) = lim t-000 2x, (t) 10000 d2, (t)
x (t)
-2 (x, + (2)
lim
t-x -27dA1 (1) + d7
-212 (1)+ (7x2237)) lim
+
1-x
7/ 2 (1)4))11)))
-211 2x, (1)
lim
A1 (1 - 2)
7/ 2
= lim
1o (10 + - -2 + 1r2
A2 + li (
X - 22 [lim--00
[x,(()
2
(-13 - 2x718)
(1,1
(- - -27713) lim
lxp (t)
121x + \ラ1Y - 2\1 + [lim
(tp
\1
D), (2)
(1) Therefore, lim;-+00 3n, (t) is the positive real root of the second-degree polynomial
P(z) = A1 + 1)(2²z An) + (A2 21)222.
(97)
Solving for > 0, we derive the explicit formula
lim 2x (t)
8000
2x (t)
(98)
x - 2A/d2 + 22
X + V
4XP12 + 88711
6AP1g + + - 4 + +1
2
Given
9x2 90 (t)
2 (t) (39,
22
7t
line
1A, (1) A2 (1)) (AF X (1 + A2 20 (2)
(AP x2)
(99)
we have
2 2x² + >3 lim 3r, (2)
(t,
lim
?t
line
(100)
()1 X2 lime Dr (1)
(1,(1)
(A1 + 13)
By substituting (98), we get
lim 7t
line
1-30
X + +-22222222 + 2 + +1
VX 4Xid2 + 8X123 - ttttg 8XPA8 4x1 X (Y) + X3)²
(101)
direct calculation reveals that X Ao > O implies lim 7 line (ka
[T((K2)
Remark F.I. For the scenario where K has an arbitrary number n of distinct eigenvalues, equation (13) remains valid. The proof parallels the one outlined above. However, in this case, the expression for limg-ro Ok is given by
12 (A1 + A + In (X} + .... +X2)'
(2 + 23 +
lim Ok k0000
(102)
where ,i,..., In-1 0 satisfy more intricate coupled system of n 1 equations.
43

---

G
Examples
Any single index model with a-pseedo Lipschitz (a v 1) activation function is covered by SGD+AL theory. In this section, we provide key learning problems within this family of models.
G.1 Binary logistic regression
We consider binary logistic regression problem with 0 where we are trying to classify two classes. We will follow a Student-Teacher model, which there exists a true vector X to the true direction such that possible labels are, y = exp(( (a))++" or - y. In order to classify the data we minimize the KI-divergence between the label and our estimate defined by the above formula,
exp( .a))
exp((X",,) a))
R(X)= E. a (X
exp( +1 + log (exp (X, (exp((X, ax)
(103)
To study the ODE dynamics of SGD in Eq. (9) needs the deterministic risk h (B), and I(B) E Ealf'((X, a), (X*, ") with B = W KW. Following the computation in Appendix D example D.4 in [15] obtain that
exp( (2B22 2)
LBB) B21
+ E [og(exp((w B11) + 1)],
(104)
(1 + exp( v B22 2))
where (O 1). The / function can also be computed explicitly by solving the following Gaussian integral, where we define g(x) def exp(a)
I+exp(y)
I(B)= = 2T det(BB / (g(x) 2(v))² exp
x ) dæ dy.
B-1
(105)
2
y/
We note that, the logistic regression is (H, RSSI with teves see section 2.2 in [15]. Its Lipschitz constant is ī(f) = 1. Using Proposition D.1 one can derive lower bound on the limiting learning of AdaGrad Norm.
For more details and more examples, see [15].
G.2 CIFAR 5m
Finally, we include an example that uses real-world data, that is, the CIFAR 5m dataset [38]. Our theory does not explicitly deal with non- Gaussaansi distributions, but find that the theoretical risk curves generalize cleanly to that case.
As are now working with discrete data points rather than a distribution, the learning setup, while closely analogous what was presented earlier, has some slight differences.
We start with a subset of the data consisting of n grayscale images, each which is 32 x 32 pixels, 1024 We fill vector b E I with the corresponding labels (0 for an image of plane, for an image of a car.) We then randomly choose a matrix W E IR 1024xx with i.i.d. Gaussian entries generate the features F' relu(AW). We want use least squares to predict the label from the features, i.c., find
that is, E 10224
arg min ElR
R(X)= = 2n
FX - 611
2n Σ
(fe X -b)2
(106)
where fi is the äth row of F. The SGD we now consider is
X = Xx
(Jiut1 X biees) ficn
(ix} iid U ,,)),
(107)
where " is the usual AdaGrad- stepsize, as in (15). Our empirical covariance matrix K (remembering that f, a row vector) is then
[ fll - -F n
K = E
(108)
We now use (64), with the AdaGrad-None stepsize, to numerically simulate the SGD loss, which we then compare to the actual loss. Our theory matches empirical results very closely.
44

---

CIAR Ara Norm Least Squares
10
2048
4296
8192
6x:10
Empittall Iiss
4x10
2x12
180
16
102
160
5GD teerationsdd
Figure 6: Predicting the training dynamics real dataset, CIFAR-5m [38], using multi-pass AdaGrad-Norm. This suggests the theory extends beyond Gaussian data and one-pass. Note that the curves look significantly different for different nı smaller values of lead an overparametrized problem, allowing least squares to memorize datapoints, whereas for larger n, least squares must learn general function mapping images of cars and airplanes their respective labels.
H Numerical simulation details
Here we provide more details for the figures that appear in the main paper.
Concentration learning rate and risk for AdaGrad-Norm on least squares problem with label noise w (left) and on a logistic regression problem with no label noise (right). For logistic, see Section G. 30 runs of MaaGrad-Nome with parameters b and 1) for each d; X ~O, Ia/d), Xu = 0, and = Ia. The shaded region represents a 90% confidence interval for the SGD runs. As the dimension increases, the risk and stepsize both concentrate around deterministic limit (red). The deterministic limit is described by an ODE in Theorem 2. The initial loss increase in the least squares problem suggesting that the learning AdaGrad-Norm naturally adapts and still the loss converges. Our ODEs predict this behavior.
Figure 1:
initially too high, but
Comparison for Exact Line Search and Polyak Stepsize a noiseless least squares problem. The left plot illustrates the convergence the risk function, while the right plot depicts the convergence of the quotient 71/ Amin (K) for Polyak stepsize and exact line search. Both ODE Tv(
Figure 2:
theory and SGD results are presented, showing close agreement between the two approaches. The covariance matrix K is generated such that the cigenvalues follow the expression Ax(K)
1/s i l,...,d, where 2 is a constant. As approaches 2,
V (LLT
the spectrum becomes more spread out, resulting in larger values of d Tr(K?). Larger values of correspond smaller spreads in the spectrum. Additionally, Tr(K)/d= for all Both plots highlight the implication of equation (13) in digh-dimensional settings, where a broader spectrum indicating slower risk convergence and poorer performance < ()))
of K results in Amin(K)
Tv(K3)
of exact line search (unmarked) as it deviates from the Polyak stepsize (circle markers). The gray shaded region demonstrates that equation (13) is satisfied.
Figure 3: Quantities effecting AdaGrad-Norm learning rate. (left): The effect of adding
noise to the targets (w 1.0) to the risk (left axis) and learning rate (right axis). Ran AdaGrad- Norm(b = 1.0, 2.5) on least squares problem with d = 500. Xg. X* ~ (N Id). single of the SGD (solid line purple) matches exactly the prediction (ODE, teal). The shaded region represents 10 runs of SGD with 90% confidence interval. The learning rate decays at the exact learning so it approaches 1. (center, right): Noiseless Depicted asymptotic
predicted rate of
7
least squares setting (w 0). (center): Prop. 4.2 predicts the avg. eig of K (Tr(K)/d) compared with Xmax effects limg Bs. Indeed, this is true. We varied the k = Amaxx Amin while keeping the Tr(K)/d /d and all other parameters fixed. All the learning rates behave identically verifying our
45

---

theory about the effect of Tr(K)/d /d on learning rates. (right): Varying the learning rate of AdaGrad norm by ||X III our predictions (dashed) match and we see the inverse relationship predicted by Prop. 4.2. See Appendix D for details. Additionally, we did the following.
Center plot: AdaGrad with b 0.5, 1j = 2.5 is run on the least squares problem with d 1000 and Xo, X Va (0, (). The covariance matrix K is generated so that the eigenvalues are
d
-1/8
(K)
i= i=1....,d d.
Σ
2/8
d+
d+1
The constant 2. When is near 2, the spectrum is more spread out, i.c., Amaa is Xmin
large. Larger values of mean smaller the spreads. Moreover Tr(K)/ /d for all In the simulations, we used E (2.1, 3.0, 3.5, 4.0, 5.5] and recorded the condition number k.
Right plot: Ran AdaGrad with = 0.5, = 2.5 the least squares problem with 1000. X and Xo VfNN(,,1) where p E {1, 2, 4, 8, 16). In this way, || - X p.
Figure 4: Power law covariance in AdaGrad Norm on least squares problem. Generated covariance K such that the density of eigenvalues are (1 3)X where 3 0.2 and
Xo Choose (XT) (X, 8/2 i= where A; is the i-th eigenvalue of K and we vary 5 E (0, 1.8) so that 0<6 8 + 3 S Setting of Prop. 4.4.
Figure 5: Convergence in Exact Line Search on noiseless least squares problem. The plot on the left illustrates the convergence of the risk function, while the center and right plots depict the (t) and the learning rate "i. respectively. Predictions from ODE theory are compared with results obtained from SGD, demonstrating close agreement between the two approaches. Initialization was performed randomly, with X, ~ ~N(0, Ia/d) and X VaL. where d 400. The covariance matrix K has two distinct eigenvalues Àι 1 > X2 0, and was constructed by specifying the spectrum, with X; sampled from a discrete uniform distribution 441, A2) for ,,....dd d 400, and setting K diag(A; formulas for the limiting behavior can be found in the Appendix F.2.
convergence of the quotient 9x, (t)
1,..., 400). Further details and
Figure Convergence on CIFAR 5m [38]. We train classifier to distinguish between images of airplanes and cars. Fix d = 2000. Then for multiple values of n. we run AdaGrad-None with initialization Xo 0, 0.1 and 7 5, randomly sampling datapoint from F every step. Details of the setup can be found in Appendix G.2.
46

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction introduce the class of problems we consider in this work. We do not exaggerate our claims and state precisely the limitations of our work, namely 'high-dimensional linear composite setting' with normally distributed data in Section 1.1.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly state all the assumptions on the model and algorithm in Section 1.1. When the theory switches to the least squares problem, we indicate clearly in the theorem statements. For example, see AdaGrad (Section 4) and Line Search (Section 3). We are very careful in the statements of our assumptions needed to prove the result.
to include all the
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper contains 2 sections detailing the exact set-up of the problem class (Section 1.1) well as the algorithmic set-up (Section 1.2). All the proofs for the main results are provided in the appendix and we are careful to include all the necessary assumptions to prove these results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Either in the captions in Section H. we state all the parameters, covariances of the data used, initialization, etc that are needed to reproduce the results. We also intend release the code and make available via github. We also remark that the numerical simulations are relatively simple to reproduce as they are generated using synthetic data and the algorithms are already use.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In the captions of the figures and/or in Section we have provided all the details to reproduce the experiments. The data is synthetic. We also intend to release the code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
47

---

Justification: Either in the captions of the figures or in Section H, we state all the parameters, covariance of the data, initialization, etc that are needed to reproduce the results. We also intend to release the code and make it available via github. We remark that the numerical simulations are relatively simple to reproduce as they are generated using synthetic data and the algorithms are already in use.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the figures (especially Figure 1), we provided statistical error bars (shaded region). Part of our theory is that the training dynamics of SGD with adaptive learning rate in high-dimensions concentrate. We then predict these training dynamics exactly. In this sense, have used statistical tests illustrate theory.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The computing resources are minimal in our case as the models are just least squares and logistic regression with synthetic data. We explicitly state all the relevant details for another person to run the experiments. We remark that due to the simplicity of our simulations, no experiment required extensive compute capabilities.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http//nuusucccc//// httpc///cbcccthhccccseeeeellelee
Answer: [Yes]
Justification: We have reviewed the code of ethics. We have made our utmost attempt to adhere to the guidelines provided by NeurIPS. We do not any human subjects nor any datasets (privacy is not a concern). We did our best to cite the relevent related work. Given that our work is in the foundational research and, in addition, the theory side, it is difficult to mitigate all the risks as the downstream effects of theory are long. The model is completely synthetic using the standard SGD) algorithm; thus we don't 't, the best of our knowledge, anticipate any risks.
10. Broader Impacts
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Question:
Answer: [NA]
Justification: We do not anticipate any negative societal impact of the work. The work is a purely theoretical result on foundational research and it is not tied to any particular application. The work uses only synthetic data, standard algorithms (e.g.. SGD, AdaGrad- Norm) that have already been used in MI/AI, and models such as least squares and logistics that are textbook. We have included a paragraph at the beginning of our appendix justifying this answer to the broader impacts of our work.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks. The work is purely theoretical and uses only synthetic data generated from a normal distribution. The models employed are standard statistical model (e.g., least squares and logistic regression) which textbook problems.
12. Licenses for existing assets
48

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: We acknowledge via citations that the algorithms we study have been intro- duced before by others. We are not using any datasets or other assets beyond numpy and thus do not need to name any license or cite any dataset.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not intend to release any new assets. Neither the model nor the algorithms are new. We are simply analyzing known algorithms and models in this paper. We intend to release code for reproducibility. particular, we want readers to able to generate the deterministic ODEs.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. The work is purely theoretical on a simple model.
15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The work does not involve crowdsourcing nor research with human subjects. The data used in this work is generated synthetically.
49


Reviews:
[{"review_id": "inUcdcTbDa", "reviewer": "Reviewer_hpAG", "summary": "The paper focuses on analytical analysis of dynamics of a class of optimization algorithms with adaptive learning rate applied to linear models with Gaussian data. The class of algorithms includes AdaGrad-Norm, RMSprop-Norm, Polyak stepsize and line search, but doesn't include, for example, Adam, classical RMSprop and AdaGrad. Also, authors consider $\\frac{1}{d}$ scaling of learning rate with dimension $d$ of the problem.\n\nFirst, the authors establish convergence of the optimization trajectory to a deterministic ODE in the limit $d\\to\\infty$. Then, they focus on analysing the resulting dynamics for quadratic loss functions and specific algorithms\n- Plolyak stepsize. Obtaining learning rate. \n- Exact line search. Obtaining learning rate for two cases of data covariance $K$: isotropic and with two distinct eigenvalues\n- AdaGrad-Norm. $O(t^{-\\frac{1}{2}})$ learning rate asymptotic for noisy observations. Also, learning rate and risk asymptotic for isotropic and power-law data covariance $K$.\n\nThe theoretical results are validated with numerical experiments on synthetic data.", "questions": "**High dimensional vs. low dimensional problems**. Some covariance matrices $K$ satisfy assumption **1** but have $\\frac{\\mathrm{Tr} K^2}{d}\\to0$ and therefore produce trivial ODE dynamics coinciding with full-batch Gradient Flow (as can be seen from eq. 11), for example \n- rank 1 with non-trivial eigenvalue $\\lambda=1$ (extreme case).\n- with power-law spectrum $\\lambda_i=i^{-\\alpha}, \\alpha>1$  (e.g. classical *capacity* and *source* conditions).\n\nIndeed, for these two examples, the dynamics converge to full-batch Gradient Flow due to $\\propto \\frac{1}{d}$ scaling of learning rate in eq. 3, while learning rate independent of $d$ would be more natural. \n- Could the techniques developed in this paper handle such problems?\n- If no, is it possible to formulate a criterion on covariance matrix $K$ to distinguish non-trivial ODE dynamics from trivial ones (i.e. coinciding with full-batch gradient flow)? For general non-quadratic problems.\n\n**Results for power-law spectrum** The last section provides variety of results in case of power-law distribution of eigenvalues (with exponent $\\beta$), and target (with exponent $\\delta$). In the non-vanishing learning rate phase $\\beta+\\delta<1$, how the obtained rate $\\mathcal{R}(t)\\sim t^{\\beta+\\delta-2}$ compares with other results for SGD under power-law spectrum (e.g. [1](https://arxiv.org/abs/2006.08212),[2](https://arxiv.org/abs/2102.03183),[3](https://arxiv.org/abs/2206.11124))? \n\n**Learning rate phase transition**. For power-law spectrum, learning rate asymptotic exhibits transition at $\\beta+\\delta=1$, as demonstrated on fig. **4**. Interestingly, on this figure we can see that in both phases learning rate reduces similarly up until it reaches very small value ($\\gamma\\sim 10^{-4}$) at quite late optimization times ($t\\sim10^{3}$). Can this dynamic transition (w.r.t. time $t$) be explained theoretically, for example, with some quantities characterizing transition scale with $d$?", "limitations": "The paper surely doesn't have any negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The main result (convergence to deterministic continuous-time ODE) is valid for general loss functions. The precise characterization of the optimization trajectory, as opposed to upper/lower bounds, is typically harder to obtain for non-quadratic loss functions\n- For AdaGrad Norm algorithm, the authors characterized the dynamics of risk and learning rate for several distinct types of covariance matrix spectrum. \n\nHowever, both these points are achieved only partially (see *weaknesses* for details). Completing them would significantly improve the paper.", "weaknesses": "The main drawback of the manuscript is its writing. The paper mostly focuses on the introduction and setting, leaving too little space for actual results and their discussion. Out of 9 pages: \n  - Five pages are reserved for introduction and setup. \n  - Two pages are occupied by Section **2**, which mainly introduces notations and demonstrates them on the quadratic problem example. Indeed, it contains Theorem **2.1**, but that one is quite short\n  - Only the last two pages contain the main results of the paper. This leaves very little space for discussing them. Some of the discussions are present in the (quite lengthy) *main contributions* part, but, for readability purposes, it would be nicer to have this discussion together with or after the results.\n  - Any kind of concluding section is absent\n\nRegarding the figures, only figure **1** is referenced in the text while figures **2**,**3**, and **4** are not referenced. This adds an extra effort to interpret them and put into right context. \n\n**New developments from previous work**. As mentioned by the authors, the current paper follows the technical framework developed in [14], extending it to adaptive algorithms. Then, it would be good if the manuscript mentioned more clearly which results/notions were already obtained in [14] and which are new additions of the current paper.  \n- In particular, Theorem **2.1** of the current paper seems to be very close to Theorem 1.1 of [14] which can also handle non-constant learning rates. If two theorems are indeed close, it would be interesting to discuss new aspects introduced by adaptive algorithms. \n- Assumptions **1-6**, and the notations of sections **1.1**, **1.2**, **2** look very close to that of [14]. If that is the case, it would be interesting to discuss more specifically their differences. Also in that case, sections **1.1**, **1.2**, **2** could be compressed by partially moving them in appendix, leaving more space for the results and their discussion.\n\n**Results for non-quadratic losses**. Although the framework, as introduced in section **2** is designed for general non-quadratic problems, the specific results are given only for quadratic problems in the main paper. Some results for non-quadratic problems are given in appendix sections **D.1**, **E**, **F.1**, **G.1** (mainly based on results of [14]). In such form of these results, it is hard to extract interesting conclusions, or determine the contributions compared to the previous literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We address below the questions of the reviewer. \n\n1. **Isotropic covariance matrix $K$.** We emphasize that our results hold for *non-isotropic data covariance matrices $K$* (under mild assumption $||K||\\_{op} \\< C$, and the average eigenvalue of $K$ is bounded independent of $d$). We point to Reviewer ipsV who also noted the anisotropic assumption on $K$.   \n     \n2. **Writing of the manuscript.** Please see some specific changes in the “All Reviewers” section of the response. We appreciate the reviewer’s suggestions.   \n     \n3. **New developments from previous work \\[14\\].** See discussion to all reviewers.   \n     \n4. **Beyond non-quadratic losses.** In Appendix D, we show that under a restricted secant inequality, similar conclusions to the quadratic case can be derived for AdaGrad-Norm.   Now the results we get are essentially the same as one gets for quadratics.  We weren’t so keen to generalize the whole paper to something like losses satisfying an RSI if, at the end of the day, we just get the same answers as for quadratics.  Identifying interesting and important non-quadratics with different phenomenology for adaptive algorithms is an interesting direction of future research (e.g. nonconvex problems like retrieval or problems exhibiting implicit bias). Using Theorem 2.1, the ODEs are already in place, and so there is a framework for finer analysis of adaptive algorithms which was missing from the literature. We believe these methods can eventually be used to provide better or more informed adaptive stepsize methods.  \n     \n5. **High-dimensional vs low-dimensional problem extensions.** Thanks for these points – they are very good.  Please see the discussion in the rebuttal to all reviewers.  We’ll remark that the $\\\\text{Tr}(K^2) \\= o(d)$ condition you observed is implied by $\\\\text{Tr}(K) \\= o(d)$, so long as we keep bounded operator norm of $K$, and so it falls into the regime where SGD and the ODEs both need to be scaled differently.  We can add a discussion of this to the paper.  \n     \n   Part of the reason we wrote *this* paper is that we needed to demonstrate that it was actually possible to analyze these ODEs and derive meaningful conclusions about optimization algorithms.  Especially, we needed to analyze these ODES in the presence of anisotropy, which can heavily degrade the value of an adaptive stepsize algorithm (e.g. line searches work great in the isotropic case – and are essentially optimal – but they are bad in the presence of anisotropy.  In contrast, adagrad norm does much better, although still fails in some power law-type setups).  So half of this paper is analyzing various algorithms and half is formulating a precise theorem which says where the ODE comes from.  \n     \n   We hope this paper could be a starting point for us (or, gladly, other researchers\\!) to refine the method further, for example:  \n* Finding the optimal covariance conditions under which these ODEs hold and making a fully non-asymptotic comparison.  \n* Understanding the extra correction terms to the ODEs in non-high-dimensional setups, such as the classical source/capacity conditions.  (The rank-1 problem you mentioned, in contrast, is not well-aligned with the goals of this type of analysis, which we think could be summarized as: how to give meaningful quantification of adaptive algorithms on anisotropic problems).  \n* Digging deep into other adaptive stepsize strategies to see if they perform better in anisotropic setups.  \n* Showing a version of Theorem 2.1 for non-Gaussian data (high-dimensionality also gives a path to showing universality of the ODEs).\n\n\n  It could be, for example, that Adagrad norm on classical capacity source setups could be analyzed in the way we did and have interesting conclusions.\n\n\n6. **SGD power-law spectrum rate.** We thank the reviewer for the references. Under our set-up, constant stepsize SGD would be $\\\\mathcal{R}(t) \\\\asymp t^{\\\\beta \\+\\\\delta \\-2}$, and there is no transition at $\\\\beta+\\\\delta \\=1$. We will add this to our paper. The references \\[1\\],\\[2\\],\\[3\\] suggested by the reviewer do consider similar assumptions on the spectrum and eigenvalues of the covariance matrix, but the papers referenced are not in the high-dimensional regime (diverging $\\\\text{Tr}(K)$). All \\[1\\],\\[2\\],\\[3\\] contain results which formally correspond to $\\\\beta \\> 1$, so a quantitative comparison is not really possible (\\[3\\] discusses our regime here, but does not seem to have rates in this regime).  Qualitatively, they all demonstrate similar results as seen here (formally similar to how the loss undergoes a phase transition as the ‘source’ parameter varies in source/capacity setups). In the next version of the paper, we will add clarification regarding the scaling with constant stepsize together with comparison and citations of the relevant works.  \n     \n7. **Learning rate phase transition.** This is a great question and worthy of future research. While we did not keep track of the exact finite dimensional $d$ effects, one could in principle use our analysis to theoretically quantify when the asymptotics for the learning rate and the risk take over.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2IgUzykT9Z", "reviewer": "Reviewer_ipsV", "summary": "This paper studies Stochastic Gradient Descent (SGD) training with adaptive step sizes. The setting is a generalization of single-index models. There is a ground truth vector X^*, and the model must find a vector X such that a loss L(X) = E_{a,epsilon}[f(<X,a>, <X^*,a>, epsilon)] is minimized. Here epsilon is a random additive error, and a is distributed according to N(0,K) for a potentially anisotropic covariance K.\n\nThis paper shows that (1) in the high-dimensional regime, the training dynamics converge to a deterministic limit given by an ODE. It studies these dynamics in two significant cases of adaptive step-size algorithms: (2) exact-line-search (i.e., greedily decreasing the risk optimally at each step), and (3) AdaGrad-Norm. For exact-line search, it is shown that this can be very suboptimal if the data covariance is highly anisotropic. For AdaGrad-Norm, it is shown that if the data covariance has lower-bounded eigenvalues the optimal step size (within a constant factor) is reached. However, for harder problems with worse conditioning of the data covariance, AdaGrad-Norm can be overly pessimistic.", "questions": "I am satisfied with the presentation in the paper, and do not have questions.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper provides a fine-grained analysis of practical optimization algorithms, and also proves a general theorem on convergence to deterministic dynamics that I believe could be used to analyze dynamics for different adaptive-learning-rate algorithms in the future. There is also a strong match between theory and experiments. I believe it is an important step in the direction of understanding these practical optimization algorithms.", "weaknesses": "In terms of techniques, Theorem 2.1 on convergence to an ODE is based in large part on modifying a previous analysis of [Collins-Woodfin ’23] to the case of adaptive step sizes, so the techniques are not completely novel.\n\nThe data is also assumed to be Gaussian, although anisotropy is allowed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their comments. We will update the paper to emphasize the difference between our analysis and that of \\[14\\]. We state below the main differences between this work and \\[14\\]:\n\n* First, the learning rate is by itself a stochastic process.  In addition, since it carries all the history of the previous gradients, the iterates of SGD are no longer a Markovian process. We extend the proof of the previous paper to handle such situations by providing an additional concentration of the learning rate to its equivalent deterministic limit.\n\n* Second, there are differences in the analysis of the Volterra equation on the least square problem. Having the learning rate as a nonlinear function of the loss makes the analysis much harder, and we had to introduce new techniques to study this new generalized form of the Convolutional Volterra equation (see Appendix D).\n\n* Third, in this paper, we study in depth the effect of different covariance matrix structures. For the line search method, we analyze in detail the example of two distinct eigenvalues.  For AdaGrad-Norm, we study different power law covariance scalings and zero eigenvalues, for which we had to utilize some nontrivial asymptotic methods.  \n\nThe assumption of the data being Gaussian could be relaxed. Most of the concentration estimates we use to prove the deterministic equivalent are based on concentration inequalities, which are valid beyond Gaussian data. In addition, Figure 6 in the paper shows that we can predict multipass SGD dynamics on more realistic data (CIFAR-5m) which suggests that our theory extends beyond the Gaussian data and the streaming setting. \n\n\\[14\\] Elizabeth Collins-Woodfin, Courtney Paquette, Elliot Paquette, and Inbar Seroussi. Hitting the high-dimensional notes: An ODE for SGD learning dynamics on GLMs and multi-index models. arXiv preprint arXiv:2308.08977, 2023\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2JHOtoWDvZ", "reviewer": "Reviewer_hf9X", "summary": "### Update after rebuttal\nAfter reading the feedback carefully, I updated my score for the paper.\n\n### Original review\n\nIn this work, the authors study SGD and its adaptive variants in the setting of noisy generalized linear models. Assuming the covariance matrix of the data is bounded by a dimension-independent constant and a few more similar assumptions, the authors establish that the expected risk of SGD and the expectation of adaptive learning rates both converge to deterministic curves described by an ODE.\n\nI found the paper rather hard to follow and I couldn't understand the significance of the obtained results. The theory requires a lot of simplifications: the prediction model is linear; the learning rates that are studied are \"idealized\", so they are not the same as those used in practice; Adagrad-norm isn't really a method popular in practice; the ODE solution is not trivial in the case of least squares, as equation (11) is implicit, so it requires further assumptions to make a statement about the solution. I did not understand what insight we can draw from all of this.", "questions": "1. What's the intuition for why $h$ exists and why would it be well-behaved? I can see a discussion in Appendix B, but it doesn't provide any high-level intuition. \n2. Why do you say in Appendix B that $R(X)$ is \"an expectation of a Gaussian vector\"? \n3. The authors wrote \"We shall remove this condition by approximation in what follows\", where is it done?\n4. The authors say they need \"mild assumptions on the learning rate\", but why are they mild?", "limitations": "The authors could do a better job when discussing the limitations of their work. There is no explicit limitations section and in the ckecklist, the authors state that they already did everything needed by clearly outlining the required assumptions. However, I think it'd benefit the paper if the authors made an explicit statement on how some aspects of the theory are too restrictive.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The theory reveals that line search leads to slower convergence that Polyak stepsize.\n2. In some simplified cases, there exist exact expressions that describe the dynamic of the studied methods.\n3. The theory is also supported by numerical simulations.", "weaknesses": "1. The theory for line search and Polyak stepsize is only for the idealized version.\n2. The insight that line search methods do not work particularly well is not exactly new, see for instance J Bolte, E Pauwels \"Curiosities and counterexamples in smooth convex optimization\"\n3. The complexity of the theory and the discussion is such that it is very difficult to draw any insight, and it seems the theory wouldn't be extended by others because of how convoluted it is.\n\n### Minor issues\nAbbreviation \"SGD\" is used before (line 23) it is introduced (line 30)  \n\"We fully expect\"  -> \"We expect\"  \n\"worst case convergence guarantees\" -> \"worst-case convergence guarantees\"  \n\"strongly-convex\" -> \"strongly convex\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer's comments required more space than 6000 characters to answer. *We will include an additional \"Official Comment\" with the answers to the reviewer's questions.*\n\n**Significance of results \\- Main contributions**\n\nThis work introduces a framework that goes beyond traditional analysis of stochastic algorithms to utilize the high-dimensionality in the problem. It allows for a finer analysis of stochastic learning rate algorithms (e.g., AdaGrad, line search, Adam, RMSprop, etc). \n\nWe also emphasize that one of the main contributions of this work is an explicit deterministic expression for the dynamics of stochastic adaptive learning rate algorithms (Thm 2.1). That is, we can predict the behavior of the loss value and learning rate (both stochastic) at any iteration without ever running the stochastic algorithm (see Fig 1). We then analyze these dynamics to gain insights into the evolution of the stochastic learning rate at any iteration. These dynamics exactly agree when the problem is “high-dimensional” (number of parameters is large) and often numerically reproduce the dynamics on real data (see Fig. 6 on CIFAR-5m dataset). \n\n**Line search and Polyak only “idealized version”**\n\nIt is true that the algorithms presented are idealized versions. However, we emphasize that a theoretical understanding of the behaviors of these idealized (stochastic) algorithms is not known \\[Note: the reference given is for a line search on a deterministic objective and less is known about the behavior of stochastic line search\\]. Adding more practical versions, which are often more complex, only adds complications to the already unknown behaviors. As such, a natural starting point for developing a theory for the exact dynamics of stochastic learning rates is to understand the behavior of these idealized versions. \n\n**Line search methods do not work well is not new**\n\nWe first politely disagree with the reviewer that line searches do not work in practice. For deterministic optimization problems, line search methods are widely used and are the default in many well known algorithms (e.g., see scipy documentation for BFGS (Armijo condition)). While we appreciate the reference that showed an example where line searches fail on a deterministic optimization problem, line searches do provably work for a large class of deterministic optimization, including the deterministic versions of the problems analyzed in this paper. The theory for deterministic line searches is well developed, but for stochastic optimization problems the theory is still in its infancy. \n\nWe do agree with the reviewer that on stochastic optimization problems, line search methods are not used in practice. In fact, one of the goals of this paper is to explain this\\! We show why they perform badly due to anisotropic data; in contrast, a standard minimax-optimal analysis says they are as good as SGD (see e.g. \\[50\\]).  Moreover, we precisely quantify how different the convergence rates of a stochastic line search method will be from a tuned fixed learning rate SGD algorithm. \n\nThe theoretical framework established in this paper (Theorem 2.1) provides the tools (albeit more complex than the “textbook analysis”) to do a finer analysis of stochastic line search methods. \n\n**Complexity of the theory is difficult**\n\nThe standard tools in textbooks for theoretically analyzing stochastic algorithms often give too crude estimates on their behavior in practice. As an example, the standard analysis would say that you need to use a decreasing learning rate with SGD, but nobody does this in practice. The point of this paper is to provide another tool that allows for a finer analysis of these algorithms – traditional minimax optimality *isn’t capable of distinguishing these algorithms* – so something more complicated *is necessary*.\n\n**Complexity of theory makes it hard to derive insights. Theory will not be extended by others.** \n\nWe politely disagree with the reviewer. The related work paragraph (Page 4 Lines 106-110) describes past works which use a high-dimensional framework for analyzing dynamics of stochastic algorithms. These past works usually assume isotropic Gaussian data ($N(0,I)$) and do not consider stochastic learning rates for their algorithm, but are part of a growing body of literature based on this style of analysis. \n\nWe know that we are doing something quite different than the vast majority of optimization researchers, but we think there is a need for better ways to distinguish performance of stochastic algorithms.  For example, the more traditional analysis of line search in Vaswani et al.  \\[50\\] suggests SGD \\+ line-search has no performance cost (their Theorems 1 and 4). We quantify the performance cost, as one incurs an extra factor of condition-number. This is a statement 99% of optimization-researchers should be able to understand, even if the methods are not to their taste, and one *needs* a more complicated setup to see it\\!\n\n\\[50\\] Vaswani, Mishkin, Laradji, Schmidt, Gidel, Lacoste-Julien. Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates. 2019\\.\n\n**Limitation paragraph.** Please see the response to \"All Reviewers\" for the exact paragraph we will add to the main document.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhJWWJJHFG", "reviewer": "Reviewer_A5Ho", "summary": "Analyzes the behavior of AdaGrad-Norm, Line-search and Polyak step size using ODEs.", "questions": "See above.", "limitations": "See above", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "This paper provides an interesting characterization of the behavior of several optimization methods using ODE tools, which are not as heavily used in optimization theory as they should be. This work seems technically sound, although I didn't review the (long) Appendix. \n\nI am in favor of acceptance, although with reservations. The methods studied are interesting to theoreticians, but they are rarely if ever used in practice, which limits the practical appeal of this work. Especially the idealized special cases for the line search and Polyak step size. The focus on quadratic problems is also a major limitation, especially since the analyzed methods are not typically used on quadratic problems.\n\nThere is definitely potential for followup work to build on this work.", "weaknesses": "- Paper spends a lot of time setting up a general framework, which it then applies to specific settings. The amount of notation introduced seems over the top for a conference paper. For instance, the results for least-squares can be presented without establishing the full general framework, which could be left to the Appendix. Section 1.2 and 2 are just .... a lot to get through.\n- Paper ends very abruptly.\n- Some plot fonts are too small to read when printed.\n- Further discussion of the implications and potential applications would be good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support of our paper. We answer some of the reviewer’s reservations below. \n\n**“The methods studied are interesting to theoreticians, but they are rarely if ever used in practice, which limits the practical appeal of this work. Especially the idealized special cases for the line search and Polyak step size. “**\n\nWe agree with the reviewer that the algorithms studied are idealized and/or not widely used in practice. However, traditional analysis of adaptive stochastic learning rate algorithms (e.g., idealized line search/Polyak, etc.) all give the same rates as the ones used in practice. The traditional theory does not distinguish between “good” practical algorithms and “bad” practical algorithms. This suggests we need a new type of analysis. Part of our goal was to provide such a new framework. While we only analyze “idealized algorithms”, we now understand problems for which these “idealized algorithms” do not perform well. This hopefully illuminates and begins closing the gap between practice and theory. \n\nIn future work, we want to derive the dynamics of more practical algorithms. We strongly believe this framework might explain theoretically why certain often used algorithms in practice perform better. \n\n**Focus on quadratic problems is a major limitation, especially since the analyzed methods are not typically used on quadratic problems.**\n\nWe agree with the reviewer that “quadratic problems” are idealized, and we want in future work to extend the analysis of these algorithms to more complex losses. On the other hand, many of the algorithms were designed by approximating the “complex” loss with a quadratic. Moreover, it is surprising that we do not even know how these algorithms perform on these quadratics. In our opinion, we should understand this simple loss in detail first. \n\nThis said, our analysis extends beyond quadratics to include generalized linear models such as logistic regression and more. Moreover, we do provide some analysis for other losses beyond quadratics. For strongly convex problems (not necessarily quadratic) we find a bound on the limiting learning rate, see Section D1. Also, Figure 1 shows our predictions matching the iterates of SGD with AdaGrad Norm for binary logistic regression.   \n\n**Response to Weaknesses**\n\n* **Too much set-up in the main paper.** Thank you for your suggestions. In the updated version of the paper, we plan to lighten the notation and move the least square results upfront; see our reply to all reviewers.   \n* **Plot fonts too small.** Thank you for pointing that out, in the next version of the paper we will update the figures and make the font larger. See the attached PDF with the revised figures.    \n* **Paper ends abruptly.** Added a “Conclusion” paragraph. See comment to “All Reviewers” for the specific paragraph.   \n* **Discussion of implications and potential applications.** Added a “Limitations” paragraph. The conclusion paragraph contains potential applications. See comment to “All Reviewers” for the specific paragraph.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7hpama64N", "reviewer": "Reviewer_Qoif", "summary": "The paper is a continuation of a broad line of work exploring deterministic limits of stochastic gradient descent. In particular, the author(s) follow one specific branch of the many, which derives a solution to a coupled integro-differential equation by passing on the complex space. The novelty lies mainly in the fact that they derive, for least squares, deterministic limits of dynamics that have adaptive step-sizes, for non identity covariance of the observed dataset. Industrially, this is important as the idealized scenario of fixed step-sizes is __very much__ idealized, and realistic datasets have structure. Thanks to the general expressions (under 6 assumptions that are claimed to be not super-restrictive), they can proceed to study the various differences between standard schedules. The limiting learning rate can be derived, as well as the scaling of the risk along time. This is done for Line Search, Polyak step-size, AdaGrad Norm. Interestingly, their bounds suggest that for covariances with largely separated eigenvalues (say, power law spectrum), there should be a change in the phenomenology (i.e. a phase transition). They verify this for a power law covariance, which exhibits three phases depending on the parameters of the problem (notably the starting distance from $X^{\\star}$). All is accompanied by experiments corroborating the predictions.", "questions": "Please note that some of the questions below are very ingenuous. \n\n- (lines 49-57) This is more of an impression. Why is $\\lambda_{\\min}(K) > C > 0$ for you a situation of strong anisotropy? The power-law case is anisotropic in that the minimum eigenvalue is not bounded below by a constant as $d\\to\\infty$. Put another way, I do not understand how the __minumum__ eigenvalue being bounded below from zero would describe the whole behavior of the covariance profile of the data-points $a$. I understand that this might be related to the fact that the optimal strategy uses the trace so you say something in the line of \"if the minimum eigenvalue is very well distant from the average then there is anisotropy\". At the very least, it appeared to me as a reader that you somehow __define__ strong anisotropy as this eigenvalue condition, while in fact it is a consequence of the algorithmic behavior that you can tune the eigenvalues in this \"anisotropic\" way and observe that \"greed can be arbitrarily bad\". \n- The original introduction of the Polyak step-size is that of a step-size that uses the optimal value of the function, and the value of the current iterate. In your case, what you call Polyak step-size has a connection with the original method, which is related to your assumptions, as you find that eventually it gets to a $\\gamma_{\\infty}$ that takes care of the geometry of the function. From a pedagogical point of view, it would be nice to comment further on this. \n- Theorem 2.1 states that the concentration is for any $\\epsilon \\in \\left(0,\\frac{1}{2}\\right)$, line 48 says \"for any inverse power of d\", and Theorem B.1, Corollary B.1  cited below Theorem 3.1 in line 195 give a result for existance of an $\\epsilon$ such that the approximate solution of the integro-differential equation is verified. Can you shed light on these differences? Since it is your main Theorem, it would be nice to have an explicit discussion of what goes on where. More in detail, why do you get from existance in the appendix to a whole interval of $\\epsilon$ (that by the way gives a rather slow decay wrt to say $\\epsilon > 1$), to claim in text \"for any inverse power\"? I have to be honest here and I did not spend time re-doing all the proofs. \n- Related to the point above, in equation (27), your supremum is over $T\\wedge \\hat{\\tau}_M$. In line 531 you state that you will not specify which stopping time it is when it is \"clear from context\", but for (27), given that you want to prove (26), my natural question is if stability is for $\\hat{\\tau}_M(S)$ or $\\hat{\\tau}_M(\\mathscr{S})$, as you only mention one. At the very least, it is not clear from context here which stopping time it is in (27). \n- Lemma C.1 requires as condition that $S(t, z)$ is an approximate solution over the $\\xi$-mesh. How do you guarantee that this condition holds in your case? If ever, it is not evident from text that this is the case. If it is a direct consequence of arguments in reference 14, I would appreciate it being made explicit. \n- Can you clarify the sentence of lines 239-240? Ignore the \"sufficiently anisotropic\" aspect which I already discussed above. The idea that smaller learning rates lead to under-performance is not entirely true. In the simplest setting if you over-shoot with your learning rate you will never converge. I believe here this is tightly linked to your Table 1, but there line search and Polyak have the same convergence rate for the risk. Where is the difference? \n- In Figure 1, how many runs do you make? It would be nice to know.\n- The same question above but for any other plot where you show error bars. \n- (line 1125) You choose as scaling for the initialization $X_0\\sim\\mathcal{N}\\left(0,\\frac{I_d}{\\sqrt{d}}\\right)$  and the ground truth $X^{\\star} = \\frac{1}{\\sqrt{d}}\\mathbf{1}_d$. The norm of the latter is $1$, but the former has norm concentrating at $\\sqrt{\\sqrt{d}}$, and this violates your assumptions (not bounded by a constant independent of $d$). Is it a typo? Or is it that the experiment does not match the assumptions? If I look at the scaling of Figure 3, line 1099, you indeed sample $X_0\\sim \\mathcal{N}\\left(0,\\frac{I_d}{d}\\right)$. \n- Proposition 4.4 is nice. What happens when $\\beta +\\delta > 2$?\n\nMy soundness score is due to the many questions. I hope I will raise it after the discussion. The overall score follows the same principle.", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The motivation in lines 17-31 is very well stated. As a reader, I would be inclined to continue parsing the paper. \n- Proposition 4.3 states that with a null initialization $X_0 = 0$ and subject to a condition on the covariance (which roughly says that $\\lambda_{\\min}$ can be small but that there is a very small number of vanishing eigenvalues), if the ground truth signal is spread out, then the learning rate is never zero. For symmetric activations (say, phase retrieval), and constant learning rate, I think you cannot really start at zero because you have a symmetry, you need at least a tiny bit of overlap. From this result, it appears that you can start from the most useless guess and still get away with it, provably. Interesting. \n- The higher arching aim of the work is very important, as knowing that the behavior of SGD is predictable is pivotal for understanding how our models work. \n- The techniques are of independent interest and are a continuation of a line of work that is systematically, step by step, developing a global answer to these questions. \n- The result that Polyak's step-size converges to the inverse of the average eigenvalue is yet another validation of a method originated from convex optimization holding in more generality. This is reassuring.", "weaknesses": "Weaknesses are questions and questions are weakenesses in some sense. Once the author(s) engage, I will reorder things that get answered and things that remain as \"concerns\".  \n\n- Proposition 4.4 supposes that $\\mathscr{D}_i^2(0)$ could potentially have a different scaling at each site $i$. I understand that the assumption includes an uninformative initialization $\\delta = 0$. However, when is it really that one can think of a setting in which you can have an initialization that scales like $\\lambda_i^{-\\delta}$ for non trivial $\\delta$? The result is very aesthetic, but I did not think through of its realistic value. Anyways, it is very nice so I am __not__ criticizing the math. Maybe I am just interested in understanding if you have in mind something. \n- In the Checklist, you state that you have been \"__very careful__ \\[...\\] to include __all__ the assumptions needed\". I agree with this statement. However, as a reader, I do not agree with this being the limitation of your work, as in some sense you claim over the whole text that your assumptions cover standard adaptive step-sizes (hence you hiddenly claim that this theory is all we need). I would be very happy to see a paragraph where you clearly state your limitations. Where does your theory fail? Where does the technique __not__ extend naturally? If you want to sell me your paper, I do not believe its value is in the assumptions. On a first aspect, the ideas and the techniques are of independent interest. Secondly, it would be sad if the bottleneck were the assumptions. \n- Your equation (7) is found across some other works. One of the earliest appearances is (Yoshida and Okada, 2019). However, from a first check, I would guess that you are missing a term on the first object. To be fair, I quickly tried to re-derive it and I would guess that you need a further $H_{2, t}\\mathscr{V}_{12, i}(t)$ in the first expression. Am I wrong? If so, please correct me. It looks like the equations are missing some symmetry of the dynamics. \n- I am concerned about the scaling. My understanding of the literature is that scalings are very important. Your assumptions require that $\\lVert K\\rVert$ is bounded independently of the dimension and that $\\lVert X_0\\rVert$ is bounded, independently of the dimension. Moreover, your step-size is $\\frac{1}{d}\\mathfrak{g}_k$. Therefore, at the beginning you move by very little, I think by $\\propto \\frac{1}{d}$. What is done in some other references is very different. Let me take an example, and correct me if I am wrong.\n1. You can derive heuristically deterministic dynamics for step-size $\\mathfrak{g}_k$ (not normalized), starting from a standard Gaussian standard gaussian vector $X_0$, with data-points being i.i.d. Gaussians $a\\sim \\mathcal{N}\\left(0,\\frac{1}{d}I_d\\right)$. Now, the step-size is not divided by dimension, the $X_0\\sim \\mathcal{N}(0, I_d)$ is not bounded in norm by something independent of $d$, and the $a$ matches your assumptions. In this case however, the signal is far stronger, as we are removing two normalizations you indeed have. Below my questions regarding this. \n2. Can you clarify why it makes sense to choose your scaling? Apart from it allowing the proofs. \n3. In particular, how is it not possible to allow for norms dependent on $d$, and in parallel, how is it that the signal is enough to be not stuck at the initialization? \n4. Does this implicitly mean that you are exploring a slow regime? Say, a Gradient Flow? What happens if in your experiments you just change the scalings to something larger in magnitude?\n\n\n#### Typos\n__NOTE__: I am including for completeness typos in \"weaknesses\". Please do not count them as such. \n- (line 560) In the reminder of $\\Omega$, you are missing a $\\max$. \n- (equation 20) there is no $\\Gamma(t)$ in the expression, right? \n- (lines 1140-1141) the way of quoting \"noise\" and \"variance\" is not the correct one, TeX-wise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review and careful reading of our paper, including the appendix. We address below all of the reviewer’s comments. *Due to space limitations, we will include an additional \"Official Comment\" to answer all the reviewer's questions.* \n\n**Practical application of power law scaling on $\\\\mathscr{D}\\_i^2(0)$.** First, it is important to note that this power law scaling is not on the initialization but rather on the distance from the initialization to the ground truth (signal) $X^\\\\star$.  If one initializes at 0, which is reasonable to do in the least squares setting, then this is a power law scaling on the ground truth.  In settings where the data has power law covariance, one can also see power law scaling in the ground truth. For more details on this, see, for example, reference \\[56\\] mentioned in the paper, particularly their discussion of scaling laws in section 6\\.\n\n**Limitations of the paper.** Added a “Limitations” paragraph. For the version going in the main paper, see “All Reviewers Rebuttal”. \n\n**Coupled ODEs in Eq (7).** Thank you for pointing out the potential issue with Equation (7). You are correct that there was an oversight in the first term of Eq(7).  We have revised the equation to include the missing term and corrected ODEs. Page 17 contains the correct ODEs for S and reflects the updated formulation. We appreciate your attention to detail and the opportunity to correct this error.\n\n**Scalings.** Thank you for the questions.  Please see the discussion for all reviewers, where we discuss other scalings beside $(1/d)$.  \n   \nYou also asked about whether a learning rate on the scale of $1/d$ is large enough for the algorithm to really move away from initialization.  The key here is that we work in the regime where the number of iterations is proportional to dimension.  Thus, even with learning rate of size $1/d$, one sees significant movement after order $d$ iterations.  Indeed, for examples such as linear or logistic regression, this learning rate yields convergence of the risk in order $d$ iterations.  \n\nYou are correct that, in certain cases (e.g. phase retrieval with a cold start), the algorithm can become stuck at initialization and require more than order $d$ iterations to converge.  This is also captured in our result.  The theorem is not a statement about the convergence of the algorithm to an optimal solution, but rather about the convergence of the SGD dynamics to a deterministic curve.  Thus, our theorem can accurately predict the risk trajectory for order $d$ iterations, even when the risk does not converge to zero in that time.  We’ll add that to capture the escape of phase retrieval from the high-dimensional saddle $O(d \\log d)$ iterations are needed, and moreover the high-dimensional limit of the risk curve will contain non-concentrating and non-vanishing effects of SGD noise (this is implicit in Tan and Vershynin. *Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrieval. 2019*).\n\n   \nFinally, you asked about our choice to bound the norm of $X$, independent of $d$.  Because we are interested in attaining a high-dimensional limit of the dynamics, we need to scale things in such a way that the risk is dimension-independent.  In our set-up the risk is always expressible as a (generally non-linear) function of the inner product $X^Ta$, so we need this inner product to be dimension-independent.  Our choice of scaling achieves this (although one could also, for example, simultaneously adjust the scaling of $X$ and $a$ to achieve an equivalent result).  There are, as you suggest, interesting problems in which the norm of $X$ may grow with $d$, but those are beyond the scope of this paper.\n\n**Typos.** Thank you for pointing out these things.  *We corrected them.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the referees for their time and constructive comments that significantly helped us improve our paper. The reviewers questions were deep and so *we will add additional information as an "Official Comment."*

**Paper structural changes.** 

In our next version of the paper, we will implement the following reorganization changes to improve the clarity and flow:

* Condense the assumptions, as much as possible, in the introduction. However, we do value stating upfront the formal assumptions in the main paper, as we do not want to hide anything from the reader. This will include, for instance, moving the discussion of the Assumptions, to the Appendix. We will also move the ODEs for the least squares problem earlier in the paper as recommended by Reviewer A5Ho.    
* Add Conclusion paragraph *(see below in "Official Comment").*  
* Add Limitation paragraph *(see below in "Official Comment")*.  
* Increase the font size in the figures *(see attached PDF)* and added references to the figures in the inline text. 

**New developments from previous work \[14\].** 

This work’s Theorem 2.1 is inspired by Theorem 1.1 from \[14\]. We note that Theorem 1.1 in \[14\] does not apply to adaptive (stochastic) learning rates. Theorem 1.1 only holds for learning rates schedules which are *deterministic* such as decaying learning rates $1/t$. The new results (Thm. 2.1) allow for *stochastic adaptive learning rates.* This is important since many of the widely used algorithms in practice would not apply to results in \[14\], but do apply to our new result Theorem 2.1. This allows us to understand why these algorithms are used more in practice over say deterministic decaying step sizes. 

Concretely, Theorem 2.1 (with stochastic adaptive learning rates) requires substantial work and a new proof that shows the stochastic learning rates and the loss are simultaneously concentrating around a deterministic function. 

We will add a discussion about the differences between Theorem 1.1 in \[14\] and the new result Theorem 2.1 to the paper. 

\[14\] Elizabeth Collins-Woodfin, Courtney Paquette, Elliot Paquette, and Inbar Seroussi. Hitting the high-dimensional notes: An ODE for SGD learning dynamics on GLMs and multi-index models. arXiv preprint arXiv:2308.08977, 2023

**Other scalings and lower rank K.** 

We have adopted a scaling convention where stepsize scales like $1/d$.  As multiple reviewers noted, this may not be the only scaling in which one sees a nontrivial limit.  An important non-asymptotic measurement of $K$ is  $\\text{Tr}(K) / ||K||\_{op}$, sometimes called intrinsic dimension $\\text{Dim}(K)$.  We believe that a version of our main Theorem 2.1 generalizes to a non-asymptotic setting where the intrinsic dimension (as opposed to the ambient dimension $d$) is large, and this affects the step-size scalings that should be used in SGD.  

1. Our theorem is really about the case that $\\text{Dim}(K)=\\Theta(d)$.  This is the case, e.g. when one has a dimension-independent condition number (a big part of our paper).  Here the $(1/d)$ scaling is where one sees non-trivial contribution to the dynamics from both the gradient flow term and the stochastic term (SGD noise).  If one were to choose a smaller scaling (e.g. $1/d^2$), the dynamics would devolve to gradient flow, whereas if one were to choose a more aggressive learning rate, the dynamics would devolve to pure stochastic noise (and the risk would generally tend to infinity).  With the $1/d$ scaling one sees the effect of both gradient flow and stochastic noise in the dynamics.  Put another way, we have chosen the largest learning rate scaling for which one has a nontrivial limit.    
2. When the $\\text{Dim}(K) \\to \\infty$ but $\\text{Dim}(K) \= o(d)$, the scaling of $\\gamma$ should not be $1/d$.  It should rather be that the scaling is $1/\\text{Dim}(K)$ – for quadratic losses, one can check the same story as above holds here: larger learning rates lead to pure-noise and smaller learning rates lead to gradient flow approximations.  On the other hand, the ODEs that we derived *are still the same ODEs*; one just needs to rescale time and learning rate in the ODEs to produce the correct equations.  The method in our paper would definitely extend to *some* intermediate growth rates of $\\text{Dim}(K)$, but in our mind, the better theorem would be a fully non-asymptotic theorem – one which shows that when $\\text{Dim}(K)$ is large, the ODEs and the SGD curves are close as a function of $\\text{Dim}(K)$.  That’s a much bigger task, and it could clearly be a standalone future project.  
3. When $\\text{Dim}(K)$ is bounded above (say in the case of classical source/capacity restrictions), the ODE approximation has to change substantially.  The losses don’t concentrate, and the mean behavior of the ODEs follow a discrete difference equation.  Now one can use tricks, like embedding the markov chain in continuous time so the mean risk follows an ODE.  But even doing this, the ODEs have additional terms (which simplify in high dimensions\!), and so they are strictly more complicated to analyze.  We expect that in some cases this is not hopeless (such as source/capacity type conditions), and the extra terms will not play a substantial role.

Both regimes 2 and 3 are interesting, and there is plenty of room for future work in both those directions (both with adaptive and non-adaptive stepsizes).  We did not directly pursue them in this paper, because we needed to be able to show that it was possible to derive meaningful conclusions about the limit system of ODEs in the high-dimensional case.  So we pursued the more modest goal of proving the ODE comparison in the maximal-dimension case.

Meta-review: This is an excellent paper that makes a number of strong theoretical and conceptual contributions to our understanding of the dynamics of adaptive stochastic optimization algorithms in high dimensions. Moreover, the mathematical techniques developed are likely to be of use to others. The authors have addressed the major concerns of the reviewers. 

Given the depth of the paper and the additional areas for further exposition and exploration, I encourage the authors to consider the suggestion of one reviewer to write an extended, journal-length version of the paper that could do the material more justice.
Decision: Accept (poster)