Corner Gradient Descent
Authors: Dmitry Yarotsky
Venue: ICLR 2026 Poster (2026)
Keywords: mini-batch stochastic gradient descent
momentum
sampling noise
convergence rates
acceleration
power laws
phase diagram
contour integration
rational approximations
asymptotic methods
MNIST
frequency response function
Primary area: optimization
TL;DR: Generalized (S)GD algorithms = contours in $\mathbb C$. Contours having a corner with external angle $\theta\pi, 1<\theta<2,$ accelerate loss convergence rates $t^{-\xi}$ to $t^{-\theta\xi}$.

Abstract: We consider SGD-type optimization on infinite-dimensional quadratic problems with power law spectral conditions. It is well-known that on such problems deterministic GD has loss convergence rates $L_t=O(t^{-\zeta})$, which can be improved to $L_t=O(t^{-2\zeta})$ by using Heavy Ball with a non-stationary Jacobi-based schedule (and the latter rate is optimal among fixed schedules). However, in the mini-batch Stochastic GD setting, the sampling noise causes the Jacobi HB to diverge; accordingly no $O(t^{-2\zeta})$ algorithm is known. In this paper we show that rates up to $O(t^{-2\zeta})$ can be achieved by a generalized stationary SGD with infinite memory. We start by identifying  generalized (S)GD algorithms with contours in the complex plane. We then show that contours that have a corner with external angle $\theta\pi$ accelerate the plain GD rate $O(t^{-\zeta})$ to $O(t^{-\theta\zeta})$. For deterministic GD, increasing $\theta$ allows to achieve rates arbitrarily close to $O(t^{-2\zeta})$. However, in Stochastic GD, increasing $\theta$ also amplifies the sampling noise, so in general $\theta$ needs to be optimized by balancing the acceleration and noise effects. We prove that the optimal rate is given by $\theta_{\max}=\min(2,\nu,\tfrac{2}{\zeta+1/\nu})$, where $\nu,\zeta$ are the exponents appearing in the capacity and source spectral conditions. Furthermore, using fast rational approximations of the power functions, we show that ideal corner algorithms can be efficiently approximated by practical finite-memory algorithms.

Published as a conference paper at ICLR 2026
CORNER GRADIENT DESCENT
Dmitry Yarotsky Applied AI Institute & Steklov Institute of Mathematics yarotskye gmail. com
ABSTRACT
We consider SGD-type optimization of dimensodd quadratic problems with power law spectral conditions. It is well-known that such problems deter- ministic GD has loss convergence rates L, = O(t which can be improved to L, Ot by using Heavy Ball with non-stationary Jacobi-based schedule (and the latter rate is optimal among fixed schedules). However, in the mini-batch Stochastic GD setting, the sampling noise causes the Jacobi HB to diverge; ac- cordingly no (() algorithm is known. In this paper we show that rates up to 0(-22) can be achieved by generalized stationary SGD with infinite memory. We start by identifying generalized (S)GD algorithms with contours in the com- plex plane. We then show that contours that have a corner with external angle fr accelerate the plain GD rate (t- to O(t-(() For deterministic GD, increasing 0 allows to achieve rates arbitrarily close to O(t-2S.. However, in Stochastic GD, increasing 0 also amplifies the sampling noise, so in general 0 needs to be opti- mized by balancing the acceleration and noise effects. We prove that the optimal rate is given by 6max min(2, V, c+t,e), where V. are the exponents appearing in the capacity and source spectral conditions. Furthermore, using fast rational approximations of the power functions, we show that ideal corner algorithms can be efficiently approximated by practical finite-mon finite-memory memory algorithms.
INTRODUCTION
well-known that Gradient Descent (GD) on quadratic problems can be accelerated using the additional momentum term (the "Heavy Ball" algorithm, Setorithm,PPoyyakk (1964). For ill-conditioned prob- lem, Heavy Ball with a suitable non-stationary ("Jacobi")) predefined schedule allows to accelerate power-law loss converge rate O(t-s) to Ot,- i.e. double the exponent Ç (Nemirovskiy & Brakhage 1987). This acceleration is the best possible for non-adaptive schedules.
Polyak 1984
On the other hand, for mini-batch Stochastic Gradient Descent (SGD) typically used in modern machine learning, the convergence rate picture is much more complicated, and much less is known about possible acceleration. The natural quadratic problem in this case is the fitting of linear model with a sampled quadratic loss. In the power-law spectral setting. it was found in (Berthier al. 2020) that plain SGD has two distinct convergent phases: either the sampling noise weak and the SGD rate is the same Ott=$) as for GD, or the convergence is slower due to the prevalence of the sampling noise. We refer to these two scenarios as signal- and noise-dominated, respectively.
This picture was refined in several other works (Paquette et al. 2024 Varre et al. 2021 Varre & Flammarion, 2022] Velikanov et al.) 2023 Yarotsky & Velikanov 2024). particular, Yarotsky & Velikanov (2024) examined generalized SGDs with finite linear memory of any size (generalizing momentum and similar terms) and proved that with stationary schedules they all have the same phase diagram plain SGD (Figure 2 left); in particular, they do not accelerate the plain GD/SGD
rate O(t (t-
On the other hand, the non-stationary Jacobi Heavy Ball accelerating deterministic GD from O(t-9) to (t (ee fails for mini-batch Stochastic GD: it eventually starts to diverge due to the accumulating sampling noise. |aarre& & Flammarion (2022) have proposed a non-stationary modification of SGD that achieves a quadratic acceleration, but only on Unite-dimensiona problems. |Yarotsky Ve- likanov 2024 have proposed non-stationary modification of the Heavy Ball/momentum algorithm

---

Published as a conference paper at ICLR 2026
that is heuristically expected (but not yet proved) achieve rates (() with some 1 < 0 < on chinite-dimensiona problems.
To sum up. the topic of SGD acceleration in ill conciiiitt quadratic problems is far from settled.
In the present paper we propose an entirely new approach to acceleration of (S)GD that both provides general geometrie viewpoint and proves that, in certain rigorous sense, SGD in the signal- dominated regime can be accelerated from O(t 3) to O(t=(() with 0 up to 2.
Our contributions:
1. A view of generalized (S)GD contours (Section We show that stationary (S)GI algorithms with an arbitrary-sized linear memory can be identified with contours in the complex plane. This identification leverages the characteristic polynomials X and the loss expansions of memory-M (S)GD from Yarotsky & Velikanov (2024). We show that all the information needed to compute the loss evolution contained in response map V { C | 2 } - C associated with X. The map V/ gives rise to the contour (1{ E C 1 = 1}) and, conversely, be reconstructed, along with the algorithm, from given
contour.
2. Corner algorithms (Section 4 A crucial role is played by contours that have corner with external angle 0x. < < We prove that the respective algorithms accelerate the plain GD rate S(t-S) to O(t However, in Stochastic GD such algorithms have the negative effect of amplifying the sampling noise. By balancing these two effects, we establish the precise phase diagram of feasible accelerations of SGD under power-law spec- tral assumptions (Figure I right). In particular, we identify three natural sub-phases in the signal-dominated phase;
of them acceleration up Ot is theoretically feasible.
3. Implementation of Corner (S)GD (Section 5). Ideal corner algorithms require an infinite memory, but can be fast approximated by finite-memrry algorithms using fast rational ap- proximations of the power function 2.. Experiments with synthetic problem and MNIST confirm the practical acceleration.
BACKGROUND
This section is largely based on the paper Yarotsky & Velikanov (2024) to which we refer for details.
Gradient descent with memory. Suppose that we wish minimize a loss function L(w) on linear space H. We consider gradient descent with size-M memory that can be written as
Wi+1 (Woi1
(et1-")) U1+1
(- D" (VL(W.)) t= 0,
The vector w; is the current step-t approximation to an optimal vector w.. and u, is an auxiliary vector representing the "memory" of the optimizer. These auxiliary vectors have the form u (u(2) ((())...((AA))) " with (m)) E H and can be viewed as size-M columns with each component belonging H. We refer to M the memory size. The parameter a (learning rate) is scalar, the parameters b, c are M-dimensional column vectors, and D is a M x M scalar matrix. The algorithm can be viewed as sequence of transformations of size-(M + 1) column vectors (W: with H-valued components. Throughout the paper, we only consider stationary algorithms, in the sense that the parameters a, b, c, D do not depend on The simplest nontrivial special case of GD with memory is Heavy Ball (Polyak] 1964), in which M = 1 and is the momentum.
Our theoretical results will rely on the assumption that quadratic:
L(w) = Hw wT q.
with a dimensional Hilbert spaces H. and slightly abuse notation by interpreting w" as the co-vector (linear functional (w.))) associated with vector w. We will assume that H has a discrete spectrum with ordered strictly positive eigenvalues d
strictly positive definite H. Throughout the paper, we will mostly be interested in infinite-
2

---

Published as a conference paper at ICLR 2026
Let be the optimal value of L such that (L(w.) = Hw, -q -q= 0, and denote Aw, We W.-
Then, if Aw; and u, are eigenvectors of H with eigenvalue X, then
Awe+1 u1+1
Aw: u, : Sx -
b² D
+ A (-t) (1,0*),
= Sx
(3)
and the new vectors AW++1 Up+1 are again eigenvectors of H with eigenvalue X. As result, performing the spectral decomposition of Aw; u, reduces the original dynamics ① acting in H & RM++ to X-indexed collection of independent dynamics each acting in R R+++
For quadratic L, evolution
admits an equivalent representation
M
M
W =
Σ m=0
Pm Wt
+
Σ m=0
4m
VL(W++m), 1= 0, 1,1,....
(4)
with constants (pm ) (9m)m such that Em=o Pm = 1. These constants are found from the characteristic polynomial
M
x(e, A) = de 5x) = P()) - AQQ)). P(K) = 1 M+1
Σ m==
Pmj Q(1) = Σ m==
Ampte. (5)
Batch SGD with memory. In batch Stochastic Gradient Descent (SGD), it is assumed that the loss has the form I(w) =x w), where is some probability distribution of data points and f(x, w) is the loss at the point x. In the algorithm ( we replace V L by V B., where B, is random batch of [ points sampled from distribution p. and VLg the empirical approximation to L, i.e LB(w) [m] ΣxEB l(x, w). The samples B, at different steps are independent.
We assume to have the quadratic form É(x, w) = w (xx"w-y(x)) for some scalar target function (xx.. Here, the inner product x" w can be viewed as a linear model acting on the feature vector x. By projecting the subspace of linear functions,
assume w.l.o.g. that the target function y(x) is itself linear in x, i.e. [(x) = x" w. with some optimal parameter vector w.. (Later we will slightly weaken this assumption to also cover unfeasible solutions w..) Then the full loss is quadratic as in Eq. (2: L(w) = Ex-p + (x" Aw)2 = A HAw, where Aw = w w, and the Hessian H Ex= xx?
Mean loss evolution, SE approximation, and the propagator expansion. Since the trajectory W; in SGD random, it convenient study the deterministic trajectory of batch-averaged losses L, = BB......._. L(w.). The sequence L, can be described exactly in terms of the second moments of Wt. u, that admit exact evolution equations. An important aspect of this evolution is that involves 4'th order moments of the data distribution and so cannot in general be solved using only the second-order information available the Hessian Fx-ppxxx]
A convenient approach to handle this difficulty is the ppectrally-Expreesibbe (SE) approximation proposed in Velikanov et al. (2023). It consists in assuming that there exist constants Ti, T2 such that for all positive definite operators C in H
Ex-p "xx" HCHH (T2 H)H
In fact, this approximation holds exactly for some natural types of distribution p (translation- invariant, gaussian). Otherwise, if the r.h.s. is only an upper lower bound for the l.h.s., this implies a respective relation between the actual losses and the losses computed under the SE approximation. Theoretical predictions obtained under assumption (6) show good quantitative agreement with ex- periment on real data. We refer to Velikanov et al. (2023); Yarotsky & Velikanov (2024) for further discussion of the SE approximation.
The main benefit of the SE approximation that it allows to write a convenient loss expansion
L
1/2 Vi+1 + Σ 2
Σ
U441 Unm -tm- Uem-tm-a Utm tmm U2- Vis)
(7)
m= 1 Oct, C...tt R041
3

---

Published as a conference paper at ICLR 2026
with scalar noise propagators U2 and signal propagators Vi. The signal propagators describe the error reduction during optimization in the absence of sampling noise, while the noise propagators describe the perturbing effect of sampling noise injected at times t ....... t.m-
For our main results in Sections 3 formulas for U2, Vz:
we will assume that T2 = O, implying particularly simple
00 U.
8
T B
Σ
Ni= = (x (b)
where eN is normalized eigenvector for Ag. and it also assumed that optimization starts from Wo = 0 so that Awo Wo W. W..
Importantly, the batch size BB| affects L only through the denominator in the coefficient in U,. The deterministic GD corresponds to the limit B - oo: in this limit U, = and 1, = V4+1
Convergence//diverennce regimes. Given expansion 7 we can deduce various convergence properties of the loss from the properties of the propagators Vi, Us.
Theorem Yarotsky& & Velikanov (2024)). Let numbers Lg be given by expansion 7 with some U, 2 0, V. 2 0. Let = Ei=1 U, and Vs: = V,.
1. |Convergence] Suppose that Us < 1. AI 00, if O(1) (respectively, V, o(1)) then also L, = 0(1) (respectively, = o(1))
2. [Divergence] If Us > 1 and V₁ > fo at least one t, then supt=1,2,.. L: 00.
3. [Signal-dominated regime] Suppose that there exist constants εv, Cv 0 such that Vi Eu max 1). Then
Cyt &v (1)) as + oc. Suppose also that Ug < and U, = 0(t Ev with some
L= Cv 1 5V 55 (1++)()) 2(1 -s)
4. [Noise-ominatedd regime] Suppose that there exist constants &y EU > 1, Cy > 0 such
that U2 = Cyt (1 + o(1) and V₁ = Ott-tt(t as + co. Let also that Us < Then
V/CC Eu (1+
L 2(1- Uz)²
(10)
Spectral power laws. The detailed convergence results in items 3, 4 of Theorem I require us to know the asymptotics of the propagators U., V,, To this end introduce power-law spectral assumptions on the eigenvalues and eigencomponents of w. in our optimization problem:
A A (1+ = Q11 (15(1+((1)), A A
+00
(11)
(12)
with some constants A, Q and exponents 0,5 > 0. Such power laws common in kernel methods or overparameterized models, and can be derived theoretically or observed empirically
(Cui al. 2 Bahri et al. 2021 Yang Salman| 2020 Atanasov al., 2021 Bordelon &
Pehlevan 2021 Basri et al. 2020 Velikanov & Yarotsky 2021). Conditions (II.. 12 (or their the capacity and source conditions, respectively (Caponnetto & De Vito 2007). The exponent Ç is akin to an inverse effective condition number: lower means that the target and the solution have a heavier spectral tail of eigencomponents with small X, making the problem harder. The exponent is akin to an inverse effective dimensionality of the problem: lower means a larger number of eigenvectors above given spectral parameter A. Only the source condition (12) matters for the non-stochastic GD rates, but in SGD the capacity condition (II) also becomes important due to the sampling noise.
weaker, inequality forms) are usually referred
O < <<<<< 1, then the source condition 12) is inconsistent with having a finite H-norm, i.e.. strictly speaking, w. is not an element of H. Such solution is called unfeasible. In fact, unfeasible scenarios are quite common both theoretically and in practice (see Section [F). The Comer SGD to be proposed in Section 4 will be especially suitable for unfeasible scenarios. Note also that if v < 2 then U, and so Tat oo, i.e. the loss immediately diverges.
4

---

Published as a conference paper at ICLR 2026
La O(t-%)), 0max min(2, cpioo)
0
Immediate divergenoe E.
I1I
1.8
max
Eventual divergence La Σg M < oo,
En A De
0.5
O
1.4
Ra MISISI MLF
C+170
Signal dominated Lu t-
Noise dominated pl/mm
6 maax
1.2
II
0.5
1/
1.5
2
Figure 1: Left: The phase diagram of stationary finite-memory SGD from Velikanov et al. (2023)
Yarotsky & Velikanov (2024). Right: Maximum acceleration factor Omax min(2, v, (+1/6 for
Corner SGD in the -gmaa-ddminated regime (see Theorem
Stability and asymptotics of the propagators. Let say that square matrix A is strictly stable all its eigenvalues are less than in absolute value. It is natural to require the matrices Sx to be strictly stable for all À spec(H), since otherwise U2 V,, and hence L,, will not generally even converge to as t + oc, At A = 0 the matrix Sx=0 has eigenvalue I and additionally the eigenvalues of the matrix D; accordingly, we will assume that D is strictly stable.
Theorem 2 (Yarotsky & Velikanov (2024)). Suppose that D) and Sx are strictly stable for all X E sp) Recalling the characteristic polynomial x(1, X) = dct( Sx) = P(u) - -QQ(), define the effective learning rate
Ckef = -Q(1) (1),
(13)
and assume that Ceff >> 0. Then, under spectral assumptions
with 2.2 the propagators
V1, Uz given by Eq. ( obey, as + oc,
Vi = (1 (())) (2
(14)
U. = (1 + o(1)) (Dlett A)2/m [(2 1/u) (21)2/u-2
(15)
BB//
Combined with Theorem this result yields the (C, 1/v) -phase diagram shown Figure I left. In particular, the region the noise effects are relatively weak and the loss convergence L, x t-S has the same exponent as plain deterministic GD. This holds for all stationary finite-M algorithms and such algorithms cannot accelerate the exponent. In the present paper we will focus on the signal-dominated phase and propose an "infinite-memon" generalization of SGD that does accelerate the exponent.
1,0 < < 1/v represents the signal-dominated phase in which
THE CONTOUR VIEW OF GENERALIZED (S)GD
We consider the propagator expansion as a basis for our arguments. Observe that write the expression (1 or ( -xx) appearing in the definition of propagator U, in Eq. (8) as
1 f
(1 ( 2ni Jul=r 1' ) Sx) -x)- )du,
(16)
where |||| = r is a contour in the complex plane encircling all the eigenvalues of Sx. Next, simple calculation (see Section A) shows that
Q(u)
1
(1 (p= 35))'(-0))) P()) -Q(())pp
(17)
((a)
where P(n) - (Q(1) is the characteristic polynomial of Sx introduced in Eq. ( and
(()) = P(p)
(18)
Q(1)*
5

---

Published as a conference paper at ICLR 2026
| 2
D
(P)))
X
Figure 2: Left: The map vp 1' for Heavy Ball with P(μ) (μx 1) 0.4) and Q(u) = -
Q
The contour 7 v(f:: : |a| 1}) encircles spec(H) (H). The map V bijectively maps {nd > 1] the exterior open domain D., with boundary y. See Sec. B for more examples and a general discussion of memory-1 contours. Right: Contour y corresponding to a corner map V with angle Ar.
We see, in particular, that the propagators Ug depend on the algorithm parameters only through the function V
U² T1
1 dj 2 |n|=r (()) A
(19)
2
A similar observation can also be made regarding the propagators V,. Indeed, VE's are different from U,'s 's in that they involve the expression (1 o" (5) instead of o" ) ())) The contour
representation for (1 o1 )5A()) ) is similar to Eq. (16). and then simple calculation gives
(p) (1 (p (d) (w(w) 1)(1
(20)
As a result,
V = Σ
Ae )²
1¹ (H)
2
(21)
2Ti |a|=r (V()) ((p-1)
k=1
Recall from Eqs. 5) that P can be any monic polynomial (i.e., with leading coefficient 1) of degree M + 1 such that P(1) = O. while Q can be any polynomials of degree not greater than M. Since by Eq. 7 the loss trajectory Le is completely determined by the propagators U,, V,, we that designing stationary SGD with memory is essentially equivalent designing rational function subject to these simple conditions. By (4), the function V = ? can be interpreted as describing the (frequency) response of the gradient sequence (VL(w.)) to the sequence (w.).
P
Let us consider the map V from the stability perspective. Recall that we expect SA, to be strictly stable for all the eigenvalues 0 E (p)c In terms of v Q this means that v (p)) * D for all u E C such that |A] > 1. This shows, in particular, that we can set the radius r = 1 in Eqs. (19), simple eigenvalue of unit absolute
21). Additionally, if D is strictly stable, then So has only value, / = 1, and so (()) + 0 for |p) = 1 * 1. Let us introduce the curve ? as the image of the unit circle under the map V. Then the last condition means that the curve ? goes through the point 0 only once, at μ 1.
In general, the curve 7 can have complicated shape with seff-intersectionss, and the map V may be injective the domain |r| 2 In particular, the singularity of v at J = is (+1--)))(()) so a vicinity of = oo the function V is injective if and only deg(() = M (and in general V may also have other singularities at [14| > 1). However, may expect natural, non-degenerate algorithms to correspond to simple non-intersecting curves and injective maps A on [и] > 1. For example, this the case for plain (S)GD and Heavy Ball, where 7 circle and ellipse, respectively (Fig. left). See Section B for general discussion of memory-1 algorithms.
Given a non-intersecting (Jordan) contour 7, denote by D., the respective exterior open domain. Then, by Riemann mapping theorem, there exists a bijective holomorphic map V {p E C: [A]] > 1} + D., Additionally, by Carathéodory's theorem"" (see e.g. Garnett & Marshall (2005), p. 13)
'Carathéodory's 's theorem considers bounded domains, but our domains {n E C |e| 1) and D. conformally isomorphic to bounded by simple transformations 1/(p 1p))

---

Published as a conference paper at ICLR 2026
y. Such maps V are non-unique, forming a three-parameter family Vy o f, where f is a conformal automorphism of {u E C : ||| > 1). However, recall that our maps Vp = Q
this map extends continuously to the boundary, V : {u C |u| =
f. had the properties (oe) = oo and
(1) 0. These two requirements for V. uniquely fix the conformal isomorphism and hence Vy,-
This suggests the following reformulation of the design problem for stationary SGD with memory.
Rather than starting with the algorithm in the matrix or sequential forms I M we start with a contour y or the associated Riemann map v. and ensure a fast decay of the respective propagators U, V2 given by 19). ( (and hence, by Theorem I of the loss Lt). Of course, the resulting map V, will not be rational in general, but we can subsequently approximate it with a rational function
P and in this way approximately reconstruct the algorithm.
Q
4 CORNER ALGORITTMS
To motivate the algorithms introduced in this section, observe from Eqs. (9. (14) that in the signal- dominated regime of stationary memory-M SGD, we can decrease the coefficient Cr. in the asymp- totic formula I, (1 + o(1))Czt~ by increasing Oeff while keeping the total noise coefficient Ug < 1. Since T (1) = 0, Oieff can be reformulated in terms of Vp as
Q(1)
dv du (1)
Oeff
dF (1)
(22)
dy
Thus, increasing Oeff means making-
d (1) possibly smaller positive number. Regarding Ux: du
Ee Us note first that, by (19, it can be written as
U3= = TI
20
1² 1 2 TI
20
do
(23)
(
14|=1 (()) A (2
(2
1 (eto
A
k=1
1 is holomorphic in {|pl 1} and vanishes as μ Do, the integrals & here vanish for all nonpositive integers / = 01, so that E, collapses to the squared T.2 norm by Parseval's 's identity. If the resulting series (3 converges, we can always ensure Uz < by making the batch size [B] large enough.
Indeed, since the function ( (v)()) Ax)-1
is then natural to try = V with a contour y having a comer at 0 with a particular angle. Denote the angle by Ou when measured in the external domain D. (Figure righht) Such contours correspond to maps V : {1e] > 1 - D, such that
(14) - co (n (1
/ ,
(24)
with the standard branch of - 1 and some constant co > O. We will refer to such V as corner maps and to the respective generalized SGD corner algorithms. Formally,
d (μ = 1) ~ c0 - 1)0 1==1+ = { +00, 0<1
(25)
dp
+0.
0>1
so we are interested in 0 > 1. At the same time, cannot take > 2, since this would violate the stability condition To [1([p]]] > 1} 0 spec(H) = Ø. Thus, the relevant range of values for 0 is [1.2].
Within this range, increasing 0 should have positive curr-related effect but a negative Us-related effect, since the contour >
<[(p = 1) is getting closer to the spectral segment [0, Amax], thus amplifying the singularity ((ee)) - Ax in Eq. (23). Our main technical result is
Theorem 3 C.. Let Vi be a holomorphic function in {o E C: : |pl| > 1} commuting with complex conjugation and obeying power law condition (24) with some < 8 < 2. Assume that Ti extends continuously to C² function the closed domain [u]] 2 1, (n) + oc as μ + co, and d ((.) du
Olp - 1 as μ + 1. Assume also that Vilp E C: [p] Z 1, + 13) n [0, Amax] = Ø, where
Amax X1 is the largest eigenvalue of H. Let power-law spectral assumptions (1) (2) hold with
some v > 1,0 < < < 2. Then propagators (19) (2I) obey the following t + oc asymptotics.
1. (Noise propagators) U, Cutt/v 2(1 o(1)), with the coefficient
Ti A 1/v Cv
(r)d 0/v < co, Fo(r) = - {
1
e
B
2 1R 2 +
7

---

Published as a conference paper at ICLR 2026
2. (Signal propagators) Vi Cut (1 Cyt-
o(1)), with the coefficient
Cv QQ=Q. 0 (((r)d <0, Fv(r) = 2Ti iR
dz
2
We see that the leading L + Do asymptotics of the propagators are completely determined by the > spectral asymptoties of the problem and the μ - 1 singularity of the map V. The functions
Fu, Fy can be written in terms of the Leffter functions Eg , Eg (see Section C
Availability of the coefficients Cv. Cy ensures that the leading asymptotics of Uz. Vir are strict power laws with specific exponents 2 2-0/v and 0C, respectively. Increasing 0 indeed improves convergence of the signal propagators, but degrades convergence of the noise propagators.
The largest acceleration of the loss exponent Ç possibly achievable with corner algorithms is by factor 9 arbitrarily close to 2, but in general will be lower since, by Theorem !| the exponent of Le is the lower of the exponents of U. and V;; accordingly, the optimal 0 is obtained by balancing the two exponents, i.e. setting OC = since otherwise the total noise coefficient Uy
v. Also, we need the noise exponent 2 - 0/v // be 1,
and Le diverges for any batch size |B] < co.
Combining these considerations, we get the phase diagram of feasible accelerations (Figure right). Theorem 4. Consider problem with power-law spectral conditions (12 in the signal- dominated phase, ie. v > 1,0 < < 2- 1/v. Let mmax denote the supremum of those @ for which there exists corner algorithu and batch size B such that Lt = Ot Pc). Then
max min (2,v, 5 (+1l/v)
2 )
(26)
The phase diagram thus has three regions:
I. Fully accelerated: mmax = 2, achieved for v > 2, < << 1/v.
II. Signal/noise balanced: Omax = 2ti/c
2 < 2, max (2/1,, 1//D << < 1/v. The
condition 1/v < ( ensures that Us is finite and less than for [B] large enough.
III. Limited by Us-finiteness: 6max = V < 2, 1 < V < 2, < ( < 1/v. The signal exponent
Omaxxe is less than the noise exponent 2 0 fmmx// /v, but increasing @ makes Ux diverge.
5 FINITE-MEMORY APPROXIMATIONS OF CORNER ALGORITHMS
Though corner maps V are irrational, they can be efficiently approximated by rational functions. It was originally famously discovered by Newman ( 1964) that the function |a|| can by approximated by order-M rational functions with error O(e 0(e-<VM This result was later refined in various ways. In particular, Gopal Trefethen (2019) establish rational approximation with a similar error bound for general power functions - 2 on complex domains. For 0 E (0, 1). this is done by writing
sin(O.) Pm
zdt
sin(
2e 0
(27)
1100 +
On
em//24/00 +
and then approximating the last integral by the trapezoidal rule with uniform spacing h = / 20/M
In our setting, we start by explicitly defining a f-corner map. This can be done in many ways; we find it convenient to set
(n) = (
do
14-1
2)/
/ e (2- e2-20)s ds
1-1+e-0) )
-1-1
(28)
1 1 145
1r
11
with scaling parameter A > 0.
Proposition 1 D For any such that
< 0 < 2, Eq.
28 defines a holomorphic map v : C 00 + C
- Ap(1 + +(1))
1 - 0o,
(29)
p 1 sin((2 0)m) (p 1) (1+)
where 20 denotes the standard branch in C (-00, 0). Also, 1 (((p) 2 13) (0,24) 00
8

---

Published as a conference paper at ICLR 2026
Loss during training
Target predictions
10-4
0.8
10-2
0.6
10-3
0.4
Plain SGD
Plain SGD smoothed, 4 =t 9247
target
10
Corner SGD
Plain SGD
-0.000
Corner SGD
10
10
10a
10
10
10
0.0
0.2
0.6
0.8
1.0
Iteration
trained to fit the target y(z) 100. The loss trajectories also shown and used estimate the exponents < in power laws L2 o S-S. Corner SGD has 1.8 and is approximated using finite memory M = 5 as in Proposition 2 We that Corner SGD indeed accelerates the power-law convergence exponent of plain SGD. See Section F for details.
Figure 3: Training loss and final predictions of the kernel model 220
(2) using either plain or corner SGD with batch size oscillate strongly, their smoothed versions
Following Gopal & Trefethen (001)). approximate the last integral in Eq. 28 as
(ss)ds =o h Σ Φ((m 1)h,, h
(30)
VM
with some fixed constant I. Note that in contrast to 27 our integral and discretization are "one- sided" (s 0), reflecting the fact that the corner map (p) is power law only μ - 1, which is related to the + +oo behavior of the integrand.
Let (M) denote the map Y discretized with M nodes by scheme 30
Observe that (M) is a
rational function, (M(M) &. where deg M + and deg Q S M (in particular, P(n)
(μ - I)II M (μ 1 e (m-11//)))) We can then associate to ((M) memory-M M algorithm with particular b, D, for example as follows.
Proposition 2 E Let h I/VM M and
D = diag(1 - 4h (M--))))
(31)
b = (l..... 1)T
(32)
(CI, CM)T
(00-))(m11211)) Cm A- (00
(33)
a = A- (2 - 0)h 1 (2-0)M))
(2-0) - (2-0)1/12
(34)
Then the respective characteristic polynomial x(x) P(1) (Q()) with E- (M)
Of course, as any stationary finite-ments finite-memory algorithm, for very large the M-discretized corner algorithm can only provide a ((t-)) convergence the loss. But, thanks to the O rational approximation bound, expect that even with moderate M, for practically relevant finite ranges of t the convergence should be close to Ott-(C) of the ideal corner algorithm.
Experiments with synthetic problem and MNIST confirm that corner algorithms accelerate the exponents of plain SGD (see Appendix F and Figure 3)
ADDITIONAL NOTES AND DISCUSSION
Extension to SE approximation with 72 * O. The key assumption in our derivation and analysis of the contour representation and corner algorithms was the Spectrally Expressible approximation

---

Published as a conference paper at ICLR 2026
with 72 0 for the SGD moment evolution (see Eq. (6)). While the SE approximation in general was justified from several points of view in Velikamov et al. (2023): Yarotsky & Velikanov (2024), a
natural question is how important is the condition T2 0. This condition substantially simplifies the representation of propagators Uz, Vi in Eqs. (8. There are distributions satisfying this condition (see, e-g., Section 7.2 in Baah (2225) and references therein), but they don't 't seem form any standard natural family. In contrast, the cases = = and 71 = L, = -1 describe wranslatooniinvaiian
and Gaussian distributions; see Velikanov et al. (0023
In fact, our analysis of the corner propagators U,, V2 can be extended from 72 0 to general 72 by perturbation theory around 72 for sufficiently large batch sizes [B, Theorem remains valid for general T2, even with the same coefficients Cu, Cv (i.e., the contribution from 72 A produces only subleading terms U., Vi)- This implies, in particular, that the acceleration phase diagram in Theorem 4 and Figure (right) is not only TL-, but also 72 rg-independent.
Appendix H we sketch an argument suggesting that, least
Computational complexity. The main overhead of aiiiielyaaprooimmteed corner algorithms comm- pared to plain SGD lies in the memory requirements: if the model has W weights (i.e., dim w, = W in Eq. (]). then memory-M algorithm needs to additionally store MW scalars in the auxiliary vectors U,. On the other hand, the number of elementary operations (arithmetic operations and evaluations of standard elementary functions) in single iteration of fnitely-aprroximated corner algorithm need not be much larger than for plain SGD.
Indeed, iteration ①1 of memory-M algorithm consists in computing the gradient V (L,,w,) and performing linear transformation. In SGD with batch size [B, the estimated gradient V BB (w,) computed by backpropagation using x [ W operations. If Corner SGD is finieel-approximmate using diagonal matrix D as in Proposition 2 then the number of operations in the linear trans- formation is (MW). Accordingly, if B > M (which should typically be the case in practice), then the computational cost of the linear transformation is negligible compared to the batch gradient estimation, and so the computational overhead of Corner SGD is negligible compared to plain SGD.
Practical and theoretical acceleration. Our MNIST experiment in Section F shows that finitely- approximated Corner SGD developed in Section S can practically accelerate learning even on real- istic problems that are not exactly linear. We note, however, that, in contrast to the ideal infinite- memory Corner SGD of Section E this finitel-approxximated Corner SGD does not theoretically accelerate the convergence exponent as ( co. (As shown in Yarotsky & Velikanov (2024). this is generally impossible for stationary algorithms with finite linear memory.) Nevertheless, one can expect that such an acceleration can be achieved with suitable non-stationary approximation.
In Yarotsky & Velikanov (2024). an acceleration with factor 0 up to 2 1/v was heuristically derived for suitable non-stationss stationary memory-1 SGD algorithm. In the more recent paper Ferbach et al. (2025), rigorous proof of accelerated power law was given for a very similar non-stationary algorithm under some assumptions involving continuization.
We remark also that if the model includes nonlincarities, then even the plain SGD in the signal- dominated regime may show a complex picture of convergence rates depending on the strength of the feature learning effects. In particular, Bordelon et al. (2024) consider a particular model where the "rich training" regime is argued to accelerate the "lazy training" exponent by the factor T+C. This is different from our factor 8max min (2, v, c+1/²) due to different acceleration mechanism.
2
REPRODUCIBILITY STATEMENT
Jupyter notebook with all experiments attached in Supplementary Materials. Experiments are described in Section F
ACKNOWLEDGMENTS
The author thanks Maksim Velikanov, Evgenii Golikov, Ivan Oseledets and anonymous reviewers for useful suggestions that helped improve the paper. The research was supported by the Russian Science Foundation grant No. 25-11-00355.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. arXiv preprint arXiv:2111.00034, 2021.
Francis Bach. On the effectiveness of the z-transform method in quadratic optimization. arXiv preprint arXiv:2507.0 004044, 2025.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprini arXiv:2102.06701, 2021.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, pp. 685-694, PMLR, 2020.
Raphaël Berthier, Francis Bach, and Pierre Gaillard. Tight nonparametric convergence rates for stochastic gradient descent under the noiseless linear model. arXiv preprint arXiv:2006.0822 2020.
M Š Birman and M Z Solomjak. Asymptotic behavior of the spectrum weakly polar integral operators. Mathematics of the USSR-Izvestiya. 465):1151-116,, oct 1970. doi: 10.10700im1990000000aaahohoo im 1.11970004400aaee00000
URL httpps////ci..rr//11...1
Blake Bordelon and Cengiz Pehlevan. Learning curves for sgd on structured features. arXiv preprint arXiv:2106.0277 2021.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. arXiv preprint arXiv:2409. 17858, 2024.
Helmut Brakhage. On ill-posed problems and the method of conjugate gradients. In Inverse and ill-posed Problems, pp. 165-175. Elsevier, 1987.
Andrea Caponnetto and Ernesto Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-36,, 2007.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34, 2021.
Damien Ferbach, Katie Everett, Gauthier Gidel, Elliot Paquette, and Courtney Paquette. Dimension- adapted momentum outscales sgd. arXiv preprint 25t. 2025.
J.B. Garnett and D.E. Marshall. Harmonic Measure. New Mathematical Monographs. Cambridge University Press, 2005. ISBN 9780521470188
Abinand Gopal and Lloyd N Trefethen. Representation of conformal maps by rational functions. Numerische Mathematik, 142:359-382. 2019.
Hans Haubold, Arak M Mathai, and Ram K Saxena. Mittag-leffler functions and their applications. Journal of applied mathematics, 201 (011(1)2298628 2011.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemne:: Convergence and gen- eralization in neural networks. arXiv preprint arXiv: 2018.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://aaa..lecu..coo//ed/mmmsm 2010.
Arkadi S Nemirovskiy and Boris T Polyak. Iterative methods for solving linear ill-posed problems under precise information. Eng. Cyber, (4):50-56, 1984.
Donald J Newman. Rational approximation to |x]. Michigan Mathematical Journal, 11(1):11, 1964.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ phases of compute- optimal neural scaling laws. arXiv preprint arXiv:2405. /5074, 2024.

---

Published as a conference paper at ICLR 2026
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compu- tational mathematics and mathematical physics, 4(5):1-17, 1964.
Sumit Roy and John J Shynk. Analysis of the momentum Ims algorithm. IEEE transactions on acoustics, speech, and signal processing, 38(12)200882008, 1990.
Stiller. An introduction to the theory of resultants. Mathematics and Computer Science, T&M University, Texas, College Station, TX, 1996.
Mchmet Ali Tugay and Yalcin Tanik. Properties of the momentum Ims algorithm. Signal Processing, 18(2):17-127, 1989.
Aditya Varre and Nicolas Flammarion. Accelerated sgd for non-strongly-convex least squares. arXiv preprint arXiv:2203.01744 2022.
Aditya Varre, Loucas Pillaud- Vivien, and Nicolas Flammarion. Last iterate convergence of sgd for least-squares in the interpolation regime. arXiv preprint arXiv:2102.03/83, 2021.
Maksim Velikanov and Dmitry Yarotsky. Explicit loss asymptotics in the gradient descent training of neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Maksim Velikanov, Denis Kuznedelev, and Dmitry Yarotsky. A view of mini-batch sgd via gener- ating functions: conditions of convergence, phase transitions, benefit from negative momenta. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023, URL net http:/orrmmmmd-paPaELLLLLLL
Greg Yang and Hadi Salman. fine-grained spectral perspective on neural networks, 2020.
Dmitry Yarotsky. Collective evolution of weights wide neural networks. arXiv preprint arXiv:1810.03974. 2018.
Dmitry Yarotsky and Maksim Velikanov. Sgd with memory: fundamental properties and stochastic acceleration. arXiv preprint 0rXiv:2410.04228, 2024.
12

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Background
3
The contour view of generalized (S)GD
Corner algorithms
Finite-memory approximations corner algorithms
6
Additional notes and discussion
References
11
Derivations of Section
14
Memory-1 contours
14
Proof of Theorem
18
The noise propagators
18
The signal propagators
23
Proof of Proposition]
29
Proof of Proposition
29
Experiments
30
The synthetic 1D example
Extending the proof of Theorem 3 to 72 + 0
33
13

---

Published as a conference paper at ICLR 2026
0
Figure 4: Contours Y
V (fp |e| 1}} corresponding different memory-] maps V/ (see
Section B). Left: plain Gradient Descent (a circle). Center: Heavy Ball (an ellipse; 3 0.5).
Right: general memory-I algorithms Zhukovsky airfoil; 3 0.65, go 0.125, 11).
A DERIVATIONS OF SECTION 3
We have
P(u) det(f So)
(35)
det Sx + A( - (10(=))))
(36)
det(f Sx) det (1+ ( 8.)
(37)
(P(w) -= S2) S
(38)
follows that
-ws))(u-S>~ 1( P(u)
((1)) 11)
(39)
Q(μ)
(40)
P(u) -())))Q(p))
Next, by Sherman-Morriss formula and the above identity,
( SA) -1 (μ So X( (10=))
(41)
(1 ( (-2 (1
o= (m- ( So)
(4-SSo -o) -1
(42)
1-\(1 0r (u -
(u- (u) -o) --a))) ( )u Sa)
(μ -u--Sp -o0-1 A
(43)
Q(a)
P(a)
Using (1 Mp (o)- 4-1 it follows that
Q(a)
(10)) -S)-1(1) =
1
+ X
P(a) #-1
(44)
11
X
Q(a)
P(n)
r(w)
Q(n)
(45)
(μ 1)
P(a)
X)
Q(x)
B MEMORY-1 CONTOURS
In figure we show different contours 7 (l[pl 1}) corresponding to memory-1 algorithms
(see Section 3 for the introduction of contours). Below we discuss memory- algorithms and their contours in the order of increasing generality.
14

---

Published as a conference paper at ICLR 2026
Plain (S)GD. In (S)GD with learning rate 0 we have P(p) 11- and Q(p) -a,
(u) / 1
(46)
CK
Thus, y is the circle [= 2 a.
Heavy Ball. Heavy Ball with learning rate a and momentum parameter / has standard stability
conditions a > 0, B E (-1.1) and Amax < 2+22
(Roy & Shynk 1990 Tugay & Tanik, 1989). We
have P(t)= (μ 1)( 3) and Q()) au, so
(1² (er - 1)( - B)
(47)
ae
If |p|| 1, then piu 1 and hence
1 ((4) = (1 + +3p 1- 33.. (3)
(48)
Writing μ = iy, we get
(() (pp=--a ((1 -w)==-((11+
(( -1
(49)
follows that y is an ellipse with the semi-axis 1+3 along I and the semi-axis 1-8 along y. The learning rate a determines the size of the ellipse while the momentum parameter 3 determines its shape. If 3 > 0, then the ellipse is elongated in the x direction, and otherwise in the y direction.
Assuming 3 0, the eccentricity of the ellipse equals / (1 (1- 3)3((1 3)² 2/B Plain T+8
GD is the special case of Heavy Ball with 3
General memory-1 (S)GD. In a general memory-1 algorithm we have P(n) = (1x - 1)( - ()) and ((+) q0 + /1/4, so
(() (w- ()
(50)
9o 91
Heavy Ball is the special case of general memory-1 algorithms with go = 0.
In Yarotsky & Velikanov 2024) it was shown that on the spectral interval (0, Amax] the strict stability of the generalized memory-1 SGD is equivalent to the conditions
1 1 3
2+ 23
-1 < B < 1, @o
20
91 -90
(51)
Amax
Amax
(note that the Heavy Ball stability conditions result by setting go 0, Q1 -a).
ZHUKOVSKY AIRFOIL REPRESENTATION.
The map V can be written as a composition of linear transformations and the Zhukovsky function
((z = 1
(52)
12
Indeed, let then
#1 = f(( = 90 + 91H4
(53)
111-90
11(1- 3)
(1))
91
(54)
f1
2 4
+ 1 3
r
(55)
11
41
Vr
11
2 91
+ +1+1 + 3
(56)
41
/r
41
where
r = 9u
1)
qu
+
+ 3
(57)
91
41
and vr is imaginary i < 0.
Thus, the contour = [([[A] = 1)) is a rescaled image of a circle under the Zhukovsky transform, i.e. a "Zhukovsky airfoil".
15

---

is inconsistent with Published as a conference paper at ICLR 2026
CONDITIONS OF INJECTIVITY. As discussed in Section 3 the case of maps V injective the domain | > 1 seems especially natural and attractive. Let us examine when the map V given by Eq. (50) is injective. We can assume without loss that qi * 0 since otherwise the map V/ is not locally injective at co.
The Zhukovsky transform can be written composition of two linear fractional transformations and the function w = 2²:
+ 1 w
w 2 /
J(p]
(58)
#
The image of a generalized disc on the extended complex plane under a linear fractional map is again a generalized disc, and the map w = x² is injective on a generalized open disc if and only if the disc does not contain 0 and oc. Hence, necessary and sufficient condition for be injective on generalized open disc is that this disc not contain the points #1. It follows that V is injective on the generalized disc [u] iff
40
+ 1.
(59)
ạ1
Let henceforth assume the stability condition separately the cases of negative and positive r.
-1
1 given in Eq. 51
Consider
1. r Σ 0 corresponds to -1 < go < -3. In this case condition 559 is equivalent to -1 20
i.e. it holds.
However, the special case = is the degenerate scenario in which the denominator of V vanishes at μ and the stability condition Q1 < 2o in Eq. (11) is violated, so we will discard this special case.
2. > corresponds to de 21 condition (59), leaving only the option
or ye -3.
The option go
m 2a > -3. 91
(a) If 20 0, then condition
59 is equivalent to
Vr
1 +
9o
(60)
91
which holds true thanks to the assumption 8 3<h 1.
(b) If qu 2 0, then condition 59 is equivalent to
41
VF S
90 91
(61)
which holds iff
20 1- 8
(62)
41
3 3
Summarizing, assuming the stability condition -1 < B < 1 and excluding the degenerate case Qo -Q,, the condition of injectivity of the map on the domain |M] > reads
40 1 1-3
(63)
41 3+3
We remark that this condition also be reached in different way. There are two obvious essary conditions of injectivity y on the set [uu] >> 1: the absence of poles of and zeros of the derivative V from this domain (the latter ensures the local injectivity). The absence of poles means that -1 < go < 1. The zeros of the derivative are given by the equation
qu
2
2 9o
(3
B =00
(64)
91
Both roots of quadratic equation μr² +aa = lie inside the closed unit circle iff [a 1+b < 2. Applying this condition (and discarding the case qu/91 = -1). we reach the same inequalities (63)
In particular, the conditions of absence of poles and the roots of the derivative turn out to be not only necessary, but also sufficient.
16

---

Published as a conference paper at ICLR 2026
ALGEBRAIC EQUATION OF THE CONTOUR. The circle |4|| = 1 is real algebraic curve defined by the polynomial equation x² + y² = 1, where μ x + iy. Images of real algebraic curves under rational complex maps again algebraic curves, and the corresponding equations can be found using, e.g.. Macaulay resultants (Stiller 199).. the particular case of unit circle the computation can be performed in terms of standard resultants follows.
P(a) Q(a) where is a polynomial of degree M + 1, and Q a polynomial of degree I M; we assume P and Q to have real coefficients. Denote w = (p), then
Recall that (())
(Q(p) P(p).
(65)
Since belongs to the unit circle, 1ul 1. Applying complex conjugation and the identity μ = 1/P to the above equation, we get the second equation
(Q(1/))==(1
(66)
Note that Q(u) - 1 and P(n) - - P(1/-p are polynomials in μ of degree M + 1
or less. It follows that satisfies two polynomial conditions:
Ti(x) 0, T2(x)
(67)
where
P())
T2(p) P(w) wQ(M),
(68)
ie. p is common root of two polynomials, (M) and T2(p). Two polynomials have common root iff their resultant vanishes. The polynomials Ti (1)) 72( have degree M + 1 or less and
include w and w linearly in their coefficients. It follows that the set (([pl 1}) can be described by the equation
res (Ts) = 0,
(69)
which is a polynomial equation in w and w of degree at most 2(M + 1).
We implement now this general program for M = 1. Given quadratic polynomials
(1) Aμ² 2 + Bu + C. T2(u) Dμ² + Ep + F,
(70)
(71)
their resultant can be written
res res(Ti, (Ti,
-AE - -BB
(72)
In our case
A = 1, B= (8 wQl), C = 3 - wgo.
(73)
D 3 TQO. E -3 1q)) F 1.
(74)
Considering real 8, go. qı and w = x + iy, we get
re) (1 (3 )2 2 2)2
(75)
-(3 - [(1 30 9i 00]x - 9ooi (a.2 + y)
(76)
- (B + 1)) + 2y²²
(77)
It follows that the contour (((lpl) = 1}) can be described by the quartic (in general) equation
(1 (3 )ox)² -0:r)22 (B² 1 + [(g] 3 41 go]= 9091 (2² + ))
(78)
(8 1) + 2
(79)
As expected, in the Heavy Ball case go = this equation degenerates into the quadratic equation
(1 -1-33 (2) (5² + (B 1) (B + 1)2
(80)
17

---

Published as a conference paper at ICLR 2026
C PROOF OF THEOREM 3
C.I THE NOISE PROPAGATORS
The function Fu. Let us introduce the values
d I
d
U A)
(81)
2Ti |+|=1 (1))
2x
(((e0))
so that, by Eq. 19). the propagator U2 can be written as
U. B
(82)
With the change of variables ò =/
-11/0-1 U(t,A)=
7/\1/0
CHA' dss
V (eisauee //+1 = --10/0-1 Fo
=-
(83)
2π
-/21/0
where we have denoted
///1/0 eir
1
Fu(r, A) =
(84)
2T
- (cess/1/)))
Recall that we assume (p) - -cu( (i1 1) + o(1)) as - 1. By formally taking the limit > in the integral, we then expect (r, converge
ers ds
Fu(r, 0) def Fu(r)
def
(85)
2T (sge /) [ +1
for any fixed r. This integral can be equivalently written as
1 f
d=
Fo(r)= 2 Ja C +
(86)
assuming the standard branch of 2 holomorphic in C (-oc,
The function Fu can be viewed (up to coefficient) as the inverse Fourier transform of the function + elsigu /2 | + 1)- Note that, thanks to the condition 0 > 1, the latter function is so F& (r) is well-defined and continuous for all r e R. The function Fu can also be written in terms of the special Hittag-Lefffee function Eg.e (see its integral representation (6.8) in (Haubold et al. 2011)):
1-1
/ cx ) (x,(22) = 2Ti J -a=
p
1 h
Fo(r) =
Eg,9
(87)
cxp
where the integration path 7 encircles the cut (-oc, 0) and the singularities of the denominator.
The following asymptotic properties of (r) can be derived from the general asymptotic expan- sions of LeffferLeffler functions (sections and 6 in (Haubold et al. 2011)), but provide proofs for completeness.
Lemma 1.
1. (r) 0. S
2. Fr(r) (1+ o(1)) (o1)) cut 0-1 as r Y 0.
3. Fu(r) + +(1)) ((1))
+oc.
Proof. 1. Consider the function f22 integrated in Eq. 86) For any r E RR and 0 E (1,2),
the function is holomorphic in any strip To {0 Rz a), a 0, and is bounded in Ta
18

---

Published as a conference paper at ICLR 2026
[f(z)] -)) It follows that the integration line iR can be deformed to iR + without changing the integral. If r < O, then by letting a - + +o we can make the integral arbitrarily small.
2. By the change of variables =
Folr) = u(r))
(88)
where
e dz
u(r) 2ricy iR
(89)
20
Cyy p
We can find No (r) as follows. Observe that the integration line iR can be deformed to the line Yas a > O, encircling the negative semi-axis:
Ta Ya,1 Va,2 Va.3,
(90)
2a {= E C Sz -a,R: S 50)
(91)
2a = {= E C |a| =a, - <argz 2 < })
(92)
1a.3 {= C 3z a, Rr 53.
(93)
Indeed, if is sufficiently small, then this deformation occurs within the holomorphy domain of the integrated function. The integral is preserved since > and since we deform in the half-plane where the argument of e= has Rz' < O.
Thus, for any fixed we have
/
dz'
e dz'
f
dz'
lim u(r) = lim 0'0
(94)
r'2 2wicH %. 10 Cyp ,
2ricy
29
2xice (0 1) 70 2
where in the last step we integrated by parts. In the last integral, thanks to the weakness of the singularity -/1-8 at (note that 1-0 - 1), we can let -
dz'
+00
-- (e-m((1-0) mi(110) - =
(95)
-10-1
si (T (2 0)
(96)
2Ti
(97)
r(e 1)²
where the last step we used the identity r(=)T(1 - 2) = sin()s)) This is essentially Hankel's representation of the Gamma function, valid for all 0 E C by analytic continuation. Summarizing,
lim u(r) ov (0 1)T 0
(98)
((T)
3. We start by performing integration by parts in Fū
-1
/ eddd
2 /
C
2"220-1d2 (ouz² + 1)2
(r)
(99)
2vir c +1
Performing again the change of variables rz 2,, we have
Fo(r) = v(r)r 9-1
(100)
where
C /
e²' 210-2 dz
(r)
(101)
2
JR (co (2'/2'/)) /r) 1)2
To compute li u(r), again transform the integration line. Let be line that lies in the domain C (-o,,), 0) and can be represented as the graph of a function Rz = f(3z) such that
f(v) 2 c c1lyl Co
(102)
with some constant C1 >> 0 and co.
19

---

Published as a conference paper at ICLR 2026
Note that the integrated function has two singular points z' E C (-00,)) where the denominator cy (a'/rr) +1 0. These two points depend linearly on r. Require additionally that y lie to the right of these points for all O, so that ilR can be deformed to y without meeting the singularities. This requirement is feasible with a small enough e1 > 0 since, by the condition < 2, the imaginary parts of the singular points negative.
With these assumptions, integration in Eq. (001) can be changed to integration over y'. Thanks to condition (102), the integrand converges exponentially fast z' + oc, and we can take the limit r + +o0:
lim e(rr) +-00000
0p0
20 dz'.
(103)
2Ti
The contour 7 can now be transformed to a contour encircling the negative semi-axis, and applying Eq. (97) we get
lim v(r)
Cp
-CV
(104)
T(1 0)
I(-0)
The formal leading term in U.. We have
U,
(L,
1x)
TI
X
2/0
F
(1X
1/0
Ax)-
(105)
B
B
To extract the leading term in this expression, we the second argument in Fu (1 Ac) 0:
T.()) def
T1
F (7 =
TI
4ut
(106)
B
B
where
a; = 2-0/00
X
2/8 F2 (1 1/0) = t Σ 32
(107)
Lemma 2.
lim Or = 1/v [" (r) 0/0 A2/1 f --00/0 F (r)
<o..
(108)
Proof. Note first that the integral on the right is convergent. Indeed, by statement of Lemma ! p1-9/u (r) r1-8/v42(0-1) ,8(2-1/v)-1 near r Since assume 1 and 0 1, the
function -1 (r) is bounded near 0. Also, by statement 3 of Lemma 1 1-10/v (r) x (1-0/v-2(0+1) 0(r
+00
For any interval / in RR,,, denote by Sr.t the part of the expansion 107 of a, corresponding to the terms with 1/0 ////0 E I:
-1-=1-0/v
Σ
(109)
Recall that the eigenvalues À are ordered and A Ak "(1 Ak- follows that for given fixed number > 0, the condition 1/0 > holds whenever k < kr. where
o(1)) by capacity condition [T].
ky = (1 + (0)))
- ac.
(110)
Then, for / = [u, u] with 0 < u < v < oc we have
lim inf S1, > A inf l (r -8/v - -0/0) (-:00
(111)
lim sup S1, < A sup /2 Fr) -0/v - -0/)).
(112)
t+000
rET
20

---

Published as a conference paper at ICLR 2026
Moreover, for any interval I = (u, u| with 0 < u< do can approximate F 0/v by integral sums corresponding to sub-divisions I= I UlU....... Uln. apply the above inequalities to each g. and conclude that
lim Sr. - - 1-00
(113)
It remains to handle the two parts of corresponding to the remaining intervals I = [0, u] and
(e, so). It suffices to show that the associated contributions St, can be made arbitrarily small uniformly in t by making u small and large enough.
Consider first the interval I = [v, oo). Note that by Lemma for all r > we can write
2 (r) S Cr Cr-20
(114)
with some constant C, and we also have for all k
A S d S A
(115)
for suitable constants A,, It follows that
S₁ 0/-0
Σ
C(UA_ 20
(116)
A 10000
2//0///0/
0/000220 CA 2
20
(117)
8/00 (t/v)
(118)
= -1000--011120011
(119)
with (11 denoting an expression bounded by a t, v-independent constant. This is the desired con-
vergence property of St,--
Similarly, for the other interval 1 -= [0, u] we use the inequality
(r) S 20 r <<
(120)
also following by Lemma Then
St. S t 8/v
Σ
(121)
k
t 0/v429 CA?
k -2
(122)
A= (t/w))/)
0(1))-1/v4 (L/u) (1/v)(1-2v)
(123)
O(1) (10)0-1
(124)
which is the desired convergence property of ST, since >> 1.
Completion of proof. We have shown that if we replace (t -Ae) by Fe Eq. 105)
we get desired asymptotics of U, in the limit t +oc. We will show now that this replacement introduces a lower-order correction o(t:/==2); this will complete the proof.
We start with a technical lemma (to be applied with f v giving lower bound for deviations of asymptotic power law functions with 0 < from real values.
Lemma 3. Suppose that f: {u C: |||| 1} + C is continuous, f(u) = -c((p (1 as 4 + 1 with some 0 (0, and > Suppose also that f((u E C |p| = 1, + 1 Amax] =
Ø for some Xuax > O. Then there exist constant C > O such that
1(eet) >1 2 C(ss| 2),
8 I- m), x E [0, Amax]
(125)
21

---

Published as a conference paper at ICLR 2026
Proof. If we fix any small 0, then, by the condition f(Lu C: |p] = 1, + 13) n [0, Amax] Ø and a compactness argument, there exist C", > 0 such that
|f(e*) - Aa > C² > C( + X); 8 EE (s, TT, E [0, Amax]-
(126)
remains to establish inequality
125 for E. Since f(4) = df - (1+ and 0 E [0,2),
15(e²*) X|
le sign())
c|2 sign())00/22
o()) + X
(127)
l slgn(()) (1s o(1) de S
(128)
2 Rie (1a_pp_(_000000000_0100000_00000 d(1) + de ve-=signs)007/4
(129)
com(00r/a((caa +
(130)
(131)
for |s| small enough.
Lemma 4.
1. (r, 2) Fu(r)] o(1 as X - 0, uniformly in all r E R.
2. (r(r, O(+) for all of the form t2 1.2.... uniformly in all X
(0, Amax]-
Proof. 1. suffices to show that, as > O, the functions
fA(s) (2T)
(122)-((-00 (- (e (-ve)) 1/A+1)- (//2///00 (s)
(132)
converge in (R) to
So(8) -(2m)- ((s
((sign s)0x 12 1)-1
(133)
Let us divide the interval |-m/1//0 N/ //>2/0] into two subsets:
(A) = [-> ,_,,, I₂(A) 1 1/8 ///4/1 N I(A),
(134)
(135)
where /t is some fixed number such that / < 4.
By Lemma T ((e>>>X")/>> 2 c uniformly for all I-T/\1/9 T/110/ / and
(0, Amax]. It Tollows that
(eeisss// 3/> - cx Xe (0, Amax]- #E12(A)
inf
(136)
for some constant c > O. Using the condition 2 < h. it follows that
f (A)
[fx(s) 0()-)/0000000000)) 100 o(1). N. 0.
(137)
Thus, we can assume without loss that the functions fA vanish outside the intervals (A). On these intervals, thanks to the condition h f we have
fa(s) = - -( - - (cwe (siga 3)x ||| (1) -1)
(138)
uniformly in E (2). We can then apply the dominated convergence theorem to the functions - fol, with dominating function C(1 + |-|0)-1, and conclude that - fo in (R),,, as desired.
2. We start by performing integration by parts in U A):
1 $f
djc'
V'(n) djc 2 |+|=1 (VL(p) - 2)
1
ut, A)
(139)
2rit
|+==
v(())
-
22

---

Published as a conference paper at ICLR 2026
implying
1 TM ((eet) 2Tt (e's) A12
(t), -
(140)
We will show that this integral O
Note first that we can replace the integration |- r| by integration [-a, a| for any 0 < a n. Indeed, by our assumptions v is C on the unit circle, and (() there only μ 1.
Accordingly, the remaining part of the integral is non-singular as À y 0 and so is uniformly bounded for all À E (0, Amax]-
Recall that by assumption (p)) O(k 110-1) as μ 1. Applying again Lemma
C 000 - C' [tt & & Jo
(s" 1)2 tA
=
(141)
with some constant C' independent of t, A. It follows that
C'
[Fo .)1 = -/-
(142)
121/0
as claimed.
We return now to proving that replacing (1x lower-order correction v(t) It suffices to prove that Da; - 0, where
1/0 AA)) by Fo (1 in Eq.
105) amounts to
Da: 12-0/0
Σ 2/0 (1
1/0 .Ax) - (1X
(143)
=t-8/v Σ
(tX (FE (x) - F2 (tx
(144)
For any interval I C R, denote by AS t the part of Aa, corresponding to the terms in
144 such
that tA 1//0 E I. By statement I of Lemma for any u > 0 we have, as - oo,
(S = 0(1)2-01
Σ
2
2/0
(145)
RatA 1/ <u
(146)
o(1),
(147)
where we have used the fact that 2v/0 > v/ 1.
Now consider the remaining interval ju, +oo). It suffices to prove that p can be made arbitrarily small uniformly in by choosing a large enough. By statement Lemma we
write
AS to) C/20//0
2/ (12
(148)
k >a
(1/)(1/
Ct 0/v
(149)
< C 0/v
(150)
with some u--iddependent constant C'. This completes the proof of statement of Theorem 3
C.2
THE SIGNAL PROPAGATORS
The proof for the signal propagators follows the same ideas as for the noise propagators, with ap- propriate adjustments.
23

---

Published as a conference paper at ICLR 2026
The function Fv. We introduce the values
( 1 T
(e
v(l, 2Ti p|=1 (V(M)) - 1) 1) 2x
(151)
(V((t 2)(000-1 -
so that, by Eq. 21, the propagators Vi can be written
V \vle
(152)
With the change of variables Ø- sA
1/1
7/11/0
(-VP )f AA
V(I,
(-V(ClsAVP)) + 1)(ciont1o -1) Fv( A),
(153)
2T
-//21/0
where
1110
17/1/0
(-VI )/Jeerrds
(r
(154)
2T
- (- (VC(AAAA 1/a /A+ ()(etasveo
We again recall that (()) = -cv(u (u-
-cv(u=1)(((((())) as μ + and formally take the pointwise limit
y in the integrand to obtain the expression
Or, (r) def
1
oye (sign s)0 | e ds
(155)
2Ti
(cige e)0x/2|s3 + +1)s
Cyge (0 1)x s 1r 2000 /2 +1) (
(156)
2x
for any fixed r. This integral can be equivalently written as
Fv
/ J. Cy +1
(157)
assuming again the standard branch of 20 holomorphic in C (-o0, Oj. The function Fv can be written in terms of the Mittag-Leffler function Eg = Eg,1 (the special case Ea,b given by Eq. 87
(r(r) = Ee
(158)
Cy
157 not absolutely summable, due to the 2- fall off of the integrand at & + co. However, the integrand is square-summable and so Fv, a Fourier transform of such function, is well-defined almost everywhere square-interrable function.
Note that, in contrast to Fu, the integrals 156).
In fact, Fl can be defined for each particular 0 by restricting the integration in (16) segments [u, o] and letting u + oo and v - oc. Indeed, the resulting Fourier transforms FV F(U,e) converge to Fy in (TR).. However, these transforms are continuous functions of r, and as u - 00, v oo they converge pointwise, and even uniformly on the sets fr : |r] > <), for any fixed > 0.
To see this last property of uniform pointwise convergence, note that the integrand in (16)) has the + xx. The component ((s-1-0) is in I,t, so the respective part of T2(a,2) converges as и + -00, + oa uniformly for all r E Regarding the -1 component, integrating by parts gives
form (s-1 +05--0(--100 (s---=))e
irs ds
irs a=1 or J eird ds
irs
+
(159)
This expression converges as + oo uniformly for {r |r| > t) with any fixed > 0, as claimed. The same argument applies f
The above argument shows, in particular, that Fy is naturally defined as a function continuous on the intervals (0, +00) and (-o0, 0).
We collect further properties of (r) in the following lemma that parallels Lemma | for Fy. The proofs are also similar to the proofs in Lemma
24

---

Published as a conference paper at ICLR 2026
Lemma 5.
1. (r)r) 0f < O.
2. (v(r) + ass 00.
3. (r(r) (1 +1))
+00.
Proof. 1. Like Lemma | this follows by deforming the integration line in Eq. 157 towards
+00.
By the change variables rz = z',
20 del
Fv(r)
(160)
2Ti JR
20 + +.g p
As in Lemmal 1 semi-axis:
the integration line iR can be deformed to the line Va. a > 0, encircling the negative
7a.1 U 7e Va.3,
(161)
7a.1 {i E C 32 -a 50,
(162)
72.2 {= E C | = =a. =a-- <rrg < })},
(163)
Ya.1 { E C: 3z = a. Re S0).
(164)
Taking the limit > 0, we get
1- c² dzl
e dz'
lim Fv-(r) lim r' 2τi
(165)
~
2 + cy
2Ti 7a
since the last integral simply amounts to the residue of ex / x'' = O.
3. Using the same contour 7 as Lemma 1
26 e di
Fv (r) = or)r u(r) 2xi Jy Gy
(166)
(z' (2/rr² +
Taking the limit > +00 and deforming the contour to the negative semi-axis as in Lemma
/
lim v(r) r-+
2 20-1 dz'
(167)
2 Jay
I(1 0)
The formal leading term in V,, We have
V = Au (VV ExceeT W2) Au).
(168)
A=
To extract the leading term in this expression, we set the second argument in Fv 1/0 Ax) to O:
((1) def
) but 06
(169)
where
b. = 00 Σ 32
(170)
The analog of Lemma is
Lemma 6.
lim b, 2 0 FF(r)dr's =QQ / 100, - ( - 1000
<
(171)
25

---

Published as a conference paper at ICLR 2026
Proof. First, observe that, by the source condition (22 and Lemma 5 the integral converges r since 8Ç O, and near = oc since Ç < 2.
We can establish convergence of the sequence b, using the same steps in Lemma 2 introduce the sums St, comprising the terms of expansion (170) such that 1
We first
intervals [= [u, e] with 0 < < < we show, using the source condition 12) and approximation by
integral sums, that
| =
lim S1 = - ((800)Q((0100))) 6-000
(172)
After that we show that the contribution of the remaining intervals (u, t00) and (0, u) can be made arbitrarily small uniformly in by adjusting u, v.
In particular, consider the interval
+00). Let R(A) E (x w.)² denote the cumulative distribution function of the spectral measure. Since the spectral measure is compactly
supported, assumption (12) implies that R(A) I Q'XC for all À > 0 with some Q' > 0. Using statement of Lemma and integration by parts, we can bound
S ((,,+00),t S 5
Σ
Au(of w. w.)x -2
(173)
k:tA 1/ 20
00
dR(A)
(2
(174)
(w/e)o
X2
R(A)
R(A)dA
C2) 2)
+
(175)
X2
(wft)a
(w/130 13
22Q''0ic-3)
25-3d
(176)
(u/2jo
(177)
with some constant C' independent of v, For the intervals I = (0, 2) we have
Sio,a). S 05
E Ae(et...)'C
(178)
k
s cttsQ((u/tt))
(179)
(180)
Completion of proof. remains to show that the correction V due to replacement of
Fv(tX// >b) by (tX in Eq. (168 is o(t- We first establish an analog of Lemma
Lemma 7. Assuming that 1/0 with 1,2....
1. -V o(1) as X + O, uniformly for > e, for any > 0.
2. Fv < C' C'min 2 1) for all t = 1,2,... and X E (0, Amax], with some X- independent constant C.
Proof. 1. The proof of this property is more complicated than the earlier proof for Fy because the integrals defining Fy are not absolutely convergent. Recall the integration by parts argument 159h used to define (r(r) as the pointwise limit of the functions F: (=(()))))) (r). We extend this ap- proach to the functions (r, A) with À > 0. Specifically, let F (r, A) be defined as Fv (r. X) in Eq. (144), but with integration restricted to the segment [-u, u/. By analogy with our convention Fv(r)= = Fv = 0). denote also F("((1 (r) = F(") (r. A 0). We will establish the following two
properties:
26

---

Published as a conference paper at ICLR 2026
(a) PV (r, A) (r, A)l for all Amax with a u, A-independent constant C.
(b) For any u, LFV (r, -) - - as 0 uniformly for R.
Observe first that these two properties imply the claimed uniform convergence o(1) as À - O. Indeed, given any å > O, first set
3C so that by (a) we have
FV (1) (r, A) Fv A) s 5/3
(181)
for all r > and 0 < X < Amax This inequality also holds in the limit À > 0, i.e.
(w) (r) Fv(r)] 6/3.
(182)
Now (b) implies that for sufficiently small À we have
IF (r, A) (r)
(r) - 6/3
(183)
uniformly in E R. Combining all three above inequalities, we see that for sufficiently small À
- -Fv
(184)
uniformly for > as desired.
remains prove the statements (a) and (b). Statement (b) immediately follows from the uniform
A > 0 convergence of the integrand in expression (154 on the interval E [-u, u].
To prove statement (a), we perform integration by parts, using the X -prriooodyy of the integrand:
1 (r. 2)
Fr(r,
(185)
1//0
) )/A)eirsds ds x178 (277011-Nuu (Cisssi1)) 1/2 A 1)(esssiia -
(186)
2
X1/8
(V )/d)etre
(187)
2πr
(VV(eist110) )/2 - 1)(ess
1) a= 1010 200 20 [l-u,
iX'0 (' )/2)(e -1) (V(( (I) 1/a 1) e
(V(ets>'))/A () 1)2
By our assumptions on V, Lemma and standard incqualities, there exist X, s-independent constants C, e >> 0 such that for all À e (0, Amax] and [- X x
(c
)1 5 C
(188)
)1 5 - - (0-1))
(189)
(e
)/1 - 1 c(1 + |s|9),
(190)
S/A/ 1| 2 cls/
(191)
Applying these inequalities to Eq.
187). we find that
1E (r,A))FF(C - (r)
(1 + uluu + 1
|s| ds
(192)
28. Fi M- (1 +1+ 1s
C
(193)
ru
as desired.
Note that
Fv(rr,)) C, C<o,
(194)
simply by setting u = 0 in the bound 192 since the first term on the r.h.s. of 192 vanishes and the second converges thanks to 0>> 1.
27

---

Published as a conference paper at ICLR 2026
It remains to prove that Fr, A) is bounded uniformly in r. X. It suffices to prove this for with some fixed e O, since for larger this follows from bound (944. Since tx this means it is sufficient to consider
s (c/t)).
(195)
To this end consider the original definition (151) of (t., A) in terms of integration over the contour {11l = 1]. We will deform this contour within the analiticity domain {o E C: |/| 2 1] to another
contour to be specified below, that fully encircles the point μ = 1:
(()) du V(L, 2ti J, (V(()) 2)(1 - 1)
(196)
is convenient to subtract the residue of 4² /(1 1) equal to 1:
V(t, A) 1 2xi f. (w - A) 1) 2ti Jr
1 ( dy
1 1 1'-id X
2ti Gr (V(u)
#-1
N)( - 1)
(197)
We define now as the original contour perturbed to include an are of radius 1/t centered at I:
7 71 72,
(198)
Joi S0i 72 (1
71 = {e "0)
(199)
(200)
where Φ1 (0,
F) tb E (5, 77 are such that y is connected. Note that Φi x & as l - 00.
Now we bound separately the contribution to the integral from and 72. For m and - < の S n use the inequalities
| Al 2 clol"
(201)
1e 2 dl
(202)
with a o, A-independent constant > 0. This gives, using Eq.
195
\/ AL, - 1)
1 du
dΦ
S C" x
s AC
+
S
C" Xt
Xt" S C.
(203)
1010T
11
For the 7/ component we use the inequalities
1 + c 1-1 Se t
(204)
(1 + 50
>1 2 ct
2 S à $2.
(205)
(Inequality (205 relies on the assumption 0 < 2 and can be proved similarly to Lemma This gives
Al S
72 (1)))) A) 1) XC [ 0 .
10 dd
de C'Xt
(206)
Fixing some O, we see from Eqs. (00)) (006 that under assumption 195 the expressions [V A) - 11. and hence (t, A)[, are uniformly bounded, as desired.
This completes the proof of the lemma.
This lemma can now be used to show that replacing (1X 1/0 Ax) by (V in Eq. 168
amounts to a lower-order correction o(t-(C) in the propagator V.. The argument is similar to the respective argument for Fe in the end of Section C.I Statement of Lemma i used show this for the contribution of the terms with u < th < v, for any 0 < u s<v< +00. Then, for terms with 1x1/9 < we use the uniform boundedness of (r A), i.e. the part (r, S C of statement 2, and show that their contribution can be made arbitrarily small by decreasing u. use the part (r, A) < C of statement 2, and show that their contribution can be made arbitrarily small by increasing v.
Finally, for terms with tx//9 >
This completes the proof of Theorem 3
28

---

Published as a conference paper at ICLR 2026
D PROOF OF PROPOSITION
To simplify notation, set A = 1; results for general A's are easily obtained by rescaling.
Note first that for any μ C \ [0. 1] the integral in Eq. (88 converges and is nonzero. To that is nonzero, note that if μ has a nonzero imaginary part, then the integral has a nonzero imaginary part of the opposite sign, hence is nonzero. On the other hand, if Ju
μ < 0, then the integral is strictly positive or negative, so also nonzero. It follows that the expression in parentheses is invertible and so (p) is well-defined for all μ C [
The asymptotics (p) -p(1 + o(1)) at + oo is obvious.
To find the asymptoties μ + make the substitution =6/(u 1) in the integral:
J. - 1+8 (u - 1)--1 H.
d82-6
1/111-1) d=2-0
(207)
1+2"
As μ - the last integral converges to standard integral:
1) d=
00
d
(2 (2-0) B)T
(208)
1+2
0
sin((2 0))))
The integration line in the last integral is any line connecting 0 to 20 in C ( (-o0, the integral
does not depend on the line thanks to the condition 1.
We prove now that (([a 2 1}) n [0 ø. Let us first show that if |u| 2 and Sp + 0, then V (() (0, +00). To this end write
(m) = ab,
(209)
-
(n- ) =
-
d82-0
(210)
11 11-1+0
1+
b (μ 1)2 J(H) 2,
(211)
where J(n) = - 1.2 is Zhukovsky's 's function.
Suppose, for definiteness, that Sy > O. Regarding note that if Sp > 0, then the imaginary part of the integrand in Eq. (210) is also positive, and so Sa >
Regarding b, recall that if Sy > 0 and |p| > 1, then 3J(()) > 0. On the other hand, if |e| 1, then (()) E [-2, 2]. Combining these observations, we see that if Sy > 0 and |e| 2 1, then either 36 > 0, or b < 0. Since Sa 0, it follows that ab (0, +oo).
We see that (n) can be real and positive only if μ E R. Clearly, (()))) > 0 if < -1, and V (H) < 0 if μ 1. It is easily checked by differentiation that (p) is monotone decreasing for
E (o0, 1], so the smallest positive value attained by V is
(--1) = 2 //
d8 do=-0 0 -1
2 2-8)
> 2.
(212)
E PROOF OF PROPOSITION 2
In terms of a, b, c. D, the components P. Q of the characteristic polynomial det(H--S) = P(u)= XQ(p) can be written
P()) = (μ 1) det(u -D),
(213)
Q(k) = - det (ar
bT
/ D = det((= - D) - D a).
(214)
(see Theorem in Yarotsky & Velikanov 2024
Accordingly,
(μ - 1)Q(H) b bT(p - D) ---1c c -
(215)
P(u)
29

---

Published as a conference paper at ICLR 2026
If D diag(d)..... dar). then
(p- - 1)Q(4)
M
UmCm
(216)
P(N)
m=1 14 dm
On the other hand, our definition of (M() implies that
(1
1)A
M
(2 (2-0)(mm1122)) 1/23/
(n) = (0- - 2)hμ Σ
(217)
+ e 1/2jA
m=
(2 0) 1/2)/ (1 e (m-1/2)0
= (0 - 2)h
e (2 (2-0) (mm (218)
1 6- 6-(m-1/2)
m=1
m=l
0)h
(2 () 1/2)/ (m--1/2))
1
(2 0)
(2- 0)h/2
# - 1 (m 1/2)h
E 0)h
(219)
By comparing this expansion with Eq.
216 we see that the values a, b, c, given in Eqs.
31
34) ensure that P/Q v((M.
F
EXPERIMENTS
The experiments in this section are performed with Corner SGD approximated in Proposition with memory size M = 5 and spacing parameter | = 5. Experiments have been performed with GPU NVIDIA GeForee RTX 4070, CPU Intel Core i5-12400F, and GB RAM; the training of all the models on GPU has taken less than half an hour.
synthetic indicator problem. Suppose that we are fitting the indicator function y(x)
1/4,9/44(2) (z) on the segment [0, using the shallow ReLU neural network in which only the out- put layer weights we are trained:
(a w)
1 N
Σ
wn
))_ (x)+ max(x, 0).
(220)
This is an exactly linear model that in the limit N + oc acquires the form
0(x) = L w(v)(x - dy = x" x'T:. w,
(221)
where x, w are understood as vectors in L 1)), and (x 14. We consider the loss L(w)
E (x" w =(z))² where (0, 1) is the uniform distribution on [].
This limiting integral problem obeys asymptotic spectral power laws IT with precisely com- putable 14 (see Appendix
(222)
The problem thus falls into the sub-phase "full acceleration" of the signal dominated phase, and we expect that it can be accelerated with corner algorithms up to 0 2.
In the experiment we set N = 10 and apply corner SGD with 0 = 1.8, see Figure The experi- mental exponent of plain SGD is close to the theoretical value Ç = 0.25. The accelerated exponent of approximate Corner SGD is slightly lower, but close to the theoretical value 0ç 1.8 -.25 0.45.
MNIST. We consider MNIST LeCun et al. 2010) digit classification performed by single- hidden-layer ReLU neural network:
H
28x28
Tr w)
(2)
Σ
w
Wann Ti ) r= -=00. ....9.
(223)
==1
1==1
A jupyter notebook with all experiments is provided in SM
30

---

Published as a conference paper at ICLR 2026
during training. [8] = 1000
Loss during training, |8|= 100
10-1
10
Loss
LOSS
10
10
Plain SGD, <.
Plain SGD, 4=
Corner SGD 8- 1.3, La
Corner SGD 6 1.3, L 1-11
Corner SGD 8 1.8. 4 0.444
Corner SGD 6 18. 4 1.82
10
10
10a
10
10a
10a
10a
10a
10:
10a
10a
10a
Iteration
Iteration
Figure 5: Training loss of neural network (223 on MNIST classification with H/ = 1000, with batch
size B = 1000 (left) or 100 (right). The full color curves show the smoothed losses.
Here, the input vector x = (Tm) ) represents a MNIST image, and the outputs y/- represent the 10 classes. We use the one-hot encoding for the targets y (x) and the quadratic pointwise loss f(x, w) 2 (x, wx) for training. The trainable weights include both first- and second-layer weights Wim. wyn
28x28
Note that the model (223) is nonlincar, but for large width H and standard independent weight initialization belongs to the approximately linear NTK regime (Jacot et al., 20188). In Velikanov & Yarotsky 2021) MNIST found to have approximate power-law spectrum with
0.25, V/ A 1.3,
(224)
putting this problem in the sub-phase III "limited by Us--initenesss" of the signal-dominate phase (see F.. Theoretically, by Theorem 4 the largest feasible acceleration in this case is 0 и. Note, however, that this theoretical prediction relied on the dimniio-d and the divergence of the series Σi= /0/v-2 The actual MNIST problem finite-dimensional, so its Us is always finite (though possibly large) and can be made 1 if [B] is large enough. This suggests that corner SGD might practically used with > v and possibly display acceleration beyond the theoretical bound max bound
the problem
Note also that with exponents (224) the signal/noise balance 2, i.c. it is not an obstacle for increasing the parameter towards 2.
++1 1+1
1000 and 100. The 0 = 1.3 version shows stable performance accelerating the plain SGD exponent Ç by factor 1.5. The 0 = 1.8 version shows lower losses, but does not significantly improve acceleration factor 1.5 at B = 1000 and is unstable [B] = 100.
In Figure 5 we test corner SGD with 0 1.3 or 1.8 on batch sizes B
In Figure we show both train and test trajectories of the loss and error rate (fraction of incorrectly classified images). The test performance is computed on the standard set of 10000 images, while the training performance is computed by averaging the training loss trajectory. We observe that, similarly to the training set performance, the test performance also improves faster with Corner SGD than with plain SGD. The instability of Corner SGD with 0 1.8 and batch size 100 observed previously on the training set is also visible on the test set.
G THE SYNTHETIC 1D EXAMPLE
Recall that in Section we consider the synthetic 1D example in which we fit the target function y(x) 1p/4./4 (x) on the segment [0, 1] with model that in the infinite-size limit has the integral form
(xz) = - w(v)(z dy x" w,
(225)
where x, w are understood as vectors in [ 1)), and x = (x - -)4. We consider the loss L(w) E. (x w =(x))),, where is the uniform distribution on [0. 1).
=
31

---

Published as a conference paper at ICLR 2026
MNIST, batchSize: 1000
MNIST, batchSize: 100
10
10
Loss
Loss
Plain SGD Train
Plain SGD Train
Test
10
Plain
Corner 1.3 Train
Corner e = Train
Corner 8 1.3 Test
Comee e-1.3 Test
Train
Comer e 8 Train
Test
100
10:
10a
10a
10*
10P
10a
10a
10a
10*
Rer stionion
Iteration
MNIST, batchSize: 1000
MNIST, batchSize: 100
10"
107
Erro rate
10-1
Error
10-1
Plain SGD Train
Plain SCD Train
SGD Test
Plain SGD Test
Corner 8- 1.3] Test
Comee 0 1.3 Test
Corner 0 1.8 Train
Corner 0 1.8 Train
Corner 8 1.8 Test
Corner 0 18 Test
10
10a
10a
10a
10a
10a
Ieeation
Iteration
Figure 6: MNIST trajectories of loss (top row) and error rate (bottom row) train set (lighter colors) and test set (darker colors). Left column: batch size 1000. Right column: batch size 100.
The asymptotic power-law structure of this problem can be derived either from general theory of singular operators and target functions, or from specific eigendecomposition available in this simple ID setting.
The eigenvalues. ator
First observe that the operator H Fxxxp[xX" in our case is the integral oper-
Hf(x) Hf(=)= 1/'' K (z, K(æ,
(v)dd, K(x, v) S (x- =1 2z.
(226)
The operator has eigenvalues (see, e.g., Section A.6 of Yarotsky 2018)) d ET where
E
π 2
nk + O k = 0,
(227)
Numerically, Eu = 1.875 so the leading eigenvalue Ao 0.0809.
In particular, the capacity condition holds with = 4.
In fact, such a power-law asymptoties a general property of integral operators with diagonal sin- gularities of a particular order (Birman & Solomjak 1970). It is casily checked that the diagonal singularity of operator (226) is of order a In dimension the exponent v has the general form = + a which evaluates to in our case d = 1.
The eigencoefficients. To establish the source condition 12). that says that for targets that are indicator function of smooth domains we have 5+a [Velikanov & Yarotsky| 202I). Alternatively, we can directly find Ç thanks to the simple structure of the problem.
invoke the general theory
32

---

Published as a conference paper at ICLR 2026
A short (though not quite rigorous) argument is to observe that the exact minimizer w. making the loss L(w) = formally has the distributional form
w. (x) ('xx 1/4) 5' (3-34 3/4)
(228)
with Dirac delta (x)). This vector w. has an infinite [[, 11) norm, in agreement with our expec- tation that
1. The eigenfunctions of the problem be explicitly found (Section A.6 of
Yarotsky 2018 )
cosh(Ex) cOs(Ex)
0% (z) = cosh((E:)) + cos(Exx) sinh(Ex) + sin (ix(Ex)
(sinh(Ea>) + sin(Exx))).
(229)
Then, formally,
e w. de, (x)
dex(x)
x Ex.
(230)
dr
x=3/4 dx 1==1/4
follows that small denoting ko (A) min(k:
Ax }}:
\x,e x
2 x Σ
(1/2 + k) 2
x kr
(231)
(S).(A)
(S).(A)
implying again (=
A rigorous proof, avoiding Dirac deltas, be given along the following lines. First note that in the setting of loss function L(w)= = 1Ex=p w w- -x)))² the vector appearing in quadratic form acquires the form q = (xxx) which in our example gives
3/4
q(x) =
(u - +yy
(232)
1/4
We get from the condition Hw. that
e q
ef W.
(233)
Av
The eigenfunctions can be written
(x) cos(Eu2)) sin(Exa) e - + (-1))t-ta(1-=)) O(e-5x),
(234)
where the last (e SSx uniform in [0, 1]. Performing integration by parts twice with vanishing boundary terms, we find that
cTa I.'d (cos(GE=)) sin(Exx) - + (-1)&e -((1)))(((1-)))) / (y (y- dydd (v=))) + O(e Ex)
Eu
S/¹ (sin(Eee)) + cos(EA2) tuo f+x + (-1)ke (-1)&e-(a(t-s))
/3// 1yda + (t)
- E
1/4
pa/a
_=² hy (-cos(Eux) + sin(E + O $t/11)
(235)
3/4
= Ex - (-n( + k))) - cos(T(}} + k))))
+ Ole O(e-Ea/4)
(236)
1/4
x ET
(237)
leading to eg w. x ET //a Eu. in agreement with Eq. 230
eg,,
H EXTENDING THE PROOF OF THEOREM TO 72 + 0
In this section we sketch (without much rigor) an argument suggesting that Theorem remains valid under assumption of SE approximation with 72 # 0 at least if the batch size |B] is large enough.
Recall that the assumption T2 = was used to write the propagators U,, V2 in the simple form g). These representations led to the representations (19 (1 of Ue, V, in terms of the contour
33

---

Published as a conference paper at ICLR 2026
map V that were instrumental in proving Theorem While not aware of a similar contour representation at T2 * O, we can expand the general T2 * propagators in terms of the spectral components of the T2 0 propagators, and in this way reduce the study of the general case to the already analyzed special case.
Specifically, let introduce the notation
Golt, = (t, -(( ((1 )5
(238)
Then formula (8) for the propagator Us can be written as
U, =
Gt, As).
(239)
In the proof of Theorem 3 it was shown that (see Eqs. 83 85
Go(l, A) (1, 2 (12
(240)
Upon substituting t\ and applying the capacity condition
this gave the leading term in
U.
U.
~
2/0 F
(1X 110)
(241)
B
T1
2 (
1/8
(242)
B
( 1-
(243)
A 1 P 0/1 1010-2
(244)
Now, the SE approximation holds with T2 + O, then the propagator formulas (8) are no longer
valid. Instead (see Yarotsky & Velikanov 2024)) the propagators can be written with the help of
the linear transition operators A, acting on [M 1) x (M + 1) matrices Z:
A = SS
X - ( =T)
(245)
B
In particular, Eqs.
238
239 get replaced by
U.
TI B
XtG(t,A)),
(246)
G(1, = TH(()) (A01))AA--00 10- (<>())")>))
(247)
Note that Eq. 238 is a special case of Fq. (247) resulting at 72 = 0 thanks to the simple factorized structure of the transformation A with vanishing second term.
Let us now write the binomial expansion of G(t, A) by choosing one of the two terms on the r.h.s. of Eq. (245) in each of the 1 iterates of A, Eq. (247). The key observation here is that each term in this binomial expansion can be written as a product of the 72 = 0 factors Go with a suitable coefficient:
T2X2 B
G(t, A) = Go(t,
Σ
(248)
m=1
x Σ
Gu(t tm, A)Gu(tm - (m- .m.1,,))..GGot22 t1, A)Go(t1, (249)
0ct St
Here, 0 < t
....
< Im < t are the iterations at which the second term in Eq. 245) was chosen.
We can now apply again approximation 240 for Go terms of Fr, and approximate summation by integration:
-2211/0
G(I,A)
22/0-2 X
13(()110++ (
+ Σ
(cx2100)
(250)
B
m=l
34

---

Published as a conference paper at ICLR 2026
where (0))(()))))))))))))))))))))))) is the (m + 1)-fold selffconvvlution of Fū:
(Ft)o(mtt) (mm))mmm))) (r)= ] Joens.....er (rm) (Tm -m--)) (m-- (7((r)) )dn ... dr (251)
The factor X1/0 in 250 results from the respective factor X in Eq. (248) the factor X2/0-2 in Eq.
240). and the integration element scaling factor X X-1/0 due to the substitution rn t,
The leading term in expansion (250) corresponds to the case 72 = O. Consider the next term, m = 1.
The respective contribution to U, is
0."h =
72 B
(
3/0 (F+)
(tx
(252)
k=1
This expression can be analyzed similarly to the leading term in Eq. (41) giving
[- (FF) 8/v 01103
0f
T1 1/v B
(253)
Note the faster decay / compared to 18/u-2 the leading term. This difference results from the different exponent 3/0 on A It also leads to the factor r rather than r² in the integral.
The coefficient in brackets in Eq. (233) is finite unless the integral diverges. To see the convergence, write
0 /0
r (F 12 (r)dr 8/v
(F (r)dr
(254)
1/ j
and use the inequality 2 0/v & (2 11 11)) + (2r1)2-00 0/00 valid since 2 - 0/v > 0:
10 2 8/u 22)))
(255)
[(2(r - ((lr-ri))** r1)))-000 + (2r1)2-00/
(r - ((
(256)
= 220-0/0 22-0/0 22-0 (r) ffo F(())dr) <0,
(257)
since (r) r 0-- as + 00 by Lemmal
Next terms in expansion (250) can be analyzed similarly, but we encounter the difficulty that, due to the associated factor Am/6 in Eq. (250). they will contain the integrals
1 ,2+m ( that diverge for sufficiently large m. For this reason, it is conve-
nient to upper bound
mm/8
max
1m-11//01/9
(258)
Then the contribution
mm(m) to U, from the term m can be upper bounded by
U (m)
T1 m Ama
(m-0)//
1/v [
(r)))) / 0/v-3
(259)
Using the incquality 2 0/v < ((m + 1) - -mm))² be bounded
0/v +...+ ((m + 1)r1)2 0/v, the integral can
0 [ 3 (())r (r) (m + 1)3-0/1
( | 220 F3())r)())d) (( (r()))))) "
co.
(260)
Summarizing, the contribution of all the terms in Up other than the leading term U (0) can be upper bounded by
I0B - 1 1S C1/10-3
(261)
with the constant
= T/O1//v (1o ,2-0/v Fa ,2-0)vE8(r)dr (r
Tz m (m- 1)/0
Amax
(m + 118-0//0/f
FE(r)dr)) ".
V
B m++1
(262)
35

---

Published as a conference paper at ICLR 2026
If
[B] > |T2/Amax 1/0 (- (r)ddr
(263)
then series (262) converges, and so 10; U3 o(U; ) as claimed.
The case of the propagators Vi can be treated similarly. Starting from 72 0, denote
Ho(t, A) (t, 1(1 0 )8 (b)1.
(264)
then by Eqs.
153
(155)
Holl, ~ ((A110)) and
00
V2
Σ
(e w.) )
(265)
The counterpart of Ho for general 72
H(t,
(266)
Expansion 228
gets replaced by
T2d2 m
At, Ho(t,
Σ
(267)
B
Golt tm A)Go(tm 2m) -Golt2 11, A)
(268)
Oct1 <... <tm <t
and expansion 250 gets replaced by
T2X B
+
Σ m=1
((F))T (E)(L))O))
(269)
The factor Am/00 can again be used to extract an negative power of / the asymptotic bounds. To avoid divergence of the integrals, we can use bound
X m/0
(m- /
(270)
max
with some sufficiently small e > 0. Arguing as before, we then find that for [B] large enough the contribution of all the terms is O (1--8(-)), i.e. asymptotically negligible compared to the leading term x 1-99
36


Reviews:
[{"review_id": "caa76sTqkD", "reviewer": "Reviewer_Fw2s", "summary": "The authors consider the problem of linear regression with random design in a (possibly infinite-dimensional) Hilbert space. They aim to estimate a vector $w^\\star$ minimizing quadratic risk. In order to do so, they apply stochastic gradient descent (SGD) with memory. It is known that in quadratic optimization problems deterministic gradient descent (GD) converges at the rate $O(t^{-\\zeta})$ while the heavy-ball method has a significantly faster rate of convergence $O(t^{-2\\zeta})$. Here $\\zeta$ reflects the signal smoothness. Unfortunately, the question of a similar acceleration for mini-batch SGD is underexplored. In [Varre, Flammarion, 2022] the authors proposed a non-stationary modification of SGD that achieves quadratic acceleration but their bounds include the condition number of the design covariance matrix. As a consequence, they are valid in finite-dimensional setups only. The present submission studies possibility of quadratic acceleration in the infinite-dimensional case. The authors show that stationary SGD algorithms with memory can be characterized by contours on a complex plane. They establish that contours with a corner at $0$ play a crucial role and call the corresponding stochastic algorithms corner gradient descent. Such algorithms can achieve the rate of convergence $O(t^{-\\theta\\zeta})$, where $1 < \\theta < 2$, provided that the external angle at $0$ is equal to $\\theta \\pi$. The only drawback of corner algorithms is that they require infinite memory. Fortunately, the authors suggest an approach how to approximate them with finite-memory SGD.", "questions": "Can you discuss how severe the requirement $\\tau_2 = 0$?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper studies an interesting and important question, which yet has not got sufficient attention in the literature. The authors suggest a non-trivial geometric interpretation of SGD with memory, which helps to understand the possibility of acceleration compared to the standard rate of convergence $O(t^{-\\zeta})$. In the signal-dominating regime, they provide a very demonstrative figure showing when one can achieve the rate $O(t^{-\\zeta})$ and when only intermediate rates are possible. I am not an expert in convex optimization but I suppose that this contribution will be interesting to the community.", "weaknesses": "1. The authors refer to [Velikanov et al., 2023] and [Yarotsky, Velikanov, 2024] for extensive discussion of the SE approximation. I took a quick look at these papers and found no examples with $\\tau_2 = 0$ in eq. (6). At the same time the key theoretical results presented in the main text are obtained under the condition $\\tau_2 = 0$. I think that restrictiveness of this assumption must be discussed.\n\n2. Rigorous theoretical guarantees are obtained in the case $\\tau_2 = 0$ only. When $\\tau_2 \\neq 0$, the authors provide only sketch of the proof.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1iX8p40D1y", "reviewer": "Reviewer_yRoH", "summary": "The paper introduces generalized stationary SGD with infinite memory, a theoretical framework for accelerating stochastic gradient descent on infinite-dimensional quadratic problems with power-law spectral conditions. The authors build upon classical results showing that deterministic Gradient Descent achieves a convergence rate of $O(t^{-\\zeta})$, and that Heavy Ball with a non-stationary Jacobi schedule can accelerate this rate to $O(t^{-2\\zeta})$. However, in stochastic mini-batch settings, no such results are known.\nThe contribution of the paper is a geometric representation of stationary (S)GD algorithms as contours in the complex plane. The authors show that contours with a corner of external angle $\\theta\\pi$ accelerate the plain GD rate from $O(t^{-\\zeta})$ to $O(t^{-\\theta\\zeta})$. In deterministic settings, increasing $\\theta$ up to $2$ gives rates arbitrarily close to $O(t^{-2\\zeta})$. In stochastic setting, increasing $\\theta$ also amplifies noise, leading to a trade-off, with optimal rate $\\theta_{\\max} = \\min(2, \\nu, \\tfrac{2}{\\zeta + 1/\\nu})$, where $\\nu$ and $\\zeta$ are the spectral exponents coming from the capacity and source conditions.\n\nTo make Corner SGD practical, the paper proposes finite-memory approximations of the infinite-memory algorithms using rational approximations of the power function $z^\\theta$. Empirical experiments on a synthetic problem and MNIST show accelerated convergence consistent with the theoretical predictions.", "questions": "1. Can the authors comment on how the framework might extend beyond quadratic case?\n\n2. How should one choose the memory size $M$ in practice?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The contour-based geometric representation of stationary (S)GD algorithms is new and interesting, connecting optimization dynamics with complex analysis.\n\n2. The paper unifies acceleration, stability, and noise amplification under a single geometric-spectral framework.\n\n3. Rigorous theoretical results.\n\n4. The finite-memory approximation gives a practical way to approximate the idealized algorithms.", "weaknesses": "1. The paper is technically rigorous and presents a new geometric perspective on SGD acceleration, but several assumptions (only quadratic case, SE, infinite-dimensional limit) narrow the practical relevance. It is unclear if the main conclusions hold beyond this idealized regime.\n\n2. Limited experimental validation (one synthetic experiment with MNIST). However, I acknowledge that the paper is of theoretical nature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxSXHb20mn", "reviewer": "Reviewer_C97c", "summary": "The paper studies the SGD behaviour for quadratic problems. The authors analyze scaling laws and their corollaries for the gradient descent with memory. They propose a connection between SGD-type optimization with memory and holomorphic maps. After deriving the analysis for optimization schemes with infinite memory, the authors propose a finite-memory relaxation, that achieve better results on MNIST dataset.", "questions": "1)For finite-memory approximations we need to settle the parameter $\\theta$, which affects iterate dynamics as well as convergence rates. While Theorem 4 claims, that there exist an upper bound $\\theta_{max}$, what will happen, if we take $\\theta > \\theta_{max}$ for finite approximation?\n\n2)Following the previous question, how can one estimate $\\theta$ for various datasets?\n\n3)What are the limitations on the batch size $B$? The authors demand $U_\\Sigma < 1$, which depends on the batch size.\n\n4)Why for finite-memory approximations we choose $\\Psi$ specifically as in equation 28? The properties of the optimization scheme depends on $\\Psi$ properties, as declared in Theorem 3.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1)Considering SGD with the instruments of complex analysis is nonstandard and provide new views on well-established theory.\n\n2)Derived theory correlates with the recent works, that analyze the spectral power laws.\n\n3)Analysis of the optimization schemes with infinite memory. This approach is not well-developed and might lead to better finite approximations.", "weaknesses": "1)The writing is complicated. Although this might be fixed with an extended number of pages, the authors should consider reorganizing the paper.\n\n2)The errors of finite-memory approximations are not given.\n\n3)Only quadratic losses are considered.\n\n4)The paper lacks comparison of finite-memory approximations of corner SGD with varying $M$.\n\n5)Minor typos ($\\mu^{t-1}$ instead of $\\mu^{k-1}$ on line 293, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xb9sWisQAS", "reviewer": "Reviewer_g74e", "summary": "The paper proposed a novel approach to the acceleration of (S)GD in ill-conditioned quadratic problems through a geometric interpretation of stationary algorithm. The authors identify a view of generalized (S)GD as contours leveraging the characteristic polynomials and loss expansion. A special class of those contours with an external angle is proved to accelerate (S)GD and the corresponding effects are analyzed. The paper further demonstrates that the ideal corner algorithm can be approximated with finite-memory algorithms, and provide experiments on a synthetical problem and MNIST.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(i) The paper is novel. It introduces a new geometric framework for understanding stationary (S)GD algorithms through complex-plane contours, offering a conceptual unification of gradient descent, momentum. The idea of characterizing acceleration via cornered contours is novel and mathematically elegant.\n\n(ii) The analysis is rigorous, theorems are clearly stated and proved, and the trade-off between acceleration and noise amplification is discussed and quantified. \n\n(iii) The authors offer practical implementation of the algorithm and tested on both synthetical experiments and MNIST dataset, which further validates the effectiveness.", "weaknesses": "My primary concern is the experiments part, while being theoretically elegant, the experiments primarily focuses on the 1d synthetic regression task and a small scale MNIST example. The results are consistent with theory but do not establish a broader applicability. Are there any specific reasons that prevents validation on a task of a larger scale? Other than that, I believe the paper is generally well-written, but some parts of it, especially the background would benefits from additional explanations and intuitions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)