Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
Authors: Jihwan Kim
Dogyoon Song
Chulhee Yun
Venue: ICLR 2026 Poster (2026)
Keywords: scaling laws
signSGD
SGD
compute-optimal curves
power-law random feature
warmup-stable-decay schedule
Primary area: learning theory
TL;DR: SignSGD sharpens compute-optimal scaling in PLRF for noise bottleneck regime.

Abstract: We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the population risk of a linear model trained with one-pass signSGD on Gaussian-sketched features. We express the risk as a function of model size, training steps, learning rate, and the feature and target decay parameters. Comparing against the SGD risk analyzed by Paquette et al. (2024), we identify a drift-normalization effect and a noise-reshaping effect unique to signSGD. We then obtain compute-optimal scaling laws under the optimal choice of learning rate. Our analysis shows that the noise-reshaping effect can make the compute-optimal slope of signSGD steeper than that of SGD in regimes where noise is dominant. Finally, we observe that the widely used warmup-stable-decay (WSD) schedule further reduces the noise term and sharpens the compute-optimal slope, when feature decay is fast but target decay is slow.

Published as a conference paper at ICLR 2026
SCALING LAWS OF SIGNSGD IN LINEAR REGRESSION: WHEN DOES IT OUTPERFORM SGD?
Jihwan Kim
Dogyoon Song
Seoul National University & KAIST InnoCORE LLM aqua 6890 snu. ww..ua.............. ac. kr
University of California, Davis dgsong@ucdavi edu
Chulhee Yun KAIST
chulhee, boulhee.yun@kiist.aee
ABSTRACT
We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the popula- tion risk of a linear model trained with one-pass signSGD on Gaussian-sketched features. We express the risk function of model size, training steps. learning rate, and the feature and target decay parameters. Comparing against the SGD risk analyzed by Paquette et al. (2224) we identify driff-normalization effect and a noise-reshaping effect unique to signSGD. We then obtain compute-optimal scal- ing laws under the optimal choice of learning rate. Our analysis shows that the noise-reshaping effect can make the compute-optimal slope of signSGD steeper than that of SGD in regimes where noise is dominant. Finally, we observe that the widely used warmup-stablee -tt-sttdeeeeayyyy (WSD) schedule further reduces the noise term and sharpens the compute-opine optimal slope, when feature decay is fast but target decay is slow.
INTRODUCTION
In large-scale language model training, neural scaling laws are well-documented empirical regu- larity: performance tends to improve predictably as data, parameters, and compute increase. |Kaplan jet al. (2020) observe that the language model cross-entropy loss scales as a power-law of model size M and number of steps N in terms of the risk formula R(M, N) t M- + N for some T1, 72 > of Also, they observe that loss scales as the power of training compute, under optimal allocation of compute between model size and number of steps.
A growing body of theory has sought to explain this phenomenon, most prominently by analyz- ing the stochastic gradient descent (SGD) optimizer under the power-law random features (PLRF) model (Paquette et al., 2024, Lin et al., 2024 2025). Yet, in practice, SGD is not the optimizer that powers today's
of-iee--f-hhhrrttt LLMs. Instead, their training is dominated by Adam (Kingma & Ba 2015) and its variants. While Adam is considerably more difficult to analyze theoretically. it is often approximated in theory by the simpler signSGD (Bernstein et al., 2018), which captures Adam's coordinate-wii adaptivity. This gap between practice and theory motivates a natural question: how do scaling laws change when we replace SGD with signSGD? Addressing this question can help align theory with optimizer choices used in practice, and clarify how adaptive updates could reshape compute-optimal scaling regimes in the PLRF setting.
1.1 OUR CONTRIBUTION
We study the scaling law of signSGD in the power-law random features (PLRF) model, and our contributions are as follows.
We use x to denote equality up multiplicative constant, i.c., f(x) zo g(z) means cig(z) S f(x) s (x) for constants C1, 62

---

Published as a conference paper at ICLR 2026
signSGD theory 0.485
10a
Constant theory = 0.333
SGD theory 0.418
WSD theory 0.253
10
Final Loss
Final Loss
10-2
10-2
signSGD trajectories
Constant trajectories
SGD trajectories
WSD trajectories
signSGD: slope=0.446
Constant: slope-0.330
SGD: 3.ope=0.378
10
WSD: slope=0.376
10a
10
10
10a
10'
10a
102
10th
10a
10*
10a
10"
10"
10"
10"
20th
10
FLOPs (6-M N)
FLOPs (6 M N)
Figure 1: Left: SGD vs. signSGD; Right: signSGD with constant vs. warmup-stable-deaay schedules. Colored lines represent the training trajectories of each algorithm, and black lines de- note the compute-optimal curves. The upper right legend in each panel shows the theoretical value of the compute-optimgg slope. SignSGD achieves steeper compute-optimal slope than SGD (left panel), and warmup-stable-den scheduling sharpens the compute-optimal slope relative to a con- stant schedule (right panel), for some parameter configurations. See Appendix for parameters used in the experiment.
We derive a scaling law of signSGD with constant learning rates involving three variables (model size M, training steps N, learning rate %0) and two PLRF model parameters (feature decay a, target decay 9); see (12). By comparing with the SGD scaling laws of Paquettt et al. (2024) and Lin et al. 2024), we observe two effects of signSGD: a driftnorrmaizztioon effect and a noise- reshaping effect, inside the scaling law (see Section 4.1).
2. Under the fixed compute budget, we balance model size M and training steps N, and optimize over learning rate %o. This allows us to characterize the compute-optimal loss decay rate and optimal model size with respect the compute budget (see Table 1.. Comparing against the compute-optimal scaling laws of SGD from Paquette et al. (2024) across regimes of the (a, 9)- parameter plane, find that signSGD can achieve better exponents the SGD noise bottleneck regimes, due to the noise-reshaping effect (see (see Figure I).
3. We show that learning rate scheduling can further reduce the stochastic noise of signSGD. We analyze a warmup-stable-decay (WSD) schedule (Hu et al., 2022) widely used in large language model training. By maintaining drift velocity during the stable interval and reducing stochastic noise by the polynomially decaying interval, this schedule increases the compute-optimal slope in the PLRF setting for large a and small (3 (Sectont 4.3 and Figure Tp.
4. We empirically validate our theory; see Figure 1 and Appendix C for details.
1.2 RELATED WORK
Here we discuss directly relevant results; additional related work is deferred to Appendix B
Empirical Scaling Laws. Modern empirical work shows that performance improves with scale across data, parameters, and compute, following power laws across many domains (Hestness et al. 2017). In language modeling, Kaplan et al. (2020) document plwer-law loss trends over multiple ders of magnitude and simple budgeting rules linking model size, data, and compute./Henighan et al. 2020) extend these curves images, video, and multimodal settings. Building this, Hoffmann et al. (2022) argue that many LMS were under-trained on tokens and propose data-optimal scaling that substantially improves accuracy at fixed compute. Tissue al. (025) investigate the empirical scaling law with learning rate annealing.
Scaling Law Theory. Our work starts from the SGD scaling law in the PLRF model in Paquette jet al. (2024) and [Lin et al. (2024). In particular, Paquette et al. (2024) derive a scaling-law formula for one-pass SGD, where M, N, and 7/ denote the model size, number of training steps, and learning
2

---

Published as a conference paper at ICLR 2026
rate, respectively, and a and are the feature- and target-decay parameters.
20+288-
20-1 2a
40
R(M,N. 7o)
M t 1- 1-28) + (Nra)
2a
M (N
(N yo) 2a
A((M)
DDGD (N,70)
-DSGD (M,N
The A(M) term corresponds the approximation error, ie.. the loss as N co. Paquette et al. 2024) explain that al D (N, (1o) represents the aligned feature loss, as it coincides with the loss for a diagonal sketch matrix S (see Section 2.2 for a formal description). They also explain that DasD (M, N, 70) corresponds to the distorted feature loss, arising from projection with a random matrix S, and it decays more slowly than the aligned feature loss. Finally, .VSCD (N, 70) captures the SGD noise, stemming from the quadratic term in the Taylor expansion of the SGD update.
Several subsequent papers extend this baseline along two axes: (i) optimizer changes and (ii) model/training-pronee
variations. On the optimizer side, Ferbach et al. (2025) investigate dimension-adapted Nesterov acceleration in the PLRF model and argue that it gives better scaling law for 2a > 1 regime. Kunstner Bach (2025) compare the gradient descent and sign descent scaling law in the linear bigram model. Comparison with their work is in Appendix B. Lin et al. (2025) cover the multi-pass SGD scaling law and identify the effect of data reuse for the scaling law. Discussion of the model/training -erraiiiinnnnooooooooooooo changes is deferred to Appendix B
Scaling Behavior of Linear Models in the Context of Kernel Methods. The power-law settings for data and targets adopted in our work are deeply rooted in the literature on kernel methods and their fintte-width approximations. In this context, the power-law decays of the covariance spectrum and target coefficients are analogous to the classical capacity and source conditions, respectively. These spectral assumptions have been extensively investigated in kernel ridge regression (Capon- netto & De Vito 2007 Cui et al. 2021) and random-feature ridge regression (Rudi Rosasco, 2017 Bach 2017 Defilippis et al. 2024). Furthermore, similar conditions are fundamental to prior
theoretical works on SGD that are closely related to our setting, including studies on one-pass SGD)
Yao al. 2007 Ying & Pontil 2008 Carratino al.. 2018 Berthier et al., 2020) and multi-pass
SGD Pillluud Vivien et al. 2018 Detailed comparison with these works is in Appendix B.3
SignSGD Dynamics. Bernstein et al. (2018) give the non-convex convergence rate of signSGD. Xiao et al. (2025) derive the SDE and ODE for one-pass signSGD in the linear regression setting with squared loss. The ODE we derive matches theirs in final form; however, we obtain it in an alternative route that does not require spectral lower bound the covariance matrix that they imposed. Detailed comparison with Xiao et (2025) is in Appendix Compagnoni et al. (2025) derive SDEs for adaptive methods, including signSGD.
PROBLEM SETUP
2.1 NOTATION
We use bold lowercase letters (e.g.. u) to denote vectors and bold uppercase letters (c.g., A) to denote matrices. For vectors u and v, we denote the outer product by u @ v := uv (A) denotes the i-th eigenvalue of the matrix A. For positive-valued functions f(x) and g(x), f(x) S g(z) if there exists C > such that f(x) < Cg(r)) for sufficiently large v, and we use f(x) g ( if there exist c, C > 0 such that cg(x) S f(x) < Cg(x) for sufficiently large x.
2.2 MODEL.
We consider the power-law random features (PLRF) model, parameterized by O T Given feature-label pair (x, y) E Rd x R, the parameter plays the role of a linear regression coefficient vector on the sketched features Sa (for some S E R M xd), and the population risk function is
L(
Ex [(Sxx 0) y)
The data are generated as follows: the feature vector x E Rd is drawn from M/ H) where H has eigenvalues 1 -2a 22 and the label is y = (ar, w²)) with (v, w²) i-8 where v, is an
3

---

Published as a conference paper at ICLR 2026
eigenvector of H corresponding eigenvalue i i- for i .,..., d; call and / feature-decay and target-decay parameters, respectively. The sketch matrix S E IR Mxd is random matrix that has i.idd entries (0, 1/M). is drawn once and then held fixed throughout training, refer to M (with M & d) as the model size. Under these model assumptions,
L(0) / (S w²) ==
We assume d 2 roM for some ro > and let d/M E Iru. ool d, M - when 2a: 1, and d/M - r E [ro, 00) when 2a < The projected optimal parameter is
0- (SH
(2)
Define wi '² - 0 so that w" 5 Tm and SHw_ 0. The loss decomposes
(( = H 5 022 + || 1/2 w1l?
where the second term represents the approximation error.
We estimate the minimizer of the population risk via empirical risk minimization using signSGD. At step k, we draw a fresh sample (æt,, 9k) from our data model and form the stochastic gradient
SignSGD.
gh = 2((S Ox) - yk) Sra.
The signSGD update rule is
0k+1 O2 YA (ggn(g))
0k 7k sign ((Sxg, 0k) yk) sign(Sxx)).
2.3 REPRESENTATION OF THE RESULT
Let R(M, N, 7 denote the loss L(Ox) under learning rate tr and fixed model size M. We define the computational budget in terms of FLOPS as MN, and consider the optimal model size M under fixed j, and optimal scaling of learning rate in the form na (M**)=' For SGD, Paquette ct al. (202)) derive cumpute-optimal scaling laws of the following form:
M f R (M', (f =F"
Our objective is to derive analogous formulas for signSGD, namely, R(M, No) and R(M*, M= -,), and compare them with the corresponding results for SGD.
ANALYZING THE SIGNSGD
In this section, we formulate the implicit integral equation for signSGD. We define
K = SHST K= diag( (a) -)1/////-KKKK K, Ko arcsin diag(()-1/3 K diag(K)-1/2). (4) where aresin is applied entry-wise; we use these matrices and notation throughout the paper. We decompose the loss via
(N( (ON 0²) ((K, @ w.. -0))
where u, wU are the right/left eigenvectors of K corresponding to the ith eigenvalue K)). This modal decomposition matches that of Xiao al. 20255) For brevity we write L(N) == L(Ox).
M
L(N)= = Σ rN(N) + 1/ 1|a.
(5)
In Appendix E.
we derive the one-step update formula for signSGD on a quadratic objective, using second-order Taylor expansion and Gaun-Gan identities. Applying this r; yields
4
2 (((k) + w K, Ku.
2
Elr,(k + - [Fx]
d K
(6)
/ L(k)
quadratic noise
drift
floating point operations per second
4

---

Published as a conference paper at ICLR 2026
1. Drift. The first term in (6) yields systematic decrease of mode i: it is proportional to the curvature A, K) and the learning rate %k. while the factor 1/v [(k) ceff-normalizes the step. Note that the directions corresponding to larger eigenvalues contract faster.
2. Quadratic noise. The second term in (6) is ( variance injection shaped by curva- ture and the sign-noise covariance. It is independent of r:(k) and may set noise floor of 7y, unless 7 decays.
Overall, one-step progress reflects a balance between drift and quadratic noise: when ry (a()) is large, the drift decreases rk);;; near the optimum, quadratic noise can dominate and cause ry(k) plateau.
Converting the one-step update formula to the continuous-timg time ODE, we obtain
dr; 471/100 d (K) ra(() dt T L(t)
27//70 un K K ui
To
=:)
-0 (1)
Compared to SGD, the drift is self-normalized by 1/ L(t) and the quadratic noise term does nor carry the extra L(t) factor present in SGD. So, for the constant learning rate, the quadratic noise does not decrease over time. The variation-of-constie
formula gives the implicit integral representation
(N) = (0) exp
T garift
(u) du
+
exp
qdrin(u (u) du x noole (2) dz. (8)
Summing modes, define
N Tdtrit du
I dr (N) Σ
ro exp
M
N
I, poise (N)
exp
harin (w) du
nuaise (z) dz.
(10)
Exact formulation of T, [.drft(N) (N) and n,noise (N) can be found in 27 of Appendix R.2
Then by
5
our
risk is decomposed as
L(N)= I.drff( + 1, (N)
H w
(11)
approx
MAIN RESULTS
4.1 LOSS FORMULA FOR CONSTANT LEARNING RATE
We now analyze (11) to express it in the form of R(M, 7o), the loss after N steps with constant learning rate To and model size M; time-varying schedules discussed later.
For L drit (N). we use a deterministic approximation (Appendix E.2.2 2024). and obtain the asymptotic self-consistent equation: with r FM = Mminn(a,,. 70-
similar to Paquette et al.
2x
2a
dnin (N) T /// d (n)-1/2 du
2a
[ I. r,driit T.drif(uu (u)-1/2 du
2a
+ M-
Solving this yields signSGD counterparts of the aligned- and distorted- feature loss terms in denoted by Deiiu NN, 7o) and Deigu (M, No); see (12) below for their precise forms.
For [.pois((N) and approximation term, we calculate the limit loss T.xo and get
L max { 1o m22mm))),2a)) |
Lastly approximation emror result from Paquette al. 20244 Lin et al. 2024h)
|/2 M -2 m,
228)
We treat ri, and the learning-rate "/ as continuous extensions, 7r/1o well-deffned for any

---

Published as a conference paper at ICLR 2026
Combining two parts yields a proxy for the loss formula, and we prove that it satisfies the implicit integral equation (II) Appendix E.3.4 and E.4.4 Finally, we get the following four-term scaling law formula for one- pass signSGD the regime + 0.5
0.5:
2(2a+29-1)
R(M, No) ~ M max(,,0011122)) + (Mminfa..... No0)
20+1
A(MM
Dilon (M,N,1o)
(12)
6a-1
2(2a-1) 1)
+ M 20+1 (Na))
20+1
M 22-min(1.20)
D:D (M.)
=Wiiiun (M,70)
Interpretation. The term A(M) is the approximation error. The terms Drigu (M, NN,%) %0) and Deig (M, No),), 70) arise from the drift's s exponential damping exp{- - fo² Terin (u) du and cor- respond the aligned and distorted feature losses of SGD scaling law Paquette al. (2024). The term Wasgu M, 70)) captures the quadratic noise from the one-step Taylor expansion, specific to one-pass signSGD.
Comparison. We compare our signSGD scaling law formula with the SGD formula ( of Paque- Ite al. (2024). Since the approximation error is ceieiizze-indeeendennt the term A(M) remains un- changed. For the N-exponent in Dai and Dais. when the absolute value of the exponent is x for SGD, then it changes 2 x in signSGD, which is strictly larger than xr Therefore, D (M, N, 70) and Deigu (M, N, %o) decrease faster in the number of steps under signSGD. By contrast, the signSGD noise term A /eign (M,7u) 77) does not decay with N, whereas the SGD noise A/SDD (N,")) does 5)
We discuss the underlying mechanism that modifies the drift terms Date Ddis, and the noise term N.
Drift terms (Drift-normalization effect): In signSGD, the drift in (6) is
A((K), whereas 47h
TVL(K)
for SGD it is 2tx A₂(K) see (4p for the definition of K and K. The diagonal preconditioning em- bedded K contributes an extra factor m 1/2), since the scale of the matrix diag( K) K)- which is multiplied in K, is M miin(a.1/2). The normalization by vL(k) replaces the effective flow time N% with %o fo L(u) 11/2 du, which accelerates progress in training whenever L(u) X 1. Thus, in the aligned/distorted drift terms, Noo is replaced by M Mmin(a.1/2) Jo L(u)-1/2 du. It leads to the self-consistent equation, which does not occur in SGD, and the solution of the self-consistent equation includes powers of Mmiin(a11/2) Noo- The absolute value of the exponent increases compared SGD due the acceleration in the regime I(u)) S
Noise term NNoise-rehaaping effect): The signSGD noise in is 272 w. K,Ku,, while for
SGD it is Th (v. Kv;) v.) I(k) with v; eigenvector of K. The normalization removes the mul- tiplicative [.(k:) in signSGD, eliminating the Volterra structure present in PPqquette et al. (2024). This difference crucial: the lack of Lkk) in the quadratic term ultimately yields a noise term that 40-1
factor which appears in the does not decay in N. In the final formula, it removes the (N7o) 2a SGD noise term, and therefore the noise term of signSGD increases the learning rate ?o grows for all (o. .3. In contrast, when the learning rate "ro grows, the noise term of SGD decreases for 0.5 and increases for < 0.5. Meanwhile, additional M-dependence arises from working in the K Keeigenbasis (rather than K K-eigenbasis) due to the sign operation.
4.2 COMPUTE-OPTIMAL RESULT UNDER OPTIMAL CONSTANT LEARNING RATE
In the constant learning-rate schedule, we allow To to scale with the model size via To = - The hyperparameter directly influences the compute-optinet scaling law
2(2n 22--)
'oor the case 8 > + 0.5, Deien (M,, N, 7o) takes form of (1 xM mmin(a.0.5) Nryo) 2a-28+1 See
Appendix E.5 for more details.
Decay with respect to depends on the choice of "Yo: the subsequent sections, we set yo as M- 6One may wonder why we do not parameterize by N. Setting no M- e is without loss of generality, since in the compute-optimal case both M and N are expressed as powers of the total compute

---

Published as a conference paper at ICLR 2026
1.0
Aa
Ba
la
Bb
8
8
Ab
Ac
lb
IV
Ad
0.2
Ic
-3.4
-32
0.0
02
0.6 0.6
3.8
1.0
1/2
1.4
0.0
-52
0.0
02
0.4
0.6
0.8
1,0
12
B
Figure 2: Left: Phase plane for signSGD; Right: Phase plane for SGD. The white region indi- cates parameter values with no power-law scaling. The dark blue area represents the region where warmup-stable-decay scheduling (Section 4.3) yields a better compute-optimal exponent.
Following Paquette et al. (2024), we distinguish the maximal and optimal learning rates for SGD. The maximal rate is the largest step that yields stable (non-exploding) recursion; for signSGD, it leads to zero compute-optimal slope (see Appendix F.I. We therefore focus the optimal learning rate To which maximizes the decay exponent 17 in
R(M* f/M", 7ō) F
where M denotes the model size minimizing R(-) fixed compute budget f.
To characterize the compute-optimet scaling, set "Yu M M = f", and N = f0- (with E [0, I)), and solve
(e",x") arg min R R,M, N,, 70) arg min R(P f1- f-))
(13)
and %o = (M*)-=" and at the optimum N* = f-==" Then M = pr" No
R(M* f/M" 70) マ f 7(a,3)
for some r(a, B) O, which we refer to as the compute-optimal slope.
In problem 13). each of the four terms in 12) scales as f-(((e,)), so minimizing R is equivalent to maximizing min((1, f2, &3, f4). The optimal value (c", is obtained by balancing three active exponents. The resulting formulas and dominant and balancing terms are summarized Table [ see Appendix F22 for details.
We follow Paquette et al. 2024) in defining phases by dominant terms; to avoid confusion with their SGD phases, we label our SignSGD phases by uppercase letters. Accordingly, any reference to Phase I-IV hereafter refers exclusively to the SGD phases of Paquette et al. (2024). For signSGD, the phase plane is simpler: when > 0.5 and / > 0.5 (Phase B) all four terms are dominant; oth- erwise (Phase A) the dominant terms A(M). Deiga (M,N, N, 7o), and Asigo (M, 7o). We declare subphases whenever the formula of at least one % M -- AM*, or R(M* f/M* v6) changes. These changes occur across the boundaries subphases total (Phase A split into four, Phase into two). We provide formula of approxima- tion, drift, and noise term for each subphase in Table 2 For context, PPaquette cantext, et al. (2024) partition the (a,, 3) -lann into four phases with six subphases for optimal learning rate (and seven for the maximal learning rate).
0.5, f 0.5, and B a 0.5, yielding six
Remark 1 (Dominant vs. balancing terms). Dominant terms refer to those that dominate the risk for some (No. M, N).. Balancing terms are the ones that tie (hence "balancing") at the compute-optimal choice (76 MM N and therefore determine the slope; they form subset of the dominant terms.
Comparison of Compute-optimal Results. For the intersection of Phase Aa, Ab, Ac. Ba and Phase II, the compute-optimal slope n10, ()) and optimal model size M the same for signSGI)

---

Published as a conference paper at ICLR 2026
Table 1: Dominant and balancing terms, optimal learning rate, compute-optimal model size, and risk across different (a, (3) phases. Refer to (1) for the definitions of the terms A, Dat.Ddis, N. See Figures
12 in the Appendix for empirical validation of the theoretical exponents.
Term structure
Compute-optimal
Phase
Dominant terms
Balancing terms
M
R (M*, M
Aa M (a+5))
2
2n-28-1
Ab M
28+1
f
2n+28-1
Phase
A, N
A, N
Ac M-
Ad M-1
f=-6
20+45-1
20--2-1
Ba M
Phase A, Dai, N
Dai.Dais,
Bai11
Bb
M
40
f
4u
and SGD. In contrast, for the area of Phase III. IV excluding the case 0.25 < < 1/3, 3 (1 (1 (2(1 - -3a)) (See Figure 4 in the Appendix for the visualization of this area), the compute-optimal slope 17(0, (3) for signSGD is steeper than that for SGD, and the optimal model size is bigger in signSGD. We refer to this region as the Area III-IV sub- Finally, for the optimal learning rate % MM=e,, the exponent e" is always bigger than SGD in signSGD, which means signSGD always has smaller optimal learning rate.
4.3 EFFECT OF WARMUP-STABLE-DEES SCHEDULING
We next study the widely used warmup-stables stable--aale schedule (Hu et al. (024)), which reduces late- stage noise via the decay interval while maintaining the drift rate over the stable interval.
For the warmup-stable-decay schedule, we set the learning rate to 7 = (k)( with
k/(wN),
t & wN N S k S pN,
s(k)
1.
(14)
(1 + -K PNN))
>> pN,
where w, P, E (0, 1) and 7 > O. In other words, the learning rate increases linearly for the first w steps, stays constant for the next (p - W)N steps, and finally decays as a polynomial of exponent c additionally assume p/2.
for the remaining (1 p)N steps. Throughout,
In Phase Aa, the f-scheduled noise bound can improve over constant LR:
moise (N) I NOM + 70" M + N (1-ej(1 )
Combining this with the drift and approximation terms, and then optimizing over of 70 M the decay parameter c, and the model size M, yields the f-scheduled risk bound
2(4a- 12224)2811) +8ad4#a0+2a-11
RM "//////MMA/// (M*)-=*) E
(15)
The absolute value of the exponent in 15) exceeds the compute-optimal slope under constant learn- 20-1 25 Thus, warmup-stable-deca. scheduling yields
ing rate when a > 0.5 and 0.5 < B < strictly larger compute-optimal slope in the upper left region of Phase Aa (marked with dark blue in Figure 2}. We will refer to this region as Area Aa* throughout the paper.
Scheduling does not improve the SGD compute-optimal exponent in Phases I-Il (see Appendix G.5 Thus, with scheduling, signSGD achieves a larger compute-optimal exponent compared to SGD in Area Aa
Thee loss bound 1.5) also holds for stable-decay scheduling without warmup stage, as well for cosine linear scheduling. Refer to Appendices [
Whethee scheduling benefits other regions of signSGD other phases of SGD remains open, since for both methods the scheduled noise upper and lower bounds do match tightly, even up constant factors.
8

---

Published as a conference paper at ICLR 2026
5 DISCUSSION: WHERE AND WHY SIGNSGD PROVIDES BENEFITS?
With a constant learning rate " = M signSGD yields improvements over SGD in Area IIIIV sub- Under sarmup-stabee-decay scheduling, we find signSGI> also provides benefits in Area Aa".
Mechanisms. These gains can be explained by noise- mooise-rsshppng,, together with triff-normalization. In Paquette et al.] (2024), Phases III-IV are the SGD botsebootteeeck regimes. By contrast, noise-reshaping in signSGD can alleviate this bottleneck with a suitable learning-rate choice, yield- ing improved compute-optimal slopes.
Role of Learning-rate Scaling. The signSGD noise term with constant LR N /siin (M, 70)
In 22-min(1,222) whereas for SGD it is N/ (N, )o) (No (4o-)))(42)))) If %o 1, A N (M, 70) is much larger than A (N, 70). making the compute-optimal slope asymptotically zero. Hence, we set " M and optimize to balance terms and obtain steep compute-optimal curve: decreasing To lowers (Mi, (M,70) 70) while increasing the drift terms D (M, NN 70) and Dais NM,,, 70), and the optimal strikes the balance.
Why Gains Arise in Area III-IV sub- For SGD, the shape of A 70) makes it dominate DSGD 1O) the compute-optimal point in Phases III-IV. It is because the absolute value of ex- ponent in A/SD (N, 70) - (o(NNo) ---1 is smaller than that of D&GD (N, To) = (Nvo) 2x 20
in Area III-VV sub- For signSGD, noise-reshaping alters A (M, 70) so it can balance against Daig (M, N, no). Note that the noise term takes completely different form: (siin(,, (o) =
0 2 therefore dominance against the aligned drift term disappears. On the other hand, drift-normalization steepens the decay of Drign (M, .N, 70) by increasing the absolute value of the exponent with respect to N. This creates room for balance in which both terms are smaller than the SGD noise (N (t) optimum, explaining the improvements in Area III-IV sub. For example, in the intersection between Phase Ba and Phase III, balancing Aaiign (M,, (o) and D (M, N,, no)
leads to f
2a+28-1
20++2
whereas A/SGD (N, no) takes bigger value f 4m-1
Why staleeayt Scheduling Helps. For a learning-rate schedule Thk = =o)((k with general f. the onlyt-only self-consistent solution in Phase Aa takes the form
12 70 F(N)) 2(20=28-11 2011
F(N)= = 1.² f(w) du.
(2
where
This can be viewed as siign (M, N₁ 70) with N replaced by F(N). This aligns with empirical obser- vations that a loss term can decay polynomially with the area under the learning-rate curve (Tissue al. 2025).
In contrast, the noise term depends most heavily on the learning rate near the end of training, since earlier noise can be damped by later drift; see (8).
preserves the total area F(N) asymptotically while shrinking the late-stage learning rate, thereby reducing noise without sacrificing drift. As a result, warmup-stable-decay scheduling yields a larger compute-optimal slope in Area Aa" (upper-left Phase Aa; see Section for for intuition). More broadly, conjecture that appropriate scheduling can further reduce the signSGD noise term, enabling improvements beyond Area III-IV sub and Area Aa".
5.1 HYPOTHESTS FOR THE POSTTION OF THE BENEFICIAL. AREA
Here, we hypothesize why the areas with improved scaling law lie near the left edge (small () and the right side (8 > a) of the phase plane.
Heuristic Criterion. Let "target decay" denote the coordinate- wise decay of 0* in (2). and let "stochastic-gradient decay" denote that of the stochastic gradient in (3). SignSGD is advantageous when the target decays more slowly than the stochastic gradient. Under SGD, coordinates with smaller gradients take smaller updates; if the target does not decay much, those coordinates still require learning targets of comparable magnitude, so more iterations are needed an inefficiency that signSGD mitigates by normalizing per-coordinate updates via the sign operation.

---

Published as a conference paper at ICLR 2026
When Does This Occur? Observations and Conjecture Let SHS UAUT be the eigende- composition. Then the expected stochastic-gradee gradient expressed in the U U-eigenbasis has ith coordinate magnitude that decays i 20, See Appendix [ for details of analysis.
Next, wc examine how the target decays in the basis of the columns of U. For that, we have to consider UT . EsS 5] I, we decompose
S = I E, E: S==
so that E represents the zero-mean fluctuation around the identity. Then we have
T = U"(SSS")''SHu"
= (S((SS""'''H(""" -Ew =U'Sw" -U -
Since SHST UAUT and the columns of U and S are well aligned, we expect that U Sw" would exhibit a decay pattern similar to w". The second term U (S thought of as stochastic error which hinders the decay. For small 3, as the decay of u² is slow, the decay of UT "w"" is expected to be slow, and therefore the overall decay of U 0 will be slow as well. If we increase the 8. the decay of U Sw" will become faster, which also drives a faster decay of U 0r However, when 3 becomes too big, the first term U "Sw" decays rapidly, the second term ( after some steep decay.
Ew" could be
dominates quickly, and therefore T will plateau quickly
Figure 22 empirically validates our intuition for the decay of UT For (a,,3) (8) (0.7, 1.1), UTO 0" plateaus quickly; for (0.7,0.1) it decays longer; and for (0.7, 0.1), since w" hardly decays, the target also shows little decay.
(0.7, -0.06)
(0.7, 1.1, -0.1))
(0.7, 0.6, -0.23)
Absolute Value of Components
100
10
These observations suggest that in the left region
(small 8) and the right region (8 > a). the tar- gets decay more slowly than the stochastic gradi- ent, whereas in the middle band (0.5 < B < a) they do not. This could potentially explain why the signSGID-benefichn beneffiial area appears near the left edge and the right side of the phase plane.
10-2
10-3
w"
UT 0"
5.2 CONJECTURE FOR ADAM
10
109
10a
10
We conjecture that Adam with 82 parameter suffi- ciently close to 1 follows the same scaling law with signSGD, based on the heuristic analysis Ap- pendix J In detail, we expect Adam to follow the same asymptotic loss formula (T2) with signSGD, and therefore to follow the same compute-optimal scaling law with respect to FLOPS in the Ta- [ We also conducted an experiment on Adam and checked that the exponents in the Table and the measured compute-optimal loss exponents and optimal model size exponents for Adam match well (see Figure 24
Index i
Figure 3: Decay of 0" in the basis of columns of compared to w*. The legend on the top shows (a, 8, fitted slope of U 0*).
CONCLUSION
We derived the scaling law of signSGD under the PLRF model and identified two distinctive effects drifcs--ddiftnnommaiiiiiooo and caao---ehhaiing-elatiiii to SGD. Analyzing compute-optimal tradeoffs, we showed that signSGD achieves steeper slopes than SGD in the noise-bottleneck regimes, and that the warmup-stable-decay schedule further improves performance in the Area Aa". Additionally, in Appendix we analyze Adam using the heuristic of Xiao et al. (2025) and observe the same scaling law signSGD. Deriving Adam's scaling law without heuristic assumptions is a compelling direction. We defer the discussion of limitations and additional future work to Ap- pendix A.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Jihwan Kim thanks Junghyun Lee for helpful discussions and insightful feedback. This work was supported in part by an Institute of Information & communications Technology Planning & Evalu- ation (IITP) grant (No. RS-2024-00457888 National AI Research Lab Project) funded by the Ko- rean government (MSIT) and the InnoCORE program of the Ministry of Science and ICT (No. N10250156).
REFERENCES
Francis Bach. On the equivalence between kernel quadrature rules and random feature expansions. Journal of Machine Learning Research, 18(21):1-38, 2017.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e231187812) 2024.
Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The geometry of sign gradient descent. arXiv preprint wwxiv:2002.080555 2020.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Arizzadenesheli, and Animashree Anandkumar. SignSGD: Compressed optimisation for non-convex problems. In International Conference Machine Learning, pp. 560-569, PMLR, 2018.
Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. SignSGD with majority vote is communication efficient and fault tolerant. In International Conference on Learn- ing Representations, volume 2019, 2019.
Raphaël Berthier, Francis Bach, and Pierre Gaillard. Tight nonparametric convergence rates for stochastic gradient descent under the noiseless linear model. Advances in Neural Information Processing Systems, 33:2576-2586, 2020.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In International Conference on Machine Learning, pp. 4345-4382. PMLR, 2024.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. Journal of Statistical Mechanics: Theory and Experiment, 2025(8):084002, 2025.
Andrea Caponnetto and Emesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-385 2007.
Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with SGD and random features. Advances in Neural Information Processing Systems, 31:10213-1022. 2018.
Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Proske, Antonio Orvieto, and Aure- lien Lucchi. Adaptive methods through the lens of SDEs: Theoretical insights on the role of noise. In International Conference on Learning Representations, volume 2025, pp. 95500-95557, 2025.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34:10131-10143, 2021.
Leonardo Defilippis, Bruno Loureiro, and Theodor Misiakiewicz. Dimension-free deterministic equivalents and scaling laws for random feature regression. Advances in Neural Information Processing Systems, 37:104630-104693, 2024.
Shihong Ding, Haihan Zhang, Hanzhen Zhao, and Cong Fang. Scaling law for stochastic gradient descent in quadratically parameterized linear regression. arXiv preprint arXiv:2502.091066, 2025.
EleutherAL. OpenWebText2: An enhanced WebText-sttle corpus of Reddit outbound links, https:: //openwebtext.2. readt wtteooccs..oo////eteeeee 2024. Accessed: 2025-

---

Published as a conference paper at ICLR 2026
Damien Ferbach, Katie Everett, Gauthier Gidel, Elliot Paquette, and Courtney Paquette. Dimension- adapted momentum outscales SGD. Advances in Neural Information Processing Systems, 38, 2025.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint htt::2010014700... 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint Siti::1712.000000 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. Advances in Neural Informatio Processing Systems, 35:30016-30030, 2022.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, volume 2024, 2024.
Mareus Hutter. Learning curve theory. arXiv preprint arXiv:2 12200400444 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836 2020.
Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signSGD and other gradient compression schemes. In International Conference on Machine Learning. pp. 3252-3261. PMLR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, volume 2015, 2015.
Tanishq Kumar, Zachary Ankner, Benjamin Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Re, and Aditi Raghunathan. Scaling laws for precision. In International Conference on Learning Representations, volume 2025, pp. 71833-71864, 2025.
Frederik Kunstner and Francis Bach. Scaling laws for gradient descent and sign descent for linear bigram models under Zipf's s law. Advances in Neural Information Processing Systems, 38, 2025.
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between SGD and Adam on Transformers, but sign descent might be. In International Conference on Learning Representations, volume 2023, 2023.
Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data. Advances in Neural Information Processing Systems, 17:60556-60608 2024.
Licong Lin, Jingfeng Wu, and Peter Bartlett. Improved scaling laws in linear regression via data reuse. Advances in Neural Information Processing Systems, 38, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, International Confer- ence on Learning Representations, volume 2019, 2019.
Bochen Lyu, Di Wang, and Zhanxing Zhu. A solvable attention for neural scaling laws. In Interna- tional Conference on Learning Representations, volume 2025, pp. 53372-53411, 2025.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ phases of compute- optimal neural scaling laws. Advances in Neural Information Processing Systems, 37:16459- 16537, 2024.
Loucas Pillaud-Vivee Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gra- dient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems, 31:8125-8135, 2018.
12

---

Published as a conference paper at ICLR 2026
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37:100535-10055 2024.
Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features.
Advances in Neural Information Processing Systems, 50:3218-3228, 2017.
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004. /0802, 2020.
Shehper. scaling.laws: An open-source implementation of scaling laws for neural language mod- els using nanoGPT. repository, accessed 2025-11-17.
laws 2025. GitHub
Howe Tissue, Venus Wang, and Lu Wang. Scaling law with learning rate annealing. Advances in Neural Information Processing Systems, 38, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 00:6000-6010, 2017.
Ke Liang Xiao, Noah Marshall, Atish Agarwala, and Elliot Paquette. Exact risk curves signSGD in dimeesson:s Quantifying preconditioning and noise-compression effects. In International Conference on Machine Learning, pp. 68391-68439. PMLR, 2025.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learn- ing. Constructive Approximation, 26(2):289-311 2007.
Yiming Ying and Massimiliano Pontil. Online gradient descent learning algorithms. Foundations of Computational Mathematics, 865):611-59,, 2008.
13

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIALS FOR "SCALING LAWS OF SIGNSGD IN LINEAR REGRESSION: WHEN DOES IT OUTPERFORM SGD?"
USAGE OF LLM
We primarily used LLMs to polish the English writing throughout the paper. They were also em- ployed to help identify additional related work beyond those we were already familiar with. When preparing well-formatted tables, we relied on LLMs for assistance. We also used LLMs to refine La- TeX code so that complicated formulas appeared clean and readable in the manuscript. Finally, we sought LLM support for debugging code used in our experiments.
OVERVIEW OF APPENDIX
(1) In Appendix A we discuss limitations and future work.
(2) In Appendix B
we discuss more related works beyond those discussed in Section 1.2 and pro-
vide a detailed comparison with closely related works.
(3) In Appendix C
we present experimental results which support our theory.
(4) In Appendix D we prove that the general setting with a feature covariance can be reduced to the diagonal covariance case without loss of generality.
(5) In Appendix E we derive the scaling law formula (2) of R(M, N, %)) under constant learning rate. We first derive one-step update formula and convert equation. We use a deterministic approximation for the integral equation with experimental results. Then we set a proxy of the loss function and verify that satisfies the integral equation.
an ODE get an integral
(6) In Appendix(Fwv discuss the maximal learning rate deferred from the main text, and derive the optimal learning rate, compute-optimal loss, and optimal model size in Table [
In Appendix G we derive the result for the warmup-stabie -tacee learning rate in Section 4.3
(8) In Appendix
provide analysis for the linear decaying scheduling and the cosine
scheduling.
In Appendix provide analysis for stochastic gradient decay deferred from Section 5.1
(10) In Appendix we derive scaling law of Adam under heuristic proposed by Xiao et al. 2025 and verify our results with experiment.
(11) In Appendix K we provide omitted analysis from Appendix B
(12) In Appendix we provide an analysis for the case with noisy labels.
A LIMITATION AND FUTURE WORK
Limitation. Our analysis assumes batch size and focuses on the PLRF setting; we leave theoret- ical extensions to minibatches for future work and provide only empirical evidence in Section C.4 We also use a deterministic approximation whose accuracy we verify empirically; tightening con- stants and extending the formal guarantees are left for future work.
Future Work. Combining signSGD with dimension-adapted acceleration (Ferbach et al. 2025) and extending the framework to more complex architectures (e.g., two-layer linear networks or self-attention) are promising avenues.
B ADDITIONAL RELATED WORK
More Related Works on Empirical Scaling Laws. Porian et al. (2024) resolve discrepancy be-
tween Kaplan et al. (2020) and Hoffmann et al. (002)) Kumar et al. (2025) investigate precision-
aware scaling law.
14

---

Published as a conference paper at ICLR 2026
More Related Works on Scaling Law Theory. There are lines of work analyzing more complex models compared to the power-law random features (PLRF) model. Bordelon et al. (2025) investi- gate the scaling law of a two-layer linear neural network with projected gradient descent, and argue the henefit compared to the PLRF model, which is one-layer. Ding et al. (2025) cover the scaling
law of quadratically parameterized linear regression with SGD. Lyu et al. (025 cover the scaling
law of linear self-attention under gradient flow.
Sharma& & Kaplan (2020) show that test loss scales power-law of model size in regression prob- lems. Hutter (2021) investigates binary classification using a tabulation learning algorithm, deriving a power- scaling with respect to dataset size. Bahri al. (2024) analyze a linear random features model with SGD, showing a power-law decay in test loss with respect to sample size (or model size, when the other is infinite). [Bordelon et al. (2024) derive a power law over model size, dataset size, and time for the linear random features model under gradient flow dynamics.
More Related Works about signSGD and sign descent. Balles et al. 2020 investigate the ge- ometry of sign gradient descent. Kunstner et al. (2023) discover that sign descent could be the key factor making the gap between SGD and Adam on Transformers, [ernstein et al. (2019) propose signSGD with majority vote, which is communication efficient and fault-tolerant, Karimireddy et al. (2019) prove that frrob-feedback can make the rate of convergence of signSGD better.
B.1 COMPARISON WITH KUNSTNER & BACH (2025)
First, their work compares the scaling laws of sign descent and gradient descent, whereas our work compares the scaling laws of signSGD and SGD. Second, they analyze a Linear Bigram Model, while we analyze for the power-law random features (PLRF) model. The ad advantage of the PLRF model is that it models two parameters each for feature vector decay and target decay, while the Linear Bigram Model has one parameter for data frequency decay. Lastly, they derived scaling law where the model size goes to infinity; in contrast, our scaling law covers both finite model size and infinite limit by representing the loss function of model size, number of steps, and learning rate. This makes it possible for us to analyze the compute-optimal scaling law.
B.2 COMPARISON WITH XIAO ET AL. 2025
ODE for signSGD in Xiao et al. (2025) equivalent to the ODE that occurs during our analysis. The reason that we were not able to directly use their ODE is that they derived it under the spectrum lower bound assumption for the covariance matrix. In our case, the spectrum of the covariance matrix SHST decays asymptotically we re-derived the ODE without the spectrum lower bound assumption. Due to the spectrum lower bound assumption, they led to exponential decay to limit risk, which is completely different from the polynomial neural scaling law derived from our paper. They discussed the rossa-ies effect on the level of SDE. In contrast, we observed noise reshaping the level of scaling law and investigated its effect on compute-optimal scaling.
22 so their assumption does not hold for setup. So
B.3 COMPARISON WITH THE WORKS IN THE CONTEXT OF KERNEL METHODS
Yao et al. (0077) study deterministic Gradient Descent and SGD under the reproducing kernel Hilbert space (RKHS) model. Their setup captures the infinice-dimensional case, while our paper handles model size M as tunable parameter to achieve optimal risk. They analyze the Early Stop- ping and that concept is closely related to the number of optimal steps N = f/M under fixed compute in our paper. Both imply that stopping the algorithm before the convergence can be help- ful. The strength of our paper compared to theirs is that we provide an asymptotic loss function with model size and number of steps (which is the same as sample count one-pass setting), while they provide upper bound of loss by polynomial of the sample count. They use the source parameter r and relation r = (2a+ + 2 - 1)/(4a) was indicated in Paquette et al. (2024). The authors derive m ff +2-- 1) rate under condition a + 3 > 0.5, where m is sample count. Our signSGD rate with respect to N for noisy labels in Section L better than their rate. Their strength compared to our paper is that they also cover the classification setting, not only the regression setting. We leave the classification setting future work.
15

---

Published as a conference paper at ICLR 2026
Ying & Pontil (2008) study online gradient descent without regularization under the reproducing kernel Hilbert space (RKHS) model. They represent the expected loss as a function of the number of online steps T. They derive loss formula 7
In T. Similar to Yao et al. (2007). signSGD rate with respect to N for noisy labels in Section L is better than their rate. Their source parameter 3 is related to the target decay parameter / in our paper. Note that they use the same Greek letter but have different meanings. They focus on the number of online steps T, while handle two variables: model size M and number of steps N. Their paper investigates the universal polynomially decaying step size and constant step size depending The first one is similar to the polynomially decaying part of the warmup-stable-dess scheduling. One major difference is that we tune the learning rate based on model size M.
the number of online steps T.
Carratino et al.] (2018) study both multiple and single pass SGD under a random feature model with a connection the RKHS setting. In their random feature model, non-linearity is included by the continuous map v. we leave the analysis of signSGD under the nonlinear model for future work. They provide a bound of risk with high probability, while we focus on the average asymptotic behavior of signSGD. They handle both model size M and number of iterations t. and it is the same as our setting. Their strength compared to our paper that they cover minibatching, while we focus on batch size 1. For the signSGD batch size bigger than 1 makes the problem significantly complicated to solve compared to the case of SGD, so we leave minibatching for future work. Their rate with sample count n is n 1000+22/((((0++))
Our signSGD rate with respect to N for noisy labels in Section is better than their rate for the case 3 > O, and theirs is better for the case / < O.
Berthier et al. (2020) has a closer setting to our paper. They study linear regression with SGD and assume noiseless label. Their upper bound of loss is n (2in((22422-1))2222 1)/((2),1-1((aaa) where n is number of samples. Later work PPaquette et al. (2024) has the same exponents for drift terms, as they also use SGD and assume a noiseless label. The difference between exponents in Berthier et al. (2020) and the exponents of the drift term in our work stems from the trift-normalization effect of signSGD. Also note that our work different in several other aspects: (i) we consider a model size parameter M; (ii) we cover the regime 2a < 1; (iii) we derive the asymptotic loss formula rather than an upper bound; (iv) we consider the compute-optimal aspect.
Pillaud- et al. (0188) investigate multi-pass SGD in least-squares regression with bounded label noise. They got a rate n 220+23-1))(22-
where n is the number of samples, and it is better than single-pass SGD in the regime 3 < 0. Compared to the signSGD rate with respect to N for noisy labels in Section L. our signSGD rate is better when / > 0 and worse for regime B < 0 than the single-pass SGD. Investigating multi-pass signSGD for 3 < 0 will be an interesting future direction.
Much earlier work Caponnetto & De Vito (2077) study kernel ridge regression in the RKHS model. 20-228-
Their rate is t with respect to N for noisy labels in Section D for the case , < 0, and worse for the case 3 >> 0.
227 where is number of samples. Their rate is better than signSGD rate
Later work Cui et al. (2021) also investigate kernel ridge regression in the RKHS model. Dif- ferent from Caponnetto & De Vito (2007), they also consider a noiseless target and get a rate of (2a++23-1) for that case, where n. is the number of samples. Our noiseless drift exponent 2(2a+22-1) is better when a > 8 + 0.5, and worse otherwise. 2a -2841
Rudi & Rosasco (0017) consider random-features ridge regression under the RKHS model. They
give a rate of n (2a+22)/(200 ((2++28+1) where n is the number of samples. Compared to our signSGI) rate with respect to N for noisy labels in Section L ours is better when > O, a 1/(48) holds, and worse otherwise.
Bach (017)) also considers random-features ridge regression under the RKHS model, and gives different upper bound rate n- where n is the number of samples. Compared to our signSGD rate with respect to N for noisy labels in Section L otherwise.
ours is better when 3 > a² a + 0.5, and worse
Defilippis et al. (024) derive a deterministic equivalent for random- ridge regression under the RKHS model. Their rate is 1n--(-11)/// for 3 < 0.5+ 2a and - (4ar)/((a++) for 3 > 0.5+ Compared to our signSGD rate with respect to N for noisy labels in Section L. ours is better when -2 + B, B < 0.5 + 2a or B > (2a+ /88 + 2). B 2 0.5 + 2a holds, and worse otherwise.
16

---

Published as a conference paper at ICLR 2026
B.4 TABLE OF ASYMPTOTIC FORMS OF APPROXIMATION, DRIFT, AND NOISE TERM FOR SIGNSGD AND SGD
We added Table 2 and Table 3 which show asymptotic forms of approximation, drift, and noise term for signSGD and SGD, for comparison.
Table 2: Asymptotic forms of approximation, drift, and noise term for signSGD in different (a, 3) phases. In this table, we provide formula of approximation, drift, and noise term for 6 subphases.
Phase Approx
Drift
Noise
2(2r+28-1 2211) 20 20-22+1
Phase Aa M M-(20+20-1) (M
M
2(20-20-1
Phase Ab M -1
(M*)
2a 21
78 22-22
2(2a=28-11
Phase Ac -
NM))
2a 2p+1
18.40 22-2-
2(2x++28-1
Phase Ad M-2a
(max(1
No 0))
22/-
78 M2-2a
2(2a-28-1)
Phase Ba M-2a
(M)/2NT0)
20 2011
+ M
20 (Noo)
2a+1
M8M
2(2++28-1)
Get-1 Za+F (Nr))
2(2a-1)
Phase Bb M
(max(1 .1/1//20, 0))
-2
M
20+11
38. M
Table 3: Asymptotic forms of approximation, drift, and noise term for SGD different (a, B) phases. In this table, we provide formula of approximation, drift, and noise term for 6 subphases.
Phase
Approx
Drift
Noise
2a 2-
40-1
Phase Ia M (2a+221-1
(Noo)
2n
7o
(oo))))
20
2a fo+-26-1
4a-1
Phase M- (22+2223-1
(No))
ro (NN
2a+266
10-1
Phase lc M-2a
(Nro)
2a
(NNo)))
2a
2a++2
2a-1
4ry-1
Phase M-2a
(Nvr)
M M-I(N7))
(N(%%)
2a
29-1
Phase III M 2zx
(Nvr)
2a
+ M (NNr)
2x
(o(
2n
2a+28-1
4a-1
Phase IV -2a
(N1))
20
(o
20
B.5 ADDITIONAL PHASE PLANE PLOTS TO COMPARE WITH PRIOR WORK
Figure indicates the area where signSGD has steeper compute-optimal slope compared to SGD, by coloring it with Mint green. It lies in Phase Ac, Ad, Ba, Bb, and covers all areas of Phase Bb. In terms of the SGD Phase, it covers all areas Phase and most of the areas Phase IV.
Figure5Jindican the area where signSGD has steeper compute-optimal slope compared to DANA- decaying in Ferbach et al. (2025), by coloring it with Lime green. lies in Phase Ac, Ad, Ba, Bb. It is smaller than the Mint green area, and this is natural, since DANA-decaying in Ferbach et al. 2022) has a steeper slope compared to SGD.
17

---

Published as a conference paper at ICLR 2026
Aa
Ba
la
Bb
8
8
Ab
Ac
lb
IV
Ad
0.2
Ic
-3.4
-02
0.0
02
0.4
0.6
0.5
1.0 3/2 1.4
--0. 0.0
0.2
0.4
0.6
0.8
1.0
12
B
Figure 4: Phase planes to compare signSGD and SGD. Mint green area covering all of Phase Bb and III, and some part of Phase Ac, Ad, Ba, IV is the area where signSGD has steeper compute- optimal slope compared SGD. The left side is the signSGD phase plane, and the right side the SGD phase plane. We placed the Mint green area for both of them for clarity. We will call this Mint green area as Area III-IV sub-
Aa
Ba
Bb
0.4
Ab
Ac
Ad
0.2
0.8
0.4
-3.2
0.0
0.2
0.6
0.6
0.8
12 1.4
B
Figure 5: Phase plane to compare signSGD and DANA-decaying in Ferbach et al. (2025). Lime green area covering some part of Phase Ac, Ad, Ba, Bb is the area where signSGD has steeper compute-optimal slope compared to DANA-decaying Ferbach et al. (2025).
C EXPERIMENTS
C.I EXPLANATION FOR FIGURE
Parameters. Left parameters: (a, B) (0.4,0.8), 0.006, 1.0 for signSGD, 0.4571 for SGD, 20 runs. Right parameters: (a,, B) (1.0,
0.002 for both, e* = 1.0 for constant,
e* = 0.833, 0.091, w 0.05, p = 0.9, T = 1 for warmup-stable-decay 10 runs.
Takeaways. In Figure the left panel demonstrates the steeper compute-optinal slope of signSGD for (a, 3) = (0.4, 0.8) in the area Phase Ac. The right panel shows the increase in compute- optimal slope achieved by warmup-stable-des and experimental compute-optimal slopes agree within errors of 0.04 (left) and 0.01 (right), which well within the error margins reported in prior works.
scheduling for (a, (1.0, 0). The theoretical
Additionally, Figure demonstrates the steeper compute-optimat slope of signSGD for (a, B) = (0.4, 1.0) in the Phase Ad and (a, B) (0.7, 1.1) in Phase Ba.
18

---

Published as a conference paper at ICLR 2026
agrisazz theery = 0.500
signSGD theory 72722
thesry
Final Loss
Final Loss
10-
10
signSGD trajectories
SGD trajectories
SGD trajectaries
SGD trajectories
sign5GD: sloge=0.486
sigrSGD: slope=0.712
SGD: slope-0.389 0.39
10
SGD: slope-0.691
10-3
10a
10*
10a
10c
10"
10"
10a
10th
10a
10a
10a
10
10'
10a
10a
FLOPs M-N)
FLOPs (6
Figure 6: comparison of SGD and signSGD on Compute-Optimal Scaling. Colored lines rep- resent the training trajectories of each algorithm, while black lines denote the compute-optimal curves. In both panels, the theoretical compute-optimat predictions closely follow the observed scal- ing. Both plots show that signSGD has steeper compute-optimal slope than SGD. Left parameters: (o,13) = (0.4, 1.0), To = 0.01, ex = 1.0 for signSGD, e* = 0.533 for SGD, 5 runs. Right parame- ters: (a, (3 (0.7, 1.1), 7o 0.01, e 1.09 for signSGD, e" for SGD, 20 runs.
10
Stted
109
10-4
10
10-3
10-1
10a
10
102 105
102
Figure 7: Deign (M, N, 70) term exponent. Blue curves: true signSGD trajectories. Black dot- ted curves: linear fits over the early-iteratio interval log-log scale. Left: parameters (a, B) = (0.75, 0), 70
400. The theoretical exponent is -2(2a + 28 - 12/ - 23 + 1) = -0.4, which matches the experiment. Right: parameters (a, 3) (1.0,022). 70 0.0006, f(2) M 400, 1600. The theoretical exponent is -2(aa + 23 - 1)/(2a 28 + -1.07, again consistent with the experiment.
0.0012, f(z)
1, M 200,
C.2 EXPERIMENT FOR ALIGNED DRIFT
In Figure 7
we examine the exponent of the Drign (M, N, 7o) term,
2(2o+28-11
(Mmin(a, 7o N)
2a-28+1
of signSGD. For the Phase Aa, the Drign NM 70) term dominates in the early iterations over sufficient interval, allowing us to evaluate the exponent by line fitting on a log-log plot. The experi- mental results align well with the theoretical formula 2(20+23-1) 1)
2a-2841
C.3 VALIDATION OF THE TABLE
Figures through 12 validates the exponent in Table for various (a, 3). On the left plots, draw multiple curves with different model size M while setting the learning rate lower envelope becomes the compute-optimal curve, and by measuring the slope in a log-log plot, validate the compute-optimal loss exponent in the Table I On the right plots, draw the
-M Then the
19

---

Published as a conference paper at ICLR 2026
optimal model size at each FLOPS. Here, the optimal model size is the model size of the curve that meets the lower envelope at that FLOPS. By measuring the slope in a log-log plot, we can validate the optimal model size exponent in the Table [ Note that use similar experimental setting to Paquette et al. (2024). most cases, the error between the measured exponent and the theoretical exponent less than 0.04, and the error less than 0.06 even for the worst case. This error lies within the error margins reported in prior works (Paquette et al. 2024 Ferbach et al.) 2025).
20

---

Published as a conference paper at ICLR 2026
==)======000 0.00, 1.00),
Theory:
(1.000 0.00, 1.00).
Theory:
10-1
St- aoo mooe pourdo
Loos
10
Optimal model
Compute-Optimal Points
fit: Mi: FLOPy"
Loss FLOPS-9340 (87-0.996)
10' Floos 16 MM NI
10
10
10"
101
(0.70, 1.00). Theory:
(0.-00.00 0.30, 1,00), 0.413, Thaory: 0.417
Optimal mocel size M
Loos
10
Optimal Ft: Ni FLOPG9403
Compute Optimal Points
Loss FLOPS-a442 (12-0.999)
10P
16
FLOPs M
B. (0.600 (0.60, 0.40, 1.00), Measure: 0.422, Theory: 0.455
=0)600 0.40, 1.00), Neasure: 0.468, 0.455
10-4
Optimal model size
1055
Optimaa model sbee FLORNOR
Fit: Loos FLOPp-8432 (87=0.990)
10
307
10t
Flops (6 M6-M-MM
10a
(a. B. e" = 10 90, 0.40, 1.50),
Messure: 0.553, Theory: 0.571
(0.900.00 1.30). 0.397. 0.357
10-3
Optimal moder stoe
Lo35
T0P
10-3
Optimal medel s22 M
10
Optimal
rit: Loss x FLOPS 0.990)
10'
10a
10'
Flops (6 NO
Figure 8: Measure of compute-optimal loss slope and optimal model size slope. validate
exponent of R [M - and M with respect to f in the Table The left plot shows the compute-optimal loss with respect to FLOPS N. The right plot shows the optimal model size with respect to FLOPS 6.M. N. Each plot includes the measured slope and the theoretical slope from the Table [
21

---

Published as a conference paper at ICLR 2026
-1.00, 1.35).
Theary: -"1000.000 0.70, 1.35), 0.415, Theory:
10"4
10'2
Optimal model sizee
1205
10"4
10-3
Compute-Optinal Points
Fi: N FLOPynan
Loss FLOPg-9.54 (27-0.990)
10
10
10
10P
101
Flops (6 N)
FLOPs
M-N)
= (0.80, 0.70, 1211.
Theory:
(a, B. #01-00.80, 0.70, 1211, Mezcure: 0A71, Theory: 0-467
Optiral mocel size M
10-2
LoG5
20"4
Optimal Fi: Hix FLOP54413
Compute-Opten Points
Loss FLOPS-a (I²=0.9999)
10P
(6
FLOPs M
= (1.00, 1.1),, Measure: 0.807, Theory:
=1.000000 1.40, 1.18), Measure 0,570, Theory: 0.583
10a
Optmal model size M
Lo05
Optimal Ft: Mix FLOPgAS0
Compute- Optimal Points Fit: Loss FLOPS-- (R²=0.998))
(8
M
e')= (0.70, 1:10, 1.09). Measure: 0.697, Theory: 0.722
(a. B. ==1= 1.10, 1.09), Measure 0.586, Thearys
Optimal model size M4
10-2
LO55
10-3
Optimal Fit: M FLOPs?
Compute- Compute-Optimal Points
Ft: Loss FLOPs (.990.
16
Figure 9: Measure of compute-optimal loss slope and optimal model size slope. We validate the exponent of R M* Mr. :6) and M with respect to f in the Table The left plot shows the compute-optimal loss with respect to FLOPS 6.M.N The right plot shows the optimal model size 22
with respect to FLOPS 6.M.N

---

Published as a conference paper at ICLR 2026
(0.60, 1.00, 1.05).
Theory:
1.00, (0.60,
0.609, Theory: 0.625
Ootimall model size
10"4
LoG5
10a
Compute-Opt mel Points
10h-
Optimal Ft:: Mix FLOPgA00
10"
Loss FLOPs (R*=0 999)
(6
FLOPs M
10.40, 0.60, 1.00).
Theory:
(r. 4")-(0.00, 0.60, 1.00), Measure: 0.533, Theory: 0.543
109
Optiaal mooe siee
10
Loos
10-2
Compute-Opten iml Points
Fit: M FLOPp3552
Loss FLOPy-0030 [R"=1.000)
101l
Hlps 18
FLOPs M-N)
e")= (0.30, 0.70, 1.00), Meacure: 0.300, Theory: 0.341
(a. - 0.70. 1.00). Measure: 0.506. Theory: 0 568
Optimal model size sidelsseee M
Compute-Opting Points
10'
Optimal Fi:: Nis FLDPgA33e
Ft:: Loss FLOPs (R'-0.999)
to. B. e")= (0.27, 0.60, 1.00).
Measure: 0,235, Theory: 0.291
(a. e") - (0.27, 0.60, 1.00). Measure: 0.499, Theory: 0.539
10
Opetmal model size M
Lo35
10-1
Optimal model Fit: N FLOPynis
Compute-Optinee Points
FR: Lass FLOPs (87=0.999)
Figure 10: Measure compute-optimal loss slope and optimal model size slope. We validate the exponent of R M* { 70 and M with respect to in the Table I The left plot shows the compute-optimal loss with respect to FLOPS 6.M.N The right plot shows the optimal model size with respect to FLOPS 6MN.
23

---

Published as a conference paper at ICLR 2026
10200020, 0.219
(c. B. - 0.65, 1.00). Measure: 0.513, Theory: 547
Optimal model size M
LOSS
Optimal model
Optimal Points
Fit: No FLOPy
Fit: Loss
FLOPs
(R'
(8'-0.
307
10
10"
10a
10a
FLOPS (6 M N)
e") (0.30, 0.45, 0.95), Measure: 0.202, Theory: 0.250
(e. =0.300.30, 0445, 0.95), Messure: 0.504, Theary: 0.500
100
Optiral mood atee M
L050
10t
Optimal model size M
30
Computo. Optimal
FLOPs3 5M
Loss FLOPs az (8'90
10P
10P
10
10
10"
10'
10"
Flops 16 B8-0-NN
FLOPs (6 M
Mi
=')= 10.40, 0.40, 0 0.90), Measure: 0.26R, Theory: 0.360
(r. 4")-(0.00, 0-40, 0.90), Messure: 0.503, Theary: 0.500
10a
131
Ootimal mood stze
30-1
10t
Optimal model size M FLOPs? 500
Compute- Optimal
Lass FLOPs 0,86 38²=1.000
107
10P
1044
10a
10a
107
10"
10
1013
Flops 16
FLOPs (6 M N)
Figure 11: Measure of compute-optimal loss slope and optimal model size slope. We validate the exponent R MM* f : 70 and M with respect to in the Table The left plot shows the compute-optimal loss with respect FLOPS 6M. The right plot shows the optimal model size with respect to FLOPS 6.A. N. Each plot includes the measured slope and the theoretical slope from the Table I
24

---

Published as a conference paper at ICLR 2026
10. (0.700 1.50, 1.08), Measure: 0.722, Theory: 0.736
10
6a. B. =0 1.50. 1.08). Measure: 0.622. Theory: 0 631
10-4
10-2
Optimal mode size M
1015
10%
Optimal model
Compute- mal Points
Fit: Mo #LOPph007
30
Rit: Lass FLOPs (R"-0 -0.901
(6
08. e")= - (0.80, 1.70, 1.15). Measure: 0.740, Theory: 0.761
10a
6a. B. =0.000000 170. 1.15). Measure: 0.601, Theory: o 6:619
Opeimal mode size M
100
LO35
10ah
Optimal model Fit: M FLOPyhae
Compute-Optinne Points
10nd
Fit: Lass FLOPs (87-0.900)
1a. B. (0.40 1.00. 1.00), Measure: 0.485, Theory: 0.500
(c. B. (0.400000 1000. 1.00). Measure: 0.595, Theory: 6:625
30
Optimel mode size M
1005
Optimal model Ft:: M FLOPyA
Compute-Opt
Fit: Loss FLOPS-RV00 (R2-0 9977
(6
Figure 12: Measure compute-optimal loss slope and optimal model size slope. We validate the exponent of R [M* - 70 and M with respect in the Table | The left plot shows the compute-optimal loss with respect to FLOPS N. The right plot shows the optimal model size with respect to FLOPS 6M N. Each plot includes the measured slope and the theoretical slope from the Table [
25

---

Published as a conference paper at ICLR 2026
C.4 EXPERIMENT FOR MINIBATCHING
In this subsection, we provide an experiment with batch sizes 10 and 128. Figures 13 and 14 show the measured compute-optimal loss slope and optimal model size slope for batch sizes 10 and 128, respectively. The theory slope in the figure is the theory value for batch size 1. We can see that the difference between the measured value for batch sizes 10 and 128 and the theoretical value for batch size is less than or equal to 0.042. Therefore, we conjecture that mini-batching with a constant- order batch size has the same compute-optimal exponents the batch size case; this is plausible because constant factors in the loss formula are ignored in the exponent analysis. Mathematically analyzing mini-batch signSGD is an important direction for research, which we leave for future work.
26

---

Published as a conference paper at ICLR 2026
(2. B. e.ee") -0 10.80 0.40, 1.00). Measure: 0416, Theory: 0.455
10. =0.600000 0.40, 1.00). Mossure: 0.450 Theory: 0.455
10rt
Optimal mode size M
Loos
10-1
Optimal model
Compute- Prrpuue-Oppimmll Points
Fit: Mi: FFF
Fit: Loss FLOPS-RAIC (R²=0 999)
10. B. e") (1.00, 0.70, 1.35), Measure: 0.703, Theary: 0.706
(a. B. 1...000. 0.70, 1.36). Measure: 0.420. Theory: 0.412
10"1
Optimal mode size M
10-2
1015
Optimal model
10-3
Compune-Optima Points
Fit: M.: FLOPy
Ri:: Lass FLOPs (87-0.997)
16
B. =')= (0.40, 0.40, 0 0.90),
Measure: 0,260, Theory: 0.300
10
(o. (0.40 0.40. 0.90). Measure: 0.515. Theery: 0.500
Optimal model size M
Lo55
3014
10f
Optimall model Fit: Mi FLOPs
Comoute- Onnoute-Optimal Points
Fi:: Lass FLOPs -0.99.)
Figure 13: Measure of compute-optimal loss slope and optimal model size slope for batch size 10. We calculate the exponent of R [M :6) and M with respect to The left plot shows the compute-optimal loss with respect to FLOPS 6.M.N The right plot shows the optimal model size with respect FLOPS 6.N, Each plot includes the measured slope and the theoretical slope for the batch size case.
27

---

Published as a conference paper at ICLR 2026
B. =')= 10.60, 0.40, 1.00), Measure: 0.428, Theory: 0.455
(o. (0.50, 0.40, 1.00). Neasure: 0.497. Theory: 0.455
10
Optimal mode size N
10"1
Loto
10"4
Optinal model
Compute-Optmin Points
Fit: N FLOPy/
Fit: Lass FLOPs IR*=0 999)
00. e1)== 00, 0.70, 1:36), Measure: 0.725, Theory: 0.706
(a. B. -11 (1.00, 0.70, 1.38). Measure: 0.445, Theory: 0.412
(1.00,
100
10-1
Optimal mode size N
10-4
1075
10"
10m0
Optimal model Fit: Mu FLOPydas
Compute-Optingg Optimal Points
Fi:: Lass FLOPs (87=0.998)
15-4
16
(2. B. -""= 10.40, 0.40, 90),
Measure: 0.267, Theory: 0.300
(o. e' =0) 0.40, 0.90). Measure: 0.531. Theory: 0.500
Opsimel mode size M
1003
10-1
Optimal model size Ft:: M FLOPy
Compute
Fit: Loss FLOPy -225 (R2=0.999)
Figure 14: Measure of compute-optimal loss slope and optimal model size slope for batch size 128. We calculate the exponent of R M of 70) and M with respect The left plot shows the compute-optimal loss with respect to FLOPS 6MN. The right plot shows the optimal model size with respect to FLOPS 6.N. N. Each plot includes the measured slope and the theoretical slope for the batch size 1 case.
28

---

Published as a conference paper at ICLR 2026
C.5 EXPERIMENT OF ADAMW AND SGD WITH TRANSFORMER
C.5.1 COMPUTE-OPTIMAR EXPONENT
We calculated the loss decaying exponent with respect to the compute for AdamW (Loshchllov & Hutter 2019) and SGD optimizer on the Transformer architecture (Vaswani et al. 2017).
We conducted an experiment based on the GitHub code of [Shehper (2025). In our exper- iment, evaluated five different model sizes: (number of layers, embedding dimension) (4, 64), (8, 64), (8, 96), (8, 128), (8, 160). We used constant learning rate and gradient clipping with 1.0 for both AdamW and SGD. We set 31 0.9, |3 = 0.95 for AdamW. W. We trained for 10 steps for each run. We set both batch size and gradient accumulation steps 1. We set dropout as 0.1, and set weight decay as 0.1. We used 1024 tokens per iteration. Amount of compute calculated by 6 x (number of model parameters) x (iterations) x (tokens per iteration). The validation loss is a cross-entropy loss with 200 sets of 1024 tokens. We used the OpenWeebeee dataset (EleutherA] 2024) for training.
Figure shows that the exponent of Adam is -0.021 and the exponent of SGD is -0.005. means Adam W has better compute-optimal scaling compared to SGD in this experiment. Our experiment implies that a practical optimizer, AdamW, on a practical deep network, Transformer, can have a bet- ter compute-optimal exponent compared to SGD. Although as an approximate surrogate of Adam and its variants and a simple linear model, our experiment implies that an advantage in the compute-optimal scaling aspect may also occur in a practical opti- mizer, AdamW, with a deep neural network Transformer.
analysis is about signSGD segndie-stuuiedd studied
C.5.2 DRIFT-NORMALIZATUUN EFFECT AND SOISE-RESHAPIR EFFECT
To observe the crift-normalization effect, we experimented with a batch size of 16 and gradient ac- cumulation steps of 32 to decrease the noise term. As the loss curve the sum the drift term, noise term, and approximation term, decreasing the noise term allows to observe the drift-normalization effect more clearly. We experimented for (number of layers, embedding dimension) = (8, 96) for each AdamW and SGD. Other experimental settings are the same as the section C.5.1 In Figure [6] we measure the slope of the loss curve in a log-log plot for the linear decaying interval, where the drift term is dominant. We can observe that the slope for AdamW is larger than SGD, and this is consistent with the nnfftnoomalliationn effect in PLRF, which increased the exponent of the drift term in signSGD compared to SGD.
To observe the noise-reshaping effect, we focus on the plateau regime of the batch size I experiment. To see how the loss value of the plateau regime is influenced by the size of the learning rate, we experiment with two learning rate values: 0.00266 and 0.00133 for both Adam and SGD. We experimented for (number of layers, embedding dimension)
(8, 96) for each AdamW and SGD. Other experimental settings are the same as the section C.5.I. including batch size and gradient accumulation steps In Figure 17 we can see that the loss value at the plateau regime, which is dominated by the noise term, increases for Adam W when we take bigger learning rate, but does not increase for SGD, This is consistent with the noise-reshaping effect in PLRF, which made the size of the noise term in signSGI increase we take a larger learning rate, in contrast to SGD.
C.6 OTHER SYNTHETIC TASK EXPERIMENT
We experimented with feature learning based on the setting of Bordelon et al. (2025). In the feature learning, the sketch matrix S becomes learnable, in contrast the fixed Gaussian sketch setting of the F model. We let S = (L) where (()) is M x M square matrix and (0) = 7. During the training, we update the square matrix (t)) at each time step with the optimizer. Other settings, except for this learnable sketch matrix, are the same the settings for PLRF
Figure 18 shows our evaluation of the compute-optimal slope for Adam, signGD (full-batch sign descent; deterministic signSGD), and GD in the feature learning setting. We experimented with full batch due to the training instability of small batch cases. We experimented for the parameter (a, 9) (1.0, 1.25) which is included in the Area III-IV sub- In this feature learning experiment, Adam and signGD had similar slopes, and those two had a steeper slope compared to GD. The result consistent with the phenomena in PLRF that signSGD has steeper compute-optimal slope
29

---

Published as a conference paper at ICLR 2026
10.00
7.400P
Valiaaion Loza
6x
0
(70-0.000))
-0.00 (P-8.550)
Compute Frundel
Compute- Opdinal Porter
12
3012
2014
10a
10th
10th
Compuse (FLOP()
Compute
Figure 15: Measure compute-optimal loss slope for Adam and SGD on Transformer ar- chitecture. Left: AdamW, Right: SGD The x-axis shows the amount of compute calculated by (number of model parameters) x (iterations) x (tokens per iteration). The y-axis shows the val- idation loss, which is cross-entropy loss with 200 sets of 1024 tokens.
#5
13x
45.00
-8320 -1000
val fet bqperer: didios
10
10
101
Figure 16: Measure of drift term slope for Adam and SGD on Transformer architecture. Left: AdamW, Right: SGD The x-axis shows the iterations. The y-axis shows the validation loss, which is a cross-entropy loss with 200 sets of 1024 tokens.
compared SGD in the Area III-V sub: and also consistent with the conjecture in PLRF that Adam has the same compute-optimal slope as signSGI).
30

---

Published as a conference paper at ICLR 2026
10.00
Valdaaee Loss
8'
10
30
10
327
10'
107
Eorations
Figure 17: Plateau loss value for different learning rate. Left: AdamW, Right: SGD The blue curve is the trajectory with a learning rate of 0.00133, and the yellow curve is the trajectory with a learning rate of 0.00266. The x-axis shows the iterations. The y-axis shows the validation loss, which is cross-entropy loss with 200 sets of 1024 tokens.
a-)= 11--11.00 1.25,
optesignGD, (a. e')=
1.0
10-1
10-4
10
1050
10
1055
10
10-5
Co Points
Compute-Optinal
FLOPS-R0rS (02=0.9922
FLOPs 1:44 (8.-0.386)
10
122
15
10
105
10
10
10a
10
101
10
10"
10"
10"
10
FLOPs
M-N)
FL.0% 16 M N)
opt=GD, (1.00. 0.00). Messure: 0.584
10
10-3
1253
10
Compute Optimal
Fl: Loss FLOPs 9.54 (R/=0.398)
10'
10a
30
30
10
10a
FLOPS 06 94
Figure 18: Compute-optimal exponent for feature learning. Left upper: Adam, Right upper: signGD, Left lower: GD. The x-axis shows the FLOPS. The y-axis shows the loss value. We perimented for (a, 3) = (1.0, 1.25). We set e of We use constant learning rate. Dimension before projection is 2000, and each loss curve experi- mented with projected dimensions 32, 64, 128, 256, 512, 1024.
- as the optimal value derived in PLRF
31

---

Published as a conference paper at ICLR 2026
D EQUIVALENCE TO DIAGONAL COVARIANCE H
In this section, we will prove that general covariance H with eigenvalues 1 1-2a ,2-2a, ,.,22....,,' 20 can be reduced diagonal covariance diag(1 -2a 2-,22
20x ..... d 2a), Thereby, for the following sections, we will assume H diag(1 [ -2a 2a d d= without loss of generality.
Recall that we assume (w); w0) 8-8 where vs eigenvector of H corresponding eigenvalue i 2n for = ......d. d.
Let D diag( 1 20 2 d 2a). Then H UDUT holds for some orthogonal matrix U by the eigenvalue decomposition. The i-th column of U can be thought as vs Then the following holds for wg -- 2
ST..
w
v2 Uww.
i=1
The signSGD update rule is
Ok+1 02 7 sign ((Sig, On) yk) sign(Sax).
With label assumption yı: (arks w*) the signSGD update rule converts to
i k++1 = 02 " sign ((Sr; 02) (Izk, w*)) sign(Sx:).
We let 2/1 U. xt. By substituting æ = Uzi and w" Uwi we get Oh1 02 The sign ((SUx) 0x) (U Uwi)) sign (Sgn(SUxkk)
As U is orthogonal, it leads to
0k+1 = 02 - 7 sign (SUx 0v) (0/ wr)) sign(SUa()..
(16)
Also, the loss formula
L(0)= = -0:00-=)111=0 = -00001111
HSSTO ww)
converts to
L( = -
U = ((SU) -w)) D(((U)) wv)-
d 20) and target wi 29.2-8 [1-8,2-8, .2......,d'- B|TT. is same with the diagonal covariance case. Lastly, the distribution of SU is identical to the distribution of S. This is because each row 8; of S follows the distribution NO Iz/M), and s,U, which each row of SU, follows the distribution (0, U LUU/M (O, Ia/M). Also note that s,s are independent and s, Us are independent.
Now the covariance of a is D
diag(1 -20 2-
So the converted update rule (6 diag(1 12a,
equivalent to the case with diagonal covariance
2 2-2a ,d-2a).
32

---

Published as a conference paper at ICLR 2026
E DERIVATION OF THE SCALING LAW FORMULA R(M, N, 70)
Goal. In this section, our goal is to derive the scaling law formula 12 R(M, N, 70). On the area a < 0.5 or 3 0.5 with -a + 0.5 < B < a + 0.5, Drigu (M,""MM,MMM No) term is smaller than at least of the other three terms. So is enough show
2(2o 28
R(M, N, m0)
M 2a-t 1-2)) (Mmin(a,0..5 No0)
N7))
2a 25+1
2 (22mmin(1,2))
=I((M)
=:D (M,N,
=:Niian
for that area.
For the area > 0.5 and B > 0.5 with -a 0.5 < 8 + 0.5, as all four terms are dominant, we will prove
2(2a+28-1) 2a-28+1
N,M,N, 7o) ~ M 20 2 122)) + (.) Noo)
A(M)
=: =:Da *M,,NMMMM (M.
6a-1
2(2a-1) 1)
+ M 2a+1 (No)
2a+1
M 22
Diii (M.N.70)
(M,1n
Proof Overview. As a first step, we obtain the ODE
dp: 4
20(1/70)270 V,.
P(E) (K) 5(1/70)
(17)
x
where P(() = L()/ho) and pz(() = (1/7/0)
Then derive the following integral equation from the ODE.
M
1A.70
f(u) JoN VL(w)
2% M
N
1d:70
S
f(u)
L(N) 1/2 w 11
(0) e
V1
e
[L(w)
f(z)² dz.
π
(18)
Going through the arguments, including the contour integral, our integral equation converts to the following equation, where Q(=) = 4na
Jo
f(a)
du.
VI(w)
22--1-1
L(N) M 2a (0.1 29)
miniaa 0.5)
Q(N))
20
(19)
approx
drift
2%
N
4 A
N
du
V
exp
dz.
(20)
π
π
VL(u)
noise
for a 0.5 B < 0.5, and
2nst 2,9
1
L(N)
M + (M11)(((N)
2x
M (M (Q(((2)))
2a
(21)
approx
drift,
driftz
2
47
N
du
V
exp
(K)
dz.
(22)
π
L(u)
noise
for > 0.5 and 3 > 0.5.
Solving the early stage and the limit stage separately, we get the following proxy for a < 0.5 or 3<0 0.5.
2(2x: + + 1)
Tppx (N) = (70 0.55 N) + 20 M min(a, 0.5) M max(0, 1-28)
p 2a + 1 28
(23)
33

---

Published as a conference paper at ICLR 2026
For >> 0.5 and 0.5, get the proxy
pxx = (Yo M PI + (NoM
M ou AN) 2
+ C,
(24)
where
2(2a 2/3 -
4a -2
P1 2a + 1 2/3
P2 20 +
As last step, we verify the proxies by proving that they satisfy the converted integral equations.
E.I ONE-STEP UPDATE FORMULA OF SIGNSGD
Xiao et al. (2025) approximate the signSGD trajectory using SDE and ODE techniques. Their proof relies spectral lower bound assumption of the covariance matrix, so their results are not directly applicable to our setting.
For quadratic function q, by Taylor's theorem, have
Elq(0 0x 1) - ((x)) FA] B((Va(On), 0u+ - 0x) Fil E (V (, 0x) Fi
where Fs = (S, 0-. 0 Since
0 0 ~ sign( 0x) yy)
We can expand the two terms using sign- iign-Gaussian identities.
Gradient term.
(((vg(Os) 0k+1 0k) Fil = (V((k)) EE (iign( (Sx)) sign (xx,, STo, w")) Fxl)
diag (iag(SHST) - 1/2 SH (ST w0"
--T (g(( aresin
VSTO, w³) H (STO,. - w))
2 aresin
diag K) K (0): - 0*)
7 Vy(Ok);
( 02 - w0)l
where K SHST
Quadratic term.
2 q, (0x+1 0x) 0x) Fr
= 2 q, E
(sign( Swxx sign (en(((x))) ((I), STO, - w"))) 02
Fr
= 2h
4zg, 4, E
(sign(Saa) (Sank) 82 Fk
2/
V2g. q. π
aresin diag (SHST) - 11/2 SHST diag (SHST) -1/2)
4. aresin diag K) K diag K) 1/2
One-step update formula. step update formula for signSGD.
Substituting the gradient and quadratic terms yields the desired one-
E(g(O - ((xx) F. = 27/x
(0. (0p--7) -0-) VL(K)
))
2 ( K₂)
Vg(Os), arcsin
Let (x) u; and denote the eigenvalue, right eigenvector, and left eigenvector of K, respec- tively. Then K DM, Xe (K) u, &9 un and I Σd U, wi:
34

---

Published as a conference paper at ICLR 2026
Define
r:(k) = (0x - (K') @ w;) - 8*)
The loss decomposes
d
L(k) = 1 or) | || = -0, (( ((00 11 Σ
r;(k) H 1/2 2
We now apply the one-step update formula to rk()). Note that (r:(k) K u, (wi 0x 6*) + w; {K uu 0g 0*),
V2r, ku + ww,, K
Approximating (r)) ~ x and using KT = K together with K K K we obtain
27
K (Ox -0")
K (0x - 0*) -0") 00)) ))
(/r + 1) (k)))) Fi] ]
(wi, 0x - 00) Ku,,
(Ku,, 0x -0") w VL(k)
π
VL(k)
k
w K Ku:
41ix
27k wi K2 Ku;-
X (
rk():) +
/L(E
It is possible to replace the linear approximation arcsin(x) = by an inequality, and the main results of our paper remain unchanged. We explain it in Appendix K.2 Hence,
4
2 w K Ku
Blra(k + 1) rr)
K
(k)
L(K)
E.2 ODE APPROXIMATION AND IMPLICIT INTEGRAL EQUATION OF SIGNSGD
Let the learning rate be 7% (k:). Define V, w. K Ku, then our one-step update formula becomes
47h
A (rK)r:()) + 2 V4.
B/ri(k + - (k)) Fx)
L(k)
Dividing by To gives
(k + 1))
(KK) s(k) + 2 Vi.
E
Fx
70
x <
Interpreting To as the time step, the discrete index h corresponds to continuous time / = kyn- Let P(t) (o))/r)) and pt(t) (t/raa) We then obtain the ODE
dp:
A(K) [(1/70) + 2f(t/7o)P V, dt π /P(()
4
(25)
From this point onward in the analysis, we treat P, L, and as their continuous extensions, allowing arbitrary positive real inputs.
Integral formulation.
Solving the ODE yields
4d So
f(u/ro) du PP(u)
4h: f(u/ral du /P(w)
2n
(t) p0)
V.
f(s/10)² ds.
M Since P(() Ei= (:(1) 1/2 112, we obtain
M 12 12+ Σ
4dg lt f(u/70) du
M
4h St (P(a)
Hlw/ro) du
P(t) = 1/2 w
pr(0)
(P(w)
+
270
V
e
f(8/700) ds.
1=1
π
0
35

---

Published as a conference paper at ICLR 2026
Integral equation in discrete form. obtain
Note that L(N) P(Nr)). With change of variables,
M
4A,70
Jo
[(a)
du
2%0
M
4A:70
5
f(u)
du
T(N) // t0
11
r((
V L(u)
+
V
e.
V [(w)
[(=)² dz.
π
1=1
(26)
Drift and noise decomposition. Define
4h,70
fo
f(u)
2%
N
42,70
JN
((u)
T. (N)
Σ e
/L(a)
L muise (N)
V
v L(w)
f(=)² dz.
i=1
(27)
Then
L(N) = || | 1/2 w/ll 1|+ + drift ( + L moiee (N),
(28)
and we will analyze || w1ll" + 1, and L, poise (N) separately.
Figure 19 shows dynamics of three terms prox, Drift Noise. The right plot Figure 19 validates the equality in 28
H 1/22 w 1|2, I, [,dr NN), I. (N) referring each as Ap-
10
10"
10
10
signSGD
Risk
10
ODE
Risk
10
signSGD
Drift
ODE
10-1
Noise
10-3
Drift
Approx
Noise
10-4
10
Approx
Sum of three terms
10-1
10"
10
10a
10a 10
10a
10
10a
10a
10a
10a
10*
10a
Iterations (k)
Iterations (k)
Figure 19: Dynamics of Drift and Noise. Left: the purple curve the 80% confidence interval the true signSGD trajectory, while the blue curve is the numerical ODE solution. The yellow, orange, and green curves correspond to the approximation. drift, and noise terms in (28). Right: the red curve shows the sum of these three terms, matching both the true trajectory and the ODE solution. Parameters: 1.0, 0, 70 0.003, f(z) 1, M 200, d 800.
E.2.1 TRANSFORMATION OF THE DRIFT TERM AND APPROXIMATION ERROR
Let
470 f(u) du, Q(=)
K, T ( 1/2
π Jo VL(u)
Then
KK HH
holds.
Define
A 1 K.Q(N) 1/2
u STO, STO w1 STO w."
From SHw.. = 0 we get
K 1/2 (w) =0,
and this implies
wi) ( (H112 (H
1/2 wi.
36

---

Published as a conference paper at ICLR 2026
Thus,
Awi = _w_: wI Aw = =w] HWw
and
T AT u Hw (0. " SHw. -w] H - _w H
Using these identities, we can convert the drift term as follows:
M
(N) = Σ
r((0) (A(KKQ)(
2m1
00 (Ku, @ (0 ( ->>,(KKIUN)
Σ (0 0²) (Ku, & (w:) . - 0*)
1=1
(0. 0*) Ke Ke- (T--KQ()) (0o 0")
(0u - 00) T SH H 1/ Se Se RQ(N)) 8*)
- (0) SH 1/2 (c- QIN 22 0*)
SSH
= (u + w ) A(u + w
-u Au u Tw + w Au w Aw
= =u =uTH'2 1/2 KIQ(N) 12 u w Hww - w Hw + w Hw
= =u H H /2 -.Q(NN) 12 Pu- |
1/2 1/
Drift term plus approximation error. Adding the approximation error gives
drift (N) + 1/2 w u (K(())
(S 6, - w) 02) KIQ(N) (K-K.qiN))
le-
Also we assume 0, = 0. then
(IQN() (H2 0. w²) w²)) (e-R (KQN)) 220
In the subsection, we will describe how apply deterministic approximation, similar Paquette et al. 20024) to the following term:
H. = (e (e-KIQ(N) 082).
where H1 E Rd.
E.2.2 DETERMINISTIC APPROXIMATION
Note that we assume d rM for some 1, and let d/M (1, co] d, M. oo when 2a 1. and d/M + (1, 2o) when 2a < 1. In our setup. S E RR Mxd have i.i.d. ( 1/M) entries, and we will write the kth column ST VM Sy Rdd columns are independent.
Define
VM
yk 1/2 8k E Rdd
ak
0.
V M
UI Ub
/ Hs
37

---

Published as a conference paper at ICLR 2026
The unnormalized baseline and the (aluum-normmaized matrices
K
1 M
yayk:
: S diag(SHS))-1/SsH
SH 1/2
ak y
M
F C+ = (2: 3z 0), define the resolvents
-1
L(z) = [ - -21) (z) = M
1
Σ E#
aayey? - =1
Note that
Yk THHB)
for matrix B. In particular,
VM a VTrH H
N Tr
Also note that
VM
a* y R.R
Tr (HRR())
yk
Tr H
and
VM
0k yk R
Tr(H R
yk
Tr H
By the Sherman-Morrison expansion,
M M- ag (M)) yky R())
R= (1)
1 M M-1 y R yk
Multiplying the left by R and sandwiching with 3 (-)k. get
M M- y. R R( yv a.y. R yk
a R Q yk
1 + M-lax y R R(kgy yv.
Now will replace terms the right side by
M
a R. R (2)
T((RR())
yk
VT H
and
M
0K V R
Th( R)
Tr H
Thus
JM
Tr(H R.
a u T RRys
R
VITH
M+ VM Tr( R())
VITH
Replacing ( by R and averaging over k. we obtain
M
Pa H RR)
VM
anys
RRys
Pa:= VITH
M
1+ M pa H
k=1
It implies
Pa Tr(HH R)
Tr(R(R + =I) R
1 M
Tr(H R)
This implies
L(z) z1
1+ -pa Tr H L(z)) H.
Pa
38

---

Published as a conference paper at ICLR 2026
Let
m(z/pa) 1+ M- Tr( H L()))
Then
L(2) ~ (-21 pam
Thus
= (-2I pam(z/ (Ki - 21)
Therefore,
m(=) 1+ M- pa Tr H R(paz))
1+ M Tr ( (-2I m(z)
holds. This fixed-point equation is identical to the one in Paquette al. (2024).
Contour representation. Let H 1/222" Rd and consider
H (KKKQ( vo2)
For any analytic on a contour I'2 enclosing Spec K)), $ 2τi Jr,
g(K1)
g(2) K1 - -=1)-² dz.
We prove
C1 (m,in(0.5,a)
1 diag diag(SHHT) SHS I 02 Mm.in(0...a))
in Section
K.4 It leads to
CI Mmin(0.5.0)
K K 3 02 M0min(0.5.a) K.
K2 has eigenvalues scaled by M0iin0.5,,o) compared to K excluding constant. Note that Pa Mm,n So, there exists a contour 12 enclosing the spectrum of K,, and its 1 1/pa-scaled version I encloses the spectrum of K.
Taking (22) - Q(N)s
$ Q(N)= (K1 =z))= 0 dz
H
2 Jrz
1 $ Q(NN== ((-2I p-pmaal//pu H) 022) dz
2 frz
2mi $
DoQ((N) ((-2I m m)) 222) dz.
Let R(x)= = (-2] +
H) (-)))= 11 then our objective converts to
H 2mi / Pa (R(2), 022) de.
E.2.3
FINAL TRANSFORMATION RESULT
Paquette et al. 2024 evaluate the contour integrals with R(2). When < 0.5 or B < 0.5, they show
f (1 -1-2- (1-2,B4
Bz B2 + BB (BBBB+)+2)) 1))2)) (f(2), 0823 de - max(0, 1-22)
2mi Jr
2a+28-1
+ (2y
2a
(29)
39

---

Published as a conference paper at ICLR 2026
When > 0.5 and B >> 0.5, they obtained
$ 2- 2 (E(2), 180) de = M M-2a+max(0,11 1-20)
2mi /
2x+22-1
(2yBr)
2a
M (2yBr)
2+ 2a.
(30)
For the case a < 0.5 or < 0.5, applying similar method to our objective yields
$ e pa Q(N)= Q(N) (f(z), 2 de M -2- max(00 1-23)
2ti f
2a+-26-1
0.5)
Q(N))
2a
(31)
with details provided in Appendix K. Hence,
RKQQN), (H w )02)
M 2a +max(U, 1-28)
2a 2-8
Munin(a, Q(N))
2xx
(32)
For the case > 0.5 and 3 > 0.5, similar argument gives
2 f e ppa e Q(N)s (f(z), v02) de ~ -ax
1
2a+22-11
+ (Mmin(a, 0.5) Q(N))
2a
-1+ 1
+ M-1 (Mminlaa 0.5) 0.5) Q(N))
2x
(33)
with details Appendix K.I Consequently,
(o K RIQ(),
(H2 w )02) ~ M 20 m,x(U,111111
2a
+ (Mminia 0.5) 0.5) Q(N))
20
+ M-1 (Mmiaaaa 0.3)
Q(N))
-1+ 2a
(34)
In summary, we obtain
20 20.-23 2.
L drift (N) + H 1/2 w 1 M maxx0 1-20) + minin 0.5) Q(N))
2xc
(35)
for a < 0.5 8 < 0.5, and
drit (N) + || 1/2
12 w 112 max 1-
20
+ (Mminna, 0.5) Q(N))
2a
-14 1
+ M-1 (M 0.5) Q(N))
20
(36)
for a > 0.5 and 3 > 0.5.
Figure 20 shows that transformed result in 35 and (36 based deterministic approximation match the true signSGD trajectory up to a constant factor. When interpreting the figure, note that our analysis is asymptotic; hence, discrepancies may appear in the very early iterations.
40

---

Published as a conference paper at ICLR 2026
signSGD 80% signODE
10a
ratio mean
deterministic
#1 std
2-
Risk
109
Ratio
1
10
0
10a
10a
10a
10a
10
10a
104 Iterations
10
Iterations
signSGD 80%
2.0
signODE
ratio mean
deterministic
1.5
#1 std
Ratio
Risk
1.0
10-1
0.5
10
0.0
'102
102
10a
10*
10a
10a
10a
10
10
Iterations
Iterations
102
signSGD 80%
signODE
ratio mean
deterministic
+ 1
Risk
101
Ratio
2
10
90:
10a
10a
10a
10
102
10a
10
10*
Iterations
Iterations
10a
signSGD
signODE
10a
ratio mean
deterministic
10a.
+ 1 std
Ratio
Risk
10
109
2-
10-1
10-4
9
102
10a
10
10a
102
10a
Iterations
Iterations
Figure 20: Verification the deterministic approximation and drift-term transformation. Left: the purple curve denotes the 80% confidence interval of the true signSGD trajectory. the blue curve represents the numerical ODE solution, and the yellow curve corresponds to the deterministic ap- proximation after drift-term transformation in (35) and (36). Deterministic approximation matches the true trajectory up to a constant factor. should be noted that our analysis asymptotic, and thus, discrepancies may occur in the very early iterations. Right: the ratio between the approxi- mation and the true trajectory remains bounded by a constant factor, confirming the validity of approach. Parameters: (a, B) (0.7, 0.3), (1.0, 0), (0.4, 0.4), (0.7, 1.1) from top to bottom, % = 0.003, f(2) = 1, M 200, = 800, 100 runs.
41

---

Published as a conference paper at ICLR 2026
E.3 CONSTANT LEARNING RATE: PROXY AND VERIFICATION FOR THE CASE 0.5 OR < 0.5 (PHASE A)
Throughout this section, set [(z) 1; hence
4 Q(V)
N
du
π
V(u)
Applying the drit/aapproximation-eee transformation the ODE solution yields the implicit rela- tion
2-1---11
L(N)
M
2n + max 23)
(M 0.5)
Q(N))
20
(37)
approx
drift
270
M
47o
N
du VL(u)
V
exp
X
K
dz.
(38)
noise
E.3.1 EARLY STAGE (DOMINANCE OF THE DRIFT TERM)
At N = 0, the noise integral is O, the approximation term is independent of N, and the drift term is large and decreases with N. Thus, the early phase,
2a-28-1
L(N)
(Mmin(a, 0.5) Q(N))
2a
(39)
Since Q(N) Ano Jo L(u)-11 du, 39 is equivalent (up to absolute constants) to
du
L(N)
2a +2
Mmin(a,00 0.5)
70
(40)
VL(u)
To obtain a proxy profile, we replace = by equality 40 and differentiate both sides:
2a
I(t)
2a
('(t) Manin(a, 0.5)
70
(41)
2a + 23 1
LL(t)
Solving 41 for (t) and separating variables gives the separable ODE
dL
dt LL < = 2a + 28 - + K
1
+ 23
M Mmnn(a 0.5)
2
2a
Assuming Ç > 1 (i.e. 2a + 28 < 4a + 1), we integrate to obtain
L(t)-(s-1)
L(I) = 1)=-1) 1/(5--1)
kt + constant
(42)
5-1
Substituting Ç
20-428- + and A
2x
2a+28-1
M mini(a, 0.55) yields the early-phase proxy
2n
2(2a + 2,3 1)
L(N) (7o Mamimia 0.5) N) P
p:= 2a 25
(43)
By construction,
satisfies
40 (hence
39
up absolute constants.
E.3.2 LIMIT STAGE (STATIONARY ANALYSIS AND FL.OrR)
With 1, the mode-wise ODE is
dp: dt
270 V.. / P(() \ p()))()
At stationarity, pt( + s; and P()) + Loes we must have
(K) 8, + 2% V, =
LoL V, Lo (1o. K, Ku.).
8
T
π
2>((K) 22;(K)
42

---

Published as a conference paper at ICLR 2026
Using the loss decomposition P(t) Σ Ps(t) + | w obtain
M H/
M
W K Ko
100
8
+
VL + H 1/2 w/ll
(i(K)
1=1
70
TT diagg K) 1))12K2 K.) V Loo + H 1/2 w1 21 11l12
2
Tr(diag(K) K) V Loo +
1| ww 11?
2
4
Solving the quadratic in Lx gives
2
To
Tr(diag( (K) + v( 4 Tr (r(diag( ( + 4 |4| 2
10
(44)
2
max (10 Tr(diag( ( || ww 11
(45)
Under setup,
M Σ
Tr(diag( K) K) 1/2
(SHST)
)a
Mmax(1 2a, ~ M 1-miaaa 0.5).
M
By the results from Paquette et al. (2024):
Lin al. (024), and note in Appendix K.3
| /2 1|
M 2a 1-29)
Hence
Loo max 10 M 2-2 an3),,,0.)) M 200 )
(46)
E.3.3 PROXY
Combining the decay
43
with the floor 46) adopt
2(2a + +23
(N) (no Miiin(a,0.. 0.5) N)
M 2-22 min(a, 0.5) M 2a mo=mmax00,1111
p
2a + 2,3
(47)
E.3.4 VERIFICATION OF THE PROXY
We show that Typx satisfies N
37 to absolute constants. Equivalently, writing Q1, (N) =
we establish
(u))
20-228-
270
4 A >₂(K) {.d
N
du
(Mminn(a, 0.5) Qu (N))
+ M 2a max(0, 1-2
V
exp
VLpx(u)
dz
J
approx
1=1
(48)
(o Mann(a 0.5) N) C.
(49) Lpx(N)
Lower Bound
We prove
drift approx noise
(70 Mmmin(a, 0.5) N) -P +C
(50)
Since [u) 2 (%o Mimin( 0.5) 0.5) PP-
2a++28 2n
NN 2 (aminn(a, 0.5). 70 /-/ (To Mimin 0.6) w/ du
4
drift (Mminla, 0.5) Quae (N))
2xc
(No M Minina 0.5) N)-P.
43

---

Published as a conference paper at ICLR 2026
Since [pp(u)) 2 C for all u,
N
du
N
1
Vapae (1) (1)
VC
Hence
270
M Σ 1=1
NN
470 (K)
noise 2
Vi
exp
de
(51)
Ju
VC
2
2
M
VC
(1-e -0 (x)
Vi
(52)
4no (k)
1=1
VC 70
7o TTddia(()) 1/3) (K)
V
to M mimn(a...0.) VC 2 To M 2- min(a,0.5)
(53)
Adding the approximation term M + 1-28) gives noise + approx 전 C. Combining with the drift contribution yields (50)
Upper Bound
We establish
drift + +pprox + noise S (70 M Mumin(a,,.5) 0.5) N) N)- +C.
(54)
Let
2(2xx + 2B -1) 1) A(N) == max (10 Mminnaa 0.5) N) c). p= 2a + 1 - 2,3
Then Lxx A(N). Define No by (70 Mmin(a, 0.5) No) -P C, i.e.
(70 Mmin(a, 0.5) N) N s No.
A(N)
C,
N > No.
There exists constant B such that
(NV S BANN
(AN 2 0).
(55)
Upper bound for the drift term. Since L S BA by 73 and Q is decreasing in its denominator,
2a-2881
2x 2a-226-1
drift (Muiin(a, 0.5) (N))
2nx
(Mmin(a, 0.5) QHa(N))
2n
We evaluate the right-hand side by cases.
Case N s No. Then A(u) = (Yo M 0.5) -) for S N, so
470 du QBA(N)
VB 70 / Aminla. 0.3) u) u/ du
BA(u)
for absolute constant
0, which implies
drift S (%o M Mmin(a..... N)
2a
(1+p/2) (70 M/min(a..... N)-P. p. 2a
Case N > No. Split the integral at No:
No (7o Mrmin(a,,.3) 0.3) w)nl du 0.3) (7o Mmiuaa,,.. No) 1tp/2 + To Mmmilaa,, 0.5) N No
0.5) 0.5) (
N du
Mminaa 0.5) QBA(N)=
To Momiaa,
VB
No v BC
VB
VBC
44

---

Published as a conference paper at ICLR 2026
Raising to the power
2a+228-1 and using (no Mminta, 0.5) No) C.
20
2x
drift x
c
1/2/2
+ 70
No
2o
1
0
(80436-13/00)
2a
C.
/ BC
Combining the two cases,
drift 2 (70 Amin((,,.5) N) + C.
(56)
Upper bound for the noise integral. By the monotonicity of r + r 112
du
du
L(u)
VB J= V((u)
Therefore,
270
N
4h
N du
noise
Vi
exp
Ar(K)
de.
(57)
TVB
VA(u)
We again split into two cases.
Case N & No. Then A(u) = (7o M Mminn(a,..5) ()-))))=>> u)- on NJ. hence
du
(7o Mmmin(,,005) 0.5) p/2 {
IN 2p/2 du (7o M ammnaa,0..) 2012p/22 N1 2l4p/2
A(u)
1 1 p/2
Plugging this into 57 and factoring,
noise-
2nd
Σ
V1
exp
470
X (K) (70 M 0.3)) p/2 N - 21/1//2
0.3))
dz
TV TVB B
1 1 1+p/2
i=1
2
47o
A(( (7o Main(a, 0.5))/ 9/2 N1-p/2
V1
exp
π
TVB
1 1 p/2
>i(K) (70 Mmiulaa, 0.53) p/2 21tp/2 TVB
470
exp
dz.
1 1 p/2
Make the change of variables becomes N /
z 21//22 so that de
1 1tp/2
2 6/2
dy and the upper limit
Ni+p/2
noise-
2
V, e N141/1
y y -1 dy.
1 + p/2
1=1
(70 M Miinn(a 0.5) 0.5) p/2
470
04 TVB
\;(K)
1 1 p/2
Let X := 41+r/2 and
g(v)
1 p/2 y 240
1 p/2
Since c is increasing and g(y) is decreasing on (0, Chebyshev's integral inequality (oppo- sitely monotone) yields
/ x X Ju
1
((v) dy - (///²
x
a.v dy) (2 x
)lv) du).
45

---

Published as a conference paper at ICLR 2026
Hence
x e e 10a evv) dy S Se-o eacx
x
-1
((y) dy
Oii
X 0
1 - --
1
1
X
24p
0i
1 + +/p
1
2tp
P
1 e
-x
X 24.
(ince (1 - (1) =1)
a;
1
1
X
2tp
N-p/2.
aii
Therefore
210
1
noise
V2
N/2/2
N
π
a4
20 Mmin(a, 0.6))
p/2
and with o;
47a TVB (k)
this becomes
1tp/2
noise I
TVVB
(1 2/2)
Σ
M
V
(1o Mmminaa...5 N) pp/2 -
2
(K)
Using E,
V. A,(K) = Tr(ddaa(( K) 112)) ~ M 1iniaa 0.5), we get
noise S 70 . 1 min(a, 0.5) (70 M min 0.5) N) N) p/2
7/ M miinna 0.5)
(%o Mmin(a 0.5) 2/ (no Mminaa 0.5) N)-
1
S 70 M 1-minn(a, 0.5)
(70 ma 0.5) N) p x (70 miiuaa, 0.5) N)-P,
VC
where we used (70 Mmin(a,0..) 0.5) Nyp/2 $ (10 Mmiin(a,0.5) No) C-11/2
Case N No. Split the z integraa at No:
No du VA(u) The first integral is the N = No case just handled, hence
noise- <
27d
M
Vi
exp
4ho TVB
A((K
dz +
N expl
TVB {x(K) 1.a VA(u)
4%0
'N du
dz
No
h
No
dz S (70 Mminfa, 0.5) No) C.
For the second integral, we use that A C on [No, N]:
N
expl
470 N du dz expl
N
470
X K
N -
dz
No
TVB
AA(u)
No
TVB
VC
vc
11
4:p w/B
X.(K)
N Nu
v0
Tne A.(K)
NVB
TVB VC
4% As(K)
Therefore,
2
AVB VC
noise S (7o Mmin(o,0.5) 0.5) No) +
V:
4% (()
20V B
M
V
%o B
B
= C
VC
= C+
+
VC Tr(diag(K)
2
K
2
i=1
C + 70 Mminno,,03
0.5)
VC S C+ VC.
VC s C.
46

---

Published as a conference paper at ICLR 2026
Combining both cases,
noise - (7 Mmiula,,..) 0.5)
N)-P + C.
(58)
Conclusion of the upper bound. From 56
58
and approx M 12-2 2a max(0, 1-
& C, wc
obtain (54).
Finally, combining the lower bound (50) and the upper bound (54) proves (49 Therefore, the proxy
67 satisfies the implicit relation up absolute constants, with the three contributions labeled
as approx, drift, and noise.
E.4 CONSTANT LEARNING RATE: PROXY AND VERIFICATION FOR THE CASE >> 0.5 AND 0.5 (PHASE B)
We handle the because 3 > 0.5, we have min(2a, 2a + 23 - 1) = 2a. Applying the driitapproximation-temm transformation the ODE solution yields
0.5 and | 0.5. Since 0.5, have min(ax 0.5) 0.5, and
2a+23-1
I(N) ~ - + Q(N))
2a
M (M(2
2a
(59)
upprox
drift
drift
270
M
400 Ae K
du
V1
exp
dz,
(60)
π
VL(u)
1=1
noise
where
470
du
Q(N)
[(uu
E.4.1
EARLY STAGE PROXIES (DRIFT) AND DRIFT2)
We extract proxies from the two drift terms in (59 by the same differentiaaee-add-ssppaaa trick as before.
drift;: (MI/3Q(N)) (28-22)1)
Assuming this term dominates and replacing ~ by equal-
ity,
2a 2a-23-I = 1/2 To
du VL(u)
L(N)
Differentiation gives the separable ODE L'(t) = -Ki L(t) with
20 1
2x + 2,3
B1 = 2a 23
M 22
2'
2a
For 81 > 1 (equivalently 2a 2 < 4a + 1) we obtain
2(2a + 2,3 - 1)
(NNN) = (70M
(61)
2a + 2,3
M-1 MM1(M1/2Q((N) -1+ 14
drift;:
Assume a > y and, in the early phase, the second drift term
1
dominates:
2a-1
N du
I(N) (M
2a
Q(N) = 70
VL(u)
Expanding the M-exppoent,
(M1/2)) -
20-1
2a
M
(2n 1)
4n
2a
2a
47

---

Published as a conference paper at ICLR 2026
hence
6a-1
2a-1 I(N) = / VL(u)
N du
L(N) ~ M
(x)((N)
2x
(62)
Raise both sides 62
to the power
so that the integral becomes linear:
Ia-T
20
60-1
Gar1 N du M
L(N)
2a-1
M 4a-2
To I(N) ~
1a-2
70 jo
(63)
VL(u)
Differentiating 663
with respect to t yields
20
2zz
1
T((1)
2a-1
T1(1) M
4a
70
2a
(L(t)
Rearranging gives separable ODE of the usual power form
20:
1
6a
(t) - Lt
B2
>
(64)
2a 1
2
4a
with
20 -
6a-1
K2
To M
4a-2
0.
(65)
2a
Since 82 > 1, solving (64) gives
L(t)-(82-1) = (32 - 10)-1) Kg + const.
Absorbing harmless absolute constants into and setting t N,
(2(N)
(10 M
da
Ga-1
N)
P2
2(2a 1)
(66)
32
2a
Crossover scale. Equating 61 and (66) gives
2x + - 48
N1
70 M"
1
4/
so R1 dominates for N S N1 and L2 for N 2 N₁ (when a > 0.5 and 0.5 < < +
E,4.2 LIMIT STAGE (APPROX AND NOISE FLOORS)
As in the case a < 0.5 or 3 < 0.5, the stationary analysis with f= yields
Loo max n Tr(diag( K) 1/23, H 1/2 11/2w_1''' ]1
Under our standing model Tr(diag( K 1/2 1/2)
~ A 0.5 and by the results from Paquette et al. (2024);
Lin et al. (2024), and note in Appendix K.3
1/2 2 = M -2a, hence the floor
C =
oo M + M-
E.4.3
COMBINED PROXY
Tapx (N) = (N(N) + L2(N) +
(67)
= (70 00.5 N0.MMNN N)-Pi + (70 Ga
+ C,
where
2(2a 23 1)
4a 2
P1 2a + 233
P2 =
2x + 1
E.4.4 VERIFICATION OF THE PROXY
We show that Tap satisfies (59 up to absolute constants.
48

---

Published as a conference paper at ICLR 2026
Lower bound.
We claim
(M.5 Lpp
(N))
2 20-1
M M- (Mos MM.sQ1p Qup (N))
1 2m
M M
(68)
dritty
drilig
approx
21
M
N
V1
exp
170 de (1(K)
dz 2 (x))
(69)
π
noise
Drift part. Using px 2 Rı inside Q
AM0.BQ2, (N)) 20420-1
2 (Masnn /o- N du
(NoMONN))= ~ LNN).
vLi(u)
Similarly, using Lxp 2 L2 inside Q
M- wx (N)) 1+ zh 2 M-1 (M-0s
/ .V
du
-1+
2 (70 M to -o -2 (2(
Ga-1
VL2(u)
Therefore,
drift: + driftz 2 LiNN + L2(N).
(70)
Noise + apprex. Since Lix 2 C,
du
N -2 2
S (px(u))
VC
As in the Equation 53
V
noise 2 %0 CovCy
Trddiag( (1) vc 70 Mo. 2 20 M.
(K)
Thus noise + approx Z C. Together with (70). this proves 69
Upper bound. We will prove
(MU3QL 2 (N))
+ M (M Suu.sQL (N))
mm
+ M M-2a 20
(71)
drilty
drifta
approx
M
2nd
4r0 \n(K) L
+
V1
exp
du
dz S L,x((N)
(72)
Let
(70 MO.
N N1. N < N < N₂.
A(N)
(70 M -= N) P2
C,
N N₂.
where N1 and N₂ are the crossover points between the three terms. There exists constant B such that
2
(xx(N) < BA(N)
(VN 2 0).
(73)
suffices to show
2a 28
20
+ M 0.5 (A.SQBAA())
1+
+ M 20x
(74)
drift,
dritz
approx
2 M
+
exp
4r X X₂
VB.A(u))
dz S LPx
(75)
49

---

Published as a conference paper at ICLR 2026
Case N S N1. It is enough to prove
20428-1
+ M M (M) -1 20
+
2
(76)
driflz
approx
2
2
M
+
Vi
470 X exp
K
du (B.A(w)
de
2 (7o M
(77)
We have M-2a < (n0 M0.6N)- directly. Also, the following holds with straightforward integra- tion.
20+28-1
(MO.QBB-A
(70 M N)
Since N S NI 2 7ot M with r) = 43
2a-1-48 following holds by integration and calculation.
(M B.
2 (no (M.5
Finally, arguing as in the N S No case of Section E.3.4
2 M
exp
exp(- - 120
120 Ax)) S(
du /BAlw))
dz S (70 (0.5N
Hence, the claim holds for N S N1.
Case N₁ I N S N2. We will show
[M (BA()))
20
M (M (N))
+
M 22
2a
(78)
drift,
driftz
approx
+
2%
2
M
exp
400 (K)
du VBA(W)
dz S (70 M
aa=2 No-
(79)
noise
where
pi = 2(2a + +23
2(2a 1)
(ro M0.5 p-.
S N1,
2a + 1 - 23
P2
2a +1
A(u) =
(70 M 20-2 MaaMaaa00
2) -po
N1 < u N,
and (N) = 47a So
du BA(a)
Approx term. Since N < N2.
Ga-1 M 2a (To M la-2 N) -P2
Drift term. If N1 S N 2N,, using the case N S N1, we get an inequality for two drift terms.
(M0 (Baa))
28--28-1
M M (QA(NN) 1t
(80)
dritt
drita
(MM.5QBA
20438-1 2n
+ M
BA (N:)) 1+
(81)
drilt,
drilg
S (vo 0
50.5 -m $ (70
M 4a
N) P2
(82)
So while covering the drift term, we will temporarily assume 2N, N.
Lower bound on QBA (N). Split the integral at N₁:
QB)(N) ~ 70/ //m du N du
V((u) + 70 JN, VA(u)
=: 7o (L1 + 12).
(83)
50

---

Published as a conference paper at ICLR 2026
For the first part, using A(u) (TodM on N1),
No !
um du (7oM0.50o0/2 N/tm//2
1 (NoM00.6)m
(84)
1+ P1/2
For the second part, using A(u) = (70M1
Ber
oa 22)- P2 on [N,
12 = (70 M 1o- jaal / upr/2 du (70 in- )
NN
(N N2tpo/3).
(85)
+ p2/2
Since we temporarily assumed N 2 2N1, we have
12 (70M aa-2 ) /
Hence, from 833
Ga-1 QBA(N) Z To (7
1a-2 poz/2 N1 x1+p//2
(86)
drift; vs. driftz. From N N1 and (86, we have QBA (N) [BA((N) It follows that
drift, (Ba
2a-20-1
M- (M -1+ driftt,
so it suffices to control driftz.
drift, bound. Using (86).
-1+ -1+ 2
drift, = M-1 (M QBA(N))
-It
$ SM-1
Tv M
1tp2/2
N1tm2/2)
(87)
Now compute the exponents of N, Yo. and M separately.
(i) N-exponent:
(1 (1++) -
(-1+ 24 1+
2a
1 -1)
4a
(-
(-20--
2a 1 1a22-11
2(2a - -PP--
2a
2a
2a
20
20: +
(ii) ex-expnntn:: the same calculation as in (i) gives -p2-
(iii) M-exponent: the total exponent equals
-1 + (-1 + 2/) (.. +
)
6a 1
p2
4a 2 2
direct simplification shows this equals
fa--1 p2. Therefore, from 187 za
Ga-1 drift S (70 M la-2 N)-P P2
(88)
Since drift, driftz, also have drift, S (70 N)
Noise bound. It suffices show
2
M
470 (1((K) Lm
N
Vi
exp(
(B-Alw)
de
(89)
π
2
M
V
expl
- L (
dz S (70 M
2a pp
(90)
π
Ni
Integral over [N1, N]. As in the case S No of Section E.3.4 with A(u) (1oul (0 20 [Ni, NJ.
on
2
M
V.
N
exp(-
4r0 (1(K) {xK)
VB-((w))
Ga-1 dz S (70 M Ia-2 N -pa
π
No
51

---

Published as a conference paper at ICLR 2026
Integral over [0,N First,
2
M
N the ((( L
2%
M
expl
du dz S B.A(w)
V1
exp - x
470
X (
du
dz.
v BB.(u))
1=1
As in the case N S No of Section
2
M
exp the ( { VB-A(w) dz S vC (10MMNN)) -Li/2 - S (70 M 2a-2 Ni) P2
Ga-1
V
π
If N S 2N1, this already implies
2
M
exp 1 (x) { N du BBA(w))
Ga-1 dz S (70 M 2a- p
V1
1=1
If N > 2N1, then
0 exp
470 X K
du ABA(
dz S N1 expl te X, (K)
du
B
B-A(W))
N1
6a 2 N Ltp2/22 (70 M 4a-2 p2/2
and, using x= x < 1
ff together with the lower bound JN
du /B.A(u)
we get
2%
N1
2%
M
N1
V1
de 2
V1
π
π
4 π
Xr K)SN
N
du VB.A(u)
V1
(7oM 6a-1
2a-2 -P2/2
70
Ax(KK
70 0.5
0.5
(1o M 2a N) -22/2
6a-1
S (70 M 2a-2 N) -P2
Ga-1
N)-
1
where the last step uses m M0. < (7o. 4a-2 N p2/ which holds from N N N2.
Combining the N] and [0, N1] bounds yields
6n noise S (70 2a-2 N)- P?
as required for the case N1 S N S N2.
Case N 2 N2. We have M -2a S C directly. As in the above case,
(MUSQB
20--3-1
S M (M (M))Q -1+
Using the estimate from the previous case,
M M- (MM. (N)) -1+ th S M
M-
(M0.Q (N2))
-1 2v
6a S (70 2 p2 SC C.
Finally, as in the N No case of Section E.3.4
2
M
N 1 \1K)
V1
exp
V dz S C.
du
Therefore, the bound holds for N 2 No as well.
E.5 NOTE ON THE REGIME 3 + 0.5
When 3 > a + 0.5, the assumption < > used in step 42 no longer holds. In this case, the first drift term takes different form:
2(26+28-1) 2J 2u
Laon, (N) = k0 Mminfa.u.3) N)
52

---

Published as a conference paper at ICLR 2026
for finite horizon and some constant Inserting the max function, represent it as global function.
2(20+28-1)
Lain (N) (max (1 7o Minin(a,0.55 N,O)
Now we explain the behavior of the term. When N is asymptotically smaller than (To ( the term is asymptotically constant. On N ~ (ro M the term suddenly drops from constant scale to 0.
For the case < 0.5 or < 0.5 the valid proxy is
2(2
(NV = [aax (1 - K " M M N, N.O)
28-20-1
+7u M 2-2 min(a, 0.5) +M - 1-23).
and for the case > 0.5 and [ 0.5 the valid proxy is
2(2a 1)
Lix (NV = maxx (1 No M N.
+M 2a+1 (N%))
2a+1
+70 M + M -2
-2x
These satisfy the implicit integral equation, same as Sections Therefore, for the case 0.5, a + 0.5,
E.3.4 and
E.4.4
2024+25-1)
R(M, N, 70) max (1 - 7o M N.O) and for the case a > 0.5 and B > 0.5,
28 2m
+ 70 M 2a + 222
(91)
2(2++28-1)
Go-1
2(2a 1)
R(M,N, 70) (max (1 K 70 M N.O)
+M
2
1 (Noo)
2a ++
+70 M -20.
(92)
53

---

Published as a conference paper at ICLR 2026
F DERIVATION OF THE COMPUTE-OPTIMAN RESULT
Goal. The main goal of this section is to derive compute-optimal scaling laws of signSGD in the following form:
R RM >d=..6) =
M =
Here R(M, N, 70) denote the L(Ox) under learning rate 7o and fixed model size M. We define the computational budget in terms of FLOPS as MN,, and consider the optimal model size M under fixed f. and optimal scaling of learning rate in the form % M
Proof Overview.
Substituting the learning rate To M into our loss formula
2(2a+28-1)
R(M, N, 70) ~ MM 20x 1-20) + (M.iin(a....)) No))
2a 2a-2p+1
6a-1 +M 2a+T (Nv))
2(2a-1)
20+1
+ 00 M m
we can represent the risk as function of three variables M, N, e, and two parameters a. 3.
Then for fixed compute f MN, we substitute M = f and N = ff- to express the risk as the function of three variables f. æ, and two parameters az, B. Four terms in the loss formula convert to four terms with exponential of FLOPS with exponent functions I to Ls.
R(f xr e, 8)
f (i((,,,e..))) + f=f-allxee..)) + falx.,,,)))
(I(,,,
Since each term is power of f. and assuming f 2 1, the loss simplifies to
2(f,2,e) f h(2.0.0.0)) where h(x, 3) min((1, L2 ls l1).
We find the optimal learning rate exponent e" and the optimal model size exponent by
x arg max e, 3).
As we optimize over two variables x and e, three terms among & balance on the optimal values x² and e*.
Then the optimal learning rate is 76 M -=",, and the optimal model size is M = fr= FFinally, the compute-optimal scaling law is
-f=((x" R(M", 1/M","0) h(xx.((x aa"="""""ee" aa.8),
and h(x", ,ee,e,,,,,, 3) will be the compute-optimal slope in absolute value.
F.1 COMPUTE-OPTIMAL RESULT FOR MAXIMAL LEARNING RATE
We now discuss the maximal learning rate case deferred from the main text. Note that Phquette al. 2024 showed that the maximal learning rate for SGD is 7o when 1. and 7o M- (112aa
when a ! 3-
Now, we discuss the maximal learning rate for signSGD. Because the noise term is no M 2-min(1, 2a), stability requires
d M 2 min(1, 2n)
S 1.
Otherwise, the signSGD noise term explodes as M grows. This condition is satisfied by choosing
M min 0.5)
which ensures 70 A m1,, x while the other terms still decay appropriately. For 0.5, the term
2/2x+28-11
2(2xx 2/3 29-1) I)
(Mmiula, 0.5) N No)
2a 2a-2841
(M-()120) N)
2a 288+1
54

---

Published as a conference paper at ICLR 2026
decreases with N but increases with M. However, under fixed compute budget MN, one allocate resources so that this term does not cause an exploding loss; hence we do not classify it as unstable.
Thus, the maximal learning rate for signSGD
M 1+min(aa
70
In this case, however, we obtain R(M, N, 70) ~ 1, so the slope of the compute-optimal curve is always zero.
F.2 DERIVATION OF COMPUTE-OPTIMAL RESULT FOR OPTIMAL LEARNING RATE
We assume a + 3 > 0.5 throughout, even for the case where it is not specified.
F.2.1
0.5, < 0.5 (PHASE Aa)
We start from
2 --1
R(M, 2o) (111/2
2a+1-20 23
+ M 1--200 220--10)-1 + 100 M.
Substitute
= M
N
M =
M
so that, up to constant factors,
R (1)x)
where
(1) (2a + +3 23-1)
(x(x) 2(2a + 23 -1)
(e (c+5)
2(2a 2/ 1) 2a
2a + 2,3
2/3
(x(x) = (1 - 2c)
We minimize the convex, piecewise-lined function f(z, e) = max, (( e) over E (0, 1) and e E R. By convexity, any interior minimizer must occur at kink where least lines are active. In our regime a + 3 > 0.5 and B < + 0.5, the only admissible triple intersection is (4, lg fz). Solving & and lg Eg yields
h² =x=)) l2(xx) = Ex(x*) 2a 29
= B
x²
=
2a +1+* 1
2a -
To verify that this kink is the global minimizer, note first that x² E (0, when > 0.5, hence it is interior. Next, the subgradient optimality condition for convex max-of-lines problems requires (0, 0) E 0f(=", e*). At (x", e²) the active lines have slopes that straddle zero in both coordinates:
0.41 - (22-+28-1)) < 0, 0.l. 2(2a 2B 1)
C (e+
0, 0,ls = 1-2e" 1122(a+ 0,
2a 1 23
and
2(2a + 2,3
1) 23
0,4 =
0.2₂
x² 0,
0.6 -2x" <0
2a 11
Since 0 lies the convex hull the active slopes in both r and e, we have (0,0) E 0f (")) so the interior triple intersection is the global minimizer; no boundary check is needed.
2o M- M x f1/22411,, R R(M' x/-) x
2a
2a+1
55

---

Published as a conference paper at ICLR 2026
F.2.2 < 0.5, 3 < 0.5 (PHASE Ab)
We start from
2(2n+22811 R(M,N, (7)
2a+1-28
+ M (2a+222)-1) +7 2-22.
Substitute
M N
M-P,
M'
so that, up constant factors,
~ (ale)
where
(1) (2a 23 1) x,
(2)2) 2(2a + 2B 1)
2(2a + -) 2a + 1 23
(a- 1)
2a + 1-1-- 23
6 (x) (2 2a 2e) x.
We minimize the convex, piecewise- linear function f(x, e) = max, la(x, e) over # E (0, 1) and E R. Under standing assumptions + B > 0.5 and 3 < 0.5, the only admissible triple intersection is (4, 6,, f.). Solving l = lg and l2 = lg gives
2. (==x))))= = (2(==)20))))) bz(x²) 2a 23
= 3 1
To certify optimality, note that x- E (0, 1) (since x- = 21) and check the subgradient condition (0,0) E 0f(z", e²)) ). At (x", the active lines have slopes straddling zero in both coordinates:
0,4 = -20 <0, d.2== = 2(2a + 23 -)
(e +1-a) > 0, = 2c" = 11-((a+ < <0,
2a + 2a 1-
and
0,41 = 0, 0.l2 2(2a 23
2a 23 x='
0,63 -2 <
Hence lies the convex hull the active slopes in both variables, so the interior kink (x", e²*) ) is the global minimizer, no boundary check is required.
70 = M (8+00.5)
M p1/2 R(M, xt=)) ~
2a
F.2.3 a < 0.5, 0.5 < 3 3 0.5 (PHASE Ac)
We start from
222+22811)
R(M,N. = (M*NN))
2a+1-20
+ M 20 + + M 2-2a
Substitute
20 = M
N M
M M=P
so that, up to constant factors,
R R (60))
where
(z(2) = 2a
b(() = 2(2a + 23 283-1 -1)
(a-e-1) (a-e -1)
2(2a + 28 282-1 -1)
(a
2a + 1 -2
2a + 1-112 2,
(x(x) = (2 2a - 2e) x.
56

---

Published as a conference paper at ICLR 2026
We minimize the convex, piccewise- objective f(z, e) max, (z e) over x (0,1) 1) and e E R. In the regime a + 3 > 0.5 and / < a + 0.5 (with a < 0.5 < B). the only admissible triple intersection is {e1, E2, ). Solving & = Lg and lz La yields
2a + 23
2a (2a + 23 - 1)
e* 1,,
x² - + 6a + 43 -
h" = 4 - (2(xx) = Ez =
-4a8 + 6a + 4, -
One checks that the denominator is positive in this regime and exceeds the positive numerator 2a 23 1, hence x² E (0, 1).
Interior optimality. At (x", , the active lines' slopes straddle zero in both coordinates:
2(2a + 23 33-11
0,h -2 <0, 0zl2
(e* (1- > 0, Ozl3 2- -2e" -22 < 0,
2x + 1 223
and
0el2 2(2xx 2/ 1)
0.41 = 0,
x² >0
0.l3 2xx <0.
2a 1- 23
Thus 0) e 0f(x* e²) and, with x (0. 1). the interior kink is the global minimizer; no bound- ary check is required.
70 = M-1 M
2a-128-1
MMM,, +
E.2.4 a> 0.5, 0.5 < 3 a + 0.5 (PHASE Ba)
We start from
R(M,N, 70) = (M 2(24+28-1)
2a+1-28
+ (M ho-1
No0)
2(20-1) 2041
+ +M- + 70 M.
Substitute 7o = M - N = =f/M, M fr. Then, up to f-independent factors,
R
maxx 1, (((,,e)
where
(xx) = 2a
(2(x) = 2(2a + 28 - 1)
2(2a + 23 - 1)
( 2)
2a + 1 23
2a + 1-+1-2 1-2 2
(x(x) 22(2a - 1)
2(2a 1)
1)2
2a + 1
2a +
4(x) = (1 - 22) x.
We minimize the convex, piecewise- linear function f(x, e) = max; fa e) over x E (0, e E 0.5, 8 0.5, the only admissible interior kink with three active lines is (12, la, ]. Solving la = la and fg = la yields
R. In the regime
e' 2a + 4,8 - x" a a+
B
2o: + 2,3 - 2a + 23
-==2(0" c0) Es(x* (x)x=="")== la(a" e')==,")===>
43
Interior optimality: First, x' E (0, since a, B > 0.5. Second, the subgradient condition (0,0) 0f _e*) holds because the active slopes straddle zero in both variables:
0,62
2(20+28-1
+ O, 0 = 2(20-1)
< O,
0,l4 1-26"
1-2ax22 28 0, 2/
2a+1 23
2a+1
and
0.6₂ 2(2n 2,8 x²' > 0, 0.l3 2(2a
> 0, 0.ls = -22" O.
2a+1-24
2a+1
Hence
(x", ee*,e") is the global minimizer among interior points. It remains to exclude e, at (x* e²);
B
2a + 2,8 - h:" 2(a + ())
6 (1*²) = 2x
5
a + 3
since 4a,3 2a 23 1) } > 0 for a, 3 0.5. Therefore the triple intersection
{b2, 6,. 4,} is the global optimum.
2a 2a+48-1
2a+22-
7o M
49
M x fat3
R
(MM,
M²
{
~
2a+25
57

---

Published as a conference paper at ICLR 2026
F.2.5 < 0.5, 3 >> a + 0.5 (PHASE Ad)
Recall the loss formula (91)
2(20+25-1) 20 20
R(M, N,,,),))))))) m)) = maxx -1- (1-670 M N,
+ +70 M2 + Ma
Note that the drift term vanishes at ~ (70 Ma)-1
Let To M Note that because the approximation error M- there is no gain from setting bigger than 1. So we will only consider the case < 1. In that case, loss is constant scale before N M*=a, and it drops to the scale M 2e-2a+2
Since a constant scale loss cannot be compute-optimal, the loss M M-2e-2a+2 at N ~ M will be a candidate for the compute-optimal point. In that case f= MN = Mi+e-a holds and leads to
M frre-e, So the loss M - -22-2a+2 has the size f
22-22a42 11c-a
Since e = 1 minimizes 22+-2 Ite-a
770 M is the optimal learning rate. This leads to the follow-
ing result.
70 = M M fz-a R(M", st-.) ~ f 2-
20
F,2.6 a 2>0.5, 0.5, 8 a + 0.5 (PHASE Bό)
Recall the loss formula (92)
2(2+28-11)
fir-1
2/2n
R(M,N, N, v0) [max (1 77 M0 N,O)
+M 2a+1 I (N70)
2a+1
+70 M -
Note that the first term vanishes at N ~ (1) At that point second term becomes
6a-1
2(2x-1)
M 20++1 (Nr))
20+1
M
As optimize over three parameters N, M, To. and one constraint f MN, have two degrees of freedom. So this means three terms may balance together at the compute-optimal point.
The first possible case is the balance of the first three terms, and in this case, 7/M M = M uपं and
N (7o M must hold. Here, the loss is M I and . M holds, so the loss is T
The second possible case is the balance of the last three terms, and after solving the equations, the loss is f 2.TI.
The first case has steeper decay, so it is the compute-optimal. This leads to the following result.
6 4a
20+1
R R(M atr:)
70
M
M
f
4a+1
F.3 DISCUSSION FOR THE SUBOPTIMAL LEARNING RATE
In this section, we calculate the compute-optimal exponent for general size of learning rate in the form of To M We will focus Phase Aa. In that phase, the maximal learning rate was To 112 and optimal learning rate was % M (a++))
In this section, we will calculate the compute-optimal exponent for general 2 1/2.
Recall that we have the following loss formula for Phase Aa.
2(2a+28-11 2a+1-22
R(M,N,00 (20
29
+ M (2a+28-1) + M.
222--28-11
For the case 1/2 < + 3), (M1
2a+1-24
and of M are dominant terms. Substi-
tuting ?0 M e and balancing them, get N M 1 - (e-1/2) As MN holds, it leads
58

---

Published as a conference paper at ICLR 2026
Compute ustinal encorent
033
04
14
122
14
Figure 21: Compute-optimal exponent with respect to e of to = - for (a, () = (0.6.0.4).
Colored line shows the compute-optimal exponent æ in the formula R M M 70) ~ f-.
(2e- 1)(24+29-11
M p2/00100000 Xc- R(M,, M" Mr: :10
f-
2ox ()x (2
For the case 2 (a 8). (
2a+1- and M 12a-222--11 are dominant terms. Sub-
stituting Yoo = MM - and balancing them, get N = M Ma-9++e As MM holds, it leads to
2a-28-1
M pl((-Btett), R R(M st. 70)
a a S+o+1
In Figure 21 we provide a graph the compute-optimal exponent with respect to of to M for (a, 3 (0.6, 0.4). As the graph is continuous, the absolute value of the compute-optimal exponent gradually decreases as we move away from the optimal choice. Also, we can observe that the degradation is smaller for the learning rates with larger in 70 - than that of the optimal learning rate. So terms of tuning the learning rate, we may aggressively set high in %o = M -c for the initial attempt, and gradually decrease the for later attempts.
59

---

Published as a conference paper at ICLR 2026
G DERIVATION FOR THE STABLE-DECAY AND WARMUP-STABLE- SCHEDULING
We first derive scaling-law bound for the stable-decay schedule, simplified variant of the stable-deayy (WSD) schedule, and then extend to the full warmup- stable-daay schedule in Section G.4
We set the learning rate as 7 = "oo f (k).. Previously, we considered the constant-learning-n case (f 1). In this section, start with general decaying learning rate by taking f to be decreasing function, and then substitute the stable-decay scheduling. Throughout, for simplicity, we assume 0.5 and / 0.5 (Phase Aa).
Recall the implicit integral equation (60
M
4dm70 //a N f(u) du (0) exp
L(N) = ||
T
2
(93)
VL(u)
1=1
270
N
N
f(u)
V
exp
42,70 x
du f(2)² dz.
(94)
π
v L(u)
Also recall Equations
27 and 28
M
4A.20 Jo
f(a)
du
2% M
4Amm
[(u) du
drift (N) = Σ e0)
n
L(a)
L noise (N)
Σ
V
e
n
L(w)
f(2) dz.
1=1
(95)
L(N) | w 1| (N) + L moise (N).
(96)
Recall also the drit//approximation transformation
2a+28-1
1, drin (N) + 1/2 wrlll = M 1222222--1) + (M ((0552(())
20x
Q2(2) = 470
π L-¹
f(u)
du.
VL(u)
Hence,
2a 2a-28-1
L(N) = MMMM--2aa (2a+23-1) + (MMS)L((N))
2a
(97)
2
N
N
f(u)
expl
4A.20
du
f(2)2
(98)
V(uu)
22
Remark 2 (Early-itcration proxy). In early iterations the drift term (A (o(.))((N))))))))))))
dom-
2a+2881
inates. Solving I(N) (M0.5
yields
2(2a-28-1) 2a+1-28
F(N) = 1-' N
L(N) (M (F(N)
f(u) du.
Now we move on to stable-decay scheduling.
Learning-rate factor S(K)
Stable Regime
Stable-decay schedule. For the stable-decay schedule, we set the learning rate to ro (k) with
k pN, f(k) = (1+ - PNN) - k> pN.
11.
(99)
Decay Regime
where P.C E 1) and > 0. In other words, the learning rate remains constant for the first pN steps, and then decays polynomially with exponent for the remaining (1 N steps.
pN Iteration
N
Figure 22: Visualization of Stable-decay 60 Scheduling.

---

Published as a conference paper at ICLR 2026
Remark 3. Note that f depends on the total train- ing steps N. To be precise, we have to represent it as fx. but for simplicity, we write it as f throughout the analysis.
First, we will make an upper bound on the noise term under stable-decay scheduling.
G.1 UPPER BOUND OF THE NOISE TERM
Fix < q < 1 close to 1 and split noise (N) as
2
9N
N
f(u) du du) [()) dz
|uoise (N)
V1
exp
40:70
VL(u)
7
2
M
f(u)
+
V
exp
4Acra
VL(u) du f(2)² dz =: TsON + TTNN
9N
Bounding T TzgN. Note that f(N) = f(z) holds for qN <I< < <
43:70
f(u) du du) f()) S(N)
expl
4h,30
N f(w) du dz.
)
exp
VL(u)
VL(u)
NN
For q sufficiently close to 1, there exist constants co, Ci > 0 such that for qN < <
cu (N - 2){(N)
.N f(u) du c1 (N- 2)((N)
VL(N)
((u)
VL(N)
Therefore,
270 TsON S
.
(N 2)
S(N)
Σ
exp
1dm7a co
dz
VI(N)
2 ~
f(N)²
Σ
M
Vi
4):70 cD [(N) ~ 7o S(N) L(N) Σ
π L(N)
M
V
π
X;
i=1
jo f(N) v [(N) Ti(diag( (()dig(())/2) K)2/2) ~ 7o f(N) VI(N) Moo.5
To summarize, we have
T>ON S 70 F(N) L(N) 0 ~ 1
Bounding TSAN. Let L(z 470 N f(w)
du. Then
v VEww)
2nd
M
/
NNN - AK)
N fiu) du Vicuo)
TSgN= =
Σ iml
(.)) K2 Ku,)
f(z)
2nd
M
Tr TT(K,Kupp /rav --MiKRI(E..)) f(2)2d2
π
2
N
M
Tr TunKKK M(KI {()))
π
1=1
2nd
NNN
M
Tr(K K
Σ 1=1
6 QL. (nn [())
π
2
CaN
Tr(K,Ke -KQ(E.N) f(z)ddz.
π
61

---

Published as a conference paper at ICLR 2026
Using aresin approximation on K = arcsin(diag(K)) 1/2 K diag((K) 1/) we get
Tr K)()NN)) -
= - Tr(K K ())().))).
Using same contour representation method and deterministic approximation with Section E.2.2 we get
27
TSNN ~
Tr(K 1E
Tr
(
Jo
27
N
Tr
zie
2 ((.
(K ([1-111)-1 ( {x))dz
2Tí
TT
2
N
Tr
pizife (= R(2))) dz1 f(z)ddz
2Ti
Jr
2
NNN
M
Tr
2/C PeQ(3.M)2 f())2
π
2ti
Jr
Adopting the method in Paquette et al. (2024) same as Section K.I we get
Tr
2mi f Z7E (=Q(: ~ (P)((2, 2+ 2+1/(20) ~
2+1/(20)
It leads to
2% TS&N
/00h 1 (M2 f(2)2dz
M
π
M 2/1a/ (000 (Q(,,N)) 2211122a) f(2)²
~ noM
Finally,
11 M/(aaa 1'm (Q(2,N)) -211/(2a) f(2)² dz
2/(aa) /(4/) lon paN
f(x)²
NNN
f(z)
~ 10
dz + 70
2/(aa) M1/(4n)
dz
(
flu)
du (2n)2-1// 2-11((2a)
PN (
Flw)
du)2-1/(20)
Vi(u)
V
1/(2a) //1/(aa)
10
dz
PN-= V1(0)
+ VLNNN JNN f(u)du)2-11(2a)
f(z)²
1/(2a) 11/(40)
70
VIINN Jo (())du)2-11((a) dz
PN
f(2)² dz
20
(/(20)
1/2/(4a)
PN-=
+ Ni-e )2 1/(2a)
dz + 70
1/(2a) M1/(1a)
JpN
Ni-c
)2-1/(2a)
V((0) VL(ENY
VL(pNN
(L(pN) )1(20))1 (pN + Ni-c )1((20)-1)
NL-c
2/(20)
(L())((
VL(pN)
NI-c )1((20)-2
1/(2n) (max
70
VT.(pN)
S m
1/(2a)
-11011101) (1 < %
1/(2a) 1/(oo) -1-O0-11)
(100)
So we have
1/(2a) M2/(1a) 1011000))0000000-1111110011111
T TSgN S 70
62

---

Published as a conference paper at ICLR 2026
G.2
COMBINING TERMS
Combining the bounds,
2(2m+28-1)
L(N) S M (2 + (M
2a+1-25
M0.5 N (T(N) + +000 M N N-(1-o) (1- 20.
2(2++28-1)
We replaced the drift part with (M0.500. N)
2a+1-22 23
temporarily based on Remark and justify
this our selected parameters in Remark
Solving the inequality asymptotically yields
2(2a+23-1)
L(N) x M (2a+23-1) + (M-s
2a+ 2
M MMN--E
2c
2a 7o
M 4o
N (1--0 (1- 2/7
Finally, substituting = - and N = f/M yields
2(20+28-1)
R(M.f) & (- + M (M---U.)
2x+1-22
M 2e g-2-
M Za (1-c) (1- za, (-11-) (1-2a)
Optimizing over M gives bound of the form R(M*, S f ((,,) and we then optimize over to maximize h(a, B, e)..
G.3
OPTIMIZING OVER C, TO MAXIMIZE h(a, 3,c,e)
Assume throughout a > 0.5, $ < 0.5, and 2a + + 1. Consider the upper bound
2(2x+23-11
Ry (M, AM-22422-21 1)+ (M-
2a-1-23 +M2+20-20 20 8-20 2- 4a 2a 64+000(1111
-11-0(1-22
For large j. define
Rmin (f) = min Ry (M, f). MS0
We show Rmii (f) (a with h" (a, B) and identify ca 3), e² c" B), and M fm (a,3).
Logarithmic reduction to exponent balancing
Let M = 1m with m R. Writing each term as T gives
(m) = - -2a + 23 - 1) m,
(101)
L2(m, e) = 2(2a + 23 - 1)
2(2a + 28 - 1)
1-+1-26 +
+
(102)
2a 1
La(m, c, = m(1 2c 2e) 2c,
(103)
+ (1 c) (1-11))) 1n)) - -11-0)1- (11-1)
La(m,c,e)
201
(104)
2x
Thus minimizing Ry is equivalent to
min m.eER, 0<e<1
max L2, L3, Lz, La).
(105)
Introduce and rewrite as min m.c.e.h
s.t. Limm, < 1,2,,,444 0 0<c<1.
(106)
At an interior optimum (0 < < 1), constraints equalize:
L Ing L.3 T. h.
(107)
63

---

Published as a conference paper at ICLR 2026
Solving the equality yields
- 80.3 + 2a + + -1
C'
(108)
16a + 8a 6a - 23 +
Sa² + 16a3 - 4a - 4,3 + 1
e'
(109)
2(4rr -1
2( 1)
m 16a + 80.5 + 2a - 28- -1
(110)
2(4a 1) (2a + 23 1)
h-
(111)
16a + 8a, + 2a - 23
Feasibility. Since 0.5, denominators are positive. The condition e² 0 is equivalent to
2a
- Sa, + 2a + 26 - > 0 3 <
2(4a == (*)a),
which is stricter than 3 < 0.5. Moreover, e* holds automatically for 8 > 0. Hence, the interior solution is feasible whenever
2a - 2(4cx - 1)
0.5
BB (a)
with (a)
(112)
In this band,
M fm
Pmin (f) ~ gh²
with m², h² in (110) Note m² 0 and ht < 0.
Result As f - co, the choice M = fm with
2(4a 1)
m 16a + 8aB + 2a - 23
-8a, + 2a + 28 -
c²
16a + +a3 6a 23
e" Sa + 16a3 - 4a 43 +
2(4a -
h- 2(4a (2a + 23 1)
16a² + 8a3 2a 28-
is optimal for >> 0.5, 0.5 a < 3< (a). where
2a (a)
2(4a 1)
This choice minimizes (I, L2, L3, L4] in 105 Consequently,
Rmin (f) (a.88)
with h"(aah 3) 0.
Remark 4 (Justification on drift term conversion). Note that M = MM and N = pl-M* holds for the selected parameters.
For yN iterations the stable-decay scheduling behaves same as the constant learning rate. Let No be the crossover point in constant learning rate. Note that N 2 Ng holds, and N is asymptotically strictly bigger than No. So L(u) S 3 + 22-2-128-1 holds for 2 No.
Also for selected M- M. Z M 22-22 holds.
So have T(u) 76 M for > No.
64

---

Published as a conference paper at ICLR 2026
2a+22-1
20-22-11 2a
M 470
PN f(u)
2a
(L(-))
du
v L(uu)
+-
M0.5 4700
SPN f(u)
VR
du
No VL(u)
2a-+20
for
MoMos
du
No VL(u)
20-+-2
PPN
2a
7o.
du
No
M
2a 28-288
+2
(pN No)
2nx
N
2a
For selected parameters -M N f -
M c², and 70- = - following holds.
2a+28-1
N
2a
2
20-28-1
As (M0.0QL(N)
Zcc
18MN-20 -c" replacing the drift term with a proxy does not alter
the argument.
G.4 ANALYSIS FOR WARMUP-STABEE S-E---TAAALLEEEE DECAY
The analysis for warmup-stable-decay is almost identical to that for stable-decay. The only differ- ence occurs in the step leading to (100), but the final bound is the same. Thus, the loss bound for wanmup-stable-de
matches that for stable-decay. We provide the corresponding analysis to the procedure (TOO) at the end of this subsection.
Finally, the bound
2(4n 1)(2a 1) +8a2+883426
Ry "/.""//M""""""""""" (M*)=+*) $
(113)
introduced in (55) also holds for warmup-stable-deea.
For the warmup-stable-deat schedule, set the learning rate to 7 rof(k) with
k/ k/(wN),
A wN, wN < k S pN,
s(k)
1. -
(114)
(1 ((N)) P,
> pN,
where w,p,c E (0,1) 1) and > 0. is the ratio for the warmup stage, and assume that w is smaller than p/2.
Following is the corresponding analysis to the procedure of 1100
12 >/ 1/4o)) (- (Q(2,N)) - 2+(2(2)) f(2)² dz
1/(2a) 1/2/(40)) joar (S VL(u) 7u
f(z)²
1/(2)) M1/(4a) fon ( f(w) vi(w)
f(z)² du 2-/2)//
dz + 70
dz
((w)
duy 2- 1/(aa)
1/(2a) M
NwN
1
20
dz
PN/2 VI(0)
VI(NN S f(u)du)2-11(2a)
65

---

Published as a conference paper at ICLR 2026
+ 70
1/(2a)
M1/(4n)
Spx ((u)du)2-11(2a) dz
(DN-2 V1(0)
+
VI(NN)
NN
f(=)²
1/(2n) M2/(1a)
VMANN SaN ((1)2))2-1/((a)) dz
PN
1/(20)
1 (//(4
1
1/(20) M 1/((aa) smm.
1
2 70
dz + 70
dz
PN12
Ni-6 12-1/(2a)
JNN (PN-2
+ Ni-
)2 1/(aa)
(1(0)
VIANN
VT10)
V((NN)
PNN
f(2)²
+ 70
1/(2n)
M1/(40)
dz
Ni-c
)2-1/(2a)
[PNN)
Seeee (eall)
1/1200
IV
PN/2
-2+1/22a)
VL(O)
22 + (4a) (0220t
/T(O)((
Ni 1/(2a) (pN WN
N
)1(1aa)-1)
VL(pN)
VL(pN)
ii ( O0+
(1 22,00)
Ni-c 1/(20)-2
VL(pN)
2/(20)
(/(20)/1 M N- 1/(2a) M (1 1/(0)))
(1N)
To
1/220) (21((4) (111
G.5 SCHEDULING ON SGD
In this subsection, we explain that the scheduling does not lift the compute-optimal exponent of SGD in the Phase I and Phase II. Assume bounded scheduling function f, and define F(k)
Jo f(=) d2.
Ferbach et al. 2025 proved
MM, N, v0) 2 M max
(N M (N 1++1/(2n)
for the risk R2,M,,, 70) with general bounded scheduling function f.
Since f is bounded, we have F(N) X N. Therefore,
Ry N, To) 2 M max+max(011-2
(20 - (xN
20-+ma(0,11-25) + (7o.V) (204333-2004- M M (70N)
Ri (M,
where R1 N,
yo) is the loss under constant schedule f f= 1.
Thus, scheduling does not improve the compute-optimal exponent of SGD in Phase and Phase II.
66

---

Published as a conference paper at ICLR 2026
H ANALYSIS FOR LINEAR DECAYING SCHEDULING AND COSINE SCHEDULING
H.1 ANALYSIS FOR LINEAR DECAYING SCHEDULING
In this section, we analyze the following linear decaying scheduling.
(1 1
f(t) =
(115)
N
N
It decays from 1 to
1 linearly. VN
We will focus on Phase Aa, and follow a similar procedure stable-decay scheduling.
Note that we have to handle the following equation, where Q2(z)= = 1Ta
So
f(a) du.
V((w)
2n- 20-2881
L(N) = M
(222-22011 + (M
2a
270 +
Vi
N
expl
exp(-42.10
42,20
N
S(u) du f(z) dz.
π
[(u)
2x
In early iterations the drift term (Mu.3
2n
dominates. Solving L(N) ~
2a+22-1
(M
2a yields
2(20-28-1) 2a+1- 2/
F(N): = J(" [()) du.
L(N)=
For linear decaying scheduling 2(2o+28-1)
F(N)
N holds, so the drift term becomes
(M.s
2-
Now we move to the noise term. We split the noise term moisee (N) as
2nd
N /VN
S(u)
noise (N)
V
exp
42:30 Tt
VL(u) du [(=)²
dz
π
2 +
M
N
f(u)
V1
exp
4di7m
VI(u) du f(z)² dz =: TSNN-VN) + +T +TZ(N-VN)) VNy
π
N-VN
Bounding T "(N-VVX
Note that f(N) ~ f(2) holds for (N - VVN < < So
f(u) du [(3) = S(N)2
N
f(u) du )z) dz.
exp 4A,70
expl
44:20
(N-VN)
L(u))
(N-VN)
VL(u)
There exist constants
co, C1 > such that for (N -VN- <
(N 2)f(N)
N f(u) du S c
(N 2){(N)
co
$
v S(NN
V(uu)
VL(N)
Therefore,
2nd T>aN <
N
(N- (N=)][(N)
J(N)²
Σ
exp
42rn0 co
dz
π
(N-VN)
VL(N)
i=1
2
((N)
V
H(N)²
Σ
4A. cu S(N) Jo S(N) VL(N)
X
30 S(N) v L(N) Th Ti(diag(A)) (a)iaa(())))) 12)
= To (NN) VL(N) 0.
To summarize, we have
T TNN-VN) X 70 S(N) (((N) 0.5
70 12/2 /
67

---

dy. Published as a conference paper at ICLR 2026
Bounding T Ts(N- /Ny
Let (22
tho Ja f(u) du.
VT(a)
By the same procedure as the stable-decaying case, we can get
2%0 (N-VN)
S(N-VN) ~ M
())) f(=)2
π
(N-VN) (Q(2, - [(2)2 dz.
/2/(da)
2 70
We have
(N-VN)
10 1/(aa)
(Q(2, 2+ 2+1/(2)) s(=) dz
(N-VN)
f(2)²
1/(2n) M2/(4a)
70
dz
(/2 f(u) du) (/))2-1((2)) V((u)
(116)
(N-VN)
f(=)²
1/120) M2/(4a)
S 90
de
( f(u) 2 2-1((a)))
(40)
(N-VN)
f(z)²
1/(20) M2/(1a) ~%
dz
U f())du)2 2-1/(2)) 1/(2n)
Let the integral term be I. First, we use the change of variables N u, which transforms the integration interval [0, N - vNJ into [vN,N] NJ. In the regime of large N, the linear schedule f(N u) can be approximated as
f(N - u) =1 1 (1-
1
N N
1
u
1+
(117)
VN
N
VN
VN
Using this approximation, we evaluate the inner integral in the denominator:
J ((8) L VN
1+
v
dr
u VN
1
(118)
VN
N
Substituting these terms back into I, obtain
)²
N
1
VN
VN
I
du.
(119)
+
2/N
To decouple the dependency on N, we apply the scaling u V Ny, which implies du VNdy. The integration limits change from [VN, N] to [1, vNJ. The integral then reformulated as
(1 + y)²
I
VN dy
(0(1 8/2))² 20
(120)
(1+
VN
y² 3o (1+ 1/23222 zm
The asymptotic behavior is determined by the convergence of the remaining integral. As y oc, the integrand behaves as
y²
1 2-2 1 1 1-2 1
(121)
2 Iu (y/2)
Integrating this term from to VN leads to the following cases depending the exponent - 2:
- No--
if 1 - >> -1 if - a= ir i
1,
I
In(vN) IN
(122)
N
const
a>1
68

---

Published as a conference paper at ICLR 2026
Simplifying the final exponents, get the asymptotic order:
0 N के )
if <0.5<a<1
I ~
0
N à ln
(123)
0
N-
1.
For 0.5 < a < 1, we have
T S(N-VN) S %0
1/(2o)
M (1-11a11--111111a))))
For 0.5 < a combining the bounds for the drift term and noise term, we have
2(2a+28-1)
L(N) x M (2(22+23-1 + (M..3370N)
2a+1-28
1oM55N-11 ((N) + n M 4m N --11-1 2202
In intersection of Area Aa* and 0.5 < a < 1, with choice of e" in 7o = M and c* we used for stable-decaying scheduling, we have
2(2a+23-1) 1-
L(N) S M (2 + (M
+ 1o..0 VL(N) + 20 M (1 22)
So in intersection of Area Aa' and 0.5 < a < 1, we have
2(4a 12
Ry(M", FC(M",fM",,, (M
16a +2a3+2a
(124)
Therefore, linear decaying scheduling has an advantage compared to constant learning rate in the intersection of Area Aa" and 0.5 < 1.
H.2 ANALYSIS FOR COSINE SCHEDULING
In this section, we analyze the following cosine scheduling.
f(t) = 1 1/N
1/N
π
C08
(125)
2
2
N
decays from N
We will focus on Phase Aa, and follow a similar procedure to stable-decay scheduling.
Note that we have to handle the following equation, where Qz(z)
1Ta
Jo
f(a) du. /L(u)
2n 20-2,8-
L(N)
M (22+222-1)
+ (M
2x
2 +
Vi
N
expl
42,70
N
S(u)
du [(2) dz.
L(u)
2x -
In early iterations the drift term (M0.52((N))
2a
dominates. Solving L(N)
2a-23-1
(MO.),((N)) 2a yields
2(2o-28-1)
F(N) :oND=-/ 1 N
L(N) = (M0.5 20 F(N))
2a+1-29
f(u)do)
2(2a+2/-11 2a+1-
For cosine scheduling F(N) N holds, the drift term becomes (MM..50,,))
69

---

Published as a conference paper at ICLR 2026
Now move to the noise term. We split the noise term m [N))
as
270
M
N-VA
N
f(u)
noiee (N)
V1
exp
47.00
f(2)22z dz VL(u)
du
π
270 +
M
f(u)
V
expl
42:70
VL(u) du [()) dz =: T: S(NN-V + +T>(N -NN-V
π
N-
==
Bounding T SNN-VN)
Note that f(N) f(2) holds for (NV VN) < N. So
[(u) du f(=)2 = S(N)2
[(u) du dz.
exp
4A:70
exp
44:30
(N-VN)
VL(u)
(N-VN)
/ LLu))
There exist constants Co, C1 > 0 such that for (N- -VN <E<N
(N xN
$ S N
f(u) du S a VL(N)
(N 2)
cu
VL(N)
VL(u)
Therefore,
270 [(N)
M
1dam (N 2)5(N)
ToNN= S
Σ
V/
exp
dz
VL(N)
(N-VN)
i=1
27d
π L(N)
V
S(N)²
Σ
Vi
47:70 co S(N) To F(N) VL(N)
π
~ %0 f(N) ([(N) Ti (iiag(K)
~ 7o f(N) v VL(N)AM M
To summarize, we have
T (N- I To f(N) VL(N) M ~ To - ( L(N).
Bounding T TSNN-vi /NY Let Q(2,N) 4ne N f(u) du. VT(aa
By the same procedure as the stable-decaying case, we can get
270
(N-VN)
T TSNN-VN)
M
())) [(=)
π
(N-VX) (Q(2, ((=)2 dz.
1o M/(ida
We have
(NN-NN
2o 2/(aaa)
-2111(22)) {(=)² dz
(N-VN)
ʃ(z)2
1/(20) (2/(
%0
(
f(w) du) 2-)2-11(22)) ((()
dz
(126)
1/(20) M1/(1a)
(N-VN)
[(=)
dz
(NN flu) du) 122a(
Vi(0)
(N-VN)
f(z)2
1/(2a) M11(40) 70
U.N ((u)dd)2-1//// 1/(2a) dz
Let the integral term be I. First, we use the change of variables 2 = N - u, which transforms the
integration interval [o, N vN] into I/N,N, The integral can be written as:
N J(N x)2
I JN (JO J(N - v) du) 2 20
dx.
(127)
70

---

Published as a conference paper at ICLR 2026
We evaluate the asymptotic magnitude of I by analyzing the dominant contributions from the lower limit (xr R vN) and the upper limit (z R N).
Contribution near the lower limit (a VNy: In the region where æ is small, the learning rate approaches its minimum, S(N - x) 2 Consequently, the cumulative sum scales linearly with the inverse of N, i.c. Jo f(N - 2) du 2 N.. Substituting these approximations, the integrand becomes: (1/N)²
(a/N)²- N N2.
N - 2+ -t
(128)
Integrating this term with respect x near the lower limit VN:
N 2a (VN) N N
(129)
Simplifying the exponents yields the scaling N dtc
Contribution near the upper limit (æ 2e N): In the region where æ is large, f(N - x) ~ O(1) and the cumulative sum scales as (zx). The integrand is dominated by x =- (2 2.), Integrating this term near the upper limit N:
1+
(130)
The asymptotic behavior of I is determined by the maximum of these two contributions. The con- tribution from the lower limit dominates when 1 Otherwise, the contribution from the upper limit dominates. Thus,
ta > -1 + 20 which corresponds to a >> 1.5.
N N
if 1.5, if < a < 1.5.
I
(131)
For 0.5 < < 1.5, we have
T TIN-VN S 70
1/(2a)
M 1/4aa) (1-111((2))
For 0.5 < a < 1.5, combining the bounds for the drift term and noise term, we have
L(N) x M (20428-1) + (M0
2(2a+28-1) 2a+1-22 23
70.55 VL(N) 70
M 4a N (1-
1
(1
In intersection of Area Aa" and 0.5 < a < 1.5, with choice of e- in 7o M-e and e* we used for stable-dccaying scheduling, we have
2(2
1
L(N) S M (20+29-1) + + (M..5
2a+1-28
+ 10.5 VL(N) + %
2a M (-(1-c")
So in intersection of Area Aa" and 0.5 < a 1.5, we have
2(4a 122
Ry f/M*, (M*)~= S
16a7 22
(132)
Therefore, linear decaying scheduling has an advantage compared constant learning rate in the intersection of Area Aa" and 0.5 < < 1.5.
71

---

Published as a conference paper at ICLR 2026
10a
10-1
magnitude
10-4
10->
empirical-ag (slope -0.91)
10
0.5 empliical-avg (Slope 1.222
empirica-aav (slope -1.47)
10
10a
Index
Figure 23: Decay of gradient under the basis of U. Colored solid lines show the average of gradi-
ents under the basis of U for the parameter (a, 3) = (0.4, 0.5), (0.6, 0.5), (0.8, 0.5). On the legend, only noted the The dotted line is fitted for the average of gradients, and noted its slope in the legend. Slope similar to 2a within error 0.13.
I ANALYSIS ABOUT HYPOTHESIS FOR THE POSITION OF THE BENEFICIAL AREA
In this section, we cover the analysis of stochastic gradient decay, which was deferred from Sec- tion 5.1
We examine the decaying structure the stochastic gradient. Assume a feature vector x: is drawn from the distribution (0, H), and its label is y (x, w") Then the stochastic gradient for that feature vector is
g 2((Sx,, O 0,-1) v) Sa. 2Sxx T ( - 0*) 2Sxx T W..
Taking the expectation of the stochastic gradient and using SHw_ = 0, we obtain
[[g] 2SHS (0 -0) HW1
2SHS (0 0").
Lin et al. 2024) proved that the eigenvalues A of SHST satisfy ~ 12a Let the eigenvalue decomposition of SHST be SHS U diag UT Then
U U'Elg 2 diag X,) (0- 0*),
which provides the intuition that Eig], expressed in the basis of the columns of U, decays as i i-2a Figure shows that the expected gradient decays similarly to i 2a Also, note that a larger a leads to steeper gradient decay.
72

---

Published as a conference paper at ICLR 2026
J SCALING LAW OF ADAM WITH HEURISTIC
First, we recall the Adam (Kingma & Ba 2015) update and notation. For the stochastic gradient
gk (S 0k) - yk) San.
Adam maintains first and second moment estimates
m, 51 mp-1 + (1 (1-31))m 31) gk:
U/. = B2 uk-1 + (1-32) 9k 62
with bias corrections m m 3 0b vu/ - -B4). The update
0 a++1 02 7 m. (e1 6k) 11/2
where denotes elementwise multiplication and the (-1/2) 12) power is taken elementwise; the usual damping (we will set
0 in the asymptotic analysis).
Xiao et al. (2055) proposed a heuristic for Adam: take 32 sufficiently close that the second moment can be treated as expectation.
We present results under same heuristic. In addition, Ferbach et al. (025) prove that SGD with momentum obeys the same scaling law as SGD; motivated by this, we set $1 0 and omit the first-moment term for simplicity.
Second-moment proxy and normalized update.
Under the heuristic Xiao et al. 2025
Gr ~ 41 (Sxx) ((Sæg, 0k) yu) Fu
Let := Swg be the sketched feature vector. Under the Gaussian assumption æg. follows multivariate Gaussian distribution with covariance K SHST.
First, we analyze the residual term. Using the decomposition w- S 0" + the residual at step is:
(Say, Oz) yA x =ISTO 02 æg w²
TTSTO O2 aI +
(Sax)) T T(ps 0³) x
= CK+
where 6g := 0g- 12-00 0" the parameter error, and C& == æ. wi is the irreducible residual term induced by approximation error.
We focus on the i-th coordinate of the second moment vector. Let X == (E); and Y = E 5y - (the residual). Since both are linear combinations of the Gaussian vector ær,, they are jointly Gaussian. We apply Isserlis' theorem:
E
We evaluate each term:
1. Variance of the feature (E[
E[(() (SHST)& (diag( K)):
2. Variance of the residual (E[{²2): By definition, the expected squared residual is the pop- ulation risk:
E[ = EE ((S O2) sx) L(
3. Covariance term (E[XY]): This term involves the correlation between the feature and the residual.
E E (E):(ET on ca)
E (2) 6x) E (Sxx) (w)
73

---

Published as a conference paper at ICLR 2026
The first part is the standard covariance calculation:
ba) (S S sxx); (SHS' ok); (Kos);
The second part vanishes due to the orthogonality property of the projected solution (SHw) = 0):
E (1)
(SH w11, 0.
Thus, [[XY] (Kr))..
Substituting these back into Isserlis' formula yields:
E x (diag (K), T((0)) 2((((() - -0"))))²
Stacking the coordinates gives the following equation.
E (Sxx) ((SIg 0k) yk)² Fk diag(K) L(Ox) (( - 00))
(133)
Moreover, by Cauchy- Schwarz,
(K(O, 3 < Kú -0 0*) < K5j I((k),
and hence the exact second moment admits the coordinate-wise bounds
diag( K) I(OO)) v |P| (Sxx) ((Sas; 0x) yx))² Fs v 3 diag K) I(Ok),
(134)
where < denotes elementwise inequality. In particular, replacing the exact second moment by
diag () ((x,))) = diag(SHS)) L(Ox)) changes the normalization by most universal constant factor.
Hence, the (elementwise) normalized update satisfies
(SSxg 02) yk) Sxx
1++1 02 ~ok (SHS) L(0x)
One-step update formula.
Recalling the Taylor expansion used for signSGD,
B[u(0x+1) - ((.)) Fi] Ie| (Vy(Ox), Oc+1 0z) Fa
+
E
V q, 1 -00822))82) Fic]
Gradient term:
B (VV(Ok) ax+1 0x) Fu] - (Vo(Os). diag S
SHST 0% SHw"
Quadratic term:
B
(0 - 0x) 02) Fi
V2gg, diag (S 1/2 Sx.ak diag(SHS))-117 ((Sæg, 02) yyk) I(Ok))
B
Fr
72 L(Ox)
((diag(SHST) sia SHST L(Ok)
diag(SHS")
+2(HHS Siag(SHS)) diag(SHS 1/2 SHSST (0x - -2)82))
Combining the two contributions,
- Fx]
(Vq(O)), K(O. - 00)) + 2
( K,)
(L(O2)
22
(SHST diag - diag SHST)-1/2 SHST, (0, - 02))
+ I(Ox)
where K, di S -
74

---

Published as a conference paper at ICLR 2026
Mode-wise recursion.
For (1))) == (8,e - (K)) X wi) (0x 0*) (cf. Appendix E.1
27k >: \:( + 2h Kw: Ku,,
2
E/r 1 [x))
(:( ri(k)
vL(00)
L(
27k.
27
A (K) (k) + c ( K Ku,)
VL(Ow)
L(Ox)
We now assume f 1, and 7i = To for simplicity. Passing to the ODE limit as in Section get following ODE for P(t) L(t//ro) and p,(t) r;(t/%))
E.2 wc
70 dp; 2 (P(t)
((K) 70 V/'
(135)
P(t)
where V/ : : K Ku.,
Interpreting the solution of the ODE an implicit integral equation and summing over i, similar Section E.2 and writing
N (VLT)) 7o [(w)
1
Q2 270
du,
we obtain the implicit integral inequality for some > O.
H 1/ w 11 +
(0) expl exp(-cc (e((-11100((1))
M
Vi
expl
-211000 J (V1.0) -2 [(u)
1
du dz
re(0) expl (c22((2((1)) 1/2 F 1|2 + Σ M
H $ L(N)= <
1=1
M
N
expl 2022/00 1. (V()
2 + 70
V
7o "[(u) du dz.
Drift transformation and limit phase. By the same drift/approximation transformation as in equation 35
2a-28-1 2a
M 20 + (Aminn(....b) Q2(N))
+ Laxus 820.5 M (M/min(a.0.5) Q2(N))
+
70
2
M
V
N
-201
12t (V()))
fimi Liu
du
f(2)² dz
i=1
20+223-
< L(N) & M-2a2 20 1-2)) + (M/minfaa,0.) (2(NN)
2xc
+ Lasus, B20.5} -- (A/miu(a.... 0.5) Q2(N))
-1+
2n
+
70
M
V
-2220 1.
VTang -o Lim) ) du
f(m)
nww)
f(z) dz.
We will first handle the limit phase, similar to Section P(I) + Toxe we must have
E.3.2
At stationarity, let pt(t) + and
-2
70
No >i(K)
s: V=
T0 Lo
V
1
(w. T K, KKKK)
1
/T.o
Loo
1
20
(x)
1
20
V
75

---

Published as a conference paper at ICLR 2026
Using the loss decomposition P(t)
E Ps(t) + |/ w obtain
Loo
8,
+
1 1/2
w
70
2
M
w;
{x(K)
K Ku
L.0
70
+
w
1=1
1=1
vx
70
Ti(diag( K) 1/22 K-) ()), VTV.
T.oc
H 1/1111111
2
30
Via
70 Tr Lox
1H112W1
2
70
V
Then we get
Loo max TY (K(daa(K))//) |/ w1
[2] [1
max 70 2-2-2mn(aa00p M 20-20 1-29) }.
So have the same floor for signSGD.
Since f is bounded and L(N)= 2 10M 2mm.000000
f(w)
VL(u)
2 M1-min(a,0.5)
fu
(of(u)
L(n)
so the subtraction inside Q2 is asymptotically negligible and Q2(N) Q(N). Hence, the drift contribution coincides with that of signSGD.
Scaling law (constant learning rate). For f = 1, Adam (under this heuristic) follows the same scaling law as signSGD:
2(20+22--)
R(M, no) = M 2a + (Muiin(a,0.5)
70)
-2-+1-2
6a-1 +(M 4a-2 70)
2(2a-1)
20+1
+ +70 M 2-2 2-2min(..0.5)
Since the loss formula R(M, No) %0) is the same signSGD, the compute-optimal scaling law will also be the same signSGD. So we expect that Adam has the compute-optina scaling law in Table [] Figure 24 shows that exponents the Table [ and measured compute-opting loss slope and optimal model size slope (in log-log plot) for Adam match well.
76

---

Published as a conference paper at ICLR 2026
(oc (0.80, 1.00), 0.399, Theory: >') (0.80, 0.20, 1.00),
Measure: 0.384, Theory: 0.384
15-1
Optimal model size
102
665
10
Compute Optimal
Optimal
Fil: M FL0P
FLOPs 190 -0"00.998)
10P
12
FLOPs (6 (6-M--M N)
10"
10
10
107
Flops (6 Mi N)
10"
10
e²) (1.00, 1.00), Measure: Theory: 0.333
a'1-
Theory:
Optimal model size
10P
Los5
10a4
Compute Optimal Points
Optimal model
Hitt Loss FLOPs-9.11 (R² = .999
Fit: M FLOP
19-
10
10
FLOPs (6
10"
10
10"
Mlops (6
10"
(o. -0.000000 0.40. 1.00), Measure: 0.444, Theory: 0.455
e'1 10.60, 0.40, 1.00). Measure: 0.500, Theory: 0.455
10
Optimal model size
10a
Compute Optimal Points
Optimal model size
Fit: F.O
Fit: Loss FLOPs 19'-1.000 (1000.000)
16
(o. B. e=)== (0.70. 1.10. 1.09). Measure: 0,703, Theory: 0.722
1a. e (0.70, 1.10, 1.09). Meesure: 0.616, Theory: 0.611
10a1
Optimal model size M
10
LO55
10"M
13"4
109
Comoute-Optima Points
Optimal model sice M
15-3
Loss FLOPy-6700 (R*=0.999)
FLOPs 16
(6.
Figure 24: Measure of compute-optimal loss slope and optimal model size slope for Adam. We
validate the exponent of R M -f- 70) and M with respect to f in the Table The left plot shows the compute-optimal loss with respect to FLOPS 6MN. The right plot shows the optimal model size with respect FLOPP 6MN. Each plot includes the measured slope and the theoretical slope from the T Parameters 3 = 0.9, B2 0.999, 10-8 70 0.002.
77

---

Published as a conference paper at ICLR 2026
K OMITTED ANALYSIS FROM SECTION E
K.1 OMITTED PROOF OF 31 AND 35
Note that the proof is almost similar to Paquette et al. (2024), but we cover it briefly for completeness. Refer to Appendix F, G, H of Paquette al. (2024) for more details.
In this section, we omitted proof of (31) and 33
is enough to prove
f Q(N)= (f(z), 22 dz M 20 max 1-2
2mi Jr
2a-22-1
(Mmin(a,,.)) 0.5) Q(N))
14300.,030003 M- (Amiiu(a, 0.5) Q(Ni)
-1+
From now on, will use similar notation to Paquette et al. 2024), except in the inevitable case, to facilitate easy comparison for the reader. Note that we use M and for model size and initial dimension before projection, while Paquette al. (2024) uses d and v.
We use I for the contour containing the spectrum of K, while Paquette et al. 2024 used F TUT for that, where T small circle containing the origin.
Let
1 F(N) == 2 (12 PaN(NN P.Q())) dz.
(136)
The exponential kernel QNN) replaces all polynomial weights in the analysis of Paquette et al. (2024). The resulting leading orders remain the same while constants and exponents are altered in transparent way: precise statements follow.
We can split the F(N) by splitting the keyhole contour r. We let
F(N) (N) + Fcaps (N)) Fc(N) (lower-order),
(137)
where Fo collects the small circle around the origin, Feaps collects the right/left caps adjacent the positive real axis, and Fc collects the central are close to [0, 1]. Refer to Appendix F of Paquuette al. (2024) for more details about the picture of contour and decomposition of contour.
In the following proposition the function (x)+ == max(æ, 0) is used.
Proposition K.1. Fo(N) is independent of N and obeys
j 20 28
[o(()) Σ
S CMM (28
1+j 2 k(d/M)
Sketch. Putting = 0 to the exponential leads to 1, so we can reduce to the analysis of Paquette al. (2024). So the error bound identical.
After
this
Fo(N) 22
M 20 max 1-28) holds by identical procedure calculating
Ej= 14j-2a 2520 x(d/M)'
Proposition K.2. There exist functions f.g 2 with
f(N) < Cexp(-paQ(N)) -PaQ(N) M 2n), g(N) s Ceexpl-pp((
so that
| capss (N) S CS(N) 2a+(1-29)- + Cg((N)
Sketch. Use [m(2) - 1| S M min(2a.1) (as in Paquette et al. 20244) on a cap pushed ((1) to [0, 1] to replace (R(2), (2 by simple partial fraction, and control the remainder by the real part of
78

---

Published as a conference paper at ICLR 2026
The main contribution arises from the are parameterized by z(u) + in(() with E [M-22
and In(2)] < u. Along this are we have the uniform approximation
m(=())) (1 2/ (c(u) + 1/(2a) -1120) M-1 I e 11(111121
(138)
for some bounded real c(u). Inserting (338 in R(z) (-2I m(=)H)-1 and extracting the imaginary part produces two canonical integrals,
Pp (N) 2a ! (28-1)/(20)
- PaQ(NNu du,
Fac(N) == CB
u -1/(2a) M PaaQ(NN)
du,
2a M-220
(139)
with cs = - Proposition K.3. There exists C' 0 such that for all NN 2 0, [o([N) S CV)
j 28 23 and C3 = 0 otherwise.
Moreover, there are A > O and a bounded function C(N) > 0 with C(N) I 1 +E whenever paQ(N) [A, M /A], and
1 C(N) (Fm( + Fac(N)) Fo(N) ((N)(Fm(N) Fac(N))
Sketch. Parameterize L'c by u and use (38) to separate real/imaginary parts. The imaginary terms integrate exactly to 139 while the real part smaller by factor (e)) since (n(2)] < u.
Proposition K.4 (Asymptotics of ppp). Assume 2a 2, and set X PaQ((N). For any 0 there exists A > such that for X 2 A,
(N) (p) S (Np),
where
(P) = (2a)-1X (1(488/))+1/221 r( 1
2a +1)
Moreover, if X < then c < (N) < C for constants c, C > 0, and if X >
A then
(P) S Č Fo(N) for some C>
> 0 independent of M.
Sketch. With the change of variables w Xu, we get
x w (220))) ddw.
(N) (2or) x (1+8/00+1
Comparing to the complete gamma integral yields the relative error bound in terms of the upper incomplete gamma tail, which can be made S E by choosing large. The remaining bounds follow by monotonicity and elementary estimates.
Proposition K.5 (Asymptotics of Fac). Let x= = PNQ(N). There exists C(a. B) > 0 such that
(F)(NN)
28 1, 2a 1,
Fac(N)
0,
23 < 1.
If in addition 2a > and 2B > 1, then for any e > 0 there is A > 0 such that whenever X E [A, 22a /A).
-N) (Nc,,), (Nc(N)
Σ
28 (2a)-1 - 1 x 1+1((2a)) M-1
j=1
Furthermore, for any Ā >>> there exist constants C, (independent of M) such that
JCM-1. X < Ā, cFo(N), X >> 2
Fac(N)
Sketch. Compare the truncated integral in (19 with its extension [0, oo) and control the two
tails [0. M -2a]] and [1, separately. The first is at most 2M- the second is bounded by M -----
Normalizing by (N) shows both are relatively small for X [A, M /A] with A large. The endpoint bounds follow from dropping the exponential and from a crude -e-Xu du X-1e-XM-- estimate when X 2 M2a
79

---

Published as a conference paper at ICLR 2026
Finally get
$ =)Q (f(z), ,02) dz Fo(N) + (c) + Fc(N)
2mi J
2a+28-1
M MM2atmax(011-29) (paQ(N))
2a
+ 1 (030.6,330.3) - (pa (PaQ(N))
-I+
2n
M
+ max
1-28)
2x-+28-1
~ M 2a 1 + (Mmin(a....) 0.5) Q(N))
2a
-1+ 1
+ |4300563005 Mminn 0.5) Q(N))
2a
K.2 NOTE ON THE arcsin = APPROXIMATION
We explain that it is possible replace the linear approximation arcsin x As x by an inequality, and the main results of our paper remain unchanged.
Replacing the arcsin-linearization by a uniform sandwich. Fix 0 < p S and define
arcsin f.
aresin arcsin p
(p()) inf S
1, c2(p)
= sup Sr
S
For x Rd with ||=] S p. the entrywise odd and monotone map aresin satisfies the componentwise bounds
ci(p) x s arcsin(x) S c2(p)
In our update, put = 0% 0* and
K Uk
Ik =
so that arcsin(xx) = DE ok.
(((O))
for some diagonal D3 diag(Fk, kk,a) with ci(p) S Kk.j S c2(p). Using KT K and K K = K KT, the one-step drift can be written
2%k
21 (w. K Ku.).
B|ri(k (kk - rx()) [x) π (V²)))
o (Kuww, + w.u Dr K un +
Since Dg is diagonal with (p)) - Dr I T, the quadratic form is sandwiched between the same expression with D& replaced by cp)T and cz(p).. Recalling the identity used earlier,
of (uuw + w.u, K) K uk 22)(K) r(()),
we obtain the two-sided one-step bound
4 cc2(p) Th TT((Op))
27 (w.KK.ul KgKK,) s B[r.(k+1) rk(b)) Fil
(K) (k:)
2n (c) 7k ((K)r((k) + (w) K Ku:).
/ T(Op)
Consequences for the ODE limit and the implicit integral equation. Let n = (k), = kyo- (1(1) == (kk), and P(t) = L(Ox), as in Appendix E.2 Then we obtain the differential inequalities
(p) ( f(t/ro) (t)) 2nd TV((()
4c1(p) X(K) f(t)0))))))))1)
270
f(t/no)3V, < p.(() < *VP(L)
f(t/no)
with V = w. K2 K u;. Solving these linear comparison incqualities yields the bounds
p (0))(1) < pc(1) Pi (()
plez) & P(t) < pla))().
80

---

Published as a conference paper at ICLR 2026
where p o and (.) denote the solutions of the ODE/integral equations from Appendix E.2 with the factor replaced by Equivalently, defining
Qc(N) = 4c 70 Lo
f(u) du,
VP(u)
the drift/noise expressions remain valid with Q(N) replaced by Qc(N), and all proofs carry through verbatim.
Only multiplicative constants change; scaling exponents and phases do not. Every appearance of QN) in the final formulas enters either through exponential e AQ(N) or through polynomial factor (M*Q(NN) - Replacing Q by Qe = cQ only multiplies these terms by constants: e XcQ converts (M"cQ)
(MM"Q) -² Hence the rates, exponents, and phase boundaries of the scaling laws are unchanged; only the prefactors are rescaled by fixed constants depending on ci (p), (2(p) E / /22. In particular, all "~" statements (equalities up absolute constants) remain valid with the same exponents.
K.3
NOTE ON APPROXIMATION ERROR
Though proof Paquette al. (2024) implicitly implies
H 1/2 wil M 202a--mx(0. 1-20)
was not explicitly specified. we clarify here.
First,
2K=21)-1 (H /2 w 22) )02) dz M -ax 1-28))
2 |=|=s
directly implied from Proposition H.3 of Paquette et al. (2024). So it is enough prove the following claim.
Claim. Let
K= 5 1/2 w" = ST wi. SH w. = 0.
For sufficiently small circle |2| enclosing only the eigenvalue of K,
K =1) =1)
(H (22 82 de = | | w
2ti
|=|=2
Proof. By the Riesz projection theorem (Dunford-Riee functional calculus), for small circle |2| enclosing only the eigenvalue 0 of K,
Ib = 2xi
(K -1 de
12/=c
is the spectral Riesz projector onto the O-eigenspace; since K is Hermitian, Ho is the orthogonal projector onto ker( ()).
Then we have
(K-2I)-1
(K
-11 (H (11 (1"02000002) de = (Io. 1/2
1/2 122 )))))
[I H/2 1/2 w 11
2 J=1=
Since ker K K) = (x: S / -0- = (Im( (1/
We have the orthogonal decomposi-
tion
1/20== = 111 0" +
1/2 w
E Elm(H Elm(H1/25T)
(Im/ (1m(H//2TT))
where the second membership uses S H w 0. Hence HI H 1/2 12ww"
H 12 and therefore
(o, (H 1/2 w )2))
2/22 w
81

---

Published as a conference paper at ICLR 2026
K.4 PROOF OF MATRIX INEQUALITY FOR S" )
We will prove the inequality in the following form in this section.
C1 _Mmin(05..a) N diag(SHS')-112 1/2 3 02 MMmin(U.,,a)
Setup. Let S R xx have i..dd entries Sty (0, 1/M), and let
H diag( 1 12a 2 2a
d d=20),
a> 0.
Then, for each E (1....,,.,,, M
1 d j
diag (SHST) = Σ
2a X3:
=
M j=1
j=1
where xi...., X2 are i..d. (1).
Remark 5 (Rough intuition for what we will prove).
{ M-
a a> 2
Σ
2a X3
= M
1--d1-20
~ M-2a a S with d d= M,
1=1
So, want obtain diag(SH (SHST) ~ Mmin((.5,.)).
Define
Sa(a)
20
ll = M Sa(a).
i=1
Hence, any probbbbbobyilityy upper/lower bounds on Sa(a) translate into corresponding bounds on diag(S (SHS) -1/2 via
1 Sa(a) SU U
(SHg)) -1/2
M I,
M
M Sala)
2 L
[diag (diag(SHHT)
-1/2
M I.
We consider two regimes and then unify them through Mmmin(0.5.n).
REGIME I: 1 (SUMMABLE WEIGHTS)
In this regime, E ((2a) < cc. Write X, = j= - -1), so that
Sa(a) [[Sa(a)] Σ X₁: E|Sa(a))
Σ5 20 < <(2a).
/=1
1=1
Moreover, Var(S>(a)) ンド j 40 < 2((4a).
Upper tail (to lower-bound diag 1122). For A = 2.
[[[AX,] = = 22 (1 22- 2aa--2y-1/2 S (ex((2)-10)
hence
E e (82(0)-BS((00)]
< exp
E S exp | ((4a))
j=1
By Markov and a union bound over the M diagonal entries, setting the per-entry failure probability to 5g = 8 MM,
Pr Pr(Sa(a) & ((20) + ((4a) + 2 log
M 8
Sa(a)
2
82

---

Published as a conference paper at ICLR 2026
Therefore, with probability at least 1 Ovotal.
VM
Blag(SHH")
1.
(C(2a) ((4a) + 2 log duas
M
1/2
Lower tail (to upper-bound diag 1/2). A Chernoff bound on the lower tail of Sa(a) (via the mgf of "xx) gives, for any E (0, 1), the existence of constant
2a
ci(a) [20-1
12 /220-1
such that
Pr (a(S( 2 (a) (log(1/6)) 120 110
1-
With d = 60 drocal / M and union bound over the M rows, with probability at least 1 Orotal
VM
(log M 2a-1 Ococal
I.
(a((a))
Conclusion for 5.4 Combining the two displays.
VM
I N diag( (HH 11/2
VM
(log
M 8potal
2a-1
2
1
(<(2a) + ((4x) + 2 log Bweil
1/2 M
(c)
1/2
REGIME II: S (DIVERGING WEIGHTS)
Assume M for some fixed r 1 (as in our setup). Then
(d ()=2))-2a_ -1
d'-2a
[[[a([a]] Σ 20
satisfies
< E[Sa(a)] S 14
1-22 2xx
1 2a
j=1
Hence R[Sa(a)) d1-2a Moreover,
Var(Sa(a)) = > j- 0(1),
a>
O(d
a ac
so cases (ar(Sa(a)) (()( as d + oc. Thus, by Bernstein and union bound over the M rows, for all sufficiently large M we get, with probability at least 1 - S
1 B[Sa(o)) < Sa(o) < [[Sa(a)).
3
2
Using d 2 rM and the integral bounds for E[Sa(a)),
1-2a -1 < Sa(a) < 1 (r.))1-2a-)
3
2(1 20))
2a
Dividing by and inverting the square-root yields constants
Cr(a,r) = (1_ 1-2a 2a
3 1-20 2a
-1/2 Cu(a,r) =
-1/2
1 1-2a
r
2(1 - 2x)
such that, with probability at least
8
Cr(a.r) M z diag(SHS 3
Ma 1aI
(a S t)
83

---

Published as a conference paper at ICLR 2026
UNIFIED STATEMENT
Combining Regimes I and II, there exist positive constants ei (a, diota)) and oz(a, r, dtoal) such that, with probability at least 1 - Ototal.
(a, r, Owoul) Mmin((,5,o) I diag(SHS) Siag(SHH")--1? 3 02( r, Omal Mmiaa(0.5,a)
with the following explicit choices:
af > 1/=
ci(a, (roal) (((2)))((4a)+2 - c2(a,, Srotal) (cs(a)) -1/2 (log M
-1/2
dune
where one admissible choice is (((a)
(22-1)
2a-1
20-1/ / 22a-1
If a S and d 2 rM:
((a,r.))CC(a,r)) c2(a,r, = Cu
with CL, Cy as defined above.
84

---

Published as a conference paper at ICLR 2026
L ANALYSIS FOR THE CASE WITH LABEL NOISE
For the case with label noise, only Phase la is solved for SGD by Lin et al. (2024). So we will focus on the Phase la where 0.5 and B 0.5 holds.
Now we set an assumption for label noise. For selected data α, we assume that label y satisfies
(z. w²) +
where é is a label noise with mean 0 and variance x² satisfying c x.
Note that the case with label noise L(0) = Ez [( 0) = and L( = | w II? are not equivalent.
So in this section, we will use notation Line (0) Ex ((Sx, 0) -y)
Then Line (0) = |
- w²) 2 + 0² = (0) + o²
Here o² is the irreducible risk. Lin al. (2024) discussed compute-optimal scaling for L(
Luue (0) - 0². Therefore, we will also discuss compute-optimal scaling for L(Ø) Live (0) o².
Also in this section, let R(M,N, N, 7o) the L0 under learning rate % and fixed model size M. We will discuss the scaling law of R(M, N, 70) - o².
L.1 DERIVING ODE AND INTEGRAL EQUATION
For quadratic function q, by Taylor's theorem, have
((+())+1) - ((x)) F = E[(Vg(Ox), 0x+1 - 0&) FR) + E. (V (0&+1
0x) Fi],
where Fs = (S, Co. 0₂) Since
0x 02 7kk sign( (Sar,, Oz) yh)
We can expand the two terms using Gaussian identities. We let label noise for the same (æx, 9k) EG and V/ (æk, w*) Eg holds.
Gradient term
(((((())) 0.+1 0x)) [i
= (Vg(O))) E sssiinnSsee (Sxx) sign (Ix STO, w") <x) Fil))
diag (SHST) -1/2 SH (STO; w²)
2
= Vg(O.). aresin
π
(STO. - -w²) H (STOg w²) +
diag(K) K) 112 K (0: - 0")
Vg(O+). aresin
V|| - w')l' + 0²
where K SHST.
Quadratic term.
222 q, (0++1 - 0z) 02) Fi
= V2 E
(sign( Sxxx) sign (en((ax) ((x), STO, w") - -a4) 82 Fx =x))
= 7
V2g, E
(sign(Sarg 82 Fi
aresin diag (SHST) - 11/2 SHST diag (SHST) -1/2)
22g. 4:
V2g. aresin (laag( K) K) K diag K) K)- 1/2)
85

---

Published as a conference paper at ICLR 2026
One-step update formula. Substituting the gradient and quadratic terms yields the desired step update formula for signSGD.
27t Vg(Oz), arcsin
K (0p -0") (VL(k) + to² ))
Kk
2 (V9. Ko)
Elq(0x-1) g(0x)) Fx)
By the same procedure the noiseless case, while v L(k) in the denominator is replaced by v (T(()) + o², we get the following ODE, where P(t) [(t/70) and pi(t) = rz(1/vo)-
dp:
P(t) 0 (K) (tt/%))p(tt) + 2f 2f(t/no) 2o
4
Vi.
(140)
dt
π
Integral equation. Also, by the same procedure as the noiseless case, while ((u)) in the de- nominator replaced by VLlu) + 0², we get the following integral equation.
L(N) = H 1/2 w
11
M
r0)
42070 fo
f(u) VI(u)+o3
du
2% π
M
V
e
42070 s
L[w)+o3
f(u) du
5(z)² dz.
1=1
(141)
By using the same dritt/approximation-eem transformation as the noiseless case, we get
L(N)
2a-2--2+1 + (A10.5 Q(N))
2
(142)
approx
drift
10
M
V
exp
47o x (K)
N {
du
dz
(143)
π
00
π
VL(u) + o²
noise
where f(z) (which means constant learning rate) and
470
f N
du
Q(N)
L.2 EARLY STAGE FOR A NOISY LABEL
Similar to the noiseless case, we first solve for the early stage. Here we have to solve the following equation
28-28-1 20
L(N) (M Q(N))
It can be converted to
2a L(N)
70 /0
du
20+23
0
(144)
/
[(u)) +
Replacing by equality in 144 and differentiating with respect to N (viewed continuous time variable 2) yields
2x
2a
I(t))
2a+2331
T'(() = M0.5
(145)
2a
2/3 -
VL(I+ +
Equivalently,
2a
T(1)A-1 T'(1)) 00.5 70
A = 2a + 233 -
(146)
VL(() +
For any a > 0 and x 2 0 we have the elementary bounds
1
1
/2 min(2 -1/2, 1/2 0-1) <
S min x -1/2
1/2 0 o-1).
(147)
Vr o
86

---

Published as a conference paper at ICLR 2026
Indeed, if o² then S + o² < 2x,
V2 1/2 1/2 < +
S x 1/2
whereas if0 S x S o² then 8² < x +
0² < 2002.0 so
1 0 +o+ S o-
v2
Combining the two cases yields 147
Applying
147
with I L(t) in 146 obtain
I(t)-1/2 1/2 L(t) 2 o², L(t) S 0².
1(1)~111(0) x M0.5
7o
(148)
This naturally splits the dynamics into a large- regime L x² and a small- regime S 0². Suppose L(t) 2 o². Then from (48) we have
L(t)a MM. (0L(t)-1/2
or equivalently
L'(t) ~ M
%o I(t))-A-22
(149)
Define
1 1 A -A
2a
1
2
2a + 23 -
The assumptions a 0.5, 0.5, and form
B 0.5 imply <> Then (149) takes the canonical
dL
~ M 70 LS.
dt
Separating variables and integrating gives
j 1 = M 70 f
((t)-((-1) ~ M005 4
where we have absorbed additive constants into the implicit comparison. Thus, in the large- L regime,
L(I)= (M0.5 To -1/(6-1)
(150)
Writing
2(2a 2 - 1)
C-1
2a 1a+1-22
we recover exactly the original early-phase exponent:
L(t) ~ (M m) ")
L(t) 2 0².
(151)
In particular, the presence of v + r² in the denominator does not change the scaling exponent in the regime where T, is larger than the noise floor o²; it only affects the constant factors hidden in
Now suppose L(f)= < o² and is sufficiently large so that the small-L regime dominates, From 148
we obtain
I(I)==II'()) M 70
Observing that L()(t) A L'(t), we can rewrite this as
d
L(t) 0 -1 dt
Integrating in & and absorbing additive constants into yields
I(1) 005 To t.
87

---

Published as a conference paper at ICLR 2026
Since A < O, we invert this relation to obtain
2a 2,
T(t)) (M0.5 10.00.00//// t/o///// (M 10//0 p' =
(152)
2a
Thus, in the small-T (noise-dominated) regime.
T(t) (M.s To 1/
I(t) - o².
(153)
Combining (15I) and [13). we obtain the following formula for the early early-stage.
2(2a + 23 1) p
2a 23
L(I) ~ (M 70 1) p (M 70 t/t/o)
pf
2a + 1 23
2x
(154)
L.3 LIMIT STAGE FOR A NOISY LABEL
By the same procedure as Appendix E.3.2 we get an equation
Loc
20?
(ddaag(K))a) Lo + 0² + |
4
Solving the quadratic equation, get
L 7ō Tr(diag( ()(d + too I((iaa(())12) + |/2
Under our setup,
M
Tr(diag( (acliag(K))/ K3 = Σ SHST
La R M
M Mmax(1-20,0) = M 11minina, 0.5).
M
By the results from
Paquette et al.
2024 Lin al. 2024), and note in Appendix K.3
H 2
[wll
M 2a 1-299)
Hence
L ~ a 1 VM M- 22-+20-1)
L.4 EVALUATING COMPUTE-OPTIMAL SCALING
Combining the early stage and the limit stage, we get
2(2a+28-1) N,M, (27)
2a+28-1
2a+1- 28
(M
2a
+%
(20++8-1)
Note that we use R instead of L when are writing the loss three-variable function.
let To = M- Also assume z (this covers values such as a = 1, 0.01, etc.).
Compute-optimal occurs when the three terms balance. For the loss formula in this section, compute-
optimal occurs when [ 2a+23-1
and a M and (2-22028-1) balances. Solving
T M M (2a-2228-) 1), we get 70 M (2 Solving (2u1/2 20-28-1
M +2a+20-1) we get = 2 and it leads to j MN M4a+23
So finally get
M (1((1420))
R(M",f/M*, "fM",, - - ==
(155)
Figure 25 shows that exponents in the 155 and measured compute-optimal loss slope and optimal model size slope (in log-log plot) for the case with the label noise match well. In the experiments, used 0.1.
88

---

Published as a conference paper at ICLR 2026
B. = 10.60, 0.40, 1.50), Measure: 0.332, Theory: 0.313
to. B. (00600 0.40, 1.5),. Measure: 0.350. Theory: 0.312
Optimel mode size M
1014
Lots
10-4
Optimal model Fit: M TLOPy
Compute-Optmet Points
Ri:: Lass FLOPs IR'-0.999)
(6-M)
(0.80, 0.20, 1.50),
Measure: 0.298, Theory: 0.278
(a. B. (0.80 0.20 1.50). Measure: 0.338, Theory: 0 0278
10a
Optimel mode size M
10-1
Loss
30
Optimal model Fit: M FLOPyA
Compute-Optine
Fit Loss FLOPs-9920 (R?-0.991) 9911)
(6
(6-M)
Figure 25: Measure compute-optimal loss slope and optimal model size slope for the case
with label noise. We validate the exponent of R [ -f Y0 and M* with respect for the case with label noise. The left plot shows the compute-optimal loss with respect to FLOPS 6.M.N. The right plot shows the optimal model size with respect FLOPS 6MN. Note that we evaluate the region with big FLOPS, as we aim to evaluate asymptotic behavior.
89


Reviews:
[{"review_id": "oCZlpv4gOl", "reviewer": "Reviewer_dH5K", "summary": "This manuscript builds on the asymptotic characterization of SignSGD for quadratic problems in Xiao et al. (2024) to study its scaling properties, in a manner reminiscent of Paquette et al. (2024). It shows that, in phases where the scaling exponent of SGD is determined by gradient noise, SignSGD can improve the scaling. The benefit is attributed to SignSGD’s ability to balance drift-normalization and noise-reshaping via learning-rate tuning.", "questions": "- As fast as I can follow, there is no label noise in the setting studied. Can you explain why?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper derives the scaling exponent for SignSGD, a simplified variant of Adam, and empirically demonstrates that the theoretical predictions also hold for Adam.", "weaknesses": "- The technical part of the paper closely follows the ideas in Xiao et al. (2024) and Paquette et al. (2024). \n- As far as I understand, the results in the paper depend on choosing a good learning rate, where the optimal rate depends on both the target and feature exponents. However, the authors do not discuss how to tune the learning rate. Discussing this point—and showing how sensitive the results are to suboptimal choices—would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wxyvfjAHEV", "reviewer": "Reviewer_1qNE", "summary": "This paper explores the scaling laws of one-pass signSGD under the power-law random features (PLRF) model with two parameters (feature decay and target decay). It derives a four-term population risk decomposition and an explicit scaling law that depends on model size M, training steps N, and learning rate. By comparing this to SGD, the authors uncover two phenomena: drift-normalization, which accelerates training under conditions,  and noise-reshaping, which yields an N-independent noise floor and additional M dependence. Based on the scaling law, this paper theoretically shows that signSGD achieves a steeper compute-optimal slope than SGD in certain regimes, and optimal learning rate scheduling (a simplified warmup–stable–decay) further improves the slope. There are empirical experiments that confirm the theory.", "questions": "Based on the phase transitions, can you provide some summary for practitioners to decide (say from coarse empirical estimates of “feature/target decay”) when to try signSGD + stable-decay vs. SGD?\n\nPlease refer to the Weakness section for other questions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The theoretical argument is rigorous and sound to me. The four-term risk formula decomposition mirrors prior SGD analyses and uncovers signSGD-specific properties in a clear way. This gives immediate intuition for when signSGD can help.\n* The paper provides further insights, including the comprehensive phase transitions of scaling laws, analysis for compute-optimal slope, and learning rate scheduling that reflects the practice. These analyses serve as a step towards understanding this optimizer in theory. \n* The paper checks predicted slopes and compute-optimal scaling across representative regimes (Appendix C) via synthetic experiments. These empirical results validate and give us confidence in the theory.", "weaknesses": "* **Limitations in Setting:** There are certain assumptions in this paper, for example, batch size = 1, diagonal covariance, one-pass training, and Gaussian sketching in the PLRF model, that limit direct generalization to more practical pipelines. The paper acknowledges some limitations such as mini-batching and momentum/Adam invariants, but these still limit the novelty to some extent, and results would be stronger with at least partial extensions (or tests) beyond this PLRF setting.\n\n* **Discussion of Theory:** The structure of the paper could be improved. In particular, the context of the problem and proof ideas occupy too much space, while some useful contents are pushed to the appendices, and this makes the important Section 5 seem rushed. The two main effects, drift normalization and noise reshaping, should definitely be discussed in more detail. For instance, from Lines 413-424 ,the mechanism of signSGD superiority is attributed to an intricate interplay between these two effects that “creates room for a balance”. It is helpful if this could be quantified further. \n\n* **Adam Optimizer:** In the introduction, one claimed relevance of this work is that the state-of-the-art LLMs are trained with the Adam optimizer (rather than SGD), which can be indirectly studied through the more tractable signSGD. Since this is why signSGD scaling law helps connect theory with empirical optimizer choice in practice, there should also be more discussion on Adam in the main text. For now, Adam is not mentioned until in the conclusion, which claims that they also analyze Adam in a heuristic manner in the Appendix (while I do not think we should mention new contents in the conclusion). Given the paper’s motivation, these parts will be selectively included earlier to make the organization less detached, and it would strengthen the case to include at least one non-PLRF synthetic setting for Adam, too. \n\nOverall, I think this paper is sound. My main concerns are in possible preliminary extension from this setting and issues in presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nulVvK0BRr", "reviewer": "Reviewer_Hokd", "summary": "This paper develops a theoretical analysis of the scaling laws for signSGD under the power-law random features (PLRF) model, extending the recent theoretical framework of Paquette et al. (2024) for SGD. The authors derive asymptotic risk formulas and compute-optimal tradeoffs for signSGD, identifying two novel mechanisms—drift-normalization and noise-reshaping—that differentiate its scaling behavior from SGD. They also propose a simplified stable-decay learning-rate schedule (related to warmup–stable–decay) that further improves compute-optimal exponents in certain parameter regimes. Theoretical predictions are validated with numerical simulations.\n\nOverall, this work makes a theoretically interesting extension of the PLRF scaling-law framework to signSGD, introducing two interpretable mechanisms that may explain optimizer-dependent scaling differences. However, it requires improved exposition, stronger intuition, and at least limited empirical grounding on neural networks to reach full impact.", "questions": "1. How sensitive are the scaling-law exponents to the Gaussian-sketch assumption? Would structured or learned feature maps change the analysis?\n2. Can the “drift-normalization” and “noise-reshaping” effects be observed empirically in loss/gradient trajectories of real models trained with Adam?\n3. The “stable-decay” schedule is claimed to approximate warmup stable-decay; can this be verified experimentally on a practical model?\n4. How does the asymptotic noise term ( N_{\\text{sign}}(M, \\eta_0) ) compare quantitatively with SGD noise in finite-N settings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Extending scaling-law theory to optimizers beyond SGD is a natural and important step, especially given that sign-based methods underlie Adam and its variants, which dominate large-scale model training. The derivation of scaling exponents for signSGD is nontrivial, and the introduction of drift-normalization and noise-reshaping as analytic effects is both insightful and original.\n2. The analysis connects optimizer dynamics (sign normalization, step-size scheduling) to compute-optimal scaling exponents, potentially offering insight into why adaptive methods empirically outperform SGD. The paper demonstrates strong technical depth, following a structured progression from stochastic dynamics → ODE approximation → scaling law formulation → compute-optimal tradeoffs. The analysis is self-contained and references relevant recent work (Paquette et al., Lin et al., Xiao et al., etc.).\n3. Experiments (though small-scale) consistently confirm the predicted exponents and qualitative trends from the theory.", "weaknesses": "1.The paper’s presentation is algebra-heavy and difficult to parse. While mathematically correct, it often lacks guiding intuition or concrete interpretation of the results in the context of real neural network training. The drift-normalization and noise-reshaping effects, while named, are only partially explained mechanistically.\n2. The experiments are limited to synthetic settings (Gaussian-sketch features) and are primarily confirmatory. No evidence is given that the derived scaling laws hold qualitatively in deep networks, or that the “beneficial areas” (e.g., Area III–IV_sub) correspond to practically relevant regimes.\n3. Although signSGD is an important extension, much of the framework—including PLRF setup, compute-optimal derivation, and phase-plane analysis—is directly inherited from prior work. The primary novelty is analytical substitution of the sign-based update rule into the existing framework.\n4. The paper’s claim that Adam follows the same scaling law “heuristically” (Appendix H) is speculative and unsupported by rigorous derivation. This is a key potential impact point, but it’s treated informally.\n5. The text is dense, with overuse of symbols and nested definitions (e.g., equations (6)–(12)). Several steps (especially in Appendices D–E) are highly opaque. Without significant simplification or visualization, the theoretical results remain inaccessible to a broad audience.\n6. The empirical mapping of where signSGD outperforms SGD in the (α, β)-plane feels ad hoc and under-motivated. The heuristic in Section 5.1 (decay mismatch between target and gradient) is suggestive but not tested quantitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZMJ9OuOzk0", "reviewer": "Reviewer_AVVQ", "summary": "This work analyzes the one-pass SignSGD algorithm on a random sketch model $f_{\\theta}(x)=\\langle Sx,\\theta\\rangle$ with $S\\in\\mathbb{R}^{M\\times d}$ a Gaussian matrix with $\\mathcal{N}(0,1/M)$ entries and data generated by a linear model $y_{k}=\\langle w_{\\star},x_{k}\\rangle$, under a power-law assumption: $x_{k}\\sim \\mathcal{N}(0,H)$ with $H_{ij}=i^{-2\\alpha}\\delta_{ij}$ and $w_{\\star,j}=j^{-\\beta}$ with $\\alpha,\\beta\\geq 0$. \n\nLeveraging an ODE description from Xiao et al. 2024, the main contribution is to characterize the scaling behavior of the risk $R \\asymp \\mathfrak{f}^{-\\eta}$ as a function of the number of flops $\\mathfrak{f}=MN$ (model size $\\times$ running time), which is summarized in the diagram in Fig. 2 (left)\n\nThe authors then compare it to one-pass SGD, concluding as long as the learning rate is properly chosen, the SignSGD rates are always equal or better than SGD, depending on the interplay between $\\alpha$ and $\\beta$. This can be understood by comparing the ODE description of these two algorithms, which differ by a noise-reshaping and drift-normalization terms that lead to improved scaling.", "questions": "1. The observation that the maximal learning rate leads to a plateau regime is intriguing. Do you have any intuition for why this is the case? How does the maximal and optimal learning rate compare in terms of scaling?  \n\n2. In L305: \n> *We therefore focus on the optimal learning rate $\\gamma_0^\\star$, which maximizes the decay exponent $\\eta$*\n\nIs the optimal learning rate $\\gamma_0^\\star$ maximizing the exponent $\\eta$ also maximizing the risk?\n\n3. This work focus on a noiseless target. Why? Previous work on SGD (Berthier et al. 2020) shows that the noiseless rates can be faster, and in the context of ridge that there can be cross-overs in the rates depending on the interplay between the noise and the regularization (Cui et al. 2021; Defilippis et al. 2024). Do you expect your results to be independent of the noise of to observe something similar here?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "With the surge of interest in scaling laws, the results are timely and of interest to the ICLR community interested in this topic. The fact that the authors can derive different scaling behavior in the $(\\alpha,\\beta)$ plane and characterize the cross-over between them is interesting. Although the paper is technically loaded, the authors make a good job in giving intuitive explanations to the technical steps.", "weaknesses": "The primary weakness - shared by much of the literature on “theory of neural scaling laws” for linear models - is the gap between the stated motivation and the actual model studied. The setting considered here is highly idealized in the context of this motivation (linear model, linear feature map, noiseless observations: no feature learning), and it remains unclear to what extent the interesting theoretical results obtained in such a simplified regime meaningfully translate to the more complex models they aim to explain.\n\nClosely related settings have been extensively investigated in the context of kernel methods and their finite-width approximations (e.g., random features, Nyström methods), where the motivation for studying scaling behavior in linear models is arguably more natural. The power-law decay of covariates and target coefficients corresponds to the classical source and capacity conditions in this literature, and has been analyzed in kernel ridge regression (Caponetto and De Vito, 2007; Cui et al., 2021), random-features ridge regression (Rudi and Rosasco, 2017; Bach, 2017; Defilippis et al., 2024), as well as in work more directly related to the present paper on one-pass SGD (Yao et al., 2007; Ying et al., 2008; Carratino et al., 2018) and multi-pass SGD (Pillaud-Vivien et al., 2018).\n\nWhile I recognize that there are substantive differences in terms of motivation, specific parameter regimes ($2\\alpha<1$), and the quantities of interest (compute versus running time), there remains a meaningful overlap in the models and regimes studied. Therefore, the absence of discussion of this prior work — some of which precides Paquette et al. (2024) and Lin et al. (2024) by decades - is an important weakness. I encourage the authors not only to acknowledge this in the introduction and related work but also to compare with the overlapping regime.\n\n- (Caponnetto and De Vito 2007) Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics 7 (3), 331-368.\n- (Cui et al. 2021) Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime. NeurIPS 2021\n- (Rudi and Rosasco 2017). Generalization properties of learning with random features.  NeurIPS 2017.\n- (Bach et al. 2017) On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions. JMLR 2017.\n- (Defilippis et al. 2024) Dimension-free deterministic equivalents and scaling laws for random feature regression. NeurIPS 2024\n- (Yao et al. 2007) On Early Stopping in Gradient Descent Learning. Constr Approx 26, 289–315 (2007).\n- (Ying et al. 2008) Online gradient descent learning algorithms. Foundations of Computational Mathematics, 8, 561-596.\n- (Carratino et al. 2018) Learning with SGD and Random Features.  NeurIPS 2018.\n- (Pillaud-Vivien et al. 2018) Statistical optimality of stochastic gradient descent on hard learning problems. NeurIPS 2018.\n- (Berthier et al. 2020) Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model. NeurIPS 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VTiEZOGDbh", "reviewer": "Reviewer_5nJx", "summary": "This paper studies the scaling law of SignSGD in linear regression. It shows that the compute-optimal scaling of SignSGD is better than SGD in certain regimes, and the stable-decay schedule can further benefit the scaling.", "questions": "1. Does SignSGD improve the exponent in area Ic? Ic is adjacent to IV, and it intersects with Ac and Ad for SignSGD. It seems that SGD and SignSGD performs differently at least in part of Ic.\n2. Does SGD benefit from stable-decay scheduling? If so, it can be unfair to compare singSGD **with** stable-decay scheduling against SGD **without** such scheduling.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A clear benefit of SignSGD compared with SGD is shown.\n2. The authors made an important attempt towards understanding the learning rate schedule by studying the stable-decay schedule.\n3. Presentation is clear overall.", "weaknesses": "1. The assumption of $H$ being diagonal and $(w_*)\\_i$ being $i^{-\\beta}$ can be problematic for SignSGD. We omit the projection $S$ for convenience. In existing works about SGD, assumptions are usually about the eigenvalue spectrum of $H$ as well as the inner product of $w_*$ with the eigenvectors of $H$. Meanwhile, we can assume that $H$ is diagonal to simplify analysis because in SGD, we can apply rotations to $x$ and $w$: If $H$ is not diagonal, then assume that $H=UDU^\\top$ where $U$ is orthogonal and $D$ is diagonal. We can then apply the mapping $x'=U^\\top x$ and $w'=U^\\top x$, then the SGD update is\n$$\nw'\\_{t+1}=U^\\top w_{t+1}=U^\\top(w_t-\\gamma_tg_t)=w_t'-\\gamma_tU^\\top g_t.\n$$\nSince $g_t=x_t(x_t^\\top w_t-y_t)$, we have $U^\\top g_t=x_t'((x_t')^\\top w_t'-y_t)\\coloneqq g'_t$, indicating that the SGD update does not change under rotation. However, this property does not hold for SignSGD because $U^\\top\\mathrm{sign}(g_t)\\neq\\mathrm{sign}(g'_t)$. Hence, we cannot apply rotations to $x$ and $w$ to make the covariance matrix of $x$ diagonal. This is my major concern because intuitively SignSGD can perform better if the model has some \"coordinate-wise\" structure, which is the case when $H$ is diagonal. I will raise my score if a good explanation is made.\n2. In terms of presentation, it can be beneficial to compare the drift and noise terms of signSGD and SGD in a table (with a comparison in each area of Figure 2). It can also be beneficial if the stable-decay structure can be introduced in greater detail, possibly in a separate section.\n3. The stable-decay schedule can be restrictive. It captures the case where the learning rate decays polynomially in the iteration number, but does not include the linear decay schedule (and the similar cosine schedule, both decaying to 0 after a finite number of iterations).\n4. (Minor) Given the existing results in Xiao et al. (2024), the technical contribution is minor. However, the insights of this paper is fruitful.\n\nXiao et al. Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)