Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate
Authors: Zhiqi Bu
Shiyun Xu
Jialin Mao
Venue: ICLR 2026 Poster (2026)
Keywords: Convex optimization
Scaling law
Hyperparameter transfer
Primary area: foundation or frontier models, including LLMs
TL;DR: Convexity emerges in deep learning for general optimizers and model architectures, precisely characterizing and predicting loss in large scale.

Abstract: Deep learning has non-convex loss landscape and its optimization dynamics is hard to analyze or control. Nevertheless, the dynamics can be empirically convex-like across various tasks, models, optimizers, hyperparameters, etc. In this work, we examine the applicability of convexity and Lipschitz continuity in deep learning, in order to precisely control the loss dynamics via the learning rate schedules. We illustrate that deep learning quickly becomes weakly convex after a short period of training, and the loss is predicable by an upper bound on the last iterate, which further informs the scaling of optimal learning rate. Through the lens of convexity, we build scaling laws of learning rates and losses that extrapolate as much as $80\times$ across training horizons and $70\times$ across model sizes.

Published as a conference paper at ICLR 2026
CONVEX DOMINANCE IN DEEP LEARNING I: A SCALING LAW OF LOss AND LEARNING RATE
Zhiqi Bu
Shiyun Xu
FAIR, Meta Superintelligence Labs
Independent Researcher
com
sh shilynxxulare@@mail.coo
Jialin Mao
Independent Researcher jial malinmaa0000 gma com
ABSTRACT
Deep learning has non-convex loss landscape and its optimization dynamies is hard to analyze or control. Nevertheless, the dynamics can be empirically convex-like across various tasks, models, optimizers, hyperparameters, etc. In this work, we examine the applicability of convexity and Lipschitz continuity in deep learning, in order to precisely control the loss dynamics via the learning rate schedules. We illustrate that deep learning quickly becomes weakly convex after short period training, and the loss is predicable by an upper bound on the last iterate, which further informs the scaling of optimal learning rate. Through the lens of convexity, we build scaling laws of learning rates and losses that extrapolate as much as 80 across training horizons and 70% across model sizes.
INTRODUCTION
Deep learning has highly non-convex and complex loss landscape, e.g. the global minimum may not be unique, and there may be many local minima and saddle points where the optimization can be trapped (arripov et al.) 2018 Choromanska et al. 20155] Dauphin et al. [0144] Jin et al. [017 [Sagun] ctt al. 2016). Nevertheless, the empirical success of optimization in deep learning has implied that some benign properties may hold and be leveraged.
In fact, there has been a long history of empirically exploring convexity in deep learning. For example, Llama training (non-convex) with AdamW (Loshchilov & Hutter 2019) is closely similar to convex optimization with SGD, in terms of the shapes of loss curves in Figure of (Schaipp et al. (025). Empirical evidence by (Zhou al. 2019) has shown that SGD follows star-convex path during the optimization of neural networks. In addition, convexity is universally observed along the direction of gradients vision and language models by (Bu & Xu 2025).
Another line of research has focused on the two-dimensional loss landscape, by (Li et al. 2018 Im al. 2016) (along two random and normalized directions) and (Allen-Zhu al.) 2019b) (along the gradient and the negatively curved direction of the Hessian), which is approximately locally convex for residual neural networks with various depth and width. Furthermore, these convex- like loss landscapes are also observed on large language models such as RoBERTa, LLaMA, Qwen, and
Mistral in (Zhong et al. 2022 Chen et al. 2025) Lee al. 2024).
Besides these low-dimensional loss landscapes, the Hessian spectrum provides rigorous local notion of curvature: a positive-definite Hessian (all eigenvalues positive) indicates that the loss landscape is locally convex around that point. Empirical observations show that at initialization, the Hessian often contains many large negative eigenvalues, which quickly shift toward zero and become much smaller in magnitude than the positive eigenvalues. Consequently, the spectrum becomes dominated by positive eigenvalues and the loss landscape becomes approximately convex (Yao et al. 2020
Papyan 2018 Sankar al. 2021: Zhhng al. 2024

---

Published as a conference paper at ICLR 2026
In addition, theoretical analysis shows that two-layer and deeper neural networks can be regarded as convex in the nideenetwork limit, under neural tangent kernel (Jacot et al. 2018 Lee et al.] 2019 Du
et al. 2018 2019 Li & Liang 2018 Allen-Zhu et al., 2019a), neural repopulation (Fang et al. 2019 2022 and mean field (Mei et al. 2018 Chizat & Bach 2018).
As matter of fact, many works in deep learning have been drawing useful understanding from convex analysis to non-convex deep learning: [Sutskever et al. (2013) shows that momentum significantly accelerates the convergence; Defazio (Defazzi and decay from a convex viewpoint; the effect of weight decay (Krogh & Hertz 1991) is first understood in ridge regression (Hoerl & Kennard 1970).
al. (2023) explains the effectiveness of learning rate warmup
Specifically, the relationship between the learning rate sequence {me} and the upper bounds of loss sequence { has been heavily studied. These bounds come from different settings, studying convex or strongly convex loss, Lipschitz continuous or smooth loss, initeeiteration or asymptotic bound, averaged or last iterate, etc. Our work is directly based on Corollary 12 of (Defazio et al.] (2023) (re-stated in (2.4)) and a simplified version can be found in (2.3). which already provides some insights as we will summarize in Example 2.3
1.1 RELATED WORK
Convex to deep learning. It is well-known in convex regime that loss can converge at 0(1/VT) and optimal learning rate is O(I/VT). However, whether when these conclusions hold in deep learning is largely unclear. From this perspective, our work is most closely related to (Schaipp et al. 2025), both following from (Defazio et al. 2023). In contrast, we not only extend major findings of (Schaipp et al. 2025) theoretically (e.g. our qualifying exam in Condition 2.5 covers any schedule), but also focus more on empirical validation. For example, O(1/VT) convergence of loss and consistent patterns across models and optimizers in Figures 6-12 are not presented in (Schaipp et al.] 2025), which focuses on one model and single training horizon like our Figures 2-5. In particular, our empirical approach data-driven and practically applicable in large-scale deep learning.
Scaling laws. Current popular scaling laws are primarily about loss (Kaplan et al. 2020 Hoffmann al., (022) (i.e. scaling laws of loss), predicting how loss changes as model sizes and training horizons change, assuming optimal learning rate. As result, learning rate is not explicitly presented in these laws. Other laws (i.e. scaling laws of learning rate) can scale learning rate across training horizons Tpeak XI " where 0.125 in (i et al. 2024), a E {0.32, 0.38, 0.42, 0.65, 0.70} by Table 5 in (Bjorck et al.] 2024), and some are horizon korizon-unaware in (Porian et al.) 2024 Wang et al.) 2024) (i.e. a = 0). Nevertheless, these laws may not predict loss. In this work, we propose a law that simultaneously predict loss and optimal learning rate for the fixed value = 0.5.
Learning rate transfer. Maximal update parameterization (muP) (Yang al. 2022 is technique
to transfer optimal learning rate across model size. While Table I of ( (Yang et al. 2022) claimed it also transfers across training horizons, this contradicts with empirical evidence in (Borrkk et al.) (see Section 3.3) and our analysis.
2024)
1.2
CONTRIBUTIONS
In this work, we study the scaling law of deep learning loss and learning rate, through the lens of convex loss (non-smooth) and bounded gradient. We will establish series of generalizations from convex theory to deep learning, which is presented in Table and summarized as follows.
1. We study the convex-like behaviors in deep learning for general model architectures. op- timizers and learning rate schedules, hence establishing a non-asymptotic mapping from learning rate sequence to loss sequence.
2. We generalize an asymptotic upper bound of loss, achieving O(L/VT) convergence when (I) the peak learning rate scaled by 1/VT and (II) the learning rate schedule is qualified.
3. We propose a data-driven method to fit the asymptotic bound, establishing a scaling law across training horizons and model sizes.
2

---

Published as a conference paper at ICLR 2026
2 CONVERGENCE OF SGD UNDER CONVEX LOSS
2.1 REVISITING NON-ASYMPTOTIC BOUND OF SGD
We consider the stochastic gradient descent (SGD) as Wi+1 W; (w-1g((e), where w is the parameters, 77 is the learning rate, g is the mini-batch gradient with (g((w) VL, 0 S < I is the iteration and T the training horizon (i.e. number of iterations). The learning rate defined by two factors via 1paak St (T): (I) the learning rate schedule, which is a function E [0. 1] (e.g. linear decay is sy (T)
t/T), and (II) the peak learning rate Ppeak E R which is positive scalar.
We briefly review the convergence analysis under the convex and bounded gradient conditions.
Condition 2.1. Denoting a differentiable function as L and its gradient as VL, then is
convex (f, L(w) L(x)= S (w x) (((W) bounded in gradient 3G s.t. Vw, (lg(())] G
(2.1)
(2.2)
Remark 2.2. In Condition 2.1 the convexity is not necessary for our analysis as it can be replaced by star-convexity and iterate convexity along the optimization path, i.e.
L(w,) L(w.)= S (we w.) (((w.))
where w. E arggmi L(w) is the minimizer, and
L(w,) L(W,) S (we-w,) V..TT.
Additionally, the bounded gradient condition reduces to Lipschitz continuity when SGD is full-batch. In the parameter space, with g: == g(w:).
|| w. ||w, w.l12 27/+1 (W: w.) g: n
For bounded gradient and convex loss, denoting La = minw L(w), we have expectation
Wr41 w. 1| El[w; w.ll 20011 L.) G²
Telescoping sum gives
T-1 Σn
0 S B|| 1 We w.ll
Σ M+ L.)
G2
Dividing by 2 2, 1 and applying Jensen's inequality, we obtain upper bound of L(Wr), where Σr-d m m141 we is the averaged iterate and D |wo w.ll. Ei-d Mt+1
EED M+ile L. + 2 2E = ((
D2 GE
BL(w,)
(2.3)
EE Me+1
Finding 2.3. For constant learning rate the loss upper bound derived from convex analysis in simplifies to I., 12 n²Ch. This aligns with the empirical trade-off in deep learning that
2.3
21 2
larger ni converges faster but to higher loss, and vice versa (See Fig 10.14 (WikiDocs [2255) J.
D Furthermore, this loss bound is minimized by .TG in convex analysis, which underlies the
fast convergence observed in deep learning such D D-adaptation (Defazio & Mishchenko 2023).
Prodigy (Msshhhenko & Defazio 2023), DoG (Ivgi et al. (023). and DoWG (Khaled et al. (023).
With extra term in (Defazio al.
2023
Corollary 12), we have bound of any single iterate:
EL (L( I L. + 2 2E 1t
D2
G 2Σ
1=1
Th
+
G
MI
Th
(2.4)
7
1= 1h Σ m
k=
Note that (2.3) and (2.4 translate an arbitrary learning rate sequence {n1} } to an upper bound of the loss value. While both bounds shed some insights on the loss dynamics, focus the bound from
(244). since (2.3) can be less precise in characterizing the loss curves (c.f. Figure and Figure in
(Schaipp al. 2025
3

---

Published as a conference paper at ICLR 2026
2.2 ASYMPTOTIC LOSS BOUND LAST ITERATION
For any training horizon T, we can characterize the loss at the last iteration wr via (2.4.. We show the upper bounds on the loss in terms of different learning rate schedules, the optimal peak learning rate meakk and the optimal loss bound in Table where the results are derived in Theorem in Appendix A
Table 1 Convergence of optimal loss and optimal learning rate by Theorem under different schedules.
learning rate schedule
i..
upper (L(w))
opcimal
optimal Kiuu
consitant
Yicak
In
1. DG
square-oot inverse
Upeas Vy
L.
4V
400
I, DGV
linear docaging
Tpeak (1 1/2)
Le + "ap
12
Moua
G2
L, 2DG
cosine decaying
Opaak 1iowiet/22
be + Pipu
Moaa 6..
1. 2DG 1imi
dammu--tablleddaay
Opeit
idt CT 11
t [1+ 11+bb1n (1t))
L. 2DGy
(1 (1ro))
Opeat F-cc
We summarize some insightful observations in Table !
The optimal loss convergence rate is O(I/vT) and it can be achieved by some schedules like linear, cosine decay and warmup coamuup-ttable-aee stable-decay (WSD or trapezoid; (Xing al. 2018 Hägele et al., 2024)), which we term as the qualified schedules. In contrast, some other schedules like constant learning rate only achieve a suboptimal rate T/TT..
The qualified schedules have common patterns: (1) the schedules are horizon-aware, i.e. f is dependent on T, in constrast to the constant and square-root inverse schedules; (2) the loss bounds take the following form for some constants @1: 92:
of
EL(wr) $ L. +
+ Mpeak = LSGD lass (Mpaak, T)
(2.5)
Trpaak
The optimal learning rate of qualified schedule is O(1/VT).
While Theorem have considered specific schedules and optimal peak learning rate, in what follows, we will generalize to non-optimal peak learning rates Section 2.3 and introduce an exam to select qualified schedules in Section 2.4
2.3
OPTIMAL CONVERGENCE UNDER SCALED LEARNING RATE
We show in Corollary 2.4 that any scaled /peak can achieve the optimal O(1 (T/VVT loss convergence, regardless of whether 1/peak is optimal. Here, we scale Upeak by dividing any reference learning rate
Thef to VT.
Corollary 2.4. Consider learning rate schedule and Theak that satisfy 2.5 for SGD under Condi- tion 2.7
1. Any Thtt E R4 achieves the asymptotically optimal convergence rate:
L last Aast (Ppeak = Thee/ VT,,TTT - I.. Q(the)(( 10thh///////00000V00000
in which we define Q(Trr) := of /hof + Vhrr4/-
2. Additionally, the optimal The = argmin
Hort Q = 91/92. and LSGD (T) L, 2/1
/T
2.4 QUALIFYING EXAM FOR LEARNING RATE SCHEDULES
We qualify learning rate schedule function (T) if 2.5 satisfies the following
LSGD-tea (Tpeak 1/VT.TV) L. LL.
4

---

Published as a conference paper at ICLR 2026
where we simply choose Tee 1 as supported by Corollary
2.4 part 1.
We highlight that the qualifying exam can be conducted in training- free way without running any model optimization. We propose employ symbolic analysis in Condition 2.5 and the full derivation from discrete summation (2.4) to continuous definite integrals can be found in Appendix A.2
Condition 2.5. We claim learning schedule function St (T) [0, 1] is qualified, the following holds with Tu (T) s (T)/VT:
D
G²
TH nidkk
dt - =
2 So nett
2
St
Theoretically, prove in Theorem that
linear decaying, cosine decaying, and WSD schedules pass the qualifying exam; constant and square-root inverse schedules fail.
Empirically, we validate the effectiveness of our qualifying exam in Figure |
6.7
2007
Figure Upper bound of SGD loss Equation (2.4) with peak learning rate Tpeak = 1/VT.
Left-most is linear decaying schedule. Center is constant schedule.
GENERALIZATIONS TO DEEP LEARNING AND ADAPTIVE OPTIMIZERS
3.1 ABSTRACT FORM ANY-ITERATION LOSS
Our goal is to generalize the understanding from convex analysis with SGD beyond the scope of (2.4), to non-convex deep learning and general optimizers. However, we cannot (2.4) directly as we do not know D, G, L.. Alternatively, we will estimate these quantities in data-driven manner and work with an abstract form of the loss in (3.1).
Generalization (generalized from 2.4 ) For general optimizers and for deep learning, we have
D
Σi Ti
(L(,,) S Loo +
ME
T
(3.1)
2Σ 11t
2
1==1
++1 ΣE=
In contrast to 2.4 we require the following modifications:
The quantities D and are no longer tied to D = |/ w.ll and G max Bl]g(w)]l i.c. we renounce the physical meaning of these quantities, because
general optimizers (e.g. momentum SGD, AdamW, Muon, efffmeter-efficient training such as LoRA) are not covered by (2.4) ever under the convex setting in Condition 2.1
D., G depend on the model architectures and datasets, which are hard to derive in practice.
-
The irreducible loss is now Lac- -lim, w,) instead L. L(w.). This accommo- dates the fact that there are infinite minima in deep learning, hence w. is not unique and L, is not well-defined.
As we will observe in the following sections, some initial training. To be clear, we examine 3.. on various
holds well at all iterations and becomes tight after 3.1
models: ResNet (He al. 2016). VIT (Dosovitskiy et al. 2020), GPT2 (Radford et al. 2019), lanion-language models (which use LLAMA3 architecture (Dubey et al.) (024) as the language backbone).
5

---

Published as a conference paper at ICLR 2026
tasks: ImageNet (Deng et al. 2009). WebTWe (Gokaslan et al. 2019). and Cauldron [aurençon et al. 2024).
optimizers: SGD, AdamW Loshchilov Hutter (2019). Muon (Jordan al. 2024).
parameter-effiite training (LoRA (Hu et al. 2022 ) etc.
learning rate schedules: linear decay, cosine decay, WSD, constant, and square-root inverse.
3.2 GENERALIZING TO DEEP LEARNING
We start with SGD (same as (2.4) but in the deep learning setting. In Figure 2 we train R8 on ImageNet dataset with SGD under 4 learning rate schedules. The goodness of fit for (3.1 is particularly evident for WSD and cyclic schedules: for WSD schedule, we see sudden decrease of loss matching the decay of learning rate; for cyclic schedule, we see the periodic oscillation of loss matching that of learning rate. particular, we use half the iterations fit (solid line) and half to predict (dashed line). Our loss prediction by (3.1) is tight after a short period of training, with R² score 2 0.95, highlighting the strong applicability of Generalization
Line
200
250
201
200
20 20
(a) schedules
(b) Linear
(c) Cosine
(d) Cyclic
(e) WSD
Figure 2
Sequence- Sequence-to-sqquence sequence prediction by
3.1 for ResNet18 on ImageNet with SGD.
3.3
GENERALIZING TO ADAPTIVE OPTIMIZERS
We further test adaptive optimizers beyond SGD. Firstly, we compare ResNet18 trained with Adam W
Figure to SGD in Figure 2 and observe the same patterns that [..1 holds with R² score 2 0.95.
5000
nestred tea
schedules
Linear
(c) Cosine
(d) Cyclic
(c) WSD
Figure 3
Sequence-to-sequeece prediction by 3.1 for ResNet18 on ImageNet with AdamW.
We further pre-train GPT2 (124M) with AdamW and Muon-NSGD (Boreiko et al. 2025) optimizers. and observe consistent patterns in Figure and Figure cases.
The prediction I2 scores are 0.95 in all
LOSS CHARACTERIZATION AT LAST ITERATION
Towards building a scaling law loss and learning rate, we focus on the last iterate Wr and further abstract the loss characterization in (3.l)
Generalization 2 (generalized from (.5)). For general optimizers under deep learning and for
qualified learning rate schedule with any peak learning rale Tppakk we have
EL(wT) L.o
Upeak -2
LID (aup.akk (ppeak T)
(4.1)
Trlpaak
We non-negative linear regression y X + Foo where y E RT. specifically L(wr), X 25
, [D,
- T
and
X =
Di-n 2i
+ - Eia Ei
2h.
Di o2

---

Published as a conference paper at ICLR 2026
Pastte sepeica
ccom
2001
(a) schedules
Linear
(c) Cosine
(d) Cyclic
(e) WSD
Figure 4
prediction by 3.1 for GPT2 Open WebText with AdamW.
Zerms
(a) schedules
(b) Linear
(c) Cosine
(d) Cyclic
(e) WSD
Figure 5
seubenncett--equenee prediction by (3.1) for GPT2 on OpenWebText with Muon-NSGD.
In contrast Generalization 1 require the following modifications:
We restrict our analysis to the qualified schedules that satisfy Condition 2.5 such as linear and cosine decay, instead of any schedules.
We renounce the exact forms of /1 and T2 (e.g. &1 D, dz G for linear decay; d2 V /1.061G for cosine decay; `1 = D/ + c) for WSD) and will derive these quantities in data-driven manner.
We focus wr rather than any iterate w, and transform the inequality (3.1 mate equality for moderately large T.
approxi-
4.1 Loss UNDER ANY PEAK LEARNING RATR
We test (4.1) using losses reported in Li et al. (025). These results were obtained under optimally tuned hyperparameters, where the language models (dense and Mixture-of-Experre trained on a mixture of web text, mathematics, and code. The training used Adam W optimizer with cosine decaying schedule (see details in Section 3.2 and Appendíx A of (Li al., 2025)).
MoE) were
We observe high scores (2 0.95) across different model sizes as well as model architectures, which empirically validates Generalization 2
2.725
2.45
6.430 1S7 -2.00
0223 (.00
2158
181 -0.0000.
2.650
2.625
2.800
2.25
2.575
10-3
30
10-3
Learning Rate
Loarning Rat:
Figure 6 Loss versus learning rate for dense models (left) and MoE models (right) by (Li et al. 2025). the right plot, the orange and green curves correspond to different architectures despite having the same model size. Specific model configurations are summarized in Appendix C.]
4.2
LOss UNDER OPTIMAL PEAK LEARNING RATE
We further derive and test 4.1 under the optimal Paaa in Generalization
7

---

Published as a conference paper at ICLR 2026
part 2). For general optimizers under deep learning for qualified learning rate schedule with optimal peak learning rate Tpeakk we have EL(wr) ~ Lox /// VT.
Generalization 3 (generalized from Corollary 2.4
To validate Generalization B we scrutinize the loss values and training horizons in (Hoffmann et al. 2022) various models from 0.074B to 12.56B, which were used to establish the compute-optimal scaling law for large language models. These runs trained Chinchilla models (same as Gopher (Rac et al. (022)) with Adam W under cosine decaying schedule. The original loss values are not publicized in (Hoffmann al. 2022] but they reconstructed by (Besiroglu et al. 2024). Note that Hoffmann et al. (0022) only gives F rather than the training iterations. Therefore, we translate the training horizon to token size=FLOPS /6/ /6mmodel size= batch size x 7.
Table 2
Summary of linear regression on EL WT and 1/vT- batch size from (Hoffmann
loss
2022 Besiroglu
linear fit (99609996)
9986)
relelive
et al. 2024). Full table is deferred to Table 6 Runs for 2B
model visualized in Figure
20-
model size(B)
num of horizons
29 02
Loo
R score
0.074
5
3.22c+04
2.825
0.991
0.140
7
3.0c++04
2.670
0.991
1/-T-BatchSia
0,279
3.29c+04
2.498
0.999
0.425
8
3.27c+04
2,430
0.998
0.632
3.17e+04
2.367
0.998
26-
1000
1.143
10
3.10c+04
2.275
0.998
1,429
3.18e+04
2.253
0.996
1.611
9
3.36e+04
2.228
0.995
2.004
3.62e+04
2.178
0.999
194
Model size(B)
2.280
7
4,41e+04
2.128
1,000
2.979
5.90c+04
2.016
0.990
Figure Upper: 2B model pre-
4,519
6
3.83e+04
2.106
0.978
cisely fits Generalization B across a wide range of training horizons. Lower: As model size increases, Tox log-linearly decreases and 2/1 roughly increases.
6.792
8
4.66c+04
2.023
0.999
9.290
4
4.29c+04
2.046
0.988
12.56
3
4.23e+04
2.053
1.000
In Table 2 we consistently observe precise fitting of Generalization 3 with R² > 0.978. up to FLOPs=Ie22 and over 300B tokens. We note that all loss values are within 1% relative error of our prediction Figure
TWO-DIMENSIONAL SCALING LAW FOR LEARNING RATE
The success of transferring insights from convex analysis to deep learning underscores the appeal of predicting and controlling the loss via learning rate. In this section, we propose two-dimensional scaling law of optimal loss and learning rate at various model sizes (N) and training horizons (T).
Notably, must take step further from Generalization! because do not know the optimal di with unknown q1 and 72.
Tpeak. except it has form
dvTT
Generalization 4 (generalized from Corollary 2.4 part 1). For general optimizers under deep
learning and for qualified learning rate schedule with scaled peak learning rate Opeak Thet/ VT. we have
EL(wr) T.oc ((het) /VT, Wher-
FFLOPPs 6 token size x model size = 6 batch size x iterations x model size. The batch size is not released in this work and assumed to he constant across all runs.
8

---

Published as a conference paper at ICLR 2026
In words, we have (TI/VT) loss convergence under L/VVT-scalee Tlpeak> including but not limited to the optimal lee in Generalization 3 Notice this expression is linear 1/ 1/VT and can be visualized by a straight line. We seek the optimal Ther by multiple small-scale runs and we estimate L and Q via linear regression in Section 5.2 and Section 5.3
Finally, we present scaling law that predicts both loss and optimal learning rate:
ELIN, T) Looo (Mhati N) + (Teci N)/VT
(5.1)
Ppaak () ~ Tpeak (N small: /mmat)// T small
where Nemall is smaller model with Nsmall S NN, and Tamall is a shorter training horizon with Tmall T, so that we can use small-scale training inform the large-scale training's hyperparameter.
5.1 EXPERIMENT SETTINGS
We train GPT2 language models on OpenWeeText for various training horizons, following na codebase (Karpathy] 2023). We apply two optimizers: AdamW and Muon-NSGD, both with 0.01 weight decay and without gradient clipping. Here Muon-NSGD is adapted from the original Muon by (1) optimizing all 2D tensors with Muon and other tensors with normalized SGD, i.e. NSGD, and (2) using a single learning rate for Muon and NSGD. We use the cosine decaying learning rate schedule that decays to 0 with 2% warm-up. We conduct 240 runs, which are settings (0. 1d/dmma, O. D.LB/Mu IB/Muon-NSGD. and NB/Mu, 10 training horizons from 100 to 500k iterations, and Met from 0.01 to 30.0). See more details in Appendix B
To illustrate the generality of our approach, we conduct additional experiments in Appendix D for parameter-efficien training (LoRA (Hu et al. 2022)) and ablations over key hyperparameters, such weight decay, gradient clipping, momentum coefficient, batch size, and random seeds. We consistently observe O(1/VT) loss convergence with scaled learning rate across different regimes.
5.2 SCALING ACROSS TRAINING HORIZONS
We train GPT2 (O.1B) for T ranging from 100 to 500k steps. We obtain good fitting of linear regression over 1/vT < 0.02, i.e. T > 2.5k which indicates that Generalization becomes predictive very early during training. This observation is consistent for two optimizers AdamW in Figure and Muon-NSGD in Figure
From the intercepts of fitted lines, we determine leverage the optimal Meeak meer VT where Tirt = 10 for Muon-NSGD and Met 0.3 for AdamW. As we observe from the zoom-in plots in Figure and later Figure 10 the extrapolation of our scaling law across training horizons is 80x.
onls odts .V
6ir
14.7
Figure
Loss values (dots) and 1/vT prediction for GPT2 (O.1B) with AdamW.
5.3 SCALING ACROSS TRAINING HORIIZNSS AND MODEL SIZES
We further train GPT2 IB and 7B models with Muon-NSGD, using the same ner that transfer from GPT O. model by (5.I) Taking a closer look at 7B model losses, we observe precise fitting of our scaling law, extrapolating across model sizes by 70x.

---

Published as a conference paper at ICLR 2026
14
Su Filne Pledate
1.
0.01
0.88
56009 0200 23000 2000 subae adta o250 o2t0 cir 5.000
0.0
0.01
6.5
Figure 9 Loss values (dots) and 1/vT prediction for GPT2 (O.1B) with Muon-NSGD.
caro 1o04 redid, c3t2 tre War
6n Marii -1 ariz (ne Neevsson
GPD (XTFL Merr. NurE)
(P7) (1R) Mas M6.3 GPD (10) Made 68.0 qum
8.0ce cabso arers ames eaion calse untew
Figure 10
Loss values (dots) and 1/vT or 1/VFLOPs prediction for 4 settings.
5.4 SCALING MULTI-MODAL MODELS
We finetune vision-language models (VLM) with =IB parameters on the Cauldron dataset (Laurençon al. 2024), following VLM codebase (Wiedmann et al. 2025). We apply Adam optimizer with cosine decaying learning rate and 3% warmup. We test three values of Vher across training horizons up to 14.4k iterations. Figure TT] we observe good fitting when T 2000, thus generalizing our scaling law multi-modal models. We note that VLM has multiple components (language backbone, vision backbone, and modality projector) and each has a separate learning rate,
all of which are 1/VT scaled.
Gut lacuat
Se Jacee
0.7
0.50
606
6.66
2.55
0.65
3.00
6000
2.00
3/7
30/
Figure
Loss values (dots) and 1/vT prediction for multi-modal VLM with Adam W. Right plot zoomed.
CONCLUSION
This paper presents path, starting from a rigorous analysis that is restricted to SGD and convex loss, and generalizing to non-convex deep learning with general optimizers beyond the support of theory. Along the path, we heavily rely on data-driven method to fit our prediction of loss and learning rate. Our findings support (I) convex-like like behavior in deep learning, which can be characterized by sequence-to -esuueeeeeeeeeeeeeeeeeeeeeeeeeee prediction in 2 (II) asymptotic prediction of O(1/VT) loss convergence and O(1/VT) learning rate in deep learning: (III) a scaling law that extrapolates across training horizons and model sizes. We note some limitations of this paper, including the fact that our approach fails to predict loss but continues predict training loss when overfitting is severe (see Figure 12). and the lack of understanding why convex- like behaviors exist in various architectures and how many iterations it takes for deep learning be characterizable by such behaviors.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíčck, Agustín Piqueres Lajarin, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big deta-centric training of a small language model, 2025. Uttppst::///xxi wttp://.gaaaaaaaaaoo/o..ooo.b
02737
20
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019a.
Zeyuan Allen-Zhu. Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep Iearning via over- parameterization. In International conference on machine learning, pp. 442-252. PMLR, 2019b.
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024. 8 21
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 2024. 2
Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal Ir across token horizons. arXiv preprini arXiv:2409. 1et::2449999913,.. 2024. 2
Valentyn Boreiko, Zhiqi Bu, and Sheng Zha. Towards understanding of orthogonalization in muon.
In High- Learning Dynamics 2025, 2025. URL /tt.:tttps///opeeeeeeee...
Forum? Dozum??d==ppmy=t.... 9EN.
20
Zhiqi Bu and Shiyun Xu. Gradient descent with generalized newton's method. In The Thirteenth International Conference on Learning Representations, 2025. URL htttps:://pperreiiiii
Huanran Chen, Yinpeng Dong, Zeming Wei, Yao Huang, Yichi Zhang, Hang Su, and Jun Zhu. Understanding pre-training and fine-tuning from loss landscape perspectives. arXiv preprint arXiv:2505. 17646, 2025. T
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems, 31, 2018. 2
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. PMLR, 2015. !
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in nigh-dimensional non-convex optimization. Advances in neural information processing systems, 27, 2014. 1
Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by d-adaptation. In Interna- tional Conference on Machine Learning, pp. 7449-7479. PMLR, 2023.
Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements. arXiv preprint arXiv:2310.0783/, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2026
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675-1685. PMLR. 2019. 2
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.0000000 2018.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. aix-v2247, 2024. 5
Cong Fang, Hanze Dong, and Tong Zhang. Over parameterized two-level neural networks can learn near optimal feature representations. arXiv preprint arXiv:1910. 1508, 2019.
Cong Fang, Yihong Gu, Weizhong Zhang, and Tong Zhang. Convex formulation of overparameterized deep neural networks. IEEE Transactions on Information Theory, 68(8):5340-3352, 2022. 2
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018. [
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. //Skylion007. github. http:hhhu...oooeieeeeeeeeee 1o/DppatWeeDTexttooppu 2019. 6
http:
Alex Hägele, Elie Bakouch, Atli Kosson, Leandro Von Werra, Martin Jaggi, et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37:76232-76264 2024. 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-63 67, 1970.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 30016-30030, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference Learning Representations, 2022. Uttpttps: http://///epeeeeeeeeeeeeeeeeeeee id=nzevKeeFYE
Daniel Jiwoong Im, Michael Tao, and Kristin Branson. An empirical analysis of the optimization of deep network loss surfaces. arXiv preprint 1611v:161 2016. [
Maor Ivgi, Oliver Hinder, and Yair Carmon. Dog is sgd's best friend: A parameter-free dynamic step size schedule, In International Conference on Machine Learning, pp. 14465-14499, PMLR, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael Jordan. How to escape saddle points efficiently. In International conference machine learning, pp.
PMLR, 2017.
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: 7/kel wew.lllajo.rdd..o. tiub.. wttthuuo...///oeee tc/posst//mun//
thub.
12

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.
arXiv preprint ar arXiv:2001.08361, 2020. 2
Andrej Karpathy. nanogpt. https://github::: httpss///..ttaab.ccmm///aaataaa/
2023.
Ahmed Khaled, Konstantin Mishchenko, and Chi Jin. Dowg unleashed: An efficient universal parameter-free free gradient descent method. Advances in Neural Information Processing Systems, 36:
9748-6769, 2023. 3
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991. 2
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- models?, 2024.
6 TO
Jachoon Lec, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019. 2
Joonhyung Lee, Jeongin Bac, Bycongwook Kim, Se Jung Kwon, and Dongsoo Lee. To fp8 and back again: Quantifying the effects of reducing precision on Ilm training stability. arXiv preprint arXiv:2405. /87.:2405./8710 10, 2024. [
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018. !
Houyi Li, Wenzhen Zheng, Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang. Shijie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i-optimal hyperparameter scaling law in large language model pretraining. arXiv preprint arXiv:2503.04715, 2025.
21
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in neural information processing systems, 31, 2018. P
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Song Mei, Andrea Montanari, and Mhan-Minhha Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665EE7)) 17671, 2018. 2
Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free leamner arXiv preprint arXiv:2306.06 20233
Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXiv preprint arXiv: 1,07062, 2018.
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimall scaling of language models. Advances in Neural Information Processing Systems, 37:100535-1003
2024. 2
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 5
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12.iv:21 2021. 8
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv: /61:: 1.11,7776 2016. [
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9481 9481-9488. 2021.
13

---

Published as a conference paper at ICLR 2026
Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Simsekli, and Francis Bach. The surprising agreement between convex optimization theory and learning-rate scheduling for large model training. In Forty-second International Conference Machine Learning, 2025.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147
PMLR, 2013. 2
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025. URL
https://arxiv org/ a /tt///////222 2502. 14786
20
Siqi Wang, Zhengyu Chen, Bei Li, Keqing He, Min Zhang, and Jingang Wang. Scaling laws across model architectures: A comparative analysis of dense and moe models in large language models.
In EMNLP, 2024. 2
Luis Wiedmann, Aritra Roy Gosthipaty, and Andrés Marafioti. nanovlm.
https://githus
com/ htmm://ugiinfaccc/nanoo Va 2025. T0
WikiDocs. "deep learning bible - 1. from scratch: Z_10. optimizers", 2025. URL https://
wikidocs net. /
82297 Accessed Nov. 16, 2025. B
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv.1802.08770, 2018. 4
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022. 2
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens the hessian. In IEEE BigData, 2020.
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhiquan Luo. Why transform- ers need adam: hessian perspective. Advances in neural information processing systems, 37:
131786-131223 2024. [
Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhua Liu, Bo Du, and Dacheng Tao. Improving sharpness-aware minimization with fisher mask for better generalization on language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4064-4085, 2022.
Yi Zhou, Junjie Yang, Huishuai Zhang, Yinghin Liang, and Vahid Tarokh. Sgd converges to global minimum in deep learning via star-convex path. In International Conference on Learning Representations, 2019. [
14

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 DERIVATION OF TABLE
Theorem 1. For SGD under Condition 2.1 Equation (.4
gives:
Case 1: Constant learning rate (1): Tppak for all t):
D²
npeak G
EL(wT) S L. +
IT = LSGD-laat-cosstnn Opeak? T)
21 Mpaak
2
The upper bound is minimized at:
IT
D
(ppaki min LSGD-Aast-coommad (ppaak T) L.. + +GV
with Tpeak (T)
GvInT
(A.1)
Case 2: Square-root inverse learning rate (T) T/peak/ Vt):
D2
Ooeak G2 In
EL(wT) S L,
LGD -Gs---is-srr---in (7peak: T))
4VTT/peak
4vT
The upper bound is minimized at:
In min LSGD- Opeak: T) L. DG
D
with Mieak (T)
(A.2)
4T
GVInT
Case 3: Linearly decaying learning rate (T) Nppak t//T)):
D2
EL(wr) S L. +
+ Gp = LLGD-ttst-iiner ( (ppakk? T)
Tilpcak
The upper bound is minimized at:
1 min LSGD las linear Opeak T) T.. +2DG T
D
with 1lpaak (T)
(A.3)
Tipek
GVT
Case 4: Cosine decaying learning rate (1)z
1peak
1+cos/rt/T))
2
D2
EL(WT) L.
+ Mpeak 22 1.061
L LssD-last-cosine (ppakk T)
Trlpcak
The upper bound is minimized at:
11066
min LSG last cosine (Tpeak T) = I., + +DDGV Byaa
with Tpaak (T)
T
GVI.061T
(A.4)
Case 5: Warmup-stable-deaay learning rate ( m)
Tpeak
ift eT ift cT
T-t !lpcak T-eT
D2
1
1 + 1-c
EI((T) & L.
(1 + c)ppak T + Ppaak G2
1
In
L wiGD--ast-wod (?peak T)-
2
The upper bound is minimized at:
1 1 In
1tc 1-c
2
min Lod Opeaki Ojeak
(ppoaks T)) + +2DG
(A.5)
(1+
D
1 ln
1 c 1-c c
with Mpeak (T)
(1+c)
1+
T.
(A.6)
G
2
15

---

Published as a conference paper at ICLR 2026
Proof of Theorem 7
For the ease of presentation, work in the continuous regime and translate
2.3 to
( nrdt
D
So dl
G
T-1
LSGD any (T)
T.a
So nit
2 So net
G²
2
Jo
Jí nelt Si-1 ne
dk.
We will later show Condition 2.5 is also the continuous version of 2..
Constant learning rate T/ 7/. Hence, S Midl (T - k)m, ST ne (T - k)nl² Then
T] ( ne
D2
G²
G2
T-1
LSGD last constaml (T) 19-1, L., =
+
dk
21
2
J nett J n.dt
0
D
G
G
T-1
dk
27
2
0
T k
D
G
+
17 T.
2T n
2
Square-root inverse learning rate = Meak / Vl+ then f
2n
11) and
1 nH = 12 T+1 and we can write
D2
17 nln(T +1)
LSGD- sqnp--i (T) - L.
G
In(VT
+1-1)
4( 4(vT +1-1)
G22
T-1
In T+1
Vk+Il 1) vk I) T vk) dk.
8
Let A vT+ and with the change of variable = A - vk, du 2] dk we write
0 Vk +
VT I -
A - 0 ( Therefore,
1 V Vk + vk) (A uyu
1
and
T+1 k
22n (1- u
In
Also notice the upper limit & corresponds to u = 2A and the lower limit A, so the integral can be approximated by
2+1 k V To (V Vk)
dk 2 -4 /
In(1 u/A)
I(T)
du.
1/(2A)
u22
This integral can be calculated exactly with change of variable
[n(1 (1 + Inc In(1 (n(1-))
4 (Ine-
4ln(T + 1) + 6.7726
( (lo())+
+o(1/VT).
A
+T+1
Combining the above we have
LSGD-last-sgr-ine (T) Lu
An(vT+1 - + I)n( +1)
D
4ln(T 11 + 6.7726
G
+o(1/VT)
+T+1
D²
+ Iln(TT
In(T)
G² + +
4nyT
4/T
VT
16

---

Published as a conference paper at ICLR 2026
Linear decay learning rate Jk vedt (T k) /372.
"I 711 - 1/T) and it is not hard get Jg n.ll n(T 12)
can write down
D
T/3
G
T-1
TTTR 1: + 1)9
LSGD-last-linear (T) - L,
+
G² +
1(T k)2 * (T - k + 1)2 dk
T/T
1/1
2
D
116
2n
T-1
1
(1 +
)dk
1/ 3
37
(T- 1:)
D
Gn
(T 1 1 InT)
T/T
3
37
D
G
(3 + 2 T/T - 2/T).
3
Cosine decay learning rate
Tt = (1 + cos(( m))
We derive
A(k) [
T
1 + cOs
1/dll -1
dl
2
1
(
(I'
k)
COs
dt)
T
k
T
1=T
m
+
sin
2
2
T
1=kk
T
k
T
n
sin
2
2T
T
and
B(k) = ["
T
T
dt 17
2
(3
cos T
+
cos
2st T
dt
- 3(T - k)
1
T
T
+
sin T
+
sin
2nt T
8
2
k
2m
12( 3(T - k)
T
T
sin
tk
sin
2nk
2x
T
16
Finally, substituting them into main expression gives
22
B(0)
G
Mh B( - 1)
+
G
A(k) - dki,
2A(
2A(0)
2
D2
Gn
G
T-1
(1 cOs( 5/
B(k: -
Tii
8
2
A(k) A(k -1)
D2
Gr
G
T-1
(1 + cos( z
3(T-k+1) sin
m(k 1)
T sin
2ar(--1)
dk.
TT
4
sin
T
sin
2n
By change of variable
4. the last integral behaves like
1/T
(L + cos(
+ + dx
T
A2(Tx)
+ cos TT))
3(1-x)
sin(mar)
Ite sin (2TT)
27
dx + d+)
2
1-z
2 sin(TTx))
2
2 2.7443 + O(2).
17

---

Published as a conference paper at ICLR 2026
Therefore,
LsGD-last-cosine (T) L. 83
D
3rG
r
x -22.443++
D Tni
(1.66 + +0
TTj
7
ift < cf ift 2 cT
Warmup-stable-decay learning rate
1/2
T-t (/--T
We can get
| nidt
T
1cTT - h:) + (1-C)TT
if k < cT,
T
if 2 cT,
and
T
1/ - k) + 1¹ (T cT)
if <π. ifk2 C.
m nidl
3CT- (T k))
n
can write
LsSD-lattwsD(E (T)
L.
D2
1 2T
(1 c)yt G2
+ c)riT
(5e-1 1/dtt
T-1 Th (JT- ni
G
cl'
dk
dk
2
St nidt 1 nitt
se Je nitt Jh nett
D2
1+2c T
(1+c)n] G2
(1
+ C)TT
G
oT
(PTTT - -k) (T cT) (T-T)
331
T-1 VT-T
3(T- eTT
(T k)s
+
dk
dk
2
(n[CT -
k) + (1-chnT )z
2
cT
[[T'cTT]
)2
D
(1+ (c)) @² G
1 + C
2c
4
1
In
+
-
1
(1 - -
(1 ccnT
3(1
2
0
3(1 + c)
3
D
(1+ (1+2c)n
G² + G
1
2c
4
In
3(1 c
2
c
3(1 + c)
3
D2
1
( l1tc)
1 n
In
(1+c)m]
2
1 1-c)
A.2 DERIVATION OF CONDITION 2.5
Recall the loss bound at last iterate in
2.4
is
L. + D
G
-=
M
2
Σ
mr
2 2Σ
-==
k= E1
11= 1h
Σt=
We can rewrite the last term by applying Theorem with x, 1/. Theorem 2. For any sequence {#t), if f/T//T- 0, then
T
T
Σ x
Σ
Tk
Σ
Zt
I
Th
k=1
11=
Dr
E 1
t=k
Ekst+1
T
17
Proof of Theorem 2 We firstly simplify the notation by denoting A
El Tt Σ. 76
Elate n Elan for l <
18

---

Published as a conference paper at ICLR 2026
With this notation, write the left hand side of Theorem
2
T
T
T
T-1
T-
Et
Ak
Σ
Ft
A8
Tt
Th
R=1
t
1he
Am1
k=1
1=
k=1
Now we can exchange the double sum by noticing that
Zt
It
Ak
E
AgT
1=1
IEKEIST-1
Therefore the left hand side of Theorem becomes
T-1
T-1
Zt
St
Ak
FT
Σ
A
E1=1 1A
Splitting the first term, we get
T-1
(
Σ T E
A
+
IT
Σ
A
(A.7)
t=1
?/1
k=1
E
which the term in big bracket can be greatly simplified by telescoping the sum,
(
1
Σ
1
A
DE Q1
k=1 E 7s
Ee
ΣS ns
Σ
R=1
In particular, with ( - T - 1, this is just 1/1r.
Substituting back right hand side (..7 we finally obtain an equivalent form of the left hand side of Theorem as follows, after using //T = 0:
1
IT
+
t=1
Lsne+1
7p
Now have
D ET=1
G T-1
m
(A.8)
t=1
k=(+1 7k
However, (A.8) has some singularity in the last term at f = T, where the denominator may be zero (if learning rate decays) and not divisible. To work around, we replace the discrete summation by continuous integral to get
D2
G2
rr² nk
L, +
2
dt
2 2 d,
Je"
A.3 SUMMARY OF GENERALIZATIONS
Table Summary of generalizations in this work and coefficients be fitted in data-driven way.
related experiments
coefficients to fit
Generalization
Figure 2-5
L.o. D,G G
Generalization
Figure
Loo, q 92
Generalization
Figure 7. Table 2
L T192
Generalization
Figure 8-11
L (
19

---

Published as a conference paper at ICLR 2026
B
MORE EXPERIMENT SETTINGS
Figure
WSD uses 10% warmup and 10% decay.
Figure 2
Optimizer is SGD without momentum or weight decay, batch size 256.
Linear decays from peak learning rate 0.225 to O. Cosine uses peak learning rate 0.225. Cyclic uses triangular cycles and peak learning rate 0.056. WSD uses 10% warmup and 10% decay, peak learning rate 0.113.
Figure 3
Optimizer is Adam W: 31 = 0.9, 52 = 0.999, weight decay= 0.01, batch size 256.
Linear decays from peak learning rate 0.0003 to O. Cosine uses peak learning rate 0.0003. Cyclic uses triangular cycles and peak learning rate is 0.0003. WSD uses 10% warmup and 10% decay, peak learning rate 0.00014.
We use two optimizers: A and Muon-NSGI. Here Muon-NSGI uses Muon and NSGD with the same learning rate in (Boreiko et al. 2025): denoting W layer's parameter and ignoring weight decay for brevity,
Figure and Figure 5
Muon: W 41 = W. 17 (
NSGD:W 6+1 W. 17 m m
where NS is the Newton-Schulz matrix iteration and m, is the momentum.
Each optimizer trained under four learning rate schedulers, i.e., linear decay with warm-up, cosine decay with warm-up, cyclic and WSD. For schedulers with warm-up, we allocate 2% of the total steps (i.e., 100 of 5000 steps) to warmup. For WSD, we start to decay the learning rate from the last 10% of the steps. For AdamW, the peak learning rate is 2e-3. For Muon-NSGD, the peak learning rate is 2e-2.
Section 5 0.95. Muon-NSGD uses momentum 0.95. O. models use batch size 512. IB/7B models use batch size 64. We train with 1024 sequence length.
All runs trained with 6 mixed-precision training. AdamW uses 31 = 0.9, 82
VLM models use 1200LMMM--600M-Ittuue (Allal et al. 2025) as the language model and siglip2- base-patch16-55 1-511 (Tschannenn et al. (0255) the vision model. These models use different learning rates instead of single one. For example, when Met 5e 3, the vision and language backbones use Thef/ VT as the learning rate while the modality projector uses 100meet/ /T. As a consequence, all learning rates are 1/vT scaled. Adam W uses default hyperparameters with batch size being 64. Sequence length 8192, max image size 2048, max images per example 8, max images per knapsack 36.
20

---

Published as a conference paper at ICLR 2026
C
DETAILS OF SECTION 4
C.I FIGURE 6
The model configurations of Figure can be found in Table and Table All other parameters not specified here follow the default setting in (Li et al. 2025).
Table 4 Dense model configurations. Model size is measured in billions of parameters (B). In columns, dmedel hidden dimension, di feed-forward width, Tihend number of attention heads,
Mlayer number of layers, B = batch size, T = total iterations.
Model Size (B)
drudee
d
Thhaus
Maaers
B
T
1.07
2048
8192
16
16
352
27743
0.54
1280
9048
10
13
352
39464
0.43
1280
9472
10
10
512
21696
0.21
960
9368
15
7
128
76293
MoE model configurations. Model size is measured in billions of parameters (B), In columns, nexp = number of experts, deep = per-expert hidden size, SED selected expert dimension, B = batch size, T = total iterations.
Table 5
Model Size (B)
dexp
SED
B
T
2.16
8
2888
8664
32
30517
2.15 (Green)
88
352
704
32
30517
2.15 (Orange)
89
352
352
32
30517
C.2 FIGURE 7 AND TABLE
We give the comprehensive table of all datapoints from (Besiroglu al. regression fits well in all cases.
2024
where the linear
21

---

Published as a conference paper at ICLR 2026
model size(B)
of horizons
201
Loo
Ra score
0.074
5
3.22e+04
2.825
0.991
0.090
3
3.19c+04
2.774
0.991
0.106
4
3.38c+04
2.706
1.000
0.117
3
3,27e+04
2.692
0.996
0,140
7
3.04c+04
2.670
0,991
0.163
3
3.11e+04
2.619
1.000
0.175
7
3.08e+04
2.619
0.995
0.196
3.14e404
2.582
0.999
0.217
6
3.54e+04
2.526
0.998
0.251
3
3.37c+04
2.517
1.000
0.279
8
3.29c+04
2.498
0.999
0.305
7
3.14c+04
2.488
0.997
0.425
8
3,27e+04
2.430
0.998
0,489
4
3.30c+04
2.404
0.999
0.552
8
3.24++04
2.382
0.999
0.586
8
3,25c+04
2.368
0.994
0.632
8
3.17e+04
2.367
0.998
0.664
3
3,46c+04
2.330
0.999
0.724
3
3.53e+04
2.320
0.999
0.817
10
3.28c+04
2.315
0.994
0,893
3
3.35e+04
2.304
0.998
1.019
7
3,06c+04
2.305
0.997
1.143
10
3.10c+04
2.275
0.998
1 1.265
10
3.05e+04
2.286
0.986
1,426
3
4.07c+04
2.214
0.984
1,429
9
3.18c+04
2.253
0.996
1.592
4
4.22c+04
2.182
0.997
1.611
9
3.36e+04
2.228
0.995
1,730
3,53e+04
2.207
0.998
1.795
11
3,41e+04
2.211
0.997
2.004
8
3,62c+04
2.878
0.999
2.280
7
4.4le+04
2.128
1.000
2.636
4.08e+04
2.113
0.998
2.979
10
5,90c+04
2.016
0.990
4.519
6
3.83e+04
2.106
0.978
6.792
8
4.66e404
2.023
0.999
9.290
4.29e+04
2.046
0.988
12.560
3
4.23e+04
2.053
1.000
Table Comprehensive version of Table
22

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
D.1 EFFECT OF OVERFITTING
Our experiments are primarily conducted when the train loss and test loss similar, without much overfitting For example, our GPT2 runs in Section OpenWebText) and test loss is close to train loss.
5 use up to 260B tokens (about 30 epochs of
In order to validate our generalization in the overfitting regime, we train VIT models and ResNet models on ImageNet for multiple runs up to epoch, and a ResNet50 on CIFARIO 10 for multiple runs up to 820 epochs. As expected, within epoch when overfitting is insignificant, O(1/VT) convergence holds for both train loss and test loss, although the coefficients of fitting may be slightly different. However, with overfitting (e.g. after 50 epochs), we observe that D(1/vT) convergence only holds for train loss but no longer for test loss (see the last plot Figure
6.75
8.75
6.50
6.50
6.25
6.25
3
5.75
Wi-Saas
5.75
ResNei-11
WTT Smal
ResWee-3 34
5.50
WiTTiyy
5.50
Reshet 50
5.25
5.25
0.015 0.020
0,030 1VT
0.005
0.049
0.045 5.050
5:00
0,010
0.015
0.020
0.025
0.000
0.535 6.640
0.045
0.250
11/7
Predicted Train Loss
Train Loss
Loss
2-
0-
203
I02
51
32
16
1/y epochs
Figure 12 Loss of multiple runs with scaled peak learning rate and AdamW. Upper left: VIT model on ImageNet (within I epoch). Upper right: ResNet models ImageNet (within epoch). Lower: ResNet50 on CIFAR10 (over epoch), where we have marked the number of epochs along x-axis.
All models are optimized by AdamW with 31 0.9, 3 0.999, weight decay= 0.01, batch size 128. Learning rate schedule is cosine decay with peak learning rate 0.01 1.01/VT.
23

---

Published as a conference paper at ICLR 2026
D.2 PARAMETER-EFRICHEE V.S. FULL MODEL TRAINING
We experiment with full model training and parameter-efficient fine-tuning of GPT2 models. We test small//medium/lage sizes E2E dataset. For parameter-effcient fine-tuning, we apply low-rank adaptation (LoRA) with rank 4. Our experiments in Figure 13 show that O O(1/vT) convergence holds within both training regimes, All models are optimized by AdamW with 3 0.9, B2 0.999, weight decay= 0.01, batch size 8. Learning rate schedule is linear with peak learning rate 0.0005 /V for LoRA and 0.00005/ VT for full model fine-tuning.
1.5-
1.2-
120
1.6-
1.15
14-
1.10
GP12-medum GPT2-aage
T2-nwdlum
GPT2 lange
1.05
0.01
0.02
0.67
0.04
0.10
Figure 13 Loss of multiple fine-tuning runs with scaled peak learning rate and AdamW, E2E dataset. Left: full model fine-tuning. Right: LoRA fine-tuning.
D.3 ABLATION STUDY ON HYPERPARAMETERS
We further evaluate our Generalization through the ablation study over five key hyperparameters, such weight decay, gradient clipping, momentum coefficient, batch size, and random seeds. For each new row Table 7 we launch 5 runs with T [602, 70?, 802, 90², 100 so in total we
summarize the statistics of 35 runs. We fit linear regression of validation loss against 1/VT, and we consistently observe goodness of fit.
Q estimation (std err)
L estimation (std err)
R fit score
default (seed=1337)
23.9094 (0.469)
2.8298 (0.004)
0.998
seed=3333
24.1820 (0.220)
2.8321 (0.003)
1.000
seed=8888
25.0086 (0.914)
2.8146 (0.012)
0.995
default (batch size=512)
23.9094 (0.469)
2.8298 (0.004)
0.998
batch size=64
39.5403 (1 (1.238)
2.9670 (0.016)
0.996
default (clipping=0.0)
23.9094 (0.469)
2.8298 (0.004)
0.998
clipping=1.0
24.3836 (0.481)
2,8207 (0.006)
0.998
default (momentum=0.95)
23.9094 (0.469)
2.8298 (0.004)
0.998
momentum=0.9
23,8508 (0.497)
2.8432 (0.007)
0.998
default (weight decay=0.01)
23.9094 (0.469)
2.8298 (0.004)
0.998
weight decay=0.0
25.1 (0.416)
2.8158 (0.005)
0.999
Table 7 Ablation study Generalization 4 over weight decay, gradient clipping, momentum coefficient, batch size, and random seeds. High R² score supports our O(1vVT) loss convergence with 1sCTed learning rate.
Here, the default setting net 1.0 for GPT2 0.IB, batch size 512(* * 1024 context length), weight decay 0.01, momentum 0.95, random seed 1337, clipping norm 0.0, Muon-NSGD optimizer and cosine learning rate schedule (with 2% warmup), the same as in Section
24


Reviews:
[{"review_id": "4ADvwXARZ0", "reviewer": "Reviewer_WNR5", "summary": "This paper explores whether convex optimization insights can reliably predict and control loss dynamics in deep learning through learning-rate (LR) schedules. Starting from a known convex, bounded-gradient last-iterate bound, the authors compute schedule-specific last-iterate bounds and show that horizon-aware “qualified” schedules (e.g., linear, cosine, WSD) achieve $\\mathcal{O}\\left(\\tfrac{1}{\\sqrt{T}}\\right)$ optimal last-iterate convergence with $\\eta_{\\text{peak}} \\propto \\tfrac{1}{\\sqrt{T}}$ (Theorem 1, Table 1). They then propose a training-free “qualifying exam” (Condition 2.4) using integral approximations to identify such schedules. Moving beyond convex SGD, they introduce data-fitted coefficients $(\\tilde{q}\\_1, \\tilde{q}\\_2, L\\_{\\infty})$ and show empirically that (i) a sequence-to-sequence mapping from LR schedule to loss captures training trajectories across models, optimizers, and schedules ($R^2 \\geq 0.95$ in most cases), and (ii) last-iterate loss scales approximately linearly in $1/\\sqrt{T}$ under $\\eta_{\\text{peak}} \\propto 1/\\sqrt{T}$, with cross-horizon and cross-size extrapolations. Analyses include new runs in vision and language and external large-scale results to support a simple two-dimensional scaling relation.", "questions": "- Will you release code, configurations, and logs (including seeds) for all experiments and figures?\n- How sensitive are $(\\tilde{q}\\_1,\\tilde{q}_2, L\\_{\\infty})$ and fit quality to weight decay, momentum/$\\beta$’s, gradient clipping, batch size, data order, and seeds? Please add ablations with error bars.\n- Can you justify Condition 2.4’s sum $\\to$ integral replacement or provide bounds connecting the integral “exam” to the discrete last-iterate behavior?\n- In your experiments, how sensitive are fitted slopes/intercepts to batch size?\n- Can you propose diagnostics to detect the onset and breakdown of the “convex-like” regime (e.g., via curvature/Hessian measures) and validate test-loss predictions where possible?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Broad, consistent empirical signal.** The sequence-to-sequence fits trained on the first half of iterations and evaluated on the second half show high $R^2$ across vision (ResNet, ViT) and language (GPT-2), optimizers (SGD, AdamW, Muon-NSGD), and LR schedules.\n- **Simple, actionable guidance.** Horizon-aware schedules with $\\eta_{\\text{peak}} \\propto 1/\\sqrt{T}$, paired with a few pilot runs to pick $\\eta_{\\text{ref}}^\\star$, provide a practical recipe for early forecasting and planning.\n- **Cross-horizon and cross-size extrapolation.** The paper demonstrates extrapolation up to $\\sim 80 \\times$ in horizon and $\\sim 70\\times$ in model size, and complements with analyses of external large-scale results.", "weaknesses": "- **Limited theoretical novelty.** Main results rely on standard convex SGD analysis; $\\mathcal{O}(1/\\sqrt{T})$ rates are classical. The main theoretical additions are schedule-wise constants (some via approximation).\n- **Reproducibility and robustness gaps.** No code/logs are released. Experiments primarily vary LR and $T$; other key hyperparameters (weight decay, momentum/$\\beta$’s, clipping, batch size) are mostly fixed (e.g., wd$=0.01$) with no ablations. \n- **Scope of applicability.** Appendix~D shows the $1/\\sqrt{T}$ law may fail for test loss under overfitting; diagnostics to detect onset/breakdown of the “convex-like” regime are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mxOMwz20Tm", "reviewer": "Reviewer_yGzE", "summary": "This paper extends the discoveries in [1], which states that the upper bound driven from convex optimization for SGD matches the loss curve of deep learning in general. In other words, in deep learning SGD has coeffients A, B that satisfy\n$$\nL(w_T) - L^{*} \\leq \\frac{A}{T\\eta_{max}} + B\\eta_{max},\n$$\nwhere $\\eta$ is learning rate, $A$ is an analogy of the diameter, and $B$ is an analogy of the maximal function value. The core contribution of this paper is extending this tendancy to other optimizers and find that the principle holds indeed. Different from SGD, the coefficients are not exactly given from convex optimization, so they propose a \"data-driven\" approach where they use certain points on the training curve to approximate $A$, $B$, and predict the rest. They validate their discoveries across various experiments: the tendancy across different max learning rates, upon the optimal learning rate, and a scaling law that predicts both loss and optimal learning rate.\n\n[1] Schaipp, Fabian, et al. \"The surprising agreement between convex optimization theory and learning-rate scheduling for large model training.\" arXiv preprint arXiv:2501.18965 (2025).", "questions": "One thing I got curious: why is this phenomenon happening? Is it really because the landscape becomes convex in the latter parts of training? If that is the case, is there a way to actually verify whether the landscape is convex or not?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe it is rare to have a nice, predictable tendancy in neural network training, and such discovery as in this paper is useful because once we can predict what will work best in practice, e.g. can compute the optimal learning rate by hand. Even though this is an extension (at least in my view), the fact that the tendancy exists for different optimizers is an important discovery, because it is not obvious. So one strength of this paper is that it proposes a property of neural network training that seems actually useful.\n\nAnother strength of this paper is an extensive experimental result with very high correlation ($R^2 \\geq 0.95$, mostly even higher). I think these results show a strong signal that the authors have indeed identified a tendancy.", "weaknesses": "One thing that was not clear to me was:\n\nso we have an upper bound that looks like\n$$\nL(w_T) - L^{*} \\leq \\frac{A}{T\\eta_{max}} + B\\eta_{max}.\n$$\n\nSo is $A, B$ a function of the optimizer, the model, and the learning rate schedule? In other words, if these three are fixed, is $A$ and $B$ fixed regardless of the maximal learning rate? If this is not the case, I think it would make the paper much weaker (and there is potential that I may lower the score), because even though there is a tendancy there is no practical benefit as for every maximal learning rate we should find $A$, $B$ again and there might not be a practical scheme. It would be great if the authors clarify this point, and it would be amazing to show a practical application of this result in the experiments section (e.g. actually tuning learning rate with the prediction). \n\nEventually, it would be good to explicitly show how $A, B$ and the various different factors in training is related.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFAJkzWHwJ", "reviewer": "Reviewer_cgXX", "summary": "The paper proposes a series of upper bounds that are used to show—through regression fitting of their parameters—that different combinations of deep learning models and optimization algorithms display a convex-like behavior across the training path. In particular, this is used to predict asymptotic $O(\\frac{1}{\\sqrt{T}})$ loss convergence and learning rate. The upper bounds are characterized as “generalizations” of bounds derived from loss functions that otherwise are convex (or at least “star-convex”) and have bounded gradient.", "questions": "Please, see the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper strengths are:\n- The paper’s motivation of predicting convex-like behavior is relevant in the understanding of the training of deep learning models. \n- The scaling laws found by the paper, which establishes evidence towards convex-like behavior in the training of deep models, are extensively characterized in their data-driven approach.\n- The paper has enough models and training procedures to properly demonstrate its claims. I really appreciate that from the authors.", "weaknesses": "Although the paper’s topic is relevant, there were multiple things that were unclear in the paper’s presentation (some of them could have been spotted after diligent proofreading). Also, there were confusing parts in the presentation of both theoretical and experimental results—I will detail this below. Given that there is still work to be done to improve the paper, I am giving the current score.\n\n**>>Important things:**\n- I find it odd that in the last sentence of the paragraph from line 057, there is reference to equations (2.3) and (2.4) and the work by Defazio et al. (2023), as if the reader could immediately recognize what the authors are referring to with such references. Moreover, it is hard to know from the sentence in lines 059-061 what exactly the authors are doing differently than Defazio et al. (2023).\n- The inclusion of “Example 1.1.” is confusing. A few notes:\n  - There is no explanation of the meaning of parameters $L_*$, $D$, $T$, and $G$. How is the reader supposed to understand Example 1.1 then?\n  - Moreover, there is another problem: it is said that the shown equation “aligns with the empirical trade-off in deep learning that larger $\\eta$ converges faster but to a higher loss, and vice versa”. Two problems with this: Firstly, **there is no citation** to back up this claim: the authors must add one. Secondly, the statement is misleading: what becomes higher is the **upper bound on the loss** that the authors have derived, and not the loss *itself* (but the reader has no way to know this, since no explanation is given in Example 1.1.).\n  - Example 1.1 needs to be urgently fixed considering what I pointed out. It may need to be fully reformulated. Also, the title “Example” sounds odd–maybe “Finding” or even \"Example of Findings” could be better.\n- Line 116: Shouldn’t $\\eta_{peak}$ be constrained to be positive, instead of being any real number?\n- Equation (2.2): It should say “$\\exists G$ such that $\\forall w”.\n- Remark 2.2: $w_*$ is stated as being **unique** because of the expression $w_*=argmin$ (instead of using the expression $w_*\\in argmin$). However, convex functions **do not necessarily** have a unique minimum (I don’t think that Condition 2.1 implies uniqueness; please, correct me if I am wrong). Why is uniqueness assumed? Does uniqueness play an important role in **any** of the derivations in the paper?\n- The paper goes through a series of elemental derivations to obtain equation (2.3). However, then the work (Defazio et al., 2023) is cited to obtain (2.4), without any mention of whether equation (2.3) was used to derive equation (2.4), and, if that is the case, there is no mention of how (2.4) came from (2.3). The question is: why derive equation (2.3) when at the end, as the paragraph that follows equation (2.4) states, only (2.4) is going to be used in the whole paper? Why not simply cite (Defazio et al., 2023) and avoid derivations that do not seem useful at all? The shown derivations would only be useful if an explicit connection and derivation is made between (2.3) and (2.4), but this is absent in the paper.\n- In line 160 and in Table 1, the term “optimal loss” **does not** refer to the optimal loss itself, but it refers to the “**optimal upper bound on the loss**”. This needs to be fixed.\n- Line 185: “Section 2.4” is mentioned when, in fact, the paper has no “Section 2.4”. What is supposed to go there?\n- It is confusing that equation (2.2) of Condition 2.1 is included as an assumption for the **whole** paper, when in reality, according to Remark 2.2., only star-convexity is needed (the unnumbered equation inside Remark 2.2). So, if all the paper results only need star-convexity (assuming Defazio et al (2023) also used star convexity), why not replace equation (2.2) by star-convexity? One could just make a simple remark saying that convexity is a stronger notion than star-convexity. Do the authors have a strong reason for keeping (2.2) inside Condition 2.1 instead of being replaced by star convexity? \n- I **strongly suggest to emphasize** in Section 3 that: **>>>>** All the results described in Section 3.2 and Section 3.3 indicate convex-like behavior, and that such behavior is due to **two factors**: the model itself and the chosen optimization algorithm. These two factors were originally present in the upper bound where the generalized bound was derived from. In other words, though the loss function of the neural model is non-convex, its behavior **along the optimization path** behaves according to a convex-like upper bound. **<<<<**. This summarization is currently absent in the important Section 3. Also, the use of the term “optimization path” as *where* the convex-like behavior happens is very useful. \n- There seem to be figures missing or a mismatch between the text and the figures caption. For example, line 261 mentions that ResNet50 is present in Figure 2, however, Figure 2’s caption shows ResNet18. Likewise, line 282 mentions that ReNet18 is present in Figure 3, while Figure 3’s caption shows ResNet50. In any case, the paper **will greatly benefit** from the presentation of both SGD and AdamW for both ResNet18 and ResNet50 models.\n- Section 4.1 does not mention which model is being trained for the experiments. I suspect that it is some sort of language model, but the model **must** be included.\n- It is really cumbersome for the reader to have all types of generalizations spread out in the paper, when they are sort of related to each other. I **strongly suggest** to put all generalization bounds on a single table to aid the reader to better understand the paper. The table can include columns **describing** (i) where/when each generalization bound is used, and (ii) which parameters need to be obtained through regression.\n- To better understand the experiments of Figure 8 and Figure 9 (something similar for Figure 10), it is important to include a text explaining that the reason why one cares about **fitting straight lines** is because the upper bound in Generalization 4 is just a line: for different values of $\\frac{1}{\\sqrt{T}}$, the slope $\\tilde{Q}$ and the offset $\\tilde{L}_*$ needs to be estimated. Unless I missed something, this explicit information of which **parameters the figures are fitting** is missed in Section 5.\n- Many citations are **missing their publication year**. This needs to be fixed.\n\n**>>Unclear things:**\n- Lines 031-032: How is it that the Llama training being mentioned is similar to the other SGD training being mentioned? What kind of metric of similarity was used? Which citation presents such a claim?\n- Line 076: uses the term “qualifying exam” without any reference of what this means and with respect to *what* something is being qualified. Moreover, the term “qualified” is used again in line 105 without any explanation of its meaning. The reader has to read past two pages to understand what these expressions mean. These expressions need to be specified early on if they are going to be used.\n- Line 183: refers to Theorem 1. The problem is that Theorem 1 is in the Appendix, which makes the paper less self-contained. It would help to include a little description of the specific cases that Theorem 1 analyzes: in other words, to move the explanations from lines 216-217 to inside the sentence that starts in line 183.\n- The important information that only half of the iterations are used to fit the bound in page 5 should be moved from footnote 1 to the main text of Section 3.2. This is an important experimental detail which is odd to relegate to a footnote.\n\n**>>Other presentation issues:**\n- Many references need to have parenthesis before the names of the cited authors; this is used when one refers to the *work* and not to the *authors themselves*. Use “\\citep{}” for this. This occurs throughout the paper and needs attention. The first time this problem appears, for example, is in lines 026-027 (it should start with “(Garipov et al, 2028; …)”). Similarly, it should be “(Krogh & …, 1991)” and “(Hoerl & …, 1970)” in lines 054-055. Multiple other parts to fix exist throughout the paper.\n - Lines 045-046: How is it that the spectral properties of the Hessian indicate “convexity”? Does it have to do with local curvature or something related to it? Please explain.\n- End of line 088: should say “for the fixed value $\\alpha=0.5$”.\n- Section 2.1: specify that $g_t:=g(w_t)$.\n- Line 140: it should say “applying Jensen’s inequality”.\n- Line 158-159: it should say “We show the upper bounds on the loss in terms of different learning rates (...)”.\n- Line 160: it says “see appendix”. Which section of the appendix does it refer to?\n- Line 161: it should say “in Table 1” instead of using the preposition “from”.\n- Line 430: include citation from where Muon comes from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iyVIOSqQ0i", "reviewer": "Reviewer_CJJk", "summary": "This paper derives the dependency of the last-iterate loss on the sequence of the learning rate for the SGD algorithm under the assumption of convexity and bounded gradient. From the fine-grained upper bound based on the specific learning rate values, the paper derives $i)$ asymptotic bounds on the loss that depends on the peak learning rate, the optimal convergence rate, as well as a qualifying exam of the learning rate schedule that achieves the optimal convergence rate. The paper also presented an extension of the regulating dynamic to the setting of deep learning, and observed empirically the coincidence between the theoretical prediction and the experimental convergence. Lastly, the paper presented a scaling law governing the model size, the number of iterations, and the reference learning rate.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The result of the paper applies to any learning rate schedules, rather than just the cosine and WSD in Schaipp et al., 2025.\n2. The paper empirically verified the similarity of the empirical training behavior of deep learning models and the theoretically derived last-iterate convergence.\n3. The paper also derived a qualifying exam for the learning rate schedule and a scaling law of the learning rate schedules.", "weaknesses": "1. The paper assumes a uniform bound on the expected gradient, which in many case does not hold (e.g. in the training of neural networks based on MSE loss). The following theoretical bound are based on this assumption, which is probably the reason why in the experiment part the paper has to adopt the Adam optimizer instead of the regular SGD (to condition out the influence of the gradient norm). In the setting of the deep learning extension, this uniform upper bound factor $G$ is modified to be a more sophisticated term that may not solely depend on the gradient norm, which can be even harder to track.\n2. The qualifying exam of the learning rate schedules takes a complicated form that requires several integrations. This can be hard to apply to a general learning rate schedules. The paper can consider to incorporate several examples that explicitly apply this qualifying exam and validate the result using experiments.\n3. The dependency on $\\eta_{\\text{ref}}$ is not discussed thoroughly in the paper. Although is can be understood that the paper's focus is on the scaling of $T$, it is certainly the case that not all choices of $\\eta_{\\text{ref}}$ will lead to a meaningful convergence, which is not accounted for in this paper.\n4. Some technical part lacks clarity. For instance, it is not clear how to obtain Eq. (2.4) from Eq. (2.3). Moreover, the definition of $N_{\\text{small}}$ and $T_{\\text{small}}$ in the scaling law part is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)