On the Provable Separation of Scales in Maximal Update Parameterization
Authors: Letong Hong
Zhangyang Wang
Venue: ICML 2025 poster (2025)
Keywords: Maximal Update Parametrization
hyperparameter tuning
zero-shot transfer
stochastic differential equations
Primary area: deep_learning->theory
TL;DR: We establish the first fundamental separation of scales in µP between macro-variables (e.g. loss landscapes) and micro variables (e.g. individual weights).

Abstract: Maximal Update Parameterization ($\mu$P) has shown significant promise in allowing zero-shot hyperparameter transfer across neural network scales, reducing the prohibitive cost of hyperparameter tuning for large models. However, the theoretical foundation behind the observed approximate transferability of hyperparameters remains underexplored. Relying on a width-dominance regime, which ensures that as width grows, certain terms of the learning dynamics dominate, we establish the first fundamental separation of scales in $\mu$P between macro-variables (e.g. loss landscapes) and micro-variables (e.g. individual weights). Our formulation explains why hyperparameter tuning can be effectively performed in early training stages, i.e., \textit{early statistics effectively approximate global hyperparameter optima}, implying the potential to further reduce the training costs required for searching optimal hyperparameters. We further apply our main theory to explain an empirical deep learning phenomenon discovered independently by prior work.

On the Provable Separation of Scales in Maximal Update Parameterization
Letong "Carina" Hong"!
Zhangyang "Atlas" Wang"
Abstract
model training pipelines adopt similar, ad-hoe approach: they identify hyperparameters from the first few epochs and assume those settings generalize throughout training to avoid prohibitively expensive full-run searches. While widely used in industry to reduce cost, this early-stage strat- egy lacks a rigorous theoretical grounding, motivating our work. Specifically, it is unclear how the dynamics of opti- mization adapt changes in model size and interact with μP to preserve hyperparameter validity. Bridging this gap requires deeper insights into the interplay between parame- terization, training stability, and the loss landscape across scales. Such understanding could further enhance the ro- bustness and efficiency of hyperparameter transfer.
Maximal Update Parameterization (nP) has shown significant promise in allowing zero-shot hyperparameter transfer across neural network scales, reducing the prohibitive cost of hyperpa- rameter tuning for large models. However, the the- oretical foundation behind the observed approxi- mate transferability of hyperparameters remains underexplored. Relying on a width-dominance regime, which ensures that as width grows, cer- tain terms of the learning dynamics dominate, we establish the first fundamental separation of scales in μP between macro-variable (e.g. loss landscapes) and viriallvs (e.g. individual weights). Our formulation explains why hyper- parameter tuning can be effectively performed in early training stages, i.c., early statistics effec- rively approximate global hyperparameter optima, implying the potential to further reduce the train- ing costs required for searching optimal hyperpa- rameters. We further apply our main theory explain an empirical deep learning phenomenon discovered independently by prior work.
This paper formalizes a novel scale-separation frame- work, showing that the macro-level descriptors (e.g.. gra- dient norms, loss landscapes) converge at (11/n), while micro-level variables (individual weights) only converge at (LL/VR). This theoretical lens provides an immediate explanation for the ad-hoc industry practice of "early-stage" hyperparameter tuning: even if individual weights are far from their eventual values, large-width models induce stable macro-level signals that can guide hyperparameter selection. As result, optimizing hyperparameters model or a small fraction of training steps can approximate globally optimal hyperparameters within (11/n) error po- tentially reducing the cost of full-run searches substantially in large-scale neural network pipelines.
smaller proxy
1. Introduction
The success of hyperparameter transfer in deep learning relies critically on the choice of network parameterization. Recent work introduced the Maximal Update parameteriza- tion (nP), which enables reliable μTraansfr of hyperparam- eters from small proxy models to larger ones ( (Yang et al., 2021). This breakthrough allows practitioners to efficiently optimize hyperparameters on smaller, cheaper models while retaining effectiveness when scaled to larger architectures.
Our work introduces several key technical advances:
We introduce a stochastic differential equation (SDE) framework that cleanly separates macro (loss land- scapes) and micro (individual weight) variables in large-width neural networks. This leads to a precise un- derstanding of why hyperparameter tuning can succeed early in training, marking the first rigorous application of such SDE techniques to hyperparameter transfer.
However, the theoretical understanding of whether or why early-stage tuning can transfer effectively across different model scales remains limited. In practice, many large-scale
We develop new mathematical treatment of how loss landscapes vary with network width n, proving that macro-level descriptors converge at (1/n)) while micro-level weights evolve rigorously more slowly. This width "regularity" underlies an (11/n) bound on hyperparameter transfer error, providing a principled foundation for zero-shot transfer across scales.
'qqual contribution 'TT AI Labs, XTX Markets. Authors listed in alphabetic order (a-9). Correspondence to: Atlas Wang <Atlas. Wang @ xtxmarkets.com.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We extend our macro-micro separation framework to

---

On the Provable Separation of Scales in Maximal Update Parameterization
incorporate integrated learning rate descriptors, yield- ing novel theoretical account of the recently observed "delay" phenomenon between dynamic learning rates and validation loss (Tissue et al., 2024).
classification task involving latent word correspondence structures, they demonstrate that gradient descent dynamics in symmetric two-headed transformer with ReLU neurons follow a three-stage progression (Yang et al., 2025).
In the early stage, the hyperparameter decisions are found to be critical, as during this period the network is still close to initialization where the theoretical guarantees hold, and once they are locked in, later training dynamics become more robust (Cohen et al., 2021). Empirically, it is folklore in industry that optimal hyperparameters can be determined in early training without having to wait until training comple- tion (Goodfellow et al., 2016). There are many algorithms for early halting, such as the irace package (López- -banezz et al., 2016; Birattari et al., 2002), successive halving (SHA) (Jamieson & Talwalkar, 2015), asynchronous successive halving (ASHA) (Jamieson & Talwalkar, 2015), and hy- perband - which applies SHA and ASHA multiple times (Li et al., 2018). (Egele et al., 2024) confirmed that even training after one epoch the hyperparameter optimization obtained often achieves competitive performance.
2. Related Work
Our framework complements classic wide-network theo- ries. Neural tangent kernel (NTK) analyses (Jacot et al., 2018), which linearize training in the infinite-width limit, illuminate macroscopic behavior through a fixed kernel but miss finite-width effects and individual weight trajectories. Mean-field approaches (Mei et al., 2019), conversely, track microscopic weight fluctuations yet do not explain the rapid stabilization of aggregate quantities such as loss landscapes or activation norms. Our finite-width, multiscale perspective extends these insights, revealing that global signals stabilize early- well before individual weights converge - and reliable guides for hyperparameter selection. Our main proof technique is based on the SDE framework used in (Xie et al., 2024), which we will discuss in Section 2.3.
2.1. pP Theory of Hyperparameter Transfer
2.3. Scaling Laws for Hyperparameters
Historically, hyperparameter tuning has been a costly step when scaling up models. Practitioners often need to retune from scratch for bigger models slow and expensive process. Utilizing tensor program formulations giving rise to a Gaussian Process interpretation of deep neural networks, the theory of maximal update parameterization (Yang & Hu, 2021; Yang et al., 2021; 2024a) provides theoretical principles that ensure consistent training behavior across different width scales of the model, enabling "zero-shot" transfer of hyperparameters from narrow to wide models.
Given the importance of hyperparameter tuning, various empirical studies attempted to fit scaling law of optimal learning rate with respect to model size. Kaplan and Mc- Candlish et al. found that larger models require a smaller learning rate to prevent divergence, while noting also there may be a dependence on network width (Kaplan et al., 2020). DeepSeek Al found that the optimal learning rate gradually decreases with the increase in compute budget C (dee, 2024). However, the empirical scaling laws may be affected by the various set-ups, such as the independent variables of choice and the underlying fitted data (Lingle, 2024).
Subsequent work further explored how μP extends to trans- fer across models of different depths (Bordelon et al., 2024) (combined with techniques such as dynamical mean field theory (Bordelon & Pehlevan, 2022)), different widths and depths (Yang et al., 2024a), and even varied network ar- chitectures (Chen et al., 2024). More recently, Lingle con- ducted large-scale experiment testing the effectiveness of μP (Lingle, 2024). They found that upon empirical verifica- tion of up to 10B transformers and training budgets of up to 1 190B tokens, u-Transfer works as intended for the majority of important cases and enumerated the exceptions.
A recent work (Xie et al., 2024) predicts training loss and optimal learning rate schedules, deriving convergence rates and escape probabilities under time-inhomogeneous SDEs, and providing an empirical scaling law for hyperparameters. In comparison, we rigorously develop width scaling theory within μP to specifically analyze the separation of scales. Unlike (Xie et al., 2024), whose width-agnostic treatment applies to any non-convex problem, we impose extra local convexity or smoothness on the meta-objective. This nar- rower scope forfeits some generality but lets us rigorously prove scale-separation results for hyperparameter choice findings that, in practice, still hold across nonconvex neural networks (as seen in our experiments).
2.2. Early Training Dynamics
Prior work has analyzed neural network training through the lens of gradient flow dynamics, revealing distinct phase- dependent behaviors. Recently, Yang et al. identified a two-stage training process naturally partitioned by gradient flow patterns, showing that training each regime separately with tailored algorithms simplifies learning rate schedules (Yang et al., 2024b). Meanwhile, in a two-mixture linear
3. A Prior: Early-Stage Coarse-Graining for Hyperparameter Tuning
We first show that, under suitable assumptions about model parameterization and how global descriptors converge, the
2

---

On the Provable Scparation of Scales in Maximal Update Parameterization
optimal hyperparameters for early-stage training approxi- mate the optimal hyperparameters for the full training.
Assumption 3.3 (Early-Stage Macro-Level Self-Similarity) The early-stage statisties (e.g., activation variance, gradi- ent norms) concentrate around stable values and exhibit self-similar behavior when subjected to coarse- gaassing transformation (such as a time integral or moving average). Formally, there exists a stable descriptor X (t) such that
Theorem 3.1. Consider a neural network parametrized using μ.. Let nt(t) denote the hyperparameter set at train- ing step and n/" (lo) be the hyperparameter optima found during early-stage training t E [0, to). where to < T. We further define TI(t) to be the hyperparameter configuration that, in the infinite-width limit, aligns with stable macro- level statistics observed up to step t. The coarse- naase--raiied hyperparameter set Ti(t)) approximates the optimal hyper- parameters 1f" for the full training duration T:
X(t) ((x(T). (0x(T).55.. T (t).,,
for t E [O, to]. with deviations of (1/n)) at large width.
Here, R represents a (coarse-granning operator-sut as S dr or weighted average-while (t) is the stable limiting reference for (T)). Empirically, many large-scale training runs show that, after a brief "burn-in" period, global descriptors like activation norms and gradient magnitudes vary minimally, supporting this assumption in practice.
((o) x 2y",
with an error bounded by O(1/n).
3.1. Assumptions
The first assumption is fundamental in the theory of μP, where the cumulative drift is limited due to finite-width effects, ensuring that those effects do not blow up or over- shadow the error scale we derive.
Macro vs. Micro Variables under μP. Throughout this work, we decompose all observation variables during the network training process into two classes of variables:
Assumption 3.1 (Width Dominance). The width of the neural network is much larger than training time: n >>>>> T.
Micro-variables, which capture fine-grained, local changes in the parameter space. A canonical ex- ample is the deviation of weights from initialization, p(t) 0(t) ((0). This micro-variable jut) tracks how individual parameters drift during training. Un- derstanding [|x(t]] or its coordinate-wise fluctuations helps quantify slow, localized changes in the network.
The next assumption recalls μP scaling (Yang et al., 2021).
Assumption 3.2 (μPP Scaling). The model parameterization follows uP rules, preserving maximal feature learning:
Hidden weights W E Raxn are initialized as W ~ No0, 1/n) with learning rate 17 Nju-
Macro-variable which represent global or aggregated statistics of the network. Typical macro-level descrip- tors include activation statistics A(t) (c.g., layerwise means/ variances), gradient norms G(t), or the global loss L(t). These variables concentrate and stabilize faster large width (Bordelon & Pehlevan, 2022).
Input weights V are initialized as V ~ ~N0 1) with learning rate 7 = non.
Output weights V are initialized as V' ~N(0,1 with learning rate 17 no/n.
An additional assumption in our framework is that, at suffi ciently large width, certain coarse-level descriptors of the network- - such as layerwise activation variances, gradient norms, or average feature magnitudes - stabilize quickly and exhibit a near time-invariant (or slowly varying) be- havior over an early training window 40). We term this property "self-similarity under coarse-graining."
Under Maximal Update Parameterization (μPP), the update rule for the network parameters (t) at training step t is
0(t
(tt 17 VL(((t)),
where rj is scaled by 1/n (or a similar rule) to ensure stable feature learning across layers.
'This assumption has strong empirical and thcoretical support from wide network studies. On the theory front, prior juP work (e.g., (Yang& & Hu, 2021; Lingle, 2024)) shows that layer-wise statistics such as activation variances and gradient norms stabilize early in training and remain nearly invariant across widths, con- sistent with infinite-widde theories where macro-level descriptors concentrate, as common in neural tangent kernel and mean mield analyses. On the empirical front, recent large-scale experiments (e.g.. (Vyas et al., 2023; Dey et al., 2024)) also confirm that op- timal hyperparameters stay stable across model scales, indirectly verifying the training dynamics exhibit self-similarrty.
GoarsiGG Transformation on Macro Variables. Motivated by the observation (Assumption 3.3) that global statistics often exhibit a self-similar or quasi-stationary be- havior early in training, we define a coarse-graining operator R acting on these macro descriptors. Concretely, for any lecro-level function ft) (such as A(t) or G(t)), we set:
T(t) R((()), where R(f(t) S. [(() dr.
3

---

Hyperparameter Evolution via A() and G()). In par- ticular, if (t(t) denotes the hyperparameter set (e.g.. learning rates, momentum) at step t, one may write its update rule as On the Provable Separation of Scales in Maximal Update Parameterization
The idea is that So [(T) dr averages (or integrates) the macro-level signal over training steps, smoothing out short- term fluctuations while preserving the long-term stable trend. If ft) is reasonably stable, then f(t) provides a representa- tive value upon which hyperparameters can be adapted.
n1, the actual optimal hyperparameters width n, 77" differs slightly. By the law of large numbers and concentration arguments at width n, fluctuations in activation variances or gradients decay as (1/ In j.P theory, such fluctuations sum to a final (1 /n) mismatch between the infinite-width and the finite- width optimum (Yang et al., 2021; Yang & Hu, 2021).
Thus, provided that (i) to < T, which does not overly constrain hyperparameter tuning, and (ii) > T, which ensures early-stage tuning is both sufficiently representa- tive (small carly-stage error) and robust under finite-width corrections (small finite-width error), these two sources of error add up to (1/n), as desired.
1)) A((t)) A(t), a(t))
where A(1) = (A(()) and G(()- = RG(!)) are coarse- grained versions of the activation / gradient statistics. In other words, the macro-variables first go through smoothen- ing of local fluctuations, and then get pushed by the update rule. At large width n, under μPP stability, these macro- level variables converge reliably, allowing vi()) to track wear-fixed-pord pointdponnt r/:
4. Main Result: Separation of Scales in μP
We now present our main result that under μP parameteriza- tion, loss landscapes converge at rate (11/n) with network width, while individual weights evolve more slowly at rate @(1/ (n) This explains why hyperparameter transfer can work effectively even before individual weights have con- verged their infinite-width behavior.
F'rr, A(t), G(t) A(t))((() = n""
reflecting that once macro-variables and hyperparameters reach a stable regime, their further updates remain small. The fixed-point analysis here leverages the notion that micro- level fluctuations in ((t do not prevent the vacro-variables from stabilizing, thereby guiding ntt) toward 17" +
Theorem 4.1. Consider a neural network of width n trained under μP parameterization. Let M denote the expected training loss after T steps with learning rate 1) (macro- variable)- in other words, M (n) EL(n) and p(t) 6, 0g denote the deviation of weights from initialization (micro-variable Under suitable conditions, for any 8 > O. with probability at least 1 - 8:
Finite-Width vs. Etrly-Stage Errors. At large width n, the difference between the coarse-grained hyperparameters (tto (determined at early time to < T) and the global optimum 17** can be decomposed into two parts:
1. For any In - -2l = 0(1) (M - (112) O(1/n).
(m(()) - 77" S (7(4)) (" (0o) + (n" (co) "
(Fast convergence of loss landscape).
(Early-Stage Error)
(Finite- Width Error)
F.For any fixed E [0, [[((]]]] 6(1/v7). (Slow convergence of weights).
Here, (Do) denotes the optimal hyperparameters one would find in the infinite-width limit, ignoring finite-width corrections. We interpret and bound these two differences:
4.1. Assumptions
Assumption 4.1 (L-smoothness). The loss function L(-) is L-smooth (standard in (Nesterov, 2014)):
1. Etrgy-Stage Crraiiin--Duration Error: (T((a)
1f (. This captures the fact that 774)) is determined from only to < T carly-stage training data. If to < T, it might not exactly match the infinite-width global op- timum r) (co) that fully accounts for the entire training horizon. As n - oc, though, the macro-level statis- ties (activation / gradient norms) quickly reach near- stationarity, meaning that hyperparameters deduced from the first to steps suffice to approximate the true infinite-wide width optimum 17" (co). This is observed empir- ically when early training reveals stable signals about learning-rate rate, momentum, etc. (Lingle, 2024).
[[V[(x) - V s L|[x - vll. Yo, y E Rdd())
The second assumption ensures that the noise introduced by different data points during the estimation of the gradient is unbiased and independent, given the weights:
Assumption 4.2 (Unbiased Estimator). Given any x E N assume that the entries of VF Gi) V f((x) are i..d. Gaussians (0, Σg) for all i E (D) where Eg is given.
2. Finite- Width Error:
|n* - r/" Even with infinitely-man many training steps to find 17" (co) at infi- nite width, real networks are at a finite but large width
These two assumptions lead to a concentration inequality whose proof is in Proposition 2 of (Xie et al., 2024).
Proposition 4.2 (Gradient Noise Trace Boundedness). The
4

---

On the Provable Separation of Scales in Maximal Update Parameterization
stochastic gradient follows:
This leads to:
vi(x)= = (L) + E, where & (00, (x))
T((VLL(.))(0..))) (F000)00000000000000000000011113
S [Vdd()--(v))) - = O(n).
with covariance (xx) satisfying the trace bound:
The subtle point is that the Lipschitz constant itself decays as L == On -1/2) under μP scaling, so the product remains only linear in instead of n//
TT{[(x)) - TM
with probability 2 1 eexp(-DI (AAi-Di//(AA(//) (4TH(E)) 2lo2)) where D is batch size, Bg is base covariance, and og its maximum eigenvalue.
Itô decomposition of the loss. Because L(O,) is twice differentiable, Itô's formula gives
The third assumption is a standard regularity guarantee ensure the convergence of mucro-variable under increasing width, for example see (Soto, 2024). It holds for standard ac- tivations (e.g., ReLU, GELU), and similar assumptions are employed in the neural tangent kernel stochastic gradient descent literature and μP papers themselves?. We emphasize that the bound is for the entire training path difference in the function space. not just local Lipschitz gradient differences.
dI.(0.) = (VL(),), do,) + L10
=-1I|VL(0,]1 //t v//m(VL(00).000))dd
" TV (()))())))
2n
where the last term comes from d0,d0, = (0())
The second-order term in the Itô expansion carries a prefac- tor 7/n = O(n ?); hence (n/n) Tr(v Σ) = Onn the same order as the drift term - Note that with- out Lemma 4.4 the trace could scale as ((n2322, the diffu- sion term would dominate, and the O(1/n) macro rate used in Proposition 4.6 could not be proved.
Assumption 4.3 (Width Regularity). For any widths n1 < n2, there exists a mapping o Id(())) + (d(nz) such that:
[[O0 m1) (())))((((()))) m()(0); S Cl - ngl/m1
4.2. Proof of Theorem 4.1: Part 1 (Macroscopic)
Taking expectation and integrating up to & (the martingale term has zero mean) we obtain:
We start by approximating the discrete SGD updates by a continuous-time SDE, by Euler-Maruyama discretization, under standard small step assumption:
B[I(O.) I((o) = - I. B B
= -1==-)\V((.)dd + Vn//no(O.)dv
2n L. ET
where (0.) E((O.) and W, Brownian motion.
By the O(n) trace bound above and 7 (1/n), both integrals are ((1/n). Hence the total drift of M M n) over T steps is (11nn) which precisely yields the statement of Lemma 4.5 used in the proof of Proposition 4.6 later:
We first establish how the macroscopic variable M (17) re- lates to feature learning in μP through the following key lemmas, whose proofs are deferred to the Appendix A.
Lemma 4.3 (Gradient Scaling). Under μP and Assumptions 3.3, for any fixed
Lemma 4.5 (Finite-horizon loss drift), For any fixed train- ing horizon T and every width n,
-
M(rn) - L(0g; = (11m).
Lemma 4.4 (Hessian Scaling). Under uP and L- smoothness (Assumption 4.1).. for any 0,:
We are now ready establish a fundamental result of width scaling, on the expected T-step training loss:
[())( & d()
Proposition 4.6 (Width Scaling in pP). For widths nı n2. under A Assumptions 4.1 4.3, let
where d(n) is the parameter dimension scaling with
M M n) E [<([[((r;n]]
"For example, (Yang & Hu, 2021) assumes that activations are smooth (or smoothed RcLU), and many proofs invoke pscudo- Lipschitz or bounded derivvaiveevv conditions to enable law-of- large- numbers-type arguments to ensure that descriptors like average activation variance loss stabilize width increases empiri- cally supported and analytically tractable.
be the expected loss after steps of training with learning rate yi laken over the training noise &. Then,
[M M(1; S 011 min(n1,
5

---

On the Provable Separation of Scales in Maximal Update Parameterization
Proposition 4.6 immediately implies that for any infinite se- quence of increasing widths ngg, (1; rig) forms Cauchy sequence, giving rise to the limit M (n; oo) and a conver- gence rate of O(1/n). Formally, this is stated as:
Choosing a + and pulling it back to the na()) evolution SDE, 4 E [I(e(t)]l is upper-bounded by the right side: at
nL E 1m(O1) + T. EIVLOOI +
2 E[T E(O))
Corollary 4.7 (Existence of Limiting Landscape). Under Assumptions 4.1- 4.3, there exists a limiting landscape (1); xo) such that (m: - M (m; xc) S ((L/n). Proposition 4.8 (Transfer Error Bound). Let n² (n) be the optimal learning rate for width n. The transfer error is:
= =0
+
Note that under μuP scaling, we have = (11nn) for hidden weights². Meanwhile, by Proposition 4.2, (r,)( is close to the base covariance Tr(Eg), which is (
We write d dt =(t)) S c (() + C, X(0) = 0 for X(1) =
(n) (2)) - (n) (n2): n2) S O1/ min(n1,7m n2)).
E (I)(t) with c == Tmma (1/nn) and C == ((1/n).
Indeed, by Proposition 4.6, the difference of M(j" (m1)) m2) and M(r (v/ (mi n1) is bounded by (1/ min(m), n2)). ). By optimality of 17 (m1) for width nı and Proposition 4.6,
By Grönwall's inequality, solving this gives
X(t) & - -ee -1 = O(1nn)
0 O
M(i (m)): & M(1" (12); < <M(n (n2): n2) + ((1 min(miin n2)).
because ct = (1//n) for fixed horizon T. Hence
EE [|((0))]= 0(1/n) = E [|(((|| = 0(1/m))
(11())) (1) as a com- Using the triangle inequality with M (1)" mon anchor, we have the desired result.
The 0(1/vn)) upper bound proved here is the first half of the tight ((1/ Vz) rate; the next subsection completes the argument with an (1/)n) lower bound.
Having shown that the loss landscape itself varies only (L/nn) with width, we now show that the minimizer of that landscape with respect to 1 inherits the same regularity:
4.3.2. PART 2.2: TIGHT (LOwer) BOUND
Lemma 4.9 (Learning Rate Transfer). Let 1f" (n) wrgminM(n) n) be the optimal learning rate for width n. Then under Assumptions 4.1 for ny < n2:
Building the differential inequality derived in $ 4.3.1 we now establish E [|ı(!)] 2 c//i.
Lemma 4.10 (Non-Vanishing Diffusion Lower Bound). Fix a width-indepennenn horizon t E [0,T] and suppose As- sumptions 3. 1-3.3 and 1.1.33 hold. Let the parameter displacement satisfy the SDE
In -nn (mz)) = O1// mi)(
That concludes the first part (macro-variable) proof.
n(() (0,)) dW, aOT = E,
du(()) = -n( VL(O.) di+ V
4.3. Proof of Theorem 4.1: Part 2 (Microscopic)
For the microscopic variable, we first establish the upper bound ($ 4.3.1), then proceed to the tight bound ($ 4.3.2).
where the LR obeys < Mmin S n(() S mmax O(n-)).
Assume 'mmx S Tmmin with 7 independent of n. and there exists constant (0,1) independent of n such that
1. PART 2.1: UPPER BOUND
Pr Th E(+,) c_ for all s E &(0,T] 2 1- QQ(n)
From the SDE representation, the evolution of p(t) follows:
du(t) = d0,- d.) + (n))o
Then, with sufficienlly small for every t E [0,
C- Tmina t Cn,
B[[la(t)]PP >
Consider the evolution of (t] just derived in the macroscopic variable part (using Itô lemma and taking the expectation for the Itō martingale's 's mean to vanish):
2n
where C (L, Mmax: T) is width indhh-inddeendeet. Moreover
there exist width-independent constants ca(t), > 0 such that
dt [[] =
(((() -
VL(O.)) 2 "E(T))())))))
n
ei (t) 112 1/2 I E[lla(01] S n2 1/2
Vt E [.,
(3)
By Young's 's inequality, for any a >0 0
In particular, for any fixed to > 0, (1))((1))1 = (
(u(t), vL(),)) 2 2a ||(t)]|2 a
1
We clarify that this refers to the learning rate per parameter, not the total per-layer update.
2
6

---

On the Provable Separation of Scales in Maximal Update Parameterization
Proof. Throughout the argument we write C& > 0 for generic constants whose numerical value may change from one line to the next, but which depend only on fixed model and schedule- parameetrss (T, L, Jmin, 7max C_, bounds on ||]|| etc.) and never on the network width n.
Step 5: fourth-moment bound. Because the drift
(t)) VT, and diffusion (n(t)/no are globally Lipschitz
(in 0) with linear growth, standard SDE estimates (Oksendal
& Øksendal, 2003) give, for any k 2 1, Bi]a(t)]| 1 C1 n n-n In partieular,
Step 1: L differential inequality. Apply Itô's formula:
Bl|((111* & Cin
(6)
d d ([))( 22m(t) (()0)) E6 ETTT [00)]
n(
+ n
Step 6: Paley-Zygmune to upgrade to L'. Let 2 :=
di
lu(t) From (2) and (6), IEZ 2 an -1 and EE <
(4) Note this is the very same differential identity that was established in $ 1. we will simply apply it in the oppo- site direction, direction, retaining only the always-positive diffusion term.
Cin with a = 3c-7mmin C > once n is large. Pa-
ley Zygguudd implies Pr|Z 2 1 JEZ] > a²
i0 - 5(1) > 0.
Whenever Z 2 EZ we have [|{(t)]] 2 /a (2n)
5(t) T
[|] 2 Priz 2 EZ
v2 n2 1/2
Step 2: bound the drift contribution. Because L is glob- ally L-smooth (Ass. 4.1). (1, VL)= I [1ul + 21 V Taking expectations and using Lemma 4.6
2n
VL I Co).
Concavity of x + Vi gives the matching upper bound B|((t)]| S vu(t) S c2(t)) 1/2 11/ This proves (3).
((t) L
B/ V 2 -=n(t)[[(llll
Co-
Implications of Theorem 4.1 and Corollaries Our anal- ysis reveals a fundamental separation between macro- variables (loss landscapes) that converge rate (l/nn) and micro-variables (weights) that evolve more slowly rate @ 1/ /n). The separation of scales is intimately connected to feature learning (Yang et al., 2024a), supplementing prior work (Vyas et al., 2023). The (1/nn) convergence of M M()) to a well-defined limit means the loss landscape stabilizes quickly with width while preserving learning capacity, and guarantees reliable transfer of learning rates across scales.
Step 3: bound the diffusion contribution. From (1) and the union bound, [T E(0.]] 2 zc-n for all E [0.7] once n is large enough, hence n(() E E(e,)] 2 1 c- Mmin-
2
Step 4: solve the inequality. bine Steps 1-3 to obtain
Let y()) = ( Com-
bb((t) nlt)) Co. : LMmax = OnO
(tt) 2 2 Mmin - c-min
L
(5) Note nt() 7max On- and 7mmax S Mmin- If is small enough such that 4C07= S L.L. Then for all widths 40o7max < 4Com I c. mmins so the last term in (4) can be bounded by c-min. Set
5. Interpreting Learning Rate Scaling Laws and Delay Phenomena
Recent empirical work (Tissue et al., 2024) suggests that cross entropy training curves can be described by scal- ing laws involving integrals of the learning rate, and that changes in learning rate produce "delays" in validation loss response. We now incorporate integrated learning rate de- scriptors into our macro-level variables, showing that these scaling laws and delay effects arise naturally. The original notations are extended to our theorem statement.
a
c_ 7min > 0,
and note a is width indeeeededdn..n.. Then (5) simplifies to
i(t) 2 a- - by(t),
v(0)= = O.
We introduce memory kernel q(7) modeling the effect of time-varying learning rate momentum in continuous limit: for example, the same concept of memory weight captures how past states influence the current update in the Generalized Langevin Equation and models integral equa- tions in the dynamical mean-field theory for SGD literature. Throughout we assume q: : 20) - TR is measurable, JS ((T) dr 1, and sup, q(T) < co.
Solving it gives
v(1) 2 2/ (1-
> al abbt?
where the second inequality uses -M < 1 - x + 3 As
bt = 11 for any fixed horizon 1. 1 T, we have
v(t) N _ = In
Theorem 5.1. Under the Maximal Update Parameter- ization (μP), consider an infinitely wide neural network trained with a time marying learning rate rj(t). We define
This establishes the lower bound (2).
7

---

On the Provable Separation of Scales in Maximal Update Parameterization
the integrated learning rate variables as follows:
Vanishing noise. Because the per-parameter learning- rate satisfies n7(s) = ([1/n) and Tr {((,,) (n), the covariance of the stochastic term in the SGD step is 77(s) Tr ( = (1/mz). By a matrix -ernsteii inequality, which is a standard exponential decay result, the cumulative noise over any finite horizon converges to almost surely as n - XC.
Si(t) (1(t)= ((t))s[(t ((()) ((T)dr dr, S2(t) sa(t)= S(² n(T))(T) dr.
Based Q(T) assumptions, Sy(t) is finite and non-negative. Assume that the loss landscape, when reduced to a one- dimensional effective manifold parameterized by x², sat- isfies t(x) x Lo + 1'x="' for large x and some A' and a' > O, in addition to the conditions in the following subsec- tion, which guarantees stable infinite-widtt wintt dynamics and concentration of stochastic effects.
(ii) ODE limit. Write p(n) for the total number of train- able parameters at width n. so the ambient space is TR(()) With (i) and the uniform I-smooth gradient bound,
weaak-converenne theory (Gra- ham et al., 1996) implies that the stochastic trajectory O. E po(n) converges (in the Skorokhod topology) to the deterministic solution of
Then there exist constants A, C. a > O such that as +
L(t)= Lo + (!) (t)
d0 (1) 7o (t))
8" (0) = 0g.
dt
The- (C) term encodes a systematic delay in how the loss L(t) responds to learning rate changes, as it incorpo- rates historical learning rate information.
Consequently, the discrete update rule
0c-dt = 0; (et=nt))((()) dt + +n(t))(()))1) aw. reduces in the - do limit to d0,/dt -n((t)
5.1. Assumptions
Assume L--) is L-smooth, as in Assumption 4.1; as well as bounded gradients under pP: we ignore (-dependency and
We assume that the dominant training dynamics can be cap- tured by a single scalar variable x(t) parametrizing a stable, low-dimensional manifold of interest, i.e. there exists a function 6 such that L(O,) = ((x(t)) for suitably chosen sr(t). On this manifold, we write dz dt = - (t))((t)), for some positive, smooth function g(x). Intuitively, g(x) is the gradient magnitude along the direction of descent. We assume 0 < C1 S g(x) S C22 along the trajectory; this holds for cross-entropy with bounded activations.
consider sup,gr [[.]l = (1) uniformly. 5 Further:
Assumption 5.1. Stable Infinite-Wiod Limit Under pP: As n + xc, define an inftnite-width limiting trajectory 0* (1) that solves the deterministic ODE:
d0 (1) dt no (0" (t)),
where g(0")= = 1im (0) scaled appropriately. This stable limit exists as uP ensures no degenerate behavior.
We also recall the assumption that for large x, {xx) Rs Lo + A'a-a a' with a' O. This assumption means that as training progresses (reflected by decreasing æ). the loss approaches Lo with power-law decay in terms of x.
Assumption 5.2. Macro-Level Descriptor Stability: Let xx,(1) (t) be a macro-level descriptor, e.g. the layer-wise acti- vation variance. Assume
From de = - -g((())( (tt), we separate variables and derive dx
xx, = (t) (t) (())))))))) O11/n), V2 to
/ g(x)
- n(() (((() di = -()
for some to < T, where T is the total training horizon and (t) is a smooth, deterministic limiting descriptor. Early- stage self-similarity means that after short transient, (1)
where we recall Si = S dr.
hovers close stable value trajectory X* (t).
Ast + co, if (tt) remains positive and integrable, St) + 0o. By integrating both sides and considering the inverse relationship, we are able to express x as function of (():
5.2. Proof
From the above assumptions, the following holds:
a1) = x(S)
*This one- dimensional manifold assumption is typical in sim- plifying large-wide wighh analyses, such as the standard neural tangent kernel or dynamical mean field theory approaches.
For large x and correspondingly large S1),,, standard asymptotic inversion yields a power-law type relationship. Specifically, there exist constants A, a > 0 such that
"Indeed, under JPP, at any time f, the per-layer learning rate should still scale with width to ensure bounded updates. Our use of varying 7tt) assumes that such width scaling is preserved pointwise f. as is common in practical p..
{(x(1)) To + (1)
8

---

On the Provable Separation of Scales in Maximal Update Parameterization
Now consider when r7(() is not fixed. We write 17(1) ((()) 5n(() where i(t) is slowiy-varying baseline sched- ule and (n()) is signed perturbation capturing fast LR adjustments. The memory descriptor therefore records only the perturbative part
6. Conclusion and Outlook
We present the first scale-separation theory for hyperparam- eter transfer under μ.P. By coupling an (1/n) macro con- vergence of the loss landscape with tight (L1/Vn) micro evolution of the weight vector, we obtained quantitative er- ror bounds that explain the empirical success of "early-stage" tuning, and learning-rate delay curves. Besides recovering existing uP phenomena, our analysis yields two design guidelines: (i) updates must be co-scaled with width to keep the diffusion term sub-dominant, and (ii) macro descriptors should be chosen so that their p.P drift O(1/n).
S2(t) = S:' (n(()) g(F) dr,
The baseline ODE dng = (x)) determines a reference trajectory, and we set xx() - (t) + x) (t). Linearizing to first order yields (discarding quadratic-order terms)
$1 (t) = it) o' (z)( (t) - 5n(() (())()).
Limitations and future directions. The clean separation relies two ju PP--pccciic facts: (a) hidden-layer learning rates scale
Solving this scalar linear ODE gives
(1) = - -ct) L
1/n, so gradients remain order-one, and (b) global statistics (activations, gradient norms) exhibit early self-similarity. Other parameterizations break at least one of these pillars. For example, in standard ("mean-field") scaling, gradients vanish as n + Do, leading to lazy dy- namics with no timescale gap: NTK scaling freezes features near initialization, again precluding a macro/micro split.
((t)S(()),
with
I(t)= = c ((s))() ds)
C(t) = ((O)(()(()) (0(2)(())
C(t) >
We therefore view generalization beyond μP as the next challenge. A promising route is the a-scaled family (a = 1 recovers uP, a the NTK) and the conjecture that a val- ues closer to may still support a weakened but non- trivill scale separation. Other open problems include (i) adap- tive optimizers (Adam, RMSProp), (ii) the minimal width needed for reliable transfer, and (iii) joint width-depth seal- ing. We believe that the techniques developed here, e.g., Itô decomposition paired with width-regularityy provide a flexible starting point for these investigations.
Starting from the baseline approximation f(x) ~ La + A ATT substituting x(t) = (t) + x (t) and retaining the first-order term gives n - 00
L(t) Lo + (t) (t2
The negative correction -C (t) captures the delay phe- nomenon: learning-rate changes are "remembered" through S2) and only gradually reflected in the loss, matching the empirical findings of Tissue et al. (2024).
Impact Statement
Interpretation. We have shown that the loss L(t) can be expressed as scaling law involving Si and (2). This learning-rate delay phenomenon arises because macro- level descriptors "register" learning rate changes rapidly, but the micro-level parameters actually implementing those changes do so more slowly. As a result, the final loss re- sponds with a lag, reflecting the time it takes for the slower (micro-level) parameter updates to "catch up" to the new learning rate regime. This lag is what appears empirically in training curves: a delay between adjusting the learning rate and seeing the expected shift in the validation loss trend.
This work aims to contribute to the advancement of Deep Learning Theory. We do not believe any specific conse- quences require particular emphasis at this time.
Acknowledgment
The authors extend their sincere gratitude to Alex Gerko, Hans Buehler, the XTX Markets leadership, and the Quant Research team for their invaluable support and for pro- viding access to exceptional research resources. We are especially grateful to the entire XTX Markets New York office for fostering vibrant and inspiring work environ- ment. This work was conducted as part of the XTY AI Res- idency Program (https:/ / www. xt wtxxarkket.... com/ job/2id==274458003)
Intuitively, one can view (1) as the "instantaneous" (1/n)) macro response predicted by Theorem 4.1, while (t) is the integrated residual that accumulates until the (n=11/2) micro dynamics have travelled a comparable dis- tance. Theorem 5.1 thus provides mechanistic instantiation of the scale-separation principle: the faster-equilibrating macro channel yields the leading AST decay, whereas the slower micro channel produces the delayed- -C'S, correc- tion observed empirically by Tissue et al. (2024).
References
Deepseek IIm: Scaling open-source language models with longtermism, 2024. URL hatps://arxiv...

---

On the Provable Separation of Scales in Maximal Update Parameterization
Abs/2441.029955
Jamieson, K. and Talwalkar, A. Non-stochastic best arm identification and hyperparameter optimization, 2015. URL https: /ttpsi///aaaiv////a//aaeeasiiava www.e/iiiorrrooooooooo.oo wrg/abs/g//a.................. haapg//abs/1522// 1tt./1122.000979
Birattari, M., Stützle, I., Paquete. L., and Varrentrapp. K. A racing algorithm for configuring metaheuristics, pp. 11-18, 01 2002.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodci, D. Scaling laws for neural language mod- els, 2020. URL https: //rxivv org/ arggabs///e abs/2001 08361.
Bordelon, B. and Pehlevan, C. Self-consistent dynamical field theory of kernel evolution in wide neural networks. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 32240-32256. Curran Associates, Inc., 2022.
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization, 2018. URL https: / //apxxv.og//aaa///../ http://wwwgg//////....... 06566
Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehle- van, C. Depthwise hyperparameter transfer residual networks: Dynamics and scaling limit. In The Twelfth International Conference on Learning Representations, 2024. URL https: http///wweeeeeeeeeeeeeeeeeeeeeep net/ forum? id=KZJehvRKGD,
Lingle, L. large-scale exploration of j--ransfer, 2024.
URL https: htt.:sss/aaaavv..rr/////////////
López-lbáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Birattari, M., and Stützle, T. The irace package: Iter- ated racing for automatic algorithm configuration. Op- erations Research Perspectives, 3:43-58, 2016. ISSN 2214-7160. doi: https///bb..ogg////006....oppppp
Chen, W., Wu, J., Wang, Z. and Hanin, B. Principled architecture- aware scaling of hyperparameters. In The Twelfth International Conference on Learning Represen- tations, 2024. URL bttpp:://pppt http///ppnneeeeeeeeeeeeee net/ forum? HdddHiddHHHH.........
002. URL UtLLhtcppss//ww.cccccoddeeeee.e. ccckeee/srriadee/l////2227133300
Mei, S., Misiakiewicz, T., and Montanari, A. Mcan-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on learning the- ory, pp. 2388-228880 2464. PMLR, 2019.
Cohen, J., Kaur, S., Li, Y., Kolter, Z., and Talwalkar, A. Gradient descent on neural networks typically oc- curs the edge of stability. In International Confer- on Learning Representations, 2021. URL https: //oppnreview.neeeeeeeeeeeeeeeeee www.nneet/oooiiiieo -id jh jh-jihh-ree rItvkGeM.
Nesterov, Y. Introductory Lectures on Convex Optimiza- tion: A Basic Course. Springer Publishing Company, Incorporated, 1 edition, 2014. ISBN 1461346916.
Dey, N., Anthony, Q., and Hestness, J. The practitioner's guide to the maximal update parameterization, 2024.
Øksendal, B. and Øksendal, B. Stochastic differential equa- tions. Springer, 2003.
Egele, R., Mohr, F., Viering. T., and Balaprakash, P. The unreasonable effectiveness of early discarding after one epoch in neural network hyperparameter optimization. Neurocomputing, 597:127964, 2024. ISSN 0925-2312. doi: https//lo..ayy/l...ooojnneooompp com/
Soto, T. Wide stable neural networks: Sample reg- ularity, functional convergence and bayesian inverse problems, 2024. URL 2407.03903
URL sci sccieeee//rrttccl//i///922222222
Tissue, H., Wang, V., and Wang. L. Scaling law with learning rate annealing, 2024. URL https:://aaseee htg/abs//444 11029.
Goodfellow, L. Bengio, Y., and Courville. A. Deep
Learning. MIT Press, 2016. http:// //ww///wwww///./.
deeplearningbook.org
Vyas, N., Atanasov, A., Bordclon, B., Morwani, D., Sainathan, S., and Pehlevan, C. Feature-learning net- works are consistent across widths at realistic scales. In Thirty-sevnnth Conference Neural Information Pro- cessing Systems, 2023. URL https: opwnrevie....eeeee net/ / f
Graham, C., Kurtz, T. G., Méléard, S., Protter, P. E., Pul- virenti, M., Talay, D., Kurtz, T. G., and Protter, P. E. Weak convergence of stochastic integrals and differential equations. Springer, 1996.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Xie, X., Yan, S., Toh, K.-C,, and Wei, Optimization hyper-parameter laws for large language models, 2024. URL https: //tp:///aivvvi/ieeeeea/eeeeeaaea www....woo//b.///0/0.0000 04777 04777.
10

---

On the Provable Separation of Scales in Maximal Update Paramcterization
Yang, G. and Hu, E. J. Tensor programs iv: Feature learning in wiith neural networks. In Meila, M. and Zhang. T. (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11777-117777 PMIR, 18-24 Jul 2021. URL press/vil vv19// htwp:/aaagnnnccc... html.
Yang, G., Hu, E., Babuschkin, I. Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, Tuning large neural networks via zero-shot hyperparameter transfer. Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 17084-17097. Curran Associates, Inc., 2021. URL https::////// /pto://////.reeeeggg....oo neurips. cc co/paper cc/pppper_flle/pppppeppeeeeeeeee 8 Paper.. pdf.
Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI: Feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Rep- resentations, 2024a. URL https: h openreview. net/ www.mrirumiiiee..e.... 1ttpiidd=ppppppp..m.
Yang, H., Kailkhura, B., Wang. Z. and Liang, Y. Training dynamics of transformers to recognize word co-occurrence via gradient flow analysis. In The eiihty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024b. URL https:// openreview.o net/ forum? d= sisSRR
Yang, H., Wang. Z., Lee, J. D., and Liang. Transformers provably learn two-mixture of linear classification via gradient flow. In The Thirteenth International Conference Learning Representations, 2025. URL https: / wwtteviiew.eetttooumiidaaaaaa

---

On the Provable Separation of Scales in Maximal Update Parameterization
A. Additional Proofs in Section 4
A.1. Proof of Lemma 4.3
Proof. Under j.P, preactivations a, remain (1) during training (bounded for all widths). The gradient for hidden weights W has the form:
VL(W) a.d. + +O1/Vn).
Under jP.. 17 ((1nn). This ensures | (())
Indeed, ,PP dictates that the learning rate and gradient magnitudes are co-scaled to ensure stable updates: gradients scale as ((1) for output layer weights (due to downstream fan-in), learning rate scale as O(1 ((/n) for output weights; thus, their product (weight update) remains (1/n), matching the expected per-step update scale. This is consistent with pP's core principle that all layers receive comparably scaled updates across widths.
A.2. Proof Lemma 4.4
Proof. By L-smoothness (Assumption 4.1), for any unit vectors u, v, |u" (2L(0,) S L. Therefore, each entry of V(.( (1). Under μul parameterization, d(n)) = (nn) for hidden layers.
A.3. Proof of Proposition 4.6
Proof. We combine three ingredients:
(i) Width regularity (Assump. 4.3): for some C > 0
[i n2
[L((:n) - L(6(0); 1 C
n1
(ii) Finite-horizon drift (Lemma 4.5): [M: L(0o:n)) (L/nn) for every
(iii) Optimality at width n2: :22) L(0; m22)
Insert L(Dos.) as common anchor and apply the triangle inequality:
(m:m)) ((, < (m; L(fo:
+ mL) L(0o;
+ [[(o; n22) M(r
The first and third terms are (1/nn, by Lemma 4.5 (fiite-horizon drift).
On the second term, apply width regularity (Assump. 4.3) with 0 Og to obtain (B) O(1mni) Symmetrically one could write the bound with 1/ng; choosing min(n1, 122) gives the tightest rate.
Combining the three bounds yields [M(::n)) M(; n2) = O(1/ min((n, n2)), which proves Proposition 4.6.
A.4. Proof of Lemma 4.9
Proof. By strong convexity of M (m: n) in (from Assumption 4.1, L-smoothness):
In (mi) 17" (n2) < (1/a)] - (,,M())))
where a is the strong convexity parameter. Meanwhile by the uniform convergence from Corollary 4.7, e.g., the fact that M (m: n) converges uniformly at rate ((1n)) the gradient difference is bounded by:
((;mm - Vyg s 011 min(n1, n2)).
12

---

On the Provable Separation of Scales in Maximal Update Paramcterization
Here we consider ((n n)as the effective scalar function describing the expected loss after training steps T under macro-level parameter 7/ For wide enough networks, macro-level fluctuations vanish, and local expansions around vf" (n) ensures a-strongly convex local basin in 77. This is akin to standard "strict local minimum" conditions in practical large-scale NNs near optimal learning rate. Importantly, this does not imply nor require global convexity of the full model's loss surface. Rather, we assume I.-L.isschtz gradients in the standard sense of gradient- methods, typically used nonconvex analyses to control the norm of the gradients and Hessians locally.
B. Simulations
We provide proof of concett run confirming the qualitative macro-mick microo scale separation predicted by Theorem 4.
Setup. Two-layer MLP, input dimension 3072 (flattened CIFAR-10) 10), hidden width n = 10 000 (ReLU), output width 10. Weights are initialized with μpP scaling, trained for 300 epochs using vanilla SGD (batch 128, no momentum/decay) and cross-entropy loss. Learning-rate grid r E (0.01, 0.02,... ,0.30.. We log the training loss L(t; n) and (ii) the squared weight drift []x[(; n) ||0, 0o every 10 epochs.
Fast stabilization of macro descriptors. We record max m1) L(t m2).. By epoch 10 the spread is already below 5x 10-4 (0.2 % of the mean loss), and remains almost flat afterwards. This corroborates the (1/nn) bound for macro variables (loss landscape) even though is fixed. A practitioner could therefore identify the best LR bucket (low 0.01-0.1, medium 0.11-0.15, high 0.16-0.30) within the first 60 epochs.
Slow evolution of micro variables. We then observe E || n) for three representative learning rates. Despite the macro loss plateauing early, [[)] keeps climbing roughly linearly until = epoch 180 (high 17) or 250 (low 1)), and its standard deviation across grows from 3.8x 10-2 at epoch 10 1.1 at epoch 120. The contrast illustrates the O(1/ (n) slow scale of microscopic drift.
Take-away. Even on a single width, the experiment shows loss differences become negligible long before training converges, and (ii) weight movement continues and separates trajectories. Both behaviours exactly what the scale- separation theory predicts; wider networks multiple would allow quantitative verification of the 1/n vs. 1/v rates, which we leave for future work.
13


Reviews:
[{"review_id": "vV3Gan9X7U", "reviewer": "Reviewer_LkSD", "summary": "The paper provides a theoretical framework to understand Maximal Update Parameterization ($\\mu$P). It introduces a decomposition of variables into macro-level descriptors (e.g., gradient norms, loss landscapes) and micro-level variables (e.g., individual weights). Via the formulation, the analysis shows why hyper-parameter tuning at early training stages can work (under certain assumptions). The paper discussed the feasibility of the proposed framework in two application cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Width-Dominance\n\nOur analysis explicitly relies on the width-dominance regime (Assumption 3.2)​ ensuring as width grows, certain terms dominate the learning dynamics. We will revise to explicitly mention it in the abs/intro. Note that it is a standard condition formalizing the intuition that we operate in the large-width limit where 𝜇P theory is intended to apply. Prior 𝜇P literature also shows partial validity beyond purely “infinite-width” conditions. Besides, our additional experiments in reply to Reviewer g228 show that scale separation behaviors indeed happen in training MLPs.\n\n2. Justifying Assumption 3.4\n\nWide-network studies provide strong empirical and theoretical support for A3.4. Prior 𝜇P work (e.g., Yang et al. 2021; Lingle 2024) shows that layer-wise statistics, e.g., activation variances and gradient norms, stabilize early in training and remain nearly invariant across widths. This is consistent with infinite-width theories where macro-level descriptors concentrate, as common in NTK and mean-field analyses. Additionally, recent large-scale experiments (e.g., Vyas et al., 2023; Cerebras “Practitioner’s Guide to 𝜇P”) also confirm that optimal HPs stay stable across model scales, indirectly verifying the training dynamics exhibit self-similarity.\n \n3. Convex Settings & Strong Convexity of 𝑀(𝜂;𝑛)?\n\nWe clarify that our main results do not require global convexity of the loss. Rather, we assume 𝐿-Lipschitz gradients in the standard sense of gradient-based methods (A4.2), typically used in nonconvex analyses to control the norm of the gradients and Hessians locally. Nowhere do we require the global objective to be strictly convex.\n\nIn Lemma 4.8, we consider 𝑀(𝜂;𝑛) as the effective scalar function describing the expected loss after training steps 𝑇 under a macro-level parameter 𝜂. For wide enough networks, macro-level fluctuations vanish, and local expansions around 𝜂*(𝑛) ensures a strongly convex local basin in 𝜂. This is akin to standard “strict local minimum” conditions in practical large-scale NNs near an optimal LR. We will clarify in the revision that this does not imply the full model’s loss surface is globally convex.\n \n4. Novelty and Paper Structure\n\nOur main innovation is the macro–micro separation framework under 𝜇P. Specifically, we derive a continuous-time SDE approximation (Sec. 4) that rigorously separates the convergence of macro-level variables at O(1/𝑛) from the slower micro-level O(1/√𝑛) behavior. This interplay had not been previously formalized; we will state more explicitly.\n\nWe will add a clear roadmap of the paper by end of Sec. 2:\n\nSec. 3 formulates the assumptions and “early-stage coarse-graining” argument.\n\nSec. 4 establishes the main SDE-based theorems (macro/micro separation).\n\nSecs 5 and 6 apply our unified theoretical framework to explain two empirical phenomenon, on HP and parameter early stability respectively:\n\nSec. 5 treats the ‘learning rate delay’ using macro-level integrated LR descriptors to explain the lag law;\n\nSec. 6 extends to the early stabilization of “critical parameter subset”.\n \n5. Comparison to Xie et al. 2024\n\nOurs and Xie et al. (2024) both leverage SDE to study training dynamics, but from different perspectives. Xie et al predict training loss and optimal LR schedules, deriving convergence rates and escape probabilities under time‑inhomogeneous SDEs, providing an empirical scaling law for HPs. In contrast, we rigorously develop a width scaling theory within μP. We prove a separation of scales between macro-level dynamics (which govern optimal HP transfer) and micro-level fluctuations, thereby explaining why HPs become width‑invariant as the network grows. In short, while Xie et al. offer complementary SDE-based insights on HP sensitivity in non-convex regimes, our work uniquely addresses the underlying mechanism of HP transfer via width-dominance—a question left open in their work.\n \nTechnically, Xie et al. is formulated for general non-convex optimization at the cost of ignoring width-specific dynamics​. Our analysis is based on conditions that allow us to rigorously prove separations, typically invoking local convexity or smoothness in the effective (meta-)objective governing HP selection. Empirically, the HP transfer phenomena appears robust even in non-convex NNs. Nonetheless, to extend our rigorous analysis fully to the general non-convex case, additional work would be needed to overcome challenges from multiple local minima and non-unique optimal HPs. We will clarify this comparison/limitation.\n \n6. Miscellaneous\n\nCiting arXivs: We will update our reference list to cite arXiv's accepted versions whenever available. Our result does not depend on Lingle (2024); the citation was meant to give context of validating and extending 𝜇P in practice.\n\nGenerality of Theorem 3.1: You are right. These assumptions are more about how the model is parameterized and how global descriptors converge rather than about specific architectures.", "claims_and_evidence": "The claims in the introduction are well-supported and justified. After reviewing the assumptions in Sections 3 and 4, it becomes clear that the results depend on several key assumptions. In particular, the theoretical analysis focuses on \"width dominance\" neural networks. It would be beneficial to highlight that the analysis pertains to such neural network properties in the abstract or technical summaries. This would better inform readers of the underlying assumptions from the outset. More comments on theoretical parts will be left in the later sections.", "methods_and_evaluation_criteria": "This is a theoretical paper. I will leave all the comments in the Theoretical Claims section.", "theoretical_claims": "1. It seems Assumption 3.4 is an important one to reach the coarse-graining transformation claim. However, there lacks empirical evidence or additional theoretical justification for such an assumption.\n\n2. The results seem limited to the convex setting.", "experimental_designs_or_analyses": "This is a theoretical paper. No experiment was involved.", "relation_to_broader_scientific_literature": "Providing understanding to reduce hyperparameter tuning cost, which can lead to less energy consumption for AI.", "essential_references_not_discussed": "The related work section seemed well converging necessary related work to support readers to understand the topic.\n\nSeveral references that the authors used to support their statement are unpublished work, such as \"This is observed empirically when early training reveals stable signals about learning rate, momentum, etc. (Lingle, 2024).\" Given those references can be dynamically updated on arxiv, it requires efforts to identify the version", "other_strengths_and_weaknesses": "This work seems interesting. The technical novelty in the proof should be further hilighted.", "other_comments_or_suggestions": "The paper structure can be further improved. For example, the connection from Section 3 and Section 4 is not very clear. Also, why the two specific applications in Sec 5 and 6 are used might not be clearly indicated in the paper, or I might have missed. Providing a roadmap for the paper structure could be helpful.", "questions_for_authors": "1. It is interesting to see the adoption of the SDE framework. The authors claimed Xie et al. 2024's analysis \"is complementary to ours but does not address the width scaling theory or the underlying mechanisms of hyperparameter transfer.\" Could you please elaborate more on the differences? \n\n2. I've noticed the referred work Xie et al. 2024 can be applied to non-convex optimization. Can this submitted work be generalized to non-convex optimization?\n\n3. Could you please explain the strong convexity for $M(\\eta; n)$ in your proof for Lemma 4.8?\n\n4. Please justify the Assumption 3.4 across different architectures or training scenarios.\n\n5. Does Theorem 3.1 hold for different architectures or training skems once they satisfy Assumption 3.2 - 3.4?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJKVkN4p4s", "reviewer": "Reviewer_iXhf", "summary": "The authors explained why hyperparameter tuning can be done effectively at early stages of training or narrower networks under the Maximal Update Parameterization scheme. They defined the seperation of scales for μP between macro-variable (loss, activation variance, gradient norms etc) and micro-variable (weight values), and proved that macro-variables converge faster than micro-variables. The authors further applied the seperation framework to explain two empirical observations: learning rate scaling laws and delay phenomena, and the \"early bird ticket\" phenomena (where small subnetworks can be trained to achieve same accuracy as original network).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and thoughtful comments. Below we address each point raised:\n\n(1)  Lemma 4.7 and the asymptotic notation: The reviewer correctly identified that if L is treated as a constant independent of n, this would typically yield O(n^(3/2)). However, in our analysis, the Lipschitz constant L in our context decays with the network width, specifically L=O(1/√n). As d(n)=O(n^2), \\sqrt{d(n)}=O(n). Therefore, L√d(n)⋅O(√n) = O(n).\n\n(2) Applicability of μP to phenomena in Sections 5 and 6: in addition to providing theoretical justifications as laid out in the submission, we have conducted small experiments on PreResNet-101 + CIFAR-10 (aligned with original You et. al. 2020  setting), and confirm that using μP, we can still replicate Table 2 accuracy results at p = 0.3, 0.5, 0.7. Due to 1-week limit, we were unable to fully replicate the LLM experiments in Tissue et al., 2024) using μP, but we will continue afterwards and updates results to our paper too once we finish.\n\n(3) Intuition on Magnitude-based vs Hessian-based pruning: While the original early-bird work focuses on iterative magnitude pruning (IMP) in the context of the lottery ticket hypothesis, subsequent research has introduced non-IMP “early ticket” methods—such as ProsPr, EarlyBird, and EarlyCroP—that incorporate additional training signals to enhance mask quality and speed up early convergence. However, to the best of our knowledge, no prior work has specifically explored Hessian-based pruning in the early-bird setting (they do exist in another related setting of “pruning at initialization” or PaI). Moreover, our paper’s analysis relies on modeling the network’s training dynamics with a gradient-flow SDE (i.e., approximating discrete SGD updates by a continuous-time stochastic differential equation), and thus does not readily apply to Hessian-based approaches. Intuitively, Hessian-based methods typically exhibit slower initial convergence but can achieve better long-term performance by using curvature information to prune weights that minimally affect the loss landscape, even if those weights do not have the smallest magnitudes. We will leave the more comprehensive study as future work.\n\nTwo other quick fixes:\n\n(1) We have fixed the assumption and theorem numbering in each section.\n\n(2) We revised the notation in Theorem 3.1 to η*(t₀), so that in general η*(t) represents the optimal hyperparameter value at time t, while η*(∞) represents the global optimum.\n\nWe are grateful to the reviewer for their valuable feedback, which has helped us improve the clarity of our manuscript.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have checked the correctness of proofs in Section 4. In the proof of Lemma 4.7, since $d(n)=O(n^2)$, why is $ L \\sqrt{d(n)} \\cdot O(\\sqrt{n}) = O(n)$ instead of $O(n^{\\frac{3}{2}})$?", "experimental_designs_or_analyses": "For section 5 and 6, the original work for the phenomena discussed were not based on Maximal Update Parameterization. How does the authors make sure that the phenomena are still valid under $\\mu$P assumptions?", "relation_to_broader_scientific_literature": "The paper established a novel framework to understand and explain empirical findings regarding hyperparameter tuning and model pruning. It provides insights on the dynamics of model training optimization and guide future researches on more efficient training.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n1. The paper is well written, with clear definitions of math notations and assumptions.\n2. Is it good that the authors provide interpretations or implications throughout the paper to help the readers understand the theorems and assumptions.\n3. It's interesting that the macro-micro separation framework can be used to explain two phenomenas that seems to be distinct, which widens the impact of this theoretical framework.", "other_comments_or_suggestions": "1. Section number and Assumptions/Theorem sharing the same numbering system is slightly confusing. Maybe consider alternative numbering schemes for the Sections.", "questions_for_authors": "1. I'm somewhat confused by the notation of $\\eta^*$, in Theorem 3.1, $\\eta^*$ refers to the hyperparameter optima found during early-stage training, while on line 2 of page 4, it is defined as the hyperparameter optima across the entire training process.\n2. Are the authors able to provide an intuition on whether Magnitude-based pruning or Hessian-based pruning will converge faster?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JYUsFRPbeg", "reviewer": "Reviewer_RWaf", "summary": "The authors try to propose a theoretical framework for hyperparameter transfer in neural networks under maximal update parameterization (µP) by trying to demonstrate a separation of scales between macro-variables (such as loss landscapes, activation norms, and gradient statistics) that converge at an $O(1/n)$ rate; and micro-variables (like individual weight updates) that evolve more slowly at $O(1/\\sqrt{n})$. Their bounds are qualitatively consistent with the empirical observation that early-stage hyperparameter tuning effectively approximates globally optimal settings, thereby justifying zero-shot transfer from smaller proxy models to larger networks. By view SGD as stochastic gradient flow, the authors derive **upper** bounds on hyperparameter transfer error and unify disparate deep learning phenomena, including learning rate delay effects and the early emergence of lottery tickets, under some very high-level and abstract assumptions.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Def. of “Separation”:\nThanks for suggesting the formal use of “separation” in learning theory. Here, we use “separation of scales” to describe the difference in convergence rates between macro- and micro-variables under μP. We show that loss landscape descriptors converge at rate O(1/n), while weights evolve at Θ(1/√n). This holds in practice: coarse statistics stabilize early even as parameters continue evolving. We agree that we shall clarify our use of Θ(1/√n) instead of O(1/√n). What we establish is a quantitative gap in convergence rates with meaningful implications for early-stage hyperparameter (HP) transfer.\n2. Initialization in Assumption 3.3 and Lemma 4.6:\nWe thank the reviewer for catching a typo in A3.3 (due to accidental copy-paste). The input weights were misstated as N(0,1/n^2); they should be N(0,1). Importantly, this typo does not affect our derivation despite the text misstating it. L4.6 already assumes that input weights are N(0,1), ensuring pre-activations stay O(1). All weight initializations after L4.6 are correct as they concern hidden-layer weight N(0,1/n). Thus, the typo does not make our result incorrect.\nWe made typos in A3.3 re the scaling of learning rate (LR). In subsequent derivations, though we wrote that η=O(1/n) for output-layer parameters under μP, we clarify that this refers to the LR per parameter, not the total per-layer update. In μP, LR and gradient norms are co-scaled so that each layer’s update remains O(1), preserving stability as width grows. For example, the gradient on output weights scales as O(1), and with η=O(1/n), the resulting update is also O(1/n), matching the intended scale.\n3. Lemma 4.6 and Gradient Norms:\nOur argument that O(1/n) LR results in O(1) weight updates can indeed be expanded. Under μP, the LR and gradient magnitudes are co-scaled to ensure stable updates: gradients scale as O(1) for output layer weights (due to downstream fan-in), LR scale as O(1/n) for output weights; thus, their product (weight update) remains O(1/n), matching the expected per-step update scale. This is consistent with μP’s core principle that all layers receive comparably scaled updates across widths. We will clarify this interaction between LR and gradient norm.\n4. Assumption 4.5: Width Regularity:\nThe assumption is standard to ensure the convergence of macro-variables under increasing width. Similar assumptions are used in the NTK/SDE literature and μP papers: e.g., Yang (2021) assumes that activations are smooth (or smoothed ReLU), and many proofs invoke pseudo-Lipschitz or bounded-derivative conditions to enable LLN-type arguments. A4.5 ensures that descriptors like average activation variance or loss stabilize as width increases—empirically supported and analytically tractable. This assumption holds for standard activations (e.g., ReLU, GELU) either directly or via minor smoothing. It does not weaken the generality or correctness of our results.\n5. Memory Kernel in Theorem 5.1:\nq(τ) is not an ad hoc invention: the term arises when modeling the effect of time-varying LR momentum in a continuous limit. A prominent example is the Generalized Langevin Equation, which includes a memory kernel to capture how past states influence the current update. The recent DMFT for SGD yields integral equations with memory kernels too. Here, it serves as a continuous-time analog of LR memory effects studied in training delay. In Tissue et al. (2024), the delayed response of validation loss to changing LRs is modeled by cumulative functions like S1(t) and S2(t). Our notation makes explicit the role of memory weight q(τ), enabling clear continuous-time modeling of past LRs. We will add citations and clarify the role.\n6. Gradient Norm O(1) in Section 5.2:\nThe claim that gradient norms remain O(1) with time-varying LR η(t) relies on Assumption 5.2, not just μP scaling. Under μP, at any time t, the per-layer LR should still scale with width to ensure bounded updates. Our use of time-varying η(t) assumes that such width scaling is preserved pointwise in t, as is common in practical μP. The theoretical foundation for this substitution comes from NTK/μP literature, where smooth η(t) can be inserted into gradient flow ODEs without breaking convergence. We will make it explicit.\n7. Tensor Programs Faithfulness:\nWhile we build on the μP works, our approach uses SDE to distill the key phenomena of macro vs micro scales. We do not replicate the exact Gaussian Process construction as our focus is on connecting HP transfer and early‐stage alignment under μP. But our theoretical lens is compatible with the same scaling laws, and we cite those results to justify certain variance and regularity assumptions. Nowhere do we claim to reproduce the entire measure-theoretic formalism from the ground up. Our vantage point is narrower—yet consistent—focusing on the rates at which global vs local descriptors converge.\n\nMinor Issues: We will correct the typo at the end of Section 4.2 and fix the ICML template.", "claims_and_evidence": "The authors' definition of \"separation\" **disagrees with the common sense** in theoretical computer science, in particular, machine learning theory. See Section \"Theoretical Claims\" for details.", "methods_and_evaluation_criteria": "Not applicable for this theoretical paper.", "theoretical_claims": "- **Fundamentally misleading claim**: $O(1/n)$ versus $O(1/\\sqrt{n})$ is not called \"separation\" in the context of learning theory. If it is something like $O(1/n)$ versus $\\Omega(1/\\sqrt{n})$, then it can be called \"separation\".\n- Many basic facts about $\\mu$P in this paper are wrong. For example, in Assumption 3.3, the element-wise initialization of the input weight $\\mathcal{N}(0, 1)$ in the original paper of $\\mu$P, instead of the $1/n^2$ written in this paper. This is a very obvious mistake.\n- Moreover, the learning rate of $\\mu$P for the output layer is indeed $O(1/n)$, but Lemma 4.6 actually is a statement about the gradient norm alone, not invoving learning rates; so the logic in the proof of Lemma 4.6 is very vague by saying a $O(1/n)$ learning rate ensures a $O(1)$ gradient norm.\n- Assumption 4.5 is too coarse to be considered as a reasonable one. It implicitly encompasses many aspects of the neural network, including the regularity of the activation function. On the other hand, it is not very clear whether the original set of MLPs considered in Greg Yang's $\\mu$P paper, i.e., those neural networks whose activation functions have pesudo-Lipschitz derivatives, satisfies Assumption 4.5; but the authors just directly uses the results in the $\\mu$P paper without further justification on this point.\n- In Theorem 5.1 and it's proof, there is no formal definition of $q(\\tau)$, the so-called \"memory kernel\", which seems to be a very artificial correction term the authors came up with in order to have the term in the conclusion to match the empirical \"systemetic delay\".\n- In Section 5.2, the authors claim the gradient norm to be $O(1)$. But actually the results in the original $\\mu$P paper is obtained under constant learning rate, while the learning rate here is allowed to be time-varying, as it did in Theorem 5.1\n\n#### Minor comment:\n- Misleading typo \"$O(1/n)$\" in the last sentence of Section 4.2.", "experimental_designs_or_analyses": "Not applicable for this theoretical paper.", "relation_to_broader_scientific_literature": "This paper is under the name of $\\mu$P but does not faithfully rely on the infinite-width neural network Gaussian process results developed in, e.g., the papers listed on Greg Yang's website: https://thegregyang.com/#tensorprograms.", "essential_references_not_discussed": "No fatal oversight.", "other_strengths_and_weaknesses": "Another minor comment is:\n**This submission does not use the correct template, so there is no line number shown on the left hand side of each page!**", "other_comments_or_suggestions": "See Section \"Theoretical Claims\" for details.", "questions_for_authors": "See Section \"Theoretical Claims\" for details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8CND5crbfe", "reviewer": "Reviewer_g228", "summary": "This paper provides a theoretical framework for analyzing the separation of “macro” and “micro” scales in large-width neural networks trained under the Maximal Update Parameterization (μP). Building on techniques from stochastic differential equations (SDEs) and tools in wide-network theory, the authors prove that global (“macro-level”) quantities (e.g., learning rates, gradient norms, loss landscapes) converge on the order of O(1/n), while individual (“micro-level”) weights converge more slowly at O(1/\\sqrt{n}). The paper then leverages this scale-separation perspective to explain important phenomena including:\n1.\tEarly-stage hyperparameter tuning can be done on smaller proxy models (or for fewer training steps) with minimal cost to final performance.\n2.\tChanges in learning rate produces delayed reactions in validation loss\n3.\tEmergence of early-bird lottery tickets in large-width networks, showing how “redundant” parameters can be pruned without harming macro-level behavior.\nThe overall argument is compelling and addresses a significant gap in our theoretical understanding of hyperparameter transfer across different model widths. The results are timely given current interest in scaling large models efficiently. Despite a purely theoretical presentation, the paper’s insights have broad potential impact.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Comparison with Earlier Wide-Network Theory Frameworks:\n\nWe appreciate the suggestion to clarify how our work provides a novel perspective complementing earlier wide‐network theories like the NTK and mean-field SDE frameworks. In our paper, we emphasize an explicit separation of scales that distinguishes between global (macro) and local (micro) dynamics in neural network training—a perspective that is not directly addressed by NTK or conventional mean-field approaches.\n\nNTK theory primarily examines the behavior of networks in the infinite-width limit, effectively linearizing the training dynamics. While this yields valuable insights into the global macroscopic behavior of the network function via a constant kernel, it largely overlooks the finite-width nuances and the evolution of individual weight updates. In contrast, mean-field SDE methods focus on modeling the microscopic evolution of weights, capturing the stochastic fluctuations of individual parameters. However, these methods do not inherently highlight how aggregated statistics, like loss landscapes or activation norms, stabilize rapidly. \n\nOur approach under µP rigorously demonstrates that macro-variables converge at an O(1/n) rate, while the micro-variables evolve more slowly at an O(1/√n) rate, building upon the SDE method used in the Xie et al. (2024), as discussed in Section 2.3 as a recent work. Thus, our work unifies these perspectives: it extends NTK analysis by accounting for finite-width effects and complements mean-field approaches by clearly delineating the roles of global and local dynamics. This multi-scale perspective not only deepens our theoretical understanding in addition to the two frameworks but also has practical implications for the effectiveness of early-stage HP tuning: even when individual weights have not yet converged, the stable, global signals are sufficiently reliable to guide HP selection. .\n\n2. Experimental results:\n\nAs proof of concept, we take a two-layer MLP where the input layer is 3072 units (flattened 3x32x32 CIFAR-10 images), the hidden layer is n=10000 units with ReLU activation, and the output layer is 10 units (for class probabilities), the SGD optimizer with LR 0.01 - 0.30 with 0.01 increments, CE loss, batch size 128, and n=300 training epochs with a sampling interval of checkpoint 10. Our experiment confirms the macro-micro scale separation we observe theoretically. In particular:\n\ni. Fast convergence of macro-variables: The relative loss differences between learning rates are very small in early epochs (at epoch 10 already, as an example, between LRs 0.07 and 0.08, the loss difference is 0.0005), indicating that the loss landscape has a consistent structure across different learning rates, as predicted by our O(1/n) convergence rate for macro-variables. We observe effective early hyperparameter tuning: in low LR (0.01-0.1), medium LR (0.1-0.15), and high LR (0.15-0.3) settings, the respective top learning rates emerge early (at epochs 60, 60, 100), indicating the loss landscape stabilizes quickly.\n\nii. Slow convergence of individual weights: Despite early identification of optimal LRs, the loss continues to decrease and finetune throughout all 300 epochs for LR < 0.1 and until 180 epochs (LR = 0.3) to 250 epochs (LR = 0.15) for larger LRs. We also see a dramatic increase in the loss gap in later epochs (the ratio between the worst and the best losses, across lrs, is 1.10 at epoch 10 and 238.33 at epoch 120), indicating that small initial differences are being amplified. This phenomenon is consistent with the idea that while the macro structure stabilizes quickly, the micro-scale adjustments (individual weight evolutions) continue refining the network, progressively enhancing the performance gap.\n\n3. Proof Completeness: We promise to include expanded proofs with explicit epsilon-delta arguments and tighter bounding constants to enhance rigor in the appendix.\n\n4. Limitations Under SP and NTK: μP provides the cleanest setting for demonstrating scale separation due to two properties: (1) consistent gradient/activation scaling preserving feature learning as width increases, and (2) self-similar behavior of global statistics that stabilize quickly. \n\nFor standard parameterization (SP), gradients shrink with width, leading to \"lazy\" training without clear timescale decoupling. Similarly, NTK parameterization preserves a limit but keeps features near initialization, yielding kernel machine behavior without pronounced scale separation. We acknowledge this limitation and view it as an exciting direction for future research to determine whether analogous separation phenomena can be rigorously established in alternative parameterization regimes. For example, one could consider the α-scaled framework (α=1 for μP, α=1/2 for NTK), and hypothesize that values closer to 1 likely maintain scale separation; however, a comprehensive characterization requires further investigation. We will add the discussion.", "claims_and_evidence": "1.\tMacro-level convergence at O(1/n): The authors rely on a continuous-time SDE approximation of SGD under μP, showing that key global descriptors (like average activation variance, gradient norms, etc.) stabilize quickly at large width.\n2.\tMicro-level weights converge at O(1/\\sqrt{n}): They derive a separate SDE for the individual parameters and show that high-dimensional fluctuations slow the final convergence of each weight.\n3.\tEarly-stage hyperparameter selection: Because macro-level signals stabilize rapidly, the hyperparameters chosen by training just the first few epochs (or a smaller proxy network) align closely with those for the full-run training.\n4.\tEarly-bird lottery tickets: Arguing via stable importance metrics for each parameter, the paper shows that “pruning” low-importance parameters early on yields minimal loss penalty.\n\nThe paper’s theoretical arguments are solidly motivated. However, the proofs occasionally mix high-level intuition with technical lemmas, and some readers might prefer more rigorous detail in certain sections (e.g., explicit epsilon–delta arguments, tighter bounding constants).", "methods_and_evaluation_criteria": "The paper is strictly theoretical. It uses a blend of:\n•\tStochastic Differential Equation analysis: Approximating the discrete SGD updates in the large-width limit.\n•\tConcentration of measure: Relying on typical wide-network bounding techniques to show that macro descriptors converge quickly.\n•\tFunctional approximation arguments: Highlighting how “macro-level” features become decoupled from the slower “micro-level” fluctuations at large nnn.\n     No experimental or empirical validation is provided. While this is understandable for a theory-oriented work, small-scale or synthetic experiments (e.g., training wide MLPs on a toy dataset) could provide a “proof of concept” that the claimed scale-separation emerges in practice.", "theoretical_claims": "The main theoretical claim is a novel form of scale separation under μP, with key results summarized in Theorem 4.1 and subsequent corollaries. These show that:\n•\tThe macro-scale loss landscape converges at rate O(1/n)\n•\tIndividual weights converge more slowly at O(1/\\sqrt{n})\n•\tHyperparameters adjusted during early training steps transfer effectively to the full training regime.\nThese claims are plausible and quite interesting, but some parts of the derivations rely on assumptions (e.g., “wide dominance,” “macro-level self-similarity,” “stable infinite-width limit under μP”) that might need additional justification or numeric demonstration.", "experimental_designs_or_analyses": "No experiments were conducted. For a paper with nontrivial assumptions and wide-network scaling arguments, a small set of synthetic experiments—e.g., wide fully connected networks on a simple classification task—could help illustrate how quickly macro-level variables stabilize in practice. Such experiments would strengthen the paper’s persuasiveness and might inspire real-world usage.", "relation_to_broader_scientific_literature": "This work fits neatly into the line of research on large-width neural networks and hyperparameter transfer (notably the μP literature). It also links to recent results on “early-bird” ticket phenomena (You et al., 2020) and a wealth of existing wide-network theory (e.g., neural tangent kernels, mean-field approaches). The authors do a good job situating their main contributions in that context and explaining how their scale-separation perspective might unify multiple observed phenomena in large-scale deep learning.", "essential_references_not_discussed": "It would be beneficial if they explicitly compared or contrasted their approach to earlier wide-network theory frameworks such as the Neural Tangent Kernel line of work and the Mean-field (SDE) approaches. Although the references are tangential, a brief discussion could further emphasize how the authors’ approach differs in focusing on the macroscale vs. microscale variables.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "1.\tEmpirical Demo: Could the authors provide a small toy experiment (e.g., a simple classification or regression with wide MLPs) to illustrate the macro–micro convergence rates in practice?\n2.\tProof Completeness: Do the authors plan to include more detailed, fully rigorous proofs (e.g., bounding constants, more explicit expansions) in the supplementary to address the theoretical nature of the paper?\n3.\tGenerality of μP Argument: The paper focuses on μP. How far might these scale-separation results extend to other parameterization schemes (e.g., standard parameterization, NTK parameterization), or do the authors see potential for broader application? A discussion on its limitations would be helpful too.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors explained why hyperparameter tuning can be done effectively at early stages of training or narrower networks under the Maximal Update Parameterization scheme. They defined the seperation of scales for μP between macro-variable (loss, activation variance, gradient norms etc) and micro-variable (weight values), and proved that macro-variables converge faster than micro-variables. The authors further applied the seperation framework to explain two empirical observations: learning rate scaling laws and delay phenomena, and the "early bird ticket" phenomena (where small subnetworks can be trained to achieve same accuracy as original network).

This paper has many strengths. For example, the paper is well written, with clear definitions of math notations and assumptions. It is good that the authors provide interpretations or implications throughout the paper to help the readers understand the theorems and assumptions. It's interesting that the macro-micro separation framework can be used to explain two phenomenas that seems to be distinct, which widens the impact of this theoretical framework. The claims in the introduction are well-supported and justified. The main theoretical claim is a novel form of scale separation under μP, with key results summarized in Theorem 4.1 and subsequent corollaries.

While the reviewers had some concerns about theoretical claims, the authors did a particularly good job in their rebuttal. Therefore, most of us have agreed to accept this paper for publication! Please include the additional discussion in the next version.
Decision: Accept (poster)