Understanding the Mechanisms of Fast Hyperparameter Transfer
Authors: Nikhil Ghosh
Denny Wu
Alberto Bietti
Venue: ICLR 2026 Poster (2026)
Keywords: hyperparameter transfer
hyperparameter tuning
scaling laws
optimization dynamics
maximal update parameterization
science of deep learning
Primary area: foundation or frontier models, including LLMs

Abstract: The growing scale of deep learning models has rendered exhaustive hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware HPs, which can enable direct transfer of optimal settings from small-scale grid searches to large models with minimal performance loss. Such approaches are useful when the optimal settings converge "fast" enough with scale. While approaches like the Maximal Update Parameterization ($\mu$P) have empirically displayed fast transfer when scaling model width, a deeper conceptual understanding of the mechanisms that enable this is still missing. Our work establishes a systematic conceptual framework for analyzing fast HP transfer across different synthetic and practical scenarios. In synthetic settings, we present various quantitative examples where transfer either offers a provable computational advantage or fails even under $\mu$P. 
  We then propose a key property that enables the fast transfer often observed in practice: through a novel decomposition of the optimization trajectory, we identify one component that rapidly converges with model width and determines the optimal HPs, and the other that continues to improve the loss with increased width but has negligible impact on HP choice. We conjecture that this decomposition elucidates the key mechanisms behind fast transfer and empirically validate it in practical settings such as LLM training.

Published as a conference paper at ICLR 2026
UNDERSTANDING THE MECHANISMS OF FAST HYPER- PARAMETER TRANSFER
Nikhil Ghosh'
Denny Wu'
Alberto Bietti'
Flatiron Institute NNew York University
(nghosh, dwu, wwiittti/eraattionnnssiititttt
ABSTRACT
The growing scale of deep learning models has rendered exhaustive hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale- aware HPs, which can enable direct transfer of optimal settings from scall-scale grid searches to large models with minimal performance loss. Such approaches are useful when the optimal settings converge "fast" enough with scale, so that transfer is asymptotically more efficient than direct tuning. While approaches like the Maximal Update Parameterization (μPP) have empirically exhibited fast transfer when scaling model width, a deeper conceptual understanding of the mechanisms enabling this behavior is still missing. Our work develops a conceptual framework for analyzing this phenomenon across different synthetic and practical scenarios. In synthetic settings, we present various examples where transfer either offers a provable computational advantage or fails even under μ.P. We then propose key property that enables fast transfer in practice: by decomposing the optimization trajectory, the final loss splits into (i) width-stable component that determines the optimal HPs and (ii) a width-sensitive component that continues to reduce loss but minimally affects HP choice. We conjecture that this condition explains fast transfer and empirically validate it holds in large language model pretraining,
INTRODUCTION
Scaling-aware hyperparameters. A central dogma in empirical deep learning is that performance will steadily improve [KMH- 20; HBM+22] as training data and parameter count increase. This paradigm incentivizes practitioners train increasingly larger models while first conducting experi- ments at smaller, more economical scales, For such small-scale experimentation to effectively inform larger training runs, becomes crucial reason about hyperparameters (HPs) in a scaling-aware framework and to analyze the behavior of the sequence of progressively scaled-up training runs. For example, when scaling the width n of a neural network, we should conceptualize the learning rate as the product of a scale-independent HP and a scaling factor n= This perspective was initially formalized in the Tensor Program series [YH21; YHB* THB+22: 22; YL23] for the scallis of neural networks. It was shown theoretically that the correct scaling for ensuring "optimal" training in the infinite- wiitth limit the Maximal Update Parameterization [YH21, μPP). This parameterization specifies per-layer initialization variance and learning rate scalings that tivations and their updates are (11), ensuring that the optimal 17 is asymptotically indepeendeenn
oc, the network ac-
Hyperparameter transfer. Empirical evidence [YHB* 22; Lin24] has demonstrated that in fact, μP enables a much stronger property which we refer to as fast hyperparameter transfer (see Section 3) that is not apparent from its theoretical derivation. At high level, fast HP transfer occurs when optimal HPs converge significantly faster with respect to width than the performance measure of interest. This phenomenon allows practitioners to perform HP selection smaller proxy models and subsequently apply these optimal values to larger-scale training runs, drastically reducing the cost of HP tuning. Despite its practical utility, this phenomenon remains primarily an empirical success, with limited theoretical understanding of its underlying mechanisms.
Our contributions. We provide insight into the puzzle of fast HP transfer by developing frame- work for reasoning about HP transfer in Section 2. Then in Section we define fast HP transfer in terms of convergence rates of optimal HPs and the loss, and connect it to the "usefulness" of transfer when performing compute-optimat grid search. We quantify these convergence rates in synthetic set- tings, and demonstrate that while valid in certain linear model, fast transfer is not guaranteed in neural networks even when using jPP. Such examples illustrate that the benefits of transfer heavily depend on structural properties of the training process emerging from data optimizer, and architecture.

---

Published as a conference paper at ICLR 2026
To illuminate this structure, in Section 4.1 we introduce novel loss decomposition obtained by decomposing the step-wise linearized loss change along the training trajectory. We empirically demonstrate that the linearization is an effective proxy for the true loss change when considering the exponential moving average (EMA) of the optimization trajectory. The faithfulness of this approximation is due to the smoothness of the resulting EMA trajectory (see Figure 7). Using the linearization, we track the loss change arising from the projection of the update in the top-k directions that maximize loss change at each matrix layer. This decomposes the total loss change into two components: the top-k loss arising from the dominant & components and the remaining residual loss.
We conjecture that the training behavior on the top-k subspaces concentrates quickly with width while the remaining directions provide perfor- mance gains as the width increases. Figure we validate this intuition by applying the loss decomposition introduced quence of Llama-style transformers of increasing width trained under μP. We observe that the lead- ing top-k loss remains approximately invariant across widths, whereas the residual loss consis- tently improves with width, especially later in training. Moreover, since the top-k loss provides the majority of the loss decrease, it is reasonable expect that the optimal learning rate for the total loss will not deviate much from the optimal learning rate for the top-k loss. Therefore, for an appropriately chosen &o < n, the top-k loss decomposition provides the following potential explanation for the phenomenon of fast transfer.
[ 0. 004 k- 60
-1.
Top Alane din-126
ALoos
Top-t slace dim 2044
Section 4 to se-
Beedial gloce drn- 121
Racdial drocns 2015
.5.0
1000
3000
4220
1000
1000
Step
Figure I: Loss decomposition into top-k & and residual
components in transformers with varying widths.
Hypothesis 1: Fast Transfer via Loss Decomposition
1. The top-k loss converges rapidly with scale n, hence so do the optimal HPs for top-k loss.
2. The optimal HPs for top-k loss approximately determine the optimal HPs for the total loss.
In the rest of this paper, we provide a formalization for this intuitive explanation and use it to provide conceptual insights into the fast transfer phenomenon with support from experiments across synthetic and realistic settings, including large language model (LLM) training.
1.1 RELATED WORK
Hyperparameter transfer. The concept of hyperparameter transfer was introduced by [YHB+22) who showed that HPs tuned on small proxy models can transfer reliably to much larger ones under μP scaling. Subsequent empirical studies confirmed the effectiveness of learning rate transfer in transformers, while also highlighting certain limitations [Lin24; VBF25J. As [YHB+22] noted, the success of hyperparameter transfer is not explained from first principles. Addressing this gap empirically, [NMH024] showed that top Hessian eigenvalues converge rapidly under μP across widths, suggesting width-stable curvature that could underpin transfer. However, the connection between these spectral statisties and the optimal learning rate remain unelear. More recently, the concurrent work of [HW25] established scale separation between certain macro- and micro- variables, thereby suggesting HPs can be effectively tuned in early training stages. contrast to these results, our work explicitly defines (via a trajectory-level loss-decomposttion and connects the fast convergence of certain statistics of the optimization path to fast convergence of optimal HPs.
Optimization trajectories. Our fast transfer hypothesis rests on the existence of low-dimensional structure in optimization trajectories. [GARDI8] observed that gradients rapidly align with top eigenvectors of the Hessian, suggesting that GD operates within a "tiny subspace." " [SAY24] refindd this view, showing that while gradient variance concentrates in the top eigenspace, motion in these directions primarily drive oscillations rather than loss reduction. This behavior consistent with edge-of-stability (EoS) [CKL-21; DNL22] and motivates our use of EMA smoothing: by averaging out oscillatory components tied to the top eigenspace, the smoothed trajectory reveals the low- dimensional subspace where genuine loss decrease occurs. Similar low-dimensional structure also appeared in theoretical settings on gradient- basee feature learning, such as the learning of multi--ndex models (AAM22; D2, 3BPV23; MHPG+23, 23; GWB25), where SGD "localizes" parameters into
2

---

Published as a conference paper at ICLR 2026
low-dimensional subspaces. Recent works have also shown that gradient updates induce spiked dignaalplus-noiss noise) eigenstructure in the conjugate kernel [BES 22; MLHD23; WWF24; DPC 244 or the Hessian [BAGHJ23; MW25] of the neural network, and the top cigenvectors contain information of the target function. Drawing inspiration from these theoretical settings, we explicitly isolate the width-invariant structure in realistic settings using spectral decomposition and link it to HP transfer.
2 PRELIMINARIES AND FORMAL FRAMEWORK
Let denote the scaling dimension. The scaled HPs used during training at scale are:
(v, = (vin ....., ,....,
,,.. "*,,
where we refer to = (MI,..., vh) as the HPs and to 7 = (71....., 77.) as the scaling exponents. Conceptually, v are a set of n-independent constants which are tuned for a specific problem and y set of exponents which specify how to scale the HPs with so that training be performed at any scale n using Hn 7). In this paper, we focus on the "optimization hyperparameters" of the abed- poeeddaaaattteizaioonn of [YL.23] (see Appendix B). This setting encompasses the width scaling of hyperparameters needed for training standard neural network architectures using common optimizers such as SGD and Adam. Thus we will often simply refer the scale n as the width; however our framework can be used more broadly for reasoning about scale-aware HPs.
In our setting the training procedure A is held fixed and we only vary n, the HPs v, and the scaling 7- The training procedure returns a stochastic output A(n, v, y). For neural network training this means that the architecture, optimizer, dataset, etc. are all fixed, and we will let A(n, V, 7) be the resulting optimization trajectory. For given configuration, from the output of A (c.g.. g. the final validation loss). We tune HPs over search space X which we take to be fi-dimensional box. We define the optimal HPs and the corresponding optimal value as
can measure scalar metric on (u;7) obtained
v² (m; 7) arg min (n) (7) == (v= (m:))
vEX
We omit the exponents y when context is clear, and abbreviate the above as @n(+), v² (n), and o%
Weak Transfer. A basic requirement for a parameterization y is that both the optimal HPs v (n) and the local sensitivity of on around >> (n) are (asymptotically) independent of the scale n. This guarantees that grid search over fixed, scale-independent grid of HPs achieves near-optimal perfor- mance for all n. To ensure that relevant asymptotic quantities converge to well-defined limits, we impose regularity assumptions on } so that well-defined limit ox exists and the optimal HPs (n) converge to the minimizer v² (oo) of &o To quantify the transfer suboptimality, we further assume that do is locally strongly convex. This condition links the suboptimality of transferred HP to loss suboptimality and ensures that performance meaningfully degrades away from the optimum. Without such condition, 6xx can be flat near its minimizer, making accurate HP selection irrelevant.
We will say that the parameterization 7 admits weak transfer (or simply that "HP transfer holds") when the above conditions satisfied. The rigorous formulation is given in Definition 3 (A (Appendix B). We regard this as the "weakest" notion of HP transfer, since it concerns only the asymptotic convergence of quantities rather than their convergence rates. Prior work [YHB+ 22; YYZH23] has argued informally that only "optimal" parameterizations can admit weak HP transfer and that μP is the unique optimal parameterization for general tasks. We formalize the first claim in Theorem 9 and assume that weak transfer holds under jPP.
33
width n=128
width n- n-2048
Width n-16384
Vollidation Lavs
Optimal
13
30'
Learning Rate n
Figure 2: Learning Gaussian k-index model using two-layer ReLU network with μP. The HP of interest is the Adam learning rate- Appendix E for details. The optimal HP ex- hibits an abrupt shift at n = 8192.
Note that weak transfer alone does not imply that trans- fer under jPP is actually useful, as it does not quantify the convergence rates and computational gain over di- rect tuning. In particular, even though the optimal HP converges to well-defined limit, if such convergence happens very slowly, then one cannot reliably infer D² (co) from small-scale proxies - see Figure for numerical example, where the optimal HP up to width 8192 becomes clearly suboptimal for larger-wide width models. In the next section, we will show that stronger conditions on the convergence rates required for fast and useful HP transfer.
3

---

Published as a conference paper at ICLR 2026
Performenc
Hyperparamderr
Ryperparamterr
(8) Loss gap an-
(b) Hyperparameter gap bn-
(c) Transfer suboptimality gap Ca-
Figure 3: Illustration of convergent quantities in Definition I; note that captures the efficiency of HP transfer.
THE PUZZLE OF FAST & USEFUL TRANSFER
In this section, we relate the convergence rate of optimal HPs (c.g., learning rate) to that of the evaluation metric (e.g., validation loss). As previously diseussed, the definition of weak transfer ensures that v² (n) + v (oo), but we need to characterize the rate of convergence in order to conclude the computational efficiency of the transfer strategy. As we will see in Proposition 1, the convergence rate of Φ - &so already implies an upper bound on the convergence rate of the minimizers v² (n) due the local strong convexity condition of é Moreover, turns that this convergence also informs whether it is computationally more efficient to use HP transfer than to directly tune the model at single width n. as we show Theorem 2. We then present toy examples where the convergence rates of the optimal HP and loss can be quantified.
3.1 CONVERGENCE RATES OF HP TRANSFER
Notation. Given deterministic sequences In RR and yo E R++ we write x,, = Oyn) to mean such that [x,,] < суn for all large nı; o(-), O()) are defined analogously. We write In ~ M/ to mean In = (za)). ). We overload the same notation for random quantities: whenever In random, 0(-), o()). ((-), and are understood in the corresponding stochastic sense (i.e., Op. Op- Op), and limits 2n + x are interpreted as convergence in probability unless stated otherwise.
there exists
Tracking the Width Dependence. We introduce three quantities that track the convergence of the optimal loss and HPs, illustrated in Figure 3.
Definition (Convergent Quantities). We define the following width width-dependent quantities.
Loss gap: on -o 0-ol. The quantity measures the discrepancy between the optimal value of the metric & at the finite-width model (over all HPs) and that of the infinite-width model.
Hyperparameter gap: bn || (n) v² (oo) |[. The quantity measures the discrepancy between the optimal HPs at width n and the infinite-wide width counterpart.
Transfer suboptimality gap: Ca = 1 (v" (n)) - &'cl- The quantity measures the performance gap between two infinite-widthh models: one with the optimal HP tuned at infinite-width, and the other with the optimal HP at a smaller width.
Fast Transfer. In the following. in addition assuming HP transfer holds (Def. 3 in Appendix C.I) we will make the mild technical assumption that the "uniform loss gap is locally tight" which implies that a, ~ || & @ ll (see Def. 4 and Lemma 12 in Appendix C.2). The following proposition uses strong convexity around the optimal infinite-widd width HP to directly connect the convergence rates of the optimal HP and the evaluation metric (see Appendix C.2).
Proposition 1. 6n (aa and en O(b)) O(an).
Since HP transfer holds, we have by definition an bns + 0 as n + CC. This being said, priori we may have 6, (a) and hence C,2 O(a,). This would indicate that the transfer suboptimality gap converges at the same rate as the loss gap. In practice however, we tend to observe a much faster rate for G, relative to an. We refer to this phenomenon formally as fast transfer.
Definition 2. We say that HP transfer is fast if en = o(a,) which occurs iff bn = o(a (1712))
For instance, if an = and bn ~8, then fast transfer occurs if and only 3 a/2.
Useful Transfer. The notion fast transfer also coincides with the computational usefulness of HP transfer when performing grid search. Consider HP tuning strategies: direct performs grid
4

---

Published as a conference paper at ICLR 2026
search on models of single width, and transfer performs grid search models of smaller width and then trains a final model of larger width using the obtained optimal HPs. Suppose we have a compute budget of F flops, and the amount of flops needed for single training run scales with n' for models of width n, where r = 2 for standard optimization algorithms on standard architectures. We say that transfer is useful if the transfer strategy yields better performance than the direct strategy for a given compute budget. The following theorem characterizes when this occurs.
Theorem 2 (Useful transfer). Suppose an n and bn ~ n -8 We are given compute budget of F flops, we tune h different HPs, and a single training run at width n. costs n² flops.
(a) Direct tuning. If we divectly conduct grid search on a width-n model, the compute-optinal performance scales as F and is obtained at width n² =Fnat.
(b) Transfer. If we grid search over width-n models, then transfer to width-M model, the compute- optimal loss scales as F + F Fuatt, obtained at widths n' ~ Fiiie and M F1/r.
-28
Hence transfer is useful if and only is >> a/2, ie., bn = olyan).
Observe that the requirement 3 > a/2 is the same condition fast transfer (Definition 2). Hence under local strong convexity of dxc with respect to v. the naive loss convergence rate already implies that transfer never underperforms the direct tuning strategy asymptotically, provided that the HPs are parameterized to be reinndependent While this supports the effectiveness of μ-transfer [YHB+ does not address the question of when the optimal HPss converge faster than what is implied naively by the loss convergence. The question turns out to be subtle, and the following subsection presents simple examples where fast transfer may be present or absent.
3.2 EXAMPLES OF FAST AND SLOW HP TRANSFER
The precise scaling of quantities in Definition 1 is infeasible to measure in large-scale scenarios: since the infinite-scabe model (n 20) is inaccessible, we must resort power-law fits from finite-n data, which requires fine grid search of HPs at each scale. Therefore, in this section we present synthetic settings where the convergence rate of (an, bn,cr) Ca) can be either analytically derived or reliably estimated from data. These settings allow us to quantify the computational gain of HP transfer.
Fast HP Transfer: Random Features Regression. First we consider tuning the ridge penalty À in a high-dimenniioal random features (RF) model with nonlinearity a, where the target function is a single-index model with link function d,, on isotropic Gaussian input in IRd We aim to select the optimal regularization parameter À R that minimizes the prediction risk R(f) = Ex - fl
In the proportional limit where the number of training data N, dimension of input features d. and model width n all diverge N, d n - 0o, NNdd + 11, n/d + 1/2 where 4/1, 4/2 E (0, oc). [MM22; GLK 20] derived precise asymptotics of prediction risk under standard assumptions (see Assumption 2). In this model, the number of trainable parameters controlled by the ratio 1/2 = n n/d, and the infinite-widd model is obtained by sending 1h2 - co. The following proposition quantifies the convergence of prediction risk and hyperparameter as a function of 1/2.
Theorem 3. Define (x)2) == arg min XER Reg (A), where Rey (A) is the asymptotic prediction risk of the RF model with width n/d- = u/2 and ridge penalty X. Then under Assumption 2, we have
Loss gap: (A*)()) - Roc (20 = O,
Hyperparameter gap: (X - =000)
Suboptimality gap: |R (X* (w22) Rox (X* (2o)) = -w
This theorem states that both the loss gap on and the HP gap bn scale as d/n v251, whereas the transfer suboptimality gap scales as On 15, < an. We conclude that the ridge regularization parameter exhibits fast transfer per Definition 2. Consequently, Theorem 2 implies that tuning the ridge penalty on a small (narrower) RF model and then transfer is more compute-efficien than directly tuning the large-width model. To our knowledge, this gives the first concrete setting where the HP transfer strategy in [YYZH23] provably offers a computational advantage.
Figure 4a presents the prediction risk of the RF ridge estimator across varying widths n/d e/2. We set σ = tanh, o_ = ReLU, 2 = 4, Gr = 1/4. The analytical curves are obtained by solving the coupled Stieltjes transforms (9) and (10). Figure 4b confirms the scaling of an buy Cn in Theorem 3.
5

---

Published as conference paper at ICLR 2026
nid =
10"2
n/d 256
411
n/d
10-11
1055 gMp 4
Prediction risk
107
10P
10
4x10-
5.9 10
gep by
10"11
101
37
subost,
0.2
0.6
08
1.0
10"
Ridge penalty
Width o2
(a) Prediction risk ridge penalty.
Scaling of optimal loss and ridge penalty.
Figure 4: Optimal ridge penalty (generalization error) for RF regression to learn Longle-indes model.
Slow HP Transfer:
Two-layer ReLU Network. Next consider classification setting with
shallow ReLU neural network: f(x) Ei=1 (wW, x) + 6;). where we aim to tune the learning rate n that minimizes the validation loss. We the target be the norm indicator function, which a well-studied function that requires wide two-layer network to approximate [SL22],
y (1120]) F. (0.5)), where (00 Ka),
where F (0.5) represents the median of chi-square distribution with degrees of freedom. This threshold ensures that the classes are exactly balanced. We set = 26, = 224, and run the Adam optimizer [KB14] 14] for T = 2 steps with batch size 2 to minimize the binary cross-entropy loss. The initialization and learning rate are set according to μP [YH21].
In Figure 5a we observe that under JPP while the optimal learning rate admits a well-defined limit, there still visible drift towards the right. Morcover, Figure 5b illustrates that under power-law fit, the optimal 77 converges slower than the validation loss, and the estimated scaling bn ~ aa, suggests that the HP convergence rate does not beat the agnostic rate from strong convexity. An interesting future direction is to analytically derive the scaling exponents in an bn, On. for jPP examples and quantify the efficiency of learning rate transfer (beyond the definition of weak transfer in Section 2).
26
of 6.37 0.108
Validation Locs
2x33
Optimal
74
width n=16384
Optimal HP
(eguto() ) -7.80 6.78
Learning π
Width o
(a) Validation loss vs. learning rate.
(b) Convergence of optimal learning rate.
Figure 5: Optimal learning rate (validation loss) for two-layer ReLU network learn the ball indicator function.
FAST TRANSFER VIA TRAJECTORY DECOMPOSITION
Section 3 demonstrates that without further assumptions, the existence of scale-independent limit of optimal HP (i.e., weak transfer) does entail that transfer more computationally efficient than direct tuning. At high-level, for useful transfer to occur in practical neural network optimization settings, the optimal HPs should depend on some statistics the training trajectory that converge faster than statistics tied to the performance metric of interest. In this section, we explicitly extract the cast-conn statistics from the trajectory and connect them to the optimal HPs. To do so, will leverage prior intuition
the din-dimensionality of optimization trajectories (see Section 1.1): intuitively speaking, we may expect the movement in a small number of sensitive directions to significantly contribute to the loss decrease and approximately determine the optimal HP. If the

---

Published as a conference paper at ICLR 2026
associated now-dimensional statistics converge sufficiently fast, then the optimal HP also becomes stable across scale. For instance, it is plausible that the loss depends on all eigenvalues of the Hessian, whereas the learning rate is decided by the top components alone (which may converge faster with scale). In the ensuing subsection, we introduce a novel layer-wise spectral decomposition to identify the low-dimensional structure that informs the HP selection.
4.1 ToP-k k Loss DECOMPOSITION
Let us consider an optimization trajectory w (Wo, ........ WT) of a neural network. Define the one-step loss change (L(w;) == [(wr+1) - [(we). so that the overall loss change is the sum
AC(W) == ((WT) ((wo)
Σ ÖC(ww). t=
Let g: V ((w)) and ow;; W1+11 w Let define do(w;) == (ge, ows)) be the linearization of (w,) Under appropriate smoothness conditions on the trajectory,
T-1 b)(w;)= = 5(())) and ((w) = Σ d0(w)) ~ AC(W).
(I)
To ensure that such smoothness conditions hold and ((w) is a useful proxy for AL(w), we take w to be an exponential moving average (EMA) of the base optimization trajectory to remove oscillatory behavior. On realistic problems, we obtain excellent agreement between ((w) and A for EMA trajectories which perform least as well as the corresponding base trajectories (see Fig. 7).
The utility of considering the linearization ((w) in our setting is that we can decompose fd(w) based on the structure of (gt, Sw:) and recover resulting decomposition of @(w). Let us consider fixed time index I. If u (W(1),.. ..... W(L) are the model parameters with corresponding gradients g = (G)1,, ..... G(L)) such that ( and (( are tensors of the same shape, then
(g, dun)
Σ (G (G0,)w())) FE[T]
Consider single summand coming from W E mR with corresponding gradient G E Rmxx, To isolate the dominant directions of descent, we analyze the alignment between the gradient and the update via the alignment matrix which is the symmetrized product:
S(G, &G) = (GT -: + OW G).
(2)
( SW) has eigenvalues A1,..., A such that |\ı| 2 ... 2 |aul. we define the top-k stepwise linear loss change dΦ (W) to be the sum of the first & eigenvalues.
86(W) Σ A and d0 (W)
Σ A
Eq. (3) captures an intuitive notion of loss change in the top-k directions of maximum change in loss. If the update o' is aligned with the gradient, i.e., G x 8W, then Sø*(W) is the sum of the top-k singular values of G. We can add the stepwise changes over time and parameters' to compute,
tt (w)) = Σ
60 ( (W!
EEIL]
For some parameters we use "row version" of 6Φ* which uses S(G (W instead of S(G, 8W). Then the index ls can span [n] for all layers so we may employ the same for each layer in Eq. (4).
Decompesition-aware HP Transfer. We can now make our earlier intuitions Hypothesis more precise using this stepwise decomposition. Assume we fix a training procedure A (see Definition 3). Then we can define the width-n trajectory trained with HPs v and scaling 7 to be the trajectory wn, (H,7) obtained from executing the training procedure A with hyperparameters H,, (v, 7). If we consider a fixed scaling y and an HP- deeeendent truncation function k which outputs an index x(v) e [n] given HPs v, we can abbreviate
On (v) = (w, ()), (5.()) = (wn()) 7))
4 (n) arg min (v), (n) = arg min (N(2) (v).
(5)
We will only decompose matrix parameters and use the full inner-product for vector parameters.
7

---

Published as a conference paper at ICLR 2026
Table I: Notation for the linearized trajectory decomposition.
Symbol
Definition
Ref.
o0(w)) o(w)
Linearized loss change at step t. Total sum of 86(w,, over trajectory.
(1)
(1)
86 (WW!
Sum of top-k components in layer at time I.
(3)
a² (ww
Sum of top-k components over layers and time.
(n(z)
Loss curve over HPs v for width Truncation function maps HPs index in [n].
k(v)
0%(()
HP adaptive top-x loss curve (% (v).
We will refer to dh as the top-k loss curve and 6, == dh. dh as the residual loss curve. For convenience, we summarize the notation for our trajectory decomposition in Table
Based on this decomposition, we have the following conceptual explanation of fast transfer: if for an appropriately chosen sequence the following are simultaneously true for large enough n,
Top-x strong convexity: The top-K,, losses of and 6h are locally strongly-convex.
Top-s invariance: The top-ii loss converges rapidly so 6% Ri 6- and V2 (n) vA (co).
Residual Flatness: Residuals on "00" are "flat"; hence Vi (n) R V" (n). V (oo) 2 v (oo).
it follows that v² (n) (0o). i..., the optimal HP remains stable across widths n (see Appendix C.5 for details). This is precisely the mechanism posited in Hypothesis 1: the top-K,, loss optimizer stabilizes quickly as width grows and the residual loss is too flat to significantly shift the optimum of the full loss. Due to the prohibitive computational cost accurately estimate the scaling of the above quantities in realistic settings, in this work we present qualitative evidence that the above conditions hold empirically (hence the notation ">="). We leave
quantitative investigation to future work.
Selecting the Truncation Function. Selecting k involves a trade-off: we must retain enough components to ensure the top-ı minimizer approximates the true optimal hyperparameters (requiring tail components (requiring smaller k). In Appendix C.5, we formalize this by defining a quantity (k) which upper bounds the HP gap b,, using quantitative measures of top-k invariance and residual flatness. By choosing truncation K,, that minimizes Jn, (K). balance these properties obtain the tightest upper bound on d,. Since directly optimizing Tn is intractable, we instead optimize proxy objective Jproxcy (r:) to obtain a truncation k(n) via Algorithm 1 (Appendix D). We validate empirically that k(n) achieves the desired qualitative properties.
larger k)., while excluding width-sensitee
4.2 EXPERIMENTAL RESULTS
We train Llama-style transformer [TLI+23] using the Adam optimizer [KB14] with warmup- stable-decay (WSD) learning rate schedule [HTH+24] on WikiText-103 (MXBS16) using μP, and sweep the peak learning rate as shown Figure 6; detailed setup can be found in Appendix E.
In Appendix A. repeat the same setup but with Muon and observe that transfer is less stable and top-k invariance is weaker. In Appendix A.2 we use CIFAR-10 training to interpret what the top and tail components are capturing by introducing sample-wise version the top-k decomposition; we use this refined decomposition to connect the quality of transfer to the "hardness" of the samples.
Fast Transfer and Linearization Faithfulness. As we can see from Figures 6 and 7, the EMA loss and the linearized loss are nearly indistinguishable, indicating that the EMA trajectory is sufficiently smooth. From Figure 7 we see that smoothing does not degrade the final loss. Our setting also clearly exhibits fast transfer since the optimal learning rate is converging rapidly with the width (see Fig. 6a) while the reducible loss improves more slowly, converging at a rate of n -0.32 (see Fig. 6b). Using the optimal learning rate obtained at width 128 for larger widths is essentially optimal, as indicated by the overlapping curves in Figure 6b. We now further probe the optimization and scaling dynamics in this fast transfer setting through the lens of our decomposition.
Dim 128, LR 0. 000
Lisearized
7.5
1300
Stop
Figure 7: Transformer training on WikiTexi- 103. Linearized and EMA losses are identical through training and match the final iterate loss.
8

---

Published as a conference paper at ICLR 2026
SMA
328
L 18:16-- 1-----8 (800) V
4.50
256
3,230 02 #0, (olmo (22000)
532
4.25
4.0-
Linerrtze
128
3.8
250
3.75
512
1021
3.6
n= 2013
3.50
3.4
Learning Rate
Width n
(a) EMA loss (solid) and linearized loss (dashed).
(b) Scaling law for the EMA loss.
Figure 6:
Training a 4-layer Llama transformer with Adam optimizer on WikiText- 103. Left: EMA and linearized losses nearly coincide, indicating small lincarization error. Right: EMA loss for each width for two HP choices: the width-dependent optimal learning rate v (n) [blue dots] and the width-128 optimal learning rate v" (128) [orange triangles]. Overlapping curves indicate perfect transfer across widths.
Decomposition Over Time. Figure displays the top-k loss with k 60 and residual loss over training, across multiple widths, at a fixed learning rate. We see that throughout training, the top-k loss nearly width-invariant and accounts for the majority of loss reduction. This indicates that the bulk of the improvement due to optimization comes from low-dimensional subspace, and the benefit of width mostly comes from improving the residual loss. Accordingly, the "width-dependent" learning largely occurs in the tail components (and their contribution increases later in training).
Decomposition Across Widths. To evaluate our fast transfer conjecture, we apply our loss de- composition across widths and compute K., A(n) using Algorithm 1. We use the largest width 7max = 2048 as an infinite-width proxy, and consider transfer from finite widths n < Mmaa to this proxy. In the right panel of Figure 8, we see that the top-k,, loss curves nearly overlap across learning rates, i.c., 6% & 6x despite the large gap in the total losses on and doe shown in the left panel. Furthermore, the minimizers of the total loss are largely determined by the minimizers of the respective top-k,, loss, in the sense that both v² (n) = v v"(n) and v (co) = v² (oc) (see Eq. (5)). In Figure 9 we plot the corresponding residuals and see that they are flatter than the top-K,, losses in neighborhood of the corresponding top-x,, minimizer. As a result, the residuals contribute less to the determination of the overall optimal learning rate.
Total
Top-K
n-
n1512
512
n- 1024
1024
66.5
-7.0
128
256
512
7.5
0= 1024
Learning Rate
Learning Rate
Figure 8: Left: Total loss curves dn across widths. Right: Top-k,, loss curve pairs din (blue dashed) and øšc (purple dashed). The top-ka pairs nearly overlap, with minimizers close to those of corresponding total losses.
Residual
Residual
$
128
128
256
-0.5
512
ALoss
n- 1024
o- 1024
1.0
-1.5
2-15
2-N
2m
2
Learning Rate
Learning Rate
Figure 9: Residual losses are flat around the top-ka minimizers, indicating less sensitivity to the learning rate.

---

Published as a conference paper at ICLR 2026
kin)
LR 0.002
500
-9
0-512
400
2003
200
-6
100
400
1000
Leariing Rale
Figure 10: Left: Computed values of R(n) using Algorithm 1. Right: Top-k losses & for LR = 0.002, which descend rapidly with & and overlap across widths over an intermediate range where top-k invariance holds.
Top-k Profile. To further interpret the decomposition and the computed values i(n), we now examine Figure 10. The left panel shows the computed values of R(n), which are approximately constant across learning rates and grow sublinearly with n. The right panel plots of for a fixed learning rate function of k. The top-k loss varies smoothly with k dropping rapidly at first and then flattening out. Notably, we can see that R(n) chosen roughly at the index & where the curve ot "peels off" from the 6% curve. This index marks the transition where increasing & starts to include width-sensitive directions and balances the tradeoff discussed at the end of Section 4.1. We see in the figure that this transition point increases with since the finite-width model shares more converged components with the infinite-width proxy. Consequently, R(n) increases with and the residual 6 F. shown in the right panel of Figure decreases in magnitude. This differs from Figure 1, where we fixed k 60 to isolate width-invariant index and the residual magnitude grew with n. Compared to the infinite-widhh residual, the finite-width residual @, á*a smaller since there are fewer tail components, hence the curves in the left panel of Figure 9 are closer to zero.
Overall, we can qualitatively see how our decomposition can account for fast transfer in the sense described in Section 4, even when the convergence the loss itself is much slower. The provides concrete evidence for our central hypothesis that there is a low-dimensional projection of the tra- jectory which remains nearly invariant across width and is responsible for deciding the learning rate. In Appendix E.4.1, we observe similar fast-transfer behavior for GPT-2 architecture trained on Win [PKL+22 224] using Adam.
CONCLUSION
This work introduces a novel conceptual framework to reason about hyperparameter transfer and its underlying mechanisms. We posit that a basic form of HP transfer, which we refer to weak transfer, can hold generically as a consequence of the asymptotics of loss and hyperparameter scaling. In synthetic settings, we demonstrate that this asymptotic condition alone does not imply the substantial computational benefit of transfer often observed in practice. We conjecture that fast & useful transfer instead requires non-trivial low-dimensional structure in the optimization dynamics. We make this idea concrete by introducing a decomposition of the dynamics based on a lincarization of an EMA-smoothed training trajectory. This decomposition provides operational way to describe the relevant oow-dimensional structure and motivates an empirically testable sufficient condition for useful transfer. Our experiments suggest that this structure appears in practice for common optimizers such as SGD and Adam, and that it may underlie the empirical success of hyperparameter transfer across scales. We hope this perspective motivates further work on identifying when efficient transfer should be expected and on developing deeper understanding of optimization dynamics across scale.
ACKNOWLEDGMENT
The authors thank Blake Bordelon, Lénaïc Chizat, Jeremy Cohen, Soufiane Hayou, Jason Lee, Yan Shuo Tan, Atlas Wang, and Greg Yang for discussion and feedback. The symbolic computation in Appendix C.4 was assisted by GPT5-Pro.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
[AAM22] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The -tared-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions two-layer neural networks. In Conference Learning Theory, pages 4782 4887. PMLR, 2022.
[AXA+25] Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv:2504.05295, 2025.
[BAGHJ23] Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, and Aukosh Jagannath. High-dimensional sgd aligns with emerging outlier eigenspaces. arXiv preprint arXiv:2310003010 2023.
[BBPV23] Alberto Bietti, Joan Bruna, and Loucas Pillaud- Vivien. On learning Gaussian multi- index models with gradient flow. arXiv preprint arXiv:2310. 19793, 2023.
[BES+22] Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptoties of feature learning: How one gradient step improves the representation. Advances in Neural Informaionn Processing Systems, 35:37932-37946, 2022.
[BMR21] Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30:87-201, 2021.
[Chi22] Lenaic Chizat. Sparse optimization on measures with over-parameterized gradient descent. Mathematical Programming. 194(1):487-532, 2022.
[CKL+21] Jeremy M Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations (ICLR), 2021.
[CN24] Lénaic Chizat and Praneeth Netrapalli. The feature speed formula: a flexible approach to scale hyper-parameters of deep neural networks. Advances in Neural Information Processing Systems, 37:62362-62383, 2024.
[DD25] Damek Davis and Dmitriy Drusvyatskiy, When do spectral gradient updates help in deep learning? arXiv preprint arXiv:2512.04299 2025.
[DLS22] Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference on Learning Theory, pages 5413-5452. PMLR, 2022.
[DNI.22] Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209. 15594, 2022.
[DPC+24] Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue M Lu, and Bruno Loureiro. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities. arXiv preprint 0rXiv:2410.189533 2024.
[DW18] Edgar Dobriban and Stefan Wager. High-dimennional asymptotics of prediction: Ridge regression and classification. The Annals of Statisties, 46(1):247-2779 2018.
[FAL255] Kevin Frans, Pieter Abbeel, and Sergey Levine. whitening optimizers? arXiv preprint 10. 2025.
What really matters in matrix-
[GARD18] Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018
[GLK+20] Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zde- borová. Generalisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning, pages 2020.
PMLR,
11

---

Published as a conference paper at ICLR 2026
[GSJW20] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(1 113011, 2020.
[GWB25] Margalit Glasgow Denny Wu, and Joan Bruna. Mean-field analysis of polynomial- width two-layer neural network beyond finite time horizon. In The Thirty Eighth Annual Conference on Learning Theory, pages 2461-2539. PMLR, 2025.
[HBM*22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optima large language models. arXiv preprint arXiv:2203. 2022.
[HL.22] Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, (113):1932-194 2022.
[HTH+24) Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. In Conference on Language Models (COLM)5, 2024.
[HW25] Letong Hong and Zhangyang Wang. On the provable separation of scales in maxi- mal update parameterization. In Forty-second International Conference on Machine Learning, 2025.
[IP Andrew lyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint SKYi:2200.006222. 2022.
[J] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.
[KB14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 4rXiv:1412.6988. 2014.
[KGG+22] Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning. arXiv preprint arXiv:2202.099331 2022.
[MH + 20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
[Lin24] Lucas Lingle. An empirical study of mup learning rate transfer. arXiv preprint arXiv:2404.05728, 2024.
[MHPG+23] Alireza Mousavi-Hosseinn Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A Erdogdu. Neural networks efficiently learn low-dimensional representations with SGD. In The Eleventh International Conference on Learning Representations, 2023.
[MLHD23] Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. arXiv preprint urXiv:2310.0788 2023.
[MM22] Song Mei and Andrea Montanari. The generalization error of random features regres- sion: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
[W25] Andrea Montanari and Zihao Wang. Phase transitions for feature learning in neural networks. In preparation, 2025.
[MXBS16] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (ttl:::6000088.. 2016.
12

---

Published as a conference paper at ICLR 2026
[NMH024] Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, and Antonio Orvieto. Super consistency of neural network landscapes and learning rate transfer. Advances in Neural Information Processing Systems, 37:102696-10277 2024.
[4] Guilherme Penedo, Hynek Kydlïček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data scale. Advances Neural Information Processing Systems, 37:30811-30844 2024.
[SAY24] Minhak Song, Kwangjun Ahn, and Chulhee Yun. Does sgd really happen in tiny subspaces? arXiv preprint arXiv:2405. 16002, 2024.
[SL.2] Itay Safran and Jason Lee. Optimization- based separations for neural networks. In Conference Learning Theory, pages 3-64. PMLR, 2022.
[TLI+23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[VBF25] Georgios Vlassis, David Belius, and Volodymyr Fomichov. A thorough reproduction and evaluation of mup. Transactions on Machine Learning Research, 2025.
[WWF24] Zhichao Wang, Denny Wu, and Zhou Fan. Nonlinear spiked covariance matrices and signal propagation in deep neural networks. In The Thirty Seventh Annual Conference on Learning Theory, pages 4891-4957 PMLR, 2024.
[WX20] Denny Wu and Ji Xu. On the optimal weighted & regularization in overparam eterized linear regression. Advances Neural Information Processing Systems, 33:10112-10123, 2020.
[YH21] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite- width neural networks. In International Conference on Machine Learning, pages 11727-11737. PMLR, 2021.
[YHB+22] Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint MrXiv:2203.03466 2022.
[YL23] Greg Yang and Etai Littwin. Tensor programs ivb: Adaptive optimization in the infinite-widn limit. arXiv preprint arXiv:2308.01884, 2023.
[YYZ223] Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning infinite-depth neural networks. arXiv preprint arXiv:2310.02244, 2023.
[ZGKS21] Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. Are larger pretrained language models uniformly better? comparing performance at the instance level. arXiv preprint arXiv:2105 06020, 2021.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
1.1
Related Work
2 Preliminaries and Formal Framework
3 The Puzzle of Fast & Useful Transfer
3.1 Convergence Rates of HP Transfer
3.2 Examples of Fast and Slow HP Transfer
Fast Transfer via Trajectory Decomposition
4.1 Top-k Loss Decomposition
4.2
Experimental Results
5
Conclusion
10
Additional Findings on Top-k Decomposition
15
A.1 Llama with Muon
15
A.2 MLP with SGD
16
Background
19
B.1 Scaling Limits and Tensor Programs
19
B.2 Muon Optimizer
20
B.3 Dion Optimizer
20
Theoretical Results
22
C.I Weak Transfer
22
C.2 Asymptotic Rates
23
C.3 Grid Search
24
C.4 Random Features Regression
25
C.5 Decomposition-aware Fast Transfer
28
D Truncation Index Selection
30
E
Experimental Details and Additional Experiments
32
E.
μP MLP Experiments
32
................................
E.2
Llama Experiments
33
................................
E.2.1
Llama Adam LR
33
E.2.2
Llama Adam 31 and 32
33
E.2.3
Llama Muon
36
E.3
Llama Dion
36
E,4
GPT-2
Experiments
38
E.4.1
GPT-2 Adam I.R
38
E.4.2
GPT-2 Muon LR
39
E.5
CIFAR-10 10 MLP SGD
41
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL FINDINGS ON TOP-k DECOMPOSITION
A.1 LLAMA WITH MUON
We repeat the same experiments in Section 4.2 but using the recently popularized Muon opti- mizer [JJB 24], which updates the model weights via the orthogonalized momentum gradients (see Appendix E.2.3 for details). Figure 1la shows that the optimal learning rate shift is more pronounced with Muon than with Adam (see Figure 6a). This larger shift leads to suboptimal performance at larger widths when using transferred hyperparameters, as demonstrated in Figure 1 1lb the scaling law using the transferred hyperparameter diverges noticeably from the one using the optimal hyperparameters. While this transfer suboptimality is offset by the improved overall performance of Muon, it is still an interesting example of "imperfect" transfer, and our decomposition reveals distinctive properties of Muon's learning dynamics.
Figure 12 reveals that top-k invariance holds only for small & (up to Ko R 10). This is in sharp contrast with the decomposition for Adam (Figure 10), in which we observe approximately invariant 6% (up h z 60) that explains large fraction of the loss reduction. For k > Ko. we clearly see that of increases with n, even though on decreases with n. This indicates that under our layer-wise decomposition, Muon is "spread- ing" the loss decrease over more directions: each di- rection contributes less, but the cumulative decrease over all directions is larger since on, decreases with n. Intuitively speaking, this lack of low-dimensional invariance is connected to the "whitening" step in Muon which increases the effective rank of the gra- dient [FAL25; DD25]. Consequently, the low-rank structure in Adam and SGD updates that enables fast transfer may not be present in full-matrix preconditioned updates.
[.R - 0.008
120
1024
Top-k losses di for Muon. The curves flatten out slowly with k, especially for large ni showing top-k invariance holds only for narrow range of k and the residual is significant.
Figure 12:
In Appendix E.4.2, we report the learning rate transfer of Muon in different problem setting: training GPT-2 on the Web dataset. There again we find that Muon displays a less stable optimal learning rate across widths, but due to the flatness of the overall loss, the transfer performance is still nearly optimal. Interestingly, in Figure 32 we observe that the decomposition looks qualitatively different for different learning rates. In particular. while the small lcarning rate decomposition resembles that in our earlier Muon experiment (Figure 12), at near-optimal learning rates we instead see more pronounced top-k invariance that is closer to the Adam decomposition (Figure 10). We speculate that this can happen when the dynamics of the layers trained with AdamW (e.g., the input and output layers, see Appendix B.2) tend to dominate at certain learning rates, but we leave careful verification of this to future work. In any case, this example highlights that the correct notion of invariance for Muon is subtle and can be hyperparameter dependent.
4.4
4.2
n=512
n-2
4.2
2-1133 18.03 11110 - (122)
1074
2010
4.0
3.8
Lincorited
3.8
3.5-
3.4
3.-
32
Leamiing Rate
Width
(a) EMA loss (solid) and lincarized loss (dashed).
(b) Scaling law for the EMA loss.
Figure 11: Same model and dataset as Fig. 6. but trained with Muon. Left: EMA and linearized losses coincide. Right: EMA loss versus width for the learning rate choices n² (ni) [blue dois] and v² (128) [orange triangles]. At large using n² (128) becomes suboptimal, indicating imperfect transfer.
15

---

Published as a conference paper at ICLR 2026
Experiment with Muon Variants. To investigate the effect of whitening on learning rate transfer and the decomposition in a more controlled manner, in Appendix B.3 we perform the same experiment using the Dion optimizer [AXA 225], which has a rank hyperparameter controlling the dimension of the whitened subspace. If we choose the Dion rank to be small, we expect a higher degree of invariance and less influence of the residual, which should lead to improved transfer. In Figure 24, we see that setting the rank equal to min(n//2 128) indeed greatly improves transfer compared to using rank of n/2, which closely resembles the Muon results (Figure 11). Perhaps surprisingly, although Dion with bounded rank does exhibit more stable transfer, the resulting top-k decomposition (Figure 26a) is more Adam-like but still looks qualitatively similar that for Muon. We speculate that there exists a different notion of invariance for optimizers like Muon and Dion, obtained by modifying our proposed decomposition, that leads more cleanly to a qualitative picture similar to what we see for SGD and Adam; for such suitable notion of invariance, we expect the corresponding residual loss will be flatter for Dion with bounded rank compared to Muon.
A.2 MLP WITH SGD
For setting with a new data modality and architecture, we now consider learning rate transfer in two-layer MLPs trained CIFAR-10 using momentum SGD. Additional details can be found in Appendix E.5. This setting will have the benefit of being lightweight enough to support a detailed analysis and interpretation of our decomposition from data-centric point of view (similar in spirit to [ZGKS21; KGG 22; IPE I) In Figure 13 we can see that our linearization described in Section 4.1 is accurate and the optimal HP is stable across width (though transfer is "imperfect" compared to Figure 6). We also observe that the benefit of width is less pronounced compared to the language setting and that loss convergence occurs at faster rate of approximately n n0777 This aligns with the intuition that CIFAR- 10 is a "simpler" task; in Appendix E.5 we further support this intuition through the top-k decomposition (Fig. 33).
1.40
EMA
129
1.42
2.57 1-.52 13.52-9-77 URii)
1.41
312
256
1:40
1024
2013
1.36
1.30
4092
Linearized
1.34
128
1.22
312
256
1.32
nw 1024
1.30
1.29
2-4
Learning Rate
2-
n= 2043
n- 1147
n= 4095
27
20
27
Width
21c
21a
2
(a) The final EMA loss (solid) and final lincarized.
(b) Scaling law for the EMA loss.
Figure 13: Two-layer MLP trained on CIFAR- 10 using SGD. We observe reliable but imperfect transfer. Left: The EMA and linearized losses coincide. Right: EMA loss across widths using the width-dependent optimal learning rate v² (n) [blue dots] and the fixed width-128 -28 choice s² (1128) [orange triangles].
Sample-wise Decomposition. We have previously seen that the top-k components account for the majority of loss decrease, while the tail components account for the improved learning at larger widths. However, this picture does not shed light on what structures the top vs. tail components are actually learning. To gain better qualitative understanding, we now apply our decomposition at a sample-wise level and examine how different components affect the loss on each example. We conjecture the following structure:
Easy Examples: Examples that are almost entirely learned by the top components are "easy" examples. Loss on these examples will show fast transfer and essentially decide the optimal HPs.
Hard Examples: Examples that rely on the tail components are "hard" examples. These examples are learned differently across widths because tail components are not width-invariant, hence slowing transfer.
For example, it could make sense to consider invariance for the rescaled top-k loss की := ( where m, (k) akkn for some a > 0, B (0, 1) and l E 1x /a). since this would capture increasing number of directions with n.
16

---

The eigenvectors u, define the Published as a conference paper at ICLR 2026
Mean Component Index. To make these notions precise, the Mean Component Index (MCI), which quantifies how much of the loss change for an example occurs in top versus tail components. Examples with small MCI correspond to the "easy" examples above and those with high MCI correspond to the "hard" examples. Let us recall the setup in Section 4.1. For simplicity, let consider single layer W with gradient G' and update oW at time i, since everything will extend to multiple layers by summing over each layer. As before, let S(G, 8W) from Eq. (2) be the alignment matrix between G and óW with eigenvalues AI...., An ordered such that Ar
introduce per-example statistic,
2 Am and corresponding eigenvectors u...... Un component directions our decomposition. If we have data points and the loss L is the average over pointwise losses l with gradients G;, then G' is the average over G; and we can decompose 5c5(W) = (G, 8W) into
P
P
P
( = (G, (W) = P Σ
3Σ n G. ow = G. OW) = P im1g=1
1
1 P
AΣ 2
(80))s
where we define (òb)) := 3 G. &Wa, to be the instantancous linearized loss change of sample i in the j component from updating parameter W. By summing (ōv)), over time steps t and matrix layers can define Vay to be the linearized loss change of sample in the component over the entire trajectory. The mean component index MCL; E [1, n] for example is defined as the quantity
MCL: Σ jPs Pj j=1
16sd and dyj == Σ leal
(ōx) (W! (O).
(6)
EEILI
We use MCI; as a scalar index placing example along the easy-to-hard spectrum described above.
Empirical Findings. In Figure 14a we show the distribution of MCI, for network of width = 128. The average MCI, is about 12.9, which is much smaller than n. and the distribution left-skewed. This indicates that most samples are learned mainly through the top components. consistent with Figures 33 and 35.
In Figure 15 we visualize the four examples with the smallest and largest MCL. Observe that examples with smallest MCI shown in Figure 15a are visually simple (top panel) and are essentially learned by the first component (bottom panel) in our decomposition (6), whereas the large MCI examples shown in Figure 15b are more complex and require non-trivial contributions from tail components (i.e., j > 10).
Figure 14b quantifies how similar the rankings of MCI, over samples are across widths and random seeds. For widths 11, f2 and random seeds $1, 82, we take the top or bottom 5% of samples ranked by MCI for each width and seed pair, giving us two subsets S1 and S₃ of [P]. For given pair of widths we compute the fraction of overlap |S, n S//00.05P and average over pairs seeds 81 and 82, ignoring
if n4 112 since this would be trivially equal to one. We observe less agreement when min n2) is smaller and we restrict to samples with higher MCI, suggesting that easy (low-MCI) examples are learned more consistently at different widths and random seeds than hard (high-MCI) examples.
Width 128, 0.016
Hardert of Samples
Eaxiect Samples
214
Frequency
Wilth
552
25:1
2545
4141
424
Mean Component Index
124
Widts
200
120 350
2014 430
(a) Histogram of MCI (CIFAR-10).
(b) Consistency of top 5% hardest and easiest examples.
Figure 14: Left: MCI computed on CIFAR-10 for n. 128 and learning rate 0.016. The distribution is left-skewed with small number of large outliers, indicating that most examples "easy". Right: For each pair of widths and random seeds, we compare the sets of top or bottom 5% of samples ranked by MCI, and plot the average fraction of shared samples. Diagonal values need not be since entries compare different seeds at the same width. Rankings are more stable for low-MCI (easy) examples and larger widths, and less stable for high-MCI examples and smaller widths.
17

---

Published as a conference paper at ICLR 2026
MCI, 3.8
MCI,
MCI, 40.4
MCI 34.6
33.7
MCI, -33.6
(a) Lowest Mcan Component Index ("casy" samples). (b) Highest Mean Component Index ("hard" samples).
Figure 15: The five examples in CIFAR-10 validation set with lowest (left) and highest (right) MCT for two-layer MI.P of width n = 128 trained with SGD using LR = 0.016. Top row: We visualize the samples and the corresponding MCT. Bottom row: We plot the t/ay values for the first 50 components j (see Eq. (6)). Note that the low-MCI examples turn out to be simple images from the airplane class which are learned early in training using simple features, while the high-MCI images are complex examples from different classes which are not learned well by the model.
Due to the consistency across scale observed in Figure 15a, we intuitively expect the optimal learning rate for these easy examples to be stable and transferrable. Hence in Figure 16 we probe the effect of easy versus hard examples on learning rate transfer. Up to this point, all losses have been computed on the full validation set. To isolate the effect of individual data points. we now evaluate learning-rate sweeps on subsets of the validation set consisting only of easy or hard examples, based on the previously computed MCI values from training a single model of with (lowest 25% MCI) we see near-perfect transfer (Figures 16a, 16b), and the optimal learning rate is very close to the one obtained on the full validation set reported in Figure 13; this suggests that the optimal HP on easy examples- - learned by the top components and remains stable across width approximately decides the optimal HP for the full loss. In contrast, on the hard subset (highest 5% MCI) we clear leftward shift in the optimal learning rate width increases and noticeably worse transfer (Figures 16c, 16d). This illustrates a potential failure mode of HP transfer in practice and shows how our sample-based decomposition can help reveal such failures: if the downstream evaluation metric & is nuuu-o--distrbutionn and concentrated on high-MCI examples, then HPs transferred from a small model can yield suboptimal performance. We leave further investigation of this sample-wise decomposition, for example in language model settings, as future work.
128. On the easy subset
Easy Examples les
Easy Examples
0.925
120
0.703 LRON) (m)
212
0.50
0,792 527 (RiI) y (220)
5.000
4095
0.88
0.875
1,00
0.86
0.630
0.80
0.625
0.82
0.80
2
Learning Rate
Width
(a) The final EMA loss (easy examples).
(b) Scaling law for EMA loss (easy examples).
Hard Examples
Hard Examples
3.05 3.050
2.04
L-1934 1340 p (80 - In)
3.025
2.02
2048
3.000
Loos
1.98
1.96
1.950
1.94
1.955
1.52
2/
Learnnng Rate
Width
(c) The final EMA loss (hard examples).
(d) Scaling law for EMA loss (hard examples).
Figure 16: EMA loss on the 25% casiest (Top) and 5% hardest (Bottom) examples in the CIFAR-10 validation set, based on MCI computed on the width n 128, LR 0.016 model. Observe that for EMA loss on the easy subset, the optimal learning rates are well-aligned across widths and match the (large width) optimal HP on the full validation set (Figure 13a), whereas the optimal learning rates for the hard subset shift significantly and yields suboptimal transfer.
18

---

Published as a conference paper at ICLR 2026
B BACKGROUND
Function Class Regularity Let X' be compact metric space, and let ((2) denote the space of real-valued continuous functions on X, equipped with the uniform norm:
|l If := sup [f())]-
ve
We say a collection & C ((X) is:
uniformly bounded if s IIfllsup S K for some K < oc,
uniformly equicontinuous if for every there exists such that
| v <
(5()) j(v))
< for all fE
We denote by (X') the space of k-times continuously differentiable functions. For C* (X), the k-th derivative is written (kk), We define f(0) = f. In multivariate settings, this refers to the k-th total derivative.
Theorem 4 AAzzlå-Ascoliy Any uniformly bounded and uniformly equicontinuous collection F C C()) is relatively compact in the uniform norm topology.
Proposition 5. Let {fn} c. C (2) such that f - uniformly and fn (vo) - L for some Vo E X and LE E E. Then fn + f uniformly for some f C. (X), and f == =g-
Proposition 6. If {ffn (X) and the derivatives fh are uniformly bounded, then {fn} is uniformly equicontinuous.
B.1 SCALING LIMITS AND TENSOR PROGRAMS
In this section we will recall some simplified background from [YH21; YL23). For concreteness, will fix the architecture more generic architectures (see Section 2.9.1 in [YL23]). An L-hidden layer MLP of width n with nonlinearity & : RR IR and no biases is parameterized by weight matrices W1 E Raxd W 4 E Rax= and WL+1 E R lxn, On an input a E Rd the network computes
L-hidden layer MLP, but all statements can be extended to much
he h(()) W of (x) E R", 2 (x) ((h R" for l = 1,...,,
and the output is f(x) L+1 x E R. Given N inputs Bi...., wN. we will abbreviate w
h' = [1:()) ... h [xN)] ROXN の = (z)))) ... 12 E 1
= ((Ex)...,,(() N
abc-parameterization Assume that we train the network using SGD. We recall the definition of abc- parameterization from [YH21] (see [GSJW20; CN24] for similar findings). An abc- parameterization width-aware HP scaling specified by set of HPs {ay, de, me (e(ee(t+1) and HP scaling exponents 7 {aa, et te such that
(a) The weights 8 receive multiplier aun
(b) We initialize each W Wa NO0, -2e), and
(c) The SGD learning rate in layer l is mem -e.
Asymptotic Notation Given sequence {2(n) of random tensors write O(n -a) and say that x has coordinates of size O(n ") if there exist constants A, B > such that, almost surely for sufficiently large n,
AS #æ(n)
Σ < B,
where #arl)) denotes the number of entries in (n). We O(n and 2(n=*) similarly.
19

---

Published as a conference paper at ICLR 2026
Dynamical Dichotomy Theorem. We recall some definitions from [YL23]. To reflect the network aftert steps of training we add a subscript to the quantities in Eq. (7). We use A to denote a one-step difference of timeddependent quantity. We say abbe-prraeterrzationn is
1. stable at initialization if
h6. = (1), Ae [L], and fo O(1).
2. stable during training if for any time? we have for any training routine
Ah Az = O(1), Ae E [L]. and Af: = (1).
3. trivial if for any time t 2 and training routine, ft fo - almost surely as n - co. We say the parameterization non-trivial otherwise.
4. is in the kernel regime if there exists K. IRN + R such that for every 2 0 and training routine, as n + 0O,
fi+1
(ff.) -
5. is feature learning if Azt (21) for some training routine and 2 0.
Theorem 7 (Dynamical Dichotomy [YH21]). A nontrivial and stable abe-parametrization either admits feature learning or is the kernel regime but not both. The kernel regime does not admit feature learning, that is, for any training routine Azt 0 for all 0.
The μP and NTK parameterization are the maximal feature learning and kernel parameterizations respectively. All other such parameterizations can be obtained from one of these by setting some initialization or learning rate zero (see Section 5.3 in [YH21] for more discussion). For adaptive optimizers such as Adam is possible to extend the definitions to abcd- parerarreeterizaiionss (see Section 2.2 in [YL.3]] for more details), for which similar Dynamical Dichotomy theorem exists.
B.2 MUON OPTIMIZER
Here we recall the Muon optimizer introduced by [JJB+24). For transformers typically the embedding and output layers, as well as vector and scalar parameters are trained using Adam. For the remaining matrix parameters, the update is built around the matrix sign function msgn(-), which for full-rank matrix X with SVD X VVV is defined as
msgn (X) UV
For parameter W Rmxn with gradient G Rm step t we form momentum matrix
M = 3M, + (1 B) G.,
with momentum parameter B E [0, 1), and then update the weights by
W = W. "V m p(M) = W. -V
mV m msgn Mt),
where 7) is the learning rate, the factor / mm/n ensures μP scaling, and p is a composition of low- degree matrix polynomial chosen approximate the function msgn. This approximation comes from an iterative Newton-Schie method where the number of iterations gives the number of polynomial compositions in p.
B.3 DION OPTIMIZER
Dion [AXA + 25] is orthonormalization. baase optimizer designed to retain the benefits of Muon- style matrix updates while remaining compatible with sharded weight layouts in large-scale LLM training. Instead of applying a Newton Schulzz iteration on each weight matrix, Dion operates momentum buffer and uses an amortized power iteration to obtain an orthonormal low-rank update, controlled by rank hyperparameter and stabilized by error feedback.
For matrix parameter W g with gradient G IR m at step t, Dion maintains residual momentum matrix M E Rmxn and a right factor Q E R where r is the rank hyperparameter. Each step begins by accumulating the new gradient into the residual momentum,
M M, G.-
20

---

Published as a conference paper at ICLR 2026
An approximate leading left subspace is then extracted using the current Qt
Pt orth M Qt),
RS
M Pt,
where (-) returns a matrix with orthonormal columns (e.g. via QR decomposition). The right factor is updated by column-wise normalization.
R
j 1,...,
R
12 +
with small for numerical stability. When Q: spans the dominant right-singular subspace of M1+1, this amortized power iteration drives P. and Q++1 toward approximate left and right singular vectors of M,+1, with both factors having unit-norm columns.
The residual momentum is updated with error feedback.
M++1 M - P.R.
where j E [0, 1) controls how aggressively the dominant rank-r component removed. In the idealized regime where P, R. captures the leading rank-r part of Mc+1, this update geometrically damps that component by a factor μ while retaining higher-rank and previously truncated directions in M1+1- These retained components can subsequently re-enter the leading low-rank subspace, so the residual acts as an error buffer that compensates for the information lost by low-rank truncation.
The weight update Dion uses is
m P.Q..11
W+1
W.
" n
where the learning rate is and the factor vm/n ensures yP scaling.
21

---

Published as a conference paper at ICLR 2026
C THEORETICAL RESULTS
C.I WEAK TRANSFER
In the following we provide a minimal set of technical conditions for the concept of HP transfer to be well-defined and align with empirical observations. We say that function is locally strongly corvex (LSC) with parameters T, 6. > 0, if for every minimizer v² arg min f.
for all such that | v 11 S (s(w) l 2 -- Recall that we assume that the HP search takes place over search space X::
X II uil. int (X)
I u.),
1=1
which is h--imensional box with bounds & u4 and interior int (X). Definition (HP Transfer). The scaling 7 admits HP transfer for training procedure A and metric over search space X if the following hold almost surely
1. on E (X) is convex and has a unique minimizer v² vn)).
2. There exists LSC deterministic 6:0 such that on + do and arg min do S int (2).
3. The family { is uniformly bounded and uniformly equicontinuous.
This definition implies the following desirable consequences. Proposition 8 (HP Transfer Properties). In the context of Definition 3, the following are true
1. doo E (X)) is convex and has a unique minimizer v² (oc)
2. On + Poos &' + O'xo 8% + 0% uniformly and (n) - (oc).
Proof. By Proposition 6, it follows that for all j {0, 1, 1, olf is uniformly equicontinuous. Since on has a limit, it must converge uniformly Φxo- Now repeatedly using Proposition 5 after passing to subsequences and invoking Arzela-Ascoli (Theorem 4), we see that / - O uniformly for j E (1,2). Now it follows that &x E (2)) since the uniform limit of continuous functions is continuous and do is convex since convexity is preserved under pointwise limits. Note that the local strong convexity condition implies that oxx, has unique minimizer. Now because X' is compact, every subsequence of (n) has convergent subsequence. By uniform convergence and the continuity of the on and oo.. it follows that the limit of this subsequence minimizer of doc- By the uniqueness of this minimizer, all the subsequences converge to 1²* (co) hence v (n) - v (Dc).
This definition gives minimal set of conditions under which the concept becomes mathematically coherent and aligns with observed empirical behavior. Requiring dn, have unique minimizer removes ambiguity about which configuration should be transferred across scales. The convexity, smoothness, and equicontinuity conditions provide technical regularity that facilitates analysis and generally hold in practice.
The local strong convexity of doo ensures that performance meaningfully degrades away from the optimum. Without this condition, 6o0 may be flat near its minimizer, making accurate hyperparameter selection potentially irrelevant in the large-n limit. The assumption that v² (oo) lies in the interior of search space ensures that this optimum remains unchanged under any enlargement of the domain; this condition rules out the n-dependent drift of the HPs of interest due to "suboptimal" scaling see Appendix B for discussions.
"Optimal" scaling limit. In [YHB+22) YYZH23J. the authors remark that key principle behind hyperparameter transfer is the "optimality" of the scaling. Heuristically if scaling yields a suboptimal limit, then it cannot exhibit hyperparameter transfer since the HPs need to undergo n-dependent rescaling to "convert" the suboptimal scaling into the optimal scaling. The following proposition formalizes this intuition. The proposition requires 0 E X' to avoid uninteresting cases where the optimal HP is zero which will generally not occur for optimization HPs normal neural network training.
22

---

Published as a conference paper at ICLR 2026
Theorem 9. Let -y be scaling that exhibits transfer over X [0, un] 1xl and all x
containing X. Any other scaling o + with these properties must satisfy
min (0: min @x (V): vEX
v'ex'
Proof of Theorem 9, For brevity define
V. arg min d00 (v and 1/ arg min 2o (v':7'). w't
VEN
For the sake of contradiction assume that 6: (v,:7) > 2 (v' 7') For large enough n, we will have (v. > Pn (v.:7) where V. V/ @ (n n 7))
vk) and V, * V. which is
contradiction. The case & (M.i)) < & (vl:n') follows analogously.
The dynamical dichotomy theorem states that all scalings induced by aladeparameeterizations except for μP lead to optimization degeneracies or non-feature learning behavior. Therefore the proposition implies that if HP transfer is possible with μP and feature learning advantageous, then should be only possible using pP. Of course, it is still a challenging problem to rigorously characterize when μPP will exhibit transfer and when feature learning actually advantageous.
C.2 ASYMPTOTIC RATES
Recall the definitions of the quantities an, bus Cn from Definition 1. We will need to reason about the following quantity which we call the uniform loss gap.
Og = |
0 Iluup"
Using the local strong convexity assumption, we will be able to directly relate the HP gap with the uniform loss gap. We first prove convenient lemma which bounds the minimizer displacement in terms of the sup-norm of the perturbation.
Let f X - R and g : X -> R such that ʃ has a unique minimizer xy and g has Lemma 10.
unique minimizer Eg. and g is strongly-convex
g(x) g(x,g)
7 x xzll? Va X
2
for some > If -9 9 gllaup I E, then
|0, - S 22(-) 1/2
Proof. Note that g(æ,)) - S {(x)) S / (1xgg S g(x.g) + e, hence
g(x,) g(x,)) s 2e.
By strong convexity, 2e 2 7 /z. =.yl² which after rearranging gives the desired conclusion.
The above lemma, along with Propositions and Taylor's 's theorem immediately yields the following. Lemma 11. Assume HP transfer holds (Def. 3), then b, O o(a!! 1/2) and c, = ((2)).
To relate the loss gap an and the uniform loss gap ag we must make further assumptions. Intuitively, would like capture the fact that typically the uniform loss gap is dominated by fairly uniform, positive loss gap for HPs which are nearly optimal.
Definition 4. We will say that the uniform loss gap is locally tight if there exists some radius F and constants 0 < S C' such for all B(v" B(v²(2o), F).
cān S 2r - (v) < Cā..
This essentially states that the uniform loss gap tightly controls the convergence rate for any nearly op- timal set of HPs. This corresponds with the empirical observation that nearly optimal hyperparameters obey identical scaling laws. Under this assumption it is easy to see that a, (aā,).
23

---

Published as a conference paper at ICLR 2026
Lemma 12. Assume HP transfer holds (Def. 3). If the uniform loss gap On is locally tight then an = (a,))
Proof.
Note that we have
@n(v"(n) 6 (v* = on (n(v((n) - ox (x=(((n)) + (v) (v=(1((0)))
2 2n''' (n)) (x(('(n))
(n(v"(n)) 2 (v" = Φn (v' (n)) o.(v"((o)) + pn (v' (oo)) - )xo
I Φn (v' (co)) dx (so)).
Since v¹ (n) + v² (co) we can apply the inequalities in Definition 4 for large enough n to yield the claim.
Proof of Proposition 1. This follows directly by applying Lemmas 11 and 12
C.3
GRID SEARCH
We now turn towards the connection between the previous asymptotic quantities and compute-optimal grid search. Define grid 9 in search space X' be collection of points {z(1), (M) contained in X. The grid resolution p(g, X) is defined as the largest distance of a point in X' to point in G, that is
p(g, X) := sup min ||w-v'l V ee
For grid G in the search space X. define (n, arg min on (v). Let us assume that we are allocated flops budget F order to perform hyperparameter search and produce a final model.
Recall that for brevity we use [xx g(ar) to mean (x) = ((g(z)). For a grid g of resolution p. we will make the following convenience assumption for Theorem 2.
Assumption (Grid proximity). For grid G of resolution p. we assume that
min | - (n)]l p. MEg
This assumption is morally true G is not chosen with knowledge of the location of v² (oo). If for given G chose vz (xx) uniformly at random within G. then this assumption holds average. Suppose we have a compute budget of F flops, and the amount of flops needed for a single training scales with n² for models of width n, where r for standard optimization algorithms standard architectures. For scaling we will evaluate the quality of a set of HPs v by performing full training run for certain width using the scaled hyperparameters H., (v,7). Let assume we perform grid search over HPs. We first consider compute optimal performance when directly tuning the HPs on a large model.
Proof of Theorem 2(a). For a grid resolution p = =(G we will have [g] ~ pー and F ~ n" p- Now observe that by uniform convergence of derivatives (Prop. 8), for n large enough oi will satisfy (r', 5'S-L.CC for some constants 7',5' 5' > 0 and so &n (v* (n, 9)) on (v (n)) ~ p² by Assumption 1. Therefore,
Φn (n, - =bn( (v" (v(("((n))) ()) - 2nlv + (n &
0² n
n 2 2-/h 2/h + n a0
We see the final expression is minimized by taking n' ~ F Fra which yields the rate
Ón (n,
oi F 2a
as claimed.
Now consider the strategy of transferring the optimal HPs from smaller model. We say that transfer is useful if this strategy achieves a better loss scaling than directly tuning the large model under the same compute budget, specified above.
24

---

Published as a conference paper at ICLR 2026
Proof of Theorem 2(b). Note that in this setting F ~ p- + Mr. The performance scaling is Φxr(v" (n, G)) - 0 - -Dar(v' (n, ()) - (m(("(n)
+ *MM(U (n)) - (M))
+ (Mr(v(M)) - -o
p² + n2 + M
n"
2/h
+ n M
F- M
Since F - M² ~ F, we should take M ~ F//r in which case the above simplifies to
n2r/h
C2 0: F2/h
+ 223 + F F-a/rr a/r
which is minimized a n² ~ Furtr and yields ΦAs (v' (n, 9)) - -Øx F F Now
Now note
that transfer.
and hat2r
2a
h3+r
23
>
ha+2r
20
if and only if 3 > a/2, which is the condition for useful
C.4 RANDOM FEATURES REGRESSION
Consider the following data generating process where the labels come from a single-index model, and we train random features ridge regression estimator N samples,
y (xx, B.)) +,, where (N, Ia), ||@- = 1, Var(s) = =
(ax o (Wx)); x)), where W E Rdxn [W] j.g ~ ~N(,
ax = argmin RER" (v
A|
We aim to select the optimal regularization parameter that minimizes the prediction risk (general- ization error) R. = (y [z))) We make the following assumptions.
Assumption 2.
Proportional limit. N,d, n - oc, oc, N/d vi, 1/2 where th,, th2 E (0,
Normalized activation. Both the student and teacher nonlinearities are normalized such that [[o.] = 0, ||o I|o.lly = 1, and also l ||o''l,, + O. We further require that is nonlinear odd function with bounded first three derivatives, and is ((1)-Lipschitz.
Remark 1. The above assumptions are standard in the high-dimensn digeedimensional asymptotic analysis of random features features models, see e.g., [MM22; GLK* 20]. The non-zero expectation of o', o', is necessary for the RF model to outperform the null estimator in the proportional regime. The assumption of odd simplifies the Gaussian equivalence computation see [HI.22; BES
22].
Asymptotic prediction risk.
Under Assumption 2, following [HL22], we know that the asymptotic prediction risk is given as by the following implicit equations,
m (A)
my (A)
lim Ely - (x=)) R(A)) = - + o²)
(A)2 1
n.d,N +00
my m, (X)2
where the Hermite coefficients μ4 = EE (z)] م3² = 1 - pf² and the coupled Stieltjes
transforms m2(2) and m²(²) E C+ U URR are uniquely defined by the following self-consistent
equations for z E C+ U R.
(m1(2) m2 (2(2)) + (z))(()) + pîm (2) (zm1 (2) 1) 0,
(9)
11
1 (mz (2) - mm(())) + 1mm (z)m²(2) (zm, (2) - 1) = 0, (10)
(2 + 11
= 1- nř. Note that being nonlinear implies μ22 * 0. We omit the argument in my (A), (A) except when tracking the X-dependence. m/, m' stand for derivative with respect to X. To further simplify the exposition, we define n := th ///22, and write the asymptotic prediction risk at width 2hs and ridge penalty À as Rv (A) R2,/n(A).
where # Eo. (o ((((((())))) (z)]
25

---

Published as a conference paper at ICLR 2026
Large-width limit. corresponds to taking 1/2 n/d + 0o or - Note that the prediction risk in this limit is well-defined and has been computed in prior works (see e.g.. [BMR21J). First recall that my, m2 0 and 2m1 remains uniformly bounded for any 1/1, hence from (10) we know that at the large-width limit,
First consider the test performance of the "infinite-widd width" model, which
pfmımy + (m m1) Fi (m1, m2) = 0, Am1 + p2m1 + p2 1 [m m, A) = 0.
Reparameterize t == + p m1 1. we have A(t)
18v1 11. - -1.. By the chain rule,
t-1
p7,(()
0am1 pGv1"
0em2 =
0,1 = t - 1)2
where we defined S(1) = 1b - (1 1)2. Hence 0 we have
mí
v/t
m mi
4
mi
S(1)
S(1)
Therefore,
Ro (A(t)) = lim Ryd. (A(t)) V2 + oz + +12)) +20
8(1)
Differentiating the risk yields the closed-form expression of the optimal ridge penalty (consistent with I8: WX20)),
((0)) 11(00 + 112
-12.
(11)
pT²
We restrict ourself to the setting where non non-sinis regularization needed at the large- width limit, i.., X3 (co) > O. Denote I, the corresponding optimal value of /. / mim the optimality
condition n3² 1.+.
(o (1.-1) implies that
Vit. t, + +1>0, O, S(t.) (Vit. t. + 1)t, + (t. 1) O.
Hence we have the following characterization of the curvature
2(13² + +
1t,
(X)()) ((t t, 1)
0, X'(t.) 12(t. 1)2 0. (12)
Note that (12) validates the local strong convexity of Rox-
Finite-width sensitivity.
Now consider the system given by (9)(10)
J2 m2, A)
E(m1, m2, X, (m,, m2) + 77 3 m2, A)
where F3 pmmmmmg (m 1). Differentiate 0 with respect to n and evaluate at 1) 0,
1gym1
0
x
Jo
0g
T
where
Jo = mm mg (2,
[2/m2
f1,m1 + et
0
(13)
Recall that Fg = 0 yields Am1 - 1 = - (nām + m2 and hence Th = - (pmm)
+
(ff22). On the other hand, by direct computation
(1)((1
det = [
+ del.o(l.) > 0.
(14)
Solving the linear system (13) yields
4m3m + 1m)2)
(t (p² + pa²t)
0,m, =
dot.Jo
26

---

Published as a conference paper at ICLR 2026
0, = (A+ + p3) m2 (n3m, 1(/22) (t 1)8 (1)t) ++ 1) 14v)t2S(t)
det.Jo
Differentiating the prediction risk with respect to yj and evaluate at the firge-width limit vj O. Ro (A) = -(p2 + oo2)
Oymi
2m, 0ppm
10mmé mf
2m mi
=2
mi
mi
where R (A) Ry,, /v (,)))) 1=
A similar determinant calculation yields
5
0yml (A + pg)
0,
detJJo
detJo
where
y == : mm (2Am1
1)mi + p (Am1 1)ml + pfm3m2
1)3
t
+ (2t + 1) t- μ + 1
(t 1)
S(t)
S(t)
Using the above, a tedious algebraic calculation gives the following expression of the sensitivity of the prediction risk (at fixed A) with respect ry:
(1 - 1)²(())
Rox 17)5(t)2
(15)
where Q(t) Σ k= ceth with coefficients
C2 1 (m² + +o?) - p22 + 0.)) + 2222211 (v/1 - 1),
Q 3n² + +
+ p2pt
1) 13(122 + oz) p2247(11 + +3),
Co (u² + +o2)
24² (2n2th 11 2n 1).
Importantly, under the optimal defined in (11). we have the factorization
Q(4+) 2(p²² (M) a2t.)(t. 1)S(tt)) 1it. to +1
and since (t. 1), (Vit, - +1) > O. we conclude the derivative (15) at t. is strictly positive Roo 2(16² + oZ) + (l..)(l. 1)) C₁ > 0.
(16)
((bitt t. 1
Putting things together. Recall that the asymptotic prediction risk R. is C2 in and C² in
7. Given the Jacobian invertibility (14) and local strong convexity (12), the implicit function theorem (IFT) implies that there exists a neighborhood defined by some 7an > 0 and a unique C
0. and (x (n)) 0. Consequently, we may take a first-order expansion and conclude (setting 1 = t/1 t//
map X [0, no) - R4, such that X (0) X (co), R,,
/ under with 4/1)
o--00, (²))) θxa, Ro (o0)) Cx-
X (v/2) X* (co) + 0 (n)
oV 2
RR (A-()c))
(17)
Note that the denominator in Os is strictly positive by (12). Moreover, since ((t..) * 0, we may write (x = (,(0,_o and compute Cx (3o) 4pf tIe
1]
This
(()
20h
confirms that the hyperparameter gap vanishes at a rate Ov
For the loss gap, denote 6, X2 (1h /n) - (o0) for 77 E [0, mo). the IFT and Taylor expansion gives
(X (X* 5n) = Ro (X (oc) + 02= (X (00)),, Rx O v/?)
1/1 + o(
(X
Cy
J/2
where (i) is due to the stationarity condition (17), and C, > 0 is explicitly given in (16). The strict positivity of C₁2 ensures that the loss gap scales exactly as O(v-"). Hence by Proposition and Theorem 2 know that the ridge penalty in RF regression exhibits fast and useful transfer, i.e., the suboptimality gap |R (X* (vz2) Rxx (X* (co)] viz" & [R,,, (A²(2))) (x" (xo())) which aligns with the observations in Figure 4 and concludes Theorem 3.
= 0 and 8, = O(n) from
27

---

Published as a conference paper at ICLR 2026
C.5 DECOMPOSITION-AWERR FAST TRANSFER
In this section we formalize quantitative bound the HP gap b, in terms of the top-k invariance and residual flatness arising from our decomposition. For the sake of simplicity we will assume that X' is small enough that the local strong convexity condition holds globally.
Definition 5. For f E (2)) define the curvature u(f) and the Lipschitz constant Lip(f) as: u(f) = inf DEX
int (")), Lip(f)
== sup vEX
suplff()) lf'(
Definition 6 (Decomposition Rate). Let On and X' be as in Definition 3, and 05 and (n) as in Eq. (5). We define the following quantities associated with the decomposition:
110% 4 Happ Top-k invariance gap: (n, u(Ag) u(Ag)vu((s))
Residual flatness gap: (nta k) == Tip(o=")
Tip(o=")
m(on) + a(dx)
Decomposition objective: [R: == 2 K) + Caae (n,
Decomposition HP gap: in == min, [(x) s.t. u((S) p((=)) >
The decomposition HP gap in is defined to be a natural upper bound on the HP gap bn as we show in Proposition 13 which makes use of Lemmas 10 and 14. This upper bound is obtained by choosing an optimal HP dependent truncation index K,, minimizing J (N) such that Einv K,,,) which quantifies top-x invariance and ftta (n,, kQ))) which quantifies residual flatness are both appropriately small. Note that t, is well- for n large enough because we can take K and from the assumptions of Definition 3 both on and @oo are strongly convex.
Proposition 13. Assume the setting of Desinition 3 where the local strong convexity global. The decomposition HP gap in in Definition 6 satisfies i, 2 bn where by is the HP gap from Definition I.
We remark that we introduce the quantity in primarily as theoretical quantity for conceptual purposes. The quantity ia will be small when top-k invariance and residual flatness holds and since In 2 by this will imply b, is small as well. We also note that it is natural to chose the optimal truncation index Kt, used in fn to be a function of the width n. This is because as n - Do we expect b, - 0 and so it is desirable that t, + 0 as well which will not be the case if we used fixed because Eaat in J (K) will converge to a non-zero value. Overall, one can view f, being small as an explicit sufficient condition for fast transfer. We conjecture that (some version of) this condition holds in practice when training neural networks on natural data with optimizers like Adam or SGD, For future work, it would be interesting to provide natural settings where such a formal statement is provably true.
To prove Proposition 13 we will first need a perturbation result similar to Lemma 10. Lemma 14. Let f : X + R and g : X + IR such that f is T strongly-convex and sup Ig (x)) S e. Let x" = arg min f(x) and w = arg min f(x) + g(x)). Then
||~-" i -
Proof. By first order optimality we have f'()) + g' hence |f'()) (g'(0)) S e. By strong convexity Ø x+1 S If (x) S e which gives the result.
Proof of Proposition 13, For a given and x such that u(o)) A m( > 0,
bn (2'(n) (n) - v'(o))]
+/) - 2³ (co)] [((()) - -n)(m)] + [12(()) - ( + [11((00) - ('0))
(n) (n)(n) + (n)
2 (n, k) + (1n(m) (n)) + 110) - v'(0))
< 2 Eiv (n, x) + Eaat (nz x)),
RRelative to the bound by implied solely by an (see Proposition 1).
28

---

Published as a conference paper at ICLR 2026
where the first inequality is the triangle inequality, the second comes from Lemma 10, and the last inequality comes from Lemma 14. Taking the minimum of the right hand side over valid x yields the claim i 2 bn.
29

---

Published as a conference paper at ICLR 2026
D TRUNCATION INDEX SELECTION
Empirically finding the minimizer x² (n) the definition of ln (Def. 6) is not tractable due to complicated nature of the decomposition objective J. Instead of trying minimize J, will use simpler surrogate process which we outline below.
Note that given a finite grid of HPSs (V1,..., "ey.....,,),. we only need to produce a truncation index for each D, with E [g]- Let K (Ki:..., (g)) represent the vector of these values where R; k<(;))
for i E [g]. We define %% to be the set of pointwise evaluations {(V,, 0% (v:)) helgi and identify with the function obtained from its linear interpolation. Our goal is return set of truncation index vectors (())..
Let Tmmax denote the largest width model under consideration and fix a width n < Tmmax- Consider the following proxy objective with parameters (TI, T2) and TI, 72
Jproxy (rE; T) = 9 selsl
(wi) - 0% nemax (4:) + Lip(on + Lip(
(18)
and define its minimizer to be K (T) arg minl Jproxy T), which can be found approximately using coordinate descent (see Algorithm 2). The objective Jproxy is similar the objective in Definition 6, except that instead of a supnorm we use average f fi-norm to promote tractability, we use maax an infinite-widee proxy, and we absorb all the curvature based scalings y(.)) into constants TI, We will set R(()) (f) for a "reasonable" choice of f. In particular, f will be that on (%"(T))
chosen be the smallest' are close to the minimizers of db. and @n respectively, assuming such f exists. The full details of Appendix
(r) and Pnamxx are approximately convex and the minimizers
process are given Algorithm
In this section we describe our procedure (Algorithm 1) for selecting k(n) (see Section 4.1). We do not claim this procedure is optimal in any sense and emphasize that we are just searching for valid R(n) so that a certain sufficient condition holds qualitatively in order to support our conjecture for fast hyperparameter transfer. For convenience, only consider the case where sweep single HP, although this can be straightforwardly extended.
As part of our algorithm, we need measure the convexity and flatness of function f given set of pointwise evaluations {(v,, })) where 1/1 <... < V and yı f(1).. For each interior index i 2, 2.....g 9 define the three-point second-derivative estimate
("( = (1:-1-1i(11-1 - vi+1) (v:
9i-1
y/3
V/+1
14-1) - v1+1) (Vi+1 - 14- 14-1) ( - v.))
-
The convexity error the fraction of interior points with negative curvature estimate:
Σ [f( <0)
CouvEnt((Pe. (:)))
(19)
The Lipschitz constant the maximum slope magnitude:
Lip(((V,, 31))%_ max 18159-1
Vi+1 1 f/i
(20)
1/+1 1/
'For definiteness, we order (71,72) 72) their sum, breaking ties in the first-coordinate.
30

---

Published as a conference paper at ICLR 2026
Algorithm I:
Compute k(n)
Input:
N
set of widths, with Timax max.N grid of g hyperparameter points candidate list of J values tolerances (convexity, argmin proximity) arrays 0% (2,) for each E
{v.
Sevxy Camin Pn E &Rxx
Output: k(n) truncation vectors in [n]] for < Mmaxx or FAIL
1. Grid: 9, = {(T1, 72) -I; 72 E T) sorted by 71 72
2. For each E E \ [Mmmax]
(a) For (T1, T2) E 9- (in order):
i. +- MINIMIZEPROXY (2n, 2 mmaax T1, T2)
11 Alg. 2; minimizes
Eq. (18)
max{ConvEiis (ConvErr(phe di,,, ConvErr(dr. (f )} ii Eevx = max
// Eq. (19)
iii. On == arg min, Pn a = arg min, On:
ax arg min, Pmmme a arg milk Prmax
A = max (la, 1oo 1 1
// argmin proximity
iv. If Eevx S Sevx and A S Eamin set k(n) + and break to the next n.
(b) If no pair in Gr is accepted, set R(n) <- FAIL.
Algorithm 2: Minimize Proxy Objective Eq. (18)
Input:
Pn, Pmmax (TI, T2)
arrays of and of Rmaas (vi), shapes g x n and X Tmmx
positive penalty weights
Output: R² E [n]o (approximate minimizer via coordinate descent)
Initialize: k +- (n/2....,,
2. Repeat until no coordinate changes:
For 1.
Local scores for each [n]] score(k) Jproxy (R) where & is with K; switched k ii Coordinate update: f- arg minkels) score(k).
3. Return x² = -R:))
31

---

Published as a conference paper at ICLR 2026
E EXPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS
E. μP MLP EXPERIMENTS
Experiment Setting. We consider regression with shallow ReLU network: f(z)
the learning rate 7) that minimizes the validation squared loss. We take the target function to be the following multi- model on isotropic Gaussian data,
E= a.o (.ow + b,), where we aim
V - 2 x~ P~, 4d La)
d
We set 15, and d 64, 2 We run the Adam optimizer for T 2 steps with batch size 256 to minimize the squared loss. The initialization and learning rate are set according to pP. As shown in Figure 2, the optimal learning rate steadily approaches 10-1 before an abrupt jump at width 8192.
width n=128
width n-
width n=2048
width n=2048
n=16384
Validation Loos
Optimal HP
Validation Loss
Optimal HP
001
Learning
Learning 7
(a) Identical all- layer learning rate (Figure 2).
(b) Different layer-wise learning rate.
Figure 17: Learning Gaussian k-index model with two-layer ReLU network under jP. The HP of interest is the Adam learning Left: default pP implementation with the learning for the first- and second-layer parameters. Right: layer-war wise learning rate where the second-layer learning rate divided by 50. Observe that the original μP yields a "bimodal" HP curve, whereas layer-wise HP leads to more stable transfer.
Layer-wiss Learning Rate. One possible explanation of the "bimodal" learning rate curves in Figure 2 (duplicated in Figure 17a) is that there exist different mechanisms to lower the loss that may dominate learning at different widths and require separate HP tuning. our two-layer ReLU network, while the first-layer neurons rotate and grow in norm, the second-layer parameters can only affect the norm (see e.g., discussion on Wasserstein vs. Fisher-Rao dynamics for mean-field neural networks [Chi22]). There no reason priori that the optimal learning rate for the two layers coincide under mean-field/ μPP, hence if one layer start to contribute more to the loss decrease at certain width, we may observe shift in the optimal learning rate that reflects the shift in the dominant mechanism (c.g.. for learning non- oon-oolnoomill multi-index models, training the second-layer parameters alone can achieve low loss only at exponential width). In such scenarios, we expect that this bimodal behavior can be remedied by the use of layer-wise learning rate.
This reasoning is empirically supported by Figure 17b where we divide the second-layer learning rate by 50. We observe that with the introduced multiplier, the learning rate curves now become roughly unimodal; moreover, the model achieves lower validation loss than the original tied learning rate setting (Figure 17a). We leave a more systematic investigation of layer-wise wise learning rate future work.
32

---

Published as a conference paper at ICLR 2026
E.2 LLAMA EXPERIMENTS
For our Llama experiments use the following Llama architecture.
/PP Base Dimension
128
Normalization
RMSNorm (prelayer norm)
Hidden Layers
4
Attention Heads
8
Feedforward Network
SwiGLU
Feedforward Dimension
4x Embedding Dimension
Rotary Embedding
0 = 10000
Table 2:
Llama architecture configuration used in all experiments.
We the following schedule for the EMA to preserve performance and linearization accuracy.
EMA Warmup We warm up the EMA decay coefficient ae linearly from Qstart = 0.98 to Oend 0.9995 over 2000 steps in the effective window
(4)) -1 To capture early-training variation without increasing linearization error, subsample the EMA trajectory every v steps, with
T itself warmed up linearly from Tstart 2 to Tend = over the same period.
E.2.1
LLAMA ADAM LR
Llama Adam Configuration
Below is the default setup for all Llama experiments using Adam.
Dataset
WikiText-103
Epochs
Optimizer
Adam (31 = 0.9, B2 0.999)
Batch Size
128
Context Length
128
LR Schedule
with 4% linear warmup 20% cooldown
LR Grid Search
LR (1.0, 1.4, 2.0, 288, 4.0, 4.8, 5.7, 6.7, 8.03 10-3
n E {128, 256, 512, 1024, 2048)
The loss L is evaluated the validation split. We average over 3 random seeds.
E.2.2 LLAMA ADAM 31 AND B2
We repeat the same experiments from Appendix E.2.1 for the Adam 3 hyperparameters, fixing I.R = 0.004. For the 31 experiment, we perform sweep over 31 with 8g = 0.999 fixed and for the 82 experiment we fix 31 = 0.9 and sweep over 32. The grids used for the sweeps are:
31 (0.63, 0.78, 0.86, 0.92, 0.95)
.2 (0.95, 0.98, 0.99, 0.995, 0.998, 0.9999)
The sweeps are performed in logspace in the effective window size (1 -
The analogs of Figures 6a, 8, 9 are presented for 3 in Figures 18a, 19, 20 respectively and for Bg in Figures 18b, 21, 22 respectively. We note that in this particular setting the performance is insensitive to 32 except when it is too small. The computed values R(n) shown in Figure 23 are fairly similar for all the hyperparameters, suggesting that the dimension of this invariant subspace may be mostly
33

---

Published as a conference paper at ICLR 2026
42
274
Lisorieed
Liseerized
8.4
Aw322
86
222
3.6
ne 1224
3.4
B:)
Adam 31 sweep.
(b) Adam B2 sweep.
Figure 18:
Same Llama and WikiText setup as Fig. 6, but sweeping Adam 81 and 32. Left: Smaller 31 values increase the lincarization error. Right: The lincarization error small for all 32. For large enough B3 the loss is near-optimal and transfer nearly perfoct up harmless fluctuations in the optimal 32.
data and architecture dependent. It will be interesting to perform similar experiments for other HPs such as weight decay and our decomposition viewpoint can shed insight onto HPs which do not show fast transfer.
Total
Top-x
-6.25
6650
n56
256
Aooss
-6.5
n=1024
n-312
1=1004
n-512
120
-7.25
2
2
1024
(1 8.)
0,)
Figure 19: Left: Total loss curves du widths for Adam 3,. Right: Corresponding top-K., losses dh" (blue dashed) and 6x (purple dashed), similar Figure 8.
Residual
Residual
%
47%
-02
Loss
-0.4
n= 512
1024
512
1024
-0.8
-1.0
21
24
2
Learning Rate
Learning
Figure 20: Residual losses around the top-Na minimizers for Adam 31, which are nearly flat as in Figure 9.
34

---

Published as a conference paper at ICLR 2026
Total
Top-K
6.4
On
-6.6
256
256
3Toss
-6.8
512
512
2048
-72
-14
0- 512
2
2
2
2
2-
2
2
2
-- 1024
11 83)
(1
--)-
Figure 21: Left: Total loss curves on, across widths for Adam S2. Right: Corresponding top-Kn. losses $ (blue dashed) and 6% (purple dashed), similar to Figure 8.
Residual
Residual
0.0
n- 128
128
-0.2
256
256
AToss
-0.4
-0.6
-0.8
2
2
2
2
2
2
2
2
2
Learning Rate
Learning Rate
Figure 22: Residual losses around the top-Kn minimizers for Adam B2, which nearly flat in Figure 9.
k(n)
R(n)
k(n)
600
128
500
256
n= 512
400
1024
300
200
100
212
2- Learingg Rate
2-0
2'
24
2
2
23
27
2/2
235
Bi)
(1 - Be)
Figure 23: Computed values of ((n) for sweeps over the Adam LR, 83, and B2.
35

---

Published as a conference paper at ICLR 2026
E.2.3 LLAMA MUON
As in Appendix 1,2.1, we train a Llyma-style transformer architecture with a warmup stable (WSD) learning rate schedule WikiText- 103, but using the Muon [JJB+24] optimizer (see Appendix B.2) instead of Adam. The training configuration is shown below. The learning rate sweeps are shown in Figure 1la.
Llama Muon Configuration.
Below is the setup used for the Muon training.
Dataset
WikiText-103
Epochs
Hidden layers
Optimizer.
Muon (3 = 0.95, Adam LR= 0.004, 31 0.9, A2 = 0.999)
Newton-Schulz Iterations
Batch Size
128
LR Schedule
WSD with 4% linear warmup & 20% cooldown
LR Grid Search
LR {0.1, 0.2, 0.4, 0.57, 0.8, 1.1, 1.6, 10-2
v. E (128, 256, 512, 1024, 2048}
E.3 LLAMA DION
As in Appendix E.2.3, we train Llama-style transformer with warmup stable (WSD) learning rate schedule on WikiText-103. but using the Dion [AXA +25] optimizer (see Appendix B.3). The training configuration is shown below. We consider two different configurations of the Dion rank hyperparameter In the bounded rank setting we set = min 1288, n/2) and in the proportional rank setting we take
n/2. The learning rate sweeps for the bounded rank setting shown in Figure 24. As we can see, the transfer is nearly perfect. On the other hand, in the proportional rank setting shown in Figure 25, we see the same imperfect transfer that we saw for Muon, but also that the performance large width benefits from larger From Figure 26a, rank setting, the de curves look more closer to the Adam profile in Figure 10. In particular, for (1024, 2048) the curves nearly overlapping. However, there is still not the strong top-k invariance that we saw with Adam which holds across all widths, suggesting that there may different notion of invariance for Dion. In contrast, Figure 26b, we that the of curves strongly resemble those of Muon in Figure 12. It is an interesting future work to pin down a more appropriate notion of invariance and understand what causes the imperfect transfer for Muon and proportional rank Dion.
that in the bounded
Dataset
WikiText-103
Epochs
1
Hidden layers
4
Optimizer
Dion (μ 0.95, Adam LR= 0.004, 81 = 0.9, 3, 0.999)
Newton-Schulz Iterations
5
Batch Size
128
LR Schedule
WSD with 4% linear warmup 20% cooldown
LR {0.1, 0.2, 0.4, 0.57, 1.1, 1.9} x 10
E { (128, 256, 512, 1024, 2048)
36

---

Published as a conference paper at ICLR 2026
4.2
4.4
1:23 20:7
3.235 25755
__ (223)
4.3
2040
Unserited
121
3.8
3.6
n
3.6
34
Leamming Rate
Width
EMA loss (solid) and linearized loss (dashed).
(b) Scaling law for the EMA loss.
Figure 24: Same model and dataset Fig. 6, but trained with Dion using bounded rank min n).
Left: EMA and linearized losses coincide. Right: EMA loss versus width for the learning rate choices v/ (n) [blue dots] and v² (128) [orange triangles]. The transfer nearly perfect contrast with Muon (Fig. 11) and Dion with proportional rank r = rt/2 (Fig. 25).
42
EMA
n11225
0.4
E 1018
LASS - 100 26.02
0.2
2-3112
4.0
4.0-
2.5
Unserited
121
3.8
3.6
3.6
1024
2045
3.4
Leaminng Rate
Width
(a) EMA loss (solid) and linearized loss (dashed).
(b) Scaling law for the EMA loss.
Figure 25: Same model and dataset as Fig. 6, but trained with Dion using proportional rank n/2. Left: EMA and linearized losses coincide. Right: EMA loss versus width n for the learning rate choices 2 (n) [blue dots] and v" (128) [orange triangles]. The transfer imperfect, similar to Muon (Fig. 11) and in contrast with Dion with bounded rank r = min( 128, r2/2) (Fig. 24).
0,908
0.008
-2
0-2424
2048
2-45
-3
-6-
850
1200
450
(a) 6 for bounded rank min (128, n/2).
(b) 0i for proportional rank r n2/2.
Figure 26: Left:
The top-k loss curves of more invariant in the bounded rank setting, but still appear qualitatively different from Adam and SGD. Right: The top-k loss curves di in the proportional rank setting look qualitatively similar to those for Muon (Fig. 12).
37

---

Published as a conference paper at ICLR 2026
E.4 GPT-2 EXPERIMENTS
For our GPT-2 experiments the following architecture with trainable position embeddings.
μdP Base Dimension
128
Normalization
RMSNom (prelayer norm)
Hidden Layers
4
Attention Heads
8
Feedforward Network
GeLU
Feedforward Dimension
4x Embedding Dimension
Table 3: GPT-2 architecture configuration used all experiments.
EMA Warmup We warm up the EMA decay coefficient 0r linearly from Ostart 0.995 to Gend = 0.996 over 2000 steps the effective window (1-a,) We subsample the EMA trajectory every steps, with itself warmed up linearly from Tstart = 1 to Tend over the same period.
E.4.1 GPT-2 ADAM LR
Similar to the experiments in Section 4.2, we investigate transfer of the Adam LR, but using the GPT-2 architecture in Table 3 and the FineWeb dataset.
GPT-2 Adam Configuration
Below is the setup for GPT-2 experiments using Adam.
Dataset
FineWeb
Tokens
IB
Optimizer
Adam (81 0.9, A3 0.95)
Batch Size
256
Context Length
1024
IR Schedule
WSD with 4% linear warmup & 20% cooldown
LR Grid Search
LR {0.3, 0.6, 0.8, 1.2, 1.7, 2.4, 3.4) x 10-2
n E {128, 256, 512, 1024, 2048]
The loss f is evaluated the validation split. We average runs 3 random seeds.
The analogs of Figures 6, 9 are presented for our GPT-2 setup in Figures 27, 29, 30 respectively. As observed in the Llama experiments, we can observe the faithfulness of the linearization and nearly perfect transfer in Figure 27. In Figures 29 and 30 see that top-k invariance and residual flatness hold approximately Although the decomposition is not as clean as was for the Llama architecture, the trends still suggestive and provide qualitative support for overall conjecture. In particular, we see essentially the same qualitative profile of o%: in Figure 28 which we saw for Figure 10. Interestingly however, we see that k(n) is much smaller in this setting. Another difference we observe is that the top-k invariance seems to break down for small learning rates and small widths in the GPT-2 setting, which did not appear be the case for the Llama architecture. We note that the conditions for our fast transfer conjecture only require invariance to hold for nearly optimal HPs, and we speculate that invariance can indeed break for suboptimal HPs. We defer more quantitative evaluation of this phenomenon to future work.
38

---

Published as a conference paper at ICLR 2026
4.6
1.50
1/ 3,051 11
L-2067
22.07
(- (RR10 (22(2))
LA00
2024
2043
4.2
4.00
Unwerboed
nm
128
3.75
322
3.0
21x
24
Learning Rate
(a) EMA loss (solid) and linearized loss (dashed).
(b) Scaling law for the EMA loss.
Figure 27: Training 4-yayer GPT-2 transformer on FineWebl IB with Adam. Left: EMA and linearized losses nearly coincide. Right: EMA loss across widths under the width-dependent optimal learning rate v (n) [blue dots] and the fixed width- 128 choice v* (128) [orange triangles] shows near perrept transfer across widths.
Rin)
LR = 0.008
200
175
n-512
--1444
-04
125
100
75
25
200
400
800
1000
1200
Leamiing Rate
Figure 28: Left: Computed values of ((n) using Algorithm 1. Right: Top-k losses d for LR 0.008. The curves descend rapidly with and overlap different n for an intermediate range of k where top-k invariance holds. The profile is qualitatively similar Figure
Total
Top-K
128
-6.0
n1512
512
ALoss
n=1024
1024
-6.5
2040(a)
-7.0
128
256
1024
2-11
Learning Rabe
Learning Rate
Figure 29: Left:
Total loss curves oh across widths for GPT-2 experiments with Adam. Right: Corresponding top-x,, losses th (blue dashed) and dxe (purple dashed), similar to Figure 8.
Residual
Residual
-025
$ KkK
$ -Kx
n= 128
n= 120
0.50
256
256
512
n= 1024
n= 3024
-100
-125
2
2-1
2
--
Learning Rate
Learning Rate
Figure 30: Residual losses around the top-k, minimizers for GPT-2 with Adam, which are nearly flat as in Eggur 9, "GPT-2 MUON LR
Next we perform a learning rate sweep for the Muon optimizer (Appendix B.2) in our GPT-2 setup (Table 3).
GPT-2 Muon Configuration Below is the setup for GPT-2 experiments using Muon.
LR Grid Search
39

---

Published as a conference paper at ICLR 2026
Dataset
FineWeh
Tokens
IB
Optimizer
Muon (Adam LR = 0.01, 91 = =09,, 82 = 0.95)
Newton-Schulz Iterations
5
Batch Size
256
Context Length
1024
LR Schedule
WSD with 4% linear warmup & 20% cooldown
LR {0.3, 0.6, 1.2, 2.0, 4.0, 6.0, 12.0} 10-2
n E {128, 256, 512, 1024, 2048}
The loss C is evaluated on the validation split. We average runs over 3 random seeds.
The result of the sweep is shown in Figure which the analog of Figure 11. We see that although the optimal learning rate seems to shift non-trivially (Figure 31)), the Muon optimizer produces "flat" hyperparameter curves so that suboptimal learning rate does not affect the transfer performance much (Figure 3 1b). However, it is still possible that the transfer suboptimality will become more visible for larger widths.
In Figure 32, we plot the top-k loss profiles o for LR E (0.001, 0.012). We see that for the smaller learning rate the profile looks very similar to Figure 12. Interestingly however, for the larger learning rate, which is closer to the optimal learning rate, we see that the profile looks a lot closer to that observed when using the Adam optimizer. One potential explanation is that the dynamics might be more heavily dominated by the layers which use the Adam optimizer for larger learning rates. is possible that this is linked with the stable Muon transfer observed in this setting as opposed to worse transfer performance observed in Figure la. More careful investigation required to disentangle these effects, potentially by only training the layers which undergo the orthogonalized updates and ablating the number of Newton-Schulle iterations.
4.5
4.50
4.4
2 go (2122
4.2
n=512
1074
4.2
2040
4.0
3.8
3.75
325
Leaming
Width
(a) EMA loss (solid) and linearized loss (dashed).
(b) Scaling law for the EMA loss.
Figure 31: Same model and dataset as Fig. 27, but trained with Muon. Although the optimal learning fluctuates across width, the insensitivity to learning rate causes transfer to be only slightly suboptimal for the given widths. Left: EMA and linearized losses nearly coincide. Right: EMA loss across widths using the two learning rate choices v² (n) [blue dots] and //" (128) [orange triangles).
40

---

Published as a conference paper at ICLR 2026
LR 0.001
LR 0.012
3.5
n= 256
-.00
1024
2048
-5.0
-4-
-6
6.5
-7
200
400
600
800
1000
1200
200 400
600
800
1000
1200
k
Figure 32: The qualitative behavior the top-k losses df, learning rate dependent for GPT-2 trained with Muon. Left: Top-k losses di, for LR = 0.001. The profile looks similar to the one observed for Muon the Llama architecture in Figure 12 since top-k invariance only holds for small ic and the descent is slow especially for large n. Right: Top-A losses 6% for LR. 0.012. The profile looks similar those observed from the Adam optimizer (see Figures 10 and 28) since the descent fast, there is approximate top-k invariance, and the tail flattens out.
E.5 CIFAR-I0 MLP SGD
We also probe the generality of our observations to different dataset, optimizer, and architecture: CIFAR-10 training using SGD on 2-layer MLP with ReLU activation and no biases.
CIFAR-10 Training Configuration
Below is the setup used for our CIFAR-10 experiment.
Dataset
CIFAR-10
Epochs
100
Layers
Optimizer
Momentum SGD (3 009)
Batch Size
512
LR Schedule
WSD with 4% lincar warmup & 20% cooldown
Data Augmentation
Mixup, Random Resized Cropping
We the same EMA warmup and discretization schedule detailed in Appendix E.2. We largest width mmax = 8192 as the infinite-width proxy for computing ((n) using Algorithm I.
LR Grid Search
I.R (1.0, 1.4, 2.0, 2.8, 4.0, 4.8, 5.7, 6.7, 8.0} 10-3
n E {128, 256, 512, 1024, 2048, 4096, 8192]
In the right panel of Figure we see that the majority of the loss decrease occurs the first few components, and in the left panel we see that the computed values of k(n) are much smaller than the width n. When we apply our decomposition using the computed values of k(n) in Figures 34 and 35, see the same qualitative picture holds in the transformer Adam experiments (for example see Figures and for Llama trained on WikiText-1033 103).
41

---

Published as a conference paper at ICLR 2026
kin)
LR 0.016
22
50
0.5
256
-0.6
1096
-07
5332
30
-10
23
2
200 400
600
800
1000
1200
Learning Rate
Figure 33: Two-layer MLP trained on CIFAR-10 10 using SGD. Left: Computed values of A(n), with ratios /n/n much smaller than in the language setting (see Figure 10). Right: Top-k losses & descend with and flatten much more quickly than in Figure 10, indicating more low-rank structure in this setting.
Total
Top-K
0.80
128
1-1
0.85
0-21024
ALoss
0.90
(192
1024
4096
1.000
2
2-
2
Learning Rate
Learning Rate
Figure 34: Left: Total loss curves widths for the two-layer MLP trained with SGD. Right: Corresponding top-K.. losses of (blue dashed) and @%x (purple dashed), similar to Figure 8.
Residual
Residual
0.000
$
128
-0.025
4096
AToss
-0.050
-0.075
-0.100
2-
2
2
Learning Rate
Learning Rate
Figure 35: Residual losses around the top-x,, minimizers for two-layer MLP trained with SGD, which approximately flat relative to curvature the top-N,, loss, as Figure 9.
42


Reviews:
[{"review_id": "Rp4hzR2B7T", "reviewer": "Reviewer_ssQX", "summary": "Scaling-aware hyperparameters propose hyperparameter tuning with a small-scale model (e.g., small width/depth) and then directly use the tuned hyperparameter (after modification with certain rules) for training a large-scale model. Recently, Maximal Update Parameter ($\\mu P$) proposed a theoretical framework for scaling-aware hyperparameter tuning when scaling model width, but in the mean-field regime only. However, such fast-hyperparameter transfer phenomenon also works with finite width/data as well, but there is no prior explanation/justification for this. This is the main motivation of this work. \n\nTo explain fast hyperparameter transfer in width-scaling, the authors propose a new decomposition of the optimization trajectory into two components: (1) a component that converges quickly with model width and determines the optimal hyperparameters, and (2) a component that improves the loss when the width is increased but has a negligible impact on HP choice. The authors then hypothesize that this one might explain the hyperparameter transfer with width scaling, and then validate their finding with empirical experiments.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well-written and easy to follow. I hardly see any typos. The motivation is well-presented and very clear. Though I am not really an expert in this field, I can follow the paper very easily. I think it is a good paper.", "weaknesses": "First things first, since I am not an expert in this field, my opinion might be biased and not fairly evaluate the contribution of this paper. \n\n1. One might argue that the paper only focuses on fast hyperparameter transfer in terms of width. In practice, one might want to know about fast hyperparameter transfer when scaling the depth instead. However, I would not consider it an issue, since it is the weakness of the Tensor Program series in general. Besides, I think that the setting is good enough, since we can simply fix the depth, train with minimum width, and it should be fast enough. \n\nSince I am not an expert, I can only give support for this paper with a low confidence score. I will leave the decision to the AC and other expert reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UouiwrHz21", "reviewer": "Reviewer_WG5j", "summary": "The paper explores the gap in understanding of what determines the fast convergence rate of zero-shot hyperparameter transfer at the infinite-width limit using $\\mu$Parameterization, as seen empirically. \nThe authors present a novel decomposition of the loss trajectory through a linearization scheme over an EMA of the original trajectory. \nThey show that the sum of the top-k eigenvalues of the gradient matrix captures the loss change in the top-k directions of maximum change in loss.\nThis measure can be seen to be invariant across width-scaling, whereas the other residual components appear to reflect scaling with width.\nThe authors posit that this indicates the existence of a subspace of the trajectory (as captured by EMA smoothing), such that tuning for those top-k components at a width $n$ can guarantee fast hyperparameter transfer as $n \\rightarrow  \n\\infty$.", "questions": "Below are the enumerated questions and suggestions.\n\nPlease note that the rating will be increased contingent on the points below being addressed/clarified.\n\n1\\. The authors could consider explaining in brief the comparative scaling of the hyperparameter value compared to the performance metric with width, as the meaning of \"fast transfer\", in the abstract.\n\n2\\. L60-63: The authors could consider referring to the relevant sections.\n\n3\\. L116: In comparison to previous literature, is one of the contribution of this paper, to show that the top-k eigenvalues of the gradient matrix can adequately capture invariance across appropriate width-scaling?\n\n4\\. L172:173: Does this specifically apply to only certain types of hyperparameters? Or only the abcd-parameterization parameters that are scaled in width?\n\n5\\. L266-269: Could the authors explain why this happens and its implications?\n\n6\\. L255-266: *Top-$\\kappa$ strong convexity*, here both variables appear same, $\\phi_{n}^{\\kappa_n}$ . If $\\kappa(\\nu)$ is an integer related to width-$n$, then what does the super- and sub-script denote?\n\n7\\. Figure 4.b): Can this be explained more please? Especially the role or choice of `Step 7185`. What is the blue dot line, in $LR(n) = \\nu^{*}(n)$?\n\n8\\. Figure 7 (right): Could different colours be used here?\n\n9\\. L460-464: Is flatness of the loss residual the issue here or the loss gap at $n \\rightarrow \\infty$?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Novel framework and formulation to try and explain the practical gains of an important theoretical finding in $\\mu$Parameterization.", "weaknesses": "* Hard to follow the notations in Section 4, especially with the incremental evolution of $\\phi$ and its usage.\n\n* Hard to see the practical implications of the finding, especially with the requirement of adjusting $k$ for any new task. Unless the only goal was to *explain* why fast HP transfer shows up in practice, the claim in abstract for validating this for LLMs might need to be revised.\n\n* The understanding of the *main* paper on its own is a bit hard, with adequate parsing of the Appendix details quite crucial to sometimes connect points made in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pmCGhSOL5z", "reviewer": "Reviewer_PzsK", "summary": "The work introduces a conceptual framework to study the transfer properties of hyperparameters of neural networks from low to large scales. A decomposition of the loss suggests a connection of certain optimization statistics to the convergence properties of optimal hyperparameter settings.", "questions": "* The authors focus on a very specific choice of hyperparameters. They state \"however our framework can be used more broadly for reasoning about scale-aware HPs\", could you elaborate on that? Where are the limitations of your framework?\n\n* What scale in terms of # of trainable parameters do your width choices correspond to?\n\nMinor comments\n* Numbering all equations would be good to facilitate discussions\n* Axis in Figures descriptions are very small", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Understanding the mechanisms behind hyperparameter transfers from low to large scales is an important issue to study as it can inform future algorithmic directions\n* Assumptions are clearly stated\n* Existing literature on analysis of hyperparameter transfer across scales and related analysis methods is well covered", "weaknesses": "* Unclear if there are any direct practical implications of the introduced conceptual framework\n* In line with previous work, the conceptual framework targets a very specific choice of hyperparameters and optimization settings.\n* Code to reproduce the experiments is not provided, although some details on the trainings and fixed hyperaparamter settings are provided. What are the specifics of the \"Llama-style\" transformer architecture you used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Br6Gktp5O9", "reviewer": "Reviewer_VXBW", "summary": "The paper develops a formal framework for understanding fast hyperparameter transfer. Fast HP transfer means that hyperparameters tuned on a smaller model (less width) transfer to bigger models (more width). They give theoretical reasoning why this works, for example, in muP. They decompose the loss into top-k components and residual loss. The top-k components determine the optimal HPs and converge quickly across widths, which explains the fast HP transfer. This is validated through experiments on transformers and MLPs.", "questions": "* How is k for top-k chosen in practice? What are the implications of it?\n* What effect would a negative change in loss have?\n* For which other HPs other than learning rate does your framework hold? Is your framework also applicable to other scale factors like depth, data, or number of training epochs?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "In summary:\n\n* The paper is well-written and nicely put into the context of the related work. \n* The framework of HP transfer is novel and provides interesting novel insights and theory on the topic.\n* The idea of the loss decomposition seems to be new as well and quite useful, possibly also for other applications. \n* The paper gives theoretical and empirical reasoning for why the HP transfer works.\n* They also verify their theory on real-world models in their experiments, next to analytical ones.\n* Understanding HP transfer is relevant given the trend to very large models, which can be tuned more computationally efficiently with HP transfer.", "weaknesses": "* It would be nice if the authors introduced the concept of muP in the introduction, since your paper is based on it. For someone not being super familiar with the topic, it needed quite a bit of effort to get into the topic.\n* Multi-Fidelity optimization is also a type of HP transfer. I would appreciate a discussion on the relation to multi-fidelity as a fairly established approach.\n* Please double-check for introducing abbreviations before using them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)