Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
Authors: Bruno Kacper Mlodozeniec
Pierre Ablin
Louis Béthune
Dan Busbridge
Michal Klein
Jason Ramapuram
marco cuturi
Venue: ICLR 2026 Poster (2026)
Keywords: muP
tensor programs
hyperparameter optimization
hyperparameter transfer
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A careful study of hyperparameter transfer (e.g. LR), including per-tensor hyperparameters, when tuned first on a small model and extended to larger scales.

Abstract: Hyperparameter tuning can dramatically impact training stability of large-scale models.
Recent works on neural network parameterisations, such as  μP, have shown that layer types and sizes should dictate how global hyperparameters should be rescaled in order to achieve efficient transfer across model sizes.
On the other hand, the established practice for hyperparameter optimisation search is to look for optimal global base values that apply at some fixed model scale.
We transfer hyperparameters across all scaling axes: width and depth, using an extension of CompleteP (Dey et al., 2025), training horizon, and batch size.
Our study covers all optimisation hyperparameters of modern models: learning rates, Adam parameters, weight decay, initialisation scales, and residual block multipliers.
Lastly, we demonstrate that hyperparameter transfer holds even in the per-layer hyperparameter regime.
We characterise the empirical challenges of navigating the high-dimensional hyperparameter landscape, and propose practical guidelines for tackling this optimisation problem.
We suggest a simplified parameterisation of the hyperparameter space that reduces the dimensionality of the search-space at no performance cost.
Our experiments demonstrate training speed improvements when applying transferred hyperparameters to Large Language Models.

Published as a conference paper at ICLR 2026
COMPLETED HYPERPARAMETER TRANSFER ACROSS MODULES, WIDTH, DEPTH, BATCH & DURATION
Bruno Mlodozeniee" University of Cambridge
Pierre Ablin Apple
Louis Bethune Apple
Dan Busbridge Apple
Michal Klein Apple
Jason Ramapuram Apple
Marco Cuturi
Apple
ABSTRACT
Hyperparameter tuning can dramatically impact training stability and final perfor- mance large-scale models. Recent works on neural network parumeterisations, such as μP, have enabled transfer of optimal global hyperparameters across model sizes. These works propose an empirical practice of search for optimal global base hyperparameters
small model size, and transfer to large size. We tend these works in two key ways. To handle scaling along most important scaling axes, propose the Complete Parameterisation that unifies scaling in width & depth - using an adaptation of CompleteP (Dey et al., 2025) - as well as in butch-size and training duration. Secondly, with our parameterisation, vestigate per-module hyperparameter optimisation and transfer. We characterise the empirical challenges of navigating the high- nigh-dimensional hyperparameter land- scape, and propose practical guidelines for tackling this optimisation problem. We demonstrate that, with the right parameterisation, hyperparameter transfer holds even in the per-module hyperparameter regime. Our study covers an extensive range of optimisation hyperparameters of modern models: learning rates, Adam W parameters, weight decay, initialisation scales, and residual block multipliers. Our experiments demonstrate significant training speed improvements in Large Lan- guage Models with the transferred per- module hyperparameters.
in-
50M HP search
50M training run
7.2B training run
3.6
Optimal global
Optimal por-module
3.5
Scaling
2.2
Final loss
3.4
Best global HPs loss
2.1
3.3
3.2
2.0
2.31 31 x speedup
1.32x speedup
3.1
0h
27h
55h
83h
0B
1B
2B
3B
4B
100B 150B 200B 250B
Runtime
Tokens
Tokens
Figure 1: (Left): We optimise hyperparameters at small 50M parameters/1.66 tokens scale (lear- ing rate, initialisation scale, Adam E, 18,, B2 and weight decay) with an evolutionary strategy. These hyperparameters (HPs) can be either optimised globally (one value across the entire model), or per- module (13 module types, some tuned per depth). (Middle): At the 50M scale, the mod-le HPs are 2.3x faster reach the same level of performance as the global HPs. (Right): Our new (dmplete Parameterisation enables direct transfer (without any subsequent tuning) to x0000 larger FL.P budget. While we can transfer both optimal global and per-module HPs, the per-module HPs retain 1.32x speed-up and improved benchmark performance (see Figure 9) at 7.2B scale.
WWrk done Apple.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The remarkable success of large transformer-based models (Vaswani al., 2017) has been driven by scaling up model size and data (Kaplan et al., 2020), However, to get the most out of these large- scale training runs, or to even successfully complete them all, a multitude of hyperparameters (HPs), such as learning rates, weight decay, or initialization scales, must be carefully set.
Parameterisation. To mitigate this HP tuning cost, recent works have introduced principled param- eterisations - such as the μ paraamtteristtion (nP) (Yang et al., 2022) that enable the transfer of hyperparameters from smaller, cheaper-to-train models to their large-scale counterparts. Effec- tively, these parameterisations automatically adapt an HP for any tensor, depending on its type and the change in width. This process has been extended (Yang et al., 2024) and further investigated for width and depth transfer in transformers with Com- pleteP (Dey et al., 2025). These methods have been demonstrated to successfully transfer optimal global hyperparameters like the learning rate. However, two questions remain open:
handle changes in depth with Depth-puP
Per-module HPs: Can tuning HPs on a per-module basis give significant gains, and do the per- module HPs empirically transfer with the right parameterisationn Given the significant performance improvements from well-optimised global HPs, it is natural to consider optimising them on a finer- grained scale well. When scaling up model with JP, Depth-zuP Completel, different tensors will receive different HPs depending on their architectural role - for instance, the learning rates for the embedding layers have be scaled differently from those for the hidden weights. It is therefore reasonable to expect that different modules, or tensors, could benefit from independent hyperparameter tuning. Put differently, there is little reason to believe the optimal por-module HPs should all collapse to the same value at some base width at which we optimise them.
How should we parameterise training for scaling beyond just model size? pP and extensions are only derived for transfer in model width & depth. However, model size is only one of the scaling axes utilised for training better models. Scaling data and batch-size (the latter larger corpus) is also critical, but the HP transfer will
Scaling modalities:
often being necessary to facilitate training not hold along these axes if relying solely on μP (Sections 2.2 and 2.3).
In this work, we systematically investigate the transfer of per-module hyperparameters across var- ious scaling modalities. Per-module hyperparameters introduce a scaling challenge: tuning per- module HPs basis creates explosion of the dimensionality of the search space, making it truly intractable at a large scale. We demonstrate that a practical methodology of optimising at the small-scale, and using an HP-transfer-enabling parameterisations works. We perform the expensive, high-dimensional search for optimal per-module HPs on a small proxy model, and demonstrate the transfer of the optimal HPs a large target model. Our contributions are:
Complete( for width & depth transfer. We refine the CompleteP parameterisation from Dey et al. (2025), extending it to modern Transformer components like Query-Key Normalisa- tion (Henry et al., 2020). We further identify and rectify minor issues the original formula- tion. We illustrate the resulting parameterisation permits robust hyperparameter transfer for all eioorttiallly-mooivatt variants of depth scaling (a E []).
() for transfer new scaling axes. We systematically study transfer beyond model size, including in token horizon and batch size. We unify our adaptation to CompleteP with batch- size and token horizon scaling rules. For batch-size scaling, we adapt the SDE reparameterisation in (Malladi et al., 2022) AdamW, extending the prescription to scaling of weight-decay. We further propose a new SDE-inspired scaling rule for transfer in the token horizon.
Per-module HP transfer. We empirically demonstrate that hyperparameler transfer holds for per- module hyperparameters with the right parameterisation. Optimising per-module hyperparameters at a small scale yields significant training speed-ups that persist after transfer to a larger scale.
A practical recipe to find por-module HPs. We empirically characterise the per-module hyper- parameter optimisation landscape. We highlight its challenging nature, marked by sharp "cliffs" where training diverges. These characteristics make it highly inefficient to use random search and can prove challenging for default Bayesian Optimisation. We show that the landscape is close to "invex", and that local search strategies work well.
2

---

Published as a conference paper at ICLR 2026
Width Transfer
Depth Transfer (a= Depth Transfer (a 1)
Final Loss
4.0
Depth
Depth
Width
3.5
1024
24
24
15336
64
64
2048
128
128
3.0
Miai 3,050
mmm
mm
mm
crinng
10
10-3
10-2
10-1
10-9
10-3
10-2
10-1
10-4
10
10
10-1
Learning Rate
Figure 2: Hyperparameter transfer for global learning rate across depth and width. Each setting is run with three independent seeds.
2 HYPERPARAMETER TRANSFER
In this section, describe the hyperparameter transfer modalities we consider, and the principles that we follow to adjust HPs while varying other aspects of the training configuration. We first de- scribe hyperparameter transfer in model size (width and depth) in Section 2.1, where introduce variant the CompleteP parameterisation (Dey et al., 2025). ). In Section 2.2, we describe principles consider hyperparameter trans-
follow for hyperparameter transfer across batch-size. Lastly, fer in the number of training tokens in Section 2.3, illustrating that optimal HPs do not transfer out of the box across token horizons. For an extended discussion of related work, see Appendix E.
Experimental Setup All experiments conducted using decoder-only transformer model (Radford et al., 2019; Phuong & Hutter, 2022) on the RedPajama dataset (Weher et al., 2024). We modern transformer variant with pre-nomaalisation Query-Key norms (Henry et al., 2020), trained with a mixture of cross-entropy and Z-loss (de Brébisson & Vincent, 2016). We always train with cosine schedule. As performance metric, we always report the final validation loss on the pre-trane ---trrining data, which is strong indicator of downstream performance (Hoffmann et al., 2022; Chen al., 2025). For remaining training and architecture details, see Appendix D.
2.1 HYPERPARAMETER TRANSFER ACROSS MODEL SIZE
The core idea underlying hyperparameter transfer across models of different sizes is view finite- size models discretisations of the infinite-sizi limit. Intuitively, two models of different sizes that are both sufficiently close to the same infinite limit will behave similarly. they are sufficiently close over set of considered hyperparameters, then they should share similar optimal hyperparameters.
The challenge is that, depending on the parameterisation - i.e. the rules for adjusting the hy- perparameters as function of size we can obtain different infinite width depth limits with fundamentally different behaviours (Yang & Hu, 202 Most these limits are pathological in var- ious ways. For instance, the Standard Parameterisation (SP) the features blowing up with size, whereas the Neural Tangent Parameterisation (Jacot et al., 2018) results in lack of feature learning (Yang & Hu, 2021). μP was identified by Yang & Hu (2021) as the unique parameterisation for Stochastic Gradient Descent (and later for a broad class of adaptive algorithms (Yang& & Littwin, 2023)) that precludes the emergence of such pathologies scale.
et al., 2020) leads to
In this work, we build upon CompleteP (Dey et al., 2025), which itself an adaptation of Depth-uP (Yang al., 2024) transformers, which we make several adaptations. These new scaling rules, which we call Complete are summarised in Table 1. Firstly, we extend the parameterisation to Query-Key (QK) normalisation layers (Henry et al., 2020), which have become a staple modern transformer implementations ( (Yang et al., 2025; Debghani et al., 2023). The challenge of QK norms that, unlike any other component in transformers, these layers share weights across transformer heads. If scaling in width is performed by increasing the number of heads while keeping the head dimension fixed (as was done (Dey et al., 2025; Yang Hu, 2021)), then QK norms introduce
We disambiguate between hyperporameters (the settings want to find optimal values for), and the training configuration (the settings change facilitate scaling: data size, model size, hatch-size).
3

---

Published as a conference paper at ICLR 2026
Table Parameterisation Comparison as a function of width (mx),, depth (mz), batch size (mg) and token count/data size (m.) ratios. For Complete differences to CompleteP (Dey et al., 2025) for width & depth scaling are shown alongside in gray.
Parameterisation:
PPP (Table 3)
Complete
Multipliers
MHA Residual
MHABlock(x)
x + m wwmmmMMm MHABBlcc((x)
MLPP Residual
MLP (lock))
+ m "MLP (lockl))
Unemb. Fwd
Unaugmented
Unaugmented
[x(my))
Input Emb.
Init. Variances
Hidden weights
xm
Xmx
Hidden biases/norms
of
Unemb. LN
Unemb. Weights
xm
XMN
[xl
Input Emb.
Learning Rates
Hidden weights
xm
xm N
x m ma-1
Hidden biases/norm
Xm, Xm
X
m2
Unemb. LN
Unemb. weights
X
XMN
[X]]
Adam c
Hidden weights/biasse/normss
x X 11
xm
x mL
QK norms
bb
NA
X
X
M
Input Emb.
X
XmN
mB
Output weights/biasss/norms
Weight decay
Hidden weights
XmN
XMN
Unemb. weights
A
XmN
XMN
X
m2
mD
Rest
x1
x1
AdamW W (1 31)
(1 31,5)
X 123
AdamW (1 - 32)
(1 32,5)
X mB
Training iterations
m
weight-sharing across the scaled dimensions. This necessitates different scaling considerations than for regular normalisation layer multipliers or biases. The adjustments for Adam (Loshchilov & Hutter, 2017) shown in Table I, which justify in Appendix B.
Secondly, we note that Dey et al. (2025) mistakenly derived the wrong scaling for the Adam scaling for the input embedding. We justify our modification in the Appendix B. Although the resulting modification is minor, we found that the lack thereof was sufficient to break a thorough sweep of the coordinate checks deseribed by Yang al. (2022) in our implementation.
Lastly, we climinate the explicit scalar multiplier on the output of the final lincar projection by reparameterising its effect into the learning rate and initialisation scale. This enables easily incor- porating memory-efficien algorithms like Cut Cross-Entrop (Wijmans et al., 2025), which avoid materialising the full logit matrix, drastically reducing GPU memory requirements for modern large vocabulary models.
In Figure 2, we verify the HP transfer with Complete across width and depth. An important factor in Depth-uuP is the depth-dependent re- re-scaling factor for the residual connection in transformers (h? the output of layer f, Fg the function applied to it):
h f+1 he mz Fe(h"), LE (1,.....,1 L}
which is governed by single parameter a E [0.5, We make the following observation:
Cdmplete with and permits hyperparameter transfer across depth.
Our parameterisation seems to allow for HP transfer with all theoretically justified values of a E [1. This is contrast to the findings of Dey et al. (2025) who notice a degradation of transfer for 0.5. The added QK norms in our implementation improve stability (see Figure 13
4

---

Published as a conference paper at ICLR 2026
for comparison without); however, removing them does not lead to the breakdown of transfer ported by Dey et al. (2025). We note that in their publicly-releasee reference implementation, they apply the same AdamW's to all weights (including embeddings), against their own recommenda- tion. Interestingly, the optimal loss is slightly better for the largest model for a = 1. potentially suggesting that the theoretical arguments for this parameterisation on the basis of feature diversity (Yang et al., 2024) might be beneficial in a language transformer context.
2.2 HYPERPARAMETER TRANSFER ACROSS BATCH-SIZE
Model size and dataset size are two levers to achieve lower loss increasing each predictably leads to model improvements, as implied by scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). To avoid the training time blowing out of proportions, one typically wants to make use of extensive parallelisation, such as that controlled by the batch-size. However, the set of practically usable batch-sizes is often heavily constrained by the memory footprint of model of a given size, and the specific parallel compute architecture available for training. In practice, the availability of compute can often change from month to month or week to week, requiring adjustments. Furthermore, these requirements might be different when tuning hyperparameters vs. when scaling up. For smaller hyperparameter sweep runs, a smaller batch-size is often desirable to reduce the per-run memory footprint and enable running parallel HP search. When training larger model, one typically wants to adjust the batch-size to make use of all available compute for a single run. Unfortunately, as with scaling model size or training duration, hyperparameters do not transfer across batch-sizes without further reparameterisation. In this work, we transfer hyperparameters across batch-size via a similar limiting argument as for transfer across model size. In particular, we follow the principles for batch- size transfer laid out in Malladi et al. (2022), and extend them to weight-decay in AdamW.
Training as discretising an SDE. The goal of the SDE reparameterisation approach to parame- terise hyperparameters such that training trajectories remain approximately invariant to batch-size changes when measured in "normalised time" (proportional to data observed). However, exact in- variance is generally impossible. We therefore aim for invariance in the limit of infinitely small "effective" step sizes (step sizes in normalised time), where the discrete optimizer iterates (c.g., Adam SGD) converge continuous stochastic process (often described by a Stochastic Dif- ferential Equation (SDE) (Malladi et al., 2022)). By identifying the parameterisation that ensures a consistent and sensible SDE limit, we can derive reparameterisation rules where the optimizer acts as a stable discretisation of the same underlying process. Hyperparameters then transfer effectively across batch-sizes, provided the "effective step size" remains small enough for the discretisation to remain valid.
Adam weight-decay reparameterisation. We extend the parameterisation of Malladi et al. (2022), which yields a consistent SDE limit for Adam, to handle the weight-decap in AdamW. To informally motivate the weight-decay scaling rule, we consider the same simplifying RMSPropW example as in Malladi et al. (2022). We consider that the gradients queried at each step & are a noisy version of a fixed direction g* = 9 + oe*; where e* are i.i.d. Gaussian vectors of identity covariance. We note that as we move towards the small step size limit - i.e. as the batch-size gets smaller and smaller- - the mini-batch gradient will be dominated by noise- - o >> |[g||. Contrary to Malladi et al. (2022), we also consider weight decay term as in AdamW. We denote the learning rate, and À the weight decay. We obtain the simplified RMSProp iterations (see (Malladi et al., 2022, Sec. 4.1) for more details) that define the iterates 0 (k:; 17, X, σ) with the equation
g + A0 ((
k k++
0²
0 (g + XOoO) ne"
which is a discretization of the SDE dO, = no (g + Ao + dW, with step-size 1/² in the sense that 0² ~ Oxp². Therefore, we find that the multipliers to keep fixed iterate distributions, i.e., such that 0 (,,; mg- In particular, if the batch-size is multiplied by к, we have m, = x - 1/2 and we find that the new hyperparameters matching the SDE limit should follow the square-root scaling rule
Ommkk; m,M myd, m,)) should verify mam? = 1, m,, = 1, and my
r¹ Vin. k' kk, and X' VRA
(2)
To the best of our knowledge, we are the first to extend the SDE reparametrisation scaling rules (Li et al., 2019; Malladi et al., 2022) to the weight decay of A, although we note that the
5

---

Published as a conference paper at ICLR 2026
4.5
4.5
Training Loss
Batch Size
Training Loss
Batch Size x
4
2 x
Base
4.0
Base
4.0
x
3.5
3.5
10-4
10-3
10-2
10-1
10-4
10-3
10-2
10-1
Global LR
Learning Rate
Figure 3: Learning rate transfer with batch-size. Left: Learning rates transfer when using the square-root rule in Equation (2) Right: Learning rates fail to transfer without adjustment. Each setting is run with three independent seeds.
same scaling rule for weight decay proposed based on other principles in recent work (Wang & Aitchison, 2025; Compagnoni et al., 2025; Bergsma et al., 2025). We report the effect of using these scaling rules in batch-size in Figure using the square-root rule critical good LR transfer. We further show in Figure 4 that the above rule is essential for transfer of weight-decay in batch-size.
AdamLH and multipliers Equation (I) mirrors the Pytorch implementation of AdamW, where the weight decay X is multiplied by the learning rate If one instead uses the original AdamW implementation, often coined AdamLH, proposed by Loshchilov & Hutter (2017). get the simplified iterations g = l 7 ^0 and we find that the multipliers are the same as for Adam W, except that my = m doubling the batch-size means that the weight decay should now be doubled. Hence, using A leads a bigger drift across batch-sizes the scaling is not done correctly, amplifying further the drift observed in Figure 3, right. We posit that this is one of the reasons why the Pytorch implementation is more widely used.
Standard
Standard
SDE Adjustment Rule
3.7
Training Loss
Batch Size
Batch Size
Batch Size
3.6
x
4
2
x
2
3.5
3.4
3.3
100
10
10
Weight Decay
Effective Weight Decay Weight Decay
Figure 4: Weight decay transfer with batch-size. Left: Weight decay fails to transfer with batch- size without any adjustments. Middle: The rescaled (effective) weight decay A/VR where the increase does transfer. Right: The effective weight decay transfers when rescaling all hyperparam- eters following our AdamW SDE scaling rule, Each setting run with three independent seeds.
2.3 HYPERPARAMETER TRANSFER IN TRAINING DURATION
Unlike transfer in model size or batch-size, transfer in the token horizon has received comparatively less attention in the literature. Nonetheless, is one of the two main levers to scaling compute. Like Bjorck et al. (2025), we observe that the optimal learning rate decays with the number of training iterations, holding all other things constant.
Optimal learning rate decay with token horizon. In Figure we notice that the optimal learning rate decays at rate roughly proportional to 1. the number of training iterations. In Figure (right), we plot the optimal learning rate from the shortest training duration (Mop)) transferred with the scaling rule Reps in gray, and observe that it /R aligns almost perfectly with the true optima. In contrast, Bjorck et al. (2025) fit scaling law to
/ where is the factor by which we' ve increased

---

Published as a conference paper at ICLR 2026
Training Loss
Tokens
3.50
0.8B
3.25
1.6B
8B
3.00
16B
2.75
mm
FIIS
10
10
10-3
10
10
100
10
10
10
10
10
10¹
Global LR
Effective LR (n x vtokens)
Figure 5: Learning rate transfer across training coorizonnajjustinn the number of tokens by changing the number of training iterations while holding batch-size constant. Left: Break-down of transfer of the global learning rate. Right: Stability of the "effective" learning rate one that preserves the AdamW SDE integration horizon. Each setting is run with three independent seeds.
Final Loss
42M params
Complete with SDE scaling CompleteP
3.50
Final FinalLoss Loss Final Loss
3.5
238M params
743M params
3.00
Scaling
Complete SDE: 2.233 $15283
2.75
CompleteP: 2.482 *** ge
Number of tokens
Figure 6: Scaling law comparison of models trained with and without Complete token horizon sealing rule. We compare Complete width depth scaling only (aka CompleteP with and QK-nomn fixes in Section 2.1) and full Complete with SDE scaling rules for token horizon transfer. The token horizon transfer rule leads to better performance at scale, as indicated by a better lower bound coefficient of the scaling law.
find the exponents 3 for the scaling rule Negt: and they identify 3 be in the ranges of 0.3 0.7 depending on the model size, which appears congruent with our scaling rule.
SDE iso-horizon scaling rule. In light of the SDE interpretation in Section 2.2, scaling the learning rate by VR while holding the batch-size constant can be seen as reducing the signal-to-noise (SNR) ratio in the SDE, while keeping the time horizon constant. Indeed, we orthogonally observe that when simulating the AdamW SDE, improving the signal-to-noise ratio (i.c. reducing the size of the diffusion coefficient) while holding other parameters constant consistently leads to improved performance. Hence, we hypothesise that the right way to scale the token horizon might be only adjusting the signal-to-noise parameter in the AdamW W SDE, while keeping all other terms constant. We validate this hypothesis in Figure 14, where we scale the number of tokens by increasing batch- observe near-
size only (which has the desired effect of changing the signal-to-noise ratio); perfect learning rate transfer across the token horizon. We further observe that the resulting scaling rule function of the token horizon leads better asymptotic performance, as predicted by scaling law, as we demonstrate in Figure 6.
The SDE iso- iso-horizon scaling rule empirically permits transfer across training horizon.
7

---

Published as a conference paper at ICLR 2026
Optimisation Horizon (Tokens per Parameter)
Learning rate
11
Final Loss
3.50
10-4
3.25
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Tokens
1e9
Figure 7: Best Learning Rate annealing over 4,842 runs for five different token horizons. The best schedule at short horizon is never a prefix for the best schedule at longer horizon. The optimal schedule cannot be found by greedy approach: the best LR annealing is not data agnostic.
We expect this transfer to break at larger batch-sizes, where the discretisation will be too coarse for AdamW W to approximate the underlying SDE, but when taken together with the batch-size reparam- eterisation rules in Section 2.2, this finding suggests how to scale all HPss across token horizons, while choosing the batch-size freely. This is the token horizon scaling procedure follow in all the HP transfer results the remainder of this paper, unless otherwise stated. We note that this finding might be specific the fixed (cosine) schedule that
Best Learning Rate (LR) annealing different token horizons. Optimal schedules might have different shapes at different token horizons (Luo et al., 2025). We conduct a greedy search to de- termine optimal learning rate schedules in the following way. We enumerate all the non-increasing piecewise-constane LR schedule over the discrete set (0.0015 1115//2.5/0 S h I 4}. We sub-divide the total training duration in 16 intervals of 77M tokens each. At the end of each interval, either the LR remains constant, either it is decayed by one more steps. For five different token horizons, we report the best scheduling among the 4,842 tested. We report the results in Figure We notice that the best scheduling at short horizon is never prefix of the best scheduling at long horizon. This em- pirical observation compatible with the findings of Luo et al. (2025): there is a tension between the optimisation bias induced by the terminal LR value (the lower the better) and the progress of optimisation which requires higher LR values at start.
INVESTIGATING TRANSFER OF PER-MODULE HYPERPARAMETERS
Equipped with the tools for hyperparameter transfer described in the preceding section, this sec- tion we investigate 1) how much there is to gain from per- per-parameter hyperparameter optimisation, and 2) how well do per-module hyperparameters transfer.
3.1 OPTIMISING PER PEDULE HYPERPARAMETERS
To show improvements and transfer of per-module hyperparameters, we need a good way to optimise them at a fixed scale. Although there is ample literature on hyperparameter optimisation in deep learning (Snock et al., 2012; Maclaurin et al., 2015; Lorraine et al., 2020), optimising HPs on a per- module basis introduces many new difficulties. Below, we highlight why many standard approaches fail in the per- parameter optimisation setting.
Per-module hyperparameter loss landscape In Figure 8, we plot slices through the per-module learning rate (LR) loss landscape - i.c. the landscape of the mapping from LRs to the final loss. We observe that, fortuitously, it's pretty close to being invex (stationary points are global minima), and hence might be tractable even despite its high dimensionality. Several other aspects, however, render challenging for common HP optimisation methods: 1) The values of per-module learning rates at which training becomes unstable are module-dependent, and can differ by multiple orders of magnitude. 2) The boundary at which training becomes unstable has a complex shape, with non- trivial interactions among different modules, implying it's difficult to predict with simple predictive models (c.g. linear models or Gaussian Processes (Williams & Rasmussen, 2006)). Our observa- tion is similar that made by Sohl-Dickstein (2024) who observed the stable regime boundary
8

---

Published as a conference paper at ICLR 2026
Layer-1 weightssh
Layer-1 wrVgttights
Layer-1 weights
Layer-5 weVittthhss
Layer-1 MLP-waights
Layer-5 OKV-eeights
Layer-5 MeP-weighhs
Layer-5 M..P-wiights
Final Loss
10-1
16-1
10
10-1
5
10
10
10"1
10"
Figure 8: The boundary for stable training has a complex shape. Each plot shows the final training loss for different combination learning rates for the modules indicated, while fixing the remaining learning rates to the optimal "global" value. MLP weights refer to the MLP in the attention layer. If unstable training results in NaNs, the last stable training loss is reported.
is fractal - but we also note a lack of emergent simple structure at the macro scale. This means common hyperparameter optimisation strategies, like random search or standard Bayesian Optimisation, struggle in this regime. For instance, random search lacks any locality bias; we ob- serve that without careful manual tuning of the search boundaries, either all runs will fail due to unstable training, the boundaries will fail to include the actual optimum. Commonly, Bayesian Optimisation relies on Gaussian Process (GP) models to guide search. However, GPs may struggle on highly non-stationn stationary data (Snoek al., 2014; Rana et al., 2017) with a fixed kernel. We did observe such irregularities in the per-module HP to loss landscape. which resulted in many failures when using BO. Many of these difficulties can be alleviated by more robust 'trust region' methods approaches that optimise in neighbourhoods of previous good solutions. We describe a simple trust-region random search variant that we use for our experiments in Appendix C.
Parameterising per-module hyperparameters We adopt depth-type Kronecker parameterisa- tion of per-module hyperparameters that is compatible with Complete transfer across width and depth. Let m M index module type within Transformer block (QKV weights, attention projec- tions, MLP weights and biases, layer norm and QK-norm multipliers). and let l E 1,... I. index (1), (2), c for module m at depth &
the depth. For a hyperparameter Sg.e E n7 2,
log Sm,e (T, N, B) = .log Sm ctype + log edepta (L) + log SDE B) + log Cmm (N. 22
(3)
type
depth
SDF batch/horizon
CompleteP*
where (TE((T: B) carries all training- -horiin T and batch-size B dependence via the AdamW SDE transfer rules, CF (N. L) is the Complete(d) scaling rule adjustment in width and depth (Table 1), and ctype, <depsh for m E M, E L are dimensionless, time-invarian multipliers that we optimise given width and depth. This factorisation reduces the number of free multipliers from [M|L to [M + L. We optimise the hyperparameter multipliers in log-space using trust-region random search (Appendix C). For the learning rate, the above multiplier post-multiplies the learning rate from the cosine schedule. When transferring the depth multipliers <- edepsh (L) to larger depth L', we interpolate them linearly with respect to f. This reasonable, as the depth-SDE (a = })) or depth- ODE limits (a ($. 1)) should still exist if the base hyperparameters (|ti.) (L) vary continuously with sufficient regularity across depth [0, 1). In this sense, the finite-depth multipliers can be seen as a discretisation of the continuous limit HPs: To transfer to large depth we simply linearly interpolate all HPs in depth.
Smi
3.2 PER-MODULE HYPERPARAMETERS MATTER
Depth multipliers matter We ablated away the effect of the learning rate per-depth multipliers, by instead considering only a search over learning rate multipliers for each layer role: Within each residual block, every parameter group gets an independent learning rate, which is shared across different residual blocks; similarly, each parameter group in the embedding and unembedding layers gets its own value. We initiate this search from a projection of the best por-module hyperparameters onto this linear subspace. In Figure 16c, we observe that the search value, although still substantially better than the best global learning rate, is worse than the that includes per-depth multipliers.
9

---

Published as a conference paper at ICLR 2026
Hence, while the majority of the gain comes from different module types within residual blocks getting different learning rates, there is still notable benefit to per-depth multipliers.
How restrictive is the depth-Kroneecke factorisation? To check how much performance we're leaving on the table with the Kroneker factorisation constraint, we continue searching for fully uncoupled per-layer learning rates from the optimal Kronecker-factorised ones. The search results are shown in Figure 16b. Crucially, we observe virtually no improvement over the Kronecker- factorised ones. While this is not conclusive evidence that the optimal per-module learning rates are factored - the fully uncoupled search-space is much higher-dimensional and more difficult to navigate, and its likely we didn't find the optimum these runs imply that most of the benefits of per mer-medule HP optimisation can be captured by Kronecker factorised learning rates.
3.3 OPTIMAL PER-MODULE HYPERPARAMETERS TRANSFER WITH SCALE
Demonstrating upsides of per-module HP optimisation would be of little practical use if the HPs have to be tuned at the target model scale. In this section, show that the improvements do persist across different model scales. Firstly, we demonstrate transfer in model size. Figure 11 illustrates that the optimal per-module learning rates transfer as we scale up both width and depth. Although we cannot easily visualise how the per-parameter HP loss landscape shifts as we vary model size (like was shown in Figure 2) due its high-dimensional nature, we instead show the final training losses for a slice (a hyperplane) going through both the scaled-up optimal per-module learning rates and the optimal global learning rate. We observe that this landscape appears stable with model size, suggesting that the optimal per-module LRs do transfer with width and depth.
Speed-ups from per-module HPs large scale. We also investigate what improvements are pos- sible when transferring per-module HPs to a compute-optimal model at a larger scale. We jointly optimise the per-module learning rate, weight-decay AdamW 31, 52, E and initialisation scale, and the residual block multipliers at the 50M parameter/11 1.B token scale. We initiate the search from the optimal global HPs. In Figure 1, we show that the transferred per-module HPs lead to 1 1.32x speed-up over the optimal global HPs in a x0000 x higher compute setting.
4 DISCUSSION & CONCLUSION
Limitations & Future Work Our study encompasses broad set of ablations, but there are limits to our empirical investigations and resulting recommendations, some of which we highlight below:
Although we identify improved per-module hyperparameters in reasonable number of trials, more trial-efficient search methods seem achievable. Exploring Bayesian Optimisation methods that address the specific difficulties of the per-module hyperparameter loss landscape highlighted in this work (e.g. Trust Region Bayesian Optimisation (Eriksson et al., 2019)), utilise early- stopis,, and exploit the structure of the training process (Lin et al., 2024), seem particularly promising.
We only evaluate one training setup (autoregressive transformer training on the RedPajama dataset). While that setup has broad practical relevance, our approach (the proposed transfer prin- ciples, especially for token horizon transfer, and benefits of per-module hyperparameters) should ideally be verified in other settings.
We only consider one fixed learning rate schedule (cosine decay) throughout. The optimal schedule for each token horizon might be very different to the recommendation in this work, and might improve upon fixed schedule.
The speed-ups of per-module hyperparameters we identified at a small scale seem to diminish slowly with model and data size. However, we do not know whether that is mostly to be explained by imperfect hyperparameter transfer in the non-asymptotic regime, or whether this is due to an asymptotic property of the infinite models. We hope future work might find computationally feasible ways of answering that question.
Conclusion In this paper, proposed new transfer rules for pypee-parameters, valid across the most important scaling axes: model's 's width, model's 's depth, token horizon, and batch size. Furthermore, these transfer rules also hold for por-medule optimisation at small scale with trust region methods produce a configuration that transfers to larger scales, and significantly improves training speed.
parameessss We demonstrate that systematic
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Power lines: Scaling laws for weight decay and batch size in I1m pre-training. arXiv preprint arXiv:2505. 13738, 2025.
Johan Bjorck. Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling op- timal across token horizons. In Y. Yue, A. Garg, N. Peng. F. Sha, and R. Yu (eds.), International Conference Representation Learning, volume 2025, pp. 2025. URL https: /ttp:////////eeee//// ngs selr. cc/ wwtp://aapppeeeeeeeeeeeeeeeeeeee cacaaper__leeee
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scal- ing laws for predicting downstream performance in LLMs. Transactions on Machine Learn- ing Research, 2025. ISSN 2835-8856. URL https:/ https///ppppeeeveeeee net forum?id Pid=
PJUbMDKOVY.
Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurelien Lucchi. Adaptive methods through the lens of sdes: Theoretical insights on the role of noise, 2025. URL https://arxiv org/ abs 241..1555.
Alexandre de Brébisson and Pascal Vincent. The z-loss: shift and scale invariant classification loss belonging to the spherical family, 2016. URL https: / / arxiv. org/ abs / /0604 08859.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa. Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International conference on machine learning, pp. 7480-7512. PMLR, 2023.
Nolan Dey, Bin Claire Zhang. Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don't be lazy: Completep enables compute-efficient deep transformers. arXiv preprint arXiv:2505.01618, 2025.
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems, 32, 2019.
Nikolaus Hansen. The ema evolution strategy: a comparing review. Towards a new evolutionary computation: Advances in the estimation of distribution algorithms, pp. 75-102, 2006.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers, 2020. URL https: htlLtttts://aaaee //axiiv.///////a/aaeaaaeataaev org/ 22122.11002222
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Au- relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Lau- rent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30016-30030. Curran Associates, Inc., 2022. URL https: / http::////ccceeddeeeeeeeoe neurips cc/ wwtpppppeeeaaeeeeeeeeeeeee lfrerr__lllee fww.ff/eeppeeeeeee /2022/ Paper- wwwpre-oooofeeeee.ccme .df.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URI. https: / /awxiv.. org/ abs 2at./200..000 08361.
Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20(40); 1-47, 2019. URL http: / // wwt.:oooooooo///./eeeeeeeeeeeeee
html.
11

---

Published as a conference paper at ICLR 2026
Jihao Andreas Lin, Sebastian Ament, Maximilian Balandat, and Eytan Bakshy. Scaling gaus- sian processes for learning curve prediction via latent kronecker structure. arXiv preprint arXiv:2410.09239, 2024.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiaaion. In Silvia Chiappa and Roberto Calandra (eds.). Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1540- 1552. PMLR, 26-28 Aug 2020. URL https: /ttss//prrocoo proceedings mlr. press /v
/100//111//////////1 0000/8/000 wwwraieeeeee0 html.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Jan Ludziejewski, Jan Matašnicki, Maciej Pióro, Michal Krutul, Kamil Ciebiera, Maciej Stefaniak, Jakub Krajewski, Piotr Sankowski, Marek Cygan, Kamil Adamczewski, and Sebastian Jaszczur. Decoupled relative learning rate schedules, 2025, URL https: // arxiv. org/ 2507
03526.
Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Maosong Sun, Zhiyuan Liu, Kaifeng Lyu, and Wenguang Chen. A multi-power law for loss curve prediction across learning rate schedules. In The Thirteenth International Conference Learning Representations, 2025. URL https:/ opwnreviee n forum? Kdd-KoSSXxIII..
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimiza- tion through reversible learning. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2113-2122, Lille, France, 07-09 Jul 2015. PMLR. URL https: //proceedings mlr prwpspssse..
html.
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35: 7697-7711, 2022.
Mary Phuong and Marcus Hutter. Formal arXiv:2207..09238. 2022.
algorithms for transformers. arXiv preprint
Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, and Atish Agarwala. Scaling collapse reveals universal dynamics compute-optimally trained neural networks. In Forty- second International Conference on Machine Learning, 2025. URL https:/ /ttp:////e//eeeeee/e//mee net/ bet//fose forumoormmmmm wwweiniidddiigggiiia
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9 2019.
Santu Rana, Cheng Li, Sunil Gupta, Vu Nguyen, and Svetha Venkatesh. High dimensional bayesian optimization with elastic gaussian process. In International conference on machine learning, pp- 2883-2891. PMLR, 2017.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Jasper Snoek, Kevin Swersky, Rich Zemel, and Ryan Adams. Input warping for bayesian optimiza- tion of non-stationary functions. In International conference on machine learning, pp. 1674- 1682. PMLR. 2014.
Jascha Sohl-Dickstein. The boundary of neural network trainability is fractal, 2024. URL https: / //arxivv.comm htwaariiw.org/bb////222 06184.
Jascha Sohl-Dickstein Roman Novak, Samuel S Schoenholz, and Jachoon Lee. On the infinite width limit of neural networks with a standard parameterization. arXiv preprint arXiv:2001.07301,
2020.
12

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL https:
org/abs/ 1706. 03762.
Jinbo Wang, Mingze Wang, Zhanpeng Zhou, Junchi Yan, Lei Wu, et al. The sharpness disparity principle in transformers for accelerating language model pre-training. In Forty-second Interna- tional Conference on Machine Learning, 2025.
Xi Wang and Laurence Aitchison. How to set adamw's 's weight decay as you scale model and dataset size. In Forty-second International Conference on Machine Learning, 2025. URL https: / //openeeview net/ dorummii-IIs
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Cha- lamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
Erik Wijmans, Brody Huval, Alexander Hertzberg, Vladlen Koltun, and Philipp Kraehenbuehl. Cut your losses large-vocabulary language models. The Thirteenth International Confer- ence on Learning Representations, 2025. URL https:/ http///ppeeeeeeeeeeeeeeeeeeeee.p wwwereeiiwwwweeeeeeeee.m catc/forunn forum?id= E4FK3YuG56.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0988:25505009888, 2025.
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural net- works. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 18-24 Jul 2021. URL https: https::/prroeeedngg mll ptw.ss/esss....................s v139/yang2lc. html.
Greg Yang and Etai Littwin. Tensor programs ivb: Adaptive optimization in the infinite-width limit, 2023. URL https: / arxii. www.iii..gg/aaaeeooooooooooeeeee atszgg/abb22300 01814.
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ry- der, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URL https: /tttpp:://rriii org/ /
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs VI: Feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representa- tions, 2024. URL https:://peenree net fortm? iideidd-111 1d177/VDDwww...
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Hyperparameter transfer
2.1
Hyperparameter transfer across model size
2.2
Hyperparameter transfer across batch-size
2.3
Hyperparameter transfer in training duration
3
Investigating transfer of per-module hyperparameters
3.1
Optimising per-module hyperparameters
3.2
Per-module hyperparameters matter
3.3 Optimal per-module hyperparameters transfer with scale
10
Discussion & Conclusion
10
Additional experiments & figures
15
A.1
Suboptimality of learning rates identified at too small scales
15
Motivation of the Complete adjustments
B.1 QK norm multiplier weights
21
B.2 Embedding layer AdamW c....
................................
B.3 Changes to the unembedding weight
21
Per-module hyperparameter search algorithm
22
Experimental details
22
D.I Best Learning Rate (LR) annealing at different token horizons
22
D.2 Baseline global hyperparameter tuning
23
D.3 Per-module hyperparameter search and transfer (Figure 1)
23
Extended related work
23
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTS & FIGURES
0.8
Global Optimal Per-module optimal
0.7
0.6
0.5
Score
0.4
0.3
0.2
0.1
0.0
Piqa
Hellaswag
Winogrande
Arc Easy
Lambada Standard
Arc Challenge
MMLU
Figure 9: Final down-stream benchmark performance for the models trained with the opti- mal per-module hyperparameters and the optimal global hyperparameters, both transferred with Complete for the 7.2B parameter/1493 143B token training runs from Figure 1.
Performance Delta (%)
4
2
Global Optimal (Ref.)
-2
Per-module Optimal
MMLU
Winogrande
Piqa
Hellaswag
Arc Challenge
Lambada Standard
Arc Easy
Figure 10: Final down-stream benchmark performance for the model trained with the optimal per- module hyperparameters for the 7.2B parameter/1453 143B token training runs from Figure
A.1 SUBOPTIMALITY OF LEARNING RATES IDENTIFIED AT TOO SMALL SCALES
As hyperparameter transfer in width & depth with Complete P motivated by the asymptotic width & depth behavior, would expect it start degrading at smaller scales. Indeed, this can
15

---

Published as a conference paper at ICLR 2026
Optimal per-module LRs
3.30
50M parameters
3.25
Final Loss
3.15
100M parameters
3.10
2.95
Optimal global LR
500M parameters
2.90
0.2
0.8
Interpolation Coefficient
Transfer across model scale of the optimal per-module learning rates. We interpolate between the optimal global learning rate multiplier, and the optimal per-module multipliers across models of different scales, and show that the optimal per-module multipliers 1) consistently improve upon the global multiplier baseline, and 2) remain close optimal the hyperplane spanned by the optimal global and local multipliers. Each setting is run with three independent seeds.
Figure 11:
Best learning rate at each scale
at 483M scale
2.99
0.010
Final loss
2.98
0.008
0.006
2.97
0.004
2.96
58M
136M
245M
483M
Model Size (#parameters) at which optimal learning rate is identified
Figure 12: Suboptimality of learning rates optimised at smaller scales. We plot the final loss on 483M parameter model (y-axis) when training with the learning optimal for a smaller model (x-axis). To optimise the (global) learning rate at each of the shown smaller scales, conduct grid search with the following set of candidates: {10-4 3.3x,, 10 5.6 x 10 10 x,3,3x, 10 10 5.6 10 -3 10 3.3x, 10 10 3.3x, 10-1). We Complete throughout. At the 136M scale, the optimal learning rate is already approximately stabilised. At the 50M scale, we incur small penalty. The final losses averages over 3 seeds.
be empirically observed with learning rate transfer in, e.g., Figure 2. Hence, there is trade-off when optimising hyperparameters with Complete transfer; going to smaller model sizes enables cheaper hyperparameter optimisation, but these hyperparameters could be slightly suboptimal at scale due to degraded hyperparameter transfer.
We illustrate this trade-off in Figure 12, where show the final loss of larger-scale model (483M) when transferring optimal hyperparameters (global learning rate) from a smaller model with dp. We that when transferring the optimal learning rate from smaller 58M model, we incur small transfer penalty. Optimising the learning rate at the 136M scale or larger seems incur virtually penalty. This suggests might have been able obtain more competitive per-module hyperparameters, at substantially higher compute cost, were we to conduct our search the 136M scale. We consider an exciting direction for future work empirically investigate at what scales hyperparameters should be optimised, and subsequently transferred, for maximum compute savings. Nonetheless, this will of course depend on the hyperparameter search method used.
16

---

Published as a conference paper at ICLR 2026
Depth Transfer (a = 0.5)
Depth Transfer (a 1.0)
Training Loss
Training Loss
x 10P
4 x 10
Depth
Depth
8
12
24
64
64
128
128
3 10
10
1700g
10
2
10
10"
10
10
10
10
10
Learning Rate
Learning Rate
(a) With QK-norms
Depth Transfer (a 0.5)
Depth Transfer (a 1.0)
Depth
Depth
Training Loss
4 100
8
Training Loss
4 x 100
12
12
64
64
Min: 3.245
Min: 3.264
3 10
3 10
10
Learning Rate
Learning Rate
(b) Without QK-norms
Figure 13: The effect of QK-norms on hyperparameter transfer of the global learning rate across depth with two variants of Complete with 1).
250M parameters
0.4B tokens
Training Loss
4.0
0.8B tokens
1.6B tokens
3,2B tokens
6.4B tokens
3.5
3.0
10-5
10-4
10-3
10-2
10-1
10"
Learning Rate
Figure 14: Learning rate transfer across token horizon when scaling up by increasing batch-size while holding training iterations constant. This scaling rule can be seen as improving the gradient signal-to-noise (SNR) ratio in the discretised Adam SDE (Malladi et al., 2022), while holding all the other SDE parameters and the integration horizon fixed.
17

---

Published as a conference paper at ICLR 2026
50M parameters
250M parameters
lokens 4B tokens
tokens
Training Loss
Training Loss
.6B tokens
4B
16B tokens
8B tokens
16B tokens
3.5
3.5
3.0
3.0
10-3
10-4
10
10-2
10
10
10
10
10-3
10
10-1
10º
Global LR
Global LR
500M parameters
0.80 B.B tokens
Training Loss
4.0
1.68 tokens
4B tokens
8B tokens
16B tokens
3.5
3.0
10-4
10-3
10-2
10-1
Global LR
Figure 15: Lack of learning rate transfer across training horizons increasing token horizon through number of iterations with fixed batch-size for different model sizes. The square-root transfer rule for the optimal learning rate identified the smallest token horizon for each model size is plotted in
18

---

Published as a conference paper at ICLR 2026
3.4 x 10
3000
Cumulative min. loss
3.38 x 10
Final trial loss
Running mean (100 trials)
3.36 10
Best grid search loss
Total Runs
3.34 x 10
3.3 10
1000
3.28 10
3.26
3.24 10
0:00
13:53
27:46
41:40
55:33
69:26
83:20
97:13
Runtime (hours:minutes)
(a) Hyperparameter search for the per-module learning rate multipliers parameterised with the Hepth--roneede factorisation.
3.32 10
3.31 x 10
Cumulative min. loss
Final trial loss
Running mean (100 trials)
600
3.3 x 10"
Total Runs
Best grid search loss
3.29 x 10
Best Kronocker-factorised LR multipliers loss
400
3.28
3.27
200
3.26
3.25
0:00
2:46
5:33
8:20
11:06
13:53
16:40
19:26
Runtime (hours:minutes)
(b) Hyperparameter search for the per-module learning rate multipliers with fully uncoupled multipliers. The search is initialised with the optimal HPs found in the search for optimal Heeth-Kroneed factorised multipliers in Figure 16a.
10
1000
Cumulative min. loss
Final trial loss
3.31 10
Running (100 trials)
3.3 100
Total Runs
Best grid search loss
3.29 x 10
Best LR multipliers loss
600
3.28
3.27
400
3.26 10
200
3.25 10
0:00
5:33
11:06
16:40
22:13
27:46
33:20
Runtime (hours:minutes)
(c) Hyperparameter search for the per- module learning rate multipliers with no depth multipliers. he search initialised with the projection of onto the constraint set of the optimal HPs found in the search in Figure 16a.
Figure 16: Hyperparameter search results with Trust Region Random Search. Each dot indicates the final loss of single trial, and the lines indicate the training duration (start & end).
19

---

Published as a conference paper at ICLR 2026
3.4 x 10
2500
3.38
Cumulative min. Joss
Final trial loss
3.36 10
Running mean (100 trials)
2000
Total Runs
3.34
Best grid search loss
1500
3.32 x 10
3.3 109
10
3.26 109
3.24 x 10"
0:00
27:46
55:33
83:20
111:06
138:53
Runtime (hours:minutes)
Figure 17: Hyperparameter search for the per-module learning rate multipliers parameterised with the Kronecker factorisation with CMA-ES. C.f. Figure 16a for the comparable Trust-region Random Search results.
20

---

Published as a conference paper at ICLR 2026
B MOTIVATION OF THE !OMPLETE ADJUSTMENTS
Here, we give a justification for each of the modifications made in Complete in Table 1. These modifications primarily concern scaling the anfnite-widdh limit. We directly rely on the properties that μP parameterised neural networks are known to possess that were formally shown in (Yang et al., 2024; Yang & Littwin, 2023). Concretetely, we note that when scaling with juP, forward hidden-layer parevvactivssoons
expected to have entries of size (1) (as defined in (Yang & Hu, 2021, Definition N.2)) and the backpropagated gradients with respect to hidden (pre- )activations are expected to have entries of size (1)/N). Furthermore, the (pre- Jactivations and the back propagated (pre-) Jantvaactivttion gradients are expected to approach i.i.d. in the infinite-width limit.
B.1 QK NORM MULTIPLIER WEIGHTS
In standard implementation of QK norms, the elementwise affine operation x +> m @ x + b with multipliers m and bias b shared across the transformer heads. When scaling width by increasing the number of heads - as is common in most of the relevant model scale parameterisation literature (Dey et al., 2025; Yang et al., 2022) this effectively means that these parameters are shared across the scaled width dimension N. For instance, for a collection of query vectors q E R Nusuda X dhand we have that the normalised query elements ůij == myJi.jj + by all share the same parameters my, by E for = l...., Naeads, where Nhezds = ((N). We denote by q:j the R RaNisdd vector (B.. : = 1,..., Nhaas (á., respectively). By the results of Yang & Hu (2021); Yang & Littwin (2023), we have that for the juP parameterisation q. has entries of size ((1) throughout training- The loss gradients for any hidden (pre-) petevvactivation, are known to have entry size (1/NN), and so the post-normalised query activation gradients Sc will also be of size @ /N). The backpropagated TQ.,
gradient with respect the multiplier m, is:
Nusada
OC
Nasada
OC oq.jj..
N
oq.
N
1=
where the rescaled random variables
qi have entry size (1)) as N + co. Informally,
u9.3.
in Yang & Hu (2021), the random variables N proach i.i.d as N - oo. Hence the sum above has Strong Law of Large Numbers like behaviour, converging to the mean of the entrywise limit of N
ac q: for = I,..., Nneada were shown to ap-
84.5
ac
qı- As such, we effectively have that
the gradient with respect to the width-shared parameters also @(1) with width. The scale of the AdamW W should match the scale of the gradient (Yang & Littwin, 2023), and so we have that the AdamW W c parameter for the width-shared multipliers should also scaled as +(1) with width. A near-identical argument follows for the bias terms.
B.2 EMBEDDING LAYER ADAMW
The ((1/N) scaling with width N for the embedding layer AdamW follows from the observation that the gradients with respect to the embedding parameters have element size (1/NN) To see this, note that the gradients with respect to the output of the embedding layer are O1/NN), whereas inputs are obviously constant with width. Hence, it naturally follows that AdamW should be scaled as (11NN) match the scale of the gradient (Yang & Littwin, 2023).
B.3 CHANGES TO THE UNEMBEDDING WEIGHT
The changes to the unembedding weight scaling rules are mostly reparameterisation of the multiplier-based μP implementation in (Dey et al., 2025). Namely, for AdamW, a weight multi- plier m has the same effect throughout training (bar the finite-precision arithmetic effects) as: 1) multiplying the initialisation variance by m 2) multiplying the learning rate by m 3) and mul- tiplying the AdamW parameter by mN (Yang & Littwin, 2023). We re-parameterise with (1) and (2), but we don't change AdamW e as it appears to have been derived incorrectly in (Dey et al., 2025). To see this, note that with jP (the Table 3 variant without output layer multiplier), the
21

---

Published as a conference paper at ICLR 2026
gradients for the unembedding layer weights expected to have scale (1) with width N. Hence, to remain of the same scale, the output embedding weight should also have a matching scale of (1) (Yang & Littwin, 2023). After reparameterisation to m output layer multiplier as is done in CompleteP the would also have had to be scaled my to match the reparameterised gradients.
C PER-MODULE HYPERPARAMETE SEARCH ALGORITHM
As described in Section 3.1, standard random search is unsuitable for the task of optimising per- module hyperparameters. We make two minimal tweaks that make it into workable method. We induce an exploitation bias by turning it into a trust region method: we constrain the search-space adaptively to the neighbourhood {x R² ||x X; x Il S r) of the current best solution x at given iteration t. Hence, the bounds move with the best solution found so far. We optimise all parameter in the Log2-space, and sample uniformly from within the bounding box. Even with this modification, however, we found that this trust-region random search quickly plateaued with relatively high variance in the final loss values. Hence, to allow the algorithm to explore promising regions more thoroughly, also decay the size of the bounding region r if the loss doesn't 't improve after a certain number of trials.
For all experiments, unless stated otherwise, we instantiate the search with the bounding box size of 1 (meaning that at each iteration, we multiply the best solution found so far by 2 with x sampled uniformly from [-1, 1)), and decay size of the trust region r by 0.7 if no improvement is observed in 100 trials. We run the algorithm asynchronously with a maximum of 100 simultaneous trials.
The goal of this paper is not to identify the best HP optimisation strategy for this setting; merely want to find a workable one order to demonstrate potential for improvements from per-module HP search. Since the above tweaks borrow from the principles underlying many evolutionary search (ES) methods, we also wanted to directly compare to a strong ES baseline to check our method performs reasonably. In Figure 17, we compare CMA Evolutionary Search (CMA-ES) (Hansen, 2006) to the Trust-region Random Search described above (c.f. Figure 16). CMA-ES is not natively an asynchronous HP search strategy, so we make a minor modification: for population size P. we wait until at least P trials sampled from the current generation have finished running. At that point, there might be more than P new finished trials (left-over trials from the previous generations), so we update the CMA-ES state with P best trials only. In this instance, Trust-region Random Search outperforms this CMA-ES variant. This gives credence to our search method of choice being able to identify good per-module HPs in reasonable runtime. We hope that future work can explore alternative strategies that might be able to severely reduce the number of trials required to find good per-module HPs.
D EXPERIMENTAL DETAILS
D.1 BEST LEARNING RATE (LR) ANNEALING AT DIFFERENT TOKEN HORIZONS
We pretrain small GPT-2 model (121M parameters). We enumerate all the non-increasing piecewise-constant LR schedule over the discrete set (0.0015/2.5*
k k.ar). We sub-divide the total training duration in L intervals of 77M tokens each. At the end of each interval, either the LR remains constant, either it is decayed by one or more steps. We chose L 16 and kmaz 4, which yields a total of 4842 runs. For efficiency, we use the same checkpoint to warm start all runs sharing the same prefix in the L.R scheduling, which cut down the computational complexity of this naive enumeration from (II to (m..). Therefore, the total compute budget is kept under 7,000 A 100 GPUh. For five different token horizons (155M, 3 310M, 621M, 932M and 1.24B) we report the best scheduling among the 4,842 tested. We report the results in Figure 7. We notice that the best scheduling at short horizon is never a prefix of the best scheduling at long horizon. This empirical observation is compatible with the findings of Luo et al. (2025): there is tension between the optimisation bias induced by the terminal LR value (the lower the better) and the progress of optimisation which requires higher LR at start.
22

---

Published as a conference paper at ICLR 2026
D.2 BASELINE GLOBAL HYPERPARAMEINE TUNING
To establish baseline, we perform extensive random hyperparameter search consisting of 2048 trials. Each trial trains 50M parameter model (dmodel = 512, L. 4) for 1.64B tokens (33 to- kens/parameter) over a discrete search space defined by:
. LR E (1x10-1.0 3 10 4x 1 , 3x 10 1x 10- 2x 2,3x
10-2, 1 x 10-1)
Adam c E (1 x 10 14 10 1 10 1 10-1 3 10
4x 10 1x 10-3}
Adam 81 E (0.8, 0.85, 0.95, 0.999)
Adam $2 E (0.9, 0.95, 0.98, 0.99, 0.999}
Weight Decay E (1 10=1,1x 10-3, x 10-2,1111100 1 10 2x 10 1001, -
The results and hyperparameter sensitivities from this search are visualized in Figures 18b to 18d. The optimal configuration from this global search achieves validation negative log-likelihoon of 3.34 nats. This result is substantially higher than that achieved by our per-parameter search strategy. underscoring the advantage of discovering optimal configurations at small scale before upscaling with principled rules like Complete"
D.3 PER-MODULE HYPERPARAMETEE SEARCH AND TRANSFER (FIGURE
Per-module hyperparameter search For the per-module hyperparameter search, we ran the trust- region random search as described in Appendix C. We ran 100 trials (training runs) in parallel, with a total budget of 5000 trials. We randomly chose a different random seed (dictating the network initialisation and data order) for each trial.
We optimised AdamW learning rate, weight-decay, e, momenta 01 (1) and ag = (1- B2), as well as the standard deviation for the initialisation, with one hyperparameter (multiplier) per module type. For module types, treat each 'tensor' within transformer block an individual type (e.g. QK-norm multipliers, QKV weights, output projection weights, first feedforward layer weights, second feedforward layer weights, etc. would all be individual types): each tensor outside the transformer blocks are also individual module types (input embedding weight, output embedding multipliers, etc. would all be individual types).
weight, output embedding bias, output layer We also optimise the per-depth transformer residual block multipliers in the depth-type Kronecker parameterisation (there are two residual multipliers in each transformer block one for the attention block, one for the feedforward block). Altogether, that leads to 79 hyperparameters to optimise.
The hyperparameter search at small scale Figure took 6730 GPU-hours on NVIDIA A100s, although 99% of the loss gains over the optimal global hyperparameters were realised within the first 3168 GPU-hours.
E EXTENDED RELATED WORK
Hyperparameter transfer in width & depth. Our work directly builds upon, extends and com- bines many existing parameterisation for transfer across different modalities. For width transfer, we directly build the Tensor Programs (Yang & Hu, 2021; Yang et al., 2022) based derivations for the p-parameterisation, and the extensions to adaptive optimisers (Yang & Littwin, 2023). Although (Yang & Littwin, 2023) contains an exposition of all the theoretical tools required to derive the right parameterisation for virtually any neural network architecture, applying these tools is still non- trivial task. Yang et al. (2024) extended similar principles to find parameterisations in depth. Dey et al. (2025) adapted these principles to derive width & depth wranfer-enabling parameterisation specifically for transformer models. Although Dey et al. (2025) directly builds upon and uses virtu- ally the same principles (Yang & Littwin, 2023) and Yang et al. (2024), they do derive the right parameterisation for a broad range of hyperparameters (initialisation scales, learning rates, Adam W weight decay, AdamW W epsilon), unlike the original μP paper (Yang et al., 2022), and they derive complete set of rules specifically for transformer models. We directly build upon and extend Com- pleteP (Dey al., 2025) for width & depth transfer part of our parameterisation, making couple of
23

---

Published as a conference paper at ICLR 2026
Validation Cross Entropy
3.00
3.55
3.50
3,3363
Number of runs
3.45
3.40
3.30
THI T
3.8
4.0
Learning Rate
1 B2
Weight
Validation Cross Entropy
Decay
(a) Number of trials of baseline 50M model evalu- ation loss.
(b) Gaussian Process fit sensitivity around optimal
identified global hyperparameters.
11-16
Validation Cross Entropy
3.8
n=15
3.7
3.6
IN a=162
n=179
n-171
n=150
n=
0-244
n-214
m=211
n-214
n=206
a=345
3=2
R=3
n-277
n=278
n-229
u-339
n=226
n=2
a=263
n=263
n-306
n-248
159
3.5
1=1
178
3.4
0,0003
0.08044
0301
0,003
000s
001
002
6,3
0.1
io-25
1--10
3e Tode
40-18
1w07
0,08
0.000
0.9
0.05
0.88
0,00
0,500
0.001
0.01
0.1
0.2
Learning Rate
Adam Eps
Betal
Beta2
Weight Decay
(c) (Marginal) sensitivity of the baseline 50M model for core hyperparameters. We show the 25%-75% quan- tiles for the validation losses with all hyperparameters sampled independently at random, conditioned on the shown hyperparameter being certain value.
3.9
0.0001
3.896
3.536
3.492
3,408
3.403
3.396
3.393
3.399
3.389
3.454
Weight Decay
3.8
Cross Entropy
0.001
3,900
3.529
3.421
3,402
3 3.404
3.393
3.423
3,480
0.01
3 3,899
3.542
3.495
3.414
3.423
3.381
3.391
3.
3.
3.413
3.7
0.1
3.903
3.528
3.485
3.421
3.371
3.368
3.358
3.376
3.415
3.634
3.6
0.2
3.903
3.525
3.482
3.101
3.391
3.371
3.393
3.142
3.491
3.719
3.5
0.4
3.917
3.509
3.473
3.408
3.412
3.411
3.171
3.5417
3.585
3.836
3.4
0.0001
0.0003
0.0004
0.091
0,00
0.004
0.t
0.02
0.03
0.1
Learning Rate
(d) Weight decay learning rate for baseline 50M model.
Figure 18: Summary of the global hyperparameter sweep on a 50M parameter model for the base- line.
small modifications (extending to QK-norms and fixing minor mistakes in the derived rules)?. This constitutes part of the Complete(d) parameterisation.
"One component that could, in principle, receive the same scaling treatment is gradient clipping. Gradient clipping can in principle be cast within the Tensor Papgrams framework, but would require the modification

---

Published as a conference paper at ICLR 2026
Transfer in batch-size and SDE scaling rules. For the batch-size transfer rules for the Complete parameterisation, we also directly build upon prior work on SDE transfer rules (Li et al., 2019), including for Adam (Malladi et al., 2022). We extend the SDEs of Malladi et al. (2022) to Adam W, allowing us to propose a scaling rule for weight-decay with batch-size. We note that similar scal- ing rules for weight-decay have been recently proposed in other works (Wang & Aitchison, 2025; Bergsma et al., 2025) prior to ours. To our knowledge, we are the first ones to motivate them theo- retically with the principle of preserving the dynamics of an AdamW stochastic differential equation (SDE), integrating it with the transfer rules in batch-size for the other Adam W hyperparameters.
Our rules are compatible with those proposed by those proposed in (Wang & Aitchison, 2025; Bergsma et al., 2025). Wang & Aitchison (2025) posit how the product of weight- and learn- ing rate should scale as a function of the batch-size. In particular, they suggest that the learning rate (B) and the weight-decay X(B) should be scaled with batch-size B so as to keep TEXA- = 7>D con- stant. This rule doesn't I specify whether 7y or À should be adjusted, but only constrains their product. Substituting in rules for (B) and X(B) from Table - which dictate that 7BB) x vB and XBB x VB- of in Wang & Aitchison (2025). They are also more specific, dictating how the learning rate and weight-decay should each be adjusted individually.
B
see that Teaa O 1. Hence, the rules we put forward are compatible with those
Compagnoni et al. (2025) also propose an Adam W SDE, and suggest the same weight-decay scaling rule, but based on different principles. Whereas we argue for preserving the dynamics of the SDE (and, hence, approximately preserving the dynamics of discrete-time Adam W) similarly to Malladi al. (2022), Compagnoni et al. (2025) propose scaling rules to try and maintain an upper bound on the final training loss. Furthermore, their derived SDE is different from ours and that of Malladi et al. (2022), whereas ours is fully compatible with that of Malladi et al. (2022). We highlight that Malladi et al. (2022) made a compelling case for the need for reparametrising the AdamW momenta terms that led to their SDE. This reparametrisation is missing from (Compagnoni et al., 2025).
Token horizon transfer rules. To the best of our knowledge, we are the first to propose our scal- ing rule in token horizon for learning rate, weight-decay, AdamW momenta and AdamW e on the grounds of the proposed SDE principles. However, prior works have explored transfer in token horizon more broadly. (Bjorck et al., 2025) demonstrate that the optimal learning rate shifts with the token horizon, and propose empirically derived scaling rule. Qiu et al. (2025) observe that adequately normalised training curves transfer across model size and token horizon when both are scaled jointly remain compute optimal. Since they rescale by the final loss the exact quantity the behaviour of which we study - it's not immediately clear how those insights could be exploited for hyperparameter transfer. Nonetheless, their observations might be closely related, and their analysis through the lens of SDEs could prove to be fruitful avenue for explaining the transfer rules across token horizons we identify in this paper.
Per-module hyperparameter selection. Independently, (Ludziejewski et al., 2025) studied the benefits of setting hyperparameters differently for different parameter groups. They similarly show improvements over global hyperparameters at fixed model scale, and demonstrate improvements persist when training a larger mixture of experts model (8x compute) with each expert being the same size as the base model. Our analysis differs in few places: 1) we thoroughly investigate transfer across model scale for the same architecture with uP and pP-derived parameterisations, 2) investigate transfer across token horizons and batch-size, 3) we investigate more fine-grained hyperparameter transfer on per-module basis, whereas Ludziejewski et al. (2025) only consider separating the parameters into broad groups, and 4) we investigate hyperparameters beyond the learning rates (weight decay, Adam W momenta, etc.), whereas Ludziejewski et al. (2025) consider learning rates and parameters of their schedules.
Recently, Wang et al. (2025) motivate why per-module learning rates might be beneficial from a curvature perspective. They consider the relative scale for the learning rates for 5 sub-modules in transformer block; more specifically. the query-key (QK) and value-output (VO) blocks; wisnt feedforward networks (FFN); normalization layers (Norm), and embedding layers (Emb). Follow- ing their theoretical analysis, their practical recommendation is to tune manually (informed by that theory) these ratios on a smaller scale and re-used as is for larger scales (which would still require
to how the norm is computed - the norm of each tensor ought be scaled differently depending on the width hefore aggregating.
25

---

Published as a conference paper at ICLR 2026
tuning global LR when changing the compute scale). Compared to work, their practical ap- proach is restricted to a far smaller set HPs: LRs their work vs. approximately hundred HP in our case, since we study 6 fundamental HPss (learning rate, initialization scale, Adam E, S1, $2 and weight decay) times (number of modules plus depth). Additionally, their perspective does not stress transfer of these hyperparameters along any scaling axes, which is contribution in work. Instead, they retune the only hyperparameter they tune multipliers for (the learning rate) at every scale they consider, making costly to apply it scale.
26


Reviews:
[{"review_id": "LOoHmPt21i", "reviewer": "Reviewer_E2dm", "summary": "This work investigates the transfer of optimization hyperparameters of language models from a lower to larger scale. The key contribution is to optimize hyperparameters at the low scale on a per-module-type basis. The per-module optimization, empirically, leads to a speed-up of 27% over global optimization in the large scale training run.", "questions": "* What do the multiple solid red lines in Figure 1 represent?\n* What does \"optimal global\" refer to, e.g., in Figure 1? Is there a direct comparison to Depth-muP and CompleteP? \n* Evaluation is done with respect to final validation loss. Is this also being used to optimize the hyperparameters at the lower scale? If so (maybe additionally) showing results on hold-out dataset would be good.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "* How to set hyperparameters for large models is a crucial question, and investigating per-module hyperparameter optimization on the lower scale is an important idea.\n* 27% speedup in the large scale training is a substantial improvement and emprirical evidence of per-module HP transfer being useful is important to a larger audience.\n* The most related work is closely discussed throughout the paper and ideas well placed in the literature\n* The paper is mostly well written\n\nI did not closely check some of the more theoretical ideas in the paper and may be missing some strengths there.", "weaknesses": "* The paper makes a point about Bayesian optimization being an ill fit for the problem setting at many points, but does not include an actual empricial comparison.\n* Saying \"Our study covers all optimisation hyperparameters of modern models\" in the abstract is too bold of a statement. This already breaks down when other optimizers or learning rate schedules are considered, or, e.g., the choice of optimizer becomes a hyperparameter. I suggest making a slight adjustment here.\n* Experimental rigor could be improved by considering more seeds and quantifying measurement error, e.g., by showing error bars as the standard error around the mean.\n* \"Even with this modification, however, we found that this trust-region random search quickly plateaued with a relatively high variance in the final loss values.\" Showing these results in the paper would be beneficial.\n* How much optimization of your search method did you do with respect to your final evaluation metric (validation loss on the 1.3B model). Seeing how your method transfers to scales / models that you have not touched during method development would be good.\n* No code is provided to reproduce the experiments. Some details are provided in the Appendix, but I doubt I could reproduce the study.\n* The authors do not discuss the limitations of their work.\n* Figure 1 \"Best grid search loss\" is hard to read (even without a red-green blindness). More explanation in the caption would be helpful.\n* Figures should make it clear that validation loss is shown\n* I would like to see a comparison where only the parameterisation is changed and the search method etc. is kept the same.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2DK2Uiqi8", "reviewer": "Reviewer_DDMZ", "summary": "The authors propose Complete(d)P, a refinement of the CompleteP parameterization that extends to modern Transformer components like Query-Key normalization. The key contribution is demonstrating that hyperparameters optimized at a per-module level on small proxy models (50M parameters) can transfer successfully to larger models (1.3B parameters). The paper covers all major optimization hyperparameters, and authors also provide practical guidelines for optimizing the challenging high-dimensional per-module hyperparameter landscape.", "questions": "1) In Figure 1, what is the \"optimal global\" baseline? Is this the best global HP found through the random search?\n\n2) You mention that removing QK-norms doesn't completely break transfer (Figure 14), contrary to Dey et al. (2025). What implementation differences might explain this discrepancy?\n\n3) Can you provide error bars or results across multiple random seeds for the main speedup claim (Figure 1)?\n\n4) Feel free to address any of the weaknesses listed above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The extension to per-module hyperparameters represents a meaningful advancement beyond prior work on global hyperparameter transfer.\n\n2) Novel weight decay scaling rule for batch size (κ scaling) derived from SDE analysis is theoretically motivated and empirically validated\n\n3) Comprehensive experimental coverage across multiple scaling dimensions (width, depth, batch size, token horizon). Empirical results show a speedup improvement of up to 27%.", "weaknesses": "1) Model sizes are relatively limited - while the sizes of the models used in this paper are comparable to CompleteP, µP was evaluated on larger models. Addressing the question of scalability would have strengthened the guildelines proposed by the authors.\n\n2) The evaluation does not include the model's performance on downstream tasks. CompleteP, for example, includes this kind of evaluation. Given that this paper builds directly on said paper, not including this in the evaluation reduces the reader's ability to assess the approach's applicability.\n\n3) Evaluation is conducted only on a Transformer-decoder architecture, so transferability to other Transformer architectures (while certainly possible) is not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NmSUKXNWw2", "reviewer": "Reviewer_rHzn", "summary": "The paper investigates hyperparameter transfer for pre-training of large Transformer models as model size scales. The proposed approach builds upon μ-parameterization (μP) and its variants, such as Depth-μP, and extends these frameworks to incorporate more recent architectural components like QK-normalization. In addition, the authors broaden their analysis to cover the effects of scaling batch size and training token budget.\n\nThe work further examines per-layer hyperparameters, showing that optimizing them independently can yield additional performance improvements. Finally, the paper introduces a local search strategy designed to more effectively navigate the sharp cliffs present in the hyperparameter landscape.", "questions": "- Line 241: Where does the term \\eta * e^k originate from?\n- Line 244: What is \\Theta_\\eta2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Extending the state of the art (Depth-μP) to include more recent architectural advances such as QK-normalization makes the proposed hyperparameter transfer framework significantly more practical and up to date.\n\n- Addressing hyperparameter transfer across batch size and token budget is both sensible and novel, as this dimension of scaling has received limited attention in prior work.", "weaknesses": "- Clarity and notation: Certain sections of the paper are difficult to follow, with inconsistent or undefined symbols. For instance, the variable θ denotes model weights in Section 2 but appears to represent hyperparameters in Section 3. The second paragraph of Section 3.1 in particular is confusing and requires clearer explanations and consistent notation.\n\n- Questionable argument against existing HPO methods: The paper claims that established hyperparameter optimization techniques (e.g., Bayesian optimization) would not meaningfully work in this setting. However, this assertion is not convincingly supported. Even if the loss landscape contains cliffs, Bayesian methods can still locate high-performing regions without fully modeling these discontinuities exactly. This holds particularly for Bayesian optimization methods that do not use Gaussian processes. The authors should provide empirical evidence or an ablation study to substantiate this claim.\n\n- Limited downstream evaluation: The primary metric used is the final validation loss on pre-training data, which the authors argue correlates with downstream performance. While reasonable, showing results on actual downstream tasks (e.g., fine-tuning accuracy or benchmark performance) would strengthen the claim that the transferred hyperparameters provide practical benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aNyShZGFeI", "reviewer": "Reviewer_36xD", "summary": "This paper extends and refines existing parameterisation schemes for hyperparameter scaling, most notably CompleteP. The authors introduce/combine several ideas: a continuous-time analysis of SGD to derive scaling rules for the learning rate, weight decay, and other hyperparameters across batch size and token horizon; corrections to inconsistencies in the original completeP work (AdamW $\\eps$); per-module and per-depth hyperparameters. They empirically demonstrate that these refinements allow efficient hyperparameter transfer across scaling axes, including model size, batch size, and token count. Impressively, they do this with not only learning rate, but also Adam parameters, initialization scales, and multipliers.", "questions": "- Can you provide an approximate compute budget (GPU-days) for the per-module hyperparameter search? Section C.1 gives partial details but not the full scope.  \n- Do you believe these scaling ideas could extend to mid- or post-training adaptation (e.g. LoRA fine-tuning)? Would the same SDE-based scaling rules still hold?  \n- For practitioners pretraining Transformers from scratch, could you recommend a minimal set of “good defaults” derived from *Complete(d)P*, avoiding the need for exhaustive search?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Tackling hyperparameter transfer across many scaling axes (width, depth, batch size, and training horizon, etc.) is very ambitious. As far as I know, no prior work has attempted such comprehensive scaling experiments.  \n\nThe derivation of batch-size and weight-decay scaling from the SDE limit of AdamW is conceptually elegant. The pragmatic “trust-region random search” for finding the per-module multipliers $\\alpha_g$ and $\\delta_\\ell$ seem effective and fairly efficient, showing that the resulting configurations generalize well to large models.  \n\nThe experiments are extensive and to have combined several ideas regarding hyperparameter scaling make the paper overall very impressive.", "weaknesses": "The paper would benefit from a concise quantitative table comparing Complete(d)P vs. CompleteP across representative settings. This would make the incremental gains clearer. Further, it is not totally clear what one ought to compare to, because there is no dedicated related-work section. This makes situating the contribution relative to prior works difficult. There seem to be missing citations, such as [1] (I would also expect to see a quantitative comparison to [1]).\n\nThe paper would be more useful to practitioners if it provided recommended base hyperparameters (e.g. base learning rate, initialization scale, etc.) for typical model sizes. Table 1 gives the scaling rules but not concrete base values. In practice, we don't want to run these grid searches (if possible).\n\n\n[1]: https://openreview.net/pdf?id=DZ6iFdVDrx", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)