Weight Decay may matter more than µP for Learning Rate Transfer in Practice
Authors: Atli Kosson
Jeremy Welborn
Yang Liu
Martin Jaggi
Xi Chen
Venue: ICLR 2026 Poster (2026)
Keywords: maximal update parametrization
llm
pretraining
hyperparameter transfer
learning dynamics
adamw
mup
weight decay
hyperparameter tuning
scaling law
transformer
Primary area: foundation or frontier models, including LLMs
TL;DR: Empirically-focused study showing µP requires weight decay to successfully transfer learning rates across model sizes in practice.

Abstract: Transferring the optimal learning rate from small to large neural networks can enable efficient training at scales where hyperparameter tuning is otherwise prohibitively expensive. To this end, the Maximal Update Parameterization (µP) proposes a learning rate scaling designed to keep the update dynamics of internal representations stable across different model widths. However, the scaling rules of µP rely on strong assumptions, particularly about the geometric alignment of a layer’s inputs with both its weights and gradient updates. In this large-scale empirical investigation, we show that these assumptions hold only briefly at the start of training in the practical setups where learning rate transfer is most valuable, such as LLM training. For the remainder of training it is weight decay rather than µP that correctly stabilizes the update dynamics of internal representations across widths, facilitating learning rate transfer. This suggests µP's scaling primarily acts as a form of implicit learning rate warmup, allowing us to largely replace it with modified warmup schedules. Together these findings fundamentally challenge prevailing beliefs about learning rate transfer and can explain empirical observations such as why µP requires the independent weight decay variant for good transfer.

Published as a conference paper at ICLR 2026
WEIGHT DECAY MAY MATTER MORE THAN μP FOR LEARNING RATE TRANSFER IN PRACTICE
Atli Kosson! Jeremy Welborn! Yang Liu! Martin Jaggi² Xi Chen! AAmzzon FAR (Frontier Al & Robotics), 2EPFL
ABSTRACT
Transferring the optimal learning rate from small to large neural networks can enable efficient training at scales where hyperparameter tuning is otherwise pro- hibitively expensive. To this end, the Maximal Update Parameterization (μP) proposes a learning rate scaling designed to keep the update dynamics of internal representations stable across different model widths. However, the scaling rules of juP rely on strong assumptions, particularly about the geometric alignment of a layer's inputs with both its weights and gradient updates. In this large-scale empirical investigation, we show that these assumptions hold only briefly at the start of training in the practical setups where learning rate transfer is most valuable, such as LLM training. For the remainder of training is weight decay rather than μP that correctly stabilizes the update dynamics of internal representations across widths, facilitating learning rate transfer. This suggests JP's scaling primarily acts as form of implicit learning rate warmup, allowing us to largely replace it with modified warmup schedules. Together these findings fundamentally challenge pre- vailing beliefs about learning rate transfer and can explain empirical observations such as why μP requires the independent weight decay variant for good transfer.
INTRODUCTION
The Maximal Update Parameterization (μPP) has become a cornerstone of efficient training at scale, particularly for large language models (LLMs). Many open-recipe LLMs have used variants of pP
(Dey et al. 2023 Hu et al. 2024 Liu et al. 2023 Falcon-LLM Team 2025 Cohere et al. 2025), and
some commercial models with unknown training details likely use similar learning rate (LR) scaling approaches. For example, Llama4 used an internal technique called MetaP, Grok-2 configuration files suggest yP use, and the GPT-4 report (Achiam al., 2023) cites a key jPP paper coauthored by OpenAl staff (Yang et al. (2021). It has also impacted theory, showing how feature learning can occur in infinitely wide networks, contrasting with the neural tangent kernel regime (Jacot et al. 2018
Independent WD
Standard WD
No Weight Decay
Final Val Loss
4.0
3.5
3.0
V
10-3
10-2
10
10-3
10-2
10
10-3
10-2
Base Learning Rate
Base Learning Rate
Base Learning Rate
Width 128
Width 256
Width 512
Width 1024
Width 2048
Min Loss
Figure 1: μP specifically requires independent weight decay for good learning rate transfer when pre- pre-training LLaMA networks with AdamW (20B tokens, width 2048 and IB params, see Appendix F.3). Independent weight decay achieves this by counteracting jAP' update scaling later in training which is needed because the core assumptions juP's scaling is based on quickly break down. Standard weight decay does not counteract μP's scaling and consequently results in poor learning rate transfer on longer experiments. No weight decay behaves somewhere in-between, see Appx. B
i Work done while interning at Amazon FAR

---

Published as a conference paper at ICLR 2026
Despite this success, several works (Wortsman et al. 2024 Blake et al. 2025 Wang & Aitchison 2025| Bergsma et al., 2025) have empirically found that learning rates only transfer well in practice when up is combined with the independent variant of weight decay (WD), see Figure | In the conventional AdamW formulation (Algorithm [T). weight decay multiplies the weights by 1 - 7/ at each step where 7/ is the learning rate and À the weight decay coefficient. With independent weight decay, the weights are decayed by 1 1 À which is not affected by JuP's learning rate scaling. Equivalently, À can be scaled to keep the product 17À constant 1) is scaled in the formulation above.
In this work, we demonstrate that independent weight decay is not merely a helpful addition but a central mechanism for enabling learning rate transfer in practice. We begin by developing a unifying framework for weight decay and μuP based on relative updates, the size of updates as a fraction of the current weights or features (Section 2). Within this framework, we show that independent weight decay directly counteracts and ultimately overrides μP's scaling during training (Section B). We then show why this is crucial: for the majority of training. is independent weight decay, not jaP, that correctly stabilizes feature learning across different model widths (Section 4 This stabilization of the feature learning seems to be what facilitates learning rate transfer and a key objective of j.P.
The need to override JP's prescribed update scaling arises because the underlying alignment as- sumptions fail hold in practice. We show this can happen when the batch size is large relative to the model width, something that commonly happens in practice unlike the infinite-wide that inspired μP (Section This analysis fundamentally reframes JP's practical role as an implicit learning rate warmup. We empirically measure this effect (Section ( and demonstrate that the practi- cal benefit of μP can largely be replicated by using stronger, explicit warmup schedules (Section 7 Together, our findings challenge prevailing belief's about learning rate transfer and provide guidance for achieving robust transfer in practice. In summary our core contributions are:
limit
We introduce a relative update framework that unifies jP and weight decay throughout training. We demonstrate empirically and analytically that μP'' core alignment assumptions fail early on. We show independent weight decay overrides μP's update scaling, stabilizing feature learning. We reframe JP's practical benefit as implicit learning rate warmup and verify this empirically.
2 A UNIFYING FRAMEWORK FOR μP AND WEIGHT DECAY
2.1 SINGLE LAYER MODEL AND REPRESENTATION CHIANGES
The core μP and weight decay concepts discussed in this work can be understood from the linear transformation of a fully connected layer in isolation. For an input feature dimension C, output feature dimension K and batch size B, we define this as:
(1) weight update WHW +AW causes the layer's outputs to change as Y for the same input X. Note that in the context of neural networks, AY refers to the local representation change from this layer alone. The total change would include potential changes A in the inputs arising from the updates of earlier layers. Local changes can be related to total changes shown by Yang et al. (023)) but we will focus on local ones in this work for simplicity.
Y W X RC (inputs), W RR (weights), YERK
(outputs)
2.2 ALIGNMENT RELATES WEIGHT MAGNITUDES TO REPRESENTATION SIZES
Central to this work and μP in general is connecting the size of the representation changes || to the size of weight updates || AWI]. We measure this in terms of the Frobenius norm
is easy for an optimizer to give a specific weight update size |AW] ||: Adam (Kingma & Ba 2015)
does this approximately and sign-based optimizers like Lion (Chen et al. 2023) can do this perfectly. However it is ultimately the size of the representation changes |Y that determines the impact of an update. For example, weight update that does not change Y at all has no immediate impact. and is the core objective of juPs LR scaling.
However, controlling ||Y|| || is much harder than || A AW|
We describe the relationship between || |AY|| and AWI| via the update alignment, defined as: |AWX] X|
OAW |AY| X TAW |X V R,,
Er,b TAw [ cos2 L(Aw;, 2ta) E [O,
Ammyl 2 my
This is weighted root-mean-squar square average of the cosine similarities between input samples X= [21,.... BB) and rows in the weight update A AW =
Note that a high update alignment OAw means that Awi; and a, tend to point in similar directions and will result in larger representation changes ||Y|| for given update size |]. We similarly define the weight alignment as aw
|Y|| X which relates the weight magnitude ||] to representation size ||Y|| ||.
2

---

Published as a conference paper at ICLR 2026
(a) Normalization Layers
(b) Counteracting Gain
(c) Homogeneous Activation
kW
LayerNorm
kW
7/K
y
<Wi
ReLU
Wa/K
Figure 2: Neural networks are commonly invariant to certain types of rescaling transformations. Examples where applying an in-place scaling factor K >> 0 does not affect the current block output y. Across such transformations, an update with a fixed absolute size ||W]] has a varying impact on the output while given relative change || | [/]] always has the same impact. This makes the relative change more informative measure of the size of updates that better captures their impact.
2.3 WEIGHT DECAY MODULATES THE RELATIVE SIZE OF WEIGHT UPDATES
Weight decay is widely thought to mainly function as regularizer despite a long line of work arguing
this is not the case in deep learning (Van Laarhoven] 2017; Zhang et al. 2019 Chiley et al. 2019; Li
& Arora 2020 Li et al. 2020; Wan et al. 2021 Li et al. 2022] Kosson et al., 2024b D Angelo et al. 2024). Instead it primarily acts secondary learning rate hyperparameter that modulates the size of relative weight updates || A W ||- Kosson et al. (2024b) describe these effects in Adam W in detail. In the standard variant (Algorithm D. weight decay will cause the weight norms to converge to fixed equilibrium value given by [W] VKC- -n/A for learning rate 77. weight decay À and matrix size K x C. At this value, the shrinking effect of weight decay and growth effect of gradient updates balance out, keeping the norm constant in expectation. Adam W's normalization of the weight updates makes || |||] x nVKKC, which in turn gives relative updates || |I|l W x vny.. Crucially, note that the learning rate and weight decay À affect the relative updates in equilibrium the same way, only their product yp matters. In summary:
|| x VKCn/>,
JAWIl [W
x Vny
(equilibrium effect of weight decay) (3)
Due to the structure of neural networks, relative changes are often more meaningful than absolute changes (Bernstein et al. 2020: Kosson et al., 2024b a). A given relative update size | W|/l/ ||| has the same impact across number of equivalent parameter states obtained via transformations like those shown in Figure 2 unlike a fixed absolute update size ||AW]. II. A similar argument extends to the representations, given change AY typically has greater impact when ||||| || is small than when it is large. Rescaling transformations do not occur in standard training but weight decay can have a similar overall effect by changing the weight norms over time while normalization layers or learnable gains can undo the effects of this on the output. We show an example of this happening in practice in Appendix G In this work we focus on relative changes rather than absolute ones based on their informativeness and connection to weight decay, deviating from existing work on jaP. Note that at the start of training these approaches are equivalent since || || initialization. However, later in training they can differ significantly.
and || Y|| are roughly determined by the
2.4 FORMULATING μP IN TERMS OF RELATIVE UPDATES
The primary design goal of μ&P is to ensure stable and non-trivial feature learning at any width. Yang
et al. (2023) formulate this as || representations and their changes neither explode nor vanish as the input dimension C' grows In our relative formulation we want to keep || |Y||/| roughly constant across widths for a given learning rate. The hope is that fixed relationship between learning rates and representation changes will result in transfer of the optimal learning rate, but this is ultimately a heuristic without guarantees. We can express the relative representation change in terms of the relative weight update as follows
| |IIMS := || |Y / VKBB = @(1) and || | ||MM = (1). i.e., the
|AY||
OAW |AW| |
(4)
Y
ow
where we call a&w /ow the alignment ratio. To stabilize the relative representation changes JPP | across different widths C. μP must perfectly counter changes in the alignment ratio by scaling the weight updates appropriately (via the learning rate). This requires knowing how both the update alignment aAw and weight alignment a' behave as a function of the network width C.
'Techniaally it is the total representation change across training that matters, but Yang et al. (202)) show this can follow from bounded local changes with assumptions about the spectral norms of weights and updates. We assume matrices are non-zero which typically the case throughout training, barring special initialization.
3

---

Published as a conference paper at ICLR 2026
At the start of training the weights are randomly initialized. If we assume the weight vector E R has independent and identically distributed (IID) zero-mean elements that are also independent of the inputs æ E Re, we can show that [ x) = El| w | [|][||=]|]]/ resulting in aw R 1/VC. This is analogous to the computation used to derive variance preserving initialization schemes which call for |W| /K, which notably does not depend on the input dimension C.
The update alignment is slightly more complicated since the update A WWIn is a function of the inputs X rather than independent, causing correlations that tend to increase the update alignment. Specifically the weight gradient is an outer product of the inputs X and the gradients with respect to Y. For SGD at batch size one, this makes the update a scaled version of the input, giving perfect alignment oAw = 1 regardless of C. Based on this we get the core μpP alignment assumptions:
aAw = =1),
aw = @(1/VC),
OAW = e(VC)
(5)
OW
With this we can scale the learning rate counteract changes in the alignment ratio in Equation 4 When the network width is multiplied C + mC, we thus get the following μPP -r-eeescribed scaling:
|AW| [W]
saW oow
x
(relative update size) v/m
1] = Taase m (Adam LR)
(6)
The 7) scaling follows from the relative one based on |AW| kVKC for Adam and |W]| VK. The optimal base learning rate Those is what we aim to transfer across widths, tuning it at a small scale rather than the target scale are interested in, saving significant compute in the process.
We emphasize that the prescribed relative update scaling in Equation i fully consistent with prior works on μP and not artifact of our formulation, see for example Appendix B.3 of Yang et al. (202T) Adafactor "Full Align" in Table Everet et al. (2024) and Section 5.4 of Yang et al. (2023). Besides scaling the learning rate of hidden layers, jP involves special handling of the final output layer and attention normalization. [Everett et al. (2024) shows these are not necessary for learning rate transfer, combining standard parameterization with layer-wise scaling of the learning rate works equally well or better in practice. We adopt their simpler version in this work although we refer to the scaling as p.P pp--precrrbbed or simply μP for brevity. See Appendix F2 for details.
INDEPENDENT WEIGHT DECAY EVENTUALLY NEUTRALIZES μP'S SCALING
There are two common formulations of AdamW. In the original work by Loshchilov & Hutter (019 the rate of weight decay was determined by X alone, i.e., at each step the weights are scaled by X. However in the default implementation in Py PyTorch the weights are instead shrunk by 1 1- 1Jλ. These equivalent since we can scale À to get one from the other, but will behave differently when is modified by j... In this work we base all discussion on the PyTorch variant but consider the following two approaches for weight decay as uP scales the learning rate 17 + n/m:
(17, X) + (n/mm, X) (7,, A) + (n/m, mA)
(standard scaling) (7)
(independent scaling) (8)
Note that independent scaling keeps the product π7λ constant, meaning the relative updates in equilibrium are not scaled at all (see Equation 3) This fundamentally differs from juP's prescribed scaling in Equation meaning that independent weight decay scaling contradicts μuP by eventually overriding its update scaling, On the other hand standard weight decay scaling results in equilibrium behavior that follows the relative update scaling prescribed by μPP This is surprising because multiple works have empirically found that using independent weight decay gives better learning rate transfer (Wortsman et al. 2024, [Blake et al. 2025| Wang & Aitchison| (025) Bergsma et al.) 2025). We
replicate this finding for LLaMA (Touvron et al. (023) training on DCLM (Li et al. 2024) next
token prediction in Figure Together these results suggest that applying the pP-thherelical scaling
of the relative update size throughout training does not work well in practice. We validate this finding through direct experimentation in Appendix H
To the best of our knowledge, this key mismatch between theory and what empirically works has not been pointed out before. In the following sections we use our unified relative framework to clarify the exact roles of both jaP's scaling and weight decay in achieving learning rate transfer in practice.
4

---

Published as a conference paper at ICLR 2026
aaw/aw dww
|
Std. WD Scaling
2
10
10-2
2'
10
10
27
10-1
1x
10
16x
10:
10-5
Ind. WD Scaling
10
10
24
10-1
10
2
10
10
16x
10-5
29
10
0
15
20
0
5
10
15
20
0
5
15
20
Step [10')
Step [10')
Step [10')
Figure 3: Standard μuP scaling fails to maintain relative representation changes across widths. Comparing the relative representation change (RRC) for μP with standard weight decay scaling (first row) and independent weight decay scaling (second row) for LLaMA training with a 10% linear warmup followed by linear decay. Plots shown for single representative layer with C' 128 at 1x width, see F.3.2 for details. Left: Standard scaling does not preserve the RRC for the same learning rate unlike independent scaling. Recall that the RRC is the product of the alignment ratio and the relative weight update (Equation 4.. Middle: The alignment ratio is similar for both widths and approximately 1. Right: To maintain the RRC across widths the relative weight update size should then be kept constant as achieved by independent weight decay rather than uP's prescribed scaling.
WEIGHT DECAY COUNTERACT. JP'' BROKEN ALIGNMENT ASSUMPTIONS
Recall that in our formulation JP's core objective becomes mapping a given base learning rate to the same rate of relative representation changes across different network widths. In the left column of Figure 3 we experimentally measure how well this is achieved under both standard and independent weight decay scaling during LLaMA pre-training. We find that independent weight decay scaling successfully maintains the relative representation changes across widths as indicated by the similarity of the traces on the lower left panel. In contrast, standard weight decay scaling fails to do this; the same base learning rate results in vastly different relative feature updates for the narrow and wide networks later in training. Since learning rates transfer well with independent weight decay (Figure I. this suggests that our goal of maintaining relative representation changes across widths can be an effective objective for learning rate transfer, but μP relies on independent weight decay to achieve this later in training.
The reason independent weight decay helps maintain relative representation changes can be un- derstood from the middle and right columns of Figure Recall from Equation 4 that the relative representation change is the product of the alignment ratio shown in the middle column and the relative weight change in the right column. We that the alignment ratio quickly falls from the width-dependent initial value to approximately one, violating jP's assumptions. Once this happens, relative weight changes should be the same across both network widths in order to produce the same relative representation changes, which is exactly what independent weight decay achieves in practice, matching theoretical models. Standard weight decay scaling results in smaller relative updates for the larger network, as prescribed by jPP. This fails to maintain the relative representation change with the lower alignment ratios that dominate training, preventing learning rate transfer.
In Appendix B we explore the effects of not using weight decay at all. Somewhat counterintuitively this also results in similarly sized relative weight updates across widths like with independent weight decay. The reason is that scale-invariant weights grow perpetually when updated without weight decay. Once the cumulated updates outweigh the initialization, the weight norm becomes proportional to the (peak) learning rate used, making both the weights and their updates proportional to it. This means their ratios, the relative weight updates, eventually lose their dependence on the peak learning rate without weight decay. We believe this is a key reason learning rates somewhat transfer without weight decay despite pP's broken assumptions. The ever decreasing relative changes, and the lack of control over them, likely also contributes the loss of performance without weight decay.
5

---

Published as a conference paper at ICLR 2026
Weight Align. dw
20
fan jajin=122
Update Align. dsw
20
Align. Ratio duw/aw
2
fan In-2048
2:²
μuP assump.
2
2F
2-1
20
2
2-6
Step [10P]
Step [10?]
Step [10º]
Figure 4: The alignment assumptions of μP do not hold in practice. Measurements of the weight alignment, the update alignment and the alignment ratio for LLaMA training at different widths. The update alignment varies significantly over time, becoming width-dependent. As a result the alignment ratio loses its width dependence. This violates pP's core scaling assumptions which only hold very early in training. Plots shown for one representative MLP layer, see F.3.3 for details.
WHY DO JuP' ALIGNMENT ASSUMPTIONS BREAK DOWN IN PRACTICE?
Figure 4 shows how JP's alignment assumptions only hold very briefly at the start of training. Both the update and weight alignment can deviate significantly, for the weight alignment this is more prominent in ResNet experiments (see Figure [3] Appendix E.. In this section investigate why and when jP's assumptions about each alignment component break down in practice.
5.1 UPDATE ALIGNMENT
In Appendix C show how the update alignment can become width-dependent when the batch size B is large compared to the input dimension C, violating juP's assumptions. This is frequently the case in practice unlike in the infinite width setting pP was originally developed for.
To understand the conceptual reason for the width-dependence, let us look at the simplified case of SGD for a single neuron i.e., K = in Equation [ Assume the input samples X₃, RC and the output gradients y E R are all zero-mean IID random and mutually independent. The gradient for each sample is then given by g, x&y', y and is also IID and zero-mean. The weight update given by Aw = = E. go, allowing us write the i-th element of the output change Ay as:
(9) The output change for each input involves one self-contribution term and B interference terms from other samples. Although random alignment weakens each interference term by factor of 1/VC relative to the fully-aligned self-contribution, the random sum across all B - terms amplifies their impact by roughly VB. As result, interference dominates when B >> C, giving an overall dependence on C. Conversely, for B & C, the interference becomes negligible making the update alignment invariant to C. In models like Transformers and CNNs, the spatial sequence dimensions act similarly to the batch size. For our LLaMA experiments, the effective batch size becomes the total number of tokens per batch (1.048,576), which far exceeds the network width (C' S 3 x 2048).
Ay; = (Aw,x) x.) = 4-n E, xoyb, x,) = b x,) D Σbes (6,,,, x,)
In practice the gradients of different samples are not uncorrelated. High initial correlation results in large update alignment that resembles the small batch-to-width setting that μP targets. The correlation decreases over time, resulting in width-dependens that violates μP's assumptions, see Appendix
5.2 WEIGHT ALIGNMENT
The initial weight alignment matches pP's assumptions, but analyzing it later in training is hard. Intuitively, we note that we can decompose the weights W. at time t into weight updates A AW, from prior time steps т, where we assume ÁW does include the weight decay term (if used):
(update composition weights) (10) If the decay rate v7λ is large or || is small compared to the updates, the Wo term quickly becomes insignificant. Afterwards, the weight alignment at time is determined primarily by the alignment of X, and recent update terms A W... If the current update and recent updates are similarly aligned with the inputs, the overall weight alignment may approximate the update alignment. This would explain the alignment ratios OAw/ w/ww s 1 that we often observe in practice (see Figure
W. = (1 - qa)² Wo + Et=b m/))=-- AW,

---

Published as a conference paper at ICLR 2026
10
10 -2
Scaling Effect St
20
2-1
|
10 3
[|wI|//
10-4
Base LR= WD=A
10
2
10
LR=1/16, WD=16X
10
10 15
20
5
10
15
20
5
10
15
20
Step (10*)
Step [10')
Step [10$)
Figure 5: uP with independent WD scaling has a warmup-like effect on relative updates. Measurements of the relative weight update (left) and relative representation change (middle) for different learning rate (LR) and weight decay (WD) combinations in a LLaMA layer with 2048 input features. Independent weight decay scaling achieves given LR-WD product using lower LR and higher WD (orange) compared to not performing juP scaling (blue). This results in smaller relative updates early in training, similar
additional learning rate warmup factor. The right panel shows this scaling effect through the ratio of the relative weight updates for the two runs Tद to asymptotically
8, = ||AW W Miue W orangell/ [I] [ biue ||| approaching 1 in roughly exponential manner. See F.3.4 for experimental details.
A onange []), which goes from
6 THE WARMUP EFFECT OF μP WITH INDEPENDENT WEIGHT DECAY
Based on the order-one alignment ratios that we find to dominate practical training, the relative weight updates should be kept roughly constant across widths. Kosson et al. (2024b) described how the magnitude of the relative updates in the steady-state only depends on the product of the learning rate v) and weight decay rather than their individual values. However, (1), A) pairs with the same product generally affect the initial phases of training differently. Specifically, high weight decay pairs like those obtained from independent jP scaling via (7, A) + (n/mm, mm) have a warmup-like effect, where the relative updates at the start of training are comparatively smaller. Therefore, uPP acts as a special form of additional learning rate warmup when used with independent WD scaling. We measure this effect empirically for LLaMA training in Figure 5 confirming the general effect still present despite the lack of the perfect scale-invariance typically assumed by works on weight decay. Note that since juP does not scale the updates appropriately for the later phases of training, this warmup effect is the only practical benefit of uP's learning rate scaling for longer runs.
The shape of the induced warmup effect generally quite complex. The initial downscaling of the relative update is always 1/m where m is width ratio used for uP's scaling. The scaling approaches one over time as the weight norms close in on their equilibrium values. The effective length of the warmup depends on the initialization value, learning rate and weight decay. In Appendix D we analyze a simplified setting with a constant learning rate where the scaling ratio becomes:
1 + p2 1)a2t
8t = V 1+ (m² p p2o
(simple warmup effect) (11)
1) 1)a2
PP
V1+ (m² 1)a2
where a == 1 - 17λ is the decay multiplier at each step. Pt is the weight RMS, Po is the initialization RMS value, and Poo = 3 is the predicted weight RMS equilibrium. In practice this model does not accurately predict the shape in Figure overestimating the time required to reach equilibrium. The reason is likely the interaction of momentum with temporal gradient correlations that do not appear in the simplified fully random and uncorrelated setting the model is derived for.
Finally we note that μP's learning rate scaling in AdamW is used to both counteract assumed changes in the aligmment ratio and the size of the relative updates. Recall that the learning rate scaled 17 x 1 //m whereas the relative updates should only scale with 1/ /mm. Even without any assumptions about the alignment ratio, it would make sense to scale the learning rate and weight decay (7, A) + (n/ /m, vm- A). This scales Pro & 1/ym,, matching how variance preserving initialization scales po 1/mm. Such scaling preserves both and Po/ Poo in Equation 11 and is thus more likely to preserve the relative weight update behavior across training. In comparison independent μPP scaling has additional warmup effect of 1/ /mm in the relative updates (which is desirable), but also decreases Pro & 1/m. These smaller weight norms could potentially cause issues (signal propagation etc.) if there are not learnable gains or normalization layers counteract them.
7

---

Published as a conference paper at ICLR 2026
4.0
4
Final
4
4x
3.0
16x
10%
exp50
50%
decayaway
10
10-2
10-1
10-3
10
10 10-3
10
10-1
Learning Rate
Learning Rate
Learning Rate
Figure 6: Stronger warmup can replicate the benefit of uP with independent WD scaling, Learning rate transfer plots for LLaMA training using different warmup approaches. Left: pP with independent scaling transfers well. Middle: No yP scaling with either the base 10% linear warmup or a longer 50% warmup. These give poor learning rate transfer in comparison. Right: No pP scaling with additional warmup incorporating the width scaling factor m can give good transfer, but can require tuning and is less stable at higher learning rates. Additional 50% exponential warmup
(Equation 12) compared a decay-away warmup (Equation 13
See F.3.5 for experimental details.
7
CAN STRONGER LEARNING RATE WARMUP REPLACE μP's LR SCALING?
Learning rate warmup is already commonly used in prac- tice, making it unclear whether the additional warmup effect from μPP is really needed. The first two panels of Figure 6 explore this for LLaMA training, compar- ing the quality of learning rate transfer with and without uP's scaling. Interestingly, we find that even long linear warmup does not necessarily give the same benefits for learning rate transfer or stability as μP with inde- pendent weight decay. This suggests that linear warmup schedules may not be very effective for certain setups despite their widespread use in practice. In Appendix E we conduct a similar experiment finding that the addi- tional warmup effect is not needed for ResNet training. This could be a part of the reason μP like learning rate scaling did not appear until transformers became popular, well-normalized convolutional networks may not need it reasonable scales.
Extra Warmup Factor
2
2.
2-
2-3
Exp-Increasing 50%
Decay-Away
2
10
15
20
Step [10²]
Figure 7: Additional warmup factors
applied on top of a linear decay schedule with 10% linear warmup for m 16, 1 0.004, and À = 0.1, matching the minimum in the right panel of Figure
We conjecture that linear warmup schedules may increase too fast early on, not allowing enough time for the alignment ratio to fall. Taking inspiration from the more exponential warmup shape from Equation 11 we experiment with two types of additional multiplicative learning rate warmup factors:
$ m min(U, 1/Tw-1)
(exp-increasing warmup) (12)
1-1
8, = (1 + (m² 1) (1- n T=0
(decay-away warmup) (13)
Both scale the initial learning rate by 1/m, similar to μP, and eventually become one. The exponen- tially increasing one explicitly defines the length of the warmup through a hyperparameter Tw which can differ from the length of the linear warmup. The decay-away warmup comes directly from the simplified case of Equation IT] this case the length of the warmup effect differs depending on the value of 7/λ, which can vary over time if the learning rate changes according to schedule 1):-
The final panel of F the effect of applying these additional warmup factors on top of the existing 10% linear warmup. This works well, offering better stability and learning rate transfer than with the linear schedules suggesting: Additional exponential warmup can sometimes replace uP's learning rate scaling in practice. Although the effect from independent scaling still seems to stabilize higher learning rates slightly better, these results further point to the warmup effect being uP's primary benefit. We note that in experiments we applied the warmup factors all parameters. This differs from the warmup effect of the independent scaling which only affects parameters whose learning rate is scaled by jP, i.e., the weight matrices of the hidden and final output layers.
8

---

Published as a conference paper at ICLR 2026
RELATED WORK
We discuss related work in more detail in Appendix[A] only briefly mentioning the strongest influences and most closely related works here. Our work adapts the layer-wise LR scaling approach of [Everett ct al. (2024) and uses a similar alignment definition, although they only consider weight alignment. We also take inspiration from the simpler more accessible approach to jPP introduced by Yang et al. (2023) to isolate the key alignment assumptions of μul and work with local changes. Finally, we build closely upon the framework for weight decay introduced by Kosson et al. (2024b) and the additional observations about relative updates and learning rate warmup from Kosson et al. (202aa).
The question of why jPP requires independent weight decay was explored before by Wang & Aitchison (2025) who also empirically demonstrated its necessity for learning rate transfer. Their analysis revolves around approximating AdamW as an exponential moving average (EMA) over past updates, which has a form similar to the geometric sum in Equation I0 From this perspective the decay rate defines a time scale for the EMA (analogous to a half-life) which is an alternative characterization of the relative update size in equilibrium. They argue the time scale should intuitively remain constant across network sizes in order for the final network parameters to form a similar average over different data points, but do not justify this formally. We note that although the timescale interpretation can be a useful mental model and approximation, it does not account for the dependency of later updates upon earlier ones. This dependency is exactly what prevents optimization from being performed in single step by simply taking an appropriate weighted average over all the data. We take a different approach by relating weight decay directly to the goals of uP through the rate of feature changes.
DISCUSSION & CONCLUSION
Originally μP was proposed a theoretical framework that aims to control feature learning in the infinite width limit at initialization. This work expands upon this by studying feature learning throughout training in practical finite-width networks through a more empirical approach. Our relative formulation provides unified view of μP and weight decay, explaining how both play a role in stabilizing feature learning at different times during training. Crucially, this reveals that uP's downscaling of the relative weight updates for wider networks is only desirable early in training. As training goes on, jPPs scaling must be counteracted by independent weight decay or similar mechanisms in order to stabilize feature learning and thereby facilitate learning rate transfer.
In this work we have focused exclusively on AdamW for two key reasons. First, Adam W is still the most common optimizer for LLM training, a key application area for μP and learning rate transfer. Second, the original μuP formulation for SGD does not scale the learning rate for hidden layers, using special multipliers instead. This can make standard weight decay behave like independent weight decay under jPPs scaling, making it harder to show the discrepancy. However, we believe the alignment dynamics have described and j(P's broken alignment assumptions occur in SGD and similar optimizers as well. Matrix-level optimizers on the other hand could significantly change these dynamics. For example, Muon (Jordan et al.) 2024) and Scion (Pethick et al.) 2025) can result in a constant low update alignment that does not vary over time, allowing them to bypass much of the complexities we have discussed in this work. This could potentially explain their reduced need for learning rate warmup and makes it a promising alternative to μP for effectively controlling feature learning in practice.
In Appendix E we show that our findings mostly generalize to ResNet training on ImageNet with AdamW. The main difference there seems to be a potentially confounding regularization benefit from higher learning rates. As the network widths increase, smaller relative representation changes minimize the training loss while larger changes minimize the validation loss. Despite this, independent weight decay still gives much better transfer than standard weight decay.
The complex effects of weight decay are unfortunately often overlooked in optimization research for deep learning. The use of weight decay allows specifying the size of relative updates twice. once at the start of training through the learning rate and again in equilibrium through the product of the learning rate and weight decay. Interestingly, juP with independent weight decay scaling actually makes use of this effect to create the beneficial implicit learning rate warmup. Although we showed a similar effect can be achieved via modified learning rate schedules, juP with independent weight decay scaling is a perfectly valid and practical way of achieving this. We hope our work can inspire confidence in this approach in practice, provide useful conceptual insights into learning rate transfer and weight decay, and finally highlight a promising way forward via matrix- optimizers.

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arwih:2303.087777 2023.
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. International Conference on Learning Representations, 2019. URL https //openreview net. forum? faarrmm?ddrrkko-nAA7F
arXiv.1812.0398]
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks kemel learners: The silent aligmment effect. In International Conference on Learning Representations, 2022. URL
ittpps:/ net wtt.://eoorr..tddd-eeoooooooooo.
arXiv:21T1.00034
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Power lines: Scaling laws for weight decay and batch size in Ilm pre-training. arXiv preprint arXiv:2505.13738 2025.
Jeremy Bernstein, Arash Vahdat, Yisong Yue, and Ming-Yo Yu Liu. On the distance between two neural networks and the stability of learning. Advances in Neural Information Processing Systemx, 33:
21370-21381, 2020. arXiv:2002.03432
Charlie Blake, Constantin Eichenberg, Josef Dean, Lukas Balles, Luke Yuri Prince, Bjöm Deiseroth, Andres Felipe Cruz-Salinas, Carlo Luschi, Samuel Weinbach, and Douglas Orr. u-$) muSp: The unit-scaled maximal update parametrization. In The Thirteenth International Conference on Learn- ing Representations, 2025. URL hattp:::////.... httpt:::://peeeeveeeree wwwpereeiiiew.tteefooeeeee wtaet//oormmiiddddddm wtw.ruumiidPPKKIIIIS arXiv:2407.17665
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, and Quoc V Le. Symbolic discovery of optimization algorithms. In seirth- Conference on Neural Information Processing Systems, 2023, URL
net. forum i d=
arXiv:2302.066675
Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32, 2019. arXiv: 19980999
Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi. Dennis Aumiller, et al. Command a: An enterprise-ready large language model. arXiv preprint (arXiv:2504. 00698, 2025.
Francesco D' Angelo, Maksym Andriushchenko, Aditya Varre, and Nicolas Flammarion. Why do need weight decay in modern deep learning? In The eighth Anmual Conference Neural Information Processing Systems, 2024. URL RtLhttps::/ http:sss///peeneeee..eeee.o
Id=YrAxxscKM2
arXiv:2310.04415
Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gptt gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv arXiv: 2023.
Nolan Dey, Quentin Anthony, and Joel Hestness.
The practitioner's guide to the
maximal update parameterization, 2024. URL https: /www wwtp://wwwwrrrrr......ooooooo
The-practitionerz: -wwigggid----oo the the-man --aaaainnn mmaximal
Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon,
Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don't 't be lazy: Completep enables compute-efficient deep transformers. arXiv preprint arXiv:2505.01618 2025.
Katie E Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jachoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. In Forty-first International Conference on Machine Learning. 2024. URL
net oetpr//iww...ettooormooooooaaaaa
arXiv.2407.05872
10

---

Published as a conference paper at ICLR 2026
Falcon-LLM Team. Falcon-hl: A family of hybrid-head language models redefining efficiency and performance, May 2025. URL hettsttts:///fccooo 1m. httpluub.iooboogo/aaaaaaaaa falcoon/
Moritz Haas, Sebastian Bordt, Ulrike von Luxburg, and Leena Chennuru Vankadara. On the surprising effectiveness of large learning rates under standard width scaling. arXiv preprint ( 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. arXiv:1512.03383
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.00339 2024.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
arXiv:1806.0757
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bemsstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https
//kellez github. ic http//psssss///
Tero Karras, Miik Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. An- alyzing and Improving the Training Dynamics of Diffusion Models In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24174-24184, Los Alamitos, CA, USA, June 2024. IEEE Computer Society. doi: 10.1099CVVPS5233.22244 02282. URL https::/doo 2024. 02282 arXiv:2312.02696
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015. | 141.1.11161.9. 26698
Atli Kosson, Bettina Messmer, and Martin Jaggi. Analyzing & reducing the need for learning rate warmup in GPT training. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL Ststraae//peneeereeveeeee//eoene
arXiw:2410.23922
Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks. In Forty-first International Conference on Machine Learning. 2024b. URL https:://pperrreiieeee
wwt/ id=Moi 1d=MO1rNNUUPP
#rXiv2230S.17212
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Kamal Mohamed Abbas, Cheng- Hsieh, Dhruba Ghosh, Joshua P Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen. Igor Vasiljevic, Sham M. Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander T Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-LM: In search of the next generation of training sets for language models. In The Thirty- eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLbttta http:::://///ppeeeeeeeeeee
net. for f=
arXiv:246..11794
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In Inter- national Conference on Learning Representations, 2020. URL URL/https:///ot URLhtttps::/opeeeei.ies net/
forum? 1d=rJg@TeSFDD 1rX.v:1910.07454
Zhiyuan Li, Kaifeng Lyu. and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 114444-445555 2020. arXiv:2010.02916

---

Published as a conference paper at ICLR 2026
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. In International Conference on Machine Learning, pp. 12656-12688 PMLR, 2022. arXiv:2202.00980)
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. arXiv preprint (arii::3322.0000
2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
arXiv:1711.0511 Bkg6RicqY7
Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg, ngpt: Normalized transformer with representation learning the hypersphere, 2025. URL https::/arxiv..rrt aabs/
2410.01133
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomozak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 106440-106473. Curran Associates, Inc., 2024. doi: 10.52202/079017-3378. URL
www.:n/ws..uu.rr.o....o. wwtp:rippppcppppprrrpeppp les psspppapr///22/ ELL:00044T2200033333333377773333 Cowppeeerrrooooceeeeeee pdf ar xXIv:2407.01800
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint 1122::::1122061222 2018.
Christian H.X. Ali Mehmeti-Göpee and Michael Wand. On the weight dynamics of deep normalized networks. In Forty-first International Conference on Machine Learning, 2024. URL https
net/ / AAAACCChJJSS
arXiv:2306.00700
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 191.01191
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Forty-second International Conference on Machine Learning, 2025. URI
ew net/
forum?id-20qml I arXiv:2502.07529
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV). 115
1-252, 2015. doi: 10.1007/s1 1263-015-08 16-y. arXiv: NW::1409 409.0575
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedingss of Machine Learning Research, pp. 4596-4604.
PMLR, 10-15 Jul 2018. BtlLattsss//pccccalgggg wwwpeggsmmrr.om http:///ww...........o./////a/ea
html arXivv1804.00.. 04435
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https: ptttps://axxxi..or//bbbe..
www.orrggaao.o abs abrrg/2bs223322 2302 13971
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint Stti::77060535500 2017.
Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet, Olivier Teytaud, and David Lapez-Paz. Meta Lingua: A minimal PyTorch LLM training library, 2024. URL
12

---

Published as a conference paper at ICLR 2026
Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynam- ies: Learning dynamics of normalized neural network using sgd and weight decay. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 6380-6391. Curran Asso- ciates, Inc., 2021. URL httpps::/procceeddiiiiiiec// 2268cC56ccc055555511666633333332 Paper pdf arXiv:00060041199
neur ps cC wwwp//pppe///2////ececc
Xi Wang and Laurence Aitchison. How to set adamw's 's weight decay as you scale model and dataset size. Forty-second International Conference on Machine Learning, 2025. URL https:
//openreview. aee
arXiv:2405.366%
Ross Wightman. Pytorch image models.
https://github com.u...commee.. wtt.://./.o.ww//tatma....m
pytorch- latta--immee--ooooee 2019.
Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jachoon Lee, Justin Gilmer, and Simon Korbblith. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learn- ing Representations, 2024. URL ptctps://opennet ew net. forum? d=d8
arXiv:2309.14522.
Ge Yang, Edward Hu, Igor Babuschkin. Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neu- ral networks via zero-shot hyperparameter transfer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.). Advances in Neural In- formation Processing Systems, volume 34, pp. 17084-17097. Curran Associates, Inc., fiwesllllsaaae #11lss/appr/22211 Paperaapppreee pdf arXiv:2203.03466
2021. URL https::/ wwwurrps.ccppaaae..
E1 File/ Fale/8dffc2es Bata:///ceccccceeeeeeaaaaabeeaaa
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 18-24 Jul 2021. URL https:
html
XrX:v:201 14..2222
14522
Greg Yang, James B. Simon, and Jeremy Bernstein. A spectral condition for feature learning. ArXiv, abs/2310. 17813, 2023. URL httptt::://aat semannicscchhhl.rr. horprr/cccccooo.oo//oorrrpss 264555180 arXiv:2310.17811
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs VI: Feature learning infi- nite depth neural networks. In The Twelfth International Conference on Learning Representations, 2024.
net. http:/eeeeeettmmmeeee//o////ooo
arXiv:2310.02244
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference Learning Representations, 2019. URL
https::/openreviie. net wtt.://rrrmmrrm//m 3Rct? arXiv:1810.12281
13

---

Published as a conference paper at ICLR 2026
LLM STATEMENT
LLMs were used for aiding and polishing writing as well as creating and modifying plotting scripts.
A
EXTENDED RELATED WORK
Adam (Kingma & Ba 2013) is a widely used optimizer in deep learning. It was originally used with (g-regularization but Loshchilov & Hutter (2019) empirically found that decoupled weight decay works better, resulting in AdamW. The version of Adam W we use is shown in Algorithm | and
corresponds to the most popular variant, the default in Py (Paszke al. 2019
Weight decay is unfortunately still widely believed to act primarily as an explicit regularizer despite a
long line of work that argues this is not the case in deep learning (Van Laarhoven 2017 Zhang et al.
2019 Chiley et al. 2019 Li & Arora 2020 Li et al. 2020; Wan et al. [022]] Li et al.) 2022 Kosson 2024b] D Angelo et al. 2024). Instead weight decay modulates some notion of an "effective"
et al.
learning rate, which can be seen as measures of the relative update size. Our work builds closely
upon Kosson et al. 2024b), who described the equilibrium dynamics of weight decay in detail for Adam W. This included originally noting the scheduling effects, how weight decay can be replaced by controlling the relative weight update, and how the balanced equilibrium behavior of Adam explains its effectiveness over Adam with &g rrrgulaarizatinn
Yang & Hu (021) proposed the Maximal Update Parameterization μPP as a way to ensure stable feature learning in infinitely wide networks. Yang et al. (021) demonstrated how uP allows transferring the
learning rates across different network widths and how this is beneficial for hyperparameter tuning.
Later works have expanded μP network depth rather than only width (Yang et al. 2024; Dey et al. 2025). The original formulation of jP is quite theoretical but Yang et al. (2023) show a simplified and more accessible derivation based spectral norms which we base discussion on. (ey et al.
(024) provide a useful overview of p.P.
Everett et al. (2024) show hyperparameter transfer is possible for other parameterizations than μ.P. including standard initialization schemes without the special multipliers used in j&, using analogous learning rate scaling. We adopt this approach in our experiments as it is simpler and results in better performance according to Everett et al. (2024). They also describe how the learning rate scaling depends on alignment metries similar to those we define Equation but use the total weight change from the start of training rather than single optimization step. Our findings fit particularly well with one case they consider, the no-alignment setting. Their analysis of AdaFactor (Shazeer & Stern 2018) with
updates in this case predicts that learning rates should transfer without any scaling. This is similar to what we observe in practice once weight decay makes the updates proportional to the weights. We expand upon their work with measurements of the update-alignment describing the role of weight decay, and how the batch size affects alignment.
Atanasov et al. (2022) studied a different form of alignment between the Neural Tangent Kernel Jacot et al. (2018) 18)) and the target labels. This alignment is not directly related to the weight and update alignment as we define them far we know, but there could be connection to the update aligmment. The NTK measures the similarity between the gradients of different samples, something we hypothesize the update alignment also depends on through the Signal-to-No Signal-to-Noise ratio described in Appendix
Haas et al. 2025) is very recent work that investigates why higher learning rates than jP prescribes can be used. They investigate alignment changes as potential cause but dismiss them in favor of the role of the loss function, showing that typical cross entropy losses help stabilize the final layer compared to &a based losses. Our alignment measure is based on the current update rather than the total update across training, which might explain the different conclusions.
14

---

Published as a conference paper at ICLR 2026
Weight Align. dw
2a
fan_ing 12_in1128
Update Align. daw
2
Align. Ratio dsw/aw
2
fan_ine 2048
2-2
uP assump.
2-2
2
2
2-4
2-
2
2-h
2-h
10
20
0
10
20
0
10
20
Step [10?]
Step [10º]
Step [10º]
Figure 8: The alignment assumptions of μP break down even without weight decay. Measure- ments of the weight alignment, the update alignment and the alignment ratio for LLaMA training dif- ferent widths. The update alignment varies significantly over time, becoming width-dependent. This violates jP s assumption which only holds very early in training. A base learning rate of = 1.6-10-2 10- is used for the narrower network and scaled down 16x for the wider network via μP's learning rate scaling. Plots shown for one representative MLP layer, layers .13 feed__eeerrrreeeee forward.. wl.
B JP'' ALIGNMENT ASSUMPTIONS STILL FAIL WITHOUT WEIGHT DECAY
In Figure [ we observed that no weight decay seems to provide better learning rate transfer than the of standard weight decay, although it is still not good with independent weight decay. The final loss is also slightly worse than the best configuration with either weight decay approach. In this section explore whether the assumptions of μP hold without weight decay.
In Figure we see that this is not the case. The alignment ratio still becomes roughly one, independent of the width, meaning that the alignment assumptions of μP break down exactly like in the weight decay case. Weight decay is therefore not the cause of jP'' violated assumptions. This matches our analysis of the update alignment (Section where the width dependence at large batch sizes does not depend on weight decay directly. With an alignment ratio that does not vary across width, the relative updates should be preserved in order keep the relative representation changes comparable.
In Figure 9 we measure how the relative updates behave across different learning rates. Notably, without weight decay the peak learning rate has little effect on the size of relative updates later in training. This could explain why the no weight decay setting seems to be less sensitive to the specific value of the learning rate in Figure Without weight decay the size of the relative updates falls over time compared to when weight decay is used, similar to an extra decay factor in the learning rate schedule. The way that this decay occurs seems to cause the relative weight updates to lose their dependence happen in a simplified setting for constant learning rates. The conceptual reason is that if the total weight growth dominates the initial value, both the update and the total weight norm have the same proportional dependency on the learning rate later in training. These factors cancel out resulting roughly identical relative weight updates across peak learning rates. This assumes the network behaves similar to a scale-invariant network, i.e., that there is no strong gradient signal that affects the weight norms. A similar effect of the peak learning rate losing its relevance over time was observed for SGD and normalization layers before (Arora et al., 2019 Mehmeti- Göpel & Wand 2024).
the peak learning rate. In Appendix D.2 we show this can
The optimal learning rate likely becomes a balance between two factors, meaningful learning and stability. Higher learning rates decrease the contribution of the initial weights to the final model, which may allow the model to better learn the distribution. However, high learning rates can also cause large relative updates early in training that can have detrimental effects as explored by Kosson cet al. 2024a),, perhaps by saturating non-linearities or causing instability. these detrimental effects occur sufficiently early, the alignment assumptions of μP may still roughly hold. In this case the optimal learning rate may be decided primarily by early behavior. This may aid optimal learning rate transfer under juP's scaling, despite its assumptions only holding briefly the start of training. We suspect the length of the learning rate warmup may also impact whether the key limiting stability period occurs when JP's alignment assumptions still hold or not.
15

---

Published as a conference paper at ICLR 2026
|/W//I
10-1
10-1
Standard WD
10-2
10-2
10
10
n=2x10
n=8x 10
10
10
10
10-5
10-5
10
0=8x 10
10-2
0=3,2x002
10-2
No WD
1.28 x 10-1
10
10
10-4
10-4
10
10
10
20
0
10 Step [103]
20
Step [103]
Figure 9: Without weight decay relative updates become similar across learning rates over time. Mcasurements of the relative weight update size and relative representation change for LLaMA training. Plots shown for one representative MLP layer, layers.. 13. feed feoeeddfoowrrdddo w1.
Overall the reason for learning rate transfer without weight decay, to the extent it occurs, would then be similar in the independent weight decay case. Early training μP correctly scales down the relative updates which helps transfer relative representation changes under the high, width-dependent, initial alignment ratios. Later in training the relative updates become approximately the same across widths which helps stabilize the rate of feature learning when the alignment ratios are identical across widths. Only standard weight decay follows μP's prescribed relative weight update scaling from Equation 6 throughout training, which is exactly why learning rates fail to transfer with standard weight decay. With either no weight decay or independent weight decay, the relative updates deviate from the prescribed scaling which helps counteract the violated alignment assumptions. However, independent weight decay seems to control the relative updates later in training more tightly, resulting in better transfer overall.
16

---

Published as a conference paper at ICLR 2026
Fan-in C= 256
Fan-in C 4096
2
2
Update Alignment daw
2-1
2-1
P
== 10
p= 10
2-
p=
2-1
2
p=
2-5
2
1//B
1//C
2-6
2
20
24
28
2
2
220
2
24
2
2
2
220
Batch Size
Batch Size B
Figure 10: Predicted update alignment o&w the simple noise model based on Equation 25 The overall behavior well approximated as o&w dashed color lines. Note how the alignment for a low signal-to-noise ratio z 0 is roughly 1/VB (no width-dependence as μP assumes) until B C' when it becomes 1/vC (uP"s assumption fails). Higher values of yp always increase the alignment regardless of B and C. This most prominent when yp > 1/C in which case we get d&w R 4/V/p-lu once >> yp This can be viewed as the batch size being effectively capped at y +1, an estimate of the critical batch size (McCandlish et al. 2018). Early in training this can give a high approximately width-indeepndent alignment, allowing juP's assumptions to hold initially even when B >> C but then fail the SNR decreases.
yp shown by the
C ANALYSIS OF THE UPDATE ALIGNMENT FOR A SIMPLE NOISE MODEL
In this section analyze the update alignment for a simple probabilistic model single neuron with fan-in of C at batch size B. On the forward pass we have:
= Ly......... yn = w X = w [X
X
(14)
where w E RC are the weights of the neuron, X E RCBB B is batch of B input vectors with dimension C each, and y Rd is the output for each input. The gradient of the final average sample
loss L = DE Lo with respect to the weights w is given by:
09
B
LLL X
=
y/xb
(15)
dw
B
B
bml
hml
where we have defined y == 0.2//dy, Note how the gradient for each sample go := y/xx is simply scaled version of the input vector X and is thus fully aligned with it.
We want to be able to introduce analytically tractable correlations between the random variables in our probabilistic model. Taking inspiration from Kosson al. (2024a), we use the Signal-to-Noiss ratio yp to capture the correlation. It is defined as the square ratio of the norms of the expected gradient (signal) to the expected deviation (noise):
gg
g Ep [gg]
(16)
We can now define a distributions for and y', that result in specific &p while respecting the mathematical form the gradient and being simple enough to analyze. chose the form of the input samples as:
1 1 //V++ y
X4 (3r Vp
(17)
where the noise component X; and signal component standard multivariate normal distribution, i.c., Xx, s ~ (0, We choose the distribution of the output gradient to be y', +1 with equal probability avoid complexities in modeling the inverse.
independent random vectors drawn from
17

---

Published as a conference paper at ICLR 2026
We could scale the variance of the distributions but this would not affect the alignment ratio. In this model the gradient becomes:
g+ y/,0 ( + V s)/ 1++
(18)
Assuming ||s||2 C' (or alternatively in expectation over s), this results in the targeted signal-to-noise ratio yp:
/|ss (1) 9 Eollly /olly/..o/v111 +[ /
C/(1 - p) =
(19)
Es ge -
c)
Note that there are other forms for x that result in the desired signal-to-noise ratio including some that give g s. but our choice results in bounded input and gradient norms in expectation for both o + and ip + oc. We will also assume that the update obtained via simple gradient descent:
Aw 7g
(20)
The update alignment involves ratio that is hard to analyze under expectation. Instead will approximate this as the ratio of the square expectations:
A X
EI| Aw [
E[aaw]= E
(21)
X
I X
we thus need to compute Bi| Aw T 11 B[|lAw| [2]. and E |||||
Starting with the input size we get:
E | |3] = B- - Es |>||1x00
| ||
(22a)
B. - Eb
1
Rb + /Vp.5/55.
S
1
Xb + //. S.
)
_
(22b)
1+ 4p
y
y/
B
1
[B + /
(Xb. s)
+ +B
||s||
(22c)
1 4p
(y6))
(C+0++p.C)
(22d)
1 i
BC
(22e)
For the update size we get:
El [|]
77² ||
(23a)
1
X + wps
= =
(23b)
B
v1 + 4
n
B Σ y
E
B /
(23c)
B² (2(1+
B Σ 1=1
B
)
E
+ 2F
Σ
&/ B ByTs))
+ E
/
|3
B²(1 (p)
(23d)
1/2
B
Σ
E| + +0+ B²
(23e)
B² + k))
172
(BC B²)C)
(23f)
B²(1 + +)
1/C 1 + Byp
(23g)
B 1+p
18

---

Published as a conference paper at ICLR 2026
Finally compute the expected output change as:
A = (xx( -ng)
(24a)
Bn²² (1 9)²
1
(x,y) + v )
E8
[x +
Vips/y
(24b)
B
272
(1 B) E [v,11
Σ
(Ke, x2i + (1+ B3) ($x,,
bzb
+ ybb
Σ
(s. x,
B4p |ss| [|s1))]
(24c)
%
b
1/2 (1 + B
18 /
||"1
E
Σ
(xe, x 9 + (1 + B)P s)
+ 5
Σ
(s. xx))s²)) 9 BB (11111) 2Bpp ||xt 11|s||/
(24d)
172 (1 B
EE |1 +
Σ
Ei x (1 B)² (xx,,, s)
1nth
+ 4
E( x,) B 2 10
2B
(24e)
1
1/2
C(C (B
(1 + BB
+ (( - B +2)
(24f)
Now we have all the factors needed to approximate the update alignment via Equation 21 giving:
E[asw] x (C+ + (B 1) + + +) BB +)
-
(25)
(1+ + B1) (1)(1++))C CB
Figure 10 plots this prediction showing C-dependent behavior for large batch-to-widh ratios as expected. We find the overall behavior in Equation 25 well approximated by:
[loAw] VC-L B-1 <p((1 + sp)
(26)
The last term is the inverse of +-1 which is an estimate of the critical batch size (MeCandlish al. 20188) where the overall gradient starts become dominated by the signal component and the benefit of increasing the batch size further diminishes quickly. In the fully uncorrelated case 0 we roughly get o&W x 1/VB when B < C' which is the regime targeted by pP, infinitely wide networks trained with finite batch sizes. the practical training settings we often have B > C which results in oaw 1/VC. This C' dependence breaks the scaling behavior of jPP When the critical batch size is small compared to C' this can instead become d&W dependence again. The overall alignment behavior is thus highly dependent on the specific settings of B and C' well as the signal-to- to-se ratio & which dynamically changes throughout training.
which eliminates the C
Thip
19

---

Published as a conference paper at ICLR 2026
D ANALYZING THE SCHEDULING EFFECTS OF WEIGHT DECAY
D.1 THE EVOLUTION OF THE WEIGHT NORM OVER TIME
Here construct a simple model of how the weight norms evolve over time, similar the used in prior work (Kosson et al. 2024b). In this section we use AW to refer to this "gradient component" of the update, treating the weight decay component separately. We will assume that the weight update (excluding weight decay) at each timestep is orthogonal to the current weights. This results in simple recurrence relation for the weight norms:
[W++1 1|2 = (1 - 7) W. 112 + ||WW.12
(27)
We will assume that the elementwise RMS size of the weight update is roughly one, for example due to the second moment normalization in Adam. Defining P: == || Will/vKe as the RMS value of the elements in Wt] and 0 == 17- as the decay multiplier we get recurrence relation:
p = at + n
(28)
This model involves several significant simplifications:
The assumption that the update is orthogonal to the current weights can either come from randomness or due to scale-invariance from normalization layers. The gradient of a scale- invariant weight is always orthogonal to the weight and zero-mean uncorrelated random gradient is orthogonal to the weights average. We note that with momentum the update is not necessarily orthogonal on average, even these special cases.
This model only holds for momentum when the gradients uncorrelated over time. In this case we can replace the update term by the total contribution of the current gradient over time (the sum of the impulse response) done by Kosson et al. 2024b). This also assumes 31 is not too large, so that the impact of single gradient happens over a short timeframe compared to the t values we are interested in. If there is correlation the behavior of the system changes in a way we can not predict without additional information. See Wan et al. (20T) for an analysis that partially accounts for this in SGDM.
The size of the update assumes that 32 can accurately track the expected magnitude of the gradient over time. This requires sufficiently small 82 values in comparison to how fast the gradient magnitude changes. We note that at the start of training the gradient magnitude often changes very rapidly.
This means that in practice Equation 28 does not accurately predict the behavior of the weight norm in neural network training. However, we believe it is the best we can do based on the hyperparameters alone without measurements of the temporal gradient correlation and gradient magnitude over time (which we expect strongly depend on the other hyperparameters too). We still find the predictions of this model to be informative for real training, particularly those based on the equilibrium value that satisfies Pt+11 = Pt. The RMS values seen in real training often stabilize near this value (Kosson) jet al., 2024b), but how fast this happens depends strongly on the complex temporal effects described above.
With this disclaimer, we proceed with the analysis of Equation 28
Rolling out the recurrence relation:
p
2
II a
II
a
(29)
T=0
==0
(=++1
In the case where the learning rate constant n) = 17, making as constant as well, we get:
2 = p a
20- a
p a 21 1) 1 a²
(30)
assuming 0 < < 1. If nu is not constant, we can still easily compute the behavior of the weight norm over time using the recurrence relation or the rolled-out form directly.
In the constant learning rate case the RMS value approaches the equilibrium value Poo exponentially:
2
p = p². + (0) p3 )
P
17
(31)
1- a2
2X
20

---

Published as a conference paper at ICLR 2026
We use this to estimate how long it takes for the weights to approximately reach equilibrium. If we define this by the timestep TEQ when the RMS value falls within a factor v2 of the equilibrium value we get:
In
P 1-1/v2
if po < 1
vpPoc
2n
TEQ ~
In
Pi/ v2-1
PD 22 otherwise
(32)
2xy
D.2
THE EVOLUTION OF THE RELATIVE WEIGHT UPDATE OVER TIME
In this simplified model the relative weight update becomes:
Th Pt
(33)
which we can compute exactly given the expressions for Pr from the previous subsection. In the constant learning rate case we get:
AW ll WI V
Will
? pi V
1/ a + 172
2nd 172
2mA
(34)
P
1- T-a
t-000
showing that the product y7À determines the size of relative weight updates in the steady-state. However, at the start of training. e.g. the first step, the relative weight update (excluding the weight decay component) is determined by just the learning rate and the initialization norm.
It is also interesting to see what happens without weight decay, X = 0. For constant learning rate, this gives:
- -
(35)
Pt
2 (P(o/1²)
which means that the relative updates decay towards zero over time roughly as 0.5 rather than stabilizing at fixed value. For this reason it is important to use learming rate schedules with weight decay, otherwise the effective learning rate measured terms of relative updates never decays which typically leads to suboptimal outcomes. This decay in the relative update size can also be detrimental, the relative update size decreases too quickly the network stops learning (sometimes referred to as loss of plasticity). An example of this was seen in the AdamW paper (Loshchilov & Hutter 2019) where the optimal weight decay value in an experiment was zero for a constant learning rate but non-zero when cosine schedule was used. It is better to vary the effective learning rate over time explicitly according to our desired schedule, ideally exactly through optimizers like LionAR (Kosson) al. 2024a) approximately via weight decay.
D.3
THE WARMUP EFFECT OF HIGH WEIGHT DECAY CONFIGURATIONS
Let us now consider two configurations for the learning rate and weight decay, (1, A) vs (n/m, mA). These correspond to the configurations from independent weight decay scaling compared to no scaling (or in standard scaling at a higher base learning rate). We first note that their product is identical, so the relative weight update will converge to the same equilibrium value. However, the very first update will also be m times smaller for the high weight decay configuration. A learning rate warmup starting at n//m and reaching over time has an identical effect on the initial update and the steady-state updates. However, the exact shape of this warmup effect is not trivial, ie. how affects the relative update size over time.
We use the previous symbols to denote quantities for the base hyperparameters (17 A) and over the symbol to denote the corresponding quantities for the scaled configuration. We are interested in the ratio the relative weight updates between the two configurations:
np/Po
St
(36)
n// Pt
21

---

Published as a conference paper at ICLR 2026
which captures the size of the warmup effect. For the simple weight norm model we can always compute this using Equation 28 even when the learning rate varies over time. For constant learning rates we would get:
1 Pt 81
(37a)
m Pt
Po 22 + 12 1-
1
1-0²
(37b)
Pd 2t + 12122
2-
1-
P8 + (p) - ~oc )a
(37c)
P (m² Pb poz )a
1 + (pg/ p2o - 1)a21
(37d)
1 (m² p// 1)a2
This is already a relatively complicated expression with behavior that depends on the initialization magnitude Po as well as the exact values of 1y and À used. The effective length of the additional warmup can vary from a single step, c.g. when po - 0, to the whole training, for example when po is large or decays slowly. We note it will also depend significantly n',, when sweeping learning rates the length of the warmup effect will differ between points.
Finally note that simply scaling the learning rate schedule for the base configuration by s, will generally not give relative updates matching the high weight decay configuration. This is because modifying the learning rate schedule also changes how the norms evolve over time. A scaling factor that would make the profiles match exactly could still be computed using the recurrence relations by keeping track of how the modifications affect the norms over time. However, exactly matching the warmup effect from higher weight decay values may not be that important since it is largely arbitrary as far as we know in the sense that it does not depend on the alignment ratio. Applying the scaling factor from Equation 36 relative or rotational optimizer like LionAR (Kosson et al. 2024a) would result in the desired warmup effect.
E RESNET EXPERIMENTS
In this section we repeat some of our experiments for ResNet training to see if they generalize to other architectures and settings. The training setup is described in Appendix [F44] Overall our main conclusions from the LLaMA experiments hold well. The main differences are that the learning rate transfer is worse, likely due to regularization effects and that the warmup effects seem to matter less. The results are shown in the following figures.
Figure compares learning rate transfer between standard and independent weight decay scaling. Independent weight decay gives better transfer but there is a shift in the optimal learning rates for both scaling approaches.
Figure 12 shows that independent weight decay scaling helps preserve relative representation changes. The behavior is essentially identical to the LLaMA setting.
Figure I3 measures the weight alignment, update alignment and alignment ratio. Just like in the LLaMA setting juP's assumptions only hold for very brief period at the start of training. The weight alignment changes more than we observed with LLaMA.
Figure I4 empiricllly measures the warmup effect from the use of high weight decay. The conclusions are identical to the LLaMA setting.
compares learning rate transfer between μP with independent learning rate scaling and not applying μP scaling. We find that both learning rate transfer quality and stability are similar. The μP configuration results in slightly higher accuracy which may at least in part be due to the warmup effect.
Figure 15
22

---

Published as a conference paper at ICLR 2026
1.8
1.6
80
1.6
1.5
1.4
1.4
76
Train Loss
1.2
1.3
1 Accuracy
1.0
1.2
0.8
70
1.1
0.4
1.0
66
0.2
4
0.9
0.0
0.8
10-3
10"²
10-1
10"
10-1
10-2
10-1
10a
10-1
10
10-1
10"
Learning Rate
Learning Rate
Learning Rate
Standard
min loss
x0 25 Width
x1 Width
Independent
max acc
Figure 11: Comparison of standard vs independent μPP scaling for ResNet training on ImageNet. Learning rates generally transfer much worse than for LLaMA. Independent scaling still gives better results than standard scaling overall. Note how optimal learning rates in terms of training loss shift downwards with independent scaling while they shift upwards for the validation loss and accuracy, strong indication of confounding regularization effect. Widths are specified relative to standard ResNet-50 but the base width for scaling the 0.25 variant. Learning rates for parameters that do not scale with width are frozen before the sweep at the optimal value for the 0.25x network.
|
azw/ (aw
/
Std. WD Scaling
10
10
10-4
10
23
0.25x
4x
29
Ind. WD Scaling
25
10
10
2
10-1
10-1
0.25x
10-4
4x
10-4
100
100
100
Step (10?)
Step (10?)
Step [10?)
Figure 12: ResNet variant of Figure Similar to the LLaMA case, standard weight decay scaling fails to preserve the relative representation change across widths. The alignment ratio is similar across widths contrary jP's assumptions, requiring maintaining relative weight updates independent weight decay scaling. Plots show for layer3. 2. convi which a 1 x convolutional layer with inmii==45 for the 4x variant. The learning rate used here 7 4 x 10-3 with epoch warmup (1%) followed by cosine decay to zero.
achieved by
23

---

Published as a conference paper at ICLR 2026
Weight Align. dw
2
fan_in=25 256
Update Align. d&w
Align. Ratio dtww//tw
2
fan 4096
2
μP assump.
2-2
2-
2-4
2a
2.
Step [10.]
Step [10?]
Step [10?]
Figure 13: ResNet variant of Figure showing how the alignment assumptions of μP do hold. Plots show for layer3. convl which is 1 1 convolutional layer. Both the weight alignment and update alignment deviate significantly from pP's assumptions. Note the larger changes in the weight alignment over time compared to the LLaMA experiment. Learning rate 17 = 4 x 10 no JPP scaling applied.
10
10
Scaling Effect
10-3
10
2-1
10 1
2-2
LR=1,
2
10-3
LR-1/16, WD-16A
10
Step [10*)
Step [10')
Step [10*]
Figure 14: ResNet version of Figure showing the warmup effect of higher weight decay configura-
tions. Plots show for 1ayer3.2 convl which a used are 17 8 x 10 3,- x = 0.1 corresponding to the best standard WD configuration in Figure I]
convolutional layer. Hyperparameters
(after μP scaling). The additional warmup effect here comparable in absolute length the LLaMA setting but proportionally shorter.
78
Val Top-1 Accuracy
76
74
72
68
64
62
10-1
10-3
10-2
10-3
Learning Rate
Learning Rate
Learning Rate
μP
ne-μP (10% WU)
x0.25 Wisth
x1 Width
no-uP (1%
max
Figure 15: Learning rate transfer comparison between pP with independent weight decay scaling (left) and no-pP scaling for ResNet training (middle/right). The left and middle panels use 1 1% linear warmup, the right panel uses 10% linear warmup (WU). The additional warmup effect of μP with independent weight decay scaling is not needed for stability or transfer, but may contribute to the slightly higher accuracy achieved on the left. Other differences such as the resulting weight norms could also matter, particularly for the final FC layer. For normalized layers the weight norms also inversely scale the gradient norms which may affect the result through interactions with the epsilon value of Adam. A similar gap also appears in Figure between standard and independent weight decay scaling.
24

---

Published as a conference paper at ICLR 2026
Algorithm 1 The PyTorch variant of AdamW. This differs from the original by Loshchilov & Hutter
(2019) which corresponds to replacing the orange term with 7/// maxe (7te) here, applying the learning rate schedule to the weight decay term but decoupling the overall strength from the peak LR max, (m.).
Require:
Learning rate n/, weight decay X, momentum 81, magnitude smoothing 8a, for numerical stability 1: Initialize: Time step 0, parameter vector Oo. momentum vector ma O. magnitude vector wo
2: while stopping criteria not met:
3: I <-< -+++
4:
+ Mini-batch gradient w.r.t. 0
5:
m, - B1 m + (1 3.)00
6:
Ut p20=-1 + 82)gi
7:
m & mz/(1 (1- 8))
8:
û. + ve/(1 (1 (32)
9:
(1 11100.11 n note/lve
F EXPERIMENTAL DETAILS
F.1 ALGORITHMS
The Adam W variant we use is shown in Algorithm | Note how the weight decay hyperparameter in line 9 scaled by the learning rate. This differs from the original variant described in [Lshhhhilov & Hutter (2019), which is often referred to as having "independent" or "fully-decoupldd" weight decay. This variant only has À instead of y in line 9, but the weight decay hyperparameter still varied according to the same schedule as the learning rate.
F.2 LEARNING RATE TRANSFER EXPERIMENTS
Our learning rate transfer experiments consist two phases. We start from an existing hyperparameter configuration and first sweep the global learning rate used across all parameter types, i.e., the input layer, hidden layers, output layer, biases and gains. This is done for the 1x scale model. We then freeze the learning rate of the input layer, biases and gains, changing only the learning rate for the layers that scale with the width, i.e., the hidden layer and output layer. This second base learning rate is scaled via jaP's rules before applying it in the optimizer, but the plots show the base rate before scaling (which is what transfers across scale). This second sweep gives the final performance shown in the learning rate transfer plots.
We adopt this approach because it better captures how well the learning rate scaling works by leaving the parameters is applied to unaffected. Compared sweeping single global learning rate across scales, this typically gives better performance with standard scaling but highlights the shift more. The reason is that without freezing the learning rate for the other layers, they can become unstable at the shifted (higher) base learning rates needed for the hidden layers to learn well. This gives worse overall results since no single global learning rate works well for all parameters anymore, but the optimal learning rates may shift less as result.
Instead of using the full Maximal Update Parameterization μP for training we combine standard parameterization with the learning rate scaling of μP as proposed by Everett et al. (2024). They find this simpler approach empirically works just as well or better. We note there are many variants of μP that achieve the same effects slightly differently, see for example Tables 3, 8 and 9 in Tensor Programs V (Yang et al. 2021). However, these are generally not equivalent with standard weight decay, which would complicate the use of full μP for experiments.
For transformer training the main differences between full μp and our experiments are the handling of the final output layer and the normalization factor applied in attention. The output layer is initialized using smaller standard deviation that scales with 1/C' rather than 1/VC like in standard parameterization. This results in an initial relative update size that is not scaled with the width compared the 1/ //mm scaling that occurs in the approach from Everett al. (0224). The other difference is that the scaling factor in the attention heads is 1/dnead rather than 1 doead- However we keep chead constant in our scaling experiments, opting to scale the number of heads instead. This difference could thus be absorbed into tunable factor that does not change across scales.
25

---

Published as a conference paper at ICLR 2026
Wr
SILU
AMSNorm
WVx
Attention (SDPA)
RMSNorm
W2
Atication Block
ML. Block (SwIGLU)
Figure 16: LLaMA transformer block diagram. The names of the matrices are used to refer to measurements in other figures. The learnable gains Yaun and "MLP demb Cemb-dimensional vectors.
F.3 LLAMA EXPERIMENT CONFIGURATION
Our LLaMA Touvron et 2023) experiments are performed using a modified version of Lin- gua (Videau et al. 2024). The networks have 20 transformer blocks, each consisting of one attention and one MLP sub-block (see Figure I6.. We vary the embedding dimension demb between 128 and 2048, which sometimes refer to Ix and 16x. The MLP block has expansion factor of exactly 3x giving exact scaling ratios for all layers. We used fixed attention head size of dhead = 128, varying the number of heads across scale. All linear layers initialized using standard deviation of 1/vC. We do not use embedding tying but the embeddings still initialized with standard deviation of 1/ L/Vád demb like the final FC layer (default behavior in Lingua).
The task is standard next token prediction pre-training using a cross-entropy loss. The dataset used DCLM (Li al. 2024), specifically 10% subset obtained by using the first global shard from the Hugging Face dataset. We use the llama3 tokenizer with a vocabulary size 128,256.
the PyTorch variant of the AdamW optimizer shown in Algorithm We vary the learning rate between runs but other hyperparameters are fixed at 31 = 0.9, B2 = 0.95, =010- and À= 0.1 (except when scaled through independent weight decay scaling). Weight decay is applied to all parameters except the gains. We train for 20,000 steps at a global batch size of 256 and sequence length of 4096 giving 1,048,576 (220) tokens in each batch and 20.97 billion tokens in total. This results in 19.23 tokens per non-embedding parameter in the largest configuration we use (embedding dim 2048). Unless otherwise noted, the learning rate schedule consists linear warmup from 0 to the specified peak learning rate, followed by a linear decay down to 0. The losses reported are the average of the last 100 training steps unless otherwise specified. This due to initial issues we had with the evaluation part of the code base.
Unless otherwise stated
10%
For an embedding dim of 2048 it takes roughly 14 hours to complete single run using H200 GPUs. This was not optimized and is slowed down by heavy logging of wide range of alignment related metries. Training was performed in bfloat 16 with mixed precision.
F DETAILS FOR FIGURE
This experiment follows our base LLaMA training and learning rate transfer setup F2 However these are more recent runs with working validation losses. For the weight decay runs, the learning rate for the input layer and gains is frozen at y = 1.6 10-2 based on an initial sweep of global learning rate at width 128 (1 x scale). For the no weight decay runs, this secondary learning rate is set to 77 = 6.4 x 10-2 The best loss values can be seen in Table |
Table 1: Minimum loss values for Figure Each cell reports the minimum validation loss with corresponding learning rate in parentheses.
Model Width
Independent WD Scaling
Standard WD Scaling
No WD
128
4.030 (8x 10-3)
4.032 (8x 10
4.037 (1.6x 10-2
256
3.607 (8x10-3)
3.597 (1.6x 10
3,634 (3.2x10
512
3.263 (1.6x10-3)
3.258 (3.2x10
3,283 (6.4x10 2)
1024
2.991 (8x10-1
2.989 (3.2x10 10-2
3,012 (6.4x10-1 2)
2048
2.801 (0-3)
2.798 10-4x10-2)
2,815 (6.4x10-3)
26

---

Published as a conference paper at ICLR 2026
F.3.2 DETAILS FOR FIGURE
This experiment shows layers. feed_forrrrrrddd w1. for a base learning rate y = 8 x 10 the optimal value for the 1x width in Figure]] I The learning rate for gains and the imput layer is 7 6.4 x 10-2 The metries are logged and plotted every 10 steps, the final 0.5% of training is excluded for numerical reasons (the changes go to zero with the learning rate). The representation based metrics are computed for 1/64th of batch (corresponding to miero-batch
the master rank). Other details follow the base setup described in F3 Additional layers can be seen in Figure 22 and Figure 23 compares the transfer of the absolute and relative representation changes.
F.3.3 DETAILS FOR FIGURE
This experiment shows layers. 13. weight decay 0.1. Note that this is the LR-WD product that gives the best results for standard scaling of the 16x configuration in Figure [ after applying the μP scaling to the LR. No juP scaling performed between the two configurations, the hyperparameters are exactly the same and the specified learning rate is applied all parameters. The metrics are logged and plotted every 10 steps, the final 0.5% of training is excluded for numerical reasons. The representation based metries are computed for 1/64th of a batch (corresponding one micro-batch for the alignment ratio of other layers. Other details follow the base setup described in F3 Similar be seen in Figure 24 for additional learning rates under independent
wl for a learning rate 4 x 10-3 and
the master rank). See Figure 21
alignment measurements weight decay scaling.
F.3.4 DETAILS FOR FIGURE
This experiment shows layers. 13. feeeeeeeeefeoee forward. w1 for learning rate 1) 4 x 10-3 (after juP scaling) and weight decay À = 0.1. This corresponds to the best configuration for standard weight decay scaling Figure and the independent scaling with the same LR-WD product. Metrics logged and displayed every 10 steps, the final 0.5% are excluded for numerical reasons, representation changes are computed on 1/64th of batch. Other details follow the base setup described in F3
F.3.5 DETAILS FOR FIGURE 6
This experiment follows our base setup for LLaMA training F.3 and learning rate transfer experiments F2 The learning rate for the gains and input layer is fixed at 1 1.6 x 10 2, The modified learning rate schedules are applied to all parameters, not only the width dependent ones. The scaling factor for exp50 and decayaway, way varies between widths as in pP, they have no effect the scale. The additional warmup factors applied on top of the existing 10% warmup and linear decay. For the longer 50% linear warmup, the run corresponding to y = 4 10 was repeated due to sudden divergence half way through, which did not occur in the repeated run with exactly the same seed and configuration. The loss values can be seen in Table 2 and are roughly comparable between methods. The exponential schedules perform marginally worse here but not extensively tuned.
Table 2: Minimum loss values for Figure 6 Each cell reports the minimum loss (average of last 100 train steps) for each method and the corresponding learning rate in parenthesis.
Model Width
Method
(128)
4x (512)
16x (2048)
Independent jP scaling + 10%
3.947 (8x 10-3)
3.232 (8x10-3)
2.792 (8x 10-3
No μP 10%
3,955 (8x10-3)
3,237 (4x10-3)
2.785 (4x10
No jaP + 50%
3.955 (1.6x10-9)
3.252 (4x10-3)
2.808 (4x10-3)
No μPP 10% Decayaway
3,953 (8x10-3)
3.239 (8x10-3)
2.800 (8~10-3)
No JP + 10% Exp50
3.953 (8x10-9)
3.245 (8x10-3)
2.808 (8x10-3)
27

---

Published as a conference paper at ICLR 2026
F.3.6 DETAILS FOR FIGURE 19
These experiments use the LionAR optimizer shown Algorithm The hyperparameters are 3 = 0.9, / = 0 (no Nesterov momentum). We follow the setup for learning rate transfer experiments from Section F2, i.e., we sweep global learning rate the narrowest width and then fix this while only changing the learning rate of the width-varying layers during the sweeps shown. The global learning rate was swept with the AdamW compatibility mapping the learning rate from Kosson et al. (2024a). This resulted in r 0.013 and 0.016 for 100 steps,
0.018 and 0.032 1,000 steps and 7/ = 0.013 and 7 = 0.032 for 20,00 000000 steps. The learning rate schedule is linear decay to zero with 10% linear warmup for the 1,000 and 20,000 step experiments. For the 100 step experiments no learning rate schedule is applied. The losses for the 100 and 1,000 step runs are averaged over 5 runs with different seeds.
is unclear why the widest network does not perform best for the 100 step training. We note that the validation loss is shown, perhaps the wider networks do not generalize well from the relatively few training examples it sees. could also be that the learning rates for the embedding and gains (which are tuned for the narrowet width) do not transfer well here.
F.4 RESNET EXPERIMENT CONFIURRATINN
Our ResNet He et al. (0166) experiments are performed using modified version of PyTorch Image Models (Wightman) 2019) All networks are 50 layers deep with bottleneck block configura- the residual stream
tion, specifically the original post-norm variant but with the downsampling performed on the 3x3 convolutions ("ResNet v1.5"). When scaling the width we scale all hidden dimensions of all layers evenly relative to the original network, going down to 0.25x width up to 4x. The 4x variant is fairly large for a convolutional network with roughly 384M parameters.
The training task is image classification using cross-entropy loss. We train on ImageNet- 1k (Rus sakovsky et al. 2015) for 100 epochs. Data augmentation consists of random cropping and random horizontal flips.
Optimization performed using the PyTorch variant Adam W. The learning rate varies between experiments. Other hyperparameters are 81 0.9, /3 0.999, independent weight decay scaling). Unless otherwise stated we use a epoch linear warmup from followed by cosine decay O. The batch size is 1024 spread over 8 GPUs without the of synchronized batch normalization during training.
10 10-8, X 0.1 (before
The training of the 4x variant takes roughly 10 hours to complete using 8 H200 GPUs. This was not optimized and is slowed down by heavy logging of a wide range of alignment related metrics. Training was performed in b with mixed precision.
28

---

Published as a conference paper at ICLR 2026
Weight Decay A
10
10-1
||A]|/ || [%]
||AY MYYIITI [%]]
X
2.0
2.0
2.815
Final Val Loss
1.5
2.810
1.0
1.0
2.805
0.5
0.5
0
2.800
0.0
0.0
10
10
10
0
10
20
10
20
Learning Rate 1
Step [10º]
Step [10º]
LR: 1.3e-4 / WD: 6.4e+0
LR: 2.5e-4 / WD: 3.2e+0
LR: 5.0e-4 WD: 1.6e+
1.0e-3/ WD: 8.0e-1
LR: 2.0e-3 / 4.0e-1
4.0e-3
WD: 2.0e-1
X Unstable
The scheduling effects of higher weight decay and lower learning rates. LLaMA experiments where the weight decay and learning rate are varied while their product kept con- stant. The effects on the final loss relatively small (left panel) and likely correspond to using different learning rate schedules which could alter the relative weight updates (middle) and rep- resentation changes (right) in a similar manner. For example lower weight decay values have an effect similar to using stronger or longer warmup schedules. Updates shown for a single layer layers. 13. feed feeddfeeeedeeeee forward.wl.
Figure 17:
G WEIGHT DECAY VS LEARNING RATE
Our framework in Equation 4 is expressed in terms of relative weight updates ||AW W]] Al- though this can applied directly by controlling the relative updates explicitly as done in Appendix H our main experimental setups with Adam W do not do this. Instead the initialization norm, learning rate and weight decay jointly modulate the size of the relative weight update over time in a complex manner (see Appendix D) The steady-state size of the relative weight updates is determined by the product y7À of the learning rate 77 and weight decay as described by Equation (this requires similar properties to hold exactly). This steady-state behavior dominates long
scale invariance training but it is still interesting to understand the secondary effects of varying the learning rate and weight decay while keeping their product fixed. We note that this was explored before in Kosson et al. (2024b) and our findings largely mirror theirs with a couple of new observations.
Experimental setup: We sweep different learning rate and weight decay pairs (17, X) in our standard LLaMA AdamW training setup for width 2048 (see Appendix(F.3). The product ya is kept constant the value that corresponds to the best configuration for independent weight decay scaling in Figure [ y = 8 10-4 The learning rate and weight decay values reported here are the actual values used in the optimizer, after any μP scaling or similar transformations. We only change the values for the width- midth--eeenndet weight matrices, the hyperparameters for other parameters like the embedding input layer and gains are kept constant.
The left panel of Figure 17 shows how the loss varies with the learning rate and weight decay when their product is fixed. We can see that the loss does vary slightly, but the changes are relatively small compared to changing the product ya significantly as happens in Figure | The independent weight decay scaling in Figure results in (n = (5 10 1.6) which is slightly suboptimal here. In contrast the best value for standard weight decay scaling is (17, A) (6.4 10-2 0.1) which has lower yjλ product and is thus not shown here, but note the significantly lower weight decay value and what this implies in light of the following observations.
Effect on the loss:
We believe the biggest impact of varying the weight decay and learning rate when their product is fixed are scheduling effects. These work similar to modifying the learning rate schedule which can be seen in the middle and right panels of Figure 17 Note in particular how higher weight decay values have warmup-like effect both the relative weight updates and relative
Scheduling Effects:
29

---

Published as a conference paper at ICLR 2026
representation changes was also discussed in Section 6 Despite including standard 10% linear warmup, the violet run with (77, X) = (4: (4.10- 1000, 0.2) has large updates at the start of training. Further increasing the learning rate results in even larger updates which are likely the cause of the unstable training we observe for those values. Kosson et al. 2024a) hypothesized that the primary benefit of learning rate warmup is to prevent these large initial representation changes.
We note that the scheduling effects are strongly influenced by the initialization norm. Increasing the initialization norm decreases the size of the initial relative updates similar manner as lowering the learning rate and increasing the weight decay. For a perfectly scale-invariant weight, for example when normalization layer immediately follows layer, the scheduling effects of varying the learning rate and the initialization norm become exactly the same (up to numerical limitations, optimizer considerations like the Adam, and so on).
Effects on the weight and representation norms: Using higher learning rates and lower weight decay values generally results in higher weight norms. For scale-invariant weight, Kosson et al. (2024b) predicts || - Vn/A for sufficiently long training with fixed hyperparameters. The representation norms are directly scaled by the weight norm (for given direction or weight alignment), but also affected by other operations such normalization layers and non-linearities. When learnable gains are used, the network has some freedom to control the representation norms independently of the weight norms. As a result weight decay does not necessarily directly constrain the representation norms. This is the case in our LLaMA experiments where there is one learnable gain in the RMSNorm at the start of each residual branch, both for attention and MLP blocks. As is common practice, we do not apply weight decay to the gains allowing them to be learned freely.
Gain RMS (Attn RMSNorm)
[i]lsss (Attn Wo)
Yillms (Attn Wo)
1.0
10-2
10a
0.5
0.0
|Y|lmss (Attn Wo)
||M (FF W₁)
|||lRMS (FF W2)
10a
10"
1.5
10a
1.0
10
0.5
10-1
0.0
20
0
10
20
0
10
20
Step [10?]
Step [10?]
Step [10
LR: 1 1.3e-4; WD: 6.4++0
LR: 2.5e-4; WD: 3.2e+0 -
LR: 5.0e-4; WD: 1.6u+0
LR: 1.0c-3; WD: 8.0c-1
LR: 2.0e-3; WD: 4.0c-1
LR: 4.0o-3; WD: 2.0c-1
Figure 18: The behavior of weight and representation norms for different weight decay values. The top row shows how the network uses the learnable gain (upper left) in the RMSNorm to counteract differences in the weight magnitude of the following layer (upper center) in order get query representations of a fixed magnitude (upper right). With only one gain per residual branch the network can not further control the output magnitudes of the residual branch (lower left). The MLP block behaves similar, the input magnitude to the SiLU non- non-linearity is roughly preserved (lower center) but the residual branch output magnitude is not (lower right). Metries shown for block 13.
30

---

Published as a conference paper at ICLR 2026
shows measurements of different weight and representation norms time during LLaMA training with different (77, A) values. We observe that the network can learn to use gains to counteract changes in the weight norm due to weight decay in way very similar to the rescaling transformations described in Section 2 This seems to happen with the attention softmax in particular. Here the gain (upper left panel) learns perfectly counteract changes in the weight norm of the query transform (upper center panel) resulting in a stable query magnitude (upper right panel). Altering the magnitude of the query and key representations would change the selectiveness of the attention detrimental way. similar to changing the softmax temperature. With only one gain per residual branch shared between the key, query, and value transformations, the network can further learn to counteract changes in the weight norm of the attention output transformation. As a result the output magnitude of the residual branch (lower left panel) strongly affected by weight decay.
Figure 18
potentially
We observe similar pattern in the MLP block though it is unclear if the network is using the gain to keep the input magnitude of the SiLU (bottom center panel) consistent or trying to balance the output magnitudes of the attention (lower left panel) and MLP blocks (lower right panel), Adding extra gain per residual branch or one per linear layer would allow the network to learn the representation norms directly, but we do not explore the effects potential benefits of this here.
Other effects: If the learning rate hyperparameter for weight matrices is also applied to other parameters like gains and biases, this becomes strong secondary effect of changing the weight decay as discussed in Kosson et al. (2024b). There could also be effects related to numerical precision. When using floats with wide exponent ranges, representations of different magnitudes can likely be encoded accurately. However, this is not the case for fixed point and narrower float formats which might have significant issues if this not compensated for.
31

---

Published as a conference paper at ICLR 2026
Algorithm 2 LionAR (Kosson et al. 2024a): A rotational version of the Lion optimizer (Chen ct al., 2023) that replaces weight decay with projections onto a sphere of fixed radius, controlling relative weight updates directly. This version does not include the AdamW learning rate compatibility mapping from Kosson et al. (2024a). making the relative updates directly proportional to the learning rate. Note that this makes the (relative) learning rate more sensitive than in AdamW W (it should be scaled more like the square root of the Adam learning rate with batch size or network width).
Require: Relative learning rate Tt, scalar learning rate "t. momentum B. Nesterov factor V 1: Initialize: Time step t +- 0, parameter vector O, ( (()), divided into sub-vectors at the granularity individual matrix rows, convolutional filters, embeddings, or biases/gains, momentum vector my
2: while stopping criteria not met
3:
t
+
4:
[g, ....,.h 9t P1] Mini-batch gradient w.r.t 0.-1, divided into sub- like 0
(o..)....
5:
for p E [l....,P]
6:
m. + 3m (n) + (1 "
7:
u;
(p) < -(1 (1-v)m, + vgg (p)
8:
if (()) E R is weight matrix row, embedding, or convolutional filter:
9:
/.0) -0- 7t-
|106 (
# &Scale update be prop to sphere radius
10:
0 + r
|1000)
# Project value back onto the sphere
11:
else:
12:
0 00 (?))( # Standard update without weight decay for gains and biases
H TRAINING WITH CONSTRAINED WEIGHT NORMS
As an alternative to weight decay, we can instead constrain the weight norms to their initialization values using projections after each step. Although this approach is still not very common in practice, it has shown promise for training diffusion models (Karras et al. 2024). in reinforcement learning (Lyle et al. 2024) and transformer language model training (Loshchilov et al. 2025). By doing this we can eliminate the more complex scheduling effects of weight decay and have the learning rate directly determine the relative weight update [WII/ W||-
We use a variant of the Lion optimizer (Chen et al. 2023) similar to LionAR (Kosson et al. (024a) but without the additional learning rate adjustments for AdamW compatibility, see Algorithm 2 In this setup the learning rate directly controls the size of the relative weight updates throughout training, there is no convergence to rotational equilibrium or similar. The weight magnitudes remain fixed throughout training. keeping the norms close to what the jP initialization strategy prescribes rather than letting them drift away as in standard training.
Figure 19 shows how the optimal learning rate stays roughly constant across width for longer runs. This directly contradicts JP's prediction for the scaling of the relative weight update size, see Equation 6 (which we emphasize is consistent with multiple prior μP works). This behavior can be explained by the changes in the alignment ratio as our framework predicts. Figure 20) compares how the relative representation change is preserved when μP scaling is applied vs μul scaling. We can see that due to the alignment ratio being roughly width independent for most of training, not applying any scaling better preserves the relative representation change. The exception is early on when the representation change can spike without jP, but here this is not detrimental enough to shift the optimal learning rate. Since no weight decay is applied we can not get an additional warmup effect from independent weight decay scaling, but exponential warmup strategies like those discussed in Section 7 could be applied if this becomes an issue for wider networks.
The LionAR experiments allow us to isolate the effects of the relative weight update without the other effects of weight decay (Appendix G). The findings are consistent with our interpretation of independent weight decay scaling, it helps by keeping the relative weight update later in training similar across network sizes in contrast to what jaP prescribes. The experiments also highlight the wide applicability of our framework by showing it holds for different optimizer (Lion) and different form of explicit weight constraints (on top of the weight decay and no weight decay settings discussed before). Finally training with constrained weight norms might be setting where our warmup interpretation of jP with independent weight decay has a direct practical application.
32

---

Published as a conference paper at ICLR 2026
100 Steps
1,000 Steps
20,000 Steps
7.2
5.00
Final Val Loss
4.75
4.0
7.0
4.50
3.5
6.8
4.25
4.00
3.0
6.6
10
10
10
10-2
Relative Learning Rate
Relative Learning Rate
Relative Learning Rate
Width 128
Width 256
Width 512
Width 1024
Width 2048
Min Loss
Figure The relative weight update size should roughly constant across width for longer runs rather than scaled as JuP prescribes. LLaMA experiments with the LionAR optimizer (Algorithm 2 where no μP--like scaling is applied to the relative learning rate across widths. We note that applying juP's prescribed scaling would be mathematically equivalent to shifting the curves to the right by (width/128) The weights are constrained to their initialization norms and no weight decay is applied. Note how there only minimal shift in the optimal learning rate across widths for the 1,000 and 20,000 step runs. In contrast the optimal learning rate shifts significantly for the 100 step run, similar to fPPs prediction from Equation 6 The prediction is that the relative weight update, and thus the relative learning rate, should decrease with the square root of the network width. See F.3.6 for experimental details.
| [%]
Cwwm Cww
[/W]///] [%]
With μP Scaling
2
fan Bon_in-128
2
fan (nn_inn22888 2048
2
JPP assump.
2
2.
2"
No uuP Scaling
2
fan fan_in=122 1n8 128 fan in-2048
2
assump.
10
15
20
5
10
15
20
0
5
10
15
20
Step [103]
Step [10*]
Step [10º]
Figure 20: Not scaling relative weight updates as μP prescribes better preserves relative representation changes for most of training. LLaMA experiments corresponding those shown in the right side of Figure| 19 Recall how the relative representation change |AY||/|| is the product of the alignment ratio a aaw /aw and relative weight update |AWI W [/| (see Equation 4). In the upper row we compare the metrics for a relative learning rate of 0.018 for width 128 and 0.0046 for width 2048, matching jPPs prescribed scaling of 1/vm = 1/4 from Equation 6 In the lower row do not apply any scaling using 0.018 for both widths. Note how the latter approach results in better transfer of the relative representation change |AY ||||||| aside from a very brief period at the start of training. Metrics shown for single layer layers 13. feed forward. wl.
33

---

Published as a conference paper at ICLR 2026
I
SUPPLEMENTARY FIGURES
Align. Ratio aowwaww
21
Emb. Dim
128
2
2048
2-1
Layer Type
attn wq
2-2
atth
2
3
attn
fiwl
Align. Ratio dowlaw
ff.w
ff
20
output
2-1
10
11
12
13
14
15
16
17
18
19
Output
Transformer
Figure 21: Halfway through training the alignment ratio of all layers is approximately one. The aligmment ratio for each layer in the LLaMA experiments shown in Figure measured step 10,000. The ratio is similar across widths, contrary to jP's assumptions. The values should be contrasted to jP's assumed alignment ratio of VC where C = demb for all layers except ff w2 where C' = 3d This would give values of roughly /2048 = 25.5 xs 45 for most orange markers. The gap between the blue and orange markers could also be compared to JP's predicted value of /2048/12 1188 = 4, where the orange marker should be higher.
34

---

Published as a conference paper at ICLR 2026
C.aw// Cly
2
Std. WD Scaling
10-2
10
10-1
27
10-3
Width
10
1x
10
16x
10-3
10-3
Ind. WD Scaling
10-2
2r
10-1
2
10
Width
10-4
10
10
15
20
15
20
15
20
Step [10?]
Step [10?]
Step [10?]
Layer
0,attn wq
D.attn.wv
0.fl.11
0.ft.
0. a.attnn wk
D.attn.wo
D.ff.22
aaw Cwy
//
Std. WD Scaling
10
10-2
10
10
Width
10-1
10-3
10-5
2
Ind. WD Scaling
10
10-2
10-1
10-1
2
Width
10-4
1x
10
16x
10-5
20
20
Step [10?)
Step (10?)
Step (10?)
Layer
19 199.attn.ww
19.attn.wv
19.ffwl
19.ffw3
19 attn wk
19.attn.wo
19.ffw2
output
Figure 22: Comparing the transfer of relative representation changes across layers. An extension of Figure 3 showing the behavior for additional layers in LLaMA, specifically those in the first transformer block (upper half of figure) and last transformer block (lower half of figure). The relative representation change (RRC) transfers well under independent weight decay scaling for almost every layer, unlike standard scaling. The worst RRC transfer for independent scaling occurs in 0. attnt where the alignment ratio for the narrower network stabilizes at value notably lower than for the wide network for unknown reasons. Compared to Figure 3, each line is further downsampled 100 roughly equally spaced points without averaging.
35

---

Published as a conference paper at ICLR 2026
|
||A IRMS
Std. WD Scaling
10
10
10
Width
10
10
16x
10-1
Ind. WD Scaling
10-2
2
10-1
Width
10
10
10-5
Step [10*)
Step [10*)
Layer
0.attn.wq
D.attn.wv
0.ftw1
0.f1.
0.attn.wk
0.attn.wo
0.ffw
|
|AY|sMM
Std. WD Scaling
10
10
10
Width
10
1x
16x
10-1
Ind. WD Scaling
10
10-1
10
Width
10
10
10-5
20.0
Step [103]
Step [10?]
Layer
19. wq
19 attn.wv wv
19.ff.w1
19.ff.w3
19.attn.wk
19.attn.wo
19.ff.w2
output
Figure 23: Comparison of relative and absolute representation changes. An extension of Figure comparing the transfer of the relative vs absolute representation change across multiple layers in LLaMA, specifically those in the first transformer block (upper half of figure) and last block (lower half of figure). The relative representation change transfers ||Y [AY|I/|X ||| well for independent weight decay scaling as shown before in Figure 22 Neither metric transfers for standard weight decay scaling. The absolute representation change |AY||RMS does not normalize for the magnitude of the inputs the layer |X]] the size of the weights |[]] For this reason does not isolate the contribution of single layer and the transfer characteristics can vary across layers depending on where they are located relative to normalization layers and gains as argued in Section 2.3 Note the poor transfer for 0. ff. w2, 19.attn. w0 and 19. ff.. w2 in particular, even for independent scaling where the LR transfers well (these are residual branch output layers). Compared to Figure each line is further downsampled to 100 roughly equally spaced points without averaging.
36

---

Published as a conference paper at ICLR 2026
Width
Width 512
Width 2048
2
Weight Align. aw
2-4
2.
29
Update Align. dow
2-2
2-
2
Align. Ratio dowlaw
24
22
2
10
15
20
10
15
20
0
10
15
20
Step [10*]
Step [10*]
Step [10*)
4.0e
IPP
Base 2.0e-03
8.De-03
Figure 24: Alignment behavior across widths and learning rates. Measurements of the weight alignment, update alignment and alignment ratio for LLaMA training for different model widths and learning rates. Independent weight decay scaling is used, the runs correspond to those shown in the left panel Figure where the optimal base learning rate Thase As 8 10-3 Note how higher base learning rates result in faster changes in the alignment ratio. This is expected as meaningful updates to the network are needed to change
behavior (for learning rate zero alignment would never change). Other details follow those for Figure 4 in Appendix F.3.3 except the lines are further downsampled to 200 roughly equally spaced points without averaging.
37


Reviews:
[{"review_id": "MYAjdrKnpN", "reviewer": "Reviewer_wA1t", "summary": "The paper studies the role of weight decay in learning rate transfer. In particular, the paper argues, both empirically and theoretically, that independent weight decay is crucial to learning rate transfer, i.e. the value of \\eta * \\lambda in PyTorch's AdamW implementation, should be fixed for different model widths. The key role of the independent weight decay is about controlling the relative update size of the feature. According the MuP's assumption at model initialization, the relative update size should scale with model widths, however the authors point out that as training proceeds, the assumption on independency between the  layer inputs and gradients, would break, and the relative update should become a constant value with respect to the model widths, which can be achieved by scaling the weight decay together with model width.\n\nThe authors then answers a couple of natural questions that arise, such as why would the assumption break and why we still need MuP scaling of learning rate anyway?\n\nThe paper conducts large scale empirical experiments on LLM pre-training under extensive hyperparameter sweep. \n\nOverall I find the paper's claims and analysis convincing; the arguments and analysis angles from the paper can inform future works on hyperparameter transfer.", "questions": "- Line 114, in the caption, should it be \"... with a fixed absolute size |\\Delta W|\" rather than \"|W|\"?\n\n- While I do undersand the argument intuitively, can the author clarify more on \"... while a given relative change ∥∆W∥/∥W∥ always has the same impact ...\", is \"same impact\" measured by the norm of \\Delta Y, or is it measured by the value of Y?\n\n- Could the fact that MuP with independent weight decay is needed because we want to restrict the absolute size of both the weights and the activations? I imagine under low precision LLM pre-training, we certainly want to weight norm and the activation norm. In particular, I believe the activation norm should depends on weight norm and the widths, and in order to prevent it from blowing up, we want to control the norm of W. Especially from the middle column in Fig.6, it seems that the consequence of not using MuP + independent weight decay is mostly on training stablity rather than the optimal learning rate starts to shift.\n\n- If \\eta * \\lambda should be kept constant and \\eta should scale with model width, do we have to scale \\eta ~ 1 / m? Can we e.g. \\eta as \\eta_base / \\sqrt(m) or \\eta_base / m^p ?\n\n- One question that I feel unanswered by the paper is: How should we pick the value of \\eta * \\lambda? The paper's experiments show that, if we keep \\eta * \\lambda fixed, then optimal \\eta_base becomes transferable, but how should we choose the value of \\lambda or the value of \\eta * \\lambda?\n\n- It seems that there are two phase changes happening during the training, the first one is when the alignment ratio goes from width dependent to a value closer to 1, the second one is when the model entering the \"equilibrium\" and Eq.3 starts to hold, are these two phase changes happening at the same time?\n\n- Why is the norm used for Frobenius norm rather than, e.g. Spectral norm?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The obesrvations and the arguments from the submission are not surprising, MuP's scalnig relies on lots of assumptions that don't hold and are unverified in later phases of the training, so it is not surprising that things will break. However, this paper is the first work, as far as I am concerned, that shows rigorously which ingredient in the MuP scaling breaks and why that is the case.\n\n- The paper addresses most of the questions that naturally arise with the arguments, e.g. how and why the MuP assumptions break or why we still need MuP scaling of learning rate in practice.\n\n- Although the paper involves lots of non-trivial technical details, I personally find the paper easy to follow, and the presentation of the results very clear.\n\n- The experiments are fairly large-scale and convincing.", "weaknesses": "- It is unclear when the phase change will happen, and how does the phase changing will happen, does it have anything to do with the model size, the model width, etc.\n\n- The paper does not seem to provide lots of new practical takeaways; the main conclusion seems to be that: One should use decoupled weight decay, which is already explicitly pointed out by many recent works.\n\n- The paper seems only to argue that if this \\eta * \\lambda is **not fixed**, then learning rate transfer would break, i.e Eq.4 would start to depend on model widths. However, there is no direct argument on why learning rate becomes transferable as we fix \\eta * \\lambda. \n\n- The paper is mostly around how to choose \\eta, but does not show how to choose the value of \\eta * \\lambda or how to choose the value of \\lambda.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a0EcivPGFo", "reviewer": "Reviewer_rEs9", "summary": "The paper studies learning-rate transfer under the Maximal Update Parameterization ($\\mu$P) and the role of weight decay. $\\mu$P scales learning rates to keep feature updates stable across widths, supporting hyperparameter transfer from small to large models. The paper observes that $\\mu$P’s theoretical alignment assumptions only hold early in training, especially in large-batch settings common in practice.\n\nEmpirically, the authors show that independent weight decay, where the product of learning rate times weight decay $\\lambda \\eta$ is scale-invariant, is essential for stable learning-rate transfer in large models, while standard weight decay fails to provide transfer. \n\nThe authors further demonstrate that the primary effect of $\\mu$P in practice is similar to an implicit learning-rate warm-up. They show that explicit warm-up strategies can partly replicate the benefits of $\\mu$P.", "questions": "How is $\\Delta Y / Y$ computed? At the same time step?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Understanding why the learning-rate transfer works under $\\mu$P, and what happens under $\\mu$P scaling over long training horizons, are certainly interesting and of fundamental importance. In this sense, the experiments on the alignment ratio are a valid contribution, showing that no matter how weight decay is scaled (Figure 3, middle plot), the alignment ratio has a very weak dependence on the width.\n\n- The experiments are very well documented in the appendix, where there are details for each Figure.\n\n- The experiments on achieving good transfer when the width scaling is incorporated only in the warm-up phase are intriguing. It seems to suggest that $\\mu$P scaling is mainly required close to initialization (although there is increased instability, probably due to the fact that the updates explode with width after the warm-up phase). These warm-up experiments isolating early-training effects suggest that $\\mu$P’s impact may be concentrated near initialization.", "weaknesses": "A central narrative of the paper is that independent weight decay “contradicts” $\\mu$P by overriding its scaling. Quoting from the paper: *“This fundamentally differs from $\\mu$P’s prescribed scaling in Equation 6, meaning that independent weight decay scaling contradicts $\\mu$P by eventually overriding its update scaling.”*\n\nMy interpretation differs: under *standard* weight-decay scaling, the contribution of weight decay vanishes as width grows (as also noted in [1]: *“However, the implied EMA timescale now changes with model size, while we hypothesize that the optimal timescale should not vary with model size.”*), which breaks width-invariance of feature updates — a core $\\mu$P premise. From this perspective, independent weight decay does not contradict $\\mu$P, but rather restores its intended property that the components of the feature updates remain width-invariant. Under standard weight decay, weight-decay’s contribution vanishes in the large-width limit (this can be shown via Tensor-Programs-style calculations), which, in my view, explains why the optimal learning rate shifts to the right in Figure 1 (middle plot): increasing the learning rate partially compensates the vanishing contribution of weight decay.\n\nThat said, the paper raises an important possibility: over long training horizons (e.g., when total steps scale with width), the training trajectory may move sufficiently far from initialization that $\\mu$P’s assumptions no longer hold. This is a compelling direction. However, the conceptual leap from “assumptions break late in training” to “learning-rate transfer fails” would benefit from further justification and clearer separation from the effect of vanishing weight decay. In fact, the paper’s experiments seem to show deviations from the $\\mu$P regime under both independent and standard weight-decay scaling (as observed in the alignment ratio plots, Figure 3 middle). To me, this suggests instead that the vanishing contribution of weight decay with width is the cause of poor learning-rate transfer. This does not contradict $muP$'s philosophy of maintaining scale-invariant contributions to the feature updates.\n\n\n**Clarification request**\n\nIs Figure 4 conducted under a setup that actually uses $\\mu$P? In the appendix, it is written: *“the hyperparameters are exactly the same and the specified learning rate is applied to all parameters”*, which seems to suggest that **no $\\mu$P scaling** was applied. If so, how can this be used to claim $\\mu$P assumptions break, if $\\mu$P was not used? I would appreciate a clarification.\n\n**Warm-up discussion**\n\nIn light of the above discussion, the result that learning-rate warm-up alone cannot replace $\\mu$P (Figure 6 left) is not surprising. This is because the exploding updates with width cause the optimal learning rate to shift left.\n\n**Overall**\n\nThis paper contains valuable experiments, but I believe the narrative requires restructuring. In particular, I would focus more on the investigation of $\\mu$P over long training horizons and the role of weight decay.\n\n\n[1] Xi Wang, Laurence Aitchison, How to set AdamW's weight decay as you scale model and dataset size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "58Cv4zsx1b", "reviewer": "Reviewer_x5KA", "summary": "The paper studies Maximal Update Parameterization ($\\mu$P), a learning rate scaling for transfer between different model widths, and its relation to weight decay. To do this, the authors develop a framework based on relative updates (i.e., the size of updates proportional to the weights). They use this to show that independent weight decay effectively counteracts $\\mu$P scaling, which counterintuitively leads to an improvement in transfer. They show that this is because the alignment assumptions of $\\mu$P do not hold as training progresses in practice. However, because of this, independent weight decay stabilizes feature learning by counteracting $\\mu$P and maintaining the same relative representation changes across network widths. The authors show that early in training, $\\mu$P provides a benefit by acting as an effective learning rate warmup and support this by showing that a stronger learning rate warmup can replace $\\mu$P’s learning rate scaling with similar results.", "questions": "How is update or weight alignment related to data eigenvector alignment (if it is at all)? I’m wondering if the “silent alignment” effect (Atanasov et al., 2022) is related to the alignment and muP “warmup” discussed here or if these are separate concepts.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper studies $\\mu$P scaling and weight decay, a subject of significant practical relevance for hyperparameter transfer. The authors identify why $\\mu$P scaling works with independent weight scaling, diverging from the reasons assumed by the theory used to develop it. The framework, approach, and results appear original. The paper supports its claims with theory and solid experiments using LLaMa and ResNet. The paper is well-written, clear, and well-motivated; the overall presentation of the paper is excellent.", "weaknesses": "As someone who is less familiar with this area, I found it somewhat challenging to follow and get an intuition for the framework being introduced—in particular, comparing and contrasting between update alignment, weight alignment, relative weight updates, alignment ratio, and relative representation. The summaries and discussion helped with getting an overall understanding, but it would have been helpful to perhaps use less jargon or to give more intuition for the different terms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9d9eEaVjEw", "reviewer": "Reviewer_aDn7", "summary": "The paper discusses a previously known fact that $\\mu$P, which is commonly thought to give learning rate transfer between different model widths, actually does not unless paired with independent weight decay.\nAuthors highlight that $\\mu$P's alignment assumptions, when viewed in terms of relative updates, are incompatible with IndepWD. This can be reconciled by the fact that ultimately $\\mu$P's alignment assumptions are not correct.\nAuthors show empirically that IndepWD enables transfer of 'relative representational changes', whereas $\\mu$P does not.\nFinally authors propose particular learning rate warmup schedules as alternatives to $\\mu$P, and show that these are comparable to $\\mu$P.", "questions": "- How did you pick which weight to show for Figures 3, and 4? The alignment ratio is almost exactly 1 for this weight, but it doesn't seem to hold in general based on Figure 16/Appendix G, and thus I'm not sure how valid the approximation for $\\alpha_{\\Delta W}/\\alpha_W$ is (in section 5.2).\n- Have you thought about $\\mu$P/IndepWD/RRC during fine-tuning? In particular with LoRA, and how your observations interact with work like [1]\n- I would be interested to see how the RRC plots behave for different optimizers?\n\n[1]: Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. \"Lora+: Efficient low rank adaptation of large models.\" _arXiv preprint arXiv:2402.12354_ (2024).\n\nMinor points:\n\n- Is there a typo in line 105/106? 'high update alignment means that $w_k$ and $x_b$ tend to point in similar directions'\n- What is $\\eta_\\tau$ in Eq. (13)?\n- 'scaling effect' should be defined in Figure 5's caption, or at the very least, the right panel should be explicitly described.\n\n\nI would generally be in favour of acceptance if the weaknesses and my questions above can be addressed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "To my knowledge, emphasising the role of _relative_  representational changes in learning-rate transfer is novel, as is the perspective that weight decay, rather than the learning rate itself, governs successful transfer. The observation that $\\mu$P effectively acts as a learning-rate warm-up schedule is particularly insightful, as is the clarification that $\\mu$P conflicts with the usual recommendation to keep $\\eta\\lambda = \\text{const}$.\n\nThe paper convincingly demonstrates how $\\mu$P’s infinite-width assumptions fail in practice—since realistic networks do not maintain a small batch-size-to-width ratio—and consequently why $\\mu$P alignment can break down.\n\nThe authors illustrate their results in both transformers and resnets.\n\nI think the observations in this paper are of great interest to those training neural networks.", "weaknesses": "It's not immediately obvious what this paper's contributions are, versus what already existed. I would appreciate the standard bulleted list at the end of the introduction to declare contributions unambiguously. I feel this would improve the clarity, and highlight the novelty of the paper.\n\nI would like to see more justification that RRC are what we should be interested in, as opposed to absolute norms of representations. As far as I can tell, there are not any plots for this. It's also unclear if transfer of the RRC appears for all weights, or just the one shown (from layer 13), possibly a figure to illustrate many/all layers should be included.\n\nI would like to see some tables with performance metrics. Some Tables in the Appendix confirming that optimal performance is similar across different settings ($\\mu$P+IndepWD, vs. $\\mu$P) would help, as well as performance figures to compare $\\mu$P to the proposed learning rate schedules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)