A thorough reproduction and evaluation of $\mu$P
Authors: Georgios Vlassis
David Belius
Volodymyr Fomichov
Venue: Accepted by TMLR ()

Abstract: This paper is an independent empirical reproduction of the claimed benefits of the $\mu$P parametrization proposed in \citet{yang2020feature} and \citet{yang2021tuning}. Under the so-called Standard Parametrization (SP), the weights of neural networks are initialized from the Gaussian distribution with variance scaling as the inverse of ``fan-in'', with the learning rate being the same for every layer. While this guarantees that (pre)activations are $\mathcal{O}(1)$ at initialization with respect to width, it causes their scale to be width-dependent during training. To address this, \citet{yang2020feature} and \citet{yang2021tuning} proposed the Maximal Update Parametrization ($\mu$P), which is also claimed to make the optimal value of various hyperparameters independent of width. However, despite its alleged benefits, $\mu$P has not gained much traction among practitioners. Possibly, this could stem from a lack of thorough independent evaluation of $\mu$P against SP. We address this by independently reproducing the empirical claims of the original works. At the same time, we substantially increase the scale of the experiments, by training $16000$ neural networks of sizes from $500$ to $1$B parameters, and empirically investigate $\mu$P's effect on outputs, gradient updates, weights, training loss and validation loss. We find that generally $\mu$P indeed delivers on its promises, even though this does not always translate to improved generalization.

Published in Transactions on Machine Learning Research (01/2025)
A thorough reproduction and evaluation of μP
Georgios Vlassis
golassis@elha.ch
D-ITET ETH Zurich
Volodymyr Fomichov
Faculty of Mathematics and Computer Science
UniDistance
David Belius
delid, Wwnddistance..dih
Paculty of Mathematics and Computer Science
UniDistance
Reviewed OpenReview: https: opeeeeve//penrevieeuu net/ forum? id= AFaEdJuQcp
Abstract
This paper is independent empirical reproduction of the claimed benefits of the pP parametrization proposed in Yang & Hu (2020) and Yang et al. (2021). Under the called Standard Parametrization (SP), the weights of neural networks are initialized from the Gaussian distribution with variance scaling as the inverse of "fan-in", with the learning rate being the same for every layer. While this guarantees that (pre)activations are O(1) at initialization with respect width, causes their scale to be width-deeendent during training. To address this, Yang & Hu (2020) and Yang et al. (2021) proposed the Maximal Update Parametrization (nP), which is also claimed make the optimal value of various hyperparameters independent of width. However, despite its alleged benefits, μP has not gained much traction among practitioners. Possibly, this could stem from a lack of thorough independent evaluation of μP against SP. We address this by independently reproducing the empirical claims of the original works. At the same time, we substantially increase the scale of the experiments, by training 16000 neural networks sizes from 500 to 1B parameters, and empirically investigate jP's effect on outputs, gradient updates, weights, training loss and validation loss. We find that generally μP indeed delivers on its promises, even though this does not always translate to improved generalization.
1 Introduction
1.1 Related works
Deep learning rescarchers and practitioners have long understood the importance of initialization and its relation to width. The work LeCun et. al.
2002) advocated that weights be sampled from a distribution with
mean zero and standard deviation Vian-in
(LeCun initialization). Glorot Bengio (2010) shed further light
why this is helpful, and Sutskever et al.
2013
showed that initialization schemes like this synergize
with momentum methods.
Subsequent theoretical work (Jacot el al. 2018 networks with these initialization schemes leads to so called "lazy training" in the infinite-width limit. Specifically, in that limit they converge to kernel machines. These limiting kernels do not depend on the dataset all (that is, they are not learned). This means that the infinite-wide limit, feature learning becomes impossible, possibly hurting performance.
Chizal et al., 2019) has demonstrated that using neural

---

Published in Transactions on Machine Learning Research (01/2025)
Motivated by the desire to escape this "lazy training trap", Yang & Hu (2020) introduced parametrization
that is supposed to maintain the ability to learn features at seffnnttewwidi) Starting from the desideratum
that (pre)activations be (1) throughout training, and using the theory developed in the Tensor Programs
(TP) scries of papers ( (Yang) 2019a b 2020a Yang & Littwin 2021 Yang 2020b; Yang & Hu 2020 Littwin
& Yang 2022 Yang et al.] 2023a 2021: 2023bj Yang Hu 2020 they arrive at the jP parametrization
For many hyperparameters, /&P is also claimed to stabilize optimal values as width varies, a property that is
exploited in Yang al. (2021) for hyperparameter optimization. In this paradigm, called p'Transfer, optimal hyperparameters are discovered cheaply for small, proxy network, and then zero-shot transferred to a big, target network. Lastly, Yang et al. (2021) claim that wider networks should in general have lower training loss under sP,, which is not necessarily true under SP.
Since its proposal, pP has been used in a limited number of published works. For the case of Large Language
Models (LLMs), has been used by Dey et al. (2023a) (Cerebras-GPT)) Li et al. (2023) (FLM--01BB)
Dey et al. (023bb (BTLM-3B-SK), Liu et al. (2023) (rystalCoder)), Hu al.) (2024) (MiniCPM) and Li
et al. 2024)) (Tele-FLM). Intriguingly AAchamm et al. (0233 (GPT-4) includes Yang & Littwin (2021 in the
references without explicitly citing it, leaving it unclear if it uses pP not. Outside of the LLM world, uP
was used in Cabannes et al. (2023) to ensure that fixed learning rate was rcasonable for every width they tested, and in Beaini et al.
2023 which included μP in their GNN library Graphium targeted at molecular
learning.
1.2 Objectives
The above works using μP nearly always assume its benefits, taking at face value that uP is preferable over SP, without ablating with respect to the paramctrization. Besides the original papers (Yang & Hu,
2020 Yang et al.) 2021), the only work that investigates the claimed advantages of jP over SP is Lingle (2024). It studies whether uP indeed stabilizes the optimal learning rate for many architectural variations of transformer, and finds that does for most but not all of these variations.
In this paper, we will thoroughly investigate the alleged benefits of μP and compare it head-to-head with SP. We expand the scale of the existing μP versus SP comparisons (Yang & Hu 2020 Yang et al.) 2021; Lingle 2024), by including additional architectures and domains, scaling to narrower and wider networks, performing a denser hyperparameter sweep, training for more random seeds and training for longer. In total, we train 15760 networks, ranging from 500 to 1B parameters. Our ultimate goal is to understand whether and to what extent the promises of μP hold in practice, and if and when should be preferred over SP.
Our work is fundamentally an independent reproduction of Yang & Hu (2020) and Yang et al. (2021). Hence, made every effort that results reproducible themselves. The complete repository of the training codeP is already available online.
1.3 Findings
Our findings can be summarized as follows:
1. Inspecting the norm of coordinate- wise network outputs reveals that they indeed are O(1) under jP, while heavily depending on width under SP. Similarly, the norms of coordinate-wise gradient updates and weights decay exponentially with width under both SP and uP.
2. For pP, and unlike SP, the best (with respect to the training loss) learning rate indoed stays ap- prozimately constant width increases. Thus, TTannserr, in contrast to "naive" hyperparameter tuning with SP, indeed enables zero-shot hyperparameter transfer, from narrow (and thus cheaply trainable) networks to wider ones.
Under μP, wider networks in general outperform (in training loss) narrower networks. Under SP this trend is much less visible, although sometimes present.
A parametrization simply a prescription according to which the weights network are initialized and updated. https://ggtthut con www.gy/assiss//nnttt
2

---

Published in Transactions on Machine Learning Research (01/2025)
4. Points 2 and do not always translate to better generalization. That is, the optimal pP network often has worse validation loss than the optimal SP network.
5. With SP, we observed some wide networks diverging. Specifically, the wider the network, the more likely it was to diverge. In contrast, almost none of the networks diverged with p.P.
The benefits of μP seem to be more evident for transformers.
In summary, we found that μP mostly performs as expected in terms stability of optimal learning rates, but does often leads to worse generalization.
2 IPP summary
We start with high-level summary of the Tensor Programs framework Yang 2019a
In this framework, the initial weights and learning rates of a neural network are scaled in terms of a parameter matrix's "fan-in" and "fan-out". Their precise meaning for different types of layers are as follows:
1. The parameter matrix of biases of linear layer has fan-in = 1, and fan-out equal to the activation dimension.
The parameters of a convolutional layer are viewed as a collection input_ channels x output channels matrices (where fan-in = input channels and fan-out = output. channels). The number of such matrices for layer kernel width kernel height.
3. Biases and weights of layer normalization layers are treated the same biases of linear layers.
4. The class embedding of Vision Transformers (ViTs) has fan-in and fan-out = d, where d is the model dimension.
5. The embedding operation of a transformer is viewed as a matrix multiplication between the embed- ding table and a one-hot vector representing token of the vocabulary. Therefore, the embedding table has fan-in vocabulary__. size and fan-out d.
With these conventions, assume faan-innfaa-oon parameter matrix of a neural network. Under SP, the initialization and update rules are:
0u ~ (() fan-in) if fan-in >>
if fan-in = 1,
(1)
0.1 + 02 k I(VO.),
(2)
for function f, where μ, e and k hyperparameters that do not scale with width. Note that 0 is possible (e.g. biases are often initialized to zero). Different choices of f lead to different optimizers (e.g. for f= id we recover SGD, while another choice leads to Adam).
Under the nP, the initialization and update rules are instead:
0o (N(,ee if fan-in = 1,
(3)
(0 52) if fan-in > 1,
0.+1 O f(V0.),
(4)
where s and y are scaled with width as specified in Table addition, the scale a self-attention layer of dimension should be changed from + to 1.
The constants e and do not have to match between SP and pP. Moreover, they can be chosen arbitrarily for every parameter matrix of the network. This allows us to make SP and μP exactly equivalent for a base width. We can do so by insorting width-independent constants front of the μ, and of jP. The constants be inserted are obtained from equating the initializations (equation and equation 3 and the update rules (equation and equation 4
3

---

Published in Transactions on Machine Learning Research (01/2025)
Table Standard deviation and learning rate scaling in 1dP
fan-out + 00
fan-in, fan-out
fan-in +
8
V/ fan-out
1/Van-in
1/la-in
y (SGD)
1
iaa--nn
7 (Adam)
1
1/fan-in
1/tan-in
3
Experimental setup
We experimented with four architectures, across five datasets. Specifically, we tested 3-layer ML. the California Housing and the MNIST datascts, a VGG11 CNN and VIT on CIFAR-10, and transformer Tiny Shakespeare and WikiText--03.
For every architecture we chose a base width, and then trained networks of widths Ç x base_widtle while varying Ç. We ran comprehensive experiments for each architocture and datasct combination. For every combination, we picked multipliers to make SP and jP exactly equivalent for the base width <= (as described in the previous section). We swept the learning rate hyperparameter k, training 16 networks for every value.
For our MLP California Housing, we followed Yang et al. 2021 Figure 5) in plotting the norm of coordinate-wise outputs to test uP's stabilizing effect on them. We also did the same for weights and gradient updates.
In all settings, we compared performance for different hyperparameter values at varying width, producing curves like those of Yang et al. (2021) Figure 1). Specifically, we collected the minimum training and valida- tion losses, and plotted their mean, along with one standard deviation error bars for both paramelrizations.
For all the experiments, we set the initialization scale to 1/10 and used the Adam optimizer (Kingma & Ba, 2017) with PyTorch's 's defaults. Additionally, we trained without weight-decay or data augmentation.
In lotal, we trained 15760 neural networks, spanning from 500 to 1B parameters, which needed 3200 hours when using an NVIDIA A100 3
3.1 MLP on California Housing
The California Housing dataset (Pace & Barry 1997] is a tabular regression dataset with the goal of predict- ing the median house value for geographical block California from eight real-valued features. It consists of 20640 samples, out of which we held out 2000 for validation and 2000 for testing.
We used an MLP with two hidden layers, and gave them base width of 16. We trained networks cor- responding to width multipliers from < 1 (width = 16, parameters = 433) to ( 512 (width = 8192, parameters = 67M). For each width we trained with 16 different learning rate multipliers k geometrically spaced between 10-5 and 1. Each training run consisted of 50000 mini-batches of size 16.
Overall, we trained 5120 MLPs, which took around 200 hours on an NVIDIA A100.
3.2 MLP on MNIST
The MNIST dataset (LeCun et al. 1998) is an image classification dataset with ten classes, one for each handwritten digit. contains 70000 greyscale images, of size 28 28. We held out 10000 images for validation and 10000 for testing.
The images were flattened to 784-dimensional vectors and passed to MLPs with two hidden layers with base width 16. Similarly to in Section 3.1 used an MLP with two hidden layers and base width 16. We varied from 1 (width 16, parameters 13K) to Ç = 256 (width = 4096, parameters 20M). The remaining training details are the same as in Section 3.1 with the difference that here we used 20000
3 At. the time of publication, this would cost. on the order 130008 on AWS Amazon
2024
4

---

Published in Transactions on Machine Learning Research (01/2025)
Overall, we trained 4608 MLPs, which needed about 100 GPU hours.
3.3 VGG11 on CIFAR-10
The CIFAR-10 dataset (Kizhevsky et al. 2009) is an image classification dataset where one tries to classify an image in one of ten classes. There are 60000 images, of size 3 x 32 x 32. We held out 10000 images for validation and 10000 for testing.
We used the VGG11 architecture (Simonyan & Zisserman 2014] with four convolutional stages. The stages had base widtl 4, 16 and 32 respectively. The classifier head had base width 20, and 0.5 dropout probability. We tested networks from
(max channels = 32, parameters = 21K) to Ç = 128 (max _chnneels = 4096, parameters = 336M). We tried eight geometrically spaced values for the learn- ing rate multiplier k, between 6 10-6 and 0.01. Each training run consisted of 50000 mini-batches, of size 32.
In aggregate, we trained 2048 CNNs, for about 500 GPU hours.
3.4 VIT CIFAR-10
We used the ViT architecture (Dosovitskiy et al. 2020 with patch size of four and six blocks of base width
32, eight heads, expansion factor of one and 0.1 dropout probability. For positional embeddings we used sinusoidal positional encodings. We tested networks from <= 1 (width = 32, parameters = 34K) to Ç = 128 (width- = 4096, paramcters = 504M). The remaining training details follow Scction 3.3.
In total, we trained 2048 ViTs, which took 1000 GPU hours.
3.5 Transformer Tiny Shakespeare
The Tiny Shakespeare dataset (Karpathy 2015) is a subset of Shakespeare's works in single 40000 lines file. Language models trained from scratch on this dataset can produce samples that look very close to the original. We tokenized the dataset with the GPT-2 (Radford et al. 2019) tokenizer, leading to 300K tokens. We held out 25K tokens for validation and 25K tokens for testing.
We used the transformer architecture Vaswani et al. (0177) with a context of 128 tokens and six blocks of base width 32, eight heads, expansion factor of four and no dropout. For positional embeddings we used simusoidal positional encodings. We tested networks from <= 1 (width 32, parameters 3.3M) to (= 32 (width 1024, parameters 180M). We tried eight geometrically spaced values for the learning rate multiplier k. between 6 10-1 and 0.1. Each training run consisted of 20000 mini-batches of size 32.
Overall, we trained 1536 transformers, for approximately 300 hours.
3.6 Transformer on WikiText-103
The 100 dataset (Merity et al., 2016) is collection of 1.8M verified Good and Featured articles from Wikipedia. We held out 4K articles for validation and 4.5K for testing. We tokenized the dataset with the GPT-2 (Radford et al. 2019) tokenizer, leading to 114M tokens.
The architecture used was scaled-up version of Section 3.5 with the context doubled to 256 tokens, twelve transformer blocks with base width 144 and twelve attention heads. We varied Ç from ( = 1= (width = 144, parameters = 17M) to Ç = 16 (width = 2304, parameters = 997M). We tried ten geometrically spaced values for the learning rate multiplier k, between 10-5 and 0.1. Each training run consisted of 8000 mini-batches, of size 512 (1B tokens).
Because each training run is much more computationally expensive than in the previous scttings, we trained four networks (instead of 16) for every value of k. Moreover, as is the norm in such settings, used B2 = 0.95 for Adam.
4The width convolational layer is simply the number of its output channels.
5

---

Published in Transactions on Machine Learning Research (01/2025)
Overall, we trained 400 transformers, in 1100 hours.
4 Main results
4.1
MLP on California Housing
4.1.1 Scale of activations
As our first experiment, we measured the average coordinate-wiss norm of the output of our MLP architecture California Housing, described in Section 3.1 We did this for width multipliers from <= to Ç 512 and for twelve batches. We then compared SP with uP to see the impact of parametrization. According to theory, outputs should be width-dependent under SP, and width-independent under pP. The results are presented in Figure [ For SP, we can see that the scale of the outputs rapidly increases as we increase the width. On the contrary, for μP, the norm is stable with respect to the width. The results are as expected, and mirror Yang al. 2022 Figure 5).
We did the same for the gradient updates and the weight norms of the last hidden layer of our MLP, According to thcoretical claims in Yang & Hu (2020), the coordinate-wise norms of the gradient updates and the weights, under both SP and μP, decay exponentially with width. Again in Figure [ we confirm these claims. Interestingly, the gradient update curves are more stable under μP, with small spikes appearing for nearly all batches under SP.
batch=0
batch=3
batch=6
batch=9
batch=12
sp.out
sp.grad mean
sp.data mean
11
0.8
comn
1
40
0.6
0.4
20
0.5
0.2
0
mn
10a
10a
10a
10a
10a
10a
10
10a
10a
C
mup.ont
mup.grad mean
myp myp.data mean
10
1
0.5
0.5
0.5
0
10a
10
102
109
10a
10a
101
10a
Ç
Figure 1: Scale of network outputs (left), gradient updates (middle) and weights (right) function of the width multiplier Ç. μP stabilizes the scale of the outputs with respect to width.
4.1.2 Train and validation loss
The results for the stability of the hyperparameters with changing width are shown in Figure 2 We observe that the training loss curves for both SP and μP are quite noisy, with the error bars for different widths overlapping. This indicates that in some cases the benefits of μP are detectable only when averaging over many training runs, highlighting the need for many random seeds.
Under SP, the optimal learning rate multiplier with respect the training loss shifts around an order of magnitude to the left as the width increases. On the other hand, it stays approximately constant under jaP.

---

Published in Transactions on Machine Learning Research (01/2025)
Moreover, under jP, the curves somewhat flatter, which means that the networks are less sensitive to the exact value of k.
For SP, wider networks do not consistently outperform narrower ones in terms of training loss, except for a small range low values of k, and the difference is slight. Meanwhile, this trend much stronger for JaP, and observed for a wider range of The validation loss curves show similar behavior, but are less noisy.
Comparing best performing networks with respect to the training loss, [ 0.66), while the best uP network has Ç 512, & = 3 10-4 and min _tt_ttraiinng loss 6.78 10 2 R 0.62). Thus, for SP the third widest network performs the best, while, consistently with theory, the widest network performs the best for uP. With respect to the validation loss, the best networks have min_val _i__all_osss = 0.47 (R² = 0.54) for pP. Hence, pP has worse best performing network in terms of training loss in comparison to SP, and a slightly better, but comparable, one in terms of the validation loss.
that the best SP network
has Ç = 128, k = 10 and min training loss 6.52 10
8, k = 10-4 and min val loss = 0.48
(R² = 0.55) for SP and Ç = 512, = 0.1 and
(=1
(=2 -(=4 ( (=
( 32 ( ( 128
= 256
Ç 512
sp
sp
10-0.5
100.2
min train loss
min val loss
10
10
10-0.2
10
10-3
10-1
10
10-3
10-1
k
mup
mup
10-0.5
min train loss
min val loss
10
10 1011
10-0.5
10-5
10-3
10-1
10-5
10-3
10-1
&
k
Figure MLP on California Housing. The y-axes correspond MSE loss, and the r-axes correspond the learning rate multiplier The solid curves show the mean, and the shaded regions + one standard deviation. Albeit the plots are quite noisy, we both that pP stabilizes the best k's, and that enables wider networks more consistently outperform their narrow counterparts.
4.2 MLP on MNIST
Training the same architecture as in Section 4.1 on MNIST, we obtained Figure 3
The noise level it significantly lower than in Figure 2 though the standard deviation the train loss is still quite large.
CConfidence intervals the minimum training and validation losses for all the settings he found in Table

---

Published in Transactions on Machine Learning Research (01/2025)
The best learning rate multiplier & in terms of training loss shifts around half an order of magnitude to the left. for SP as the width increases. Contrary to theory, it shifts about the same, but to the right, for μP. This was actually the only setting in which we observed the optimum significantly shifting for pP, We suspect that this is due to the narrower widths being subject to finite size effects (such as the non-vanishing variance of the network output, discussed in Section D.2 of Yang et al. (202T)). However, in terms of validation loss, SP, while indeed staying constant for jaP.
the optimum lt shifts approximately two orders of magnitude Even for SP, we that wider networks tend to outperform their narrower counterparts, both in terms of training and in terms of validation loss (with the exception of Ç 256). This trend nevertheless stronger for j.P.
With respect to the training loss, the optimal SP network has Ç = 256, k = 10 and min training loss = 0 (acc. = 99.06%) and the optimal uP network has (= 256,
10 3 and min _it____rainng____ loss = 3 10
(acc. 98. 25%). With respect the validation loss, the best performing SP network has (= 256, 3-10
and min val loss 2.94-10-2 (acc. 96. 96.88% for SP and Ç < 128, 10-3 and min_ loss 3.41-10-2
(acc. 96.69%) for μP. Therefore, in this setting, SP marginally perforrs μP in terms of both training (contrary to theory) and validation losses.
(=1 2 5= - (= 16 5
32
( 64 ( 128
256
sp
10a
mnin train loss
val. losss
10"
10
10
10
10-5
10
10
10-5
10-3
10-1
k
mup
mup
10
miil Irain loss
val. loss
10
10
10-1
10
10-5
10-3
10-1
10-5
10-3
10-1
k
Figure 3: MLP on MNIST. The plots show mean (solid curves) =1 std. dev. (shaded regions) of cross entropy loss versus the learning rate multiplier k. In this setting μP does not stabilize the best k's as much as in our other exporiments, but it does enable wider networks more consistently outperform narrow ones. The flat regions are due to clipping values below 10-7
4.3 VGG11 CIFAR-10
The results for the VGG1L architecture on CIEAR-10 are shown in Figure
As in Figure 2 the loss curves are quite noisy. Unlike Figure 2 here the noise levels of validation and training loss curves similar.

---

Published in Transactions on Machine Learning Research (01/2025)
Under SP, the best learning rate multiplier with respect to the training loss shifts around half order of magnitude to the left as the width increases. On the other hand, it stays roughly constant under μPP For SP, wider networks consistently outperform narrower ones in terms of training loss only for k I 10-4. Mcanwhile, consistently with theory, for μP this trend is observed for every k. As for the validation loss curves, they are very similar to the ones for the training loss.
Comparing best performing networks with respect the training loss, we see that the optimal network for SP has
128, & 6 10-5 and min training loss 2.61 10-5 (acc. 99.50%), while uP has - 64, k = 3 10 and min training loss 1.27 - (acc. 99. 99)). Hence, as in Section 4.2 and in contrast to thcoretical predictions, it is actually not the widest network that performs the best for μPP With respect to the validation loss, the best performing network has 128, k = 6-10- and min val loss 0.74 10 -3 and min min_val vaa__alll__ loss = 0.76 (acc. = 76.94%) for pPP In summary, even though the differences are small, the best performing SP networks outperform the best performing jP networks terms of both training and validation loss.
(acc. = 77.66%) for SP and Ç = 128,
(=
== 1=16 (= 32
16
128
sp
10a
min train loss
mn1 val loss
10
10
10-7
10
5
10
10
3
10
2
10
k
mup
mup
10a
Coenn crann 1100
min train loss
mn1 val loss
10-3
100
10-7
1.1
10
5
10"
4
10
3
10
10
10
10
10
k
Figure VGGI1 on CIFAR-10. We plot cross entropy loss vs learning rate multiplier k. Under μP, the best k's do not shift and wider networks perform better.
4.4 ViT on CIFAR-10
The results for the VIT architecture CIFAR-10 are shown in Figure
The error bars are much tighter than in Figure 2 and Figure both for the training and for the validation loss curves.
Under SP, we see that the best learning rate multiplier & with respect the training loss shifts around two orders of magnitude to the left as the width increases. On the other hand, it stays almost constant under nP.

---

Published in Transactions on Machine Learning Research (01/2025)
For SP, for some k, we can wider networks outperforming narrower ones in terms of training loss. However, this can only be observed for small range of learning rates close to the smallest we tried, and the difference is slight. Meanwhile, for pP the loss is roughly monotonically decreasing in width for a larger range of k, centered on the (approximately width-independent)) optimum.
Comparing best performing networks with respect to the training loss, the best SP network was the second widest, with Ç <= 64, k 3 10-8 and min training loss 5.62 10-3 (acc. 98.20%), while for uP it was obtained for the third widest, with Ç = 32, k = 3 10-3 and min training loss 0.01 (ace. 97.41%). Hence, though the μP networks exhibit better stability than the SP networks in terms of best learning rate, as well as higher monotonicity of the train loss relative to width, the best SP network in fact slightly outperforms the best pP in terms of training loss by half an order of magnitude.
The shape of the validation loss curves is qualitatively similar to the training loss curves, with the notable exception of the validation loss curve of the widest yP network, which has peculiar peal
Furthermore, for SP and contrary to theory also for μP wider networks perform worse in terms of validation loss. With respect to the validation loss, the best performing SP network had Ç 4, k 10 and min_val loss = 1.07 (acc. 62.56%), and the best performing μP network had Ç = 4, k = 3 10-3 and min_ val_ _lsss = 1.09 (acc. = 61.87%). Hence, also in terms of validation loss, the best SP network outperforms the best μP network.
Another interesting observation that, as seen Table 2 some SP networks diverged during training. Specifically, one network diverged for Ç 32, two networks diverged for Ç 64 and five networks diverged for < 128. By contrast, no μP networks diverged. The pattern suggests that SP networks become increasingly unstable as we increase the width, while pP networks are more stable, consistently with the theory behind μP. Eight networks is tiny mumber compared to the 1280 total networks we trained, so this could go unnoticed had the scale of our experiments been smaller, However, this could prove crucial for the training of extremely big networks.
4.5
Transformer on Tiny Shakespeare
The results for the transformer the Tiny Shakespeare dataset are shown in Figure
The training curves are very similar to those reported for a transformer language model in Yang et al. (2021 Figure 1). Moreover, we notice that the training and validation curves have significantly higher standard deviation for SP.
Under SP, we see that the best learning rate multiplier & with respect the training loss shifts around two orders of magnitude to the left the width increases. On the other hand, stays almost constant under μP, Furthermore, like in Figure 2 under μP the curves are flatter, meaning that the networks are less sensitive to the value of k. For small range of i wider SP networks outperform narrower ones in terms of training loss, while (as predicted by thcory) for μP this behavior is much more consistent, for almost every k.
Quantitatively, the best network for SP was obtained for ( = 32,
3- 10-4 4 with min twwininnnnnaaano loss = 0.17 (perplexity = 1.18), while for uP it was obtained for Ç = 32, k = 6 10-3 with min training loss = 0.18 (perplexity 1.20). Hence, terms of training loss, the best SP network somewhat outperformed the best μP network.
The validation loss curves are qualitatively similar to their training loss counterparts. For SP, the best network in terms of validation loss was obtained for Ç 16, k = 10 with min_ val loss = 96.54 (perplexity = 4.06), while for μP it was obtained for Ç = 32, k = 6 10-3 with min val loss = 4.72 (perplexity = 112.16). Hence, in terms of validation loss, the best SP network moderately outperformed the best uP network.
We could not clarify the origin of this peak. To rule bug the implementation we carefully reviewed the code and reran the experiment for this width, producing the same results.
10

---

Published in Transactions on Machine Learning Research (01/2025)
8
16
<= 32
64
128
100.1
min train loss
109
min vall loss
10-1
100.2
10
100
10
5
10
10
3
10
10
10
10
3
10
mup
mup
minn train loss
100
min val loss
100.2
10
100.1
10
10-3
10
10
10
10-3
10-4
10
10
Figure 5: VIT on CIFAR-10. We again plot the mean (solid) + std. dev. (shaded) of cross entropy loss vs learning rate multiplier Broadly speaking, uP fixes the best k's and leads to wider networks mostly outperforming narrower ones, consistently with theory.
Moreover, we again in Table 2 that (precisely 74) networks diverged under SP. Similarly to Sec- tion 4.4 we observed that, the wider the network, the more likely it is to diverge. For uP the other hand no networks diverged.
4.6 Transformer on WikiText-100 103
Lastly, the results for the transformer on the WikiText-103 dataset are shown in Figure
The loss curves very similar to those in Section 4.5 but exhibit even less noise. This could be explained by the significantly bigger batch size (512 vs 32), which reduces the stochasticity of the training trajectory.
We see that for SP the best learning multiplier & with respect to the training loss decreases roughly an order of magnitude as we make the architecture wider. In contrast, μP stabilizes this optimum. Moreover, under μP the trend of wider networks outperforming narrower ones in terms of training loss stronger, even though it can also been observed for k < 10-3 under SP.
In terms of training loss, the best SP network has Ç 16, 6 10-4 with min training loss 1.71 (perplexity 5.52), and the best μP network has 16, lt 6 10-3 with min training loss 1.40 (perplexity = 4.06). Thus, this is the only setting in which jP significantly outperformed SP. This also holds for the validation loss, where the best SP network has (= = 8, k = 10-4 with min val loss 2.99 (perplexity = 19.88), and the best jP network has < = 16, & = 6 10-4 with min val loss = 1.40 (perplexity = 19.10).
In Table we see that here some (precisely four) pP networks also diverged, although their number was still much lower than the SP networks (namely twelve) which diverged.
11

---

Published in Transactions on Machine Learning Research (01/2025)
<=1
(
(=8
Ç
sp
sp
train loss
min val loss
100.5
10"
100.7
10
1
10-2
10
10
10
10
10
mup
mup
100.8
min train loss
min vll losss
100.75
10
100.7
10-4
10-3
10-2
10
10
10
10
10-1
Figure 6: Transformer Tiny Shakespeare. The y-axes show cross entropy loss, and the 2-axes show the learning rate multiplier k. Under SP, the best k's shift significantly, and wider networks do not always lead to better performance. The results for uP are better in both regards.
5 Summary
Our most important empirical results are summarized in Table 2 but worse than SP in four of the six settings, significantly worse than SP in setting, and significantly better than SP in one setting. Thus, the claim that μP encourages feature learning and therefore better generalization than SP is not consistently supported by our experiments.
We that μP performed similarly
Table 2: Summary of our results, showing mean minimum train and validation losses + one standard de- viation. For the majority of cases, the standard deviation is relatively low, indicating that we can indeed draw conclusions from the mean of the minimum training and validation losses.
Setting
Parametrization
min train loss
min val loss
Networks diverged
SP
(6.52 + 0.12) 10-2
0.48 + 0.01
0
MLP on California Housing
μP
(6.78 - 0.10) 10-2
0.47 0.01
0
0.00 0.00
(2.94 0.37) -
MLP MNIST
μP
(3 (3 10-6
10-
(3.41 1 0.36) 10-2
0
SP
:.15) 10
0.74 0.01 0.76 0.02
0
VGG11 on CIFAR-10
μP
(1.27 +: 0.25) 10 10
SP
(6.62 1.66) 10
1.07 + 0.01
8
VII on CIFAR-10
μP
0.01 = 0.00
1.09 0.01
0
SP
0.17 0.02
4.57 1.022
74
Transformer Tiny Shakespeare
uP
0.18 0.02
4.72 1.001
0
SP
0.03
0.01
12
Transformer on WikIText-100
jP
1.40 0.06
2.95 0.01
12

---

Published in Transactions on Machine Learning Research (01/2025)
(=2 (=
( (=8
sp
train loss
min mi loss
100.8
109.6
100.6
10-6 5
10 2 10
6
10
3
10
mup
mup
100.8
train loss
min. val Joss
100.5
100.6
100.4
cin cime Lime
Limn LIME LIME Linnt
10-5 10-4 10-3 10-2 10-1
10-5
10-4 10-3 10-2 10-1
Transformer on WikiText-103. We plot mean (solid) +1 std. dev. (shaded) of cross entropy loss versus the learning rate multiplier k. These are the cleanest results: we clearly see that the best k stays almost constant under μP, and that wide networks (mostly) outperform narrow ones.
Figure 7:
6 Conclusion
This paper is head-to-head comparison between SP and pP. We independently reproduced the empirical claims of Yang Hu 2020) and Yang et al. (021), while at the same time significantly increasing the scale of the experiments. We confirm that μP indeed has a number of benefits over SP, even though one might not observe all of them in every setup. In general, μP stabilizes the optimal learning rate as a function of width and makes wider networks outperform narrower ones. Furthermore, alleviates divergence issues. However, in terms of both train and validation error, the best μP network is often worse than the best SP network.
While our results do confirm that 4'Transfer is viable paradigm for hyperparameter optimization, the alternative is not necessarily tuning the hyperparameters of the biggest network directly. In practise, for SP, it more common to search for hyperparameterss by training the biggest network for fewer iterations. would be interesting to compare that protocol with narrow-to-wide μ TTaansfer for the same compute budget.
Since μP is theoretically well-founded and empirically has consistent stabilizing effect, it merits further investigation. In particular, future rescarch should investigate under what circumstances μP networks gen- eralize better than their SP counterparts, and whether uP can be adapted to more consistently outperform SP in terms of generalization.
Acknowledgments
Calculations were performed at sciCORE (attp://scicore unibas. ch/) scientific computing center University of Basel. We grateful for the support of Prof. Jiri Cerny of University of Basel.
13

---

Published in Transactions on Machine Learning Research (01/2025)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Nge Akkaya, Florencia Teoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.0877 4.7.7774.. 2023.
Amazon. Amazon EC2 P4d Instances AWS mnstance-tyessppl 2024. [Accessed 15- 15-12-2024).
aws.amnazon.com. https://aws. https://aws.amazzon.co wwt.:aaaazoo.coooooo com/ec2/
Dominique Beaini, Shenyang Huang, Joao Alex Cunha, Gabriela. Moisescu-Pareje Oleksandr Dymov, Samuel Maddrell-Mander, Callum McLean, Frederik Wenkel, Luis Müller, Jama Hussein Mohamud, et al. Towards foundational models for molecular learning on large-scale multi-task datasets. arXiv:2310.0420 2023.
Vivien Cabannes, Elvis Dohmatob, and Alberto Bietti. Associative memories with heavy-tailed data. Tn NewrIPS 2023 Workshop Heavy Tails in Machine Learning, 2023.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
Nolan Dey, Gurprect Gosal, Hemant Khachanc, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hes- tness, et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv:2304.00 033008 2023a.
Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Robert Myers, Jacob Robert Steeves, Natalia Vassilieva, et al. Btlm-3b-8k: 7b parameter performance in a 3b parameter model. arXiv:2309.11 11568, 2023b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010. 11929, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249- 256. JMLR Workshop and Conference Proceedings, 2010.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv:2404.06306 2024.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Andrej Karpathy. The unreasonable effectiveness of recurrent neural networks, 2015. URL https:
M/karpathy.githiee
Accessed: 2024-06-26.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXic:1412.000 6980, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009, URL https://wwk. cs. toronto.
features--erttte--ereeeeeeeeeeee TR.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-50. Springer, 2002.
Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, et al. Flm-101b: An open Ilm and how to train it with $100 k budget. ar Xrx::22309.0 03852, 2023.
Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zihan Wang, Yu Zhao, Xin Wang. Yuyao Huang, et al. Tele-flm technical report. arXiv:2404.166 16645, 2024.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Lucas Lingle. A large-scale exploration of p-transfer. arXiu:2404.05728, 2024.
Etai Littwin and Greg Yang. Adaptive optimization in the oo-width limit. In The Eleventh International Conference on Learning Representations, 2022.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yugi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source 1lms. arXiv:2312.0663 06.500 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv:1609.078: 07843, 2016.
Robert Kelley Pace and Ronald Barry, Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291-297, 1997.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409. 1556, 2014.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, pp. 1139- 1147. PMLR, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097, 2021.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv:1902.0 0eee0477600 760, 2019a.
Grog Yang. Wide feedforward or recurrent neural networks of any architecture are Gaussian processes. Advances in Newral Information Processing Systems, 32, 2019b.
Greg Yang. Tensor programs II: Neural tangent kernel for any architecture. arXiv:2006.145/6 2020a.
Greg Yang. Tensor programs III: Noural matrix laws. arXiv:2000.1000 10685, 2020b.
Greg Yang and Edward J Hu. Feature learning in infinito-width neural networks. NNN:2011.14422 2020.
Greg Yang and Etai Littwin. Tensor programs IIb: Architectural universality neural tangent kernel training dynamics. In International Conference on Machine Learning, pp. 11762-11772. PMLR, 2021.
Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learning. 21X.v:2310.17711 2023a.
Greg Yang, Dingli Yu, Chen Zhu, and Soufianc Hayou. Feature learning in infinite depth neural networks. In The T Twelfth International Conference Leurning Represenlations, 2023b.
15


Reviews:
[{"review_id": "rnNojt8xXX", "reviewer": "Reviewer_XHzd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The writing is overall clear, with a well-defined scope and objective.\n- The empirical study is comprehensive and the authors provide clear set of findings, added by details like the number of GPU hours and other relevant experimental setups for reproducibility.\n- The findings on generalization and the benefits of $\\mu$P, particularly in the context of Transformers, are timely and relevant to current research trends.\n\nWeaknesses: \n- While the scope is quite limited, the current version also has some similarities to a technical report in terms of presentation. I think it'd strengthen the paper if authors can highlight/discuss why the observed effects occur in each section Or if such behaviour are indeed theoretically expected. I see answers to these question hinted in only a few cases. I think the authors could provide more thorough discussions, analysis/implications and conclusions from the results as the current paper lacks these.\n\nQuestions:\n- Section 4.1.2: I'm not sure if one may conclude that SP outperforms $\\mu$P in terms of training loss since the observed difference is quite small and curves overlap significantly. Wouldn’t it be better to just interpret these results as $\\mu$P exhibiting greater stability in wider networks, rather than claiming that one clearly outperforms other in terms of training loss?\n- Section 4.3: I don't understand how the validation loss curve is reminiscent of double descent. The double descent is generally observed across increasing capacity of the network. However, I can't conclude that clearly from Figure 4. Can the authors clarify that?\n- One of the findings mentioned $\\mu$Transfer enables zero-shot hyperparameter transfer, from narrow networks to wider ones. Have the authors explored $\\mu$Transfer experimentally across models/datasets? For example, applying transfer from VGG to ResNet or using ViT on CIFAR-10 as a base setup for training bigger ViT on ImageNet?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides an in-depth study on the advantages of Maximal Update Parametrization ($\\mu$P) compared to Standard Parametrization (SP). The authors start by providing a background on $\\mu$P. They then conduct an extensive set of empirical experiments to identify scenarios where $\\mu$P should (or not) be preferred over SP. The findings highlight that $\\mu$P offers more stability as the width increases, making it a robust choice for scaling. However, it is also observed that $\\mu$P underperforms relative to SP in general settings. Additionally, the study reveals that SP, while effective in several cases, diverges on very wide networks. These insights provide comprehensive comparison on the practical trade-offs between the two methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors could possibly consider using a log scale for all the train/val loss plots to improve readability to better highlight the differences, especially for smaller values.\n- Maybe make the figure captions self-contained by providing clear descriptions of what the reader should expect or infer from the figures.\n- In Figure 2, some curves are still not clearly visible because of shaded regions. While the color choices show trends, increasing the transparency of the shaded areas could further improve clarity and visibility.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pWmTyryJZF", "reviewer": "Reviewer_8XW8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe goal of thorough and independent empirical evaluation or strong theoretical findings is an important and potentially very impactful one. The paper scales up evaluation of $\\mu P$ to networks ranging from 500M to 0.5B parameters, which is a wide enough range to provide insightful information. The authors also make sure to present results that are comparable between $\\mu P$ and $SP$. Interpretations and actionable insights from this kind of evaluation would be very valuable to practitioners.\n\nWeaknesses:\n\n- I found the results from this study difficult to interpret. For instance, the authors report training and validation losses in all their settings, to make conclusions about the effective performance in different settings. However, it is difficult to compare and interpret absolute values of losses. The significance of loss values like 0.17 vs. 0.18 (section 4.4) is unclear.\n- Since the contribution of this work is an empirical evaluation, similar to a benchmarking study, I find the numbers of architectures (4) and datasets (1 per architecture) to be very low. Evaluating performance differences in such few settings is likely to lead to high-variance conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to conduct a thorough and head-to-head comparison between the standard parameterization under LeCun initialization of neural networks and the more recently proposed $\\mu P$ parameterization, which has been theoretically shown confer several benefits to training but seems to have gained little traction in practice. The authors independently reproduce the empirical claims of the original works that proposed this scheme, scaled up the experiments for more diverse evaluation and presented their findings. Notably, their results indicate that several claims made about optimal learning rates, output norms, gradient norms and weight norms are well-founded and empirically verifiable. However, the authors also present evidence to suggest that these claims do not necessarily lead to better generalization.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Typo: I believe the authors intended to write \"500M to 0.5B parameters\" in the abstract and section 1.2.\n- Are the authors able to report evaluation metrics that are more interpretable, for example, accuracies, etc. on the tasks that the models are trained on?\n- If the authors can present results on a wider variety of datasets in each setting, their claims about the overall performance differences between the two parameterizations would be better supported.\n- What actionable insights can readers and/or practitioners derive from the experiments of this paper?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RjmkDRDiRy", "reviewer": "Reviewer_T9E4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The question addressed by this work—probing the real utility of µP, given its limited use in published works—is significant.\n2. The paper is clearly presented and supported by convincing results.\n\nLimitations:\n1. The experiments focus only on small datasets, e.g., CIFAR-10 and the 4000-line Tiny Shakespeare dataset. These may not truly reflect the results on larger-scale datasets, such as those used for LLM pretraining. Additionally, the largest model considered in the paper is relatively small, with only 0.5B parameters.\n2. No new methods or insights are provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an independent empirical reproduction of the claimed benefits of the µP parametrization. The experiments are extensive, ranging from 500 to 0.5B parameters, with evaluations on over 10k neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be better to report the test accuracy for CIFAR-10 training.\n2. Include the base learning rate and explain how it was determined.\n3. Conduct large-scale training experiments, e.g., using at least a 1B-parameter model and 1B tokens.\n4. It is better to give some practical guidelines for improving the SP/µP training.", "broader_impact_concerns": "No ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an empirical study comparing Standard Parametrization (SP) and μP across settings. Reviewer recommendations are mixed (1 leaning accept, 2 leaning reject); for some reviewers the authors have NOT addressed many key concerns through their revision:

Strengths:

- Clear experimental methodology and comprehensive evaluation within scope
- Significant computational investment (3200 A100 hours)
- Added interpretable metrics (accuracy) and new experimental settings
- Tempered claims to match empirical evidence
- Important practical insights about μP's benefits and limitations

Remaining Concerns:

- Some performance differences are small and hard to interpret definitively
- Some theoretical explanations for observed behaviors could be strengthened

Given TMLR's criteria emphasizing thorough empirical evaluation and reproducibility, and considering the authors' careful revisions and responses, I believe this paper makes a valuable contribution despite its limitations. The work provides important practical insights about μP's applicability while maintaining appropriate scope and claims supported by evidence.

Recommendation: Accept with minor revision to address remaining clarity issues in result interpretation and theoretical discussion.
Decision: Accept (poster)