Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Authors: Greg Yang
Edward J Hu
Igor Babuschkin
Szymon Sidor
Xiaodong Liu
David Farhi
Nick Ryder
Jakub Pachocki
Weizhu Chen
Jianfeng Gao
Venue: NeurIPS 2021 Poster (2021)
Keywords: hyperparameter tuning
scaling law
transformer
language model pretraining
infinite-width neural networks

Abstract: Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters.
We show that, in the recently discovered Maximal Update Parametrization ($\mu$P), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call *$\mu$Transfer*: parametrize the target model in $\mu$P, tune the HP indirectly on a smaller model, and *zero-shot transfer* them to the full-sized model, i.e., without directly tuning the latter at all.
We verify $\mu$Transfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup. See arxiv.org for the full, up-to-date version of this work.

Tensor Programs V:
Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Greg Yang** x
Edward J. Hu
Igor Babuschkin
Szymon Sidor Weizhu Chen OpenAl
Xiaodong Liu Jianfeng Gao*
David Farhi
Nick Ryder
Jakub Pachocki
Microsoft Corporation
Abstract
Hyperparameter (HP) tuning in deep learning is expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, the recently discovered Maximal Update Parametrization (μP), many optimal HPs remain stable even model size changes. This leads to new HP tuning paradigm we call pTransfer: parametrize the target model in paP, tune the HP indirectly on a smaller model, and zero-shot trunsfer them to the full-sized model, i.e., without directly tuning the latter all. We verify p'rransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, outperform published numbers of BERT-large (350M parameters), with total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at at-githe github. wwt/miocooooooooooooooooooooooot crosof t/mup
Introduction
7.0
6.5
Hyperparameter (HP) tuning is critical to deep learning. Poorly chosen HPs result in subpar performance and training instability. Many pub- lished baselines are hard to compare to another due to varying degrees of HP tuning. These issues are exacerbated when training ex- tremely large deep learning models, since state- of-the-art networks with billions of parameters become prohibitively expensive tune.
Midth
Tranino
2040
1080
3.5
$152
stable
ug..eamiisgaat
lggutoaminghate
Figure Training loss against learning rate on Transformers of varying dmodel trained with Adam. Conventionally and in contrast with our technique, different widths do not share the same optimal hy- perparameter; wider networks do not always per- form better than narrower ones; in fact they under- perform the same-wide width networks in our technique even after tuning learning rate. See Sections and 4 for experimental setup.
Recently, 45 showed that different neural net- work parametrizations induce different infinite- width limits and proposed the Maximal Update Parametrization (abbreviated μP) (summarized in Table 3) that enables "maximal" feature learn- ing in the limit. Intuitively, it ensures that each layer is updated on the same order during train- ing regardless of width [3) In contrast, while the standard parametrization (SP) ensures activations are of unit order at initialization, actually causes
Work done partly during Microsoft Al Residency Program
" "Equal contribution. Order random. Correspondence to (gregyang, edvaardhu)@siccroofftt 2 "eee arxiv. org for the full, up-to-date version of this work.
ii... the updates" effect on activations becomes roughly independent of width in the large width limit,
35th Conference Neural Information Processing Systems (NeurIPS 2021).

---

Algorithm 1 Tuning a Large Target Model via pTrrnnsfrr
1: Parametrize target model in Maximal Update Parametrization (jPP)
2: Tune a smaller version (in width and/or depth) of target model
3: Copy tuned hyperparameters to target model
Table 1:
Hy pyperparametrss That Can Be T'aansffrre,, Not yTrranferred, or pTransferred Across, with few caveats discussed in [Section 5.1 * means empirically validated only on Trans- formers. while all others additionally have theoretical justification.
Not j'Transferable
i iTTransferred Across
optimization related, init, parameter multipliers, etc
regularization
width, depth*. batch size*, training time*, seq length
(dropout, weight decay, etc)
them to blow up in wide models during training [5 essentially due an imbalance of per-layer learning rate (also see Fig. 8). We leverage jPP to zero-shol transfer HPs from small models to large models in this work that is, obtain near optimal HPss large model without directly tuning it at all! While practitioners have always guessed HPs of large models from those of small models, the results are hit-or-miss at best because of incorrect parametrization. For example, as shown in Fig. I] in Transformer, the optimal learning rate is stable with width in jP (right) but far from so in standard parametrization (left). In addition to width, we empirically verify that, with a few caveats, HPs can also be transferred across depth (in (Section 5.1) as well batch size, language model sequence length, and training time (in Appendix 1.2.T). This reduces the tuning problem of an (arbitrarily) large model to that of (fixed-sized) small model. Our overall procedure, which we call p'Transfer, is summarized in Algorithm T and Fig. 2
and the HPs we cover summarized in
Tables and
There are several benefits to our approach: 1. Better Per- formance: j'Transfer is not just about predicting how the optimal learning rate scales in SP. In general, we expect the I'Transferred model to outperform its SP counterpart with learning rate optimally tuned. For example, this is the case
$55555
Standard
in Fig. with the width-8199 Transformer. We discuss the reason for this in Appendices B and C. 2. Speedup: It pro- vides massive speedup to the tuning of large models. For example, we are able to outperform published numbers of (350M) BERT-large [9] purely by zero-shot HP transfer, with tuning cost approximately equal to BERT-large pre- training. Likewise, we outperform the published numbers of the 6.7B GPT-3 model 66) with tuning cost being only 7% of total pretraining cost. For models on this scale, HP tuning not feasible at all without our approach. 3. Tune Once for Whole Family: For any fixed family of models with varying width and depth (such as the BERT family or the GPT-3 family), we only need to tune a single small model and can reuse its HPs for all models in the family Fl For example, we will use this technique to tune BERT- (1 10M parameters) and BERT-large (350M parameters) simultaneously by transferring from a 13M model. 4. Better Compute Utilization: While large model training needs to be distributed across many GPUs, the small model tuning can happen on individual GPUs, greatly increasing the level of parallelism for tuning (and in the context of organizational compute clusters, better scheduling and utilization ratio). 5. Painless Transition from Exploration to Scaling Up: Often, researchers explore new ideas on small models but, when scaling up, find their HPs optimized during exploration work poorly on large models. ,Trrnnfer would solve this problem.
Direclly tund longe model
Muthethod
$$
Tune
Figure 2: Illustration of μ TTransfer
Nevertheless, j T/rrnnfer still has several limitations. For example, while it is very effective for pretraining, it cannot transfer regularization HP$ so it's generally not applicable the finetuning of pretrained models. We discuss other limitations carefully in Secciioo 5.1
bbtt possibly not for different data and/or tasks. 'It can transfer regularization HPs to the extent they help training but it may not transfer their effect on testing.

---

Table 2: Examples of T/rranfferbble Hyperparameters. All of the below can also be specialized to per-layer hyperparameters.
Optimizer Related
Initialization
Parameter Multipliers
learning rate (LR), momentum, Adam beta, LR schedule, etc
per-layer init. variance
multiplicative constants after
weight/biases, etc
Our Contributions
We demonstrate it is possible to zero-shot transfer near optimal HPs to large model from small version via the Maximal Update Parametrization (μP) from 445
While 45
only covered SGD, here we derive μP for Adam as well (Table 3).
We propose new HP tuning technique, JTrranffe, for large neural networks based on this observation that provides massive speedup over conventional methods and covers both SGD and Adam training;
We thoroughly verify our method on machine translation and large language model pretrain- ing (in infSection 6.3) as well as image classification (in Appendix I.1);
We release PyTorch [27] package for implementing μ TTrnnsfer painlessly. sketch of this package is given in Appendix
Terminologies Sometimes, to be less ambiguous, we often refer to the "large model" as the target model, as it is the model we wish ultimately tune, while we refer to the "small model" as the proxy model, as it proxies the HP tuning process, We follow standard notation dmodel, dhead Fig. 11 for refresher.
dy, duy Rheads ff regarding dimensions in Transformer,
Tensor Programs Series This paper the 5th installment of the Tensor Programs series. While the target audience here are practitioners and empirical researchers, this paper presents the first major practical payoff of the theoretical foundation built in previous works 441.66]
2 Parametrization Matters: A Primer
In this section, we give a very basic primer on why the correct parametrization can allow HP transfer across width, but see Appendices L.1 to L.3 for more (mathematical) details.
The Central Limit Theorem (CLT) says that, if æ1...., Hn are iid samples from zero-mean, unit- variance distribution, then Vn (I1 + .......... xn) converges to a standard Gaussian A(00 1) as n - co.
Therefore, we can say that _ is the right order of scaling factor e, such that (, + ... + n.,) vn converges to something nontrivial. In contrast, if we set e, = 1/n, then (x +... + x..) - 0: or c, 1, then e, (2) +... xx2)) blows up in variance as n - oc.
Now suppose we would like to minimize the function
(.(c)
def
E Finno
[((xx + xn))
over E IR, for some bounded continuous function f: R. If we reparametrize c == a./ / for O E RR then by CLT, Gz def Fa(c) + ()()(0, stabilizes into a function of o: as n - co.
Then for sufficiently large n, the optimal a def arg min, Ga should be close to a'y for any N > n, and indeed, for N = xo this precisely means we can transfer the optimal cx, or o for smaller problem (say F,) to a larger problem (say FN): GN approximately minimized by a and Fy is approximately minimized by c,, eivnn /n/N. Because the transfer algorithm is simply copying a, we say the parametrization
a/ /n the correct parametrization for this problem.
In the scenario studied in this paper, .1...., Fy are akin to randomly initialized parameters of width-n neural network, is akin to HP such as learning rate, and f is the test-set performance of the network after training, in this example, if we parametrize the learning rate and other HPs correctly, then we can directly copy the optimal HPs for a narrower network into wide network and expect approximately optimal
that F,, gives its expectation over random initializations. Just as

---

performance - this is the hyperparameter transfer we propose here. It turns out the Maximal Update Parametrization (nP) introduced in 45 is correct (akin to the parametrization in a above), while the standard parametrization (SP) is incorrect (akin to the parametrization in c). We will review both parametrizations shortly. Theoretically, a jPP network has well-defined infinite-width limit akin to (x1 + ....... + kn)/ / y having a (0. 1) limit by CLT - while a SP network does not (the limit will blow up) [5 In fact, hased the theoretical foundation laid in 45), we argue in Appendix L.3 that μP should also be the unique parametrization that allows HP transfer across width.
We emphasize that, to ensure transferability of any hyperparameter (such as learning rate), it's not sufficient to reparametrize only that hyperparameter, but rather, we need to identify and correctly reparametrize all hyperparameters in Table 2| For example, in Fig. 1] the wide models in SP still underperform their counterparts in juP, even with learning rate tuned optimally. This is precisely because SP does not scale parameter multipliers and input/output layer learning rates correctly in contrast to uP (see TTale 3). See Appendix C for more intuition via a continuation of our example here. We shall also explain this more concretely in the context of neural networks in Appendix B.
3 Hyperparameters Don't Transfer Conventionally
In the community there seem to be conflicting assumptions about HP stability. A priori, models of different sizes don't have any reason to share the optimal HPs. Indeed, papers aiming for state- of-the-art results often tune them separately. On the other hand, a nontrivial fraction of papers in deep learning fixes all HPs when comparing against baselines, which reflects an assumption that the optimal HPss should among models of different designs - therefore, such comparisons are fair. Here, we demonstrate HP instability across width explicitly in MLP and Transformers in the standard parametrization. We will only look at training loss to exclude the effect regularization.
stable not only among the same model of different sizes but also
MLP with Standard Parametrization We start with 2-hidden-layer MLP with activation func- tion Φ using the standard parametrization with LeCun initialization akin to the default in Py PyTorch:
f(g) = W @(W2T T(W + b²) + +²) with init. W 1.^~NO NNO 1/atn), w1z.3) ~ NNO,, 1/n), ((1,2) = O,
where Wi E RddmxxnD D² E R" W2 E Ruxx 8² E R", W Raxdout and din, n, and dout are the input, hidden, and output dimen- sions. The particular MLP we use has = ReLU and a cross-entropy (xent) loss function. We define the width of MLP as the hidden size n. which is varied from 256 to 8192. The mod- els are trained on CIFAR-10 for 20 epochs, which is more than enough to ensure convergence.
2.0
wichh
2.0
254
1.5
512
3.5
1524
Traineg
1.0
2040
1.0
4096
0.5
0.5
0.0
0.0
-10
-8
-1
-8
loguleaminggatee
logul RaritgRage
MLP width different hidden sizes trained for 20 epoch on CIFAR-10 using SGD. Left uses stan- dard parametrization (SP); right uses maximal update parametrization (μP). pP networks exhibit better learning rate stability than their SP counterparts.
Figure
As shown on the left in Fig. 3] the optimal learning rate shifts by roughly an order of magnitude as the width increases from 256 to 8192; using the optimal learning of the smallest model on the largest model gives very bad performance, if not divergence.
Transformer with Standard Parametrization This perhaps unsurprising observation holds for more complex architectures such as Transformer as well, as shown in Fig. (left). We define width
'The more theoretically astute reader may observe that SP with @(1/width) learning rate induces well-detined infinite-wide limit exists as well. Nevertheless, this does not allow HP transfer because this limit is in kernel regime shown in 45). See Appendix L.3 for more discussions.
" "i. the default parametrization offered by common deep learning frameworks. See Table 3 for a review. The key here is that the init. variance x 1/ 1/fan in, the same insights here apply with e.g. He initialization.

---

Table 3: (P45 and SP for General Neural Networks, Basic Form. This basic form emphasizes the scaling with width (fan in or fan out); in practice, we may insert tunable multipliers in front of fan_in and lan_out as in Eq. (4) Notations: 1) v is the "master" learning rate. 2) The fan_out of a bias vector is its dimension (whereas fan_in is 1). 3) Purple text highlights key differences from standard parametrization (SP): Gray text recalls the corresponding SP. SGD (resp. Adam) here can be replaced by variants such as SGD with momentum (resp. Adagrad, Adadelta, etc). In general, the three columns here can be interpreted as linear layers that have [finite, infinite, infinite] input dimension and (infinite, finite, infinite) output dimension in an infinite-widd width network; this description generalizes more readily other parameters such as those of layernorm. Transformer JP requires one more modification (1 (1/d attention instead of 1 1/v dd;; see Definition 4.1 This version of μPP gets rid of parameter multipliers; for the version similar to that in [4]. see Table 13. Also see Table 12 for yP formulation that is easier to implement (and compatible with input/output weight sharing).
Input weights & all biases
Output weights
Hidden weights
Init. Var. SGD LR
1/tan in 3- --fa__out (1)
1/ann in?
(4tan_in) (7)
1/tan in
1/tan
Adam LR
17
1/an
(T)
a/tan in
(n)
as dmodet. with dy dg = du fmoda//meed and fn 4dmodel. The models are trained on wikitext-2 for epochs. in Fig. 18 in the appendix we also show the instability of initialization scale and other HPs.
4 Unlocking Zero-Shot Hyperparameter Transfer with μP
We show that μP solves the problems we see in Section
MLP with pP For the MLP in Section 3 to switch to jPP we just need to modify Eiq. (2)
initialization of the last layer and its learning rates of the first and last layer as well as of the biases.
The basic form i
initialize W ~ NO, 1a..), W ~ N(0,1/1) W ~ -N(00 1/n²), 6012) = 0
(3)
with SGD learning rates
7/ = 7762 T)2 = 17n, 7/22 = 17. 7/23 = 1pn
Here, 17 specifies the "master" learning rate, and we highlighted purple the differences in the two parametrizations. This basic form makes clear the scaling with width n of the parametrization, but in practice we will often insert (possibly tune-able) multiplicative constants in front of each appearance of n. For example, this is useful when we would like to be consistent with a SP MLP at a base width
no- Then we may insert constants as follows: For ñ def n/no:
initialize W ~ (O, 1/a..,, W ~ N1 1/),, W ~ ((0.1/nn)) b(1.2) = 0
(4)
with SGD learning rates T/w1 = 776- 762 = nñ, 7w2 = 7/3 = Tpñ
Then width n no. all purple factors above are 1, and the parametrization is identical SP
(Eq. (2) at width ng. Of course, as n increases from no. then Eq. (4) quickly deviates from Eq. (2) In other words, for a particular n., juP and SP can be identical up to the choice of some constants (in this case nt)), but μP determines a different "set" of networks and optimization trajectory than SP as one varies n. As we will see empirically in the next section, this deviation is crucial for HP transfer.
Indeed, in Fig. riight,,, we plot the CIFAR10 10 performances, over various learning rates and widths, of μP MLPs with no 128. In contrast to SP, the optimal learning rate under jPP is stable. This means that, the best learning rate for a width-129 -28 network is also best for a width-8192 network in juP i.e. HP transfer works but not for SP. In addition, we observe performance for a fixed learning rate always weakly improves with width j(P, but not in SP.
This MLP μP example can be generalized easily to general neural networks trained under SGD or Adam, as summarized in Table 3 which is derived in Appendix L.
While superficially different, this parametrization is equivalent to the μP defined in 45
5

---

4.5
Training
1.5
2000
4000
1.0
575
7.0
5.0
5.0
525
Tranng
5.0
4.2
435
4.0
4.1
233
4.0
425
1.5
39
402
533
Figure 4: Empirical validation of the stability of four representative hyperparameters on pre- LN Transformers in μaP: learning rate, last layer weight multiplier Okoutputs weight initialization standard deviation, and learning rate schedule. We use the following learning rate schedules: (a) linear decay; (b) StepL.R @ [5k, 8k] with decay factor of 0.1; (c) Stepl R @ [4k, 7k] with decay factor of 0.3; (d) cosine annealing; (c) constant: (f) inverse square-root decay. All models are trained on wikitext-2 for 10k steps. When not specified in the legend, the width used is 256,, depth 2, batch size 20, sequence length 256, and LR schedule constant. We sweep particular HP, corresponding to each column, while fixing all others constant. See Section 5.1 for discussion of these results.
Transformers with pP We repeat the experiments with base width no 128 for Transformers:
Definition 4.1. The Maximal Update Parametrization (pP) Transformer is given by Table 3 and 1/d attention instead of 1/ 1/Vd, i.e. the attention logit is calculated as q k/d instead of k/ Vd where query q and key k have dimension d d!
The results are shown on the right in Fig. where the optimal learning rate stable, and the performance improves monotonically as width increases.
Which Hyperparameters Can Be /Transferred?
In this section, we explore how common HPs fit into our framework. general, they can be divided into three kinds, summarized in Table 1)
1. those that can transfer from the small to the large model, such as learning rate Table 2):
2. those that primarily control regularization and don't work well with our technique; and
3. those that define training scale, such as width as discussed above as well as others like depth and batch size, across which we transfer other HPs.
Those in the first category transfer across width, as theoretically justified above in Section 2] To push the practicality and generality of our technique, we empirically explore the transfer across the other dimensions in the third category. Note that j/Transfer across width is quite general. e.g. allows varying width ratio of different layers or number of attention heads in a Transformer: Appendix G.2. This will be very useful in practice. For the second category, the amount of regularization (for the purpose of controlling overfitting) naturally depends on both the model size and data size, so we should not expect transfer to work if the parametrization only depends on model size. We discuss these HPs in more detail in Appendix G.I.
10 is roughly because during training, and l will be correlated so q & actually scales like d due to Law of Large Numbers, in contrast to the original motivation that q. & are uncorrelated at initialization Central Limit applies instead. See Appendix L.2.1 for more in-depth discussion.

---

5.1 Empirical Validation and Limitations
Our empirical investigations focus on Transformers (here) and ResNet (in Appendix I.1.1), the most popular backbones of deep learning models today. We train 2-layer pre-layernorm | Transformer with attention heads on Wikitext-2. We sweep one of four HPs (learning rate, output weight multiplier, initialization standard deviation, and learning rate schedule) while fixing the others and sweeping along width and depth (with additional results in Fig. on transfer across batch size, sequence length, and training time). Fig. 4ssoww the results averaged over 5 random seeds.
Empirically, we find that for language modeling on Transformers, HPs generally transfer across scale dimensions if some minimum width (e.g. 256), depth (e.g.. 4), batch size (c.g., 32), sequence length (e.g. 128), and training steps (e.g., 5000) are met, with some caveats discussed below. While the exact optimum can shift slightly with increasing scale, this shift usually has very small impact on the loss, compared to SP (Figs. and (eftt).. However, there are some caveats. For example, the best initialization standard deviation does not seem to transfer well across depth (2nd row, 3rd column), despite having a stabler optimum across width. In addition, while our results on width, batch size, sequence length, and training time still hold for post-layernorm (Fig. 17) The transfer across depth only works for pre-layernorm Transformer. Nevertheless, in practice (e.g. our results in Section 6.3) we find that fixing initialization standard deviation while tuning other HPs works well when transferring across depth.
6 Efficiency and Performance of TTransfer
Now that the plausibility of TTransfer has been established in toy settings, we turn to more realistic scenarios to see if one can achieve tangible gains. Specifically, we perform HP tuning only smaller proxy model, test the obtained HPs on the large target model directly, and compare against baselines tuned using the target model. We seek to answer the question: Can /1 ''Transfer make HP tuning more efficient while achieving performance on par with traditional tuning? As we shall see by the end of the section, the answer is positive. We focus on Transformers here, while experiments on ResNets RIFAR10 and Imagenet can be found as well in Appendix I.1. All of experiments run on V100 GPUs.
6.1 Transformer IWSLT14 De-En
Setup 1WSLT14 De-En is a well-known machine translation benchmark. We use the default IWSLT (post-layemorm) Transformer implemented in fairseq (25 with 40M parameters, which we denote as the 1x model 1 For y'Transfer, we tune on a 0.25x model with 1/4 of the width, amounting to 4M parameters. For this experiment, we tune via random search the learning rate n, the output layer parameter multiplier O'outpat, and the attention key-projection weight multiplier Cattnn See the grid and other experimental details in Appendix H.1.
We compare transferring from the 0.25x model with tuning the Ix model while controlling the total tuning budget in FL.OPs [4 To improve the reproducibility of our result: 1) we repeat the entire HP search process (a trial) 25 times for each setup, with number of samples as indicated in Table and report the 25th, 50th, 75th, and 100th percentiles in BLEU score; 2) we evaluate each selected HP combination using 5 random initializations and report the mean performance
We pick the HP combination that achieves the lowest validation loss Tol for each trial. The reported best outcome is chosen according the validation loss during tuning. We compare against the default in fairseq, which is presumably heavily tuned. The result is shown in Table 4
Ha2 layers" means the model has 2 self-attention blocks. To compare with SP Transformer, see Fig. 18. in fact, post-layernorm Transformers are much more sensitive HPs than pre-layemorm, our technique is more crucial for them, especially for transfer across width. Fig. 1 Jses post-layernorm.
/httts://githhi
www.cott////eeeeeeooo wwwttorch/aairreg/l..o.o. http///aateeeeeeee wttpeeeeeeeaaeeeeeeeeeeeeeeeeeee wttpeeeesstaaaaaaoo..nooomaaomoe www.aaannnnnnnnnnanaaeaaeee /REAEREEADE...E... md
16 "Ideally we would like to measure the wall clock time used for tuning. However, smaller models such as the proxy Transformer used for IWSLT are not efficient on GPUs, so wall clock time would not reflect the speedup for larger models like GPT-3. Thus, measure in FLOPs, which is less dependent hardware optimization. 15 We do not report the standard deviation over random initializations to avoid confusion. We find this provides more reliable result than selecting for the best BLEU score.
7

---

Table 4: Transformer IWSLT14 De-En, 1x and 0.25x refers to scaling of width only. Compared to traditional tuning ("Tuning on Ix"), jutransfer from 0.25x provides better and more reliable outcome given fixed amount of compute. On the other hand, naive transfer (i.e. with SP instead of μP) fails completely. The percentiles are over independent trials, with each trial involving the entire tuning process with a new HP random search.
Val. BLEU Percentiles
Setup
Total Compute
#Samples
25
50
75
100
fairseq 25 default
35.40
Tuning on 1x
1x
5
33.62
35,00
35.35
35.45
Naive transfer from 0.25x
1x
64
training diverged 35.33
Transfer from 0.25x (Ours)
1x
64
35.27
35.45
35.53
Performance Pareto Frontier The result
454
above only describes a particular compute bud- get. Is Transfer still preferable when we have lot more (or less) compute? To answer this question, we produce the Pareto frontier in Fig. S(left), where we repeat the above experiment with different compute budgets. Evidently, our approach completely dominates conventional tuning.
252
8 150
540
244
Convensionl
33
Figure 5: Efficiency-performee
Pareto fron- tier of j'Transfer compared to conventional tuning, on IWSLT Transformer, using random HP search as the base method. We plot the median BLEU score over 25 trials (Left) against relative compute budget in log scale and (Right) against number of HP samples taken. While with the same num- ber of samples, / TTransfer slightly underperforms conventional tuning, this gap vanishes with more samples, and in terms of compute, our Pareto fron- tier strongly and consistently dominates that of conventional tuning. Note that, in larger models (e.g. BERT or GPT-3, not shown here), we believe our efficiency advantage will only widen as our small proxy model can stay the same size while the target model grows.
Sample Quality of Proxy Model vs Target Model The Pareto frontier in Fig. 5 rright sug- gests that, given fixed number of random sam- ples from the HP space, 1) tuning the target model directly yields slightly better results than tuning the proxy model (while taking much more compute of course), but 2) this perfor- mance gap seems vanish as more samples are taken. This can be explained by the intu- ition that the narrower proxy model is "noisy estimator" of the wide target model (45) With few samples, this noise can distort the random HP search, but with more samples, this noise is suppressed.
6.2 Transformer on WMT14 En-De
We scale up to WMT14 En-De using the large (post-layernorm) Transformer from 37 with 21 parameters. We tune on proxy model with 15M parameters by shrinking dmodel, dffas and Nhead- For this experiment, we tune via random search the learning rate 7). the output layer parameter multiplier Coorppat, and the attention key-projection weight multiplier Cattn following the grid in Appendix H.2. The result shown in Table 5) While random search with HP samples far underperforms the fairseq default, we are able to match it via transfer using the same tuning budget.
Table 5: Transformers on WMT14 En-De. 1x and 0.25x refers to scaling of width only. We report BLEU fluctuation over independent trials, i.e., 3 independent random HP searches.
Val. BLBU Percentiles Worst
Setup
Total Compute
#Samples
Median
Best
fairseq 25 default
26.40
Tuning on 1x Naive transfer from 0.25x # Trannsfr from 0.25x (Ours)
1x
3
training diverged
25.69
1x
64
training diverged
1x
64
25.94
26.34
26.42
8

---

6.3 BERT
Finally, consider large-scale language model pretraining where HPP tuning is known be challeng- ing. Using Megatron (pre-layernorm) BERT [32 as a baseline, we hope to recover the performance of the published HPs by only tuning proxy model that has roughly 13M parameters, which we call BERT-prototype. While previous experiments scaled only width, here we will also scale depth, as discussed in [Section 5|and validated in Fig. 4 We use a batch size of 256 for all runs and follow the standard finetuning procedures. For more details on BERT- prototye, what HPss tune, and how finetune the trained models, see Appendix H.3.
During HP tuning, we sample 256 combinations from the search space and train each combination on BERT-prototype for 108 steps. The total tuning cost measured in FLOPs is roughly the same as training large for the full 10 steps; the exact calculation is shown in Appendix H.3. The results are shown in Table 6 Notice that on BERT-large, we obtain sizeable improvement over the well-tuned Megatron BERT large baseline.
Table 6: BERT pretraining. HP transfer outperforms published baselines without tuning the full model directly at all. We tune BERT-base and BERT-large simultaneously via a single proxy model, BERT-prototype. The total tuning cost the cost of pretraining single BERT-large. Model speedup refers to the training speedup of BERT-prototype over BERT-base or BERT-large. Total speedup in addition includes time saving from transferring across training steps. Both speedups can be interpreted either as real-time speedup on V100s FLOPS speedup (which turn out to be empirically very
similar this case).
Model
Method
Model Speedup
Total Speedup
Test loss
MNLI (m/mm)
QQP
BERT BERTaase base
Megatron Default
1x
1x
1.995
84,2/84.2
90.6
BERT BERTbase Tasse
Naive Transfer
4x
40x
training diverged
BERT base
Transfer (Ours)
4x
40x
1.970
84.3/84.8
90.8
BERT targe
Megatron Default
1x
1x
1.731
86.3/86.2
90.9
BERT targe
Naive Transfer
22x
220x
training diverged
B BERT
#Transfer (Ours)
22x
220x
1.683
87.0/86.5
91.4
6.4 GPT-3
In order further verify y'Transfer scale, we applied it to GPT-3 6.7B (6) This Transformer model (the target model) consists of 32 residual blocks with width 4096. We form the small proxy model by shrinking width to 256, resulting in roughly 40 million trainable parameters, 168 times smaller than the target model. HPs were then determined by random search on the proxy model. The total tuning cost was only 7% of total pretraining cost. Details of the HP sweep can be found in Appendix H.4.
In order to exclude code difference
possible confounder, we also re- re-trained GPT-3 6.7B from scratch using the original HPs from (6]. During training of the j v/Transfer model encountered numerical issues that lead to frequent divergences. order to avoid them, the model was trained
using FP32 precision, even though the original 6.7B model and our re-run were trained using FP16
18
The resulting y uTransfer model outperforms the 6.7B from (6). and is in fact comparable to the twice-as-large 13B model across our evaluation suite (see Table 9). Selected evaluation results can be
found in Table 7 and further details are given in Table 8 and Appendix H.4.
Related Works
Hy Hyperparameter Tuning Many have sought to speed up HP tuning beyond the simple grid or
random search, such as via Bayesian optimization [34 35 or multi-arm bandits [55 18). There are also dedicated tools such as Optuna [4] and Talos 3 which integrate with existing deep learning frameworks and provide easy way apply more advanced tuning techniques. Our work is
1hhile we mainly focused the efficacy of Trannsfer regardless of precision, it would be interesting ablate the effect of precision in our results, we did not have enough resources to rerun the baseline FP32 IIt is quite interesting that Transfer identified a useful region of hyperparameters leading to much improved performance, which probably would be difficult to discover normally because 1) researchers usually change hyperparameters to accomodate precision and 2) there was precise enough justification to go against this judgment until yrrrnnffr.

---

investigated the right Table 7: GPT-3 6.7B Pretraining. Selected evaluation results for the GPT-3 6.7B model tuned with j Transfer (transfered from small proxy model of 40M parameters), compared to the results published in [6] and re-run with original HPs. Note that the perplexities in this table are based on a custom tokenization and are not comparable to the literature. The validation loss refers to the loss achieved on a random held-out part of our dataset. Zero-shot, One-Shor and Few-Shot refer to the number of additional query and answer pairs passed in the context when performing the sampling-bast based evaluations. See Appendix H.4 for full evaluation results.
Task
Metric
6. 6.7B+μuP
6.7B re-run
6.7B from 6
Validation loss
cross-entropy
1.98
2.03
PTB
perplexity
11.4
13.0
WikiText-100
perplexity
8.56
9.13
One Billion Words
perplexity
20.5
21.7
LAMBADA Zero-Shot
accuracy
73.5
70.8
70.3
LAMBADA One-Shot
accuracy
69.9
64.8
65.4
LAMBADA Few-Shot
accuracy
74.7
77.1
79.1
HellaSwag Zero-Shot
accuracy
72.0
66.7
67.4
HellaSwag One-Shot
accuracy
71.1
65.9
66.5
HellaSwag Few-Shot
accuracy
72.4
66.4
67.3
complementary to the above, they can be used to tune the proxy model. it is only for scientific reasons that we primarily did random search throughout this work.
Hyperparameter Transfer Many previous works explored transfer learning of HP tuning (e.g. [8 36 49]). However, to the best of our knowledge, our work the first to explore zero-shot HP transfer. In addition, we focus on transferring across model scale rather than between different tasks or datasets. Some algorithms like Hyperband [19] can leverage cheap estimates of HP evaluations (like using a small model to proxy a large model) but they are not zero-shot algorithms, so would still be very expensive to apply to large model training. Nevertheless, all of the above methods are complementary
they can be applied the tuning of our proxy model.
Previously Proposed Scaling Rules of Hyperparameters M 23 31 33)
way to scale learning rate with batch size, with sometimes conflicting proposals. which we summarize in Appendix F. (26) studied how learning rate (and batch size) should scale with width for MLPs and CNNs trained with SGD in NTK or standard parametrizations. We provide a detailed comparison of our work with theirs in Appendix F.
Many previous works proposed different initialization or parametrizations with favorable properties, such as better stability for training deep neural networks [0] [3] 21 30 47 48 51]. Our work differs from these in that we focus on the transferability of optimal HPs from small models to large models in the same parametrization.
8 Conclusion
Leveraging the discovery of a feature learning neural network infinite width limit, we hypothesized and verified that the HP landscape across NNs of different width is reasonably stable if parametrized according to Maximal Update Parametrization (μP). We further empirically showed that it's possible to transfer across depth, batch size, sequence length, and training time, with a few caveats. This allowed to indirectly tune very large network by tuning its smaller counterparts and transferring the HPs to the full model.
Venues of Improvement Nevertheless, our method has plenty of room to improve. For example, initialization does not transfer well across depth, and depth transfer generally still does work for post-layernorm Transformers. This begs the question whether a more principled parametrization in depth could solve these problems. Additionally, Fig. 4 shows that the optimal HP still shifts slightly for smaller models. Perhaps by considering finite-wir width corrections jaP one can fix this shift. Finally, it will be interesting to consider if there's 's way to transfer regularization HPs as a function of both the model size and data size.
10

---

Broader Impact Our work makes HP tuning of large models more efficient. This enables large models to be better tuned given the compute budget, thereby increasing the performance per cost. Organizations large and small can focus their research on small models and scale up only once with reasonable confidence that the training would go well. We do not foresee any direct negative societal impact.
References
[1] WVDDIADDepLLummnnkkkaapppe DeepLeearningExxamplees
v2 license. URL hicensss.UJLLhppps////gaabbooooD
[2] Davidnet, mit license, 2019. URL /tt.://ee/ghttbbb.oo//////ggggee
[3] Autonomio talos, mit license, 2019. URL. httt:///gtthub..ccm//ttttoooooo/
[4] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework, 2019.
[5] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cot- trell, and Julian McAuley. ReZero is All You Need: Fast Convergence at Large Depth. arXiv:2003. 04887 [cs, stat], June 2020. URL http:///xxxxv.orabbeees 2003.04887.
04887
arXiv:
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[7] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014.
[8] Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl. Attentive language models beyond a fixed-length context, 2019.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv: 1810.488000 /cs], May 2019. URLhttpp //tpx//axvvvv/////////////aaav www/////////aaeeeeee/
04805
arXiv: 1810.04805
version: 2.
[10] Xavier Glorot and Yoshua Bengio. Understanding the difficulty training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249 229,,56, Chia Laguna Resort, Sardinia, Italy, May 2010. PMLR. URL http:////// mlr. wwwpp/reessv wwt.rpeeeeeeorooollleeee glw.ott0a..h........
[11] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better:
closing the generalization gap in large batch training neural networks. arXiv:1705.0 07744 [cs, stat], May 2017. URL http::tttt:////tt org/ abs 1wtp/1705...0 08741
arXiv: 1705.08741.
[12] Samuel Horváth, Aaron Klein, Peter Richtárik, and Cédric Archambeau. Hyperparameter transfer learning with adaptive complexity. CoRR, 0bs/2102.118800, 2021. URL https
//arxiv. www.orggg../.ooooooooo//ooooooee abs//102...22 22022 12810
[13] Xiao Shi Huang and Felipe Pérez. Improving Transformer Optimization Through Better Initialization. page 9.
[14] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. arXiv: /00v: 075572 (cs, math, stat], June 2018. URL http://arxiv.orgg
1806. 07572 00000 arviv: 1806.07572.
[15] Kevin Jamicson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparame- ter optimization, 2015.

---

[16] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv:2001.08.0 083661 [cs, stat], January 2020. URL http:/// http://arxiv org/ abs/2001 08361
arXiv: 2001 .08361.
[17] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning, 2020.
[19] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A Novel Bandit-Based Approach to Hyperparameter Optimization. JMLR 18, page 52.
[20] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747-5763, Online, November 2020. Association for Computational Linguistics. doi: 10. 0tt55//111222.mmmlpp.. URL https:////attt aclweb. org/ anthologyy200000 emn main. 463
[21] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the Difficulty of Training Transformers. arXiv:2004.088 002449 [cs, stat], September 2020. URL
http:://rxxx org. wttggabs/22000 08249
arXiv: 2004.08249.
[22] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487 4487.4496,, Florence, Italy, July 2019. Association for Computational Linguistics. URL https: /wtp:://wwww.co www. aclweb. org/ anthology/
P19-1441
[23] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An Empirical Model of Large-Bater Batch Training. arXiv:1112.06162 [cs, stat], December 2018. http
//arxiv. orw/oos////////o/oo//ooooee abs /ats../1..... 06162,
arXiv: 1812.06162.
[24] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
(25) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling, mit license. In Proceedings of NAACL-HIT 2019: Demonstrations, 2019.
[26] Daniel S. Park. Jascha Sohk-Dickstein, Quoe V. Le, and Samuel L. Smith. The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study. May 2019. httpt://////:iiii org/ abs/ 1905. 03776v1
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmai- son. Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, performaaee deep learning library, bsd-style li- cense. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'lché Buc, E. Fox. and R. Garnett, editors, Advances Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers: net neurips cc/paper/ 9015-pytor watpppyooohaa--mnrrrraeeeeeeeeee an- -ww.merratiieeeyyl----i-ee wttp://hwhiiggpoooooorooooeoeeea wwwiiingng-bbbrrr library pdf
[28] Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable Hyperparameter Transfer Learning. NeurlPS 2018, page 11.
[29] Martin Popel and Ondicj Bojar. Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguisties, 110(1):43-0,, April 2018. ISSN 1804-0462. doi: 10.2478/ pralin-2018-000 URL http:://ontte:: sciendo. conm/vieea/oounnall/paalla 1/article-p43. xm1
12

---

[30] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Infor- mation Propagation. arXiv:161 1.01232 [cs, stat), November 2016. URL http::tttp//i..... org/ abs/1611.
arXiv: 1611.01232.
[31] Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the Effects of Data Parallelism Neural Network Training. arXiv: 181 1.03600 [cs, stat], November 2018. http:////aaaii wrw.aa//.rgg/////./e//obo/b aa////aaa/11....111 03600 arXiv: 181 1.03600.
[32] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. CoRR, 0bs//199.00005, 2019.
[33] Samuel I.. Smith, Pieter-Jan Kindermans, and Quoe V. Le. Don't Decay the Learning Rate, Increase the Batch Size. arXiv: .0489 [cs, stat], November 2017. URL http:ttp://arx org/abs/ 1atbs///11..000
arXiv: 1711.00489.
[34] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012.
[35] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks, 2015.
[36] Danny Stoll, Jörg K.H. Franke, Diane Wagner, Simon Selg, and Frank Hutter. Hyperparameter transfer across developer adjustments, 2021. URL U wttp:/iwwwweeeffrullie.. WPOOvDYLXem
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. CoRR, abs//100.00362, 2017.
URL http::/aaax org/ abs/ 1706. 03762
[38] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding.
EMNLP 2018, page 353, 2018.
[39]
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[40] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 1112-1122, Association for Computational Linguistics, 2018. URL http://aclweb wwgg/anthhoogg///1eeeoooo 1101
[41] Greg Yang. Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Archi- tecture are Gaussian Processes. 110.171910.124778 /cond-mat, physies:math-plh December 2019. /tR///ttt.//aaawvoo////b//////// 12478
[42] Greg Yang. Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation. arXiv:1902. 1400. [cond-mat, physics:math-phh stat], February 2019. URL http:h/tt:///aeeee http:://axxvvv.ooe wwt.:/r////b///0// 04760
[43] Greg Yang. Tensor Programs II: Neural Tangent Kernel for Any Architecture. arXiv:2006.14548 /cond-matt, stat), August 2020. htt.:ttpp//arrii.oor//bbb///////
14548
[44] Greg Yang. Tensor Programs III: Neural Matrix Laws. arXin:2009.10685 [cs, math], September 2020. URL htllttt/p///xiii org/ abs 10685
[45] Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks. arXiv, 2020.
[46] Greg Yang and Etai Littwin. Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics. arXiv:210s.03703 [cs, math], May 2021. URL http:
//arxiv. www.oogg//////eeeeeo/ooooeeeeeee abs/2105....// 03703
13

---

[47] Greg Yang and Sam S. Schoenholz. Deep Mean Field Theory: Layerwise Variance and Width Variation Methods to Control Gradient Explosion. February 2018. URL https
//openreview net/forummidd<JGGGGht
[48] Greg Yang and Samuel S. Schoenholz. Mean Field Residual Networks: On the Edge of Chaos. 1XXv:1712.00866 [cond-mat, physicx:mlinn, December 2017. URL http:////aaxx org/abs/
1712.08966 00000 arXiv: 1712.08969.
[49] Dani Yogatama and Gideon Mann. Efficient Transfer Learning Method for Automatic Hyperpa- rameter Tuning. Artificial Intelligence and Statistics, pages 1077-1085. PMLR, April 2014. URL http::///proeeeediiii
mlr, prwss/ v33/ v33/yogatamas htmmmmmmmm.mm.1mm.1mmmmmmmmmmmmm ISSN: 1938-7228.
[50] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2017.
[51] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual Learning Without Normalization via Better Initialization. In International Conference on Learning Representations, 2019. URL
14


Reviews:
[{"review_id": "n5jXD0CS2IX", "reviewer": "Reviewer_Ty3V", "summary": "Based on recently proposed Maximal Update Parametrization ($\\mu$P), this paper finds that hyperparameters can be copied (i.e. transferred with zero cost) across model sizes. Therefore, it is possible to tune hyperparameters in a small model, and to apply hyperparameters in a large model.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although this paper claims it is Zero-Shot Hyperparameter Transfer, I would rather say that they find some scaling rules for hyperparameter tuning. This paper heavily depends on [40], and can be seen as an empirical validation of the theory proposed by [40]. I suggest that the authors weaken the claim that this is the first paper to explore zero-shot hyperparameter transfer, because learning rate scaling rules [31] / initialization rules (Kaiming initialization and Xavier initialization etc) all fall into this category of \"zero-shot hyperparameter transfer\".\n\nLarge models like GPT-3, T5 are not only difficult to train, but also difficult to tune. Even research institutions in large companies like google and facebook cannot afford grid-search tuning of hyperparameters for these large models. Therefore, **the demonstrations in this paper can have a huge impact for training large models**. It is thrilling to see that we can tune hyperparameters in a small Transformer and expect better performance in a large Transformer, which is also validated in ResNets. Based on the potential impact, I give a positive review. But to be honest, the technical novelty is limited, cause this paper is a direct application of the theory in [40].\n\nI'd like to increase the score if my concerns are addressed:\n\n(1) \"this is the first paper to explore zero-shot hyperparameter transfer\" is an over-claim. Please tone down the claim. Section 2 (Parametrization Matters: A Primer) talks about proper scaling of hyperparameters, not hyperparameter transfer. I think everyone can understand the importance of finding the proper rule of hyperparameters scaling with respect to model scaling. In my humble opinion, section 2 is uninformative. Maybe authors can move some content from the supplementary material to the main paper to replace section 2.\n\n(2) why regularization cannot be transferred? Since the main goal of this paper is to verify hyperparameter scaling rules, intuitively there should also be a scaling rule for regularization factors.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "YSYBDoPHun0", "reviewer": "Reviewer_5dwZ", "summary": "The authors build on the recently developed Maximal Update Parameterisation\n($\\mu$P) to develop a framework for using the same hyperparameters on\nmodels with variable numbers of hidden units or channels. The authors\ndemonstrate the value of this approach on large benchmark tasks, such as\nIESLT14 De-En, WNT14 En-De and training BERT-large on wikitext-2.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Originality\n\n> Are the tasks or methods new? Is the work a novel\n> combination of well-known techniques? (This can be valuable!) Is it clear\n> how this work differs from previous contributions? Is related work\n> adequately cited?\n\nThis is an application of the Maximal Update Parameterisation to the\nproblem of hyperparameter tuning. The paper provides theoretical and\nextensive experimental exploration of this topic.\n\nThe experimental results showing transfer of hyperparameters regardless of\nmodel size is the main original contribution of this paper.\n\n## Quality\n\n> Is the submission technically sound? Are claims well supported\n> (e.g., by theoretical analysis or experimental results)? Are the methods\n> used appropriate? Is this a complete piece of work or work in progress? Are\n> the authors careful and honest about evaluating both the strengths and\n> weaknesses of their work?\n\nExperimental results are concrete and described in great detail. The\ntheory described in the paper describes the Maximal Update Parameterisation\nand the background for it's development. The aim of this parameterisation\nis to be invariant to hyperparameters so it is natural that it would be\nuseful for this task.\n\nTables 4, 5 and 6 list experimental results where the training of models\nthat are being compared against diverged. This does not give a fair\ncomparison to existing work. In practice, practitioners would try to tune\nthe model hyperparameters to the new setting using hyperparameter tuning\ntechniques or personal expertise. A standardised way to train the models so\nthat they do not diverge would be a fairer comparison.\n\nA natural choice for this would be to automate the traditional method of\ntuning hyperparameters on a subset of the full dataset or a single\nminibatch before running it on the full dataset. Random search over\nhyperparameters could provide a good baseline using this method to compare\nagainst.\n\nThe experiments presented expand upon the cases where this method of\nhyperparameter transfer performs well but fail to explore where this model\nfails. For example, it is mentioned that the method does not transfer to\narchitectures using different regularisation methods but this is not\nexplored in experiment.\n\nIt is not clear how this model behaves while varying the hyperparameters\nthat can be transferred across, as listed in Table 1. The authors do not\nexplore the regions where the maximal update parameterisation breaks down.\nIn other words, how much does the underlying architecture have to change\nbefore training with hyperparameters chosen on a small network breaks down?\n\n## Clarity\n\n> Is the submission clearly written? Is it well organized? (If not, please\n> make constructive suggestions for improving its clarity.) Does it\n> adequately inform the reader? (Note that a superbly written paper provides\n> enough information for an expert reader to reproduce its results.)\n\nThe work very clearly states its strengths and immediately explains why the\nparameterisation is ideal for this task. Figure 1 is well placed to explain\nthis to the reader and communicates the main thesis of the paper clearly.\n\nThe paper editorializes unnecessarily. For example, at the start of Section\n3 and Section 5.1, the authors make a statement about community opinion\nwithout citation. Footnotes generally also express the author's opinions.\n\nIn the introduction a sequence of numerical bullet points are presented in\na paragraph instead of as a list. It would be clearer as a list.\n\n## Significance\n\n> Are the results important? Are others (researchers or practitioners) likely\n> to use the ideas or build on them? Does the submission address a difficult\n> task in a better way than previous work?  Does it advance the state of the\n> art in a demonstrable way? Does it provide unique data, unique conclusions\n> about existing data, or a unique theoretical or experimental approach?\n\nParameterisation in deep learning is important. This work demonstrates a\nvaluable application of a specific parameterisation and how this can\nmassively improve contemporary deep learning practice.\n\nThe experiments provided are extensive and could provide a useful reference\nto researchers who would like to use the proposed parameterisation.\n\nOne detriment to the significance of this work is that it did not introduce\nthe Maximal Update Parameterisation. The experiments in this work follow\ndirectly from that development without extra development work. The major\ncontribution of this work is the experimental results demonstrating that\nthis parameterisation still works on large deep learning problems.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Comparisons in the experiment section to similar works or alternative\nsolutions for hyperparameter search are lacking. As noted in the main\nreview, the authors list that the methods they are comparing against\n\"diverged\" in Tables 4,5 and 6. It would make more sense to compare against\na contemporary work focused on the same problem of hyperparameter transfer.\nSee the main review for suggestions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SFTV8fEKj5J", "reviewer": "Reviewer_Z5cb", "summary": "Summary:\n\nThis paper makes the observation that when initializing neural network with the *maximal \n\nupdate parametrization* technique, certain hyperparameters such as learning rate and learning rate schedule are transferrable across different scales of the network (e.g. width of each layer). This insight enables to authors to tune hyperpameters using a small network and directly transfer the tuned results to much larger networks, saving the cost of tuning larger networks. Although such tuning-and-transfer approach is probably already a standard practice, the authors point out that such practice is not optimal if we initialize neural networks using the standard technique and using the proposed method generally leads to much better transfer result. \n\nI find this work to be interesting and see the potential of this submission to inspire future work. That being said, I want to point out that the *maximal update parametrization* technique is already published in a previous work and this submission should be viewed more as an add-on to this prior work. Practically speaking, I should also note that not all hyperparameters are transferrable as pointed out by the authors candidly. Certain hyperpameters such as regularization strength are not transferrable across scale. Such hyperparameters may have complex interactions with other hyperparameters and therefore make the proposed method less useful. Given these two reasons, I am leaning toward accept but have some reservation. I detail my other concerns in the following section.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- the submission is novel and aims at solving a practical problem\n- the paper is well-written\n\nCons/Questions:\n\n- the submission is largely built upon the prior work \"maximal update parametrization\"\n- I wish the author spent more time on explaining the intuition of the finding, namely in section 2. In line 91, the authors mention that the hyperparameters are similar to \"c\". However, I fail to see this connection. I am wondering whether is it reasonable to think of the  test-set perforance as a bounded continuous function of the hyperparamter. I would suggest the authors to expand on this analogy and also try to provide more intuition of this finding.\n- I want to double check my understanding of the experimental details.\n    - In Table 6, how is the total speed up ratio calculated? Why is it different from model speedup?\n    - How does such direct transfer of hyperparameter compare to tuning the hyperparameter with a short amount of time? In my experience, the training loss decreases much slower toward the second half of pretraining and we should be able to tell the better hyperparameters early on. If the authors have saved all the intermediate checkpoints for BERT pretraining, this experiment should be easy to add.\n- nitpick: not sure if the terminology \"zero-shot hyperparameter transfer' makes sense to me because it is not clear what \"few-shot\" or \"many-shot\" hyperparameter transfer would be. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see negative societal impact from this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have ethical concerns with this work. ", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "DdF6XeA2UC", "reviewer": "Reviewer_rfmG", "summary": "The paper proposes an alternative method for tuning hyperparameters which is to tune hyperparameters on a small width model and transfer these sets of hyperparameters to the full network. The core idea is to utilize the maximal update parameterization so that optimization hyperparameters remain stable under model size changes. This enables a zero-shot transfer and makes the hyperparameter optimization process more efficient both in computation and memory. Empirically, the authors show that the careful hyperparameter selection on the small model with the proposed approach outperforms BERT-large on MNLI and OOP and also requires less total training costs.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n* The core principle of the method is based on Maximal Update Parametrization (\\mu P) and the work is incremental to show hyperparameters such as learning rate are stable under model size changes. \n* However, the idea of zero-shot hyperparameter transfer is interesting and introduces a different perspective in hyperparameter optimization.\n* It is clear how this work differs from previous contributions and the related work is adequately cited.\n\nQuality: \n* The submission is technically sound and claims made in the paper are well supported. The authors carefully discussed the weakness and strengths of their proposed approach.\n\nClarify:\n* The submission is clearly written and well-organized. It was a good read overall.\n\nSignificance: \n* The results presented in the paper are important and interesting. I expect other researchers or practitioners likely to use the idea or build on them. \n* One concern I have with the paper is the limited novelty. The core foundation of the paper is based on Maximal Update Parametrization (\\mu P). However, the investigation of several hyperparameters' sensitivity on \\mu P and experiments to analyze this behaviour is interesting and promising. \n* Another concern I have is the limited generalizability (scalability). The proposed parameters cannot handle regularization hyperparameters such as weight decay and dropout. Since the modification of such regularization hyperparameters may change the optimization hyperparameters, in the end, to obtain the state-of-the-art results, I feel like hyperparmaeter search on the full model is inevitable. I believe that future works are necessary for the proposed model to be fully utilized in practice.\n\nMinor Comments:\n* In footnote 8, it should be \"in fact\" -- > \"In fact\".\n* The use of \\times instead of x would be helpful in section 6.\n\n--------\n\nI thank the authors for their response. I acknowledge that I read the authors' responses and other reviewers' comments. The authors addressed some of my concerns on the novelty of the proposed approach. Regarding the second issue, I understand that the proposed approach achieved good results in GPT-3 that doesn't require much regularization. However, for models that require careful tuning of regularization hyperparameters to obtain the state-of-the-art results (e.g. ImageNet classification), hyperparameter search on the full model is inevitable and the proposed method wouldn't be helpful. Given these reasons, I would like to remain at my current score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adeptly address the potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BblrzsfTmD8", "reviewer": "Reviewer_dYRN", "summary": "The authors demonstrate the effectiveness of zero-shot transfer optimal hyperparameters from small models to large models within the same model family. They verify their method on machine translation and large language model pre-training. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro: \nThe paper is well written and easy to follow. They first parameterize the target model in Maximal Update Parameterization, then tune the smaller version of the target model and copy the optimal hyperparameters to the large model.\n\n- The authors acknowledge the limitation of \\muTransfer clearly: It applies to pretraining the architectures in the same family to the same data, but cannot transfer regularization parameters or for fine-tuning for other datasets. \n\nCons:\n\n- The paper is mainly based on the Maximal Update Parameterization, which was proposed in [40]. However, the authors assume readers are familiar with the work.\n\n- The authors find that for language modeling on Transformers, hyperparameters generally transfer across scale dimensions if some minimum width, depth, batch size, sequence length, and training steps are met. Not too much explanation is provided here. How are these specific conditions obtained? Does this indicate the transability of the method is only applicable in certain conditions.\n\n- The method enables HP transfer across width, however, a stronger model in the model family usually comes from increased depth, which also contributed the most to the latency of the model with the number of weights being the same compared to a wider model. It is not clear whether the method can generalize to depth. \n\n- The authors mostly take shallow networks for illustrations or experiments to show the instability of optimal HPs across width, such as Figure 3 & 4. However, it is not clear whether it still holds when the network depth increases. \n\n- In Fig 9, the variation of optimal HPs for ResNets with SP on CIFAR-10 is not quite evident. Also for large datasets such as ImageNet, the figures of instability of SP are not provided and the improvement of \\muTransfer over SP is not that significant (0.41%). What is the gain of HPO in comparison with the default HPs, such as LR=0.1, momentum 0.9 in the original paper? It is not clear to me why some wired HPs used in the ResNets experiment in D.1.2, such as 2.048, 3.05, 0.875.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for "zero-shot transfer" of optimal hyperparameters from small models to large models, which enables efficient tuning of large models such as GPT-3. Such a study on efficient tuning of large models is highly anticipated by our community, and may inspire future research in this area. Reviewers generally recommended acceptance (4 out of 5), acknowledging the significance and experimentation of this work. They, with one referee still below the bar, pointed out several important concerns. Authors shall revise their paper to: 1. Elaborate the novelty by connecting to the Maximal Update Parameterization [40], highlighting clearly which part is due to [40] and which part is novel in this paper; 2. Town down the claim of "zero-shot" hyperparameter transfer, because hyperparameters of the regularization and the optimizers cannot be transferred, and the hyperparameter transfer can only be applied within the same model family; 3. Document clearly in what cases their approach is applicable, e.g. which hyperparameters can be transferred and which ones cannot, and how practitioners can deal with the difficult ones -- simply bypassing difficult cases is not good, because it will make scope of the approach quite narrow. I would stress that the author rebuttal did not respond to all the concerns, so the acceptance is conditioned on the authors' revision, in which at least the meta reviews must be well addressed.
Decision: Accept (Poster)