Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors demonstrate the effectiveness of zero-shot transfer optimal hyperparameters from small models to large models within the same model family. They verify their method on machine translation and large language model pre-training. 
main_review: Pro: 
The paper is well written and easy to follow. They first parameterize the target model in Maximal Update Parameterization, then tune the smaller version of the target model and copy the optimal hyperparameters to the large model.

- The authors acknowledge the limitation of \muTransfer clearly: It applies to pretraining the architectures in the same family to the same data, but cannot transfer regularization parameters or for fine-tuning for other datasets. 

Cons:

- The paper is mainly based on the Maximal Update Parameterization, which was proposed in [40]. However, the authors assume readers are familiar with the work.

- The authors find that for language modeling on Transformers, hyperparameters generally transfer across scale dimensions if some minimum width, depth, batch size, sequence length, and training steps are met. Not too much explanation is provided here. How are these specific conditions obtained? Does this indicate the transability of the method is only applicable in certain conditions.

- The method enables HP transfer across width, however, a stronger model in the model family usually comes from increased depth, which also contributed the most to the latency of the model with the number of weights being the same compared to a wider model. It is not clear whether the method can generalize to depth. 

- The authors mostly take shallow networks for illustrations or experiments to show the instability of optimal HPs across width, such as Figure 3 & 4. However, it is not clear whether it still holds when the network depth increases. 

- In Fig 9, the variation of optimal HPs for ResNets with SP on CIFAR-10 is not quite evident. Also for large datasets such as ImageNet, the figures of instability of SP are not provided and the improvement of \muTransfer over SP is not that significant (0.41%). What is the gain of HPO in comparison with the default HPs, such as LR=0.1, momentum 0.9 in the original paper? It is not clear to me why some wired HPs used in the ResNets experiment in D.1.2, such as 2.048, 3.05, 0.875.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary:

This paper makes the observation that when initializing neural network with the *maximal 

update parametrization* technique, certain hyperparameters such as learning rate and learning rate schedule are transferrable across different scales of the network (e.g. width of each layer). This insight enables to authors to tune hyperpameters using a small network and directly transfer the tuned results to much larger networks, saving the cost of tuning larger networks. Although such tuning-and-transfer approach is probably already a standard practice, the authors point out that such practice is not optimal if we initialize neural networks using the standard technique and using the proposed method generally leads to much better transfer result. 

I find this work to be interesting and see the potential of this submission to inspire future work. That being said, I want to point out that the *maximal update parametrization* technique is already published in a previous work and this submission should be viewed more as an add-on to this prior work. Practically speaking, I should also note that not all hyperparameters are transferrable as pointed out by the authors candidly. Certain hyperpameters such as regularization strength are not transferrable across scale. Such hyperparameters may have complex interactions with other hyperparameters and therefore make the proposed method less useful. Given these two reasons, I am leaning toward accept but have some reservation. I detail my other concerns in the following section.
main_review: Pros:

- the submission is novel and aims at solving a practical problem
- the paper is well-written

Cons/Questions:

- the submission is largely built upon the prior work "maximal update parametrization"
- I wish the author spent more time on explaining the intuition of the finding, namely in section 2. In line 91, the authors mention that the hyperparameters are similar to "c". However, I fail to see this connection. I am wondering whether is it reasonable to think of the  test-set perforance as a bounded continuous function of the hyperparamter. I would suggest the authors to expand on this analogy and also try to provide more intuition of this finding.
- I want to double check my understanding of the experimental details.
    - In Table 6, how is the total speed up ratio calculated? Why is it different from model speedup?
    - How does such direct transfer of hyperparameter compare to tuning the hyperparameter with a short amount of time? In my experience, the training loss decreases much slower toward the second half of pretraining and we should be able to tell the better hyperparameters early on. If the authors have saved all the intermediate checkpoints for BERT pretraining, this experiment should be easy to add.
- nitpick: not sure if the terminology "zero-shot hyperparameter transfer' makes sense to me because it is not clear what "few-shot" or "many-shot" hyperparameter transfer would be. 
limitations_and_societal_impact: I don't see negative societal impact from this work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors build on the recently developed Maximal Update Parameterisation
($\mu$P) to develop a framework for using the same hyperparameters on
models with variable numbers of hidden units or channels. The authors
demonstrate the value of this approach on large benchmark tasks, such as
IESLT14 De-En, WNT14 En-De and training BERT-large on wikitext-2.
main_review: 
## Originality

> Are the tasks or methods new? Is the work a novel
> combination of well-known techniques? (This can be valuable!) Is it clear
> how this work differs from previous contributions? Is related work
> adequately cited?

This is an application of the Maximal Update Parameterisation to the
problem of hyperparameter tuning. The paper provides theoretical and
extensive experimental exploration of this topic.

The experimental results showing transfer of hyperparameters regardless of
model size is the main original contribution of this paper.

## Quality

> Is the submission technically sound? Are claims well supported
> (e.g., by theoretical analysis or experimental results)? Are the methods
> used appropriate? Is this a complete piece of work or work in progress? Are
> the authors careful and honest about evaluating both the strengths and
> weaknesses of their work?

Experimental results are concrete and described in great detail. The
theory described in the paper describes the Maximal Update Parameterisation
and the background for it's development. The aim of this parameterisation
is to be invariant to hyperparameters so it is natural that it would be
useful for this task.

Tables 4, 5 and 6 list experimental results where the training of models
that are being compared against diverged. This does not give a fair
comparison to existing work. In practice, practitioners would try to tune
the model hyperparameters to the new setting using hyperparameter tuning
techniques or personal expertise. A standardised way to train the models so
that they do not diverge would be a fairer comparison.

A natural choice for this would be to automate the traditional method of
tuning hyperparameters on a subset of the full dataset or a single
minibatch before running it on the full dataset. Random search over
hyperparameters could provide a good baseline using this method to compare
against.

The experiments presented expand upon the cases where this method of
hyperparameter transfer performs well but fail to explore where this model
fails. For example, it is mentioned that the method does not transfer to
architectures using different regularisation methods but this is not
explored in experiment.

It is not clear how this model behaves while varying the hyperparameters
that can be transferred across, as listed in Table 1. The authors do not
explore the regions where the maximal update parameterisation breaks down.
In other words, how much does the underlying architecture have to change
before training with hyperparameters chosen on a small network breaks down?

## Clarity

> Is the submission clearly written? Is it well organized? (If not, please
> make constructive suggestions for improving its clarity.) Does it
> adequately inform the reader? (Note that a superbly written paper provides
> enough information for an expert reader to reproduce its results.)

The work very clearly states its strengths and immediately explains why the
parameterisation is ideal for this task. Figure 1 is well placed to explain
this to the reader and communicates the main thesis of the paper clearly.

The paper editorializes unnecessarily. For example, at the start of Section
3 and Section 5.1, the authors make a statement about community opinion
without citation. Footnotes generally also express the author's opinions.

In the introduction a sequence of numerical bullet points are presented in
a paragraph instead of as a list. It would be clearer as a list.

## Significance

> Are the results important? Are others (researchers or practitioners) likely
> to use the ideas or build on them? Does the submission address a difficult
> task in a better way than previous work?  Does it advance the state of the
> art in a demonstrable way? Does it provide unique data, unique conclusions
> about existing data, or a unique theoretical or experimental approach?

Parameterisation in deep learning is important. This work demonstrates a
valuable application of a specific parameterisation and how this can
massively improve contemporary deep learning practice.

The experiments provided are extensive and could provide a useful reference
to researchers who would like to use the proposed parameterisation.

One detriment to the significance of this work is that it did not introduce
the Maximal Update Parameterisation. The experiments in this work follow
directly from that development without extra development work. The major
contribution of this work is the experimental results demonstrating that
this parameterisation still works on large deep learning problems.
limitations_and_societal_impact: Comparisons in the experiment section to similar works or alternative
solutions for hyperparameter search are lacking. As noted in the main
review, the authors list that the methods they are comparing against
"diverged" in Tables 4,5 and 6. It would make more sense to compare against
a contemporary work focused on the same problem of hyperparameter transfer.
See the main review for suggestions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an alternative method for tuning hyperparameters which is to tune hyperparameters on a small width model and transfer these sets of hyperparameters to the full network. The core idea is to utilize the maximal update parameterization so that optimization hyperparameters remain stable under model size changes. This enables a zero-shot transfer and makes the hyperparameter optimization process more efficient both in computation and memory. Empirically, the authors show that the careful hyperparameter selection on the small model with the proposed approach outperforms BERT-large on MNLI and OOP and also requires less total training costs.
main_review: Originality: 
* The core principle of the method is based on Maximal Update Parametrization (\mu P) and the work is incremental to show hyperparameters such as learning rate are stable under model size changes. 
* However, the idea of zero-shot hyperparameter transfer is interesting and introduces a different perspective in hyperparameter optimization.
* It is clear how this work differs from previous contributions and the related work is adequately cited.

Quality: 
* The submission is technically sound and claims made in the paper are well supported. The authors carefully discussed the weakness and strengths of their proposed approach.

Clarify:
* The submission is clearly written and well-organized. It was a good read overall.

Significance: 
* The results presented in the paper are important and interesting. I expect other researchers or practitioners likely to use the idea or build on them. 
* One concern I have with the paper is the limited novelty. The core foundation of the paper is based on Maximal Update Parametrization (\mu P). However, the investigation of several hyperparameters' sensitivity on \mu P and experiments to analyze this behaviour is interesting and promising. 
* Another concern I have is the limited generalizability (scalability). The proposed parameters cannot handle regularization hyperparameters such as weight decay and dropout. Since the modification of such regularization hyperparameters may change the optimization hyperparameters, in the end, to obtain the state-of-the-art results, I feel like hyperparmaeter search on the full model is inevitable. I believe that future works are necessary for the proposed model to be fully utilized in practice.

Minor Comments:
* In footnote 8, it should be "in fact" -- > "In fact".
* The use of \times instead of x would be helpful in section 6.

--------

I thank the authors for their response. I acknowledge that I read the authors' responses and other reviewers' comments. The authors addressed some of my concerns on the novelty of the proposed approach. Regarding the second issue, I understand that the proposed approach achieved good results in GPT-3 that doesn't require much regularization. However, for models that require careful tuning of regularization hyperparameters to obtain the state-of-the-art results (e.g. ImageNet classification), hyperparameter search on the full model is inevitable and the proposed method wouldn't be helpful. Given these reasons, I would like to remain at my current score.
limitations_and_societal_impact: The authors adeptly address the potential negative social impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Based on recently proposed Maximal Update Parametrization ($\mu$P), this paper finds that hyperparameters can be copied (i.e. transferred with zero cost) across model sizes. Therefore, it is possible to tune hyperparameters in a small model, and to apply hyperparameters in a large model.
main_review: Although this paper claims it is Zero-Shot Hyperparameter Transfer, I would rather say that they find some scaling rules for hyperparameter tuning. This paper heavily depends on [40], and can be seen as an empirical validation of the theory proposed by [40]. I suggest that the authors weaken the claim that this is the first paper to explore zero-shot hyperparameter transfer, because learning rate scaling rules [31] / initialization rules (Kaiming initialization and Xavier initialization etc) all fall into this category of "zero-shot hyperparameter transfer".

Large models like GPT-3, T5 are not only difficult to train, but also difficult to tune. Even research institutions in large companies like google and facebook cannot afford grid-search tuning of hyperparameters for these large models. Therefore, **the demonstrations in this paper can have a huge impact for training large models**. It is thrilling to see that we can tune hyperparameters in a small Transformer and expect better performance in a large Transformer, which is also validated in ResNets. Based on the potential impact, I give a positive review. But to be honest, the technical novelty is limited, cause this paper is a direct application of the theory in [40].

I'd like to increase the score if my concerns are addressed:

(1) "this is the first paper to explore zero-shot hyperparameter transfer" is an over-claim. Please tone down the claim. Section 2 (Parametrization Matters: A Primer) talks about proper scaling of hyperparameters, not hyperparameter transfer. I think everyone can understand the importance of finding the proper rule of hyperparameters scaling with respect to model scaling. In my humble opinion, section 2 is uninformative. Maybe authors can move some content from the supplementary material to the main paper to replace section 2.

(2) why regularization cannot be transferred? Since the main goal of this paper is to verify hyperparameter scaling rules, intuitively there should also be a scaling rule for regularization factors.
limitations_and_societal_impact: Limitations are discussed in the paper.
review_title: null

Meta-review: This paper proposes an approach for "zero-shot transfer" of optimal hyperparameters from small models to large models, which enables efficient tuning of large models such as GPT-3. Such a study on efficient tuning of large models is highly anticipated by our community, and may inspire future research in this area. Reviewers generally recommended acceptance (4 out of 5), acknowledging the significance and experimentation of this work. They, with one referee still below the bar, pointed out several important concerns. Authors shall revise their paper to: 1. Elaborate the novelty by connecting to the Maximal Update Parameterization [40], highlighting clearly which part is due to [40] and which part is novel in this paper; 2. Town down the claim of "zero-shot" hyperparameter transfer, because hyperparameters of the regularization and the optimizers cannot be transferred, and the hyperparameter transfer can only be applied within the same model family; 3. Document clearly in what cases their approach is applicable, e.g. which hyperparameters can be transferred and which ones cannot, and how practitioners can deal with the difficult ones -- simply bypassing difficult cases is not good, because it will make scope of the approach quite narrow. I would stress that the author rebuttal did not respond to all the concerns, so the acceptance is conditioned on the authors' revision, in which at least the meta reviews must be well addressed.
Decision: Accept (Poster)