Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies neural architecture search over autoregressive transformer models, and proposes a training-free approach to search the Pareto frontier of multiple objectives: latency, memory and perplexity. The proposed approach is based on an important finding that the perplexity is highly correlated with the number of parameters in the decoder, which offers a cheap and convenient way of model evaluation.

strengths_and_weaknesses: Strengths:
- The paper proposes a training-free NAS approach to discover the Pareto front of auto-regressive transformer architectures, which can significantly reduce the searching cost in designing efficient models.
- The authors provide numerous empirical results to justify the training-free proxy, together with its relation with the latency and memory of target devices.
- Well written and easy to follow.

Weakness:
There are still several concerns with the finding that the perplexity is highly correlated with the number of decoder parameters. 
- According to Figure 4, the correlation decreases as top-10% architectures are chosen instead of top-100%, which indicates that the training-free proxy is less accurate for parameter-heavy decoders. 
- The range of sampled architectures should also affect the correlation. For instance, once the sampled architectures are of similar sizes, it could be more challenging to differentiate their perplexity and thus the correlation can be lower.


Detailed Comments:
1. Some questions regarding Figure 4: 1) there is a drop of correlation after a short period of training, which goes up with more training iterations; 2) the title "Top-x%" should be further explained;
2. Though the proposed approach yields the Pareto frontier of perplexity, latency and memory, is there any systematic way to choose a single architecture given the target perplexity?

questions: Please address the comments above.
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of efficient deployment of autoregressive transformer models in resource constrained environment. Finding architectures that can solve the dual problem of latency and accuracy in such environments can be difficult. This problem gets exacerbated when one tries to deploy across multiple hardware platforms. NAS, via its SuperNet methodology,  has been proposed to manage this problem. However, SuperNet training is expensive and can lead to large memory requirement. Additionally, training supernets is non-trivial as subnets can interfere with each other and rank correlation between sub architectures is not preserved. Non SuperNet approach to NAS generally have an expensive architecture evaluation phase where accuracy on a training data is measured, which makes it difficult to use in practice.

This paper makes the following contribution in this problem domain -
1. The paper finds the surprising correlation that for autoregressive transformer model, the number of decoder parameters is a good proxy for validation perplexity. This allows for a very simple architecture evaluation phase.
2. Building on this insight, they develop a lightweight transformer search algorithm that combines evolutionary search with this simple accuracy proxy and real runtime measurement on hardware to develop a framework that provides a better accuracy-runtime pareto frontier than the traditional method of layer scaling. 
strengths_and_weaknesses: Strengths - 
1. Comparison with a large number of simple accuracy proxy methods from literature
2. Ablation studies like the one to understand the impact of topology of the model on accuracy and how it impacts the proposed proxy method for accuracy
3. The proposed solution can easily adapt to any hardware platform, while not losing its efficiency
 

Weaknesses -
1. The entire paper hinges on the idea that parameter count is a good proxy for validation perplexity. However, the validation of this claim is not done cleanly. The authors rely on training results on wikitext-103,  an extremely small dataset. Additionally, the use the same training hyperaparameter for different models to verify their claim for parameter count serving as a proxy for validation perplexity. Autoregressive models are trained on far larger datasets[2][3]. Infact, training on small datasets can have a significant impact on validation perplexity [2]. Additionally, dataset is expected to scale with size of the model.  Finally, [1] also shows that using the same training hyperparameters for models at different scale can lead to poor accuracy. These facts indicate that this claim is based on weak experimentation methodology. As a result, it becomes unclear if a more appropriate experiment method will lead to the same conclusion.

2. Additionally, using validation metric as a measure of quality of an autoregressive model is also questionable. Autoregressive models have shown immense benefit in generative, zero shot and few shot/prompt tuned tasks. Additionally, these are the tasks for which these models are being currently deployed in the industry (see cohere.ai, OpenAI, AI21 APIs for these models). Given that these tasks represent the way trained autoregressive models are consumed in the industry, measuring accuracy based on some of these tasks would make sense. Its unclear whether validation perplexity is a good proxy for these tasks or not. This further makes it difficult to understand the impact of this work and the usefulness of the result.

3. Their claims on impact of topology and relevance of the proxy metric for iso-parameter, different width/depth factor models is also suffering from similar methodology problem. As you make models deeper and thinner, initialization becomes extremely important to get good perplexity score. As a result, if the init of parameters is not explored, accuracy will suffer, making the claims harder to justify.

[1] https://www.microsoft.com/en-us/research/blog/%C2%B5transfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks/

[2] https://arxiv.org/abs/2203.15556

[3] https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html
questions: Questions - 

Can the authors comment on Weakness (1), (2) and (3)?
limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use the number of decoder parameters as a training-free metric for autoregressive transformer architecture search for language models. The paper then shows that the metric correlates strongly with the final perplexity of the model over the entire model sizes and topologies. Because of the training-free nature of the proxy proposed, the authors show that using otherwise standard search algorithms such as evolutionary search, it is possible to optimise for architectures that trade off perplexity, memory and latency on devices with modest computational power directly, and it is possible to recover much better Pareto frontier using the method proposed compared to the naive scaling of baseline architectures. 

strengths_and_weaknesses: Overall I believe this is a sound paper with potential for real impact, and the writing and presentation are clear and good throughout. I have outlined my detailed comments on strengths and weaknesses below.  Note that I have combined this section with my questions (next section) as many of them are related, and given the time constraint for the rebuttal, I do not expect the authors to respond thoroughly to all of my questions and concerns especially where additional experiments are necessary; in these cases some qualitative response would suffice.


## Strengths

The algorithm proposed is simple and effective in the search spaces investigated by the authors: the number of decoder parameters is very cheap to obtain, and is arguably cheaper than many of the previous zero- or low-cost proxies proposed in the NAS literature (also considered by the authors in Fig 5). The ability to directly search for better architectures on target devices with computational constraints as a result of the lowered computational cost can lead to very tangible benefits, so this paper has potential to be very impactful.


## Weaknesses & questions

1. A concern I have is that the fact that performance (perplexity in the NLP context) is correlated with some measure of #params is highly expected in general and is also demonstrated to be the case in NAS in particular (see the results of the size search space of NATS-Bench, an extension of NAS-Bench-201 [1]). In many cases, some measure of #params has been used as a search objective alongside performance [2] (not in this case, as the authors used mem usage and latency); this shows that the correlation between that and performance is assumed. While the paper does show some nuance by demonstrating the superiority of #decode params over the total #params (Fig 6), I am still a bit unconvinced by the fact that most of the analysis (Sec 4.1 - 4.3) use the absolute performance as the outcome variable, even though ultimately we are interested in the *trade-off* between performance and #params by finding a family the architectures that maximise performance at different part of the Pareto frontier instead of just maximising performance in Sec 4.4. It is worth noting that the existing zero-cost proxies the authors compared like synflow and jacob_norm were largely originally evaluated in cell-based search spaces where we often only aim to find a single architecture with its accuracy optimised [3,4], and that’s why it is more justified to use the absolute performance as the dependent variable in that case.  
	
      Given this goal of finding Pareto-optimal family of architectures, I feel that analyses should instead pivot on the predictive ability of the # decoder params after the effect of other confounding variables are removed: for example, how well does #decoder param predicts perplexity on architectures with similar mem usage and/or latency (using a metric such as the partial correlation)? I feel this is a more direct test of how well the proposed proxy can truly help discover more Pareto-optimal architectures. While the authors did discuss these briefly in Appendix C, I believe these analyses are rather important and should be expanded: for example, 1) the paper considers both latency and mem_usage as optimization objectives, but App C only considers latency; 2) similar to how NAS predictor can often predict well over the entire search space but break down in high-accuracy regions (which are also what we care about), the proxy proposed might also suffer degradations nearer to the Pareto frontier and it would be interesting to investigate whether that is the case and 3) the results in App C do show that the proxy quality is worse at larger models, and it could be interesting to explain/investigate why it might be the case and discuss potential remedies.

    On a related note, I feel it would also be good for the authors to include some explanations or intuitions on why do they think why # decoder params is a great proxy in this case. I feel that this could potentially be inspirational for future works that would like to develop on the idea but, for example, on a different search space or domain of application. It is also helpful in this particular case as this is the main innovation (the search techniques and other part of the algorithms are otherwise standard).

2. Experiments: Sec 4.4 is where the main experiments are demonstrated to show promise of the method. My main concern is that while the results are promising, the baselines seem a bit weak that the authors simply scale the base, homogenous Transformer and GPT models. I am wondering whether it is possible also to compare against a single or a family of NAS architectures from, for example, one or more of the prior works the authors cited in Related Works?

3. Compatibility/sensitivity with varying training hyperparameters: One thing also noted by the authors is that the best Transformers is often obtained by also tuning the training hyperparameters in addition to the architectural hyperparamaters (footnote 2). One interesting thing to investigate is whether the the suggested proxy still predict performance well when there are changes in hyperparameters. 

4. Related work: there is also a recent work that proposes training-free transformer search proxy [5]. While I recognise that since this paper is only published at CVPR 2022 it is concurrent to the current submission (and my rating is independent of this paper), the authors are encouraged to include some qualitative discussions & comparisons w.r.t this work.

5. Train code: the authors have not provided code for the paper and mention they will open source the model upon acceptance. Would you please also clarify are you going to open-source the code? I think it would greatly enhance the reproducibility & benefit the broader community more if both code and models will be released.


## References

[1] Dong, X., Liu, L., Musial, K., & Gabrys, B. (2021). Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence.

[2] Chen, M., Peng, H., Fu, J., & Ling, H. (2021). Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12270-12280).

[3] Abdelfattah, M. S., Mehrotra, A., Dudziak, Ł., & Lane, N. D. (2021). Zero-cost proxies for lightweight NAS. ICLR.

[4] White, C., Zela, A., Ru, R., Liu, Y., & Hutter, F. (2021). How powerful are performance predictors in neural architecture search?. Advances in Neural Information Processing Systems, 34, 28454-28469.

[5] Zhou, Q., Sheng, K., Zheng, X., Li, K., Sun, X., Tian, Y., ... & Ji, R. (2022). Training-free Transformer Architecture Search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10894-10903).
questions: Please see Strengths And Weaknesses
limitations: The authors have adequately discussed the limitations of their work both in Conclusion and in appendix. The authors should also be given credit for interleaving several broader impact discussions in the paper (rather than as a standalone section), such as the estimation of both GPU hours and $CO_2$ emission statistics in Table 1.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper adapts the number of decoder parameters as the search proxy to accelerate the design loop for decoder-only language models. The authors verified the assumption using 2800 trained data points  and based on the assumption they show 1.6-2.5x faster runtime speed and 1.3-2.0x lower peak memory on various hardware. 
strengths_and_weaknesses: Strength

- The paper tackles on an important problem. 
- The problem formulation is clear and writing is well presented.
- Evaluation in Figure 3 shows the correlation between #num parameters and final model performance.
- The pareto-frontier exploration demonstrates significant improvements compared with naive transformer scaling.


Weakness 

- The biggest assumption highly relies on decoder-only architecture. L377: Our focus is entirely on autoregressive, decoder-only transformer. This is not a general solution to explore NAS on tasks like NMT.
- Only compared with scaled transformers, which is not considered as strong baseline. Since this is NAS work, the latency and accuracy from models derived with AutoML techniques should be discussed.
- Free proxy does not mean free evaluation — the generated architecture still need to perform full training to obtain the model weights. Then even the search cost is reduced, the total cost will not be saved much compared to once-for-all studies like [1, 2].

1. Hat: Hardware-aware transformers for efficient natural language processing.
2. Prune Once for All: Sparse Pre-Trained Language Models
questions: In Figure 3, WikiText-103 is a relative small dataset compared to standard GPT-2 level model trainign (e.g., WikiText of billions of tokens). If the dataset size becomes larger, will the precision of #num_of_parameters proxy become better or worse?

In Figure 8, the authors show PPL on evaluation. However, in Algorithm 1 there is no description about how the model is trained. Is every model trained separately, or there is parameter sharing techiniques?
limitations: Overall this is a good paper with clear method and solid experiment results. If the authors can address the aformentioned weakness above, I would like to improve the score.
review_title: null

Meta-review: metareview: Building on the novel observation that there is a strong correlation between model quality and number of parameters in the decoder of autoregressive Transformers, this work proposes a training-free NAS algorithm for this class of models.

The main concerns raised by reviewers are related to the key observation the paper is built on (this correlation), that may not necessarily hold in all situations. However, authors were able to clarify important points, provide additional motivations and empirical results, eventually leading to all reviewers leaning towards acceptance.

Considering the popularity of this class of models, I believe this work should be of significant interest to researchers and practitioners in the field, and I am thus recommending acceptance.
confidence: Less certain
Decision: Accept