Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides various ablation studies to show the effect of scaling various architecture parameters on the performance of a downstream task. The studies are performed on encoder-decoder transformer architectures that follow the T5 architecture as the backend. As a result of the ablation studies, the authors provide a heuristic on how to design architectures that lie on a better Pareto front compared to the previously proposed T5 architectures.
main_review: Strengths:
- The findings in the paper are useful for researchers in this domain from two folds: Firstly, by open-sourcing the checkpoints, others can take advantage of the trained models to foster research in various NLP applications. Secondly, the heuristics provided can potentially guide the design of better architectures for downstream tasks.

Weaknesses: 
- The plots are often hard to read and have very bad visualization. As an example, the labels in Figure2-c are unintelligible. As such, it is hard to drive conclusions from the plots, rather the reader has to refer only to the text, which is not adequate. It would be great if the authors improve the plots and ensure that all text/labels can be easily read.
summary_of_the_review: Please see above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims at providing insights from scaling Transformers for pre-training and finetuning. Based on extensive experiments involving pre-training and fine-tuning over different transformer configurations, the authors find that the model shape matters a lot besides model size when considering the downstream performance, and scaling strategies differ at different compute regions. With these new findings, the authors propose the DeepNarrow scaling strategy and verify it on additional experiments including tasks in different domains (language and vision).
main_review: **Strengths**

* The paper is clearly written and the figures and tables are illustrative.
* The empirical efforts on extensive experiments deserve to be appreciated. The model checkpoints and codes will be publicly released to the community.

**Weaknesses**

* The novelty of this paper is limited. 
  * Regarding the scaling laws: the motivation of this paper is that [1] only consider the pre-training performance (loss/perplexity on upstream datasets) and does not investigate the downstream performance. The authors try to fix the bias of the previously established scaling law by investigating the downstream performance. Incremental experiments investigating the downstream performance are provided. Besides, the authors do not provide further quantitative analysis like that in [1] under this new setting.
  * Regarding the proposed DeepNarrow strategy: based on the observation of the experiments, the authors propose to preferentially increase the model's depth when scaling the models. However, this finding has been discussed in [2] and the authors of [2] provide both theoretical and empirical analysis on the depth-width tradeoff of Transformer models. The authors should include this work and discuss it.

[1] Kaplan et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

[2] Levine et al. "The Depth-to-Width Interplay in Self-Attention." NeurIPS 2020.

summary_of_the_review: Considering the novelty of this paper, I vote for 'weak rejection'. If the authors can addressed my concerns, I would like to increase my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents scaling insights from pretraining and fine-tuning Transformers empirically. The main findings are as follows: (i) model shape matters for downstream tasks; (ii) scaling protocols operate differently at different compute regions; (iii) T5-base and T5-large are not Pareto-efficient. They will publicly release over 100 pretrained checkpoints for further research.
main_review: Strengths: This paper is the first one to discover the scaling insights on both pretraining and fine-tuning Transformers. The authors perform comprehensive experiments and provide insights from different perspectives. Even though some insights are not surprising, this paper provides evidence to support some previous intuitions.

Weaknesses: Since this paper focuses on empirical conclusions, the novelty is somewhat weak. Furthermore, the experiment results may be effected by the randomness and I don't find the error bar in the paper, which is the most concern of mine. Also, I have the following questions about the paper.

1. In the caption of Figure 1, I don't see variable P (default model parallelism) in the table. Can you provide some explanation about it? 
2. As discussed, do you perform repeated experiments to avoid the influence of randomness? I guess the main conclusions in your paper like Figure 1 should be performed repeatedly.
3. T5 is a sequen-to-sequence model. I am curious about your conclusions for different models like ViT or BERT. Can you perform some experiments on different models to verify the generalization of your conclusions?
summary_of_the_review: The paper study the empirical findings of scaling insights for Transformers on both pertaining and fine-tuning and provide a list of conclusions. Even though the conclusions are not surprising to me, I think this paper provides some evidence to prior works and future works. One main concern of mine is the randomness of the paper since this paper performs comprehensive experiments and concludes insights based on the results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents solid experiments across many NLP benchmarks to show that the perplexity of the upstream language model can be a deceiving indicator of downstream quality. They also show that popular language model like T5-base and T5-large are relatively inefficient and scaling strategies differ at different compute regions, i.e., applying the same strategies at different compute regions has a different effect on model quality. Finally, they give a simple DeepNarrow strategy that can be applied to different model sizes and make them efficient while preserving their performance. 


main_review: Strengths
* This paper conducts solid experiments across many NLP benchmarks to demonstrate the discrependcy between upstream perplexity and downstream performance. 
* This paper also shows scaling strategies differ at different compute regions, i.e., applying the same strategies at different compute regions (small vs large) has a different effect on model quality.
*  This paper presents a DeepNarrow strategy that is applicable to all model sizes and can reduce model parameters and improve speed while preserving downstream performance.

Weakness
* The discrepancy between upstream perplexity and downstream performance is somewhat related to model selection with dev set in natural language generation task. Model selection with dev set according to perplexity may underperform model selection strategy according to downstream metric, e.g. BLEU score. Therefore, the discrepancy found in this paper is not "surprising"!
* DeepNarrow strategy seems also to be discussed before in other neural architectures. In Figure 6.7 of [1], discussed Conv nets with the same parameters but different layers and found that deeper models perform better. Therefore, the novelty of this strategy is not significant.

[1] Goodfellow et al. Deep Learning. 2016 
summary_of_the_review: This paper conducts solid experiments to support its claims, e.g. discrepancy between upstream perplexity and downstream performance and scaling strategies difference at different compute regions and DeepNarrow strategy to improve model speed and reduce parameters. Some of the findings provided in this paper is kind of intuitive and the DeepNarrow strategy has been shown in previous literature, which hurt its novelty.
review_title: null

Meta-review: The results reported in this paper and the model checkpoints released are of interest and broad utility to the community in the opinion of the NLP.  While one reviewer was somewhat negative, most reviewers were in favor of acceptance of this paper, which expands the results from [1] to downstream tasks. The AC therefore recommends acceptance.
Decision: Accept (Poster)