Scale Efficiently: Insights from Pretraining and Finetuning Transformers
Authors: Yi Tay
Mostafa Dehghani
Jinfeng Rao
William Fedus
Samira Abnar
Hyung Won Chung
Sharan Narang
Dani Yogatama
Ashish Vaswani
Donald Metzler
Venue: ICLR 2022 Poster (2022)
Keywords: transformers
attention
deep learning

Abstract: There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.

Published as a conference paper at ICLR 2022
SCALE EFFICIENTLY: INSIGHTS FROM PRE-TRAINING AND PINE-TUNING TRANSFORMERS
Yi Tay", Mostafa Dehghani",. Jiifeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama' Ashish Vaswani, Donald Metzler Google Research & DeepMind (yitay, dehghani C @google.com
ABSTRACT
There remain many open questions pertaining to the scaling behaviour of Trans- former architectures. These scaling decisions and findings can be critical, training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. (2020) presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear these set of findings transfer downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are follows: (1) we show that aside from only the model size, model shape matters for downstream fine- fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficci we present improved scaling protocols whereby redesigned models achieve similar downstream fine-tuning quality while having 50% fewer parameters and training 40% faster compared the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.
To this end,
INTRODUCTION
Training Transformers incurs both financial and envirommental costs (Schwartz et al., 2019; Patterson al., 2021). To this end, researchers and practitioners often have to work around fixed compute budgets and figure out the best ways to train their models. In lieu of the rising computation demand for training state-of-the-art Transformer (Vaswani al., 2017: Devlin al., 2018; Raffel et al., 2019; Brown et al., 2020; Fedus et al., 2021) models, the goal of this paper is to present insights and lessons from scaling Transformers and making them efficient and effective for transfer learning on downstream tasks.
Despite the insights offered in scaling laws research (Kaplan et al., 2020; Hernandez et al., 2021) there remain unresolved questions: Should one follow fixed scaling ratios? If not, should one scale by depth? Or by width? Will scaling experiments on upstream pre-training generalize for downstream transfer? Do scaling protocols for small models generalize to larger models? Are scaling behaviours similar in all compute regions? We hope the insights presented in this paper can be useful to both practitioners and researchers in informing their scaling decisions.
Neural scaling laws (Kaplan et al., 2020) is a common resource that many look to for advice on scaling Transformer architectures. However, this paper limited its scope to exhaustive study of upstream cross entropy on language modeling tasks. It is furthermore unclear if findings from (Kaplan al., 2020) will transfer to downstream applications. Specifically, Kaplan al. (2020) proposed that the performance of Transformer language model strongly depends on model size and only weakly on its shape. They also argue that many model configurations with the same number of parameters perform similarly regardless of architectural details. Our work empirically confirms this on upstream training but finds distinct discrepancy when considering practical downstream performance key insight that we believe is highly important.
Equul contribution

---

Published as a conference paper at ICLR 2022
To this end, conduct extensive experiments involving pre-training and fine-tuning over 200 transformer configurations ranging from 5M to 30B parameters. To the best our knowledge, this is the largest empirical study of practical scaling of transformer date that considers both upstream and practical downstream transfer. While there have been many proposed scaling protocols for ConvNets (Tan and Le, 2019; Bello et al., 2021), there is still limited advice scaling of transformer architectures, apart from (Kaplan et al., 2020; Li et al., 2020). Hence, the key goal of this paper is to distill our experiences and insights with scaling Transformer architectures and share them with the broader community.
Contributions
The overall findings and insights of the paper can be summarized as follows:
We find that scaling laws may differ in upstream and downstream setups. Specifically, contrary to Kaplan et al. (2020), we find that downstream performance strongly depends on shape and not only on model size. Henee, pretraining performance may not necessarily transfer to downstream applications. (Figure 1).
Our findings show that pre- vre-training perplexity can often be a deceiving indicator of down- stream quality and therefore model building based on upstream perplexity can be challenging. Scaling laws can differ substantially when considering metrics on actual downstream fine- tuning. (Figure 1)
Given that empirical scaling laws differ when considering quality on the downstream, our work investigates the pareto-frontier of transformer configurations in this setup. We find that the canonical model configurations such as 75-Base and LarL sizes (Raffel et al., 2019) are relatively inefficient (Figure 2). Note that these sizes are based off the canonical BERT (Devlin et al., 2018) base and large sizes.
We find that scaling strategies differ at different compute regions, i.e., applying same strategies at different compute regions (small vs large) has different effect model quality. This has practical implications since finding strategies at small scale might not necessarily transfer or generalize to higher compute regions (section 4.2).
After extensive empirical exploration of the pareto- fronterr of transformer models, we propose a simple but effective scaling strategy which we call the DeepNarrow strategy. We show that we are able to obtain model quality par better than canonical model sizes (e.g., base) with 50% less parameters and being 40% faster. While we highlight the limitations of this strategy, we also show that this DeepNarrow strategy is applicable to all model sizes. (Table 4).
To consider how generalized these scaling strategies are, we conduct additional experiments Vision Transformers (VIT; Dosovitskiy et al., 2020) to verify them in the vision domain. Moreover, on top of the 17 GLUE (Wang et al., 2018) / SuperGL.UE (Wang et al., 2019) and SQuAD (Rajpurkar al., 2016) tasks employed in extensive study, we verify findings via additional downstream experiments across 12 diverse language tasks (section 22)
We release (1) the pre-trained checkpoints for our T5 models with improved scaling proto- cols and (2) all 100+ model checkpoints, including intermediate training checkpoints to the research community. We believe that this is a treasure trove of data to study the behaviour of large LM pretraining and finetuning especially pertaining to scaling laws. The check- points and code will be released https: /estps://giie /ttttttttttu//////// com/ google-reses.oooooe cacch/ The checkpoints are now publicly available at our Google Cloud Bucket gs: //:://sces /sceniieeieeee cubucket/ s scal iiliinng__ nig_exploreee sear/scaainn wewaaaanleepplrrrr_eeeeee More recently, these checkpoints are also now available on Huggingface https: tttpps:/husse http://hgggggccceeeee/e// co /ww//oooelllseeooo ? hoth deep-ne -arrow.
htt////ttttssaaaaeeeeeee ltriiingggnan_ieeeeee trans
2 RELATED WORK
Transformers ( (Vaswani al., 2017) have become ubiquitous in the modern deep learning stack and have seen widespread impact across not only language (Devlin et al., 2018; Raffel et al., 2019; Brown al., 2020) but also computer vision (Dosovitskiy et al., 2020; Amnb et al., 2021), reinforcement
2

---

Published as a conference paper at ICLR 2022
Table 1: Table of model configurations. Nr, is the number of layers, drf is the size of the MLP, dmodet is the hidden size of the model. dxu the size of each key-value vector. Nr is the number of heads. P is the default model parallelism.
Table 2: Description of different knobs used in the paper to define scaling operations.
Scaling Op
Description
Model
N1
dys
dmodel
dRv
H
#Params
NL
Num. layers
Tiny
4/4
1024
256
16M
EL
Num enc. layers
Mini
4/4
1536
384
32
8
31M
DL
Num. dec. layers dmodel
Small
6/6
2048
512
32
8
60M
DM
Base
12/12
3072
768
64
12
220M
KV
dRv
Large
24/24
4096
1024
64
16
738M
NH
Num. of heads
24/24
16384
1024
128
32
3B
FF
dys
XXL
24/24
65536
1024
128
128
11B
SH
Shared heads
XXXL
28/28
131072
1280
128
256
30B
SKV
Tied key-values
learning (Parisotto al., 2020) and computational biology (Senior et al., 2020). To this end, discovering empirical scaling laws of these models is research area that has garnered considerable interest (Kaplan et al., 2020; Henighan al., 2020; Hernandez et al., 2021; Bahri et al., 2021).
Discovering empirical scaling laws that govern neural language model scaling has been recent subject of keen interest (Kaplan al., 2020; Hernandez et al., 2021; Bahri al., 2021). Many of these works present scaling laws across variety of axis such as model size, compute and/or dataset size. It is worth to note that many of these works evaluate on autoregressive language modeling and use cross entropy loss to measure performance (Kaplan et al., 2020; Hernandez et al., 2021). There are a multitude of interesting findings presented (Kaplan et al., 2020) whereby the authors show that performance (loss) scales power-law with model size and dataset size. However, notable claim is that architectural details (e.g., network depth and width) have minimal effects. Subsequently, Hernandez al. (2021) builds upon the work of Kaplan al. (2020), evaluating scaling laws for 'transfer'. To this end, the authors study the effect of dataset scaling on unsupervised transfer learning and finetuning. That said, the experiments of Hernandez et al. (2021) are mainly targeted dataset transfer between two different distributions (language and code) and make the same assumptions as Kaplan al. (2020) about model scaling. similar vein, Henighan et (2020) studied empirical scaling laws for different domains for generative modeling in vision, video and multimodal setups.
There have been increasing demand for training and scaling Transformers (Shoeybi al., 2019; Raffel et al., 2019 Fedus et al., 2021; Conneau et al., 2019; Lin et al., 2021). Despite the benefits from improved performance, there are financial considerations and environmental costs (Schwartz et al., 2019; Patterson et al., 2021) to training these models. Given that every moment spent on hardware accelcrators a cost incurring activity, we believe that research in distilling practical scaling insights and recommendations to be highly crucial (Li et al., 2020; Kaplan et al., 2020; Bello et al., 2021).
Notably, the research problem of making transformers efficient have also been tackled from an exten- sive number of angles such as (but not limited to) distillation (Hinton et al., 2015), compression (Zafrir et al., 2019), parameter sharing (Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021). efficient atten- tion (Tay et al., 2020c; Kitaev al.,, 2020; Choromanski al., 2020; Tay al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-base architectures (Tolstikhin et al., 2021: Tay al., 2021a; 2020a: Lee-Thorp et al.. 2021). With so much extensive research into novel techniques to improving the efficiency of transformers. it is surprising that the standard configurations (e.g., base, large) of transformers BERT (Devlin al., 2018) or T5 (Raffel et al., 2019) have not been rethought.
METHODS
This section describes our main experimental setup.
Architecture We study a Transformer encoder-decode architecture that uses relative attention based of the T5 architecture (Raffel al., 2019). The choice of adopting Seq2Seq architec-
3

---

Published as a conference paper at ICLR 2022
tures (Sutskever et al., 2014) is mainly due to their universality and ability to both subsume encoder (Bert-like) and decoder (language) models within an identical framework. Moreover, the universality of Seq2Seq architectures also allow to fine-tune across broad range of tasks. Our implementation and experiments are performed in Mesh Tensorflow! (Shazeer et al., 2018) using the default T5 library".
Model Configurations We first define eight Transformer sizes, i.e., tiny, mini, small, base, large, XL, XXL and XXXL. The small, base, large, XL and XXL corresponds to the canonical T5 sizes that are released in Raffel al. (2019). We use three other sizes as starting points, e.g., tiny and mini since there is a lack of representation of transformers at lower compute regions.
Pretraining We pretrain on the Colossal Cleaned Common Crawl Corpus (C4; Raffel et al., 2019). We pre-train encoder-decoder models using the span-based masked language modeling (MLM) objective (Fedus et al., 2018; Devlin et al., 2018). We pretrain all our models for 2 steps using 16 TPU-v3 chips. For larger models, we run our models with 64 TPU-v3 chips. We use 219 steps since majority of the experiments in (Raffel et al., 2019) were conducted in the same fashion. We would also like to emphasize that the official released T5 checkpoints were pretrained on 17 tokens (1 million steps with batch size of 2048). Given that this extended long pretraining setup is infeasible given the number of experiments we would have to run, we opt to follow the standard ablation setup (Raffel et al., 2019) which pretrains on more manageable number of tokens.
Downstream Tasks We consider myriad of downstream tasks. In total, we consider 17 tasks. We finetune on a mixture of GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019), SQuAD (Rajpurkar et al., 2016) for the key downstream experiment results and report aggregate GLUE/SaperGiL.UR scores. We believe that an aggregate of 17 tasks in natural language understand- ing that commprises of both high-resource and low-resource tasks gives us a good overview of a model's ' downstream performance. Finetuning is typically performed with 16 TPU-v3 chips.
Notation for Scaling Operators For the remainder of the paper, use shortform code for each scaling operator applied on a standard transformer size. For example, NL32-SM refers to scaling small (SM) transformers to 32 layers (NI32)). We use EL,DL to represent scaling encoder and decoder layers independently, KV to represent scaling each key-value size, DM to represent scaling dmoddl- NH to represent modifying the number heads and FF to represent scaling dFF. The initial/standard model sizes are tiny, mini, small, base, large, XL, XXL and XXXL. This is described in Table 2.
Convention With the exception of Figure 1, all charts are plotted with FLOPS as the main compute metric. We use number of params for Figure downstream results are plot with SuperGIUUE accuracy (Wang et al., 2019) as the Y-axis. Due to the lack of space, we report charts/plots of other metrics (params of speed) and other tasks (GI.UF or SQuAD) in the supplementary material. All parameter counts also include the embedding parameters. We re-emphasize that it is critical to take into account multiple facets of efficiency and therefore report all three key metries (FLOPs, throughput/speed and parameter count) in the supplementary material.
align with Kaplan et (2020). All of the
Model and Data Parallelism By default, our models are trained across multiple workers via data parallelism. As per convention in the T5 paper, our larger models the default model parallelism. Specifically, this is set to 2 for large models, 8 for XL models and 32 for XXL models.
ANALYSIS AND RESULTS
This section presents our overall findings and key results.
httpp::/ggithbb.cmm/eeeeffrfooo
4

---

Published as a conference paper at ICLR 2022
NL12-00XL
NL32-XL
-140
Negabive Log-9 Log-Perploxe
CNL--OXXX
EXt
121-0000
ANL12-00 XXL
SuperGrue Accuracy
D
KLB 3L--33
AN 12-XXL
NLG-XXL
NL36
NLO-XXL
ONL OXXI-XXXL
NL32-LG
72
ARL24
VNB8-XL
O OlarguB-XX
ONL36
NLG-XXLL
Fril
-166
CNL32
276-0
2.le19
42642
1.6119 1.7e100
17668 54440
Liee
4,3019
0.6619 1.7-10
Params
Params
(a) Pre-training scaling
(b) Fine- fine-tuning scaling
Figure 1: The predictability and unpredictability pre-training versus fine-tuning. While the upstream pre-training performance measured by negative log perppeeeyyy scales with model size quite independently from the model shape, the downstream performance (SuperGlue (avg) score) does not. This indicates that the shape of models plays an important role on how performs on the target task and the performance is not merely a function parameter size.
4.1 MODEL SHAPE MATTERS
We extend the results of Kaplan et (2020) to fine-tuning and present model shape dependence not highlighted in Hernandez et al. (2021). Kaplan et al. (2020) studies pre-training (upstream) and concludes that performance depends only weakly on model shape, but strongly on model size. Hernandez et al. (2021) extends this work to measure effective data transfer measure when pre- training and then fine-tuning on Python dataset. However, this work does not consider details of model shape, and instead focused on the relative predictability with model scale alone. Our work stands in contrasts since we find that model shape matters considerably for downstream fine-tuned performance.
Figure shows dozen Transformers. The models considered are sampled diversely within two-order of magnitude band of model scales. We adjust the model shape primarily through depth variations, starting with configurations such as XXXL (33B), XXL (11B), XL (3B) and LG (750M) parameters but have their depths/lengths modified. From Figure reveals strong correlation of the upstream performance with model size, corroborating the neural scaling laws of Kaplan al. (2020). But the strong pre- training corrllation largely vanishes when fine-tuning these models on SuperGI.UE (Wang et al., 2019). While we confirm the findings Kaplan al. (2020) that performance scales strongly dependent on model size but weakly model shape, we find that model shape (such depth-scaling is highly important for downstream transfer a dimension that is not considered Kaplan et al. (2020).
scatter plots for pre-training (left) and fine-tuning (right) over a
As substantiating point and additional context to Figure 1, we also show via counter-example that pretraining perplexity not indicative of transfer performance, i.e., we explicitly show that a case (in Table 3) where model can have outstanding pre-training perplexity but substantially undeliver when it comes to downstream performance. To the best of our knowledge, while this has been mentioned implicitly in several existing works (Narang al., 2021), this is the first work explicitly shows this point.
Zooming in Versus Zooming Out Here, one may argue that a general trend (even on downstream) may still exist if we zoom out and cover very wide range of model sizes (e.g., very tiny very large). A tiny model is not likely to outperform very large model no matter how well-configured might be. Our purpose here is not to contradict this general trend but distinguish between both arguments. We argue that, in practical setups, comparisons between models and scaling decisions often made when zooming-in in and pairwise comparisons above not largely different
5

---

Published as a conference paper at ICLR 2022
Table Upstream performance does not guarantee downstream performance. Example points from Figure 1. A model with improved upstream quality (as measured by validation perplexity) can do significantly worse on transfer if the shape setting is not right. Hence, pre- pre-training perplexity can be misleading.
Name
N .
dys
dmodei
dgc
NH
#Params
PPL
Downstream
NI.12-XXI.
12
65536
1024
128
3.6B
-1.46
85.1/76.5/88.1
NL.32-XI.
32
16384
1024
128
32
3.8B
-1.49
86.9/79.9/89..
Sugue Arsureyy
Sugur Pagurrruu Accuraryy
O
D422
Super
A
a2g
18-0
20012
1840
356-3
Me-1
3540 74--13
(a) T5-Small Model
(b) T5-Base Model
(c) TS-Large Model
Figure 2: Downstream scaling properties scale-dependent. The downstream performance on SuperGLUE has qualitatively different scaling properties across models sizes. From left to right, fine-tune model configurations closely matched to T5-Small, T5-Base and T5-Large.
models, rather those that are on the same neighborhood in the size (close in the x-axis). Thus, what claim is that when you zoom in, which is what happen in practice, it is not uncommon to cases similar to the models in Table where taking the upstream perplexity into account may lead to sub-optimal choice. It also worth to mention that zoom-ing in upstream returns very different trends compared to zoom-ing in on downstream results.
4.2 SCALING BEHAVIOUR DIFFERENT COMPUTE REGIONS DIFFERENT
In this section, we evaluate how each scaling hyperparameter and model shape influences model's position on the compute-performance chart. Figure 2 shows three plots which varying different scaling knobs. Given three starting points, small, base and large, we scale the starting points across different knobs. is clear from Figure that the effect of applying different scaling operators is very different across different compute regions. We also observe that the Pareto-boundary is also very different at different compute regions. The implications of this finding is nontrivial, since this effectively means that finding improvements at (plausibly) smaller scale in hopes that it will generalize at large scale might not be an effective strategy. This corroborates recent findings of Bello al. (2021) which studied scaling properties of ImageNet models. Their paper demonstrates that the compound scaling rules (Tan and Le, 2019) derived in small-scale regime, lead to Pareto-inefficient models when then extrapolated larger scales.
4.3
NOT ALL SCALING STRATEGIES AND MODEL SHAPES ARE CREATED EQUAL
From Figure we can also observe that different scaling strategies results in very different outcome. A common pattern amongst all three model sizes is that the NL operator has strong influence on the Pareto-frontier. On the other hand, settings such as KV (varying dew) seem to result in models that are less Pareto-efficient. We notice mixed signals from varying dmodet. In this case, scaling down results in model on the pareto-fiee mureto-frontier, but scaling up, i.e., DM2K results a highly model. When compared to scaling up, NL is substantially superior option to dmodel- We describe scaling recommendations in subsequent sections. Overall, does seem like d and Nu does not influence the Pareto frontier much as other knobs.
iretto-neeffccient

---

Published as a conference paper at ICLR 2022
Panea
Maed Mand
DMUX
Mixa
NCgettee
FLOPS
FLOPS
FLOPS
FLOPS
(a) Upstream
Supur
106913
014413 JW4C 4.052
186413 A2410
3.341 1.3422
-8642 Mar3
1.8937 42012
(b) Downstream
Figure 3: Different scaling with respects to different knobs, in upstream and downstream. On the plots, DM refers scaling model dimension, FF refers scaling FFN hidden size, NH is number of heads, and NL is number layers. Size each circle indicates the model size in terms of number of trainable parameter parameters.
Effeet of scaling different knobs Figure illustrates the effect of scaling different knobs on the compute-pefformance boundary. It becomes clear that not all strategies influence the boundary with the same impact. For example, NL has the biggest impact while NH does not influence the model's position on the graph much. Finally, also note that the effect of scaling upstream and downstream is quite different. For example, FF2K is clearly better option than the canonical base model in downstream but not upstream.
Scaling Depth vs Width In Figure 3, we also note that the NL scaling operator (depth) has generally impact the Pareto-boundary compared to width (FF). For instance, FF12K (in Figure 3 clearly not Pareto-optimal, being outclassed by configurations such as NL16- SM, EL12-SM. Likewise in the base setup, FF9K and F12K are less Pareto-efficient compared to NL16 and NL24.
that
4.4 SCALING RECOMMENDATIONS
We generally recommend a DeepNarrow strategy where the model's depth preferentially increased before considering any other forms of uniform scaling across other dimensions. This is largely due to how much depth influences Pareto-frontier as shown earlier sections of the paper. Specifically, tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise, a tall base model might also generally more efficient compared to a large model. We generally find that, regardless of size, even if absolute performance might increase the relative gain of Pareto- eefret--ffiiiency diminishes as we increase the layers, converging at 32 to 36 layers. Finally, note that our notion of efficiency here relates any one compute dimension. i.e., params, FLOPs or throughput (speed). We report all three key* efficiency metrics (number of params,
continue to stack layers,
Our concurrent work Charformer (Tay al., 2021b) makes use of a DeepNarrow inspired strategy which referred Tall in the paper.
'It is often assumed that number of parameters, speed and FLOPs tend correlate. We find that this is not always the case especially when dealing with modeling choices that influences parallelism (depth width). Therefore, we emphasize the importance reporting all key efficiency metrics.
7

---

Published as a conference paper at ICLR 2022
Table Efficient DeepNarrow alternatives the canonical T5 model sizes using the DeepNarrow strategy. Models are all Pareto efficient at least to one or more aspect of compute and one more downstream task. XXL and XL32L models are trained on 64 TPU-V3 chips and so they are faster.
Model
#Params
#TFlops
Steps/s
Ppl (C4)
GLUE
SGLUE
SQuAD
AVG
Small Mini-SL
61M
3.7
23
-2.021
77.45
62.88
80.39
73.57
50M
3.2
24
-2.056
77.11
63.35
80.12
73.52
Base
223M
11
9
-1,752
82.53
69.80
85.14
79.16
Small 16L
134M
7.2
13
-1,825
82.57
69.51
84.12
78.73
Small
164M
8.6
-1.798
83.22
69.44
85.23
79.30
Small 22L
179M
9.3
10
-1.798
82.52
70.68
85.39
79.54
Small 24I.
193M
10
9
-1.783
83.11
71.11
85.45
79.92
Small 32EL
143M
10
10
-1.897
82.77
70.66
86.01
79.81
Large
738M
34
4
-1.605
85.08
75.97
87.55
82.87
Base 36L
621M
29
3
-1.626
85.26
75.57
87.84
82.89
XI.
2.9B
64
1
-1,487
86.49
77.99
88.70
84.38
Large 361.
I.IB
50
2
-1,564
87.22
79.34
89.21
85.27
XXL.
11.3B
367
1
-1,430
86.91
79.20
89.50
85.20 85.42
XI. 32L
3.8B
169
3
-1.500
86.94
79.87
89.46
FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.
Table describes this phenomena in which we list efficient alternatives to the canonical model sizes using the DeepNarrow strategy. Note that this list is not exhaustive. Firstly, we find that significantly increasing the depth of the small model does substantially better in terms of the compute-performant performance trade-off and may result in pareto-efficient models. The Small 16L model achieves comparable performance to Base while being 40% faster, cost 50% less parameters and has only about 63.1% of total FLOPs. Alternatively, the Small 24L model has 87% of FLOPs of the base model, similar speed (steps/s). and only 16% parameter savings and yet outperforms Base all three downstream tasks. Meanwhile the canonical large model can be outperformed by base model of 36 layers with 16% parameter saving and lower flops cost. The Large 36L model is only 37% of the parameter cost of the XL model and yet outperforms the XL model on all three downstream tasks. Finally, the XI. 32L, model is only a third the size of the XXL. model, approximately consume 44% the number of FLOPs of the XXL model and is about times faster on the same hardware.
Efficient Alternatives TS-Base/L
The Limits of Depth Width We note obvious limitation with our advice. Scaling depth has an obvious limiter, i.c., they are non-paralled pon-paaallelizable across different machines or devices and every computation has to always wait for the previous layer. This is unlike width. which be easily parallelizable over thousands or hundreds of thousands of devices. Within the limitation of scaling to 64 workers with model parallelism of 32, still find that scaling depth Pareto-efficiency of models. From our experiments, from Table 4, we see that the efficient small DeepNarrow models (e.g., Small 16L etc) are still much faster than the base models. Things get tricky as we approach larger models where model parallelism makes it difficult to compare the utility between wide and deep models. To this end, believe the proposed scaling protocol holds within certain hardware limit. Scaling to extreme number of machines via model parallelism (of width) is out of scope of this paper. Another potential drawback to depth-scaling is that this may influence the stability of training these models (due to vanishing gradients). However, we did not observe this in our experiments with T5 models.
still improve the
Relationship of Model Depth with Pareto-frontier Figure shows the performance of scaling small, base and large models by depth. It is clear that the small model (green) dominates the Pareto- frontier initially but slowly tapers off at a certain point. Here, we note that the depth-scaled small model is Pareto-efficient compared to the base model. After the Pareto-efficiency of the small
8

---

Published as a conference paper at ICLR 2022
-1.5
78
-1.6
Large
Base
perplexety
SuperGlue Accuracy
RLA0
Base
B-LG
NL4-GG
-1.
Small Nunss
NLB-SM
Negative
2.0
NLGGN
0NL--L6
Smal
NL5 SM
gl NNL--- dNLL
82
60-
NL4-5M
NL2-LG
56
MM2-5M
2.3
NN.--M
56-
-2.4
56-
FLOPS
FLOPS
(a) Negative Log- Perpeexlyy
(b) SuperClue Score (Avg)
Figure 4: Compute-Performee trade-off when scaling model depth of different starting points
(Small, Base, and Large).
model tapers off, the base model (red line) becomes Pareto-efficient. Similarly, this tapers off and the large model becomes Pareto-efficient.
CONCLUSION
In this paper, we present several important findings pertaining training and practical usage of efficient transformer architectures. Specifically, we discovered that scaling laws differ in upstream and downstream. Contrary to prior work (Kaplan et al., 2020), we emphasize the importance model shape for ensuring strong downstream performance. Next, we also discovered that scaling happens rather differently different compute regions and scaling small model would behave differently from scaling large model. We highlight that this has implications since model development in certain compute region could potentially not transfer to another compute region. We go on to show that not all model knobs are created equal, i.e., some knobs (such as depth) has larger impact on the Pareto-frontier. frontier. Finally, we propose simple but highly effective improved scaling protocol. With this strategy, obtain models with similar downstream finetuning quality while having 50% fewer parameters and/or training 40% faster.
Pareto
REFERENCES
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek. Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08883 2020.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint arXiv:2103. /5691, 2021.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2 102.06701,
2021. 2021.
Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint arXiv:2103.075 2021.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, hbs/1903.04561. 2019. URL http:://///. // aariiii www.vvorree orrg/bbb 903. 04561.

---

Published as a conference paper at ICLR 2022
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.1473: 147994 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: 1911.0216,, 2019.
Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. 2021.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 10X0vv///0.048000 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. Multi-news: large-scale multi-document summarization dataset and abstractive hierarchical model, 2019.
William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the_. arXiv preprint 1801 100111003777 .0736,,777 2018.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03961 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Danny Hemaandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01299) 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 2015.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08330 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL https:://ppeneeeeeeeeeee
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1942, 2019.
James Lee-Thorp, Joshua Ainslie, Ilya Eekstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.0388 2021.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning, pages 5958-5968. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2022
Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding. Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. M6: A chinese multimodal pretrainer. arXiv preprint arXiv:2103.00823, 2021.
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on multitask benchmark. arXiv preprint arXiv:2103. 13009, 2021.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142- 150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
org/ www.ooogggoaooooooooooooo antlologhhoyyy/////oo (atgg////11-- PP1-11111.1....11
Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Kar- ishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, et al. Do transformer modifications transfer across implementations and applications? arXiv preprint 1rXiv:2102.11977 2021.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/ 808.08745, 2018.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pages 7487-7498. PMLR, 2020.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang. Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10:: 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.1068. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: wxXiv:6606.05250 2016.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point: Summarization with pointer-generator networks. CoRR, abs/1704.04368, 2017. URL htttp://axxi...rr//aa 1704. 04368.
Anderw Senior, John Jumper, Demis Hassabis, and Pushmeet Kohli. Alphafold: Using ai for scientific discovery. DeepMind. Recuperado de: https://deepmind. com/blog/alphafold, 2020.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414-10423, 2018.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv: 001199999000.5 2019.
David R So, Chen Liang, and Quoe V Le. The evolved transformer. arXiv preprint arXiv:1901.11111 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data deep learning era. In Proceedings of the IEEE international conference on computer vision, 2017.
11

---

Published as a conference paper at ICLR 2022
Iyy Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. arXiv preprint arXiv: 1409.325,, 2014.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks
In International Conference on Machine Learning, pages 6105-6114, PMLR, 2019.
Yi Tay, Aston Zhang, Luu Anh Tuan, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. Lightweight and efficient neural natural language processing with quaternion networks. xrXiv preprint arXiv: 1900:9996.000 0406,, 2019.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.00743, 2020a.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. arXiv preprint arXiv:2002.112900 2020b.
Yi Tay, Mostafa Dehghami. Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arKiv:2009. 06732,, 2020c.
Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. Are treitrained convolutions better than pre-tree ire-trained transformers? arXiv preprint arXiv:2105.0332 2021a.
Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106. 12672, 2021b.
llya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01660 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355 Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1W118-5444 URL https:///// //t:://wwwww /www. aclweb. org/ hatg//nnhology///// W1 5446.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpon language understanding systems. arXiv preprint arXiv: 0e011.95500277 2019.
Ellery Wulezyn, Nithum Thain, and Lucas Dixon. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pages 1391- 1399. Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN //doi. (tloi.org//0... 1tt1:/...111/3338888
30. doi: 10.1145/003911..05559.. URL https: 3052591.
Ofr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. arXiv preprint arXiv: IxXiv:I910.000 1910.0618... 2019.
Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. arXiv preprint 07X1 2021.
12

---

Published as a conference paper at ICLR 2022
APPENDIX
.1 TRANSFERABILITY OF RESULTS TO VISION TRANSFORMERS (VIT)
Following our language experiments, and as per the advice of (Narang et al., 2021), order to examine our DeepNarrow scaling strategy in another domain and check if the observations extends to the cases where Transformers are applied on other modalities, we pre-retrained several different configurations of Vision Transformer (ViT; Dosovitskiy et al., 2020) and evaluated on downstream few-shot image recognition task. We focused on investigating the pareto-efficient of DeepNarrow small models compared to base models.
We follow the exact same setup as Dosovitskiy et al. (2020) and pre-train VIT on the JFT dataset (Sun al., 2017) with 18k classes and 303M images, for 7 epochs. We evaluate model on ImageNet 10-shot classification. In our experiments, we use the patch size 32 x 32.
Table 5: Results on image recognition task. All models are trained with the same batch size using 64 TPU-V3 chips.
Model VIT-S
#Params
GFLops
Steps/s
ImageNNet-OOShol
62M
1.37
8.83
45.3
ViT-B
102M
4.44
6.74
58.9
ViT-S L224
87M
3.94
6.11
59.7
ViT-S 2-2
99M
4.58
5,36
61.6
Results Table 5 report results on ViT experiments. When considering the number of trainable parameters FLOPs, we observe that DeepNarrow scaling of the ViT-S model achieves better Pareto efficiency compared to the ViT-B model. Notably, when L = 24, the model achieves better few-shot accuracy with 15% less parameters, 11% less FLOPs and achieves +1.4% percentage improvement in accuracy. With respect to the step per seconds (speed), given ViT-S or ViT-SL=2s add to the sequential operations depth, they become bit slower than ViT-B. However, we consider 1s and 6.74s to be reasonably within the same ballpark. In short, the ViT-S,,-24 is still a compelling alternative.
.2 HOW TRANSFERABLE ARE THESE RESULTS ACROSS MULTIPLE DIVERSE NLP TASKS?
To verify that scaling protocols transfer to tasks outside of the 17 tasks explored in (GL.EE/SpprGLLUE and SQuaD), we run additional experiments on a myriad of diverse NLP tasks. Verifying whether the findings generalize outside GLUE and SuperGLUE. is important since we do not want to fall prey to the benchmark lottery problem (Dehghani et al., 2021). As such, the purpose of this additional experiment to verify if our results are universal enough for a general recommendation.
Setup In this experiment, we compare the base T5 transformer with the efficient 24 layer small model using the DeepNarrow strategy, which has 14% less parameters and 10% less FLOPS compared to the T5 base model. This finetuning protocol uses constant learning rate of 10-3 and a batch size of 128 for all tasks. Note that we used the same pretrained models as earlier sections that produced finetuned results on SuperGLUE and GLUE.
Diverse NLP tasks We conduct experiments on total of 12 extra tasks, i.c., 6 tasks of Rain- bow (Lourie et al., 2021) which contains commonsense reasoning tasks, seneration//ummarizattioo tasks (XSum (Narayan et al., 2018), CNN/Dailymail (See et al., 2017) and MultiNews (Fabbri et al., 2019)), along with 3 text classification tasks (civil comments (Borkan al., 2019), wiki toxicity (Wul- czyn et al., 2017) and IMDb reviews (Maas et al., 2011)). )). For the Rainbow suite, we co-tro co-train all tasks in (Lourie al., 2021) and report peak validation results.
13

---

Published as a conference paper at ICLR 2022
Table 6: Rainbow dataset.
Table 7: Generation tasks (Rouge-L).
Task
Base
DeepNarrow
Task
Base
DeepNarrow
ANLI
65.7
65.7
XSum
32.3
33.1
CosmoQA
69.9
70.0
CNN//ailymail
38.9
38,9
HellaSwag
49.7
48.9
MultiNews
20.2
20.5
PQA
73.7
74.4
SQA
65.1
66.0
Wino
65.3
65.9
Results on other tasks Table 6, Table 7 and Ta- ble reports results on these 12 tasks. On all additional diverse NLP tasks considered, we show that the Pareto-efficient alternative outperforms or ties with the base T5 model on 11 out of 12 tasks where 4 of them are ties. It is good to bear in mind that this is model which is effectively smaller in parameters and has 10% less FLOPs.
Table 8: Classification tasks (Acc).
Task
Base
DeepNarrow
CivilComments
88.2
88.4
WikiComments
95.4
95.4
IMDb
94.4
94.4
14

---

Published as a conference paper at ICLR 2022
Cre
O911-3331
12-03
17-331
12-231
Dkt
Daga
Oh 4-221
10
11
Out
Ouaig
1/Thnoughpue
(a) Upstream: Negative Log Perplexity.
Ohal2 1006
01111120000
Aug ass
Superdue
Supendue
Ouars
8-300
O
Aniz
Params
FLOPS
3/T s/Trroughuut
(b) Downstream: SuperGLUE Accuracy
Oy 17-0000
12-1231
Oug 12-222)
De
Dx
0g8-33331
O1/26 Garps
Oru R122
Ca
Vaston
33rt 139m
Siart
Clire
Slerta
205
1/Thenughest
(c) Downstream: GLUE Accuracy
Cep E20
13-41
D2
OuL1D 2006
Dg
91.11-0001
Dg. O2122000
tynooun Oyts 001
Og 1001
Oy6.35
OyLt 1000.
Dle Shpro
Cle
Vigg vi.
Ch632
An124
Cyus VOL
m53
Parame
1/Thenughest
(d) Downstream: Squad Accuracy
Figure Performance on upstream and different downstream tasks with respect to number of parameters, FLOPs, and throughput for models presented in Figure lb.
15

---

Published as a conference paper at ICLR 2022
Nngetiv Log- wwo.-Prrrrrtttt
1367
20
44-10
(a) Upstream: Negative Log Perplexity.
Nig
2
Superciie Accuraty
Super Glue Accuray
25 1193
c0 S
bV
SM
N.3
<ee 7 2100200 L/Throughiut
FLOPS
(b) Downstream: SuperGLUE Accuracy
12iv Accuracy
17007
Dhit
3/Throughhut
(c) Downstream: GLUE Accuracy
Squad Assuuaso
Squed
Squad
1307
Libr
33ed
329922
Ided Kded 1ar 100.00000 100 1/Throughiux
(d) Downstream: Squad Accuracy
Figure 6: Performance on upstream and different downstream tasks with respect to number of parameters, FLOPs, and throughput for small models presented in Figure 2a.
16

---

Published as a conference paper at ICLR 2022
Log Porplookyy
0U2
046
5638
PLOPS
1/Thmmughout
(a) Upstream: Negative Log Perplexity.
1
Sape Dine Recureyy
SuperCluv Accurccy
Gapertoing Rccurcy
A
DM3
Deg
Parama
FLOPS
1/Throughpu:
(b) Downstream: SuperGLUE Accuracy
CUPPt
DURME DMY
Glie RODUNN
Detgge
%
DM106
Glve Arcuresyy
de0
rugg)
24g.
Star7
6707
Stace
Levi
Maric
K0d2 LoetLde 101330 1/Thoughpu
(c) Downstream: GLUE Accuracy
Equaed Recurity
Squaed Repurnty
H-A
Orqued Repurety
012
Parami
FLOPS
4/Throughhux:
(d) Downstream: Squad Accuracy
Figure 7: Performance on upstream and different downstream tasks with respect to number of parameters, FLOPSs, and throughput for base models presented in Figure 2b.
17

---

Published as a conference paper at ICLR 2022
DL240
L2g
2424
010
Perwax
FLOPS
1/Thoughpu:
(a) Upstream: Negative Log Perplexity.
Suparcou
M 8
Bogg
04 4
37
Lind
3305
18:1 FLOPS
65
1/Thooughut
(b) Downstream: SuperGLUE Accuracy
Ohe Accumary
eg
04/2810
22ye
naga
0
1364
Params
PLOPS
I/Troughhut
(c) Downstream: GLUE Accuracy
Squad Aecuaaty
Recurety
Accuroty
Sqqaed
Params
FLOPS
A/Thioughout
(d) Downstream: Squad Accuracy
Figure 8: Performance on upstream and different downstream tasks with respect to number of parameters, FLOPs, and throughput for large models presented in Figure 2c.
18


Reviews:
[{"review_id": "w_ZmllU_4N", "reviewer": "Reviewer_C5AB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* This paper conducts solid experiments across many NLP benchmarks to demonstrate the discrependcy between upstream perplexity and downstream performance. \n* This paper also shows scaling strategies differ at different compute regions, i.e., applying the same strategies at different compute regions (small vs large) has a different effect on model quality.\n*  This paper presents a DeepNarrow strategy that is applicable to all model sizes and can reduce model parameters and improve speed while preserving downstream performance.\n\nWeakness\n* The discrepancy between upstream perplexity and downstream performance is somewhat related to model selection with dev set in natural language generation task. Model selection with dev set according to perplexity may underperform model selection strategy according to downstream metric, e.g. BLEU score. Therefore, the discrepancy found in this paper is not \"surprising\"!\n* DeepNarrow strategy seems also to be discussed before in other neural architectures. In Figure 6.7 of [1], discussed Conv nets with the same parameters but different layers and found that deeper models perform better. Therefore, the novelty of this strategy is not significant.\n\n[1] Goodfellow et al. Deep Learning. 2016 ", "summary_of_the_paper": "This paper presents solid experiments across many NLP benchmarks to show that the perplexity of the upstream language model can be a deceiving indicator of downstream quality. They also show that popular language model like T5-base and T5-large are relatively inefficient and scaling strategies differ at different compute regions, i.e., applying the same strategies at different compute regions has a different effect on model quality. Finally, they give a simple DeepNarrow strategy that can be applied to different model sizes and make them efficient while preserving their performance. \n\n", "summary_of_the_review": "This paper conducts solid experiments to support its claims, e.g. discrepancy between upstream perplexity and downstream performance and scaling strategies difference at different compute regions and DeepNarrow strategy to improve model speed and reduce parameters. Some of the findings provided in this paper is kind of intuitive and the DeepNarrow strategy has been shown in previous literature, which hurt its novelty. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tuDz2GdCk5", "reviewer": "Reviewer_SEDW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The paper is clearly written and the figures and tables are illustrative.\n* The empirical efforts on extensive experiments deserve to be appreciated. The model checkpoints and codes will be publicly released to the community.\n\n**Weaknesses**\n\n* The novelty of this paper is limited. \n  * Regarding the scaling laws: the motivation of this paper is that [1] only consider the pre-training performance (loss/perplexity on upstream datasets) and does not investigate the downstream performance. The authors try to fix the bias of the previously established scaling law by investigating the downstream performance. Incremental experiments investigating the downstream performance are provided. Besides, the authors do not provide further quantitative analysis like that in [1] under this new setting.\n  * Regarding the proposed DeepNarrow strategy: based on the observation of the experiments, the authors propose to preferentially increase the model's depth when scaling the models. However, this finding has been discussed in [2] and the authors of [2] provide both theoretical and empirical analysis on the depth-width tradeoff of Transformer models. The authors should include this work and discuss it.\n\n[1] Kaplan et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).\n\n[2] Levine et al. \"The Depth-to-Width Interplay in Self-Attention.\" NeurIPS 2020.\n", "summary_of_the_paper": "This paper aims at providing insights from scaling Transformers for pre-training and finetuning. Based on extensive experiments involving pre-training and fine-tuning over different transformer configurations, the authors find that the model shape matters a lot besides model size when considering the downstream performance, and scaling strategies differ at different compute regions. With these new findings, the authors propose the DeepNarrow scaling strategy and verify it on additional experiments including tasks in different domains (language and vision).", "summary_of_the_review": "Considering the novelty of this paper, I vote for 'weak rejection'. If the authors can addressed my concerns, I would like to increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cfUQKi-0Ech", "reviewer": "Reviewer_8qst", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper is the first one to discover the scaling insights on both pretraining and fine-tuning Transformers. The authors perform comprehensive experiments and provide insights from different perspectives. Even though some insights are not surprising, this paper provides evidence to support some previous intuitions.\n\nWeaknesses: Since this paper focuses on empirical conclusions, the novelty is somewhat weak. Furthermore, the experiment results may be effected by the randomness and I don't find the error bar in the paper, which is the most concern of mine. Also, I have the following questions about the paper.\n\n1. In the caption of Figure 1, I don't see variable P (default model parallelism) in the table. Can you provide some explanation about it? \n2. As discussed, do you perform repeated experiments to avoid the influence of randomness? I guess the main conclusions in your paper like Figure 1 should be performed repeatedly.\n3. T5 is a sequen-to-sequence model. I am curious about your conclusions for different models like ViT or BERT. Can you perform some experiments on different models to verify the generalization of your conclusions?", "summary_of_the_paper": "This paper presents scaling insights from pretraining and fine-tuning Transformers empirically. The main findings are as follows: (i) model shape matters for downstream tasks; (ii) scaling protocols operate differently at different compute regions; (iii) T5-base and T5-large are not Pareto-efficient. They will publicly release over 100 pretrained checkpoints for further research.", "summary_of_the_review": "The paper study the empirical findings of scaling insights for Transformers on both pertaining and fine-tuning and provide a list of conclusions. Even though the conclusions are not surprising to me, I think this paper provides some evidence to prior works and future works. One main concern of mine is the randomness of the paper since this paper performs comprehensive experiments and concludes insights based on the results.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TO5iE07aP3e", "reviewer": "Reviewer_826H", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The findings in the paper are useful for researchers in this domain from two folds: Firstly, by open-sourcing the checkpoints, others can take advantage of the trained models to foster research in various NLP applications. Secondly, the heuristics provided can potentially guide the design of better architectures for downstream tasks.\n\nWeaknesses: \n- The plots are often hard to read and have very bad visualization. As an example, the labels in Figure2-c are unintelligible. As such, it is hard to drive conclusions from the plots, rather the reader has to refer only to the text, which is not adequate. It would be great if the authors improve the plots and ensure that all text/labels can be easily read.", "summary_of_the_paper": "This paper provides various ablation studies to show the effect of scaling various architecture parameters on the performance of a downstream task. The studies are performed on encoder-decoder transformer architectures that follow the T5 architecture as the backend. As a result of the ablation studies, the authors provide a heuristic on how to design architectures that lie on a better Pareto front compared to the previously proposed T5 architectures.", "summary_of_the_review": "Please see above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The results reported in this paper and the model checkpoints released are of interest and broad utility to the community in the opinion of the NLP.  While one reviewer was somewhat negative, most reviewers were in favor of acceptance of this paper, which expands the results from [1] to downstream tasks. The AC therefore recommends acceptance.
Decision: Accept (Poster)