Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Authors: Yi Tay
Mostafa Dehghani
Samira Abnar
Hyung Won Chung
William Fedus
Jinfeng Rao
Sharan Narang
Vinh Q. Tran
Dani Yogatama
Donald Metzler
Venue: EMNLP 2023 Findings (2023)
Keywords: language models
architectures
scaling laws
TL;DR: How do different architectures scale?

Abstract: There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour?  How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.

Scaling Laws vs Model Architectures: How Does Inductive Bias Influence Scaling?
Yi Tay
Mostafa Dehghani
Samira Abnar
Hyung Won Chung'
William Fedus!
Jinfeng Rao
Sharan Narang Vinh Q. Tran
Dani Yogatama
Donald Metzler
Google
dehghani@google..com
Abstract
interaction between architecture and scaling laws crucial as designing models that perform well at diverse scales will likely have significant impact.
There have been a lot of interest in the scal- ing properties of Transformer models (Kaplan et al., 2020). However, not much has been done on the front of investigating the effect of scaling properties of different inductive bi- ases and model architectures. Do model ar- chitectures scale differently? If so, how does inductive bias affect scaling behaviour? How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transform- ers, Switch Transformers, Universal Transform- ers, Dynamic convolutions, Performers, and re- cently proposed -Mxeess Via extensive experiments, we show that (1) architecture is an indeed an important consideration when per- forming scaling and (2) the best performing model can fluctuate at different scales. We be- lieve that the findings outlined in this work has significant implications to how model architec- tures currently evaluated in the community.
This paper is attempt to understand the ef- feet of inductive bias (architecture) on scaling laws of language models. To this end, we pre-train and finetune over ten diverse model architectures across multiple compute region and scales (e.g., from 15M to 40 Billion parameters). In total, we pre-train and finetune over 100 different models of different ar- chitectures and sizes and present insights and chal- lenges at scaling these ten diverse architectures.
We consider broad spectrum of models in our extensive experiments. Concretely, we con- sider several well-established Transformer vari- ants (Vaswani et al., 2017) such as Evolved Trans- former (So al., 2019), Universal Transformers (Dehghani et al., 2018) and Switch Transformers (Fedus et al., 2021). We also consider lightweight models such as ALBERT (Lan et al., 2019) and/or efficient Transformers (Tay et al., 2020) such as Performer (Choromanski et al., 2020) and Funnel Transformers (Dai al., 2020). In our comparison, also interested in finding out if general im- provements the Transformer architectures such as Mixture-of-Softme (Yang et al., 2017) and/or Gated Linear Units (Dauphin et al., 2017: Shazeer, 2020) influence the scaling behaviour of models. Finally, we also evaluate models outside the fam- ily of Transformers including Lightweight convo- lutions (Wu et al., 2019). Dynamic convolutions (Wu et al., 2019) and the recently proposed MLP- Mixers (Tolstikhin al., 2021). Figure illustrates an overview about the experiments we run.
1 Introduction
There have been lot recent interest in the scaling properties of Transformer models (Kaplan al., 2020; Hernandez et al., 2021; Bahri et al., 2021; Henighan et al., 2020; Tay et al., 2021b; Abnar et al., 2021). However, not much is understood about the scaling properties of different inductive biases imposed by model architectures. Improve- ments at specific scale (compute, size etc) are often assumed to transfer to different scales and compute regions (So al., 2019; Choromanski al., 2020; Lan et al., 2019; Dehghani al., 2018) and new research often presented a point-wise fashion with respect to scale. In short, is not un- common for new methods to be presented with data points at very specific or limited compute regions (e.g., base size). We believe that understanding the
We also note that scaling these models is not as straightforward as it seems, i.c., there are intricate details of scale that are intertwined with architec- tural choices which we study in detail in this pa- per. For example, a distinct feature of Universal Transformers (and ALBERT) is parameter sharing. Hence, compared with standard Transformers, this
"ii and Mostafa contributed equally.
Work completed while at Google.

---

-1.4
1.5-
1.5
watdc XLS
Negative Log hig-Perplexity
SuperGlue Accuracy
O
O
2.0
70-
OGNA
2.2
68-
2.4
60
Q
mm O Swethia
O
35-
2.8
-3.
1,2111226112
4.6612
88
1.8e-13 3.5 1840114
1.les123ee12 4.4e+12
5.8e-112
1.tee13
3.5e-13 7.0=-144-144
FLOPS
FLOPS
(a) Upstream: Negative Log-Perplexity
(b) Downstream: Accuracy
Figure I: An overview compute-performance (FLOPs performance) plot of all the diverse models and architectures we pretrained and finetuned in this study. Colors represent different model architectures and size of the circles represent the size of the model (parameters).
architectural choice significantly warps the scaling behaviour not only with respect to performance but also amongst compute metrics such FLOPs. speed and number of parameters (Dehghani et al.. a0) Conversely, models such Switch Trans- formers are on the other end of the spectrum with an uncommon relationship between FLOPs and number of parameters, i.c., they have high parame- ter to FLOPs ratio. This difficulty makes navigating this landscape challenging.
scalability with pointwise comparisons at a cer- tain compute-region.
We find that when it comes to scaling different model architectures, upstream pre-training per- plexity might not correlate well with downstream transfer. Hence, the underlying architecture and inductive bias is also crucial for downstream transfer.
We highlight the difficulties of scaling with cer- tain architectures and show that some models do not scale (or scale with negative trend). We also find concerning trends where linear-time at- tention models such as Performer struggle with scaling up.
Our Contributions and Insights The key con- tributions of this paper are as follows:
For the first time, derive scaling laws for dif- ferent inductive biases and model architectures. We find that this scaling coefficient differs greatly from model to model. We believe this is an impor- tant consideration in model development. It turns out that amongst all ten architectures that we con- sider, the vanilla Transformer has the best scal- ing behaviour, even if its absolute performance at each compute region is not the greatest.
Related Work
Kaplan et al. (2020) studied empirical scaling laws of the decoder-only Transformer language mod- els. They focused on the standard left-to-right lan- guage modeling objective with the cross-entropy loss as the performance metric. One of the main findings is that the loss scales as power-law with three major characteristics of the model training: model size, dataset size and the training compute. Another somewhat surprising finding is that the model shapes such as width or depth of the Trans- former network have minimal effects on the cross- entropy loss for a wide range of scales. Subsequent works (Henighan et al., 2020; Hernandez et al.,
We observe that models that operate well in one compute-scale region is not necessarily the best in another compute-region. Moreover, find that certain models have difficulty scaling de- spite performing decently (comparably) lower- compute regions. This has implications, since difficult to get the fulll picture of a model's

---

2021) made similar conclusions for autoregressive generative modeling and for transfer learning, re- spectively. This finding also generally supported by (Tay et al., 2021b) but discrepancies were found for the gap between pretraining and finetuning highlighting the fact that observing downstream performance of large language model is indeed im- portant. In (Tay et al., 2021b), the effect of depth unusually pronounced for downstream perfor- mance.
ily support causal masking and autoregressive de- coding. Due this, we consider encoder-decoder configurations trained with span corruption, and evaluate on downstream finetuned tasks. This cre- ates a more level playing field for architectures that do not support in-contex: learning. As such we follow (Raffel et al., 2019) to fix the number of pretraining tokens (i.e. sequence length, training steps, batch size) seen by each model. Given the large space of model architectures and scales we aim study, this also fixes the data size dimension, making our empirical study more tractable. Since we finetune models until convergence, we antici- pate the effect of pretraining token amount to be less pronounced than studied in (Hoffmann et al., 2022).
Raffel et al. (2019) studied the effect of pre- training objectives, model structures (e.g.. encoder- decoder, decoder-only), pre-training dataset size and training strategy on the transfer learning. They showed that the downstream performance mono- tonically increases with the model scale (from 60M 11B parameters). While they studied several model structures, the Transformer implementation is mostly the same as the original Transformer by Vaswani et al. (2017). Conneau et al. (2020); Goyal et al. (2021) scaled-up multilingual encoder- only architectures up to 1IB parameters while maintaining the original Transformer implemen- tation. They found that scaling the model improves its cross- gross-lingual ability. Fedus al. (2021) scaled a sparse model based on Mixture of Experts (MoF. models up to trillion parameters.
3 Methods
This section outlines our experimental setup.
3.1 Models
This section describes the models we evaluate in our experiments. Our models are largely im- plemented in a sequence sequence framework (Sutskever et al., 2014) following the convention of T5 (Raffel al., 2019). Encoder-decoder models are natural choice for this experimentation be- cause they can universally express both encoding and decoding tasks.
While previous studics have repeatedly shown the benefits of scale for language understanding tasks for both dense and sparse Transformers and cross-lingual abilities, all of these used the same Transformer implementation within each studies. With a plethora of improved Transformer architec- tures proposed in the literature, it is timely to inves- tigate which of these improved architecture has the best scaling properties. The main goal of this pa- per is to systematically study how inductive biases imposed by these Transformer variants affect the scaling behavior in shared software and hardware settings. This is in similar spirit to (Narang al., 2021) that studies the impact of architectures on performance. Our analysis extends that of (Narang et al., 2021) to the model scale axis.
Transformer Variants We consider several stan- dard Transformer variants.
Transformers (Vaswani et al., 2017) The basic vanilla Transformer architecture. Our basic setup considers the T5-style of Transformers (Raffel et al., 2019), which largely follows the vanilla Transformer except that uses relative attention instead of sinusoidal position embeddings and pre-layer normalization, i.e. layer normalization is applied before each sublayer.
Evolved Transformers (So et al., 2019) A transformer architecture learned via AutoML. The architecture comprises of convolutions and attention. We scale Evolved Transformers fol- lowing the same pattern as vanilla Transformers.
We note that increasingly the number of tokens seen during pretraining has been incorporated in the study of scaling laws (Hoffmann et al., 2022;
Muennighoff al., 2023), Hoffmann et al. (2022) trains decoder-only Transformer language models with casual langauge modeling and evaluates on zero and shot tasks. In this work we consider architectures modifications that do not necessar-
Universal Transformers (UT) (Dehghani al., 2018) - A Transformer architecture with shared parameters and recurrent-like computation for transform layers. Scaling UTs are challenging
because of parameter sharing. While we are able

---

to also increase drr domodet the increase in parameters is of magnitude Nuyers than stan- dard Transformers. Another axis of exploration is to scale r the number of repeated computation at each UT layer this increases computation (number of FLOPs) but does not increase the parameter size of the model.
Ton-Transformer Architectures We are inter- ested the scaling behaviour of non- Transformer based architectures such as convolutions and/or mixer architectures.
Lightweight Convolutions (Wu et al.. 2019) Lightweight depthwise convolutions that have shown promise over Transformer architectures.
Switch Transformer (Fedus et al., 2021)
Dynamic Convolutions (Wu et al., 2019) An extension of the Lightweight Convolution to cre- ate time-dependent kernels.
sparsely activated mixture-of-expert architec- ture. The Sparse Transformer is another model with an unusual relationship between number of parameters and compute. When we scale this model uniformly, the number of parameters eas- ily reaches the ballpark of 40B.
MLP-Mixers (Tolstikhin al., 2021) Mixers are recently proposed architectures that learn a lightweight mixing of tokens. Since Mixers have not been used in autoregressive decoding, we only use token-mixen on the input encoder.
Efficient Transformer Variants These class of models are mainly concerned reducing computa- tional costs, memory usage, or parameter count of models.
4 Experiment Setup
Performer (Choromanski et al., 2020) A linear time attention model using generalizable kernel attention. For simplicity, we adopt the relu kernel variant for our experiments. We scale Performer in the similar fashion (i.e., uniform scaling) as vanilla Transformers.
Our setup, along with all models, are implemented in Mesh TensorFlow (Shazeer et al., 2018), li- brary with similar interface to TensorFlow but en- ables distributed model parallelism across multiple workers. For fair comparison, all models are pre- trained for 2 steps on the english C4 corpus op- timized using an inverse square root learning rate with Adafactor (Shazeer and Stern, 2018), All mod- els use the same SentencePiece tokenizer (Kudo and Richardson, 2018) containing 32K subwords. This closely follows the setup in the T5 paper (Raf- fel et al., 2019). Finetuning is performed for 100K steps mixture of GLUE (Wang al., 2018), SuperGLUE (Wang et al., 2019) and SQuAD (Ra- jpurkar et al., 2016). We evaluate on both upstream (pre-training) validation perplexity as well as down- stream transfer for NLU tasks (GLUE + Super- GLUE SQuAD) after fine-tuning. We pretrain and finetune our models with 16 TPU-v3 chips with data parallelism. All large models have model parallelism of 2 and XL models have model par- allelism of 8.
Funnel Transformer (FT) (Dai al., 2020) A Transformer architecture that downsamples the input sequence across the layer stack. Our imple- mentation uses FT only in the encoder and reverts to vanilla Transformer in the decoder following Narang et al. (2021).
ALBERT (Lan et al., 2019) A lightweight trans- former architecture that shares parameters across all layers and factorizes the embedding and out- put softmax layers. For our seq2seq ALBERT, we also share the weights of encoder and decoder.
General Improvements We consider general im- provements that are not necessarily tied to Trans- formers. We select candidates that have shown to do well in Narang et al. (2021).
We consider several different model sizes for each architecture. For models that are straightforward to scale, we simply follow the stan- dard convention in Raffel al. (2019), moving from small base, large and XL. We include tiny version of each model to observe how differ- ent models behave lower compute regions. For models where was not straightforward to scale (e.g., Universal Transformers, ALBERT), we tried
Model Sizes
Mixture of Softmaxes (Yang et al., 2017) A transformer architecture adopting the MoS method at the Softmax layer.
Gated Linear Units with GeLU (GLU- Transformer) Replacing position-wise feed- forward-networks Transformers with Gated Linear Units (Dauphin et al., 2017).

---

pots Lags
8.00%
RCW
-012
1010
(a) ALBERT
(b) DCony
(c) Evalved
(d) Funnel
Sape
Panc
Lings
(e) Transformer-GLU
LCony
(g) MLP Mixer
(h) MoS Transformer
Sasmorr Adomt
400%
19070 (armee
(i) Performer
(i Switch Transformer
(k) Universal Transformer
Figure 2: Upstream Negative Log-Perplexio of vanilla Transformer compared to other models.
scale them in similar fashion but faced obvi- ous limitations such as getting ALBERT to have the same number of parameters as T5 XL without incurring huge number of cost terms of FLOPs. For convolutional models, we consider dmodel to be the hidden size (i.e., channel depth) for the one- dimensional convolution layers. Values such as duv, NH then become redundant. Details on scal- ing details' of each architecture can be found in the supplementary material.
gates of GLUE, SuperGLUE and SQuAD. While we use the same Mesh TensorFlow- codebase used by Raffel et al. (2019) and hence expect our experimental results to match theirs, we verify that our T5 base does achieve similar results to what is reported in Raffel et al. (2019).
5.1 Do all models scale the same way?
We compare on both upstream perplexity and down- stream finetuning performance here.
5 Main Results
Upstream Perplexity Figure 2 reports the scal- ing behaviour all models as we increase the number of FLOPs. We observe that the scaling be- haviour of all models are quite unique and distinct, i.e., most of them are quite different from standard Transformers. Perhaps the biggest finding here is that most models (e.g., LConv, Evolved) all seem be on-par or better than standard Transformers but fail to scale with higher compute budget. Another interesting trend is that "lincar" Transformers such as Performer fail to scale as shown in Figure 2i.
We report the main results of this paper in Table
1. We report the number of trainable parameters, FLOPs (of single forward pass) and speed (steps per second). We also report on validation perplex- ity (on upstream pre-training) and results 17 downstream tasks. The results reported aggre-
'The largest Switch transformer was scaled in pretty sub-optimal way. So we don't 't think is representative of the full potential of the Switch family. Take the last data point of Switch with pinch of salt.

---

Ooor feit
TLOFS
The
1800 Teckeen
(a) ALBERT
(b) DConv
(c) Evolved
(d) Funnel
Parm
10000
HDD
(c) Transformer-GLU
(f) LCony
(g) MLP Mixer
(h) MoS Transformer
Deea
Lange
7L000
Lmnoan2
14:40
(i) Performer
Switch Transformer
(k) Universal Transformer
Figure 3: Downstream accuracy of vanilla Transformer compared to other models.
The pre-training perplexity metric only decreases by 2.7% going from base to large scale compared to 8.4% of the vamilla Transformer.
5.2 Are the best models at each scale different?
Figure shows the Pareto-frontier when plotting compute against upstream and downstream perfor- mance. Since the colors of the plot represent dif- ferent models, we can observe that the best model for every scale and compute region might be differ- ent. Moreover, from Figure 3, we can also observe this. For example, the Evolved Transformer seems to do well against the standard Transformer at tiny small region (downstream) but this quickly changes when scaling the model up. We also ob- serve this with MoS-Transformer where it clearly outperforms vanilla Transformers at some regions but not at others.
Downstream Transfer Figure reports the seal- ing curves of all models on downstream transfer. The overall finding that most models have distinct scaling curves compared to Transformers is also evident in downstream tasks. It is also noteworthy that most models have different upstream and downstream scaling curve. We find that some mod- els such as Funnel Transformer and LConvs that seem to hold out pretty well on upstream but suffer substantially on downstream. As for Performer, the performance (disparity) seems to be even greater in downstream as compared to upstream. Notably, the SuperGL.UF downstream tasks generally re- quire pscudo tross-attention the encoder, which models such convolutions are equipped to handle (Tay al., 2021a). To this end, we find that certain models may have difficulty learning the downstream tasks despite good upstream per- formance.
5.3 Scaling Law for Each Model
Table 2 presents the slope of the fitted linear line for each model across multiple scenarios. We derive by plotting (FLOPs), U (upstream per- plexity), D (downstream accuracy). P (number parameters). In general, most values of a depict

---

Table 1: Results on pre-training and finetuning ten different model architectures. Full results (further varying
hyperparameters of these models) can be found in the Appendix.
Model
#Params
FLOPs
Speed
Neg Log Ppl
GLUE
SGLUE
SQuAD
Transformer Tiny
16M
1.21
38.4
-2.47
69.3
56.9
73.6
Transformer Small
60M
3.70
22.7
-2.02
78.1
65.3
81.9
Transformer Base
223M
11.4
9.3
-1.75
83.8
74.0
86.3
Transformer Large
738M
34.3
3.6
-1.61
86.4
78.3
88.6
Transformer XL
2.9B
63.8
1.3
-1.49
87.8
81.5
89.5
Evolved Transformer Tiny
19M
1.31
39.7
-2.45
69.6
57.1
69.6
Evolved Transformer Small
79M
4.23
23.7
-2.44
75.7
66.2
80.2
Evolved Transformer Base
218M
10.2
8.9
-1.79
83.0
70.5
84.8
Evolved Transformer Large
L.BB
49.3
2.1
-1.62
86.2
77.1
88.0
Evolved Transformer XL
2.2B
71.3
0.8
-1.55
87.0
78.3
88.2
Universal Transformer Tiny
11M
1.77
38.1
-273
69.8
56.1
62.3
Universal Transformer Small
52M
7.30
18.3
-2112
76.8
64.2
75.4
Universal Transformer Base
127M
20.3
8.4
-1.1
80.0
67.9
80.1
Universal Transformer Large
283M
27.6
1.6
-1.67
84.0
73.4
85.4
Switch Transformer Tiny
174M
3.25
29.7
2201
78.2
63.8
80.7
Switch Transformer Small
460M
4.63
22.3
-185
80.3
68.0
82.9
Switch Transformer Base
2.0B
12.7
8.4
-1.66
84.2
74.1
86.5
Switch Transformer Large
3.9B
-156
84.6
75.8
87.9
Switch Transformer XL
29.6B
43.3
0.8
-162
84.0
75.2
87.5
Performer Tiny
16M
1.14
42.0
-2.88
50.5
48.8
15.0
Performer Small
61M
3.50
39.0
-2.44
57.8
51.1
31.1
Performer Base
224M
10.8
11.7
-2.33
61.4
53.4
37.8
Performer Large
739M
32.8
-2.16
62.4
52.4
30.8
Funnel Transformer Tiny
16M
1.10
39.9
-2.58
63.4
49.4
54.6
Funnel Transformer Small
61M
2.96
-2111
70.0
58.5
75.1
Funnel Transformer Base
223M
8.10
11.9
-1.83
76.3
62.9
81.6
Funnel Transformer Large
739M
22.6
5.0
-1.99
79.8
67.1
83.8
Funnel Transformer XL
2.9B
40.3
1.89
-1.61
79.8
68.0
83.7
ALBERT Small
15M
3.57
42.0
-2.66
73.7
62.0
77.1
ALBERT Base
21M
16.4
-2.88
69.0
57.2
64.3
ALBERT Large
34M
31.6
5.1
-2.00
62.9
54.1
27.3
Too--Trasformer Tiny
27M
1.29
39.7
-2.37
70.6
57.9
74.1
MoS-TransTormer Small
8IM
3.70
26.3
-1.98
79.7
67.1
83.1
ToS-Transformer Base
257M
11.4
8.6
-1.70
84.5
73.9
86.8
MoS-Transformer Large
800M
35.0
-1.56
86.5
79.7
89.1
NoS-Transformer XL
2.9B
112
1.2
-1.45
88.2
81.4
90.0
GLU-Transforner Tiny
26M
1.29
31.7
-2.35
70.5
57.0
74.2
GLU-Transformer Small
77M
3.70
26.4
-1.97
79.1
67.4
83.0
GLU-Transformer Base
248M
11.4
8.6
-1.71
84.6
74.5
87.2
GLU-Transformer Large
748M
35.0
3.4
-1.56
84.2
74.3
86.2
GLU--Trasffrmer XI.
2,85B
61.3
1.0
-1.49
87.6
82.9
89.4
L.Conv Tiny
I7M
1.20
-2.50
51.1
51.3
49.5
LCony Small
67M
3.80
12.8
2.10
71.8
59.9
64.7
LCCnv Base
210M
10.6
12.8
-1.95
73.8
63.6
70.3
LConv Large
741M
41.0
3.0
-1.76
76.8
65.6
76.3
LConv XL
2.3B
77.0
1.0
-1.75
73.3
64.1
72.9
DConv Tiny
22M
1.39
27.3
-2.66
51.1
48.9
30.2
DConv Small
96M
4.97
19.8
-2.88
68.6
57.4
64.3
DConv Base
324M
15.3
7.6
-1.90
72.9
60.1
63.7
DConv Large
1.2B
78.0
1.1
-1.82
70.8
58.5
58.2
ML.--Mxx Small
67M
3.83
22.3
2.15
65.4
55.1
58.7
MLP-Mixer Base
233M
12.4
10.7
-1.90
64.4
58.1
60.5
MI.--Mixe Large
739M
38.3
3.9
-1.73
52.2
47.8
60.9
MLP-Mixer XL
2.86B
48.3
1.2
-1.61
57.3
58.9
65.7

---

Table 2: Slope of a fitted linear line for each model, when compare FLOPs upstream performance (F, U). FLOPs vs. downstream performance (F, D), parameter size vs. upstream performance U), pa- rameter size downstream performance (P, D), and finally upstream vs. downstream performance (U, 1?).
Model
DF
C.P.U
aPPDD
QU,D
Transformer
0.54
0.28
0.47
0.24
0.49
GLU-Trans.
0.49
0.24
0.42
0.22
0.46
L.Covv
0.32
0.13
0.29
0.11
0.48
(a) Upstream Neg, Log-PPI..
(b) Downstream Accuracy.
Funnel
0.47
0,22
0.38
0.18
Figure 4: Scaling depth
Switch
0.23
0.14
0.13
0.08
0.58
Universal
0,50
0.20
0.22
ALBERT
0.08
-0.12
0.13
-0.21
-1.67
Pubeel Prpc
Evolved
0,22
0.42
0.21
Performer
0.25
0,05
0.24
0.05
PTC Spe
MoS-Trans,
0.43
0.21
0.43
0.20
MLP-Mixer
0.32
-0.03
0.26
0.65
-0.02
how well model scales. For example OFF is plot- ting FLOPs against Upstream performance. The only exception is agD which is a measure of up- stream downstream performance. A high ou, value means that the transfer the downstream tasks is better as a model scales. Overall, the a value is a metric that represents how well model performs relatively across all scales
(a) Upstream Neg, Log-PPL.
(b) Downstream Accuracy.
Figure Scaling width of FFN
scaling property of the model.
Analysis of Slope for each Model In general, we find that the vanilla Transformer has the highest values of a. Models such as Evolved Transformer, GLU-Transformer MoS- Transformer and Funnel Transformer tend have similar scaling properties to the vanilla Transformer. The GLU-Transformer has similar and slightly worse scaling properties to the vanilla Transformer, even if it was observed to do better in absolute sense on some compute- regions. On the other hand, also observe that there are models which difficult scale such LCony, UT, MLP-Mixer and Performer. This is even more evident on downstream task. We also note that ALBERT scales (trends) negatively" (gets worse) as we scale the model up. On the other hand, the metric oUDD measures how the downstream per- formance scales with upstream performance. Over- all. the Switch Transformer does the best on this metric where downstream performance scales well with upstream performance. Generally, models that make less changes to the main Transformer ar- chitecture -Trn-ssoooorrerr, MoS-Tron Transformer) tend to retain similar scaling behaviours and chang- ing the inductive bias also significantly alters the
5.4 Do Scaling Protocols influence model architectures in the same way?
We are interested in how different scaling protocols influence the model architectures. Figure 4 shows the effect of scaling depth of four model archi- tectures MoS--rransfommer, Transformer, Evolved Transformer and LConv). Figure shows the ef- feet of scaling width on the same four architec- tures. Firstly, on upstream (negative log perplex- ity) curves, we note that while different architec- tures have a distinct difference in absolute perfor- mance, the scaling trend remains quite similar. On downstream, depth scaling (Figure 4) seems to act equally most architectures with the exception of LCony. Meanwhile, for width scaling, it seems that Evolved Transformers scale slightly better when applying scall-s is also interesting to note that depth has a much more substan- tial impact on downstream scaling as opposed to width-scaling.
Epilogue and Conclusion
In this paper, conducted extensive experiments, pretraining and finetuning of up to 100 models
"This version of ALBERT shares parameters across en- coder and decoder which may partially explain why we had hard time scaling up.
ranging from well-established Transformer and
non- Transformer architectures. We showed that

---

different model architectures can have different scaling behaviours and models performing well in one compute region (or model size) may not do identically well in another compute region. We also showed that model architectures may do well on upstream perplexity but fail to transfer to down- stream tasks. Hence, practitioners should be cau- tious about developing architectures that not only scale well with respect the upstream perplexity, but also based on downstream performance. While we certainly do not expect researchers to always report model performance across all scales (espe- cially large-scale), we believe that it is good to keep in mind that architectures can perform quite differently at different compute regions. Hence, this might be good dimension to consider when designing new inductive biases. As such, perform- ing evaluation at a certain compute region may be insufficient to capture the full picture. is also good to consider if different inductive biases will result in different extents of emergent capabilities (Wei et al., 2022a; Abnar et al., 2020).
huge number of models considered in this work. while we scaled each model the best of our abil- ity and present details on how they were scaled, there could always be unexplored hyperparameter settings and other tricks that could get model to "work" at larger scales. Beyond this work, one could also study the differences in prompting tech- niques, e.g. chain- thain-of-thoughu prompting (Wei et al., 2022b), between different architecture and scales. Such findings would be of importance for the re- search community the future. Although in either case here, we believe that our findings, i.c. models scale differently and need to be tested, will con- tinue to be relevant. This space will only continue to grow, and future researchers and practitioners must continue to assess the scalability of new mod- els under new use cases.
References
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. 2021. Exploring the limits of large scale pre-training. arXiv preprint arXiv: 10.02095.
We also showed that different model architec- tures may react differently to different scaling pro- tocols, reaffirming that comparing and benchmark- ing these models can be very challenging (De- hghami et al., 2021b). When comes to scaling large models, we show that novel inductive biases can be indeed quite risky which might explain why most state-of-the-art large language models (Rae et al., 2021: Chowdhery et al., 2022; Tay et al., 2022) are based on relatively vanilla architectures. Our advice is to be cautious when staking an expen- sive run on an architecture that drastically modifies the attention mechanism. Finally, we acknowledge that not every practitioner or researcher would re- quire models that are able to scale to billions of parameters. In that case, inductive biases that are tailored to small or low compute will be sufficient.
Samira Abnar, Mostafa Dehghani, and Willem Zuidema. 2020. Transferring inductive biases through knowl- edge distillation. arXiv preprint arXiv:2006.00555
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explaining neural scaling laws. arXiv preprint arXiv:2102.0670..
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sar- los, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, al. 2020. Rethinking attention with performers. arxiv preprint arXiv:2009. 147994.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton. Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 2rXiv:204..02111.
7 Limitations
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross cross-lingus lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
As with all empirical studies, ours come with its own set of limitations. We only present a sampling of Transformer variants, and it is not exhaustive. Our selection is aimed towards sampling a diversity of architecture approaches to have representation across the entire space of Transformer architec- tures. As such, do not claim that our findings hold within a subcategory; for example efficient Transformer variants, which there are many recent works not covered here. Additionally, given the
Zihang Dai, Guokun Lai, Yiming Yang, and Quoe V Le. 2020. transforr: Filtering out sequential redundancy for efficient language processing. arXiv preprini 2rX00:2006.03236.

---

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated con- volutional networks. In Proceedings of the 34th In- ternational Conference on Machine Learning, vol- ume 70 of Proceedings of Machine Learning Re- search, pages 933-941, PMLR. PMLR.
2019. Albert: lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv: 1909. 11942.
Niklas Muennighoff. Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models.
Mostafa Dehghami, Anurag Amab, Lucas Beyer, Ashish Vaswani, and Yi Tay, 2021a. The efficiency mis- nomer. arXiv preprint
Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, et al. 2021. Do transformer modifications trans- fer across implementations and applications? arXiv preprint arXiv:2102.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser, 2018. Universal transformers. arXiv preprint 110.:1887.00811,
Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021b. The benchmark lottery. arXiv preprint 20700220...07002
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 1r2:v:211..11446.
William Fedus, Barret Zoph, and Noam Shazeer. 2021, Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arxiv:2001.0090
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2019, Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv preprint arXiv: 1900.
Naman Goyal, Jingfei Du Myle Ott, Giri Ananthara- man, and Alexis Conneau. 2021. Larger-Seale Seale Trans- formers for Multilingual Masked Language Model- ing. arXiv e-prints, page arXiv:2105.0057.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1600.00250.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701.
Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXh::200..05202
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish, 2021. Scaling laws for transfer. arXiv preprint 1rXiv:2102.01293.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et 2018. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Informa- tion Processing Systems, pages 10414-10423.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai. Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan. Erich Elsen. Oriol Vinyals, Jack Rae, and Laurent Sifre. 2022. Training compute-optinet large language models. Advances Neural Information Processing Systems, volume 35, pages 30016-30030. Curran Associates, Inc.
Noam Shazeer and Mitchell Stern. 2018 Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. pages 4596 4694.4404. PMLR.
David So, Quoc Le, and Chen Liang. 2019. The evolved transformer. In International Conference Machine Learning, pages 5877-5886. PMLR.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint 2001 108611
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2020. Efficient transformers: A survey, arXiv preprint arXiv:2009.06732.
Taku Kudo and John Richardson. 2018. Sentencepiece: simple and language independent subword tok- enizer and detokenizer for neural text processing. arXiv preprint urXiv:1808.062.
Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. 2021a. Are pre-trained convolutions better than pre-trained trans- formers? arXiv preprint arX5
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.

---

Yi Tay, Mostafa Dehghani. Jinfeng Rao. William Fedus. Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Met- zler. 2021b. Scale efficiently: Insights from pre- training and fine-tuning transformers. arXiv preprint arXiv:2 109.. 10686.
Appendix
8.1 Scaling Details for Individual Models
For most models, it was reasonable to follow the uniform scaling method in the main T5 sizes. At each size, the hyperparameters are follows:
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Dara Bahri, Tal Schuster. Huaixiu Steven Zheng. Neil Houlsby, and Donald Metzler. 2022. Unify- ing language learning paradigms. arXiv preprint arXiv:2205.05131.
Model
N₁
def
dmnddl
des
N₂
#Params
Tiny
4/4
1024
256
32
4
16M
Small
6/6
2048
512
32
8
60M
Buse
12/12
3072
768
12
220M
Large
24/24
4096
1024
16
738M
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jes- sica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. 2021. Mip-mixer: An all-mip architec- ture for vision. arXiv preprint S.ddv:2105.0166
XL
24/24
16384
1024
128
32
3B
Table 3: Table of model configurations, Nr is the num- ber of layers, dyf is the size of the MLP, dmoodt is the hidden size of the model. dæv is the size of each key- value vector. No is the number of heads.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
Scaling for Switch Transformer For Switch Transformers, we use the following scaling:
Model
N2
dgf
dmodei
des
Na
Ng
#Params
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general purpose language understand- ing systems, arXiv preprint arXiv: 1905,00537.
Tlny
4
1024
512
64
12
32
173M
Small
2048
64
12
460M
Base
12
3072
768
64
12
32
2B
Large
24
3072
768
64
12
32
8B
XL
3072
768
64
12
128
30B
Table 4: Scaling for Switch Transformer.
Ng is the
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0P 07461..61.
number of experts.
Scaling for Universal Transformer Scaling UTs are generally difficult as described in the main text. There were main considerations for scal- ing UTs. Initially we tried scaling the number of recurrent operations. However, we found that even with an increase of FLOPS, this does not lead to improved performance. Overall, the UT model might be pretty slow and therefore a model with the same hparams as vanilla XL might be infeasible to run. Hence, we explored increasing the width of the MLPs 32K to see if UTs would scale in this manner.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint 2266.220607768
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2022b. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824 24837. Curran Associates, Inc.
Model
Ng
drf
disodet
dyc
N1
#Params
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv: 1907. 10430.
TT
373
1024
128
32
8
IIM
UT Small
3/3
2048
512
32
8
52M
UT Base
3/3
3072
768
64
12
127M
Large
32768
64
283M
Table 5: Table of model configurations. Ng is the num- ber recurrent operations, d is the size of the MLP. dmodet is the hidden size of the model. da. is the size of each key-value vector. Nu is the number of heads.
Zhilin Yang. Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2017. Breaking the softmax bot- tleneck: A high-rank mn language model. arXiv preprint 1rX.v:1711.03953.
8.2 Full Results

---

Lirg
1C284
10 Moze
1223 Uarg
LOwe7 2m
Omm
5mm
Smcr
+507 13001
1920 10007100
110%5
17
30:
PL/R0
200
40/000
Figure 6: Quality-FLOP trade off for the upstream Negative Log-Perplexity vanilla Transformer compared to other models.

---

DCam
Eediedd
Tandll
Hapton lgg NwW
Derter 2m.0
San Lante
1a Log
Jaforrer
30100
Juc 4344
32:40
32.00
Parame
Oee
Johnn
Figure 7: Quality-Parannn -arrity- trade off for the upstream Negative Log-Perplexity of vanilla Transformer compared to other models.

---

DCov
Fasta
721
Lanel Tay
LOwn
NG Thanshorer
Perbereer
Darn Leng
Lover
14024
07
10.00
10
Figure 8: trade off for the upstream Negative Log-Perplexity of vanilla Transformer compared to other models.

---

Abett
DCOF
Distvo
Puredl
Brchop Smull
LiatLaree
L2om
NLP Hiorr
MSS Treraforter
Poforra
1
Jiee Tiyy
Swith
Tearefomerou
LT
Larg
1aar
Figure 9: Quality-FLOP trade off for the downstream SuperGlue Accuracy of vanilla Transformer compared to other models, with respect to FLOPs, number of parameters, and throughput.

---

Fanml
Lantted Eeis
fravings
HJT
TrandorTRr
Tortrrer
Xin
Twy
2017 1agge
1501 1200
Swich
Ovid
Large
Joinn
Figure 10: Quality-Parameter trade off for the downstream SuperGlue Accuracy of vanilla Transformer compared other models.

---

1709
163 Marr
burh and
Ealoome 77
Switt
17
Deca Largs
Yawr
Sqpa
AiThooggoou
Figure 11: Quality- Throuuuth trade off for the downstream SuperGlue Accuracy of vanilla Transformer compared other models.

---

Abetl
DCor
Puned
Lerg
Zome Tem
500m
Loom
NLP Hiarr
MSS Treraforter
Poform
Spaararf Rdl
Oue Tay
Swith
Treretomersu
LT
Figure 12: Quality-FLOP trade off for the downstream Glue Accuracy of vanilla Transformer compared to other models.

---

Kirdd Lial
HL.
TrandorTR
Large
Mancitp Dert
Lod Nour Lagg
2017
Swith
2011
Figure 13: Quality-Parameter trade off for the downstream Glue Accuracy of vanilla Transformer compared to other models.

---

Bark
Aue Umall
Joort Eert
1000
16P Marr
Weth
Omm
Layg
Swde
A/Thoaggood
Figure 14: other models.
trade off for the downstream Glue Accuracy vanilla Transformer compared to

---

Funnel
Jone Tryy
MA1 1001
5000
4010
1947
Figure 15: Quality-FLOP trade off' for the downstream Squad Accuracy of vanilla Transformer compared to other models.

---

Abas
Bihhed
Pannd
Ed0
131 2014
43-7
Lown
Smaal
20300
Swithh
TeeraformrrL
Avic
Figure 16: Quality- Maality-Parameter trade off for the downstream Squad Accuracy of vanilla Transformer compared to other models.

---

Fuorel
120
Voder 2819
112 22
Sony
Figure 17: Qualit--Throughee trade off for the downstream Squad Accuracy of vanilla Transformer compared other models.


Reviews:
[{"review_id": "3zCtoOpCP8", "reviewer": "Reviewer_n3js", "summary": "", "questions": "Question A: From line 387 to line 389, \"A high αU,D value means that the transfer to the downstream tasks is better as a model scales.\" Isn't it more likely that upstream perplexity is not closely related to downstream metrics, rather than the model's transfer ability? \n\nQuestion B: In Figure 4(b), there is a huge fluctuation in the performance of the MoS model from 16 to 36 layers, have you tried deeper layers like 100 to compare the results?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and give your thoughtful comments!\n\nRe: Only C4 pretraining. As the main goal of the work is to derive a relationship between various model architectures and their scaling behavior, we intentionally decided to fix the pretraining experimental settings between all models for fair comparison. As C4 is a widely used and publicly available dataset, as well as one where many of our considered architectures developed their models on, it was a natural choice for this. While deriving the relationship between pretraining dataset, architecture, and scale would be very interesting, the cross product becomes infeasible, given the huge number of experiments we run as the reviewer mentioned.\n\nRe: Encoder-decoder vs decoder-only. Similar to the pretraining dataset, we decide to also fix this variable as many of the architecture variants we consider can be expressed as either encoder-decoder or decoder only. Some however, such as Funnel Transformer or MLP-Mixer are only applicable to the encoder side. This made encoder-decoder be a more suitable backbone for more of the modifications we wanted to evaluate.\n\nQuestion A: Depends on what the reviewer means by “transfer ability” but we see from Table 2 that the relationship between upstream and downstream performance does fluctuate quite a bit between architectures, showing that upstream performance does not necessarily transfer to strong finetuning performance. \n\nQuestion B: Unfortunately we did not, in order to bound the number of experiments, we tested all models only to the FLOP region shown in the paper.\n\nFinally, thank you for your notes on the presentation, we will revise the paper to improve it for the final version.\n\nThanks again for your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A huge number of experiments. Pretrain and finetune over 100 models (10 model architectures with different sizes) and get valuable findings in the scaling laws with different architectures.\n2. Well-written paper.", "reasons_to_reject": "1. The c4 dataset is the only one for all experiments and not sure whether the pretrain dataset will have an effect.\n2. All the models are Encoder-Decoder arch while Decoder-only missed. But I believe this should not detract from the fact that this is a good paper that would contribute to the field.", "questions_to_authors": "Question A: From line 387 to line 389, \"A high αU,D value means that the transfer to the downstream tasks is better as a model scales.\" Isn't it more likely that upstream perplexity is not closely related to downstream metrics, rather than the model's transfer ability? \n\nQuestion B: In Figure 4(b), there is a huge fluctuation in the performance of the MoS model from 16 to 36 layers, have you tried deeper layers like 100 to compare the results?", "paper_topic_and_main_contributions": "The topic of this paper is the relationship between scaling laws and model architecture The authors attempt to explore whether inductive bias inside the model architecture affects the scaling performance.\nThe contributions of this paper are:\n1. Comprehensive experiments on over 100 models in pretraining and finetuning to derive scaling laws for different model architectures.\n2. They find that models perform well in one scale region and do not consistently be the best in another.  While the vanilla Transformer performs best in scaling.\n3. They observe that some architectures are hard to scale up, such as the performer, suggesting that the linear Transformer may have a bottleneck in scaling.\n\nOverall a good article with very detailed experiments and very valuable findings learned with a lot of computing budget, especially in the trend of LLMs. These findings will be very helpful for model selection in both research and engineering.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure 1, there are some circles named Mini or Tiny but no architecture names.\n2. In Figures 2 and 3, red and green may not be friendly to red-green color blindness.\n3. Table 1 is too large. Maybe make it smaller and leave the full results in the appendix."}}, {"review_id": "RDW6HXHBWk", "reviewer": "Reviewer_rySs", "summary": "", "questions": "- The vanilla transformer seems to be winning across the board, particularly in the downstream evaluations. On the other hand, that is also the architecture that has been around for longer. Would it be reasonable to hypothesize that this gives an advantage to transformers in terms of the training workflows (including optimizers, schedulers, normalization blocks, starting hyperparameter values, etc and their interplay) having grown adapted to this architectures for longer? In other words, were the community to spend as much time as it has training transformers on hyperoptimizing, say, MLP-Mixer, couldn't that change the reported results?\n- It is a bit puzzling that the paper didn't mention anything about the effect of dataset size, and data-optimal scaling like the \"chinchilla scaling\", since data availability is starting to be the bottleneck in training LLMs, at least is some conditions, and the interplay between model size and dataset size is currently arguably the de facto criterion to allocate training budget. Is it still possible to convincingly justify the choice of ignoring the effect of scaling dataset size in this type of studies? Wouldn't it be reasonable to expect that different model architecture will also display different scaling depending on where they are located in terms of allocating flops to parameters rather than data? If that is the case, the main results of this study might change considerably at different data regimes.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper! These are great questions. We address your comments below:\n\nRe: Chinchilla optimality / Dataset size. The importance of pretraining dataset size is limited in this study, as unlike the setting studied in Hoffman et al. and many recent LLM works, we study the relationship between architecture, model size, and *downstream finetuning performance*, whereas Hoffman et al. studies language modeling / few-shot learning. For few-shot, evaluation is done directly on the pretraining checkpoint without finetuning, and thus pretraining convergence and the diversity of knowledge available in pretraining dataset matters. In our experiments, the finetuning tasks supply all models with exactly the same examples required to learn the task, and we are able to finetune each model until convergence to accommodate for different convergence behaviors.\n\nAs such, we intentionally fix the pretraining experimental settings between all models for fair comparison and to narrow the search space: all models see the same number of pretraining tokens before finetuning. The pretraining setup for our encoder-decoders in this paper closely follows the well established ablation setting in Raffel et al. 2019. Since we follow the T5 setting, we also pretrain with span corruption, which has been shown to not perform well on few-shot learning (Tay et al. 2022, UL2), making Hoffman et al. especially not applicable.  \n\nOne benefit of comparing models in this way is precisely because it is less dependent on pretraining design to acquire in-context learning (ICL) ability. It is still an open question whether some of the architectures we study in this work can even exhibit ICL abilities in the same way as the normal Transformer, which seems like a separate open research question. For example, it is not possible to apply a token-level causal mask to a Funnel Transformer, making the usage of pure causal language modeling (CLM) pretraining objective not possible. This complicates comparisons with models that can indeed train purely with CLM, which works better for ICL. Designing the experiments as we did levels the playing field in this regard and allows us to focus solely on the comparison of inductive biases and scale.\n\nThis is not to say that Scaling Laws vs. Model Architectures for ICL is not important, but it is simply outside the scope of this work and the diversity of architectures we evaluate. We are happy to change the framing and/or add this discussion to the paper to clarify in the final version, as well as reference Hoffman et al. An aside on Chinchilla optimality: the process proposed by Hoffman et al. requires plotting multiple isoFLOP curves at smaller FLOP bands in order to extrapolate to larger FLOP bands. Hoffman et al. only showed this extrapolation process to be accurate for their relatively vanilla Transformer architecture. In order for us to properly apply Hoffman et al. to our work here, we would have to not just derive compute-optimal scaling laws as they do for every architecture, but also empirically verify that the extrapolation assumptions of Hoffman et al. still hold for all architectures considered. This would also be a different research question than the one studied by this paper.\n\nRe: Lack of mechanistic explanations or actionable recommendations. We did indeed aim to provide an empirical study of parameter scaling multiple diverse architectures, and not a mechanistic interpretation of any particular architecture. We do believe though that there are many significant findings of interest to the community of which we thoroughly describe in the results and conclusions of our paper. Mainly, different architectures scale differently, and what might be the best architecture at one compute region may not be the best at another compute region. This has major implications for how researchers and practitioners should go about conducting and interpreting architecture research, especially as many architecture papers only show a narrow slice of the compute region.\n\nRe: Compute required for this study. Line 290 clearly states “We pretrain and finetune our models with 16 TPU-v3 chips with data parallelism,” Table 1 lists the training speed in steps per second for each model, and Line 278 states that we pretrain for 2^19 (524k) steps, which is sufficient to estimate the degree of compute required for this study. \n\nRe: Vanilla Transformer and hyperparameters. This is a true limitation of this study that we discuss in the limitations section. As finding the optimal way of training specific architectures is a never ending quest for the research community, it will never be possible to fully disentangle this. However, we have tuned each model to the best of our ability, reaching out to authors / experts when possible.\n\nThanks again for your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Very extensive empirical evaluation of the scaling properties across different model architectures spanning a range of 15M to 40B parameters\n- The paper is clearly written and straightforward\n- The paper presents a simple, yet solid and unified framework for the evaluation of a large variety of models", "reasons_to_reject": "- The paper doesn't analyze a crucial factor determining the scaling properties of model performance which is dataset size, and doesn't even refer to the literature pertaining to data-optimal scaling laws like the so-called \"Chichilla scaling\"\n- The paper is rich in empirical results, but unfortunately limited in terms of mechanistic explanations and actionable recommendation. With the exception of some passing observations like noticing the importance of cross-attention in the encoder to guarantee downstream performance, the paper does not provide many insights on what architectural modules support favorable scaling\n- The paper stands out for its extensive empirical evaluation, but that is not accompanied by a detailed enough report of the computation and hardware budget that required this study. The paper mentions the use of TPU-v3 pods, but doesn't mention how many pods for how long had to be used in the study, and what the resulting energy and CO2-emission footprint was. This might be a concern for this type of large-scale studies. To be clear, this type of studies are clearly important also because, despite their possibly large footprint, clarifying models scaling could potentially help amortize that cost over the savings of future more efficient architectures. But in the current conjuncture of GPU scarcity, concerns over energy costs and growing sensitivity for climate issues, there seems to be a growing expectation that these points be explicitly addressed", "questions_to_authors": "- The vanilla transformer seems to be winning across the board, particularly in the downstream evaluations. On the other hand, that is also the architecture that has been around for longer. Would it be reasonable to hypothesize that this gives an advantage to transformers in terms of the training workflows (including optimizers, schedulers, normalization blocks, starting hyperparameter values, etc and their interplay) having grown adapted to this architectures for longer? In other words, were the community to spend as much time as it has training transformers on hyperoptimizing, say, MLP-Mixer, couldn't that change the reported results?\n- It is a bit puzzling that the paper didn't mention anything about the effect of dataset size, and data-optimal scaling like the \"chinchilla scaling\", since data availability is starting to be the bottleneck in training LLMs, at least is some conditions, and the interplay between model size and dataset size is currently arguably the de facto criterion to allocate training budget. Is it still possible to convincingly justify the choice of ignoring the effect of scaling dataset size in this type of studies? Wouldn't it be reasonable to expect that different model architecture will also display different scaling depending on where they are located in terms of allocating flops to parameters rather than data? If that is the case, the main results of this study might change considerably at different data regimes.", "paper_topic_and_main_contributions": "This paper provides an extensive empirical evaluation of the scaling properties of different encoder-decoder models for seq2seq tasks including transformers, efficient variants thereof like Performer and ALBERT, and non-transformer architectures like Dynamic Convolutions and MLP-Mixer. In particular, the question under investigation is how the inductive bias due to the architecture affects the scaling of the perplexity loss as a function of model size (measured as flops, as the training dataset is fixed). The main finding of this effort (which spans a remarkable 100 different models ranging from 15M to 40B parameters) is that architecture indeed has an effect on the performance of the pretraining seq2seq task, and an even starker effect on downstream fine-tuning tasks (NLU tasks coming from GLUE and SuperGLUE and SQuAD). In addition, regular transformers are however mostly still the winner in regard of how performance scales as a function of model size.\n\n### Changes after rebuttal\n* soundness increased to good\n* confidence increased to quite sure", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Hoffmann et al. \"Training Compute-Optimal Large Language Models\", NeurIPS 2022"}, "extra_text": {}}, {"review_id": "Ud1XOzHAE5", "reviewer": "Reviewer_MfUv", "summary": "", "questions": "Is there available some repository with your code somewhere?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time in reading our paper, providing your valuable feedback, and for recognizing that this would be an important contribution to EMNLP!\n\nRe: 40B and emergent abilities. Emergent abilities as described by Wei et al. 2022 pertains only to few-shot tasks such as BIG-Bench Hard (BBH). Our work only studies the relationship between model scale, architecture and downstream finetuning performance. There are a variety of reasons that we formulate our study in this way as explained in further detail in our rebuttal to Reviewer rySs. In short, it allows us to consider a wider diversity of architectures than that of those known to exhibit in-context learning ability or emergent abilities, and we are able to compare them in a more controlled way. We would further add that for traditional finetuning performance, emergent abilities based on model size are not known to exist as evidenced by the SuperGLUE leaderboard which shows a more gradual relationship between increasing model size and finetuning performance, e.g. T5 11B and PaLM 540B is only 1pt apart.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper highlights another dimension to consider when introducing new architectures for language modelling, namely their scale. Since the same architecture can perform differently on different scales, researchers who propose new architectures should consider reporting how their proposal scales with FLOPS (or number of parameters); as to characterize their proposal more thoroughly. \n\n- The paper additionally provides a large set of results that can be used for comparison (i.e. as baselines). Specially by those researches who want to introduce novel architectures or mechanisms for language modelling (see point above).\n\nI believe these two reasons are enough to make this paper an important contribution to EMNLP. ", "reasons_to_reject": "- The paper considers models up to 40B parameters. Yet the so-called *emergent properties* of language models seem to appear for larger model sizes (Wei et al. 2022). It’s therefore a valid question whether the scaling of different architectures breaks down after crossing the (architecture-dependent) critical scale of emergent properties. Indeed, it would have been interesting to additionally study this architecture-dependent critical scale itself.\n\n- There is little to no details available to reproduce the results of the paper. \n\n*Links to references*\n\nWei et al. (2022): https://arxiv.org/pdf/2206.07682.pdf\n", "questions_to_authors": "Is there available some repository with your code somewhere?", "paper_topic_and_main_contributions": "This paper studies how the performance of different language model architectures changes with their scale (i.e. FLOPS or number of parameters) on both upstream and downstream tasks. The authors considered up to 10 different architectures and empirically showed that the latter can display very different scaling behaviors, thereby highlighting a new dimension to consider (i.e. model scale) when constructing/proposing novel architectures. In particular they showed that, although a given architecture might perform well at a given scale, its scaling behavior might be suboptimal. \n\nThe authors also demonstrated that the same architecture can display different scaling wrt. the upstream and downstream tasks; a result which mirrors previous findings, for fixed-size architectures. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I1XIvFGCjf", "reviewer": "Reviewer_1kqg", "summary": "", "questions": "Can the authors provide the scaling effect only on the MLP layers? Beacuse the MLP is the fundamental architecture for neural networks.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We address your concerns below:\n\nRe: Vector graphics. All diagrams used in the paper are indeed vector graphics, and appear to be rendering correctly as such in Google Chrome.\n\nRe: Metrics. In Table 1 we do indeed show the negative log perplexity of the pretrained models, which is a function of the loss. Regardless, it has been known in the NLP community that pretraining perplexity does not necessarily correlate with downstream performance, as we additionally reaffirm in Table 2. Finetuning loss especially does not correlate with performance, as it could mean overfitting. In general, we base our evaluations off of downstream metrics as those are the metrics most important to practitioners.\n\nRe: Metric formulas. We only use standard common metrics and tasks in our work, and for particular datasets we refer to the original dataset papers in Section 4, that provide detailed descriptions of their metrics and averaging conventions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are detailed and adequate.\n2. Investigating the scaling behaviour of the neural networks is of importance to the community.", "reasons_to_reject": "1. Although the experiments are adequate with detailed illustration, I cannot see the words or numbers clearly enough.\n2. The metrics used in this paper is indirect with the learning dynamics of the transformers. I wonder how the loss is affected by the scaling behaviour, because the training process is direcly pushing the gradients w.r.t loss, instead of other metrics.\n3. I suggest the authors clearly define all metrics used in the paper with mathematics formula.", "questions_to_authors": "Can the authors provide the scaling effect only on the MLP layers? Beacuse the MLP is the fundamental architecture for neural networks.", "paper_topic_and_main_contributions": "This paper aims to find out the scaling properties on network architectures in transformer models. Specifically, they investigate the effect on upstream and downtream tasks, respectively. Experiments are done in different variants of transformers.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. As discussed before, please make sure all the figures are clear to be seen."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Pros:
- Extensive experiments with many models.
- No reviewers mentioned this, but I think that the discussion of how hyperparameter settings can change scaling laws could be very valuable on a practical level, as it could support hyperparameter selection decisions.

Cons:
- They only consider encoder-decoder models. Because the architectures are limited, there is a missed opportunity to consider inductive biases of radically different architectural approaches.
- The analysis and discussion is limited. Given that the paper is framed as an exploration of inductive bias, there is little to no discussion of the source or nature of those inductive biases when using different architectures. This makes the title a little misleading; perhaps it should be called “How Does Architecture Influence Scaling” instead.
- They only consider up to 40B parameter models, which is below the range claimed to show dramatic breakthroughs on capabilities like chain of thought.
Decision: Accept-Findings