Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Authors: Yi Tay
Mostafa Dehghani
Samira Abnar
Hyung Won Chung
William Fedus
Jinfeng Rao
Sharan Narang
Vinh Q. Tran
Dani Yogatama
Donald Metzler
Venue: Submitted to ICLR 2023 (2023)

Abstract: There have been a lot of interest in the scaling properties of Transformer models \citep{kaplan2020scaling}. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour?  How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.

Under review as a conference paper at ICLR 2023
SCALING LAWS VS MODEL ARCHITECTURES: How DOES INDUCTIVE BIAS INFLUENCE SCALING?
Anonymous authors Paper under double-blind review
ABSTRACT
There have been lot of interest in the scaling properties of Transformer models (Kaplan et al. 2020). However, not much has been done on the front of inves- tigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does induc- tive bias affect scaling behaviour? How does this influence upstream (pretrain- ing) and downstream (transfer)? This paper conducts a systematic study of scal- ing behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers. Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) ar- chitecture is indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architec- tures are currently evaluated in the community.
INTRODUCTION
There have been a lot recent interest in the scaling properties of Transformer models (Kaplan al, Hernandez et al. |022 Bahri et (222 Henighan et al. 2020 (Ta et al. [222 Abnar et al., different inductive biases imposed by model architectures. Improvements at a specific scale (compute, size etc) are often assumed to transfer different scales and compute regions (So et al.) 2019; Choromanski et al.] 2020 Lan et al. 2019 Dehghani et al. 2018) and new research is often presented in a point-wise fashion with respect to scale. short, is not uncommon for new methods to presented with data points very specific limited compute regions (c.g., base size). We believe that understanding the interaction between architecture and scaling laws is crucial as designing models that perform well at diverse scales will likely have significant impact.
2020
2021
However, not much is understood about the scaling properties
This paper is an attempt to understand the effect inductive bias (architecture) on scaling laws of language models. To this end, we pre-train and finetune over ten diverse model architectures across multiple compute region and scales (e.g., from 15M to 40 Billion parameters). In total, we pre-train and finetune over 100 different models of different architectures and sizes and present insights and challenges at scaling these ten diverse architectures.
We consider broad spectrum of models in our extensive experiments. Concretely, we consider several well-established Transformer variants (Vaswani et al. (017)) such as Evolved Transformer (So et al. 2019), Universal Transformers (Dehghani al.) (2018) and Switch Transformers (Fedus) et al. 2021). We also consider lightweight models such as ALBERT (Lan et al., 2019) and/or ef- ficient Transformers (Tay et al. 2020) such as Performer (Choromanski et al., 2020) and Funnel
Transformers (Dai et al.) 2020). In our comparison, we are also interested in finding out if gen- eral improvements the Transformer architectures such as Socture-of-Softmatt (Yang et al. 2017) and/or Gated Linear Units (Dauphin et al.) [2117] [Shazeer (020) influence the scaling behaviour of models. Finally, we also evaluate models outside the family of Transformers including Lightweight convolutions (Wu al. (2019). Dynamic convolutions (Wu al. 2019) and the recently proposed MLP-Mixers (Tolstikhin et al. 2021). Figure [ illustrates an overview about the experiments we run.
We also note that scaling these models is not as straightforward as it seems, i.c.. there are intri- cate details of scale that intertwined with architectural choices which we study in detail in this

---

Under review as a conference paper at ICLR 2023
85-
-14-
00-
-18-
Negative Log-P Par--eeleiitty
D
SuperGlue Accuracy
Qvine
22.0
70-
O O+
66-
Ounim
24-
60-
O
Ou
52-
-3
90
1.te1127e-12
ddbe1)
8.6+17
1,8-13
355613 7.0e18.0016
1220112
1.8:113
330113 7.0c1154c
FLOPS
FLOPS
(a) Upstream: Negative Log-Perplexity
(b) Downstream:
Accuracy
Figure I: An overview cumpute-performance (FLOPs performance) plot of all the diverse mod- els and architectures we pretrained and finetuned in this study, Colors represent different model architectures and size of the circles represent the size of the model (parameters).
paper. For example, distinct feature of Universal Transformers (and ALBERT) is parameter shar- ing. Hence, compared with standard Transformers, this architectural choice significantly warps the scaling behaviour only with respect to performance but also amongst compute metrics such as FLOPs, speed and number of parameters (Dehghani et al.) 222 Conversely. models such as Switch Transformers are on the other end of the spectrum with an uncommon relationship between FLOPs and number of parameters, i.e., they have high parameter to FLOPs ratio. This difficulty makes navigating this landscape challenging.
Our Contributions and Insights
The key contributions of this paper are as follows:
For the first time, we derive scaling laws for different inductive biases and model architec- tures. We find that this scaling coefficient differs greatly from model to model. We believe this is an important consideration in model development. turns out that amongst all ten architectures that we consider, the vanilla Transformer has the best scaling behaviour, even if its absolute performance each compute region is not the greatest.
We observe that models that operate well in compute-scale region is not necessarily the best in another compute-region. Moreover, we find that certain models have difficulty scaling despite performing decently (comparably) lower-compute regions. This has im- plications, since is difficult to get the fulll picture of a model's scalability with pointwise comparisons at a certain compute-region.
We find that when it comes scaling different model architectures, upstream pre-training perplexity might not correlate well with downstream transfer. Hence, the underlying archi- tecture and inductive bias is also crucial for downstream transfer.
We highlight the difficulties of scaling with certain architectures and show that some mod- els do not scale (or scale with a negative trend). We also find concerning trends where linear-time attention models such as Performer struggle with scaling up.
RELATED WORK
Kaplan et al. (200)) studied empirical scaling laws of the decoder-only Transformer language mod- els. They focused on the standard left-to-right language modeling objective with the cross-entropy loss as the performance metric. One of the main findings is that the loss scales as a power-law with three major characteristics of the model training: model size, dataset size and the training
2

---

Under review as a conference paper at ICLR 2023
compute. Another somewhat surprising finding is that the model shapes such width depth of the Transformer network have minimal effects on the eross-entropy loss for a wide range of scales. Subsequent works (Henighan al.] 2020; Hermandez et al. 2021 made similar conclusions for autoregressive generative modeling and for transfer learning. respectively. This finding is also gen- erally supported by (Tay al. (202Tb) but discrepancies were found for the gap between pretraining and finetuning highlighting the fact that observing downstream performance of large language model indeed important. (Tay al. 2021b), the effect of depth was unusually pronounced for downstream performance.
Raffel et al. 2 studied the effect of pre-training objectives, model structures (e.g., encoder- decoder, decoder-only), pre-training dataset size and training strategy on the transfer learning. They showed that the downstream performance monotonically increases with the model scale (from 60M to 11B parameters). While they studied several model structures, the Transformer implementation mostly the same as the original Transformer by Vaswani et al.. (2017). [Cnneau al.) (2020); Goyal al. (2021) scaled-up multilingual encoder-only architectures up 1IB parameters while maintaining the original Transformer implementation. They found that scaling the model improves its cross-lingual ability. Fedus al. (202T) scaled sparse model based Mixture of Experts (MoE) models up to trillion parameters.
While previous studies have repeatedly shown the benefits of scale for language understanding tasks for both dense and sparse Transformers and cross-lingual abilities, all of these used the same Trans- former implementation within each studies. With plethora of improved Transformer architectures proposed in the literature, it is timely to investigate which of these improved architecture has the best scaling properties. The main goal of this paper is systematically study how inductive biases imposed by these Transformer variants affect the scaling behavior in a shared software and hardware settings. This is in similar spirit to (Narang et al., (02T)) that studies the impact of architectures on performance, Our analysis extends that of (Narang et al. 2021) to the model scale axis.
3
METHODS
This section outlines our experimental setup.
3.1
MODELS
This section describes the models we evaluate in our experiments. Our models are largely imple- mented in sequence to sequence framework (Sutskever et al. 2014) following the convention of T5 (Raffel et al.] 2019). Encoder-decoder models are a natural choice for this experimentation because they can universally express both encoding and decoding tasks.
Transformer Variants
We consider several standard Transformer variants.
Transformers (Vaswani et al. 2017) The basic vanilla Transformer architecture. Our ba- sic setup considers the T5-style of Transformers (Raffel et al.) 22019), which largely follows the vanilla Transformer except that it uses relative attention instead of sinusoidal position embeddings and pre-layer normalization, i.e. layer normalization applied before each sublayer.
Evolved Transformers (So et al. 2019) A transformer architecture learned via AutoMI.. The architecture comprises of convolutions and attention. We scale Evolved Transformers following the same pattern as vanilla Transformers.
Universal Transformers (UT) (Dehghani et al., 2018) Transformer architecture with shared parameters and recurrent-like computation for transform layers. Scaling UTs challenging because of parameter sharing. While we are able to also increase dFF or dmoodets the increase in parameters is of magnitude Niayers than standard Transformers. Another axis of exploration is to scale r the number of repeated computation at each UT layer this increases computation (number of FLOPs) but does increase the parameter size of the model.
Switch Transformer (Fedus et al., 2021) a sparsely activated mixture-of ceature-offexpertss archi- tecture. The Sparse Transformer another model with unusual relationship between
3

---

Under review as a conference paper at ICLR 2023
number of parameters and compute. When we scale this model uniformly, the number of parameters easily reaches the ballpark of 40B.
Efficient Transformer Variants These class of models are mainly concerned reducing compu- lational costs, memory usage, or parameter count of models.
Performer (Choromanski et al. 2020) A linear time attention model using generalizable kernel attention. For simplicity, we adopt the relu kernel variant for our experiments. We scale Performer in the similar fashion (i.e., uniform scaling) as vanilla Transformers.
Funnel Transformer (FT) (Dai et al., 2020) A Transformer architecture that downsamples the input sequence across the layer stack. Our implementation uses FT only in the encoder and reverts to vanilla Transformer in the decoder following Narang et al. (2021).
(Lan al. 2019) A lightweight transformer architecture that shares parameters across all layers and factorizes the embedding and output softmax layers. For our seq2seq ALBERT, we also share the weights of encoder and decoder.
ALBERT
General Improvements We consider general improvements that are not necessarily tied to Trans- formers. select candidates that have shown to do well in Narang et al. (202T).
Mixture of Softmaxes (Yang al. 2017) A transformer architecture adopting the MoS method at the Softmax layer.
Gated Linear Units with GeLU (Gnn-Tranforrmrr) Replacing position- -asse feed- forward-networks in Transformers with Gated Linear Units (Dauphin al., 2017).
Non-Transformee Architectures We are interested in the scaling behaviour of Ton-Transformer based architectures such as convolutions and/or mixer architectures.
Lightweight Convolutions (Wu et al. 2019) Lightweight depthwise convolutions that have shown promise over Transformer architectures.
Dynamic Convolutions (Wu et to create time-dependent kernels.
2019) An extension of the Lightweight Convolution
MLP-Mixers (Tolstikhin et al. 2021
Mixers are recently proposed architectures that learn lightweight mixing of tokens. Since Mixers have not been used in autoregressive decoding, we only use token-mixers on the input encoder.
3.2 EXPERIMENT SETUP
Our setup, along with all models, are implemented in Mesh TensorFlow (Shazeer et al. 2018 library with similar interface to TensorFlow but enables distributed model parallelism across multi- ple workers. For fair comparison, all models are pretrained for 219 steps on the english C4 corpus optimized using an inverse square root learning rate with Adafactor (Shazeer & Stern) 2018). All
models use the same SentencePiece tokenizer (Kudo & Richardson 2018) containing 32K sub-
words. This closely follows the setup in the T5 paper (Raffel et al.) 2019). Finetuning is performed for 100K steps on a mixture of GLUE (Wang et al. 2018), SuperGL.UE (Wang et al. 2019) and both upstream (pre-training) validation perplexity
SQuAD (Rajpurkar al. 2016). We evaluate as well as downstream transfer for NLU tasks (GLUE + SuperGLUE + SQuAD) after fine-tuning. We pretrain and finetune our models with 16 TPU-v3 chips with data parallelism. All large models have a model parallelism of 2 and XL models have model parallelism of 8.
Model Sizes We consider several different model sizes for each architecture. For models that are straightforward to scale, we simply follow the standard convention in Raffel et al. (2019), moving from small to base, to large and XL. We include a tiny version each model to observe how different models behave at lower compute regions. For models where it was not straightforward to scale (e.g.. Universal Transformers, ALBERT), we tried to scale them in a similar fashion but faced obvious limitations such as getting ALBERT to have the same number of parameters T5 XL without incurring a huge number cost in terms FLOPs. For convolutional models, we consider dmodel to be the hidden size (i.e., channel depth) for the one-dimensional convolution layers. Values such
4

---

Under review as a conference paper at ICLR 2023
Am
1005
(a) ALBERT
(b) DConv
(c) Evolved
(d) Funnel
Virge
(e) Transformer-Gue
(1) LConv
(g) MLP Mixer
(h) MoS Transformer
Performer
(i) Switch Transformer
(k) Universal Transformer
Figure 2: Upstream Negative Log-Perplexity of vanilla Transformer compared to other models.
as dgv, NH then become redundant. Details on scaling detail of each architecture can be found in the supplementary material.
3.3 MAIN RESULTS
We report the main results of this paper in Table I ters, FLOPs (of single forward pass) and speed (steps per second). We also report validation perplexity (on upstream pre-training) and results on 17 downstream tasks. The results are reported aggregates of GLUE, SuperGLUUE and SQuAD. While codebase used by Raffel et al. (2019) and hence expect our experimental results to match theirs, we verify that our T5 base does achieve similar results to what is reported in Raffel al. 2019).
We report the number of trainable parame-
the same Mesh TensorFlow-basee
3.4 DO ALL MODELS SCALE THE SAME WAY?
This section investigates if all model architectures scale in the same way.
Upstream Perplexity Figurerrepon the scaling behaviour of all models as we increase the num- ber of FLOPs. We observe that the scaling behaviour of all models quite unique and distinct, i.e, most of them are quite different from standard Transformers. Perhaps the biggest finding here is that most models (e.g., I.Conv, Evolved) all seem to be on-par better than standard Transformers but fail to scale with a higher compute budget. Another interesting trend is that "lincar" Transform- ers such Performer fail scale shown Figure The pre-training perplexity metric only decreases by 2.7% going from base to large scale compared to 8.4% of the vanilla Transformer.
'The largest Switch transformer was scaled in pretty sub-optimal way. So we don't 't think is representative of the full potential of the Switch family. Take the last data point Switch with pinch of salt.

---

Under review as a conference paper at ICLR 2023
Lu mm
10.00
FLOB5
1099
10
RL50
(a) ALBERT
(b) DConv
(c) Evolved
(d) Funnel
CLUANO
Lage
PLPRS
(c) Transfomer-GLL
(f) LConv
(g) MLP Mixer
(h) MoS Transformer
Large
Performer
(i Switch Transformer (k) Universal Transformer
Figure 3: Downstream accuracy of vanilla Transformer compared to other models.
Downstream Transfer Figure 3 reports the scaling curves of all models on downstream transfer. The overall finding that most models have distinct scaling curves compared to Transformers is also evident in downstream tasks. It is also noteworthy that most models have different upstream and downstream scaling curve. We find that some models such as Funnel Transformer and L.Covss that seem to hold out pretty well on upstream but suffer substantially on downstream. As for Performer, the performance (disparity) seems be even greater downstream as compared upstream. No- tably, the SuperGLUE downstream tasks generally require pseudo cross- tross-attention on the encoder, which models such as convolutions are not equipped to handle (Tay et al. 2021a). To this end, we find that certain models may have difficulty learning the downstream tasks despite good upstream performance.
3.5 ARE THE BEST MODELS EACH SCALE DIFFERENT?
Figure [egure[shor shows the Pareto-frontier when plotting compute against upstream and downstream perfor- mance. Since the colors of the plot represent different models, we can observe that the best model for every scale and compute region might be different. Moreover, from Figure we also observe this. For example, the Evolved Transformer seems to do well against the standard Transformer at tiny small region (downstream) but this quickly changes when scaling the model up. We also ob- serve this with MoS- Too-Trrasfrrmer where it clearly outperforms vanilla Transformers at some regions but others.
3.6 SCALING LAW FOR EACH MODEL
Table 2 presents the slope of the fitted linear line for each model across multiple scenarios. We derive by plotting F (FLOPs), U (upstream perplexity). D (downstream accuracy). P (number

---

Under review as a conference paper at ICLR 2023
Table Results on pre-training and finetuning ten different model architectures. Full results (further
varying hyperparameters of these models) be found in the Appendix.
Model
#Params
FLOPs
Speed
Neg Log Ppl
GLUE
SGLUE
SQuAD
Transformer Tiny
16M
1.21
38.4
-2.47
69.3
56.9
73.6
Transformer Small
60M
3.70
22.7
-2.02
78.1
65.3
81.9
Transformer Base
223M
11.4
-1.75
83.8
74.0
86.3
Transformer Large
738M
-1.61
86.4
78.3
88.6
Transformer XL
2.9B
63.8
1.3
-1.49
87.8
81.5
89.5
Evolved Transformer Tiny
19M
1.31
39.7
-2.45
69.6
57.1
69.6
Evolved Transformer Small
79M
4.23
23.7
-2144
75.7
66.2
80.2
Evolved Transformer Base
218M
10.2
8.9
-1.79
83.0
70.5
84.8
Evolved Transformer Large
1.0B
49.3
-1.62
86.2
77.1
88.0
Evolved Transformer XI.
2.2B
-1.55
87.0
78.3
88.2
Universal Transformer Tiny
11M
1.77
38.1
-273
69.8
56.1
62.3
Universal Transformer Small
52M
7.30
18.3
2.12
76,8
64.2
75.4
Universal Transformer Base
127M
20.3
8.4
-1.91
80.0
67.9
80.1
Universal Transformer Large
283M
27.6
1.6
-167
84.0
73.4
85.4
Switch Transformer Tiny
174M
3.25
2.01
78.2
63.8
80.7
Switch Transformer Small
460M
22.3
-185
80.3
68.0
82.9
Switch Transformer Base
2.0B
12.7
8.4
-166
84.2
74.1
86.5
Switch Transformer Large
3.9B
23.0
4.1
-1556
84.6
75.8
87.9
Switch Transformer XL.
29.6B
43.3
0.8
-1.62
84.0
75.2
87.5
Performer Tiny
16M
1.14
42.0
-2.88
S0.5
48.8
15.0
Performer Small
61M
3.50
39.0
-2.44
57.8
51.1
31.1
Performer Base
224M
11.7
-2,23
61.4
53.4
37.8
Performer Large
739M
32.8
4.4
-2.16
62.4
52.4
30.8
Funnel Transformer Tiny
16M
1.10
39.9
-2.58
63.4
49.4
54.6
Funnel Transformer Small
61M
2.96
32.7
-2.11
70.0
58.5
75.1
Funnel Transformer Base
223M
8.10
11.9
-1.83
76.3
62.9
81.6
Funnel Transformer Large
739M
-1.69
79.8
67.1
83.8
Funnel Transformer XL
2.9B
40.3
1.89
-1.61
79.8
68.0
83.7
ALBERT Small
15M
3.57
42.0
-2.66
73.7
62.0
77.1
ALBERT Base
21M
9.40
16.4
-2.88
69.0
57.2
64.3
ALBERT Large
34M
31.6
5.1
-2.0
62.9
54.1
27.3
MoS Transformer Tiny
27M
1.29
39.7
-2.37
70.6
57.9
74.1
MoS-1 Transformer Small
81M
26.3
-1.98
79.7
67.1
83.1
MoS- Transformer Base
257M
11.4
8.6
-1.00
84.5
73.9
86.8
MoS -SSTTranffrrrrr Large
800M
35.0
3.4
-1.56
86.5
79.7
89.1
ToS-rransfrrmer XL
2.9B
112
1.2
-1.45
88.2
81.4
90.0
GLU-Translrmerr Tiny
26M
1.29
31.7
-2.35
70.5
57.0
74.2
GLU--Trnsformer Small
77M
3.70
26.4
-1.97
79.1
67.4
83.0
GLU-Transformer Base
248M
-1.71
84.6
74.5
87.2
GLU-Transformer Large
748M
35.0
3.4
-1.56
84.2
74.3
86.2
GLU-Transformer XI.
2.85B
61.3
1.0
-1.49
87.6
82.9
89.4
LConv Timy
17M
1.20
31.2
-2.50
51.1
51.3
49.5
LConv Small
67M
3.80
12.8
2.10
71.8
59.9
64.7
I.Conv Base
210M
10.6
12.8
-1.95
73.8
63.6
70.3
LConv Large
741M
3.0
-1.6
76.8
65.6
76.3
LConv XI.
2.38
77.0
1.0
-1.75
73.3
64.1
72.9
DConv Tiny
22M
1.39
27.3
-2.66
51.1
48.9
30.2
DConv Small
96M
4.97
19.8
-2.88
68.6
57.4
64.3
DConv Base
324M
15.3
7.6
-1.90
72.9
60.1
63.7
DConv Large
1.2B
78.0
-1.82
70.8
58.5
58.2
MLP-Mixe. Small
67M
3.83
22.3
-2.15
65.4
55.1
58.7
MLP-Mixer Base
233M
12.4
10.7
-1.90
64.4
58.1
60.5
MLP-Mixer Large
739M
38.3
3.9
-1.73
52.2
47.8
60.9
Mixe-Mixer XI.
2.86B
48.3
1.2
-1.61
57.3
58.9
65.7

---

Under review as a conference paper at ICLR 2023
Table 2: Slope of fitted linear line for each model, when we compare FLOPs vs. upstream per- formance (F. U). FLOPs vs. downstream performance (F, D), parameter size vs. upstream per- formance U), parameter size vs. downstream performance (P, D), and finally upstream perfor- mance downstream performance (U, D).
Model
Q.F.U
Q.P.D
Q.P.U
QP.DD
QU,D
Transtormer
0.54
0.28
0,47
0.24
0,49
GLU-Trans.
0.49
0.24
0.42
0.22
0.46
LConv
0.32
0.13
0.29
0.48
Funnel
0.47
0.22
0.38
0.18
0.46
Switch
0,23
0.14
0.13
0.08
0.58
Universal
0.50
0.20
0.56
0.22
0.35
ALBERT
0.08
-0.12
0.13
-0.21
-1.67
Evolved
0.44
0.22
0.42
0.21
0.47
Performer
0.25
0.05
0.24
0.05
0.24
MoS-Trans.
0.43
0.21
0.43
0.20
0.47
MLP-Mixer
0.32
-0.03
0.26
0.65
-0.02
of parameters). In general, most values of a depict how well model scales. For example aF is plotting FLOPs against Upstream performance. The only exception is ap,D which is measure of upstream downstream performance. A high ou, value means that the transfer to the downstream tasks is better as model scales, Overall, the a value is a metric that represents how well a model performs relatively across all scales
Analysis of Slope for each Model In general, we find that the vanilla Transformer has the high- est values of a. Models such as Evolved Transformer, GLU-Transformer, MoS- Trs--rrasformrr and Funnel Transformer tend to have similar scaling properties the vanilla Transformer. The GLU- Transformer has similar and slightly worse scaling properties to the vanilla Transformer, even if observed to do better in absolute sense observe that there are models which are difficult to scale such as LConv, UT, MLP-Mixer and Per- former. This is even more evident on downstream task. We also note that ALBERT scales (trends) negatively (gets worse) as we scale the model up. On the other hand, the metric OU.D measures how the downstream performance scales with upstream performance. Overall, the Switch Transformer does the best on this metric where downstream performance scales well with upstream performance. Generally, models that make less changes to the main Transformer architecture (GLu-Transformer., MoS- MoS--ransformer tend to retain similar scaling behaviours and changing the inductive bias also significantly alters the scaling property of the model.
compute-regions. On the other hand, also
3.7 Do SCALING PROTOCOLS INFLUENCE MODEL ARCHITECTURES IN THE SAME WAY?
We are interested how different scaling protocols influence the model architectures. Figure 4 shows the effect of scaling depth of four model architectures Moo-Traasformer. Transformer, Evolved Transformer and LConv). Figure 5 shows the effect scaling width on the same four architectures. Firstly, upstream (negative log perplexity) curves, note that while different architectures have a distinct difference absolute performance, the scaling trend remains quite sim- ilar. On downstream, depth scaling (Figure seems act equally most architectures with the exception of LConv. Meanwhile, for width scaling, it seems that Evolved Transformers scale slightly better when applying width-scalle bidth-scaling. also interesting note that depth-scaling has much more substantial impact on downstream scaling as opposed to sadth-scaling.
3.8 EPILOGUE AND CONCLUSION
In this paper, we conducted extensive experiments, pretraining and finetuning of to 100 models ranging from 10 well-established Transformer and non- Transformer architectures. We showed that different model architectures can have different scaling behaviours and models performing well in one compute region (or model size) may not do identically well in another compute region.
Thii version of ALBERT shares parameters across encoder and decoder which may partially explain why had hard time scaling up.
8

---

Under review as a conference paper at ICLR 2023
BI-E Mo
Hirt Surfind NL.34
MUSEE
Sindil NLLL
MILLS
Accurco
NLLS
Aic
Done N12
2.710
44613
14h/11
30w13
FLOPS
(a) Upstream Neg. Log-PPL.
(b) Downstream Accuracy.
Figure 4: Scaling depth
H1X
HM12
7725
4..0
LOat P112K
LCam ITC
1Con Lann ITDonn
1840
CIRC LAM0 A3H12
FLOPS
(a) Upstream Neg. Log-PPL.
(b) Downstream Accuracy.
Figure 5: Scaling width of FFN
We also showed that model architectures may do well on upstream perplexity but fail to transfer to downstream tasks. Hence, practitioners should be cautious about developing architectures that not only scale well with respect to the upstream perplexity, but also based on downstream performance. While certainly do not expect researchers to always report model performance across all scales (especially large-scale), we believe that is good to keep in mind that architectures can perform quite differently different compute regions. Hence, this might be good dimension consider when designing new inductive biases. As such, performing evaluation at a certain compute region may be insufficient to capture the full picture. It also good consider if different inductive biases will result in different extends of emergent capabilities (Wei et al. 2022 Abnar et al. 2020).
We also showed that different model architectures may react differently to different scaling proto- cols, which further expands on the narrative that comparing and benchmarking these models can be very challenging (Dehghani et al.) 2021b). When it comes to scaling large models, this paper shows that novel inductive biases can be indeed quite risky which might explain why most state-of-the-ar large language models Rae et al. 20221: Chowdhery et al. (002)): Tay et al. (2022) are based on relatively vanilla architectures. Our advice is be cautious when staking expensive Transformer architecture that drastically modifies the attention mechanism (e.g.. Mixers and Per- formers generally high risk options that not every practitioner or researcher would require models that are able to scale to billion of parameters. In that case, inductive biases that are tailored to small or low compute will be sufficient.
in experiment results). Finally, acknowledge

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samira Abnar, Mostafa Dehghami, and Willem Zuidema. Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006. 00555, 2020.
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:21 10.02095, 2021.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2002.06701, 2021.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint urXiv:2204.02331 2022.
Alexis Conncau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. http3//11222..ammmmii... 7/ 7/www aclweb. o anthologg/2220 acl-aallaae main. 747
URL https:
Zihang Dai, Guokun Lai, Yiming Yang, and Quoe V Le. Funnel-transforne Filtering out sequen- tial redundancy for efficient language processing. arXiv preprint wxXiv:2066003366 2020.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learn- ing Research, pp. 933 941. PMLR, 06-11 Aug 2017. URL httt:://prcceeee:: press/v70/dauppl hat.naa.hmmmmmm
mlr.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.038/9 2018
Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency mis- nomer. arXiv preprint arXhv:2110.1288/ 2021a.
Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2/07.07000 2021
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 2021.
Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. Larger-Scale Trans- formers for Multilingual Masked Language Modeling. arXiv e-prints, art. arXiv:2105.00572. May 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 2000::0010..4701,, 2020.
Danny Hermandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. 2020.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:/888.000 1808.8888..666 2018.
10

---

Under review as a conference paper at ICLR 2023
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXi:: 2019.
Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Kar- ishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, et al. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11972 2021.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11422 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 1606.05250, 2016.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool,
Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Nesh-tensorfloww Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pp. 10414- 10423, 2018.
David So, Quoe Le, and Chen Liang. The evolved transformer. In International Conference Machine Learning, pp. 5877-5886. PMLR, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. arXiv preprint arXhv:1409.3213: 2014.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint arXiv:2009.067:: 2020.
Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. Are pre-trained convolutions better than pre-trained transformers? arXiv preprint arXiv:2105.033322, 2021a.
Yi Tay, Mostafa Dehghami, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109 10686, 2021b.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint N.X1:2205500131. 2022.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: wwX.vv:800..07460 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for purpose language understanding systems. arXiv preprint arXiv:1905.0mm 1905.0057, 2019.

---

Under review as a conference paper at ICLR 2023
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language 2022.
models. arXiv preprint
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint 1rXiiv1901.100 2019.
Zhilin Yang. Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: A high-rank mnn language model. arXiv preprint arXiv: 2017.
12

---

Under review as a conference paper at ICLR 2023
4 APPENDIX
4.1 SCALING DETAILS FOR INDIVIDUAL MODELS
For most models, it was reasonable follow the uniform scaling method in the main T5 sizes. At each size, the hyperparameters are as follows:
Model
N₂
dyf
dmodei
dge
N
#Params
Tiny
4/4
1024
256
32
4
16M
Small
6/6
2048
512
32
8
60M
Base
12/12
3072
768
64
12
220M
Large
24/24
4096
1024
64
16
738M
24/24
16384
1024
128
32
3B
Table 3: Table of model configurations. Nr is the number of layers, dss is the size of the MIP, dmodel is the hidden size of the model. deo the size of each key-value vector. NH is the number of heads.
Scaling for Switch Transformer For Switch Transformers, we use the following scaling:
Model
Nr.
def
dmodel
day
Nu
Ng
#Params
Tiny
1024
512
12
32
173M
Small
6
2048
512
12
460M
Base
12
3072
768
64
12
32
2B
Large
24
3072
768
64
12
32
8B
XL
48
3072
768
64
12
128
30B
Table 4: Scaling for Switch Transformer, NE is the number of experts.
Scaling for Universal Transformer Scaling UTs are generally difficult as described in the main text. There were two main considerations for scaling UTs.. Initially we tried scaling the number of recurrent operations. However, we found that even with an increase FLOPS, this does not lead to improved performance. Overall, the UT model might be pretty slow and therefore model with the same hparams as vanilla XL might be infeasible to run. Hence, we explored increasing the width of the MLPs to 32K to see if UTs would scale in this manner.
Model
NR
dyf
dmodet
dec
Nи
#Params
UT Tiny
3/3
1024
128
32
8
11M
UT Small
3/3
2048
512
32
8
52M
UT Base
3/3
3072
12
127M
UT Large
3/3
32768
1024
64
16
283M
Table Table of model configurations. Nn the number of recurrent operations, dyf is the size of the MLP, dmodet is the hidden size of the model. deu is the size of each key-value vector. NH is the number heads.
13

---

Under review as a conference paper at ICLR 2023
A59
(a) FLOPS
(b) Number of Parameters
(c) Throughput
Figure 6: Quality-cos trade of for the upstream Negative Log- Log-Perplexity of vanilla Transformer compared to other models, with respect to FLOPs, number of parameters, and throughput.
14

---

Under review as a conference paper at ICLR 2023
(a) FLOPs
(b) Number of Parameters
Throughput
Figure 7: Quality-com trade of for the downstream SuperGlue Accuracy of vanilla Transformer compared to other models, with respect to FLOPs, number of parameters, and throughput.
15

---

Under review as a conference paper at ICLR 2023
(a) FLOPs
(b) Number of Parameters
(c) Throughput
Figure 8: Quality-cost trade of for the downstream Glue Accuracy vanilla Transformer compared to other models, with respect to FLOPs, number of parameters, and throughput.
16

---

Under review as a conference paper at ICLR 2023
100
(a) FLOPs
(b) Number of Parameters
(c) Throughput
Figure 9: Quality-cost trade of for the downstream Squad Accuracy of vanilla Transformer comparredrrr to other models, with respect to FLOPs, number of parameters, and throughput.
17


Reviews:
[{"review_id": "meY-CGN9WoL", "reviewer": "Reviewer_1jM9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts a systematic study of scaling behavior of different model architectures. \n", "summary_of_the_review": "This article systematically studies model architecture’s influence on scaling law. Extensive experiments are carried out. However, All of the conclusions drawn from experiments are pretty under expectation and there are no surprises. Majority of findings have been seen before. In addition, some SOTA model architectures are not included in this paper.", "strength_and_weaknesses": "*Strength:*\n\nExtensive experiments have been done in this paper.\n\n*Weaknesses:*\n\n(1). There is a lack of innovation. All of the conclusions drawn from experiments are pretty under expectation and there are no surprises. Majority of findings have been seen before.\n\n(2). This paper mainly compares a series of models published by google such as Evolved Transformers, Universal Transformers, Switch Transformers, Performer, Funnel Transformer, Albert, MLP-mixer, and ignores some other state-of-the-art models. For example, although the Switch Transformer was published first, the subsequent models such as BASE layers [1], Hash layers [2] are much better than the Switch Transformer.  As a systematic study paper, the author should choose SOTA models for comparison. \n\n(3). One of the conclusions from this paper is “the MLP-Mixer is hard to scale”.  I doubt whether this can explain the structure of MLP hard to scale, or because the author only uses mlp-mixer at input encoder part. Actually, MLP structure can also be applied to decoder-only models such as [3]. \n\n(4). I wonder why this paper does not compare encoder-decoder model architecture v.s. decoder-only model. The encoder-decoder model seems to be rarely used for large language models. The largest T5 is only 11B, however, decoder only models can easily pass 100B parameters. In addition, I wonder how to scale up encoder-decoder models (increase depth or width? How to balance the encoder part and decoder part). This article systematically compares model structures, but ignores this one.\n\n*References:*\n\n[1]. Lewis, Mike, et al. \"Base layers: Simplifying training of large, sparse models.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2]. Roller, Stephen, Sainbayar Sukhbaatar, and Jason Weston. \"Hash layers for large sparse models.\" Advances in Neural Information Processing Systems 34 (2021): 17555-17566.\n\n[3]. Yu, Ping, et al. \"Efficient Language Modeling with Sparse all-MLP.\" arXiv preprint arXiv:2203.06850 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "grYtPuvaVWg", "reviewer": "Reviewer_4um9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors examine how a wide variety of transformer architecture choices (and a few non-transformer architectures such as MLP-Mixers and dynamic convolutions) scale over a wide range of model sizes. Their models are implemented in a sequence-to-sequence manner, and evaluated with pretraining perplexity and SuperGlue accuracy. They make the following observations: the vanilla transformer has the best scaling behavior even though its absolute performance in each compute region is not the best; performance does not scale consistently for a given architecture; pretraining quality (negative log-perplexity) does not always correlate with downstream quality (SuperGlue and SQuAD accuracy); some architectures scale negatively.", "summary_of_the_review": "Well-motivated and will be of useful to the field if the methodological and editorial shortcomings are addressed.", "strength_and_weaknesses": "**Strengths**\n\nI agree with the motivation that new methods are not always evaluated across a sufficiently broad range of resource scales.\n\nThese sorts of studies are resource-intensive and not particularly exciting to conduct, but provide high-level perspective that can be invaluable to the field.\n\n**Weaknesses**\n\nThe authors point out the shortcomings of different measures of model “scale” (parameters and flops). In the end, what most people care about is the resources required to use these models. The authors provide “speed” (steps/sec) values for each model in their main results table, but do not spend much time discussing it in the text. I think speed should be more strongly emphasized.\n\nInsufficient hyperparameter information (see comment in below section)\n\nNumerous works have shown that the optimal amount of training data scales proportionally to the size of the model being trained. But you train all your models with the same amount of data. Is it possible that the scaling results you observe for larger models are confounded by their being trained on an insufficient amount of data?\n\nPerhaps I’m misunderstanding something about the scaling quality metric, but it seems like it may be a poor metric if it is showing the vanilla transformer as a superior choice to the GLU-Transformer. The GLU-Transformer is superior to the vanilla Transformer for every model size when evaluated w/ perplexity (Figure 2) and every model size except one when evaluated on SuperGlue (Figure 3), yet the vanilla transformer “scales better” than the GLU-Transformer and is repeatedly emphasized as the best model choice. Similarly, the Switch Transformer has superior perplexity and similar SuperGlue accuracy to the vanilla transformer at all but one model size.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LMihT-QYgNc", "reviewer": "Reviewer_7UyJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper conducts extensive experiments to study the scaling behavior of different neural network models on language problems. The experiment results show: \n1. Different models architectures scale differently when the sizes of the models are changed,\n2. At different compute regions, the model that perform best is different.\n3. Scaling models in different ways, such as scaling by width or depth, shows different scaling behaviors.\n\n", "summary_of_the_review": "This work conducts extensive experiments, but lacks deeper thinking and analysis of the mechanism that causes the empirical results. ", "strength_and_weaknesses": "Strength:\n\n1. The experiments are comprehensive. Many model architectures are tested in a large range of model scales. \n2. Both upstream and downstream performances are considered and studied.\n3. Careful attention is paid on how to scale the models.\n\nWeaknesses:\n\n1. The paper presents a lot of empirical results, but does not summarize new patterns from the observations.\n2. The paper does not discuss the mechanism behind the results, such as why different models have different scaling behaviors. \n2. No new insight is provided on how neural networks work in different scales. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EUCAsMe1DX", "reviewer": "Reviewer_6T2s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the scaling laws of different sequence model architectures, showing that different Transformer and non-Transformer models scale differently with model size. This study is carried out throughout, as it compares many mainstream models with a wide range of model sizes for both upstream and downstream tasks.", "summary_of_the_review": "The paper is well motivated, clearly written, and technically sound. However, I would rate the paper slightly above the acceptance threshold due to the lack of new guidance/practice on scaling models.", "strength_and_weaknesses": "Strength: The evaluation in this paper is extensive as it examines a wide range of inductive bias/different model architectures from tiny to large model scales. In addition, this paper also provides results on both upstream and downstream tasks and points out the potential discrepancy in the performance of upstream and downstream tasks.\n\nWeaknesses:\n1. Why use FLOPS instead of speed to show scaling laws? Although FLOPS is a good indicator of speed, FLOPS does not really reflect how well the model can scale on real-world hardware.\n2. It is clear to me that some architectural changes might not generalize well to large-scale models or downstream tasks. However, it remains unclear to me how to leverage lessons learned from this study to help designing new architectures. For example, Chinchilla [1] suggests using more data to achieve compute-optimal language models.\n3. This paper also lacks enough studies on scaling protocols. Although the paper touches slightly on the impact of scaling protocols for different architectures, I wonder if it is possible to get a general scaling rule for sequence models.\n\n[1] Training Compute-Optimal Large Language Models, arxiv'22\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject