Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper conducts a systematic study of scaling behavior of different model architectures. 

strength_and_weaknesses: *Strength:*

Extensive experiments have been done in this paper.

*Weaknesses:*

(1). There is a lack of innovation. All of the conclusions drawn from experiments are pretty under expectation and there are no surprises. Majority of findings have been seen before.

(2). This paper mainly compares a series of models published by google such as Evolved Transformers, Universal Transformers, Switch Transformers, Performer, Funnel Transformer, Albert, MLP-mixer, and ignores some other state-of-the-art models. For example, although the Switch Transformer was published first, the subsequent models such as BASE layers [1], Hash layers [2] are much better than the Switch Transformer.  As a systematic study paper, the author should choose SOTA models for comparison. 

(3). One of the conclusions from this paper is “the MLP-Mixer is hard to scale”.  I doubt whether this can explain the structure of MLP hard to scale, or because the author only uses mlp-mixer at input encoder part. Actually, MLP structure can also be applied to decoder-only models such as [3]. 

(4). I wonder why this paper does not compare encoder-decoder model architecture v.s. decoder-only model. The encoder-decoder model seems to be rarely used for large language models. The largest T5 is only 11B, however, decoder only models can easily pass 100B parameters. In addition, I wonder how to scale up encoder-decoder models (increase depth or width? How to balance the encoder part and decoder part). This article systematically compares model structures, but ignores this one.

*References:*

[1]. Lewis, Mike, et al. "Base layers: Simplifying training of large, sparse models." International Conference on Machine Learning. PMLR, 2021.

[2]. Roller, Stephen, Sainbayar Sukhbaatar, and Jason Weston. "Hash layers for large sparse models." Advances in Neural Information Processing Systems 34 (2021): 17555-17566.

[3]. Yu, Ping, et al. "Efficient Language Modeling with Sparse all-MLP." arXiv preprint arXiv:2203.06850 (2022).
clarity,_quality,_novelty_and_reproducibility: Clarity:

This paper is well written and easy to understand.

Novelty:

As stated in weaknesses, this paper lacks novelty. In addition, instead of trying to compare SOTA model architectures, most models in this paper come from google. I understand that this will greatly reduce the workload, but as an article of a systematic study, more SOTA model structures should be selected.

summary_of_the_review: This article systematically studies model architecture’s influence on scaling law. Extensive experiments are carried out. However, All of the conclusions drawn from experiments are pretty under expectation and there are no surprises. Majority of findings have been seen before. In addition, some SOTA model architectures are not included in this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the scaling laws of different sequence model architectures, showing that different Transformer and non-Transformer models scale differently with model size. This study is carried out throughout, as it compares many mainstream models with a wide range of model sizes for both upstream and downstream tasks.
strength_and_weaknesses: Strength: The evaluation in this paper is extensive as it examines a wide range of inductive bias/different model architectures from tiny to large model scales. In addition, this paper also provides results on both upstream and downstream tasks and points out the potential discrepancy in the performance of upstream and downstream tasks.

Weaknesses:
1. Why use FLOPS instead of speed to show scaling laws? Although FLOPS is a good indicator of speed, FLOPS does not really reflect how well the model can scale on real-world hardware.
2. It is clear to me that some architectural changes might not generalize well to large-scale models or downstream tasks. However, it remains unclear to me how to leverage lessons learned from this study to help designing new architectures. For example, Chinchilla [1] suggests using more data to achieve compute-optimal language models.
3. This paper also lacks enough studies on scaling protocols. Although the paper touches slightly on the impact of scaling protocols for different architectures, I wonder if it is possible to get a general scaling rule for sequence models.

[1] Training Compute-Optimal Large Language Models, arxiv'22


clarity,_quality,_novelty_and_reproducibility: This paper is well written and easy to understand. The quality of the work is good, as the approaches to studying scaling laws, the evaluation setup, and the derived conclusions all seem reasonable to me. The novelty of the paper is moderate, as it does not really provide new practices for designing new architectures and scaling up models.
summary_of_the_review: The paper is well motivated, clearly written, and technically sound. However, I would rate the paper slightly above the acceptance threshold due to the lack of new guidance/practice on scaling models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper conducts extensive experiments to study the scaling behavior of different neural network models on language problems. The experiment results show: 
1. Different models architectures scale differently when the sizes of the models are changed,
2. At different compute regions, the model that perform best is different.
3. Scaling models in different ways, such as scaling by width or depth, shows different scaling behaviors.


strength_and_weaknesses: Strength:

1. The experiments are comprehensive. Many model architectures are tested in a large range of model scales. 
2. Both upstream and downstream performances are considered and studied.
3. Careful attention is paid on how to scale the models.

Weaknesses:

1. The paper presents a lot of empirical results, but does not summarize new patterns from the observations.
2. The paper does not discuss the mechanism behind the results, such as why different models have different scaling behaviors. 
2. No new insight is provided on how neural networks work in different scales. 


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written.

Quality: The empirical study is brought out with high quality, with extensive experiments on many different settings.

Novelty: Empirically, experiments on the scaling behavior of neural networks with such a large coverage have not been done before. Technically, no new message is given. 

Reproducibility: I did not check the reproducibility of the experiments. 
summary_of_the_review: This work conducts extensive experiments, but lacks deeper thinking and analysis of the mechanism that causes the empirical results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors examine how a wide variety of transformer architecture choices (and a few non-transformer architectures such as MLP-Mixers and dynamic convolutions) scale over a wide range of model sizes. Their models are implemented in a sequence-to-sequence manner, and evaluated with pretraining perplexity and SuperGlue accuracy. They make the following observations: the vanilla transformer has the best scaling behavior even though its absolute performance in each compute region is not the best; performance does not scale consistently for a given architecture; pretraining quality (negative log-perplexity) does not always correlate with downstream quality (SuperGlue and SQuAD accuracy); some architectures scale negatively.
strength_and_weaknesses: **Strengths**

I agree with the motivation that new methods are not always evaluated across a sufficiently broad range of resource scales.

These sorts of studies are resource-intensive and not particularly exciting to conduct, but provide high-level perspective that can be invaluable to the field.

**Weaknesses**

The authors point out the shortcomings of different measures of model “scale” (parameters and flops). In the end, what most people care about is the resources required to use these models. The authors provide “speed” (steps/sec) values for each model in their main results table, but do not spend much time discussing it in the text. I think speed should be more strongly emphasized.

Insufficient hyperparameter information (see comment in below section)

Numerous works have shown that the optimal amount of training data scales proportionally to the size of the model being trained. But you train all your models with the same amount of data. Is it possible that the scaling results you observe for larger models are confounded by their being trained on an insufficient amount of data?

Perhaps I’m misunderstanding something about the scaling quality metric, but it seems like it may be a poor metric if it is showing the vanilla transformer as a superior choice to the GLU-Transformer. The GLU-Transformer is superior to the vanilla Transformer for every model size when evaluated w/ perplexity (Figure 2) and every model size except one when evaluated on SuperGlue (Figure 3), yet the vanilla transformer “scales better” than the GLU-Transformer and is repeatedly emphasized as the best model choice. Similarly, the Switch Transformer has superior perplexity and similar SuperGlue accuracy to the vanilla transformer at all but one model size.

clarity,_quality,_novelty_and_reproducibility: Datapoint labels (model names) are illegible in Figure 1. Please increase the text size if possible.

In the Related Work section, you say “​​Fedus et al. (2021) scaled a sparse model based on Mixture of Experts (MoE) models up to trillion parameters.” but you don’t report their findings.

There is insufficient information about hyperparameters. You mention training for 2^19 steps on C4, but do not mention batch size or LR/LR schedule, for example.

Needs another round of proofreading: “Improvements at a a specific scale”; “Details on scaling details”; many of the section titles are followed immediately by a single sentence that restates the same information that’s provided by the section title, e.g. “Do All Models Scale the Same Way” is followed by a single sentence “This section investigates if all model architectures scale in the same way”.

summary_of_the_review: Well-motivated and will be of useful to the field if the methodological and editorial shortcomings are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers recommend rejecting the paper (5-3-5) or are unsure and place the paper in the borderline category (6). 
All reviewers agree that the paper is well-written and easy to follow. They underline and appreciate the extensive experimental investigation of the scaling behaviour of transformer models. 
However, they also raise several concerns, mostly related to the weak discussion of the results and their implications. For example, the paper does not provide new guidance or insights on scaling models, it does not summarize new patterns from the empirical observations, nor does it try to discuss the mechanisms behind the results.
Unfortunately, none of these concerns has been discussed during the author-reviewer discussion phase. The authors did not reply to the reviewers' comments, nor did they provide any clarifications or additional experiments.
justification_for_why_not_higher_score: Fair concerns and elements of discussion were raised by the reviewers, but none of them have been addressed in a rebuttal.
Decision: Reject