Scaling Laws for Neural Machine Translation
Authors: Behrooz Ghorbani
Orhan Firat
Markus Freitag
Ankur Bapna
Maxim Krikun
Xavier Garcia
Ciprian Chelba
Colin Cherry
Venue: ICLR 2022 Spotlight (2022)
Keywords: Scaling Laws
Neural Machine Translation
NMT
Model Scaling

Abstract: We present an empirical study of scaling properties of encoder-decoder Transformer models used in neural machine translation (NMT). We show that cross-entropy loss as a function of model size follows a certain scaling law. Specifically (i) We propose a formula which describes the scaling behavior of cross-entropy loss as a bivariate function of encoder and decoder size, and show that it gives accurate predictions under a variety of scaling approaches and languages; we show that the total number of parameters alone is not sufficient for such purposes. (ii) We observe different power law exponents when scaling the decoder vs scaling the encoder, and provide recommendations for optimal allocation of encoder/decoder capacity based on this observation. (iii) We also report that the scaling behavior of the model is acutely influenced by composition bias of the train/test sets, which we define as any deviation from naturally generated text (either via machine generated or human translated text). We observe that natural text on the target side enjoys scaling, which manifests as successful reduction of the cross-entropy loss. (iv) Finally, we investigate the relationship between the cross-entropy loss and the quality of the generated translations. We find two different behaviors, depending on the nature of the test data. For test sets which were originally translated from target language to source language, both loss and BLEU score improve as model size increases. In contrast, for test sets originally translated from source language to target language, the loss improves, but the BLEU score stops improving after a certain threshold. We release generated text from all models used in this study.

Published as a conference paper at ICLR 2022
SCALING LAWS FOR NEURAL MACHINE TRANSLA- TION
Behrooz Ghorbani; Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba & Colin Cherry Google Al
ABSTRACT
We present an empirical study of the scaling properties of encoder-decoder Trans- former models used in neural machine translation (NMT). We show that cross- entropy loss as function of model size follows a certain scaling law. Specifically (i) We propose a formula which describes the scaling behavior of cross-entropy loss as a bivariate function of encoder and decoder size, and show that it gives accurate predictions under a variety of scaling approaches and languages; that the total number of parameters alone is not sufficient for such purposes. (ii) We observe different power law exponents when scaling the decoder vs the encoder, and provide recommendations for optimal allocation of encoder/decoder capacity based this observation. (iii We also report that the scaling behavior of the model is acutely influenced by composition bias of the train/test sets, which we define as any deviation from naturally generated text (either via machine generated human translated text). We observe that natural text on the target side enjoys scaling, which manifests as successful reduction of the cross-entropy loss. (iv) Finally, we investigate the relationship between the model size, cross-entropy loss and the quality of the generated translations. We find two different behaviors, depending on the nature of the test data. For test sets which were originally translated from target language to source language, both loss and BLEU score improve as model size increases. In contrast, for test sets originally translated from source language to target language, the loss improves, but the BLEU score stops improving after a certain threshold. We release generated text from all models used in this study.
show
INTRODUCTION
Scaling properties of neural networks have long been an intriguing topic of study (Ahmad & Tesauro 1988; Amari al. 1992). Along with the practical success of modern neural networks at scale,
theoretical understanding of the factors governing the quality and training dynamics of large neural
networks has also being developing (Advani et al. 22200] Rosenfeld et al.) 2019 Geiger et al. 20200]
Ghorbani et al. 2021 Cho 2020 Hutter 2021 Bahri et al. [021
et al. 2021). In particular,
scaling model sizes, datasets and the total computation budget has been identified as a reliable approach to improve generalization performance on several machine learning tasks. For many of
these tasks the scaling behavior of neural networks is highly predictable; model fit or test loss can be described precisely
function of its number of parameters (Hestness et al. 2017 Kaplan let al. 2020 Henighan et al. 2020 Hernandez et al. 2021]] Rosenfeld et al. 2019). Neural machine iranslation (NMT) has long enjoyed the benefits of scaling (Huang al. 2019: Arivazhagan et al.)
2019 Lepikhin et al.] 2021). but studies investigating the scaling behavior of NMT models are
missing. We present a large-scale systematic study of scaling laws for encoder-decoder Transformer models applied to NMT (Vaswani et al. 2017).
We start with highlighting the major differences between decoder-only language models, where the majority of the previous work has focused, and encoder-decoder (conditional) language models applied NMT. The two differ along a few crucial dimensions. The first difference results from the very nature of the separate architectures being used, i.e. decoder-only vs encoder-decoder. The presence of separate architectural components complicates the study of scaling properties due the
"Correspondence ghorbant @googl e. com.

---

Published as a conference paper at ICLR 2022
increased degree of freedom. Second, contrary to language modeling, the task of machine translation is conditional: the task is predictive rather than fully generative. Furthermore, this prediction task is right answer for a given source, and translations
ambiguous: there is depending on the translator's ss incentives. This manifests itself as different scaling benefits for different test sets. To take extreme example, test set translated by someone who writes nearly word- for-word translations may benefit less from model scaling than one translated by someone who considers each translation a work of art. In this work, these differences in difficulty coincide with the translation direction of the test set; that is, whether the source was translated into the target chource-original or vice versa (target-original, Source-original data has translated text on the target side, which contains several artifacts of "translationese" that distinguish it from text originally written in that language, often lacking the diversity and complexity of "natural" text (Koppel Ordan 201T), while target-original data requires the prediction of more complex natural text on the target side. Finally, unlike language models, NMT is evaluated on metrics that quantify generation quality against reference translations (for eg. BLEU) (Papineni et al. 2002) instead of evaluating model fit (perplexity) on an evaluation set.
vary substantially
In this paper, we aim to provide empirical answers the following research questions:
1. Does the encoder-decoder architecture for NMT share the same scaling law function as language models? Contrary to previous work on LM, we show that a univariate law depending on the total number of parameters in the network does not adequately describe the scaling behavior of NMT models. Our scaling laws parameterize the cross entropy loss bivariate function of the number of encoder parameters and the number of decoder parameters as separate variables. Our results indicate that the scaling behavior is largely determined by the total capacity of the model, and the capacity allocation between the encoder and the decoder.
2. How does the naturalness of source/target side data affect scaling behavior? We study the effect of naturalness of the source and the target text, both for training and evaluation. When evaluating with target side natural text, scaling the model capacity continues improving model quality throughout our range of measurements. On the other hand, improvements on cross-entropy saturate (or reaches the irreducible error region) on source side natural evaluation sets even for moderately- sized models.
3.
Do scaling improvements in cross-entropy translate into corresponding improvements in generation quality? Finally we study the relationship between generation quality and cross-entropy and how their correlation changes as we: (i) Scale different components of the model (encoder vs decoder) and (ii) Evaluate on source-natural or target-natural test sets.
Our results on multiple language pairs and training/test data compositions validate that model scaling predictably improves the cross-entropy on validation data. However, our findings also raise several questions regarding the effect of naturalness of training and evaluation text and how cross-entropy eventually relates with generation quality for auto-regressive generative models.
2
EFFECT OF SCALING ON CROSS-ENTRES
2.1
EXPERIMENTAL SETTING
Model Architectures and Training We train series of pre-layer norm Transformer networks with varying sizes (Xiong et al. 2020). Models are trained with per-token cross-entropy loss and Adafactor optimizer (Shazeer & Stern 2018). All models are trained with a fixed batch-size of 500k tokens and dropout rate of 0.1 for residuals, feed-forward activations and attention. All models trained near convergence for 500k training steps. Details of the model haper-parameters described in Appendix C
Model Scaling Transformer architecture consists of Transformer Blocks: a cascade of self-attention, cross- coss-attention and fred-forwart layers, each having multiple adjustable
(e.g.
A few weeks before the publication of this manuscript on Arxiv, Arxiv,Gordon et al. 2022 appeared on Open- Review. While both papers study scaling laws for NMT, our studies focus different parameter regimes (393K-56M vs 100M-4B). See Appendix A for comparison of the results.
2

---

Published as a conference paper at ICLR 2022
model-dimension, number of attention heads, attention projection dimension etc.). Considering the combinatorial expansion of the search space for scaling each one, in this study we choose to vary only the total number of Transformer Blocks, while keeping the internal hyper-parameterr intact across different scales. In other words, we scale the depth of the Transformers while keeping width and We use GPipe pipeline parallelism for our experiments (Huang et al. 2019). [
other variables fixed.
In an encoder-decoder Transformer architecture for NMT, depth scaling can naturally be implemented by varying encoder-decoder blocks independently or symmetrically. Hence, we examine the change in the cross-entropy loss as the number of parameters increase with three depth scaling approaches:
Encoder Scaling: vary encoder depth (2 to 64) while decoder depth is fixed (6 layers).
Decoder Scaling: vary decoder depth (2 to 64) while encoder depth is fixed (6 layers).
Symmetric Scaling: increasing decoder and encoder layers together (from 2 to 64), i.e. the number of Transformer Blocks in the encoder and decoder being equal.
For all experiments, configuration of the individual layers is unchanged: the model dimension, width of the feed-forward layer, and number of attention heads are fixed respectively 1024, 8192, and 16. Fl Each encoder layer adds approximately 20M parameters to the model while each decoder layer adds around 25M parameters. We train 200 such models which scale the model size by approximately a factor of 50 (from 80M parameters to 4B parameters). Following the convention, we do not count the parameters in the embedding and softmax layers towards the model size. We focus our study on large-scale models: our smallest models require 200 TPUv3 days to train to convergence while our largest models require 2700 TPUv3 days of training.
Language Pairs We report results on four language pairs: English GGrrma, German- English->Chinese *Chinese, and Chinese->Englic We use in-house web-crawled training datasets with around 2.2 billion sentence pairs (approximately 55 billion tokens) for English++German and 781 million sentence pairs for English++Chines..
Evaluation Sets We use a variety of test sets for evaluation covering different domains: (i) Web- Domain (ii) News-Domain (iii) Wikipedia (iv) Patents. The news-domain test sets come from the WMT2019 (Barrault al. 2019) evaluation campaign (newstest2019) for all language pairs. The other test sets are internal test sets representing the different domains, ranging from 500 to 5000 sentence pairs. For each domain, we randomly sample sentences in the source language and use professional translators to generate a reference translation in the target language. Throughout the paper, we will refer this type of test sets as source-original as the source sentences have been crawled from the web while the reference translations are added later. For most of the domains, we also have a target-original counterpart which is generated in the opposite direction: Sentences are crawled in the target language and human translated into the source language. Earlier work (Freitag et al. 2019 2020 Grahamm et al. 2020) showed that it is important to differentiate between the two different kinds of test sets as the style of natural sentences and human (or machine) translations (translationesee) is quite different. Cross-entropy loss is evaluated on the different test sets during training. To reduce the variation caused by the parameter fluctuations the end of the training, we present the median loss over the last 50k steps of the training as the final loss.
2.2 RESULTS
Figure [ shows the empirical evolution of the test loss on the Web-Domain test sets for encoder and decoder scaling for English-+German. To compare the empirical results with the scaling laws present in the literature for decoder only models (Kaplan et al.] 2020 Henighan et al., 2020), we have fitted a power law of the form f.(N)= = N + T.oe to the data. Here, N is the total numher of parameters outside of embedding / softmax layers and {a, p. Lo are fitted parameters. As Figure 1 suggests, scaling the encoder has different effects on the test loss compared to scaling the decoder. As such, simple power-laws that only consider the total parameter count, fail capture the correct scaling behavior of the model.
II Appendix H we examine the role of scaling approach in our results. See Section 2 for more details.
3 A complete description of the model architecture is provided in Appendix *Details of the curve fitting procedure are presented in Appendix |
3

---

Published as a conference paper at ICLR 2022
Web Domain orig), RA; 78.8, 4. 0.028
Web Domain orig), RP: 96.8, -l 0.021
1.100
1.050
Encoder Scaling
Encoder Scaling
1.075
Decoder Scaling
1.025
Decoder Scaling
1.050
Fitted Power Law
1.000
Fitted Power Low
Text Loss
1.025
0.975
1.000
0.950
0.975
0.925
0.950
0.900
0.075
0.925
0.850
0.4
1.0
0.8
Total Number of Porameters (Excluding ng Embeddings)
Total Number of Parameters (Excluding Emboddings)
Figure 1: Evolution of the test loss as function of the total model parameters for English->German German. Scaling the encoder has different effects compared to scaling the decoder. As such, traditional scaling laws are unable to capture the correct scaling behavior. R-squared (100 x explained variance and maximum absolute deviation (l |loc) are reported for each fit.
total variance
Proposed Scaling Law To tackle this issue, we present a new scaling law that reflects the encoder- decoder nature of the architecture as well as the bilingual format of the data. Let Ne and Na be the number of non-embedding parameters in the encoder and the decoder respectively. Then, our proposed scaling law has the form
L(N., Na) aN Pe N + Loc
(1)
where (a. Pe: Pd: ] are test set specific (fitted) parameters. In this formulation, a corresponds to the maximum loss reduction that one can hope from scaling, while Pe and pa are the scaling exponents for encoder and decoder respectively. Los corresponds to the irreducible loss of the data.
In Figure 2 we fit our scaling law to the empirical loss values achieved by encoder/decoder scaling models on Web-Domain test sets. The results suggest that our scaling law almost fully captures the variation in the data (R²2 > 99%). In particular, in contrast to traditional scaling laws, our proposed formulation is able to capture the diffcrent scaling behaviors of encoder and decoder scaled models.
While these improved fits are encouraging, to truly show that a candidate scaling law captures the model scaling behavior in NMT, it is necessary to verify the out-of-sample prediction power of the law. We provide two such verifications; we compare the predictions of the scaling law fitted on encoder decoder scaling models with the empirical loss values from (i) symmetrically scaled models. (ii) models randomly sampled from the 2D grid of all potential encoder-decoder configurations Symmetrically scaled models are chosen for this verification as they are widely used by the NL.P community (Raffel al. 2020). Randomly sampled models are chosen ensure that our results are not biased towards any particular model class.
Figures present the out-of-sample results. As the plots suggest, the predictions of the scaling law match the empirical results with remarkable accuracy. These results suggest that Eq. (1) is able to capture the model scaling behavior across different scaling approaches and model sizes.
To ensure that our results generalize across different language pairs, we examine the fit of our scaling law on encoder / decoder scaling models trained on German Engllshh (De->En). Chinese-to-Enge tnnEnss (Zh->En), and Aaglish-to-Chinese on in-domain held-out data. Similar to the En->De case, our proposed functional form able to closely capture the scaling behavior of these models.
(En->Zh) translation tasks. Figure presents the results evaluated
To keep the scope of our experiments manageable, we focus on depth scaling in our analysis. In Appendix H we ran large-scale experiments to see if other scaling approaches show significant deviations from the trends identified here. For large-scale models, which are the focus of this study, observe that different scaling approaches yield very similar results: the final achieved test loss across different scaling approaches shows just 1 2% variation. We observed that for small models (< 100M parameters). this variation tends to grow, suggesting that model shape plays a key role in the performance of small models. We leave further analysis of this phenomenon to future research.
SSee Appendix F for the full deseription of the models.
4

---

Published as a conference paper at ICLR 2022
Web Domain orig). &P: 99.4, J2 - 0.006
Web Dumain liglt orig), (9:99.4 [ 0.000
1,050
Encoder Scaling
Encoder Scaling
1.075
Sculing
1.025
Desedler Scaling
jointyy Fitzed Scating Law
Jointly Fitted Scaling Law
1,050
L 20
L 20
1.025
0.975
0,950
0.975
0.900
0.950
0.925
0.850
0.900
0.2
0.4
0.6
0.0
1.0
1.2
144
1.
1.8
0.2
0.4
0.6
0.8
1.0
1.2
144
1.6
1.8
Total Number of Parametets (Excluding Embodddings)
Total Number of Par armeters Eaclatiin Entediings
Figure 2: Evolution of test loss as a function of the model size for English Beggish-~Germee German models. Eq. is jointly fitted to the empirical loss values from encoder scaling and decoder scaling experiments. Our proposed scaling law is able to capture more than 99% of the variation the data. We anticipate some fluctuations around the predicted trend (ā 0.003) caused by the randomness in the training pipeline (see Appendix See Appendix D for similar results on our other test sets.
Web Domain (src orig), 99.8. J ((- 0.008
Web Domain (lgt orig). A 99.8, LL Qi. 0.005
1.165
1.140
Symmetri i. Scaling (Main Saud)
Symeeitri Scaling (Main Soudl
1.115
Symmetric Scaling ID fffrent Soods)
1.081
Symmetric Scaling (Different Sooda)
Sealing Law Prediction I'
Sealing Prediction
Text Loss
1.066
1.019
1.020
Test
0.960
0.975
0.905
0.933
0.053
0.8%
0.078
0.166
0.357
0.765
1.640
3.515
0.803
0.078
0.166
0,357
0.765
1.640
3.515
Total Numbar of Parameters (Excluding Embeddings)
Total Numbar of Parameters (Excluding Embeddings?
Figure 3: Comparison of the out-of-sample predictions of the scaling law with the empirical test loss values from symmetric scaling Enerish models. Eq. [I) is fitted only using the encoder / decoder scaling data and then just evaluated on (Ne, Na) pairs corresponding to the symmetric scaling model. Our scaling law able to almost fully capture the variation in the data (R² 99.8%). To examine the randomness in the results, we have repeated a subset of training runs with 4 different random seeds (see Appendix for details). See Appendix Dooo similar results on our other test sets.
2.3 ANALYSIS
The above results suggest that the scaling law formalized in Eq. ① captures the scaling behavior of the Transformer NMT models in multiple language pairs. As such, we can study the fitted coefficients to fully understand the scaling properties of these models. Several observations are in order:
Decoder Encoder Scaling: On all our test sets, the decoder exponents were observed to be
larger than the encoder exponents, Pa > (see Figure 9). As a result, when improving the test loss is concerned, it is much more effective scale the decoder rather than the encoder. This is contrary to the usual practice; due to latency considerations, many practitioners train NMT models with deep encoders and shallow decoders (Kasai et al., (2021). Our results suggest this practice could be sub-optimal in terms of loss reduction. Proposition [ beloow provides guidance on how to allocate parameters between the encoder and decoder optimally. The proof is presented in Appendix [G]
(Optimal Scaling), Assume the loss performance of the model is described by Eq. ( Let B denote the budget for the total number of parameters. Then, the optimal encoder / decoder sizes (denoted respectively by N and N are N Pe B and N
Proposition 1
pd
B. In addition,
Petps
Put
when optimally scaling the model, the scaling law reduces to Lope(B) = aa B (patte) + Lox where a" = a (De+Da y (De+pay yos
N- Pe. the optimal scaling behavior can be achieved. Inspection Proposition [suggests that when N₂ pu of the functional form of Eq. ( suggests that as long as Na/N, is fixed as the model scales (i.e. encoder and decoder grow proportionally together), the optimal scaling exponent, (p. + pa), can
5

---

Published as a conference paper at ICLR 2022
Prodicted Test Lass. ClOuu of Sampie))
Web Domain 1 (ire orgg), Co 0.9.03 0.00.07
0.007
Predicted Text Locs, 110utoofSamppi
Web Domain 1 (igt origl. Carti. ().i...))0.990)) 00000
1.00
x=y line
x=y
2+
0.92
[+
0.98
0.95
0.00
0.94
0.86
0.92
0.84
0.90
0.82
Tiaa
0.90
0.92
0.96 Empirical Test Loss, 2
0.94
0.98
1.00
0.84
0.85 Empirical Test Loss,
0.88
0.90
0.92
Figure 4: Comparison of the out-of-sample predictions the scaling law with the empirical test loss values from the randomly sampled English-+Geraa models, The predictions are more than 99% correlated with the empirical test loss values. See Appendix for similar results on other test sets.
Da Ein
- 0.00n
Taxk: 7h
ga.n P 0.004
Taol: 2h .7 06.7, p 1 0.007
0.85
Encocer Scaling
Scaling
3.10
Dccodar Scaling
Scaing
Emooder Scaling
Docoder Scaling
pannly Titted Sceling Baw
jointly Fitted Sowing Law
jernely Fitted Sealing Law.
0.50
0.95
Test
0.90
0.95
0.00
0.
1.0
340
Nan-ebeediigggs Parameters
Tonal Nen-Fmbeddingg Parameters
Figure 5: Fitted scaling laws for other language pairs. Eq. ①1) is jointly fitted to the empirical loss value from encoder/decoder scaling experiments. Similar to EDe case, the law is able to describe the empirical scaling behavior of the models with high accuracy. See the appendix for more details.
be achieved, albeit with potentially sub-optimal multiplicative constant, a# To examine the magnitude of this sub-optimality, in Figure To we compare the multiplicative constants resulting from proportional scaling of the encoder and decoder with different values of Na/N, The results suggest that as long as the parameter allocation is not extremely far from (Nr, Nj). the scaling scheme is approximately optimal. In particular, symmetrically scaling the encoder and decoder layers, is barely distinguishable from the optimal scaling scheme.
EFFECT OF DATASET COMPOSITION BIAS ON SCALING BEHAVIOR
Translation deals with the problem of mapping a sequence in one language into another language. good translation should not only be adequate and fluent, but should ideally also adopt the style of a sentence naturally written in the target language. This necessitates MT models to make sense of natural looking inputs and generate natural looking outputs. As mentioned in Section 2 the examples used to train or test NMT models carry a critical bias, which we refer to as composition bias. Composition bias is introduced because of the unavailability of source-target examples (pairs) that are both natural in the accessible data generating distribution. For any given naturally generated text in language, the corresponding text in the other language is either translated by humans, introducing translationese bias or translated by other machine translation systems, introducing MT bias. We consider both biases affecting the problem from a similar angle, hence we bundle them and call composition bias. While machine translation by design has composition bias in the training/test sets employed (Freitag et al. 2020; Riley et al.) 2020), its effect on model scaling is unknown. In this section investigate the role of composition bias in scaling and identify critical factors playing role.
We caution the reader to not take the composition bias problem specific to NMT. In fact as most training corpora in NMT are web-crawled. they can contain machine translation output on either the source target side. Considering the growth of generated content in the web by machine learning

---

Published as a conference paper at ICLR 2022
models 0+
it is not improbable that a proportion of the content collected and used by machine learning models is going to be biased by other models that are continuously generating content.
The Effect of Test Set Construction: We first examine the impact of composition bias on the test sets. shows the fitted scaling exponents for all our test sets. The plot suggests that the scaling powers for source-original test sets are drastically different from those of target-original test sets. This behavior is in direct contrast with language modeling setting (Kaplan et al. (2020) where it was observed that the evaluation on different test sets merely acted as scaling penalty that only changed the multiplicative constants of the scaling law.
To elucidate this phenomenon further, in Figure of the appendix, we compare the scaling trends for different source and target original test sets. Several observations are in order: Test sets with a similar composition approach (source or target original) have a qualitatively similar scaling behavior. However, scaling behavior is vastly different between the two composition approaches. Reducible loss quickly decays to zero for source original test sets. fact, we observe that scaling our baseline 6L-6L model by a factor of 2.5 is sufficient for ensuring that reducible loss is below 0.05 for all source original test sets. In contrast, on target original test sets, the loss decays much more slowly with model size. For comparison, ensure that reducible loss is below 0.05 for all target original test. sets, we estimate that the baseline model has to be scaled up by factor of 11.
Because of this behavior, the value of larger models in NMT is closely tied to their evaluation sets: On source original data, due to larger scaling exponents, even moderate increases in model size are sufficient for pushing the reducible loss close to zero. Hence, beyond few hundred million parameters, there is no benefit in increasing the model size. In contrast, for target original data, which generally have smaller scaling exponents, large models are needed to push the reducible loss zero.
The Effect of Training Set Construction: In this section, we briefly examine the role of training data construction on the scaling behavior. To do this, we generate two En-+ee datasets, that were not used in the previous experiments. One fully target original and another completely source original.
To generate the target original dataset, we compile a set of German documents from the web. Documents are screened to ensure the data is not machine generated. We use Hybrid model (with 380M parameters) (Chen et al. 2018) to back-translate (BT) these documents to English. Similarly, for the source original data, collect human generated English documents and (forward) translate them to German using a hybrid model (with approximately 327M parameters). Both datasets provide us with approximately 2.2 billion training examples. We mimic the experimental setup of Section 2
Note that even though these datasets are not human generated, they reflect important aspects of training large NMT models. Many modern NMT datasets are harvested from the web and result, are contaminated with machine generated data. Moreover, many popular data augmentation algorithms such as Back Translation (Sennrich al. 2016), sequence level distillation (Kim & Rush 2016) and self training (He et al. (020) purposefully add machine generated data into the training pipeline in order to take advantage of monolingual data.
Figure 6 describes the scaling behavior for models trained on target-original data. We observe that even though larger models are successful in reducing the training loss, they are unable improve the test loss after roughly 400M parameters. Once this size threshold is exceeded, models overfit the training data and the test loss starts to deteriorate across all of our test sets. We hypothesize that this size threshold corresponds to the capacity of the original back-translation model. This assertion suggests that in order for back-translation to be beneficial for training large models, it has to be performed with models with comparable capacity or higher. Although quite intriguing, we leave the verification of this hypothesis to future work.
Figure | paints another interesting picture for the models trained on the source-original data only, implying the target side having the composition bias, expected to be simpler, dull and not rich in its content, in short- not natural looking. As experiments suggest, even our smallest models are able to achieve extremely low loss values (roughly 0.16), with an apparent overfitting pattern. We believe the same phenomenon is also related to the "data simplification" effect seeked by non-autoregressive models in NMT (Zhou et al. 202T).
hhtn:://ogooaaaccoom/mooggggggg/
7

---

Published as a conference paper at ICLR 2022
1.42
Wab Domain ( orig)
1:21
Wob Domain 1 loge orig)
0.57
Mochine Trandiated Data iTrininn Dist
Denader Sea
Cras-ERrrpy L200
1.39
Cross Cross Entroyy Loge
1.20
Crose-Poe Preee-rrrrryy Lowe
0.54
1.36
1.38
0.51
1:37
0.49
Ensocer Scatig
1:15
Encoder Scaling
0.46
1.28
0.2
0.6
0.8
1.4 3.6
0.4 0.6
0.8
1.4
0.4
0.6
1.0
1.2
Number Parametors
Total Number Porameters
Total Number Parameters
Figure 6: Scaling behavior of models trained on tackssraasleted data. Right: Increasing the model size successfully reduces the loss on the training distribution. However, on the test data (left and center) the loss increases after approximately 400M parameters.
2.17
Web Damaiin (srt orig)
2.43
Web Domain 1 fig orig)
0.17
Machine Translated Data (Training Gist
Cross Entogy Loss
2.03
Cross Cross Entrpy Loss
2.34
0.15
Cree-heemey
1.91
2.26
0.13
1.01
2:38
0.12
1.71
Broocer Scaling Docoder Scaling
2:31
Encoder Scaling Decoder Scaling
1.63
0.2
0.4
0.6
0.8
1.0
1.2
1.4 16
2:04
0.2
0.4
0.6
0.8 1.0
12
1.4 1.6
0.2
0.4
0.6
0.8
1.0
12
14
Number Parerttess
Total Namber Paramaters
Total Kumber Parameters
Figure 7: Scaling behavior of models trained on forward translated data. Left center: early stopping
test loss on Web- Web-Domain. Right: loss at the end of the training for subset of the training data.
4 EVOLUTION OF GENERATION QUALITY
We examine the effects of scaling on the output quality as measured by BLEU score For this analysis, we focus on output generated via beam search (Wu et (016)) For tractability purposes, we do not attempt to tune the (many) hyper-parameters of beam-search for each model. Instead, we use the configuration optimized for the baseline model (listed in Appendix My in all the decoding tasks.
Figure X presents the co-evolution of BLEU score and cross-entropy loss throughout the training for all of our models. Depending on the construction of the test sets, two different empirical behaviors emerge. On target-original test sets, larger models are able to improve (lower) the test loss. These improvements in the loss are accompanied with consistent improvements (increases) in BLEU score. In fact, we observe that a simple power law can capture the relationship between BLEU score and cross-entropy loss for high-quality models.
| In contrast, on source-original test sets, this relationship is absent; larger models consistently achieve better test losses, however, beyond a certain threshold, BLEU scores begin to deteriorate. Figures 27 and [8 exhibit that this phenomenon is not due to over-training; the BLEU score gap between large and small models is persistent throughout training.
To ensure that this observation truly reflects the generation quality of the models (as opposed to potential biases of BL.UU score), we repeat our analysis with BLEURT score (Sellam et al., [020a]] b). The results are presented in Appendix K The results show that BLEURT scores closely mirror the behavior of BLEU scores with respect to model scaling.
careful look at the left-subplots of Figures 8 brrings up another interesting trend. At similar values of the test loss, encoder-scaled models result in better generation quality compared to decoder-scaled models. This findings agrees with previous work that relied on encoder-scaling when optimizing for BLEU and inference latency (Kasai et al. 2021). Whether these differences in the effects of incoder-scaling and decoder-scalin are caused by insufficient search algorithms, or just different model fits from different architectural priors is left future work.
We computed the BLEU scores using an internal reimplementation of Moses scorer: mt evvevall 1 -v 13a.p) Fitting details are presented in Appendix
8

---

Published as a conference paper at ICLR 2022
-n De Web Damain (igt orig!
En De Web Domein (src orig)
0.446
Symmettic Staling
0.345
Symmentic Sceling
Encoder Scaling
0.341
Encoder Scaling
Test Score
0.427
Occocel Scoling
tedd Power Law
0.337
Docoder Scalling
0.408
0.333
0.390
0:329
0.373
0.325
0.32
0.79
0.95
1.16
Test Cross-Entropy Loss
Test Crass Entropy Loss
Figure 8: Log-log plot of the evolution of BLEU score as function of test loss. For each scaling approach, warmer colors represent larger models. Each individual color represents different check- points of a single model during training. On target original data, improvements to cross-entropy loss lead to consistent improvements in BLEU score. Dashed lines correspond to power-law fits. The relationship breaks down for source original data. See Appendix J for more details.
CONCLUSION AND LIMITATION
In this work we have attempted to quantify the evolution of model quality as a function of model capacity for encoder-decoder NMT models. While univariate scaling law describing the cross- entropy as a function of the total number of parameters in the model is insufficient, a bivariate law treating the number of encoder and decoder parameters as separate variables adequately describes the scaling behavior of these models under various scaling strategies. We validate this behavior on a variety of language pairs and evaluation sets. Whether this behavior is intrinsic to the encoder-decoder architecture, or arising from the nature of the NMT task, requires further study.
Next, show that this scaling behavior is highly dependent the composition of the evaluation data, specifically on whether the source or target sentences are "original".. Our findings indicate that target-original evaluation sets continue benefiting from model scaling, while the reducible error on source-original evaluation sets quickly saturates to O. This could be an artifact of the lack of diversity in translated text; simpler target distribution doesn't 't require much capacity to model while generating fluent or natural- natural-looking text could benefit much more from scale.
We also study how the composition of training data affects the scaling behavior of models. When training on target-original (back-translated) text, model quality keeps improving until point after which the trend saturates. In our study the capacity where saturation manifests first is perilously close to the capacity of the model used for back-translation indicating that the capacity of the generative model used to generate synthetic text might have a role to play, but this requires further investigation. When training source-original text, even low-capacity models are sufficient to reach the irreducible loss region, painting a gloomy picture for synthetic data. While we have explored these ideas in the context of machine translation, given the proliferation of generative models this problem will likely be a challenge for future practitioners training on web-scraped monolingual datasets as well. For low-resource languages, the proliferation of machine translated text is already a problem given that significant portion of web text in these languages is machine translated.
Finally, we attempt to understand how generation quality evolves with the improvements in cross- entropy resulting from model scaling. As with our previous findings, dataset composition plays a major role in determining the trends. For source-original evaluation sets, the correlation between cross-entropy and generation quality breaks down. On target-original evaluation, we observe an inverse correlation between cross- coos-entropy and BLEU, suggesting that improved model fit results in corresponding improvement in generation quality. The slope of this relationship is different for encoder-scaling and decoder-scaling. with encoder-scaled models performing better on BLEU than decoder-scaled models, the same level of cross- cross-entropy loss. Whether this is artifact of our search strategy or the difference in architectural priors is something that requires further investigation.
Our findings suggest that scaling behavior of encoder-decoder NMT models is predictable, but the exact formulation of scaling laws might vary depending on the particular architecture or task being studied. Our empirical findings also raise concerns regarding the effect of synthetic data model scaling and evaluation, and how proliferation of machine generated text might hamper the quality of future models trained web-text.

---

Published as a conference paper at ICLR 2022
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensionn dynamics of general- ization error neural networks. Neural Networks, 132:428-4466 2020.
Subutai Ahmad and Gerald Tesauro. Scaling and generalization in neural networks: a case study. Advances in Neural Information Processing Systems, 1:160-168, 1988.
S. Amari, Naotake Fujita, and S. Shinomoto. Four types of learning curves. Neural Computation, 605-618, 1992.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs/1907.05015 2019. BRL.http://axxiv.oo wwtpaaawww...ra.oaaaaoooo abs / 907. 0501 9
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws, 2021.
Loic Barrault, Ondiej Bojar, Marta R Costa-Jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, al. Findings of the 2019 conference on machine translation (wmt19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). pp. 1-61, 2019.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation, 2018
Kyunghyun Cho. Scaling laws of recovering bernoulli. Blog Post, 2020. me/ ottp://wwwnoopoeeeeeaaaaaaaaeaaa wnn.----eeettttttttoteoooooaaaa
URL /
Markus Freitag, Isaac Caswell, and Scott Roy. APE at Scale and Its Implications on MT Evaluation Biases. In Proceedings of the Fourth Conference Machine Translation, pp. 34 44, Florence, Italy, August 2019. Association for Computational Linguistics. URL http: / www. . ac aclweb
org/ awtahaoooogeee /1 5204
Markus Freitag, David Grangier, and Isaac Caswell. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 61-71, Online, November 2020. Association for Computational Linguistics. doi: 11286633122220.emm/ 2020. maanpp-main...
main.5. URL https: /www. aclweb org/ antho logy/
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d' Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2)003401, Feb 2020. ISSN 1742-5468. doi: 1..1088/174225446//666666 URI
neural networks in high dimension. The Annals of Statistics, abbbbb3/bbccccco
http::/dxx doi org 10. 1088 1114114422
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers 2021.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 5915-5922, Online and Punta Cana, Dominican Republic, November 2021. Associatinn for Computational Linguistics. doi: htttps://aclanttee logy. org 2021 emnlp-ma i n. 478
mammlp-main.47. main.478. URL
Yvette Graham, Barry Haddow, and Philipp Koehn. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 72-81, 2020.
Junxian He, Jiatao Gu, Jiajun Shen, and Mare' Aurelio Ranzato. Revisiting self self-training for neural sequence generation, 2020.
10

---

Published as a conference paper at ICLR 2022
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXfv:2010.14701, 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.00400 2017.
Yanping Huang. Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le. Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In NeurIPS, 2019.
Marcus Hutter. Learning curve theory. CoRR, abs/2102.04074, 2021. URL https: /tthttps:://eett //arxiv
brg/abs/2102 04074
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.085: .00361, 2020.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation, 2021.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation, 2016.
Moshe Koppel and Noam Ordan. Translationese and Its Dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies Vol- ume 1. pp. 1318-1326, 2011. 2002636
URLhttp: /ttp://dll /ww.//laao acm. wtgggiiatiin.n..ooooio...o...a.e wtw.:iioon.cmmmmmm0000200000
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. [GS) Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Representa- fions, 2021. / http://wwwweeeettmmddddddaeeeeee
Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting Association for Computational Linguistics, pp. 318-318. Association for Computational Linguistics, 2002. URL
https: wwwwwwwww..mmmm.mmm/..wwo ac web. org antho Pogg/P02-1042
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667, 2020. URL
http::/ http://www...gggppppeee./// http///pppprr//1///00// 074.html
Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. Translationese language in "multilingual" nmt. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 7737-7746. 2020.
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. International Conference on Learning Representations, 2019.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.088 04696, 2020a.

---

Published as a conference paper at ICLR 2022
Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. Learning to evaluate translation beyond english: Bleurt submissions to the wmt metrics 2020 shared task. arXiv preprint arXiv:2010.04297, 2020b.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL (1), 2016.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596 4596-4604, PMLR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.00 2016.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533, PMLR, 2020.
Chunting Zhou, Graham Neubig, and Jiatao Gu. Understanding knowledge distillation non- autoregressive machine translation, 2021.
12

---

Published as a conference paper at ICLR 2022
A CONNECTIONS TO RECENT LITERATURE
In this section, we provide a more detailed overview of the scaling laws literature immediately relevant to our investigation. [Hestness et al. (2017) examine data scaling laws for LSTM- encoder-decoder NMT models. They show that the model performance exhibits an empirical power- law relationship with the size of the training data. Rosenfeld et (2019) study model and data scaling laws for image classification and language modeling tasks. They provide a bivariate scaling law that jointly capture the model and data scaling behaviors for each of these tasks. Crucially, they show that scaling laws can be used to effectively extrapolate beyond model/data configurations used for fitting them. particular, they show that one can use smaller models to fit the scaling law and then use the law to accurately predict the performance of larger models. Our results (Figures 3 Jnd validate that this behavior also holds for our setting.
Kaplan et al. 2020) study the scaling behavior of decoder-only Transformer models used in language modeling. They identify three factors that principally determine the model performance: the number of (non-embedding) model parameters, the size of the training dataset, and the amount of compute used for training. In the scenario where data and compute are abundant, they show that model performance exhibits empirical power-law relationship with the total number of (non-embedding) parameters in the network. Henighan et al. (2020) verify this power-law behavior for other auto- regressive generative modeling tasks such as image modeling, video modeling, and mathematical problem solving. In contrast, in our setting, we observe that the total number of parameters is not good predictor of the test loss in NMT. Instead, we show that mncoder-decodér NMT models scale according to bivariate function of the number of encoder parameters and the number of decoder parameters.
Kaplan et al. 2020) also examine how scaling affects the in language models. Their results suggest that different test sets exhibit rather similar scaling trends as model size increases. In comparison, in the NMT setting, we show that different test sets, depending on their naturalness structure, exhibit drastically different scaling behaviors.
generalization behavior
The most closely related study to ours is the work of Gordon et al. (2021). They also examine scaling laws for NMT. albeit at a different parameter regime: Gordon et al. focus on small models (393K- 56M parameters) while here, focus on large scale models (100M-4B parameters). In addition, we examine the general scenario where encoder and decoder can scale asymmetrically (which is common practice in NMT) while Gordon al. only consider symmetric scaling. Despite the differences in the experimental setup, when focusing on the symmetric scaling setting, the conclusions of both studies mostly align. In particular, when symmetrically scaling the model, our scaling law simplifies to
L(N) aN + Loo
which agrees with the conclusions of Gordon et al. One difference in our conclusions is the presence of the Loo term in our scaling law. This term, which captures the irreducible loss in the learning problem, does not appear in Gordon al's analysis. We suspect that this might be due the small models used in that analysis which cannot statistically observe such asymptotic terms.
Gordon al. also examine the relationship between test cross-entropy loss and BLEU score. They observe that as models scale and cross-entropy loss improves, BLEU score also improves in tandem. In comparison, we unveil more intricate relationship our study: We show that when the test data is origiall improvements in the test cross-entropy loss are correlated with improvements in generation quality (as measured by BLEU/BLEURT scores). However, when the test set source- original, the correlation between cross-entropy loss and generation quality breaks down. In this setting, we observe that generation quality improves up until certain threshold and then deteriorates (see the right plot in Figure 8)
13

---

Published as a conference paper at ICLR 2022
B MISCELLANEOUS FIGURES
0.5
0.5
Q
0.4
0.4
x
x=y
0.2
orig
orig
0.1
tgt orig
Train Data
Train Data
0.0
0.0
0.5
0.3
0.4
0.5
Pe
Pa
Fitted linin
Fitted Scaling
0.5
0.5
0.4
0.3
X-y
0.2
X-y line
orig
0.1
orig
0.1
tgt orig
Train Data
Train Data
0.2
0.5
0.2
0.4
0.5
Pe
Pe
Figure 9: Fitted scaling exponents for En++Zh and En translation tasks. Across all the test sets under consideration, we observe pa > Pe-
Effect of Parameter Allocation in a²
Additiive Test Loss Penalty Due Suboptimal o
WMT nowitit22LD
Xx100
Web Domain 1 figt ori::
Web Domwin 7
3.0
Patents Domatn Imised)
WiMpedia Domain
WMI newsteet201
11
2.0
Domain (orc
1.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Na/N
NoIN
Figure 10: We use our fitted scaling laws to evaluate the effect of encoder decoder parameter allocation ratio when proportionally scaling the encoder and the decoder. Left: a /as for different parameter allocation schemes. Right: The predicted additive loss penalty, (L- Lopt), for model with 10 total (non-embedding) parameters. Each line corresponds to different test
14

---

Published as a conference paper at ICLR 2022
Scaling Comparison Source Target Original Damains)
Linear Axes
Log-log Axes
0.586
0.5
Bomain (ert
Domain fgg
0.4
Domain fig
Reducible Los,
Rodu ditbleb Loss,
Baseiire
0.057
0.2
0.026
0.1
0.012
0.006
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.078
0.166
0.357
0.765
1.640
3.515
Total Number Parameters (Excluding Embeddings)
Total Number of Parameters (Excluding Embeddings)
Figure 11: A comparison of scaling behavior across source and target original test sets. We use our fitted scaling laws estimate the evolution of reducible loss for each test set. All scaling trends correspond to symmetrically scaling the encoder and decoder layers.
C ARCHITECTURE AND
DETAILS
As described in Section 2 all our models use similar configuration for their Transformer Blocks. In particular, we fix model dimension 1024, feed-forward layer dimension 8192, number of attention heads to 16, and attention hidden dimension to 1024. Our models use sentence-piece vocabulary of size 32000.
We dropout of 0.1 for residuals, feed-forward activations and attention. Models are trained with label smoothing of magnitude 0.1. To improve the training stability, all models use logit clipping of 10.
Regularization:
We use Adafactor (Shazeer & Stern 2018) optimizer for training our models. We use 40k linear warm-up steps and an inverse square root learning rate schedule. For Adafactor we used momentum with 0.9 and factored second moment save memory.
Optimizer:
Table I (and Table 2 resp.) describes the parameter decomposition of the encoder scaling (decoder scaling) models. Table describes the parameter counts for the symmetric scaling models. The largest model we used (64L-64L) has more than 3 billion parameters while the smallest model we used (2L-2L) has only 92M non-embedding parameters.
15

---

Published as a conference paper at ICLR 2022
Table I: Parameter decomposition of the encoder scaling models. The total number of parameters includes 98M parameters representing the softmax and embedding layers.
Encoder
Decoder
Layers
Parameters
Layers
Parameters
Total Parameters
2
42M
6
151M
291M
84M
6
151M
333M
8
168M
6
151M
417M
252M
6
151M
501M
16
336M
6
151M
585M
20
420M
6
151M
669M
24
504M
6
151M
753M
28
588M
6
151M
837M
32
672M
6
151M
921M
36
756M
6
151M
1005M
40
840M
6
151M
1089M
48
1007M
6
151M
1257M
56
1175M
6
151M
1425M
64
1343M
6
151M
1593M
Table 2: Parameter decomposition the decoder scaling models. The total number of parameters includes 98M parameters representing the softmax and embedding layers. Note that the 6L-6L model the baseline model we used for hyper-parameter tuning.
Encoder
Decoder
Layers
Parameters
Layers
Parameters
Total Parameters
126M
2
50M
275M
126M
4
101M
325M
6
126M
6
151M
375M
126M
8
202M
426M
6
126M
12
302M
527M
126M
16
403M
627M
6
126M
20
504M
728M
126M
24
605M
829M
6
126M
28
705M
930M
126M
32
806M
1030M
126M
36
907M
1131M
126M
40
1008M
1232M
126M
48
1209M
1433M
126M
56
1411M
1635M
126M
64
1612M
1836M
16

---

Published as a conference paper at ICLR 2022
Table 3: Parameter decomposition of the symmetric scaling models trained for English German translation task. The total number of parameters includes 98M parameters representing the softmax and embedding layers.
Encoder
Decoder
Layers
Parameters
Layers
Parameters
Total Parameters
2
42M
2
50M
191M
63M
3
76M
237M
4
84M
4
101M
283M
105M
5
126M
329M
8
168M
8
202M
468M
336M
16
403M
837M
20
420M
20
504M
1022M
24
504M
24
605M
1207M
32
672M
32
806M
1576M
48
1007M
48
1209M
2315M
56
1175M
56
1411M
2684M
64
1343M
64
1612M
3054M
17

---

Published as a conference paper at ICLR 2022
D SCALING LAWS FOR OTHER TEST SETS
In order to keep the discussion in the main text focused, we only presented scaling laws for Web Domain test sets. These sets were chosen as they had domain similar the training data (i.e. web). In this appendix, we repeat the same analysis for our other test sets. The details of these test sets are described in Section
Figure 12 demonstrates how well the scaling law in Eq. ① fits the empirical scaling behavior of our models on all our test sets. For each test set, we have fitted the law jointly on the final test loss achieved by the encoder and decoder scaling models. We measure the final test loss by the median test loss over steps 450K 500K. Details of the fitting procedure are provided in Appendix !|
The results suggest that Eq. I is closely capturing the scaling behavior of the model for all the test sets domains. In the last row, we also demonstrate the fit for the training data (left column) and the training loss (cross entropy on training data plus regularization, right column). We observe that the scaling law is almost perfectly fitting the empirical data in these cases.
To examine the fit more closely, in Figure 13 we have plotted the same data but with several modifications:
1. Instead of plotting the final loss, we plot the reducible component of the final loss (L Loo) As the true value of Loe is unknown, use the value given by the fit of the scaling law.
2. For encoder scaling models, plot the (reducible) loss against the number of encoder parameters (as opposed to the total number parameters). Similarly, for decoder scaling models, we plot the loss against the number of decoder parameters.
3. We use log-log scaling on the axes.
4. We use the results of Table provide confidence region around our predictions. This con- fidence region quantifies our expected uncertainty caused by randomness in the initialization and training pipeline.
Eq. predicts that the relationship between the empirical final loss values from the encoder (decoder) scaling models and the number of encoder (decoder) parameters should appear linear on these plots. Figure suggests that the empirical scaling behavior of these models conforms closely to these predictions.
To see if Eq. ① continues to capture the scaling behavior of the models out-of-sample, compare predictions of the scaling law with the empirical loss values for symmetric scaling models. In other words, examine how well the scaling laws fitted only using encoder / decoder scaling models predict the final test loss achieved by symmetric scaling models of different sizes. Figure 14 shows this comparison for all of our test sets. We observe remarkable match between the predictions of the scaling law and the empirical loss values across the board. These observations confirm that Eq. (1) is able to capture the scaling behavior of the model regardless of the scaling approach.
18

---

Published as a conference paper at ICLR 2022
Web Domain Isrc orig), AP: 99.4. IL 00000006
0.006
Web Domain a fig origi. 99.4, BL Ll.: 0.000
1.100
Erooder Scaling
1,550
Ercoder Scaling
1.075
Decodor Scaling
1.025
Decader Scaling
1,050
Jointyy Fitted Scaling
1.000
Fitted Scaling 20
1.025
20
0.975
0.050
0.925
0.900
0.950
0.875
0.925
0.850
0.900
0.2
0.4
0.5
0.8
Total Number Parameters (Eacluding Enbediings)
le
Total Number of Parameters (Excluding Emboddiing))
10
WHT 9 A 99.1, Q1: 10.009
WMT newstes20119 ligt orig), 8P: 99.6, [- 0.011
1.025
Encoder Scaling
Enredder Scaling
1.000
Decoder Scaling
Decoder Scaling
Jointly fitted Sceling law I' 20
by Fisted Scaling Law
I 20
1.05
0.925
Test
0.900
1.00
0.875
0.95
0.850
0.825
0.90
0.2
0.4
0.6
0.8
1.0
12
1.4
1.6
1.8
0.2
0.4
0.5
0.8
3.0
1.2
1.4
1.0
3.5
Total Number Paramatars (Excluding Entociiings)
Total Numrowr Paramatera (Excluding Emblddiings)
Web Domain Isre origt, 98.4 JL -L- 000000000
0,008
Web Domain 2 figt origi, R?: 99.4, E. Q.. 0.010
Ercoder Scaling
Encoder Scading Decoder Scaling
0.88
Decoder Scaling
Jointyy Fitted Sceling Law
1.05
risted Scaling law
Lons
2G
0.84
1.00
0.95
0.80
0.2
0.4
0.5
0.5
1.0
1.2
1.5
0.2
0.4
0.5
0.8
1.0
1.2
1.4
1.5
Total Number of Parameters (Excluding Embeddiings
Total Number of Paramaters (Excluding Embeddiingg)
10
Pataett Gomain (miowd)), Rt: 99.3, I - 0.005
Web Domein 3 [urc orig), n 982, Ji --- 0.013
0.74
Encoder Scaling
Encoder Scaling
0.72
Decoder Scaling
Decoder Scal
Jointly Fitted Scaling Law
1.15
Jointly Filted Scaling Law
0.70
La 20
Loss
I' 20
Test
3.10
0.66
0.64
0.69
0.4
0.6
0.8
1.0
Total Number Parameters (Excluding Embediings)
Total Numbor Paramaters (Excludig Emboddinns))
Wilized Domain (ste orig), 99.5, II 1. 0.014
Web Domain (er orig), 99 II p- 0.807
1.45
Encoder Scaling
1.000
Encoder Scaling
Decoder Sealing
Decuder Scalin
Jointly Fiteed Scaling Law
0.975
Jointly Filted Scaling Law
2
Lost
0.550
1.30
0.925
1.25
0.900
1.20
1.15
0.850
1.10
0.2
0.4
0.6
0.3
10
12
1.4
1.6
0.2
0.4
0.5
0.8
1.0
122
1.4
155
1.3
Total Number of Panameters (Pockuding Embeddingss
14
Total Number of Panameters (Pacluding Tribeedings)
Train Data Subset (mixed), RP, 99.6. [L -- 0.013
Training Loss, 8?: 987, JL 0.008
Ercoder Scaling
Encoder Scaling
0.95
Jointly Fitted Scaling Law
Eneropy Lees
Jointly Scaling 20
Cross Entrrperepp
0.90
26
2.15
0.85
2.10
2.05
0.75
2:00
0.4
0.5
0.8
(Eschuding Embeddinggs
(Eacuding Embeddiing) 165
Figure 12: Fitted scaling law (Eq. 11) for English++Gernne translation task. The scaling law captures the scaling behavior of the models over a diverse collection of test sets and domains. The last row describes the evolution of the cross-entropy loss on the training data (with and without regularization effect).
19

---

Published as a conference paper at ICLR 2022
0.232
Web (src orgg). 99.4,
0.271
origi,
Encoder Scaling
0.131
0.203
Docoder Scaling
Jointy Fitted Staling Law
0.074
0.152
Rodurble Loss:
Recucblee Look,
0.042
0.110
Encoder Scaing
Decoder Sceling
0.013
Jointyy fitted Sceling
0.004
0.008
0.035
0.079
0.178
0.400
0,903
2,036
0.035
0.079
0.178
0.400
0,903
2.036
Number of Encoder Decoder Parameters
Number of Encoder / Decoder Parameters
WMT n005iess201 iocc orig), A 99.1 JL 0.0000
WMT newslast2019 (igt origl. 8": 99.6, IL Al. 0.011
0.235
0.371
Encoder Scaling Scaling
Rcductii Look -12
0:318
Scaling
0.060
0.214
0.030
0.163
0.015
Encoder
Reductte
8.12
Sceling Jointy Fibted Scaling
0.008
0.09
L 20
0.005
0.071
0.178
Encoder
ongl. JL- ip.
ongll R-
Encoder Scaling
0.069
Decoder Scaling
Jointly Fiired Scaling Law
Redudble Loss:
0.013
Redudble
Encoder Scaling
Desodar Scelinng
0.003
Jointyy Fitted Scaling Law
0.001
0.035
0.079
0.178
0.400
0,903
2,036
0.035
0.079
0.178
0.400
0.503
2.066
Number of Ensoder
Decoder Par amallerss
Namber of Encoder Decoder Parametars
Patents Domatn imixedi, R": 98.3, JL E).: 0.005
Web Domain 3 (dnc ong). R:: 98.8, LL- Ep. 0,013
0.145
0286
0.097
0.163
0.065
0.093
Rodudbbe Loss
Reductbe Loss,
0.044
0.053
0.030
Freoder Scaiin
Sceling
0.030
Freoder Scaling
Scaling
0.020
Jointy Staling
0.017
Jointly Fitted Scaling
[ F-
++
0.013
0.079
0.010
2.006
Number Encoder Decoder Parametors
Number Encoder Decoder Parametors
140
--- 0.014
origt, [
Reducbic Loss
Red Redictie
Staling Decoder Scaliing
Enoodar Scaling
Decoder Scaling
Jointly Fitted Staling Law
Jointly Fitted Scaling Law
0.637
0.035
0.079
0:178
6.000
0.903
7
2.036
0.015
0.079
0:178
0.400
0.901
2.036
Number of Encedor Decader Parametors
Number of Enceder Decoder Parameters
Trais Date Sibset (mised), B²: 6 | - 6.013
Training lows, & 96.7, JI j 0.008
0.390
0449
Encoder Sealing Recoderr fcwiing
Encoder Scaling Deroder Scwling
0.323
0.375
Fitted Scaling
Scaling
Seducde uass
0.267
20
0.313
0.211
0.262
0.383
0.19
0.151
0183
0.175
0.51
0.35
0.079
0.178
0:400
0.400 0.903
2.036
Number Encoder Decoder Perameters
1x9
Number Encoder Decoder Parameters
Figure 13: Fitted scaling law (Fq. (1) for plot order to inspect the fit more closely. Shaded cyan regions correspond to the uncertainty region given by x deviation. Per test set standard deviations provided in Table
translation task. Here, we use log-log
20

---

Published as a conference paper at ICLR 2022
Web Domain Isre origi. 99.8, [[ -i 0.008
Web Domain 1 ligh ong).. : [I- 0.005
1.103
1.248
Symmorrcc Scaling (Moin Scodl
Synmetric Scalin (Vain Soadi
1.115
Symmeticc Scaling (Bffferent Seeds)
1.081
Symmetric Scaling (Diffrent Seeds)
Scaling Law mediction
Scaling Law Prediction
1.06b
L2
1.019
L 20
1.020
0.960
0.975
0.905
0.933
0853
0.883
0.078
0.166
0.357
0.765
1.640
3.515
0.803
00078
0.166
0.357
0,765
1.640
3.515
Total Number Parameters (Excluding Embeddings)
Total Number of Parameters (Excluding Embeddinges
WMT newitest2019 (sre orig), R²: 99.8, P p 0.012
WHT newctest2019 (iyr orig), 8²: 99.7, J j-- 0.007
1.116
1.306
Symmetric Seatin (Main Seed]
Symmettc Sea tning (Main Seed)
1.057
Symmetric Seatin (Dittenent Seeds)
12122
Symmetnc Sis inging (Diteren Seeds
Scoling Low Prediction
Scaling Prediction
1.000
1.126
Test
0.947
1.045
0.896
0.970
0.849
0.901
0.803
0.078
0.166
0.357
1.640
3,515 14
08
0.078
0.166
0.357
0.765
1.600
3.515
Total Number Parameters (Excluding Embeddiings
Total Number of Parameters (Excluding Embedei
S4
Web Domain Isrc ortal. R?; 99.9, R Lp: 0.008
Domain 2 (tgt ori 99.a. IL- Q 0.005
0.962
1.199
Symmetri: Scating (Mai Sead]
Symmetnc Sea ing (Mam Swed!
0.923
Symmetrcc Scating (Driierent Seeds)
Symmetric Sca (Diienent Sendu) Sealing law Prediction
Graling tow Predittion
0.889
2e
1.055
20
0.854
1.003
0.821
0.945
0.788
0.891
0.758
0.840
0.078
0.166
0.357
0.785
1.640
3.515
0.078
0.166
0.357
0,705
1.000
3.515
Total Number Parameters (Excluding Embeddingss
Total Number of Parameters (Exclading Embiddings
Patents Domain (mixedi, R"; 99.9, JL [ 0.004
Web Domain 3 (arc origi, R²: 99.5, IL- q 0.011
0.805
1275
Symmetticc Scaling (Main Socdl
Symmotric Sca ing (Main Soadi
0.768
Symmetric Scaling (Dfffcrent Scods)
1.216
Symmetric Sca ing (Diffrrnt Seodal
Scaling Law Preictiion
Scating Prediction
Test Lops
0.732
3.100
20
0.698
1.105
0.885
1.055
0.634
1.05
0.605
0.960
0.078
0.166
0.357
O. 765
1.660
3,515
0.078
0. 0.166
0.357
0.765
1.40
3:515
Total Number of Parameters (Excluding Embeddings)
140
Total Number of Parametors (Excluding Egp
See
Wilipedia orig), 98.7, --q 0.072
Domain orig), p 0.006
1.597
1.082
Symmetric Scating Seed]
(Main
1.491
Symmetrcc Scaling (Dificrent Scods)
1034
Symmebric Sca (Diffrrnn Seoda)
Scoling Law Predittion
Scalinng Prediction
1.392
2o
0.988
20
1.300
1.213
0.002
1.133
0.802
1.0
0.765 Total Number Parameters (Excluding Embeddings)
0.357
3.515
0.
0.166
0.357
0.765
3.515
Number Parameters (Excluding Embedd nggsl
Train Data Subset Imixedi, R 99.3, [t ic- 0.013
Training Loss, R²: 99.4, [t Ic- 0.014
1.091
2.383
Symmetri Scating [Main Seed]
Symmeint Sexing (Vaan Seed!
1.011
Symmerrcc Scaling (Different Seeds)
2.303
Symmetric Sea lng (Diffrent Seeds) Scaling Law Predictio
Scaling Law Predictioon
0.936
20
2.225
20
0.867
2.150
Cross
0.803
2.078
0.743
2.008
0.688
0,078
0.166
0.357
0.785
1.640
3,515
1.960
0.78
0.166
0.357
0.765
1.640
3,515
Total Number of Parameters (Fucluding Embeddiings)
Total Number of Parameters (Excluding Embeddinngt
Figure 14: Out-lf- prediction accuracy of English-+Germant scaling laws on symmetric scaling models. Scaling laws are fitted only using the encoder and decoder scaling models. Nevertheless, they accurately predict the scaling behavior of symmetric scaling models.
21

---

Published as a conference paper at ICLR 2022
(sre org)) 97.9, [[
dig ongi 9f:99.7 [t-[]_
Encoder Scaling
1.100
Encoder Scaling
1.05
Decocer Scaling
Decoder Scaling
Jointly Fitted Sca ing Law
Jointly Fitted Scaling Law
1.04
1.050
1.075
1.02
1.000
1,00
0.975
0.950
0.98
0.8
0.2
Total Number Farameters (Excluding Embeddings)
103
Total Numbor Farameters (Excluding Emocddings)
WWT nowstest20LD farc origi, 99.7. IL Qp 0.004
WWT rowstest2019 (tge orig). A1: 93.7. (L-..0:.000 0.009
Encoder Sceling
1.15
Encoder Scaling
Decoder Staling
Decoder Scaling
1.10
Juntly Fitted Scalin Law
1.10
Joindy Fiteed Scaling Law
Ted Lo53
1.05
Loes
1.05
1.05
1.04
1.00
1.02
0.95
1.00
0.6
0.8
1.2
Number
(Excluding Embedding))
103
(Eecluding Emteddiing))
Web Domain (sre orig), 98.4, JI -..00.0000
Web Domain 2 (ig origt, 99.6, p p=I].:0000
1.05
Decocer Scating
Encoder Scaling
0.90
Encader Scaling
Decoder Scaling
Jointly Fiited Sca
0.88
Fiteed Sceiing Lsy
1.04
Test
0.86
1.02
0.54
1.00
0.80
0.98
0.6
0.8
0.2
0.4
Total Number Parameters (Excluding Embeddings)
Total Number Paramaters (Excluding Emooddings)
Web Domain tigt ongl, R 99.5, JL 11..000000
Wikipedis Domain ltgt org). 993. 12- 12-Q-= -Q010.011 0.013
1.075
Encader Scaling
1.45
Encoder Sealing
1.050
Decacer Scating
1.40
Decoder Scaling
1.025
Jointly Fitted
1.35
Fiited Scaling
1.000
Loss
Tost
0.975
1.30
0.950
1.25
0.925
1.20
0,900
0.2
0.4
0.6
0.8
1.0
1.2
1-6
1.5
0.2
0.4
0.5
0.8
1.0
1.2
144
1.6
Total Number Parameters (Excluding Embeddings)
Total Number of Parameters (Excluding Emoeddings)
Web Domain fut origi, 62; 99.9, Il I.: 0.003
Patentss Domain (mised), R²: 99.0, J -l 0.008
1.35
Encoder Scaling
0.70
Encader Scaling
Decocer Scaling
Decoder Scaling
1.30
Jointly Fitted Scalin ing Law
0.74
Jointy Fitted Scaling Law
Test Lo00
0.72
1.20
1.15
0.2
0.8
0.6
0.8
1.0
1.2
14
1.6
1.8
0.2
0.4
0.5
0.0
1.0
1.2
14
1.6
1.5
Total Numbar Farameters (Excluding Emboddings)
Total Number of Parameters (Excluding Emooddings)
Train Data Subset (mixed), R3: 98.8 J Q 0.008
Training Loss (Reguibried), 8?: 99.9, [ QQj:: 0.007
0.95
Encoder Scaling
Sraling
2.20
Encoder Scaling
Scaling
CRisEntro Los
Jointly Filled Scalng Law
Jointly Fitee Scaling Law
0.90
Enerapy
2.15
0.85
2.30
0.80
2.00
0.75
2.00
0.2
1.4 Total Number of Patameters (Escluding Embeddingg)
0.4
0.6
0.8
1.0
122
1.6
1.3
0.2
0.4
0.5
0.8
1.0
1.2
1.4
1.6
1.3
189
Total Number of Panameters (Escluding Emoedding))
Figure 15: Fitted scaling law (Eq. 1T) for German German->Englist translation task. The scaling law captures the scaling behavior of the models over a diverse collection of test sets and domains.
22

---

Published as a conference paper at ICLR 2022
Web 57.9. 0.000
Domain R9: 99.7, Q) 0.007
Encoder Scaling
Scaling Decoder Scaling
Decoder Scaling
Jointly Fiteed Scaling Law
Joindy Fitted Scaling Law
Reducible Loss,
0.047
Reducible Loss.
0.140
0.030
0.107
0.019
0.082
0.000
0.048
0.035
0.078
0,174
0.385
0.867
1,935
0.035
0.078
0.174
0.589
0,867
1.635
Number of Encoder/ Decoder Parameters
Number of Encoder Decoder Parameters
orig),
Higr orig), p p-
Reducible Loss, L-C L-
Encoder Scaling
Encoder Scaing
0.091
Decoder Scaling
0.250
Decoder Scaling
jointly Fitted Scaling
Fitted Scaling
0.053
Reducible Lass.
0.193
0.034
0.150
0.013
0.090
0.008
0.069
0.078
0.174
0.867
1,935
0.035
0.078
0:174
0.389
0.067
1.935
Number of Encoder Decoder Parameters
1x3
Number Encoder Decoder Parameters
orig), 90.4, _
(tgt onig), I [ _
0.125
0.183
Reducible Loss, L-(-
Encoder Scaling
Encoder Scaing
0.084
Decoder Staling
0.144
Decoder Scaling
Jointly Scaling
Fitted Scaling
0.056
Reducible Loss,
0.113
0.038
0.089
0.017
0.055
0.035
0.174
1,935
0.047
0.035
0.078
0.359 Number Encoder Decoder Parameters
0.174
1.935 Lu
Decoder
Web (tgt orgg), 99.5, Q 0.010
Wikipedia (tgt origi, 99.7, J Q
0.288
0.33
Encoder Scaling
Encoder Scalnn
0.200
Decoder Scaling
0.267
Decoder Scaling
Jointly Scaling
Filted Scaling
Reducible Loss
0.155
Reducible Loss.
0.191
0.098
0.072
0.070
0.056
0.050
0.174
Decoder
Number Decoder Parameters
140
Web Domain ligtt origl, R?: 99.9, [L- -[- 0.003
Patents Domain (mixed), 8": 19999.0 I [ 0.008
0.700
0.145
Encoder Scaling
Encoder Scaiing
0.200
Decoder Scaling
0.108
Decoder Sceling
Jointly Scaling
Scaling Law
Reducible Loss
0.155
Reducible Loss.
0.081
0.086
0.045
0.084
0.033
0.048
0.025
0.389
0.867
0.174
Parametors
Encoder Decoder Parameters
140
Train Data Subset (mised), At 99.8, [ :: 0.008
Trainin loss (Regularived) At: 99.9, II 0f0:0.007
0.373
0.418
Encoder Scaling Decoder Scaling Jointly fitted Scaling Law
Enroder Scaing
0.317
0.300
Scaling
Jointly Fitted Scaling Law
Reducible LosS,
Reducible Loss,
0.185
0.230
0.166
0.198
0.141
0.171
0.035
0.078
0.176
0.389
0.967
0.033
0.078
0.174
0.389
0.067
1.915
Number Encoder/ Decoder Parameters
Number Enceder Decoder Parameters
Figure 16: German FFnglish scaling law fits on log-log scale.
23

---

Published as a conference paper at ICLR 2022
WMT newstest2019 (src orig). 8": 98.4, Qp:: 0,015
WMT newstest2019 (1et orial. 99.7. [ 0.009
1.275
1.50
1.250
1.225
Encoder Scaling
Decoder scaling
1.45
Encoder Scaling
Decoder Scaling
1.200
joindy Fitted Scaling Law
1:40
jointly Fitted Scalinn Law
1.175
1.35
1.150
1.30
1.125
1.100
1:25
1.075
1.20
0.7
0.5
1.0
Number Parametors
Embeddings)
Number Parameters (Excluding Embeddingsi
Web Domain (sre origt, At: 08.7, J U-- 0.013
Web Domain (igt orig), 99.8. p p-f 0.008
Encoder Scoling
Encoder Scaling
Decoder Scaling
Decader Scaling
Jointly Scaling
1.45
Jointyy Fitted Scaling
3.40
Test
1.35
3.35
1.30
1.25
1.25
0.2
0.4
0.5
0.8
1.0
1.2
144
1.6
1.8
0.2
0.4
0.6
0.8
1.0
1-2
1.4
1.8
Total Number Parametors uExig Embeddingss
Total Number Parameters (Excluding Embeddingsi
News Domain 2 (arc orig). 8 98.3, [ 0,014
News Domain 2 (tgt origi. s [L- 0.008
1.500
1.525
Encoder Scaling
Decoder Scaling
1.25
Encoder Scaling
Decader Staling
1.475
joindly Filted Scaling Law
1.20
Jointly Fitted Scaling Law
1.450
1.425
Test
3:15
1.400
1.375
1.05
1.350
1.325
2.00
0.2
0.4
0.6
0.8
1.0
1,2
1.6
1.8
0.2
0.4
0.6
0.8
1.0
1.7
1.4
1.6
1.0
Total Number Parameters (Faclusing Embediiinss)
Total Numberr Parameters [Pacluding Fmbbedinngs)
145
Witipedia (igt origt, R2: P -
Domain ligh orig),
P p- 0.007
1.30
Encoder Scaling
1.125
Encoder Scaling
Decoder Scaling
1.300
Decader Scaling
Jointly Fitted Scaling
1.075
Jointyy Fitted Scaling
1.20
1.050
1.025
1.15
1.000
0.975
0.950
1.05
0.925
0.5
0.8
0.6
0.8
Total Number Paremotors (Exclucing Embeddings)
Total Number Parameters (Evcluding Embeddingsi
Web Demain 3 (bet origt. R?: 99.77 [L Q 0.008
Web Domain (tge orig), 8?: 98.7. [ L-- 0.012 0.012
Encoder Scaling
0.975
Encoder Scaling
Decoder
Decader Staling
1.50
Jointly Filted Scaling
0.950
Jointly Fitted Scaling Law
1.45
1.40
0.875
1.35
0.850
0.825
0.2
0.4
0.5
0.8
1.0
1.7
3.4
1.6
1.8
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.0
Total Number of Parameters lusingging Embeddiingsi
Total Number of Parameters iEscluding Embeddings)
Web Domain (tgt origi, R?: 99.7. - -e 0.006
Web Demain (tgr orig), & 99.7.
-ip: 0.007
1.150
1.35
1:125
Facoder Scaling
Decoder Scaliin
1.100
Decoder Scaling
Jointly Fitted Scaling Law.
1.075
Jointly Fitted Scaling Law
1.050
1.25
1.000
1.20
0.975
0.2
0.5
0.8
1.2
1.4
1.6
1.8
0.2
12
1.4
1.9
Number Parametors (Exclucing Embeddings)
Number Parameters (Excluding Embeddings)
TED (mixed), R 99.8. L----:0.000 0.010
Train Data Out imixec), 99.7. JL- -[p-:00007 0.007
2.05
Encoder Scaling
1.05
Encoder Scaling Decoder Staling
2.00
Decoder Scaling
1.00
1.35
Jointly Filted Scaling
Jointly Fiteed Scaling
1.00
0.95
1.85
0.90
0.85
1.75
1.70
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.8
0.2
0.4
0.6
0.8
1.2
1.4
1.6
188
Total Number Patameters lucinniinn Embeddings)
Total Namber Parameters [Focliding Embeediings)
Figure 17: Chinese Fnggishh scaling law fits.
24

---

Published as a conference paper at ICLR 2022
WMT newstest2019
newstest2019
19.7, Qp 0.009
Encoder Scaling
Encoder Scaling
0.197
Decoder Scaling
0.385
Decoder Scaling
0.144
Jointly Fitted Scaling Law
Jointl Fitted Scaling Law
0.106
0.256
0.078
0.208
0.049
0.188
1.915
Number Encoder/ Decoder Parameters
Number Encoder Decodor Parameters
Domain (stc origl, R+t 98.7 JI _.
Web (igr origi, 99.8. p -f]. 0.008
0.285
0.414
Encoder Scaing
0.197
Decoder Scaling
0.137
Scaling
0.095
0.185
Encoder Scaling
0.045
Jointly Fitted Scaling Law
Decoder Scaling
0.118
0.031
0,092
0.035
0.078
0.174
0,389
0.867
1,935
0.035
0.078
0.174
0.388
0.867
1.935
Number of Encoder/ Decoder Parameters
Number of Encoder Decoder Parameters
News Domain 2 (arc orig), R²: 98.3, IL- Cp 0.014
0.014
News Domain (tot origi. R 39.7. IL Q 0.008
0.245
Encoder Scaling
0.158
0.427
Scaling
0.099
0.364
Jointly Fitted Scaling Law
0000..9
0.040
Ereoder Scaling
0.266
0.025
Decoder Scaling
jointly Scaling Law
0.016
0.184
0.035
0.078
0.174
D. 389
0.067
1,035
0.035
0.078
0.174
0.389
0.867
1,915 teb
Number of Enceder/ / Decoder Parameties
Number of Encoder Decoder Parameters
orig),
orig), p-f 0.007
Ercoder Scaling
Encoder Scaing
0.393
Decoder Staling
0.319
Decoder Scaling
fitted Scaling
Firted Scaling Law
0.284
0.241
0.242
0,209
0.175
0.035
0.867
0.157
0.078
Decoder Parameters
Decodar Parametors
Web Domain 3 (igt origi, 997, IL Qq 0.008
Web Domain (tgt orig), 8 98.7. Il- Q. 0.012
0.300
0.502
Encoder Scaling
Encoder Scaling
Decoder Scaling
Decoder Scaling
0.256
Jointly Scaling Law
0.433
Jointly Scaling Law
0.209
0.170
0.139
0.347
0.113
0.035
0.078
0.174
D. 0.389
0.067
1.935
0.035
0.078
0:174
0.389
0.867
1.915
Number of Encoder/ / Decoder Parameters
Number of Encoder Decoder Parameters
Web Domain 5 orig), e 99.7, - i 0.006
Web Domain (igr orig), R²: 99.7. -iq 0.007
0.382
0.347
Focader Sealing Decoder Scaling
Sceling
0.335
0.202
Decoder Scaling
0.287
Jointly ritted Scaling Law
0.265
jointly Fitted Scaligg Law
0.245
0.210
0.175
0.154
0.124
0.035
0.078
0.174
0.389
0.867
1,035
0.035
0.078
0.174
1.935 La0
Number Encoder Decoder Parameters
Number Encoder Decoder Parameters
(mixed), JL-
0.010
Out Imixed), 8 JL [p.
0.522
1.021
Encoder Scaling
Encoder Scaling
0.013
Decoder Scaling
0.977
Decoder Scaling
Jointly Scaling
Fitted Scaling
0.299
0.878
0.162
0.794
0.128
0.755
0.035
0.078
0.174
1.389
1.995
0.035
8.078
0.174
0.389
0.867
1.915
Number Encoder/ (ecoder Parameters
Number Frncoder Decoder Parameters
Figure 18: Chinese Ennglss scaling law fits on log-log scale.
25

---

Published as a conference paper at ICLR 2022
WMT newstest2019 orig), 99.7. [L [k- 0,006
newstest2019 (tgt 99.7. -- 0.008
1.30
1.525
Encoder Scaling
Encoder Scaling
1.500
Decoder Scaling
1.25
Decoder Scaling
1.475
Jointly Fitted Scaling Law
Lozz
Jointly Fitied Scaling Law
1.450
1.20
1.400
1.425
1.15
1.375
1.10
1.350
Total Number of Parameters (Excluding Embeddings)
Total Number Parametors (Excludig Embediingg)
Web Domain (see orig), Rt: 99.3, II p0-
Web Domain lgg orig), 99 JI [- 0.006
1.500
1.475
Enceder Scaing
Decoder Scaling
1.45
Decoder Scaling
Frcader Scaling
1.425
1.450
Jointly Fitted Sceling
Loos
jointly Sceling
1.400
2.35
1.375
1.30
1.350
1.325
1.25
1.300
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1.0
1.4
Total Number of Parameters (Excluding Embeddings)
Total Number of Farometers (Excluding Embeddings)
News Domain tarc orig). A 99.4. L Qi: 0.006
News Domain 2 (bgt orig). R²: 99.9. JL Li. 0.005
1.34
Encoder Scaling
1.50
Encoder Scaling
Decoder Scaling
Decodeer Scaling
1.32
Jointly Fizted Scaling Law
1.45
Jointly Fitted Scaling Law
1:30
1.40
1.20
1.26
1.35
1.24
1.30
1.25
0.2
0.4
0.8
0.8
1.0
0.2
0.4
0.6
0.8
1.0
3.4
Total Number of Parameners (Escluding #mbeddings)
100
Total Number Parameters (Fxc Embbediing))
1e3
Domain origi, v p 0.005
ligh orig), R²: 100.0, I
1.375
Encoder Seainn
1.95
Erceder Scaling
1.350
Decoder Scaling
1.50
Decoder Scaling
1.325
Jointly Fitted Scaling
jointly Fitted Scaling Law
1.300
1.80
1.275
1.75
1.250
1.225
1.70
1.200
1665
Total Number of Parameters [Excluding Embeddings)
Total Number Parametors (Exclding Embedoings)
Web Domain 3 (bot origl, 99.9. IL- -L- -10000000 0.006
Web Domain (tgt orig), 87: 99.9, [L 0.000
0.004
Encoder Scaling
Encoder Scaling
2.20
Decoder Scaling
1.45
Decoder Scaling
2.15
Jointly Fitted Scaling Law
Jointly Fitied Scaling Law
2.40
2.05
Tert
2.00
1.35
1.35
1.30
1.90
0.2
0.4
0.8
0.8
1.0
1.4
0.2
0.4
0.6
0.8
1.0
1.4
Total Number of Parameters (Escliding #mbeddings)
Total Number of Parameters (F ludig Embeddingg)
Web Domain 5 (sre orig), &*: 99.0, Ji - ij 0.009
Web Domain (sre orig), g²: 98.4, -i 0.007
Scaling
1.22
Scaling Decoder Scaling
1.42
Decoder Scaling
1.40
Jointly Fitted Scaling Law
jointly Fitted Scaling Law
1.18
1.36
2.16
1.34
1.14
1.30
0.4
0.8
0.4
1.0
1.4
Total Number of Parameters [Excluding Embeddings)
b00
Total Number Parameters (Excluding Embeddings)
TED (mixed), 99.8, [L- Qp. 0.005
Train Data Out (mixed), A?: 99.8. pLLL 0.004
2.5
Encoder Scaling
Encoder Scaling
Decoder Scaling
Decoder Scaling
Test Lo05
2.4
Jointly Fitted Scaling
Jointly Scaling
2.3
0.95
0.50
0.4
0.8
0.8
1.2
0.4
0.6
1.0
14
Total Number Parameters (Facluding Embeddiigs)
Total Number Parameters (F uuting Embredding))
Figure 19: English scaling law fits.
26

---

Published as a conference paper at ICLR 2022
WHT newstest2019
99:9977 0.000.006
newstest2019
19.7, 0.009
Encoder Scaling
Encoder Scaling
0.181
Decoder Scaling
0.264
Decoder Scaling
0.126
Jointly Fitted Scaling Law
0.209
Joindy Fitted Scaling Law
0.008
0.061
0.130
0.000
0.081
0:074
Number Encoder/ Decoder Parameters
Number Encoder Decodar Parameters
0.252
Domain (sre origi, Rt: 99.3. g - 0.009
0.411
Web
(igr origi, 99.8. P - 0.006
Ercoder Scaling
Encoder Scaing
0.277
Decoder Scaling
0.342
Decoder Scaling
0.125
Jointly Scaling
0.283
Scaling
0.088
0.35
0.043
0.161
0.031
0.035
0.074
0.156
0,326
0.683
1,430
0.035
0.074
0.156
0.326
0.683
1.430
Number of Encoder/ Decoder Parameters
Number of Encoder Decoder Parameters
News Domain 2 (arc orig), 8²: 99.4, IL- <- 0.006
News Domain (tgt orig). R 99.9, 9, IL [ 0.005
0.175
0.402
Encoder Scaling
Encoder Scaling Jointly Fitted Scaling Law
0.120
Decoder Scaling
0.317
0.083
Jointly Fitted Scaling Law
0.50
0.057
0.039
0.155
0.027
0.018
0.086
0.035
0.074
0.156
D. 1.326
0.683
1,490 tea
0.035
0.074
0.156
0.326
0.683
1,430 teb
Number of Enceder/ / Decoder Parametiess
Number of Ercoder Decoder Parameters
Witipedia orig),
origi, P i 0.002
Ercoder Scaling
Encoder Scaing
0.197
Decoder Staling
0.435
Decoder Scaling
jointly Fitted Scaling
Fitted Scaling Law
0.112
0.288
0.084
0.234
8.063
0.047
0.683 Encoder/ Decoder Parameters
0.155
0.074
Parameters
Web Domain 3 (tat orig). R 99.9, 9, 14 Q. 0.006
Web Domain 4 (tot orial, R?: 99.9. JL Li- 0.004
0.533
0.345
Encoder Scaling Decoder Scaling
Encoder Scaling
Decoder Scaling
0.308
Jointly Scaling
0.234
Jointly Scaling Law
0.234
0.192
0.178
0.135
0.130
0.103
0.035
0.074
0.156
D.326
0.683
1,430
0.035
0.074
0.156
0.326
0.683
1.430
Number of Encoder / Decoder Parameters
Number of Ercoder Decoder Parameters
Web Domain S ferc orig), R²: 99.0, J - -i 0.009
Web Domain (sre orig), R², 98.4, -i 0.007
0.189
0.344
Focader Decoder Scaling Jointly ritted Scaling Law
0.140
0.095
0.103
Loss
0.062
0.077
0.041
0.057
0.027
Encoder Scaling
Jointly Fitied Staling Law
0.031
0.013
0.035
0.074
0.156
0.326
0.683
1,430
0.035
0.074
0.156
1.430
Number Enceder Decoder Parameters
Number Encoder Decoder Parameters
(mixed), 39.9. JL- 0.006
Data Out Imixed), 8²: IL Q
0.570
0.565
Encoder Scaling Decoder Scaling
Encoder Scaling
0.397
0.509
Decoder Scaling
Jointly Scaling
Fitted Scaling
Test
0.193
0.413
0.094
0.335
0.065
0.301
0.035
0.074
0.156
0.378
3,430
0.035
0.074
0.156
0.326
0.683
1.430
Number Encoder/ (ecoder Parameters
Number Frcodde Decoder Parameters
Figure 20: English->Chiness scaling law fits on log-log scale.
27

---

Published as a conference paper at ICLR 2022
E QUANTIFYING THE RANDOM VARIATIONS IN THE RESULTS
Note that the final test loss achieved by the model a random quantity. Randomness incorporated into the training pipeline through the initialization step, data order, and hardware failures/ preemptions. To quantify the magnitude of the fluctuations caused by this randomness, we retrain a subset of our models (2L-2L, 3L-3L, 4L-4L, 5L-5L, 8L-8L. 16L-16L, 24L-24L, and 32L-32L) with different seeds. Figure 22 standard deviation (left) and maximum difference (right) of the final test loss values observed for each model.
lenn Domaii our ang)
Ranartte Damann (mboed)
Web Domain Ogr ent)
0.030
Useo Daman 3 ters ong)
(MM1 (AMSAAAA200 lars ong
Sniippolia Dornan bes, snu!
reweten2005 fipt ong)
Wen Derann ong!
trieedt
0.010
0.025
Standard Deviatio
0.008
0.000
0.006
0.004
0.010
0.005
0.000
0.000
2121
31.31
4L4L
34.11
BLBL
18LTOL 24L24L 32L32L
21.1
31.11
AL4L
51S1
88.00
30L1OL 24124L 32U32L
Models
Models
Figure 21: Variability of the final test loss across four different seeds.
Table 4: Variability final test loss for each test dataset (averaged over all models).
Dataset
Average Standard Deviation
Average Maximum Deviation
Web Domain (src orig)
0.0030
0.0078
Web Domain (tgt orig)
0.0030
0.0077
WMT newstest2019 (sre orig) WMT newstest2019 (tgt orig)
0.0027
0.0067
0.0028
0.0071
Web Domain (src orig)
0.0024
0.0062
Web Domain (tgt orig)
0.0030
0.0076
Patents Domain (mixed)
0.0021
0.0053
Web Domain (src orig)
0.0037
0.0094
Wikipedia Domain (sre orig)
0.0057
0.0146
Web Domain (sre orig)
0.0026
0.0066
Train Data Subset (mixed)
0.0026
0.0067
Training Loss
0.0025
0.0064
28

---

Published as a conference paper at ICLR 2022
F RANDOMLY SAMPLED MODELS
To ensure that our scaling laws are not biased towards a particular class of models, we test their out-of-sample predictions 10 models where the number of encoder and decoder layers are decided randomly. We use the following code to generate randomly sampled models:
random. r (99)
decoders np.random. randint (3, 40, size=10) encoders np. random, randint (3, 40, size=10)
Table 5 lists the parameter allocation of these randomly sampled models.
Table 5: Parameter decomposition of the randomly sampled models trained for English translation task. The total number of parameters includes 98M parameters representing the softmax and embedding layers.
Encoder
Decoder
Layers
Parameters
Layers
Parameters
Total Parameters
5
105M
12
302M
506M
11
231M
8
202M
531M
15
315M
7
176M
590M
23
483M
4
101M
682M
3
63M
21
529M
690M
32
672M
4
101M
871M
39
819M
4
101M
1018M
18
378M
26
655M
1131M
26
546M
38
957M
1601M
30
630M
38
957M
1685M
29

---

Published as a conference paper at ICLR 2022
Domain orig). Com1L..)00.993, JL
:00007
onig). Comit, 0.999, L -o--00000000
Sample)
Predicted Test Loss (Out-df-Sanpla
1.00
x-w line
x line
2
0.96
0.94
0.86
0.92
0.84
0.90
0.82
0.90
0.92
0.96
0.98
1.00
0.84
0.86
0.88
0.90
0.92
Empirical Test
Empinical Test
WMT newstest2019 (src origi. Corni. O) 0.990.
01:0000
WMT newstest2011 origl. ContL, O 998, 0 0.005
(Out Sample)
Predicted Test Loss (Out-ot-Sample)
0.90
28
x-y
1.00
x-y
28
0.88
0.86
0.92
0.84
0.90
0.82
0.88
0.86
0.82
0.84
0.86
0.88
0.90
0.88
0.90
0.92
0.94
0.96
0.98
1.00
Empirical Test Loss
Empinical Test Loss
Web Domain 2 1src orig). Cornil, DI: 0.991, BL
- 0.005
Web Domain (tgt.orig). orig). Corit., Q): 0.997, J - 0.006
0.83
Prodicted Test LOSS (Dut-of-Samplee
0.98
0.82
28
E
0.96
0.81
0.80
0.94
0.79
0.78
0.77
0.88
0.76
0.86
0.76
0.77
0.78
0.80
0.81
0.82
0.88
0.90
0.92
0.94
0.96
0.98
Empirical Test Loss
Empirical Test Loss
0.68
Petents Domain (mixed]) CorriL. 0)99.9,,
0.004
Predicted Test LoSS (Dut-af-Sampl) Samplll
Web Domain (src orig), CorriL, 0.991, |L
0.009
line
1.10
line 26
0.67
25
3.08
0.66
2.06
0.65
1.04
0.64
1.02
0.63
1.00
0.62
0.98
0.96
1.02
1.10
Empirical Loss
Empincal
Wikipedia orig). Cortt., (1:0.99 -.. 10.013
Domain orig), Corrtl. O 0.986, J fT. 000007
Predicted Test Loss ao--Samlle
1.275
x-y line
Prodicted Test Loss (Out-of-Sangle)
0.92
x-y line
1.250
[+ 26
[=
1.225
0.90
1.200
0.88
1.175
3.125
1.100
1.100
1.125
1.150
1.175
1.200
1.225
1.250
0.34
0.85
0.86
0.87
0.88
0.89
0.90
0.91
0.92
Empirical Test
Empirical Tost
Figure 22: Comparison the out-of-sample predictions the scaling law with the empirical test loss values from the randomly sampled Anglish->Germa models.
30

---

Published as a conference paper at ICLR 2022
G PROOFS
G.1 PROOF OF PROPOSITION
Proof.
The optimal encoder decoder sizes are optimal parameters of the following optimization
problem:
minimize No.Na
a. Po Pd Ne + Na S
(2)
s.t.
To convert the problem to a convex problem, we instead consider the log objective and adopt the following change of variables:
= Log((N)),
w= log((N)).
(3)
The transformed optimization problem is the form:
minimize
P.u P&u exp(u) exp(u) - B
(4)
s.t.
Note that 4 is now convex and therefore, we can use KKT conditions to solve for the optimum. The Lagrangian has the form:
C(u, A) -P P pau + 2 B exp(() -ex((e)
(5)
Solving for the first-order conditions yield:
De (exp(u")
(6)
Pp Xexp(u").
Since the constraint is binding, À * 0. Therefore, can divide both sides of the equations above which yields:
Pe exp(u") N
(8)
Pa exp(1**)
N
Substituting
8 the constraint yields:
Pe B,
Pd B.
N
N
Pe + Pd
Pe + Pd
Finally, we substitute
( in the scaling law which yields:
Teopt B =
+ Pa
Pe
Pe + Pa
Pa
+ Tro
(10)
B
PaB
+
Po
Pe + Pa
Pa
B ppet Loe
(11)
Pe
Pa
31

---

Published as a conference paper at ICLR 2022
H OTHER SCALING APPROACHES
In the main text, we focus our empirical analysis on depth scaling we increase the depth of the encoder or the decoder while keeping the model dimension (1024), the width of the feed-forward layer (8196), and the number of attention heads (16) fixed. In this section, we examine the role of the scaling approach in our results. We consider the following scaling approaches:
Width Scaling: The number of encoder/decoder layers is fixed six. The model dimension increases from 768 all the way to 4096.
Depth Scaling- Wide: The number of encoder/decoder layers changes from 3 the way to 32. Model dimension is fixed at 2048.
Depth Scaling Narrow: The number of encoder/decoder layers changes from 3 all the way to 32. Model dimension fixed at 768.
Uniform Scaling: Model depth, model dimension, feed-forward layer size, and number of attention heads are increase in tandem together.
Similar to the models the main text, we use a dropout of 0.1 for residuals, feed-forward activations and attention. Models are trained with label smoothing of magnitude 0.1. To improve the training stability, all models use logit clipping of 10. We use Adafactor (Shazeer & Stern 2018) optimizer for training models. We use 40k linear warm-up steps and an inverse square root learning rate schedule. For Adafactor we used momentum with 0.9 and factored second moment to save memory.
Table provides detailed description of the models used in this analysis. In total, consider 22 models with sizes ranging from 170M parameters all the way to nearly 4B parameters. For the sake of computational tractability, only focus on symmetrically scaled models.
We examine how much the scaling behavior of these models deviates from the scaling trends computed in the main text. Figure 23 presents the percentage deviation of each model from the scaling law, computed as
Empirical Loss Predicted Loss
Percentage Deviation
100.
(12)
Empirical Loss
The results suggest that the different scaling approaches seem to yield very similar scaling behaviors. In fact, for most of our models, the deviation from the scaling laws of the main text are limited to 1- percents. This suggests that the phenomena identified in the main text are robust to the scaling approach.
The results of Figure parameters) the deviations from the scaling law seem to grow. We believe two factors contribute to this phenomena: (i) Model shape plays key role in the final performance of small scale models. The influence of model shape diminishes as model size increases. (ii) Our scaling laws are parametrized focus on the large model regime. Given that our scaling law singular when either N, Na equal to zero, we do not anticipate getting high accuracy predictions from them the small model regime.
2 suggest that as we move to smaller models (less than 100M non-embedding
32

---

Published as a conference paper at ICLR 2022
Table 6: Description of the models used for alternative scaling.
Model Class
#Layers
Model Dim
FF Width
# Heads
Total Parameters
Width Scaling
6L6L
768
8192
16
267M
Width Scaling
6L6L
1024
8192
16
375M
Width Scaling
6L6L
1536
8192
16
620M
Width Scaling
6L6L
2048
8192
16
902M
Width Scaling
6L6L
2560
8192
16
1221M
Width Scaling
61.61.
4096
8192
16
2407M
Depth Scaling- -Wide
3L3L
2048
8192
16
549M
Depth Scaling-Wir Wide
6L61.
2048
8192
16
902M
Depth Scaling-Wick
8L8L
2048
8192
16
1137M
Depth Scaling-Wiie Wide
I2L12L
2048
8192
16
1607M
Depth Scaling-Widd
16L16L
2048
8192
16
2077M
Depth Scaling-Wii Wide
24L24L
2048
8192
16
3016M
Depth Scaling-Wide Wide
32L32L
2048
8192
16
3957M
Depth Scaling-Narrow
3L3L
2048
8192
16
171M
Depth Scaling-Narron
61.61.
2048
8192
16
267M
Depth Scaling-Narrov
8L8L
2048
8192
16
332M
Depth Scaling-Narrow
12L12L
2048
8192
16
461M
Depth Scaling-Narrow
16L16L
2048
8192
16
590M
Depth Scaling-Narron
24L24L
2048
8192
16
848M
Depth Scaling
Scaling-Narrow Narrow
32L32L
2048
8192
16
1107M
Uniform Scaling
6L6L
768
3072
12
173M
Uniform Scaling
12L12L
1024
4096
16
451M
Uniform Scaling
12L.11
1536
6144
16
941M
Uniform Scaling
12L12L
2048
8192
24
1604M
Uniform Scaling
16L16L
2560
10240
32
3183M
33

---

Published as a conference paper at ICLR 2022
orig)
Hitt org)
Scaling (lamoo)) Depin Stareo mekee wodth Scaiig
Depth Scaling (Namow)) Depen Soaing (wee) Wort Scaviy
0.603
2,308
Total
(iko-Smbbeddingl
'9
WMT remstes12019 juc orig)
WKT rewntes/201 orig)
Depth Sorto (aarow
Depth soaro (amoow)
Unittor
Uniferm
Percertoge
0011
0.158
0.309
0.603
1.100
2,508
4.515
0.081
0.158
0.309
0.603
1.100
2,308
4.515
Total Number of (Non Embeddingi Parameters
Total Number of Ilton (ooo--mmedding Parameters
orig)
Higt org)
Soare Depth Scaling (Mec) Widn Seaing
Eipth Scaling (Miie)
Wides Seuling
Seaking
Sealing
Forcntge
-3
2,508
0.309
0.603
2,308
Embeddingl
/MMo-Emeeddttgg
Patents Domain (mixed)
Espth Sciling (Narow
Depo: Scdling (Harow)) Dept: Scaling (Mial
Scaling
Pencentgp
0.158
000
1.180
2,308
0.603
1.180
2.308
Total Number of (Non- Embe-diggnnn Parameters
Total Number of (Noo-Embedding) Parameters
Willpedls Domain forc orig)
Web Domain tsrc orig)
Staleo
(Mer)
Deviston
withs Keaingg
Uniorm Scaling
with Sesing
Unioor Scaling
Parcentae
088
0.158
0:309
0. 6.003
1.360
2,308
4.515
0.081
0.158
0,309
1.180
2.308
4.515
Total INDo Enbedddndddggg Parameters
(Non- (ooo-Embbddinn) Parameters
Figure 23: Percentage deviation from the scaling laws fitted the English-te tosssh-t-Gaar German models presented in the main text. Different scaling approaches seem to yield similar results; for the majority of our models, different scaling approaches deviate only - 2% from the scaling trends presented in the main text. Deviation from the trend increases for smaller models suggesting that model shape plays an important role in the performance small scale models. Models that experienced severe training instabilities/ NaNs during training removed from the plots.
34

---

Published as a conference paper at ICLR 2022
I CURVE FITTING DETAILS
We use ec.pt..pt imi mize. least_squee susares function for curve fitting throughout this paper have some robustness to outliers, we use the loss= 'soft_1l' option which is popular option for robust regression. The code snippet below shows the exact arguments we use for fitting the scaling laws:
T0
To
def (unc X, y): """Fitting a bivariate scaling law.
p: A 1-D array of dim 4, correspanding alpha, pLe, PLd, c.
x: A matrix dimension n \times 2. First column encoder params, second col decoder params.
A 1-D array of log-ppix of dim n. "." """
ba_ xl:, 0]
x_d ND_bar xi:, 1]
return p[0] np.power (x_e P(I]) nower (x_d p(21) P[3] -
def fit model (x, f_scale): X x.to s.to__umpy(())) ()
y y.to t)mpy().coppy if np. mp.isnan(X) any or np.isnan(y) (y). any ():
raise ValueError (' Data contains NaNs
if len sy. 1 s. != X. [0]p
raise ValueError Error in shapes')
p0 = ne.zeros| (4,)) (4,))
p0[0] 0.2 # alpha
p0111 0.4 p_e
p0121 0.6 p_d
p0[3] 1.0
fit least_squares (func, p0, soft_"soft_l1 f_scale-f_scale
args- (X, y), max_nfev=10000, bounds- (0, 10))
return fit
The 'soft_11 11' loss chosen above applies & penalty on small residuals and a -like penalty on outlier residuals. The argument f_scale determines the boundary where the transition between the two different behaviors occur. For the results presented in this paper, we choose f_scale from the grid given np geomspace (0. 001, 0. 025, num=25). Choosing = s_s_clll000 025 effectively yields least-squares regression while smaller values add more robustness to outliers.
"nttps://docs. http:///wttttttoooooo/////////ss ze.
ceast_squares..
35

---

Published as a conference paper at ICLR 2022
J ANALYSIS OF THE GENERATION QUALITY
Decoding:
As described in Section we use beam-search for decoding (Wu et al. 2016). To keep the hyper-parameters of beam-search for each
the experiments tractable, did not attempt (optimized for the baseline model) for all our model. Instead, we use the same hyper-parameter decoding jobs. particular, fix the length normalization parameter 1.0 and number of beams to 4.
BLEU-Cross Entropy Loss Co-Evolution: Figure 24 presents the relationship between BLEU score and cross-entropy loss for various test datasets. The results closely mimic the phenomenon observed in Figure 8 On target original data, improvements to cross-entropy loss are accompanied with improvements BLEU score. On source original data however, beyond certain point, cross- entropy loss and BLEU score exhibit diverging behaviors.
We observe that in large well-trained models, the relationship hetween BLEU and cross-entropy loss on target-original data is well captured by a power law of the form
BLEU = c Pa
CB. PB > 0.
(13)
The fit achieved by this power law is plotted in figures. We observe that fitted power laws for encoder scaling models consistently attain larger exponents compared to decoder or symmetrically scaled models. This reflects the fact that encoder scaling models are more successful in improving the generation quality (as measured by BLEU).
Finally, we observe number of deviations from the predictions Eq. (3) particular, models
with shallow decoders (6L2L, 6L4L, 6L6L) seem to outperform the trend (Figure 25). Moreover, we observe that models in the beginning of the training process tend deviate from the overall trend (Figure 26). We postpone an in-depth analysis of these phenomena to future work.
36

---

Published as a conference paper at ICLR 2022
BLEU Score vs Cross-Entropy Loss for Engllsh-to-German Task
WMT newstest2019
WMT newctest2019 orig)
0.41
Symmetric Scaling
Symmetric Scaling
0.41
Decoder Scaling
Frcoder Sealing
0.40
Docoder Scolinn
Focoder Stwing
Fitted Power-Law
0.39
0.38
Test BLEU
0.38
0.36
Test
0.37
0.34
0.37
0.36
0.92
1.03
1.15
0.80
0.85
0. 0.90
0.96
1.02
1.08
Test Cross- Entropy Lass
Test Cross Entropy Loss
Web Domein (igt orig)
Web Domain 2 (src orig)
0.41
Symmetric Scaling Encoder scaling
0.38
0.40
Score
Fitted Power Law
0.39
0.36
Test BLEU
0.39
0.34
0.38
Symmetric Scaiing
0.32
Encoder Scaling
Decoder Sceling
0.37
Test Cross-Entropy Loss
Test Cross Entropy Loss
BLEU Cross-Entropy
German-to-English Task
WMT newstes(2019 (tgt orig)
WMT newstest2019 (src orig)
0.492
0.440
Encoder Scaling
Enceder Scaling
0.476
Decoder Scaling
0.433
Decoderr Scalin
Test BUEU Score
0.460
Test BLEU
0.445
0.418
0.431
0.411
0.417
0.403
0.39%.99
0.98
1.14 Test Cross-Entropy Loss
1.34
1.10
1.22
1.28
Test Cross-Entropy loss
Domain (tgt orig)
Domain (src orig)
0.506
0.416
Encoder Sceling
Encuder Staling
Decoder Scaling
Decoder Scaling
Text BLEU Score
Fitted Power-Lav
Telt BLEU Score S
0.482
0.407
0.471
0.402
0.461
0.397
0.450
0.392
0.440
0.77
0.81
0.86
0.92
0.97
1.03
0.387
0.96
1.00
1.04
1.08
1.13
1.18
Test Cross-Entropy Loss
Test Cross-Entropy Loss
Figure 24: Log-log plot of the evolution of BLEU score function of cross-entropy loss for different models. For each scaling approach, warmer colors represent larger models. Each individual color represents different checkpoints of a single model during training. On target original data (left column), improvements to cross-entropy loss lead to consistent improvements in BLEU score. Dashed lines correspond fit achieved by Eq. (13). The relationship breaks down for source original data (right column).
37

---

Published as a conference paper at ICLR 2022
0.433
Web Domain 1 (tgt origg
0.410
WMT newstest2019 itat orig)
0.300
Domain (tgt arigl
0.414
Fitted Powen
0.389
Power Law
0.366
Power
Test BUEU Scone
Test BUEU BIEU Score
0.395
0.368
0.346
0.377
BLEU
0.349
0.326
0.360
0.330
0.300
0.344
0.313
0.291
0.328
0.296
0.275
0.83
0.91
1.00
1.09
1.19
1.31
0.87
0.97
3.09
1.22
1.37
1.58
0.87
0.95
1.04
1.24
1.25
1.36
Test Cross Entropy Loss
Test Cross-Enyyy Loss
Test Cross- Entropy Loss
0.459
Fn Web Domain orig)
0.486
Fr. WMT newstest2019 ltgt orig)
0.501
Do
Web Domain orig!
0.447
Fided
0.468
0.488
Test BLEU Score
Test BLEU Score
0.436
0.452
0.475
0.426
BLEU
0.435
0.452
0.415
0.420
0.405
0.405
0.438
0.308
0.92
0.99
1.06
1.14
1.23
1.32
0.390
0.90
0.99
1:08
1.19
1.30
1.42
0.426
0.77
0.82
0.88
0.94
1.01
1.09
Test Cross Entropy Loss
Test Cross Cross-Entropy Loss
Test Cross Entropy Loss
Figure 25: Models with shallow decoders tend to outperform predictions of Eq. dark green color represent different checkpoints of 6L2L, 6L4L, and 696 models.
13 Points with
En- De Web Domain (ig orig)
En De WHT newstest2019 (tgl orig)
En De Web Domain 2 (Egt orig)
0.440
0.423
0.394
Treader Sealing
Enceder Scaling
Enceder Scaling
0.421
sted Poasend an
0.399
Fitted Powser AM
0.373
Fitted Fowerdan
Test BIEU Soore
Test BLEU Spore
0.403
0.376
0.353
0.386
0.354
0.334
0.360
Text
0.334
0.353
0.315
0.300
0.338
0.851
0.920
0.994
1.074 1.161
1255
0.298
0.901
0.990
1 1.088
1.195
1.313 1.443
0 0288
0.893
0.964
1001
1.125 1215
1.312
Test Cross Entropy Loss
Test Cross Entropy Loss
Test Cross- Entropy Loss
En De Web Domain (ig orig)
En - De WHT newstes12099 (tgt orig)
En De Web Domain 2 (Egt orig)
0.479
0.460
0.405
becode Sewing
Decoder Sexing
Decader Seating
0.317
thad Poaserd mn:
0.287
Fited Powsr av
0.325
Fitted Fowerd AM
Test BLEU Sconee
Spore
0.210
0.179
0.200
0.139
0.112
0.209
0.092
0.070
0.061
0.084
0.134
0.040
0.782
1.122
1.610
2.311
3,318
4.702
0.027
0.821
1.193
1.725
2.522
3.067
5.331
0.108
0.821
1.369
1.604
2,369
3.373
4,802
Test Cross Entropy Loss
Test Cross Entropy Loss
Test Crost-Entropp Entropy Loss
Figure 26: On some of the test sets, data points corresponding to early training checkpoints exhibit deviations from the overall trend.
38

---

Published as a conference paper at ICLR 2022
Decoder Scaling: Web Domain 1 (src orig)
Encoder Scaling: Web Domain 1 (src orig)
0.345
0.345
0.339
0.339
BLEU Score
BLEU Score
0.332
0.332
0.326
0.326
0.320
0.320
0.314
0.314
Training Steps
1c5
Training Steps
1e5
Decoder Scaling: Web Domain (tgt orig)
Encoder Scaling: Web Domain (tgt orig)
0.435
0.435
0.415
0.415
BLEU Score
BLEU Score
0.396
0.396
0.378
0.378
0.361
0.361
0.344
Training Steps
1e5
Training Steps
1e5
Figure 27: The evolution of BLEU score during the training for English-to-German Web Domain test sets. Warmer colors correspond to larger models. Top row: On source original test data, our largest models achieve lower BLEU scores compared to mid-sized models throughout the training. Bottom row: On target original test data, increasing the model size yields consistent improvements in BLEU score throughout the training.
39

---

Published as a conference paper at ICLR 2022
Decoder Scaling: Web Domain (src orig)
Encoder Scaling: Web Domain 1 (src orig)
0.430
0.430
0.423
0.423
BLEU Score
BLEU Score
0.415
0.415
0.408
0.408
0.401
0.401
0.394
0.394
Training Steps
1c5
Training Steps
1e5
Decoder Scaling: Web Domain (tgt orig)
Encoder Scaling: Web Domain (tgt orig)
0.465
0.465
0.450
0.450
BLEU Score
BLEU Score
0.435
0.435
0.421
0.421
0.407
0.407
0.394
0.394
Training Steps
1e5
Training Steps
1e5
Figure 28: The evolution of BLEU score during the training for German-to-English Web Domain test sets. Warmer colors correspond to larger models. Top row: On source original test data, our largest models achieve lower BLEU scores compared to mid-sized models throughout the training. Bottom row: On target original test data, increasing the model size yields consistent improvements in BLEU score throughout the training.
40

---

Published as a conference paper at ICLR 2022
K
BLEURT SCORES
En -a De Web Domain 1 (ggt orig)
En - De Web Domalin (src
Scaling
Symmutric Scaling
0.600
Scwiig
Test BLEURT Score
0.575
Decotlr Scaling
Test BLEURT Score
Decoder Scaling
0.550
0.525
0.500
0.475
0.450
Test Cross-Entropy
Pross-Entropy Loss
De En Web Domain (tgt orig)
De En Web Domain orig)
Encoder Scalin
Encoder
0.55
Test BLOURT Scote
Score
0.44
0.54
0.53
0.43
0.52
0.42
0.51
0.50
0.41
Test Cross-Entropy Loss
Test Cross-Entropy Loss
Figure 29: The evolution of BLEURT score as a function of cross-entropy loss different models.
For each scaling approach, warmer colors represent larger models. Each individual color represents different checkpoints of single model during training. On target original data (left column), improvements to cross-entropy loss lead to consistent improvements in BLEURT score. This relationship breaks down for source original data (right column).
41

---

Published as a conference paper at ICLR 2022
WMT newstest2019 orig)
newstest2019 orig)
Symmetric Staling
Symmetric Sealing
0.50
Decocer Staling
Enuuder Scaling
Test BLEURT Score
0.05
Decoder Scaling
Erxode Scaling
0.45
Test BLEURT
0.00
-0.05
0.35
0.30
-0.10
0.9
1.0
1.1
1.2
1.3
0.80
0.85
0.90
0.95
1.00
1.05
1.10
Cross-Entropy orig)
Eross-Entropy
orig)
0.600
Symmetric Scaling
0.36
Symmetric Scaling
0.575
Encoder Scaling
Brooder Scaling
Test DLEURT Score
Decocer Sca ing
BLEURT Score
Decoder Scaling
0.550
0.34
0.525
0.32
0.475
0.450
0.425
0.28
0.85
1.10
1.15
1.20
0.775
0.800
0.850
0.900 0.525 0.950
Entropy
Enoss-Entrooy
De - En WMT newstest2019 ltgt orig)
De - En WMT newstest2019 (src origl
0.54
Encoder Scaling
Encoder Scaling
Decader Scaling
0.36
Decocer Sta
Test BLEURT Score
0.52
Test BLEURT Score
0.34
0.50
0.32
0.48
0.46
0.95
1.00
1.05
1.10
1.15
1.20
1.25
1.30
1.00
1.05
1.10
1.15
1.20
1.25
Test Cross-Entropy Loss
Test Cross- Eooss-Entropy Less
De -e Web Domain (tat orig]
De En Web Domain (sre eria)
Decader Scaling
Encader Scaling
0.435
Encoder Staling Decocer
0.60
0.59
0.425
Test BLEURT
0.58
0.420
0.415
0.57
0.410
0.405
0.400
0.395
0.50
0.85
0.90
0.95
1.00
0.975
1.000
1.025 1.050 1.075 1.100 1.125 1.150 1.175 Test Cross-Entrapy Loes
Test Cross-Entropy Loss
Figure 30: The evolution of BLEURT score as function of cross-entropy loss for different models. For each scaling approach, warmer colors represent larger models. Each individual color represents different checkpoints of single model during training. On target original data (left column), improvements to cross-entropy loss lead to consistent improvements in BLEURT score. This relationship breaks down for source original data (right column).
42


Reviews:
[{"review_id": "mGktNUxeZKt", "reviewer": "Reviewer_Dbxb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- I really enjoyed reading this paper. The research questions are clearly stated, and the results are fascinating to go through. Well-done!\n- The number of experiments is impressive. This adds weight to the papers conclusions, which are certainly thought-provoking. For example, the conclusion that scaling law should be bivariate, treating the number of encoder and decoder layers separately, makes me think we should not train models of symmetric number of layers anymore. \n\nCons: \n- The footnote on comparison to Gordon et. al. in EMNLP2021 is too terse and should be expanded. That paper studied scaling laws for NMT as well, but in a different parameter regime. Do your conclusions affirm, reject, or extend theirs? ", "summary_of_the_paper": "This paper studies scaling laws for NMT. It confirms and extends some of the existing work on scaling laws, and as such it is a valuable contribution to the field. In particular, new questions addressed include: does the encoder-decoder architecture of NMT share the same scaling law as decoder-only LMs? How does training data (backtranslated or not) affect scaling? ", "summary_of_the_review": "This is a strong empirical paper that extends our understanding of scaling laws in deep learning. The research questions are clear and the results are convincing. I think it will be cited often in the future. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "h0jFEMR35F", "reviewer": "Reviewer_TAXy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. They conduct extensive experiments on four language pairs and five different test domains.\n2. It is interesting that contrary to previous findings, this paper suggests that it is better to scale the decoder than the encoder, providing guidance for future research on deep NMT.\n3. The analysis on the translationese provides insights on the correlations between cross-entropy loss and generation quality, and its effect on the scaling laws.\n\nWeaknesses:\n1. The scaling laws have several test set-specific parameters, which makes it questionable if the laws have practical values. For example, if we want to get the optimal allocation of encoder/decoder capacity using their method, we have to train models of different variants and fit their proposed formula, so why don’t we just do parameter search with the same training budget?\n2. I appreciate that the authors make efforts to measure the outputs in both BLEU and BLEURT, but it would be better to include human evaluation results that can better reflect the generation quality of models.\n3. The paper mainly focuses on the relationship between cross-entropy loss and the number of Transformer layers in large-scale settings, while there are a lot of other factors that can be considered. For example, it is unclear whether the scaling laws will hold if the architecture varies or if the hidden size changes.\n", "summary_of_the_paper": "The paper provides scaling laws for machine translation models. They experiment on different language pairs and domains and focus on large-scale Transformer-based NMT models.\n\nThey find that the scaling behavior of cross-entropy loss can be described as a bivariate function of encoder and decoder size and make recommendations for optimal allocation of encoder/decoder capacity. They also find that “translationese” can affect the model scaling behavior. Finally, they study the correlations between cross-entropy loss and BLEU/BLEURT scores.", "summary_of_the_review": "While the paper has some minor issues, it conducts extensive experiments in large-scale settings and the results can provide insights for future research, thus I’m leaning towards an acceptance of the paper", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Zux0VK0nU_s", "reviewer": "Reviewer_afSG", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n- the research questions answered in this paper are interesting and help to better understand the mechanics of NMT. As an expert in MT, I really found this paper interesting.\n- findings are very insightful\n- this paper will be impactful: it will guide NMT  (or any other kind of encoder-decoder model such a BART) practictionners to build more efficient and effective models \n- another proof that target-original test sets are not suitable for NMT evaluation (Figure 8 should go on Twitter when you publish your paper!)\n- evaluation is not only performed with BLEU (BLEURT is also used)\n\n\nWeaknesses:\n- almost the entire evaluation is not reproducible: undisclosed datasets, use of a non-standard BLEU implementation, pre-processing of datasets not detailed, etc.\n- the paper misses a related work section (more details than in the introduction). The reader may not know related work on language models for instance", "summary_of_the_paper": "This paper presents a study on the scaling power of NMT Transformer (encoder-decoder).\n\nIt shows: \n- how the scaling of encoder-decoder compares to language model\n- that NMT behaves very differently on translationese texts \n- propose to model the scaling of NMT encoder-decoder\n", "summary_of_the_review": "All the \"strengths\" pointed above justifies my recommendation for the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "MbFNo-29jSI", "reviewer": "Reviewer_q9c2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The proposed function of model capacity for encoder-decoder NMT model performance is useful for parameter allocation of NMT models training.\n2. This paper provides experiments to explore how the composition of training/test data affects the test/training loss for scaling NMT models. It's an important question to connect the other two questions.\n3. It is important to investigate how generation quality evolves with the improvements in cross-entropy resulting from model scaling. The problem is of great value for both NMT practice and research.\n\n\nCons:\n1. The authors have proposed three research questions, while the relationship among them is not clear.\n2. The scaling property of the composition bias of the train/test sets has been less studied in previous work, and is most interesting to the reviewer. This part can be more convincing if the authors can provide more experiments and analyses.", "summary_of_the_paper": "The paper provides interesting scaling properties for autoregressive neural machine translation models, which are validated on comprehensive experiments. The authors investigate the scaling properties from three perspectives:\n1. Quantify the evolution of model quality as a function of model capacity for encoder-decoder NMT models.\n2. How the composition of training/test data affects the scaling behavior of NMT models?\n3. How generation quality evolves with the improvements in cross-entropy resulting from model scaling?", "summary_of_the_review": "The research questions explored in the study are important and interesting, e.g., the scaling properties of composition bias and the influence of source/target naturalness on the scaling behaviors. However, the relationship among these research questions and findings is not clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a strong empirical paper that studies scaling laws for NMT in terms of several new aspects, such as the model quality as a function of the encoder and decoder sizes, and how the composition of data affects scaling, etc. The extensive empricial results offer new insights to the questions and provide valuable guidance for future research on deep NMT. The datasets used in the study are non-public, which may make it hard to reproduce the evaluation.
Decision: Accept (Spotlight)