Data Scaling Laws in NMT: The Effect of Noise and Architecture
Authors: Yamini Bansal
Behrooz Ghorbani
Ankush Garg
Biao Zhang
Colin Cherry
Maxim Krikun
Behnam Neyshabur
Orhan Firat
Venue: ICLR 2022 Submitted (2022)
Keywords: Scaling laws
Neural Machine Translation

Abstract: In this work, we empirically study the data scaling properties of neural machine translation (NMT). We first establish that the test loss of encoder-decoder transformer models scales as a power law in the number of training samples, with a dependence on the model size. We then systematically vary various aspects of the training setup to understand how they impact the data scaling laws. In particular, we change the (1) Architecture and task setup, to a Transformer-LSTM Hybrid as well as a Decoder-only transformer with language modeling loss (2) Noise level in the training distribution, starting with noisy data with filtering applied as well as clean data corrupted with synthetic iid noise. In all the above cases, we find that the data scaling exponents are minimally impacted, suggesting that marginally worse architectures or training data quality can be compensated for by adding more data. Lastly, we find that changing the training distribution to use back-translated data instead of parallel data, can impact the scaling exponent.

Under review as a conference paper at ICLR 2022
DATA SCALING LAWS IN NMT: THE EFFECT OF NOISE AND ARCHITECTURE
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, empirically study the data scaling properties of neural machine translation (NMT). We first establish that the test loss of encoder-decoder trans- former models scales dependence on the model size. We then systematically vary various aspects of the training setup understand how they impact the data scaling laws. In par- ticular, we change the (1) Architecture and task setup, to Transformer-LSTM Hybrid as well as Decoder-only transformer with language modeling loss (2) Noise level in the training distribution, starting with noisy data with filtering ap- plied as well as clean data corrupted with synthetic iid noise. In all the above cases, we find that the data scaling exponents are minimally impacted, suggesting that marginally worse architectures or training data quality can be compensated for by adding more data. Lastly, find that changing the training distribution use
power law in the number of training samples, with
data instead of parallel data, can impact the scaling exponent.
INTRODUCTION
In deep learning, a reliable recipe to improve the generalization performance a model is to increase the amount of compute and data used to train it (Krizhevsky et al.] 2012 Brown et al. (020)). Many recent advances deep learning are directed towards increasing the efficiency through advancements in architecture (Vaswani al. 2017), improving the quality of the data with filtering, or incorporating entirely new sources of data into training (Radford et al. 2021 Chen et al., 2020). In this new paradigm of machine learning, it is crucial understand the efftaeefficency of our training methods, and whether improvements to them at small scale translate into improvements large scale.
digesting data
Recent work (Hestness et al. 2017 Rosenfeld et al. 2019 Kaplan et al. 2020) on scaling laws offers useful tool to answer such questions they show that the test loss of model scales predictably and smoothly as power law in the relevant quantities of interest such as dataset size (D), model size (N) and amount of compute (C'). We take these findings step further and ask how different interventions to the training setup impact the data scaling curves in the setup of neural machine translation (NMT):
How do changes to the training setup (such as architecture, noise) impact the data scaling curves in NMT?
Practically, data scaling laws can be leveraged to make experimental decisions for future large scale experiments, and decide where computational and research efforts should be focused. Currently, NMT models are trained using massive web-scale (uta;(Arivazhageee et al. (2019) used 25B training examples (approx. IT tokens) and with the advent of the self-supervised learning techniques (Liu et al. 2020 Raffel et al.) 2020, (Siddhant et al.) 2020) this number can easily reach 10+T tokens. At such large scales of data, it is unfeasible to "just perform the experiment" and scaling laws can be used to drive training decisions. For instance, if small changes in architecture do not lead to change in scaling exponent (as shown in Figure T3,, then architecture choice can be driven by other factors such computational efficiency by paying small penalty in the form of more data. In the opposite scenario, if a training setup performs worse at given dataset size but scales much better, it should be chosen for future larger scale experiments.

---

Under review as a conference paper at ICLR 2022
Changing model size Exponent = 0.251
Changing architecture Exponent 0.285
Changing Noise level Exponent 0.296
2.744
2.547
3.836
Layers Enc, Layers
Decoder-Decader
No notse
Decoder only
Source noise
Text lag-terlleity (t)
1.968
6 Enc, 6 Dec Layers
Enc. Dec Layers
Test log porplesityy (L)
Language modeling
Transtomer Encoder,
2,800
Target nelsz
Layers
LSTM Decoder
1.412
1,458
1.771
That tag-
1.013
1.103
1.207
A
0.727
Sentence pairs dataset (D)
10a
10'
10F
B
0.835
Sentence pairs dataset (D)
10'
0.822
Sentence pairs In dataset (D)
10"
10'
10
Data scaling exponent is minimally impacted by changes to the training set up We train series of models on increasing amounts of data (500 sentence pairs while (A) Changing the depth of the encoder-decoder transformer (B) Changing the architecture and task set- up (C) Changing the noise in the training distribution. For each figure, we fit the data scaling law similar Eq. | and find that a common exponent p provides good fits for the empirical observations. Full details in Section & 4 respectively.
Figure 1:
1.1 OUR CONTRIBUTIONS
In this work, we first establish that the test log-perplexity of encoder-decoder transformer models trained perform neural machine translation (NMT) follows a power law in the dataset size, with a dependence on the model size (Figure [A]. We demonstrate that our scaling law predicts experi- mental results over 3 orders of magnitude (from 500K-512M) training examples (sentence pairs in NMT) Then, we systematically vary aspects of the training setup to understand how these impact the scaling laws.
In particular, we consider the effect of (1) Changing architecture and task setup from a vanilla Trans- former SincoderDeccode to Transformer Enoder-LSTM Decoder Hybrid as well as a Decoder-only Transformer with language modeling objective (2) Changing the level of filtering in the dataset (3) Changing the amount of noise on the source (input) side and the target (output) side (4) Using back- translated data. Surprisingly, we find that, with the exception of back tran-translttion, these changes do not impact the scaling exponents much (See Fig. for example of changing model size, architecture and level of noise). Some of these changes affect the bias or the final loss value at infinite data.
Our work suggests that many of the common operations used to boost performance, such as small changes to the architecture or data filtering, can be replaced by adding additional constant factor of data, That is, in some cases, sub-optimalities the architectures and noise in the datasets can both be considered as (model and data) scaling penalties.
1.2 EXPERIMENTAL SETUP
Our experiments are conducted on pre-layer transformer networks (Xiong et al. 2020 Models are trained with per-token cross-entropy loss and Adafactor optimizer (Shazeer Stern 2018). All models are trained with fixed batch-size of 500K tokens and dropout rate of 0.1 for residuals, feed-forward activations and attention. For the small dataset sizes, the models are trained to early stopping (as measured on the log-perplexity of a held-out development set) and for large dataset sizes they are trained for up to 500K gradient steps. The hyperparameters for these models were optimized for 6 encoder layer and 6 decoder layer model trained on 2.2 billion sentence pairs. We train different model sizes (2L6L, 6L2L, 6L6L, 6L28L, 28L6L), where 28L6L means 28 encoder layers and 6 decoder layers.
Models:
We also train two decoder-only models with a language modeling loss with (9L. 13L), and three Hybrid-LSTM model with {6L21, 6L6L, 6L12L). All the hyperparameters are matched as closely as possible between these models to provide an apples-to-apples comparison.
Corresponding 27.6 billion tokens.
2

---

Under review as a conference paper at ICLR 2022
2.910
cting performance
Loss vs. BLEU score
0.2L
1004
5.749
OOD Test Loss
Tost perslstty (1)
2.131
Out sample
Layers
0.375
0.400
0.9L
2.5L
20L01
0.28L
Datnset millinns (D)
perpietyy
3.325
Vee Eemulr
Vee Eeruan
AeB Doman
bec Lapers
54
Train Onte Suttet
2.556
0.325
3.923
0.300
8
J6
2.136
0.275
1.112
0.225
A
0.829
Sentence dataset
10a
10'
B
1.0
Test log perppeelyyxityyyyooooomooooo.eee (L)
1.5
2,0
C
0.643
Sentence dataset
10*
10'
10'
Figure 2: (A) Using the joint data and model scaling law from Eq. 2 to predict the performance of previously unseen model. (B) BLEU score and test loss display an almost-linear relationship (C) Scaling law for different OOD test sets for a 6L6L model (Results plotted for Web Domain 1 test set.)
Training data our experiments, the models are trained on two large scale datasets. The first set of experiments on data size scaling are using the dataset sampled from an in-house parallel corpora containing up to 2.2B sentences translated from English to German. We sample training datasets of sizes (1M, 2M, 4M, 8M, 16M, 32M, 64M, 128M, 256M, 512M} independently to study the data scaling laws. The second set of experiments are conducted with Paracrawl dataset (Bañón et al.) 2020) and with and without filtering applied. The details are described in Section
Test data The model performance is measured on a held-out dataset from the training distribution. Additionally, also measure performance various uut-of-diitribution test datasets. These test datasets have different domain composition and sampling methods. The domains considered are (i) Web-Domain (ii) News-Domain (iii) Wikipedia and (iv) Patents. The news-domain test sets come from the WMT2019 evaluation campaign (newstest2019) for all language pairs. The other test sets internal test sets representing the different domains, ranging from 500 5000 sentence pairs for each set.
DATA SCALING LAWS
We begin our investigation by training a series of large scale encoder-decoder transformer models the in-house English to German parallel corpus, with the parameters ranging from 170M to 800M. Our dataset sizes range from 500K to 512M sentence pairs, covers 28B tokens, which is 10 times larger than the experiments conducted in prior work on scaling laws for NMT Gordon et al. (02TI. Our goal is to find function in the relevant variables that predicts test performance of our experiments. We will mainly focus on the test log- perpllextx on heldout dataset from the training distribution, but we will also discuss scaling of BLEU scores (Section 2. and performance other distrr-distiibution test sets (Section 2.2 later sections.
Form of the scaling law: The chosen scaling law must exhibit decreasing loss with dataset size D, and at infinite data D oo, the model must converge finite constant. Additionally, Kaplan al. (2020); Bahri et al.) (2021), conjecture that when D + the models are in a "variance-limiter regime and the loss should scale as Dl) These desiderata are satisfied by the following scaling law for fixed model size:
Do D +
where a is multiplicative constant, p is the scaling exponent and C' is a model-dependent constant that are fitted empirically (D, = 1e6 a fixed normalization constant use for numerical stability). We find that this scaling law indeed provides good fit for the experimental observations. Moreover, when we fit just the last few datapoints (See Figure 5) we find that the loss scales as O(1DD), thus confirming the conjecture even for large-scale supervised learning setup.
3

---

Under review as a conference paper at ICLR 2022
Now we change the model size (depth, width, depth--idth aspect ratio etc), it is not apriori ob- vious if these different models, say encoder-heavy vs. decoder- heavy models, should have the same data scaling parameters a. For instance, Ghorbani et al. (202T) observe that parameter scaling for encoder-decoder models depends separately on the number of encoder and number of decoder models. Thus, we train 5 different models {21.1.,, 61.21., 61.6I., 28L61., 61.281.) including very asymmetric ones and find that their scaling parameters a, P are in fact very similar (See Appendix A.I for individual fits for Equation to each model). Additionally, the constant C' must depend on the model size, since at infinite data D = oc different models will converge to different loss values aC", with larger models converging lower loss values. Figure IA shows the fit for common o, and 5 different [C; for each different model. We find that common exponent is sufficient to provide a good fit for the experimental data.
We can go step further and replace the constant C such that it recovers the parameter scaling curve D = oo. To do so, we leverage the parameter scaling law found by Ghorbani et al. (2021) and fit the following final scaling law:
Do + B No= No + Troo L(D, No, No) = ={D
1/0
(2)
The only free parameters this equation are a and p. and the parameters in the right term (3, No, Na, Ioo) are directly borrowed from Ghorbani et al. (022), such that they converge to the parameter scaling law at D = oc. The fit using this scaling law is shown in Figure D Since we have joint scaling law with dataset size and parameters, of models that were not used to fit the scaling law. For example, in Figure 24 we find a, p for all models while holding out the model with 6L22L and later are able to predict the performance of this out-of-sample model. Note that this scaling law differs from Gordon et al. (202T) in two ways (1) Using the encoder and decoder parameters separately vs. using the total number of parameters (2) An additional Lioe tern that remains even when the No, Na, xc.
can use this to predict the test loss
Implications: Equation 1 suggests that there exists two operating regimes for data scaling: (i) data- limited regime where Do
D >> C, and (ii) capacity limited regime where Do & C. Fitted exponents in Figure I Juggess that, the data limited regime, loss scales as O(D-1/4), suggesting marginal value of (D-5/44 for additional data. Increasing the model capacity in this regime has negligible impact on the loss. In the capacity limited regime however, the loss scales as O(D-1) suggesting a (significantly smaller) marginal value of O(D-2) for additional data. In this regime, the loss value is dominated by the model-dependent constant C' and most of the improvement can be had by increasing the model size. There is a smooth phase transition between these two regimes at approximately CD 1 (See Appendix C for illustration). Thus, by increasing the model size (which reduces C), one can push the transition to larger values of D and leverage the available data more efficiently.
2.1 BLEU SCORE
Language tasks can roughly be categorized into two groups understonding tasks where given piece of text is tasked to be encoded for downstream classification (eg. sentiment analysis, named entity recognition), and generation tasks where a representation of a piece of text is used (conditioned) to generate another arbitrary length sequence of text (eg. summarization, question answering). Ma- chine translation, without loss of generality, belongs to both of the categories: source content first needs to be understood and the corresponding target sequence must be generated. Given this cat- egorization, we not only care about the model score on reference target sequence (measured in log-perplexity) but we also care about the generation quality. Once sequences are generated from a sequence model, in autoregressive fashion or not, we resort to automatic measures like BLEU
Gordon al. (2021) use relatively smaller model sizes. At small Ne, No. the Tro term is dominated by the 1/N Ng term, which may explain why they did not need to use an additional Loo term
4

---

Under review as a conference paper at ICLR 2022
score
We find that the BLEU score and test log-perplexity have nearly linear relationship as shown in Figure 23. Note that this linear relationship is known to break down at very low loss values Ghorbani al. 202T.
2.2 OOD TEST DATASETS
Practically, we are also interested the performance of our models on out-of-distribution (OOD) test sets, outside the heldout set. These test sets are related to the training distribution but may have different support or composition. To understand how performance on such test sets scales with dataset size, we measure the performance on the test sets as described in Section 1.2 We find that the test loss on these test sets also follows a similar power law in the dataset size as Equation I Figure 2 shows the test loss fits for all the test sets for 6L6L model. We find that most of the test sets have similar scaling exponents (See Figure@0p for exact values). That is, we do not observe any major differences in scaling on the basis the test set composition and source-original test sets scale similarly. Additionally, since the different test sets have similar scaling exponents. this implies that the in-distribution loss and the out-of-distribution loss must have a nearly-linear relationship (See Figure (C). This is in line with previous findings in vision (Miller al. 202T). Why these distributions scale similarly (or why they have a linear relationship) still an open theoretical question.
both target-original
THE EFFECT OF ARCHITECTURE
Innovations in architecture have been fundamental to progress in deep learning. It is often suggested that architectures confer certain inductive biases that enable learning. However, recent trend in machine learning largely confirms the trade-off between architectural biases and the amounts of data, namely tendency towards architectures with low inductive biases being trained on large amounts of data, compute and parameters. Thus, we believe it is essential to determine whether these "low bias" architectures have similar scaling behavior with respect the dataset size.
To understand the relationship between the amounts of data and architectural bias, we pick three common architecture and loss set ups that commonly used for machine translation. We take an encoder-decoder transformer described in detail in Section as the baseline. Next, we pick a Hybrid architecture with a transformer encoder and an LSTM decoder (Chen al. 2018 due their wide adoption by the industry applications | This allows us to compare the scaling of transformer vs. an LSST both popular but different sequence-tor sequence architectures. Finally, we use a decoder- only tranformer that is trained with a language modeling (LM) loss. Thus, the last model changes not only the architecture, but also the loss (by including an LM loss on the source side). This setup is identical GPT-3, and thus allows to compare if these two different set ups for doing translation have similar data efficiency.
To compare the scaling of these models, we train series of models on increasing subsets of the data. All the three models have ~ 300M parameters. Then, we a scaling law with a common exponent p, but model-dependent {ag, C The results of our experiment fitting parameters are shown in Appendix D. We find that this scaling law with a common exponent
shown in Figure 13. The
'While we report BLEU in addition to log-perplexine scores in our study, we acknowledge and would like to bring the recent findings on the deficiency of BLEU as an automatic metric to the readers attention. As the MT systems got better over the years, BLEU scores (along with several other automatic metrics) started to losc their sensitivity approximate human judgement (Zhang & Toral 2019) Mathur al. 2020 [Fritag et all 2020 (000|20T:/ Kocmi el al., 2021) and the translation community has started to experiment with learned metrics such as COMET (Rei et al. 2020) or BLEURT (Sellam et al. 2020). We find the ongoing discussions around the established automatic metrics quite important, but refrain from dilating the scope our study, hence report
both model scores and BLEU scores.
*BLEU score a precision based metric that compares a reference translation with the generated hypothesis by the model and yields score between 0 and 1, taking into account n-gram overlap between reference and hypothesis while compensating for the lack of recall with brevity penalty.
http:://wwwmmiiiicio oec.rhaoooe.e.....aaa...o.ooooon
html
5

---

Under review as a conference paper at ICLR 2022
Effect of filtering Exponent 0.278
Effect of transltianni
2.252
2.915
No fitter
CDS Fitering
0T model 6l
1.027
fittering
Test log-perplostty (t)
2.126
BI model X2L0L
model 64L6.
Paralicl data
1.482
1.551
1.202
1.131
0.976
10"
10'
10a
0.825
100
107
10
A
Sentence pairs dataset (D)
B
Sentence pairs dataset (D)
Figure 3: (A) Effect of filtering: We apply two different filtering algorithms to the raw Paracrawl dataset and evaluate the data scaling curve. We find that a common exponent provides good fit for the experimental observations. (B) Effect of back-translation: We train a 6L6L model on back- translated data from 4 different back toacstrraslation models {21.61, 61.61,, 32L61., 64L61.). We find that the scaling exponent for Back-translated data worse than that for clean parallel data.
describes the observed experimental data well. Please refer to Appendix D for more results with different model sizes of the Hybrids and Decoder-only models.
Our experiments show that while subtly different architectures may have different performances on a fixed dataset size, the architectures may nevertheless scale similarly. Thus, we can compensate for a marginally worse architecture by adding more data. Crucially, the factor of additional data be added does not depend on the loss value, it will always be 1/p, where (11, 02 are the multiplicative constants for the two architectures and p the common scaling exponent. On the other hand, if the exponents were different, the factor of data to be added to achieve equal performance would increase exponentially with decreasing loss. This suggests that when choosing between multiple architectures that have similar data scaling, the decision can freely be driven by other considerations such as compute efficiency, multi-task abilities, compressibility for deployment or generation latency etc.
ag
4 THE EFFECT OF NOISE
Large scale parallel corpora are essential to building high-quality machine translation models (Ari- vazhagan al. 2019 Lepikhin et al. 2020 Fan al. 2020). Such datasets created by crawling web pages and performing various post- post-processing steps to create parallel data such as document alignment and sentence alignment (El-Kishky et al., 2020 Bañón al. 2020). However, there are misaligned sentences, copied URLs, typos,
many ways in which noise can enter this pipeline mistranslations and so on. Such noise can potentially be detrimental to translation quality. For in- stance, prior work (Khayrallah & Koehn 2018) has found that adding large amount of noisy data to high quality parallel data can have a catastrophic effect on the performance of NMT models. In this section, will study the effect of noise, not just by comparing noisy and clean data single dataset size, but by comparing their scaling for increasing dataset sizes. We approach this question in two opposite and complementary ways (1) We start with noisy web-crawled corpus, Paracrawl English to German (Bañón et al. 2020) and apply filtering algorithms to (Section 4.1 (2) We start
with clean parallel corpus and add different types of noise it (Sections 4.2. 4.2.2
4.1 DATA FILTERING
Due to the prevalence of noise in web-crawled corpora and its impact machine translation models, a variety of algorithms and heuristics have been developed to filter out noisy sentences (Wang et al.
2018 Junczys-Dowwnn Dowmunt 2018 Sancrez-Sänchez al. 2020). Our goal here is to understand the impact of data filtering on the data scaling properties NMT.
Our setup: To understand the effect of data filtering, English- German dataset(Bañón et al.) 2020). We lightly filter the raw dataset with de-duplication, length filtering (< 256) and language IID filtering. We also remove duplicates of our test sets
the largest publicly available Paracrawl

---

Under review as a conference paper at ICLR 2022
with 10-gram overlap. This leaves 750M noisy sentence pairs. We train 6L6L model on this dataset with increasing dataset sizes {1M, 2M...256.M.. To study the effects of the filtering, we use two filtering algorithms. For the first filtering algorithm, we use the Bicleaner scores (Ramirez- Sánchez et al. 2020) that are publicly released along with the Paracrawl dataset v8. 1. This is a score ranging from (0, T). with higher score indicating higher sentence quality. The bicleaner scores in- cludes various hard-coded rules, language-moder fluency scores, and scores from a classifier trained to detect mutual translations. We use threshold 0.5 and discard all sentences below this threshold leaving us with ~ 300M sentence pairs. For the second filtering algorithm, we choose Contrastive Data Selection (CDS) (Wang et al., 2018), which belongs to a family of cross-entropy- based filter- ing algorithms (Moore & Lewis 2010; Junczys-Dowman 2018). CDS scores the quality of each sentence pair according to the difference in cross entropy scores between two related translation models: a clean model that was fine-tuned on a trusted dataset, and noisy one that was not. We choose the top 50% of the
sentence pairs.
Results:
The results for the data scaling law for all three training datasets are shown in Figure 3
and fitted coefficients in Table T We measure the test log perplexity on the Web Domain I test set. We fit scaling law with common exponent p and separate a, C for each training set. We find that
a common exponent fits the experimental data well. We make the following observations:
Multiplicatiiv constant a shift: The filtered dataset has lower a, implying that at given dataset size, the loss for filtered data is lower. Thus, if you were constrained by compute to use a small fixed dataset size, it would be ad- visable use filtered dataset.
a
C
p
No filter
2,501
0.034
CDS
2.235
0.054
0.278
Bicleaner
2.130
0.064
Loss at convergence aCP is the same: In our experiments, we find that the three datasets converge to the same loss for a 6L6L model. This implies that at large dataset sizes, a 61.61. model is unable to distinguish the differences be- tween a filtered and unfiltered dataset. We would like to emphasize that this is consequence of the particular filtering algorithms used. For instance, it is possible to construct a filtering algorithm that is very biased and discards all data from certain domain, which may converge to higher loss value.
Table I: Coefficients for filtering
Similar exponents p. We find that common exponent is sufficient to describe the experimental data. This finding has the following important implication: We can compensate for noise in the dataset simply by adding more data!
There is no standard, task-independent definition or measure of sentence quality or noise. Thus, any filtering algorithm always runs the risk of biasing the training dataset, say, towards the in- domain trusted dataset used for filtering. For example, a recent study by Gao (202I) shows that very aggressive filtering can negatively impact the downstream performance of language models. Our results show that the effect of unfiltered noisy sentence pairs is not catastrophic while some amount of filtering may be desirable for computational efficiency, we can replace filtered data with more unfiltered data.
4.2 ADDING NOISE
While understanding the effect of filtering on data scaling curves is practically informative, filtering combines many different types of noise and heuristics together. To get a more detailed understanding clean dataset have finer control over the dataset.
of the effect of noise, we now add noise In particular, we make two different types of distinctions. First, we add noise either only to the source side (the input sentences) or only to the target side (the output sentences). Second, we consider independent vs. dependent noise. Independent noise, includes noise such as changing character to a random character or deleting a random word - the noise does not depend on the source/target sentence itself. By dependent noise, we mean that noise added to the sentence depends on the sentence itself, for eg: if the word 'cat' is always mistranslated. We believe that this is a natural distinction since the effect of the former type of noise can (at least information information-thcorrtiilll theoreticallll) be reversed we are provided with more data. On the other hand, dependent noise can bias the distribution in more drastic, irreversible ways.
7

---

Under review as a conference paper at ICLR 2022
4.2.1 INDEPENDENT NOISE
We start with a simple setup of the following types of iid noise added to the source and target side respec- tively: (1) Character level: We perturb = 0.1 frac- tion of the characters in the sentences to random char- acters (alphumeric + punctuations), (2) Word level: We delete p 0.15 fraction of the words from either the source or the target side, and (3) Sentence level: For p 0.1 fraction of the sentences, we shuffle the map- ping sentences of the sentence pairs. These noise types have also been considered in prior work Khayrallah Koehn (2018). Next, we train 6L6L trans- former model on increasing subsets of the noisy training datasets. The results are shown in Figure IC and Table 3
C
No noise
1.969
0.064
Source noise
2.222
0.067
0.296
Target noise
2.772
0.323
Table 2: Coefficients for added noise
Our first observation is that, similar to the case of data filtering, we can fit power law of Equation D with a common exponent p, but different a, C' for the different training sets to our experimental observations. On the other hand, unlike filtering, both the source and target noise datasets do not converge to the same loss value as the clean dataset at large dataset sizes D - co. These results show that while the exponents for different datasets can be similar, it is also important consider the C' or the loss where these models converge. In this particular case, more data cannot always offset the effect of noise. It is open question if this is because the model size 61.6 is too weak to learn in the presence of this amount of noise, that these types of noise are disruptive to neural network training irrespective of model size, or that this problem is hard solve at finite samples for any class of models. It is also an interesting open question to see if we can predict which types of noise will lead a different C.
Lastly, another important observation is that target noise is much more harmful than source noise for the same amount of noise. This may help explain why backtranslation (Sennrich et al.. 2016), a widely used data augmentation technique for machine translation, works since it it uses clean monolingual target corpus and noisy back-translated source sentences. This in contrast with observations in the vision domain Bahri et al. (2021). where changes to the input distribution change the exponent, but changes to the output distribution keep the exponent unchanged.
4.2.2 DEPENDENT NOISE: BACK-TRANSLATION
Now we turn our attention to changing the training distribution by using back-translated data instead of parallel data. Back-translation (BT) (Sennrich et al., 2016) is common auga-augmentation tech- nique employed in MT to increase the amount of training data. Say you are training your model to translate English to German sentences. With back-translation, one would use reverse model trained from German to English, and clean monolingual German corpus to generate English-Gern German sen- tence pairs. Back-transsation can be considered type of noise that is added to the source side. However, it differs from the independent noise considered in Section in in that noise depends on the source/target sentence itself. For instance, if the BT model was never trained on any sentences on the topic of animals, it will make systematic errors on such sentences. These errors can be impossi- ble for the downstream model reverse, no matter how much training data is provided it. This makes it an interesting setup to study data scaling, as it is not apriori obvious if such a distributional change would impact just the bias of the model C' or also the scaling exponent p.
Setup: In our setup, to minimize the number of confounders, we extract the same German target side of the same dataset that was used to train our models in Section [ This keeps at least the target distribution the same as that in Sections 2 and 4 We use four different German to English encoder- decoder models of sizes (21.6l,, 61.61,, 321,6T, 641661.) to translate the German sentences, thus giving us four different datasets with English source sentences of varying quality, with the smallest model producing the 'noisiest" source sentences. Note that these German to English models are trained on different dataset, although there may be some sentence overlaps with the English to German datasets. To evaluate the data scaling, we train a 6L6L model on increasing dataset sizes for all these datasets.
The results of our experiments are shown in Figure 3B. We fit a scaling law with com- mon 0, and dataset dependent C. Interestingly, we find that the scaling exponent of
8

---

Under review as a conference paper at ICLR 2022
the BT trained models is slightly lower (~ 0.2) than the scaling exponents of the par- allel dataset (~ 0.28) as is also visible in Figure 33. We did not get good fits when tried to jointly fit the BT data with the clean parallel data with common exponent. Moreover, the BT datasets converge to a worse loss value at the infinite data regime. Another point of note is that different BT model sizes have similar scaling, but larger models con- verge to marginally better test log-perplexities. Thus, if we are in the extreme data- limited regime, almost any back-translation provide good improvements, but as we approach very large dataset sizes, will be more beneficial to use larger BT model.
a
C
p
BT model 2L61.
2.343
0.059
BT model 6L61.
2.288
0.054
model will
0.0198
BT model 32L6L
2.251
0,040
BT model 64L6L
2.224
0.037
Parallel data
1.196
0.048
0.271
Taken altogether, these three experiments show that noise has less impact than one might have expected on the exponent characterizing the data- scaling behavior of NMT, given the strong emphasis on data quality in the NMT community. Both filtering natural noise and adding artificial independent noise have no impact on the exponent. How- ever, they do impact the multiplicative constant, meaning that for fixed computation or data budget, filtering remains quite relevant. Crucially, the techniques outlined here give practitioners tools they can use to help determine when effort should be put into improving removing noise, and when they should focus on collecting more data.
Table 3: Coefficients for BT
5 RELATED WORKS
Our work builds extensively on prior work on scaling laws Hestness et al. (2017); ) Rosenfeld et al. (211)) and in particular Kaplan et al. (2020). The papers most closely related our work are Hestness et al. (2017); Gordon et al. (2021) who provide data and parameter scaling laws for NMT. In addition to conducting experiments a much larger scale, our work goes beyond the experimental setup in these papers and considers the question of how various change to the training setup impact the data scaling law. Additionally, the form of our scaling law departs slightly from the scaling laws presented in Gordon et al. (2021) as discussed in Section 2 Our scaling law differs from Hestness ct al. (2017) in that they conduct experiments with LSTMs and their law does not scale as OI/DD) when D + co. Despite these differences, the scaling exponents found in both these papers are in the same range as ours ~ 0.3. Gordon et al. (202T) also conduct experiments on multiple language pairs and find that their exponents in the same range. Note that these exponents
much higher
than those found for the unconditional language modeling case by Kaplan et al. (2020); Hestness) al. (2077) (0.1 vs. 0.3). [Hoimm et al. (2021) conduct an analysis of data scaling curves in the vision domain with a focus on architectures and pre-training.
6 DISCUSSION AND CONCLUSIONS
In this work, we study the data scaling of models trained to perform neural machine translation under various interventions to the training pipeline, including changes to architecture as well as the training distribution. We find that a majority of these changes lead only to a multiplicative shift in the scaling curves, and the exponent changes minimally. Practically, this suggests that many advancements that seem significant at smaller scale, could be equivalently achieved by moving further across the data scaling curve and adding more data.
Apart from the practical implications, this work also raises some interesting theoretical questions. If so many interventions to the training pipeline keep the exponents unchanged, then this may be indicative of deeper commonality in the mechanism by which these deep networks learn. For instance, recent work (Bahri et al.) 2021 Sharma Kaplan (020)) conjecture that the data scaling exponent captures the "dimension of the data manifold" it is represented by the model. If this is the case, then our experiments suggest that some changes to the architecture or certain types of changes to the distribution do not change this 'manifol''.
In conclusion, we believe that understanding how different training methods change the scaling of the test performance can yield important insights both theoretically and practically.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry. Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively multilingual neural machine translation in the wild: Findings and chal- lenges, 2019.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2/02.06701, 2021.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel I. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramirez-Sáncher
Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. ParaCrawl: Web-scale acquisition of parallel cor- pora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2020.acl- main.417. |tt wtt.://antnnn.oo.ooo/o//ooaaee/e acl-mmm............mmmmm.mmmmmaa wwt.:///0.0................
acl
417
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzman, and Philipp Kochn. Ccaligned: A mas- sive collection of cross-lingual web-document pairs, 2020.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multilingual machine translation, 2020.
Markus Freitag, David Grangier, and Isaac Caswell. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. URL
htttps://aclanthologgs org. emmmem20m022mm11pppppp emnlp main.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation, 2021.
Ieo Gao. An empirical exploration in quality filtering of text data, 2021.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation, 2021.
Mitchell Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. ACL Rolling Review-May, 2021, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Frederick Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. ArXiv, abs/ abs//712.00049, 2017.
Derek Hoiem, Tanmay Gupta, Zhizhong Li, and Michal Shlapentokh-Rotthmal Learning curves for analysis of deep networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4287 4296. PMLR, 18-24 Jul 2021. URL Actps: http:///ppeeeegg....// .mlr. mlr.
preas/v139/hor / em html
10

---

Under review as a conference paper at ICLR 2022
Marcin Junczys-Dommunt. Dual conditional cross-entropy filtering of noisy parallel corpora. arXiv preprint 1.X.:v:889.001110 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0830 2020.
Huda Khayrallah and Philipp Koehn. On the impact of various types of noise on neural ma- chine translation. Proceedings of the 2nd Workshop on Neural Machine Translation and Gen- eration, 2018. doi: 10. 10.18653/1//118-7000 URL httpp://// doi wttp:/.......11.11553.3.13
018-2709
Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Manczys-Dowmun,, Hitokazu Matsushita, and Arul Menezes. To ship metrics for machine translation, 2021.
ship: An extensive evaluation of automatic
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li. Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre- pre-training for neural machine translation, 2020.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Transla- tion, pp. 688-725 Online, November 2020. Association for Computational Linguistics. URL https: hattps///ccaans wttpss//aclanhhhllgy.oooom org/20200020000 wmt 1m7-1.77
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.
Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACI 2010 Conference Short Papers, pp. 220-224, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL haagsss//ccaatholooyyo oorghhooggggogg//// 210-2044
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00m0 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
Gema Ramírez-Sánchie Jaume Zaragoza-Bern Bernabeu, Marta Bañón, and Sergio Ortiz Rojas. Bifixer and bicleaner: two open-source tools clean your parallel data. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 291-298, Lishoa, Portugal, November 2020. European Association for Machine Translation. URL https::/ 1.33
sclantholooy..rg//2200
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. URI.
htt attt:::/aaclanttooooy..oos
cannlpmee m...naainn....... 213
Jonathan Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv: 1909. 12673, 2019.
11

---

Under review as a conference paper at ICLR 2022
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics. URL https://
ac aanaaoologys wwwg////000 acl -mmin. 704
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data, 2016.
Utkarsh Sharma and Jared Kaplan. neural scaling law from the dimension of the data manifold, 2020.
Noam Shazeer and Mitchell Stemn. Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Ari- vazhagan, and Yonghui Wu. Leveraging monolingual data with self-supervision for multilingual neural machine translation, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection, 2018.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architec- ture, 2020.
Mike Zhang and Antonio Toral. The effect of translationese machine translation test sets, 2019.
12

---

Under review as a conference paper at ICLR 2022
A DETAILS ABOUT POWER LAW FITTING
A. SEPARATE FITS AND VARIATION IN THE EXPONENT
In Section we showed that single common exponent p gives reasonable fits to the experimental observations for different model sizes. We now examine the difference in the co-efficients for each of these models. To do so, we fit separate power law from Equation [ to each model as shown in TA. The scaling exponents for all models are not exactly the same they do have minor differences.
Some of these variations can be attributed to the sensitivity of these scaling parameters to ran- domness in the training procedure. There are multiple sources of randomness in training - imitial random seed, randomness over sampling of the training set, randomness in SGD training such as batch order. To understand the effect of these, we sample 5 different versions of the training set for dataset sizes (250K, 500K, 1M) (we choose these as they require low compute train conver- gence) and train networks on these datasets from scratch. Figure I shows the standard deviation observed in the test loss for the 6L.1 and 61.28L models. As we can see, the variance up to 2% of the loss. The variance in larger datasets expected to be lower than those for smaller datasets.
2.745
Individual
SL2L p p-0.243
X (Sid-Dev of Mean Los))
Standard deviation loss
Medel
Distribution exponents
2t5 pro.291
Text Ing-perplexty (1)
1.966
2016L, p-0.211
SLSL p-0.271
1.4
35
1.2
30
25
1.408
Frequency
20
0.8
15
1.009
0.6
A
0.722
Sentence peirs in dataset (D)
B
Dataset size ID)
0.6
1.0
Scoling Exponent
0.23 0.24
Figure 4: (A) Separate power law fits from Equation ! to different model sizes. We observe small differences in the scaling co-efficients (B) Standard deviation in the loss due to randomness in train- ing. The loss varies by up to 2% (C) Distribution the scaling exponents from a monte-carlo simulation assuming 2% standard deviation in the loss.
To understand how this variance would affect the final observed scaling co-efficients, we do the following monte carlo experiment: We assume that the loss distributed as A/1 0.021) where I is the loss for a given dataset size and model. We then simulate different loss values from this distribution for the dataset sizes, and scaling co-efficients to it. This gives us a distribution the scaling co-efficients. Figure 4C shows the distribution of the scaling exponents obtained from this procedure for the 28L6L model. As we can see, a 2% randomness in the test loss, gives standard deviation of 0.02 in the scaling exponent. This provides benchmark in comparing exponents obtained from two different experiments (say two different architectures). Please note that this is only a rough calculation that is meant to give some intuition for how these sacling parameters may be affected by various sources of randomness.
A.2 VARIANCE-LIMITED REGIME
We fit the scaling law L = (Do/Dy' + to dataset sizes >= 32M. If the "variance-limited" conjecture is correct, then the scaling exponent should be p 1. Figure 5 indeed shows that this is the case. Moreover, the transition point to this regime should occur later for larger model sizes. As see, the exponents for the larger models has not reached 1.
B SCALING LAWS FOR DIFFERENT DATASETS
In all main experiments, study the scaling laws for heldout test set from the same distri- bution as the training set. But, practically, we may be interested in the performance of some other out-of-distribution test sets. These other test sets have some overlap with the training distribution but
13

---

Under review as a conference paper at ICLR 2022
Variance-limited regime
1.1849
6L2L, p=0.902
2L6L, p-1.179
log perplexity (L)
61 p-1.076
1.0566
28L61, p-0.608
6L28L, p=0.582
0.9421
0.8400
0.7490
10
Sentence pairs in dataset (D)
Figure 5: Power law fits for large dataset sizes show that the loss decays as O(1/D)
may have different support or composition. To understand how performance on such test sets scales with dataset size, we measure the performance on various other test as described Section ??.
5.749
1.2
8.925
3.344
Tost 1o3 wer.eeo.oon per
1.923
1838
0.9
1.112
2.011
A
0.643
Sentence pairs in detaset (D)
10"
B
0.250 0,275 0.300 0.350 0.375
Expoment
C
0.556
Inddistribution
Figure 6: Scaling laws for various OOD test sets for 6L6L model
We find that the test loss these additional test sets also follows a similar power law in the dataset size as Equation T Figure shows the test loss fits for 14 different test sets for 6L6L model. We find that most of the test sets have similar scaling exponents. That is, do not observe any major differences on the basis on the test set composition both target and source original test sets scale similarly.
Additionally, since the different test sets have similar scaling properties, this implies that the in- distribution loss and the out-of-distribution loss must have nearly-linear relationship.
C DATA SCALING PHASE TRANSITION
We fit the scaling law shown in Equation
Do L = CoaaDoot D
(D+cc²
(3)
This equation displays two scaling regimes:
1. Over- pveepparameterized (or small D): In this regime, the Do/ term dominates the loss and the loss scales Ol)
2. Under-parameterizee (or large D): In this regime, we can take a Taylor's approximation for the small term Do/ which leads the loss scaling O(1DD).
14

---

Under review as a conference paper at ICLR 2022
Equating the derivatives of the two expressions provides expression for the point where the marginal value of data transitions (and hence the model moves to capacity limited regime). sim- ple calculation shows that this point occurs CDD Do- Figure provides an illustration for this transition the simple scenario where Do =
376.7
V
ap
D-1-P
8.1
D
CD=1
0.2
0.0
0.0
10
10
10
10a
10a
Number Sentence Pairs,
Figure 7: Phase transition from data-limited regime to model-limited regime. For simplicity, we
assume Do 1.
D SCALING LAWS WITH DIFFERENT ARCHITECTURES
The fitting parameters for Figure B are shown here.
C
Encoder-Decoder
1.969
0.057
Decoder-only
1.817
0.11
0.285
Hybrid-LSTM 2.011
0.078
We now show additional plots for decoder-only and transformer-LSTM hybrid models, with indi- vidual fits for each model.
Decoder-only Models
2.411
Hybrid models
91, p-0 309
2.256
6L21, p-0.243
13L, p-0.327
6L61, p60.266
Test peppperlextty
1.906
Test log-per cog-peplextty (L)
1.793
1.506
1.425
1.132
0.900
10f
107
10a
10a
107
10P
Sentence pairs dataset (D)
Sentence pairs dataset (D)
Figure 8: Separate fits for architecture with different depths (A) Decoder only (B) Transfomer-
LSTM Hybrids
15


Reviews:
[{"review_id": "y2sx3E23GM", "reviewer": "Reviewer_XWAu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. Well executed empirical study that complements the work of Ghorbani et al. 2021 by considering different architectures such as transformer LMs, transformer-LSTM hybrids, and data filtering along with results on publicly available web-crawled corpora.\n\n2. The paper offers clear suggestions on when to use noisy data based on available compute and model sizes.\n\nCons:\n\n1. A lot of the analysis and scaling results are already present in Ghorbani et al. 2021 which this paper borrows from. For example, 1) the asymmetric encoder-decoder scaling 2) OOD test set results on almost identical test sets 3) backtranslated vs forward translated data in this paper vs “original src” and “original tgt” in Ghorbani et al etc. \n\nQuestions:\n\n1. The paracrawl.eu website reports a lot less than 750M En-De sentences for Paracrawlv8. It would be best to report the exact source from which this data was downloaded.\n", "summary_of_the_paper": "The paper studies model and data scaling laws for Neural Machine Translation. It studies how the test cross-entropy loss scales with 1) Different seq2seq model architectures (transformer seq2seq, transformer LM and a transformer-LSTM hybrid) and asymmetric scaling of the encoder and decoder 2) (a) Training data size and composition - backtranslated data from models of varying size, filtered data using cleaning tools, synthetically introduced noise in the source and target languages (b) In-domain vs out-of-domain test data.", "summary_of_the_review": "Overall this is a very thorough empirical study of NMT scaling that is sure to benefit the NMT research community immensely when read alongside Ghorbani et al. 2021.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l33mrMSVZkX", "reviewer": "Reviewer_Pwfc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons for score:\n\nI have major concerns about the contributions this paper adds compared to [1] and [2]. Most of the conclusions in the paper have already been confirmed by these previous works.\n\n- The scaling law of dataset size has also been thoroughly studied and verified by [2]. The form of scaling law used in this paper is the same as what in [2] except an extra $L_\\infty$ term to control the convergence.\n- [2] has already shown the effect of larger models on data scaling laws with models with different numbers of parameters. [1] has also thoroughly studied the effect of different model sizes on the scaling laws of neural machine translation models.\n\nIn addition, some conclusions from the paper are also not convincing:\n\n- The paper only conducts experiments on a single language pair (English - German), which doesn't necessarily mean all the patterns found in the paper would generalize to other datasets.\n- When testing for different models, there is only one data point for a type of model at a specific dataset size. However, the performance of a model is determined by lots of other hyper-parameters like the choice of the optimizer, the learning rate, dropout rate, etc. The authors didn't discuss how they decide these hyper-parameters and how they affect the scaling law.\n- In Figure 3A, the curve of the no-filter case misses the data points a lot. Also, while the three cases in this figure obviously have different test log-perplexity for the largest dataset case, the fitted curve converges to the same point, and in the main text the authors claim that \"Loss at convergence $\\alpha C^p$ is the same\". This conclusion is based on errored fitted curves and is consequently wrong.\n\nOther comments:\n\n- \"However, recent trend in machine learning largely confirms the trade-off between architectural biases and the amounts of data, namely tendency towards architectures with low inductive biases are being trained on large amounts of data, compute and parameters.\" Please add references for this claim.\n\nMinor Comments:\n\n- On page 4, please define $\\beta, N_e, N_d$ instead of only referring to previous works.", "summary_of_the_paper": "This paper verifies the scaling laws for neural machine translation models in terms of dataset size. In addition, with different model sizes, model architectures, and different noise level of the training data, the scaling law still holds. This means the effect of suboptimal model architectures and noisy data can be compensated with simply adding more data.", "summary_of_the_review": "The contributions of this paper are incremental and the conclusions are also not convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9fDBuwmQqdJ", "reviewer": "Reviewer_Nuk2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n- The paper has strong empirical experiments at scale to validate the scaling laws. Compare with previous work (Gordon et. al), the training data is much larger, up to 512M sentence pairs for English->German. While I believe that the models used in the paper have a significantly large number of parameters, I couldn’t find this information in the paper. Could the authors elaborate the size of these models?\n\n- There are some interesting findings in the paper when scaling up the training data. For example, it seems that data filtering can be replaced with more unfiltered data. \n\n- The experiments with data noise are carefully designed and controlled. The results provide useful insights.\n\n**Weaknesses and Questions**\n\n- As the authors already called out in the paper, one major weakness I found is the use of log-perplexity as a proxy for language generation quality. *To what extent would improvement in test log-perplexity reflect in human evaluation (HE) and would the improvement in human evaluation follow dismissing returns laws as the amount of data increases?* Although the authors discussed about another potential metrics (e.g., COMET, BLEURT) I wonder if the authors have consider using human evaluation for one of those model (e.g., 6L6L) at different data scales? If yes, what prevented it from being done?\n\n- While the paper focuses mostly on test log-perplexity in scaling laws, I think there are other aspects of translation quality that are important for making deployment decisions. Larger models trained on large amounts of data could potentially exacerbate some known issues of NMT such as gender-biases. I understand that this is not the main focus of the paper, but I think it might be useful to have this angle in the discussions when deployment decisions are rarely made based on perplexity and BLEU scores.\n\n- As the training data is crawled from the web, I imagine that some part of the data could have translationese source. I wonder if the tetsets used in the work are source-original? Perhaps, my question would be *what the graph of fitting the scaling laws look like when evaluating on translating in the natural direction?*\n\n- While all the models in the paper are dense model, I wonder if the scalling laws hold for sparse models (e.g, Sparse Mixture of Experts)\n\n- What is the value of $L_\\infty$? Do you set it fixed for fit it?\n\n**Few recommendation for paper presentation:**\n- In the paragraph below equation 2, I think you meant 6L2L as a held out model, not 6L28L\n- In subsection 4.2.1 at the end of the first paragraph, I think you meant table 2 (i.e.The results are shown in Figure 1C and Table 3.)\n- Table 3 isn’t mentioned in subsection 4.2.2 DEPENDENT NOISE: BACK-TRANSLATION\n- If possible, I would recommend putting the figures where they are discussed. \n- In equation 2, are $p_e$ and $p_d$ are free parameters? They are not mentioned in the following paragraph.\n- I think it's worth to explain $L_\\infty$.\n- Some part of the paper should have been proof-read and cleaned up. In the implication paragraph, could the author ellaborate where the addtional amount of data $O(D^{−5/4}) $ and $O(D^{-2})$ come from?\n", "summary_of_the_paper": "The paper provides an empirical study of scaling laws for neural machine translation with respect to data noise (filtered/unfiltered/back-translation) and model architectures (transformer encoder-decoder, transformer decoder, transformer-encoder LSTM decoder). Focus on data aspect in scaling laws, the paper carried out a series of experiments on multiple data scales ranging from low-resource (500K sentence pairs) to large scale in industrial setup (512M sentence pairs)\n\nScaling Laws: the paper shows that their scaling laws (in equation 1) can be fit five different architectures (2L6L, 6L2L, 6L6L, 28L6L, 6L28L) on in-domain testsets and 11 out-of-domain testsets using 6L6L models. The scaling laws also can be used to predict the test log-likelihood of an unseen model (Figure 2A).\n\nArchitecture: Experimenting with two different encoder-decoder architectures, namely transformer encoder-decoder and transformer-encoder LSTM decoder, the paper shows that marginally worse architecture can be boosted up by adding more data. \n\nData noise: Throughout a series of control experiments, the experiments show that data filtering can be replaced by adding more unfiltered data. The scaling laws can fit both independent noise and dependent noise (via back-translation)\n", "summary_of_the_review": "The paper shows strong empirical results on fitting scaling laws. The experiments are carried out on large scale dataset and large models. While this provides valuable insight for practictioners on making decision when training a NMT system, using log-perpelxity without human evaluation is problematic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0-KKAAN7SPX", "reviewer": "Reviewer_fUfX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- With the recent advances in large language models, scaling laws for NMT is timely and important.\n- A large number of empirical studies are performed to investigate the impacts of different model & data choices on the final NMT scaling laws.\n- The depicted scaling laws could provide empirical guidance to the NMT community.\n\nWeaknesses:\n- The studies are mostly focused on one language pair (English-to-German), there isn't clear evidence whether their findings could generalize to different language families.\n- Since the paper is mostly an empirical work, there's lack of novelty in methodology contributions. However, I am not sure if we should count too much on the methodology contribution for such scaling laws papers.\n- Some findings don't seem very informative, e.g., the scaling laws on model sizes and model family. \n- The following paper seems very relevant, which should be cited:\nScale Efficiently: Insights from Pre-training and Fine-tuning Transformers", "summary_of_the_paper": "The paper studies data scaling laws for NMT. Specifically, it investigates the following factors empirically:\n- model architecture\n- model sizes (different numbers of encoder or decoder layers)\n- data filtering\n- different types of manually-added noises.\n\nThe paper has performed a large number of studies to demonstrate some empirically-useful findings for training the NMT models. ", "summary_of_the_review": "The major takeaways for the paper, I think, are:\n- Changing in model sizes and architectures have very similar scaling patterns, with a shared component. \n- Different noise reduction filtering methods (e.g., CDS and Bicleaner) can also be fit into the shared component.\n- Adding different types of noises can dramatically downgrade the NMT performance, while independent noises can still fit into the share the component, except the dependent noises via back-translation.\n\nOverall, these findings emphasizes again on the importance of data quality for NMT models, which is somehow known to the community. The paper further quantified such importance with the data scaling laws. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the data scaling laws in NMT tasks with different network architectures and data qualities. The main purpose of this paper is to investigate how such different experimental setup affects the scaling law. The authors found that those difference does not have strong impact on the scaling exponent, and a small difference of model architecture and data noise can be compensated by larger data size.

This paper gives nice justification of data scaling law from some different aspects which is instructive to some extent. On the other hand, the paper has some weakness as listed in the following: (1) The scaling law itself has been analyzed by many papers, and its novelty is rather limited. I acknowledge that this paper investigates different aspects of the data scaling law and the size of experiments are larger than existing work. However, the result is rather unsurprising. (2) The experiments are conducted mostly on one language pair (English-to-German), it is still unclear whether the findings are universal to other language pairs. As the authors responded, exhaustive experiments over all language pairs are unrealistic but some more investigation to more general data sets could be conducted to strengthen the paper.  

This paper is around the borderline. Some reviewers were rather positive to this paper. However, they also pointed out the concerns I listed above and they do not show strong support on the paper.  
In summary, although this paper shows some instructive findings, it is still a bit below the threshold of acceptance.
Decision: Reject