Data Scaling Laws in NMT: The Effect of Noise and Architecture

1 INTRODUCTION . In deep learning , a reliable recipe to improve the generalization performance of a model is to increase the amount of compute and data used to train it ( Krizhevsky et al. , 2012 ; Brown et al. , 2020 ) . Many recent advances in deep learning are directed towards increasing the efficiency of digesting data through advancements in architecture ( Vaswani et al. , 2017 ) , improving the quality of the data with filtering , or incorporating entirely new sources of data into training ( Radford et al. , 2021 ; Chen et al. , 2020 ) . In this new paradigm of machine learning , it is crucial to understand the data-efficiency of our training methods , and whether improvements to them at small scale translate into improvements at large scale . Recent work ( Hestness et al. , 2017 ; Rosenfeld et al. , 2019 ; Kaplan et al. , 2020 ) on scaling laws offers a useful tool to answer such questions — they show that the test loss of a model scales predictably and smoothly as a power law in the relevant quantities of interest such as dataset size ( D ) , model size ( N ) and amount of compute ( C ) . We take these findings a step further and ask how different interventions to the training setup impact the data scaling curves in the setup of neural machine translation ( NMT ) : How do changes to the training setup ( such as architecture , noise ) impact the data scaling curves in NMT ? Practically , data scaling laws can be leveraged to make experimental decisions for future large scale experiments , and to decide where computational and research efforts should be focused . Currently , NMT models are trained using massive web-scale data ; Arivazhagan et al . ( 2019 ) used 25B training examples ( approx.1T tokens ) and with the advent of the self-supervised learning techniques ( Liu et al. , 2020 ; Raffel et al. , 2020 ; Siddhant et al. , 2020 ) this number can easily reach 10+T tokens . At such large scales of data , it is unfeasible to ‘ just perform the experiment ’ , and scaling laws can be used to drive training decisions . For instance , if small changes in architecture do not lead to a change in scaling exponent ( as shown in Figure 1B ) , then architecture choice can be driven by other factors such as computational efficiency by paying a small penalty in the form of more data . In the opposite scenario , if a training setup performs worse at a given dataset size but scales much better , it should be chosen for future larger scale experiments . 1.1 OUR CONTRIBUTIONS . In this work , we first establish that the test log-perplexity of encoder-decoder transformer models trained to perform neural machine translation ( NMT ) follows a power law in the dataset size , with a dependence on the model size ( Figure 1A ) . We demonstrate that our scaling law predicts experimental results over 3 orders of magnitude ( from 500K-512M ) training examples ( sentence pairs in NMT ) 1 . Then , we systematically vary aspects of the training setup to understand how these impact the scaling laws . In particular , we consider the effect of ( 1 ) Changing architecture and task setup from a vanilla Transformer Encoder-Decoder to Transformer Enoder-LSTM Decoder Hybrid as well as a Decoder-only Transformer with language modeling objective ( 2 ) Changing the level of filtering in the dataset ( 3 ) Changing the amount of noise on the source ( input ) side and the target ( output ) side ( 4 ) Using backtranslated data . Surprisingly , we find that , with the exception of back-translation , these changes do not impact the scaling exponents much ( See Fig . 1 for example of changing model size , architecture and level of noise ) . Some of these changes affect the bias or the final loss value at infinite data . Our work suggests that many of the common operations used to boost performance , such as small changes to the architecture or data filtering , can be replaced by adding an additional constant factor of data . That is , in some cases , sub-optimalities in the architectures and noise in the datasets can both be considered as ( model and data ) scaling penalties . 1.2 EXPERIMENTAL SETUP . Models : Our experiments are conducted on pre-layer transformer networks ( Xiong et al. , 2020 ) . Models are trained with per-token cross-entropy loss and Adafactor optimizer ( Shazeer & Stern , 2018 ) . All models are trained with a fixed batch-size of 500K tokens and dropout rate of 0.1 for residuals , feed-forward activations and attention . For the small dataset sizes , the models are trained to early stopping ( as measured on the log-perplexity of a held-out development set ) and for large dataset sizes they are trained for up to 500K gradient steps . The hyperparameters for these models were optimized for a 6 encoder layer and 6 decoder layer model trained on 2.2 billion sentence pairs . We train 5 different model sizes { 2L6L , 6L2L , 6L6L , 6L28L , 28L6L } , where 28L6L means 28 encoder layers and 6 decoder layers . We also train two decoder-only models with a language modeling loss with { 9L , 13L } , and three Hybrid-LSTM model with { 6L2L , 6L6L , 6L12L } . All the hyperparameters are matched as closely as possible between these models to provide an apples-to-apples comparison . 1Corresponding to 27.6 billion tokens . Training data In our experiments , the models are trained on two large scale datasets . The first set of experiments on data size scaling are using the dataset sampled from an in-house parallel corpora containing up to 2.2B sentences translated from English to German . We sample training datasets of sizes { 1M , 2M , 4M , 8M , 16M , 32M , 64M , 128M , 256M , 512M } independently to study the data scaling laws . The second set of experiments are conducted with Paracrawl dataset ( Bañón et al. , 2020 ) and with and without filtering applied . The details are described in Section 4 . Test data The model performance is measured on a held-out dataset from the training distribution . Additionally , we also measure performance on various out-of-distribution test datasets . These test datasets have different domain composition and sampling methods . The domains considered are ( i ) Web-Domain ( ii ) News-Domain ( iii ) Wikipedia and ( iv ) Patents . The news-domain test sets come from the WMT2019 evaluation campaign ( newstest2019 ) for all language pairs . The other test sets are internal test sets representing the different domains , ranging from 500 to 5000 sentence pairs for each set . 2 DATA SCALING LAWS . We begin our investigation by training a series of large scale encoder-decoder transformer models on the in-house English to German parallel corpus , with the parameters ranging from 170M to 800M . Our dataset sizes range from 500K to 512M sentence pairs , covers 28B tokens , which is 10 times larger than the experiments conducted in prior work on scaling laws for NMT Gordon et al . ( 2021 ) . Our goal is to find a function in the relevant variables that predicts test performance of our experiments . We will mainly focus on the test log-perplexity on a heldout dataset from the training distribution , but we will also discuss scaling of BLEU scores ( Section 2.1 ) and performance on other out-of-distribution test sets ( Section 2.2 ) in later sections . Form of the scaling law : The chosen scaling law must exhibit decreasing loss with dataset size D , and at infinite data D =∞ , the model must converge to a finite constant . Additionally , Kaplan et al . ( 2020 ) ; Bahri et al . ( 2021 ) , conjecture that when D → ∞ , the models are in a “ variance-limited ” regime and the loss should scale as O ( 1/D ) . These desiderata are satisfied by the following scaling law for a fixed model size : L = α ( D0 D + C ) p ( 1 ) where α is a multiplicative constant , p is the scaling exponent and C is a model-dependent constant that are fitted empirically ( D0 = 1e6 is a fixed normalization constant use for numerical stability ) . We find that this scaling law indeed provides a good fit for the experimental observations . Moreover , when we fit just the last few datapoints ( See Figure 5 ) , we find that the loss scales as O ( 1/D ) , thus confirming the conjecture even for large-scale supervised learning setup . Now as we change the model size ( depth , width , or depth-width aspect ratio etc ) , it is not apriori obvious if these different models , say encoder-heavy vs. decoder-heavy models , should have the same data scaling parameters α , p. For instance , Ghorbani et al . ( 2021 ) observe that parameter scaling for encoder-decoder models depends separately on the number of encoder and number of decoder models . Thus , we train 5 different models { 2L6L , 6L2L , 6L6L , 28L6L , 6L28L } including very asymmetric ones and find that their scaling parameters α , p are in fact very similar ( See Appendix A.1 for individual fits for Equation 1 to each model ) . Additionally , the constant C must depend on the model size , since at infinite data D = ∞ different models will converge to different loss values αCp , with larger models converging to lower loss values . Figure 1A shows the fit for a common α , p and 5 different { Ci } 5i=1 for each different model . We find that a common exponent is sufficient to provide a good fit for the experimental data . We can go a step further and replace the constant C such that it recovers the parameter scaling curve at D =∞ . To do so , we leverage the parameter scaling law found by Ghorbani et al . ( 2021 ) and fit the following final scaling law : L ( D , Ne , Nd ) = α ( D0 D + β ( 1 Npee 1 Npdd + L∞ ) 1/p ) p ( 2 ) The only free parameters in this equation are α and p , and the parameters in the right term ( β , Ne , Nd , L∞ ) are directly borrowed from Ghorbani et al . ( 2021 ) , such that they converge to the parameter scaling law at D = ∞ . The fit using this scaling law is shown in Figure 2A . Since we have a joint scaling law with dataset size and parameters , we can use this to predict the test loss of models that were not used to fit the scaling law . For example , in Figure 2A we find α , p for all models while holding out the model with 6L28L and later we are able to predict the performance of this out-of-sample model . Note that this scaling law differs from Gordon et al . ( 2021 ) in two ways ( 1 ) Using the encoder and decoder parameters separately vs. using the total number of parameters ( 2 ) An additional L∞ term2 that remains even when the Ne , Nd , D =∞ . Implications : Equation 1 suggests that there exists two operating regimes for data scaling : ( i ) datalimited regime where D0D C , and ( ii ) capacity limited regime where D0 D C. Fitted exponents in Figure 1 suggest that , in the data limited regime , loss scales as O ( D−1/4 ) , suggesting a marginal value of O ( D−5/4 ) for additional data . Increasing the model capacity in this regime has negligible impact on the loss . In the capacity limited regime however , the loss scales as O ( D−1 ) suggesting a ( significantly smaller ) marginal value of O ( D−2 ) for additional data . In this regime , the loss value is dominated by the model-dependent constant C and most of the improvement can be had by increasing the model size . There is a smooth phase transition between these two regimes at approximately CDD0 = 1 ( See Appendix C for an illustration ) . Thus , by increasing the model size ( which reduces C ) , one can push the transition to larger values of D and leverage the available data more efficiently .

Summary: The paper provides an empirical study of scaling laws for neural machine translation with respect to data noise (filtered/unfiltered/back-translation) and model architectures (transformer encoder-decoder, transformer decoder, transformer-encoder LSTM decoder). Focus on data aspect in scaling laws, the paper carried out a series of experiments on multiple data scales ranging from low-resource (500K sentence pairs) to large scale in industrial setup (512M sentence pairs)  Scaling Laws: the paper shows that their scaling laws (in equation 1) can be fit five different architectures (2L6L, 6L2L, 6L6L, 28L6L, 6L28L) on in-domain testsets and 11 out-of-domain testsets using 6L6L models. The scaling laws also can be used to predict the test log-likelihood of an unseen model (Figure 2A).  Architecture: Experimenting with two different encoder-decoder architectures, namely transformer encoder-decoder and transformer-encoder LSTM decoder, the paper shows that marginally worse architecture can be boosted up by adding more data.   Data noise: Throughout a series of control experiments, the experiments show that data filtering can be replaced by adding more unfiltered data. The scaling laws can fit both independent noise and dependent noise (via back-translation)