Data Scaling Laws in NMT: The Effect of Noise and Architecture

1 INTRODUCTION . In deep learning , a reliable recipe to improve the generalization performance of a model is to increase the amount of compute and data used to train it ( Krizhevsky et al. , 2012 ; Brown et al. , 2020 ) . Many recent advances in deep learning are directed towards increasing the efficiency of digesting data through advancements in architecture ( Vaswani et al. , 2017 ) , improving the quality of the data with filtering , or incorporating entirely new sources of data into training ( Radford et al. , 2021 ; Chen et al. , 2020 ) . In this new paradigm of machine learning , it is crucial to understand the data-efficiency of our training methods , and whether improvements to them at small scale translate into improvements at large scale . Recent work ( Hestness et al. , 2017 ; Rosenfeld et al. , 2019 ; Kaplan et al. , 2020 ) on scaling laws offers a useful tool to answer such questions — they show that the test loss of a model scales predictably and smoothly as a power law in the relevant quantities of interest such as dataset size ( D ) , model size ( N ) and amount of compute ( C ) . We take these findings a step further and ask how different interventions to the training setup impact the data scaling curves in the setup of neural machine translation ( NMT ) : How do changes to the training setup ( such as architecture , noise ) impact the data scaling curves in NMT ? Practically , data scaling laws can be leveraged to make experimental decisions for future large scale experiments , and to decide where computational and research efforts should be focused . Currently , NMT models are trained using massive web-scale data ; Arivazhagan et al . ( 2019 ) used 25B training examples ( approx.1T tokens ) and with the advent of the self-supervised learning techniques ( Liu et al. , 2020 ; Raffel et al. , 2020 ; Siddhant et al. , 2020 ) this number can easily reach 10+T tokens . At such large scales of data , it is unfeasible to ‘ just perform the experiment ’ , and scaling laws can be used to drive training decisions . For instance , if small changes in architecture do not lead to a change in scaling exponent ( as shown in Figure 1B ) , then architecture choice can be driven by other factors such as computational efficiency by paying a small penalty in the form of more data . In the opposite scenario , if a training setup performs worse at a given dataset size but scales much better , it should be chosen for future larger scale experiments . 1.1 OUR CONTRIBUTIONS . In this work , we first establish that the test log-perplexity of encoder-decoder transformer models trained to perform neural machine translation ( NMT ) follows a power law in the dataset size , with a dependence on the model size ( Figure 1A ) . We demonstrate that our scaling law predicts experimental results over 3 orders of magnitude ( from 500K-512M ) training examples ( sentence pairs in NMT ) 1 . Then , we systematically vary aspects of the training setup to understand how these impact the scaling laws . In particular , we consider the effect of ( 1 ) Changing architecture and task setup from a vanilla Transformer Encoder-Decoder to Transformer Enoder-LSTM Decoder Hybrid as well as a Decoder-only Transformer with language modeling objective ( 2 ) Changing the level of filtering in the dataset ( 3 ) Changing the amount of noise on the source ( input ) side and the target ( output ) side ( 4 ) Using backtranslated data . Surprisingly , we find that , with the exception of back-translation , these changes do not impact the scaling exponents much ( See Fig . 1 for example of changing model size , architecture and level of noise ) . Some of these changes affect the bias or the final loss value at infinite data . Our work suggests that many of the common operations used to boost performance , such as small changes to the architecture or data filtering , can be replaced by adding an additional constant factor of data . That is , in some cases , sub-optimalities in the architectures and noise in the datasets can both be considered as ( model and data ) scaling penalties . 1.2 EXPERIMENTAL SETUP . Models : Our experiments are conducted on pre-layer transformer networks ( Xiong et al. , 2020 ) . Models are trained with per-token cross-entropy loss and Adafactor optimizer ( Shazeer & Stern , 2018 ) . All models are trained with a fixed batch-size of 500K tokens and dropout rate of 0.1 for residuals , feed-forward activations and attention . For the small dataset sizes , the models are trained to early stopping ( as measured on the log-perplexity of a held-out development set ) and for large dataset sizes they are trained for up to 500K gradient steps . The hyperparameters for these models were optimized for a 6 encoder layer and 6 decoder layer model trained on 2.2 billion sentence pairs . We train 5 different model sizes { 2L6L , 6L2L , 6L6L , 6L28L , 28L6L } , where 28L6L means 28 encoder layers and 6 decoder layers . We also train two decoder-only models with a language modeling loss with { 9L , 13L } , and three Hybrid-LSTM model with { 6L2L , 6L6L , 6L12L } . All the hyperparameters are matched as closely as possible between these models to provide an apples-to-apples comparison . 1Corresponding to 27.6 billion tokens . Training data In our experiments , the models are trained on two large scale datasets . The first set of experiments on data size scaling are using the dataset sampled from an in-house parallel corpora containing up to 2.2B sentences translated from English to German . We sample training datasets of sizes { 1M , 2M , 4M , 8M , 16M , 32M , 64M , 128M , 256M , 512M } independently to study the data scaling laws . The second set of experiments are conducted with Paracrawl dataset ( Bañón et al. , 2020 ) and with and without filtering applied . The details are described in Section 4 . Test data The model performance is measured on a held-out dataset from the training distribution . Additionally , we also measure performance on various out-of-distribution test datasets . These test datasets have different domain composition and sampling methods . The domains considered are ( i ) Web-Domain ( ii ) News-Domain ( iii ) Wikipedia and ( iv ) Patents . The news-domain test sets come from the WMT2019 evaluation campaign ( newstest2019 ) for all language pairs . The other test sets are internal test sets representing the different domains , ranging from 500 to 5000 sentence pairs for each set . 2 DATA SCALING LAWS . We begin our investigation by training a series of large scale encoder-decoder transformer models on the in-house English to German parallel corpus , with the parameters ranging from 170M to 800M . Our dataset sizes range from 500K to 512M sentence pairs , covers 28B tokens , which is 10 times larger than the experiments conducted in prior work on scaling laws for NMT Gordon et al . ( 2021 ) . Our goal is to find a function in the relevant variables that predicts test performance of our experiments . We will mainly focus on the test log-perplexity on a heldout dataset from the training distribution , but we will also discuss scaling of BLEU scores ( Section 2.1 ) and performance on other out-of-distribution test sets ( Section 2.2 ) in later sections . Form of the scaling law : The chosen scaling law must exhibit decreasing loss with dataset size D , and at infinite data D =∞ , the model must converge to a finite constant . Additionally , Kaplan et al . ( 2020 ) ; Bahri et al . ( 2021 ) , conjecture that when D → ∞ , the models are in a “ variance-limited ” regime and the loss should scale as O ( 1/D ) . These desiderata are satisfied by the following scaling law for a fixed model size : L = α ( D0 D + C ) p ( 1 ) where α is a multiplicative constant , p is the scaling exponent and C is a model-dependent constant that are fitted empirically ( D0 = 1e6 is a fixed normalization constant use for numerical stability ) . We find that this scaling law indeed provides a good fit for the experimental observations . Moreover , when we fit just the last few datapoints ( See Figure 5 ) , we find that the loss scales as O ( 1/D ) , thus confirming the conjecture even for large-scale supervised learning setup . Now as we change the model size ( depth , width , or depth-width aspect ratio etc ) , it is not apriori obvious if these different models , say encoder-heavy vs. decoder-heavy models , should have the same data scaling parameters α , p. For instance , Ghorbani et al . ( 2021 ) observe that parameter scaling for encoder-decoder models depends separately on the number of encoder and number of decoder models . Thus , we train 5 different models { 2L6L , 6L2L , 6L6L , 28L6L , 6L28L } including very asymmetric ones and find that their scaling parameters α , p are in fact very similar ( See Appendix A.1 for individual fits for Equation 1 to each model ) . Additionally , the constant C must depend on the model size , since at infinite data D = ∞ different models will converge to different loss values αCp , with larger models converging to lower loss values . Figure 1A shows the fit for a common α , p and 5 different { Ci } 5i=1 for each different model . We find that a common exponent is sufficient to provide a good fit for the experimental data . We can go a step further and replace the constant C such that it recovers the parameter scaling curve at D =∞ . To do so , we leverage the parameter scaling law found by Ghorbani et al . ( 2021 ) and fit the following final scaling law : L ( D , Ne , Nd ) = α ( D0 D + β ( 1 Npee 1 Npdd + L∞ ) 1/p ) p ( 2 ) The only free parameters in this equation are α and p , and the parameters in the right term ( β , Ne , Nd , L∞ ) are directly borrowed from Ghorbani et al . ( 2021 ) , such that they converge to the parameter scaling law at D = ∞ . The fit using this scaling law is shown in Figure 2A . Since we have a joint scaling law with dataset size and parameters , we can use this to predict the test loss of models that were not used to fit the scaling law . For example , in Figure 2A we find α , p for all models while holding out the model with 6L28L and later we are able to predict the performance of this out-of-sample model . Note that this scaling law differs from Gordon et al . ( 2021 ) in two ways ( 1 ) Using the encoder and decoder parameters separately vs. using the total number of parameters ( 2 ) An additional L∞ term2 that remains even when the Ne , Nd , D =∞ . Implications : Equation 1 suggests that there exists two operating regimes for data scaling : ( i ) datalimited regime where D0D C , and ( ii ) capacity limited regime where D0 D C. Fitted exponents in Figure 1 suggest that , in the data limited regime , loss scales as O ( D−1/4 ) , suggesting a marginal value of O ( D−5/4 ) for additional data . Increasing the model capacity in this regime has negligible impact on the loss . In the capacity limited regime however , the loss scales as O ( D−1 ) suggesting a ( significantly smaller ) marginal value of O ( D−2 ) for additional data . In this regime , the loss value is dominated by the model-dependent constant C and most of the improvement can be had by increasing the model size . There is a smooth phase transition between these two regimes at approximately CDD0 = 1 ( See Appendix C for an illustration ) . Thus , by increasing the model size ( which reduces C ) , one can push the transition to larger values of D and leverage the available data more efficiently .

Summary: The paper studies data scaling laws for NMT. Specifically, it investigates the following factors empirically: - model architecture - model sizes (different numbers of encoder or decoder layers) - data filtering - different types of manually-added noises.  The paper has performed a large number of studies to demonstrate some empirically-useful findings for training the NMT models.