Scaling Laws for Neural Machine Translation

1 INTRODUCTION Scaling properties of neural networks have long been an intriguing topic of study ( Ahmad & Tesauro , 1988 ; Amari et al. , 1992 ) . Along with the practical success of modern neural networks at scale , theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing ( Advani & Saxe , 2017 ; Rosenfeld et al. , 2019 ; Geiger et al. , 2020 ; Ghorbani et al. , 2020 ; Cho , 2020 ; Hutter , 2021 ; Bahri et al. , 2021 ; Loureiro et al. , 2021 ) . In particular , scaling model sizes , datasets and the total computation budget has been identified as a reliable approach to improve generalization performance on several machine learning tasks . For many of these tasks the scaling behavior of neural networks is highly predictable ; model fit or test loss can be described precisely as a function of its number of parameters ( Hestness et al. , 2017 ; Kaplan et al. , 2020 ; Henighan et al. , 2020 ; Hernandez et al. , 2021 ; Rosenfeld et al. , 2019 ) . Neural machine translation ( NMT ) has long enjoyed the benefits of scaling ( Huang et al. , 2019 ; Arivazhagan et al. , 2019 ; Lepikhin et al. , 2021 ) , but studies investigating the scaling behavior of NMT models are missing . We present a large-scale systematic study of scaling laws for encoder-decoder Transformer models applied to NMT ( Vaswani et al. , 2017 ) . We start with highlighting the major differences between decoder-only language models , where the majority of the previous work has focused , and encoder-decoder ( conditional ) language models applied to NMT . The two differ along a few crucial dimensions . The first difference results from the very nature of the separate architectures being used , i.e . decoder-only vs encoder-decoder . The presence of separate architectural components complicates the study of scaling properties due to the increased degree of freedom . Second , contrary to language modeling , the task of machine translation is conditional : the task is predictive rather than fully generative . Furthermore , this prediction task is ambiguous : there is no one right answer for a given source , and translations can vary substantially depending on the translator ’ s incentives . This manifests itself as different scaling benefits for different test sets . To take an extreme example , a test set translated by someone who writes nearly wordfor-word translations may benefit less from model scaling than one translated by someone who considers each translation a work of art . In this work , these differences in difficulty coincide with the translation direction of the test set ; that is , whether the source was translated into the target ( source-original ) or vice versa ( target-original ) . Source-original data has translated text on the target side , which contains several artifacts of “ translationese ” that distinguish it from text originally written in that language , often lacking the diversity and complexity of “ natural ” text ( Koppel & Ordan , 2011 ) , while target-original data requires the prediction of more complex natural text on the target side . Finally , unlike language models , NMT is evaluated on metrics that quantify generation quality against reference translations ( for eg . BLEU ) ( Papineni et al. , 2002 ) instead of evaluating model fit ( perplexity ) on an evaluation set . In this paper , we aim to provide empirical answers to the following research questions : 1 . Does the encoder-decoder architecture for NMT share the same scaling law function as language models ? Contrary to previous work on LM , we show that a univariate law depending on the total number of parameters in the network does not adequately describe the scaling behavior of NMT models . Our scaling laws parameterize the cross entropy loss as a bivariate function of the number of encoder parameters and the number of decoder parameters as separate variables . Our results indicate that the scaling behavior is largely determined by the total capacity of the model , and the capacity allocation between the encoder and the decoder . 2 . How does the naturalness of source/target side data affect scaling behavior ? We study the effect of naturalness of the source and the target text , both for training and evaluation . When evaluating with target side natural text , scaling the model capacity continues improving model quality throughout our range of measurements . On the other hand , improvements on cross-entropy saturate ( or reaches the irreducible error region ) on source side natural evaluation sets even for moderately-sized models . 3 . Do scaling improvements in cross-entropy translate into corresponding improvements in generation quality ? Finally we study the relationship between generation quality and cross-entropy and how their correlation changes as we : ( i ) Scale different components of the model ( encoder vs decoder ) and ( ii ) Evaluate on source-natural or target-natural test sets . Our results on multiple language pairs and training/test data compositions validate that model scaling predictably improves the cross-entropy on validation data . However , our findings also raise several questions regarding the effect of naturalness of training and evaluation text and how cross-entropy eventually relates with generation quality for auto-regressive generative models . 1 2 EFFECT OF SCALING ON CROSS-ENTROPY 2.1 EXPERIMENTAL SETTING Model Architectures and Training We train a series of pre-layer norm Transformer networks with varying sizes ( Xiong et al. , 2020 ) . Models are trained with per-token cross-entropy loss and Adafactor optimizer ( Shazeer & Stern , 2018 ) . All models are trained with a fixed batch-size of 500k tokens and dropout rate of 0.1 for residuals , feed-forward activations and attention . All models are trained to near convergence for 500k training steps . Details of the model hyper-parameters are described in Appendix C. Model Scaling Transformer architecture consists of Transformer Blocks : a cascade of self-attention , cross-attention and feed-forward layers , each having multiple adjustable hyper-parameters ( e.g . model-dimension , number of attention heads , attention projection dimension etc. ) . Considering the combinatorial expansion of the search space for scaling each one , in this study we choose to vary only 1A few weeks before the publication of this manuscript on Arxiv , Gordon et al . ( 2021 ) appeared on OpenReview . While both papers study scaling laws for NMT , our studies focus on different parameter regimes ( 393K-56M vs 100M-4B ) . See Appendix A for a comparison of the results . the total number of Transformer Blocks , while keeping the internal hyper-parameters intact across different scales . In other words , we scale the depth of the Transformers while keeping width and other variables fixed . 2 We use GPipe pipeline parallelism for our experiments ( Huang et al. , 2019 ) . In an encoder-decoder Transformer architecture for NMT , depth scaling can naturally be implemented by varying encoder-decoder blocks independently or symmetrically . Hence , we examine the change in the cross-entropy loss as the number of parameters increase with three depth scaling approaches : Encoder Scaling : vary encoder depth ( 2 to 64 ) while decoder depth is fixed ( 6 layers ) . Decoder Scaling : vary decoder depth ( 2 to 64 ) while encoder depth is fixed ( 6 layers ) . Symmetric Scaling : increasing decoder and encoder layers together ( from 2 to 64 ) , i.e . the number of Transformer Blocks in the encoder and decoder being equal . For all experiments , configuration of the individual layers is unchanged : the model dimension , width of the feed-forward layer , and number of attention heads are fixed respectively at 1024 , 8192 , and 16 . 3 Each encoder layer adds approximately 20M parameters to the model while each decoder layer adds around 25M parameters . We train 200 such models which scale the model size by approximately a factor of 50 ( from 80M parameters to 4B parameters ) . Following the convention , we do not count the parameters in the embedding and softmax layers towards the model size . We focus our study on large-scale models : our smallest models require 200 TPUv3 days to train to convergence while our largest models require 2700 TPUv3 days of training . Language Pairs We report results on four language pairs : English→German , German→English , English→Chinese , and Chinese→English . We use in-house web-crawled training datasets with around 2.2 billion sentence pairs ( approximately 55 billion tokens ) for English↔German and 781 million sentence pairs for English↔Chinese . Evaluation Sets We use a variety of test sets for evaluation covering different domains : ( i ) WebDomain ( ii ) News-Domain ( iii ) Wikipedia ( iv ) Patents . The news-domain test sets come from the WMT2019 ( Barrault et al. , 2019 ) evaluation campaign ( newstest2019 ) for all language pairs . The other test sets are internal test sets representing the different domains , ranging from 500 to 5000 sentence pairs . For each domain , we randomly sample sentences in the source language and use professional translators to generate a reference translation in the target language . Throughout the paper , we will refer this type of test sets as source-original as the source sentences have been crawled from the web while the reference translations are added later . For most of the domains , we also have a target-original counterpart which is generated in the opposite direction : Sentences are crawled in the target language and human translated into the source language . Earlier work ( Freitag et al. , 2019 ; 2020 ; Graham et al. , 2020 ) showed that it is important to differentiate between the two different kinds of test sets as the style of natural sentences and human ( or machine ) translations ( translationese ) is quite different . Cross-entropy loss is evaluated on the different test sets during training . To reduce the variation caused by the parameter fluctuations at the end of the training , we present the median loss over the last 50k steps of the training as the final loss . 2.2 RESULTS Figure 1 shows the empirical evolution of the test loss on the Web-Domain test sets for encoder and decoder scaling for English→German . To compare the empirical results with the scaling laws present in the literature for decoder only models ( Kaplan et al. , 2020 ; Henighan et al. , 2020 ) , we have fitted a power law of the form L̂ ( N ) = αN−p+L∞ to the data . 4 Here , N is the total number of parameters outside of embedding / softmax layers and { α , p , L∞ } are fitted parameters . As Figure 1 suggests , scaling the encoder has different effects on the test loss compared to scaling the decoder . As such , simple power-laws that only consider the total parameter count , fail to capture the correct scaling behavior of the model . 2In Appendix H , we examine the role of scaling approach in our results . See Section 2 for more details . 3A complete description of the model architecture is provided in Appendix C 4Details of the curve fitting procedure are presented in Appendix I. laws are unable to capture the correct scaling behavior . R-squared ( 100× explained variancetotal variance ) and maximum absolute deviation ( ‖ · ‖∞ ) are reported for each fit . Proposed Scaling Law To tackle this issue , we present a new scaling law that reflects the encoderdecoder nature of the architecture as well as the bilingual format of the data . Let Ne and Nd be the number of non-embedding parameters in the encoder and the decoder respectively . Then , our proposed scaling law has the form L̂ ( Ne , Nd ) = αN −pe e N −pd d + L∞ ( 1 ) where { α , pe , pd , L∞ } are test set specific ( fitted ) parameters . In this formulation , α corresponds to the maximum loss reduction that one can hope from scaling , while pe and pd are the scaling exponents for encoder and decoder respectively . L∞ corresponds to the irreducible loss of the data . In Figure 2 , we fit our scaling law to the empirical loss values achieved by encoder/decoder scaling models on Web-Domain test sets . The results suggest that our scaling law almost fully captures the variation in the data ( R2 > 99 % ) . In particular , in contrast to traditional scaling laws , our proposed formulation is able to capture the different scaling behaviors of encoder and decoder scaled models . While these improved fits are encouraging , to truly show that a candidate scaling law captures the model scaling behavior in NMT , it is necessary to verify the out-of-sample prediction power of the law . We provide two such verifications ; we compare the predictions of the scaling law fitted on encoder / decoder scaling models with the empirical loss values from ( i ) symmetrically scaled models , ( ii ) models randomly sampled from the 2D grid of all potential encoder-decoder configurations.5 Symmetrically scaled models are chosen for this verification as they are widely used by the NLP community ( Raffel et al. , 2020 ) . Randomly sampled models are chosen to ensure that our results are not biased towards any particular model class . Figures 3 & 4 present the out-of-sample results . As the plots suggest , the predictions of the scaling law match the empirical results with remarkable accuracy . These results suggest that Eq . ( 1 ) is able to capture the model scaling behavior across different scaling approaches and model sizes . To ensure that our results generalize across different language pairs , we examine the fit of our scaling law on encoder / decoder scaling models trained on German→English ( De→En ) , Chinese-to-English ( Zh→En ) , and English-to-Chinese ( En→Zh ) translation tasks . Figure 5 presents the results evaluated on in-domain held-out data . Similar to the En→De case , our proposed functional form is able to closely capture the scaling behavior of these models . To keep the scope of our experiments manageable , we focus on depth scaling in our analysis . In Appendix H , we ran large-scale experiments to see if other scaling approaches show significant deviations from the trends identified here . For large-scale models , which are the focus of this study , we observe that different scaling approaches yield very similar results : the final achieved test loss across different scaling approaches shows just 1− 2 % variation . We observed that for small models ( < 100M parameters ) , this variation tends to grow , suggesting that model shape plays a key role in the performance of small models . We leave further analysis of this phenomenon to future research . 5See Appendix F for the full description of the models . 2.3 ANALYSIS The above results suggest that the scaling law formalized in Eq . ( 1 ) captures the scaling behavior of the Transformer NMT models in multiple language pairs . As such , we can study the fitted coefficients to fully understand the scaling properties of these models . Several observations are in order : Decoder vs Encoder Scaling : On all our test sets , the decoder exponents were observed to be larger than the encoder exponents , pd > pe ( see Figure 9 ) . As a result , when improving the test loss is concerned , it is much more effective to scale the decoder rather than the encoder . This is contrary to the usual practice ; due to latency considerations , many practitioners train NMT models with deep encoders and shallow decoders ( Kasai et al. , 2021 ) . Our results suggest this practice could be sub-optimal in terms of loss reduction . Proposition 1 below provides guidance on how to allocate parameters in between the encoder and decoder optimally . The proof is presented in Appendix G. Proposition 1 ( Optimal Scaling ) . Assume the loss performance of the model is described by Eq . ( 1 ) . Let B denote the budget for the total number of parameters . Then , the optimal encoder / decoder sizes ( denoted respectively by N∗e and N ∗ d ) are N ∗ e = pe pe+pd B and N∗d = pd pe+pd B . In addition , when optimally scaling the model , the scaling law reduces to L̂opt ( B ) = α∗B− ( pd+pe ) +L∞ , where α∗ ≡ α ( pe+pdpe ) pe ( pe+pdpd ) pd . Proposition 1 suggests that when NeNd = pe pd , the optimal scaling behavior can be achieved . Inspection of the functional form of Eq . ( 1 ) suggests that as long as Nd/Ne is fixed as the model scales ( i.e . encoder and decoder grow proportionally together ) , the optimal scaling exponent , ( pe + pd ) , can be achieved , albeit with a potentially sub-optimal multiplicative constant , α # . To examine the magnitude of this sub-optimality , in Figure 10 , we compare the multiplicative constants resulting from proportional scaling of the encoder and decoder with different values of Nd/Ne . The results suggest that as long as the parameter allocation is not extremely far from ( N∗e , N ∗ d ) , the scaling scheme is approximately optimal . In particular , symmetrically scaling the encoder and decoder layers , is barely distinguishable from the optimal scaling scheme . 3 EFFECT OF DATASET COMPOSITION BIAS ON SCALING BEHAVIOR Translation deals with the problem of mapping a sequence in one language into another language . A good translation should not only be adequate and fluent , but should ideally also adopt the style of a sentence naturally written in the target language . This necessitates MT models to make sense of natural looking inputs and generate natural looking outputs . As mentioned in Section 2 , the examples used to train or test NMT models carry a critical bias , which we refer to as composition bias . Composition bias is introduced because of the unavailability of source-target examples ( pairs ) that are both natural in the accessible data generating distribution . For any given naturally generated text in a language , the corresponding text in the other language is either translated by humans , introducing translationese bias or translated by other machine translation systems , introducing MT bias . We consider both biases affecting the problem from a similar angle , hence we bundle them and call it composition bias . While machine translation by design has composition bias in the training/test sets employed ( Freitag et al. , 2020 ; Riley et al. , 2020 ) , its effect on model scaling is unknown . In this section we investigate the role of composition bias in scaling and identify critical factors playing a role . We caution the reader to not take the composition bias as a problem specific to NMT . In fact as most training corpora in NMT are web-crawled , they can contain machine translation output on either the source or target side . Considering the growth of generated content in the web by machine learning models 6 7 , it is not improbable that a proportion of the content collected and used by machine learning models is going to be biased by other models that are continuously generating content . The Effect of Test Set Construction : We first examine the impact of composition bias on the test sets . Figure 9 shows the fitted scaling exponents for all our test sets . The plot suggests that the scaling powers for source-original test sets are drastically different from those of target-original test sets . This behavior is in direct contrast with language modeling setting ( Kaplan et al. , 2020 ) where it was observed that the evaluation on different test sets merely acted as a scaling penalty that only changed the multiplicative constants of the scaling law . To elucidate this phenomenon further , in Figure 11 of the appendix , we compare the scaling trends for different source and target original test sets . Several observations are in order : Test sets with a similar composition approach ( source or target original ) have a qualitatively similar scaling behavior . However , scaling behavior is vastly different between the two composition approaches . Reducible loss quickly decays to zero for source original test sets . In fact , we observe that scaling our baseline 6L-6L model by a factor of 2.5 is sufficient for ensuring that reducible loss is below 0.05 for all source original test sets . In contrast , on target original test sets , the loss decays much more slowly with model size . For comparison , to ensure that reducible loss is below 0.05 for all target original test sets , we estimate that the baseline model has to be scaled up by a factor of 11 . Because of this behavior , the value of larger models in NMT is closely tied to their evaluation sets : On source original data , due to larger scaling exponents , even moderate increases in model size are sufficient for pushing the reducible loss close to zero . Hence , beyond a few hundred million parameters , there is no benefit in increasing the model size . In contrast , for target original data , which generally have smaller scaling exponents , large models are needed to push the reducible loss to zero . The Effect of Training Set Construction : In this section , we briefly examine the role of training data construction on the scaling behavior . To do this , we generate two En→De datasets , that were not used in the previous experiments . One fully target original and another completely source original . To generate the target original dataset , we compile a set of German documents from the web . Documents are screened to ensure the data is not machine generated . We use a Hybrid model ( with 380M parameters ) ( Chen et al. , 2018 ) to back-translate ( BT ) these documents to English . Similarly , for the source original data , we collect human generated English documents and ( forward ) translate them to German using a hybrid model ( with approximately 327M parameters ) . Both datasets provide us with approximately 2.2 billion training examples . We mimic the experimental setup of Section 2 . Note that even though these datasets are not human generated , they reflect important aspects of training large NMT models . Many modern NMT datasets are harvested from the web and as a result , are contaminated with machine generated data . Moreover , many popular data augmentation algorithms such as Back Translation ( Sennrich et al. , 2016 ) , sequence level distillation ( Kim & Rush , 2016 ) and self training ( He et al. , 2020 ) purposefully add machine generated data into the training pipeline in order to take advantage of monolingual data . Figure 6 describes the scaling behavior for models trained on target-original data . We observe that even though larger models are successful in reducing the training loss , they are unable to improve the test loss after roughly 400M parameters . Once this size threshold is exceeded , models overfit the training data and the test loss starts to deteriorate across all of our test sets . We hypothesize that this size threshold corresponds to the capacity of the original back-translation model . This assertion suggests that in order for back-translation to be beneficial for training large models , it has to be performed with a models with comparable capacity or higher . Although quite intriguing , we leave the verification of this hypothesis to future work . Figure 7 paints another interesting picture for the models trained on the source-original data only , implying the target side having the composition bias , expected to be simpler , dull and not rich in its content , in short - not natural looking . As experiments suggest , even our smallest models are able to achieve extremely low loss values ( roughly 0.16 ) , with an apparent overfitting pattern . We believe the same phenomenon is also related to the `` data simplification '' effect seeked by non-autoregressive models in NMT ( Zhou et al. , 2021 ) . 6https : //openai.com/blog/gpt-3-apps/ 7https : //blog.google/products/translate/one-billion-installs/ 4 EVOLUTION OF GENERATION QUALITY We examine the effects of scaling on the output quality as measured by BLEU score 8 . For this analysis , we focus on output generated via beam search ( Wu et al. , 2016 ) . For tractability purposes , we do not attempt to tune the ( many ) hyper-parameters of beam-search for each model . Instead , we use the configuration optimized for the baseline model ( listed in Appendix J ) in all the decoding tasks . Figure 8 presents the co-evolution of BLEU score and cross-entropy loss throughout the training for all of our models . Depending on the construction of the test sets , two different empirical behaviors emerge . On target-original test sets , larger models are able to improve ( lower ) the test loss . These improvements in the loss are accompanied with consistent improvements ( increases ) in BLEU score . In fact , we observe that a simple power law can capture the relationship between BLEU score and cross-entropy loss for high-quality models . 9 In contrast , on source-original test sets , this relationship is absent ; larger models consistently achieve better test losses , however , beyond a certain threshold , BLEU scores begin to deteriorate . Figures 27 and 28 exhibit that this phenomenon is not due to over-training ; the BLEU score gap between large and small models is persistent throughout training . To ensure that this observation truly reflects the generation quality of the models ( as opposed to potential biases of BLEU score ) , we repeat our analysis with BLEURT score ( Sellam et al. , 2020a ; b ) . The results are presented in Appendix K. The results show that BLEURT scores closely mirror the behavior of BLEU scores with respect to model scaling . A careful look at the left-subplots of Figures 8 brings up another interesting trend . At similar values of the test loss , encoder-scaled models result in better generation quality compared to decoder-scaled models . This findings agrees with previous work that relied on encoder-scaling when optimizing for BLEU and inference latency ( Kasai et al. , 2021 ) . Whether these differences in the effects of encoder-scaling and decoder-scaling are caused by insufficient search algorithms , or just different model fits from different architectural priors is left to future work . 8We computed the BLEU scores using an internal reimplementation of Moses scorer : mteval-v13a.pl . 9Fitting details are presented in Appendix J . 5 CONCLUSION AND LIMITATION In this work we have attempted to quantify the evolution of model quality as a function of model capacity for encoder-decoder NMT models . While a univariate scaling law describing the crossentropy as a function of the total number of parameters in the model is insufficient , a bivariate law treating the number of encoder and decoder parameters as separate variables adequately describes the scaling behavior of these models under various scaling strategies . We validate this behavior on a variety of language pairs and evaluation sets . Whether this behavior is intrinsic to the encoder-decoder architecture , or arising from the nature of the NMT task , requires further study . Next , we show that this scaling behavior is highly dependent on the composition of the evaluation data , specifically on whether the source or target sentences are “ original ” . Our findings indicate that target-original evaluation sets continue benefiting from model scaling , while the reducible error on source-original evaluation sets quickly saturates to 0 . This could be an artifact of the lack of diversity in translated text ; a simpler target distribution doesn ’ t require much capacity to model while generating fluent or natural-looking text could benefit much more from scale . We also study how the composition of training data affects the scaling behavior of models . When training on target-original ( back-translated ) text , model quality keeps improving until a point after which the trend saturates . In our study the capacity where saturation manifests first is perilously close to the capacity of the model used for back-translation , indicating that the capacity of the generative model used to generate synthetic text might have a role to play , but this requires further investigation . When training on source-original text , even low-capacity models are sufficient to reach the irreducible loss region , painting a gloomy picture for synthetic data . While we have explored these ideas in the context of machine translation , given the proliferation of generative models this problem will likely be a challenge for future practitioners training on web-scraped monolingual datasets as well . For low-resource languages , the proliferation of machine translated text is already a problem given that a significant portion of web text in these languages is machine translated . Finally , we attempt to understand how generation quality evolves with the improvements in crossentropy resulting from model scaling . As with our previous findings , dataset composition plays a major role in determining the trends . For source-original evaluation sets , the correlation between cross-entropy and generation quality breaks down . On target-original evaluation , we observe an inverse correlation between cross-entropy and BLEU , suggesting that improved model fit results in a corresponding improvement in generation quality . The slope of this relationship is different for encoder-scaling and decoder-scaling , with encoder-scaled models performing better on BLEU than decoder-scaled models , at the same level of cross-entropy loss . Whether this is an artifact of our search strategy or the difference in architectural priors is something that requires further investigation . Our findings suggest that scaling behavior of encoder-decoder NMT models is predictable , but the exact formulation of scaling laws might vary depending on the particular architecture or task being studied . Our empirical findings also raise concerns regarding the effect of synthetic data on model scaling and evaluation , and how proliferation of machine generated text might hamper the quality of future models trained on web-text . REFERENCES Madhu S. Advani and Andrew M. Saxe . High-dimensional dynamics of generalization error in neural networks , 2017 . Subutai Ahmad and Gerald Tesauro . Scaling and generalization in neural networks : a case study . Advances in Neural Information Processing Systems , 1:160–168 , 1988 . S. Amari , Naotake Fujita , and S. Shinomoto . Four types of learning curves . Neural Computation , 4 : 605–618 , 1992 . Naveen Arivazhagan , Ankur Bapna , Orhan Firat , Dmitry Lepikhin , Melvin Johnson , Maxim Krikun , Mia Xu Chen , Yuan Cao , George F. Foster , Colin Cherry , Wolfgang Macherey , Zhifeng Chen , and Yonghui Wu . Massively multilingual neural machine translation in the wild : Findings and challenges . CoRR , abs/1907.05019 , 2019 . URL http : //arxiv.org/abs/1907.05019 . Yasaman Bahri , Ethan Dyer , Jared Kaplan , Jaehoon Lee , and Utkarsh Sharma . Explaining neural scaling laws , 2021 . Loïc Barrault , Ondřej Bojar , Marta R Costa-Jussa , Christian Federmann , Mark Fishel , Yvette Graham , Barry Haddow , Matthias Huck , Philipp Koehn , Shervin Malmasi , et al . Findings of the 2019 conference on machine translation ( wmt19 ) . In Proceedings of the Fourth Conference on Machine Translation ( Volume 2 : Shared Task Papers , Day 1 ) , pp . 1–61 , 2019 . Mia Xu Chen , Orhan Firat , Ankur Bapna , Melvin Johnson , Wolfgang Macherey , George Foster , Llion Jones , Niki Parmar , Mike Schuster , Zhifeng Chen , Yonghui Wu , and Macduff Hughes . The best of both worlds : Combining recent advances in neural machine translation , 2018 . Kyunghyun Cho . Scaling laws of recovering bernoulli . Blog Post , 2020 . URL http : // kyunghyuncho.me/scaling-law-of-estimating-bernoulli . Markus Freitag , Isaac Caswell , and Scott Roy . APE at Scale and Its Implications on MT Evaluation Biases . In Proceedings of the Fourth Conference on Machine Translation , pp . 34–44 , Florence , Italy , August 2019 . Association for Computational Linguistics . URL http : //www.aclweb . org/anthology/W19-5204 . Markus Freitag , David Grangier , and Isaac Caswell . BLEU might be guilty but references are not innocent . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pp . 61–71 , Online , November 2020 . Association for Computational Linguistics . doi : 10.18653/v1/2020.emnlp-main.5 . URL https : //www.aclweb.org/anthology/ 2020.emnlp-main.5 . Mario Geiger , Arthur Jacot , Stefano Spigler , Franck Gabriel , Levent Sagun , Stéphane d ’ Ascoli , Giulio Biroli , Clément Hongler , and Matthieu Wyart . Scaling description of generalization with number of parameters in deep learning . Journal of Statistical Mechanics : Theory and Experiment , 2020 ( 2 ) :023401 , Feb 2020 . ISSN 1742-5468. doi : 10.1088/1742-5468/ab633c . URL http : //dx.doi.org/10.1088/1742-5468/ab633c . Behrooz Ghorbani , Song Mei , Theodor Misiakiewicz , and Andrea Montanari . Linearized two-layers neural networks in high dimension , 2020 . Mitchell A Gordon , Kevin Duh , and Jared Kaplan . Data and parameter scaling laws for neural machine translation . In ACL Rolling Review - May 2021 , 2021 . URL https : //openreview . net/forum ? id=IKA7MLxsLSu . Yvette Graham , Barry Haddow , and Philipp Koehn . Statistical power and translationese in machine translation evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pp . 72–81 , 2020 . Junxian He , Jiatao Gu , Jiajun Shen , and Marc ’ Aurelio Ranzato . Revisiting self-training for neural sequence generation , 2020 . Tom Henighan , Jared Kaplan , Mor Katz , Mark Chen , Christopher Hesse , Jacob Jackson , Heewoo Jun , Tom B Brown , Prafulla Dhariwal , Scott Gray , et al . Scaling laws for autoregressive generative modeling . arXiv preprint arXiv:2010.14701 , 2020 . Danny Hernandez , Jared Kaplan , Tom Henighan , and Sam McCandlish . Scaling laws for transfer , 2021 . Joel Hestness , Sharan Narang , Newsha Ardalani , Gregory Diamos , Heewoo Jun , Hassan Kianinejad , Md Patwary , Mostofa Ali , Yang Yang , and Yanqi Zhou . Deep learning scaling is predictable , empirically . arXiv preprint arXiv:1712.00409 , 2017 . Yanping Huang , Youlong Cheng , Ankur Bapna , Orhan Firat , Dehao Chen , Mia Xu Chen , HyoukJoong Lee , Jiquan Ngiam , Quoc V Le , Yonghui Wu , et al . Gpipe : Efficient training of giant neural networks using pipeline parallelism . In NeurIPS , 2019 . Marcus Hutter . Learning curve theory . CoRR , abs/2102.04074 , 2021 . URL https : //arxiv . org/abs/2102.04074 . Jared Kaplan , Sam McCandlish , Tom Henighan , Tom B Brown , Benjamin Chess , Rewon Child , Scott Gray , Alec Radford , Jeffrey Wu , and Dario Amodei . Scaling laws for neural language models . arXiv preprint arXiv:2001.08361 , 2020 . Jungo Kasai , Nikolaos Pappas , Hao Peng , James Cross , and Noah A. Smith . Deep encoder , shallow decoder : Reevaluating non-autoregressive machine translation , 2021 . Yoon Kim and Alexander M. Rush . Sequence-level knowledge distillation , 2016 . Moshe Koppel and Noam Ordan . Translationese and Its Dialects . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies - Volume 1 , pp . 1318–1326 , 2011 . URL http : //dl.acm.org/citation.cfm ? id=2002472 . 2002636 . Dmitry Lepikhin , HyoukJoong Lee , Yuanzhong Xu , Dehao Chen , Orhan Firat , Yanping Huang , Maxim Krikun , Noam Shazeer , and Zhifeng Chen . { GS } hard : Scaling giant models with conditional computation and automatic sharding . In International Conference on Learning Representations , 2021 . URL https : //openreview.net/forum ? id=qrwe7XHTmYb . Bruno Loureiro , Cédric Gerbelot , Hugo Cui , Sebastian Goldt , Florent Krzakala , Marc Mézard , and Lenka Zdeborová . Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model , 2021 . Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . BLEU : a Method for Automatic Evaluation of Machine Translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , pp . 311–318 . Association for Computational Linguistics , 2002 . URL https : //www.aclweb.org/anthology/P02-1040 . Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J. Liu . Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Learning Research , 21 ( 140 ) :1–67 , 2020 . URL http : //jmlr.org/papers/v21/20-074.html . Parker Riley , Isaac Caswell , Markus Freitag , and David Grangier . Translationese as a language in “ multilingual ” nmt . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp . 7737–7746 , 2020 . Jonathan S Rosenfeld , Amir Rosenfeld , Yonatan Belinkov , and Nir Shavit . A constructive prediction of the generalization error across scales . In International Conference on Learning Representations , 2019 . Thibault Sellam , Dipanjan Das , and Ankur P Parikh . Bleurt : Learning robust metrics for text generation . arXiv preprint arXiv:2004.04696 , 2020a . Thibault Sellam , Amy Pu , Hyung Won Chung , Sebastian Gehrmann , Qijun Tan , Markus Freitag , Dipanjan Das , and Ankur P Parikh . Learning to evaluate translation beyond english : Bleurt submissions to the wmt metrics 2020 shared task . arXiv preprint arXiv:2010.04297 , 2020b . Rico Sennrich , Barry Haddow , and Alexandra Birch . Improving neural machine translation models with monolingual data . In ACL ( 1 ) , 2016 . Noam Shazeer and Mitchell Stern . Adafactor : Adaptive learning rates with sublinear memory cost . In International Conference on Machine Learning , pp . 4596–4604 . PMLR , 2018 . Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . Attention is all you need . CoRR , abs/1706.03762 , 2017 . URL http : //arxiv.org/abs/1706.03762 . Yonghui Wu , Mike Schuster , Zhifeng Chen , Quoc V Le , Mohammad Norouzi , Wolfgang Macherey , Maxim Krikun , Yuan Cao , Qin Gao , Klaus Macherey , et al . Google ’ s neural machine translation system : Bridging the gap between human and machine translation . arXiv preprint arXiv:1609.08144 , 2016 . Ruibin Xiong , Yunchang Yang , Di He , Kai Zheng , Shuxin Zheng , Chen Xing , Huishuai Zhang , Yanyan Lan , Liwei Wang , and Tieyan Liu . On layer normalization in the transformer architecture . In International Conference on Machine Learning , pp . 10524–10533 . PMLR , 2020 . Chunting Zhou , Graham Neubig , and Jiatao Gu . Understanding knowledge distillation in nonautoregressive machine translation , 2021 . A CONNECTIONS TO RECENT LITERATURE In this section , we provide a more detailed overview of the scaling laws literature immediately relevant to our investigation . Hestness et al . ( 2017 ) examine data scaling laws for LSTM-based encoder-decoder NMT models . They show that the model performance exhibits an empirical powerlaw relationship with the size of the training data . Rosenfeld et al . ( 2019 ) study model and data scaling laws for image classification and language modeling tasks . They provide a bivariate scaling law that jointly capture the model and data scaling behaviors for each of these tasks . Crucially , they show that scaling laws can be used to effectively extrapolate beyond model/data configurations used for fitting them . In particular , they show that one can use smaller models to fit the scaling law and then use the law to accurately predict the performance of larger models . Our results ( Figures 3 and 4 ) validate that this behavior also holds for our setting . Kaplan et al . ( 2020 ) study the scaling behavior of decoder-only Transformer models used in language modeling . They identify three factors that principally determine the model performance : the number of ( non-embedding ) model parameters , the size of the training dataset , and the amount of compute used for training . In the scenario where data and compute are abundant , they show that model performance exhibits an empirical power-law relationship with the total number of ( non-embedding ) parameters in the network . Henighan et al . ( 2020 ) verify this power-law behavior for other autoregressive generative modeling tasks such as image modeling , video modeling , and mathematical problem solving . In contrast , in our setting , we observe that the total number of parameters is not a good predictor of the test loss in NMT . Instead , we show that encoder-decoder NMT models scale according to a bivariate function of the number of encoder parameters and the number of decoder parameters . Kaplan et al . ( 2020 ) also examine how scaling affects the out-of-distribution generalization behavior in language models . Their results suggest that different test sets exhibit rather similar scaling trends as model size increases . In comparison , in the NMT setting , we show that different test sets , depending on their naturalness structure , exhibit drastically different scaling behaviors . The most closely related study to ours is the work of Gordon et al . ( 2021 ) . They also examine scaling laws for NMT , albeit at a different parameter regime : Gordon et al . focus on small models ( 393K56M parameters ) while here , we focus on large scale models ( 100M-4B parameters ) . In addition , we examine the general scenario where encoder and decoder can scale asymmetrically ( which is a common practice in NMT ) while Gordon et al . only consider symmetric scaling . Despite the differences in the experimental setup , when focusing on the symmetric scaling setting , the conclusions of both studies mostly align . In particular , when symmetrically scaling the model , our scaling law simplifies to L ( N ) = αN−p + L∞ which agrees with the conclusions of Gordon et al . One difference in our conclusions is the presence of the L∞ term in our scaling law . This term , which captures the irreducible loss in the learning problem , does not appear in Gordon et al ’ s analysis . We suspect that this might be due to the small models used in that analysis which can not statistically observe such asymptotic terms . Gordon et al . also examine the relationship between test cross-entropy loss and BLEU score . They observe that as models scale and cross-entropy loss improves , BLEU score also improves in tandem . In comparison , we unveil a more intricate relationship in our study : We show that when the test data is target-original , improvements in the test cross-entropy loss are correlated with improvements in generation quality ( as measured by BLEU/BLEURT scores ) . However , when the test set is sourceoriginal , the correlation between cross-entropy loss and generation quality breaks down . In this setting , we observe that generation quality improves up until a certain threshold and then deteriorates ( see the right plot in Figure 8 ) . B MISCELLANEOUS FIGURES

Summary: The paper provides interesting scaling properties for autoregressive neural machine translation models, which are validated on comprehensive experiments. The authors investigate the scaling properties from three perspectives: 1. Quantify the evolution of model quality as a function of model capacity for encoder-decoder NMT models. 2. How the composition of training/test data affects the scaling behavior of NMT models? 3. How generation quality evolves with the improvements in cross-entropy resulting from model scaling?