Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs

1 INTRODUCTION . End-to-end neural machine translation ( NMT ) ( Bahdanau et al. , 2014 ; Sutskever et al. , 2014 ; Luong et al. , 2015 ; Sennrich et al. , 2015b ; Vaswani et al. , 2017 ; Britz et al. , 2017 ; Gehring et al. , 2017 ; Klein et al. , 2017 ; Johnson et al. , 2017 ; Wang et al. , 2017 ; Hassan et al. , 2018 ; Aharoni et al. , 2019 ; Ng et al. , 2019 ) has been widely adopted as the state-of-the-art approach for MT . For bilingual MT , sequence-to-sequence models ( SEQ2SEQ for short ) are trained and tested on publicly available benchmarks , the size of which ranges from tens of thousand for low-resource languages to tens or hundreds of million for widely used languages . Recent progress in natural language understanding ( NLU ) has proved that large-scale pretraining such as BERT ( Devlin et al. , 2018 ) or Elmo ( Peters et al. , 2018 ) will lead to a significant leap-forward in SOTA results . Mostly due to the lack of bilingual training data ( Song et al. , 2019 ) no comparable success has been made in the field of MT , and two important questions have mostly remained unanswered : WHETHER and HOW we can push the performance of existing neural models if we increase the amount of bilingual data by orders of magnitude , e.g. , if we are presented with ten of billions of bilingual training sentence pairs . The answers to these questions are not immediately clear and several aspects distinguish this problem from all previous NMT works : • Scale : Firstly , an NMT model ’ s expressivity is limited by infrastructures such as GPU memory , and indefinitely increasing the size of the training data might not improve the performance . Secondly , training on a massively large dataset with tens of billion sentence pairs can be prohibitively slow . • Noise and Out-of-domain data : A dataset with tens of billions of bilingual sentences pairs must span a wide range of domains and comes from extremely noisy resources . It is widely accepted that translation quality is very vulnerable to out-of-domain data and noisy data ( Chen et al. , 2016 ; Niehues & Waibel , 2010 ; Koehn & Schroeder , 2007 ; Eidelman et al. , 2012 ) . and small amounts of noisy training instances can have negative effects on translation quality ( Belinkov & Bisk , 2017 ) . This means blindly increasing the size of training data or adding noisy training data may not necessarily lead to better performances , but may lead to worse performances . In this paper , we investigate the problem of training neural machine translation systems on a dataset with more than 40 billion bilingual sentence pairs , which is orders of magnitude larger than the largest dataset to date . To tailor existing WMT systems to the massively large training dataset , we propose a pipelined strategy which involves large-scale pretraining and domain-specific finetuning . We handle the trade-off between full-dataset optimality and convergence speed for pretraining and demonstrate that the combination of large-scale pretraining will significantly improve NMT performances . Combing other existing NMT techniques ( Hassan et al. , 2018 ; Ranzato et al. , 2015 ; Li et al. , 2016 ) , we are able to push to the BLEU score of WMT 2017 Chinese-English translation to 32.2 , with a significant performance boost of +3.1 over existing SOTA results . 2 RELATED WORK . Large scale pretraining has proved to be of significant importance in NLP , from the early time of word vectors such as word2vec or Glove ( Pennington et al. , 2014 ; Mikolov et al. , 2013 ) to recent language model pretraining such as BERT ( Devlin et al. , 2018 ) or Elmo ( Peters et al. , 2018 ) . Pretraining has not achieved comparable success in MT . This is mostly because of the difficulty to get a large parallel corpus . Most NMT pretraining work focuses on unsupervised NMT ( Lample & Conneau , 2019 ) or pretraining using monolingual data ( Song et al. , 2019 ) . Other relevant works for SEQ2SEQ pretraining include using an auto-encoder to pretraining the encoder-decoder network ( Dai & Le , 2015 ; Ramachandran et al. , 2016 ) and transfer learning from rich-source languages to low-source languages ( Zoph et al. , 2016 ; Firat et al. , 2016 ) . Our work is related to a group of previous work in domain adaptation for MT . In the context of phrase-based MT , Hildebrand et al . ( 2005 ) selects sentences similar to the topic of the test set to construct a new training corpus to avoid topic discrepancies between the training and test datasets . Xiao et al . ( 2012 ) estimate word translation probabilities conditioned on topics , and then adapt lexical weights of phrases by these topic-specific probabilities . In the context of NMT , Chen et al . ( 2016 ) feeds neural networks with topic information ( which on human-labeled product categories ) on the decoder side ; Zhang et al . ( 2016b ) first runs topic models ( Blei et al. , 2003 ) on the training data ( both sources and targets ) , and incorporate topic representations into the encoder-decoder model . Mixture models have been widely studied used in MT . In the context phrase-based MT . Foster & Kuhn ( 2007 ) propose a three-step pipelined strategy which first splits , the training corpus into different sub-corpora according to some predefined criterion , then trains different MT models on different sub-corpora , and in the end combine different models for translation . Foster & Kuhn ( 2007 ) ’ s work lated was extended in subsequent work ( Niehues & Waibel , 2010 ; Koehn & Schroeder , 2007 ; Eidelman et al. , 2012 ) , regarding different constituent aspects such as how to split the training corpus ( Axelrod et al. , 2011 ) and how to combine different models ( Civera & Juan , 2007 ; Sennrich , 2012 ; Foster et al. , 2010 ) . In NMT , mixture models ( Shen et al. , 2019 ; He et al. , 2018 ) are inspired by deep latent variable generation models ( Kingma & Welling , 2013 ; Kim et al. , 2018 ; Bowman et al. , 2015 ) . Zhang et al . ( 2016a ) augment NMT systems with a single Gaussian latent variable , and this work was further extended by Schulz et al . ( 2018 ) which associated each target word with a latent Gaussian variable . He et al . ( 2018 ) propose to use a soft mixture model to generate diverse translations . Further , Shen et al . ( 2019 ) comprehensively evaluate different design choices of mixture models such as parameterization and prior distribution . Due to the massively large dataset , we have to split it for training . This makes this paper relevant to a wide range of previous work on distributed training for deep networks ( Dean et al. , 2012 ; Yadan et al. , 2013 ; Li et al. , 2014 ; Krizhevsky , 2014 ; Das et al. , 2016 ; Smith et al. , 2017 ) . 3 DATA SETUP . The most commonly used Chinese-to-English ( Zh-En ) translation dataset is the WMT ’ 17 . The dataset consists of 332K sentence pairs from the News Commentary corpus , 15.8M sentence pairs from the UN Parallel Corpus , and 9M sentence pairs from the CWMT Corpus . We followed the pre-processing criterion in ( Hassan et al. , 2018 ) , resulting in about 20M training pairs . Newsdev2017 is used as the development set and Newstest2017 as the test set . In addition to WMT2017 , we collected a Chinese-English parallel dataset that significantly extends those studied by previous works . The original dataset consists of roughly 50 billion sentence pairs in total . The data comes from from diverse sources such as web pages ( ∼2 billion ) , digitized books ( ∼1 billion ) and private purchase from translation agencies ( ∼46 billion ) . The dataset is extremely noisy in two ways : ( 1 ) significant proportion of files are of the PDF format . ( 2 ) texts are aligned at the document level rather than sentence level . The size of large documents can be up to thousands of pages , with figures , tables and charts annoyingly inserted . For the first part , we developed a PDF document parsing system to decode PDF files . Since this part are out of the scope of this paper , we omit the details . Succinctly , a lexical analyzer is first used to decompose PDF files into the basic lexical tokens according to PostScript syntax . Next , we build a parser to analyze the abstract syntax tree ( AST ) , and then decode the data into figures , tables , and texts . For the second part , our goal is to transform doc-level alignments to sentence-level alignments . We use a hierarchical pipeline which consists of two stages : ( 1 ) aligning paragraphs and ( 2 ) aligning sentences within each aligned paragraph . We adopted many of the techniques in Uszkoreit et al . ( 2010 ) : Paragraphs/sentences are discarded if both sides are identical or a language detector declares them to be in the wrong language . We used a standard dynamic programming sentence/paragraph alignment algorithm using sentence length and translation probability as features . Pairs are discarded if pairing scores are less than a certain threshold . We encourage the readers to refer to ( Uszkoreit et al. , 2010 ) for details . After post-processing , we are left with 41 billion sentence pairs . We randomly select 1M instances as the development set . 4 MODELS AND ARCHITECTURES . 4.1 PIPELINES : PRETRAINING AND FINE-TUNING . Translation quality is very vulnerable to out-of-domain and noisy data ( Chen et al. , 2016 ; Niehues & Waibel , 2010 ; Koehn & Schroeder , 2007 ; Eidelman et al. , 2012 ) . Since the dataset we use comes from very noisy resources and covers a wide range of domains , it would be less favorable if we directly apply a trained model on this large but noisy dataset to the test set . One option to handle this issue is do data selection and filtering before training ( Belinkov & Bisk , 2017 ; Hassan et al. , 2018 ) . Hassan et al . ( 2018 ) proposes to first learn sentence representations from the provided training data in WMT2017 ( target domain ) , and then reject training instances if their similarity with below a specified threshold . We did not choose this choice for two reasons : ( 1 ) Hassan et al . ( 2018 ) selects different training instances for different target domains . This means every time we encounter a new domain , we have to retrain the model ; ( 2 ) the value of data filtering threshold is crucial but hard to decide : it is unrealistic to tune its value since each threshold value corresponds to a different filtered training set , on which a brand new model has to be trained . Inspired by large-scale pretraining strategies such as BERT ( Devlin et al. , 2018 ) or Elmo ( Peters et al. , 2018 ) , we used a pipelined approach : we first pretrain an NMT model on the massively large dataset , and then fine-tune the model on the training set of the target domain . This strategy naturally addresses the aforementioned two caveats of data pre-filtering approach : the pretrained model can be easily adapted to a set of training data of an arbitrary domain , and there is no issue of data selection threshold . Moreover , since the model will be fine-tuned in the latter stage , it is more immune to the data noisiness in the first stage . We used the combination of WMT 20M data and 40B data for pretraining , and then fine-tune the model on the 20M data .

Summary: The paper proposes an approach to train NMT models on extremely large parallel corpora. Because of the dataset size, training several epochs on the full dataset with a single model is too expensive. As a result, the dataset is split into several chunks, on which different models are trained independently. The different models are combined to form an ensemble model. Different strategies are proposed to split the dataset effectively. The resulting model achieves a performance of 32.3, outperforming the previous SOTA by 3.2 BLEU.