XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering

1 INTRODUCTION . Recent work on pretraining natural language processing systems ( Devlin et al. , 2018 ; Radford et al. , 2018 ; Howard & Ruder , 2018 ; Peters et al. , 2018 ; McCann et al. , 2017 ) has led to improvements across a wide variety of natural language tasks ( Wang et al. , 2018 ; Rajpurkar et al. , 2016 ; Socher et al. , 2013 ; Conneau et al. , 2018 ) . For several of these tasks , data can be plentiful for high-resource languages like English , Chinese , German , Spanish and French , but both the collection and proliferation of data is limited for low-resource languages like Urdu . Even when a large language model is pretrained on large amounts of multilingual data ( Devlin et al. , 2018 ; Lample & Conneau , 2019 ) , languages like English can contain orders of magnitude more data in common sources for pretraining like Wikipedia . One of the most common ways to leverage multilingual data is to use transfer learning . Word embeddings such as Word2Vec ( Mikolov et al. , 2013b ) or GloVe ( Pennington et al. , 2014 ) use large amounts of unsupervised data to create task-agnostic word embeddings which have been shown to greatly improve downstream task performance . Multilingual variants of such embeddings ( Bojanowski et al. , 2017 ) have also shown to be useful at improving performance on common tasks across several languages . More recently , contextualized embeddings such as CoVe , ElMo , ULMFit and GPT ( McCann et al. , 2017 ; Peters et al. , 2018 ; Howard & Ruder , 2018 ; Radford et al. , 2018 ) have been shown to significantly improve upon aforementioned static embeddings . BERT ( Devlin et al. , 2018 ) employs a similar strategy by using a masked version of the language modeling objective . Unlike other approaches , BERT also provides a multilingual contextual representation which is enabled by its shared sub-word vocabulary and multilingual training data . Often , for languages for which large amounts of data is not available , aforementioned techniques for creating embeddings ( static or contextualized ) is not possible and additional strategies need to be employed . We demonstrate the effectiveness of cross-lingual data augmentation ( XLDA ) as a simple technique that improves generalization across multiple languages and tasks . XLDA can be used with both pretrained and randomly initialized models without needing to explicitly further align the embeddings . To apply XLDA to any natural language input , we simply take a portion of that input and replace it with its translation in another language . This makes XLDA compatible with recent methods for pretraining ( Lample & Conneau , 2019 ; Devlin et al. , 2018 ) . Additionally , the approach seamlessly scales for many languages and improves performance on all high- and low-resource languages tested including English , French , Spanish , German , Greek , Bulgarian , Russian , Turkish , Arabic , Vietnamese , Chinese , Hindi , Swahili and Urdu . This paper makes the following contributions : • We propose cross-lingual data augmenation ( XLDA ) , a new technique for improving the performance of NLP systems that simply replaces part of the natural language input with its translation in another language . • We present experiments that show how XLDA can be used to improve performance for every language in XNLI , and in three cases XLDA leads to state-of-the-art performance . • We demonstrate the ability of our method to improve exact-match and F1 on the SQuAD question-answering dataset as well . 2 BACKGROUND AND RELATED WORK . Multilingual Methods . Much prior work that seeks to leverage multilingual data attempts to first train word embeddings from monolingual corpora ( Klementiev et al. , 2012 ; Zou et al. , 2013 ; Hermann & Blunsom , 2014 ) and then align those embeddings using dictionaries between languages ( Mikolov et al. , 2013a ; Faruqui & Dyer , 2014 ) . Some instead train multilingual word embeddings jointly from parallel corpora ( Gouws et al. , 2014 ; Luong et al. , 2015 ) . Johnson et al . ( 2017 ) demonstrate how training multilingual translation in a single model can be used for zero-shot translation ( i.e. , for translation pairs with no parallel training data ) . This approach also attained state-of-the-art results for many languages . More recently , similar techniques have been adapted for extremely low resource languages ( Gu et al. , 2018 ) . Neubig & Hu ( 2018 ) showed how to further fine-tune a multilingual model by explicitly using a high-resource language with a linguistically related low-resource language to improve translation quality . More recently , Conneau et al . ( 2017 ) and Artetxe et al . ( 2018 ) show how to obtain cross-lingual word embeddings through entirely unsupervised methods that do not use any dictionaries or parallel data . Natural Language Inference . The Multi-Genre Natural Language Inference ( MultiNLI ) corpus ( Williams et al. , 2017 ) uses data from ten distinct genres of English language for the the task of natural language inference ( prediction of whether the relationship between two sentences represents entailment , contradiction , or neither ) . XNLI ( Conneau et al. , 2018 ) is an evaluation set grounded in MultiNLI for cross-lingual understanding ( XLU ) in 15 different languages that include low-resource languages such as Swahili and Urdu . XNLI serves as the primary testbed for our proposed method , XLDA , which improves over the baseline model in all languages and achieves state-of-the-art performance on Greek , Turkish , and Urdu even without state-of-the-art pretraining Lample & Conneau ( 2019 ) introduced concurrently to our work . Question Answering . We also include experiments on the Stanford Question Answering Dataset ( SQuAD ) ( Rajpurkar et al. , 2016 ) . This dataset consists of context-question-answer triplets such that the answer is completely contained , as a span , in the context provided . For this task we translate only the training set into 4 languages using a neural machine translation system . Due to the fact that ( machine or human ) translation may not necessarily retain span positions , the translation of this dataset is more nuanced than the classification datasets discussed previously . For this reason , we do not tamper with the original SQuAD validation data ; the test set is not publicly available either , so we constrain ourselves to a setting in which XLDA is used at training time but the target language remains English during validation and testing . Unsupervised Language Models . Recently , large , pretrained , unsupervised language models have been used for XNLI , SQuAD , and the GLUE benchmark ( Wang et al. , 2018 ) to improve performance across the board . BERT ( Devlin et al. , 2018 ) pretrains deep bidirectional representations by jointly conditioning on both left and right contexts in all layers . BERT can then be fine-tuned for a specific task with an additional output layer . BERT achieved significant improvements on both XNLI and SQuAD , and is this model that serves as the base for the application of XLDA across these tasks . Back-translation . Akin to XLDA , back-translation is often used to provide additional data through the use of neural machine translation systems . As a recent example , QANet ( Yu et al. , 2018 ) employed this technique to achieve then state-of-the-art results on question answering , and machine translation systems ( Sennrich et al. , 2016 ) regularly use such techniques to iteratively improve through a backand-forth process . XLDA also translates training data from a source language into a target language , but XLDA does not translate back into the source language . In fact , experimental results show that XLDA provides improved performance over using even the original source data , let alone a noisier version provided through back-translation . This indicates that signal in multiple languages can be beneficial to training per se rather than only as an intermediary for back-translation . 3 XLDA : CROSS-LINGUAL DATA AUGMENTATION . Let D = { ( xi , yi , zi ) } be a dataset of input text sequences xi and yi with labels zi . We create a new dataset Dlm = { ( x ( l ) i , y ( m ) i , zi ) } , where x ( l ) i is the translation of xi into language l and y ( m ) i the translation of yi into language m by a neural machine translation system . When l = m , all inputs are in the same language , which is the monolingual setting . When a training set is the union of multiple Dll , this is the disjoint , multilingual setting ( DMT ) as multiple languages are used for training , but each individual example only contains one language . When l 6= m , each input is in a different language , which is the cross-lingual setting . Experiments below demonstrate that the cross-lingual setting can improve upon the monolingual and DMT settings for natural language inference and a combination of monolingual and cross-lingual training yields the best results for span-extractive question answering . 4 EXPERIMENTS AND RESULTS . We experiment with using various subsets of DXLDA and demonstrate empirically that some provide better learning environments than the monolingual and disjoint , multilingual settings . First , we provide details on how different translation systems T were used to generate cross-lingual datasets for both tasks under consideration . 4.1 CROSS-LINGUAL DATA . The first task we consider in these experiments is MultiNLI ( Williams et al. , 2017 ) . The XNLI ( Conneau et al. , 2018 ) dataset provides the disjoint , multilingual version of MultiNLI necessary for XLDA . To create the XNLI dataset , the authors used 15 different neural machine translation systems ( each for a different language pair ) to create 15 separate single language training sets . The validation and test sets were translated by humans . The fact that XNLI is aligned across languages for each sentence pair allows our method to be trained between the examples in the 15 different training sets . Since XLDA only pertains to training , the validation and test settings remain the same for each of the 15 languages , but future work will explore how this method can be used at test time as well . Because we have human translated validation and test sets for XNLI , it is the primary task under examination in our experiments . We follow a similar process for the Stanford Question Answering Dataset , SQuAD ( Rajpurkar et al. , 2016 ) . Given SQuAD is a span-extraction problem , translation of the training set required special care . Questions , context paragraphs , and gold answers were translated separately . We then used exact string matching between the translated answers and the translated context paragraphs to determine if the translated answer could still be found in the translated context . If the answer exists , we take its first instance as the ground truth translated span . 65 % of German , 69 % of French , 70 % of Spanish , and 45 % of Russian answer spans were recoverable in this way . To translate the rest of the questions we placed a special symbol on both sides of the ground truth span in the English context paragraph before translation . Occasionally , even with this special symbol , the translated answers could not be recovered from the marked , translated context . Additionally , the presence of the symbol did influence gender and tense in the translation as well . Using this approach on examples that failed span-recovery in the first phase , we were able to recover 81 % of German , 96 % of French , 97 % of Spanish , and 96 % of Russian answer spans . However , adding this additional data from the marked , translations led to worse performance across all of the languages . Hence , we only use the translated examples from the first phase for all training settings below , effectively reducing training set size . The validation and test splits for SQuAD remain in English alone1 . This still allows us to explore how XLDA can be used to leverage multilingual supervision at train time for a single target language at validation time .

Summary: The paper provides an analysis of a cross-lingual data augmentation technique dubbed XLDA, which consists of replacing parts of an input text with its translation in another language. Building on the mBERT approach, the authors show that at fine-tuning time it is beneficial to augment the training set of XNLI with cross-lingual hypotheses and premises instead of in-language pairs. For each language in XNLI, they show results by augmenting with each of the 14 other languages in the dataset, and show significant improvements over per-language performance.