From English to Foreign Languages: Transferring Pre-trained Language Models

1 INTRODUCTION . Pre-trained models ( Devlin et al. , 2019 ; Peters et al. , 2018 ) have received much of attention recently thanks to their impressive results in many down stream NLP tasks . Additionally , multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer . Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages . Wu & Dredze ( 2019 ) show the potential of multilingual-BERT ( Devlin et al. , 2019 ) in zero-shot transfer for a large number of languages from different language families on five NLP tasks , namely , natural language inference , document classification , named entity recognition , part-of-speech tagging , and dependency parsing . Although multilingual models are an important ingredient for enhancing language technology in many languages , recent research on improving pre-trained models puts much emphasis on English ( Radford et al. , 2019 ; Dai et al. , 2019 ; Yang et al. , 2019 ) . The current state of affairs makes it difficult to translate advancements in pre-training from English to non-English languages . To our best knowledge , there are only three available multilingual pre-trained models to date : ( 1 ) the multilingualBERT ( mBERT ) 1 that supports 104 languages , ( 2 ) cross-lingual language model ( XLM ; Lample & Conneau , 2019 ) 2 that supports 100 languages , and ( 3 ) Language Agnostic SEntence Representations ( LASER ; Artetxe & Schwenk , 2019 ) 3 that supports 93 languages . Among the three models , LASER is based on neural machine translation approach and strictly requires parallel data to train . Do multilingual models always need to be trained from scratch ? Can we transfer linguistic knowledge learned by English pre-trained models to other languages ? In this work , we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way ( Strubell et al. , 2019 ) . As the first step , we focus on building a bilingual language model ( LM ) of English and a target language . Starting from a pre-trained English LM , we learn the target language specific parameters ( i.e. , word embeddings ) , while keeping the encoder layers of the pre-trained English LM fixed . We then fine-tune both English and target model to obtain the bilingual LM . We apply our approach to autoencoding language models with masked language 1https : //github.com/google-research/bert/blob/master/multilingual.md 2https : //github.com/facebookresearch/XLM 3https : //github.com/facebookresearch/LASER model objective and show the advantage of the proposed approach in zero-shot transfer . Our main contributions in this work are : • We propose a fast adaptation method for obtaining a bilingual BERTBASE of English and a target language within a day using one Tesla V100 16GB GPU . • We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks , namely natural language inference ( XNLI ; Conneau et al. , 2018 ) and universal dependency parsing . We show that our models offer competitive performance or even better that mBERT . • We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task . 2 BILINGUAL PRE-TRAINED LMS . We first provide some background of pre-trained language models . Let Ee be English wordembeddings and Ψ ( θ ) be the Transformer ( Vaswani et al. , 2017 ) encoder with parameters θ . Let ewi denote the embedding of word wi ( i.e. , ewi = Ee [ w1 ] ) . We omit positional embeddings and bias for clarity . A pre-trained LM typically performs the following computations : ( i ) transform a sequence of input tokens to contextualized representations [ cw1 , . . . , cwn ] = Ψ ( ew1 , . . . , ewn ; θ ) , and ( ii ) predict an output word yi at ith position p ( yi|cwi ) ∝ exp ( c > wieyi ) . Autoencoding LM ( BERT ; Devlin et al. , 2019 ) corrupts some input tokens wi by replacing them with a special token [ MASK ] . It then predicts the original tokens yi = wi from the corrupted tokens . Autoregressive LM ( GPT-2 ; Radford et al. , 2019 ) predicts the next token ( yi = wi+1 ) given all the previous tokens . The recently proposed XLNet model ( Yang et al. , 2019 ) is an autoregressive LM that factorizes output with all possible permutations , which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context . Here we assume that the encoder performs necessary masking with respect to each training objective . Given an English pre-trained LM , we wish to learn a bilingual LM for English and a given target language ` under a limited computational resource budget . To quickly build a bilingual LM , we directly adapt the English pre-traind model to the target model . Our approach consists of three steps . First , we initialize target language word-embeddings E ` in the English embedding space such that embeddings of a target word and its English equivalents are close together ( §2.1 ) . Next , we create a target LM from the target embeddings and the English encoder Ψ ( θ ) . We then fine-tune target embeddings while keeping Ψ ( θ ) fixed ( §2.2 ) . Finally , we construct a bilingual LM of Ee , E ` , and Ψ ( θ ) and fine-tune all the parameters ( §2.3 ) . Figure 1 illustrates the last two steps in our approach . 2.1 INITIALIZING TARGET EMBEDDINGS . Our approach to learn the initial foreign word embeddings E ` ∈ R|V ` |×d is based on the idea of mapping the trained English word embeddings Ee ∈ R|Ve|×d to E ` such that if a foreign word and an English word are similar in meaning then their embeddings are similar . Borrowing the idea of universal lexical sharing from Gu et al . ( 2018 ) , we represent each foreign word embedding E ` [ i ] ∈ Rd as a linear combination of English word embeddings Ee [ j ] ∈ Rd E ` [ i ] = |Ve|∑ j=1 αijEe [ j ] = αiEe ( 1 ) where αi ∈ R|Ve| is a sparse vector and ∑|Ve| j αij = 1 . In this step of initializing foreign embeddings , having a good estimation of α could speed of the convergence when tuning the foreign model and enable zero-shot transfer ( §5 ) . In the following , we discuss how to estimate αi ∀i ∈ { 1 , 2 , . . . , |V ` | } under two scenarios : ( i ) we have parallel data of English-foreign , and ( ii ) we only rely on English and foreign monolingual data . Learning from Parallel Corpus Given an English-foreign parallel corpus , we can estimate word translation probability p ( ej | ` i ) for any ( English-foreign ) pair ( ej , ` i ) using popular word-alignment ( Brown et al. , 1993 ) toolkits such as fast-align ( Dyer et al. , 2013 ) . We then assign : αij = p ( ej | ` i ) ( 2 ) Since αi is estimated from word alignment , it is a sparse vector . Learning from Monolingual Corpus For low resource languages , parallel data may not be available . In this case , we rely only on monolingual data ( e.g. , Wikipedias ) . We estimate word translation probabilities from word embeddings of the two languages . Word vectors of these languages can be learned using fastText ( Bojanowski et al. , 2017 ) and then are aligned into a shared space with English ( Lample et al. , 2018b ; Joulin et al. , 2018 ) . Unlike learning contextualized representations , learning word vectors is fast and computationally cheap . Given the aligned vectors Ē ` of foreign and Ēe of English , we calculate the word translation matrixA ∈ R|V ` |×|Ve| as A = sparsemax ( Ē ` Ē > e ) ( 3 ) Here , we use sparsemax ( Martins & Astudillo , 2016 ) instead of softmax . Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word . This property is desirable in our approach since it leads to a better initialization of the foreign embeddings . 2.2 FINE-TUNING TARGET EMBEDDINGS . After initializing foreign word-embeddings , we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM . We then fine-tune only foreign word-embeddings on target monolingual data . The training objective is the same as the training objective of the English pre-trained LM ( i.e. , masked LM for BERT ) . Since the trained encoder Ψ ( θ ) is good at capturing association , the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task . For example , if the words Albert Camus presented in a French input sequence , the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned . 2.3 FINE-TUNING BILINGUAL LM . We create a bilingual LM by plugging foreign language specific parameters to the pre-trained English LM ( Figure 1 ) . The new model has two separate embedding layers and output layers , one for English and one for foreign language . The encoder layer in between is shared . We then fine-tune this model using English and foreign monolingual data . Here , we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer ( §3 ) . In this step , the encoder parameters are also updated so that in can learn syntactic aspects ( i.e. , word order , morphological agreement ) of the target languages . 3 ZERO-SHOT EXPERIMENTS . In the scope of this work , we focus on transferring autoencoding LMs trained with masked language model objective . We choose BERT and RoBERTa ( Liu et al. , 2019 ) as the source models for building our bilingual LMs , named RAMEN4 for the ease of discussion . For each pre-trained model , we experiment with 12 layers ( BERTBASE and RoBERTaBASE ) and 24 layers ( BERTLARGE and RoBERTaLARGE ) variants . Using BERTBASE allows us to compare the results with mBERT model . Using BERTLARGE and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source pre-trained model . RoBERTa is a recently published model that is similar to BERT architecturally but with an improved training procedure . By training for longer time , with bigger batches , on more data , and on longer sequences , RoBERTa matched or exceed previously published models including XLNet . We include RoBERTa in our experiments to validate the motivation of our work : with similar architecture , does a stronger pre-trained English model results in a stronger bilingual LM ? We evaluate our models on two cross-lingual zero-shot tasks : ( 1 ) Cross-lingual Natural Language Inference ( XNLI ) and ( 2 ) dependency parsing . 3.1 DATA . We evaluate our approach for six target languages : French ( fr ) , Russian ( ru ) , Arabic ( ar ) , Chinese ( zh ) , Hindi ( hi ) , and Vietnamese ( vi ) . These languages belong to four different language families . French , Russian , and Hindi are Indo-European languages , similar to English . Arabic , Chinese , and Vietnamese belong to Afro-Asiatic , Sino-Tibetan , and Austro-Asiatic family respectively . The choice of the six languages also reflects different training conditions depending on the amount of monolingual data . French and Russian , and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource . For experiments that use parallel data to initialize foreign specific parameters , we use the same datasets in the work of Lample & Conneau ( 2019 ) . Specifically , we use United Nations Parallel Corpus ( Ziemski et al. , 2016 ) for en-ru , en-ar , en-zh , and en-fr . We collect en-hi parallel data from IIT Bombay corpus ( Kunchukuttan et al. , 2018 ) and en-vi data from OpenSubtitles 20185 . For experiments that use only monolingual data to initialize foreign parameters , instead of training word-vectors from the scratch , we use the pre-trained word vectors6 from fastText ( Bojanowski et al. , 2017 ) to estimate word translation probabilities ( Eq . 3 ) . We align these vectors into a common space using orthogonal Procrustes ( Artetxe et al. , 2016 ; Lample et al. , 2018b ; Joulin et al. , 2018 ) . We only use identical words between the two languages as the supervised signal . We use WikiExtractor7 to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs ( §2.3 ) . We do not lowercase or remove accents in our data preprocessing pipeline . We tokenize English using the provided tokenizer from pre-trained models8 . For target languages , we use fastBPE9 to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively . We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models . Precisely , the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa . We use XNLI dataset ( Conneau et al. , 2018 ) for classification task and Universal Dependencies v2.4 ( UD ; Nivre et al. , 2019 ) for parsing task . Since a language might have more than one treebank in Universal Dependencies , we use the following treebanks : en ewt ( English ) , fr gsd ( French ) , ru syntagrus ( Russian ) ar padt ( Arabic ) , vi vtb ( Vietnamese ) , hi hdtb ( Hindi ) , and zh gsd ( Chinese ) . Remark on BPE Lample et al . ( 2018a ) show that sharing subwords between languages improves alignments between embedding spaces . Wu & Dredze ( 2019 ) observe a strong correlation between the percentage of overlapping subwords and mBERT ’ s performances for cross-lingual zero-shot transfer . However , in our current approach , subwords between source and target are not shared . A subword that is in both English and foreign vocabulary has two different embeddings . 4The first author likes ramen . 5http : //opus.nlpl.eu/ 6https : //fasttext.cc/docs/en/crawl-vectors.html 7https : //github.com/attardi/wikiextractor 8https : //github.com/huggingface/pytorch-transformers 9https : //github.com/glample/fastBPE

Summary: In this work, the authors propose a way to transfer a pre-trained English BERT model to a new language within a short amount of time. The key insight is to map English embeddings to the foreign language and have separate embeddings for both English and the foreign language. The resulting bilingual LM is evaluated for zero-shot transfer learning on two tasks: XNLI and dependency parsing.