From English to Foreign Languages: Transferring Pre-trained Language Models
Authors: Ke Tran
Venue:  (2020)
Keywords: pretrained language model
zero-shot transfer
parsing
natural language inference

Abstract: Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT-base model within a day and a foreign BERT-large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.

Under review as a conference paper at ICLR 2020
FROM ENGLISH To FOREIGN LANGUAGES: TRANSFERRING PRE-TRAINED LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre- trained models enables zero-shot transfer of NLP tasks from high resource lan- guages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain foreign BERT hase model within a day and a foreign BERT within two days. Furthermore, evaluat- ing our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and depen- dency parsing.
INTRODUCTION
Pre-trained models (Devlin et al., 2019 Peters et al. 2018) have received much of attention re- cently thanks their impressive results in many down stream NLP tasks. Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer. Zero-shot cross- Pinsulingual transfer has shown promising results for rapidly building applica- tions for low resource languages. WW & Dredze (2019) show the potential of mulilingual-BERT (Devlin et al., 2019) in zero-shot transter for large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing.
Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English (Radford et al. [2019] Dai et al.) 2019 Yang et 2019). The current state of affairs makes it difficult to translate advancements in pre- pre-training from English to non-Enge -onnEEnlish languages. To our best knowl- edge. there are only three available multilingual pre-trained models to date: (1) the multilingual- BERT (mBERT()) that supports 104 languages, (2) cross-lingual language model (XLM; Lample &
Conneau 2019 that supports 100 languages, and (3) Language Agnostic SEntence Representations (LASER: Artetxe& & Schwenk, 2019 3 that supports 93 languages. Among the three models, LASER based on neural machine translation approach and strictly requires parallel data train.
Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowl- edge learned by English pre-trained models to other languages? In this work, we develop tech- nique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way (Strubell et al. 2019). As the first step, we focus on building a bilingual language model (LM) of English and a target language. Starting from pre- pre-trained English LM, we learn the target language specific parameters (i.e., word embeddings), while keeping the encoder layers of the pre-trained English LM fixed. We then fine-tune both English and target model to obtain the bilingual LM. We apply our approach to autoencoding language models with masked language

---

Under review as a conference paper at ICLR 2020
model objective and show the advantage of the proposed approach in zero-shot transfer. Our main contributions this work are:
We propose a fast adaptation method for obtaining a bilingual BERTBASE of English and a target language within day using one Tesla V100 16GB GPU.
We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference (XNLI; Conneau al.) 2018) and universal de- pendency parsing. We show that our models offer competitive performance or even better that mBERT.
We illustrate that our bilingual LMs can serve as excellent feature extractor in supervised dependency parsing task.
BILINGUAL PRE-TRAINED LMS
We first provide some background of pre-trained language models. Let E. be English word- embeddings and (0) be the Transformer (Vaswani et al. 2017) encoder with parameters 0. Let ew,, denote the embedding of word w; (i.e., Ew, Ee [ We omit positional embeddings and bias for clarity, A pre-trained LM typically performs the following computations: (i) transform
sequence of input tokens to contextualized representations [CuLi..,, Cw.. (eee:...... ,w, 0), and (ii) predict an output word y/ at position (.,) x exp(c Cy.)
input tokens w, by replacing them with a special token [MASK]. It then predicts the original tokens Yr = w, from the corrupted to- kens. Autoregressive LM (GPT-2; Radford al. 2019) predicts the next token (y/ W:+1) given all the previous tokens. The recently proposed Xet model (Yang et al.) 2019) is an autoregressive LM that factorizes output with all possible permutations, which shows empirical performance im- provement over GPT-2 due to the ability to capture bidirectional context. Here we assume that the encoder performs necessary masking with respect each training objective.
Autoencoding LM (BERT; Devlin al. 2019) corrupts
Given an English pre-trained LM, we wish to learn bilingual LM for English and given target language l under a limited computational resource budget. To quickly build bilingual LM, directly adapt the English pre-traind model to the target model. Our approach consists of three steps. First, we initialize target language word-embeddings E, in the English embedding space such that embeddings of a target word and its English equivalents are close together ($2.1) Next, we create a target LM from the targer embeddings and the English encoder (O). We then fine-tune target embeddings while keeping V (0) fixed (.2.). Finally, we construct bilingual LM of E., E,, and Ve (0) and fine-tune all the parameters approach.
Figure illustrates the last two steps in our
Fine- Sene-tuning foreign embeddings
Fine-tuning both foreign English
n'ess
pas
pipe
JOLA1
pais
piase
Rioc
4
4
4
A
Englishh
Layers
Layara
Layera
[c:S]
eecL
[VASA
pank
UPb
Pipe
To-43
Cecck
iTost
[MASK]]
piae
[6LS]
[NACK3
DaLIton
Figure 1: Pictorial illustration of our approach. Left: fine-tune language specific parameters while keeping transformer encoder fixed. Right: jointly train bilingual LM and update all the parameters.
2.1 INITIALIZING TARGET EMBEDDINGS
Our approach learn the initial foreign word embeddings Eg E [ based the idea of mapping the trained English word embeddings E. E Rlvl to E, such that if foreign word and an English word are similar meaning then their embeddings are similar. Borrowing the idea of universal lexical sharing from Gu et al. (018), represent each foreign word embedding
2

---

Under review as a conference paper at ICLR 2020
Eg E Rd as a linear combination of English word embeddings Eelil R
V. ai
Edi
E.li E.
where O2; E IR. VVl is sparse vector and Σ
Vel Oki 1.
In this step of initializing foreign embeddings, having a good estimation of a could speed of the convergence when tuning the foreign model and enable zero-shot transfer (:5). In the following, we
discuss how to estimate a; Vi E f1, 2..... [Ve]} under two scenarios: (i) we have parallel data of and (ii) we only rely on English and foreign monolingual data.
Learning from Parallel Corpus Given an English-foreign parallel corpus, we can estimate word translation probability p(e, (,) for any (English-foreign) pair (e,, (.) using popular word-alignment (Brown et al. 1993) toolkits such as fast-align (Dyer et al. 2013). We then assign:
aij p(e; (4,)
(2)
Since a; is estimated from word alignment, it a sparse vector.
Learning from Monolingual Corpus For low resource languages, parallel data may not be avail- able. In this case, we rely only on monolingual data (e.g.. Wikipedias). We estimate word translation probabilities from word embeddings of the two languages. Word vectors of these languages can be learned using fastText (Bojanowski
al. (017)) and then are aligned into a shared space with En- glish (Lample et al.) 2018b Joulin et al. 2018). Unlike learning contextualized representations, learning word vectors is fast and computationally cheap. Given the aligned vectors Eg of foreign and Eg of English, we calculate the word translation matrix E RRV x as
A = sparsemax E,E.)
Here, use sparsemax (Martins & Astudillo 2016 instead of softmax. Sparsemax is sparse version softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to given foreign word. This property is desirable in our approach since leads to better initialization of the foreign embeddings.
2.2 FINE-TUNING TARGET EMBEDDINGS
After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word word-embeddings to obtain the foreign LM. We then fine-tune only foreign cord-embeddings on target monolingual data. The training objective is the same as the training objective of the English pre-trained LM (i.e., masked I.M for BERT). Since the trained encoder (0) is good capturing association, the purpose of this step is further optimize tar- get embeddings such that the target LM can utilized the trained encoder for association task. For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned.
2.3 FINE-TUNING BILINGUAL LM
We create a bilingual LM by plugging foreign language specific parameters to the pre-trained En- glish LM (Figure D. The new model has two separate embedding layers and output layers, for English and one for foreign language. The encoder layer in between is shared. We then fine-tune this model using English and foreign monolingual data. Here, keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer (33). In this step. the encoder parameters are also updated so that in can learn syntactic aspects (i.e., word order, morphological agreement) of the target languages.
3 ZERO-SHOT EXPERIMENTS
In the scope of this work, we focus on transferring autoencoding LMs trained with masked lan- guage model objective. We choose BERT and RoBERTa (Liu et al. 2019) as the source models
3

---

Under review as a conference paper at ICLR 2020
for building our bilingual LMs, named RAMEN for the ease of discussion. For each pre-trained model. we experiment with 12 layers (BERT BASE and RoBERTagAsE) and 24 layers (BERTLARGE and RoBERTa variants. Using BERT BASE allows to compare the results with mBERT model. Using BERTLARG and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source pre-trained model. RoBERTa is a recently published model that is similar to BERT architecturally but with an improved training procedure. By training for longer time, with bigger batches, on more data, and on longer sequences, RoBERTa matched or exceed previously published models including XLNet. We include RoBERTa in our experiments to validate the motivation of our work: with similar architecture, does a stronger pre-trained English model resulty in a stronger bilingual LM? We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingus lingual Natural Language Inference (XNLI) and (2) dependency parsing.
3.1 DATA
We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (az), Chinese (2h), Hindi (hi), and Vietnamese (vi). These languages belong to four different language families. French, Russian, and Hindi are Indo-European languages, similar to English. Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan and Austro-Asiatic family respectively. The choice of the six languages also reflects different training conditions depending the amount of monolingual data. French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource.
For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of Lample & Conneau (2019). Specifically, we use United Nations Parallel Corpus (Ziemski et al.) 2016) for en-ru, en en-ar, en zh, and en fr. We collect en-hi paral- lel data from IIT Bombay corpus (Kunchukuttan al. 2018) and en-vi data from OpenSubtitles 2018 For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vector from fastText (Bo- janowski et al. 2017) to estimate word translation probabilities ()..3). We align these vectors into common space using orthogonal Procrustes (Artetxe et al. 2016 Lample et al. 2018b Joulin et al, 2018). We only use identical words between the two languages the supervised signal. We use WikiExtractof to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs ($2.) We do lowercase remove accents in our data preprocessing pipeline.
We tokenize English using the provided tokenizer from pre-trained models For target languages, Pi we use fastB to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively. We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models. Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa.
We use XNLI dataset (Conneau et al. 2018) for classification task and Universal Dependencies v2.4 (UD: Nivre et al. 2019) for parsing task. Since language might have more than one tree- bank in Universal Dependencies. we use the following treebanks: en-ewt (English), fr.gss (French), rur rulsyntagrus (Russian) ar ar.padt (Arabic), vi_vtb (Vietnamese), hi hi.h (Hindi), and zh-gsd (Chinese).
Remark on BPE Lample et al. (2018a) show that sharing subwords between languages improves alignments between embedding spaces. Wu & Dredze (2019) observe strong correlation between the percentage of overlapping subwords and mBERT's performances for cross-lingual zero-shot transfer. However, in our current approach, subwords between source and target are not shared. A subword that is in both English and foreign vocabulary has two different embeddings.
'The finst author likes ramen.
httea//gittnbb..oom//mmmm///////
4

---

Under review as a conference paper at ICLR 2020
3.2 ESTIMATING TRANSLATION PROBABILITIES
Since pre-trained models operate subword level, we need to estimate subword translation proba- bilities. Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align (Dyer et al. 2013).
Estimating subword translation probabilities from aligned word vectors requires an additional pro- cessing step since the provided vectors from fastText are not at subword leve We use the follow- ing approximation to obtain subword vectors: the vector e₂ of subword is the weighted average of all the aligned word vectors Cwy that have subword
e,- = p(w))
Σ my: NEWs
Ewj
(4)
n
where p(wy) is the unigram probability of word wy and n. Ew sEw, p(wy). We take the top 50,000 words in each aligned word-vectors to compute subword vectors.
In both cases, not all the words in the foreign vocabulary can be initialized from the English word- embeddings. Those words are initialized randomly from Gaussian (200 1/e))
3.3 HYPER- PARAM-ERR
In all the experiments, we tune RAMEN for 175,000 updates and RAMEN for 275,000 updates where the first 25,000 updates are for language specific parameters. The sequence length is set to 256. The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN and RAMEN LARGE respectively. For tuming bilingual LMs, we use a mini-batch size of 64 for RAMEN BASE and 24 for RAMEN where half of the batch are English sequences and the other half are foreign sequences. This strategy of balancing mini-batch has been used in multilingual
neural machine translation (Firat al. 2016 Lee et al. 2017).
We optimize RAMEN BASE BAsE using Lookahead optimizer (Zhang et al. 2019) wrapped around Adam with the learning rate of 10-4, the number of fast weight updates k 5, and interpolation parameter = 0.5. We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam). For Adam optimizer, we linearly increase the learning rate from 10 to 10 in the first 4000 updates and then follow an inverse square root decay. All RAMEN, arGE models are optimized with Adam due to memory limit")
When fine-tuning RAMEN on XNLI and UD we use a mini-bateh size of 32, Adam's 's learning rate of 10-5 The number of epochs are set and 50 for XNLI and UD tasks respectively.
All experiments are carried out on a single Tesla V100 16GB GPU. Each RAMEN model is trained within a day and each RAMENLARGE is trained within two days
RESULTS
In this section, we present the results out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing.
4.1 CROSS-LINGUAL, NATURAL LANGUAGE INFERENCE
Table I shows the XNLI test accuracy. For reference, we also include the scores from the previ- ous work, notably the state-of-the-ar system XLM (Lample Conneau 2019). Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN +BERR trained with monolingual only.
In our preliminary experiments, learned the aligned subword vectors but it results in poor performances. Because Lookahead optimizer needs an extra copy of the model's 's parameters 1219 and 36 GPU hours, he precise. Learning alignment with fast-align takes less than 2 hours and we do account for training time of fastText vectors.
5

---

Under review as a conference paper at ICLR 2020
Fr
E1
V1
2b
Conneau 2018
67.7
65.4
64.8
64.1
66.4
65.8
Artetxe
2019
71.9
71.5
71.4
65.5
72.0
71.4
Lumple Conneau
2019
(MLM)
76.5
73.1
68.5
65.7
72.1
71.9
Lamplc Conncau
2019
MLMM)
78.7
75.3
73.1
69.6
76.1
76.5
mBERT (Wu & Dredze
2019
73.8
69.0
64.9
60.0
69.5
69.3
RAMENEASE
75.2
69.4
68.2
62.2
71.0
71.7
BERT
77.0
68.8
68.7
62.8
74.0
70.4
79.2
72.4
71.6
63.4
74.9
73.3
RoBERTa
78.2
73.1
72.0
65.0
73.7
74.2
RAMEN ARGE
78.1
71.2
72.4
76.0
73.3
BERT
71.7
71.0
66.1
75.5
81.7
73.6
72.8
79.2
RoBERTa
81.7
76.4
75.0
68.5
79.7
77.7
Table 1:
Zero-shot classification results on XNLI. M indicates parallel data is used. RAMEN only uses parallel data for initialization. The best results are marked in bold.
We first discuss the transfer results from BERT. Initialized from fastText vectors, RAMEN slightly outperforms mBERT by 1.9 points average and widen the gap of 3.3 points Ara- bic. RAMEN gains extra 0.8 points on average when initialized from parallel data. With triple number of parameters, RAMENLARGE offers an additional boost in term of accuracy and initializa- tion with parallel data consistently improves the performance. It has been shown that BERT large significantly outperforms BERT BASE on 11 English NLP tasks (Devlin al. 2019), the strength of BERTLARGE also shows when adapted to foreign languages.
Transferring from RoBERTa leads to better zero-shot accuracies. With the same initializing con- dition, RAMEN BRoMEERTTRR outperforms RAMEN on average by 2.9 and 2.3 points when initializing from monolingual and parallel data respectively. This result show that with sim- ilar number of parameters, our approach benefits from a better English pre-trained model. When transferring from ROBERTaLARGE: we obtain state-of-the-ard results for five languages.
Currently, RAMEN only uses parallel data to initialize foreign embeddings. RAMEN can also exploit parallel data through translation objective proposed XLM. We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks. We leave this exploration to future work. summary, starting from BERT BASE. our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI. Our approach shows the accuracy gains when adapting from a hetter pre-trained model.
4.2 UNIVERSAL DEPENDENCY PARSING
We build on top of RAMEN graph-based dependency parser (Dozat & Manning 2016). For the purpose of evaluating the contextual representations learned by our model, we do not use part-of- speech tags. Contextualized representations are directly fed into Biaffine layers predict are and label scores. Table parsing. Unlabeled Attachment Scores are provided in Appendix A.T
2 presents the Labeled Attachment Scores (LAS) for zero-shot dependency
We first look at the fairest comparison between mBERT and monolingually initialized RAMEN The latter outperforms the former on five languages except Arabic. We ob- serve the largest gain of +5.2 LAS for French. Chinese enjoys +3.1 LAS from our approach. With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RA- MEN+RoBERTa performs better than RAMEN+BERT for most of the languages. Arabic and Hindi benefit the most from bigger models. For the other four languages, RAMEN, ARGE renders a modest improvement over RAMEN RASE-

---

Under review as a conference paper at ICLR 2020
A
Fr
ruu
hi
vi
MBERT
71.6
65.2
36.4
30.4
35.7
26.6
RAMEN
76.8
66.1
32.9
33.0
36.8
BERT
77.2
66.7
35.1
35.1
37.3
78.4
66.5
37.4
33.9
39.1
30.0
RoBERTa
78.1
67.1
35.2
34.1
38.8
30.7
RAMEN,
78.2
61.0
38.8
36.2
37.2
31.3
BERT
78.1
65.7
36.4
37.4
38.2
31.3
79.4
68.5
39.3
32.5
39.7
31.6
RoBERTa
79.8
66.6
42.4
39.5
39.6
29.3
Table 2: LAS scores for zero-shot dependency parsing. ization. Punctuation are removed during the evaluation. The best results are marked in bold.
M indicates parallel data used for initial-
5 ANALYSIS
5.1 IMPACT OF INITIALIZATION
Initializing foreign embeddings the backbone of our approach. good initialization leads to better zero-shot transfer results and enables fast adaptation. To verify the importance of a good initializa- tion, we train RAMEN tRBBRTTa with foreign word-embeddings that are initialized randomly from from.N(0) Mo, 1/d²). For fair comparison, we use the same hyper-parameters in {3.3) Table shows the results of XNLI and UD parsing of random initialization. aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN by 15.9% for XNLI and 27.8 points for UD parsing on average. We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization.
comparison to the initialization using
Fr
a.
hi
vi
zh
A
XNLI
rnd
65.3
41.9
56.9
43.2
65.4
66.5
vec
79.2
72.4
71.6
63.4
74.9
73.3
15.9
rnd
27.8
17.5
9.3
5.3
33.9
24.2
vec
78.4
66.5
37.4
33.9
39.1
30.0
27.8
Table 3: Comparison between random initialization (rnd) of language specific parameters and ini- tialization using aligned fastText vectors (vec).
We also compare RAMEN a bilingual BERT trained from scratch at 1,000,000 updates. On average, RAMEN outperforms bilingual BERT by 10.5% accuracy on XNLI task and 13.9 LAS on dependency parsing. The detail scores are provided in Appendix A.2
5.2 ARE CONTEXTUAL REPRESENTATION FROM RAMEN ALSO GOOD FOR SUPERVISED PARSING?
All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks. It is reasonable expect RAMENS do well those tasks have shown in our experi- ments. But are they also a good feature extractor for supervised tasks? We offer partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets.
fr
hi
vi
zh
mBERT
92.1
93.1
83.6
91.3
62.2
85.1
RAMEN BASE
92.2
93.1
83.8
92.1
63.4
84.4
BERT
RAMEN LARGE
92.2
93.6
84.6
92.3
64.4
85.3
RAMEN BASE
92.7
93.5
85.1
92.3
65.3
85.7
RoBERTa
RAMEN LARGE
93.1
94.0
85.3
92.6
66.1
86.9
Table 4: Evaluation in supervised UD parsing. The scores LAS.
7

---

Under review as a conference paper at ICLR 2020
We used train/dew/tsst splits provided in UD to train and evaluate our RAMEN-basse parser. Ta- ble summarizes the results (LAS) of our supervised parser. For a fair comparison, we choose mBERT the baseline and all the RAMEN models initialized from aligned fastText vectors. With the same architecture of 12 Transformer layers, BAM performs competitive to mBERT and outshines mBERT by +1.2 points for Vietnamese. The best LAS results obtained by LARGE tRBERTA with 24 Transformer layers. Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks.
5.3 How DOES LINGUISTIC KNOWLEDGE TRANSFER HAPPEN THROUGH EACH TRAINING STAGES?
We evaluate the performance of RAMEN+ MOMMENN+RBERTaa.e. (initialized from monolingual data) at each training steps: initialization of word embeddings (OK update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates). The results are presented Figure
Zero-shot XNLI RoBERTa
Zero-shot parsing RoBERTA
fr
45.0
70.3
76.4
77.4
76.4
77.3
77.1
79.2
fr
5.9
50.3
78.3
78.0
78.8
78.6
78.2
78.4
72.4
nu
66.1
66.5
Language
37.4
55.7
69.5
67.8
69.6
71.6
70.8
71.6
0.9
4.2
34.7
36.6
36.0
35.3
35.7
37.4
37.7
52.1
61.3
62.5
63.8
64.3
60.8
63.4
6.4
33.5
35.4
32.6
36.7
33.9
vi
37.3
60.5
73.2
71.7
73.9
74.6
73.7
74.9
vi
2.6
6.4
37.3
38.1
38.9
39.1
38.1
39.1
39.9
59.9
70.6
72.7
72.0
70.2
71.9
73.3
3.4
5.6
30.2
30.3
31.1
31.3
31.1
30.0
25
50
75
100
125
150
175
25
50
75
100
125
150
175
Number of updates (x 1000)
Number of updates (x 1000)
Figure 2: Accuracy and LAS evaluated at each checkpoints.
Without fine-tuning, the average accuracy of XLNI 39.7% for three-ways classification task, and the average LAS score is 3.6 for dependency parsing. We see the biggest leap in the performance after 50K updates. While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder. This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps capture language specific structures. We observe a substantial gain in LAS from 25 to 40 LAS for languages just after 25K updates of the encoder.
Language similarities have more impact transferring syntax than semantics. Without tuning the English encoder, French enjoys 50.3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4.2 and 6.4 points using the SVO encoder. Although Chinese has SVO order, is often seen as head-final while English is strong head-initial. Perhaps, this explains the poor performance for Chinese.
CONCLUSIONS
In this work, we have presented simple and effective approach for rapidly building a bilingual LM under limited computational budget. Using BERT as the starting point, we demonstrate that our approach performs better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing. We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models. We also find that RAMEN also a powerful feature extractor in supervised dependency parsing. Finally, hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained trained English models to other languages.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019. doi: 10.1 a 00288.
Mikel Artetxe, Gorka Labaka, and Eneko Agiree Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2289-2294, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. doi: 10. 10.1162/tacl_a 2tt2//acclla0005555 a_00551.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The math- ematies of statistical machine translation: Parameter estimation. Computational Linguistics, 19 (2):263-311, 1993.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp- 2475-2485, Brussels, Belgium, October-November 2018. Association for Computational Lin- guisties.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xt: Attentive language models beyond a fixed-length context. In ACI, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. Deep biaffine attention for neural dependency parsing. In ICLR, 2016.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644-648, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine trans- lation with shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 866-875, San Diego, California, June 2016. Association for Computational Linguistics. doi: 18.18/ 1/116-1101.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 344-354, New Orleans, Louisiana, June 2018. Association for Com- putational Linguistics. doi: 10.1866//1/N18--1 1/N11/ 1032.
Armand Joulin, Piotr Bojanowski. Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in trans- lation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Languuge Processing, pp. 2979-2984,, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC-2018), Miyazaki, Japan, May 2018. European Languages Resources As- sociation (ELRA).

---

Under review as a conference paper at ICLR 2020
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. NeurIPS, 2019.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations, 201 8a.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018b.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5: 365-378, 2017. doi: 10. 1..11622aall..00000
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019.
André F. T. Martins and Ramón F. Astudillo. From softmax to sparsemax: A sparse model of at- tention and multi-label classification. In Proceedings of the 33rd International Conference on In- ternational Conference on Machine Learning Volume 48, ICML' 16, pp. 1614-1623, JMLR.org, 2016.
Joakim Nivre et al. Universal dependencies 2.4, 2019. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 11136655///// 1-N1811 1202.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 1665-3650, Florence, Italy, July 2019. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp.
Curran Associates, Inc., 2017.
Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 833-844, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-100
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. NeurIPS, 2019.
Michael R. Zhang, James Lucas, Geoffrey E. Hinton, and Jimmy Ba. Lookahead optimizer: k steps forward, step back. NewrIPS, 2019.
Michat Ziemski, Marcin Junczys-Dowmunt, Dowmunt, and Bruno Pouliquen. The united nations parallel cor- pus v1.0. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation ( (LREC 2016), Paris, France, may 2016. European Language Resources Association (ELRA). ISBN 978-2-9517408-9911
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1
UNLABELED ATTACHMENT SCORES AND LABEL ATTACHMENT SCORES
fr
hi
vi
zh
MBERT
82.7/71.6
78.8/65.2
53.3/36.4
43.7/30.4
55.4/35.7
53.6/26.6
RAMEN
84.1 /76.8
80.2/66.1
50.4/32.9
46.0/33.0
56.1/36.8
59.4/29.7
BERT
84.5/77.2
79.5/66.7
51.9/35.1
49.4/35.1
56.9/37.3
60.1/30.
85.5/78.4
81.9/6..5
55.9/37.4
48.7/33.9
59.6/39.1
59.7/30.0
RoBERTa
85.5/78.1
8144/7.1
53.6/35.2
49.9/34.1
58.7/38.8
61.2/30.7
RAMEN
85.1/78.2
74.2/61.0
55.5/38.8
48.4/36.2
56.2/37.2
61.8/31.3
+ BERT
84.7/78.1
79.5/65.7
51.5/36.4
50.5/37.4
58.8/38.2
61.1/31.3
86.5/79.4
82.8/68.5
53.6/39.3
49.3/32.5
58.9/39.7 36.0//9.5
61.0/3166
RoBBRTTa
86.5/79.8
81.5/66.6
58.6/42.4
59.7/39.6
/0.2/29.3
Table 5: UAS/LAS scores for zero-shot dependency parsing. initialization. Punctuation are removed during the evaluation.
M
indicates parallel data is used for
A.2 COMPARISTON TO BILINGUAL BERT
For each language pair, we train a bilingual BERT (b-BERT) from scratch using the hyperparame- ters provided in section [.33] We learn 60,000 BPE codes concatenation of English and foreign language monolingual data. All b-BERT models are trained for 1,000,000 updates, which signifi- cantly longer than RAMEN, Table show the results of XNLI and UD.
Task
Model
fr
hi
vi
zh
B-BERT
69.7
55.5
59.1
46.4
58.3
66.0
XNLI
RAMEN
79.2
72.4
71.6
63,4
74.9
73.3
10.5
B-BERT RAMEN
69.2
41.0
17.8
14.9
26.3
22.6
UD
78.4
66.5
37.4
33,9
39.1
30.0
13.9
Table 6: Comparison between b-BERT trained from scratch for 1,000,000 updates and RAMEN trained for 175,000 updates.


Reviews:
[{"review_id": "rylWllu1qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose a way to transfer a pre-trained English BERT model to a new language within a short amount of time. The key insight is to map English embeddings to the foreign language and have separate embeddings for both English and the foreign language. The resulting bilingual LM is evaluated for zero-shot transfer learning on two tasks: XNLI and dependency parsing.\n\nPros:\n- The authors provide good details into their hyperparameter settings and about how the obtain the foreign language word embeddings.\n- By leveraging existing pre-trained models, they’re able to do pre-training for their bilingual LM within 2 days.\n\nCons:\nI find that a key comparison point in this paper is missing, which is Bilingual BERT trained on just the two languages that are being considered for their RAMEN system. This is not a fair comparison while mBERT which is trained on 100+ languages is not.\nAll comparisons are not fair since a simple baseline of just training mBERT on two languages with monolingual data and with a shared WPM is not evaluated here.\nThe proposed system has an unfair advantage over mBERT since it’s initialized from BERT/RoBERTA and fine-tuned only on two languages. Hence most of the parameters are used for just the two languages while mBERT uses the parameters for 104 languages.\nGiven this unfair comparison, I’m not sure if we can draw a meaningful conclusion from all the experiments. \n\nRating justification:\nGiven the lack a fair comparison between the bilingual and multilingual BERT models, I don't think the conclusions are insightful.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lhNKE0Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to efficiently transfer pre-trained english language model to bilingual language model. The obtained representations are evaluated on downstream NLP task (natural language inference and dependency parsing) with state-of-the-art performances.\n\n\nPros:\n\n- Experiments clearly show that, using the proposed method, stronger pre-trained English embedding leads to stronger bilingual language model and thus to better performances for downstream foreign tasks.\n\nCons: \n\nWhile it is generally  intelligible, some structural modifications could be done to  improved the clarity of the paper. For instance, the method used to align foreign word vectors with English word vectors, when no aligned corpus is available, should appear sooner. It is described in 3.1 but should probably appear in 2.1 subsection Learning from Monolingual Corpus.\n\nMinor issues:\n\n- in section 3: RoBERA -> RoBERTa\n- in section 5.1: the third sentence is syntactically incorrect\n- in Conclusion: our approach produces better than -> our approach performs better than\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SyxkmrNAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to adapt a pretrained BERT model from English to another languages with a limited time/GPU budget. Evaluation on 6 target languages shows good performance for natural language inference and dependency parsing. \n\nConcretely, the proposed approach consists of, starting from a pretrained English language model, first training language-specific embeddings and then fine-tuning the entire pretrained model on English *and* the target language, using those embeddings. The language-specific embeddings are initialized based on the English embeddings (the authors propose two different ways for doing that).\n\nI like about the paper that the approach is simple and fast. The experiments seem reasonable, too. The only minor negative point is that the approach is not particularly exciting.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to transfer a pretrained language model in one language (English) to a new language. The method first learns word embeddings for the new language while keeping the the body of the English model fixed, and further refines it in a fine-tuning procedure as a bilingual model. Experiments on XNLI and dependency parsing demonstrate the benefit of the proposed approach.

R3 pointed out that the paper is missing an important baseline, which is a bilingual BERT model. The authors acknowledged this in their rebuttal and ran a preliminary experiment to obtain a first set of results. However, since the main claim of the paper depends on this new experiment, which was not finished by the end of the rebuttal period, it is difficult to accept the paper in its current state. In an internal discussion, R1 also agreed that this baseline is critical to support the paper.

As a result, I recommend to reject this paper for ICLR. I encourage the authors to update their paper with the new experiment for submission to future conferences (given consistent results).
Decision: Reject