Pre-training via Leveraging Assisting Languages and Data Selection
for Neural Machine Translation

arXiv:2001.08353v1 [cs.CL] 23 Jan 2020

Haiyue Song1 , Raj Dabre2 , Fei Cheng1 , Zhuoyuan Mao1 , Sadao Kurohashi1 and Eiichiro
Sumita2
1
Kyoto University
2
National Institute of Information and Communications Technology{song, feicheng, zhuoyuanmao, kuro}@nlp.ist.i.kyoto-u.ac.jp,{raj.dabre, eiichiro.sumita}@nict.go.jp
Abstract
Sequence-to-sequence (S2S) pre-training usinglarge monolingual data is known to improve performance for various S2S NLP tasks in low-resource
settings. However, large monolingual corpora
might not always be available for the languages ofinterest (LOI). To this end, we propose to exploit
monolingual corpora of other languages to complement the scarcity of monolingual corpora for
the LOI. A case study of low-resource Japanese–
English neural machine translation (NMT) reveals
that leveraging large Chinese and French monolingual corpora can help overcome the shortage ofJapanese and English monolingual corpora, respectively, for S2S pre-training. We further show how
to utilize script mapping (Chinese to Japanese) toincrease the similarity between the two monolingual corpora leading to further improvements intranslation quality. Additionally, we propose simple data-selection techniques to be used prior to
pre-training that significantly impact the quality ofS2S pre-training. An empirical comparison of our
proposed methods reveals that leveraging assistinglanguage monolingual corpora, data selection andscript mapping are extremely important for NMT
pre-training in low-resource scenarios.

1

Introduction

Neural Machine Translation (NMT) [Sutskever et al., 2014;
Bahdanau et al., 2015] is known to give state-of-the-art(SOTA) translations for language pairs with an abundance ofparallel corpora. However, most language pairs are resource
poor (Russian–Japanese, Marathi–English) as they lack large
parallel corpora and it is possible to compensate the lack ofbilingual training data by leveraging large monolingual corpora. One popular approach for this is data augmentation,e.g. by back-translation [Sennrich et al., 2016a] of monolingual data to produce pseudo-parallel corpora. The limitation of this approach is that the initial MT systems used
for back-translation should be robust enough to yield pseduoparallel corpora of decent quality. Recently, another approachhas gained popularity where the NMT model is pre-trained

through tasks that only require monolingual data [Song et al.,2019; Qi et al., 2018].
Pre-training has seen a surge in popularity in NLP eversince models such as BERT [Devlin et al., 2018] have led to
new state-of-the-art results in text understanding. However,
BERT-like models were not designed to be used for NMT in
the sense that they are essentially language models and not
sequence to sequence (S2S) models. To address this, Songet al. [2019] recently proposed MASS, a S2S specific pretraining task for NMT and obtained new state-of-the-art results in low-resource settings. MASS assumes that a large
amount of monolingual data is available for the languages involved but this may not always be true. Such language pairsthat lack both parallel corpora and monolingual corpora are“truly low-resource” and challenging.
Fortunately, languages are not isolated and often belong
to “language families” where they have similar orthography (written script; shared cognates) or similar grammar orboth. Guzmán et al. [2019] leveraged linguistically similar resource-rich Hindi to improve translation involving
Nepalese through unsupervised NMT and semi-supervised
NMT. Motivated by this, in this paper we hypothesize that weshould be able to leverage large monolingual corpora of other(assisting) languages to help the monolingual pre-training of
NMT models for the languages of interest (LOI) that maylack monolingual corpora. Methods such as MASS focus
more on the pre-training task but not as much on the data
being used. In the past, data pre-processing and filtering
methods shown to impact MT performance as they minimize
the “differences” between the datasets used for training MT
systems [Axelrod et al., 2011]. Following this, we further
hypothesize that subjecting the pre-training corpora to script
mapping and simple but rigorous data-selection techniquesshould help minimize the vocabulary and distribution differences, respectively, between the pre-training, main training (fine-tuning) and testing time datasets. This should help
the already consistent pre-training and fine-tuning objectives
leverage the data much better and thereby, possibly, boost
translation quality.
To this end, we experiment with ASPEC Japanese–English
translation in a variety of low-resource settings for the parallel
corpora. We focus on sequence to sequence pre-training using MASS that uses a variety of sizes of Japanese and English

monolingual corpora filtered with data-selection techniques
to establish strong baselines. We then simulate low monolingual corpora situations for Japanese and English and complement the lack of corpora using Chinese (for Japanese) and
French (for English). Our experiments reveal that while it’s
possible to leverage unrelated languages for pre-training,
using related languages is extremely important. We identified that, it is absolutely important to maximize the similarities between the assisting languages and the languages ofinterest. We showed that Chinese to Japanese script mapping
can boost translation quality by maximum 7.4 BLEU in a realistic low-parallel and monolingual corpus setting.
The contributions of our work are as follows:
1. Leveraging assisting languages: We give a novel studyof leveraging monolingual corpora of related and unrelated
languages for NMT pre-training.
2. Pre-processing: We show the strong impact of robust andrigorous data-selection and script mapping on the translation
quality.
3. Empirical evaluation: We make a comparison of existingand proposed techniques in a variety of corpora settings toverify our hypotheses.
4. Analysis: We analyze our results and share the lessons wehave learned.

2

Related work

Our research is at the intersection of works on monolingual
pre-training for NMT and leveraging multilingualism for lowresource language translation.
In monolingual pre-training1 approaches, all or part of amodel is first trained on tasks that require monolingual data.
Pre-training has enjoyed great success in other NLP tasks
with the development of methods like BERT [Devlin et al.,2018]. To address this, Song et al. [2019] recently proposed MASS, a new state-of-the-art NMT pre-training taskthat jointly trains the encoder and the decoder. Our approachbuilds on the initial idea of MASS, but focuses on complementing the potential scarcity of monolingual corpora for the
languages of interest using relatively larger monolingual corpora of other (assisting) languages.
On the other hand, leveraging multilingualism involves
cross-lingual transfer which solves the low-resource issue
by using data from different language pairs. One can use
a richer language pair [Zoph et al., 2016], or several language pairs at once [Dabre et al., 2019; Dong et al., 2015].
Murthy et al. [2018] also proposed to re-order the assistinglanguages to be similar to the low-resource language. Dabreet al. [2017] showed the importance of transfer learning between languages belonging to the same language family but
corpora might not always be available in a related language.
A mapping between Chinese and Japanese characters [Chuet al., 2012] was shown to be useful for Chinese–Japanesedictionary construction [Dabre et al., 2015]. Such mappings
between scripts or unification of scripts [Hermjakob et al.,1This is an instance of “transfer learning” just like Cross-lingualtransfer. “Pre-training” often implies that the training task differsfrom the target task.

Monolingual data
Assistinglanguages

Targetlanguages

ScriptMapping

Parallel data

Data
Selection

Targetlanguages

Mixed data

Unsupervised pre-training

Pre-trainedmodel

Fine-tuning

NMT model

Figure 1: An overview of our proposed method consisting of script
mapping, data selection, pre-training and fine-tuning

2018] can artificially increase the similarity between languages which motivates part of our work.

3

Our Method: Using Assisting Languages

We propose a novel monolingual pre-training method for
NMT which leverages monolingual corpora of assisting languages to overcome the scarcity of monolingual and parallel
corpora of the languages of interest (LOI). The framework of
our approach is shown in Figure 1 which consists of script
mapping, data selection, pre-training and fine-tuning.

3.1

Data Pre-processing

Simply pre-training a NMT model on vast amounts of monolingual data belonging to the assisting languages and LOI can
improve translation quality. However, divergences between
the languages and also the distributions of data between different training phases is known to impact the final result.
Choosing linguistically related assisting languages is the bestsolution. However, even if the assisting languages are linguistically related, a difference in orthography can have a negative impact. Motivated by past works on orthography mapping/unification [Hermjakob et al., 2018; Chu et al., 2012]and data selection for MT [Axelrod et al., 2011], we propose
to improve the efficacy of pre-training by reducing data andlanguage divergence.
Script Mapping
Previous research has shown that enforcing shared orthography [Sennrich et al., 2016b; Dabre et al., 2015] has a strong
positive impact on translation. Following this, we propose toleverage existing script mapping rules2 or script unification
mechanisms to, at the very least, maximize the possibility of
cognate sharing and thereby bringing the assisting language
closer to the LOI. This should strongly impact languages belonging to the same family but written in different scripts.
However, for languages such as Korean, Chinese andJapanese there may exist a many to many mapping between
their scripts. As such, incorrect mapping of characters (basic unit of a script) might produce wrong words and reduce
cognate sharing. We propose two solutions to address this.
2
Transliteration is another option but transliteration systems are
relatively unreliable compared to handcrafted rule tables.

1. One-to-one mapping: Here we do not care about wordlevel information and map each character in one language toits corresponding character in another language. Here, we
just select the first mapping in the mapping list.
2. Many-to-many mapping with LM scoring: A more sophisticated solution is where for each tokenized word-levelsegment in one language we enumerate all possible combinations of mapped characters and use a language model in theother language to select the character combination with thehighest score as the result.
Note: Chinese–Japanese script mapping
Japanese language is written in Kanji which was borrowed
from China. Over time the written scripts have diverged andthe pronunciations are naturally different but there are a significant number of cognates written in both languages. As
such pre-training on Chinese should benefit translation involving Japanese. Chu et al. [2012] created a mapping tablebetween them which can be leveraged to further increase thenumber of cognates.
Data Selection
Existing NMT pre-training mechanisms relying on monolingual data do not focus on data selection. Often, the pretraining monolingual data and the fine-tuning parallel data
belong to different domains [Axelrod et al., 2011; Wang andNeubig, 2019] have shown that proper data selection can reduce the differences between the natures of data between different training phases. In this paper we propose to use a language model (LM) based and a simple but novel sentence
length based data selection mechanisms.
1. LM based data selection: To select pre-training data
which has similar data distribution with target domain, we use
language model trained by in-domain data to sort sentencesfrom the highest LM score and use the top N sentences thatare expected to be similar to the in-domain data.
2. Length based data selection: See Algorithm 1. We
first calculate the target length distribution (the ratio of alllengths in T argetF ile) and then we fill the length distribution for InputF ile by adding lines where the ratio of thelength of that line is lower than the upperbound value. These
SelectN um lines contain sentences with length distributions
similar to the in-domain dataset thereby increasing the similarity between the datasets.

3.2

NMT Modeling

In order to train a NMT model we first use the pre-processedmonolingual data for pre-training and then resume training
this model on parallel data to fine-tune for the languages ofinterest. In our work, we do not make any assumptions about
the NMT architecture.
Pre-training and fine-tuning (MASS)MASS is a pre-training method for NMT proposed by Songet al. [2019]. In MASS pre-training, the input is a sequenceof tokens where a part of the sequence is masked and the pretraining objective is to predict the masked fragments using
an encoder-decoder model. The NMT model is pre-trainedwith the MASS task, until convergence, jointly for both thesource and target languages. Thereafter training is resumed

Algorithm 1: Length Distribution Data Selection
Input : TargetFile, InputFile, SelectNum
Output: SelectedLines
1 TargetDistribution ← {};2 CurrentDistribution ← {};3 SelectedLines ← {};4 TargetNum = # of Lines in TargetFile;
5 foreach Line ∈ TargetFile do
6TargetDistribution[len(Line)]+ = 1 ;7 foreach Line ∈ InputFile do
8
if CurrentDistribution[len(Line)]/SelectNum <TargetDistribution[len(Line)]/TargetNum then9
CurrentDistribution[len(Line)]+ = 1;
10
SelectedLines ← SelectedLines ∪ {Line};on the parallel corpus, a step known as fine-tuning [Zoph etal., 2016]. We refer the readers to the original paper by Songet al. [2019] for further details.

4

Experimental Settings

We conducted experiments on Japanese–English (Ja–En)
translation in a variety of simulated low-resource settings using the “similar” assisting language pairs Chinese (Zh) and
French (Fr) and the “distant” assisting language pairs Russian (Ru) and Arabic (Ar).

4.1

Datasets

For the parallel corpora, we used the official ASPEC Ja–En
datasets [Nakazawa et al., 2016] provided by WAT 20193 .
The official split consists of 3M, 1790 and 1872 train, dev
and test sentences respectively. We sampled parallel corpora
from the top 1M sentences for fine-tuning. Out of the remaining 2M sentences, we used the En side of the first 1M
and the Ja side of the next 1M sentences as monolingual data
for language modeling for data selection. Additionally, weused Common Crawl4 monolingual corpora for pre-training.
To train LMs for data-selection of the assisting languages corpora, we used news commentary datasets 5 . For the rest of thispaper we consider the ASPEC and news-commentary monolingual sentences as in-domain and the rest of the pre-training
sentences as out-of-domain.

4.2

Data Pre-processing

Normalization and Initial Filtering
We applied NFKC normalization to data of all languages. Juman++ [Tolmachev et al., 2018] for Ja tokenization, jieba6 forZh tokenization and NLTK7 tokenization for other languages.
We filtered out all sentences from the pre-training data that3
http://lotus.kuee.kyoto-u.ac.jp/WAT/WAT2019/index.html#
task.html
4
http://data.statmt.org/ngrams/5
http://data.statmt.org/news-commentary/v14/
6
https://github.com/fxsjy/jieba
7https://www.nltk.org

contain fewer than 3 and equal or more than 80 tokens. Especially for Chinese data, which we found to be noisy, we
filtered out sentences containing fewer than 30 percent Chinese words or more than 30 percent English words.

have a little monolingual data of target languages and 3) zeroresource scenario, where we do not have any monolingual
data of target languages. We assumed that we have abundantmonolingual data of the assisting languages.

Script Mapping
Out of all assisting languages Chinese is the only one thatcan be mapped to Japanese reliably. In this paper, we focusedon converting Chinese to Japanese script to make them moresimilar by using the mapping table from Chu et al. [2012]
and the mapping approaches mentioned in the previous section. French and English are written using the Roman alphabet and do not need any script manipulation. As for Arabic
and Russian, we did not perform script mapping in order toshow the impact of using distant languages (script-wise as
well as linguistically).

4.3

Data selection
We performed data selection through both implicit and explicit data distribution selection methods.
1. LM Data Selection:We used language models trained on
the in-domain monolingual data and sort the common crawlmonolingual sentences on the LM score (highest to lowest)
and selected the top N for pre-training. N varies from 1M
to 20M sentences. We used KenLM [Heafield, 2011] to train
5-gram LMs.
2. Length Distribution Data Selection We used the ASPEC dev set to obtain an estimate of the test time in-domain
sentence length distribution. We found that the length distribution of 2,000 lines and 1M lines randomly selected fromASPEC training set are very similar, with same median andsimilar stdev. This justified our choice of using the dev set tobetter approximate test set length distributions.
Following Algorithm 1, when selecting English lines from
Common Crawl English monolingual data, we took English
side of ASPEC dev set as T argetF ile, 100M CommonCrawl English monolingual data as InputF ile ans 20M as
SelectN um. We calculated the number of lines in ASPEC dev set as T argetN um and calculated the number of
each length of all lines in ASPEC dev set. We filled theCurrentDistribution line by line in Common Crawl file if
the ratio of the length of current line had been less than theratio of this length in target length distribution. The Japanesedata selection was performed similarly with that of English.
We did not apply length distribution selection to data of theassisting languages because we lack an equivalent ASPEClike development set.
Since there is no conflict between LM data selection andlength distribution data selection, we tried all four combinations.
Dataset mixing for pre-training
We combined monolingual data of all assisting languages andlanguages of interest (LOI; Japanese and English). When
mixing datasets of different sizes, we always oversampled thesmaller datasets to match the size the the largest one.
We experimented with several scenarios when we have different sizes of monolingual data for Japanese and English
namely, 1) resource-rich scenario, where we have a lot ofmonolingual data 2) resource-poor scenario, where we only

Training and Evaluation Settings

We used the tensor2tensor framework [Vaswani et al.,2018] 8 with its default “transformer big” setting, such as
dropout=0.2, attention dropout=0.1, optimizer=adam with
beta1=0.9, beta2=0.997.
We created a shared sub-word vocabulary using Japaneseand English data from ASPEC mixing with Japanese, English, Chinese and French data from Common Crawl. We
used SentencePiece [Kudo and Richardson, 2018] and obtained a vocabulary with the size of roughly 64k . We used
this vocabulary in all experiments except unrelated language
experiment where Arabic and Russian were used instead ofChinese and French data.
For all pre-training models, we saved checkpoints every 1000 steps and for all fine-tuning models, we saved
checkpoints every 200 steps. We used early-stopping using
approximate-BL E U as target and stops when no gain after
10,000 steps for pre-training and 2,000 steps for fine-tuning.
We fine-tuned different fine-tune settings from the last checkpoint of each pre-trained model.
For decoding we averaged 10 checkpoints of the finetuning stage with α = 0.6 and beamsize = 4. We usedsacreBLEU9 to evaluate BLEU score for all translation evaluation.

4.4

Models Evaluated

Pre-trained Models
We separated pre-training settings into different blocks asshown in Table 1. Baseline model without fine-tuning
is shown as A1. Zero (0M), low (1M) and rich (20M)monolingual-corpus scenarios are shown in parts B, C andD, respectively. Pre-trained models aim to identify the best
setting considering pre-training data size, mapping method,data-selection method and related languages. These are labelled from E to H respectively.
Fine-tuned Models
We evaluated both Ja→En and En→Ja models with four parallel dataset size settings, 3K, 10K, 20K and 50K, selectedfrom the previously selected 1M ASPEC parallel sentences.

5

Results and Analysis

In Table 1, we show results of several experimental settings toanalyse the effect of: pre-training data size, Zh→Ja mapping
methods, data selection methods and choices of unrelated languages versus related languages.
Chinese to Japanese mapping
In part B, we compared two mapping methods with themethod without mapping at all. Results showed that the oneto-one mapping (character-level mapping) gives better BLEU
89

https://github.com/tensorflow/tensor2tensor, version 1.14.0.
https://github.com/mjpost/sacreBLEU

Pre-training#

Data pre-processing

Fine-tuning
Zh

Ja

En

Fr

3K

En→Ja
10K 20K

50K

3K

Ja→En
10K 20K

50K

A1

-

-

-

-

-

2.5

6.0

14.4

22.9

1.8

4.6

10.9

19.4

B1
B2
B3

w/o Zh→Ja mapping
1-to-1 Zh→Ja mappingLM-scoring Zh→Ja mapping

20M
20M
20M

20M
20M
20M

20M
20M
20M

20M
20M
20M

0.8
7.0
6.3

0.9
13.4
12.7

1.5
19.3
18.1

2.6
25.7
24.7

0.6
5.9
5.7

1.1
11.1
10.3

1.4
15.0
13.5

2.3
19.8
18.9

C1
C2
C3
C4
C5
C6
C7
C8

randomLM-scoring
LM-scoring
LM-scoring
LM-scoring
LM-scoringLD
LM-scoring + LD

-

20M
1M
5M
10M
15M
20M
20M
20M

20M
1M
5M
10M
15M
20M
20M
20M

-

7.6
3.5
5.8
4.9
6.9
4.7
9.4
7.9

14.7
8.9
12.8
11.2
14.6
11.7
17.2
14.4

19.5
13.7
18.1
16.9
19.4
16.6
21.7
19.7

26.1
22
25.6
24.2
26.0
23.9
27.8
26.4

6.7
3.8
5.3
5.0
6.5
4.5
8.2
7.1

11.8
8.3
11.2
9.6
11.7
9.1
13.8
11.5

14.8
11.7
14.5
12.9
15.1
12.9
17.1
15.2

19.5
16.8
19.6
17.7
19.3
18.3
20.6
20.7

D1
D2
D3

1-to-1 Zh→Ja mapping + LD
LD
1-to-1 Zh→Ja mapping + LD

20M
20M

-

-

20M
20M

5.3
3.4
2.1

14.5
9.1
6.7

20.0
14.9
12.6

26.1
23.4
21.9

3.7
2.1
2.2

11.2
6.3
6.3

15.6
11.3
10.7

20.5
17.7
16.8

E1
E2
E3
E4

LD
1-to-1 Zh→Ja mapping + LD
LD
1-to-1 Zh→Ja mapping + LD

20M
20M

1M
1M
1M
1M

1M
1M
1M
1M

20M
20M

7.7
8.3
8.3
7.1

15.8
16.4
15.3
15.2

20.7
20.2
19.3
19.4

26.3
26.9
26.7
26.5

7.2
7.5
6.8
6.6

12.7
12.5
12.3
12.0

15.7
16.3
15.4
15.4

19.6
20.7
20.4
19.9

F1
F2
F3
F4

LD
1-to-1 Zh→Ja mapping + LD
LD
1-to-1 Zh→Ja mapping + LD

20M
20M

15M
15M
15M
15M

15M
15M
15M
15M

20M
20M

9.6
9.7
7.7
7.7

17.2
17.1
15.0
14.9

21.5
21.6
19.8
19.7

28.0
27.2
26.3
26.1

8.6
8.3
6.3
6.5

13.5
13.3
11.7
11.4

16.8
16.7
15.1
15.4

20.9
20.6
20.2
19.8

G1
G2
G3

LM-scoring
1-to-1 Zh→Ja mapping + LM-scoring
LM-scoring + Ar20M + Ru20M

20M
-

20M
20M
20M

20M
20M
20M

20M
-

4.7
7.0
4.8

11.7
13.4
12.1

16.6
19.3
18.1

23.9
25.7
25.1

4.5
5.9
4.4

9.1
11.1
10.2

12.9
15.0
13.5

18.3
19.8
18.9

Table 1: Low-resource pre-training experiments. LD is with the meaning of “length distribution”. Best results of each part are in bold.

score than word-level mapping consistently on most finetuning settings, about 0.7 to 1.0 in most cases.
Also, if we do not perform Zh→Ja mapping, we foundthere are a lot of OOVs in Chinese monolingual data which
will dramatically affect the model in a negative way leading
to very low performance after fine-tuning. This shows theimportance behind data consistency between the pre-training
and fine-tuning stages.
Pre-training size and data selection experiments
We tried all combinations of proposed two data selection
strategies for monolingual data and show results in part C of
the table.
1. Random selection
2. Using length distribution selection
3. LM score sorting based selection
4. Combining 2 and 3.
Comparing using ‘random’ (C1) and ‘LM-scoring’ (C6)methods, we found that randomly selecting sentences will
give better results than using sentences with best LM score.
We suppose this is due to the divergence between selected
data distribution and true data distribution. The LM may
cause a data distribution preference, which is also a kindof bias. We also found that adding an explicit constraint,
the length distribution, gave better results. Random+length

distribution selection (C7) gave about 2 to 3 BL E U score
improvement compared with random selection (C1). LM
score+length distribution (C8) also gave 2 to 3 BL EU score
improvement on most settings over only LM-scoring (C6).
In part C, we also explored how different sizes of pretraining data affect the translation quality. We saw that using even 1M monolingual Japanese and English data (C2)gives much better results than without pre-training (A1) whenfine-tune on small dataset. For example on 10K fine-tuning
dataset. We observed about 3 BL EU score improvement on
En→Ja side and 3.7 score improvement on Ja→En side.
As we added monolingual data gradually, we saw that the
BL E U score improved gradually, where using 5M monolingual data (C3) gave much higher scores than using 1M (C2).
But the BLEU score is not perfectly consistent with monolingual data size as 15M monolingual data (C5) gives the best
results on most of the fine-tuning settings. We did not explore whether this phenomenon is caused by the quality ofthe monolingual dataset or other factors. We experimented
with LM-selection for these data size variation experiments
as our purpose was to show that even small amounts of pretraining data is enough to give improvements via fine-tuning.
In the future we will explore data size variation for all otherdata selection approaches. For the remaining experiments we
decided to use length distribution (LD) based data selectionand 1 to 1 script mapping wherever applicable.
Monolingual Low-resource scenario
The results of low-resource scenario are shown in parts D and
E of Table 1. We call this low-resource because in this setting we used very limited or even no monolingual data forJapanese and English. We experimented with different lowresource settings: zero-resource setting (part D) where wehave no Japanese and English data, limited-resource setting(part E) where we have 1M monolingual Japanese and English data.
In part D, we observed large improvements, a maximumof 8.5 BLEU score over the baseline setting (A1), on all finetuning settings over model without fine-tuning when using
only Chinese monolingual data (D1). Using only French dataalso gives better results on almost all fine-tuning settings, but
not as large as that of using only Chinese data. When combining Chinese and French data, they seemed to conflict witheach other leading to slightly reduced scores.
In part E of the table, when there are 1M Japanese andEnglish monolingual sentences, combining them with 20M
Chinese data also gives improvements up to 1.1 BLEU points
over A1. Combining with French data only gives occasional
improvements. In this setting too, combining Chinese andFrench data led to reduction in performance.
Although French and English share cognates and have similar grammar, we have not performed explicit script mapping
like we did for Chinese to make it more similar to Japanese.
Nevertheless we have no satisfactory explanation for this unexpected phenomenon. French-English token-level mapping
might benefit the model better, but we leave this for the future.
We can draw the following conclusions,
1. Script mapping (Chinese to Japanese) will consistently
give better results.
2. Using a linguistically similar languages (French and English) will sometimes give better results.
3. There may be conflicts between data of different assistinglanguages.
Monolingual resource-rich scenario
In part F, we found that there is less need to combine related
language data when we use a large monolingual data of target
languages. Only combining with Chinese data (F2) is comparable with pure Japanese-English monolingual pre-training(F1). Combining French data degrades the translation quality
in most settings. Thus, assisting languages become interfering languages in scenarios where large amounts of monolingual data are available for languages to be translated.
Unrelated language VS related language
In part G of the table, we compare pre-training on related languages versus unrelated languages. We saw that using Arabic
and Russian as unrelated assisting languages gives about 0.1
to maximum 1.5 improvement over the baseline (A1). Thisis surprising and it shows that leveraging any additional language is better than not leveraging them. However, using Chinese and French yields about 2 to 2.7 BLEU score improvements. Although we have only performed simple mapping ofChinese to Japanese and none for French to English, consider
this result with the results in part B of the table where we

contrasted script mapping and without script mapping. This
gives a clear answer that increasing relatedness on top of already using related languages is definitely important. In the
future, we will consider more rigorous ways of increasing relatedness between pre-training corpora by using existing dictionaries instead of simple script mapping.

6

Conclusion

In this paper we showed that it is possible to leverage monolingual corpora of other languages to pre-train NMT models for language pairs that lack parallel as well as monolingual data. We showed that maximizing the similarity between
the other (assisting) languages and the languages to be translated is crucial and standard script mapping helps improve thetranslation quality. We further showed that simple data selection methods help make the pre-training corpora more similar to the fine-tuning corpora which positively impacts NMT
performance. In the future, we plan to experiment with evenmore challenging language pairs such as Japanese–Russian
and attempt to leverage monolingual corpora belonging todiverse language families.We might be able to identify subtle relationships among languages and approaches to better
leverage assisting languages for several NLP tasks.

References
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. Domain
Adaptation via Pseudo In-Domain Data Selection. In Proceedings of the 2011 Conference on Empirical Methods in
Natural Language Processing, pages 355–362, Edinburgh,
Scotland, UK., July 2011. Association for Computational
Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural Machine Translation by Jointly Learning to Alignand Translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego,
USA, May 2015.
Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi.
Chinese Characters Mapping Table of Japanese, Traditional Chinese and Simplified Chinese. In Proceedingsof the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2149–2152, Istanbul, Turkey, May 2012. European Language Resources
Association (ELRA).
Raj Dabre, Chenhui Chu, Fabien Cromieres, Toshiaki
Nakazawa, and Sadao Kurohashi. Large-scale Dictionary
Construction via Pivot-based Statistical Machine Translation with Significance Pruning and Neural Network Features. In Proceedings of the 29th Pacific Asia Conferenceon Language, Information and Computation, pages 289–
297, Shanghai, China, October 2015.
Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. An Empirical Study of Language Relatedness for Transfer Learning in Neural Machine Translation. In Proceedings of the
31st Pacific Asia Conference on Language, Information
and Computation, pages 282–286. The National University (Phillippines), November 2017.

Raj Dabre, Atsushi Fujita, and Chenhui Chu. Exploiting
Multilingualism through Multistage Fine-Tuning for LowResource Neural Machine Translation. In Proceedingsof the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLPIJCNLP), pages 1410–1416, Hong Kong, China, November 2019. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding.
CoRR,
abs/1810.04805, 2018.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng
Wang. Multi-Task Learning for Multiple Language Translation. In Proceedings of the 53rd Annual Meeting of the
Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1723–1732, Beijing, China, July 2015. Association for Computational Linguistics.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino,
Guillaume Lample, Philipp Koehn, Vishrav Chaudhary,and Marc’Aurelio Ranzato. The FLORES Evaluation
Datasets for Low-Resource Machine Translation: Nepali–
English and Sinhala–English. In Proceedings of the 2019
Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages
6097–6110, Hong Kong, China, November 2019. Association for Computational Linguistics.
Kenneth Heafield. KenLM: Faster and Smaller Language
Model Queries. In Proceedings of the Sixth Workshop
on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, July 2011. Association for Computational
Linguistics.
Ulf Hermjakob, Jonathan May, and Kevin Knight. Out-ofthe-box Universal Romanization Tool uroman. In Proceedings of ACL 2018, System Demonstrations, pages 13–18,
Melbourne, Australia, July 2018. Association for Computational Linguistics.
Taku Kudo and John Richardson. SentencePiece: A simple
and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71,
Brussels, Belgium, November 2018. Association for Computational Linguistics.
V. Rudra Murthy, Anoop Kunchukuttan, and Pushpak Bhattacharyya. Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages. CoRR, abs/1811.00383, 2018.
Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto,
Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, andHitoshi Isahara. ASPEC: Asian Scientific Paper Excerpt
Corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC),

pages 2204–2208, Portorož, Slovenia, May 2016. European Language Resources Association.
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. When and Why Are Pre-Trained
Word Embeddings Useful for Neural Machine Translation?
In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 529–535, New Orleans, Louisiana, June 2018.
Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving Neural Machine Translation Models with Monolingual
Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers), pages 86–96, Berlin, Germany, August 2016. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural
Machine Translation of Rare Words with Subword Units.
In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016.
Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan
Liu. MASS: Masked Sequence to Sequence Pre-training
for Language Generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 915 June 2019, Long Beach, California, USA, pages 5926–
5936, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence toSequence Learning with Neural Networks. In Proceedingsof the 27th Neural Information Processing Systems Conference (NIPS), pages 3104–3112, Montréal, Canada, 2014.
Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. Juman++: A Morphological Analysis Toolkit for
Scriptio Continua. In Proceedings of the 2018 Conferenceon Empirical Methods in Natural Language Processing:
System Demonstrations, pages 54–59, Brussels, Belgium,
November 2018. Association for Computational Linguistics.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois
Chollet, Aidan Gomez, Stephan Gouws, Llion Jones,
Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2Tensor
for Neural Machine Translation. In Proceedings of the 13th
Conference of the Association for Machine Translation in
the Americas (Volume 1: Research Papers), pages 193–
199, Boston, USA, March 2018. Association for MachineTranslation in the Americas.
Xinyi Wang and Graham Neubig. Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural
Machine Translation. In Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics,
pages 5823–5828, Florence, Italy, July 2019. Association
for Computational Linguistics.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight.
Transfer Learning for Low-Resource Neural MachineTranslation. In Proceedings of the 2016 Conferenceon Empirical Methods in Natural Language Processing(EMNLP), pages 1568–1575, Austin, USA, 2016.