Exploiting Out-of-Domain Parallel Data through Multilingual Transfer
Learning for Low-Resource Neural Machine Translation
Aizhan Imankulova†

Raj Dabre‡ Atsushi Fujita‡ Kenji Imamura‡†Tokyo Metropolitan University
6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan
imankulova-aizhan@ed.tmu.ac.jp‡
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{raj.dabre, atsushi.fujita, kenji.imamura}@nict.go.jp

Abstract
This paper proposes a novel multilingual multistage fine-tuning approach for
low-resource neural machine translation(NMT), taking a challenging Japanese–
Russian pair for benchmarking.
Although there are many solutions for lowresource scenarios, such as multilingual
NMT and back-translation, we have empirically confirmed their limited successwhen restricted to in-domain data. We
therefore propose to exploit out-of-domain
data through transfer learning, by using it to first train a multilingual NMT
model followed by multistage fine-tuning
on in-domain parallel and back-translated
pseudo-parallel data. Our approach, which
combines domain adaptation, multilingualism, and back-translation, helps improve the translation quality by more than
3.7 BLEU points, over a strong baseline,
for this extremely low-resource scenario.

1

Introduction

Neural machine translation (NMT) (Cho et al.,2014; Sutskever et al., 2014; Bahdanau et al.,2015) has enabled end-to-end training of a translation system without needing to deal with wordalignments, translation rules, and complicated decoding algorithms, which are the characteristics ofphrase-based statistical machine translation (PBSMT) (Koehn et al., 2007). Although NMT canbe significantly better than PBSMT in resourcerich scenarios, PBSMT performs better in lowresource scenarios (Koehn and Knowles, 2017).
c 2019 The authors. This article is licensed under a Creative
Commons 4.0 licence, no derivative works, attribution, CCBY-ND.

Only by exploiting cross-lingual transfer learning techniques (Firat et al., 2016; Zoph et al.,2016; Kocmi and Bojar, 2018), can the NMT performance approach PBSMT performance in lowresource scenarios.
However, such methods usually require an NMT
model trained on a resource-rich language pairlike French↔English (parent), which is to befine-tuned for a low-resource language pair likeUzbek↔English (child). On the other hand, multilingual approaches (Johnson et al., 2017) propose
to train a single model to translate multiple language pairs. However, these approaches are effective only when the parent target or source language
is relatively resource-rich like English (En). Furthermore, the parents and children models shouldbe trained on similar domains; otherwise, one has
to take into account an additional problem of domain adaptation (Chu et al., 2017).
In this paper, we work on a linguistically distant and thus challenging language pairJapanese↔Russian (Ja↔Ru) which has only 12klines of news domain parallel corpus and hence isextremely resource-poor. Furthermore, the amountof indirect in-domain parallel corpora, i.e., Ja↔En
and Ru↔En, are also small. As we demonstrate in
Section 4, this severely limits the performance ofprominent low-resource techniques, such as multilingual modeling, back-translation, and pivotbased PBSMT. To remedy this, we propose a novel
multistage fine-tuning method for NMT that combines multilingual modeling (Johnson et al., 2017)and domain adaptation (Chu et al., 2017).
We have addressed two important research questions (RQs) in the context of extremely lowresource machine translation (MT) and our explorations have derived rational contributions (CTs)as follows:

RQ1. What kind of translation quality can we obtain in an extremely low-resource scenario?
CT1. We have made extensive comparisons withmultiple architectures and MT paradigms toshow how difficult the problem is. We havealso explored the utility of back-translationand show that it is ineffective given the poor
performance of base MT systems used to generate pseudo-parallel data. Our systematic
exploration shows that multilingualism is extremely useful for in-domain translation withvery limited corpora (see Section 4). This
type of exhaustive exploration has been missing from most existing works.
RQ2. What are the effective ways to exploit outof-domain data for extremely low-resource
in-domain translation?
CT2. Our proposal is to first train a multilingual NMT model on out-of-domain Ja↔En
and Ru↔En data, then fine-tune it on indomain Ja↔En and Ru↔En data, and further fine-tune it on Ja↔Ru data (see Section 5). We show that this stage-wise finetuning is crucial for high-quality translation.
We then show that the improved NMT models lead to pseudo-parallel data of better quality. This data can then be used to improve theperformance even further thereby enabling
the generation of better pseudo-parallel data.
By iteratively generating pseudo-parallel data
and fine-tuning the model on said data,
we can achieve the best performance forJapanese↔Russian translation.
To the best of our knowledge, we are the firstto perform such an extensive evaluation of extremely low-resource MT problem and propose anovel multilingual multistage fine-tuning approach
involving multilingual modeling and domain adaptation to address it.

2

Our Japanese–Russian Setting

In this paper, we deal with Ja↔Ru news translation. This language pair is very challenging
because the languages involved have completely
different writing system, phonology, morphology,
grammar, and syntax. Among various domains,
we experimented with translations in the news domain, considering the importance of sharing news
between different language speakers. Moreover,news domain is one of the most challenging tasks,

Ru

Ja

En

#sent.

X
X

X
X
X

X

913
173
276
0
4
287
1
1,654

X
X

X
X

X
X
Total

test
600
-

Usage
development
313
173
276
-

Table 1: Manually aligned News Commentary data.

due to large presence of out-of-vocabulary (OOV)
tokens and long sentences.1 To establish and evaluate existing methods, we also involved English as
the third language. As direct parallel corpora arescarce, involving a language such as English forpivoting is quite common (Utiyama and Isahara,2007).
There has been no clean held-out parallel data
for Ja↔Ru and Ja↔En news translation. Therefore, we manually compiled development and test
sets using News Commentary data2 as a source.
Since the given Ja↔Ru and Ja↔En data sharemany lines in the Japanese side, we first compiled
tri-text data. Then, from each line, corresponding parts across languages were manually identified, and unaligned parts were split off into a newline. Note that we have never merged two or morelines. As a result, we obtained 1,654 lines of datacomprising trilingual, bilingual, and monolingual
segments (mainly sentences) as summarized in Table 1. Finally, for the sake of comparability, we
randomly chose 600 trilingual sentences to createa test set, and concatenated the rest of them andbilingual sentences to form development sets.
Our manually aligned development and test setsare publicly available.3

3

Related Work

Koehn and Knowles (2017) showed that NMT isunable to handle low-resource language pairs as
opposed to PBSMT. Transfer learning approaches(Firat et al., 2016; Zoph et al., 2016; Kocmi and
Bojar, 2018) work well when a large helping parallel corpus is available. This restricts one of thesource or the target languages to be English which,in our case, is not possible. Approaches involving
bi-directional NMT modeling is shown to drasti1

News domain translation is also the most competitive tasks
in WMT indicating its importance.
2
http://opus.nlpl.eu/News-Commentary-v11.
php3
https://github.com/aizhanti/JaRuNC

cally improve low-resource translation (Niu et al.,2018). However, like most other, this work focuseson translation from and into English.
Remaining options include (a) unsupervised MT(Artetxe et al., 2018; Lample et al., 2018; Marieand Fujita, 2018), (b) parallel sentence mining
from non-parallel or comparable corpora (Utiyama
and Isahara, 2003; Tillmann and Xu, 2009), (c)generating pseudo-parallel data (Sennrich et al.,2016), and (d) MT based on pivot languages(Utiyama and Isahara, 2007). The linguistic distance between Japanese and Russian makes it extremely difficult to learn bilingual knowledge, suchas bilingual lexicons and bilingual word embeddings. Unsupervised MT is thus not promisingyet, due to its heavy reliance on accurate bilingual
word embeddings. Neither does parallel sentence
mining, due to the difficulty of obtaining accurate bilingual lexicons. Pseudo-parallel data can beused to augment existing parallel corpora for training, and previous work has reported that such data
generated by so-called back-translation can substantially improve the quality of NMT. However,this approach requires base MT systems that cangenerate somewhat accurate translations. It is thus
infeasible in our scenario, because we can obtain
only a weak system which is the consequence ofan extremely low-resource situation. MT based on
pivot languages requires large in-domain parallel
corpora involving the pivot languages. This technique is thus infeasible, because the in-domain parallel corpora for Ja↔En and Ru↔En pairs are also
extremely limited, whereas there are large parallel
corpora in other domains. Section 4 empiricallyconfirms the limit of these existing approaches.
Fortunately, there are two useful transfer learning solutions using NMT: (e) multilingual modeling to incorporate multiple language pairs into asingle model (Johnson et al., 2017) and (f) domain
adaptation to incorporate out-of-domain data (Chuet al., 2017). In this paper, we explore a novel
method involving step-wise fine-tuning to combine
these two methods. By improving the translation
quality in this way, we can also increase the likelihood of pseudo-parallel data being useful to further improve translation quality.

4

Limit of Using only In-domain Data

This section answers our first research question,[RQ1], about the translation quality that we canachieve using existing methods and in-domain par-Lang.pair
Ja↔Ru

Ja↔En

Ru↔En

Partition
train
development
test
train
development
test
train
development
test#sent.
12,356486
600
47,082589
600
82,072313
600#tokens
341k / 229k
16k / 11k
22k / 15k
1.27M / 1.01M
21k / 16k
22k / 17k
1.61M / 1.83M
7.8k / 8.4k
15k / 17k

#types
22k / 42k
2.9k / 4.3k
3.5k / 5.6k
48k / 55k
3.5k / 3.8k
3.5k / 3.8k
144k / 74k
3.2k / 2.3k
5.6k / 3.8k

Table 2: Statistics on our in-domain parallel data.

allel and monolingual data. We then use thestrongest model to conduct experiments on generating and utilizing back-translated pseudo-parallel
data for augmenting NMT. Our intention is to empirically identify the most effective practices aswell as recognize the limitations of relying only
on in-domain parallel corpora.
4.1

Data

To train MT systems among the three languages,i.e., Japanese, Russian, and English, we used all
the parallel data provided by Global Voices,4 more
specifically those available at OPUS.5 Table 2
summarizes the size of train/development/test
splits used in our experiments. The number of parallel sentences for Ja↔Ru is 12k, for Ja↔En is
47k, and for Ru↔En is 82k. Note that the three
corpora are not mutually exclusive: 9k out of 12k
sentences in the Ja↔Ru corpus were also includedin the other two parallel corpora, associated withidentical English translations. This puts a limit on
the positive impact that the helping corpora canhave on the translation quality.
Even when one focuses on low-resource language pairs, we often have access to larger quantities of in-domain monolingual data of each language. Such monolingual data are useful to improve quality of MT, for example, as the source ofpseudo-parallel data for augmenting training data
for NMT (Sennrich et al., 2016) and as the training data for large and smoothed language models for PBSMT (Koehn and Knowles, 2017). Table 3 summarizes the statistics on our monolingual
corpora for several domains including the news
domain. Note that we removed from the Global
Voices monolingual corpora those sentences thatare already present in the parallel corpus.
4

https://globalvoices.org/
http://opus.nlpl.eu/GlobalVoices-v2015.
php
5

CorpusGlobal Voices5
Wikinews6
News Crawl7
Yomiuri (2007–2011)8IWSLT9
Tatoeba10

Ja
26k
37k
19M
411k
5k

Ru
24k
243k
72M
64k
58k

En
842k
194M
66k
208k

Table 3: Number of lines in our monolingual data. Whereas
the first four are from the news corpora (in-domain), the lasttwo, i.e., “IWSLT” and “Tatoeba,” are from other domains.

We tokenized English and Russian sentences using tokenizer.perl of Moses (Koehn et al., 2007).11
To tokenize Japanese sentences, we used MeCab12
with the IPA dictionary. After tokenization, we
eliminated duplicated sentence pairs and sentenceswith more than 100 tokens for all the languages.
4.2

MT Methods Examined

We began with evaluating standard MT paradigms,
i.e., PBSMT (Koehn et al., 2007) and NMT(Sutskever et al., 2014). As for PBSMT, wealso examined two advanced methods: pivot-basedtranslation relying on a helping language (Utiyama
and Isahara, 2007) and induction of phrase tablesfrom monolingual data (Marie and Fujita, 2018).
As for NMT, we compared two types of
encoder-decoder architectures: attentional RNNbased model (RNMT) (Bahdanau et al., 2015) andthe Transformer model (Vaswani et al., 2017). In
addition to standard uni-directional modeling, tocope with the low-resource problem, we examined
two multi-directional models: bi-directional model(Niu et al., 2018) and multi-to-multi (M2M) model(Johnson et al., 2017).
After identifying the best model, we also examined the usefulness of a data augmentation method
based on back-translation (Sennrich et al., 2016).
PBSMT Systems
First, we built a PBSMT system for each of
the six translation directions. We obtained phrase
6

https://dumps.wikimedia.org/
backup-index.html (20180501)
7
http://www.statmt.org/wmt18/
translation-task.html
8
https://www.yomiuri.co.jp/database/glossary/9
http://www.cs.jhu.edu/˜kevinduh/a/
multitarget-tedtalks/10
http://opus.nlpl.eu/Tatoeba-v2.php
11
https://github.com/moses-smt/
mosesdecoder12
http://taku910.github.io/mecab,
version
0.996.

tables from parallel corpus using SyMGIZA++13
with the grow-diag-final heuristics for wordalignment, and Moses for phrase pair extraction.
Then, we trained a bi-directional MSD (monotone, swap, and discontinuous) lexicalized reordering model. We also trained three 5-gram language
models, using KenLM14 on the following monolingual data: (1) the target side of the parallel data, (2)the concatenation of (1) and the monolingual datafrom Global Voices, and (3) the concatenation of(1) and all monolingual data in the news domain in
Table 3.
Subsequently, using English as the pivot language, we examined the following three types ofpivot-based PBSMT systems (Utiyama and Isahara, 2007; Cohn and Lapata, 2007) for each of
Ja→Ru and Ru→Ja.
Cascade: 2-step decoding using the source-toEnglish and English-to-target systems.
Synthesize: Obtain a new phrase table from synthetic parallel data generated by translating
English side of the target–English training
parallel data to the source language with theEnglish-to-source system.
Triangulate: Compile a new phrase table combining those for the source-to-English andEnglish-to-target systems.
Among these three, triangulation is the most computationally expensive method. Although we had
filtered the component phrase tables using the statistical significance pruning method (Johnson etal., 2007), triangulation can generate an enormous
number of phrase pairs. To reduce the computational cost during decoding and the negative effects of potentially noisy phrase pairs, we retainedfor each source phrase s only the k-best translations t according to the forward translation probability φ(t|s) calculated from the conditional probabilities in the component models as defined in
Utiyama and Isahara (2007). For each of the retained phrase pairs, we also calculated the backward translation probability, φ(s|t), and lexicaltranslation probabilities, φlex (t|s) and φlex (s|t), in
the same manner as φ(t|s).
We also investigated the utility of recent advances in unsupervised MT. Even though we began with a publicly available implementation of
13
14

https://github.com/emjotde/symgiza-pp
https://github.com/kpu/kenlm

ID(a1), (b1)(a2), (b2)(a3), (b3)SystemJa→Ru or Ru→Ja
Ja→En or En→Ja
Ru→En or En→Ru
Ja→Ru and Ru→Ja
Ja→En and En→Ja
Ru→En and En→Ru
M2M systems

Ja↔Ru
12k
12k
12k→82k

Parallel data
Ja↔En
Ru↔En
47k
82k
47k
82k
47k→82k
82k

Total size of
training data
12k
47k
82k
24k
94k
164k
492k

Vocabulary
size
16k
16k
16k
16k
16k
16k
32k

Table 4: Configuration of uni-, bi-directional, and M2M NMT baseline systems. Arrows in “Parallel data” columns indicate
the over-sampling of the parallel data to match the size of the largest parallel data.

unsupervised PBSMT (Lample et al., 2018),15 it
crashed due to unknown reasons. We therefore followed another method described in Marie and Fujita (2018). Instead of short n-grams (Artetxe etal., 2018; Lample et al., 2018), we collected a setof phrases in Japanese and Russian from respective monolingual data using the word2phrase
algorithm (Mikolov et al., 2013),16 as in Marieand Fujita (2018). To reduce the complexity, weused randomly selected 10M monolingual sentences, and 300k most frequent phrases made ofwords among the 300k most frequent words. For
each source phrase s, we selected 300-best target phrases t according to the translation probability as in Lample et al. (2018): p(t|s) =P exp(β cos(emb(t),emb(s)), where emb(·) stands0
t0 exp(β cos(emb(t ),emb(s))
for a bilingual embedding of a given phrase, obtained through averaging bilingual embeddings ofconstituent words learned from the two monolingual data using fastText17 and vecmap.18
For each of the retained phrase pair, p(s|t) was
computed analogously. We also computed lexicaltranslation probabilities relying on those learned
from the given small parallel corpus.
Up to four phrase tables were jointly exploited by the multiple decoding path ability ofMoses. Weights for the features were tuned using KB-MIRA (Cherry and Foster, 2012) on thedevelopment set; we took the best weights after 15
iterations. Two hyper-parameters, namely, k forthe number of pivot-based phrase pairs per source
phrase and d for distortion limit, were determinedby a grid search on k ∈ {10, 20, 40, 60, 80, 100}and d ∈ {8, 10, 12, 14, 16, 18, 20}. In contrast, weused predetermined hyper-parameters for phrase
table induction from monolingual data, following

the convention: 200 for the dimension of word andphrase embeddings and β = 30.
NMT Systems
We used the open-source implementation of
the RNMT and the Transformer models in
tensor2tensor.19 A uni-directional model foreach of the six translation directions was trained on
the corresponding parallel corpus. Bi-directionaland M2M models were realized by adding an artificial token that specifies the target language tothe beginning of each source sentence and shuffling the entire training data (Johnson et al., 2017).
Table 4 contains some specific hyperparameters20 for our baseline NMT models.
The hyper-parameters not mentioned in this tableused the default values in tensor2tensor.
For M2M systems, we over-sampled Ja→Ru and
Ja→En training data so that their sizes match thelargest Ru→En data. To reduce the number of
unknown words, we used tensor2tensor’s
internal sub-word segmentation mechanism.
Since we work in a low-resource setting, we usedshared sub-word vocabularies of size 16k for theuni- and bi-directional models and 32k for theM2M models. The number of training iterations
was determined by early-stopping: we evaluatedour models on the development set every 1,000
updates, and stopped training if BLEU score
for the development set was not improved for
10,000 updates (10 check-points). Note that thedevelopment set was created by concatenating
those for the individual translation directions
without any over-sampling.
Having trained the models, we averaged the last
10 check-points and decoded the test sets with abeam size of 4 and a length penalty which was15

https://github.com/facebookresearch/
UnsupervisedMT
16
https://code.google.com/archive/p/
word2vec/
17
https://fasttext.cc/
18
https://github.com/artetxem/vecmap

19

https://github.com/tensorflow/
tensor2tensor, version 1.6.6.
20
We compared two mini-batch sizes, 1024 and 6144 tokens,and found that 6144 and 1024 worked better for RNMT andTransformer, respectively.

ID
(a1)
(a2)
(a3)
(b1)
(b2)
(b3)
(c1)

System
Uni-directional RNMT
Bi-directional RNMT
M2M RNMT
Uni-directional Transformer
Bi-directional Transformer
M2M Transformer
Uni-directional supervised PBSMT

Ja→Ru
0.58
0.65
1.51
0.70
0.19
3.72
2.02

Ru→Ja
1.86
1.61
4.29
1.96
0.87
8.35
4.45

Ja→En
2.41
6.18
5.15
4.36
6.48
10.24
8.19

En→Ja
7.83
8.81
7.55
7.97
10.63
12.43
10.27

Ru→En
18.42
19.60
14.24
20.70
22.25
22.10
22.37

En→Ru
13.64
15.11
10.86
16.24
16.03
16.92
16.52

Table 5: BLEU scores of baseline systems. Bold indicates the best BLEU score for each translation direction.

tuned by a linear search on the BLEU score forthe development set.
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with unidirectional NMT models.
4.3

Results

We evaluated MT models using case-sensitive andtokenized BLEU (Papineni et al., 2002) on test
sets, using Moses’s multi-bleu.perl. Statisticalsignificance (p < 0.05) on the difference of
BLEU scores was tested by Moses’s bootstraphypothesis-difference-significance.pl.
Tables 5 and 6 show BLEU scores of all themodels, except the NMT systems augmented withback-translations. Whereas some models achieved
reasonable BLEU scores for Ja↔En and Ru↔En
translation, all the results for Ja↔Ru, which is our
main concern, were abysmal.
Among the NMT models, Transformer models(b∗) were proven to be better than RNMT models(a∗). RNMT models could not even outperform theuni-directional PBSMT models (c1). M2M models (a3) and (b3) outperformed their corresponding
uni- and bi-directional models in most cases. It is
worth noting that in this extremely low-resource
scenario, BLEU scores of the M2M RNMT modelfor the largest language pair, i.e., Ru↔En, werelower than those of the uni- and bi-directional
RNMT models as in Johnson et al. (2017). In contrast, with the M2M Transformer model, Ru↔En
also benefited from multilingualism.
Standard PBSMT models (c1) achieved higher
BLEU scores than uni-directional NMT models (a1) and (b1), as reported by Koehn andKnowles (2017), whereas they underperform the
M2M Transformer NMT model (b3). As shownin Table 6, pivot-based PBSMT systems always
achieved higher BLEU scores than (c1). The
best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible
BLEU gains with substantial reduction of OOV
tokens (3047→1180 for Ja→Ru, 4463→1812 for

SystemPBSMT: Cascade
PBSMT: Synthesize
PBSMT: Synthesize / GoldPBSMT: Synthesize + GoldPBSMT: Triangulate
PBSMT: Triangulate / GoldPBSMT: Synthesize / Triangulate / GoldPBSMT: Induced
PBSMT: Induced / Synthesize / Triangulate / GoldRNMT: Cascade
RNMT: Synthesize
RNMT: Synthesize + Gold
Transformer NMT: Cascade
Transformer NMT: Synthesize
Transformer NMT: Synthesize + Gold

Ja→Ru
3.65
3.37
2.94
3.07
3.75
3.93
4.02
0.37
2.85
1.19
1.82
1.62
2.41
1.78
2.13

Ru→Ja
7.62
6.72
6.95
6.62
7.02
7.02
7.07
0.65
6.86
6.73
3.02
3.24
6.84
5.43
5.06

Table 6: BLEU scores of pivot-based systems. “Gold” refers
to the phrase table trained on the parallel data. Bold indicatesthe BLEU score higher than the best one in Table 5. “/” indicates the use of separately trained multiple phrase tables,
whereas so does “+” training on the mixture of parallel data.

Ru→Ja). However, further extension with phrase
tables induced from monolingual data did not push
the limit, despite their high coverage; only 336 and677 OOV tokens were left for the two translation
directions, respectively. This is due to the poor
quality of the bilingual word embeddings used to
extract the phrase table, as envisaged in Section 3.
None of pivot-based approaches with unidirectional NMT models could even remotely rival
the M2M Transformer NMT model (b3).
4.4

Augmentation with Back-translation

Given that the M2M Transformer NMT model (b3)achieved best results for most of the translation directions and competitive results for the rest, we
further explored it through back-translation.
We examined the utility of pseudo-parallel data
for all the six translation directions, unlike thework of Lakew et al. (2017) and Lakew et al.
(2018), which concentrate only on the zero-shot
language pair, and the work of Niu et al. (2018),
which compares only uni- or bi-directional models. We investigated whether each translation direction in M2M models will benefit from pseudoparallel data and if so, what kind of improvementtakes place.

ID

System#1–#10Ja∗→Ru and/or Ru∗→Ja
Ja∗→En and/or En∗→Ja
Ru∗→En and/or En∗→Ru
All

Pseudo12k→82k
47k→82k
82k
All of the above

Parallel data
Ja↔Ru
Ja↔En
12k→82k
47k→82k×2
12k→82k×2 47k→82k
12k→82k×2 47k→82k×2
12k→82k
47k→82k

Ru↔En
82k×2
82k×2
82k
82k

Total size of
training data
984k
984k
984k
984k

Table 7: Over-sampling criteria for pseudo-parallel data generated by back-translation.
ID
(b3)
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10

Ja∗→RuX
X
X

Pseudo-parallel data involved
Ru∗→Ja Ja∗→En En∗→Ja Ru∗→En
X
X
X
X
X
X
X
X
X
X
X
X

En∗→Ru
X
X
X

Ja→Ru
3.72
• 4.593.74• 4.56
3.71
3.62
3.61
3.80
3.77
3.37
• 4.43

Ru→Ja
8.35
8.63
• 8.85• 9.09
8.05
8.10
7.94
8.37
8.04
8.03
• 9.38BLEU score
Ja→En En→Ja
10.24
12.43
10.64
12.94
10.13
13.05
10.57 • 13.23• 11.00
12.66
9.92 • 14.06• 11.51• 14.38
10.67
13.00
10.52
12.43
10.19
12.79
• 12.06• 14.43

Ru→En
22.10
22.21
22.48
22.48
22.17
21.66
22.22
22.51
• 22.8522.77• 23.09

En→Ru
16.92
17.30
17.20
• 17.89
16.76
16.68
16.80
• 17.73
17.13
17.26
17.30

Table 8: BLEU scores of M2M Transformer NMT systems trained on the mixture of given parallel corpus and pseudo-parallel
data generated by back-translation using (b3). Six “X∗→Y” columns show whether the pseudo-parallel data for each translation
direction is involved. Bold indicates the scores higher than (b3) and “• ” indicates statistical significance of the improvement.

First, we selected sentences to be backtranslated from in-domain monolingual data (Table 3), relying on the score proposed by Moore andLewis (2010) via the following procedure.
1. For each language, train two 4-gram language
models, using KenLM: an in-domain one on
all the Global Voices data, i.e., both paralleland monolingual data, and a general-domain
one on the concatenation of Global Voices,IWSLT, and Tatoeba data.
2. For each language, discard sentences containing OOVs according to the in-domain language model.
3. For each translation direction, select the T best monolingual sentences in the news domain, according to the difference between
cross-entropy scores given by the in-domain
and general-domain language models.
Whereas Niu et al. (2018) exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them (Johnson et al.,2017), setting T to the number of lines of parallel data of given language pair.
Selected monolingual sentences were thentranslated using the M2M Transformer NMT
model (b3) to compose pseudo-parallel data. Then,the pseudo-parallel data were enlarged by oversampling as summarized in Table 7. Finally, new
NMT models were trained on the concatenation of
the original parallel and pseudo-parallel data fromscratch in the same manner as the previous NMT
models with the same hyper-parameters.
Table 8 shows the BLEU scores achievedby several reasonable combinations of six-way
pseudo-parallel data. We observed that the use of
all six-way pseudo-parallel data (#10) significantly
improved the base model for all the translation directions, except En→Ru. A translation direction
often benefited when the pseudo-parallel data forthat specific direction was used.
4.5

Summary

We have evaluated an extensive variation of MT
models21 that rely only on in-domain parallel andmonolingual data. However, the resulting BLEU
scores for Ja→Ru and Ru→Ja tasks do not exceed
10 BLEU points, implying the inherent limitationof the in-domain data as well as the difficulty ofthese translation directions.

5

Exploiting Large Out-of-Domain Data
Involving a Helping Language

The limitation of relying only on in-domain data
demonstrated in Section 4 motivates us to explore
21

Other conceivable options include transfer learning using
parallel data between English and one of Japanese and Russian as either source or target language, such as pre-training
an En→Ru model and fine-tuning it for Ja→Ru. Our M2M
models conceptually subsume them, even though they do not
explicitly divide the two steps during training. On the other
hand, our method proposed in Section 5 explicitly conducts
transfer learning for domain adaptation followed by additional transfer learning across different languages.

Domain \ language pairin-domain
out-of-domain

DirectA, X
C, ×

One-side shared
B, X
D, X

Lang.pair
Ja↔En
Ru↔En

CorpusASPEC
UN
Yandex#sent.
1,500,000
2,647,243
320,325#tokens
42.3M / 34.6M
90.5M / 92.8M
8.51M / 9.26M#types234k / 1.02M
757k / 593k
617k / 407k

Table 9: Classification of parallel data.
Table 10: Statistics on our out-of-domain parallel data.

other types of parallel data. As raised in our second research question, [RQ2], we considered theeffective ways to exploit out-of-domain data.
According to language pair and domain, parallel
data can be classified into four categories in Table 9. Among all the categories, out-of-domain
data for the language pair of interest have been exploited in the domain adaptation scenarios (C→A)(Chu et al., 2017). However, for Ja↔Ru, no
out-of-domain data is available. To exploit outof-domain parallel data for Ja↔En and Ru↔En
pairs instead, we propose a multistage fine-tuning
method, which combines two types of transfer
learning, i.e., domain adaptation for Ja↔En and
Ru↔En (D→B) and multilingual transfer (B→A),
relying on the M2M model examined in Section 4.
We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data.
5.1

Multistage Fine-tuning

Simply using NMT systems trained on out-ofdomain data for in-domain translation is known
to perform badly. In order to effectively use
large-scale out-of-domain data for our extremely
low-resource task, we propose to perform domain
adaptation through either (a) conventional finetuning, where an NMT system trained on out-ofdomain data is fine-tuned only on in-domain data,or (b) mixed fine-tuning (Chu et al., 2017), where
pre-trained out-of-domain NMT system is finetuned using a mixture of in-domain and out-ofdomain data. The same options are available fortransferring from Ja↔En and Ru↔En to Ja↔Ru.
We inevitably involve two types of transfer
learning, i.e., domain adaptation for Ja↔En and
Ru↔En and multilingual transfer for Ja↔Ru pair.
Among several conceivable options for managing
these two problems, we examined the following
multistage fine-tuning.
Stage 0. Out-of-domain pre-training: Pre-train
a multilingual model only on the Ja↔En
and Ru↔En out-of-domain parallel data
(I), where the vocabulary of the model isdetermined on the basis of the in-domain
parallel data in the same manner as the M2M
NMT models examined in Section 4.

Stage 1. Fine-tuning for domain adaptation:
Fine-tune the pre-trained model (I) on thein-domain Ja↔En and Ru↔En parallel
data (fine-tuning, II) or on the mixture of
in-domain and out-of-domain Ja↔En and
Ru↔En parallel data (mixed fine-tuning, III).
Stage 2. Fine-tuning for Ja↔Ru pair: Further
fine-tune the models (each of II and III) for
Ja↔Ru on in-domain parallel data for thislanguage pair only (fine-tuning, IV and VI)
or on all the in-domain parallel data (mixed
fine-tuning, V and VII).
We chose this way due to the following two reasons. First, we need to take a balance between several different parallel corpora sizes. The other reason is division of labor; we assume that solving
each sub-problem one by one should enable gradual shift of parameters.
5.2

Data Selection

As an additional large-scale out-of-domain parallel data for Ja↔En, we used the cleanest 1.5M
sentences from the Asian Scientific Paper Excerpt
Corpus (ASPEC) (Nakazawa et al., 2016).22 As
for Ru↔En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task.23 We retained Ru↔En sentence pairsthat contain at least one OOV token in both sides,
according to the in-domain language model trainedin Section 4.4. Table 10 summarizes the statistics
on the remaining out-of-domain parallel data.
5.3

Results

Table 11 shows the results of our multistage finetuning, where the IDs of each row refer to thosedescribed in Section 5.1. First of all, the final models of our multistage fine-tuning, i.e., V and VII,
achieved significantly higher BLEU scores than(b3) in Table 5, a weak baseline without using any
monolingual data, and #10 in Table 8, a strong
baseline established with monolingual data.
22

http://lotus.kuee.kyoto-u.ac.jp/ASPEC/http://www.statmt.org/wmt18/
translation-task.html
23

ID

Initialized

(b3)
I
II
III
IV
V
VI
VII
I’
II’
III’

I
I
II
II
III
III
I
I

Out-of-domain data
Ja↔En Ru↔En
X
X
X
X
X
X
X
X

In-domain data
Ja↔Ru Ja↔En Ru↔En
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X

Ja→Ru
3.72
0.00
0.20
0.23
• 5.44• 6.90• 5.91• 7.49• 5.31• 6.30• 7.53

Ru→Ja
8.35
0.15
0.70
1.07
• 10.67• 11.99• 10.83• 12.10• 10.73• 11.64• 12.33

BLEU score
Ja→En En→Ja
10.24
12.43
4.59
4.15
• 14.10• 17.80• 13.31• 17.74
0.12
3.97• 14.34• 16.93
0.26
2.18• 14.63• 17.51• 14.41• 16.34• 14.29• 16.83• 14.19• 16.77

Ru→En
22.10
• 25.22• 28.23• 28.73
0.11• 27.50
0.18• 28.51• 27.46• 27.53• 27.94

En→Ru
16.92
• 20.37• 24.35• 25.223.66• 23.17
1.10• 24.60• 23.21• 23.00• 23.97

Table 11: BLEU scores obtained through multistage fine-tuning. “Initialized” column indicates the model used for initializing
parameters that are fine-tuned on the data indicated by X. Bold indicates the best BLEU score for each translation direction.
“• ” indicates statistical significance of the improvement over (b3).

The performance of the initial model (I) depends
on the language pair. For Ja↔Ru pair, it cannot achieve minimum level of quality since themodel has never seen parallel data for this pair.
The performance on Ja↔En pair was much lower
than the two baseline models, reflecting the crucial mismatch between training and testing domains. In contrast, Ru↔En pair benefited the mostand achieved surprisingly high BLEU scores. The
reason might be due to the proximity of out-ofdomain training data and in-domain test data.
The first fine-tuning stage significantly pushed
up the translation quality for Ja↔En and Ru↔En
pairs, in both cases with fine-tuning (II) and mixed
fine-tuning (III). At this stage, both models performed only poorly for Ja↔Ru pair as they have
not yet seen Ja↔Ru parallel data. Either model
had a consistent advantage to the other.
When these models were further fine-tuned only
on the in-domain Ja↔Ru parallel data (IV and VI),
we obtained translations of better quality than thetwo baselines for Ja↔Ru pair. However, as a result of complete ignorance of Ja↔En and Ru↔En
pairs, the models only produced translations ofpoor quality for these language pairs. In contrast,
mixed fine-tuning for the second fine-tuning stage(V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks
to the gradual shift of parameters realized by indomain Ja↔En and Ru↔En parallel data. Unfortunately, the translation quality for Ja↔En and
Ru↔En pairs sometimes degraded from II and III.
Nevertheless, the BLEU scores still retain the large
margin against two baselines.
The last three rows in Table 11 present BLEU
scores obtained by the methods with fewer finetuning steps. The most naive model I’, trainedon the balanced mixture of whole five types ofcorpora from scratch, and the model II’, obtained
through a single-step conventional fine-tuning ofI on all the in-domain data, achieved only BLEU
scores consistently worse than VII. In contrast,
when we merged our two fine-tuning steps into asingle mixed fine-tuning on I, we obtained a model
III’ which is better for the Ja↔Ru pair than VII.
Nevertheless, they are still comparable to those of
VII and the BLEU scores for the other two language pairs are much lower than VII. As such, we
conclude that our multistage fine-tuning leads to amore robust in-domain multilingual model.
5.4

Further Augmentation with
Back-translation

Having obtained a better model, we examinedagain the utility of back-translation. More precisely, we investigated (a) whether the pseudoparallel data generated by an improved NMT
model leads to a further improvement, and (b)whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data
will result in a model better than training a newmodel from scratch as examined in Section 4.4.
Given an NMT model, we first generated sixway pseudo-parallel data by translating monolingual data. For the sake of comparability, we used
the identical monolingual sentences sampled in
Section 4.4. Then, we further fine-tuned the given
model on the mixture of the generated pseudoparallel data and the original parallel data, following the same over-sampling procedure in Section 4.4. We repeated these steps five times.
Table 12 shows the results. “new #10” in thesecond row indicates an M2M Transformer modeltrained from scratch on the mixture of six-way
pseudo-parallel data generated by VII and the orig-

No

InitializedBT#10
new #10
VIII
IX
X
XI
XII

VII
VIII
IX
X
XI

(b3)
VII
VII
VIII
IX
X
XI

Ja→Ru
4.43
• 6.55• 7.83• 8.03• 7.76• 7.85• 8.16

Ru→Ja
9.38• 11.36• 12.21• 12.55• 12.59• 12.97• 13.09

BLEU score
Ja→En En→Ja
12.06
14.43
• 13.77• 15.59• 15.06• 17.19• 15.07• 17.80• 15.08• 18.12• 15.26• 17.83• 14.96• 17.74

Ru→En
23.09
• 24.91• 28.49• 28.16• 28.18• 28.49• 28.45

En→Ru
17.30
• 20.55• 23.96• 24.27• 24.67• 24.36• 24.35

Table 12: BLEU scores achieved through fine-tuning on the mixture of the original parallel data and six-way pseudo-paralleldata. “Initialized” column indicates the model used for initializing parameters and so does “BT” column the model used togenerate pseudo-parallel data. “• ” indicates statistical significance of the improvement over #10.

Investigation step
Uni-directional Transformer: (b1) in Table 5
M2M Transformer: (b3) in Table 5+ six-way pseudo-parallel data: #10 in Table 8
M2M multistage fine-tuning: VII in Table 11+ six-way pseudo-parallel data: XII in Table 12

Ja→Ru
0.70
3.72
4.43
7.49
8.16

Ru→Ja
1.96
8.35
9.38
12.10
13.09

Table 13: Summary of our investigation: BLEU scores of thebest NMT systems at each step.

inal parallel data. It achieved higher BLEU scoresthan #10 in Table 8 thanks to the pseudo-parallel
data of better quality, but underperformed the base
NMT model VII. In contrast, our fine-tuned model
VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for alltranslation directions, except Ru→En. Althoughfurther iterations did not necessarily gain BLEU
scores, we came to a much higher plateau compared to the results in Section 4.

6

Conclusion

In this paper, we challenged the difficult task of
Ja↔Ru news domain translation in an extremely
low-resource setting. We empirically confirmed
the limited success of well-established solutions
when restricted to in-domain data. Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observedthat it substantially improves Ja↔Ru translationby over 3.7 BLEU points compared to a strong
baseline, as summarized in Table 13. This paper
contains an empirical comparison of several existing approaches and hence we hope that our paper
can act as a guideline to researchers attempting totackle extremely low-resource translation.
In the future, we plan to confirm further finetuning for each of specific translation directions.
We will also explore the way to exploit outof-domain pseudo-parallel data, better domainadaptation approaches, and additional challenging
language pairs.

Acknowledgments
This work was carried out when Aizhan
Imankulova was taking up an internship at
NICT, Japan. We would like to thank the reviewers for their insightful comments. A part of thiswork was conducted under the program “Promotion of Global Communications Plan: Research,
Development, and Social Demonstration of Multilingual Speech Translation Technology” of the
Ministry of Internal Affairs and Communications(MIC), Japan.

References
Artetxe, Mikel, Gorka Labaka, and Eneko Agirre.
2018. Unsupervised statistical machine translation.
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
3632–3642, Brussels, Belgium.
Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In Proceedings of the
3rd International Conference on Learning Representations, San Diego, USA.
Cherry, Colin and George Foster. 2012. Batch tuning
strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages
427–436, Montréal, Canada.
Cho, Kyunghyun, Bart van Merriënboer, ÇaglarGülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learningphrase representations using RNN encoder–decoder
for statistical machine translation. In Proceedings ofthe 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724–1734, Doha,
Qatar.
Chu, Chenhui, Raj Dabre, and Sadao Kurohashi.
2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 385–391,
Vancouver, Canada.

Cohn, Trevor and Mirella Lapata. 2007. Machinetranslation by triangulation: Making effective use ofmulti-parallel corpora. In Proceedings of the 45th
Annual Meeting of the Association of Computational
Linguistics, pages 728–735, Prague, Czech Republic.

Lample, Guillaume, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018.
Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing,pages 5039–5049, Brussels, Belgium.

Firat, Orhan, Kyunghyun Cho, and Yoshua Bengio.
2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL
HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages
866–875, San Diego, USA.

Marie, Benjamin and Atsushi Fujita. 2018. Unsupervised neural machine translation initialized by unsupervised statistical machine translation. CoRR,
abs/1810.12703.

Johnson, Howard, Joel Martin, George Foster, andRoland Kuhn. 2007. Improving translation quality by discarding most of the phrasetable. In Proceedings of the 2007 Joint Conference on Empirical
Methods in Natural Language Processing and Computational Natural Language Learning, pages 967–
975, Prague, Czech Republic.
Johnson, Melvin, Mike Schuster, Quoc Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,
Fernanda Viégas, Martin Wattenberg, Greg Corrado,
Macduff Hughes, and Jeffrey Dean. 2017. Google’s
multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351.
Kocmi, Tom and Ondřej Bojar. 2018. Trivial transfer
learning for low-resource neural machine translation.
In Proceedings of the Third Conference on MachineTranslation: Research Papers, pages 244–252, Brussels, Belgium.
Koehn, Philipp and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver, Canada.
Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra
Constantin, and Evan Herbst. 2007. Moses: Open
source toolkit for statistical machine translation. In
Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion
Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic.
Lakew, Surafel M, Quintino F. Lotito, Matteo Negri,
Marco Turchi, and Marcello Federico. 2017. Improving zero-shot translation of low-resource languages. In Proceedings of the 14th International
Workshop on Spoken Language Translation, pages
113–119, Tokyo, Japan.
Lakew, Surafel M, Mauro Cettolo, and Marcello Federico. 2018. A comparison of transformer and recurrent neural networks on multilingual neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics,
pages 641–652, Santa Fe, USA.

Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 2, pages 3111–3119, Lake Tahoe, USA. Curran Associates Inc.
Moore, Robert C. and Will Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the 48th Annual Meeting of the Association
for Computational Linguistics (ACL) Short Papers,
pages 220–224, Uppsala, Sweden.
Nakazawa, Toshiaki, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao
Kurohashi, and Hitoshi Isahara. 2016. ASPEC:
Asian scientific paper excerpt corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation, pages 2204–2208,
Portorož, Slovenia.
Niu, Xing, Michael Denkowski, and Marine Carpuat.
2018. Bi-directional neural machine translation withsynthetic parallel data. In Proceedings of the 2nd
Workshop on Neural Machine Translation and Generation, pages 84–91, Melbourne, Australia.
Papineni, Kishore, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. BLEU: A method for automatic
evaluation of machine translation. In Proceedingsof the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia,USA.
Sennrich, Rico, Barry Haddow, and Alexandra Birch.
2016. Improving neural machine translation models with monolingual data. In Proceedings of the54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
86–96, Berlin, Germany.
Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014.
Sequence to sequence learning with neural networks.
In Proceedings of the 27th International Conference
on Neural Information Processing Systems, pages
3104–3112, Montréal, Canada.
Tillmann, Christoph and Jian-ming Xu. 2009. A simple sentence-level extraction algorithm for comparable data. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North
American Chapter of the Association for Computational Linguistics, pages 93–96, Boulder, USA.

Utiyama, Masao and Hitoshi Isahara. 2003. Reliablemeasures for aligning japanese-english news articles
and sentences. In Proceedings of the 41st Annual
Meeting of the Association for Computational Linguistics, pages 72–79, Sapporo, Japan.
Utiyama, Masao and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statisticalmachine translation. In Human Language Technologies 2007: The Conference of the North American
Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages
484–491, Rochester, USA.
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Proceedings of 30th Advances in Neural Information Processing Systems, pages 5998–
6008.
Zoph, Barret, Deniz Yuret, Jonathan May, and Kevin
Knight. 2016. Transfer learning for low-resource
neural machine translation. In Proceedings of the2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin,USA.