Rapid Adaptation of Neural Machine Translation to New Languages

Graham Neubig, Junjie Hu
Language Technologies Institute, Carnegie Mellon University{gneubig,junjieh}@cs.cmu.edu

Abstract
This paper examines the problem of adapting neural machine translation systems to new,
low-resourced languages (LRLs) as effectively
and rapidly as possible. We propose methods based on starting with massively multilingual “seed models”, which can be trained
ahead-of-time, and then continuing training ondata related to the LRL. We contrast a number of strategies, leading to a novel, simple,
yet effective method of “similar-language regularization”, where we jointly train on both a
LRL of interest and a similar high-resourced
language to prevent over-fitting to small LRL
data. Experiments demonstrate that massively
multilingual models, even without any explicit
adaptation, are surprisingly effective, achieving BLEU scores of up to 15.5 with no datafrom the LRL, and that the proposed similarlanguage regularization method improves overother adaptation methods by 1.7 BLEU points
average over 4 LRL settings.1

1

Introduction

When disaster strikes, news and social media areinvaluable sources of information, allowing humanitarian organizations to rapidly mitigate crisis
situations and save lives (Vieweg et al., 2010; Neubig et al., 2011; Starbird et al., 2012). However,language barriers looms large over these efforts,
especially when disasters occur in parts of theworld that use less common languages. In thesecases, machine translation (MT) technology canbe a valuable tool, with one widely-heralded success story being the deployment of Haitian Creoleto-English translation systems during the earthquakes in Haiti (Lewis, 2010; Munro, 2010).
However, data-driven MT systems, particularly
neural machine translation (NMT; Kalchbrenner
1
Code to reproduce experiments at https://github.
com/neubig/rapid-adaptation

and Blunsom (2013); Bahdanau et al. (2015)), require large amounts of training data, and creating
high-quality systems in low-resource languages(LRLs) is a difficult challenge where research efforts have just begun (Gu et al., 2018). Anotherhurdle, which to our knowledge has not been covered in previous research, is the time it takes tocreate such a system. In a crisis situation, timeis of the essence, and systems that require days or
weeks of training will not be desirable or even feasible.
In this paper we focus on the question: how can
we create MT systems for new language pairs as
accurately as possible, and as quickly as possible?
To examine this question we propose NMT methods at the intersection of cross-lingual transfer
learning (Zoph et al., 2016) and multilingual training (Johnson et al., 2016), two paradigms that, to
our knowledge, have not been used together in previous work. Our methods, laid out in §2 follow theprocess of training a seed model on a large number of languages, then fine-tuning the model to improve its performance on the language of interest.
We propose a novel method of similar-language
regularization (SLR) where training data from asecond similar languages is used to help prevent
over-fitting to the small LRL dataset.
In the experiments in §3, we attempt to answer
two questions: (1) Which method of creating multilingual systems and adapting them to an LRL is
the most effective way to increase accuracy? (2)How can we create the strongest system possible
with a bare minimum of training time? The results are sometimes surprising – we first find thata single monolithic model trained on 57 languages
can achieve BLEU scores as high as 15.5 with notraining data in the new source language whatsoever. In addition, the proposed method startingwith a universal model then fine-tuning with the
SLR proves most effective, achieving gains of 1.7

BLEU points averaged over several language pairs
compared to previous methods adapting to only
the LRL.

2

Training Paradigms

In this paper, we consider the setting where wehave a source LRL of interest, and we want totranslate into English.2 All of our adaptation
methods are based on first training on larger data
including other languages, then fine-tuning themodel to be specifically tailored to the LRL. We
first discuss a few multilingual training paradigms
from previous literature (§2.1), then discuss our
proposed adaptation methods (§2.2).
2.1

Multilingual Modeling Methods

We use three varieties of multilingual training:
Single-source modeling (“Sing.”) is the first
method, using only parallel data between the LRLof interest and English. This method is straightforward and the resulting model will be most highly
tailored to the final test language pair, but themethod also has the obvious disadvantage thattraining data is very sparse.
Bi-source modeling (“Bi”) trains an MT systemwith two source languages: one LRL that wewould like to translate from, and a second highly
related high-resource language (HRL): the helper
source language.3 This method is inspired by
Johnson et al. (2016), who examine multilingualtranslation models to/from English and two highly
related languages such as Spanish/Portuguese or
Japanese/Korean. The advantage of this method isthat it allows the LRL to learn from a highly similar helper, potentially increasing accuracy.
All-source modeling (“All”) trains not only on acouple source languages, but instead creates a universal model on all of the languages that we haveat our disposal. In our experiments (§3.1) this entails training systems on 58 source languages, toour knowledge the largest reported in NMT experiments.4 This paradigm allows us to train a single
2
Translation into LRLs, is a challenging and interesting
problem in it’s own right, but beyond the scope of the paper.
3“Related” could mean different things: typologically related or having high lexical overlap. In our experiments our
LRLs are all selected to have an helper that is highly similar
in both aspects, but choosing an appropriate helper when thisis not the case is an interesting problem for future work.
4
In contrast to Gu et al. (2018), who train on 10 languages.
Malaviya et al. (2017); Tiedemann (2018) train NMT on over
1,000 languages, but only as a feature extractor for downstream tasks; MT accuracy itself is not evaluated.

model that has wide coverage of vocabulary andsyntax of a large number of languages, but also
has the drawback in that a single model must beable to express information about all the languages
in the training set within its limited parameter budget. Thus, it is reasonable to expect that this model
may achieve worse accuracy than a model createdspecifically to handle a particular source language.
In the following, we will consider adaptation
methods that focus on tailoring a more generalmodel (i.e. bi-source or universal) to a more specific model (i.e. single-source or bi-source).
2.2

Adaptation to New Languages

As noted in the introduction, there are two major
requirements: the accuracy of the system is important and the training time required from when
we learn of a need for translation to when we canfirst start producing adequate results. Throughout
the discussion, we will compare various adaptation paradigms with respect to these two aspects.
2.2.1

Adaptation by Fine-tuning

Our first adaptation method, inspired by Zoph
et al. (2016) is based on fine-tuning to the source
language of interest. Within our experiments, wewill test this setting, but also make two distinctions
between the types of adaptation:
Seed Model Variety: Zoph et al. (2016) performed experiments taking a bilingual systemtrained on a different language (e.g. French) andadapting it to a new LRL (e.g. Uzbek). We canalso take universal model and adapt it to the new
language, a setting that we examine (to our knowledge, for the first time) in this work.
Warm vs. Cold Start: Another contrast iswhether we have training data for the LRL of interest while training the original system, or whether
we only receive training data after the originalmodel has already been trained. We call the former
warm start, and the latter cold start. Intuitively,
we expect warm-start training to perform better,as having access to the LRL of interest during thetraining of the original model will ensure that it
can handle the LRL to some extent. However, thecold-start scenario is also of interest: we may wantto spend large amounts of time training a strongmodel, then quickly adapt to a new language that
we have never seen before in our training data asdata becomes available. For the cold-start models,
we start with a model that is only trained on the
HRL similar to the LRL (Bi− ), or a model trained

LRL
aze
bel
glg
slk

train
5.94k
4.51k
10.0k
61.5k

dev
671
248
682
2,271

test
903
664
1,007
2,445

HRL
tur
rus
por
ces

train
182k
208k
185k
103k

Table 1: Data sizes in sentences for LRL/HRL pairson all languages but the LRL (All− ).
2.2.2

Similar-Language Regularization

One problem with adapting to a small amount ofdata in the target language is that it will be very
easy for the model to over-fit to the small training set. To alleviate this problem, we propose amethod of similar language regularization: while
training to adapt to the language of interest, we
also add some data from another similar HRL
that has sufficient resources to help prevent overfitting. We do this in two ways:
Corpus Concatenation: Simply concatenate thedata from the two corpora, so that we have a small
amount of data in the LRL, and a large amount ofdata in the similar HRL.
Balanced Sampling: Every time we select a minibatch to do training, we either sample it from the
LRL, or from the HRL according to a fixed ratio. We try different sampling strategies, including sampling with a 1-to-1 ratio, 1-to-2 ratio, and1-to-4 ratio for the LRL and HRL respectively.

3

Experiments

3.1

Experimental Setup

We perform experiments on the 58-language-toEnglish TED corpus (Qi et al., 2018), which isideal for our purposes because it has a wide varietyof languages over several language families, somehigh-resourced and some low-resourced. Like
Qi et al. (2018), we experiment with Azerbaijani(aze), Belarusian (bel), and Galician (glg) to English, and also additionally add Slovak (slk), aslightly higher resourced language, for contrast.
These languages are all paired with a similar HRL:
Turkish (tur), Russian (rus), Portuguese (por), andCzech (ces) respectively. Data sizes are shown in
Table 1.
Models are implemented using xnmt (Neubig et al., 2018), commit 8173b1f, and startwith the recipe for training on IWSLT TED5 . The
model consists of an attentional neural machinetranslation model (Bahdanau et al., 2015), using
bi-directional LSTM encoders, 128-dimensional
5Found in examples/stanford-iwslt/

word embeddings, 512-dimensional hidden states,
and a standard LSTM-based decoder.
Following standard practice (Sennrich et al.,2016; Denkowski and Neubig, 2017), we breaklow-frequency words into subwords using thesentencepiece toolkit.6 There are two alternatives for creating subword units: jointly learning
subwords over all source language, or separately
learning subwords for each source language, thentaking the union of all the subword vocabularies as
the vocabulary for the multilingual model. Previous work on multilingual training has preferred theformer (Nguyen and Chiang, 2017), but in this paper we use the latter for two reasons: (1) because
data in the LRL will not affect the subword units
from the other languages, in the cold-start scenario we can postpone creation of subword units
for the LRL until directly before we start training on the LRL itself, and (2) we need not be
concerned with the LRL being “overwhelmed” by
the higher-resourced languages when calculating
statistics used in the creation of subword units, because all languages get an equal share.7 In the experiments, we use a subword vocabulary of 8,000
for each language.
We also compare with two additional baselines:
phrase-based MT implemented in Moses,8 andunsupervised NMT implemented in undreamt.9
Moses is trained on the bilingual data only (training multilingually reduced average accuracy), andundreamt is trained on all monolingual data
available for the LRL and English.
3.2

Experimental Results

Table 2 shows our main translation results, withwarm-start scenarios in the upper half and coldstart scenarios in the lower half.
Does Multilingual Training Help? To answer
this question, we can compare the warm-startSing., Bi, and All settings, and find that the answer is a resounding yes, gains of 7-13 BLEU
points are obtained by going from single-source
to bi-source or all-source training, corroborating
previous work (Gu et al., 2018). Bi-source models tend to perform slightly better than all-source
models, indicating that given identical parameter6

https://github.com/google/
sentencepiece, using the unigram training setting.
7
Preliminary experiments found both comparable: withscores of 20.1 and 19.4 for separate and joint respectively.
8
http://statmt.org/moses
9
https://github.com/artetxem/undreamt

bel/rus
10.5
0.3
2.8
15.8
16.7
16.3
17.5
18.3
18.3
17.5
17.9
2.5
3.5
11.8
15.3
17.4
16.0
17.1
17.4

glg/por
22.3
0.4
16.2
27.3
26.5
27.5
28.2
28.8
28.8
29.1
28.5
8.6
15.5
25.4
26.5
28.4
28.0
28.3
28.4

slk/ces
23.0
0.0
24.0
26.5
25.0
27.1
27.4
28.2
28.3
28.2
27.9
5.4
7.3
26.8
27.6
28.0
28.2
27.9
27.6

Avg.
15.4
0.2
11.4
20.1
19.5
20.6
20.8
21.8
21.4
21.4
21.3
5.1
7.5
18.2
19.5
21.2
20.7
21.0
21.1

Table 2: BLEU for single-source (Sing.), bi-source(Bi), and all-source universal (All) models, withadapted counterparts. 1-1, 1-2, 1-4 indicate balanced
sampling from §2.2. Bold indicates highest score.

capacity, training on a highly resourced language
is effective. Comparing with the phrase-based
baseline, as noted by Koehn and Knowles (2017)
NMT tends to underperform on low-resource settings when trained only on the data available forthese languages. However, multilingual trainingof any variety quickly remedies this issue; all outperform phrase-based handily.
More interestingly, examining the cold-start results, we can see that even systems with no datain the target language are able to achieve nontrivial accuracies, up to 15.5 BLEU on glg-eng.
Interestingly, in the cold-start scenario, the All−
model bests the Bi− model, indicating that massively multilingual training is more useful in thissetting. In contrast, the unsupervised NMT model
struggles, achieving a BLEU score of around 0 for
all language pairs – this is because unsupervised
NMT requires high-quality monolingual embeddings from the same distribution, which can betrained easily in English, but are not available in
the low-resource languages we are considering.
Does Adaptation Help? Regarding adaptation,
we can first observe that regardless of the original model and method for adaptation, adaptationis helpful, particularly (and unsurprisingly) in the
cold-start case. When adapting directly to only
the target language (“→Sing.”), adapting from themassively multilingual model performs better, indicating that information about all input languages
is better than just a single language. Next, comparing with our proposed method of adding similar

aze-tur

0.15
0.12
BLEU

aze/tur
5.9
0.0
2.7
10.9
9.7
11.4
10.1
11.7
10.2
11.0
11.1
3.8
3.7
8.7
8.8
10.7
10.5
10.7
11.0

Sing.
Bi
All-→Sing.
All-→Bi
All-→Bi 1-1

0.09
0.06
0.03
0

BLEU

Cold Start

Warm Start

Strategy
Phrase-based
Unsupervised NMT
Sing.
Bi
All
Bi →Sing.
All→Sing.
All→BiAll→Bi 1-1
All→Bi 1-2
All→Bi 1-4
Bi−All−
Bi− →Sing.
All− →Sing.
All− →Bi
All− →Bi 1-1
All− →Bi 1-2
All− →Bi 1-4

0.21
0.18
0.15
0.12
0.09
0.06
0.03
0

0

1

2

3
4
5
Hours Training

6

7

8

9

bel-rus

Sing.
Bi
All-→Sing.
All-→Bi
All-→Bi 1-1
0

1

2

3

4
5
6
Hours Training

7

8

9

10

Figure 1: Example of adaptation on the aze-eng andbel-eng development setslanguage regularization (“→Bi”), we can see that
this helps significantly over adapting directly to
the LRL, particularly in the cold-start case where
we can observe gains of up to 1.7 BLEU points.
Finally, in our data setting, corpus concatenation
outperforms balanced sampling in both the coldstart and warm-start scenarios.
How Can We Adapt Most Efficiently? Finally,
we revisit adapting to new languages efficiently,
with Figure 1 showing BLEU vs. hours training
for the aze/tur and bel/rus source language pairs(others were similar). We can see that in all cases
the cold-start models (All− →) either outperform
or are comparable in final accuracy to the fromscratch single-source and bi-source models. In addition, all of the adapted models converge fasterthan the bi-source from-scratch trained models, indicating that adapting from seed models is a goodstrategy for rapid construction of MT systems in
new languages. Comparing the cold-start adaptation strategies, we can see that in general, thehigher the density of target language training data,
the faster the training converges to a solution, but
the worse the final solution is. This suggests that
there is a speed/accuracy tradeoff in the amountof similar language regularization we apply during fine-tuning.

4

Related Work

While adapting MT systems to new languages
is a long-standing challenge (Schultz and Black,2006; Jabaian et al., 2013), multilingual NMT ishighly promising in its ability to abstract acrosslanguage boundaries (Firat et al., 2016; Ha et al.,2016; Johnson et al., 2016). Results on multilingual training for low-resource translation (Guet al., 2018; Qi et al., 2018) further demonstrates
this potential, although these works do not consider adaptation to new languages, the main focus
of our work. Notably, we did not examine partial freezing of parameters, another method provenuseful for cross-lingual adaptation (Zoph et al.,2016); this is orthogonal to our multi-lingual training approach but the two methods could potentially be combined. Finally, unsupervised NMT
approaches (Artetxe et al., 2017; Lample et al.,2018, 2017) require no parallel data, but rest on
strong assumptions about high-quality comparable
monolingual data. As we show, when this assumption breaks down these methods fail to function,
while our cold-start methods achieve non-trivial
accuracies even with no monolingual data.

5

Conclusion

This paper examined methods to rapidly adapt MT
systems to new languages by fine-tuning. In bothwarm-start and cold-start scenarios, the best results were obtained by adapting a pre-trained universal model to the low-resource language whileregularizing with similar languages.

Acknowledgements
The authors thank Jaime Carbonell, Xinyi Wang,
Rebecca Knowles, Arya McCarthy, and anonymous reviewers for their constructive commentson this paper.
This work is sponsored by Defense Advanced
Research Projects Agency Information Innovation
Office (I2O). Program: Low Resource Languages
for Emergent Incidents (LORELEI). Issued by
DARPA/I2O under Contract No. HR0011-15C0114. The views and conclusions contained inthis document are those of the authors and should
not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized toreproduce and distribute reprints for Government
purposes notwithstanding any copyright notation
here on.

References
Mikel Artetxe, Gorka Labaka, Eneko Agirre, andKyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In Proc. ICLR.
Michael Denkowski and Graham Neubig. 2017.
Stronger baselines for trustable results in neural machine translation. In Proc. WNMT.
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio.
2016. Multi-way, multilingual neural machinetranslation with a shared attention mechanism. In
Proc. NAACL, pages 866–875.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK
Li. 2018. Universal neural machine translation for
extremely low resource languages. Proc. NAACL.
Thanh-Le Ha, Jan Niehues, and Alexander Waibel.
2016. Toward multilingual neural machine translation with universal encoder and decoder. arXiv
preprint arXiv:1611.04798.
Bassam Jabaian, Laurent Besacier, and Fabrice
Lefevre. 2013. Comparison and combination oflightly supervised approaches for language portability of a spoken language understanding system.
IEEE Transactions on Audio, Speech, and Language
Processing, 21(3):636–648.
Melvin Johnson et al. 2016. Google’s multilingual neural machine translation system: Enabling zero-shottranslation. TACL.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent
continuous translation models. In Proc. EMNLP,
pages 1700–1709.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. Proc. WNMT.
Guillaume Lample,
Ludovic Denoyer,
andMarc’Aurelio Ranzato. 2017.
Unsupervisedmachine translation using monolingual corpora
only.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018.
Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755.
William D. Lewis. 2010. Haitian Creole: how to build
and ship an MT engine from scratch in 4 days, 17hours, & 30 minutes. In Proc. EAMT.
Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In Proc. EMNLP.
Robert Munro. 2010. Crowdsourced translation for
emergency response in Haiti: the global collaboration of local knowledge. In Proc. AMTA Workshop
on Collaborative Crowdsourcing for Translation.
Graham Neubig, Yuichiroh Matsubayashi, Masato
Hagiwara, and Koji Murakami. 2011. Safety information mining - what can NLP do in a disaster -. In
Proc. IJCNLP, pages 965–973.

Graham Neubig, Matthias Sperber, Xinyi Wang,
Matthieu Felix, Austin Matthews, Sarguna Padmanabhan, Ye Qi, Devendra Singh Sachan, Philip
Arthur, Pierre Godard, John Hewitt, Rachid Riad,and Liming Wang. 2018. XNMT: The extensible
neural machine translation toolkit. In Proc. AMTA,
Boston.
Toan Q. Nguyen and David Chiang. 2017. Transfer
learning across low-resource, related languages forneural machine translation. pages 296–301.
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When andwhy are pre-trained word embeddings useful forneural machine translation? In Proc. NAACL, New
Orleans, USA.
Tanja Schultz and Alan W Black. 2006. Challenges
with rapid adaptation of speech translation systems
to new language pairs. In Proc. ICASSP, volume 5.
IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words withsubword units. In Proc. ACL, pages 1715–1725.
Kate Starbird, Grace Muzny, and Leysia Palen. 2012.
Learning from the crowd: Collaborative filtering
techniques for identifying on-the-ground Twitterers
during mass disruptions. In Proc. ISCRAM.
Jörg Tiedemann. 2018. Emerging language spaces
learned from massively multilingual corpora. arXiv
preprint arXiv:1802.00273.
Sarah Vieweg, Amanda L Hughes, Kate Starbird, andLeysia Palen. 2010. Microblogging during two natural hazards events: what Twitter may contribute tosituational awareness. In Proc. CHI, pages 1079–
1088.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin
Knight. 2016. Transfer learning for low-resource
neural machine translation. In Proc. EMNLP, pages
1568–1575.