Rapid Adaptation ofMachine Translation to NewLanguages
Graham Neubig, Junjie Hu@ EMNLP 11/2/2018

Inspiration: Rapid Disaster Response
නාවල%&ය යටකළ වැ,
ව-ර - #HiruNews#StandBy
ගංව-ර හා නායයා2 ත4වය5ෙග57පතට ප4 වූ %:;වලට සහනසැපයූ ;ෙ=>ඡා ක@ඩාය2දCව5ට D:%& ඉFලා පළ කළසමාජ ජාල පIවුඩ පසුKය Lනවල
දD5නට ලැMI .

Disaster in Sri Lanka
Photo Credit: Wikimedia Commons

How can we effectively
and rapidly adapt MT to
new languages?

Some Crazy Ideas• Cross-lingual transfer: can we create a machinetranslation system by transferring across language
boundaries? [Zoph+16]• Zero-shot transfer: can we do it with no data in the low-resource language?

Multi-lingual Training[Firat+16, Johnson+17, Ha+17]•

Train a large multi-lingual MT system, and apply it to a low-resource language

fraporrustur...

belazeengTwo Multilingual Training Paradigms
Warm-start training: (indicated w/ "+")

•frapor••••••We already have some data in the test languagerus

Train a model starting with that datatur.
belCold-start training: (indicated w/ "-")

azeWe initially have no data in the test language
Possibilities for completely unsupervised transfer
Suitable for rapid adaptation to new languages

eng

fraporrustur.
belxazeeng

Experiments: Training Setting•

TED multi-lingual corpus (Qi et al. 2018)https://github.com/neulab/word-embeddings-for-nmt•

57 source languages, plus English•

Testbed languages: Azerbaijani (aze), Belarusian (bel), Galician (glg), Slovak (slk)•Related languages: Turkish (tur), Russian (rus), Portuguese (por), Czech (ces)Systems••

Test Systems:•Single-source Neural MT (Sing.): Test source language only•

Bi-source Neural MT (Bi.): Test source language and related source•

All-source Neural MT (All): All source languages

Other Baselines:•Phrase-based MT: Shown to be strong in low-resource settings•

Unsupervised MT [Artetxe+17]: Learn system using only monolingual data insource/target languages (cited as effective in low-resource settings)

How does Cross-lingual Transfer Help?
30

PBMT

Unsupervised

NMT Sing.

NMT Bi+NMT All+

22.5

15

7.5

0aze/turbel/rusglg/por

Unsupervised translation not competitive• Without transfer, NMT worse than PBMT
• With transfer NMT significantly better (transfer barely helped PBMT)•slk/cesHow Does Cold-start Compare?
30

NMT Bi+NMT All+

NMT Bi-NMT All-

22.5

15

7.5

0aze/turbel/rus

glg/por•

Large drop, but still much better than nothing•

Up to 15 BLEU with no training data in test language

slk/cesAdaptation to New Languages•Training on all languages can be less effective, esp. in cold-start case•Can we further adapt to new languages?

•Problem: overfitting

Adaptation(All→Sing.)

Pre-trainingaze

fraporrus
tur...
belazeeng

eng

Adaptation w/Similar Language Regularization(All→Bi.)turazeeng

Warm-start + Adaptation
30

NMT Sing.

NMT Bi+NMT All+

All+ -> Sing.

All+ -> Bi

22.5

15

7.5

0aze/turbel/rus•

Adaptation helps!

•Helps more w/ similar language regularization

glg/porslk/cesCold-start + Adaptation
30

NMT Sing.

NMT Bi-NMT All-All- -> Sing.

All- -> Bi

All+ -> Bi

22.5

15

7.5

0aze/turbel/rus

glg/por•

Adaptation w/ similar-language regularization gains more•

Approaches quality of warm-start; doesn't need data a-priori

slk/cesHow Fast can we Adapt?
Cold-start adaptation reaches good point
faster than training from scratch

0.21
0.18

BLEU

0.15

Sing.
Bi
All-→Sing.
All-→Bi
All-→Bi 1-1

0.12
0.09
0.06
0.03
001

2

3456
Hours Training

78

910

Take-aways•

NMT with massively multi-lingual cross-lingual transfer: a stable recipe for lowresource translation•Better results than phrase-based, unsupervised MT in real low-resource
languages•Adaptation w/ similar language regularization: simple and effective, even in coldstart scenarios

https://github.com/neubig/rapid-adaptation

Questions?