Exploiting Multilingualism through Multistage Fine-Tuningfor Low-Resource Neural Machine Translation
Raj Dabre†

Atsushi Fujita†

Chenhui Chu∗{raj.dabre, atsushi.fujita}@nict.go.jp, chu@ids.osaka-u.ac.jp†National Institute of Information and Communications Technology (NICT), Kyoto, Japan∗
Osaka University, Osaka, Japan

1. Summary

2. Key Aspects Of SettingMotivation● Training Data
● N-way multilingual corpus● ALT Dataset (Document Splits)● Wikinews Domain● Train/Dev/Test:18000/1000/1106

Small Multilingual In-Domain Corpus• Improving translation from English to low resource Asian languages

Bn-Fil-Id-Ja-Km-Ms-Vi-En• Focus on Languages from the Asian Language Treebank (ALT):
Bengali, Filipino, Indonesian, Japanese, Khmer, Malay, Vietnamese

Out-Of-Domain ● Helping DataCorpus● Bilingual corpus

Constraints• Language pair specific training data is extremely small

En-Zh● IWSLT Dataset● Spoken Language Domain● 209,491 lines• Extremely small data is N-way multilingual (content redundant)• Availability of a helping corpus with an unrelated target language
Contributions

FTMFTFinetuning

MixedFinetuning● Transferlearning● Black-box● Simple● Chu+, 2017

• Multi-stage training involving strongly trained multilingual models• Stronger pre-training via multilingualism through N-way corpora• Target language mismatch between helping and training corpora• Robust transfer learning giving 3–9 BLEU improvements• Reliance on off-the-shelf techniques

3. Multilingual Multistage Fine-Tuning
En-XX

En-YY1

En-YY2. . .

4. Experimental Results
1-to-1, 3.Pure
1-to-7, 2.Mix
1-to-7, 2.Mix+3.Pure

En-YYN

1-to-2, 1.Pre+3.Pure
1-to-8, 1.Pre+2.Mix
1-to-8, 1.Pre+2.Mix+3.Pure4035Pre-process

30

En-XX(prep)

En-ALL(prep)

En-YYk(prep)

BLEU

25
20
15
10Pre-train
NMT Model

Fine-tune
NMT Model

Fine-tune
NMT Model

En-XX
Model

En-AllModel

En-YYkModel

Pre-training(Pre)

Mixed pre-training
or fine-tuning (Mix)

Pure fine-tuning(Pure)

50Bn

JaKm

Tl

Id

ViMs

Observations• Improvements despite different target languages between datasets• The size of the helping corpus has a significant impact• Multilingual models are better for fine-tuningExpectation• Multilingualism for strongly pre-trained intermediate models– Multilingual MFT gives robust pre-trained models– N-way corpora help despite content redundancy• Multistage fine-tuning for efficient learning via division of labor

5. Experimental Details

6. Conclusion and Future Work

Pre-processingConclusion• ALT data was used in pre-tokenized form

• Robust multilingual models are useful for transfer learning• Chinese and Japanese segmented with KyotoMorph and JUMAN– Multistage fine-tuning is better than direct fine-tuning• Source sentences pre-pended with “XX” representing target language– Division of labor leading to improved final translation quality• Oversampled smaller corpus to match larger corpus size

• Strong evidence for the importance of multilingualism• Used internal sub-word segmentation of tensor2tensor– N-way training data is content redundantSettings• All implementations done using tensor2tensor v1.6
• Vocabulary sizes chosen according to dataset size (see paper)• Training and decoding (“transformer_base”)– Trained till convergence on development data
– Averaged the last 10 checkpoints for decoding– Beam size and length penalty of 4 and 0.6, respectively– External corpus is useful despite mismatched target language• Improvements of 3–9 BLEU for all low resource languages
Future Work• Experiment with several helping corpora• Study impact of domain and helping corpora sizes on final translation• Apply the approach to many-to-many translation