言語処理学会 第26回年次大会 発表論文集 (2020年3月)

Domain Adaptation of Neural Machine Translation
through Multistage Fine-Tuning
Haiyue Song†

Raj Dabre‡

Atsushi Fujita‡

Sadao Kurohashi††‡1

Kyoto University
National Institute of Information and Communications Technology{song, kuro}@nlp.ist.i.kyoto-u.ac.jp{raj.dabre, atsushi.fujita}@nict.go.jp

Introduction

3

Neural machine translation (NMT) approach [Bahdanau et al., 2015] performs well when data is abundant. For general domain or domain with lots ofparallel data, we can easily train an NMT model ofgood quality. However, for domains with very little
data, such as lecture domain, the translation quality
is extremely bad.
Existing domain adaptation methods such as finetuning can leverage out-of-domain parallel datasets.
However, when there are more than one out-ofdomain datasets, current methods simply mix them
together and thus do not explore the full potential ofmultiple out-of-domain datasets.
In this paper, we explore the potential of multiple out-of-domain datasets for educational lecturetranslation, where we propose an inflation-deflation
multistage fine-tuning strategy and domain relevancemeasurement of out-of-domain datasets, and reporton experiments and analysis showing strong improvements of translation quality over baselines.

2Related WorkIn the case of the news domain, there are many corpora, e.g., News Commentary [Tiedemann, 2012],
containing large number of parallel sentences that enable high-quality translation. In contrast, for otherdomains such as educational lectures translation,
only relatively small datasets are available. Domain
adaptation through fine-tuning an out-of-domain
model on the in-domain data [Zoph et al., 2016, Chuet al., 2017] is the most common way to overcome
the lack of data. However, approaches based on finetuning suffer from the problem of over-fitting, eventhough they could be addressed by strong regularization techniques [Hinton and Salakhutdinov, 2006,
Thompson et al., 2019]. Furthermore, the domain
divergence between the out-of- and in-domain corpora is another issue by which the training process
may not go smoothly.

Proposed Methods

We propose an inflation-deflation multistage finetuning strategy for domain adaption and use language models to measure domain relevance ofdatasets.

3.1

Multistage Fine-tuning

We propose an inflation-deflation method for multistage fine-tuning which puts out-of-domain datasets
into different fine-tuning stages appropriately.
Suppose we have n datasets,
D = hd1 , d2 , ...dn i,where the elements are sorted in increasing relevanceto the in-domain dataset dn .
Following the curriculum learning paradigm, thetraining process contains a sequence of tasks,
T = [t1 , ..., ti , ..., tf ],where each task ti is a translation task with a specificdata distribution and tf is our target task, the translation task for in-domain data. The task sequence T
is sorted in increasing relevance to the target task tf ,which makes the knowledge transfer smoothly fromthe distant tasks to the target task.
We design a sequence of the tasks and a method
to select an appropriate subset of D for each task.
Figure 1 shows the training procedure. We begin
with a randomly initialized NMT model m0 . In thefirst task, we train the model using the most irrelevant dataset d1 until convergence and save it as m1 .
In the second task (stage), we perform mixed finetuning [Chu et al., 2017] using the mixture of thesecond most irrelevant dataset d2 with d1 called d1,2 ,where we oversample the smaller one to fit the sizeof the larger one, to get model m2 . In the same way,
we get new mixed datasets d1,2,...,i , ..., until d1,2,...,nwhich is the mixture of all the n datasets. We finetune mi using dataset d1,...,i+1 to get model mi+1 in― 461 ―

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Datad1

Model

Task

m0

d1,2

d2…n

m2

m1
Inflation1

d1…ndnmn

Inflation2

mn+1

m2n-1

Deflation1

Deflationn-1

Figure 1: Inflation - deflation multistage fine-tuning strategy
each stage. We call this inflation phase because weuse one more dataset in each stage than its preceding
stages. Totally there are n inflation stages.
In the deflation phase, we remove irrelevant
dataset gradually from the training set. In the firstdeflation task, we fine-tune the last model from thelast inflation stage mn using all datasets except forthe most irrelevant one, forming d2,...,n . In the following steps, we gradually remove the most irrelevant dataset in the previous step, forming d3,...,n , andso on and so forth until only one in-domain dataset
dn remains. The model completes each translation
task with different data distribution by fine-tuning
on a mixed dataset from previous model. In the deflation phase, the data distribution of training set
gradually approaches that of the in-domain dataset.

3.2

DatasetCoursera
TED
ASPEC

Datasets

We used Coursera dataset1 from lecture domain as
our in-domain dataset. It contains automaticallyextracted English–Japanese parallel subtitles fromopen-course website. The size is small but thealignment of parallel sentences is of high quality.
We employed another two datasets as out-of-domain
datasets: One is TED corpora from TED talks [Cettolo et al., 2012] in spoken language domain. Though
it does not exactly belong to the lecture domain, thepurpose is educating people. The other is ASPEC(Asian Scientific Paper Excerpt Corpus) [Nakazawaet al., 2016], which contains 3M parallel sentences ofscientific paper domain. We only used the cleanest
1 https://github.com/shyyhs/CourseraParallelCorpusMining

Train
40k
223k
1.0M

Dev
541
1,354
1,790

Test
2,005
1,194
1,812

Table 1: Number of sentence pairs in each corpus weused.

CorpusLM
ASPEC
TED
Coursera

Dataset Relevance Measurement

We measure the domain relevance by training a language model (LM) on the in-domain dataset dn , bywhich we calculate the log-likelihood of all datasets.
The higher the log-likelihood, the more relevant thedataset is with the domain of interest.

41M parallel sentences of it. Corpora sizes are shownin Table 1.

ASPEC

TEDCoursera-1.147-2.962-2.658-3.013-1.097-2.335-2.926-2.255-0.760

Table 2: Per-token log-likelihood.
We trained a 4-gram LM on the lower-cased version of English side of each training set. Using theseLMs, we calculated the per-token log-likelihood ofeach datasets. As shown in Table 2, TED is more
relevant with in-domain dataset, Coursera, than ASPEC, presumably because they comprise spoken language unlike ASPEC. We relied on this as the clue tosort the dataset list Dn=3 as [d1 =ASPEC, d2 =TED,
d3 =Coursera] to perform multistage fine-tuning.

5MT Experiments

To empirically confirm that the proposed inflationdeflation multistage fine-tuning better leverages outof-domain data than other single-stage and multistage fine-tuning methods, we conducted an MT experiment, where we evaluated and compared MT systems trained by different fine-tuning schedules.

― 462 ―

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

IDA1A2A3A4A5A6A7A8A9A10A11A12A13A14A15A16

Training schedule
AA
A
A
A
A
A
A
A
A
A
A
A
A
A
A

AT
ATAT
AT
AT
ATAT
AT

ATC
ATC
ATC
ATC

TC
TC
TC
TC

ATC
ATC
ATC
ATC

TC
TC
TC
TC

C
C
C
C

C
C
C
C

Ja→En
13.6
25.6
27.5
25.9
24.4
24.7
26.9
24.3
23.8
25.7
25.2
24.3
24.3
25.4
23.8
21.6

En→Ja
10.4
13.5
18.0
17.6
17.7
18.5
17.5
17.6
17.2
17.9
17.4
17.5
17.8
17.6
17.1
16.9

ID
B2B3B4B5B6B7B8B9B10B11B12B13
B14
B15
B16

Training schedule
ATAT
AT
ATAT
AT
AT
AT

ATC
ATCATCATCTC
TC
TC
TC

ATCATCATCATCTC
TC
TC
TC

Ja→En

En→Ja

24.5
26.8
25.1
23.8
24.1
26.4
23.9
22.9
22.2
22.0
21.2
21.2
15.3
16.1
6.2

13.3
17.0
17.0
17.7
17.8
17.2
17.5
17.7
15.8
15.4
16.6
16.5
11.2
12.2
6.4

CC
C
C

C
C
C
C

Table 3: BLEU scores for all the 31 (= 25 − 1) sub-paths of the A→AT→ATC→TC→C flow. Bold indicatesthe initial training, and red-, blue-, and grey-colored cells mean inflation , deflation , and replacement oftraining data, respectively.

5.1

Experiment settings

5.1.1

Data Preprocessing

α as 0.6. The trained results were evaluated withBleu scores computed by sacreBLEU.3

We perform NFKC normalization to all the data andapply Juman++ [Tolmachev et al., 2018] and NLTK2
tokenization to all the Japanese and English data, respectively. Henceforth, we refer to the ASPEC training data of 1.0 million lines as A, the TED training data of 0.2 million lines as T, and the Coursera
training data of 40k lines as C. We denote the concatenated corpus by a concatenation of the lettersrepresenting them: e.g., ATC for the concatenationof ASPEC data with 5 times oversampled TED data
and 25 times oversampled Coursera data.
5.1.2

Machine Translation Settings

We used the tensor2tensor NMT framework [Vaswaniet al., 2018] with its default “transformer base” settings, such as dropout=0.2 and optimizer=adam.
We created a shared sub-word vocabulary forJapanese and English from ASPEC and TED training set using BPE [Sennrich et al., 2016] with roughly
32k merge operations and used it for all experiments.
In every experiment, we used eight GPUs withbatch size of 4,096 sub-word tokens. We saved one
checkpoint every after 1,000 steps and used early
stopping on approximate BLEU score computed on
the development set where the training process stops
when the score shows no gain larger than 0.1 for 10
checkpoints (10,000 steps).
In the decoding step, we used the average of last 10
checkpoints, with beam size as 4 and length penalty

5.2

Results and Analysis

We focus on the proposed full inflation-deflation
schedule A→AT→ATC→TC→C, while thoroughly
evaluating all of its sub-paths.
The inflation-deflation strategy showed large improvements compared with mixed training (B10) or
one stage fine-tuning (A10, B13). We saw morethan 5 Bleu points improvement on Japanese-toEnglish (Ja→En) direction (A3 over B10) and 2
points on English-to-Japanese (En→Ja) direction(A6 over B13).
The inflation strategy always brought an improvement. A3 gave the highest Bleu score on Ja→En
direction. It shows more than 20 points gain over themodel trained only on the in-domain data (B16) andmore than 5 points over the same data without multistage fine-tuning (B10). Using the same training
data, one-stage fine-tuning, A10 and B3, also showedsome gain over the model without fine-tuning B10,
but not as large as multistage fine-tuning.
Combining deflation strategy sometimes gave better results. Finally fine-tuning the model on thesmall in-domain dataset gave the best performanceon En→Ja direction (A6), 0.5 point higher than themodel without deflation (A3). B5 and B6 also gave
better performance than B3. We can also see somegain using pure deflation in B12 compared with B10.
The improvement was not consistent, suggesting thenecessity of hyper-parameter tuning for fine-tuning
on deflation stages.

2 https://www.nltk.org

3 https://github.com/mjpost/sacreBLEU― 463 ―

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

This shows the important of exhaustively exploring all settings, which confirmed and revealed thefollowings.
• Multistage inflation-deflation method can leverage multiple out-of-domain datasets better than
one stage fine-tuning or mixed training.
• Inflation strategy is often safe and we may need
hyper-parameter fine-tuning when using deflation strategy.

6Conclusion

We proposed an inflation-deflation multistage finetuning method for domain adaptation. Our experiments show that our method outperforms mixed
training and commonly-used one-stage fine-tuning
by a large margin.

Acknowledgments
This work was carried out when Haiyue Song wastaking up an internship at NICT. A part of this work
was conducted under the program “Research and Development of Enhanced Multilingual and Multipurpose Speech Translation System” of the Ministry ofInternal Affairs and Communications (MIC).

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua
Bengio. Neural Machine Translation by Jointly
Learning to Align and Translate. In Proceedingsof the Third International Conference on Learning
Representations (ICLR), San Diego, USA, May
2015.
Mauro Cettolo, Christian Girardi, and Marcello Federico. Wit3 : Web Inventory of Transcribed andTranslated Talks. In Proceedings of the 16th
Conference of European Association for MachineTranslation, pages 261–268, Trento, Italy, May
2012.
Chenhui Chu, Raj Dabre, and Sadao Kurohashi.
An Empirical Comparison of Domain Adaptation
Methods for Neural Machine Translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2:
Short Papers), Vancouver, Canada, July 2017.

Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka
Uchimoto, Masao Utiyama, Eiichiro Sumita,
Sadao Kurohashi, and Hitoshi Isahara. ASPEC:
Asian Scientific Paper Excerpt Corpus. In Proceedings of the Tenth International Conference onLanguage Resources and Evaluation (LREC), Portorož, Slovenia, May 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
Neural Machine Translation of Rare Words withSubword Units. In Proceedings of the 54th Annual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), Berlin, Germany, August 2016.
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. Overcoming
Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers), Minneapolis, USA, June2019.
Jörg Tiedemann. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources andEvaluation (LREC), Istanbul, Turkey, May 2012.
Arseny Tolmachev, Daisuke Kawahara, and Sadao
Kurohashi. Juman++: A Morphological Analysis Toolkit for Scriptio Continua. In Proceedingsof the 2018 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations, Brussels, Belgium, November 2018.
Ashish Vaswani, Samy Bengio, Eugene Brevdo,
Francois Chollet, Aidan Gomez, Stephan Gouws,
Llion Jones, Lukasz Kaiser, Nal Kalchbrenner,
Niki Parmar, Ryan Sepassi, Noam Shazeer, andJakob Uszkoreit. Tensor2Tensor for Neural Machine Translation. In Proceedings of the 13th Conference of the Association for Machine Translationin the Americas (Volume 1: Research Papers),Boston, USA, March 2018.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin
Knight. Transfer Learning for Low-Resource Neural Machine Translation. In Proceedings of the2016 Conference on Empirical Methods in Natural Language Processing, Austin, USA, November
2016.

Geoffrey Hinton and Ruslan Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786):504 – 507, 2006.

― 464 ―

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.