Pre-training via Leveraging Assisting Languages
for Neural Machine Translation
Haiyue Song1, Raj Dabre2, Zhuoyuan Mao1, Fei Cheng1,Sadao Kurohashi1, Eiichiro Sumita2
1Kyoto University

2NICT1

Enough parallel data ← Neural Machine Translation
Large
L1-L2
ParallelCorpus

Train

L1-L2
NMT model

Parallel dataTime

NMT bySeq2Seq
Jointwith NN
Learning(Sutskev(Bahdaner+)
au+)

2014

2015

2016

2018

2019

2

Lack of large parallel corpora → Low performance
MediumLow performance

L1-L2Corpus

L1-L2
MTParallel data
Seq2Seq
withNN(Sutskever+)

NMT byJointLearning(Bahdanau+)

2014

2015

Time2016

2018

2019

3

Lack of large parallel corpus ← Back Translation

L1, L2Monolingual
Corpus

L1-L2
MT

L1-L2
CorpusEasier to get
Parallel data

Monolingual data

Seq2SeqwithNN(Sutskever+)

NMT byJoint
Learning(Bahdanau+)

BackTranslat
ion(Sennrich+)

2014

2015

2016

Time2018

2019

4

BT: Initial MT → Final MT performance

L1, L2Monolingual
Corpus

L1-L2
MT

L1-L2
Corpus
2. Translate

1. Train

Initial MT

Seq2SeqwithNN(Sutskever+)

NMT byJoint
Learning(Bahdanau+)

BackTranslation(Sennrich+)

2014

2015

2016

Time2018

2019

5

BT: Small parallel corpus → Low performance
Small

L1, L2Monolingual
CorpusLow performance
L1-L2
MT

L1-L2
Corpus

2. Translate

1. Train

Initial MT ←Bottleneck

Seq2SeqwithNN(Sutskever+)

NMT byJoint
Learning(Bahdanau+)

BackTranslation(Sennrich+)

2014

2015

2016

Time2018

2019

6

Low-resource situation ← Pre-train
Large
Small

L1, L2Monolingual
Corpus

L1-L2
MT

L1-L2
CorpusPre-training
Parallel data

Monolingual data

Seq2SeqwithNN(Sutskever+)

NMT byJoint
Learning(Bahdanau+)

Backtranslation(Sennrich+)

Pretrained
wordembedding

2014

2015

2016

2018

Time

MASS

(Song+)

(Qi+)

2019

7

Extreme Low-resource: Lack both parallel and monolingual

Small

L1, L2Monolingual
Corpus

L1-L2
MT

L1-L2
CorpusLack of monolingual data?
Parallel data

Monolingual data

Seq2Seq
withNN(Sutskever+)

NMT byJoint
Learning(Bahdanau+)

Backtranslation(Sennrich+)

Pretrained
wordembedding

2014

2015

2016

2018

Time

MASS

(Song+)

(Qi+)

2019

8

Lack of monolingual data ← Proposed method

Small

L3
Monolingual
Corpus

L1-L2
MT

L1-L2
CorpusPre-training via
Assisting languages
Parallel data

Data from
Monolingual data assisting languages

Seq2SeqwithNN(Sutskever+)

NMT byJointLearning(Bahdanau+)

Backtranslation(Sennrich+)

2014

2015

2016

Pretrained
wordembedding

MASS(Song+)

Pretrainingvia
AssistingLanguage(Proposed)

2019

2020

(Qi+)

Time
2018

9

Proposed method: Overview

Goal!

L1-L2
MT model10

Proposed method: Overview
Monolingual data

Parallel data

① Pre-train
Pre-trainedmodel② Fine-tune
L1-L2
MT modelGoal!
11

Proposed method: Overview
Monolingual data
L3, L4...
Assistinglanguages

L1, L2(Optional)

Parallel data
L1-L2

① Pre-train
Pre-trainedmodel② Fine-tune
L1-L2
MT modelGoal!
12

Proposed method: Overview
Monolingual data
L3, L4...
Assistinglanguages

L1, L2(Optional)

Mapping

Mixeddata

Data
Selection① Pre-train
Pre-trained
model

Parallel data
L1-L2

② Fine-tune
L1-L2
MT modelGoal!
13

Proposed method: Mapping
Goal:

Maximize the cognate sharing

L3
L1

汉漢

14

Proposed method: Mapping
Goal:

Maximize the cognate sharing

L3L1

汉

Map漢漢15

Proposed method: Mapping
ChineseHanzi

Goal:

Maximize the cognate sharing
Example:Chinese Hanzi and Japanese Kanji

Early ages

漢JapaneseKanji

漢16Proposed method: Mapping
ChineseHanzi

Goal:

Maximize the cognate sharing
Example:Chinese Hanzi and Japanese Kanji

Early ages

漢Japanese
Kanji

漢

Background:
Kanji borrowed from Hanzi

17Proposed method: Mapping
ChineseHanzi

Goal:

Maximize the cognate sharing
Example:Chinese Hanzi and Japanese Kanji
Background:
Kanji borrowed from Hanzi
Over time the written scripts diverged

Early ages

JapaneseKanji

漢

漢

汉

漢

Simplified(1950s)
Now

18

Proposed method: Mapping
ChineseHanzi

Goal:

Maximize the cognate sharing
Example:Chinese Hanzi and Japanese Kanji
Background:
Kanji borrowed from Hanzi
Over time the written scripts diverged
Method:Map Hanzi to Kanji by a mapping table(Chu et al., 2012)

Early ages

Japanese
Kanji

漢漢

汉

漢

Simplified(1950s)Now

Mapping

19

Proposed method: Mapping
Goal:

Maximize the cognate sharing

Method: Map Hanzi to Kanji
One Hanzi may map to many Kanji

Hanzi

Kanji

机[机, 機]构[構, 搆]20

Proposed method: MappingGoal:

Maximize the cognate sharing

Method: Map Hanzi to Kanji
One Hanzi may map to many Kanji
Method 1: one-to-one mapping

Hanzi

Kanji

机

[机, 機]

构[構, 搆]21

Proposed method: Mapping
Goal:

Maximize the cognate sharing

Method: Map Hanzi to Kanji
One Hanzi may map to many Kanji
Method 1: one-to-one mapping

Hanzi

Kanji

机

[机, 機]

构[構, 搆]

22

Proposed method: Mapping
Goal:

Maximize the cognate sharing

Method: Map Hanzi to Kanji
One Hanzi may map to many Kanji

Hanzi

Kanji

机[机, 機]构[構, 搆]Method 1: one-to-one mapping
Method 2: many-to-many mapping

Chinese word

机构Japanese word(Synthetic)[机構, 机搆,機構,機搆]23

Proposed method: Mapping
Goal:

Maximize the cognate sharing

Method: Map Hanzi to Kanji
One Hanzi may map to many Kanji

Hanzi

Kanji

机

[机, 機]

构[構, 搆]Method 1: one-to-one mapping
Method 2: word-to-word mapping

Chinese word

机构Japanese word
(Synthetic)[机構, 机搆,機構,機搆]Japanese LM24

Proposed method: Mapping
Goal:

Maximize the cognate sharing

Method: Map Hanzi to Kanji
One Hanzi may map to many KanjiHanziKanji

机[机, 機]

构[構, 搆]Method 1: one-to-one mapping
Method 2: word-to-word mapping

Chinese word

机构Japanese word
(Synthetic)[机構, 机搆,機構,機搆]Japanese LM25

Proposed method: Data Selection
Goal:

Reduce difference between train and test

Monolingual data
L3, L4...
Assistinglanguages

L1, L2(Optional)

Mapping

Mixed
dataDataSelection

26

Proposed method: Data Selection
Goal:

Reduce difference between train and test

Method:Data Selection

Method 1: LM based data selectionMonolingual data
L3, L4...
Assistinglanguages

L1, L2(Optional)

Mapping

Mixed
data

DataSelection27

Proposed method: Data Selection
Goal:

Reduce difference between train and test

Method:Data Selection

Method 1: LM based data selection
Method 2: Length based data selection

Monolingual data
L3, L4...
Assistinglanguages

L1, L2(Optional)

Mapping

Mixed
data

DataSelection28

Proposed method: Data Selection
Length Distribution (LD) of target data

Goal:

LD of train data

Reduce difference between train and test

Method:Data Selection

Method 1: LM based data selection
Method 2: Length based data selection

29

Proposed method: Data Selection
Length Distribution (LD) of target data

Goal:

LD of train data

Reduce difference between train and test

Method:Data Selection

Method 1: LM based data selection
Method 2: Length based data selection

LD
Data selection

LD of selected train data30

Proposed method: Data Selection
Goal:

LD of target data and original train data

Reduce difference between train and test

Method:Data Selection

Method 1: LM based data selection
Method 2: Length based data selection31

Proposed method: Data Selection
Goal:

LD of target data and original input data

Reduce difference between train and test

Method:Data Selection

Method 1: LM based data selection
Method 2: Length based data selection
LD of target data and selected input data32

Pre-train: MASS (Song+, 2019)
Method:Input: Monolingual sentencewith tokens [MASK]ed
Target: [MASK]ed tokens

Experiment settings:
Interested languages:

Japanese and English

Assisting languages:

Chinese, French, Arabic and Russian34

Experiment settings:
Interested languages:
Japanese and English
Assisting languages:
Chinese, French, Arabic and Russian

Dataset:Pre-train:Ja, En: ASPEC (Nakazawa+, 2016)
Others: Common Crawl*
Fine-tune:
Ja-En: ASPEC(Nakazawa+, 2016)
No overlap with pre-train data

Data for LM: News commentary*

Data pre-processing:Normalization and filtering
Script mapping for Zh->Ja
KenLM to train LM*http://data.statmt.org/ngrams/*http://data.statmt.org/news-commentary/v14/35

Experiment settings:
Interested languages:
Japanese and English
Assisting languages:
Chinese, French, Arabic and Russian

Train and evaluate:

Dataset:Pre-train:
Ja, En: ASPEC (Nakazawa+, 2016)
Others: Common Crawl*
Fine-tune:
Ja-En: ASPEC(Nakazawa+, 2016)•••

Tensor2tensor (Vaswani+, 2018) with‘transformer_big’ setting
Shared vocab of 64k, using SentencePiece(Kuro+, 2018)sacreBLEU

No overlap with pre-train data

Data for LM: News commentary*
Data pre-processing:Normalization and filtering
Script mapping for Zh->Ja
KenLM to train LM*http://data.statmt.org/ngrams/*http://data.statmt.org/news-commentary/v14/

36

Results:

1. Extreme Low Resource Situation

Compared with baseline, using monolingual data from assisting languages helps.
There may be conflicts between data of different assisting languages.

37

Results:

2. Low Resource Situation

Compared with baseline, using monolingual data from assisting languages helps.
There may be conflicts between data of different assisting languages.

38

Results:

3. Rich Resource Situation

Data from assisting languages does not help.

39

Results:

Mapping:

1-to-1 Zh->Ja mapping is better than many-to-many mapping
Japanese LM cannot directly apply to Chinese mapped data
Segmentation granularity of Chinese and Japanese data is different

40

Results:

Data Selection:

Sentence length distribution selection is better than LM score method
Maybe the data used to train the LM is not in-domain.

41

Results:

Different assisting languages:

Similar languages performs better than randomly selected languages

42

Conclusions:• Leveraging monolingual data from other languages to improve NMT ispossible.
• Script mapping is a good way to improve data similarity thus improve
performance.

Future work:• Explore data selection methods• Experiments with more challenging language pairs such as Japanese-Russian43Thanks for listening!
Pre-training via Leveraging Assisting Languages
for Neural Machine Translation
Haiyue Song1, Raj Dabre2, Zhuoyuan Mao1, Fei Cheng1,Sadao Kurohashi1, Eiichiro Sumita2
1Kyoto University

2NICT44