Multi-Source
Neural Machine Translation
with Data Augmentation
Yuta Nishimura1, Katsuhito Sudoh1, Graham Neubig2,1, Satoshi Nakamura1
1Nara Institute of Science and Technology
2Carnegie Mellon University

Overview of this research (1/2)
Multi-lingual corpora usually have missing translationsдоброе утроспасибо

HolaHello

Buenos díasGood morning

Thank you

In multi-source machine translation,
we cannot use the translation surrounded by a red circle

We would like to use all available translations
2

Overview of this research (2/2)
We would like to use all available translationsПриветHolaHelloдоброе утроBuenos díasGood morningспасибо

GraciasThank you• We augment with pseudo-translations using multi-source
NMT• Our proposed method achieved good result
3

Multi-lingual Corpus• There are many corpora which have multiple languages• Video captions for talks or movies[Cettolo et al., 2012; Tiedemann, 2009]• Europarl [Kohen, 2005], UN [Ziemski et al., 2016]

These corpora have good, manually curated translations
in a number of languages
Complete corpusЗдравствуйтедоброе утроспасибо

Hola
Buenos días
GraciasHelloGood morning
Thank you  4

Multi-lingual corpus with missing data
It is unusual that sentences of all languages exist(such as subtitles for TED Talks)
Goal
Generating good translations in the remaining languages forwhich do not yet have translations in a multilingual corpus
Incomplete corpusдоброе утроспасибо

HolaBuenos días

HelloGood morning
Thank you
5

Neural Machine Translation (NMT)
One-to-one NMT

Multi-lingual NMT

J Better!
We use multi-lingual NMT to generate translations

But there are some types of Multi-lingual NMT6

Multi-lingual NMT• Multi-Source, One-Target[Zoph and Knight, 2016; Garmash and Monz, 2016]• One-Source, Multi-Target[Firat et al., 2016]• Multi-Source, Multi-Target[Johonson et al., 2017; He et al., 2016]

We’d like to improve NMT by the help of
the other curated translations on the source side at the test time

We focused on Multi-Source and One-Target
7

Multi-Source NMT | Multi-Encoder NMT• Multi-Encoder NMT [Zoph and Knight, 2016]• Multiple Encoder and one Decoder• Multiple sentences are each encoded separately,
then all referenced during decoding process

EnglishEncoderSpanishEncoderDecoderFrench8

The disadvantage of Multi-Source NMT
Multi-Source NMT assumes we have data in all of languages
we cannot use the translation
if some source translations are missing
EnglishHelloThank youFrenchSpanish

Bonjour
Merci

We cannot use the translation“Thank you” and “Merci”
in multi-source NMT

Hola9About our research
We would like to use all available translations
even if the corpus has missing data
Incomplete corpusдоброе утроспасибо

We can use only the translation in blue frame

Hola
Buenos días

HelloGood morning
Thank you


Our research is the first study on
how to handle incomplete corpora
10Our Previous WorkProblem
Multi-Source NMT
assumes that we do
not have any missing data

ProposedReplacing each missing
input sentence witha special symbol <NULL>[Nishimura et al., 2018]

Original

EnglishSpanishHow are youMissing

Original

Comment ça va

French<NULL>

This method achieved higher translation accuracy
11

Our Previous Work’s Problem
In case, the corpus has many missing data
The model will be trained on corpora witha large number of NULL symbols
Problem

The source condition will be much different
between train time and test time12

Proposed Method | Overview
Problem
The source condition isvery different between
train and test
EnglishOriginal

How are you
PseudoSpanishProposedUsing a pseudo-corpusthat fills missing data withmulti-source NMT outputs

Cómo está

Data Augmentation with
Trained multi-source NMT{English, French}-to-Spanish

Original

Comment ça va

FrenchOriginal

How are you

EnglishComment ça va

FrenchOriginal

13

Proposed Method | 1st stepFinal Goal : Get French Translation• Train a multi-encoder NMT model(Source: English and French, Target: Spanish)• If there is a missing input, we replacea missing input sentence with a special symbol <NULL>
How are you
Comment ça va
Original

How are you

EnglishComment ça vaFrench

Original

Train multi-source NMT{English, French}-to-Spanish

14

Proposed Method | 2nd step
• Create Spanish pseudo-translations
using multi-encoder NMT which was trained on the 1st step
• We conducted three types of augmentation

How are you
PseudoSpanishCómo está

Data Augmentation withTrained multi-source NMT{English, French}-to-Spanish

Comment ça va
Original

How are you

EnglishComment ça va

FrenchOriginal

15

Proposed Method | 3rd step
• Train a multi-encoder NMT model(Source: English and Spanish, Target: French)• Spanish translations have pseudo-translations
EnglishOriginal

How are you
PseudoSpanishOriginal

Comment ça va

FrenchCómo está
How are you
Comment ça va16

Three types of augmentation(1) : “fill-in”• Where only missing parts in the corpus are filled up withpseudo-translations

Original

See you

EnglishOriginal

How are you
Original

Hasta luego
SpanishFrench
Original
À bientôt
Original

Comment ça vaPseudo

Cómo está
17

Three types of augmentation
The reason of making three types• Translations of TED talks are unreliable• Translations are created from many independent volunteers

The effectiveness of applying back-translation for
an unreliable part of a provided corpus[Morishita et.al. 2017]

We proposed the methods not to use
unreliable original translations
18

Three types of augmentation(2) : “fill-in and replace”• Augment the missing part and replace original translations
with pseudo-translations• The motivation is not to use unreliable translation
Original

See youEnglish

Original

How are you
PseudoHasta pronto
SpanishHasta luego

FrenchOriginal
À bientôt
Original

Comment ça va

PseudoCómo está
19

Three types of augmentation(3) : “fill-in and add”
• Augment the missing part and added pseudo-translations
from original translations• The motivation : prevent noise of the complete
replacement of the 2nd method
Original

EnglishSee you

FrenchSee you

Original
À bientôt
Original
À bientôt
Original

Original
Original

How are you
Original

SpanishPseudoHasta pronto

PseudoCómo está

Hasta luego

Comment ça va

Hasta luego
20Experiment | Data• Corpus• A collection of transcriptions of TED Talks• Language Pair
• English (en), Croatian (hr), Serbian (sr)• English (en), Slovak (sk), Czech (cs)• English (en), Vietnamese (vi), Indonesian (id)Pair
en-hr/sren-sk/cs
en-vi/idTrg
hrsrsk
csvi
id

train
11894913355810060059918
16098482592

missing
35564 (29.9%)50203 (37.6%)
58602 (57.7%)17380 (29.0%)87816 (54.5%)
9424 (11.4%)• train• the number of available
training sentences• missing• the number and thefraction of missing
sentences in comparison
with English ones
21

Experiment | Data• Corpus• A collection of transcriptions of TED Talks• Language Pair
• English (en), Croatian (hr), Serbian (sr)• English (en), Slovak (sk), Czech (cs)• English (en), Vietnamese (vi), Indonesian (id)Pair
en-hr/sr

Trg
hr

train
118949missing35564 (29.9%)• train• the number of available
training sentences• missing• the number and thefraction of missing
sentences in comparison
with English ones
22

Experiment | Baseline Methods
One-to-oneNMTMulti-encoder NMT
with back-translation

Multi-encoder
NMT with <NULL>

Standard NMT
model from one
source language
to anothertarget language

A multi encoder NMT
system using pseudotranslation from Englishto-X NMT

A multi-encoder
NMT system using a
special symbol<NULL>

Original
Original

Original[Luong et al., 2015]

Original

PseudoData Augmentation
with Trained
one-to-one NMT
English-to-X

Original<NULL>

Original

Original[Nishimura et al., 2018]23

Baseline | One-to-one NMT
One-to-one
NMTStandard NMT
model from one
source language
to anothertarget language
Original
Original

OriginalOriginal

Original[Luong et al., 2015]24

Baseline | Multi-encoder NMT
with back-translation
Multi-encoder NMT
with back-translation
A multi encoder NMT
system using pseudotranslation from Englishto-X NMT
Original
Original

Original

Original

PseudoData Augmentation
with Trained
one-to-one NMT
English-to-X

Original

Original

Original

25Baseline | Multi-encoder NMT with <NULL>
Multi-encoder
NMT with <NULL>
A multi-encoder
NMT system using a
special symbol<NULL>
Original
Original

Original<NULL>

Original

Original[Nishimura et al., 2018]26

ResultResult in BLEU
baseline method
Pairen-hr/sr
en-sk/cs
en-vi/id

proposed method

Trg one-to-one multi-encoder NMT multi-encoder NMT
fill-in and fill-in
fill-in(En-to-Trg) (fill up with symbol) (back-translation)
replace and add
hrsrsk
csvi
id

20.21
16.42
13.79
14.72
24.60
24.89

28.18
23.85
20.27
19.88
25.70
26.89

27.57
22.73
19.83
19.54
26.66
26.34

29.17
24.41
20.26
20.78
26.73
26.40

29.37
24.96
20.43
20.90
26.48
25.73

29.40
24.15
20.59
20.61
26.32
26.21

27

Result | baseline vs proposed
Result in BLEU
baseline methodPair
en-hr/sren-sk/cs

proposed method

Trg one-to-one multi-encoder NMT multi-encoder NMT
fill-in and fill-in
fill-in(En-to-Trg) (fill up with symbol) (back-translation)
replace and add
hr
sr
sk
cs

20.21
16.42
13.79
14.72

28.18
23.85
20.27
19.88

27.57
22.73
19.83
19.54

29.17
24.41
20.26
20.78

29.37
24.96
20.43
20.90

29.40
24.15
20.59
20.61

• en-hr/sr, en-sk/cs• Proposed methods > Baseline Method• proposed method is an effective way to use incomplete
multilingual corpora
28Result | baseline vs proposed
Result in BLEU
baseline method
Pairen-vi/id

proposed method

Trg one-to-one multi-encoder NMT multi-encoder NMT
fill-in and fill-in
fill-in(En-to-Trg) (fill up with symbol) (back-translation)
replace and add

vi
id

24.60
24.89

25.70
26.89

26.66
26.34

26.73
26.40

26.48
25.73

26.32
26.21

• en-vi/id• Baseline Method > Proposed Method• The improvement by the use of multi-encoder NMT against
one-to-one NMT in the baseline was small
29

Result | Three types of augmentation
Result in BLEUproposed methodPair
en-hr/sren-sk/cs
en-vi/idTrg

fill-inhrsrsk
csvi
id

29.17
24.41
20.26
20.78
26.73
26.40

fill-in and fill-in
replace and add
29.37
24.96
20.43
20.90
26.48
25.73

29.40
24.15
20.59
20.61
26.32
26.21

• There were almost no differencesamong three types of augmentation
Detail analysis

We created three types of augmentation with
one-to-one NMT output
30

Analysis | Three types of augmentation
Our expectation
The aggressive use ( “fill-in and replace” and ”fill-in and add” )of low quality pseudo-translations
Contaminate the training data and todecrease the translation accuracy

We created three types of augmentation
with one-to-one NMT output
31

Analysis | Three types of augmentation
Result in BLEU (Augment with one-to-one NMT)
Pair

en-hr/sren-sk/cs
en-vi/id

Trg
hr
srsk
cs
vi
id

Multi-encoder NMT(back-translation)
fill-in andfill-infill-in
replaceand add
27.57
24.05
24.79
22.73
17.77
22.02
19.83
16.75
18.16
19.54
17.04
18.40
26.66
26.39
26.65
26.34
23.90
26.67

large difference
large differencefew difference• en-vi/id : there are few differences in three types of augmentation• one-to-one NMT was better than other language pairs
32Analysis | Three types of augmentation
Result in BLEU
(Augment with one-to-one NMT)
Pair

Trg

Multi-encoder NMT(back-translation)
fill-in andfill-in
fill-in
replaceand add

Train Data statistics
Pair
en-hr/sren-sk/cs
en-vi/id

en-vi/id

id

26.34

23.90

Trg
hrsrskcsvi
id

missing
35564 (29.9%)50203 (37.6%)
58602 (57.7%)17380 (29.0%)87816 (54.5%)
9424 (11.4%)

26.67• Target=Indonesian : ”fill-in and add” got highest BLEU
• much smaller fraction of the missing parts in Indonesian corpus
33

Analysis | Iterative Augmentation• Update the multi-source NMT systems into the twotarget languages iteratively
EnglishSpanishOriginal

How are you
PseudoComment ça va

FrenchCómo estámake French
pseudo translation

make Spanish
pseudo translation

PseudoOriginal

SpanishOriginal

Hola

Bonjour

FrenchOriginal English

Hello34

Analysis | Iterative Augmentation
Result in BLEU (and BLEU gains compared to step1)
Pair
en-hr/sr

Trg
hrsr

step1
29.17 (+0.00)
24.41 (+0.00)

step2
29.03 (-0.14)
24.18 (-0.23)

step3
29.10 (-0.07)
24.17 (-0.24)

step4
29.95 (-0.12)
23.95 (-0.46)• BLEU decreased gradually in every step
• We observed very similar results in the other language pairs

The iterative training may be introducing more noise

35

Analysis | Non-Parallelism
Example of the Serbian pseudo-translation
TypeSentence

Original (En)So let me conclude with just a remark to bring it back to the theme of choices.

Original (Sr)Da zaključim jednom konstatacijom kojom se vraćam na temu izbora.

Pseudo (Sr)Dozvolite mi da zaključim samo jednom opaskom, da se vratim na temu izbora.

• The Serbian original translation does not have a phrase
corresponding to “let me”• The Serbian pseudo translation have a phrase corresponding
to “let me”“fill-in and replace” or “fill-in and add” can be used tocompensate for the missing information
36

Conclusion and future workConclusion• Our research is the first study on how to handle
incomplete corpora in multi-source NMT• We proposed three types of augmentation• Our proposed methods proved better than baseline
systems, though results depend on the language pair
Future Work
• A set of three languages is that missing parts in thetest sets could not be filled in, we will conduct
experiments using more languages
37