1

Multi-Source Neural Machine Translation
with Missing Data
Yuta Nishimura, Nonmember, Katsuhito Sudoh, Nonmember, Graham Neubig, Nonmember and Satoshi
Nakamura, Fellow, IEEE

Abstract—Machine translation is rife with ambiguities inword ordering and word choice, and even with the advent ofmachine-learning methods that learn to resolve this ambiguity
based on statistics from large corpora, mistakes are frequent.
Multi-source translation is an approach that attempts to resolvethese ambiguities by exploiting multiple inputs (e.g. sentences inthree different languages) to increase translation accuracy. Thesemethods are trained on multilingual corpora, which include
the multiple source languages and the target language, andthen at test time uses information from both source languages
while generating the target. While there are many of these
multilingual corpora, such as multilingual translations of TED
talks or European parliament proceedings, in practice, many
multilingual corpora are not complete due to the difficulty toprovide translations in all of the relevant languages. Existingstudies on multi-source translation did not explicitly handle suchsituations, and thus are only applicable to complete corporathat have all of the languages of interest, severely limiting theirpractical applicability. In this paper, we examine approaches
for multi-source neural machine translation (NMT) that canlearn from and translate such incomplete corpora. Specifically,
we propose methods to deal with incomplete corpora at bothtraining time and test time. For training time, we examine
two methods: (1) a simple method that simply replaces missing
source translations with a special NULL symbol, and (2) a data
augmentation approach that fills in incomplete parts with source
translations created from multi-source NMT. For test-time, we
examine methods that use multi-source translation even when
only a single source is provided by first translating into an
additional auxiliary language using standard NMT, then using
multi-source translation on the original source and this generated
auxiliary language sentence. Extensive experiments demonstratethat the proposed training-time and test-time methods bothsignificantly improve translation performance.
Index Terms—Neural machine translation, multilinguality,
data augmentation.

I. I NTRODUCTION
NE of the major challenges in Machine Translation (MT)systems is the inherent ambiguity in translating acrosslanguages, and recent methods perform machine learningover large corpora to learn models to resolve this ambiguity.
However, in many real situations, it is difficult to create large
corpora, for a particular language of interest, and thus translation accuracy may suffer. One way to deal with this paucity
of data is to use data from multiple languages to improve thetranslation accuracy [1], [2], [3], [4]. There are a significant

O

Y. Nishimura, K. Sudoh and S. Nakamura are with Nara Institute of Scienceand Technology, 8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan e-mail:(nishimura.yuta.nn9@is.naist.jp).
G. Neubig is with Carnegie Mellon University, 5000 Forbes Avenue,
Pittsburgh, PA 15213, USA.

number of multilingual document collections that can be used
for this purpose, such as the proceedings of the European
parliament [5] or the United Nations [6]. Documents in these
corpora are manually translated into all official languages of
the respective organizations. There are also voluntary translation effort multilingual captions such as those for talks[7] and movies [8]. In addition, OPUS (http://opus.nlpl.eu)
provides many parallel corpora, and a large number of them
contain multi-way translations: EMEA, software localizations(GNOME, PHP, etc.), the Bible, OpenSubtitles, and so on.
However, in the case of such voluntary translation efforts,
large portions of the corpus are not translated, especially into
languages with a relatively small number of speakers.
In this paper, we focus on this sort of multilingual scenario, specifically using multi-source translation [9], [10], [11].
Multi-source translation is an approach to exploit multiple
inputs (e.g. in two different languages) to decrease ambiguity
and increase translation accuracy. Specifically, in the context ofneural machine translation (NMT), there are several methods
proposed to do so. For example, Zoph and Knight [10]proposed a method where multiple sentences are encoded
and then passed into a single decoding process (the “multiencoder” method). In addition, Garmash and Monz [11] proposed a method where NMT systems over multiple inputs are
ensembled together to make a final prediction (the “mixtureof-NMT-experts” method).
These paradigms generally assume, and have been testedon the case where we have data in all of the source languages. However, it is unusual that translations in all of theselanguages are provided – other than parliament proceedings
where it is legally mandated that translations in all languages
be generated, most multi-lingual corpora have gaps wheretranslators have not been able to generate translations. This paper focuses on multi-source NMT with missing data, proposing
methods to leverage multiple languages to improve accuracy,
while still maintaining the ability to train and test on corporathat are not complete.
First, we conducted a simple implementation of multisource NMT using such an incomplete multilingual corpus byusing a special symbol (arbitrarily, NULL ) to represent
the missing sentences. This makes it possible to both train andtest using existing multi-source NMT implementations without
any modifications by simply appending this symbol in placeof missing sentences. At test time, we can expect the system
to ignore the NULL symbol, with the decoder choosing
hypotheses using other input sentences. Experimental resultswith a real incomplete multilingual corpus of TED Talks show

2

EnglishEncoder

FrenchEncoder

ArabicEncoder

Gating Network

Decoder

SpanishFig. 1: Multi-encoder NMT

EnglishEncoder

Decoder

FrenchEncoder

Decoder

ArabicEncoder

Decoder

SpanishPre-trainthat the proposed method is effective for allowing training
for multi-source NMT in situations where full multilingual
corpora are not available.
However, this first method essentially throws away the
information provided by the utilization of multiple inputs whensuch inputs are not available. To alleviate this problem, we
propose a second method that performs data augmentation attraining time, creating a pseudo-parallel corpus that fills in
the missing sentences with an output of a multi-encoder NMT
system. Experimental results show that the proposed method
is more effective than simply filling up the sentences with
NULL .
Finally, we turn our eyes to methods that can utilize multisource translation at test time. The NULL augmentation
method can technically be applied at test time but essentiallyreduces to single-source translation, and the pseudo-parallel
corpus augmentation method cannot be used at test time
because we do not know the gold reference translation. As analternative, we propose a two-step method for multi-encoder
NMT decoding motivated by pivot machine translation. Pivot
machine translation uses an pivot language that has a large
amount of data to get better translation accuracy on a language
pair which has a small amount of data [12]. In other words,
pivot machine translation gets better translation accuracy usinga pivot language that is easier to machine-translate than thelanguage pair that we actually want to translate. In multiencoder NMT, we can consider the language in which wehave missing translations as a pivot language, and create
automatic translations. Specifically, our method creates multiple hypotheses in the missing language using normal one-toone NMT, and chooses the appropriate hypothesis for multiencoder NMT. We confirm the effectiveness of this proposed
method in experiments over a test set which is missing one of
the two sources.1
II. R ELATED W ORK
A. Multi-Source NMT
At the present, there are two major approaches to multisource NMT: multi-encoder NMT [10] and mixture of NMT
experts [11]. We first review them in this section.
1) Multi-Encoder NMT: Multi-encoder NMT [10] is similar
to the standard attentional NMT framework [15] but uses
multiple encoders corresponding to the source languages anda single decoder, as shown in Figure 1.
1 This journal article is based on content previously presented as twoworkshop proceedings [13], [14]. We have greatly expanded the explanation,and this article adds entirely new content related to test-time usage of multisource methods, corresponding to the Sections IV and VI.

Fig. 2: Mixture of NMT Experts
Suppose we have two LSTM-based encoders and theirhidden states and cell states at the end of the inputs are h1 ,h2 and c1 , c2 , respectively. The multi-encoder NMT method
initializes its decoder hidden states h and cell state c asfollows:
h = tanh(Wc [h1 ; h2 ])(1)
c = c1 + c2

(2)

Attention is then defined over each encoder at each time
step t and resulting context vectors c1t and c2t are concatenated
together with the corresponding decoder hidden state ht tocalculate the final context vector h˜t .
h˜t = tanh(Wc [ht ; c1t ; c2t ])(3)

Our implementation is based on Zoph and Knight [10],but we use global attention while they used local-p attention.
local-p attention focuses only on a small subset of the source
positions for each target word [16], while global attention
attends to all words on the source side for each target word.
We use global attention as it is the standard method used in
the great majority of recent NMT work.
2) Mixture of NMT Experts: Garmash and Monz [11] proposed another approach to multi-source NMT called mixture of
NMT experts. This method ensembles together independentlytrained encoder-decoder networks. Each NMT model is trainedusing a bilingual corpus with one source language and thetarget language, and the outputs from the one-to-one models
are summed together, weighted according to a gating network
to control contributions of these independent models, as shownin Figure 2.
The mixture of NMT experts determines an output symbol
at each time step t from the final output vector pt , which is
the weighted sum of the probability vectors from one-to-one
models denoted as follows:
pt =

m
X
gtj pjt(4)

j=1

where pjt and gtj are the probability vector from the j-th modeland the corresponding weight at time step t, respectively. mis the number of one-to-one models. gt is calculated by thegating network as follows:
gt = softmax(Wgate tanh(Whid [ft1 (xt1 ); ...ftm (xtm )])) (5)3where ftj (x) is the input vector to the decoder of the j-thmodel. In this work, xjt is the input vector at the time step t,which concatenates the embedding vector at the time step t
and the context vector of the j-th model at the previous time
step t-1.
B. Data Augmentation
Sennrich et al. proposed a method to use monolingual
training data in the target language for training NMT systems,with no changes to the network architecture [17]. This method
first trains a seed target-to-source NMT model using a parallel
corpus and then translates the monolingual target language
sentences into the source language to create a synthetic parallel
corpus. It finally trains a source-to-target NMT model using
the seed and synthetic parallel corpora. This simple method
called back-translation makes effective use of available resources, and achieves substantial gains in accuracy. Imamura etal. proposed a method that generates multiple source sentences
via sampling as an extension of the back-translation [18].
Firat et al. proposed a data augmentation framework for zeroshot translation that uses pseudo source language sentencestranslated from a pivot language [19].
There are also other approaches for data augmentation
other than back-translation. Wang et al. proposed a methodof randomly replacing words in both the source sentence
and the target sentence with other random words from theircorresponding vocabularies [20]. Kim and Rush proposeda sequence-level knowledge distillation in NMT that uses
machine translation results by a large teacher model to train asmall student model as well as ground-truth translations [21].
Our work is similar to the back-translation approach, but
specifically tailored to multilingual and multi-source situations.
III. T RAINING - TIME DATA AUGMENTATION FORM ULTI - SOURCE T RANSLATIONAs mentioned in the introduction, multi-source NMT assumes that we have data in all of the languages that go
into our multi-source system. For example, if we decide that
English and Spanish are our input languages and that wewould like to translate into French, we are limited to training
and testing only on data that contains English, Spanish, andFrench. However, it is unusual that translations in all of theselanguages are provided, particularly for all of the training datathat we have available; there will usually be many sentenceswhere we have only one of the sources. In this section, we
propose two methods for dealing with this method at training
time: NULL augmentation by just replacing missing sentenceswith a special symbol and pseudo-translation augmentation
by filling machine translation outputs there.
A. NULL Augmentation
The first proposed method is very simple; we just replaceeach missing input sentence with a special symbol. We referto this special symbol as NULL , although the choice of
this character string is arbitrary. By training the model using

EnglishHow are you?

FrenchComment ça va?

Arabic__NULL__¿Cómo está?

SpanishFig. 3: Multi-encoder NMT with a missing input sentence. In
this example, the Arabic input sentence is missing.
this method, we expect it to learn to ignore the NULL
symbol in multi-encoder NMT when it appears, and translateusing the other translations. This method can be applied easily
to any existing implementation of multi-encoder NMT without
modification. Figure 3 provides a concrete example of thismethod. Here we suppose the source languages are Spanish,French, and Arabic and the target language is English, and that
the Arabic input sentence is missing. In this case, the Spanish
and French input sentences are passed into the corresponding
encoders, and a special symbol NULL is passed to the
Arabic encoder. In the experiments described later, we applied
this method for both multi-encoder NMT and mixture of NMT
experts and compared them.
B. Pseudo-translation Augmentation
1) Motivation and overall framework: While the first proposed method is simple, if the model is trained on corpora
with a large number of NULL symbols on the sourceside, a large number of training examples will be different
from test time when we actually have multiple sources. Thus,
these examples will presumably be less useful in training amodel intended to do multi-source translation. We thus additionally propose an improved method for utilizing multi-source
examples with missing data: using a pseudo-corpus whose
missing translations are augmented with machine translation
outputs using a trained multi-source NMT system as shown in
Figure 3.
Here we use three languages to explain our proposed
method; English, Spanish and French. Our goal is to generate a
Spanish translation, and we suppose there are not any missing
data on the English side, but Spanish and French translations
have some missing data.
2) Step-wise Augmentation Procedure: Our augmentation
procedure consists of the following three steps. The procedure
is illustrated in Fig. 4
1) Train a multi-encoder NMT model (Source: English andSpanish, Target: French) to get French automatic translations using the baseline method, that replaces missing
input sentences with a special symbol NULL .
2) Create French automatic translations using multi-encoder
NMT, which was trained on the data generated in the firststep. We conduct three types of augmentation in this step,
the details of which we introduce later.
3) Switch the role of French and Spanish, in other words, wetrain a new multi-encoder NMT model (Source: English
and French, Target: Spanish). At this time, we use French
automatic translations on the source language side.

4

Original
EnglishHow are you?
PseudoFrenchComment ça va?

3: Train a {English,French}-toSpanish NMT model using an
augmented corpus.
Original¿Cómo está?

EnglishOriginal

Hello

FrenchSpanishHola

Original
Bonjour
Original

Original

How are you?
2: Generate pseudotranslations by{English,Spanish}-toFrench NMT.

SpanishOriginal

EnglishEnglishSpanishFrench

1: Train a {English,Spanish}-to-French
NMT model. Missing French sentences
are replaced with a symbol __NULL__.

SpanishComment ça va?

(a) fill-in
Original
EnglishFig. 4: Example of multi-encoder NMT with data augmentation using an incomplete corpus. The language pair is English,
French-to-Spanish and the translation of French is missing.

¿Cómo está?

PseudoOriginal¿Cómo está?

Original

How are you?

Original

Hello

FrenchSpanishHola

PseudoSalut
OriginalThis method is similar to back-translation but takes advantageof the fact that we have an additional source of knowledge(Spanish or French) when trying to augment the other language(French or Spanish respectively).
3) Filling strategies: We propose three specific types ofdata augmentation for multi-encoder NMT; “fill-in”, “fillin and replace” and “fill-in and add.” Figure 5 illustrates
examples in the {English, French}-to-{Spanish} case where
one Spanish sentence is missing.
(a) fill-in: where only missing parts in the corpus are filled
up with pseudo-translations.
(b) fill-in and replace: where we both augment the missing
parts and replace original translations with automatictranslations in the source languages other than English(as English has no missing data). This method has thepotential to override wrongly-aligned or non-literal translations existing in the original data similarly to workby Morishita et al. [22], which demonstrated the effectiveness of applying back-translation for an unreliable
part of a provided corpus. Translations of TED talks
are from many independent volunteers, so there may besome differences between translations other than original
English, or they may even include some free or oversimplified translations. We aim to fill such a gap using
data augmentation.
(c) fill-in and add: where we both augment the missing
sentences and add automatic translations to sentences that
already have original translations in the source language
other than English. This helps prevent introduction of too
much noise due to the complete replacement of originaltranslations with automatic translations in the second
method.
IV. T EST- TIME DATA AUGMENTATION FORM ULTI - SOURCE T RANSLATION
Our previous two proposed methods provide a better way toutilize missing multilingual corpora at training time, but theycannot be used trivially if there are missing source sentencesat test time. This is because the first proposed method can be

EnglishPseudoFrenchOriginal

How are you?

¿Cómo está?

SpanishComment ça va?

(b) fill-in and replace
Original
EnglishOriginal

Hello

FrenchBonjourEnglishHelloSpanishHola

Original

Original
Original

FrenchSpanishHola

PseudoSalut
Original
EnglishPseudoFrenchOriginal

How are you?

¿Cómo está?

SpanishComment ça va?

(c) fill-in and add

Fig. 5: Example of three types of augmentation. The language
Pair is {English, French}-to-{Spanish} and French translation
corresponding to “How are you?” is missing. In this example,
the white background indicates the pseudo-translation produced from multi-source NMT and the colored backgroundmeans the original translation.

applied at testing time but simply ignores the missing sentence,
reverting back to one-to-one translation (or worse), and thesecond proposed method uses translations of a target language
when filling up the missing source translations, and these target
translations are not available at test time. Thus we propose thethird method to deal with an incomplete corpus at test time,
specifically a multi-target machine translation method, which
generates multiple languages at the same time to increasetranslation accuracy [23]. Our specific approach is motivated5

Original !#%()%Pseudo !"#$$
Pseudo !"#$'
Pseudo !"#$!&%

% , !% )+(!"#$&!&$$ , ! $)+(!"#$&!&''+(!"#$, !&')(*

Fig. 6: Example of considering n-best source translations

by work on pivot machine translation, which uses a pivot language that has a large amount of data to get better translation
accuracy on language pairs that have a small amount of data
[12]. In other words, pivot translation gets better translation
accuracy using a pivot language where machine translation iseasier than the language pair in which we actually want to domachine translation. In multi-encoder NMT, we can consider alanguage which has a missing translation as a pivot language,
and create an automatic translation for this language. We
propose the method to create multiple source translations of a
missing language using normal one-to-one NMT, and choose
an appropriate one from multiple translation hypotheses for
multi-encoder NMT, as shown in Fig 6. Below, we describe
the proposed method in detail.
We assume we have a multi-encoder NMT system fm
whose source language sentences are xs1 and xs2 , and atarget language sentence is xt . xs2 is a missing sentence.
We additionally assume a trained one-to-one NMT system fo
whose source language is s1 and target language is s2 . Thenwe output n-best translations X̃s2 = {x̃1s2 , ..., x̃ns2 } using thetrained one-to-one NMT with beam search to fill in the missing
sentence xs2 . The sentence probabilities of these n-best translations are Ps = {p(x̃1s2 |fo , xs1 ), ..., p(x̃ns2 |fo , xs2 )}. Then we
output multi-encoder NMT translations Xt = {x1t , ..., xnt }using n-best translations into the additional source sentence.
The sentence probabilities using n-best source translations are
Pt = {p(x1t |fm , xs1 , x̃1s2 ), ..., p(xnt |fm , xs1 , x̃ns2 )}. Finally, wechoose the most appropriate translation xt defined by thefollowing equations.
a(x̃ns2 , xnt ) = λ ln p(x̃ns2 |fo , xs1 )+(1−λ) ln p(xnt |fm , xs1 , x̃ns2 )(6)x˜s2 , xt = arg max a(x̃ns2 , xnt )(7)nxn
s2 ,xt ∈Xs2 ,Xt

λ in the equation 6 is a hyperparameter that we use to consider
which probability between the source translation and the multiencoder translation is more important.
V. T RAINING - TIME DATA AUGMENTATION E XPERIMENTS
A. Experiment Outline
In this section, we examine the efficacy of our methods
for training-time data augmentation, including both “NULL
augmentation” (Section III-A) and “pseudo-corpus augmentation” (Section III-B). We first validate our general approach ofutilizing missing data in multi-source NMT training, focusing

on the comparison between NULL augmentation and otherbaseline methods. Then we move on to compare pseudocorpus augmentation with NULL augmentation and othernatural baselines focusing on data augmentation.
B. NMT settings
NMT2 settings are the same for all the methods in theexperiments. We use bidirectional LSTM encoders [15] andglobal attention with input feeding for the NMT model [16].
The number of dimensions is set to 512 for the hidden andembedding layers. Subword segmentation was applied using
SentencePiece [24]. We trained one subword segmentation
model for the training corpus that concatenated all of thelanguages in a language set. For parameter optimization, weused Adam [25] with gradient clipping of 5. The number ofhidden state units in the gating network for the mixture of
NMT experts experiments was 256. We used BLEU [26] asthe evaluation metric and SacreBLEU3 [27] as the evaluation
tool. We performed early stopping, saving parameter values
that had the best log likelihoods on the validation data and usedthem when decoding the test data. However, in the experimentswith mixture of NMT experts, we found this likely to resultin over-fitting, so we train our models for only 1 epoch andsaved parameter values.
C. Experiment: NULL Augmentation on a Pseudo-incomplete
Multilingual Corpus (UN6WAY)
First, we conducted experiments using a complete multilingual corpus and a pseudo-incomplete corpus derived by
excluding some sentences from the complete corpus. The
purpose of creating a pseudo-incomplete corpus is to extensively compare the performance in complete and incomplete
situations.
1) Data: We used UN6WAY [6] as the complete multilingual corpus. We chose Spanish (Es), French (Fr), andArabic (Ar) as source languages and English (En) as a target
language. The training data in the experiments were 800,000
sentences from the UN6WAY corpus whose sentence lengths
were less than 40 words. We excluded 200,000 sentences
for each language except English for the pseudo-incomplete
multilingual corpus as shown in Table I. “Sentence No.” inTable I represents the line number in the corpus, and thex means the part removed for the incomplete multilingual
corpus. We also chose 1,000 and 4,000 sentences for validationand test from the UN6WAY corpus, apart from the training
data. Note that the validation and test data here had no missing
translations.
2) Setup: We compared multi-encoder NMT and the mixture of NMT experts in the complete and incomplete situations.
The three one-to-one NMT systems, Es-En, Fr-En, and ArEn, which were used as sub-models in the mixture of NMT
experts, were also compared for reference.
First, we conducted experiments using all of the 800,000
sentences in the complete multilingual corpus, Complete
2 We used pytorch as a neural network toolkit. https://github.com/pytorch/
pytorch.git
3 https://github.com/awslabs/sockeye/tree/master/sockeye contrib/sacrebleu

6

TABLE I: Settings of the pseudo-incomplete UN multilingual
corpus (x means that this part was deleted)
Sentence No.
1-200,000
200,001-400,000
400,001-600,000
600,001-800,000

Es
x

Fr

Ar

En

x
x

(0.8M). In case of the mixture of NMT experts, the gating
network was trained using the 800,000 sentences.
Then, we tested in the incomplete data situation. Here there
were just 200,000 complete multilingual sentences (sentence
No. 600,001-800,000), Complete (0.2M). Here, a standard
multi-encoder NMT and mixture of NMT experts could betrained using this complete data. On the other hand, the
NULL
could be trained usmulti-source NMT withing 800,000 sentences (sentence No. 1-800,000), Pseudoincomplete (0.8M). Each one-to-one NMT system could betrained using these 800,000 sentences, but the missing sentences replaced with the NULL tokens were excluded soresulting 600,000 sentences were actually used.
3) Results: From results in Table II, we can see that themulti-source approaches achieved consistent improvements
over the one-to-one NMT models in the all conditions. This is
in concert with previous multi-source NMT studies. Our main
focus here is Pseudo-incomplete (0.8M), in which the multisource results were slightly worse than those in Complete(0.8M) but better than those in Complete (0.2M). This suggests
the additional use of incomplete corpora is beneficial in multisource NMT compared to the use of only the complete partsof the corpus, even if just through the simple modification of
replacing missing sentences with NULL .
With respect to the difference between multi-encoder NMT
and mixture of NMT experts, multi-encoder NMT achievedmuch higher BLEU in all conditions. One possible reason hereis the model complexity; the multi-encoder NMT model uses alarge single model while one-to-one sub-models in the mixtureof NMT experts can be trained independently.
D. Experiment: NULL Augmentation on a Real Incomplete
Corpus (TED Talks)
In addition to the experiments with a pseudo-incomplete
multilingual corpus, we also checked our proposed methodwith an actual incomplete multilingual corpus.
1) Data: We used a collection of transcriptions of TED
Talks and their multilingual translations. Because these translations are created by volunteers, and the number of translations
for each language is dependent on the number of volunteers
who created them, this collection is an actual incomplete
multilingual corpus. The great majority of the talks are originally in English, so we chose English as a source language.
We used three translations in other languages for our multisource scenario: Spanish, French and Brazilian Portuguese. We
prepared three tasks choosing one of these three languages as
the target language and the others as the additional source
languages. Table III shows the number of available sentencesin these tasks, chosen so that their lengths are less than

40 words, and Table IV shows the percentage of available
sentences compared to the number of English sentences inthese tasks.
2) Setup: We compared multi-encoder NMT, mixture of
NMT experts and one-to-one NMT with English as the source
language. The validation and test data for these experiments
were also incomplete. This is in contrast to the experiments on
UN6WAY where the test and validation data were complete,
and thus this setting is arguably of more practical use.
3) Results: Table V shows the results in BLEU and BLEU
gains with respect to the one-to-one results. The multi-source
NMT systems achieved consistent improvements over the oneto-one baseline as expected, but the BLEU gains were smaller
than those in the previous experiments using the UN6WAY
data. This is possibly because the baseline performance wasrelatively low compared with the previous experiments and thesize of available resources was also smaller.
We analyzed the results using the TED data in detailto investigate the mixed results above. Figure 7 shows thebreakdown of BLEU in the test data, separating the results forcomplete and incomplete multilingual inputs. When all source
sentences are present in the test data, multi-encoder NMT has
convincingly better performance than mixture of NMT experts.
However, when the input is incomplete, mixture of NMT
experts achieves performance better except on {En,Es,Pt(br)}to-Fr. From this result, we can assume that mixture of NMT
experts, with its explicit gating network, is better at ignoring
the irrelevant missing sentences. This indicates the need for abetter test-time strategy than NULL augmentation such as theone that we propose in Section IV and evaluate in Section VI.
4) Translation examples: Table VI shows a couple oftranslation examples in the {English, Spanish, French}-toBrazilian Portuguese experiment.
In Example (1), there is only the English sentence in thesource sentences. We can see that sentences which are generated from all models are the same as the reference sentences,even though French and Brazilian Portuguese sentences are
missing. Therefore multi-source NMT models work properly
even if there are missing sentences.
In Example (2), BLEU+1 of mixture of NMT experts
and multi-encoder NMT are larger than one-to-one (Englishto-Brazilian Portuguese) because of the Spanish sentence,even though the source sentence of French is missing. In
mixture of NMT experts and one-to-one NMT output “Bem,”means “Well,” in English, but this was not output when using
multi-encoder NMT. Furthermore, the Spanish source sentence
doesn’t have a word which means “Well,”. It is assumed thatthe multi-encoder NMT could use information of the otherlanguages more effectively than mixture of NMT experts.
E. Experiment: Augmentation with Pseudo-translations
Finally, we examine the efficacy of our Pseudo-translation
Augmentation method.
1) Data: Similarly to above, we used a collection oftranscriptions of TED Talks and their multilingual translations.
The numbers of these voluntary translations differs significantly by language. We chose three different language sets for7

TABLE II: Results in BLEU for one-to-one and multi-source ({Es, Fr, Ar}-to-En) translation on UN6WAY data (parentheses
are BLEU gains against the best one-to-one results).
Condition
Complete (0.8M)
Complete (0.2M)
Pseudo-incomplete (0.8M)

Es-En
51.11
48.92
51.02

One-to-one
Fr-En
Ar-En
47.18
43.29
44.82
39.46
46.63
43.43

Multi-encoder

Mix. NMT Experts

59.45 (+8.34)
55.87 (+6.95)
57.94 (+6.92)

53.16 (+2.05)
50.76 (+1.84)
52.09 (+1.07)

40

TABLE III: Data statistics in the tasks on TED data (in thenumber of sentences). Note that the number of target sentences
is equal to that of English for each task.
Training

Valid.

Test

179,784
162,764
158,807

3,880
3,547
3,451

5,200
4,484
4,459

176,473
162,764
157,384

3,972
3,547
3,490

4,546
4,484
4,113

169,791
158,807
157,384

3,689
3,451
3,490

4,544
4,459
4,113

36.57

30

BLEU

Source{En, Fr, Pt(br)}-to-Es
EnglishFrenchBrazilian Portuguese{En, Es, Pt(br)}-to-Fr
EnglishSpanishBrazilian Portuguese{En, Es, Fr}-to-Pt(br)EnglishSpanishFrench

38.62

31.49

32.28

20

10

0

Complete (4091)

Incomplete (1109)

The type of test data
Multi-encoder NMT

Mixture of NMT experts(a) TED: {En,Fr,Pt(br)}-to-Es
40
36.87

TABLE IV: The percentage of sentences without missing
sentences on TED data.
Training
83.83
85.40
88.77

Valid.
85.08
83.11
89.48

Test
78.67
89.99
90.03

29.22
BLEU

TargetSpanishFrenchBrazilian Portuguese

Multi-encoder
37.21 (+3.28)
36.19 (+2.72)
38.79 (+3.48)

Complete (4091)

Incomplete (455)

The type of test data
Multi-encoder NMT

Mixture of NMT experts(b) TED: {En,Es,Pt(br)}-to-Fr

Mix. NMT
Experts
35.62 (+1.69)
35.07 (+1.60)
36.87 (+1.56)

the experiments: {English (en), Croatian (hr), Serbian (sr)},{English (en), Slovak (sk), Czech (cs)}, and {English (en),Vietnamese (vi), Indonesian (id)}. The data in the experiments
was chosen from the TED Talks corpus, and we limited tosentences for which lengths were less than 40 words. The
experiments were designed for the translation from English
to another language with the help of the other language inthe language set, because the great majority of TED talks
are in English, there are not any missing portions in the
English sentences. Table VII shows the number of training
sentences for each language set. At test time, we conducted
the experiment with a complete corpus where both source
sentences are represented, as this method cannot easily handle
sentences where one of the sources is missing at test time.

20

0

40
39.20

36.99

34.72

30

BLEU

{En, Fr, Pt(br)}-to-Es{En, Es, Pt(br)}-to-Fr{En, Es, Fr}-to-Pt(br)One-to-one(En-to-Trg)
33.93
33.47
35.31

28.04

10

TABLE V: Results in BLEU (and BLEU gains) by one-toone and multi-source NMT on TED data. Note that the target
language in each row differs so the results in different rows
cannot be compared directly.
Task

35.73

30

35.83

20

10

0

Complete (4091)

Incomplete (453)

The type of test data
Multi-encoder NMT

Mixture of NMT experts(c) TED: {En,Es,Fr}-to-Pt(br)Fig. 7: Detailed comparison of BLEU on the TED test data.
Complete indicates the part of test data in which there is no
missing translation, and Incomplete indicates the part in which
there are some missing translation. The number in parenthesesis the number of translations.

8

TABLE VI: Translation examples in {English, Spanish, French}-to-Brazilian Portuguese translation.
Type
Example (1)Source (En)
Source (Es)
Source (Fr)Reference
En-to-Pt(br)Multi-encoder
Mix. NMT experts
Example (2)Source (En)
Source (Es)
Source (Fr)Reference
En-to-Pt(br)Multi-encoder
Mix. NMT experts

Sentence
Here’s one of my favorite pictures of Mars.
NULL
NULL
Eis uma de minhas fotos favoritas de Marte.
Aqui est uma das minhas fotos favoritas de Marte.
Aqui est uma das minhas fotos favoritas de Marte.
Aqui est uma das minhas fotos favoritas de Marte.

0.54
0.54
0.54

Well, it means everything’s fine until this happens.
Significa que todo est bien hasta que sucede esto.
NULL
Significa que tudo vai bem at que isso acontea.
Bem, significa tudo isso tudo bem at isso acontecer.
Significa que tudo est bem at que isso acontece.
Bem, significa que tudo est bem at que isso acontece.

0.21
0.52
0.40

TABLE VII: “train” shows the number of available training
sentences, and “missing” shows the number and the fractionof missing sentences in comparison with English ones.
Pair
en-hr/sren-sk/cs
en-vi/id

Trg
hr
sr
sk
cs
vi
id

train
115,127
129,461
58,10997,488150,829
77,936

BLEU+1

missing
34,116 (29.6%)
48,450 (37.4%)
16,772 (28.9%)
56,151 (57.6%)
81,945 (54.3%)
9,052 (11.6%)

2) Baseline Methods: We compared the proposed methods
with the following three baseline methods.
One-to-one NMT A standard NMT model from one source
language to another target language. The source language
is English or the remaining language in the language
set. If the target language part is missing in the parallel
corpus, such sentences pairs cannot be used in training
so they are excluded from the training set.
Multi-encoder NMT with back-translation A
multiencoder NMT system using English-to-X NMT to fill up
the missing parts in the other source language X.4
Multi-encoder NMT with NULL A
multi-encoder
NMT system using a special symbol NULL to fillup the missing parts in the other source language X.
Shared-encoder NMT A single shared-encoder multilingual
NMT system as Johnson et al. [3] trained using all
possible language directions (e.g., en-hr, en-sr, hr-sr, hren, sr-hr, and sr-en for en-hr/sr) with target language tags.
3) Main Results: Table VIII shows the results in BLEU
[26]. We can see that our proposed methods demonstratelarger gains in BLEU than baseline methods in all language settings. On these pairs, this demonstrates that the
proposed method is an effective way for using incomplete
multilingual corpora, exceeding other reasonable baselines.
On the other hand, in {English, Vietnamese, Indonesian},our proposed methods obtained smaller gains in BLEU than
4 This is not exactly back-translation because the source translations are notfrom the target language but from the other source language (English) in ourmulti-source condition. But we use this familiar term here for simplicity.

baseline methods compared to the other two language sets:{English, Croatian, Serbian} and {English, Slovak, Czech}.
We observed that the improvement by the proposed method iscorrelated to the performance of the one-to-one NMT with
non-English source languages. This suggests the benefit of
the proposed method comes from the effectiveness of the
non-English source languages. The pseudo translations help
the multi-source NMT to improve further especially when asimilar language is included in the source languages as with{English, Croatian, Serbian} and {English, Slovak, Czech}.
In {English, Vietnamese, Indonesian} case, Vietnamese andIndonesian do not help each other so much as suggested by
the one-to-one results, so the additional pseudo translations
helped just a little. Our proposed method is affected bywhich languages it uses, and the proposed method is likely
more effective for similar language pairs because the expected
accuracy of the data augmentation gets better by the helpof lexical and syntactic similarity including shared subword
entries. The shared-encoder systems showed similar or slightly
better performance in BLEU than the one-to-one models but
were much worse than the multi-encoder methods in en-hr/sr
and en-sk/cs. It is possibly due to the difference in the input
information. The shared-encoder approach takes a single input
sentence while the multi-encoder approach utilizes multiple
input sentences in different languages.
4) Different Types of Augmentation: We examined three
types of augmentation: “fill-in”, “fill-in and replace”, “fillin and add”. In Table VIII, we can see that there wereno significant differences among them, despite the fact that
their training data were very different from each other. We
conducted additional experiments using incomplete corpora
with lower quality augmentation by one-to-one NMT to investigate the differences of the three types of augmentation. We
created three types of pseudo-multilingual corpora using backtranslation from one-to-one NMT and trained multi-encoder
NMT models using them. Our expectation here was that theaggressive use of relatively low quality augmented translations
may contaminate the training data and decrease the translation
accuracy.
Table IX shows the results. In {English, Croatian, Serbian}9

TABLE VIII: Main results in BLEU for English-Croatian/Serbian (en-hr/sr), English-Slovak/Czech (en-sk/cs), and EnglishVietnamese/Indonesian (en-vi/id).

Pair

Trg

en-hr/sren-sk/cs
en-vi/idbaseline single-source
one-to-one
one-to-one
shared-encoder(En to Trg)
(Other to Trg)

hr
sr
sk
cs
vi
id

21.42
17.65
16.13
15.52
25.13
23.67

21.50
17.43
14.55
15.49
25.00
25.85

baseline multi-source
multi-encoder
multi-encoder(fill up with symbol)(back translation)

27.30
23.31
18.75
18.06
17.61
15.96

TABLE IX: The difference of three types of augmentation
in BLEU for English-Croatian/Serbian (en-hr/sr), EnglishSlovak/Czech (en-sk/cs), and English-Vietnamese/Indonesian(en-vi/id). We used one-to-one model to produce pseudotranslations.

Pair
en-hr/sren-sk/cs
en-vi/id

Trg
hr
sr
sk
cs
vi
id

multi-encoder NMT (back-translation)
fill-in andfill-infill-in
replaceand add
27.49
23.73
20.00
19.87
25.02
25.91

24.22
20.09
15.94
16.89
24.36
25.08

25.96
22.27
18.06
18.91
25.22
26.26

and {English, Slovak, Czech}, we obtained significant drops
in BLEU scores with the aggressive strategies (“fill-in andreplace” and “fill-in and add”), while there are few differencesin {English, Vietnamese, Indonesian}. One possible reason isthat the quality of pseudo-translations by one-to-one NMT
in Indonesian and Vietnamese was better than the otherlanguages; in other words, the BLEU from one-to-one NMT in
Table VIII was sufficiently good without multi-encoder NMT.
Thus the translation performance for Croatian, Serbian, Slovak
and Czech could not improve in the experiments here due
to noisy augmented sentences in those languages. Contrary,
the BLEU from “fill-in and add” was the highest when thetarget language was Indonesian and Vietnamese. In the casethe target language is Indonesian, we hypothesize that this isdue to much smaller fraction of the missing parts in Indonesian
corpus as shown in Table VII, so there should be little room
for improvement if we fill in only the missing parts even if theaccuracy of the augmented translations is relatively high. On
the other hand, in the case the target language is Vietnamese,
we hypothesize that the improvement by use of multi-encoder
NMT against one-to-one NMT in the baseline was smaller
than the other language sets as Table VIII shown, so there is
not much difference between augmented machine translations
using multi-encoder NMT and one-to-one NMT.
5) Iterative Augmentation: It can be noted that if we havea better multi-source NMT system, it can be used to produce
better pseudo-translations. This leads to a natural iterative
training procedure where we alternatively update the multisource NMT systems into the two target languages.
Table X shows the results of all languages sets. We can see

27.58
25.03
20.66
20.93
24.88
25.86

27.49
23.73
20.00
19.87
25.02
25.91

proposed method
fill-in andfill-infill-in
replaceand add
30.21
25.16
22.03
23.21
25.33
25.55

29.67
25.52
21.50
23.16
24.68
24.36

30.33
25.16
22.12
23.13
25.13
26.00

that this iterative method demonstrate larger gains in BLEU
than that at first step in all language sets except when the target
language was Vietnamese, though we didn’t get monotonicallyincreases in BLEU at each iterative step. We hypothesize thereason is that automatic translations at first step were createdfrom multi-encoder NMT which was trained with the corpus
whose missing sentences are filled up with NULL , and weused automatic translations created from multi-encoder NMT
trained with the corpus filled with automatic translations after
the first step. In other words, BLEU score improved if we filled
up missing sentences with better quality automatic translations.
On the other hand, BLEU score decreased compared to the firststep in the case where the target language is Vietnamese. The
reason may be that multi-encoder NMT can perform similarly
to one-to-one NMT when Vietnamese was the target.
6) Non-parallelism: One problem in the use of multilingual
corpora is non-parallelism, even in the allegedly manuallycreated and verified translations. In the case of TED multilingual captions, they are translated from English transcripts
independently by many volunteers, which may cause somedifferences in details of the translation in the various target
languages. For example in {English, Croatian, Serbian}, Croatian and Serbian translations may not be completely parallel.
Table XI shows such an example where the Croatian translation does not have a phrase corresponding to “To be sure,”
This kind of non-parallelism may be resolved by overriding
such translations with pseudo-translations using our proposed
strategies; “fill-in and replace” and “fill-in and add”. Here, theCroatian pseudo-translation includes the corresponding phrase“Da bi bila sigurna” and can be used to compensate for the
missing information. This would be one possible reason of the
improvements by “fill-in and replace” or “fill-in and add”.
VI. T EST- TIME DATA AUGMENTATION E XPERIMENTS
Finally, we perform an experiment to confirm the effectiveness of the proposed test-time data augmentation strategy.
A. Data
We used the same data set and language sets as the experiment of augmentation with pseudo-translations in Section V-E,
but here we used a different test set with missing source
language sentences, while all source language sentences werecomplete in the test set in the previous experiment. Table XII
shows the number of test sentences for each language set.

10

TABLE X: BLEU (and BLEU gains compared to step 1) in each step of iterative augmentation.
Pair
en-hr/sren-sk/cs
en-vi/id

Trg
hr
sr
sk
cs
vi
id

step 130.21 (+0.00)
25.16 (+0.00)
22.03 (+0.00)
23.21 (+0.00)
25.33 (+0.00)
25.55 (+0.00)

step 230.69 (+0.48)
26.27 (+1.11)
22.87 (+0.84)
23.81 (+0.60)
25.30 (-0.03)
25.93 (+0.38)

step 331.34 (+1.13)
26.24 (+1.08)
22.51 (+0.48)
24.48 (+1.27)
25.16 (-0.17)
25.84 (+0.29)

step 431.05 (+0.84)
26.55 (+1.39)
22.93 (+0.90)
24.10 (+0.89)
25.10 (-0.23)
25.86 (+0.31)

TABLE XI: Example of the Croatian pseudo-translation. This pseudo-translation is the output of {English, Serbian}-to-Croatiantranslation.
Type
Original (En)
Original (Hr)
Pseudo (Hr)Sentence
To be sure, governments have different resources to bring to the table.
Vlade imaju razne resurse kojima raspolau.
Da bi bila sigurna, vlada ima razliite resurse koje treba za stol.

TABLE XII: “Available test sentences” shows the number ofavailable test sentences for each target language.
Pair
en-hr/sren-sk/cs
en-vi/id

Trg
hr
sr
sk
cs
vi
id

Available test sentences
1,145
896
602
1,966
1,405
333

TABLE XIII: BLEU results with incomplete test sets
for English-Croatian/Serbian (en-hr/sr), English-Slovak/Czech(en-sk/cs), and English-Vietnamese/Indonesian (en-vi/id). In
this table, “1-best” means multi-encoder NMT with an incomplete corpus whose missing parts were filled with 1-best
translations with the beam width of 5.

Pair

B. Baseline methods
We compared the proposed method with the following twobaseline methods:
One-to-one NMT: whose source language is English.
Multi-encoder NMT with 1-best translations: whereanincomplete corpus’s missing parts were filled with 1-best
translations using beam search (beam width=5) fromone-to-one NMT.
We didn’t use the mixture of NMT experts as a multi-source
NMT in this experiment because the result of it is worse thanthat of multi-encoder NMT as shown in Chapter V.
C. NMT settings
We conducted the experiment with the same NMT settings
as the experiment “augmentation with pseudo-translations” inthe section V-E. The multi-encoder NMT model itself wastrained with the “fill-in” strategy. We took 5 as the size of then-best in our proposed method.
D. Results
Table XIII shows the results in BLEU. “proposed” in the
results shows the best BLEU with the appropriate hyperparameter λ. We chose this λ by performing a grid search from 0 to
1 with increments of 0.05, and chose the value where BLEU
is the highest on the development data. In this table, “1-best”
means multi-encoder NMT with an incomplete corpus whose
missing parts were filled in with 1-best translations using beam
search (beam width=5) from one-to-one NMT.
First of all, we can see that our proposed method achievedlarger gains in BLEU than almost all of the multi-encoder

en-hr/sren-sk/cs
en-vi/id

Trg
hr
sr
sk
cs
vi
id

baseline method
one-to-one
1-best(En-to-Trg)
22.58
16.38
14.16
15.13
22.62
26.41

22.55
15.71
16.57
13.63
22.96
26.23

proposed method
22.43
16.07
16.59
13.85
23.39
26.96

NMT baseline methods with 1-best automatic translations
from one-to-one NMT. These results show that our proposed
method can effectively consider which translations in the nbest are better to fill in. However, our proposed method wasworse than the “one-to-one NMT” baseline in English-toCroatian, English-to-Serbian and English-to-Czech. In the casewhere the target language is Czech, the amount of training datais larger than that the case where the target language is Slovak,
as shown in Table VII. Because of this, Czech-English canmore easily get good accuracy than Slovak-English translation,and thus it is less likely that using Slovak as an additionalsource will be helpful. Thus it is reasonable that the BLEU
of our proposed method in Czech was lower than that of oneto-one NMT, while BLEU of our proposed method in Slovak
is about 2 points larger than that of one-to-one NMT. In thecase where the target language is Croatian and Serbian, thedifference of the amount of training data between Croatianand Serbian is small, and we can assume that our proposed
method is less effective if the difference of the amount oftraining data is small.
In conclusion, the proposed method is effective for using
an incomplete corpus at test time, although this is dependent
on the amount of data available in each of the languages.

11

0.20

hr
sr
sk

BLEU differences

0.00

cs
id
vi

-0.20

-0.40

-0.60
0.00

0.25

0.50

0.75

1.00

λ

Fig. 8: BLEU differences for each target language by changing
λ. In this figure, BLEU when λ is equal to 0.5 is used as abaseline, and the BLEU difference at each λ is compared tothis baseline.
TABLE XIV: λ on the development data and the test
data when BLEU is the highest for English-Croatian/Serbian(en-hr/sr), English-Slovak/Czech (en-sk/cs), and EnglishVietnamese/Indonesian (en-vi/id). The best BLEU on test datawith an appropriate λ for test data is also shown.
Pair
en-hr/sren-sk/cs
en-vi/id

Trg
hr
sr
sk
cs
vi
id

dev. data
λ
0.50
0.50
0.85
0.40
0.35
0.35

test data
λ
BLEU
0.85
22.62
0.35
16.17
0.35
16.91
0.55
13.89
0.25
23.42
0.45
26.97

E. Discussion
We examined which λ value is the best by changing λ byincrements of 0.05. Figure 8 shows the BLEU differences fromthe one with λ = 0.5 for different values of λ. The BLEU
differences were negative for a larger value of λ, except when
the target language is Croatian (hr). From the definition in
Eq. (6), a larger value of λ means the probabilities of multiencoder NMT are more heavily weighted than one-to-one
NMT when filling in the missing parts. The Croatian results
show the use of 1-best English-to-Serbian results to fill themissing parts contributed the final performance, as also shownby the results in Table XIII. From the other results, we cansee that we get larger gains in BLEU if the probabilities of themulti-encoder NMT outputs are more important. On the other
hand, the λ that got the best BLEU on the development data
is sometimes very different than the λ that got the best BLEU
on the test data as shown in Table XIV. Thus, it is assumed
that λ is really affected by which sentences we use.
VII. C ONCLUSION
In this paper, we examined approaches for multi-source
NMT using incomplete multilingual corpora . This problemof missing translations in multi-source NMT happens at bothtraining and test time. Thus we proposed three methods, twofor training time and one for test time.

For training time, we proposed to fill missing sentences by
NULL at first and improved the performance of multisource NMT using an incomplete part of a multilingual corpus.
Then we improved it further by data augmentation with multisource NMT. For test time, we proposed to consider n-best
translations for a missing source language sentence by oneto-one NMT and achieved better BLEU results than just using
1-best translations. This work is the first study on the problem
of multi-source NMT using an incomplete multilingual corpus.
One remaining issue with the third proposed method is thatit could not get improvements compared to one-to-one NMT
in some language sets when some source sentences were notavailable at test time. Since the third method relies on allof the techniques proposed in this paper, it is not clear thatthe poor performance is due to training or test time problem.
Future work includes further investigation of the relationshipbetween training and test time augmentation to solve theproblems jointly. Language combinations are also important
for the proposed approach as we found in some mixed results,
so further extensive investigation on related factors such asmorphological and syntactic similarities.
ACKNOWLEDGMENT
Part of this work was supported by JSPS KAKENHI Grant
Numbers and JP16H05873 and JP17H06101.
R EFERENCES[1] D. Dong, H. Wu, W. He, D. Yu, and H. Wang, “Multi-Task Learningfor Multiple Language Translation,” in Proceedings of the 53rd Annual
Meeting of the Association for Computational Linguistics. Beijing,
China: Association for Computational Linguistics, July 2015, pp. 1723–
1732.
[2] O. Firat, K. Cho, and Y. Bengio, “Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism,” in Proceedingsof the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
San Diego, California: Association for Computational Linguistics, June
2016, pp. 866–875.
[3] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen,
N. Thorat, F. Vigas, M. Wattenberg, G. Corrado, M. Hughes, andJ. Dean, “Google’s Multilingual Neural Machine Translation System:
Enabling Zero-Shot Translation,” Transactions of the Association for
Computational Linguistics, vol. 5, pp. 339–351, 2017.
[4] T.-L. Ha, J. Niehues, and A. Waibel, “Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder,” in Proceedingsof the 13th International Workshop on Spoken Language Translation,
Seattle, Washington, December 2016.
[5] P. Koehn, “Europarl: A Parallel Corpus for Statistical Machine Translation,” in Conference Proceedings: the tenth Machine Translation
Summit, AAMT. Phuket, Thailand: AAMT, 2005, pp. 79–86.
[6] M. Ziemski, M. Junczys-Dowmunt, and B. Pouliquen, “The United
Nations Parallel Corpus v1.0,” in Proceedings of the Tenth International
Conference on Language Resources and Evaluation (LREC 2016),
N. C. C. Chair), K. Choukri, T. Declerck, S. Goggi, M. Grobelnik,
B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis,
Eds. Paris, France: European Language Resources Association (ELRA),
May 2016.
[7] M. Cettolo, C. Girardi, and M. Federico, “WIT3 : Web Inventory ofTranscribed and Translated Talks,” in Proceedings of the 16th EAMT
Conference, May 2012, pp. 261–268.
[8] J. Tiedemann, “News from OPUS - A Collection of Multilingual
Parallel Corpora with Tools and Interfaces,” in Recent Advances in
Natural Language Processing, N. Nicolov, K. Bontcheva, G. Angelova,
and R. Mitkov, Eds. Borovets, Bulgaria: John Benjamins, Amsterdam/Philadelphia, 2009, vol. V, pp. 237–248.
[9] F. J. Och and H. Ney, “Statistical Multi-Source Translation,” in Proceedings of the eighth Machine Translation Summit (MT Summit VIII),
September 2001, pp. 253–258.

12[10] B. Zoph and K. Knight, “Multi-Source Neural Translation,” in Proceedings of the 2016 Conference of the North American Chapterof the Association for Computational Linguistics: Human Language
Technologies. San Diego, California: Association for Computational
Linguistics, June 2016, pp. 30–34.
[11] E. Garmash and C. Monz, “Ensemble Learning for Multi-Source Neural
Machine Translation,” in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.
Osaka, Japan: The COLING 2016 Organizing Committee, December
2016, pp. 1409–1418.
[12] H. Wu and H. Wang, “Pivot language approach for phrasebased statistical machine translation,” Machine Translation, vol. 21,
no. 3, pp. 165–181, 2007. [Online]. Available: https://doi.org/10.1007/
s10590-008-9041-6[13] Y. Nishimura, K. Sudoh, G. Neubig, and S. Nakamura, “Multi-Source
Neural Machine Translaion with Missing Data,” in Proceedings ofthe 2nd Workshop on Neural Machine Translation and Generation.
Association for Computational Linguistics, July 2018, pp. 92–99.
[14] ——, “Multi-source neural machine translation with data augmentation,”in 15th International Workshop on Spoken Language Translation(IWSLT), Brussels, Belgium, October 2018. [Online]. Available:
https://arxiv.org/abs/1810.06826[15] D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by
Jointly Learning to Align and Translate,” in Proceedings of the 3rdInternational Conference on Learning Representations, May 2015.
[16] T. Luong, H. Pham, and C. D. Manning, “Effective Approaches toAttention-based Neural Machine Translation,” in Proceedings of the2015 Conference on Empirical Methods in Natural Language Processing.
Lisbon, Portugal: Association for Computational Linguistics,
September 2015, pp. 1412–1421.
[17] R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” in Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers). Association for Computational Linguistics, 2016, pp.
86–96.
[18] K. Imamura, A. Fujita, and E. Sumita, “Enhancement of encoder
and attention using target monolingual corpora in neural machinetranslation,” in Proceedings of the 2nd Workshop on Neural
Machine Translation and Generation. Association for Computational
Linguistics, 2018, pp. 55–63. [Online]. Available: http://aclweb.org/
anthology/W18-2707[19] O. Firat, B. Sankaran, Y. Al-onaizan, F. T. Yarman Vural, and
K. Cho, “Zero-resource translation with multi-lingual neural machinetranslation,” in Proceedings of the 2016 Conference on Empirical
Methods in Natural Language Processing. Austin, Texas: Association
for Computational Linguistics, Nov. 2016, pp. 268–277. [Online].
Available: https://www.aclweb.org/anthology/D16-1026[20] X. Wang, H. Pham, Z. Dai, and G. Neubig, “Switchout: an efficient data
augmentation algorithm for neural machine translation,” in Proceedingsof the 2018 Conference on Empirical Methods in Natural Language
Processing, 2018.
[21] Y. Kim and A. M. Rush, “Sequence-level knowledge distillation,” inProceedings of the 2016 Conference on Empirical Methods in Natural
Language Processing. Austin, Texas: Association for Computational
Linguistics, November 2016, pp. 1317–1327. [Online]. Available:
https://aclweb.org/anthology/D16-1139[22] M. Morishita, J. Suzuki, and M. Nagata, “NTT Neural Machine Translation Systems at WAT 2017,” in Proceedings of the 4th Workshop on
Asian Translation (WAT2017), 2017, pp. 89–94.
[23] G. Neubig, P. Arthur, and K. Duh, “Multi-target machine translation
with multi-synchronous context-free grammars,” in Proceedings of the2015 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies. Denver,
Colorado: Association for Computational Linguistics, May–June 2015,
pp. 293–302. [Online]. Available: http://www.aclweb.org/anthology/
N15-1033[24] T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” in Proceedings of the56th Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers). Association for Computational Linguistics,
2018, pp. 66–75.
[25] D. P. K. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in Proceedings of the 3rd International Conference on Learning
Representations, May 2015.
[26] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a Method forAutomatic Evaluation of Machine Translation,” in Proceedings of the40th Annual Meeting of the Association for Computational Linguistics(ACL), Philadelphia, July 2002, pp. 311–318.
[27] M. Post, “A call for clarity in reporting BLEU scores,” inProceedings of the Third Conference on Machine Translation:
Research Papers, WMT 2018, Belgium, Brussels, October 31- November 1, 2018, 2018, pp. 186–191. [Online]. Available:
https://aclanthology.info/papers/W18-6319/w18-6319

13

Yuta Nishimura received his B.E. from Tohoku
University, Miyagi, Japan in 2017, and his M.E. in
information science from Nara Institute of Scienceand Technology in 2019. From 2019, he has been
doing research and development at PLAID, Inc.,
where he is pursuing research in user analysis witha focus on machine learning approach.

Katsuhito Sudoh is an associate professor of Nara
Institute of Science and Technology. He received
a bachelor’s degree in engineering in 2000, and amaster’s and Ph.D. degree in informatics in 2002
and 2015, respectively, from Kyoto University. He
was in NTT Communication Science Laboratories
from 2002 to 2017. He currently works on machinetranslation and natural language processing. He is amember of the Association for Computational Linguistics (ACL), the Association of Natural Language
Processing (ANLP), the Information Processing Society of Japan (IPSJ) and the Acoustical Society of Japan (ASJ).

Graham Neubig received his B.E. from University
of Illinois, Urbana-Champaign in 2005, and his M.S.
and Ph.D. in informatics from Kyoto University
in 2010 and 2012 respectively. He is currently an
assistant professor at Carnegie Mellon University
and an visiting associate professor at Nara Instituteof Science and Technology. His research interests
include speech and natural language processing,with a focus on machine learning approaches forapplications such as machine translation, speech
recognition, and spoken dialog.

Satoshi Nakamura is Professor of Graduate Schoolof Information Science, Nara Institute of Scienceand Technology, Honorarprofessor of Karlsruhe Institute of Technology, Germany, and ATR Fellow.
He received his B.S. from Kyoto Institute of Technology in 1981 and Ph.D. from Kyoto University
in 1992. He was Director of ATR Spoken Language
Communication Research Laboratories in 2000-2008
and Vice president of ATR in 2007-2008. He was
Director General of Keihanna Research Laboratories
and the Executive Director of Knowledge Creating
Communication Research Center, National Institute of Information and Communications Technology in 2009-2010. He is currently Director of Augmented
Human Communication laboratory and a full professor of Graduate Schoolof Information Science at Nara Institute of Science and Technology. He
has been Elected Board Member of International Speech Communication
Association, ISCA, since June 2011 and IEEE Signal Processing Magazine
Editorial Board member since April 2012, IEEE SPS Speech and Language
Technical Committee Member since 2013, and IEEE Fellow since 2016.