IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

1

PAPER

Leveraging Neural Caption Translation with Visually Grounded
Paraphrase Augmentation
Johanes EFFENDI†a) , Nonmember, Sakriani SAKTI†,†† , Member, Katsuhito SUDOH†,†† , Nonmember,and Satoshi NAKAMURA†,†† , Member

SUMMARY
Since a concept can be represented by different vocabularies, styles,
and levels of detail, a translation task resembles a many-to-many mappingtask from a distribution of sentences in the source language into a distribution of sentences in the target language. This viewpoint, however, is not
fully implemented in current neural machine translation (NMT), which is
one-to-one sentence mapping. In this study, we represent the distribution
itself as multiple paraphrase sentences, which will enrich the model context understanding and trigger it to produce numerous hypotheses. We use
a visually grounded paraphrase (VGP),which uses images as a constraintof the concept in paraphrasing, to guarantee that the created paraphrases
are within the intended distribution. In this way, our method can also be
considered as incorporating image information into NMT without using
the image itself. We implement this idea by crowdsourcing a paraphrasing
corpus that realizes VGP and construct neural paraphrasing that behaves as
expert models in a NMT. Our experimental results reveal that our proposed
VGP augmentation strategies showed improvement against a vanilla NMT
baseline.
key words: visually grounded paraphrase, data augmentation, neuralmachine translation

1.

Introduction

Sequence-to-sequence problems are usually solved by assuming that the model is mapping from the source sequence
distribution to the target sequence distribution. The representation of these distributions can be varied depending on
the task. In question-answering tasks, the source and target
distributions can be represented by questions and answers.
Similarly, in machine translation (MT), the source and target
distributions are represented by source and target sentences.
Since these distributions are the concept’s representations,
their sentences can be replaced with other equivalent ones.
However, common approaches in machine translation(MT) assumes that this task is simplified by a one-to-one
sentence mapping problem. In this study, we propose torepresent this mapping task by a many-to-many approach,specifically by depicting the distribution with multiple paraphrase sentences. Although all of these sentences are valid,
the model’s understanding can be enriched and triggered to
produce multiple valid hypotheses. From the linguistic perManuscript received March 6, 2019.
Manuscript revised August 1, 2019.
† The authors are with the Augmented Human Communication
Lab., Nara Institute of Science and Technology, Ikoma-shi, 6300192 Japan.
† The author are with the RIKEN, Center for Advanced Intelligence Project AIP, Ikoma-shi, 630-0192 Japan.
a) E-mail: johanes.effendi.ix4@is.naist.jp
DOI: 10.1587/trans.E0.??.1

spective, this definition also resembles Hirst’s (2003) idea,
which defines paraphrasing as describing a situation by manydifferent way of expressions [1].
Furthermore, to represent the source and target distributions in multiple paraphrase sentences, a semantic boundaryneeds to be defined. Unconstrained paraphrasing on a sentence might lead to an inappropriate shifts in semantic meaning that diverged from the original idea of the paraphrased
sentence. For this reason, we need a boundary within the
distribution to ensure that there is a common ground exists
among several paraphrases.
In this study, we propose a new variant of paraphrases
called visually grounded paraphrases (VGP), which we define VGP as a set of paraphrases that describe about thesame image. Different phrases and wording between twoparaphrase sentences can also be associated with an object
or an action shown in the image, which work as a pivot[2]. This association implies that multiple paraphrases canbe produced by operating with these associated phrases orwords. Additionally, from the amount of information contained, an image has more information than a single captionthat represents it. Therefore, to sufficiently represent an image, multiple sentences are preferable for representing theinformation contained in it than just a single sentence.
Grounding visual description and its image representation enables a wide variety of functionality, especially inmultimodal settings. VGP can be used in image captioning[3] and visual question answering [4] to improve the understanding of both textual and visual information. In thisstudy, we also showed that VGP is useful for neural machinetranslation as an augmentation method, in which we use several variations of VGP as multi-source input for a neural MT(NMT). In all of its use cases, VGP improves the relations
between textual and visual data. In addition, it can also
represent visual data in a textual representation.
We evaluated our proposed VGP concept using a caption translation task to measure its effectiveness in a practicaltask and compared it with other approaches. This task hosted
by the Second Conference of Machine Translation (WMT17)seeks to translate a given image description into the target
language. Most of multimodal approaches focus on utilizing
image features in addition to the information from a single
caption of the source language. However, the results frommost submitted systems show that the additional image features only slightly contribute to system performance. As
pointed out by Calixto et al. [5], the image-text latent repre-

Copyright © 200x The Institute of Electronics, Information and Communication Engineers

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x2
Table 1

Image caption and example paraphrases

Operation
Image Caption
Deletion
Insertion
Paraphrase
Substitution
Reordering

Sentence
Two white dogs are running on the grass.
Two white dogs are running.
Two white dogs wearing collars are running on the grass.
On the grass, two white dogs are running.
Two white dogs are racing on the grass.

1. Proposed the concept of visually grounded paraphrases
that introduce a new way of creating a paraphrase corpus
through image captions;common usages of paraphrases in MT fail to demonstratethese wide and flexible definitions of paraphrases. Froma bigger perspective, we can say that this issue happens because the sentence being paraphrased doesn’t have a tangible
representation of ideas. Further intrusive operation on thisoriginal sentence, might completely change its meaning due
to a lack of grounded representation of the idea.
Therefore, we proposed a visually grounded paraphrase(VGP)† , to implement a wider definition of paraphrasing
and completely utilize its benefits. We also implement paraphrases for each visual representation as a set of captions
being paraphrased by a set of paraphrase operations. Thisidea covers every possible paraphrase that is being made andenables the tracking of the operation that has been done on
the source sentence. The grouping of possible paraphrase
operations is described in the next subsection.

2. Generated VGP sentences of the WMT17 Multimodal
Translation Task dataset through crowdsourcing;

2.2

sentation combination approach has not yielded significant
improvement on WMT 2017 Multimodal shared task dataset
testing.
However, compared to the previous attempts, our approach can be considered as diffusing the image information
with visually-grounded paraphrases without using the image
itself. The resulting paraphrase captions are then utilized
within a multi-source and multi-expert NMT model. Another advantage of our textual approach is that it uses fewer
computational resources than the multimodal approach, andstill maintains comparable performance.
The following are the contributions of this work:

Paraphrase Elementary Operations

3. Developed automatic paraphrase generation in a semisupervised manner;
4. Utilized multi-expert translation in neural machinetranslation using our proposed paraphrases;
5. Improved the baseline used at WMT17 with a 13.2
BLEU score margin, which is near the top score thatused a multimodal model.
2.

Visually Grounded Paraphrases Generation

2.1

Defining Visually Grounded Paraphrases

Although the logical definition of a paraphrase is generally
as simple as a “semantic equivalence", linguists accept abroader, approximate equivalence that allows more relaxedparaphrasing action constrained by an idea. Hirst (2003)
argued that paraphrases aren’t fully synonymous because
they have some pragmatic differences [1]. Such hard-todefine definitions causes paraphrasing-related technologies
to use the safest available operations to avoid producing nonvalid paraphrases.
For example, by analyzing the nature of the operations
found in paraphrase applications in MT, we found that only
non-intrusive paraphrase operations like phrase substitution
or phrase reordering were used. Simply adding or flexibly rephrasing information is impossible because the only
source of information that can be used as the inspiration forthe paraphrasing is the source sentence. Nevertheless, such

Fig. 1[6].

Reference image for captioning and paraphrasing shown in Table 1

To train a paraphrasing model with our VGP, first, weneed to build a set of paraphrased source sentences withimages as the basis for paraphrasing. This requirement resembles image captioning datasets such as the Microsoft
Common Object in Context (MSCOCO) dataset [7]. The
captions of this dataset can be regarded as paraphrase sets,such as those done by Prakash et al. for their neural paraphrase generation study [8]. They reported how their annotators described the most obvious things in an image andconcluded that several captions of an image can be counted
as paraphrases. Although this result may be true, we cannot† Not to be confused with Chu et al.’s (2018) work which uses
the same term that refers to a synonymous set of clustered phrases
with the help of object grounding in an image [2]. In our work, we
focus on generating paraphrase sentences.

EFFENDI et al.: LEVERAGING NEURAL CAPTION TRANSLATION WITH VISUALLY GROUNDED PARAPHRASE AUGMENTATION

3

define what operations have been done from the original sentence to the paraphrase. Consequently, the arbitrary natureof the corpus distribution might cause the paraphrases to be
regarded as noise by each other.
To prevent this situation, a set of paraphrase operations
covering every possible paraphrase variation needs to be defined. Bhagat and Hovy categorized the variations of humanparaphrases [9] and argued that “although the logical definition of paraphrases requires strict semantic equivalence, linguistics accept a broader, approximate, equivalence.” Basedon this idea, they analyzed paraphrase characteristics invarious studies and in corpora and established 25 quasiparaphrase operations, including tense changes, metaphor
substitution, and function-word variations.
Some quasi-paraphrase operations have very small frequency in the MTC [10] and MSRP corpora [11], as reportedby them. On the other hand, creating 25 kinds of paraphrases
from one original sentence in corpus creation might be too
difficult and too fine-grained. Then, we grouped them into
four elementary paraphrase operations: deletion, insertion,
reordering, and substitution. These groups cover the possible quasi-paraphrases defined, while reducing the number ofparaphrases needs to be made from one sentence. Moreover,
we call our paraphrase as elementary operation because it’s
possible to utilize more than one operations to create a newparaphrases, which has multiple elementary operations. For
this study, we focused on constructing a paraphrase corpus
based on these four operations.
However, the process of manually collecting paraphrases is expensive and time-consuming. On the other
hand, Resnik et al. (2013) proposed that corpus creation
with a crowdsourcing platform provides such advantages as
low cost, effectiveness, and reasonable quality [12]. The
paraphrase collection was done through a crowdsourcing
platform on part of the WMT17 Multimodal Translation
Task dataset [6]. After that, we constructed our automaticneural paraphrase model based on partial data to generateparaphrase sentences of the full WMT17 dataset. The details are described below.

substitution), and Fig. 1 shows the reference image. Since
paraphrasing the entire 29k triplet training dataset (29k training dataset) with crowdsourcing is inefficient in terms of cost
and time, we crowdsourced only 10k triplets of this dataset(10k training dataset) along with the entire development andtesting datasets.
We used Figure Eight† as the crowdsourcing platform.
Each crowdworker was instructed to paraphrase at least two
image captions per session, based on the image to which
it refers. As shown in Fig. 2, the crowdworker are always
be able to see the reference image and the original caption.
With this way, the created paraphrases are always refers to
the original caption, and its definition doesn’t stray away
because an image is always provided as reference.
We limited the task to English speakers or those who
spoke English as their second language to maintain quality.
We discarded such invalid sentences as those with randomly
inputted characters, empty strings, or captions that aren’t
in English. The crowdsourcing process took about threemonths with 201 participants from 16 countries including
the United States, Philippines, and Malaysia. Each worker
created an average of 50.1 paraphrase quintuplets.

2.3

2.4

Crowdsourcing Paraphrases on Partial WMT17
DatasetThe WMT17 Multimodal Translation Task dataset [6] contains a set of images with triplets of captions in English,
German, and French. The dataset was created from the
Flickr30K Entities dataset of image captions in English [13]that was extended to include manually translated German andFrench captions. The data respectively consists of 29000,
1014, and 1000 triplets respectively for the training, development and testing. An out-of-domain dataset consisting
461 images taken from the MSCOCO dataset [7] was also
introduced, which contains ambiguous verbs [14].
We focused on paraphrasing English sentences, which
are considered the source language. Table 1 shows an example of a paraphrased image caption based on the four
elementary operations (deletion, insertion, reordering, andFig. 2Form layout during the crowdsourcing.

Corpus Analysis Method

In this subsection, we show the statistics of the corpus toshow how each operation is implemented in the crowdsourcing process. There are two kinds of analysis: word-based
and part-of-speech-based. First, we created the following
word-based measurements to show the statistics of our corpus based on each operation:
Ratio of target sentence vs. source sentence: By calculating this ratio, we identified the compression or inflation rate of each sub-operation corpus with which we
can evaluate the effectiveness of each operation.
Average number of deleted words per sentence: This metric shows how many words were deleted from the source† http://www.figure-eight.com

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

4sentence on average.
Average number of inserted words per sentence: This metric shows how many words were inserted to the target
sentence on average.
Average reordered word distance: This metric shows how
a word was reordered in the sentence by comparing its
position distance between the source and target sentences.
Average number of substituted words per sentence: This
metric shows how many words were substituted between the source and target sentences on average. For
reordering and substitution operation analysis, we used
the word alignment between the source and target language sentences estimated using Fast Align [15].
Next, we performed a part-of-speech-based analysis bycomparing the source and target sentences with their partof-speech (POS) tags predicted by the Stanford POS Tagger[16]. For this analysis, we show the top three POS tagsfor every operation. This will give an overview on how
the crowdworker prefer which kinds of words to be deleted,inserted, reordered, or substituted.
Then, in order to measure the semantic consistency ofthe paraphrases, we calculate the average semantic embedding distance from original source caption to its paraphrased
counterpart. We used Bidirectional Encoder Representations from Transformers (BERT) [17] to generate sentence
embedding. As the baseline, we compare with the distanceof source sentence and random sentence in dataset. We use
mean squared error (MSE) and cosine distance between two
sentence embedding vectors as distance measurement.
2.5

Corpus Analysis(1)

Word-based operation characteristics

To calculate the deletion operation effect on the source sentences, we compared the ratio of the number of words inthe target sentence to the number in the source sentence andfound that the words in the target sentences decreased by
25.61%, where 3.20 words were deleted per sentence on average (Table 2). We did the same calculation in the insertion
sub-corpus and found that the number of words in the target
sentences increased by 25.24%, where an average of 2.89
words were inserted per sentence.
Table 2the source and target sentences and found that those in thelatter shifted on average by as many as 5.35 words. The distance calculated in the reordering happened when the source
sentence was paraphrased into its passive form. Anotherreordering approach switched the order of such sentence information as time, place, or tool.
For the substitution sub-corpus, we measured how many
words were replaced by checking whether the source andtarget words matched in an alignment. If they are different,
then we counted that as one word substitution. As seen in
Table 2, we found an average of 1.66 word substitutions per
sentence in the substitution sub-corpus. This means that at
most 1 or 2 words were substituted in a sentence.
(2)

POS-based operation characteristics

We counted the types of words that were most deleted andinserted. For reordering and substitution, we counted a pairof source and target word types to identify which word typewas usually preferred by the crowdworkers.
Table 3
Deletion
NN
IN
JJ

Top Three Most Operated POS Tags
Insertion
NN
JJ
IN

Reordering
DT DT
NN NN
IN IN

Substitution
NN NN
NNS NNSVBG VBG

Table 3 shows that nouns (NN), adjectives (JJ), andconjunctions (IN) were usually deleted or inserted. This
correlates with how most sentences are deleted or inserted:by adding or removing time, place, or tool information, suchas “on a dark night" or “on a beautiful house".
For reordering the sub-corpus, we found that no word
type was changed by the reordering, showing that reordering
just changes a sentence’s order or its structure without making major alterations to the words or the semantics. This also
happens in substitutions where a word is usually replaced byanother word with the same meaning. Such target words areusually hypernyms or synonyms of the source word.
(3)

Embedding-based operation characteristics

Sub-corpus characteristics

Parameter
Average number of deleted words per sentence
Average number of inserted words per sentence
Average distance of reordered word
Average number of word substitutions per sentence# word
3.197
2.877
5.348
1.663
Fig. 3

To measure the reordering elementary operation for the
sub-corpus, we calculated the shift distance of a word in

Semantic embedding distance from original source sentence

In Fig. 3 we show the average distance between original
source sentence embedding and the paraphrases (see column:

EFFENDI et al.: LEVERAGING NEURAL CAPTION TRANSLATION WITH VISUALLY GROUNDED PARAPHRASE AUGMENTATION

5

deletion, insertion, reordering, substitution) which are relatively similar. Compared with the distance to the random
sentence (see column: random), our paraphrased sentence
has much lower semantic embedding distance. This showsthat the paraphrase operation maintained the consistency ofthe source sentence semantic meaning.
2.6

Semi-supervised Paraphrase Generation on Full
WMT17 Dataset

To complete the paraphrasing on the full WMT17 dataset,
we used 10k quintuplets of crowdsourced paraphrases andconstructed a neural paraphrase models using four encoderdecoder, long short-term memory (LSTM) models with attention [18] for each paraphrase operation. We tuned andtested our automatic neural paraphrase model using thesecrowdsourced paraphrases of the development and testingdatasets. With these four paraphrasing models, we generated VGPs on the remaining 19k image captions.
3.

Improving MT with Visually Grounded Paraphrase

This section describes several approaches for using our proposed paraphrase operations to improve NMT. The scoresof these approaches will then be compared with the WMTbaseline and our encoder-decoder LSTM NMT baseline.
3.1

Two Scenarios of Data Usage

Given the created VGP corpus, there are two usage possibilities. First, the simplest form of data augmentation is toconcatenate all the paraphrases with the original sentence.
However, this simple approach suffers from disadvantages,
because the relationships between the paraphrases and their
original sentences are lost in the concatenation process withother quintuplets. Therefore, we also investigated the useof both multi-source and ensemble NMTs to preserve thisrelationship between paraphrases.
3.2

Combining All Data in a Single Model

This method was done by just using the paraphrases as ameans for data augmentation on the source side, such asreported by Nichols et al. (2010), to leverage SMT systems[19]. All of the paraphrases and their original sentence werecombined, and the target sentence was duplicated by the
number of multiple paraphrases. This approach measured
the baseline performance with augmented data.
3.2.2

3.2.3

Uniform-weighted Ensemble Model

For the uniform-weighted ensemble model, we trained NMT
models whose source sentences were paraphrased based on
the four elementary operations and another that uses theoriginal source sentence to create five expert NMT models.
Next, these five models were ensembled by averaging each
output layer probability distribution, so that every model
was weighted uniformly. We used this model to compare theperformance with the mixture-of-experts model described in
the next subsection, where each expert model has different
weight.
The training of this translation model consists of two
steps. The first step is to train five translation models basedon each paraphrase as a source sentence using the 56k dataset(the combination of the original and paraphrased source sentences). Five of these models are trained against the sametarget sentence. Each model is then regarded as an expert.
Each of the expert models operates on the sub-word level,
tokenized by Sentencepiece with 3000 vocabulary units† .
3.2.4

Mixture-of-experts Model

Next we adopted the mixture-of-experts model proposed byGarmash and Monz (2016). Here, instead of using the linear layers proposed in their study [21], we implemented anexpert model into a single LSTM layer hid that receives
concatenated decoder hidden state output hn :Proposed Approaches

In this section, we listed our proposed approaches to improve
MT using our created VGP dataset.
3.2.1

Zoph and Knight (2016) to incorporate various paraphrase
inputs with one output [20]. For their model, the encoded
representation and attention were combined by concatenation. Zoph and Knight reported that this model has anadvantage of information triangulation to reduce ambiguity. In their paper, they used several translation pairs suchas {French, German} with English for which this language
triplet shares a similar language structure. However, giventhese advantages, using this model as monolingual input has
never been investigated.

Multi-source Model

We implemented a multi-source NMT model proposed by

ct = tanh(LST Mhid ([h0, h1, ..., hn ]))g0:i = so f tmax(Wgate D(ct ) + bgate ).
We then applied a so f tmax function to obtain theweights of each expert model’s output layer on . Assuming Wn is the weight of the output layer from expert n. Then,the aggregated weight Wagg is a linear combination functionof each of those weights:
Wagg = g0W0 + g1W1 + ... + gn Wn .
For this model, a 50% dropout D will be applied on thehidden representation after tanh nonlinearity was applied.
The regularized representation was further transformed by
the gate layer whose output size matches the number ofexperts.
† https://github.com/google/sentencepiece

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

6

Fig. 4

Diagram of proposed mixture-of-experts neural caption translation model

A diagram of a mixture-of-experts neural caption translation model using our proposed approach is shown in Fig.
4. First, the source sentence is paraphrased into four different paraphrases used to train each of the expert model.
Then, each expert passes its abstract decoding state into
mixture model that will produce weights as the number ofexperts. The resulting weight distribution is a linear combination function between each expert’s output probability
distribution and the gating weight produced by the mixture
model.
4.

ensembling, improving the ensembled model’s robustness asa whole.
Furthermore, for the experiment, we followed the training, development, and test set-up of the WMT17 shared task
supplemented using our augmented training data. All results were scored using multeval [22] with lowercased andtokenized sentences. We used BLEU [23] and METEOR[24] as evaluation metrics.
Table 4

Experiments

The purpose of this experiment is to choose the approachthat is the most suitable for our VGP by comparing the scores
between Bahdanau et al.’s NMT baseline and several popular
multi-source NMTs.
4.1

Setup and Dataset Composition

We combined the generated 19k dataset described in Section 2.6 with the original crowdsourced 10k training dataset.
These 29k paraphrased datasets are combined with the original dataset, creating a 58k-triplet training dataset for each
operation. The 29k paraphrased training dataset functions
as a regularizer for the original dataset. These final data
will be used to train a mixture-of-experts translation model,
which will be described in the next section. The data will be
publicly available to augment the WMT17 dataset.
Based on our empirical observations, using paraphrased
data on the development and test dataset will reduce the performance of the overall system. When using paraphrased
data on development, the training objective becomes unclear,and the loss returned will not represent the actual loss. Given
that, we emphasize that using the paraphrased dataset in thetranslation step was done in the training steps in combination with the original dataset. In this stage, the paraphrases
acted as a regularizer of the source sentences and a way of

Paraphrasing model result in BLEU and METEOR

Operation
Deletion
Insertion
Reordering
Substitution

4.2

BLEU
53.0
56.1
47.2
59.6

METEOR
42.2
40.5
42.0
44.8

Model Specification and Implementation Details

We used single depth bidirectional LSTM with the size of
512 for all of our encoders in paraphrasing and translation
model. For the decoder in all model, we used unidirectional LSTM with linear attention over encoded representation [18]. For neural paraphrasing, we trained four encoderdecoder model with attention for each elementary paraphrase
operation. Note that although the architecture of the neuralparaphrasing model is similar with NMT, the input and output are monolingual. On the other hand, in NMT, to receive
five sentences as input, the multi-source NMT has five encoder and one decoder as a single model. At the same time,the multi-expert has five NMT model with one encoder and
one decoder, combined with the combination layer.
The paraphrasing and translation model are optimized
using Adam optimizer [25] with 1e-3 learning rate, while the
combination layer of the multi-expert model are trained with
RMSprop optimizer [26] with 1e-4 learning rate. We usedunweighted softmax cross entropy loss for all of our models.

EFFENDI et al.: LEVERAGING NEURAL CAPTION TRANSLATION WITH VISUALLY GROUNDED PARAPHRASE AUGMENTATION

7
Table 5

Performance of proposed neural caption translation in comparison with baseline.

Textual Model
Our NMT Baseline
Combining all data
Multi-source NMT
Uniform weighted ensemble
Mixture-of-experts ensemble

Table 6

Textual Model
Official WMT Baseline
Zhang et al. (2017)
Multimodal Model
Madhyastha et al. (2017)
Calixto et al. (2017)
Ma et al. (2017)
Helcl and Libovicky (2017)
Caglayan et al. (2017)

Table 7

Translation Model(Data)
Baseline
Single Paraphrase Model
Uniform Weight
Mixture-of-experts(Data)

Test 2016
BLEU METEOR
37.7
55.6
36.7
53.9
37.6
55.4
39.6
56.9
40.5
57.6

Test 2017
BLEU METEOR
30.1
49.7
29.6
47.7
30.1
49.4
31.4
50.7
32.5
51.3

Test COCO 2017
BLEU METEOR
25.0
44.6
25.1
43.7
24.4
44.3
26.7
46.0
28.0
46.8

Existing submission systems in official WMT17 shared task.

Test 2016
BLEU METEOR
32.5
52.5
-

Test 2017
BLEU METEOR
19.3
41.9
31.9
53.9

Test COCO 2017
BLEU METEOR
18.7
37.6
28.1
48.5

Test 2016
BLEU METEOR
41.3
59.2
36.8
53.1
41.0
60.4

Test 2017
BLEU METEOR
25.0
44.5
29.8
50.5
31.0
50.6
31.1
51.0
33.4
54.0

Test COCO 2017
BLEU METEOR
21.4
40.7
26.4
45.8
27.4
46.5
26.6
46.0
28.5
48.8

Examples of resulting sentences in mixture-of-experts model.

Type
Original
Original
Deletion
Insertion
Reordering
Substitution
Ensemble
Ensemble
TargetSentences
BLEU1
two motorcycles drive on a road along the river .
zwei motorradfahrer fahren auf einer straße entlang .
0.75
zwei motorräder fahren auf einer straße am fluss .
0.87
zwei motorradfahrer fahren auf einer straße am fluss .
0.84
zwei motorradfahrer fahren auf einer straße am fluss entlang .
0.95
zwei motorradfahrer fahren auf einer straße am flussufer .
0.82
zwei motorradfahrer fahren auf einer straße am fluss .
0.84
zwei motorräder fahren auf einer straße am fluss entlang .
0.97
zwei motorräder fahren auf einer straße dem fluss entlang .

In every increase of development loss, the learning rate is
decayed by half with the exception of the first five epochs.
We set a maximum of 7 decays for paraphrase models, and5 decays in translation models for training early stopping.
After the training stops, model with the lowest development
loss was selected was selected and used for decoding withthe beam size of five. All implementation was done with
Chainer deep learning framework version 3.0 [27].
4.3 Evaluation of Neural Paraphrase Model
Table 4 lists the scores of the paraphrases produced with our
automatic paraphrasing model. The substitution operation
produced the highest BLEU score, and the reordering operation produced the lowest BLEU score. We expected thisresult because the reordering operation sometimes includes
the changes of the active/passive properties of a sentence.
Overall, we believe this score is high enough to paraphrase
the remaining 19k WMT dataset.
4.4 Translation Model Results
Table 5 shows the performance of our proposed neural cap-tion translation. All of the results using our VGP outperformed the NMT baseline. No improvements were gained
from combining all of the data, which is the simplest form ofdata augmentation. This simple combination of data severs
the relation that existed between each paraphrase that mentioned the same image. Furthermore, we cannot be sure thateach source sentence has the same amount of paraphrases.
By considering these factors, we utilized multi-source NMT
and multi-expert NMT, which yielded better BLEU and METEOR scores.
This performance increase indicates that each expert
model is slightly different between each other, and worked
well in the uniform-weighted ensemble and mixture-ofexperts scenario. This model also outperformed the uniformweighted NMT in three cases. Moreover, the mixture-ofexperts model performed better in out-of-domain ambiguous MSCOCO test dataset, suggesting that overfitting did
not occur with such data augmentation. By applying tothese several models, we can conclude that our elementary
operation paraphrase is suitable to be used as a means for
ensembling.

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x8

Table 6 shows the current submission systems in theofficial WMT17 shared task whose submissions consist of
one textual model [28] and several multimodal models. Our
proposed approach outperformed the baseline in WMT17
by a 13.2 BLEU score margin. Our proposed model, eventhough it is textual, produced competitive results with othermultimodal models. The mixture-of-experts model outperformed several multimodal models, including another WMTsubmission [29]–[32]. Even in the out-of-domain dataset of
COCO 2017, the mixture-of-experts model also performed
reasonably well with a 28.0 BLEU score. Nevertheless, ourscore was almost the best score, proving that paraphrasing
the source side also helped our model work with unseen dataand prevented overfitting.
4.5 Discussion
To further analyze the contribution of the experts trainedon the original data and on the paraphrased data, we compared the translation process step-by-step in our proposed approach. The source sentence shown in Table 7 was translatedusing each baseline model (an expert), resulting five different translation hypotheses. Each expert has been trainedwith slightly different paraphrased source sentence. We calculated the BLEU1 scores for each hypothesis against thetarget, resulting the source-reordered expert model yielded
the best result between all experts.
The aim of the proposed mixture-of-experts model task
is to ensure that the best part of each model is kept, as well as
removing any noise or error that might occur in each model
result. As seen from the comparison of the German result
from the mixture-of-experts model with the target sentence,
the only difference is the word “am" in which the correct one
should be “dem".
In this example, in the deletion translation result, theword “motorräder" is decoded instead of “motorradfahrer".
Another example is the phrase “fluss entlang" which is only
found in the reordering translation result. Such quality ofeach expert model however, must be kept by the mixture
model by distributing the correct word for every word being
decoded. In conclusion, the final result of the ensemble ofexpert model combines the correct phrases in each expert
model.
Quantitatively, the mixture-of-experts model successfully retained the good features of the best performing 0.87
and 0.95 BLEU1 scores that were yielded in the sourcedeleted and source-reordered model results, resulting in a
0.97 BLEU1 score. This is a significant improvement compared with the BLEU1 score of the uniform weighted modelthat only increased to 0.84.
5.

Conclusions and Future Works

A single caption cannot represent all the information in image to which it refers. In this study, we elaborated an image
by various paraphrase operations. This enables us to incorporate additional knowledge from the image to the translation
process, without using the image itself, but diffused in a formof paraphrase.
We successfully generated visually grounded paraphrase (VGP) sentences of the WMT17 Multimodal Translation Task dataset by crowdsourcing and our evaluation showsthe effectiveness of our corpus creation method. We used
this corpus to construct an automatic paraphrase generation model, and employed it within various multi-source andmulti-expert approaches in NMT. In the future, we are interested on how the combinations of the elementary operations
can also be employed in this various NMT methods.
From the usage perspective, our results indicate thatour proposed paraphrase elementary operations are optimalfor ensembling, especially with multi-expert ensembling settings. We proved our hypothesis that regularizing models byparaphrasing the source sentences is effective. In the future,
we will further investigate various methods of incorporating visual information into NMT models. Furthermore, wewould also investigate a large-scale paraphrase augmentation
to enable ensembling in self-attentional NMT approach suchas the Transformer model [33].
6.

Acknowledgement

Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237.
References[1] G. Hirst, “Paraphrasing paraphrased,” Invited talk at the ACL International Workshop on Paraphrasing, 2003.
[2] C. Chu, M. Otani, and Y. Nakashima, “iparaphrasing: Extracting
visually grounded paraphrases via an image,” Proceedings of the 27th
International Conference on Computational Linguistics, pp.3479–
3492, Association for Computational Linguistics, 2018.
[3] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A
neural image caption generator,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.3156–3164, 2015.
[4] Q. Wu, D. Teney, P. Wang, C. Shen, A.R. Dick, and A. van den Hengel, “Visual question answering: A survey of methods and datasets,”
Computer Vision and Image Understanding, vol.163, pp.21–40,
2017.
[5] I. Calixto, Q. Liu, and N. Campbell, “Doubly-attentive decoder for
multi-modal neural machine translation,” ACL, 2017.
[6] D. Elliott, S. Frank, K. Sima’an, and L. Specia, “Multi30k: Multilingual english-german image descriptions,” Proceedings of the 5thWorkshop on Vision and Language, pp.70–74, 2016.
[7] T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollár, and C.L. Zitnick, “Microsoft coco: Common objects incontext,” European conference on computer vision, pp.740–755,
Springer, 2014.
[8] A. Prakash, S.A. Hasan, K. Lee, V. Datla, A. Qadir, J. Liu, andO. Farri, “Neural paraphrase generation with stacked residual lstm
networks,” Proceedings of COLING 2016, the 26th International
Conference on Computational Linguistics: Technical Papers, Osaka,
Japan, pp.2923–2934, The COLING 2016 Organizing Committee,
December 2016.
[9] R. Bhagat and E. Hovy, “What is a paraphrase?,” Computational
Linguistics, vol.39, no.3, pp.463–472, 2013.
[10] G.D. Shudong Huang, David Graff, Multiple-Translation Chinese
Corpus, Linguistic Data Consortium, University of Pennsylvania,
2002.
[11] M. Research, “Microsoft research paraphrase corpus,” mar 2005.

EFFENDI et al.: LEVERAGING NEURAL CAPTION TRANSLATION WITH VISUALLY GROUNDED PARAPHRASE AUGMENTATION

9[12] P. Resnik, O. Buzek, Y. Kronrod, C. Hu, A.J. Quinn, and B.B. Bederson, “Using targeted paraphrasing and monolingual crowdsourcing
to improve translation,” ACM Transactions on Intelligent Systems
and Technology (TIST), vol.4, no.3, p.38, 2013.
[13] B.A. Plummer, L. Wang, C.M. Cervantes, J.C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting regionto-phrase correspondences for richer image-to-sentence models,”
CoRR, vol.abs/1505.04870, 2015.
[14] D. Elliott, S. Frank, L. Barrault, F. Bougares, and L. Specia, “Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description,” Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark,September 2017.
[15] C. Dyer, V. Chahuneau, and N.A. Smith, “A simple, fast, and effective
reparameterization of IBM model 2,” Proceedings of the Conferenceof the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies (HLT-NAACL ’13),
Atlanta, GA, USA, 2013.
[16] K. Toutanova, D. Klein, C.D. Manning, and Y. Singer, “Feature-richpart-of-speech tagging with a cyclic dependency network,” Proceedings of the 2003 Conference of the North American Chapter ofthe Association for Computational Linguistics on Human Language
Technology (HLT-NAACL ’03), Stroudsburg, PA, USA, pp.173–180,
2003.
[17] J. Devlin, M.W. Chang, K. Lee, and K. Toutanova, “Bert: Pretraining of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
[18] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation
by jointly learning to align and translate,” CoRR, vol.abs/1409.0473,
2014.
[19] E. Nichols, F. Bond, D.S. Appling, and Y. Matsumoto, “Paraphrasing
training data for statistical machine translation,” Journal of Natural
Language Processing, vol.17, no.3, pp.3_101–3_122, 2010.
[20] B. Zoph and K. Knight, “Multi-source neural translation,” CoRR,
vol.abs/1601.00710, 2016.
[21] E. Garmash and C. Monz, “Ensemble learning for multi-source neural machine translation,” COLING, 2016.
[22] A.L. Jonathan Clark, Chris Dyer and N. Smith, “Better hypothesis
testing for statistical machine translation: Controlling for optimizer
instability,” Proceedings of the Association for Computational Lingustics, 2011.
[23] K. Papineni, S. Roukos, T. Ward, and W. jing Zhu, “Bleu: a method
for automatic evaluation of machine translation,” pp.311–318, 2002.
[24] A. Lavie and A. Agarwal, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” pp.65–72,
2005.
[25] D.P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol.abs/1412.6980, 2014.
[26] A. Graves, “Generating sequences with recurrent neural networks,”
CoRR, vol.abs/1308.0850, 2013.
[27] S. Tokui, K. Oono, S. Hido, and J. Clayton, “Chainer: a nextgeneration open source framework for deep learning,” Proceedingsof Workshop on Machine Learning Systems (LearningSys) in The
Twenty-ninth Annual Conference on Neural Information Processing
Systems (NIPS), 2015.
[28] J. Zhang, M. Utiyama, E. Sumita, G. Neubig, and S. Nakamura,“Nict-naist system for wmt17 multimodal translation task,” WMT,
2017.
[29] P.S. Madhyastha, J. Wang, and L. Specia, “Sheffield multimt: Using object posterior predictions for multimodal machine translation,”
Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, Copenhagen, Denmark, pp.470–476,
Association for Computational Linguistics, September 2017.
[30] I. Calixto, K.D. Chowdhury, and Q. Liu, “Dcu system report on the
wmt 2017 multi-modal machine translation task,” Proceedings of theConference of Machine Translation (WMT), 2017.
[31] M. Ma, D. Li, K. Zhao, and L. Huang, “Osu multimodal machinetranslation system report,” Proceedings of the Second Conference on
Machine Translation, Volume 2: Shared Task Papers, Copenhagen,Denmark, pp.465–469, Association for Computational Linguistics,
September 2017.
[32] J. Helcl and J. Libovický, “CUNI system for the WMT17 multimodaltranslation task,” CoRR, vol.abs/1707.04550, 2017.
[33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N.
Gomez, L.u. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems 30, ed. I. Guyon,
U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan,and R. Garnett, pp.5998–6008, Curran Associates, Inc., 2017.

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

10Johanes Effendi
received his B.S. degree in
Computer Science (cum laude) from Universitas
Indonesia, Indonesia in 2015 and his M.Eng. degree in 2018 from Nara Institute of Science andTechnology (NAIST), Japan. He is now a Ph.D.
Student at Augmented Human Communication
Laboratory, NAIST, Japan. He is a recipientof the Japanese Ministry of Education, Culture,
Sports, Science and Technology (MEXT) scholarship. His research interests include neural machine translation, paraphrasing, and natural language processing.

Sakriani Sakti
received her B.E. degree
in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In
2000, she received DAAD-Siemens Program
Asia 21st Century Award to study in Communication Technology, University of Ulm, Germany,and received her MSc degree in 2002. During
her thesis work, she worked with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between 20032009, she worked as a researcher at ATR SLC
Labs, Japan, and during 2006-2011, she worked as an expert researcher
at NICT SLC Groups, Japan. While working with ATR-NICT, Japan, she
continued her study (2005-2008) with Dialog Systems Group University
of Ulm, Germany, and received her PhD degree in 2008. She actively
involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003-2007), A-STAR and U-STAR (2006-2011). In 2009-2011,
she served as a visiting professor of Computer Science Department, University of Indonesia (UI), Indonesia. In 2011-2017, she was an assistantprofessor at the Augmented Human Communication Laboratory, NAIST,Japan. She served also as a visiting scientific researcher of INRIA ParisRocquencourt, France, in 2015-2016, under “JSPS Strategic Young Researcher Overseas Visits Program for Accelerating Brain Circulation”. Currently, she is a research associate professor at NAIST, as well as a researchscientist at RIKEN, Center for Advanced Intelligent Project AIP, Japan.
She is a member of JNS, SFN, ASJ, ISCA, IEICE and IEEE. She is also
the officer of ELRA/ISCA Special Interest Group on Under-resourced Languages (SIGUL) and a board Member of Spoken Language Technologies
for Under-Resourced Languages (SLTU). Her research interests include statistical pattern recognition, graphical modeling framework, deep learning,
multilingual speech recognition & synthesis, spoken language translation,affective dialog system, and cognitive communication.

Katsuhito Sudoh
is an associate professorof Nara Institute of Science and Technology anda visiting researcher at RIKEN, Center for Advanced Intelligent Project. He received a bachelor’s degree in engineering in 2000, and a master’s and Ph.D. degree in informatics in 2002
and 2015, respectively, from Kyoto University.
He was in NTT Communication Science Laboratories from 2002 to 2017. He currently works
on machine translation and natural language processing. He is a member of the Association for
Computational Linguistics (ACL), the Association of Natural Language
Processing (ANLP), the Information Processing Society of Japan (IPSJ)
and the Acoustical Society of Japan (ASJ)

Satoshi Nakamura
is Professor of Graduate
School of Science and Technology, Nara Institute of Science and Technology, Japan, Project
Leader of Tourism Information Analytics Teamof RIKEN, Center for Advanced Intelligence
Project AIP, Honorarprofessor of Karlsruhe Institute of Technology, Germany, and ATR Fellow. He received his B.S. from Kyoto Instituteof Technology in 1981 and Ph.D. from Kyoto
University in 1992. He was Associate Professor of Graduate School of Information Scienceat Nara Institute of Science and Technology in 1994-2000. He was Director of ATR Spoken Language Communication Research Laboratories
in 2000-2008 and Vice president of ATR in 2007-2008. He was Director General of Keihanna Research Laboratories and the Executive Directorof Knowledge Creating Communication Research Center, National Institute of Information and Communications Technology, Japan in 2009-2010.
He is currently Director of Augmented Human Communication laboratory
and a full professor of Graduate School of Information Science at Nara
Institute of Science and Technology. He is interested in modeling and systems of speech-to-speech translation and speech recognition. He is one of
the leaders of speech-to-speech translation research and has been serving
for various speech-to-speech translation research projects in the world including C-STAR, IWSLT and A-STAR. He received Yamashita ResearchAward, Kiyasu Award from the Information Processing Society of Japan,
Telecom System Award, AAMT Nagao Award, Docomo Mobile ScienceAward in 2007, ASJ Award for Distinguished Achievements in Acoustics.
He received the Commendation for Science and Technology by the Ministerof Education, Science and Technology, and the Commendation for Scienceand Technology by the Minister of Internal Affair and Communications. He
also received LREC Antonio Zampolli Award 2012. He has been Elected
Board Member of International Speech Communication Association, ISCA,
since June 2011, IEEE Signal Processing Magazine Editorial Board Member since April 2012, IEEE SPS Speech and Language Technical Committee
Member since 2013, and IEEE Fellow since 2016.