Enhancing Neural Machine Translation withImage-based Paraphrase Augmentation
Johanes Eﬀendi1
1

Sakriani Sakti1,2

Katsuhito Sudoh1,2Nara Institute of Science and Technology

2

Satoshi Nakamura1,2

RIKEN AIP

{johanes.effendi.ix4, ssakti, sudoh, s-nakamura}@is.naist.jp

1

Introduction

WMT17 Multimodal Translation Task dataset

As defined by De Beaugrande and Dressler, athrough crowdsourcing.
3. Utilize multi-expert translation in neural ma-paraphrase is an approximate conceptual equivalencechine translation using our proposed para-among outwardly diﬀerent material [4]. In this research, we treat an image as a representation of sen-phrase; and4. Improve the baseline used at WMT17 with atence idea, which can be regarded as the basis of

13.2 BLEU score margin, which is close to theparaphrasing. Furthermore, as paraphrasing to en-

top score that used a multimodal model.

able multi-source information in NMT is not muchinvestigated yet, in this study we explore the useof this image-based paraphrasing to leverage NMT2

Proposed Method
By having five caption paraphrases and theirquality.
Recently, the Second Conference on Machinetranslation, our proposed translation model consists

Translation (WMT17) accelerated a Multimodal
Machine Translation shared task that aimed to gen-of two steps. The first step is to train five transla-

erate image descriptions in a target language. The

sentence using the 56k dataset consisting of the orig-

results from most submitted systems reveal that theinal both 29k dataset and paraphrased dataset. Fiveadditional image features could only slightly con-of those models are trained against the same target

tribute to system performance although the model

sentence. Each model is then regarded as an expert

itself uses a lot of resource.

model.

Here, we attempt to go in another direction intion models based on each paraphrase as the source

2.0.1

Uniform-weighted Ensemble Model

which we incorporate the image information by us-For this uniform weighted ensemble model, we en-ing image-based paraphrasing without using the image itself. Then, we applied this proposed way ofsembled five expert models by averaging each output layer probability distribution so every model wasparaphrasing to build a paraphrase corpus in whichweighted uniformly.

we developed a neural paraphrasing model.

Fur-thermore, this initiates multi-expert neural machine

2.0.2

Mixture-of-experts Modeltranslation (NMT) model that translates an image

Next, we adopted the mixture-of-experts model tocaption without using the image itself, but instead

determine the weight for each output layer probabil-using a series of visual descriptions describing theity distribution:

image.
In summary, the contributions of this work include:
1. Introduce a novel way of image-based paraphrasing.
2. Generate multiple paraphrase sentences of thect = tanh(LST Mhid ([h0 , h1 , ..., hn ])g0:i = sof tmax(Wgate .D(ct ) + bgate ).
Here, the expert model is implemented into a single
LSTM layer hid that receives the concatenated decoder hidden state output hn . A sof tmax function

Table 1: The performance of proposed neural caption translation in comparison with the baseline.
Textual Model
Our NMT Baseline
Combine all data
Uniform weighted ensemble
mixture-of-experts ensemble

Test 2016
BLEU METEOR
37.7
55.6
36.7
53.939.6
56.9
40.5
57.6

Test 2017
BLEU METEOR
30.1
49.7
29.6
47.7
31.4
50.7
32.5
51.3

Test COCO 2017
BLEU METEOR
25.0
44.6
25.1
43.7
26.7
46.0
28.0
46.8

Table 2: Existing submission systems in oﬃcial WMT17 shared task.
Textual ModelOﬃcial WMT Baseline
Zhang et al. (2017)
Multimodal Model
Caglayan et al. (2017)

Test 2016
BLEU METEOR
32.5
52.5
-Test 2017
BLEU METEOR
19.3
41.9
31.9
53.9

Test COCO 2017
BLEU METEOR
18.7
37.6
28.1
48.5

Test 2016
BLEU METEOR
41.0
60.4

Test 2017
BLEU METEOR
33.4
54.0

Test COCO 2017
BLEU METEOR
28.5
48.8

is then applied to obtain the weights of each expert
model’s output layer on . Assuming Wn is the weight

The collection was done through a crowdsourcingof the output layer from expert n. Then, the aggre-

we constructed our automatic neural paraphrasing

gated weight Wagg is a linear combination function

model based on partial data to generate the para-of each of those weights:
Wagg = g0 .W0 + g1 .W1 + ... + gn .Wn .
The resulting weight distribution in the output

phrase sentences of the full WMT17 dataset.

platform on the partial WMT17 dataset. After that,

3.2

layer is the linear combination function between each

3Corpus Creation

3.1

Image-based paraphrase definition

theThe WMT Multimodal Translation task data con-expert’s output probability distribution and gating
weight produced by mixture model.

Partially
crowdsource
WMT17 dataset

sists of 29000, 1014, and 1000 triplets respectively
for the training, development and testing [5]. An
out-of-domain dataset consisting 461 images was also
introduced, which contains ambiguous verbs.
As paraphrasing the whole 29k triplet training

Our proposed paraphrase needs a set of sentences

dataset (29k training dataset) using crowdsourcing

with an image as its representation of idea. The caption of this dataset can be regarded as paraphrase,would not be eﬃcient in terms of cost and time, wecrowdsourced only 10k triplets of this dataset (10ksuch as done by Prakash et al. for their neural para-training dataset), along with the whole development

phrase generation study [7] using MSCOCO dataset

and testing datasets.

[6]. While this is suﬃcient, we cannot define what

we paraphrased it using a neural paraphrase modelkind of operation has been done from the original

trained using the crowdsourced dataset.

For the remaining dataset,

sentence to the paraphrase. Therefore, this mightcause the paraphrase to become noise to each other.

4Experiments

To prevent this, we need to define a set of para-

4.1

Setup

phrase operation which covers all possible para-

We constructed four encoder-decoder LSTM mod-phrase variations. Based on Bhagat and Hovy’s ideaels with attention [1] for each elementary paraphraseof quasi-paraphrases [2], we constructed a paraphrase

operation. Each model has a bidirectional encoder

corpus based on various elementary operations which

and attentional decoder with one layer, 50% dropout

is grouped from their quasi-paraphrases into 4 el-ratio, and 512 hidden layer size.

ementary paraphrase operations such as: deletion,insertion, reordering, and substitution, and used theWMT17 Multimodal Translation Task dataset [5].

4.2Translation Model Results

jointly learning to align and translate. CoRR,Table 1 shows the performance of our proposed
neural caption translation.

All results using ourmulti-paraphrase outperformed the NMT baseline.
There are no improvements gained from combining

Vol. abs/1409.0473, , 2014.
[2] Rahul Bhagat and Eduard Hovy.

What is aparaphrase? Computational Linguistics, Vol. 39,
No. 3, pp. 463–472, 2013.

all data, which is the simplest form of data augmentation. This simple combination of data breaks the[3] Ozan Caglayan, Walid Aransa, Adrien Bardet,relation existed between each paraphrases that men-Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈ction the same image.

Barrault, Marc Masana, Luis Herranz, and Joost

The mixture-of-expert model performed bettervan de Weijer.

LIUM-CVC submissions forthan uniform-weighted NMT in three cases. From

WMT17 multimodal translation task.

applying to these several models, we can conclude

Vol. abs/1707.04481, , 2017.

that our elementary operation paraphrase is suitable
to be used as a means for ensembling.
Table 2 shows the current submission systems inCoRR,[4] R. De Beaugrande and W.U. Dressler. Introduction to text linguistics. Longman linguistics library. Longman, 1981.

the oﬃcial WMT17 shared task which submissions
consist of one textual model [8] and a multimodal[5] D. Elliott, S. Frank, K. Sima’an, and L. Specia.

model. Our proposed approach outperformed theMulti30k: Multilingual english-german image de-baseline in WMT17 with a 13.2 BLEU score mar-scriptions. In Proceedings of the 5th Workshop ongin. Our proposed model, although it is textual,

Vision and Language, pp. 70–74, 2016.

could produce competitive result with other multimodal models except [3].

[6] Tsung-Yi Lin, Michael Maire, Serge Belongie,
James Hays, Pietro Perona, Deva Ramanan, Pi-In this study, we elaborated an image by variousotr Dollár, and C Lawrence Zitnick. Microsoft
coco: Common objects in context. In Euro-paraphrase operations which enables us to incorpo-pean conference on computer vision, pp. 740–755.

rate additional knowledge from image to the translation process. Furthermore, we successfully generated

Springer, 2014.

5Conclusions

multi-paraphrase sentences of the WMT17 Multimodal Translation Task dataset through crowdsourcing. We constructed an automatic paraphrase generation model, and used it with the multi-expert approach within NMT. The results indicate that ourproposed paraphrase elementary operations are best
to be used for multi-expert ensembling settings.

6Numbers

JP17H06101

andJP17K00237.

with stacked residual lstm networks.

In Pro-ceedings of COLING 2016, the 26th International Conference on Computational Linguistics:
Technical Papers, pp. 2923–2934, Osaka, Japan,[8] Jingyi Zhang, Masao Utiyama, Eiichiro Sumita,
Graham Neubig, and Satoshi Nakamura. Nictnaist system for wmt17 multimodal translation
task. In WMT, 2017.

References[1] Dzmitry Bahdanau,Oladimeji Farri. Neural paraphrase generation

Committee.

Part of this work was supported by JSPS
Grant

Vivek Datla, Ashequl Qadir, Joey Liu, andDecember 2016. The COLING 2016 Organizing

Acknowledgement

KAKENHI[7] Aaditya Prakash, Sadid A. Hasan, Kathy Lee,

Kyunghyun Cho,andYoshua Bengio. Neural machine translation by