言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Enhancing Neural Machine Translation withImage-based Paraphrase Augmentation
Johanes Eﬀendi1
1

Sakriani Sakti1,2

Katsuhito Sudoh1,2Nara Institute of Science and Technology

2

Satoshi Nakamura1,2

RIKEN AIP

{johanes.effendi.ix4, ssakti, sudoh, s-nakamura}@is.naist.jp

1

Introduction

As defined by De Beaugrande and Dressler, aparaphrase is an approximate conceptual equivalence
among outwardly diﬀerent material [4]. In this research, we treat an image as a representation of sentence idea, which can be regarded as the basis ofparaphrasing. Furthermore, as paraphrasing to enable multi-source information in NMT is not muchinvestigated yet, in this study we explore the useof this image-based paraphrasing to leverage NMTquality.
Recently, the Second Conference on MachineTranslation (WMT17) accelerated a Multimodal
Machine Translation shared task that aimed to generate image descriptions in a target language. The
results from most submitted systems reveal that theadditional image features could only slightly contribute to system performance although the model
itself uses a lot of resource.
Here, we attempt to go in another direction inwhich we incorporate the image information by using image-based paraphrasing without using the image itself. Then, we applied this proposed way ofparaphrasing to build a paraphrase corpus in which
we developed a neural paraphrasing model. Furthermore, this initiates multi-expert neural machinetranslation (NMT) model that translates an image
caption without using the image itself, but instead
using a series of visual descriptions describing theimage.
In summary, the contributions of this work include:
1. Introduce a novel way of image-based paraphrasing.
2. Generate multiple paraphrase sentences of theWMT17 Multimodal Translation Task dataset
through crowdsourcing.
3. Utilize multi-expert translation in neural machine translation using our proposed paraphrase; and4. Improve the baseline used at WMT17 with a
13.2 BLEU score margin, which is close to thetop score that used a multimodal model.

2

Proposed Method

By having five caption paraphrases and theirtranslation, our proposed translation model consistsof two steps. The first step is to train five translation models based on each paraphrase as the source
sentence using the 56k dataset consisting of the original both 29k dataset and paraphrased dataset. Fiveof those models are trained against the same target
sentence. Each model is then regarded as an expert
model.
2.0.1

Uniform-weighted Ensemble Model

For this uniform weighted ensemble model, we ensembled five expert models by averaging each output layer probability distribution so every model wasweighted uniformly.
2.0.2

Mixture-of-experts Model

Next, we adopted the mixture-of-experts model todetermine the weight for each output layer probability distribution:
ct = tanh(LST Mhid ([h0 , h1 , ..., hn ])g0:i = sof tmax(Wgate .D(ct ) + bgate ).
Here, the expert model is implemented into a single
LSTM layer hid that receives the concatenated decoder hidden state output hn . A sof tmax function― 261 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Table 1: The performance of proposed neural caption translation in comparison with the baseline.
Textual Model
Our NMT Baseline
Combine all data
Uniform weighted ensemble
mixture-of-experts ensemble

Test 2016
BLEU METEOR
37.7
55.6
36.7
53.9
39.6
56.9
40.5
57.6

Test 2017
BLEU METEOR
30.1
49.7
29.6
47.7
31.4
50.7
32.5
51.3

Test COCO 2017
BLEU METEOR
25.0
44.6
25.1
43.7
26.7
46.0
28.0
46.8

Table 2: Existing submission systems in oﬃcial WMT17 shared task.
Textual ModelOﬃcial WMT Baseline
Zhang et al. (2017)
Multimodal Model
Caglayan et al. (2017)

Test 2016
BLEU METEOR
32.5
52.5
-Test 2017
BLEU METEOR
19.3
41.9
31.9
53.9

Test COCO 2017
BLEU METEOR
18.7
37.628.1
48.5

Test 2016
BLEU METEOR
41.0
60.4

Test 2017
BLEU METEOR
33.4
54.0

Test COCO 2017
BLEU METEOR
28.5
48.8

is then applied to obtain the weights of each expert
model’s output layer on . Assuming Wn is the weight

The collection was done through a crowdsourcingof the output layer from expert n. Then, the aggre-

we constructed our automatic neural paraphrasing

gated weight Wagg is a linear combination function

model based on partial data to generate the para-of each of those weights:
Wagg = g0 .W0 + g1 .W1 + ... + gn .Wn .
The resulting weight distribution in the output

phrase sentences of the full WMT17 dataset.

platform on the partial WMT17 dataset. After that,

3.2

layer is the linear combination function between eachPartially
crowdsource
WMT17 dataset

theThe WMT Multimodal Translation task data con-expert’s output probability distribution and gating

sists of 29000, 1014, and 1000 triplets respectively

weight produced by mixture model.

for the training, development and testing [5]. An3

Corpus Creation

3.1

Image-based paraphrase definition

out-of-domain dataset consisting 461 images was alsointroduced, which contains ambiguous verbs.
As paraphrasing the whole 29k triplet training

Our proposed paraphrase needs a set of sentences

dataset (29k training dataset) using crowdsourcing

with an image as its representation of idea. The caption of this dataset can be regarded as paraphrase,would not be eﬃcient in terms of cost and time, wecrowdsourced only 10k triplets of this dataset (10ksuch as done by Prakash et al. for their neural para-training dataset), along with the whole developmentphrase generation study [7] using MSCOCO dataset

and testing datasets.

[6]. While this is suﬃcient, we cannot define what

we paraphrased it using a neural paraphrase modelkind of operation has been done from the original

trained using the crowdsourced dataset.

For the remaining dataset,sentence to the paraphrase. Therefore, this mightcause the paraphrase to become noise to each other.

4

Experiments

To prevent this, we need to define a set of para-

4.1

Setup

phrase operation which covers all possible para-

We constructed four encoder-decoder LSTM mod-phrase variations. Based on Bhagat and Hovy’s idea

els with attention [1] for each elementary paraphraseof quasi-paraphrases [2], we constructed a paraphrase

operation. Each model has a bidirectional encoder

corpus based on various elementary operations which

and attentional decoder with one layer, 50% dropout

is grouped from their quasi-paraphrases into 4 el-ratio, and 512 hidden layer size.

ementary paraphrase operations such as: deletion,insertion, reordering, and substitution, and used theWMT17 Multimodal Translation Task dataset [5].

― 262 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

4.2Translation Model Results

jointly learning to align and translate. CoRR,Vol. abs/1409.0473, , 2014.

Table 1 shows the performance of our proposed
neural caption translation.

All results using our[2] Rahul Bhagat and Eduard Hovy.

multi-paraphrase outperformed the NMT baseline.

What is aparaphrase? Computational Linguistics, Vol. 39,

There are no improvements gained from combining

No. 3, pp. 463–472, 2013.

all data, which is the simplest form of data augmentation. This simple combination of data breaks the[3] Ozan Caglayan, Walid Aransa, Adrien Bardet,relation existed between each paraphrases that men-Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈ction the same image.

Barrault, Marc Masana, Luis Herranz, and Joostvan de Weijer.

The mixture-of-expert model performed better

LIUM-CVC submissions forthan uniform-weighted NMT in three cases. From

WMT17 multimodal translation task.

applying to these several models, we can conclude

Vol. abs/1707.04481, , 2017.

that our elementary operation paraphrase is suitable
to be used as a means for ensembling.

CoRR,[4] R. De Beaugrande and W.U. Dressler. Introduction to text linguistics. Longman linguistics li-Table 2 shows the current submission systems inbrary. Longman, 1981.

the oﬃcial WMT17 shared task which submissions
consist of one textual model [8] and a multimodal[5] D. Elliott, S. Frank, K. Sima’an, and L. Specia.

model. Our proposed approach outperformed theMulti30k: Multilingual english-german image de-baseline in WMT17 with a 13.2 BLEU score mar-scriptions. In Proceedings of the 5th Workshop ongin. Our proposed model, although it is textual,

Vision and Language, pp. 70–74, 2016.

could produce competitive result with other multimodal models except [3].

[6] Tsung-Yi Lin, Michael Maire, Serge Belongie,James Hays, Pietro Perona, Deva Ramanan, Pi-In this study, we elaborated an image by various

otr Dollár, and C Lawrence Zitnick. Microsoft
coco: Common objects in context. In Euro-paraphrase operations which enables us to incorpo-pean conference on computer vision, pp. 740–755.

rate additional knowledge from image to the translation process. Furthermore, we successfully generatedSpringer, 2014.

5Conclusions[7] Aaditya Prakash, Sadid A. Hasan, Kathy Lee,

multi-paraphrase sentences of the WMT17 Multi-Vivek Datla, Ashequl Qadir, Joey Liu, andmodal Translation Task dataset through crowdsourc-

Oladimeji Farri. Neural paraphrase generation

ing. We constructed an automatic paraphrase gen-with stacked residual lstm networks.

eration model, and used it with the multi-expert ap-

ceedings of COLING 2016, the 26th Interna-

proach within NMT. The results indicate that ourtional Conference on Computational Linguistics:

proposed paraphrase elementary operations are best

Technical Papers, pp. 2923–2934, Osaka, Japan,

to be used for multi-expert ensembling settings.

6December 2016. The COLING 2016 Organizing
Committee.

Acknowledgement
Part of this work was supported by JSPS

KAKENHI

Grant

Numbers

JP17H06101[8] Jingyi Zhang, Masao Utiyama, Eiichiro Sumita,

andJP17K00237.

Graham Neubig, and Satoshi Nakamura. Nictnaist system for wmt17 multimodal translation
task. In WMT, 2017.

References[1] Dzmitry Bahdanau,

In Pro-Kyunghyun Cho,andYoshua Bengio. Neural machine translation by― 263 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.