Multi-paraphrase Augmentation
to Leverage Neural Caption Translation
Johanes Eﬀendi1, Sakriani Sak21,2, Katsuhito Sudoh1,2, Satoshi Nakamura1,2{johanes.eﬀendi.ix4,ssak2,sudoh,s-nakamura}@is.naist.jp
1Nara Ins)tute of Science and Technology, Japan

2RIKEN, Center for Advanced Intelligence Project AIP, Japan© NAIST 2018

Outline• Introduction• Image-based Paraphrasing• Proposed Idea
• Corpus Creation• Experimental Settings• Experiment Results• Conclusion and Future Works

Mul$-paraphrase Augmenta$on to Leverage Neural Cap$on Transla$on – IWSLT 20182 / 36Introduction

Mul$-paraphrase Augmenta$on to Leverage Neural Cap$on Transla$on – IWSLT 20183 / 36

Machine Translation• Text-to-text translation• Parallel text dataset• What about similar sentences?
• Concept-to-concept translation– Mapping latent representation into another latent representation

Source sentence ENMT

Target sentence DE(this latent representation can berepresented into different sentences)
Mul$-paraphrase Augmenta$on to Leverage Neural Cap$on Transla$on – IWSLT 20184 / 36

Multiple sources or references• Multiple sources into one target
Source sentence
Source sentence
Source sentence
Source sentence
Source sentence

NMT

Target sentence

NMT

Target sentence
Target sentence
Target sentence
Target sentence
Target sentence• Multiple references

Source sentence

Mul$-paraphrase Augmenta$on to Leverage Neural Cap$on Transla$on – IWSLT 2018

5 / 36

Multimodal NMT• WMT17 Multimodal Translation Task– Translate a caption with the image provided• Based on concept-to-concept idea:

Source sentence ENMT

Target sentence DE(this latent representation can berepresented into different sentences)

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 2018

6 / 36

Multimodal NMT (cont.)• Common approach:– Incorporate latent image representation in various NMT components• Caglayan et al. (2016,2017), Calixto et al. (2017)source_sent

Enc

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 2018

Dec

target_sent

7 / 36

Multimodal NMT (cont.)• Zhang et al. (2017) integrated similar image information asadditional input

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 20188 / 36

Difficulties with Multimodal NMT• Powerful, but complicated• The image encoder (VGG, ResNet) are resource intensive• Difficulties combining latent spaces from different modalities– Not all information is useful for translation• Improvement reached might not be as rewarding as the effort

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 2018

9 / 36

Image-based Paraphrasing

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201810 / 36Image-based Paraphrasing• Represent image as texts

NMTSource sentenceCommon approachCaption 1
Caption 2
CapIon 3
Caption 4
Caption 5

NMT

Our proposed approach• Image-based paraphrase / Visually Grounded Paraphrase• Rewrite source sentence with image as basis of paraphrasing• Enable multi-source information in NMT

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201811 / 36

Difference with common MT paraphrases• Paraphrasing to elaborate source language data
• Augment the dataset size in SMT• (Nichols et al., 2010, He et al., 2011)• Recent work: only reordering and substitution are used
• In this work: with image as the basis of paraphrasing, deletion
and insertion of information is possible

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201812 / 36

How to generate paraphrase from image?
• If random paraphrase is inputted, it might become noisy to eachother– How many variations?

• Bhagat and Hovy (2013) studied on how many paraphrase
operations language can possibly make– 25 quasi-paraphrases– Survey the occurrence of each quasi-paraphrases in Microsoft ResearchParaphrase Corpus (MSR Corpus)

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201813 / 36

Quasi Paraphrases - Frequency
No
name%Freq in MSR
1 Synonym substitution
192 Antonym substitution
03 Converse substitution
04 Change of voice
15 Change of person
1Pronoun/Co-referent
6 substitution
17 Repetition/Ellipsis
48 Function word variations
30
9 Actor/Action Substitution
0Verb/Semantic-role noun
10 substitution

Manipulator/Device
11 substitution
General/Specific
12 substitution
13 Metaphor substitution

0031NoName%Freq in MSR
14 Part/Whole substitution
015 Verb/Noun conversion
3Verb/Adjective
16 conversion
017 Verb/Adverb conversion
0Noun/Adjective
18 conversion
0Verb-preposition/Noun
19 substitution
020 Change of tense
121 Change of aspect
022 Change of modality
023 Semantic implication
4
Approximate numerical
24 equivalences
225 External knowledge
32

Some quasi-paraphrases have
low frequency in MSR Corpus

Bhagat, R., & Hovy, E. (2013). What Is a Paraphrase? Computational Linguistics, 39(3), 463-472.
Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201814 / 36

Simplify into four elements• Some quasi-paraphrases have low frequencies• Some quasi-paraphrases are too fine-grained• Having 25 kinds of input sentences might be too difficult

25 kinds ofparaphrases?

Source sentence
Source sentence
Source sentence
Source sentence
Source sentence

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 2018

NMT

Target sentence

15 / 36

Simplify into four elements (cont.)• We grouped it into four elementary operations:– Deletion– Insertion– Reordering– Substitution

Simplify

OriginalDeleNon
InserNon
Reordering
SubsNtuNon

NMT

Target sentence• Each source sentence now paraphrased into four paraphrase

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201816 / 36Proposed Idea

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201817 / 36

Two PossibiliGes on Data Usage• Several paraphrase as input enables two scenario:– data augmentation– multi-source• Simple data augmentation == combining all dataparaphrase• Multi-source: separate dataset per paraphrase operationparaphrase

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201818 / 36

Determining Integration Point• Multi-source combination:– preserves relation between paraphrases– on which NMT stage?

• Decoding phase and result space for this work
• Other phase is omitted for further study
Featurecombinationand selection

Variable length,
different alignment
problemCombining
attention andencoded rep.

Expertensembling

Featurespace

Encodingphase

Decodingphase

Resultspace(this study)

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201819 / 36

Multi-source NMT• Modification from Zoph and Knight (2016)– Used for purely NMT task {Fr, De} -> En– In this research, used for monolingual input or pair
– Investigate various combination functions

Could beParaphrasing
Each otherMulti-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201820 / 36

Multi Expert NMT• Garmash et al. (2016) proposed that using decoder hidden state
to predict weight combination yields better result.
– Combination of several encoder-decoder model regarded as expert– Used for paraphrased source sentences in this study– Mixture model predicts weight for every model– Final aggregated output weight is the linear combination:!"## = %& !' + %) !) + ⋯ + %+ !+Could beParaphrasing
Each other

Mul$-paraphrase Augmenta$on to Leverage Neural Cap$on Transla$on – IWSLT 201821 / 36Overall System: Paraphrase + Translation

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201822 / 36

Corpus Creation

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201823 / 36

Multi-paraphrase corpus creation• Paraphrase WMT 2017 Multimodal Translation corpus– using crowdsourcing• Using image as the basis of paraphrasing, the crowdworker
paraphrase– Original -> {deletion, insertion, reordering, substitution}• 3 months; 201 workers; 16 countries– English speaking countries, or at least English as second language• Crowdsourced 10k of training data, dev, testCaption: A little gray dog jumps over a small hurdle.
Deletion: A little gray dog jumps over a hurdle.
Insertion : A little gray dog jumps over a small hurdle successfully.
Substitution : A little gray dog pass over a small hurdle.
Reordering : Over a small hurdle, a little gray dog jumps.

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201824 / 36

Generating the remaining paraphrases• WMT dataset size is 29k pair
• Crowdsourcing successfully paraphrased 10k sentences• Trained LSTM Encoder Decoder models for each paraphrase
operation– Using 10k crowdsourced paraphrase– To generate remaining 19k paraphrase

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201825 / 36Experimental Settings

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201826 / 36

Data Composition• Combined paraphrased dataset with original dataset– Resulting 58k training data for each operation– The paraphrased data works as regularizer• For dev and test dataset:– For paraphrasing : paraphrased dataset is used
– For translation: original dataset is used

For each expert translation model:
Training data 58k
Original data 29k

Paraphrased data 29k
Crowdsourced 10k

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 2018

Generated 19k

27 / 36Experiment Results

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201828 / 36Experiment Result - Paraphrasing• BLEU and METEOR are actually metrics for translation• In this result, it is used to measure the performance ofparaphrasing model– To give some sense of the paraphrasing performance
Operation

BLEU

METEOR

Deletion

53.0

42.2

Insertion

56.1

40.5

Reordering

47.2

42.0

Substitution

59.6

44.8

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201829 / 36Experiment Result - Translation
Model Name

Test 2016BLEU

METEOR

Our NMT Baseline

37.7

Combine all data

Test 2017

Test COCO 2017

BLEU METEORBLEUMETEOR

55.6

30.1

49.7

25.0

44.6

36.7

53.9

29.6

47.7

25.1

43.7

Multi-source

37.055.0

30.8

49.6

25.0

44.3

Uniform weighted

39.6

56.9

31.4

50.7

26.7

46.0

Mixture of Expert

40.5

57.6

32.5

51.3

28.0

46.8

• Combining all data shows decrease in performance• Mixture of Expert yields the best result• Test COCO 2017 (ambiguous situation)

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201830 / 36

Result comparison with other models
Model NameType

Test 2016

Test 2017

Test COCO 2017

BLEU

MTR

BLEU

MTR

BLEU

MTR

Official WMT Baseline

Textual

32.5

52.5

19.3

41.9

18.7

37.6

Zhang et al. (2017)

Textual--

31.9

53.9

28.1

48.5

Madhyastha et al. (2017)

Multimodal--

25.0

44.5

21.4

40.7

Calixto et al. (2017)

Multimodal

41.3

59.2

29.8

50.5

26.4

45.8

Ma et al. (2017)

Multimodal

--

31.0

50.6

27.4

46.5

Helcl and Libovicky (2017)

Multimodal

36.8

53.1

31.1

51.0

26.6

46.0

Caglayan et al. (2017)

Multimodal

41.0

60.4

33.4

54.0

28.5

48.8

(Ours) Mixture-of-Expert

Textual

40.5

57.6

32.5

51.3

28.0

46.8

• Outperform almost all models, except one• Works in par with other multimodal model– Only using textual information
Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201831 / 36

Result Example - Unsuccessful
Type(Data)
Translation
ModelBaseline/NMTSingle
Paraphrased
ModelSource Sentences

Original
Typea little girl climbing metal rope cables wearing a long pink skirt and black t-shirt .
Correct!

Target Sentences

Correct!

BLEU-1

Original

ein kleines mädchen klettert metall an einem seil , das einen langenrosafarbenen rock und einem schwarzen t-shirt klettert .

0.9

Deletion

ein kleines mädchen klettert metall seilen und einem schwarzen t-shirt
klettert .

0.47

Insertion

ein kleines mädchen klettert mit einem langen rosafarbenen rock aneinem seil hoch .

0.61

Reordering

ein kleines mädchen in einem langen pinkfarbenen rock undschwarzem t-shirt klettert metall an einem seil .

0.76

Substitution ein kleines mädchen klettert an einem seil seil seilen und einem
schwarzen t-shirt klettert .

0.63

Uniform
WeightEnsembleein kleines mädchen
klettert an einem seil seil und einem schwarzen tFinal hypothesis is quite different with target
shirt hoch .

0.59

Mixture-ofExpert

Ensembleein kleines mädchen klettert metall mit einem langen rosafarbenen
Isn’t produced in any hypothesis
rock und einem schwarzen t-shirt .

0.67(Data)

Targetein kleines mädchen , das an metallseilen hochklettert und einenrosafarbenen
Multi-paraphrase Augmentation to Leverage Neural langen
Caption Translation– IWSLT 2018 rock und ein schwarzes t-shirt trägt .

32 / 36

Result Example - Successful
Type(Data)

OriginalTranslation
ModelBaseline/NMTType

Source Sentences
The word“dem“motorradfahrer”
shouldtwo motorcycles
drive on a road along
the river. fluss“ isbe “motorräder fahren“ missing
Target Sentences

BLEU-1

Original

zwei motorradfahrer fahren auf einer straße entlang .

0.75

Deletionzwei motorräder
fahrenappears
auf einer on
straße am fluss .
“fluss“

0.87

SingleInsertion
Paraphrased
Reordering
Model
Substitution

paraphrased
zwei motorradfahrer
fahren aufmodel
einer straße am fluss .

0.84

Correctphrasezwei motorradfahrer
fahrenauf einer straße am fluss entlang .

0.95

zwei motorradfahrer fahren auf einer straße am flussufer .

0.82

Uniform
WeightEnsemble

zwei motorradfahrer fahren auf einer straße am fluss .

0.84

Mixture-ofExpert

Ensemblezwei motorräder fahren auf einer straße am fluss entlang .

0.97(Data)

Targetzwei motorräder fahren auf einer straße dem fluss entlang .

-Corrected in final result
Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 2018

Corrected in final result
33 / 36

Conclusions and Future WorksMulti-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201834 / 36

Conclusions• A single caption cannot represents all the information of theimage to which it refers to• Generated multi-paraphrase of the WMT17 Multimodal
Translation Task
– Partially crowdsourcing with image as the basis of paraphrasing– Neural paraphrasing to complete the paraphrasing in semi-supervised way• Proposed a textual model, in which the image information is notincluded in the model, but diffused in form of paraphrased
caption• +2.4 BLEU improvement over our NMT baseline

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201835 / 36

Future Works• Try different combination strategies/integration point• Investigate this proposed approach for another usage– Not limited for image caption translation• Further investigate various methods of incorporating visual
information

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201836 / 36• Thanks for your attention!

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201837 / 36References• Rahul Bhagat and Eduard Hovy. What is aparaphrase? ComputaHonal Lin- guisHcs, 39(3):463–472, 2013.
• Philipp Koehn and Rebecca Knowles. Six challenges
for neural machine trans- laHon. In Proceedings ofthe First Workshop on Neural Machine TranslaHon,pages 28–39. AssociaHon for ComputaHonal
LinguisHcs, 2017.
• Yee Seng Chan, Hwee Tou Ng, and David Chiang.
Word sense disambiguaHon improves staHsHcal
machine translaHon. In ACL, 2007.
• Marine Carpuat and Dekai Wu. Improving staHsHcal
machine translaHon using word sense
disambiguaHon. In In Proceedings of the 2007 Joint
Conference on Empirical Methods in Natural
Language Processing and ComputaHonal NaturalLanguage Learning, pages 61–72, 2007.
• Katharina Wäschle and Stefan Riezler. IntegraHng alarge, monolingual corpus as translaHon memory
into staHsHcal machine translaHon. In Proceedingsof the 18th Annual Conference of the European
AssociaHon for Machine TranslaHon, 2015.
• Çaglar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun
Cho, Loı ̈c Barrault, Huei- Chi Lin, Fethi Bougares,
Holger Schwenk, and Yoshua Bengio. On using
mono- lingual corpora in neural machine translaHon.
CoRR, abs/1503.03535, 2015.
• Rico Sennrich, Barry Haddow, and Alexandra Birch.
Improving neural machine translaHon models withmonolingual data. In Proceedings of the 54th Annual
MeeHng of the AssociaHon for ComputaHonal
LinguisHcs (Volume 1: Long Pa- pers), pages 86–96.
AssociaHon for ComputaHonal LinguisHcs, 2016.

••••

•

•

•

42Rajen Chalerjee, Maleo Negri, Marco Turchi,
Marcello Federico, Lucia Specia, and Frédéric Blain.
Guiding neural machine translaHon decoding withexternal knowledge, 01 2017.
Liang Zhou, Chin-Yew Lin, and Eduard Hovy. ReevaluaHng machine trans- laHon results withparaphrase support. In Proceedings of the 2006
Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 77–84,
Stroudsburg, PA, USA, 2006.
Santanu Pal, Pintu Lohar, and Sudip Kumar Naskar.
Role of paraphrases in pb-smt. In Proceedings of the15th InternaHonal Conference on ComputaHonal
LinguisHcs and Intelligent Text Processing - Volume8404, CICLing 2014, pages 242–253, Berlin,Heidelberg, 2014. Springer-Verlag.
NiHn Madnani and Bonnie J. Dorr. GeneraHng
targeted paraphrases for im- proved translaHon.
ACM Trans. Intell. Syst. Technol., 4(3):40:1–40:25,July 2013.
Eric Nichols, Francis Bond, D Scol Appling, and YujiMatsumoto. Paraphrasing training data for staHsHcal
machine translaHon. Journal of Natural Language
Processing, 17(3):3 101–3 122, 2010.
Wei He, Shiqi Zhao, Haifeng Wang, and Ting Liu.
Enriching smt training data via paraphrasing. In
IJCNLP, 2011.
Anabela Barreiro. SPIDER: A System for Paraphrasing
in Document EdiHng and Revision — Applicability inMachine TranslaHon Pre-ediHng, pages 365– 376.
Springer Berlin Heidelberg, Berlin, Heidelberg, 2011.
Chris Callison-Burch, Philipp Koehn, and MilesOsborne. Improved staHsHcal machine translaHon

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 2018•••••using paraphrases. In Proceedings of the Main
Conference on Human Language Technology
Conference of the North American Chapter of theAssociaHon of ComputaHonal LinguisHcs (HLT-NAACL’06), pages 17–24, Stroudsburg, PA, USA, 2006.
43Michel Simard, Cyril Goule, and Pierre Isabelle.
StaHsHcal phrase-based post- ediHng. In Proceedingsof the Main Conference on Human Language
Technology Conference of the North American
Chapter of the AssociaHon of ComputaHonal
LinguisHcs (HLT-NAACL ’07), pages 508–515,Rochester, NY, USA, 2007.
Ekaterina Garmash and Christof Monz. Ensemble
learning for mulH-source neural machinetranslaHon. In COLING, 2016.
Barret Zoph and Kevin Knight. MulH source neuraltranslaHon. arXiv preprint arXiv:1601.00710, 2016.
Ozan Caglayan, Walid Aransa, Adrien Bardet,Mercedes Garcı ́a-Mart́nez, Fethi Bougares, Loı ̈cBarrault, Marc Masana, Luis Herranz, and Joost vande Wei- jer. LIUM-CVC submissions for WMT17mulHmodal translaHon task. CoRR, abs/1707.04481,2017.
Jingyi Zhang, Masao UHyama, Eiichiro Sumita,
Graham Neubig, and Satoshi Nakamura. Nict-naist
system for wmt17 mulHmodal translaHon task. In
WMT, 2017.
Jindrich Helcl and Jindrich Libovický. CUNI system for
the WMT17 mulHmodal translaHon task. CoRR,
abs/1707.04550, 2017.

38 / 36

Multi-paraphrase Augmentation to Leverage Neural Caption Translation – IWSLT 201839 / 36