Enhancing Neural Machine Translation with
Johanes Effendi¹, Sakriani Sakti¹,², Katsuhito Sudoh¹,², Satoshi Nakamura¹,² {johanes.effendi.ix4,ssakti,sudoh,s-nakamura}@is.naist.jp 1Nara Institute of Science and Technology ²RIKEN, Center for Advanced Intelligence Project AIP ● Introduction • Image-based Paraphrasing •   Proposed Idea • Corpus Creation . Experimental Settings . Experiment Results . Conclusion and Future Works 2 Introduction m
. Parallel text dataset . What about similar sentences? . Concept-to-concept translation — Mapping latent representation into another latent representation (this latent representation can be represented into different sentences) 4
. Multiple sources into one target Source sentence Source sentence Source sentence Source sentence Source sentence Target sentence . Multiple references Source sentence Target sentence Target sentence Target sentence Target sentence Target sentence
WMT17 Multimodal Translation Task • — Translate a caption with the image provided Based on concept-to-concept idea: . (this latent representation can be represented into different sentences) 6
. Common approach: — Incorporate latent image representation in various NMT components •  Caglayan et al. (2016,2017), Calixto et al. (2017) 7
. Zhang et al. (2017) integrated similar image information as additional input 8
• Powerful, but complicated . The image encoder (VGG, ResNet) are resource intensive ● Difficulties combining latent spaces from different modalities — Not all information is useful for translation •   Improvement reached might not be as rewarding as the effort Image-based Paraphrasing INAISTI 10
. Represent image as texts Common approach . Image-based paraphrase . . Rewrite source sentence with image as basis of paraphrasing Enable multi-source information in NMT Caption 1 Caption 2 Caption 3 Caption 4 Caption 5 Our proposed approach
Paraphrasing to elaborate source language data . •   Augment the dataset size in SMT •   (Nichols et al., 2010, He et al., 2011) •  Recent work: only reordering and substitution are used •  In this work: with image as the basis of paraphrasing, deletion and insertion of information is possible
•   If random paraphrase is inputted, it might become noisy to each other – How many variations? •   Bhagat and Hovy (2013) studied on how many paraphrase operations language can possibly make —  25 classes of quasi-paraphrases — Survey the occurrence of each classes in Microsoft Research Paraphrase Corpus (MSR Corpus)
Bhagat, R., & Hovy, E. (2013). What Is a Paraphrase? Computational Linguistics, 39(3), 463-472. Some quasi-paraphrases have low frequency in MSR Corpus
Some quasi-paraphrase classes: . – have low frequencies — are too fine-grained . Having 25 kinds of input sentences might be too difficult 25 kinds of paraphrases? Source sentence Source sentence Source sentence Source sentence Source sentence Target sentence
. We grouped it into four elementary operations: – Deletion – Insertion – Reordering – Substitution Target sentence Each source sentence now paraphrased into four paraphrase . Proposed Idea 17
● Several paraphrase as input enables two scenario: — data augmentation — multi-source Simple data augmentation == combining all data . •   Multi-source: separate dataset per paraphrase operation
NAIS

• Multi-source combination: — preserves relation between paraphrases – on which NMT stage? . Decoding phase and result space for this work . Other phase is omitted for further study
. Change input into paraphrases . Multi-source NMT – Zoph and Knight (2016) .
Multi-expert NMT
— Garmash and Mond (2016) — Final aggregated output weight is the linear combination: Wagg = goWo + g1W1 + … + gnWn
.
For multi-expert NMT:
Corpus Creation 22
•   Paraphrase WMT 2017 Multimodal Translation corpus — using crowdsourcing •   Using image as the basis of paraphrasing, the crowdworker paraphrase – Original -> {deletion, insertion, reordering, substitution} • 3 months; 201 workers; 16 countries — English speaking countries, or at least English as second language • Crowdsourced 10k of training data, dev, test Caption : A little gray dog jumps over a small hurdle. Deletion : A little gray dog jumps over a hurdle. : A little gray dog jumps over a small hurdle successfully. Insertion Substitution : A little gray dog pass over a small hurdle. Reordering   : Over a small hurdle, a little gray dog jumps.
• WMT dataset size is 29k pair Crowdsourcing successfully paraphrased 10k sentences . Trained LSTM Encoder Decoder models for each paraphrase • operation — Using 10k crowdsourced paraphrase — To generate remaining 19k paraphrase Experimental Settings 25
. Combined paraphrased dataset with original dataset — Resulting 58k training data for each operation — The paraphrased data works as regularizer . For dev and test dataset: — For paraphrasing    : paraphrased dataset is used — For translation : original dataset is used For each expert translation model: Training data 58k
Experiment Results 27
•   Combining all data shows decrease in performance • Mixture of Expert yields the best result •  Test COCO 2017 (ambiguous situation)
● Outperform almost all models, except one •   Works in par with other multimodal model –  Only using textual information
(Data)
“dem fluss“ is missing
(Data) Target zwei motorräder fahren auf einer straße dem fluss entlang Corrected in final result Corrected in final result Enhancing Neural Machine Translation with Image-based Paraphrase Augmentation NAIS 32 Conclusions and Future Works 33
•   A single caption cannot represents all the information of the image to which it refers to • Generated multi-paraphrase of the WMT17 Multimodal Translation Task — Partially crowdsourcing with image as the basis of paraphrasing — Neural paraphrasing to complete the paraphrasing in semi-supervised way •  Proposed a textual model, in which the image information is not included in the model, but diffused in form of paraphrased caption •   +2.4 BLEU improvement over our NMT baseline
. Try different combination strategies/integration point • Investigate this proposed approach for another usage — Not limited for image caption translation •   Further investigate various methods of incorporating visual information Thanks for your attention! .