Multitask Learning fromMultilingual
Multimodal Data
Desmond Elliott
University of Copenhagen
6th Workshop on Asian Machine Translation
November 4th 2019@delliott1

Contextual Language Understanding● “You shall know the meaning of a word by the company it keeps” (Firth, 1957)

Verb captures 2019
WKC Masters Agility
Grand Champion title

Verb: https://www.youtube.com/watch?v=GhKmykc02Bw

2

Cross-lingual Grounded Understanding3

Cross-lingual Grounded Understanding4

(mostly)

Machine Translation: an NLP success story

A baseball player in a black shirt justtagged a player in a white shirt.

Ein Baseballspieler in einem
schwarzen Shirt fängt einenSpieler in einem weißen Shirt.

5The Need for Visual Context in Translation●Post-editing of translations in Multi30K (Frank et al. JNLE 2018)6

Example of noun sense post-editing

Frank et al. JNLE 20187

Example of noun sense post-editing

Frank et al. JNLE 2018

8

The Need for Multilingual Captions●Speakers of different languages have different world knowledge

A strange looking
wood trailer isparked in a street
in front of stores.

Een draaiorgel ineen winkelstraatmet voetgangers.
(A street organ ina shopping streetwith pedestrians.)
van Miltenburg et al. INLG 20189

Multimodal Machine Translation

ModelEin Vogel fliegtüber das Wasser

A bird flies
over the water(Elliott et al. 2015; Specia et al. WMT 2016; Hitschler et al, ACL 2016; inter-alia)

10Use Cases for Multimodal Translation● Localised alt-text generation across the Web● Richer e-commerce experiences● Audio described movies for more languages

The Danish flag flying against a cloudy sky
Det danske flag vajende mod en blå himmel
11

Overview1. Multimodal Learning for Multilingual NLP
Elliott and Kadar (IJCNLP 2017)

2. Understanding Multimodal Translation
Elliott (EMNLP 2018), Gella et al. (NAACL 2019), Chowdhury and Elliott (LANTERN 2019)

3. Multilingual Learning for Multimodal NLP
Kadar et al. (CoNLL 2018)12

Multimodal learning for Multilingual NLP

Elliott and Kádár
Imagination improves Multimodal Translation
IJCNLP 201713

Decomposing Multimodal Translation● Solve as two separate tasks:
1. Learning to translate:
2. Learning to ground:● Multitask learning shared parameters (Caruana, 1997)
I. Are images necessary for inference?
II. How useful is external data for multimodal translation?
14

Task 1: Learning to Translate

15

Task 2: Learning to Ground16

Joint Multitask Learning Model

17

Optimisation● Translation task:● Image prediction task:● Joint objective:
18

Data: Multi30K● 32K English-captioned images with German, French, and Czech translations
A group of people are eating noodles.
Eine Gruppe von Leuten isst Nudeln.
Un groupe de gens mangent des nouilles.
Skupina lidí jedí nudle.
Translation: 32K Image-Sentence-Translation
Comparable: 155K Independent Image-Sentence
Elliott et al. V&L 2016, WMT 2017; Barrault et al. WMT 201819

Translation

Image Prediction improves Translation20

Translation

Further Improvements with External Data

240K
En-De

414K
En-Img

4.3M
En-De
21

Translation

Why does MTL help?

22

Take-away messages● Predicting the visual features during training
improves multimodal translation● Framework makes it easy to train withexternal parallel text or monolingual
described images

23

Translation

Empirical progress on Multi30K(Incomplete graph: not every paper reports performance on Test 2016)24

Understanding Multimodal Translation
Elliott EMNLP 2018
Gella et al. NAACL 2019
Chowdhury and Elliott LANTERN 201925

Adversarial Evaluation

Elliott (EMNLP 2018)26

Measuring Image AwarenessSource language
Target language

Congruent image
Incongruent}}Congruent:
should be better

Evaluation measure

Incongruent:
should be worse● Train on standard Multi30K training set● Evaluate on 5 random shuffles of En-De-Img
27

Models
1. Visual modulation of the the target embeddings (Caglayan et al. 2017)

2. Initialise decoder with image features (Elliott et al. 2015; Caglayan et al. 2017; inter-alia)

3. Source and image hierarchical attention (Libovický and Helcl 2017; inter-alia)

We are grateful to Ozan Caglayan and Jindřich Libovický for sharing their pre-trained models

28

Translation

Results:= Meteor● Models 1 and 2 are notaffected by the incongruent
visual data.
● Model 3 is more affected byincongruent visual data. Thismay be because it calculates
independent context vectors.

Model 1Model 2Model 3

(Grönroos et al. WMT 2018 report a related result when training with the mean of the image vectors.)

29

Translation

Understanding the Roles of the Modalities● Train with entity and colour masking (Caglayan et al. NAACL19)● Pre-trained models are sensitive to textual perturbations(Chowdhury and Elliott LANTERN 2019; Best Poster Award)

A group of young people dressed up for halloween.
Eine Gruppe jünger Menschen verkleidet.
Two groups of young people dressed up for halloween.
Zwei Gruppen von jungen Menschen in Japan.
30

MultiSense: Towards Targeted Evaluations●995 images with ambiguous verb senses (Gella et al. NAACL 2019)○Measure verb sense accuracy and translation quality

A large herd of sheep isblocking the road.
abdecken

Eine große Herde Schafe
blockiert die Straße.
verdecken

abblocken

blockieren

Senses for“blocking” inMultiSense

See also: Ambiguous COCO (Elliott et al. WMT 2017); Multimodal Lexical Translation (Lala and Specia, LREC 2018)31

Take-away messages● Awareness estimates the contribution ofadditional context in MMT models● Textual and visual adversaries offer usefulhints about the strengths of our models● More effort in creating and evaluating models
on challenging datasets
32

Multilingual learning for Multimodal NLPKádár, Elliott, Côté, Chrupała, Alishahi.
Lessons learned in multilingual grounded language learning.
CoNLL 201833

Cross-modal retrieval● Given a sentence, retrieve that it describes (and vice-versa)

34

Multi-task Multimodal Multilingual Model

Sharedencoder● Loss:Image - English

or

Image - GermanorGerman - English

When is multilingual data useful for cross-modal retrieval?
Gella et al. EMNLP 2017; Faghri et al. BMVC 201835

Translation

Multilingual data improves retrieval

36ComparableTranslation

High-to-low multilingual resource transferTranslationTranslation

Translation

Translation

Comparable

Comparable

37

Comparable

Controlling for 155K Training Data Points

DeDeDeDeEnEnEn

En

Single-task
monolingual

Multitask and multilingual38

Take-away messages● Multilingual learning improves image- tosentence and sentence-to-image retrieval● Improvements hold in “low-resource” settings● Caption-to-caption retrieval is an powerful
additional objective: more data doesn’t
entirely explain the improvements
39

Some Open Problems
●
●
●
●
●
●
●
●Larger multilingual multimodal datasets (Sanabria et al. 2018, Wang et al. 2019)Naturally-occurring multilingual grounded data (e.g. Schamoni et al. 2018)
Modelling audio, video, and text (e.g. Sanabria et al. 2018, Caglayan et al. 2019)
Robustness to adversaries (e.g. Elliott 2018; Caglayan et al. 2019)
Learning from unaligned data (e.g. Su et al. 2019)
More linguistically diverse language pairs (e.g. Parida et al. 2019)
Combining translation and ranking (e.g. Nikolaus et al. 2019)
Multilingual learning with disjoint character sets40

Acknowledgements
●
●
●
●
●
●
●
●

Afra Alishahi
Loic Barrault
Fethi Bougarres
Iacer Calixto
Koel Dutta Chowdhury
Grzegorz Chrupała
Marc-Alexandre Côté
Stella Frank

●
●●
●
●
●
●
●Spandana Gella
Eva HaslerÁkos Kádár
Frank Keller
Mirella Lapata
Lucia Specia
Khalil Sima’an
Arjen de Vries

41

Final Conclusions● Two views on multilingual multimodal data
○ Translation task: multimodality is useful○ Retrieval task: multilinguality is useful● Multitask learning was key to success○ Jointly solve multiple tasks○ Easily integrate external resources

42Our Work
S. Frank et al. Assessing Multilingual Multimodal Image Description: Studies of Native Speaker Preferences and Translator Choices. JNLE 2018.
D. Elliott, S. Frank, and E. Hasler. Multilingual Image Description with Neural Sequence Models. arXiv cs.CL 1510.04709.
L. Specia et al. A Shared Task on Multimodal Machine Translation and Crosslingual Image Description. In WMT 2016.
D. Elliott et al. Multi30K: Multilingual English-German Image Descriptions. Workshop on Vision and Language.
D. Elliott et al. Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description. In WMT 2017.
L. Barrault et al. Findings of the Third Shared Task on Multimodal Machine Translation. In WMT 2018.
D. Elliott. Adversarial Evaluation of Multimodal Translation. In EMNLP 2018.
S. Gella, D. Elliott, and F. Keller. Cross-lingual Visual Verb Sense Disambiguation. In NAACL 2019.
K. D. Chowdhury and D. Elliott. Understanding the Effect of Textual Adversaries in Multimodal Machine Translation. In LANTERN 2019.
Á. Kádár et al. Lessons learned in multilingual grounded language learning. In CoNLL 2018.

43

References
J. Hitschler, S. Schamoni, and S. Riezler. Multimodal Pivots for Image Caption Translation. In ACL 2016.
R. Caruana. Multitask learning. Machine learning 28.1 (1997): 41-75.
I. Calixto and Q. Liu. Incorporating Global Visual Features into Attention-based Neural Machine Translation. In EMNLP 2017.
J. Toyama, et al. Neural machine translation with latent semantic of image and text. arXiv preprint arXiv:1611.08459.
O. Caglayan, et al. LIUM-CVC Submissions for WMT17 Multimodal Translation Task. In WMT 2017.
J. Libovický and J. Helcl. Attention Strategies for Multi-Source Sequence-to-Sequence Learning. In ACL 2017.
Grönroos et al. The MeMAD Submission to the WMT18 Multimodal Translation Task. In WMT 2018.
O. Caglayan, et al. Probing the Need for Visual Context in Multimodal Machine Translation. In NAACL 2019.
C. Lala and Lucia Specia. Multimodal lexical translation. In LREC 2018.
S. Gella et al. Image Pivoting for Learning Multilingual Multimodal Representations. In EMNLP 2017.
F. Faghri, et al. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. In BMVC 2018.
S. Parida et al. Hindi Visual Genome arXiv:1907.08948.
M. Nikolaus et al. Compositional Generalization in Image Captioning. In CoNLL 2019.

44Figure Credits●

Slides 3 and 4: https://www.pinclipart.com/pindetail/iiwhTw_prank-machine-rube-goldberg-machine-scissor-clipart/●

Slide 7: https://www.pexels.com/photo/children-playing-soccer-2898317/●

Slide 10: https://da.wikipedia.org/wiki/Fil:Dannebrog_3.jpg, https://www.dyslexi.org/term/movietalk●

Slide 23: travelandleisure.com; modified with the GIMP●

Slides 24 and 24: Emojis by Jyoti Vyas CC-BY●

Slide 29: https://www.amazon.com/dp/B06X9YW32T/?tag=097-20&ascsubtag=v7_1_4_62m_wd1_4_x01_-srt-,
https://www.extremetech.com/extreme/135490-why-cut-down-on-carbon-emissions-when-you-can-curb-climatechange-by-simply-blotting-out-the-sun,●Slide 32: https://worldjusticeproject.org/photo-essays/brazil-female-warriors-fight-level-playing-field●

Slide 33: Modified version of Gella et al. EMNLP 201745