Multimodal Neural Machine Translation
based on Image-Text Semantic Correspondence
画像とテキストの意味対応に基づくマルチモーダルニューラル機械翻訳
Yuting Zhao

東京都⽴⼤学 システムデザイン研究科 ⼩町研究室（現在︓九州⼤学 システム情報科学研究院）
第10回AAMT⻑尾賞学⽣奨励賞招待講演
2023年6⽉21⽇

1Outline
The content is organized as follows:• Introduces the background and overview of this work.
• Describes existing works of the MNMT task.
• Details the proposed method of the region-attentive MNMT model.
• Details the proposed method of the word-region alignment-guided MNMT model.
• Makes a conclusion of this thesis and describes future directions.
• Introduces the social impacts of this work.

2

BackgroundWhy multimodal machine translation (MMT) ?
-- Semantics still poorly used in MT.
⭐An Example of English->German translation task.

MT canʼt learn everything from textual context alone.
3

Overview of the MMT Task
Research questions:
How to integrate images into the MT model?
Can images improve the performance of MT?

MT Modela woman sitting ona very large rocksmiling at thecamera with trees
in the background .

eine frau sitzt vor bäumen
im hintergrund auf einem
sehr großen Stein undlächelt in die kamera .

4

Existing MMT Works (1/2)
I. How to represent image feature?
Global visual feature

Whole image

Local visual feature

Equally-sized grid

Semantic image regions
5

Existing MMT Works (2/2)
II. How to integrate image feature?

Sourcewords[Calixto and Liu, 2017] Iacer
Calixto and Qun Liu. Incorporating
global visual features into
attention-based neural machinetranslation. In EMNLP , pages
992–1003, 2017.

Sourcewords[Calixto et al., 2017] Iacer
Calixto, Qun Liu, and NickCampbell. Doubly-attentive
decoder for multi-modal neuralmachine translation. In ACL,pages 1913–1924, 2017.

6

Challenge in MMT
Research questions:
How to enhance the translation of the text by leveraging
their semantic correspondence to the images effectively?

MT Modela woman sitting ona very large rocksmiling at thecamera with trees
in the background .

eine frau sitzt vor bäumen
im hintergrund auf einem
sehr großen Stein undlächelt in die kamera .

7

Region-Attentive
Multimodal Neural Machine Translation(Neurocomputing, Vol.476, pp.1-13, 2022.)8

Object detection

Overview: Region-Attentive MNMTSrc: man in red shirt watches dog on an agility course.

Image-attention
Trg: un homme en polo
rouge regarde son chien
sur un parcours d'gilité.
Text-attention

9Methodology: RA-RNN
homme

en

Decoderyt-1

ytchemise

rouge

st-1

st<eos>

h1

h2

hi

zt

h4

hn

r1

r2

r3

ResNet101

rprm
Image encoder

Sentence encoder

ct

Faster R-CNN

Maninredxi

shirt

<eos>

10

Methodology: RA-TRANS
Outputs Probabilities
SoftmaxLinear
N×Addition
Addition

Feed Forward

Feed Forward

Layer Norm

Decoder

EncoderN×Layer Norm
Addition
Addition

Text
Cross-Attention

Image
Cross-Attention

Self-Attention

Layer Norm
Positional
Encoding

Input Embedding

man in red shirt
watches dog onan agility course.

100 Semantic Image
Region Features
Layer Norm
Addition
Self-Attention

Layer Norm

Faster R-CNN+ResNet101

Positional
Encoding
Input Embedding

Outputs (shifted right)

11

Experiments: Datasets
Each image was paired with image descriptions expressed by both theoriginal English sentences and the German and French translations.
v Multi30k [1]:v Translation tasks:•Train set: 30k training images.

•English->German (En-De)•Valid set: 1,014 validation images.

•English->French (En-Fr)•Test sets: 1,000 testing images.

v Evaluation metrics: BLEU and METEOR.
[1] Elliott, D., Frank, S., Sima’an, K., Specia, L., Multi30k: Multilingual English-German Image Descriptions, in VL@ACL 2016, pp. 70–74.

12Experiments: Setup•Architectures: RNN-based models and Transformer-based models.

v Baselines:
RNN/TRANS: the text-to-text RNN/Transformer model, wherein only the textual
sentences were used.
GA-RNN/GA-TRANS: the Grid-attentive multimodal RNN/Transformer model, using
7*7 equally-sized gird local visual features from each image extracted by a ResNet101
pretrained on ImageNet. Each grid-based local visual feature was represented as a2,048-dimension vector.
v Proposed Method:
RA-RNN/RA-TRANS: the RNN-based/Transformer-based Region-attentive MNMT
model, using 100 semantic image region visual features from each image extracted byFaster R-CNN pre-trained on Visual Genome with a ResNet101 pretrained on ImageNet.
Each region-based local visual feature was represented as a 2,048-dimension vector.
13

Results: RNN-based Models

14

Results: TRANS-based Models

15

Analyses: Qualitative Analysis
Visualization within RNN-Based Models.
chaisesdurayées

volSrc (En)

two people are sitting fishing on
striped beach chairs in a body of water .

Ref (Fr)deux personnes sont assises dans des fauteuils de plage rayés ,pêchant dans une étendue d's eau .

RNNdeux personnes sont assises sur une structure de plage rayée(a striped beach structure) dans un plan d's eau .

GA-RNN

deux personnes sont assises , pêchent surune plage de sable (a sandy beach) dans un plan d's eau .

RA-RNN

deux personnes sont assises à pêcher surdes chaises rayées (striped chairs) dans un plan d's eau .

Src (En)men playing volleyball , with one player missing the ballbut hands still in the air .

Ref (Fr)des hommes jouant au volleyball , avec un joueur ratant leballon mais avec les mains toujours en l's air .

RNN

des hommes jouant au volleyball , un joueur à l's attraper ,mais les autres mains ayant toujours dans les airs (in the air) .
. hommes jouant au volley-ball , avec un joueurdesqui le regarde dans les airs (in the air) .

GA-RNNRA-RNNdes hommes jouant au volleyball , avec un joueur qui passer
le ballon mais les mains du vol (of the flight).

16

Analyses: Qualitative Analysis
Visualization within Transformer-Based Models.
manipuleSrc (En)
Ref (Fr)TRANS
GA-TRANS

the woman in blue is operating a camera in front of two other women .
la femme en bleu manipule un appareil photo devant deux autres femmes .

la femme en bleu manie (wields) une caméra en face de deux autres femmes .
la femme en bleu fait fonctionner (function) un appareil photo devant deux autres femmes .
RA-TRANS la femme en bleu manipule (manipulate) un appareil photo devant deux autres femmes .
en
two women wearing tank tops are looking at the camera .
Src (En)
deux femmes portant des débardeurs regardent l's objectif .
Ref (Fr)deux femmes vêtues (wearing) de débardeurs regardent l's objectif .
TRANS
GA-TRANS deux femmes portant (wearing) des débardeurs regardent l's objectif .
RA-TRANS deux femmes en (in) débardeurs regardent l's objectif .

17

Contributions•

I have proposed region-attentive MNMT method that combines object detection with anadditional region-dependent attention mechanism to fully exploit image-text semantic
correspondence on NMT architectures, which are named : RA-RNN and RA-TRANS.

•

Extensive experimental results show that the proposed method can signiCicantly improve
MT performance over different kinds of baselines on both RNN and Transformer
architectures.

•Further analysis demonstrates that the proposed method can enhance the translation of text
by attending to their semantic corresponding image regions effectively with an additional
region-dependent attention mechanism.

18Limitation
Limitation:
Too much object-detected image regions
to be focused on is dif4icult.

MT Modeleine frau sitzt vor bäumen
im hintergrund auf einem
sehr großen Stein undlächelt in die kamera .

a woman sitting ona very large rocksmiling at thecamera with trees
in the background .

19

Word-Region Alignment-Guided
Multimodal Neural Machine Translation(IEEE/ACM Transactions on Audio, Speech, and Language Processing,Vol.30, pp.244-259, 2022)

20

Overview: WRA-Guided MNMTWRA acts as an auxiliary input to guide interactions between the textual and visual
information inside the entire multimodal neural machine translation (MNMT) model.
WRA: word-region alignment

Src: [man] in [red shirt]
watches [dog] onan agility course .

aag nilit
ycourse.

onredshw irt
atchesdoginm
an

[man]
[red shirt][dog]NMT
att: 0.7

att: 0.8Trg: un homme en polo rouge regarde son chien
sur un parcours d'gilité .

21

WRA Generation
calculate the semantic relevance between thei-th region and the whole source sentence.

[standing man]c1

[white dog]c2

[red shirt]• Soft WRA:

FastText

🌟Two types of explicit WRA:

c3[yellow shirt]

cm-1[wooden ladder]cm
w1 w2 w3 w4

Src: man in red shirt

course .

wn-1 wn

FastText

Source words:• Hard WRA:

Image regions:

WRA containing explicit semantic interactions
between the source words and image regions.
22

Integration Strategy: W2R (1/2)
🌟Word-to-Region (W2R):

x1x2x3xnh1h2h3hnDot
Product• Generate WRA-guided textual representations:Hs'ts"tdecoder

st-1ytzt

yt-1

textual encoder

WRA

stct

h'2 h'3h'mr2r3

rm

a1

a2a3am• Generate multimodal representations:

ResNet-101

visual encoder

r1

W2R:
word-to-region

h'1

Faster R-CNN

23

Integration Strategy: W2R (2/2)
Outputs Probabilities

Lineara1r2a2r3

a3

h1 h2rmam

ResNet-101

WRA

Faster R-CNN

h3hn

Add & Norm
Feed Forward

Textual encoderr1

Add & Norm
Add & Norm
Feed Forward

Visual Cross-Attention

Add & Norm

Textual Cross-Attention

Decodervisual encoder

W2R:
word-to-region

Softmax

Add & Norm

Self-Attention

Self-Attention

N×N×

Positional
Encoding

Positional
Encoding
Input Embedding

Output Embedding

Source Sentence

Outputs (shifted right)

24

Experiments: Datasets
v Multi30k:

v Translation tasks:•Train set: 30k training images.

•English->German (En-De)•Valid set: 1,014 validation images.

•English->French (En-Fr)•Test sets:oTest2016: 1,000 testing images.

oTest2017: 1,000 testing images.

v Evaluation metrics: BLEU and METEOR.

25

Experiments: Setup•Architectures: RNN-based models and Transformer-based models.

v Baselines:
NMT: the text-to-text NMT model, wherein only the textual sentences were used.
MNMTR: the doubly attentive MNMT model using regional visual features, without
integrating WRA to process W2R strategy.
v Proposed Method:MNMTW2R(sa): the proposed MNMT model incorporating soft WRA to guide W2R stage.
MNMTW2R(ha): the proposed MNMT model incorporating hard WRA to guide W2R stage.

26

Results: En-De Task

27

Results: En-Fr Task

28

EnFrEn

De

Analyses: Visualization

RNN-based

Transformer-based

RNN-based

Transformer-based

Text

Text

Text

Text

Image_(Independent)

Image_(Independent)

Image_(Independent)

Image_(Independent)

Image_(MNMTW2R(sa))

Image_(MNMTW2R(sa))

Image_(MNMTW2R(ha))

Image_(MNMTW2R(ha))

29Analyses: Case Study
hautsarrière-courdébardeurscourSrc (En): a man is grilling out in his backyard .

Src (En): two women wearing tank tops are looking at the camera .

Ref (Fr): un homme fait un barbecue dans son arrière-cour .

Ref (Fr): deux femmes portant des débardeurs regardent l&apos; objectif .

MNMTW2R(ha): un homme fait griller quelque chose dans sa arrière-cour .

MNMTW2R(sa): deux femmes portant des débardeurs regardent l&apos; objectif .

MNMTR: un homme fait griller quelque chose dans sa cour (yard) .

MNMTR: deux femmes vêtues de hauts (tops) regardent l&apos; objectif .

NMT: un homme fait griller quelque chose dans sa cour (yard) .

NMT: deux femmes portant des hauts (tops) regardent l&apos; objectif .

30

Contributions•

I have proposed the WRA to bridge multimodal inputs based on semantic correlation.

•I have proposed a novel integration strategy W2R to guide MNMT to translate certain
source words attending to semantically corresponding image regions.

•Experiments validated the consistent eﬃcacy of the proposed method and revealed
that it signiﬁcantly improved baselines based on diﬀerent evaluation metrics.

•Further analysis demonstrates that the proposed method can achieve bettertranslation performance owing to better visual information use.

31

Thanks for your listening!