Do Context-Aware Translation
Models Pay the Right Attention?
Kayo Yin1, Patrick Fernandes1,2, Danish Pruthi1, Aditi Chaudhary1
André F.T. Martins2,3, Graham Neubig1

1

2

3
1

Why is Context Important for Translation?

We’ll have to get rid of that mole.

2Why is Context Important for Translation?

Things could start to get dangerous if the ministers ﬁnd out.
We’ll have to get rid of that mole.

3

Why is Context Important for Translation?

Things could start to get dangerous if the ministers ﬁnd out.
We’ll have to get rid of that mole.

4Why is Context Important for Translation?

Could it be anything serious, Doctor?
We’ll have to get rid of that mole.

5Why is Context Important for Translation?

Could it be anything serious, Doctor?
We’ll have to get rid of that mole.

6

Why is Context Important for Translation?
English:Things could start to get dangerous if the ministers ﬁnd out.
We’ll have to get rid of that mole.
French:Les choses pourraient commencer à devenir dangereuses si
les ministres le découvraient.
Nous devrons nous débarrasser de cette taupe.
7Why is Context Important for Translation?
English:Could it be anything serious, Doctor?
We’ll have to get rid of that mole.
French:Serait-ce quelque chose de grave, docteur ?
Nous devrons nous débarrasser de cette taupe.
cet grain de beauté
8

Why is Context Important for Translation?
English:So you see how bad the implications are.
Yes, they are quite devastating.
French:Vous voyez donc à quel point les implications sontmauvaises.
Oui, ils sont assez dévastateurs.
9Why is Context Important for Translation?
English:So you see how bad the implications are.
Yes, they are quite devastating.
French:Vous voyez donc à quel point les implications sontmauvaises.
Oui, ils sont assez dévastateurs.
elles
dévastatrices

10

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

11

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

12

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

13

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

14

Outline
1.

What context is useful during ambiguous
translations?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

15Outline
1.

What context is useful duringtranslation?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

16

User Study

17

User Study

18

User Study

19

User Study

20

What Context do Human Translators Use?

21

What Context do Human Translators Use?

22

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

23

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)24

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Oui, [il / elle] est à l’inﬁrmière.

25

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Oui, [il / elle] est à l’inﬁrmière.

26

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

27

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

28

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

29

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

30

What Context do Human Translators Use?
(Word Sense Disambiguation)

31

What Context do Human Translators Use?
(Word Sense Disambiguation)

32

What Context do Human Translators Use?
(Word Sense Disambiguation)
Your charm is only exceeded by your frankness.
Ton [charme / portebonheur] n’a d’égal que ta franchise.

33

What Context do Human Translators Use?
(Word Sense Disambiguation)nsubj:pass

obl:agentYour charm is only exceeded by your frankness.
VERB
NOUN .
Ton [charme / portebonheur] n’a d’égal que ta franchise.

34

What Context do Human Translators Use?
(Word Sense Disambiguation)nsubj:pass

obl:agentYour charm is only exceeded by your frankness.
VERB
NOUN .
Ton [charme / portebonheur] n’a d’égal que ta franchise.

35

What Context do Human Translators Use?
(Word Sense Disambiguation)nsubj:pass

obl:agentYour charm is only exceeded by your frankness.
VERB
NOUN .
Ton [charme / portebonheur] n’a d’égal que ta franchise.

SCAT: Supporting Context for
Ambiguous Translations dataset (14K)36

Outline
1.

What context is useful duringtranslation?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

37ModelEncoderCurrent source
sentence

DecoderCurrent target
sentence

38Model5 previous source
sentences

5 previous target
sentences

EncoderCurrent source
sentence

DecoderCurrent target
sentence

39

ModelEncoder

Decoder40

Quantifying Human-Model Alignment with SCATEn Have we got her report?
Yes, it’s in the inﬁrmary
already.

En Have we got her report?
Yes, it’s in the inﬁrmary
already.

FrFr

On dispose de son rapport?
Oui, il est à l’inﬁrmière.

On dispose de son rapport?
Oui, il est à l’inﬁrmière.

41

Quantifying Human-Model Alignment with SCATEn Have we got her report?
Yes, it’s in the inﬁrmaryalready.

En Have we got her report?
Yes, it’s in the inﬁrmary
already.

FrFr

On dispose de son rapport?
Oui, il est à l’inﬁrmière.

...
sdiOn dispose de son rapport?
Oui, il est à l’inﬁrmière.

...
seopt
de son porpra?

i,u
O

ils
di

seopt
de son porpra?

i,uO

il42

Quantifying Human-Model Alignment with SCAT.5.1.3.1

43

Quantifying Human-Model Alignment with SCATSort.5.1.3.5.3.1.1.1

44

Quantifying Human-Model Alignment with SCAT

Sort.5.1.3

12.5.3.1.1.1

45

Alignment Results

46

Alignment Results

47

Alignment Results

48

Alignment Results49Outline
1.

What context is useful duringtranslation?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

50Attention Regularization

OpenSubtitles18

Context-aware
MT Model51

Attention Regularization

OpenSubtitles18

Context-aware
MT ModelSCAT
52

Evaluation● BLEU● COMET53

Evaluation● BLEU● COMET● Pronouns F-measure54

Evaluation● BLEU● COMET● Pronouns F-measure● Contrastive Evaluation

Oui, il est déjà à l’inﬁrmerie.

Oui, elle est déjà à l’inﬁrmerie.
55

Results

56

Results

57

Results+9+858

Results

59

Results

60

Results

Have we got her report?
Yes, it’s in the inﬁrmary already.
Baseline

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

Have we got her report?
Yes, it’s in the inﬁrmary already.
Attention-RegOn dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
61

ResultsHave we got her report?
Yes, it’s in the inﬁrmary already.
More experiments Baseline& results in paper:
On dispose de son rapport?
➔ Increased usage of supporting
Oui, elle est déjà à l’inﬁrmerie.
context➔ Regularizing encoder self-attention
contributes the mostHave we got her report?
➔Yes, it’s in the inﬁrmary already.
Attention-Reg
On dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
62

Results

Have we got her report?
Yes, it’s in the inﬁrmary already.
More experiments Baseline& results in paper:
On dispose de son rapport?
➔ Increased usage of supporting
Oui, elle est déjà à l’inﬁrmerie.
context➔ Regularizing encoder self-attention
contributes the mostHave we got her report?
➔Yes, it’s in the inﬁrmary already.
Attention-Reg
On dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
63

Results

Have we got her report?
Yes, it’s in the inﬁrmary already.
More experiments Baseline& results in paper:
On dispose de son rapport?
➔ Increased usage of supporting
Oui, elle est déjà à l’inﬁrmerie.
context➔ Regularizing encoder self-attention
contributes the mostHave we got her report?
➔ Little diﬀerence in WSD performance
Yes, it’s in the inﬁrmary already.
Attention-Reg
On dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
64Summary

14k SCAT

65

Summary

14k SCAT

Baseline
Model66

Summary

14k SCAT

Baseline
Model67

Summary

Regularized
Model
14k SCAT

Baseline
Model68

Summary

Regularized
Model
14k SCAT➔

Better alignment, context usage, translation
performance

Baseline
Model69

Summary

Regularized
Model
14k SCAT

Baseline
Model➔

Better alignment, context usage, translation
performance➔

Code & Data:
github.com/neulab/contextual-mt
70