Extending Neural Machine Translation toDocuments and Signed Languages
Kayo Yin
Talk @ University of Pittsburgh
October 7 2021

1

Introduction➔

Neural Machine Translation is the state-of-the-art in automated translation2

Introduction➔

Neural Machine Translation is the state-of-the-art in automated translation➔

However, they are usually limited to sentence-level translations

3

Introduction➔

Neural Machine Translation is the state-of-the-art in automated translation➔

However, they are usually limited to sentence-level translations➔

Current NLP systems also cannot process signed languages

4

Today’s Agenda➔

Do context-aware machine translation models pay the right attention?

5

Today’s Agenda➔

Do context-aware machine translation models pay the right attention?

➔

When does translation require context?

6

Today’s Agenda➔

Do context-aware machine translation models pay the right attention?

➔

When does translation require context?

➔

How do we resolve coreference in signed languages?
7

Do Context-Aware Translation
Models Pay the Right Attention?
Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary
André F.T. Martins, Graham Neubig(ACL 2021)

8

Why is Context Important for Translation?

We’ll have to get rid of that mole.

9Why is Context Important for Translation?

Things could start to get dangerous if the ministers ﬁnd out.
We’ll have to get rid of that mole.

10Why is Context Important for Translation?

Things could start to get dangerous if the ministers ﬁnd out.
We’ll have to get rid of that mole.

11Why is Context Important for Translation?

Could it be anything serious, Doctor?
We’ll have to get rid of that mole.

12

Why is Context Important for Translation?

Could it be anything serious, Doctor?
We’ll have to get rid of that mole.

13

Why is Context Important for Translation?
English:Things could start to get dangerous if the ministers ﬁnd out.
We’ll have to get rid of that mole.
French:Les choses pourraient commencer à devenir dangereuses si
les ministres le découvraient.
Nous devrons nous débarrasser de cette taupe.
14

Why is Context Important for Translation?
English:Could it be anything serious, Doctor?
We’ll have to get rid of that mole.
French:Serait-ce quelque chose de grave, docteur ?
Nous devrons nous débarrasser de cette taupe.
cet grain de beauté
15

Why is Context Important for Translation?
English:So you see how bad the implications are.
Yes, they are quite devastating.
French:Vous voyez donc à quel point les implications sontmauvaises.
Oui, ils sont assez dévastateurs.
16Why is Context Important for Translation?
English:So you see how bad the implications are.
Yes, they are quite devastating.
French:Vous voyez donc à quel point les implications sontmauvaises.
Oui, ils sont assez dévastateurs.
elles
dévastatrices

17

Context-Aware NMT● Many approaches have been proposed for context-aware
machine translation

18

Context-Aware NMT● Many approaches have been proposed for context-aware
machine translation○ Concatenation, Multi-Encoder, Cache-Based, Hierarchical…

19

Context-Aware NMT● Many approaches have been proposed for context-aware
machine translation○ Concatenation, Multi-Encoder, Cache-Based, Hierarchical…

● Most of these approaches perform poorly on
document-level translation
20

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

21

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

22

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

23

Context-Aware NMT

Have we got her report?
Source input Yes, it’s in the inﬁrmary already.
Context-aware NMT output

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

24Outline
1.

What context is useful during ambiguous
translations?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

25Outline
1.

What context is useful duringtranslation?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

26

User Study27

User Study28

User Study

29

User Study

30

What Context do Human Translators Use?

31

What Context do Human Translators Use?

32

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

33

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

34

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Oui, [il / elle] est à l’inﬁrmière.

35

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Oui, [il / elle] est à l’inﬁrmière.

36

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

37

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

38

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

39

What Context do Human Translators Use?
(Pronoun Anaphora Resolution)

Have we got her report? It’s important. Yes, it’s in the inﬁrmary already.
On dispose de son rapport? Il est important. Oui, [il / elle] est à l’inﬁrmière.

40

What Context do Human Translators Use?
(Word Sense Disambiguation)

41

What Context do Human Translators Use?
(Word Sense Disambiguation)

42

What Context do Human Translators Use?
(Word Sense Disambiguation)
Your charm is only exceeded by your frankness.
Ton [charme / portebonheur] n’a d’égal que ta franchise.

43

What Context do Human Translators Use?
(Word Sense Disambiguation)nsubj:pass

obl:agentYour charm is only exceeded by your frankness.
VERB
NOUN .
Ton [charme / portebonheur] n’a d’égal que ta franchise.

44What Context do Human Translators Use?
(Word Sense Disambiguation)nsubj:pass

obl:agentYour charm is only exceeded by your frankness.
VERB
NOUN .
Ton [charme / portebonheur] n’a d’égal que ta franchise.

45

What Context do Human Translators Use?
(Word Sense Disambiguation)nsubj:pass

obl:agentYour charm is only exceeded by your frankness.
VERB
NOUN .
Ton [charme / portebonheur] n’a d’égal que ta franchise.

SCAT: Supporting Context for
Ambiguous Translations dataset (14K)

46

Outline
1.

What context is useful duringtranslation?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

47ModelEncoderCurrent source
sentence

DecoderCurrent target
sentence48

Model5 previous source
sentences

5 previous target
sentences

EncoderCurrent source
sentence

DecoderCurrent target
sentence49

ModelEncoder

Decoder50

Quantifying Human-Model Alignment with SCATEn Have we got her report?
Yes, it’s in the inﬁrmary
already.

En Have we got her report?
Yes, it’s in the inﬁrmary
already.

Fr

Fr

On dispose de son rapport?
Oui, il est à l’inﬁrmière.

On dispose de son rapport?
Oui, il est à l’inﬁrmière.

51

Quantifying Human-Model Alignment with SCATEn Have we got her report?
Yes, it’s in the inﬁrmary
already.

En Have we got her report?
Yes, it’s in the inﬁrmary
already.

Fr

Fr

On dispose de son rapport?
Oui, il est à l’inﬁrmière.

...
s
di

On dispose de son rapport?
Oui, il est à l’inﬁrmière.

...
se
o
p

t
de son por
p
ra

?

i,
u
O

il
s
di

se
o
p

t
de son por
p
ra

?

i,
u
O

il
52

Quantifying Human-Model Alignment with SCAT

.5

.1

.3

.1

53

Quantifying Human-Model Alignment with SCAT

Sort
.5

.1

.3

.5

.3

.1

.1

.1

54

Quantifying Human-Model Alignment with SCAT

Sort
.5

.1

.3

1

2

.5

.3

.1

.1

.1

55

Alignment Results

56

Alignment Results

57

Alignment Results

58

Alignment Results

59Outline
1.

What context is useful duringtranslation?

2.

Are models paying attention to thiscontext or not?

3.

If not, can we encourage them to do so?

60

Attention Regularization

OpenSubtitles18

Context-aware
MT Model61

Attention Regularization

OpenSubtitles18

Context-aware
MT ModelSCAT
62

Evaluation● BLEU● COMET

63

Evaluation● BLEU● COMET● Pronouns F-measure64Evaluation● BLEU● COMET● Pronouns F-measure● Contrastive Evaluation

Oui, il est déjà à l’inﬁrmerie.

Oui, elle est déjà à l’inﬁrmerie.
65

Results

66

Results

67

Results+9+868

Results

69

Results

70

Results

Have we got her report?
Yes, it’s in the inﬁrmary already.
Baseline

On dispose de son rapport?
Oui, elle est déjà à l’inﬁrmerie.

Have we got her report?
Yes, it’s in the inﬁrmary already.
Attention-Reg

On dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
71

Results

Have we got her report?
Yes, it’s in the inﬁrmary already.
More experiments Baseline& results in paper:
On dispose de son rapport?
➔ Increased usage of supporting
Oui, elle est déjà à l’inﬁrmerie.
context➔ Regularizing encoder self-attention
contributes the mostHave we got her report?
➔
Yes, it’s in the inﬁrmary already.
Attention-Reg
On dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
72

Results

Have we got her report?
Yes, it’s in the inﬁrmary already.
More experiments Baseline& results in paper:
On dispose de son rapport?
➔ Increased usage of supporting
Oui, elle est déjà à l’inﬁrmerie.
context➔ Regularizing encoder self-attention
contributes the mostHave we got her report?
➔
Yes, it’s in the inﬁrmary already.
Attention-Reg
On dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
73

Results

Have we got her report?
Yes, it’s in the inﬁrmary already.
More experiments Baseline& results in paper:
On dispose de son rapport?
➔ Increased usage of supporting
Oui, elle est déjà à l’inﬁrmerie.
context➔ Regularizing encoder self-attention
contributes the mostHave we got her report?
➔ Little diﬀerence in WSD performance
Yes, it’s in the inﬁrmary already.
Attention-Reg
On dispose de son rapport?
Oui, il est déjà à l’inﬁrmerie.
74

When Does Translation Require Context? A
Data-driven, Multilingual Exploration
Kayo Yin*, Patrick Fernandes*, André Martins, Graham Neubig(Ongoing work)*Equal contribution

Evaluating Document-Level Machine Translation➔

In machine translation (MT), context is crucial to translate certain discourse phenomena

76

Evaluating Document-Level Machine Translation➔

In machine translation (MT), context is crucial to translate certain discourse phenomena➔

However these phenomena represent only a small portion of the words in natural language data77

Evaluating Document-Level Machine Translation➔

In machine translation (MT), context is crucial to translate certain discourse phenomena➔

However these phenomena represent only a small portion of the words in natural language data➔

Common translation metrics don’t provide a clear picture of performance in these78

Evaluating Document-Level Machine Translation➔

Recent work on context-aware MT side-steps this by using contrastive datasets

79Evaluating Document-Level Machine Translation➔

Recent work on context-aware MT side-steps this by using contrastive datasets➔

However the availability of these datasets is limited

80Evaluating Document-Level Machine Translation➔

Recent work on context-aware MT side-steps this by using contrastive datasets➔

However the availability of these datasets is limited➔

Also this type of evaluation does not measure translation performance directly81Evaluating Document-Level Machine Translation➔

In this work, we propose data-driven, semi-automatic methodology for identifying salient phenomena

82

Evaluating Document-Level Machine Translation➔

In this work, we propose data-driven, semi-automatic methodology for identifying salient phenomena➔

We create a ﬁrst-of-its-kind multilingual benchmark testing these discourse phenomena

83

Evaluating Document-Level Machine Translation➔

In this work, we propose data-driven, semi-automatic methodology for identifying salient phenomena➔

We create a ﬁrst-of-its-kind multilingual benchmark testing these discourse phenomena➔

We evaluate multiple CAMT models, both trained by us and commercially available, on this benchmark

84

Measuring Context Usage➔

Previously, we proposed conditional cross-mutual information (CXMI)

85

Measuring Context Usage➔

Previously, we proposed conditional cross-mutual information (CXMI)➔This is corpus-level metric that tells us how well the context helps modelling a dataset

86

Measuring Context Usage➔

We propose a sentence-level extension, Pointwise Cross Mutual Information (P-CXMI)

87

Measuring Context Usage➔

We propose a sentence-level extension, Pointwise Cross Mutual Information (P-CXMI)➔

It can also be extended to word-level88

Which Translation Phenomena Beneﬁt from Context?

89

Which Translation Phenomena Beneﬁt from Context?

➔

Look at POS tags with high mean P-CXMI

90

Which Translation Phenomena Beneﬁt from Context?

➔

Look at POS tags with high mean P-CXMI➔

Look at vocabulary items with high mean P-CXMI

91

Which Translation Phenomena Beneﬁt from Context?

➔

Look at POS tags with high mean P-CXMI➔

Look at vocabulary items with high mean P-CXMI➔

Look at individual tokens with high P-CXMI

92

Which Translation Phenomena Beneﬁt from Context?

➔~120k parallel sentences from TED talk transcripts

93

Which Translation Phenomena Beneﬁt from Context?

➔~120k parallel sentences from TED talk transcripts➔

14 language pairs: English → Arabic, German, Spanish, French, Hebrew, Italian, Japanese, Korean,

Dutch, Portuguese, Romanian, Russian, Turkish and Mandarin Chinese

94

Which Translation Phenomena Beneﬁt from Context?

95

Which Translation Phenomena Beneﬁt from Context?

96

Which Translation Phenomena Beneﬁt from Context?

97

Which Translation Phenomena Beneﬁt from Context?

98

Which Translation Phenomena Beneﬁt from Context?

99

Which Translation Phenomena Beneﬁt from Context?

100Which Translation Phenomena Beneﬁt from Context?

101

Which Translation Phenomena Beneﬁt from Context?

102

Which Translation Phenomena Beneﬁt from Context?

103

Which Translation Phenomena Beneﬁt from Context?

104

Which Translation Phenomena Beneﬁt from Context?

105

Multilingual Discourse-Aware (MuDA) Benchmark

106

Multilingual Discourse-Aware (MuDA) Benchmark➔

Lexical Cohesion: tag target words y if the aligned source and target words pair (x,y) appears at least 3
times in the document

107

Multilingual Discourse-Aware (MuDA) Benchmark➔

Lexical Cohesion: tag target words y if the aligned source and target words pair (x,y) appears at least 3
times in the document➔

Formality: tag target words that are T-V pronouns/verbs or honoriﬁc terms

108

Multilingual Discourse-Aware (MuDA) Benchmark➔

Lexical Cohesion: tag target words y if the aligned source and target words pair (x,y) appears at least 3
times in the document➔

Formality: tag target words that are T-V pronouns/verbs or honoriﬁc terms➔

Pronoun choice: tag target pronouns if the corresponding source pronoun has multiple possible
translations

109

Multilingual Discourse-Aware (MuDA) Benchmark➔

Lexical Cohesion: tag target words y if the aligned source and target words pair (x,y) appears at least 3
times in the document➔

Formality: tag target words that are T-V pronouns/verbs or honoriﬁc terms➔

Pronoun choice: tag target pronouns if the corresponding source pronoun has multiple possible
translations➔

Verb form: tag target verbs if it has a verb form such that the corresponding source verb form has
multiple possible translations

110

Multilingual Discourse-Aware (MuDA) Benchmark➔

Lexical Cohesion: tag target words y if the aligned source and target words pair (x,y) appears at least 3
times in the document➔

Formality: tag target words that are T-V pronouns/verbs or honoriﬁc terms➔

Pronoun choice: tag target pronouns if the corresponding source pronoun has multiple possible
translations➔

Verb form: tag target verbs if it has a verb form such that the corresponding source verb form has
multiple possible translations➔

Ellipsis: tag target verbs, nouns and pronouns if the source sentence contains an ellipsis and the target
word is not aligned to any source word
111

Multilingual Discourse-Aware (MuDA) Benchmark

112

Multilingual Discourse-Aware (MuDA) Benchmark

113

A Cross-lingual, Cross-Model Exploration of Context-aware MT➔

We evaluate a sentence-level MT model and context-aware MT model on our system◆

We use a transformer small◆

For the context-aware method, we prepend the previous target context sentences to the current
target114

A Cross-lingual, Cross-Model Exploration of Context-aware MT

115

A Cross-lingual, Cross-Model Exploration of Context-aware MT➔

To evaluate more powerful models, we also ﬁnetune a large, pretrained model on this task◆

We do this for DE, FR, JA and ZH◆

We use a transformer large◆

We pretrain on Paracrawl, JParacrawl and Backtranslated News116

A Cross-lingual, Cross-Model Exploration of Context-aware MT

117

A Cross-lingual, Cross-Model Exploration of Context-aware MT➔

Finally we consider two commercial engines and evaluate them on our benchmark◆

the Google Cloud Translation v2 API◆

the DeepL v2 API

118

A Cross-lingual, Cross-Model Exploration of Context-aware MT

119Signed Coreference Resolution
Kayo Yin, Kenneth DeHaan, Malihe Alikhani(EMNLP 2021)

120

Coreference Resolution

Coreference Resolution

Signed Coreference Resolution

Signed Coreference Resolution

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context➔ Better understanding of grounding in diﬀerent forms of communication

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context➔ Better understanding of grounding in diﬀerent forms of communication➔ Broaden the scope of NLP to multiple modalities

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context➔ Better understanding of grounding in diﬀerent forms of communication➔ Broaden the scope of NLP to multiple modalities➔ Enable Sign Language Processing technologies

Outline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Outline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent➔ Assign a locus to the referent

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent➔ Assign a locus to the referent

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent➔ Assign a locus to the referent

Pronominal Pointing Signs

Pronominal Pointing Signs

Complexities of Pointing Signs➔ Pointing signs can serve other functions

Complexities of Pointing Signs➔ Pointing signs can serve other functions➔ Diﬃcult to distinguish between diﬀerent pointing signs based solely on
local visual features

Complexities of Pointing Signs
English Pronouns+ Carry some meaning on its own

ASL Pointing Signs

Complexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- Use the same handshape,
harder to distinguish on its ownComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its ownComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its own+ 1 locus = 1 referentComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its own+ 1 locus = 1 referent- Loci can be reassigned to
diﬀerent referentsComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its own+ 1 locus = 1 referent- Loci can be reassigned to
diﬀerent referents- Referents can be assigned
multiple loci

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)➔ It can help us better understand multimodal communication

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)➔ It can help us better understand multimodal communication◆ Spatial iconicity and situated referents in signed languages

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)➔ It can help us better understand multimodal communication◆ Spatial iconicity and situated referents in signed languages➔ Widen the accessibility of language technologies

Outline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Signed Coreference Resolution

Signed Coreference Resolution

1. Mention Detection

Signed Coreference Resolution

2. Coreference Resolution

DGS-Coref Dataset

Public DGS Corpus (Hanke et al., 2020)

DGS-Coref Dataset

DGS-Coref Dataset➔ 16m30s of signing➔ 3 conversations➔ 5 diﬀerent signers➔ 288 signed sentences➔ 1,457 glosses◆ 95 <I> signs◆ 8 <YOU> signs◆ 93 <INDEX> signs

A: WITH TRIP INDEX SHIP INDEX
A: We went there with an excursion boat.

Outline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution Task & Data
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Unsupervised Continuous MultigraphUnsupervised Continuous MultigraphUnsupervised Continuous MultigraphUnsupervised Continuous MultigraphPositive Relations
1. I and I

Positive Relations
1. I and I
2. You and You

Positive Relations
1. I and I
2. You and You
3. I and You

Positive Relations
1.
2.
3.
4.

I and IYou and You
I and You
Temporally Close Index

Positive Relations
1.
2.
3.
4.
5.

I and IYou and You
I and You
Temporally Close Index
Noun PhrasePositive Relations
1.
2.
3.
4.
5.
6.

I and IYou and You
I and You
Temporally Close Index
Noun Phrase
Spatially Close Index

Negative Relations
1. I and I

Negative Relations
1. I and I
2. You and You

Negative Relations
1. I and I
2. You and You
3. I and You

Negative Relations
1.
2.
3.
4.

I and IYou and You
I and YouDiﬀerent Person

Negative Relations
1.
2.
3.
4.

I and IYou and You
I and You
Spatially Far Index

Weight Assignment
Positive Relations

Negative Relations

1. I and I

1. I and I

2. You and You

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index

Weight Assignment
Positive Relations

Negative Relations

1. I and I

1. I and I

2. You and You

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index-∞

Weight Assignment
Positive Relations
1. I and I
2. You and You

Negative Relations
1. I and I+0.5

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index-∞

Weight Assignment
Positive Relations
1. I and I
2. You and You

Negative Relations
1. I and I+0.5

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index+(10-t)/20-∞

Weight Assignment
Positive Relations

Negative Relations

1. I and I
2. You and You

1. I and I+0.5

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index+(10-t)/20+(50-s)/50-∞

Clustering

ClusteringOutline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution Task & Data
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Results

Results

Results

Examples

TO-SEE YOU GOOD YOU
I think you could do a good job there.

GEST-DECLINE I CAN NOT TO-SAY TO-HOLD-ON I
I can’t keep that promiseExamples

P_IAndYou

TO-SEE YOU GOOD YOU
I think you could do a good job there.

P_YouAndYou

GEST-DECLINE I CAN NOT TO-SAY TO-HOLD-ON I
I can’t keep that promise

P_IAndIExamples

STUTTGART NUM-1 NAME INDEX NUM-1 FREIBURG
Once we were in Stuttgart, once in Ingolstadt and once in Freiburg.

Examples

P_NounPhrase

STUTTGART NUM-1 NAME INDEX NUM-1 FREIBURG
Once we were in Stuttgart, once in Ingolstadt and once in Freiburg.

Examples

WITH TRIP INDEX SHIP INDEX
We went there with an excursion boat.

Examples

P_TemporallyCloseIndex
P_SpatiallyCloseIndex

WITH TRIP INDEX SHIP INDEX
We went there with an excursion boat.

Examples

I TO-LEARN INDEX HAMBURG INDEXI learned it in Hamburg.

Examples

P_TemporallyCloseIndex
P_SpatiallyCloseIndex

I TO-LEARN INDEX HAMBURG INDEXI learned it in Hamburg.

Summary193

Today’s Agenda➔

Do context-aware machine translation models pay the right attention?
◆

No, but attention regularization on human rationales can encouragethem to do so!

➔

When does translation require context?
◆➔

Ambiguous pronouns, lexical cohesion, verb forms, formality, ellipsis

How do we resolve coreference in signed languages?
◆

Linguistically-informed heuristics and unsupervised multigraph194

Today’s Agenda➔

Do context-aware machine translation models pay the right attention?
◆

No, but attention regularization on human rationales can encouragethem to do so!

➔

When does translation require context?
◆➔

Ambiguous pronouns, lexical cohesion, verb forms, formality, ellipsis

How do we resolve coreference in signed languages?
◆

Linguistically-informed heuristics and unsupervised multigraph195

Today’s Agenda➔

Do context-aware machine translation models pay the right attention?
◆

No, but attention regularization on human rationales can encouragethem to do so!

➔

When does translation require context?
◆➔

Ambiguous pronouns, lexical cohesion, verb forms, formality, ellipsis

How do we resolve coreference in signed languages?
◆

Linguistically-informed heuristics and unsupervised multigraph196

Today’s Agenda➔

Do context-aware machine translation models pay the right attention?
◆

No, but attention regularization on human rationales can encouragethem to do so!

➔

When does translation require context?
◆➔

Ambiguous pronouns, lexical cohesion, verb forms, formality, ellipsis

How do we resolve coreference in signed languages?
◆

Linguistically-informed heuristics and unsupervised multigraph197