Natural Language Processing
for Signed Languages
Kayo Yin
TTIC NLP Seminar, February 9 2023

Signed Languages● Fully-ﬂedged natural languages● Expressed through various cues
● Independent of spoken languages

2

Signed Languages● 200 signed languages● ~70m deaf people3Signed Languages● Primary and preferred means ofcommunication for Deaf communities

4

Who Beneﬁts from Natural Language Processing?

5

Who Beneﬁts from Natural Language Processing?

6

Who Beneﬁts from Natural Language Processing?

7

Who Beneﬁts from Natural Language Processing?

Let’s allow everyone to beneﬁt from technology using their preferredlanguage!

8

Who is Working on Sign Language Processing?

9

Who is Working on Sign Language Processing?
Mostly computervision

Little NLPinvolvement
10Who is Working on Sign Language Processing?

Current models ignore the linguistic structure of signed languages
11

Who is Working on Sign Language Processing?

Incorporate linguistic insight intoSign Language Processing
12Extending NLP to Signed Languages● Both spoken and signed languages express the grammar of naturallanguages

Named EntityRecognition
Syntactic Parsing

Coreference
Resolution
POS Tagging
13

Extending NLP to Signed Languages● Both spoken and signed languages express the grammar of naturallanguages● Extend core NLP tools to signed languages

Named Entity
Recognition
Syntactic Parsing

Coreference
Resolution
POS Tagging
14

Challenges: Data Scarcity● Need large, realistic datasets

Taxonomy of language resources(Joshi et al., 2020)15

Challenges: Data Scarcity● Need large, realistic datasets● All signed languages are
extremely low-resource

Taxonomy of language resources(Joshi et al., 2020)16

Challenges: Data Scarcity● Diﬃcult to recruit and record signers for data collection

17Challenges: Data Scarcity● Diﬃcult to recruit and record signers for data collection● Finding / training annotators is challenging

18

Challenges: Data Scarcity● Diﬃcult to recruit and record signers for data collection● Finding / training annotators is challenging● 1 minute of labelled data requires 600 minutes of data collection
19Challenges: Spatial Dependencies● Grounding in signing space20Challenges: Spatial Dependencies● Grounding in signing space● We need to model the spatial
discourse21Natural Language Processing for Signed Languages

In this talk, we explore:● Data augmentation for Sign Language Translation

22

Natural Language Processing for Signed Languages

In this talk, we explore:● Data augmentation for Sign Language Translation● Coreference resolution for pronominal indexing signs

23

Data Augmentation forSign Language Gloss Translation
Amit Moryossef*, Kayo Yin*, Graham Neubig, Yoav Goldberg(MTSummit21 AT4SSL Workshop)*Equal contribution
24

Sign Language Translation25

Sign Language Translation

26

Overcoming Data Scarcity● Gloss-to-text translation = extremely low resource MT

27

Overcoming Data Scarcity● Gloss-to-text translation = extremely low resource MT● How is the relationship between a signed and spoken language
diﬀerent from two spoken languages?

28

Overcoming Data Scarcity● Gloss-to-text translation = extremely low resource MT● How is the relationship between a signed and spoken language
diﬀerent from two spoken languages?
● Can we improve gloss-to-text translation using pseudo-parallel
data?

29

Signed vs. Spoken Languages● Lexical similarity● Syntactic similarity30

Signed vs. Spoken Languages● Lexical similarity● Syntactic similarity

31

Signed vs. Spoken Languages● Lexical similarity● Syntactic similarity

32

Signed vs. Spoken Languages

33

Signed vs. Spoken Languages

34

Signed vs. Spoken Languages➔ Signed-spoken language pairs are lexically similar but syntacticallydiﬀerent
35

Data Augmentation
I’m looking forward to seeing the children tomorrow.

36

Data Augmentation
I’m looking forward to seeing the children tomorrow.

37

Data Augmentation
I’m looking forward to seeing the children tomorrow.

LOOK FORWARD SEE CHILD TOMORROW

38

Data Augmentation
I’m looking forward to seeing the children tomorrow.

FORWARD LOOK TOMORROW CHILD SEE

39

Data
● NCSLGR (SignStream, 2007)○ American Sign Language (ASL) - English○ 1,875 parallel sentences● PHOENIX 2014T (Camgoz et al., 2018)○ German Sign Language (DGS) - German○ 8,257 parallel sentences

40Model Training

Synthesized
Data

NMT (Yin and Read, 2020)

Real Data

41

Model Training

Synthesized
Data

NMT (Yin and Read, 2020)

Real Data

42

Model Training

Synthesized
Data

NMT (Yin and Read, 2020)

Real Data

43

Model Training

Synthesized
Data

NMT (Yin and Read, 2020)

Real Data

44

Results

45

Results46

Results

47

Results● Consistent translation improvements using data augmentation toleverage lexical similarities and handle syntactic diﬀerences● Data augmentation using monolingual spoken language data is apromising approach48

Signed Coreference Resolution
Kayo Yin, Kenneth DeHaan, Malihe Alikhani(EMNLP 2021)

49

Coreference Resolution

Coreference Resolution

Signed Coreference Resolution

Signed Coreference Resolution

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context➔ Better understanding of grounding in diﬀerent forms of communication

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context➔ Better understanding of grounding in diﬀerent forms of communication➔ Broaden the scope of NLP to multiple modalities

Signed Coreference Resolution➔ Novel challenges in modeling discourse and spatial context➔ Better understanding of grounding in diﬀerent forms of communication➔ Broaden the scope of NLP to multiple modalities➔ Enable Sign Language Processing technologies

Outline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent➔ Assign a locus to the referent

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent➔ Assign a locus to the referent

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent

Pronominal Pointing Signs➔ Pointing signs with a pronominal function➔ Referents are established in the signing space➔ Point to the actual location of the referent➔ Assign a locus to the referent

Pronominal Pointing Signs

Pronominal Pointing Signs

Complexities of Pointing Signs➔ Pointing signs can serve other functions

Complexities of Pointing Signs➔ Pointing signs can serve other functions➔ Diﬃcult to distinguish between diﬀerent pointing signs based solely on
local visual features

Complexities of Pointing Signs
English Pronouns+ Carry some meaning on its own

ASL Pointing Signs

Complexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- Use the same handshape,
harder to distinguish on its ownComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its ownComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its own

My mother never liked Alice, she thought she was up to no good.

Complexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its own+ 1 locus = 1 referentComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its own+ 1 locus = 1 referent- Loci can be reassigned to
diﬀerent referentsComplexities of Pointing Signs
English Pronouns

ASL Pointing Signs+ Carry some meaning on its own- The same word can refer tomultiple entities at once

- Use the same handshape,
harder to distinguish on its own+ 1 locus = 1 referent- Loci can be reassigned to
diﬀerent referents- Referents can be assigned
multiple loci

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)➔ It can help us better understand multimodal communication

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)➔ It can help us better understand multimodal communication◆ Spatial iconicity and situated referents in signed languages

Why study Signed Coreference Resolution in NLP?
➔ Theories of coreference in spoken languages may be extended tosigned languages◆ Discourse Representation Theory (Kamp et al., 2011; Steinbach and Onea 2016)◆ First mention eﬀect (Gernsbacher and Hargreaves, 1988; Wienholz et al., 2020)➔ It can help us better understand multimodal communication◆ Spatial iconicity and situated referents in signed languages➔ Widen the accessibility of language technologies

Outline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Signed Coreference Resolution

Signed Coreference Resolution

1. Mention Detection

Signed Coreference Resolution

2. Coreference Resolution

DGS-Coref Dataset

Public DGS Corpus (Hanke et al., 2020)

DGS-Coref Dataset

DGS-Coref Dataset➔ 16m30s of signing➔ 3 conversations➔ 5 diﬀerent signers➔ 288 signed sentences➔ 1,457 glosses◆ 95 <I> signs◆ 8 <YOU> signs◆ 93 <INDEX> signs

A: WITH TRIP INDEX SHIP INDEX
A: We went there with an excursion boat.

Outline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Unsupervised Continuous MultigraphUnsupervised Continuous MultigraphUnsupervised Continuous MultigraphUnsupervised Continuous MultigraphPositive Relations
1. I and I

Positive Relations
1. I and I
2. You and You

Positive Relations
1. I and I
2. You and You
3. I and You

Positive Relations
1.
2.
3.
4.

I and IYou and You
I and You
Temporally Close Index

Positive Relations
1.
2.
3.
4.
5.

I and IYou and You
I and You
Temporally Close Index
Noun PhrasePositive Relations
1.
2.
3.
4.
5.
6.

I and IYou and You
I and You
Temporally Close Index
Noun Phrase
Spatially Close Index

Negative Relations
1. I and I

Negative Relations
1. I and I
2. You and You

Negative Relations
1. I and I
2. You and You
3. I and You

Negative Relations
1.
2.
3.
4.

I and IYou and You
I and YouDiﬀerent Person

Negative Relations
1.
2.
3.
4.

I and IYou and You
I and You
Spatially Far Index

Weight Assignment
Positive Relations

Negative Relations

1. I and I

1. I and I

2. You and You

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index

Weight Assignment
Positive Relations

Negative Relations

1. I and I

1. I and I

2. You and You

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index-∞

Weight Assignment
Positive Relations
1. I and I
2. You and You

Negative Relations
1. I and I+0.5

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index-∞

Weight Assignment
Positive Relations
1. I and I
2. You and You

Negative Relations
1. I and I+0.5

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index+(10-t)/20-∞

Weight Assignment
Positive Relations

Negative Relations

1. I and I
2. You and You

1. I and I+0.5

2. You and You

3. I and You

3. I and You

4. Temporally Close Index

4. Spatially Far Index

5. Noun Phrase
6. Spatially Close Index+(10-t)/20+(50-s)/50-∞

Clustering

ClusteringOutline
1. Pronominal Pointing Signs
2. Signed Coreference Resolution
3. Unsupervised Continuous Multigraph
4. Results & Discussion

Results

Results

Results

Examples

TO-SEE YOU GOOD YOU
I think you could do a good job there.

GEST-DECLINE I CAN NOT TO-SAY TO-HOLD-ON I
I can’t keep that promiseExamples

P_IAndYou

TO-SEE YOU GOOD YOU
I think you could do a good job there.

P_YouAndYou

GEST-DECLINE I CAN NOT TO-SAY TO-HOLD-ON I
I can’t keep that promise

P_IAndIExamples

STUTTGART NUM-1 NAME INDEX NUM-1 FREIBURG
Once we were in Stuttgart, once in Ingolstadt and once in Freiburg.

Examples

P_NounPhraseSTUTTGART NUM-1 NAME INDEX NUM-1 FREIBURG
Once we were in Stuttgart, once in Ingolstadt and once in Freiburg.

Examples

WITH TRIP INDEX SHIP INDEX
We went there with an excursion boat.

Examples

P_TemporallyCloseIndex
P_SpatiallyCloseIndex

WITH TRIP INDEX SHIP INDEX
We went there with an excursion boat.

Examples

I TO-LEARN INDEX HAMBURG INDEXI learned it in Hamburg.

Examples

P_TemporallyCloseIndex
P_SpatiallyCloseIndex

I TO-LEARN INDEX HAMBURG INDEXI learned it in Hamburg.

Conclusion● Signed language data is scarce and hard to obtain

Conclusion● Signed language data is scarce and hard to obtain○ Data augmentation from monolingual spoken language data is
one promising way to mitigate thisConclusion● Signed language data is scarce and hard to obtain○ Data augmentation from monolingual spoken language data is
one promising way to mitigate this

● The meaning of certain signs rely on spatial context

Conclusion● Signed language data is scarce and hard to obtain○ Data augmentation from monolingual spoken language data is
one promising way to mitigate this

● The meaning of certain signs rely on spatial context○ Signed Coreference Resolution as a new challenge○ Unsupervised Continuous Multigraph for SCR

Future Work

● Pre-training the target side decoder with spoken language dataFuture Work

● Pre-training the target side decoder with spoken language data

● Resolve other types of ambiguous signs

Summary of the talk➔ Sign language processing is relatively nascent and NLP plays a crucialrole towards its progress➔ Example 1: data augmentation for sign language translation➔ Example 2: new dataset and unsupervised model for signed
coreference resolution

Summary of the talk➔ Sign language processing is relatively nascent and NLP plays a crucialrole towards its progress➔ Example 1: data augmentation for sign language translation➔ Example 2: new dataset and unsupervised model for signed
coreference resolution

Summary of the talk➔ Sign language processing is relatively nascent and NLP plays a crucialrole towards its progress➔ Example 1: data augmentation for sign language translation➔ Example 2: new dataset and unsupervised model for signed
coreference resolution

Summary➔ New challenge: Signed Coreference Resolution➔ Annotation software & DGS-Coref dataset

Summary➔ New challenge: Signed Coreference Resolution➔ Annotation software & DGS-Coref dataset➔ Unsupervised Continuous Multigraph for SCR

Summary➔ New challenge: Signed Coreference Resolution➔ Annotation software & DGS-Coref dataset➔ Unsupervised Continuous Multigraph for SCR➔ Code & data: https://github.com/kayoyin/scr

Future Work➔ Detect reassignment of loci
➔ Detect diﬀerent functions of indexing signs➔ Keep track of the dynamic signing space➔ Directly process videos➔ Resolve other types of pronominal signs➔ Resolve other types of ambiguous signs