Unlocking Resources for
Under-resourced Languages
Graham Neubig
Shruti Rijhwani, Xinyi Wang
Antonios Anastasopoulos, Daisy Rosenblum,
Sebastian Ruder

2

Language technologies in a multilingual world

2

Language technologies in a multilingual world
Considerable recent progress in expanding NLP to many languages!

2

Language technologies in a multilingual world
Considerable recent progress in expanding NLP to many languages!
Multilingual benchmark
datasets for NLP tasks
WMT MasakhaNER
TyDiQA XNLI FLORES-101
UD Treebank XTREME

2

Language technologies in a multilingual world
Considerable recent progress in expanding NLP to many languages!
Multilingual benchmark
datasets for NLP tasks

Pretrained multilingual
language models

WMT MasakhaNER
TyDiQA XNLI FLORES-101
UD Treebank XTREME

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

2Language technologies in a multilingual world
Considerable recent progress in expanding NLP to many languages!
Multilingual benchmark
datasets for NLP tasks

Pretrained multilingual
language models

Commercial models that
support many languages

WMT MasakhaNER
TyDiQA XNLI FLORES-101
UD Treebank XTREME

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRVoice assistants
Predictive keyboards Translation
Web search Text analytics

3

Language technologies in a multilingual world
Pretrained multilingual
language models
XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

3Language technologies in a multilingual world
Pretrained multilingual
language models
XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRTrained on unlabeled text corpora(e.g., Wikipedia and Common Crawl)

3

Language technologies in a multilingual world
Pretrained multilingual
language models
XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRSupport 100 – 200 languages

3

Language technologies in a multilingual world
Pretrained multilingual
language models
XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRSupport 100 – 200 languages

Enables NLP applications
through cross-lingual transfer• Named entity recognition• Entity linking• Web search• Machine translation• Question answering…

4

Language technologies in a multilingual world
Pretrained multilingual
language models
XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRSupport 100 – 200 languages

4Language technologies in a multilingual world
Pretrained multilingual
language models

fi

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRSupport 100 – 200 languages

Multiple societal bene ts of NLP
that includes many languages!

4Language technologies in a multilingual world
Pretrained multilingual
language models

Support 100 – 200 languages

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRMultiple societal bene ts of NLP
that includes many languages!
Access to information andeducation from other languages

fiAutomatic translationof information

4

Language technologies in a multilingual world
Pretrained multilingual
language models
XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRSupport 100 – 200 languages

Multiple societal bene ts of NLP
that includes many languages!
Access to information andeducation from other languagesfiLanguage technologies that serve
many more people!

5

Language technologies in a multilingual world
Pretrained multilingual
language models

Support 100 – 200 languages

2%

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULRThere are over 7000 living languages!

98%

5

Language technologies in a multilingual world
Pretrained multilingual
language models

Support 100 – 200 languages

2%

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

6500+ languages are underrepresented in NLP

98%

5

Language technologies in a multilingual world
Pretrained multilingual
language models

Support 100 – 200 languages

2%

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

6500+ languages are underrepresented in NLP

98%

Over 2.2 billion people areunder-served by modern
language technologies

5

Language technologies in a multilingual world
Pretrained multilingual
language models

Support 100 – 200 languages

2%

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

98%What’s stopping us fromexpanding NLP systems
to more languages?

6

The unlabeled text bottleneck

0EnglishCebuanoSwedishGermanFrenchDutch
Russian
SpanishItalianEgyptian Arabic
PolishJapanese
Vietnamese
Waray
ChineseArabicUkrainianPortuguese
PersianCatalanSerbianIndonesian
Norwegian (Bokmål)KoreanFinnishHungarian
Czech
TurkishSerbo-Croatian
ChechenSouthern Min
RomanianBasqueTatar
Malay
Esperanto
Hebrew
Armenian
BulgarianDanishSouth Azerbaijani
SlovakKazakhMinangkabau
Estonian
Belarusian
Croatian
Greek
LithuanianSimple English
Azerbaijani
Galician
Slovene
Urdu
Norwegian (Nynorsk)
Georgian
Hindi
Thai
Tamil
Uzbek
Latin
Welsh
Asturian
Volapük
Macedonian
Cantonese
Bengali
Latvian
Tajik
BurmeseAfrikaans
Malagasy
BosnianOccitan
Albanian
Low German
Marathi
Kyrgyz
Malayalam
Belarusian (Taraškievica)
TeluguNewar
BretonSwahili
Venetian
Piedmontese
Javanese

Number of Wikipedia Articles

6

The unlabeled text bottleneck
7,000,000

5,250,000

3,500,000

1,750,000

0EnglishCebuanoSwedishGermanFrenchDutch
Russian
SpanishItalianEgyptian Arabic
PolishJapanese
Vietnamese
Waray
ChineseArabicUkrainianPortuguese
PersianCatalanSerbianIndonesian
Norwegian (Bokmål)KoreanFinnishHungarian
Czech
TurkishSerbo-Croatian
ChechenSouthern Min
RomanianBasqueTatar
Malay
Esperanto
Hebrew
Armenian
BulgarianDanishSouth Azerbaijani
SlovakKazakhMinangkabau
Estonian
Belarusian
Croatian
Greek
LithuanianSimple English
Azerbaijani
Galician
Slovene
Urdu
Norwegian (Nynorsk)
Georgian
Hindi
Thai
Tamil
Uzbek
Latin
Welsh
Asturian
Volapük
Macedonian
Cantonese
Bengali
Latvian
Tajik
BurmeseAfrikaans
Malagasy
BosnianOccitan
Albanian
Low German
Marathi
Kyrgyz
Malayalam
Belarusian (Taraškievica)
TeluguNewar
BretonSwahili
Venetian
Piedmontese
Javanese

Number of Wikipedia Articles

6

The unlabeled text bottleneck
7,000,000

5,250,000

3,500,000

1,750,000

0EnglishCebuanoSwedishGermanFrenchDutch
Russian
SpanishItalianEgyptian Arabic
PolishJapanese
Vietnamese
Waray
ChineseArabicUkrainianPortuguese
PersianCatalanSerbianIndonesian
Norwegian (Bokmål)KoreanFinnishHungarian
Czech
TurkishSerbo-Croatian
ChechenSouthern Min
RomanianBasqueTatar
Malay
Esperanto
Hebrew
Armenian
BulgarianDanishSouth Azerbaijani
SlovakKazakhMinangkabau
Estonian
Belarusian
Croatian
Greek
LithuanianSimple English
Azerbaijani
Galician
Slovene
Urdu
Norwegian (Nynorsk)
Georgian
Hindi
Thai
Tamil
Uzbek
Latin
Welsh
Asturian
Volapük
Macedonian
Cantonese
Bengali
Latvian
Tajik
BurmeseAfrikaans
Malagasy
BosnianOccitan
Albanian
Low German
Marathi
Kyrgyz
Malayalam
Belarusian (Taraškievica)
TeluguNewar
BretonSwahili
Venetian
Piedmontese
Javanese

Number of Wikipedia Articles

6

The unlabeled text bottleneck
7,000,000

5,250,000

3,500,000

1,750,000

7,000,000

0EnglishCebuanoSwedishGermanFrenchDutch
Russian
SpanishItalianEgyptian Arabic
PolishJapanese
Vietnamese
Waray
ChineseArabicUkrainianPortuguese
PersianCatalanSerbianIndonesian
Norwegian (Bokmål)KoreanFinnishHungarian
Czech
TurkishSerbo-Croatian
ChechenSouthern Min
RomanianBasqueTatar
Malay
Esperanto
Hebrew
Armenian
BulgarianDanishSouth Azerbaijani
SlovakKazakhMinangkabau
Estonian
Belarusian
Croatian
Greek
LithuanianSimple English
Azerbaijani
Galician
Slovene
Urdu
Norwegian (Nynorsk)
Georgian
Hindi
Thai
Tamil
Uzbek
Latin
Welsh
Asturian
Volapük
Macedonian
Cantonese
Bengali
Latvian
Tajik
BurmeseAfrikaans
Malagasy
BosnianOccitan
Albanian
Low German
Marathi
Kyrgyz
Malayalam
Belarusian (Taraškievica)
TeluguNewar
BretonSwahili
Venetian
Piedmontese
Javanese

Number of Wikipedia Articles

6

The unlabeled text bottleneck• Annotated datasets: need existing text dataor have to recruit speakers to create it.

5,250,000

3,500,000

1,750,000

7,000,000

5,250,000

0EnglishCebuanoSwedishGermanFrenchDutch
Russian
SpanishItalianEgyptian Arabic
PolishJapanese
Vietnamese
Waray
ChineseArabicUkrainianPortuguese
PersianCatalanSerbianIndonesian
Norwegian (Bokmål)KoreanFinnishHungarian
Czech
TurkishSerbo-Croatian
ChechenSouthern Min
RomanianBasqueTatar
Malay
Esperanto
Hebrew
Armenian
BulgarianDanishSouth Azerbaijani
SlovakKazakhMinangkabau
Estonian
Belarusian
Croatian
Greek
LithuanianSimple English
Azerbaijani
Galician
Slovene
Urdu
Norwegian (Nynorsk)
Georgian
Hindi
Thai
Tamil
Uzbek
Latin
Welsh
Asturian
Volapük
Macedonian
Cantonese
Bengali
Latvian
Tajik
BurmeseAfrikaans
Malagasy
BosnianOccitan
Albanian
Low German
Marathi
Kyrgyz
Malayalam
Belarusian (Taraškievica)
TeluguNewar
BretonSwahili
Venetian
Piedmontese
Javanese

Number of Wikipedia Articles

6

The unlabeled text bottleneck• Annotated datasets: need existing text dataor have to recruit speakers to create it.

• Multilingual LMs: performance is limited by
the amount of text available!

3,500,000

1,750,000

7

The unlabeled text bottleneck

Number of Wikipedia Articles

7,000,000

5,250,000

3,500,000

1,750,000

07The unlabeled text bottleneck

Number of Wikipedia Articles

7,000,000

5,250,000

3,500,000

1,750,0000Some text isavailable in few
hundred languages

7

The unlabeled text bottleneck

5,250,000

Some text isavailable in few
hundred languages

3,500,000

1,750,0000fi

Number of Wikipedia Articles

7,000,000

Thousands of languages
languages without suf cient text
for developing NLP systems

7

The unlabeled text bottleneck

Number of Wikipedia Articles

7,000,000

5,250,000

3,500,000

1,750,0000Some text isavailable in few
hundred languages

How do we move towards
including these languages inmodern NLP systems?

Text resources do exist in many more languages!

8Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

8

Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

Printed books

8

Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

Printed books

Handwritten notes

8Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

Printed books

Handwritten notes

Typewritten documents

8Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

Printed books

Handwritten notes

or in other formats such as bilingual lexicons

Typewritten documents

8Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

Printed books

Handwritten notes

or in other formats such as bilingual lexicons

Typewritten documents

8Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

Printed books

Handwritten notes

or in other formats such as bilingual lexicons

Typewritten documents

8Text resources do exist in many more languages!
But locked away in formats that are not machine-readable

Printed books

Handwritten notes

or in other formats such as bilingual lexicons

what can we do?!

Typewritten documents

8Text resources do exist in many more languages!

9Text resources do exist in many more languages!

9Text resources do exist in many more languages!

Unlocking nontraditional
resources9Text resources do exist in many more languages!
Enable NLP for under-resourced languages

Unlocking nontraditional
resources9Text resources do exist in many more languages!
Enable NLP for under-resourced languages
Expand multilingual LMs
to more languages

Unlocking nontraditional
resources

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

9Text resources do exist in many more languages!
Enable NLP for under-resourced languages
Expand multilingual LMs
to more languages
Annotate datasets fordownstream NLP tasks

Unlocking nontraditional
resources

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

9Text resources do exist in many more languages!
Enable NLP for under-resourced languages
Expand multilingual LMs
to more languages

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

Annotate datasets fordownstream NLP tasks

Unlocking nontraditional
resources

Support communities that speak these languages

9

Text resources do exist in many more languages!
Enable NLP for under-resourced languages
Expand multilingual LMs
to more languages

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

Annotate datasets fordownstream NLP tasks

Unlocking nontraditional
resources

Support communities that speak these languages
Make native texts digitally
accessible and searchable

9Text resources do exist in many more languages!
Enable NLP for under-resourced languages
Expand multilingual LMs
to more languages

XLM-R mBERT
mT5 mBART ERNIE-M
Turing ULR

Annotate datasets fordownstream NLP tasks

Unlocking nontraditional
resources

Support communities that speak these languages
Make native texts digitally
accessible and searchable
Aid language researchers, educatators, libraries…

9

10

Unlocking Un-digitized Text
Shruti Rijhwani, Antonios Anastasopoulos, Graham Neubig.
OCR Post-Correction for Endangered Language Texts.
EMNLP 2020.
Shruti Rijhwani, Daisy Rosenblum, Antonios Anastasopoulos, Graham Neubig.
Lexically-Aware Semi-Supervised Learning for OCR Post-Correction.
TACL 2021.

11

Extracting text from scanned documents

Scanned document

11

Extracting text from scanned documents

Scanned document

Scan from a book of
folk tales in Griko

11

Extracting text from scanned documents

Scanned document

11

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document

11

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

12

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

12

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

• High accuracy on languages that have easily available resources!

12

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

• High accuracy on languages that have easily available resources!
• Off-the-shelf tools support many scripts and languages

12

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

• High accuracy on languages that have easily available resources!
• Off-the-shelf tools support many scripts and languages{

Google Vision
Tesseract
EasyOCR…

12

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

• High accuracy on languages that have easily available resources!
• Off-the-shelf tools support many scripts and languages
Support 80-100 languages{

Google Vision
Tesseract
EasyOCR…

12

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

• High accuracy on languages that have easily available resources!
• Off-the-shelf tools support many scripts and languages• Little to no prior work on very low-resourced settings

13

Extracting text from scanned documents
Optical Character
Recognition (OCR)

Scanned document• Little to no prior work on very low-resourced settings”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Machine readable text

13

Extracting text from scanned documents
Optical Character
Recognition (OCR)”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Scanned document

Machine readable text

• Little to no prior work on very low-resourced settings
Evaluation dataset
Promises and pitfalls of existing methods
Neural models for improving OCR performance in low-resource settings

Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

13

Extracting text from scanned documents
Optical Character
Recognition (OCR)”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Scanned document

Machine readable text

• Little to no prior work on very low-resourced settings
Evaluation dataset
Promises and pitfalls of existing methods
Neural models for improving OCR performance in low-resource settings

Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

13

Extracting text from scanned documents
Optical Character
Recognition (OCR)”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Scanned document

Machine readable text

• Little to no prior work on very low-resourced settings
Evaluation dataset
Promises and pitfalls of existing methods
Neural models for improving OCR performance in low-resource settings

Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

13

Extracting text from scanned documents
Optical Character
Recognition (OCR)”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Scanned document

Machine readable text

• Little to no prior work on very low-resourced settings
Evaluation dataset
Promises and pitfalls of existing methods
Neural models for improving OCR performance in low-resource settings

Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

Semi-supervised learning to improve performance with unlabeled images

Rijhwani, Rosenblum, Anastasopoulos, Neubig. TACL 2021.

14

Evaluation dataset for low-resource OCR

Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Yakkha(Nepal)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Yakkha(Nepal)Kwak’wala(Canada)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)• Orthographically, typologically,geographically diverse

Yakkha(Nepal)Kwak’wala(Canada)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Latin• Orthographically, typologically,geographically diverse

Yakkha(Nepal)Kwak’wala(Canada)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Latin• Orthographically, typologically,geographically diverse
Latin+Greek

Yakkha(Nepal)Kwak’wala(Canada)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Yakkha(Nepal)Latin• Orthographically, typologically,geographically diverse
Latin+Greek

Devanagari

Kwak’wala(Canada)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Latin• Orthographically, typologically,geographically diverse
Latin+Greek

Yakkha(Nepal)Devanagari

Kwak’wala(Canada)Boas
Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Yakkha(Nepal)• Orthographically, typologically,geographically diverse• The languages currently have:• No Wikipedia/Common Crawl text
• Not supported by multilingual LMs• No easily accessible bilingual lexica

Kwak’wala(Canada)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

14

Evaluation dataset for low-resource OCR
Ainu(Japan)Griko(Italy)Yakkha(Nepal)• Orthographically, typologically,geographically diverse• The languages currently have:• No Wikipedia/Common Crawl text
• Not supported by multilingual LMs• No easily accessible bilingual lexica• <1000 transcribed lines per language

Kwak’wala(Canada)Rijhwani, Anastasopoulos, Neubig. EMNLP 2020.

15

Existing OCR methods

15

Existing OCR methods
Supervised

Large neural networks

Requires: 10000s oftranscribed images

15

Existing OCR methods
Supervised

Unsupervised• Unlabeled images• Language model

Large neural networks

Requires: 10000s oftranscribed images

Requires: text corpus
or lexicon in thetarget language

15

Existing OCR methods
Supervised

Unsupervised• Unlabeled images• Language model

Large neural networks

Requires: 10000s oftranscribed images

Requires: text corpus
or lexicon in thetarget language

Off-the-shelf• Support ~100
languages• Not trained on ourtarget languages• Can act as a generalcharacter recognizer
for many scripts

15

Existing OCR methods
Supervised

Unsupervised• Unlabeled images• Language model

Large neural networks

Requires: 10000s oftranscribed images

Requires: text corpus
or lexicon in thetarget language

Off-the-shelf• Support ~100
languages• Not trained on ourtarget languages• Can act as a generalcharacter recognizer
for many scripts

15

Existing OCR methods
Supervised

Unsupervised• Unlabeled images• Language model

Large neural networks

Requires: 10000s oftranscribed images

Requires: text corpus
or lexicon in thetarget language

Off-the-shelf• Support ~100
languages• Not trained on ourtarget languages• Can act as a generalcharacter recognizer
for many scripts

16

Existing OCR methods: promises and pitfalls

16

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

16

Existing OCR methods: promises and pitfalls
Google Vision% Word Error Rate

word edit distance between
100prediction and reference
number of words in reference

Ocular

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

16

Existing OCR methods: promises and pitfalls
Google Vision

OcularLower isbetter!

% Word Error Rate

100

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

16

Existing OCR methods: promises and pitfalls
Google Vision

Lower isbetter!

% Word Error Rate

100

Off-the-shelf models for languages in 29 scripts
Script-speci c models

75

50

25

0

fi

Ocular

Ainu

Griko

YakkhaKwak'wala

16

Existing OCR methods: promises and pitfalls
Google Vision

OcularLower isbetter!

% Word Error Rate

100

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

16

Existing OCR methods: promises and pitfalls
Google Vision

Ocular75

Google Vision performs well if the script isknown, even without any language speci c data
50

25

0

Ainu
fi

Lower isbetter!

% Word Error Rate

100

Griko

YakkhaKwak'wala

16

Existing OCR methods: promises and pitfalls
Google Vision

OcularScript not known
to the model

Lower isbetter!

% Word Error Rate

100

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

17

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

75

50

Example in Griko:exi i kaddinàra!

_खारिनङ्गो

eχi i kaḍḍinàra!

ङ्खाॽिनङ्गो

25

0

Ainu

Griko

YakkhaKwak'wala

17

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

75

50

Example in Griko:

25

0

Ainu

exi i kaddinàra!

_खारिनङ्गो

exi
i
kaddinàra!
eχi i kaḍḍinàra!

ङ्खाॽिनङ्गो

Griko

YakkhaKwak'wala

17

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

Mixed scripts

75

50

Example in Griko:

25

0

Ainu

exi i kaddinàra!

_खारिनङ्गो

exi
i
kaddinàra!
eχi i kaḍḍinàra!

ङ्खाॽिनङ्गो

Griko

YakkhaKwak'wala

17

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

Mixed scripts

75

50

Example in Griko:

25

0

Uncommon diacritics

Ainu

exi i kaddinàra!

_खारिनङ्गो

exi
i
kaddinàra!
eχi i kaḍḍinàra!

ङ्खाॽिनङ्गो

Griko

YakkhaKwak'wala

18

Existing OCR methods: promises and pitfalls

18

Existing OCR methods: promises and pitfalls
Google Vision

OcularLower isbetter!

% Word Error Rate

100

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

18

Existing OCR methods: promises and pitfalls
Google Vision

Lower isbetter!

% Word Error Rate

100

OcularOpen-source unsupervised OCR software
Requires an LM in the target language

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

18

Existing OCR methods: promises and pitfalls
Google Vision

OcularLower isbetter!

% Word Error Rate

100

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

18

Existing OCR methods: promises and pitfalls
Google Vision

OcularLower isbetter!

% Word Error Rate

100

75

Ocular is highly
reliant on the LM

50

25

0

Ainu

Griko

YakkhaKwak'wala

18

Existing OCR methods: promises and pitfalls
Google Vision

OcularLower isbetter!

% Word Error Rate

100

Ocular is better
for Kwak’wala

75

50

25

0

Ainu

Griko

YakkhaKwak'wala

19

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

75.0

75

82.1

47.5

50

31.6
25

38.2

15.6 15.8
6.3

0

Ainu

Griko

YakkhaKwak'wala

19

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

75.0

75

82.1

• Considerable room for
improvement comparedto high-resource languages

47.5

50

31.6
25

38.2

15.6 15.8
6.3

0

Ainu

Griko

YakkhaKwak'wala

19

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

75.0

75

82.1

• Considerable room for
improvement comparedto high-resource languages

47.5

50

31.6
25

38.2

15.6 15.8
6.3

0

Ainu

Griko

YakkhaKwak'wala• Recognizes the majority ofwords correctly

19

Existing OCR methods: promises and pitfalls
Google Vision

Ocular% Word Error Rate

100

75.0

75

82.1

• Considerable room for
improvement comparedto high-resource languages

47.5

50

31.6
25

38.2

• Reliable starting point for
further improvements

15.6 15.8
6.3

0

Ainu

Griko

• Recognizes the majority ofwords correctly

YakkhaKwak'wala20

Improving the results of existing OCR systems

20

Improving the results of existing OCR systems

20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:fi

OCR output (“ rst pass”)

20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:
OCR output (“ rst pass”)

fi

OCR output has
some errors

20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:fi

OCR output (“ rst pass”)

Automatic OCR
Post-Correction

20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:fi

OCR output (“ rst pass”)

Automatic OCR
Post-Correction”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Corrected transcription

20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:

Automatic OCR
Post-Correction

OCR output (“ rst pass”)

Corrected transcription
Previous work: improve results for
unseen fonts, layouts, domains.
This talk: low-resourced languages.

fi”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:

Automatic OCR
Post-Correction

OCR output (“ rst pass”)

Corrected transcription

Text-based sequenceto-sequence taskfi

”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:

Automatic OCR
Post-Correction

OCR output (“ rst pass”)

fi

Text input”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Corrected transcription

Text-based sequenceto-sequence task

20

Improving the results of existing OCR systems”Ma ti exi’ pu klei’?””Ikleo ka ìtela _armastò.”I vekkia aggale tria dattilitia:

Automatic OCR
Post-Correction

OCR output (“ rst pass”)

fi

Text input”Ma ti eχi’ pu klei’?””Iklèo ka ìtela n’armastò.”I vèkkia àggale tria dattilìtia:Corrected transcription

Text-based sequenceto-sequence task

Text output

21

Adapting to low-resource settings
Prior work: character-level encoder-decoder with attention• Add structural biases to the model• Diagonal attention loss, copy mechanism, coverage mechanism

21

Adapting to low-resource settings
Prior work: character-level encoder-decoder with attention• Add structural biases to the model• Diagonal attention loss, copy mechanism, coverage mechanism• Leverage additional information from the source document

22

What additional information is available?

• Many documents containing text in lowresource languages also contains atranslation of the text
• Interlinear glosses, dictionaries, linguistic
documentation, language learning material…

23

Multi-source model for post-correction

23

Multi-source model for post-correction

Low-resource
language encoder…

…
Attentionfi

Griko rst pass OCR

exi i kaddinàra!

23

Multi-source model for post-correction
Italian rst pass OCR

Low-resource
language encoder

possiede la gallinaia…

…
Attentionfi

fi

Griko rst pass OCR

exi i kaddinàra!

23

Multi-source model for post-correction
Italian rst pass OCR

possiede la gallinaia

High-resource
translation encoder…

Low-resource
language encoder…

…
Attentionfi

fi

Griko rst pass OCR

exi i kaddinàra!

23

Multi-source model for post-correction
Italian rst pass OCR

possiede la gallinaia
Attention

High-resource
translation encoder…

…

Low-resource
language encoder…

…
Attentionfi

fi

Griko rst pass OCR

exi i kaddinàra!

23

Multi-source model for post-correction
Italian rst pass OCR

possiede la gallinaia
Attention

High-resource
translation encoder…

…

Concatenate
context vectors…
Decoder LSTM

Low-resource
language encoder…

…
Attentionfi

fi

Griko rst pass OCR

exi i kaddinàra!

23

Multi-source model for post-correction
Italian rst pass OCR

possiede la gallinaia
Attention

High-resource
translation encoder…

…

Concatenate
context vectors…
Decoder LSTM

Low-resource
language encoder…

…
Attentionfi

fi

Griko rst pass OCR

exi i kaddinàra!

Supervised training with asmall amount of annotateddata (<1000 lines)

24

Experiments: how do existing post-correction methods perform?

24

Experiments: how do existing post-correction methods perform?

% Word Error Rate

50

37.5

First pass OCR
Encoder-decoder
Ours

25

12.5

0

Ainu

Griko

YakkhaKwak'wala

24

Experiments: how do existing post-correction methods perform?

% Word Error Rate

50

37.5

ä

ä

Dong and Smith, 2018
H m l inen and Hengchen, 2019
Todorov and Colavizza, 2020
Lyu et al., 2021
Duong et al., 2021
…

25

12.5

0

ä

First pass OCR
Encoder-decoder
Ours

Ainu

Griko

YakkhaKwak'wala

24

Experiments: how do existing post-correction methods perform?

50% Word Error Rate

Cross
validation

37.5

First pass OCR
Encoder-decoder
Ours

25

12.5

0

Ainu

Griko

YakkhaKwak'wala

24

Experiments: how do existing post-correction methods perform?

Lower isbetter!

% Word Error Rate

50First pass OCR
Encoder-decoder
Ours

37.5

Ocular
GoogleVisionGoogleVision25
GoogleVision

12.5

0

Ainu

Griko

YakkhaKwak'wala

24

Experiments: how do existing post-correction methods perform?

Lower isbetter!

% Word Error Rate

50

37.5

First pass OCR
Encoder-decoder
Ours

25

12.5

0

Ainu

Griko

YakkhaKwak'wala

24

Experiments: how do existing post-correction methods perform?

Lower isbetter!

% Word Error Rate

50

37.5

First pass OCR
Encoder-decoder
Ours

25

12.5

0

fi

The baseline sometimes
increases the error rateover the rst pass

Ainu

Griko

YakkhaKwak'wala

25

Experiments: do the adaptions help low-resource learning?

Lower isbetter!

% Word Error Rate

50

37.5

First pass OCR
Encoder-decoder+ Proposed Adaptations

25

12.5

0

Ainu

Griko

YakkhaKwak'wala

25

Experiments: do the adaptions help low-resource learning?

Lower isbetter!

% Word Error Rate

50

37.5

First pass OCR
Encoder-decoder+ Proposed Adaptations

38%
reduction
34%
reduction

25
17%
reduction

52%
reduction

12.5

0

Ainu

Griko

YakkhaKwak'wala

25

Experiments: do the adaptions help low-resource learning?

Lower isbetter!

% Word Error Rate

50

37.5

First pass OCR
Encoder-decoder+ Proposed Adaptations

38%
reduction
34%
reduction

25
17%
reduction

52%
reduction

12.5

0Which adaptations were most helpful?

Ainu

Griko

YakkhaAblations: all adaptations are useful, copy
mechanism impacts performance the mostKwak'wala26

Improving performance without additional annotation

Current model

26

Improving performance without additional annotation

Supervised training

Very small number ofmanually transcribed pages

Current model

27

Improving performance without additional annotation

Very small number ofmanually transcribed pages

Rijhwani, Rosenblum, Anastasopoulos, Neubig. TACL 2021.

27

Improving performance without additional annotation

Very small number ofmanually transcribed pages

Relatively larger numberof raw images that need
to be digitizedRijhwani, Rosenblum, Anastasopoulos, Neubig. TACL 2021.

27

Improving performance without additional annotation

Our dataset:
Documents contain
300 – 800 pages
Very small number ofmanually transcribed pages

Only ~30 are
manually transcribed
Relatively larger numberof raw images that need
to be digitizedRijhwani, Rosenblum, Anastasopoulos, Neubig. TACL 2021.

27

Improving performance without additional annotation

Very small number ofmanually transcribed pages

Relatively larger numberof raw images that need
to be digitizedSemi-supervised learningfor ef cient use of theunlabeled imagesfi

Rijhwani, Rosenblum, Anastasopoulos, Neubig. TACL 2021.

28

Self-training for OCR post-correction

28

Self-training for OCR post-correction

First pass OCR on
unlabeled images

28

Self-training for OCR post-correction

Trained postcorrection modelFirst pass OCR on
unlabeled images

28

Self-training for OCR post-correction

Trained postcorrection modelFirst pass OCR on
unlabeled images

Previous bestsupervised model28

Self-training for OCR post-correction

Trained postcorrection modelFirst pass OCR on
unlabeled images

Post-correction
predictions

29

Self-training for OCR post-correction
Retrain the model

Trained postcorrection modelFirst pass OCR on
unlabeled images

Post-correction
predictions

30

Self-training may introduce noise

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)g·ilᵋmēsē (5)…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)g·ilᵋmēsē (5)g·îlᵋmᵋsē (2)…

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)g·ilᵋmēsē (5)g·îlᵋmᵋsē (2)…

g·îlᵋmiᵋsē (2)30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)g·ilᵋmēsē (5)g·îlᵋmᵋsē (2)…

g·îlᵋmiᵋsē (2)}Different subsets
of characters areincorrect

Empirical observations

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)g·ilᵋmēsē (5)g·îlᵋmᵋsē (2)…

g·îlᵋmiᵋsē (2)}Noise from self-training
is typically inconsistentat the word-level

Empirical observations

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)g·ilᵋmēsē (5)g·îlᵋmᵋsē (2)…

g·îlᵋmiᵋsē (2)}Correct form of theword ends up being
more frequent
Noise from self-training
is typically inconsistentat the word-level

Empirical observations

30

Self-training may introduce noise
Can we bias post-correction towards generating correct words?

g·îlᵋmēsē (7)Can we use the word frequency
information to bias the model
towards correct forms?
…

g·ilᵋmēsē (5)g·îlᵋmᵋsē (2)g·îlᵋmiᵋsē (2)Correct form of theword ends up being
more frequent31

Incorporating word frequency information

31

Incorporating word frequency information

P(y) = plstm(y)

pfreq

31

Incorporating word frequency information

P(y) = plstm(y)Next character
probability

pfreq

31

Incorporating word frequency information

P(y) = plstm(y)Next character
probability

Decoder probability

pfreq

31

Incorporating word frequency information

P(y) = plstm(y)pfreqFrequency-based
probability to explicitly
bias the model

31

Incorporating word frequency information

P(y) = plstm(y)pfreq
How do we get
probabilities based onword frequency?

32

Modeling word frequency

32

Modeling word frequency
Simple model for word frequency: count-based language model

32

Modeling word frequency
Simple model for word frequency: count-based language model

Predictions from self-training32

Modeling word frequency
Simple model for word frequency: count-based language model

Train a smoothed
unigram LM

Predictions from self-training32

Modeling word frequency
Simple model for word frequency: count-based language model

Train a smoothed
unigram LM

Predictions from self-training

Using counts of
the word forms

32

Modeling word frequency
Simple model for word frequency: count-based language model

Train a smoothed
unigram LM

Predictions from self-training

Frequency-based wordlevel probabilities

32

Modeling word frequency
Simple model for word frequency: count-based language model

Train a smoothed
unigram LM

Predictions from self-training

Frequency-based wordlevel probabilities

Noisy weighted
lexicon of the words
in the predictions

32

Modeling word frequency
Simple model for word frequency: count-based language model

Train a smoothed
unigram LM

Predictions from self-training

Frequency-based wordlevel probabilities

33

Lexically-aware decoding for post-correction

33

Lexically-aware decoding for post-correction

P(y) = plstm(y)

pfreq

33

Lexically-aware decoding for post-correction

P(y) = plstm(y)pfreq
We have frequencybased probabilities
from the unigram LM!

33

Lexically-aware decoding for post-correction

P(y) = plstm(y)pfreq
But these are at the wordlevel: how we getcharacter-level scores?

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

34

Scoring at the character-level
Weighted Finite State Automaton (WFSA)
representation of the LM

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

34

Scoring at the character-level
Weighted Finite State Automaton (WFSA)
representation of the LM
Set of states withweighted transitions

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

34

Scoring at the character-level
Weighted Finite State Automaton (WFSA)
representation of the LM

start

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

34

Scoring at the character-level
Weighted Finite State Automaton (WFSA)
representation of the LM

start

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

34

Scoring at the character-level
Weighted Finite State Automaton (WFSA)
representation of the LM

d
start

5
7
.
/0

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

34

Scoring at the character-level
Weighted Finite State Automaton (WFSA)
representation of the LM

d
start

5
7
.
/0

o

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LM

d
start

5
7
.
/0

o

g

1

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0

o

g

1

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0

o

g

Probability=1.0 Probability=1.0

134

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0

o

g

Probability=1.0 Probability=1.0

1Weight of path for “dog” = 0.75
Same as the word-level LM!

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0

o

g

1

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0
d/

0. 2

o

g

1

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0
d/

0. 2

o

o

g

1

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0
d/

0. 2

o

g

o

o

1

34

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Weighted Finite State Automaton (WFSA)
representation of the LMdstart
Probability
from the countbased LM

5
7
.
/0
d/

0. 2

o

g

o

o

1

r

2

35

Scoring at the character-level

d
start

5
7
.
/0
d/

0. 2

o

g

o

o

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

1

r

2

35

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

1

r

2

35

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

1

r

2

35

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

1

r

2

35

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

1

r

2

35

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

1

r

2

35

Scoring at the character-level
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r _ d
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

1

r

2

35

Scoring at the character-level
space, punctuation
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r _ d
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

space, punctuation

1

r

2

35

Scoring at the character-level
space, punctuation
Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

Output:d o o r _ d
d
start

5
7
.
/0
d/

0. 2

o

g

o

o

space, punctuation

1

r

2

36

Scoring at the character-level
space, punctuation

d/
start

5
7
0.

d/

0. 2

o

g

o

o

space, punctuation

1

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

r

2

36

Scoring at the character-level
space, punctuation
unk

d/
start

5
7
0.

d/

0. 2

o

g

o

o

space, punctuation

1

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

r

2

36

Scoring at the character-level
space, punctuationunkЄ / 0.05

d/start

P(<unk>)

5
7
0.

d/

0. 2

o

g

o

o

space, punctuation

1

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

r

2

36

Scoring at the character-level
Character n-gram
language model to score
unknown sequences

space, punctuationunkЄ / 0.05

d/
start

5
7
0.

d/

0. 2

o

g

o

o

space, punctuation

1

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

r

2

36

Scoring at the character-level
Character n-gram
language model to score
unknown sequences

space, punctuationunkЄ / 0.05

d/
start

5
7
0.

d/

0. 2

o

g

o

o

Determinization andminimization for a compact
and ef cient representationfispace, punctuation

1

Example LM with two words:• P(‘dog’) = 0.75
• P(‘door’) = 0.2• P(<unk>) = 0.05

r

2

37

Lexically-aware decoding for post-correction

37

Lexically-aware decoding for post-correction

P(y) =

plstm(y)pfreq(y)37

Lexically-aware decoding for post-correction

P(y) =

plstm(y)pfreq(y)37

Lexically-aware decoding for post-correction

P(y) =

plstm(y)pfreq(y)37

Lexically-aware decoding for post-correction3r

1 o 2 g 4

P(y) =

plstm(y)pwfsa(y)37

Lexically-aware decoding for post-correction3r

1 o 2 g 4

P(y) =

plstm(y)pwfsa(y)WFSA representation gives
character-level scores

38

Lexically-aware decoding for post-correction
Linear interpolation to combine the probabilities for joint inference3r

1 o 2 g 4

P(y) = (1 − λ) * plstm(y) + λ * pwfsa(y)38Lexically-aware decoding for post-correction
Linear interpolation to combine the probabilities for joint inference3r

1 o 2 g 4

P(y) = (1 − λ) * plstm(y) + λ * pwfsa(y)Hyperparameter for
linear interpolation

39

Experiments: does self-training improve performance?

39

Experiments: does self-training improve performance?

% Word Error Rate

40

First passSupervised model
Self-training

30

20

10

0

Ainu

Griko

YakkhaKwak'wala

39

Experiments: does self-training improve performance?

% Word Error Rate

40

First passSupervised model
Self-training

30

20

10

0

Ainu

Griko

YakkhaKwak'wala

39

Experiments: does self-training improve performance?

% Word Error Rate

40

First passSupervised model
Self-training

30

20

10

0

Ainu

Griko

YakkhaKwak'wala

39

Experiments: does self-training improve performance?

% Word Error Rate

40

First passSupervised model
Self-training

30

20

10

0

Ainu

Griko

YakkhaKwak'wala

39

Experiments: does self-training improve performance?

% Word Error Rate

40

First passSupervised model
Self-training

13%
reduction

30

15%
reduction

20
2.3%
increase

14%
reduction

10

0

Ainu

Griko

YakkhaKwak'wala

39

Experiments: does self-training improve performance?

% Word Error Rate

40

First passSupervised model
Self-training

13%
reduction

30

15%
reduction

20
2.3%
increase

14%
reduction

10

0

Ainu

Griko

YakkhaKwak'wala40

Experiments: does lexically-aware decoding counteract noise?

% Word Error Rate

40

First passSupervised model
Self-training+ Lexically-aware decoding

30

20

10

0

Ainu

Griko

YakkhaKwak'wala40

Experiments: does lexically-aware decoding counteract noise?

% Word Error Rate

40

First passSupervised model
Self-training+ Lexically-aware decoding

18%
reduction

30
22%
reduction

20
15%
reduction

15%
reduction

10

0

Ainu

Griko

YakkhaKwak'wala40

Experiments: does lexically-aware decoding counteract noise?

% Word Error Rate

40

First passSupervised model
Self-training+ Lexically-aware decoding

18%
reduction

30
22%
reduction

20
15%
reduction

15%
reduction

10

0

Ainu

Griko

YakkhaKwak'wala41

Summary41

Summary• Thousands of languages do not have easily accessible text to build NLP models

41

Summary• Thousands of languages do not have easily accessible text to build NLP models• Text data does exist in many of these languages!

41

Summary• Thousands of languages do not have easily accessible text to build NLP models• Text data does exist in many of these languages!
• Locked away in non-machine-readable formats like printed books

41

Summary• Thousands of languages do not have easily accessible text to build NLP models• Text data does exist in many of these languages!
• Locked away in non-machine-readable formats like printed books• OCR post-correction improves text extraction in very low-resourced settings

41

Summary• Thousands of languages do not have easily accessible text to build NLP models• Text data does exist in many of these languages!
• Locked away in non-machine-readable formats like printed books• OCR post-correction improves text extraction in very low-resourced settings
Multi-source model: ↓ WER 17% – 52%

41

Summary• Thousands of languages do not have easily accessible text to build NLP models• Text data does exist in many of these languages!
• Locked away in non-machine-readable formats like printed books• OCR post-correction improves text extraction in very low-resourced settings
Multi-source model: ↓ WER 17% – 52%
Semi-supervised with lexically-aware decoding: ↓ WER 29% – 59%

42

Impact case study: Kwak’wala42

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from

42

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation

Produced by Franz
Boas in 1921

42

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation• Tremendous cultural and linguistic value!

42

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation• Tremendous cultural and linguistic value!
• Minimally accessible to researchers42

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation• Tremendous cultural and linguistic value!
• Minimally accessible to researchers• Manual search in scanned images

42

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation• Tremendous cultural and linguistic value!
• Minimally accessible to researchers• Manual search in scanned images• Legacy orthography that is hard to read

43

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation

43

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation• 1500+ pages converted to a machinereadable format

43

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation• 1500+ pages converted to a machinereadable format• Searchable!

43

Impact case study: Kwak’wala• Collaborating with documentary linguists and Kwak’wala speakers• Identify documents that would be most useful to extract text from
Boas texts: 10 volumes of Kwak’wala language and community documentation• 1500+ pages converted to a machinereadable format• Searchable!
• Legacy orthography can be automaticallytransliterated to modern writing systems

43

Impact case study: Kwak’wala• 1500+ pages converted to a machinereadable format• Searchable!
• Legacy orthography can be automaticallytransliterated to modern writing systems

44

Impact and applications: beyond this talk44

Impact and applications: beyond this talkOur software is open-source and has
been used on many other languages!

44

Impact and applications: beyond this talkOur software is open-source and has
been used on many other languages!

BhutiaSanskrit Quechua
Igbo Tibetan
Piaroa Secwepemctsín
Pintupi-Luritja

Image credits: Ijemma Onwuzulike, Jorge Labrada, Ben Foley

44

Impact and applications: beyond this talkOur software is open-source and has
been used on many other languages!

Extracting text to train
machine translation for
Pintupi-Luritja

BhutiaSanskrit Quechua
Igbo Tibetan
Piaroa Secwepemctsín
Pintupi-Luritja

Image credits: Ijemma Onwuzulike, Jorge Labrada, Ben Foley

44

Impact and applications: beyond this talkOur software is open-source and has
been used on many other languages!

Automatic extraction of
handwritten speechtranscriptions in Piaroa

BhutiaSanskrit Quechua
Igbo Tibetan
Piaroa Secwepemctsín
Pintupi-Luritja

Image credits: Ijemma Onwuzulike, Jorge Labrada, Ben Foley

44

Impact and applications: beyond this talkOur software is open-source and has
been used on many other languages!

BhutiaSanskrit Quechua
Igbo Tibetan
Piaroa Secwepemctsín
Pintupi-Luritja

Print dictionaries in
Igbo are high-coverage,
but not digitized
Image credits: Ijemma Onwuzulike, Jorge Labrada, Ben Foley

45

Unlocking Bi-lingual Lexicons

Xinyi Wang, Sebastian Ruder, Graham Neubig.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation.
ACL 2022.

46

Multilingual Pretrained Models

Pretrained Model

Eng Task

T TaskFinetune

Inference

Pretrain/Finetune

46

Multilingual Pretrained Models

Pretrained Model

Eng Task

T TaskFinetune

Inference

Pretrain/Finetune

How to adapt the model for the language T?

47

Adaptation: Monolingual Data

T MonoPretrained Model

MLM• e.g. Continued Masked Language Modeling(MLM) using monolingual data in the target
language T

48

Adaptation: Parallel Data

Eng-T
Parallel

NMT

Eng Task

T TaskPretrained Model

Finetune• e.g. Parallel Data: use best NMT system available to translate English task data into thetarget language T

49Languages without Conventional Data
1% Covered by mBERT
4% Wikipedia/CommonCrawl

23%

77%

Bible
Other

49

Languages without Conventional Data
1% Covered by mBERT
4% Wikipedia/CommonCrawl

23%

77%

Bible
Other

• Majority of languages in the world cannot bene t from progress in NLP due to lack of

fi

data

50

Two Groups of Low-resource Languages

50

Two Groups of Low-resource Languagesfi• Majority of World’s languages cannot bene t from progress in NLP (Joshi et al. 2020)

50

Two Groups of Low-resource Languages

No-Text
80% of languagesfi• Majority of World’s languages cannot bene t from progress in NLP (Joshi et al. 2020)• No-Text: virtually no resource

50

Two Groups of Low-resource Languages

No-Text
80% of languages

Few-Text
5% of languagesfi• Majority of World’s languages cannot bene t from progress in NLP (Joshi et al. 2020)• No-Text: virtually no resource• Few-Text: very limited resource

51

Alternative Data Source

https://panlex.org/

51

Alternative Data Source• Linguists have been documenting languages for years in formats such as lexicons

https://panlex.org/

51

Alternative Data Source• Linguists have been documenting languages for years in formats such as lexicons• PanLex: open-sourced database of lexicons with much better language coverage
https://panlex.org/

51

Alternative Data Source
1% Covered by mBERT
4% Wikipedia/CommonCrawl

23%

77%

Bible
Other
Lexicon• Linguists have been documenting languages for years in formats such as lexicons• PanLex: open-sourced database of lexicons with much better language coverage
https://panlex.org/

52

Synthesizing Data Using Lexicons
T MonoPretrained Model

PseudoMono

LexiconEng Mono

Pseudo MLM

52

Synthesizing Data Using Lexicons
T MonoPretrained Model

PseudoMono

LexiconEng Mono

Pseudo MLM• Pseudo Mono Data: replace words in English monolingual data to its
corresponding translation in the target language T

52Synthesizing Data Using Lexicons
T MonoPretrained Model

PseudoMono

LexiconEng Mono

Pseudo MLM

Eng Mono

Anarchism calls for the abolition of the state , which it holds to be undesirable , unnecessary , and harmful .

Pseudo Mono

Anarchism calls ghal il abolition ta’ il stat , lima hi holds ghal tkun undesirable , bla bzonn , u harmful .

• Pseudo Mono Data: replace words in English monolingual data to its
corresponding translation in the target language T53Synthesizing Data Using Lexicons
LexiconPretrained Model

Eng Task

PseudoTask

Pseudo Fine-tune

53

Synthesizing Data Using Lexicons
LexiconPretrained Model

Eng Task

PseudoTask

Pseudo Fine-tune• Pseudo Task Data: replace words in English task data to its correspondingtranslation in the target language T53Synthesizing Data Using Lexicons
LexiconPretrained Model

Eng Task

PseudoTask

Pseudo Fine-tune

Eng Task

I suspect the streets of Baghdad will look as if a war is looming this week .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT

Pseudo Task

jien iddubita il streets ta’ Bagdad xewqa hares kif jekk a gwerra is looming dan gimgha .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT• Pseudo Task Data: replace words in English task data to its correspondingtranslation in the target language T

54Synthesizing Data Using Lexicons
T MonoPretrained Model

PseudoMono

LexiconEng Mono

Pseudo MLM

Eng Task

PseudoTask

Pseudo Fine-tune

54

Synthesizing Data Using Lexicons
T MonoPretrained Model

PseudoMono

LexiconEng Mono

Pseudo MLM• Use either pseudo MLM or Pseudo Fine-tune, or both

Eng Task

PseudoTask

Pseudo Fine-tune

55

Experiments

55

Experiments• Model: mBERT

55

Experiments• Model: mBERT• Tasks:• NER• POS tagging• Dependency Parsing

55

Experiments• Model: mBERT• Tasks:• NER• POS tagging• Dependency Parsing• Languages: 19 languages not covered by mBERT pretraining

56

Results

56

F1 gain with synthetic
data

Results
No-text

16

Few-text

12
8
4
0
NER

POS

Parsing

56

F1 gain with synthetic
data

Results
No-text

16

Few-text

12
8
4
0
NER

POS

Parsing

56

F1 gain with synthetic
data

Results
No-text

16

Few-text

12
8
4
0
NER

POS

Parsing

56

F1 gain with synthetic
data

Results
No-text

16

Few-text

12
8
4
0
NER

POS

Parsing• Using synthetic data leads to signi cant improvements for both no-text and few-textfisetting

57

Label NoiseEng Task

I suspect the streets of Baghdad will look as if a war is looming this week .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT

Pseudo Task

jien iddubita il streets ta’ Bagdad xewqa hares kif jekk a gwerra is looming dan gimgha .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT57

Label NoiseEng Task

I suspect the streets of Baghdad will look as if a war is looming this week .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT

Pseudo Task

jien iddubita il streets ta’ Bagdad xewqa hares kif jekk a gwerra is looming dan gimgha .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT• “xewqa” is a noun meaning “desire,will”

57

Label NoiseEng Task

I suspect the streets of Baghdad will look as if a war is looming this week .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT

Pseudo Task

jien iddubita il streets ta’ Bagdad xewqa hares kif jekk a gwerra is looming dan gimgha .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT• “xewqa” is a noun meaning “desire,will”• But the original English POS tag is inconsistent with the replaced word

58

Label Noisefi• Use the ne-tuned model to “correct” the labels for the Pseudo task data

58

Label NoiseT MonoPretrained Model

PseudoMono

LexiconEng Mono

Pseudo MLM

Eng Task

PseudoTask

Pseudo Fine-tunefi• Use the ne-tuned model to “correct” the labels for the Pseudo task data

58

Label NoiseT MonoPseudoMono

LexiconEng Mono

Eng Task

PseudoTask

Label Distill
Pretrained Model

Pseudo MLM

Pseudo Fine-tunefi• Use the ne-tuned model to “correct” the labels for the Pseudo task data

59

Label NoiseEng Task

I suspect the streets of Baghdad will look as if a war is looming this week .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT

Pseudo Task

jien iddubita il streets ta’ Bagdad xewqa hares kif jekk a gwerra is looming dan gimgha .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT• “xewqa” is a noun meaning “desire,will”

59

Label NoiseEng Task

I suspect the streets of Baghdad will look as if a war is looming this week .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT

Pseudo Task

jien iddubita il streets ta’ Bagdad xewqa hares kif jekk a gwerra is looming dan gimgha .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCTLabel Distilled

PRON VERB DET NOUN ADP PROPN NOUN NOUN SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT• “xewqa” is a noun meaning “desire,will”

59

Label NoiseEng Task

I suspect the streets of Baghdad will look as if a war is looming this week .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT

Pseudo Task

jien iddubita il streets ta’ Bagdad xewqa hares kif jekk a gwerra is looming dan gimgha .
PRON VERB DET NOUN ADP PROPN AUX VERB SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCTLabel Distilled

PRON VERB DET NOUN ADP PROPN NOUN NOUN SCONJ SCONJ DET NOUN AUX VERB DET NOUN PUNCT• “xewqa” is a noun meaning “desire,will”• The model is able to assign the correct label of noun

60

Label Noise60

F1 gain by Label Distillation

Label Noise
No-text

6

Few-text

4.5
3
1.5
0
NER

POS

Parsing

60

F1 gain by Label Distillation

Label Noise
No-text

6

Few-text

4.5
3
1.5
0
NER

POS

Parsing

60

F1 gain by Label Distillation

Label Noise
No-text

6

Few-text

4.5
3
1.5
0
NER

POS

• Label Distillation is especially helpful for syntactic tasks

Parsing61

Comparison to Few-shot Learning

61

F1 gain over mBERT
on NER

Comparison to Few-shot Learning
30

Best Adapted

10-shot

100-shotBest Adapted+100-shot

22.5
15
7.5
0
-7.5
hau

wol

lug

ibo

61

F1 gain over mBERT
on NER

Comparison to Few-shot Learning
30

Best Adapted

10-shot

100-shotBest Adapted+100-shot

22.5
15
7.5
0
-7.5
hau

wol

lug

ibo

61

F1 gain over mBERT
on NER

Comparison to Few-shot Learning
30

Best Adapted

10-shot

100-shotBest Adapted+100-shot

22.5
15
7.5
0
-7.5
hau

wol

lug

ibo

61

F1 gain over mBERT
on NER

Comparison to Few-shot Learning
30

Best Adapted

10-shot

100-shotBest Adapted+100-shot

22.5
15
7.5
0
-7.5
hau

wol

lug

ibo

61

F1 gain over mBERT
on NER

Comparison to Few-shot Learning
30

Best Adapted

10-shot

100-shotBest Adapted+100-shot

22.5
15
7.5
0
-7.5
hau

wol

lug

ibo

• Few-shot learning needs more annotated data for languages with limited text

61

F1 gain over mBERT
on NER

Comparison to Few-shot Learning
30

Best Adapted

10-shot

100-shotBest Adapted+100-shot

22.5
15
7.5
0
-7.5
hau

wol

lug

ibo

• Few-shot learning needs more annotated data for languages with limited text

61

F1 gain over mBERT
on NER

Comparison to Few-shot Learning
30

Best Adapted

10-shot

100-shotBest Adapted+100-shot

22.5
15
7.5
0
-7.5
hau

wol

lug

ibo

• Few-shot learning needs more annotated data for languages with limited text
• Combining adaptation and few-shot doesn’t bring consistent improvements

62

Conclusion

63

Conclusion

63

Conclusion

63

Conclusion• Methods to unlock new resources for human or machine use in under-resourced languages

63

Conclusion• Methods to unlock new resources for human or machine use in under-resourced languages• What’s next?

63Conclusion• Methods to unlock new resources for human or machine use in under-resourced languages• What’s next?
• Should we put some linguistics in the models?

63

Conclusion• Methods to unlock new resources for human or machine use in under-resourced languages• What’s next?
• Should we put some linguistics in the models?
• Morphologically aware soft constraints for OCR?

63

Conclusion• Methods to unlock new resources for human or machine use in under-resourced languages• What’s next?
• Should we put some linguistics in the models?
• Morphologically aware soft constraints for OCR?
• Morphologically/syntactically aware data synthesis using lexicons?

63

Conclusion• Methods to unlock new resources for human or machine use in under-resourced languages• What’s next?
• Should we put some linguistics in the models?
• Morphologically aware soft constraints for OCR?
• Morphologically/syntactically aware data synthesis using lexicons?
• Should we use the models in language learning or linguistics?

63

Conclusion• Methods to unlock new resources for human or machine use in under-resourced languages• What’s next?
• Should we put some linguistics in the models?
• Morphologically aware soft constraints for OCR?
• Morphologically/syntactically aware data synthesis using lexicons?
• Should we use the models in language learning or linguistics?
• Large-scale extraction of text or inter-linear glosses for use in developing language materials?