Can we Automatically Create
Language-learning Textbooks?
https://www.autolex.co/
Graham Neubig
w/ Aditi Chaudhary
Antonios Anastasopoulos, David Mortensen,
Zaid Sheikh, Arun Sampath, Ashwin Sheshadri,
Adithya Pratapa, Yulia Tsvetkov, Kayo Yinsource: https://www.reddit.com/r/linguistics/comments/pusln/world_languages_map_xpost_from_rmapporn/2source: https://www.reddit.com/r/linguistics/comments/pusln/world_languages_map_xpost_from_rmapporn/

Remarkable language diversity!
2source: https://www.reddit.com/r/linguistics/comments/pusln/world_languages_map_xpost_from_rmapporn/

Remarkable language diversity!
2

en
mr

source: https://www.reddit.com/r/linguistics/comments/pusln/world_languages_map_xpost_from_rmapporn/

Remarkable language diversity!
2

en
mr

SOV

source: https://www.reddit.com/r/linguistics/comments/pusln/world_languages_map_xpost_from_rmapporn/

Remarkable language diversity!
2

en
mr

SOV

source: https://www.reddit.com/r/linguistics/comments/pusln/world_languages_map_xpost_from_rmapporn/

Remarkable language diversity!
2

SVO
en
mr

SOV

source: https://www.reddit.com/r/linguistics/comments/pusln/world_languages_map_xpost_from_rmapporn/

Remarkable language diversity!
2

Different Systems Governing Language

Syntax3

Different Systems Governing Language

Syntax3Morphology

Different Systems Governing Language

eating

Syntax3eat+Morphology

ingDifferent Systems Governing Language

eating

eat

+

ing

खाते

खा

+

ते

/khate/

/kha/

/te/

+

तो
/to/

Syntax3Morphology

Different Systems Governing Language

eating

eat

+

ing

खाते

खा

+

ते

/khate/

/kha/

/te/

+

तो
/to/

Syntax3Morphology

Semantics

Different Systems Governing Language

eating

eat

+

ing

खाते

खा

+

ते

/khate/

/kha/

/te/

+

तो
/to/

Syntax3Morphology

Semantics

Different Systems Governing Language

eating

eat

+

ing

तां ळ
tandul‘raw rice’

खाते

खा

/khate/

/kha/

+

ते
/te/

+

तो
/to/

Syntaxदू

3Morphology

भात
bhaat
‘cooked rice’

Semantics

Describing Systems of Language4Describing Systems of Language

4

Language LearningDescribing Systems of Language

4

Language LearningDescribing Systems of Language

4

Language LearningDescribing Systems of Language

4

Language LearningDescribing Systems of Language

4

Language Learning

Language Documentation

Describing Systems of Language

4

Language Learning

Language Documentation

What are Language Descriptions?

आना

हे

5

सफरचंद

खातेआWhat are Language Descriptions?

आनासफरचंद

खाते

आ

Anna

apple

eatingisHow are subjects ordered
wrt verbs in Marathi?

हे5

What are Language Descriptions?

आनासफरचंद

खाते

आ

Anna

apple

eating

is

PROPN

NOUN

VERB

AUX

How are subjects ordered
wrt verbs in Marathi?

हे5

What are Language Descriptions?
subj
obj

आना

सफरचंद

खाते

आ

Anna

apple

eating

is

PROPN

NOUN

VERB

AUX

How are subjects ordered
wrt verbs in Marathi?

हे5

What are Language Descriptions?
subj
obj

आना

सफरचंद

खाते

आ

Anna

apple

eating

is

PROPN

NOUN

VERB

AUX

How are subjects ordered
wrt verbs in Marathi?

हे5

Subject

ObjectVerb

Why is this challenging?

How are subjects ordered
wrt verbs in Marathi?

6

Subject

ObjectVerb

Why is this challenging?
How are subjects ordered
wrt verbs in Marathi?

Subject
6

ObjectVerb

Why is this challenging?
How are subjects ordered
wrt verbs in Marathi?

Subject
6

ObjectVerb

Why is this challenging?
How are subjects ordered
wrt verbs in Marathi?

Subject
6

ObjectVerb

Could we help in some way?

NLP already does some of this!
subj
obj

आना

सफरचंद

खाते

आ

Anna

apple

eating

is

PROPN

NOUN

VERB

AUX

How are subjects ordered
wrt verbs in Marathi?

हे7

Subject

ObjectVerb

NLP already does some of this!
subj
obj

Dependency Parsingआनासफरचंद

खाते

आ

Anna

apple

eating

is

PROPN

NOUN

VERB

AUX

How are subjects ordered
wrt verbs in Marathi?

हे7

Subject

ObjectVerb

POS Tagging

Our Proposed Pipeline

Chaudhary, et al. "Automatic extraction of rules governing morphological agreement." EMNLP (2020).
Chaudhary, et al. "When is Wall a Pared and when a Muro?--Extracting Rules Governing Lexical Selection." EMNLP (2021).
Chaudhary, et al. "AUTOLEX: An Automatic Framework for Linguistic Exploration." arXiv (2022).
Chaudhary, et al. "Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning." arXiv (2022).

8

Our Proposed Pipeline(Low-resource) Language
Analysis

Chaudhary, et al. "Automatic extraction of rules governing morphological agreement." EMNLP (2020).
Chaudhary, et al. "When is Wall a Pared and when a Muro?--Extracting Rules Governing Lexical Selection." EMNLP (2021).
Chaudhary, et al. "AUTOLEX: An Automatic Framework for Linguistic Exploration." arXiv (2022).
Chaudhary, et al. "Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning." arXiv (2022).

8

Our Proposed Pipeline(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectVerb

Chaudhary, et al. "Automatic extraction of rules governing morphological agreement." EMNLP (2020).
Chaudhary, et al. "When is Wall a Pared and when a Muro?--Extracting Rules Governing Lexical Selection." EMNLP (2021).
Chaudhary, et al. "AUTOLEX: An Automatic Framework for Linguistic Exploration." arXiv (2022).
Chaudhary, et al. "Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning." arXiv (2022).

8

Our Proposed Pipeline(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectVerb

Chaudhary, et al. "Automatic extraction of rules governing morphological agreement." EMNLP (2020).
Chaudhary, et al. "When is Wall a Pared and when a Muro?--Extracting Rules Governing Lexical Selection." EMNLP (2021).
Chaudhary, et al. "AUTOLEX: An Automatic Framework for Linguistic Exploration." arXiv (2022).
Chaudhary, et al. "Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning." arXiv (2022).

8

Our Proposed Pipeline(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation
Chaudhary, et al. "Automatic extraction of rules governing morphological agreement." EMNLP (2020).
Chaudhary, et al. "When is Wall a Pared and when a Muro?--Extracting Rules Governing Lexical Selection." EMNLP (2021).
Chaudhary, et al. "AUTOLEX: An Automatic Framework for Linguistic Exploration." arXiv (2022).
Chaudhary, et al. "Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning." arXiv (2022).

8

AutoLEX: Automatic Language Explorer

9AutoLEX: Automatic Language Explorer
Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats

9AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

9

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats

AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Text corpus

9Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats

AutoLEX: Automatic Language Explorer
Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats

How are subjects
ordered in Marathi?

Text corpus

9

AutoLEXAutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Formulate the linguistic question
into a classi cation task

Extract Features andConstruct Training Data

Text corpus

Learn an Interpretable Model

Extract and Visualize Rulesfi

9AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Formulate the linguistic question
into a classi cation task

Extract Features andConstruct Training Data

Text corpus

Learn an Interpretable Model

Extract and Visualize Rulesfi

9AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Formulate the linguistic question
into a classi cation task

Extract Features andConstruct Training Data

Text corpus

Learn an Interpretable Model

Extract and Visualize Rulesfi

9AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Formulate the linguistic question
into a classi cation task

Extract Features andConstruct Training Data

Text corpus

Learn an Interpretable Model

Extract and Visualize Rulesfi

9AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Formulate the linguistic question
into a classi cation task

Extract Features andConstruct Training Data

Text corpus

Learn an Interpretable Model

Extract and Visualize Rulesfi

9AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Predict relevant features(e.g. POS tags, dependency tree)

Formulate the linguistic question

Construct Training Data

Expert annotations
for feature extraction model

Learn an Interpretable Model

Expert annotations(e.g. UD Treebank)

10

Extract and Visualize Rules

AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Predict relevant features(e.g. POS tags, dependency tree)

Formulate the linguistic question

Construct Training Data

Expert annotations
for feature extraction model

Learn an Interpretable Model

Expert annotations(e.g. UD Treebank)

10

Extract and Visualize Rules

AutoLEX: Automatic Language Explorer
How are subjects
ordered in Marathi?

Extract and visualize answers to different linguistic questions
in both human- and machine-readable formats
Predict relevant features(e.g. POS tags, dependency tree)

Formulate the linguistic question

Construct Training Data

Expert annotations
for feature extraction model

Learn an Interpretable Model

Expert annotations(e.g. UD Treebank)

10

Extract and Visualize Rules

Types of Grammar Aspects covered in AutoLEX

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word Order

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word OrderAf x Usagefi

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word OrderAf x Usage

Case Markingfi

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word OrderAf x Usage

Case Marking

Word Usagefi

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word OrderAf x Usage

Case Marking

Word Usagefi

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word Order
Morpho-Syntax

Af x Usage

Case Marking

Word Usagefi

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word Order
Morpho-Syntax

Af x Usage

Case Marking

Word Usagefi

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Types of Grammar Aspects covered in AutoLEX
Agreement

Word Order
Morpho-Syntax

Af x Usage

Case Marking

Lexical Semantics

fi

11

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Usage

Types of Grammar Aspects covered in AutoLEX
Linguistic Question

Agreement

Formulate the linguistic question
into a classi cation task

Word Order
Morpho-Syntax

Af x Usage

Extract Features andConstruct Training Data

Case Marking
Learn an Interpretable Model

Lexical Semantics
Extract and Visualize Rules
AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission
fi

fi

11

Word Usage

Types of Grammar Aspects covered in AutoLEX
Agreement

into a classi cation task

Extract Features andConstruct Training Data

Word Order

Af

Case Marking
Learn an Interpretable Model

Extract and Visualize Rules
AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submissionfi

11

Word Order

12

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order

12

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order

Only the most dominant order is annotated in WALS

12

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order

13

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order

Anna

13

is

eating

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

an

apple

Word Order

13

Anna

is

eating

an

What

is

Anna

eating

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

apple

?

Word Order
obj

Anna

is

eatinganapple

Verb-Object?

Object-Verb

VERB

obj

What

is

Anna

eatingVERB

13

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order
obj

Anna

is

eatinganapple

Verb-Object?

Object-Verb

VERB

obj

What

is

Anna

eatingVERBWhen are objects before or after the verbs?
13

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Formulate Linguistic Question

When are objects before or after the verbs?
14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Formulate Linguistic Question

When are objects before or after the verbs?

14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Formulate Linguistic Question

When are objects before or after the verbs?

14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Formulate Linguistic Question
Input

When are objects before or after the verbs?

14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

LabelWord Order: Formulate Linguistic Question
Input

f(

When are objects before or after the verbs?

14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

eating

apple

Label

)

Word Order: Formulate Linguistic Question
Input

f(

When are objects before or after the verbs?

14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

eating

apple

Label

)

BeforeWord Order: Formulate Linguistic Question
Input

When are objects before or after the verbs?

14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submissionf(eating

apple

f(

What

eatingLabel)

Before)Word Order: Formulate Linguistic Question
Input

When are objects before or after the verbs?

14

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submissionf(eating

apple

f(

What

eatingLabel)

Before)AfterWord Order: Formulate Linguistic Question
When are objects before or after the verbs in English?

15

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Formulate Linguistic Question
When are objects before or after the verbs in English?

Anna

15

is

eating

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

an

apple

Word Order: Formulate Linguistic Question
When are objects before or after the verbs in English?

Anna

What

15

eating

is

is

Anna

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

aneatingapple?

Word Order: Formulate Linguistic Question
When are objects before or after the verbs in English?

Anna

eating

is

an

VERB

What

is

Anna

PRON

15

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

apple

POS Tagging

NOUN

eatingVERB?

Word Order: Formulate Linguistic Question
When are objects before or after the verbs in English?

Dependency Parsing

obj

Anna

eating

is

an

VERB

apple

POS Tagging

NOUN

obj

What

is

Anna

PRON

15

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

eatingVERB?

Word Order: Formulate Linguistic Question
When are objects before or after the verbs in English?

Dependency Parsing

obj

Anna

eating

is

an

VERB

apple

POS Tagging

NOUN

Morphological Analysis

Tense=Pres

obj

What

15

is

Anna

eatingPRON

VERBPronType = Int

Tense=Pres

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission?

Word Order: Extract Features and Training Data
When are objects before or after the verbs in English?

16

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Extract Features and Training Data
When are objects before or after the verbs in English?

16

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Extract Features and Training Data
When are objects before or after the verbs in English?

Input

16

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

FeaturesLabel

Word Order: Extract Features and Training Data
When are objects before or after the verbs in English?

Input

f(

16

eatingFeaturesapple)

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Label

BeforeWord Order: Extract Features and Training Data
When are objects before or after the verbs in English?

Input

16

Featuresf(eating

apple

f(

What

eatingLabel

Before))AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

AfterWord Order: Extract Features and Training Data
When are objects before or after the verbs in English?

Input

f(
f(
16

eatingWhat

Featuresapple

eating))Label

is-object-NOUN

Beforeis-object-PRON

After

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Extract Features and Training Data
When are objects before or after the verbs in English?

Input

f(
f(

eatingWhat

Featuresapple

eating)is-object-NOUN)

is-verb-tense-present

is-object-PRON is-object-Interrogative
is-verb-tense-present

16

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Label

BeforeAfterWord Order: Learn a Model
When are objects before or after the verbs in English?

17

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Learn a Model
When are objects before or after the verbs in English?

root
is-object-PRON
is-object-Interrogative

17

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Learn a Model
When are objects before or after the verbs in English?

root
is-object-PRON
is-object-Interrogative

Leaf 1:
Before17

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Learn a Model
When are objects before or after the verbs in English?

root
is-object-PRON
is-object-Interrogative

Leaf 1:
Before17

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Leaf 1:AfterWord Order: Learn a Model
When are objects before or after the verbs in English?

root
is-object-PRON
is-object-Interrogative

Leaf 1:
BeforeLeaf 1:
After

Rule: When object is an interrogative pronoun, it comes BEFORE the verb.

17

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Data
When are objects before or after the verbs in English?

18

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Data
When are objects before or after the verbs in English?

• Compare the model in a clean setting ⟶ Syntactic Universal Dependencies (SUD)18

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Data
When are objects before or after the verbs in English?

• Compare the model in a clean setting ⟶ Syntactic Universal Dependencies (SUD)• Prefers syntactic heads over content heads ⟶ more conducive to our goal of rule extraction

18

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Data
When are objects before or after the verbs in English?

• Compare the model in a clean setting ⟶ Syntactic Universal Dependencies (SUD)• Prefers syntactic heads over content heads ⟶ more conducive to our goal of rule extraction• Expert-annotated syntactic analysis for >60 languages

18

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Automated Evaluation
When are objects before or after the verbs in English?

Apply modelon held-out sentences

19

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Automated Evaluation
When are objects before or after the verbs in English?

SyntaxBaseline

Apply modelon held-out sentences
Baseline: most frequentlabel in test data

Accuracy

99
98.5
98
97.5
97

19

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

en-object-verb

Word Order: Automated Evaluation
When are objects before or after the verbs in English?

SyntaxBaseline

Apply modelon held-out sentences
Baseline: most frequentlabel in test data

Accuracy

74
70.5
67
63.5
60

19

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

es-adjective-noun

Word Order: Automated Evaluation
When are objects before or after the verbs in English?

SyntaxBaseline

Apply modelon held-out sentences
Baseline: most frequentlabel in test data

Accuracy

74
70.5
67
63.5
60

Using syntactic signals insufficient!
19

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

es-adjective-noun

Word Order: Types of Features
grandes
great

20

librosbooks

fueronwere

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

compradosbought

por
by

la
the

niña
girl

pequeña
small

Word Order: Types of Features20

grandes
great

libros
books

ADJ

NOUN

fueronwere

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

compradosbought

por
by

la
the

niña
girl
NOUN

pequeña
small
ADJ

Word Order: Types of Features
mod

mod

20

grandes
great

libros
books

ADJ

NOUN

fueronwere

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

compradosbought

por
by

la
the

niña
girl
NOUN

pequeña
small
ADJ

Word Order: Types of Featuresmodsyntactic

mod20

grandes
great

librosbooks

fueron
were

ADJ

NOUN

NOUN

ADJ

is-adjis-ordinalhead-is-noun
deprel-is-mod

head-is-noun
deprel-is-mod

is-adj

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

compradosbought

por
by

la
the

niña
girl

pequeña
small

Word Order: Types of Featuresmod20

grandes
great

libros
books

ADJ

NOUN

NOUN

ADJ

syntactic

is-adjis-ordinalhead-is-noun
deprel-is-mod

head-is-noun
deprel-is-mod

is-adjlexicalmod

fueronwere

dep-lemma-is-granhead-lemma-is-libro

dep-lemma-is-niña

head-lemma-is-pequeño

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

compradosbought

por
by

la
the

niña
girl

pequeña
small

Word Order: Types of Features

21

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Types of Features

21

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Types of Features

21

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Types of Features

90

Baselinesyntactic
syntactic + lexical

Accuracy

82.5

75

67.5

60

21

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

es-adjective-noun

Word Order: Types of Features

21

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Types of Featuresroot
is-adj

Leaf 1:
Beforenode
is-adjdep-lemma-is-granLeaf 2:
Beforenode
is-adjdep-lemma-is-nuevo
Leaf 2:
Beforenode…
21

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission…

is-ordinal

Word Order: Types of Features22

Word Order: Types of Features• Word Embeddings ⟶ capture semantic/syntactic similarity

22

Word Order: Types of Features• Word Embeddings ⟶ capture semantic/syntactic similarity• BUT how to interpret what each dimension means?

22Word Order: Types of Features• Word Embeddings ⟶ capture semantic/syntactic similarity• BUT how to interpret what each dimension means?
1• SPINE ⟶ sparse and non-negative transformation over pretrained embeddings

22

1. SPINE: SParse Interpretable Neural Embeddings
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Eduard Hovy

Word Order: Types of Features• Word Embeddings ⟶ capture semantic/syntactic similarity• BUT how to interpret what each dimension means?
1• SPINE ⟶ sparse and non-negative transformation over pretrained embeddings

dim-0: wrist, shoulder, ligament, ankle, thigh22

1. SPINE: SParse Interpretable Neural Embeddings
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Eduard Hovy

Semantic similarity

Word Order: Types of Features• Word Embeddings ⟶ capture semantic/syntactic similarity• BUT how to interpret what each dimension means?
1• SPINE ⟶ sparse and non-negative transformation over pretrained embeddings22

dim-0: wrist, shoulder, ligament, ankle, thigh

Semantic similarity

dim-1: extensively, adequately, traditionally, royally, fully

Syntactic similarity

1. SPINE: SParse Interpretable Neural Embeddings
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Eduard Hovy

Word Order: Types of Features• Word Embeddings ⟶ capture semantic/syntactic similarity• BUT how to interpret what each dimension means?
1• SPINE ⟶ sparse and non-negative transformation over pretrained embeddings

dim-0: wrist, shoulder, ligament, ankle, thigh

Semantic similarity

dim-1: extensively, adequately, traditionally, royally, fully

Syntactic similarity

dim-2: torque, joystick, grip, wrist, swinging

22

1. SPINE: SParse Interpretable Neural Embeddings
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Eduard Hovy

Word Order: Types of Features• Word Embeddings ⟶ capture semantic/syntactic similarity• BUT how to interpret what each dimension means?
1• SPINE ⟶ sparse and non-negative transformation over pretrained embeddings22

dim-0: wrist, shoulder, ligament, ankle, thigh

Semantic similarity

dim-1: extensively, adequately, traditionally, royally, fully

Syntactic similarity

dim-2: torque, joystick, grip, wrist, swinging

Multi-Aspect Information

1. SPINE: SParse Interpretable Neural Embeddings
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Eduard Hovy

Word Order: Types of Featuresmod23

grandes
cuatrogreat
four

libros
books

ADJ

NOUN

syntactic

is-adjis-ordinalhead-is-noun
deprel-is-mod

head-is-noun
deprel-is-mod

lexicalmoddep-lemma-is-granhead-lemma-is-libro

dep-lemma-is-niña

fueronwere

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Compradosboughtpor laby the

niña
girl
NOUN

pequeña
small
ADJ

is-adjhead-lemma-is-pequeño

Word Order: Types of Featuresmod23

grandes
cuatrogreat
four

libros
books

ADJ

NOUN

syntactic

is-adjis-ordinalhead-is-noun
deprel-is-mod

head-is-noun
deprel-is-mod

semantic lexicalmoddep-lemma-is-granhead-lemma-is-libro

dep-lemma-is-niña

head-lemma-is-pequeño

word-is-likeprimera

head-word-is-likepágin, publicación

word-islike-niño

head-word-islike-nueva, vuelta

fueronwere

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Compradosboughtpor laby the

niña
girl
NOUN

pequeña
small
ADJ

is-adjWord Order: Types of Features

24

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Types of Features

24

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Types of Features

24

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Types of Features

Baselinesyntactic + lexcial

Accuracy

90
82.5
75
67.5
60

24

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

es-adjective-noun

syntactic
syntactic + semantic

Word Order: Extract rules

25

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Extract rules

Leaf 1:25

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Extract rules
Leaf 1:After: 1000
Leaf 1:25

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Before: 8000

Word Order: Extract rules
Leaf 1:After: 1000
Leaf 1:25

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Before: 1800

Word Order: Extract rules
Leaf 1:After: 1000

Before: 1800

Leaf 1:

Statistical threshold

Signi cancetest 2
Observed distribution
is significant

fi

𝝌

25

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

EffectSize
Magnitude ofsignificance is large

Word Order: Extract rules
Leaf 1:After: 1000

Before: 1800

Label: Before

Statistical threshold

Signi cancetest 2
Observed distribution
is significant

fi

𝝌

25

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

EffectSize
Magnitude ofsignificance is large

Word Order: Visualize rules

26

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Visualize rules

26

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Visualize rules

Dominant order in the corpus

26

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Visualize rules

Dominant order in the corpus

Features that lead to a leaf makes a rule

26

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Visualize rules
Rule: Adjective like “Primera” come before noun

27

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Visualize rules
Rule: Adjective like “Primera” come before noun

27

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Word Order: Visualize rules
Rule: Adjective like “Primera” come before noun

Exceptions!

27

AUTOLEX: An Automatic Framework for Linguistic Exploration
Chaudhary, Sheikh, Mortensen, Anastasopoulos, Neubig. In Submission

Agreement

28

Agreement

28

Automatic Extraction of Rules Governing Morphological Agreement
Chaudhary, Anastasopoulos, Pratapa, Mortensen, Sheikh,Tsvetkov, Neubig. EMNLP 2020

Agreement

Marathiहे28तो/that.M/

मुलगा
/boy/

जेवतो
/eating.M/

Automatic Extraction of Rules Governing Morphological Agreement
Chaudhary, Anastasopoulos, Pratapa, Mortensen, Sheikh,Tsvetkov, Neubig. EMNLP 2020

आ/is/

Agreement

Marathiहे28तो/that.M/मुलगी/girl/

जेवतो/eating.M/

Automatic Extraction of Rules Governing Morphological Agreement
Chaudhary, Anastasopoulos, Pratapa, Mortensen, Sheikh,Tsvetkov, Neubig. EMNLP 2020

आ/is/

Agreement

Marathiतो/that.M/मुलगी/girl/

जेवतो/eating.M/

आ
/is/

NOUN

VERB

AUX

det

DET

हे

28

Automatic Extraction of Rules Governing Morphological Agreement
Chaudhary, Anastasopoulos, Pratapa, Mortensen, Sheikh,Tsvetkov, Neubig. EMNLP 2020

Agreement

Marathiमुलगी/girl/

जेवतो/eating.M/

आ
/is/

DET

NOUN

VERB

AUX

ती
/that.F/मुलगी/girl/

जेवते/eating.F/

आ
/is/

तो
/that.M/
det

हे

हे

28

Automatic Extraction of Rules Governing Morphological Agreement
Chaudhary, Anastasopoulos, Pratapa, Mortensen, Sheikh,Tsvetkov, Neubig. EMNLP 2020

Agreement
When should a head-dependent agree on
gender and when it shouldn’t?

Marathiमुलगी/girl/

जेवतो/eating.M/

आ
/is/

DET

NOUN

VERB

AUX

ती
/that.F/मुलगी/girl/

जेवते/eating.F/

आ
/is/

तो
/that.M/
det

हे

हे

28

Automatic Extraction of Rules Governing Morphological Agreement
Chaudhary, Anastasopoulos, Pratapa, Mortensen, Sheikh,Tsvetkov, Neubig. EMNLP 2020

Quality Evaluation

29

Quality Evaluation

29

Quality Evaluation
Correctness29

Quality Evaluation
Correctness29

Quality Evaluation
Correctness

Prior knowledge29

Quality Evaluation
Correctness

Prior knowledge29

Quality Evaluation
Correctness

Prior knowledge

Readability

29

Percentage of rules

Results: Quality Evaluation for Greek

30

Results: Quality Evaluation for Greek

Percentage of rules

Does model discover grammaticallyvalid rules?

30

Results: Quality Evaluation for Greek
1

precisetoo-speci c

too-general

not-a-rule

0.75

Total: 161 rules

0.5

80% rules are valid,
40% valid rules too speci c/general

0.25

Percentage of rules

CaseMarking

WordOrder

fi

30
fi

Does model discover grammaticallyvalid rules?

Agreement

Results: Quality Evaluation for Greek
1

precisetoo-speci c

too-general

not-a-rule

0.75

Total: 161 rules

0.5

80% rules are valid,
40% valid rules too speci c/general

0.25

Percentage of rules

Does model discover grammaticallyvalid rules?

CaseMarking

WordOrder

Agreement

Are the rules human-readable?

fi

fi

30

Results: Quality Evaluation for Greek
1

precisetoo-speci c

too-general

not-a-rule

0.75

Total: 161 rules

0.5

80% rules are valid,
40% valid rules too speci c/general

0.25

Percentage of rules

Does model discover grammaticallyvalid rules?

CaseMarking

WordOrder

Agreement1
yespartially-correct

no

0.75

Are the rules human-readable?

0.5

69% rules are informative and readable

0.25
0

CaseMarking

WordOrder

fi

fi

30

Agreement

Results: Quality Evaluation
Does the model discover *new rules?

31

Results: Quality Evaluation
Does the model discover *new rules?

Even for well studied languages,
system discovers *new rules31

An aside: Parser quality matters a lot!

32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

An aside: Parser quality matters a lot!
• Applied AutoLEX on Hmong Daw (mmw) which has NO syntactic parser available

32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

An aside: Parser quality matters a lot!
• Applied AutoLEX on Hmong Daw (mmw) which has NO syntactic parser available
1— Zero-shot transfer using a multilingual UDIFY parser

32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

An aside: Parser quality matters a lot!
• Applied AutoLEX on Hmong Daw (mmw) which has NO syntactic parser available
1— Zero-shot transfer using a multilingual UDIFY parser— Discovered 30 rules for Subj-V, Obj-V, Num-N, Adj-N, Adp-N word orders
but, only 16% showed valid patterns

32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

An aside: Parser quality matters a lot!
• Applied AutoLEX on Hmong Daw (mmw) which has NO syntactic parser available
1— Zero-shot transfer using a multilingual UDIFY parser— Discovered 30 rules for Subj-V, Obj-V, Num-N, Adj-N, Adp-N word orders
but, only 16% showed valid patterns• Applied AutoLEX on Kannada (kan) which has NO syntactic parser
but has related language data with small in-person data32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

An aside: Parser quality matters a lot!
• Applied AutoLEX on Hmong Daw (mmw) which has NO syntactic parser available
1— Zero-shot transfer using a multilingual UDIFY parser— Discovered 30 rules for Subj-V, Obj-V, Num-N, Adj-N, Adp-N word orders
but, only 16% showed valid patterns• Applied AutoLEX on Kannada (kan) which has NO syntactic parser
but has related language data with small in-person data

— Few-shot transfer using a multilingual UDIFY1 parser with all available data32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

An aside: Parser quality matters a lot!
• Applied AutoLEX on Hmong Daw (mmw) which has NO syntactic parser available
1— Zero-shot transfer using a multilingual UDIFY parser— Discovered 30 rules for Subj-V, Obj-V, Num-N, Adj-N, Adp-N word orders
but, only 16% showed valid patterns• Applied AutoLEX on Kannada (kan) which has NO syntactic parser
but has related language data with small in-person data

— Few-shot transfer using a multilingual UDIFY1 parser with all available data
— For Subj-V, Obj-V word orders nearly 50% showed valid patterns

32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

An aside: Parser quality matters a lot!
• Applied AutoLEX on Hmong Daw (mmw) which has NO syntactic parser available
1— Zero-shot transfer using a multilingual UDIFY parser— Discovered 30 rules for Subj-V, Obj-V, Num-N, Adj-N, Adp-N word orders
but, only 16% showed valid patterns• Applied AutoLEX on Kannada (kan) which has NO syntactic parser
but has related language data with small in-person data

— Few-shot transfer using a multilingual UDIFY1 parser with all available data
— For Subj-V, Obj-V word orders nearly 50% showed valid patterns

With improvements in syntactic parser, quality of rules also improves!
32

1

75 Languages, 1 Model: Parsing Universal Dependencies Universally,
Kondratyuk and Straka, 2019

Revisiting our research goal

33

Revisiting our research goal
(Low-resource) Language
Analysis

33

Revisiting our research goal
(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

33

ObjectVerb

Revisiting our research goal
(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

33

ObjectVerb

Revisiting our research goal
(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation

33

Language Education

34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Computer-assisted language learning (CALL) systems are in high demand!

34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Computer-assisted language learning (CALL) systems are in high demand!

34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Computer-assisted language learning (CALL) systems are in high demand!

34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Computer-assisted language learning (CALL) systems are in high demand!

34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Computer-assisted language learning (CALL) systems are in high demand!

Limited inlanguage coverage!

34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Computer-assisted language learning (CALL) systems are in high demand!

Limited inlanguage coverage!

• Creating a curriculum is a challenging process ⟶ grammar coverage, examples, exercises …

34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Computer-assisted language learning (CALL) systems are in high demand!

Limited inlanguage coverage!

• Creating a curriculum is a challenging process ⟶ grammar coverage, examples, exercises …

• AutoLEX has shown potential in doing some aspects of this automatically34

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word Usage: Semantic Subdivisions

35

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

Word Usage: Semantic Subdivisions• Different languages carve up the semantic space differently

35

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

Word Usage: Semantic Subdivisions• Different languages carve up the semantic space differently
Outside

muro
wall
pared

Inside35

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

Word Usage: Semantic Subdivisions• Different languages carve up the semantic space differently
Outside

muro
wall
pared

Inside• Crowdsourced study where participants recruited online had to predict correct word usage in context

35

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

Learner accuracy

Results
0.9
0.675

Spanish0.45

7 learners, 9 words

0.225
0

5

10

20

30

with rules

40

50

80

without rules

Learner accuracy

0.9

Greek

0.675

9 learners, 10 words

0.45
0.225
0

5

10

20

30

40

Number of Examples
36

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

50

120

Learner accuracy

Results
0.9
0.675

Spanish0.45

7 learners, 9 words

0.225
0

5

10

20

30

with rules

40

50

80

without rules

Learner accuracy

0.9

Greek

0.675

9 learners, 10 words

0.45
0.225
0

5

10

20

30

40

Number of Examples
36

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

50

120

Learner accuracy

Results
0.9
0.675

Spanish0.45

7 learners, 9 words

0.225
0

5

10

20

30

40

50

80

When shown rules, learners learn better and faster!
with rules

without rules

Learner accuracy

0.9

Greek

0.675

9 learners, 10 words

0.45
0.225
0

5

10

20

30

40

Number of Examples
36

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

50

120

Language Education

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Apply AutoLEX to extract grammar aspects for teaching two Indian languages

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Apply AutoLEX to extract grammar aspects for teaching two Indian languages
Marathi - मराठी

Indo-Aryan language family

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Apply AutoLEX to extract grammar aspects for teaching two Indian languages
Marathi - मराठी

Indo-Aryan language family

Kannada - ಕ ಡDravidian language family“Classical language status”

ನ್ನ

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Apply AutoLEX to extract grammar aspects for teaching two Indian languages outside of India
Marathi - मराठी

Indo-Aryan language family

Kannada - ಕ ಡDravidian language family“Classical language status”

ನ್ನ

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Apply AutoLEX to extract grammar aspects for teaching two Indian languages outside of India
Marathi - मराठी

Kannada - ಕ ಡDravidian language family
Indo-Aryan language family

ನ್ನ

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

“Classical language status”

Language Education• Apply AutoLEX to extract grammar aspects for teaching two Indian languages outside of India
Marathi - मराठी

Kannada - ಕ ಡDravidian language family
Indo-Aryan language family“Classical language status”• Under-resourced settings w.r.t pedagogical resources as well as NLP models/resources

ನ್ನ

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Language Education• Apply AutoLEX to extract grammar aspects for teaching two Indian languages outside of India
Marathi - मराठी

Kannada - ಕ ಡDravidian language family
Indo-Aryan language family“Classical language status”• Under-resourced settings w.r.t pedagogical resources as well as NLP models/resources• Access to in-service teachers that teach these languages to English speakers

ನ್ನ

37

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Communities and Consultations

38

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Communities and Consultations• Kannada ⟶ Kannada Academy largest organization in the world
70 learning centers, 800 volunteer teachers

38

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Communities and Consultations• Kannada ⟶ Kannada Academy largest organization in the world
70 learning centers, 800 volunteer teachers• Marathi ⟶ Marathi Vidyalaya, Randolph, New Jersey
Marathi Shala, Pittsburgh
Small independent volunteer-run schools

38

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects

39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers

39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Info
39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Info
39

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Info
39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Info
39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Infofi

39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAf x Usage

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Infofi

39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAf x Usage

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Infofi

39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAf x Usage

AutoLEX: Selection of Grammar Aspects• In consultation with the curriculum designers• Perused existing Kannada textbooks to identify popular grammar points

General Infofi

39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAf x Usage

Word Usage

AutoLEX: Selection of Grammar Aspects

General Infofi

39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAf x Usage

Word Usage

AutoLEX: Selection of Grammar Aspects

General Infofi

39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAf x Usage

Word Usage

AutoLEX: Selection of Grammar Aspects
Roman transliteration

English translations

General Infofi

39

Agreement

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word OrderAf x Usage

Word Usage

AutoLEX: Extracting Language Descriptions

40

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions

kn-en

40

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions

kn-en

mr-en

40

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions

kn-en

kn parser

mr parser
mr-en

40

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions

?
kn-en

kn parser?
mr parser
mr-en

40

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions

?
kn-en

kn parser😕
?
mr parser

mr-en

40

kn has NO SUD treebank/model

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

mr has VERY SMALL and NOISY SUD treebank/model

AutoLEX: Extracting Language Descriptions

41

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions
There is ALWAYS some data to be found!

41

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

☺

AutoLEX: Extracting Language Descriptions
There is ALWAYS some data to be found!
Paninian Treebanks

kn

mr

41

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

☺

AutoLEX: Extracting Language Descriptions
There is ALWAYS some data to be found!
Paninian Treebanksknconvert to
SUD

mr

41

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

☺

AutoLEX: Extracting Language Descriptions
There is ALWAYS some data to be found!
Paninian Treebankskntrain model for
POS tags, lemmatization,
morphological analysis

convert to
SUD

mr

41

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

☺

AutoLEX: Extracting Language Descriptions
There is ALWAYS some data to be found!

☺

Paninian Treebanks
Hindi

kn
train model for
POS tags, lemmatization,
morphological analysis

convert to
SUD

mr

41

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

train model for
dependency parse

AutoLEX: Extracting Language Descriptions
There is ALWAYS some data to be found!

☺

Paninian Treebanks
Hindi

kn
train model for
POS tags, lemmatization,
morphological analysis

convert to
SUD

mr

41

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

train model for
dependency parse

AutoLEX: Extracting Language Descriptions

kn-en

kn parser

mr parser
mr-en

42

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions

kn-en

kn parser

mr parser
mr-en

42

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions
General Info

Agreement
kn-en

kn parser

Word Ordermr parser

Af x Usage

mr-en

Word Usagefi

42

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Extracting Language Descriptions

kn-en

General Info

Linguistic Question

Agreement

Formulate the linguistic question
into a classi cation task

kn parser

mr parser

Word Order

Extract Features andConstruct Training Data

Af x Usage

Learn an Interpretable Model

Word Usage

Extract and Visualize Rules

mr-en

fi

fi

42

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Quality Study
General Info

Agreement

Word OrderAf x Usage

Word Usagefi

43

AutoLEX: Quality Study

Af x Usagefi

43

AutoLEX: Quality Study
What are the common suf xes for Kannada nouns
and when is each used?

Af x Usage

fi

fi

43

AutoLEX: Quality Study
What are the common suf xes for Kannada nouns
and when is each used?

Af x Usage
AutoLEX

Baseline

80

Accuracy

60
40
20
0

NOUN

VERB

Automated Evaluation
fi

fi

43

AutoLEX: Quality Study
What are the common suf xes for Kannada nouns
and when is each used?

Af x Usage
AutoLEX

Baseline

80

NOUN: 9*/18 valid
VERB: 7/13 valid

Accuracy

60
40
20
0

Expert Evaluation

fi

fi

43

NOUN

VERB

Automated Evaluation

AutoLEX: Quality Study
What are the common suf xes for Kannada nouns
and when is each used?

Af x Usage

NOUN: 9*/18 valid
VERB: 7/13 valid

Expert Evaluation

fi

fi

43

What are the common suf xes for Kannada nouns
and when is each used?

Af x Usage⟶+ಇ+‘she’s doing it for herself’

‘mAD’

+

‘i’

‘koL’

(ಉ+)

ಉ

+ ಇ

+ಆ

+ ಎ

‘ut.Ta’

‘id’

‘A.L’

‘e’

sandhigender/number

reflexive

ಳ್

ದ್

ತ್ತಾ

ಳ್ಳ್

ಕೊ

ಡ್

ಮಾ

fi

ಳೆ

ದ್ದಾ

ತ್ತಿ

ಳ್ಳು

ಕೊ

ಡಿ

43
fi

ಮಾ

AutoLEX: Quality Studypresenttense

AutoLEX: Teacher Perception Study

44

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• Evaluate the relevance, utility, and presentation of the materials

44

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• Evaluate the relevance, utility, and presentation of the materials• Recruited 12 Kannada teachers and 5 Marathi teachers

44

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• Evaluate the relevance, utility, and presentation of the materials• Recruited 12 Kannada teachers and 5 Marathi teachers

Introduce AutoLEX
to teachers
44

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• Evaluate the relevance, utility, and presentation of the materials• Recruited 12 Kannada teachers and 5 Marathi teachers

Introduce AutoLEX
to teachers
44

1-2 weeks for
exploration of materials

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• Evaluate the relevance, utility, and presentation of the materials• Recruited 12 Kannada teachers and 5 Marathi teachers

Introduce AutoLEX
to teachers
44

1-2 weeks for
exploration of materials

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Fill the questionnaireAutoLEX: Teacher Perception Study

45

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• How relevant are the extracted materials to the teaching needs?

45

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• How relevant are the extracted materials to the teaching needs?
KannadaMarathi

Percentage of materials

70

52.5

35

17.5

0

ffi

45

General Info

Word UsageSu x Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word Order

Agreement

AutoLEX: Teacher Perception Study• How relevant are the extracted materials to the teaching needs?
KannadaMarathi

Percentage of materials

70

52.5

35

17.5

0

ffi

45

General Info

Word UsageSu x Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word Order

Agreement

AutoLEX: Teacher Perception Study• How relevant are the extracted materials to the teaching needs?
KannadaMarathi

Percentage of materials

70

Underlying corpora basedon formal and traditional usage
52.5

35

17.5

0

ffi

45

General Info

Word UsageSu x Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word Order

Agreement

AutoLEX: Teacher Perception Study• How relevant are the extracted materials to the teaching needs?
KannadaMarathi

Percentage of materials

70

52.5

35

17.5

0

ffi

45

General Info

Word UsageSu x Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word Order

Agreement

AutoLEX: Teacher Perception Study• How relevant are the extracted materials to the teaching needs?
KannadaMarathi

Percentage of materials

70

School’s primary focus
on beginner-levels
52.5

35

17.5

0

ffi

45

General Info

Word UsageSu x Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word Order

Agreement

AutoLEX: Teacher Perception Study• How relevant are the extracted materials to the teaching needs?
KannadaMarathi

Percentage of materials

70

School’s primary focus
on beginner-levels
52.5

35

17.5

0

ffi

45

More beneficial for advanced
learners

General Info

Word UsageSu x Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Word Order

Agreement

AutoLEX: Teacher Perception Study46

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• For what purposes would the teachers use the extracted materials for their teaching needs?

46

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

AutoLEX: Teacher Perception Study• For what purposes would the teachers use the extracted materials for their teaching needs?
Lesson Preparation

Student self-exploration

Student evaluation

Percentage of teachers

90

67.5

45

22.5

0

ffi

46

General Info

Word Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Su x Usage

Word Order

Agreement

AutoLEX: Teacher Perception Study• For what purposes would the teachers use the extracted materials for their teaching needs?
Lesson Preparation

Student self-exploration

Student evaluation

Percentage of teachers

90

67.5

45

22.5

0

ffi

46

General Info

Word Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Su x Usage

Word Order

Agreement

AutoLEX: Teacher Perception Study• For what purposes would the teachers use the extracted materials for their teaching needs?
Lesson Preparation

Student self-exploration

Student evaluation

Percentage of teachers

90

67.5

45

22.5

0

ffi

46

General Info

Word Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Su x Usage

Word Order

Agreement

AutoLEX: Teacher Perception Study• For what purposes would the teachers use the extracted materials for their teaching needs?
Lesson Preparation

Student self-exploration

Student evaluation

Percentage of teachers

90

67.5

45

22.5

0

ffi

46

General Info

Word Usage

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Chaudhary, Sampath, Sheshadri, Anastasopoulos, Neubig. In Submission.

Su x Usage

Word Order

Agreement

AutoLEX: Teacher Testimonials
The illustrative examples, and the groupedsynonyms. It does need some work on accuracy
in some places, but this is a great start!

I used this tool to teach an American adult
who takes private lessons, found this tool
helpful in addressing her grammar questions.

given that these word pairs have been extracted from natural text, its interesting to seethat there are certain word senses which are so frequently used in the real world whichcurrently we haven't covered in our lesson but are we are now thinking of adding them.

If this tool could be used to target the older kids it would be very helpful. However, the pastpresent and future tenses of the verbs are interesting and this tool managed to impress mewith the vast database, Unfortunately, the words used are very technical, and make excellent
tool to improve writing skills.
Providing teachers the ability to input a curated set of data (stories
written in good and correct language) to prepare relevant examples
from may be helpful. Working more collaboratively will help us a lot
47

Other Applications

48

Other Applications

Language
Descriptions

48

Other Applications
Automatic Multilingual Grammar Checker

Language
Descriptions

48

Other Applications
Automatic Multilingual Grammar Checker

Language
Descriptions

48

Other Applications
Automatic Multilingual Grammar Checker

Language
Descriptions

Evaluating the Morphosyntactic Well-formedness of Generated Texts
Pratapa, Anastasopoulos, RIjhwani, Chaudhary, Mortensen, Sheikh, Neubig,Tsvetkov. EMNLP 2020

48

Other Applications

Evaluating the Morphosyntactic Well-formedness of Generated Texts
Pratapa, Anastasopoulos, RIjhwani, Chaudhary, Mortensen, Sheikh, Neubig,Tsvetkov. EMNLP 2020

Language
Descriptions

48

Other Applications
Evaluating Context-Usage in MT models

Evaluating the Morphosyntactic Well-formedness of Generated Texts
Pratapa, Anastasopoulos, RIjhwani, Chaudhary, Mortensen, Sheikh, Neubig,Tsvetkov. EMNLP 2020

Language
Descriptions

48

Other Applications
Evaluating Context-Usage in MT models

Evaluating the Morphosyntactic Well-formedness of Generated Texts
Pratapa, Anastasopoulos, RIjhwani, Chaudhary, Mortensen, Sheikh, Neubig,Tsvetkov. EMNLP 2020

Language
Descriptions

When is Wall a Pared and when a Muro? Extracting Rules Governing Lexical Selection
Chaudhary,Yin, Anastasopoulos, Neubig. EMNLP 2021

48

Other Applications
Evaluating Context-Usage in MT models

Evaluating the Morphosyntactic Well-formedness of Generated Texts
Pratapa, Anastasopoulos, RIjhwani, Chaudhary, Mortensen, Sheikh, Neubig,Tsvetkov. EMNLP 2020

Language
Descriptions

Do Context-Aware Translation Models Pay the Right Attention?
Yin, Fernandes, Pruthi, Chaudhary, Martins, Neubig. ACL 2021

48

Other Applications
Evaluating Context-Usage in MT models

Evaluating the Morphosyntactic Well-formedness of Generated Texts
Pratapa, Anastasopoulos, RIjhwani, Chaudhary, Mortensen, Sheikh, Neubig,Tsvetkov. EMNLP 2020

Language
Descriptions

Do Context-Aware Translation Models Pay the Right Attention?
Yin, Fernandes, Pruthi, Chaudhary, Martins, Neubig. ACL 2021

48

Contributions

49

Contributions• AutoLEX: framework to extract and visualize language descriptions

49

Contributions• AutoLEX: framework to extract and visualize language descriptions

49

Contributions• AutoLEX: framework to extract and visualize language descriptions
http://www.autolex.co/interface/49

Contributions• AutoLEX: framework to extract and visualize language descriptions
http://www.autolex.co/interface/• Real-World Utility: established collaborations with teacher communities

49

Contributions• AutoLEX: framework to extract and visualize language descriptions
http://www.autolex.co/interface/• Real-World Utility: established collaborations with teacher communities• Under-Resourced NLP: effectively utilize existing data and collect new data49

(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation

https://www.autolex.co/50

(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation

https://www.autolex.co/
What’s Next?

50

(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation

https://www.autolex.co/
What’s Next?

• We demonstrated utility on 4 languages, about 7,000 more to go

50

(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation

https://www.autolex.co/
What’s Next?

• We demonstrated utility on 4 languages, about 7,000 more to go• Low-resource language analysis still doesn’t work well enough

50

(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation

https://www.autolex.co/
What’s Next?

• We demonstrated utility on 4 languages, about 7,000 more to go• Low-resource language analysis still doesn’t work well enough• Better rule extraction methods
50

(Low-resource) Language
Analysis

AutoLEX: Automatic
Language Explorer
Subject

ObjectApplications

Verb

Language
Education andDocumentation

https://www.autolex.co/
What’s Next?

• We demonstrated utility on 4 languages, about 7,000 more to go• Low-resource language analysis still doesn’t work well enough• Better rule extraction methods• Close link w/ data provenance (conversational text >> legal text)50