Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Authors: Aditi Chaudhary
Arun Sampath
Ashwin Sheshadri
Antonios Anastasopoulos
Graham Neubig
Venue: EMNLP 2023 Findings (2023)
Keywords: NLP for pedagogy
second language learning
low-resource languages
TL;DR: We extract descriptions from a natural text corpus to automatically create learning materials for second language learning and evaluate their utility with in-service teachers.

Abstract: One of the challenges in language teaching is how best to organize  rules regarding syntax, semantics, or phonology in a meaningful manner.
This not only requires content creators to have pedagogical skills, but also  have that language's deep understanding.
While comprehensive materials to develop such curricula are available in English and some broadly spoken languages, for many other languages, teachers need to manually create them in response to their students' needs.
This is challenging because i) it requires that such experts be accessible and have the necessary resources, and ii) describing all the intricacies of a language is time-consuming and prone to omission.
In this work, we aim to facilitate this process by automatically discovering and visualizing grammar descriptions.
We extract descriptions from a natural text corpus that answer questions about morphosyntax (learning of word order, agreement, case marking, or word formation) and semantics (learning of vocabulary).
We apply this method for teaching two Indian languages, Kannada and Marathi, which, unlike English, do not have well-developed  resources for second language learning.
To assess the perceived utility of the extracted material, we enlist the help of language educators from schools in North America to perform a manual evaluation, who find the materials have potential to be used for their  lesson preparation and learner evaluation.

Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
Aditi Chaudhary?; Arun Sampath Ashwin Sheshadri Antonios Anastasopoulos: Graham Neubig
CCrneeie Mellon University, Kannada Academy, *George Mason University adi tichaud@google. com gneubigêcs, cmu. edu antonis@gmu. edu
jeanddaaaddnnnnndddnnn... (arun. sampath, ashwin sheshadri htt.://nnensssiarllllaaaaaaaaaao com
Abstract
In the language learning context this entails de- signing materials for different learning levels, cov- cring different grammar points, finding relevant examples, and even creating evaluation exercises. For second language (L2) learning, it is not straight- forward to reuse existing curricula even in the same language, the requirements of L2 learners could be vastly different from the traditional first lan- guage (LI) setting (Munby, 1981). Given that only handful of languages, in particular English, have a plethora resources for L2 learning. but for most of the world's 4000+ written languages (Eberhard et al., 2022), it is struggle to find even a suffi- ciently large and good quality text corpus (Kreutzer et al., 2021), let alone teaching material. In this paper, we explore to what extent tion of NLP techniques and corpus linguistics as- sist language education for languages with limited teaching as well as text resources?
One the challenges in language teaching is how best to organize rules regarding syntax, semantics, or phonology in meaningful man- ner. This not only requires content creators have pedagogical skills, but also have that language's 's deep understanding. While compre- hensive materials to develop such curricula are available in English and some broadly spoken languages, for many other languages, teach- ers need manually create them in response to their students' needs. This is challenging because i) requires that such experts ac- cessible and have the necessary resources, and ii) describing all the intricacies of language is time-consuming and prone to omission. In this work, aim to facilitate this process by auto- matically discovering and visualizing grammar descriptions. We extract descriptions from natural text corpus that answer questions about morphosyntax (learning of word order, agree- ment, case murking, or word formation) and se- mantics (lcarning of vocabulary). We apply this method teaching two Indian languages, Kan- nada and Marathi, which, unlike English, do not have well-developed resources for second language learning. To assess the perceived util- ity of the extracted material, we enlist the help language educators from schools in North America perform a manual evaluation, who find the materials have potential to be used for their lesson preparation and learner evaluation.
combina-
With technology advancements, teachers have used corpus-based methods (Yoon, 2005) ana- lyze large text corpora and find patterns such as collocations and relevant examples of language use, (Davies, 2008; Cobb, 2002), to supplement their vocabulary teaching. Now, with advances in NLP, we can extract instructional material for more complex use cases. For instance, the popular tasks of Part-of-Speech (POS) tagging and dependency parsing do answer questions about some local as- pects of the language such as 'what is the function of words or their relations'. AutoLEX (Chaudhary et al., 2022) uses this local analysis for extract- ing answers to linguistic questions in both human- and
1 Introduction
Recently, computer-assisted learning systems have gained tremendous popularity, especially during the COVID-19 pandemic when in-person instruc- tion was not possible, leading to the need for user- friendly and accessible learning resources (Li and Lalani, 2020). Because these materials are curated by subject experts, this makes curriculum design challenging process, especially for languages where experts or resources are inaccessible.
format. Given question (e.g. "how are objects ordered with respect to verbs in English"), AutoLEX formalizes it into an NLP task and learns an algorithm which extracts not only the common patterns (e.g. "object is before/after the verb") but also the conditions which trigger each of them (c.g. "objects come after verbs except for interrogatives").
CCurrently works at Google Research

---

want material to teach subjects ordered wutt verbs
Cunbolun
Dedignr
Werd Order
Formalize question classification Predictt whether subjects come beforelafter verbs
Generaly the
sekject sudject wet
Feature Extraction Training Construction (POS lagging, dependency parsing, ...)
govern sarped westo ponaued
edjjctt (Sampie))
Simp
Irpul
Learn a model conducive to Intorpretation (deccision trees)
How material
the tnashing needs?
Toot Corpora
would bo ueed?
Extract and Visualize Language Descriptions
Is the presontation mator do eady Fnllww
Automatic Extraction of Learning Matarial
Teacher Perception Survey
Figure 1: NLP researchers work with curriculum designers to understand their teaching needs. We then formulate an NLP task learn a model from which we can extract and visualize learning material. Finally, we work with in-service teachers understand their perception of the extracted materials for relevance, utility and presentation.
In this paper, we take a step further by exam- ining the utility of such extracted descriptions for language education. We collaborate with in-service teachers, where we tailor automatically extracted grammar points to their teaching objectives. The process starts with identifying "teachable grammar points" which are individual syntactic or seman- tic concepts that can be taught to a learner. For example, with respect to word order, a teachable grammar point could be understanding "how sub- jects are positioned with respect to verbs". We apply AutoLEX to extract human-readable explana- tions directly from the text corpora of the language of interest. Finally, present the extracted ma- terials in-service teachers evaluate the utility in their teaching process. Figure 1 outlines this collaborative design. To our knowledge, use of such automatically extracted insights have not been explored for language teaching (see section 7 for related work). We summarize our contributions below-
We conduct a survey with 17 in in-service teach- ers to understand their perception of the ex- tracted materials 85% Kannada teachers and 40% Marathi teachers note that they will likely use this material for lesson preparation, and for providing additional material to students for self-exploration.
2 Why Marathi and Kannada?
Although these languages are spoken primarily in India, small but significant populace of speak- ers has emigrated, resulting in a demand main- tain language skills within this diaspora. We iden- tified schools in North America that teach these languages to English speakers including children and adults, with their primary objective to a) pre- serve and promote the language and culture and, b) help speakers communicate with their community. While there are existing textbooks, they cannot be used as-is, as they are based on more traditional I. teaching (Selvi and Shehadeh, 2018), where the language is taught from the ground up, from introducing the alphabet, its pronunciation and writ- ing, to each subsequent grammar point. Teachers have instead adapted the existing material and con- tinue design new material to suit the L2 speak- ers' needs. Additionally, in comparison English, both these languages have far fewer L2 learning sources. Both Marathi and Kannada are not part of any popular tools (e.g. Duolingo or Rosetta Stone (Stone, 2010)). For Marathi, there an online learning tool Barakhadi* which, however, is not
We explore how NLP methods can play role in language pedagogy, especially for lan- guages which lack resources. This entails de- signing teaching points by including real users (teachers in our case) in a collaborative design process and further evaluating its its practical utility with a large human study.
We automatically extract leaning materials for Kannada and Marathi, two Indian languages, and present them through online interface! and release the code/data publicly.
teacher-perception 'ht duwlingo...ooooooo con/ "https: com/
>https://githun cow/reviewforml..

---

free of cost. Therefore, these languages are under- resourced with respect pedagogical resources, and will likely benefit from this exercise.
to verbs in English", it is formalized it into an NLP task "predict whether the object comes before or after the verb". The second step to learn a model for this prediction task, for which training data is constructed by identifying and extracting features from the text corpora that are known to govern the said phenomenon (e.g. POS tagging and dependency parsing). Next, AutoLEX learn algorithm which extracts not only the common pat- terns (e.g. "object before/after the verb") but also the conditions which trigger each of them (e.g. "ob- jects come after verbs except for interrogatives"). Importantly, for each pattern, illustrative examples with examples of exceptions are extracted from the corpora. Finally, the extracted conditions are visu- alized with illustrative examples through an online interface. We adapt the above process to extract descriptions for all the teachable grammar points defined in Table 1. Of those, AutoLEX already out- lines the process for agreement and word order and for the others we adapt the process as shown below.
3 Proposed Work
Although language education has been widely stud- ied in literature, there is no one 'right' method of teaching. Since our focus is to create peda- gogical content to assist teachers in their process, we conduct a pilot study and collaborative design with two Kannada teachers who are deeply in- volved in the curriculum designing. We first iden- tify some "teachable grammar points" which, as defined above, are points that can be taught to a learner and are typically included in learning cur- riculum. Next, following AutoLEX, we formulate each grammar point into an NLP task and extract human-readable descriptions, as shown below.
3.1 Identify Teachable Grammar Points
As a first step towards the curriculum design sce- nario, we performed an inventory of the aspects taught in existing teaching materials. We manu- ally inspected three of the eight Kannada textbooks shared by the curriculum designers, which are orga- nized by increasing learning complexity, and iden- tified grammar aspects such as identification of word categories (e.g. nouns, verbs, etc.), vocabu- lary, and suffixes. In Table 1, we show examples of the teachable grammar points that we attempt to extract, which we group under five grammar aspects namely General Information, Vocabulary, Word Order, Suffix Usage and Agreement. We only cover the above subset out of all grammar aspects described in the textbooks because they satisfied two desiderata: a) the Kannada teachers identified these to be widely studied and important in their curriculum and b) the underlying linguistic ques- tions can be formulated into an NLP task thereby allowing us to extract descriptions.
Word Order and Agreement Both Marathi and Kannada are morphologically rich, with highly in- flected words for gender, person, number: morpho- logical agreement between words is also frequently observed. Both languages predominantly follow SOV word order, but because syntactic roles are of- ten expressed through morphology, there are often deviations from this order. Therefore, learners must understand both the rules of word order and agree- ment to produce grammatically correct language. In AutoLEX, word order and agreement grammar points are extracted by formulating questions, as shown in Table 1, and learning a model to answer each question. As outlined above, to train these models, we must first identify the relevant elements (e.g. for word order, subjects, and verbs) to con- struct the training data. To do so, the corpus of that language must be syntactically annotated with POS tags, morphological analyses, lemmas, and depen- dency parses. Next, to discover when the subject is before or after, AutoLEX extracts syntactic and lexical signals from other words in that sentence and uses them to train classifier. To obtain inter- pretable patterns, we use decision trees (Quinlan, 1986), similar to AutoLEX, which extract "if X then Y" style patterns that can, if presented appropri- ately, be interpreted by teachers learners. Exam- ple word order patterns extracted from this model for Marathi are shown in Figure 1. While AutoLEX uses English as the meta- veta-language to present ex-
3.2 Extract Learning Material
As noted above, we build on AutoLEX (Chaudhary et al., 2022) to extract learning material. AutoLEX takes as input raw text corpus, comprising of sen- tences in the language of interest, and produces human- and machine-readable explanations of dif- ferent linguistic behaviors, following four step process. The first step formulating a linguistic question into an NLP task. For example, given a question "how are objects ordered with respect

---

Aspects
Teachable Grammar Points
General Information
What gender values does Marathi show? (e.g. masculine, feminine, neuter)
Which type of words show these values? (e.g. nouns, verbs)
What are some example word usages?
Vocabulary
What words to use for popular categories (c.g. food, animals, etc.)
What are some adjectives, their synonyms and antonyms? Which word to use when?
Word Order
Are subjects before or after verbs in Marathi?
If both, when is subject before and when is it after?
Suffix Usage
What are the common suffixes for Marathi nouns?
When should a particular suffix (e.g. - ''laa') be used?
Agreement
Do some words need to agree on gender?
If so, when should they necessarily agree and when they need not?
Table 1: Example aspects of grammar, vocabulary and teachable grammar points covered in our material.
tracted descriptions, we use combination of L1 (English) and L2. This is in alignment with meth- ods such as Transsaatinnnnioo (Doggett, 1986) that encourage learning using both L1 and L2.
tifying such word pairs, along with explanations their usage, using parallel sentences between English and the L2. Each pair of sentence transla- tions is first run through automatic word aligner (Dou and Neubig, 2021), which extracts word-by- word translations, producing a list of English words with their corresponding L2 translations. On top of this initial list, filtrtion steps are applied to ex- tract those word pairs that show fine-grained diver- gences. Training data is then constructed to solve the task of lexical selection, i.e. for a given L1 word (e.g. 'rice") in which contexts to one L.2 word over another (e.g. 'bhaat' vs 'tandul'). Be- cause most of our learners have English as their L1, we extract signals from both the LI and L2 corpora to train the classifier and thereby derive style patterns which contain both L1 and L2. Com- municative Approach (Johnson and Brumfit, 1979) focuses on teaching through functions (e.g. self- introduction, identification of relationships, etc.) over grammar forms; therefore, we also organize vocabulary around popular categories. We run a word-sense disambiguation (WSD) model (Pasini et al., 2021) on English sentences, which helps us to identify the word sense for each word in context (e.g. 'bank.n.02' refers to a financial institution while 'bank.n.01' refers to a river edge). Given the hierarchy of word senses expressed in Word- Net (Miller, 1995), of each sense to find whether it belongs to any of the pre-defined categories (e.g. food, relationships, animals, fruits, colors, time, verbs, body parts, vehicle, elements, furniture, clothing). Example Marathi words extracted are shown in Figure 4. We also identify popular adjectives, their synonyms,
Suffix Usage Along with understanding sentence structure, it is equally important to understand how inflection works at the word level. The first step is identify the common suffixes for each word type (e.g. nouns) and then ask "which suf- fix to use when". For that, we need identify the POS tags and produce morphological analy- sis for each word, like we did above. To identify the suffix, we train a model that takes as input a word with its morphological analysis (e.g. 'de- shaala,N, Aac,,,,,,,,inngg and outputs the decom- position (e.g. 'desh + laa'). Next, a classification model is trained for each such suffix (e.g. '-laa') extract the conditions under which one suffix is typically used over another (Figure 2).
Vocabulary Vocabulary is probably one of the most important components of language learning (Folse, 2004). There are several debates on the best strategy for teaching vocabulary; we follow prior literature (Groot, 2000; Nation, 2005; Richards et al., 1999), which use a mixture of definitions with examples of word usage in context. Specifi- cally, we organize the material around three ques- tions, as shown in Table 1. There are some cate- gories of words where the same LI (English) word can have multiple L2 translations, with fine-grained semantic subdivisions (e.g. "bhaat" and 'tandul" both refer to 'rice' in Marathi, but the latter refers to raw rice and the former refers to cooked rice). Chaudhary et al. (2021) propose a method for iden-
traverse the ancestors

---

and antonyms, also extracted from WordNet, and present them in a similar format (Figure 5).² For each word, we also present accompanying exam- ples that illustrate its usage in context, along with its English translations. For the benefit of users who are not familiar with the script of L2 languages, we automatically transliterate into Roman script us- ing Bhat et al. (2015).
4 Automatic Evaluation
Since our objective is to evaluate whether such au- tomatically derived linguistic insights can be useful for language pedagogy, we first conduct a pilot study to evaluate quality and properties of the ex- tracted materials (section 5). Next, we conduct a study with several in-service teachers, both in Kan- nada and Marathi, to evaluate relevance, usability, and presentation of the extracted materials (sec- tion 6). In addition to human evaluation, we follow Chaudhary et al. (2022) to automatically evaluate the quality of extracted descriptions. This provides a quick sanity check on whether our trained models are able to learn the said linguistic phenomena.
General Information In addition to answering these morpho-syntax and semantic questions, we also present salient morphology properties at the language level. Specifically, from the syntactically parsed corpus of the target language, hope to answer basic questions such "whaa morphologi- cal properties (c.g. gender. person, number, tense, case) does this language have", as shown in Ta- ble 1. These questions were inspired from the Kan- nada texthooks shared by experts, which introduces the learner to basic syntax and morphology. Un- derstanding syntax patterns are crucial, especially for Kannada and Marathi, which show significant variations in inflection. Through the previous vo- cabulary section, learners can learn the L2 words for action verbs, and through this section, they can learn how to use those verbs for different genders, tenses, etc. For each question, we organize the information by frequency, a common practice in language teaching where textbooks often comprise of frequently used examples (Dash, 2008).
Word order and Agreement Chaudhary et al. (2022) automatically evaluate the learnt model by measuring the accuracy on held-out sentences. For example, for subject-verb model. the gold label is the observed word order which can be determined from the POS and dependency parses (i.e. whether the subject is 'before' or 'after' the verb), which is then compared with the model prediction to com- pute the accuracy. This model is compared with baseline that assigns the most frequent observed pattern in the training data as model prediction.
Suffix Usage We use similar most-frequent baseline where the most frequent suffix pattern is used as model prediction for the baseline accuracy, where the observed suffix is the gold label.
Vocabulary We follow the accuracy computa- tion from Chaudhary et al. (2021) to evaluate the model used for extracting semantic subdivisions for each sentence the model prediction is compared with the gold label which is the observed L2 word for the L1 word. The baseline uses the most fre- quently observed L2 word translation for the given L1 word and compared with the gold label to compute the baseline accuracy.
Along with relevant content, the format in which the material is presented equally important. Smith Jr (1981) outline four steps involved in lan- guage teaching: presentation, explanation, repeti- tion of material until it is learned, and transfer of materials in different contexts, which have no fixed order. For example, some teachers prefer the pre- sentation of content (e.g. reading material, exam- ples, etc.) first followed by explanation (e.g. gram- mar rules), while Smith Jr (1981) discuss that, for above-average learners, explanation followed by presentation may be preferable. In our design, we extract and present both (i.e. rules and examples) without any specific ordering, allowing educators to decide based their requirements. By provid- ing illustrative examples from the underlying text each step, we hope to address the transfer step. where learners are exposed to real language.
4.1 Setup
Data Since our goal is to create teaching material for learners having English as LI, we use the paral- lel corpus of Kannada-English and Marathi-English from SAMANANTAR (Ramesh et al., 2022) com- prising of 4 million sentences, as our starting point. This covers text from variety of domains such news, Wikipedia, talks, religious text, movies.
Model As mentioned in section 3, the first step in the extraction of materials is to parse sentences for POS tags, morphological analysis and dependency parsing. To obtain this analysis for our corpus,
'Fistt we automatically identify frequent cross-lingual word pairs from our corpus. Next, we identify the adjec- tives using POS tagging and use WordNet to extract the antonyms/snnnyys of their English counterparts.

---

we use UDIFY (Kondratyuk and Straka, 2019) that jointly predicts POS tags, lemma, morphology and dependency tree over raw sentences. However, UD- IFY requires training data in the UD annotation scheme (McDonald et al., 2013). Kannada has no UD treebank available and for Marathi the tree- bank is extremely low-resourced covering only 300 sentences. Therefore, we train outlined in subsection A. To learn models for tracting descriptions, we follow the same modeling setup as Chaudhary et al. (2021) and Chaudhary et al. (2022) and use decision trees (Quinlan, 1986) to extract the patterns, explanations and accompa- nying examples (subsection A.2). For suffix usage, we additionally train morphology decomposition model (Ruzsics et al., 2021) which breaks a word into its lemma and suffixes, over which we learn classification model.
as-is, 'vaidyaro" is the plural form and 'vaidayaru' formal way of saying a doctor.
Word Order For word order, experts evaluate the rules for subject-verb and object-verb order. For subject-verb, seven grammar rules explaining the different word order patterns were extracted explaining when the subject can oceur both before and after the verb, rules informing when sub- jects occur after, and showing the default order of "before"). Of the seven rules, experts found four to be valid patterns. For object-verb word order, of the six rules extracted by the model, experts marked that 2 rules precisely captured the patterns, while one rule was too fine-grained Interestingly, in addition correctly identifying the dominant or- der, all the rules which were deemed valid showed non-dominant patterns. The rules marked as in- valid were invalid because the syntactic parser that generated the underlying syntactic analyses incor- rectly identified the subjects/objects. Such errors are expected given that there is not sufficient quan- tity/quality of expertly annotated Kannada syntac- tic analyses available train high-quality parser. However, we would argue that these results are still encouraging because i) despite imperfect syntactic analysis, the proposed method was able to extract several interesting counter-examples to the domi- nant word order, and ii) further improvements in the underlying parsers for low-resource languages may be expected through active research.
parser as
4.2 Results
In Table 5 we report results for word order, suffix usage and agreement. We can see that in most cases, the rules extracted by the model outperform the respective baselines, suggesting that the model is able extract decent first-pass rules, with 98% avg. accuracy for Kannada word order, 48% for agree- ment, 85% for suffix usage, 68% for vocabulary, 98% for Marathi word order, 61% % for agreement, 85% for suffix usage and 70% for vocabulary.
5 Human Quality Evaluation
Suffix Usage We extract different suffixes used for each word type (e.g. nouns, adjectives, etc.) but in the interest of time asked experts to evaluate only the suffixes extracted for nouns and verbs. Of the 18 noun suffixes, 7 were marked as valid, 2 suffixes were suffixes in traditional terms but arise due "sandhi" i.c. transformation in the characters at morpheme boundaries. Similarly, for verb suffixes, 53% (7/13) were marked as valid. Experts mentioned that understanding suffix usage is particularly important in Kannada, as it is an agglutinative language with different affixes for different grammar categories.
We conduct a limited study for a sanity check with two Kannada teachers.
Vocabulary We present both experts with an au- tomatically generated list of 100 Iagllsh-Kannade word pairs, where one English word has multiple translations showing fine-grained semantic sub- divisions. Both experts found 80% of the word pairs to be valid, according to the criterion that they show different usages. For example, for 'doc- tor', the model discovered four unique translations, namely 'vaidya, vaidyaro, daktor, vaidyaru' in Kannada which the expert found interesting for teaching as they demonstrated fine- graiind diver- gences, both semantically and syntactically. For instance, 'vadiya" is the direct translation of 'doc- tor', whereas 'daktor' is the English word used
Teacher Perception Study
For Kannada, we work with teachers from the Kan- nada Academy² (KA), which is one of the largest organizations of free Kannada teaching schools in the world and recruit 12 volunteer teachers. For
"Because there tends to be strong agreement in these lan- guages, there is class imbalance which probably led to the low performance of the agrcement classifier.
Phttps://wwwr httt::://www.kkmaaacaddeeee kannadaacademy.coo com/

---

Marathi, there is no central organization as for Kannada, but there are many independent schools in North America. We reached out to Marathi Vidyalay in New Jersey, that teaches learners in the age group of 6-15, and Shala in Pittsburghh. Marathi Vidyalay is a small school consisting of seven volunteer teachers, of whom four agreed to participate, while Shala has one teacher. All par- ticipants are volunteer teachers; teaching is not their primary profession. Since we extract learn- ing materials automatically from publicly available corpora which may contain material which is age- inappropriate, we purposefully chose to share these materials with the teachers who we feel are best suited to decide how to use them.
existing curriculum. This is notable given that the underlying corpus is not specifically curated for language teaching and contains rather formal lan- guage. All teachers noted that especially for be- ginners they prefer starting with simpler and more conversational language style, but 5 teachers ex- plicitly mentioned that for advanced learners this would be very helpful. In fact, one of the teachers having 3+ years of experience teaching intermedi- ate advanced learners explicitly mentioned that-
"The examples are well written, how- ever, for beginners and intermediates, this might be too detailed. The corpus could be from wider data source. The use of legal terms is less commonly used in day-to-day life. Advanced learners will certainly benefit from this."
Perception Survey To answer the rescarch ques- tion of whether materials are practically usable and, if so, with regard to what aspects, we analyze the Kannada and Marathi teachers' perception regard- ing relevance, utility and presentation of the mate- rials. First, a 30-60 minute meeting is conducted for the teachers, in which we introduce the tool, the different grammar points covered in it, and how to navigate the online interface. Teachers have one week to explore the materials. Finally, all teachers receive a questionnaire Table that requires them assess the relevance, utility and presentation."
Utility We find that for all grammar concepts, most teachers (nearly 80%) expressed that they were likely to use the materials for lesson prepara- tion. The per-grammar category results are in Table 2. In fact, one teacher who used our materials to teach suffix usage to an adult learner said-
"I used this tool to teach an Ameri- can adult who takes private lessons and found it helpful in addressing her gram- mar questions. I liked how it was clearly segregated i.e. the suffixes for nouns vs proper-nouns and how it is different from one another. It is definitely great tool to refer for adults but again the vocabulary is perfect to improve written skills than the spoken language".
6.1 Kannada Results and Discussion
We report individual results in Table 2. 12 teach- ers with varying levels of teaching experience participated in this study."! I1 All teachers have used some online tools, but mostly for creating assignments for the learners (e.g. Google Class- room, 2 ), or conducting classes (e.g. Zoom). However, none had used immersive online tools similar to our tool.
Some teachers also mentioned that they could present the material to students for self-exploration. and about 70% teachers noted that it would be espe- cially helpful for vocabulary learning. When asked what aspects of the presented material they would consider using, all teachers said that they would use the illustrative examples for all sections except for the word order and agreement sections. For agreement and word order sections, although they liked the general concepts presented in the mate- rial (for example, the non-dominant patterns shown under each section), 88% of the teachers felt that the material covered advanced topics outside the current scope. Although the quality evaluation of the rules was not part of this study, teachers noted that if the accuracy of the rules, particularly for
Relevance We see that teachers, on average, find 50-60% of material presented as relevant to their
"nttps://marathhviish http:://mrrttiiiiiis.oooor/raaaa
"ht //ww.. mmpgh. www.oorggggiihaaaaaaaa shtml 10 "Although we conducted a manual evaluation of a subset of extracted materials in section 5, we did not remove those items that were marked as incorrect by the experts as we wanted to understand how the materials, as directly ohtained automat- ically without significant human intervention, are perceived when presented as-is. This is close to the real world setting where human evaluation of each grammar point is not feasible.
FFor teachers had less than three years of experience, four teachers between 3-10 years, and the remaining had 10+ years of experience. Four teachers teach only beginners, while others have experience teaching higher levels well. 12 tht http///tahootommooooo 13 Phttps: http:///uiizzttoooooo com/

---

Grammar Concept
Relevance % of relevant curriculum covered
Utility
Presentation
% of tenchers likely to use
% of tenchers that would use for
% of teschers
found this to navigute
mr
mr
General Information
62.1%
15%
highly likely; 8.3%
Jesson 1.8%
100%
very easy: 33.3%
likely; 83.3%
40%
student exploration: 54.5%
50%
somewhat casy: 58.3%
80% 20%
not likely; 8.3%
60%
student evaluation: 10%
difficull: 8.3
Vocabulary
67.5%
16%
highly likely; 33.3%
Jesson piep: 72.7%
100%
very easy: 36.3%
likely: 58.3%
40%
student exploration: 72.7%
50%
somewhat easy:
100%
likely: 8.3%
60%
student evaluation: 45,5%
50%
difficult: 8.3%
Suflix Usage
52.5%
9%
highly likely; 9.1%
Jesson prep: 77.8%
100%
very easy: 36.4%
likely: 72.7%
40%
student exploration: 55.6%
50%
somewhut easy: 63.6%
100%
not likely: 18.2%
60%
student evaluation: 33.3%
difficult: (%%
Word Order
66%
8%
highly likely; 10%
lesson prep: 88.9%
100%
very easy: 27.3%
likely:
40%
student exploration:
50%
somewhat easy: 72.7%
likely: 20
60%
student evaluation: 22.2%
difficult: 0%
20%
Agreement
53.75%
5%
highly likely; 203
lesson prep: 77.8%
100%
very easy: 36.4%
likely; 60%
40%
student exploration: 44.4%
50%
somewhal easy: 45.5%
80%
not likely: 20%
60%
student evaluation; 22.4%
difficult: 18.2%
20%
Table 2: Perception study results from 12 Kannada and 5 Marathi teachers
suffix usage, could be further improved, they could foresee this tool being used in classroom teaching, suffixes are essential in Kannada.
is much less than what the Kannada teachers re- ported, probably because the Marathi schools' pri- mary focus is teaching beginners. For beginners, teachers begin by introducing the alphabet, simple vocabulary, and sentences. In our tool, currently we do not curate the material according to learner age/experience, and we have extracted the learning materials from a public corpus which comprises of news articles that are not beginner-oriented.
Presentation In terms of presentation of the ma- terials, all teachers found them easy to navigate through, although took some getting used to. This is expected given that the teachers spent only a few hours (5-6) over the course of week exploring all materials. 8 teachers noted that for a new user the materials could be overwhelming navigate but for instance, the two Kannada experts, who also participated in the quality study, have had weeks of exposure to the tool and therefore rated very easy to navigate. We also find that these results vary for different grammar categories covered, for example, for the Vocabulary section, generally the materi- als presented were 'somewhat-easy" to "easy' to navigate, while for the Agreement section, 18.2% teachers found the materials difficult navigate. One of the reasons could be the meta-language used describe the materials, for instance for Agree- ment the rules consisted of formal linguistic jargon (for example, most teachers were unfamiliar with the term 'lemma' or the different POS tags))4.
Unlike in the Kannada findings, where 85% teachers said that were 'likely to use" the ma- terials, for Marathi, 60% teachers said 'not likely to use'. The main reason being that the Marathi teachers mainly teach beginner levels and found the materials more suited for advanced learners. The teachers who marked that were 'likely to use' noted that they would use them for lesson preparation. 50% of those also said that they could provide the materials to advanced students for self-exploration, to encourage them to explore the materials and ask questions. Similarly the Kannada study, two teachers found the illustrative examples be of the most utility, as they demonstrate a variety of usage. However, they did note that because the underlying corpus was too restricted in genre, they would benefit more from applying this tool to their curated set of stories, which are age-appropriate.
Utility
6.2 Marathi Results and Discussion
For Marathi, five teachers participated in the study, all of whom teach at the beginner level with a few intermediate learners.
Presentation 88% of the teachers found the ma- terials 'somewhat casy' to navigate and similar to the Kannada teachers, mentioned that it did require some time understand the format. The teachers also said that currently the material is too content heavy and not visually engaging, if the presentation could be improved along those aspects, would
Relevance Teachers find only 10-15% of the ma- terials as relevant to their existing curriculum. This
'ooor reference, we had created a documentation for the teachers in the study, which provides definition of these formal terms along with examples, but it is hard to determine whether the teachers consulted them frequently while evaluating.

---

make the tool more inviting.
reveal promising results. Revita (Katinskaia et al., 2017) automatically create exereises for several endangered languages within the Russian Federa- tion. Specifically, they use morphological analyz- ers to construct cloze-test questions which requires readers to provide the correct surface form of the missing word in text
7 Related Work
Below, we discuss some relevant literature for lan- guage learning.
Automatic Assessment Automatically assessing a learner's progress is perhaps the most popular NLP application explored in the past. For instance, Zou et al. (2022) automatically generate true/false question assess an English learner's reading comprehension. Wambsganss et al. (2022) pro- vide feedback on erroneous argument structures to help improve an English learner's ' essay writing skills. While most work has been for English, some works have developed assessment tools for other languages, for example, Weiss and Meurers (2022) assess sentence readability for German 1.2 learn ers, Imperial et al. (2022) build the first readability model for Cebuano which assesses the readabil- ity level of children's books. Similar us, they also use interpretable models (c.g. SVM, Random- Forests) trained using linguistic features extracted from a text corpora. However, their focus is on clas- sifying the content into three learner levels, while our focus is towards extracting teaching content from the corpus.
LLMs for Learning With the advent of Large Language Models (LLMs), many research and com- mercial applications are exploring their use for lan- guage learning, especially for retrieving examples of word collocations or creative writing samples. For example, DuoLingo Max 19 use GPT-4 (Ope- nAI, 2023) for conversation practice across dif- ferent scenarios (e.g. going on vacation versus ordering in restaurant), which is a useful feature for learning real-world language use. However, this feature is only available for learning high-resource languages of French and Spanish for English speak- ers. As more languages are added to LLMs, such automatic fcatures can be leveraged across lan- guages and speakers. Additionally, our main focus is to extract interpretable patterns for understand- ing complex grammatical aspects, which currently is not straightforward to extract from LLMs.
Educational Tools Over the years there has been surge in language learning tools such Rosetta Stone (Stone, 2010), Duolingo LingQ'6, Lear- nALanguage", Omniglot!*. Most of these tools have learning content manually curated with the help of subject matter experts, which, however, makes it difficult to extend them to numerous lan- guages. Recently, NLP tools have been used to develop resources for low-resource languages, for instance, Ahumada et al. (2022) use combina- tion of linguistic resources (e.g. grammars), NLP tools (e.g.- morphological analyzers) and commu- nity resources (e.g. dictionaries) to build learn- ing tools for the indigenous language Mapuzugun. Specifically, they design an orthography recognizer which identifies which of the three alphabets the input text is in, converts across orthographies if required, performs word segmentation and analy- sis and maps to them user-readable phrases/words. These are presented to Mapuzugun students which
8 Next Steps
In this work, we have explored one combination of NLP techniques and corpus linguisties to assist language education. The perception study shows that teachers do find the selected grammar points relevant and interesting, which highlights the im- portance of collaborative design; however, all note that the content is more suitable for advanced learners. Although in the current state, the ma- terials cannot be used as-is but the teachers find this overall effort very promising, as this tool can be applied to a corpus of their choice, which is more suited for the learning requirements. Among the different features, teachers find the illustrative examples to be most useful, especially for under- standing the non-dominant linguistic behaviors or exceptions to general rules. Additionally, the tool has the capability to extract numerous example us- ages which the teachers noted as a big plus, as it can provide a starting point for them to build upon rather than them having to find examples manually.
"htttps//www www.ingo..ccooo com/
"hhttps://ww/ www.....no..................... www.c//ccccce/ttte hamm/en/grammart g
"https://ww/ //www. www.Leaanaaanguage com/
18 Whttps: htwww//wwwwccco
http::///wwwwwwwww/eee/ www.oonnllll... llaii.toooo///. con/
19

---

9 Limitations
Bhide of the Bruhan Maharashtra Mandal (BMM) Shala for providing initial feedback on the inter- face. We also thank Pruthwik Mishra and Dipti Misra from IITT-Hyderabad for sharing the Marathi and Kannada treebanks for training the parser. This work is sponsored by the Waibel Presidential Fel- lowship and the National Science Foundation under grants 1761548, 2125466 and 2109578. This re- search was performed under protocol number 2018- 00000208 approved by the Carnegie Mellon Uni- versity Institutional Review Board. Antonios Anas- tasopoulos is supported by NSF grant IIS 233-232.....
Currently, major limitation of the tool, noted by the teachers, is that the content is not organized by learner age/experience. next step would be to invite teachers to organize the content by each level, taking the learner incrementally through the com- plexities of language. For beginner learners, lan- guage properties are built through engaging stories with little use of formal grammar terms. Therefore, using simpler meta- mneta-language to explain the gram- mar points and including engaging content would be a worthwhile addition. Even for the teachers, the materials took some time getting used to, especially the formal linguistic terms, therefore, addition to simplifying the language, educating the teachers in the tool format will also be necessary for effective learning. We hope that our work drives more such practical research in language education where we consult the real users (teachers in our work) to bet- ter understand what they need and work with them in collaboration.
References
Cristian Ahumada, Claudio Gutierrez, and Antonios Anastasopoulos. 2022. Educational tools for ma- puzugun. In Proceedings of the 17th Workshop In- novative Use of NLP for Building Educational Appli- cations (BEA 2022), pages 183-196, Seattle, Wash- ington. Association for Computational Linguisties.
Irshad Ahmad Bhat, Vandan Mujadia, Aniruddha Tam- mewar, Riyaz Ahmad Bhat, and Manish Shrivastava. 2015. liit-h system submission for fire2014 shared task on transliterated search. In Proceedings of the Forum for Information Retrieval Evaluation, FIRE '14, pages 48- 18-53, New York, NY, USA. ACM.
10 Ethical Considerations
We acknowledge that there several ethical con- siderations keep in mind while creating content or tools that will be directly used by human learners. Since currently we use public corpora to extract the learning materials, they may contain unwanted bias or age-inappropriate language or even culturally- insensitive materials. That is why we work in close collaboration with the respective educators in this work.
Riyaz Ahmad Bhat, Rajesh Bhatt, Annahita Farudi. Prescott Klassen, Bhuvana Narasimhan, Martha Palmer, Owen Rambow, Dipti Misra Sharma, Ash- wini Vaidya, Sri Ramagurumurthy Vishnu, et al. 2017, The hindi/urdu treebank project. In Handbook of lin- guistic annotation, pages 659-697. Springer.
Aditi Chaudhary, Zaid Sheikh, David R Mortensen, An- tonios Anastasopoulos, and Graham Neubig. 2022. Autolex: An automatic framework for linguistic ex- ploration. arXiv preprint arXiv:2203. 0.02003.1390. 13901.
Acknowledgements
We are grateful to Charles Perfetti and Lin Chen from the University of Pittsburgh, for the illumi- nating discussions on L. acquisition. We thank all teachers, without whom this work would not have been possible or meaningful. Specifically, we are grateful to Kannada Academy teachers- Arun Sampath, Ashwin Sheshadri, Manasa Kashi, Mukta Hendi, Sunita Sundaresh, Aravind Ganga- iah, Shashi Basavaraju, Madhu Rangappagowda, Gayathri Hebbar, Shruthi A, Naina Sharma, Gowri Gudi and P Tantry. We are also grateful to the Marathi Vidyalay. New Jersey teachers- Sudhir Ambekar, Aparna Potdar, Sujata Kulkarni, Varsha Joshi and Archana Kakirde and the Marathi Shala teacher from Pittsburgh- Pranati Talnikar. We also thank Sunanda Tumne, Komal Chaukkar, and Sona
Aditi Chaudhary, Kayo Yin, Antonios Anastasopoulos, and Graham Neubig. 2021. When is wall a pared and when a muro?: Extracting rules governing lexical selection. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing, pages 6911-6929, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Cobb. 2002. Web compleat lexical tutor/vocabulary profile. Retrieved on September, 4:2011.
Niladri Sekhar Dash. 2008. Corpus linguistics: An introduction. Pearson Education India.
Mark Davies. 2008 The corpus of contemporary ameri- can english (coca): 560 million words, 1990-present.
Gina Doggett. 1986. Eight approaches to language teaching. ERIC.

---

Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine nhe-tunnng embeddings on parallel corpora. arXiv preprint arXiv: (oXxx:2011.00 101.08231.
Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal Dependency an- notation for multilingual parsing. In Proceedings of the S1st Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 92-97, Sofia, Bulgaria. Association for Com- putational Linguistics.
David M. Eberhard, Gary F. Simons, and Charles D. Fennig (editors). 2022. Ethnologue: Languages of the world.
Pedregosa Fabian. 2011. Scikit-leamn Machine learning in python. Journal of machine learning research 12, page 2825.
George A Miller. 1995. Wordnet: lexical database for english. Communications of the ACM, 38(1 (8(11):3 11:34441.
Keith S Folse. 2004. Myths about teaching and learning second language vocabulary: What recent research says. TESL. reporter, 37: 13.13
John Munby. 1981. Communicative syllabus design: A sociolinguistic model for designing the content of purpose-specifi language programmes. Cambridge university press.
Peter JM Groot. 2000. Computer assisted second lan- guage vocabulary acquisition. Language learning lechnology, 4(1):56-77.
ISP Nation. 2005. Teaching and learning vocabulary. Handbook of research in second language teaching and learning, pages 605-620. Routledge.
Joseph Marvin Imperial, Lloyd Lois Antonie Reyes, Michael Antonio Ibanez, Ranz Sapinit, and Mo- hammed Hussien. 2022. A baseline readability model for Cebuano. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 27-32. Seattle, Washington. Association for Computational Linguistics.
OpenAL. 2023. GPT-4 technical report. CoRR. abs/2303.08774.
Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. XL-WSD: An extra-large and cross- lingual evaluation framework for word sense disam- biguation. In Proc. of AAAI.
Keith Johnson and Christopher Brumfit. 1979. The com- municative approach to language teaching. Oxford University Press.
J. Ross Quinlan. 1986. Induction of decision trees. Ma- chine learning. 1(1):81-066.
Anisia Katinskaia, Javad Nouri, and Roman Yangarber. 2017. Revita: system for language learning and sup- porting endangered languages. In Proceedings of the joins workshop on NI.P for Computer Assisted Lan- guage Learning and NLP for Language Acquisition, pages 27 -5, Gothenburg, Sweden. LiU Electronic Press.
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj. Mayank Jobanputra, Raghavan AK. Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Di- vyanshu Kakwani, Navneet Kumar, al. 2022. Samanantar: The largest publicly available parallel corpora collection for 11 indic languages. Transac- rions of the Association for Computational Linguis- rics, 10: 14:14
Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing universal dependencies universally. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing GEMNLP-IJCNLAS pages 2779- 2795, Hong Kong, China. Association for Computa- tional Linguisties.
Jack C Richards, David Singleton, and Michael Long. 1999. Exploring the second language mental lexicon. Cambridge University Press.
Tatyana Ruzsics, Olga Sozinova, Ximena Gutierrez- Vasques, and Tanja Samardzic. 2021. Interpretabilityy for morphological inflection: from character- predictions to subword- rules. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3189-3201, Online. Association for Computational Linguistics.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch. Nasanbayar Ulzi--Orhikk, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, al. 2021. Quality glance: An audit of web-crawled multilingual datasets. arXiv preprint arXiv:2103. 12028.
Ali Fuad Selvi and Ali Shehadeh. 2018. Approaches and Methods in English for Speakers of Other Lan- guages & Non-native English-speaking Teachers (NNESTs). Wiley-Blackwell.
Cathy Li and Farah Lalani. 2020. The covid-19 pan- demic has changed education forever. In World eco- nomic forum, volume 29, The rise of online learn- ing during the COVID-19 pandemicl World Eco- nomic...
Philip D Smith Jr. 1981. Second Language Teaching: Communicative Strategy. The Foreign & Second Language Educator Series. ERIC.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach- Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slay Petroy, Hao Zhang. Oscar
Rosetta Stone. 2010. Rosetta stone.

---

Juhi Tandon, Himani Chaudhry, Riyaz Ahmad Bhat, and Dipti Sharma. 2016. Conversion from paninian karakas to Universal Dependencies for Hindi depen- dency treebank. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016). pages 141-150, Berlin, Ger- many. Association for Computational Linguistics.
Thiemo Wambsganss, Andrew Caines, and Paula But- tery. 2022. ALEN app: Argumentative writing sup- port to foster English language learning. In Proceed- ings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 134-140, Seattle, Washington, Association for Computational Linguistics.
Zarah Weiss and Detmar Meurers. 2022. Assessing sentence readability for German language learners with broad linguistic modeling readability formu- las: When do linguistic insights make difference? In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 141-153, Seattle, Washington. Association for Computational Linguistics.
Hyunsook Yoon. 2005. An investigation of students' experiences with corpus technology in second lan- guage academic writing. Ph.D. thesis, The Ohio State University.
Bowei Zou, Pengfei Li, Liangming Pan, and Ai Ti Aw. 2022. Automatic true/false question generation for educational purpose. Proceedings of the 17th Workshop on Innovative Use of NL. for Building Educational Applications (BEA 2022), pages 61-70, Seattle, Washington, Association for Computational Linguistics.

---

A Appendix
with the original treebanks and report all results in Table 5 The running time of the model is approximately 2-5 mins.
A.1 Learning a Parser for Marathi and Kannada
After learning a decision tree, we extract rules from each leaf. However, given that there could be spurious correlations that led leaf, simply using the majority label of Ieaf as the grammar rule would be incorrect. Therefore, we apply a sta- tistical threshold, as outlined in (Chaudhary et al., 2022) to re-label each leaf. We design two hypoth- esis, a null hypothesis Ho and a hypothesis to be tested H1, upon which we apply the the thi-squared goodness of fit test where we compute the expected probability distribution for Ho considering uni- form distribution. Below, we define the Ho and Hi for the grammar aspects:
As mentioned in the main text, we train our own parser for both Marathi and Kannada to get the POS tags, dependency parses and lemmas. To train a parser for Marathi and Kannada, we use the train- ing data collected by ILIT-Hyderabade, which is annotated in the Paninian Grammar Framework (Bhat et al., 2017). However, UDIFY requires train- ing data in the UD annotation scheme (McDonald et al., 2013), we follow Tandon et al. (2016) convert between the two formats to obtain POS tags, lemmatization and morphological analysis. However, this converted data does not have depen- dency information. To obtain dependency data, we train UDIFY in related language (Hindi) and ap- ply it directly to the converted data above.?! 21 We then train new model on this converted data and augment it with the Hindi data, and upply the result- ing model on the 4 million Marathi and Kannada raw sentences. The performance of the resulting parser is seen in Table 4.
Morphological Agreement The task is formu- lated as given a head (c.g. a verb) and depen- dent (e.g. a noun) in a syntactic relation, when is the agreement for a morphological attribute (e.g. gender) required. This is formulated as a binary classification task, where label is 1 if the values of the head and dependent for the morphological attribute match (e.g. gender = feminine) and 0 oth- erwise. To extract rules for agreement, those leaves where the majority label is 1. How- ever, simply relying on the majority label could be misleading, as might be an artifact of any spuri- ous correlations in the training data, we apply the statistical threshold to filter such leaves. Particu- larly, the null hypothesis Ho states that cach leaf denotes chance-agreement i.e. any observed agree- ment, say for gender between the dependent and its head, is not required rather is purely an artifact of the dataset, while H1 states that the leaf being considered denotes required -ggremmett.eeemett. If the ob- served example distribution under leaf is deemed to be statistically sigmificant when compared to an expected empirical distribution (computed over the training data), we can reject Ho and accept H1.
A.2 Model for Extracting Learning Material
consider
To extract descriptions in human-readable format, we follow Chaudhary et al. (2022) and Chaudhary et al. (2021) and learn interpretable models such as decision tree (Quinlan, 1986) and SVM as they are conducive to interpretation.
For the grammar aspects of Agreement and Word order, we use XGBOOST to learn a decision tree for each language and setting separately, with the following hyperparame- ters: Learning-rate:0.11 n-estimators: subsample:0.6
colsample-bytree:/ 8, objective: multi: :nti::sofon softprob. We perform grid-search over two criterion, namely, gini and entropy and depths ranging from 3-20, and select the best performing tree based on the validation set. However, to keep the rules concise, we limit the tree max-depth to 10 and as find that balances the model performance while keeping the number of rules we derive from the trees also concise. We use the standard train/dev/test splits as provided
Word Order The task is formulated a given a head (e.g. verb) and its dependent (e.g. sub- ject nouns), when is the head before or after its dependent. Similar to above, we design H, as both the labels i.c. before and after are equally likely, and Hi that the leaf takes the label domi- nant under that leaf. Leaves that pass the statistical threshold are assigned the dominant label and syn- tactcc/exicc//morphoooiccc features that lead up to the leaf form the rule.
//ttps://Itre. liit. ac.ttac.iiii wwn.is//wwwiii php?
filenane=dowmllodds/klii
2l "iiidi and Marathi both belong to the same IndoAryan language family and share vocabulary, grammar and even script. Although Kannada belongs to the Dravidian language family, it is still related to Hindi via Sanskrit on which all (Hindi, Marathi and Kannada) are based on.

---

For extracting rules for suffix usage and word usage, we follow Chaudhary et al. (2021) and use SVM classifier. The respective tasks are formulated as follows:
Suffix Usage The task is formulated as given suffix (e.g. -laa) determine the conditions under which this suffix is observed.
Word Usage The task is formulated as given different target language word translations (e.g. 'bhaat' vs 'tandul' for rice). determine the con- ditions under which particular translation is used. Both these tasks are formulated as multi-class classification tasks, and since Chaudhary et al. (2021) find SVMs outperform decision trees, follow their same setup. Specifically, we use the LinearSVM model from sklearn (Fabian, 2011) and perform grid search over the hyperparame- ters: C [0.001, 0.01J, class weight =[ bablanecedd' None]. We select top-20 features for each word to extract the rules. Furthermore, all rules are format- ted using human-readable templates, as shown in Table 2 of Chaudhary et al. (2021).
A.3 Perception Survey
Table 3 we present the questions posed to the teachers for assessing the extracted learning mate- rial. Consent of all subjects was collected before the study, questions regarding personal information such as name, age, gender, were made optional and all results have been aggregated and presented without revealing individual details.

---

Suffix Type
Some examples where no suffi used
Examples
If suffix is usec when:
surnet word's lemma e- (man)
(bhaag)
Examples
ourmnt word's lemme a- tот (raye)
English you would usa tha following word=
0 suffi is used when:
ourrent word is the neighborhood of words WESe8400 (ghatnashhal) (prakaran)
ourrent word is the neighborhood of the words Stovee (thikzan)
Examples
(gnatrasthar)
current word's lemma ge fिया (thiksan)
[Ina) sufto used
neig
aummnt word is the mighhortooo of the word- geent (srowal]]
oureet word with Number Bing
Examples
ourrent word with Case Auz
Diment word is the- colique
Figure 2: Marathi suffixes for nouns with their usages.
For reference, we also show the corresponding English translation for each example immediately below it Examples: The word with suffix सपा is denoted by
(yaa) bाााa (hallyazt) ouguraatrauu (iarthia Rिssपयीया (khidiyanchya) अराा (knacho) ोाकीगर (doshhil) şरयरय (tootya)
the blast also smashed some windows of trigh building
NOUN
190
ROUNT
nerima (mrianmadrhye) (eka) (saha) "an mannyarzahy
(eiee) žalr (delhil) ammber (samavesh) ar (aahe)
the dead include ""eightmoninoii"" child
Figure 3: Illustrative examples extracted for suffix usage. Each example also has Marath transliteration well the English translation to help learners.

---

Vocabulary covering nouns (n), verbs (v), adjectives (a), adverbs (r)
(e.g. rice)
Type
chocolate पffeरèz (chocolate), Examples
pepper - fीts (meeth). Examples
sugar साबा (saakhar). Examples
fodder (chaara). Examples
(ann), Examples
rice - कтă (tandul). Examples
nutrient - अरय (as), Examplee
food (n)
liquor उक (daaru). Examples
stock येअर (share), Examples
flour (peeth), Examples
lemon fिra (limb), Examples
chop पि (chir). Examples
produce Afai (nirmiti), Examples vegetable भाडी (bhaaji), Examples
sibling भहद (aahe), Examples brother HDS (bhasu), Examples dad पााा (baba), Examples
relationships (n)
Figure 4: Marathi lemmas organized by basic categories. Each lemma contains link to illustrative examples which shows its usage in full sentential context, with its English translations.
Adjectives with synoryms antonyms
mon)
English Word
Definiionn
Manath Word
Synoeyms
Antonyma
Nnst
the or nigheet ordering series
(iei (pahit)) Examples
party people ussembiad promole accisballty communat activiy
(sashal), Cormples
doquited discovered
duttton::
Tht Exsmpion
Important
great sigrefcunce value
(mahsha)) Bearrgien
(mothea) Exampies
Army
Exampiess
(paarvejnn))
Figure 5: Marathi adjectives with their definitions, synonyms and antonyms.

---

Type
Question
Answer Choices
Teacher Background
Name (Optional)
Age (Optional)
Gender (Optional)
How long have you been teaching Kumnada?
What level learners you teuch?
Have you used computer- tools for your teaching?
Relevance
1. What percentage of the materials presented
0-100%
in the tool existing curriculum?
Utility
2. How likely are inclined to use this tool in your teaching?
3: Highly likely
2: Likely
Not likely
2.1. likely. for what purpose do you foresce this being used? (multiple answers can be selected):
For lesson preparation. knowledge For evaluating students
Present to the students for self-exploration Other (please specify the reason)
2.2. likely, aspects would you use:
The general concept introduced by material
The rules described the tool
(multiple answers can be selected):
Illustrative examples that accompany the rule Other (please specify the reason)
2.3. NOT likely, why?
material outside the scope
material unclear and needs improvement
(multiple answers can selected):
material already covered by existing curriculum
Other (please specify the reason)
Presentation
3. How did you find the tool?
Very easy use and navigate
Somewhat casy but took time get used
Difficult
Feedback
4.1 What did you like about the tool?
4.2 What did not like about the tool?
4.3 What would you like improve the tool?
Table 3: Usability study: Questions posed to the in-service teachers for evaluating the learning materials on relevance, utility and presentation. This of questions asked for each grammar concept.
Language
Train Dev Test
POS
Morphological Analysis
Lemmatization
Marathi (PAN)
11518 1490 / 1503
85.9
70.1
82.2
Kannada (PAN)
7244 1956 /40
90.3
79.3
90.6
Table 4: We evaluate the parser performance on the respective test sets of the Paninian treebanks (PAN). Since we only have gold annotations for POS, morpho- logical analyses and lemmas, only report results for those. In the second column, we report the number of train/ev/test sentences used the UDIFY parser.

---

Kannada
Marathi
Grammar Concept
Type
AUTOLEX
baseline
AUTOLEX
baseline
Word Order
subject-verb
97.02 (7)
96.97
97.8 (13)
97.7
object-verb
99.11 (6)
99.06
97.89 (13)
96.78
numeral-noun
98.63 (5)
98.36
99.54 (1)
99.54
adjective-noue noun
99.92 (1)
99.92
noun-adposition
99.14 (2)
99.14
Agreement
Gender
71.87 (31)
65.69
61.11 (84)
81.44
Person
24.73 (29)
25.16
Suffix Usage
NST
91.58 (13)
50
90.44 (1)
93.77
NUM
85.2 (5)
82.63
85.91 (1)
93.61
NOUN
78.61 (19)
39
70.23 (13)
67.8
PRON
87.13 (15)
58.03
75.66 (10)
65.07
PART
94.73 (13)
89.35
90.58 (4)
76.77
ADJ
87.74 (14)
66.82
87.55 (4)
83.83
VERB
63.19 (14)
30.52
78.44 (10)
65.87
PROPN
74.57 (16)
46.68
65.6 (9)
71.19
SCONJ
96.85 (9)
64.6
97.59 (5)
86.9
DET
99.53 (4)
61.83
83.91 (2)
81.71
AUX
76.92 (5)
38.46
92.8 (6)
81.57
ADV
75.19 (12)
37.27
86.84 (9)
65.89
ADP
93.55 (6)
76.43
97.12 (6)
67.63
Vocabulary
Semantic Subdivisions
68.68 (385)
58.48
70.58 (285)
56.26
Table 5: Automated evaluation results for learning materials extracted for each grammar concept. Number in the bracket denotes the number of rules extracted for word order, agreement and suffix usage, while vocabulary it denotes the number of word pairs that show fine-grained distinctions.


Reviews:
[{"review_id": "YDBW0wp3uu", "reviewer": "Reviewer_iWaN", "summary": "", "questions": "Were the evaluators (Section 4) and teachers (Section 6) remunerated?\n\nDoes the assessment made in the perception survey (Table 2) include the items identified as incorrect (Section 5)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments and suggestions. We have attempted to clarify some in this response and will work towards completely considering all in the camera-ready. \n\n**what is the level definition or if both schools used the same definition** \n>Thank you for asking this question! The Kannada and Marathi schools that we collaborated with didn’t use any global standards (e.g. CEFR) as they found them unsuitable for their needs, instead they came up with their own. As described in Section 3.1, the teachers shared the eight textbooks they used upon which we built our study. Given that their textbook levels are not based on CEFR, it is not straightforward to assign a standard level (e.g. A1) to our  materials. Additionally, our goal in this work is to provide the materials to the teachers and not the learners directly, as the teachers are best suited to select the appropriate material for different learners. \n\n**lack of contextualization about paper's position on supporting the teacher (e.g., material development) and/or the student (e.g., free exploration and feedback generation)**\n>Our focus is on supporting the teachers primarily, for which we extract both grammar rules and illustrative examples, the motivation being that examples could assist the teachers in their teaching process (as we indeed find in Table 3 that these materials could be useful for lesson prep). Interestingly, the teachers found these illustrative examples could also be given to the students for their own exploration. Moreover, since these materials are automatically extracted and could be prone to errors or may contain material which is age-inappropriate, we chose to share these materials with the teachers who are best suited to decide how to use them. We will add this clarification in our camera-ready.\n\n**Were the evaluators (Section 4) and teachers (Section 6) remunerated?**\n>No, there was no remuneration, all teachers volunteered.\n\n**Does the assessment made in the perception survey (Table 2) include the items identified as incorrect (Section 5)?**\n>Yes, we did not remove those items as we wanted to understand how the materials, as directly obtained automatically without significant human intervention, are perceived when presented as-is. This is close to the real world setting where human evaluation of each grammar point is not feasible. Thanks for asking this question, we will add this explanation in the camera-ready.\n\n**The objective stated is imprecise concerning what has been presented. The authors claim they \"explore to what extent can a combination of NLP techniques and corpus linguistics assist language education for languages.\" However, only a few methods to support language teaching are used.**\n>Thank you for pointing it out, we would like to clarify that by this statement we meant that in this work, we want to explore *one* such combination for automatically extracting language learning materials. Although, comparing different extraction methods would be an interesting future direction. We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work partially fills the gap at the interface between NLP and teaching practice.\n\nAn end-to-end pipeline for 2 resource-poor languages.", "reasons_to_reject": "There is a lack of curriculum. The results were considered more suitable for advanced learners and not relevant for beginners. However, it is not clear what is the level definition (or even if both schools use the same definition). In addition, the linguistic features studied are usually more associated with beginner levels (e.g., A1 and A2 in CEFR).\n\nThe objective stated is imprecise concerning what has been presented. The authors claim they \"explore to what extent can a combination of NLP techniques and corpus linguistics assist language education for languages.\" However, only a few methods to support language teaching are used.\n\nThere is a lack of contextualization about the paper's position on supporting the teacher (e.g., material development) and/or the student (e.g., free exploration and feedback generation).", "questions_to_authors": "Were the evaluators (Section 4) and teachers (Section 6) remunerated?\n\nDoes the assessment made in the perception survey (Table 2) include the items identified as incorrect (Section 5)?", "paper_topic_and_main_contributions": "This paper presents a study of the applicability of NLP tools for assisting language teaching. The authors used their tool (AutoLEX) to extract grammar-like rules concerning word order, word agreement, suffixes and vocabulary in Kannada and Marathi languages (both resource-poor languages). After evaluating the extraction, the authors assess the utility of the output in a teaching context. To that end, 12 (Kannada) and 5 (Marathi) non-professional teachers evaluated the results concerning relevance, utility and presentation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t4jANBYrEH", "reviewer": "Reviewer_r9y4", "summary": "", "questions": "- Do you think this material can be used directly by a learner? If so, it would have been nice to include some learners in your analysis to see how much they find the material helpful as someone who is trying to learn a second language.\n- In A2, there are not many details for the specifics of the model you trained. Please include that.\n- The survey covers relevance and utility, but not accuracy/correctness. I think this should have been part of the survey since extractions are automatic.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and careful review, and providing suggestions for presentation improvement. Please find our response below.\n\n**Authors are mainly following the approach proposed by Chaudhary et al., 2022. So I don't find the approach very novel.**\n>We would like to clarify the novelty of our work, our main contribution is to explore how NLP methods can play a role in language pedagogy especially for languages which lack resources. This entails a) identifying what learning materials to extract, for which we collaborate with teachers, b) extracting the materials from raw text, c) conducting a principled study with in-service teachers to assess its real-world utility.  \nThe approach proposed by Chaudhary et al., 2022.  is just used for b), and could be replaced by some other model which can extract human-readable explanations. Furthermore, we show how to make the approach proposed by Chaudhary et al., 2022 work with true low-resource languages that lack syntactic information (eg. POS tags, dependency parses, morphology) that Chaudhary et al., 2022. assume exists in a particular UD schema. Moreover, we design our teaching points by including real users (teachers in our case) in a collaborative design process and further evaluate its practical utility with a large human study.\n **In sum, the main novelty of our work is a comprehensive experimental study with actual teachers of low-resource languages, but in order to make this study possible we also made a few novel methodological improvements as well. We will make this more clear in the camera-ready.**\n\n**Based on the human analysis, it doesn't seem like the method works that well. For Kannada-relevance was 45-60% and Marathi-relevance was 10-15%. If extracted material is not relevant, then how helpful could it be?**\n>We do agree that currently the materials extracted are not highly relevant but as we also note in L529 and L398, the only sufficiently large publicly available Kannada and Marathi corpora comprised mostly of domains such as news, Wikipedia, talks, religious text, movies which are not particularly suitable for pedagogy. But despite this domain-shift, the teachers do find that the proposed technique has potential and nearly 80% Kannada teachers and 60% Marathi teachers said they were likely to use it in practice, more details on utility per grammar point is in Section 6. The teachers also noted that in L340 that  if the underlying corpora was more learner oriented then the resulting materials would be even more useful. Given that this is the first study of its type (to our knowledge), we also believe there is value in pointing out some of the difficulties in applying these methods, so that they may be further resolved in future work.\n\n\n**Also, there is not much discussion on the correctness of the presented material**\n>Regarding correctness of the material – in Section 5 we conduct a limited human evaluation where the Kannada experts evaluate some questions about word order, suffix usage and vocabulary. Given that its not feasible to manually evaluate all the grammar points, which comprises >200 rules (Table 5), we also conduct an automatic evaluation as shown in Table 5 which shows that the model outperforms the respective baselines. If you have other suggestions on how to better present the results on correctness we would be happy to consider modifications for the camera ready.\n\n**Do you think this material can be used directly by a learner? If so, it would have been nice to include some learners in your analysis to see how much they find the material helpful as someone who is trying to learn a second language.**\n>The materials in the current state cannot be directly used by a learner, because currently the materials are designed for the teachers. We do this for two reasons: a) the teachers are best suited to decide how to present the materials to the learners, and b) given that the materials are automatically extracted over public data, the examples derived from the dataset could be age-inappropriate or could contain offensive content, hence we do not want to expose learners to unverified materials.  \nGiven that our focus was on teacher assistance, for the materials to be directly useful for a learner, they need to be categorised by learner level, verified by experts and the language modified to age-appropriate, which is not a straightforward task and we leave it for future work. Interestingly, one of the Kannada teachers did use some of our extracted materials to teach suffix usage in their lesson, as shown in L552. The teacher did note that they liked our suffix usage material, but that the examples for those were more suited for written language learning rather than spoken language. \n\n**The survey covers relevance and utility, but not accuracy/correctness. I think this should have been part of the survey since extractions are automatic.**\n>Thank you for the suggestion, we do conduct a limited human evaluation of the materials with two teachers in Section 5 accompanied by an automatic evaluation of all rules in Section 4. However, it is not feasible to conduct this evaluation for all grammar rules, which are more than 200 (Table 5), making this a time-consuming task. For reference, the limited evaluation by the two experts itself took almost two weeks. \n\n**Model section should include more details, related work in appendix**\n>Thank you for pointing it out, we will add more details on the modeling approach to avoid any confusion. Due to the brevity of space, we moved the related work to appendix but when we are granted the extra page in camera-ready we could move it back to the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- authors study an interesting and useful task\n- their approach is simple, straightforward, and explainable.\n- I like the fact that they are able to include specific examples and exceptions for extracted rules.\n- they target two low-resource languages\n- they include a very thorough human evaluation (although I have questions about how they designed it)", "reasons_to_reject": "- Authors are mainly following the approach proposed by Chaudhary et al., 2022. So I don't find the approach very novel.\n- Based on the human analysis, it doesn't seem like the method works that well. For Kannada-relevance was 45-60% and Marathi-relevance was 10-15%. If extracted material is not relevant, then how helpful could it be? Also, there is not much discussion on the correctness of the presented material.", "questions_to_authors": "- Do you think this material can be used directly by a learner? If so, it would have been nice to include some learners in your analysis to see how much they find the material helpful as someone who is trying to learn a second language.\n- In A2, there are not many details for the specifics of the model you trained. Please include that.\n- The survey covers relevance and utility, but not accuracy/correctness. I think this should have been part of the survey since extractions are automatic.", "paper_topic_and_main_contributions": "The authors propose methods for automatically extracting learning materials for two Indian languages, Kannada and Marathi, which have low resources for second-language learners. These learning materials include teachable grammar points for word order and agreement, suffix usage, vocabulary, and general information. \n\nThe work is built upon AutoLEX (Chaudhary et al., 2022), which formulates a linguistic question into an NLP prediction task. Models are then trained for this prediction task by extracting relevant features from text corpora, such as POS tagging and dependency parsing, which are known to influence the targeted task. Specific examples can also be presented to the learner.\n\nThe authors use AutoLEX for word order and agreement but adapt it for the rest of above mentioned teachable grammar points.\nThe paper includes a very thorough human evaluation, which was based on a survey completed by multiple language educators.\nThe proposed approach is simple and interesting, and targets low-resource languages, however, I see mixed results in the manual evaluations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- related work is in the appendix which is a bit unusual IMO.\n- footnote 17 is missing \n- Model section should include more details so that the reader doesn't have to look into Chaudhary et al. 2021, 2022"}}, {"review_id": "Z6Tlx8w1H1", "reviewer": "Reviewer_61GB", "summary": "", "questions": "The system heavily depends on AutoLex (a previously published work),\nand the manuscript is difficult to understand for those who are not familiar with AutoLex.  \nI suggest to add to the manuscript a short paragraph to describe what AutoLex does and how. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their encouraging words on the motivation of our work and for their valuable feedback. Please find our response below:\n\n**The described system already works, but it seems to be still in development, not clear which parts of the system might be suitable for teaching of beginner-learners, and which for advanced learners, and how to tune such aspects**\n>We completely agree that this is still in development. The main goal of our work was to investigate whether even using such automatic methods can produce pedagogical materials especially for languages which lack pedagogy resources and we believe that through our work the answer is “yes”. We hope this drives more such practical research in language education where we consult the real users (teachers in our work) to better understand what they need and work with them in collaboration.\nIts indeed an interesting research problem on how to better curate the materials according to learner levels, especially when the schools do not use global standards such as CEFR but instead create their own. One option could be to run our approach on raw text which is more suited to a particular level, but this is not enough as we would need to know what concepts to be introduced at each level as well. \n\n**I suggest to add to the manuscript a short paragraph to describe what AutoLex does and how.**\n>Thanks for pointing it out, we will add more details on AutoLEX to help the reader better understand.\n\n**Line 396: what is \"popular adjectives\" ? Did you mean most frequent ones based on a corpus?**\n>We automatically identified those cross-lingual word pairs that were frequently aligned with each other in our corpus, identified using POS which of those were adjectives and used WordNet to identify the antonyms/synonyms of their English counterparts. We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a very interesting work and quite important, in the following aspects.\nFirst, it utilizes NLP approaches for preparation of educational materials, which is a very practical but often neglected aspect of NLP.\nSecond, it does so in the languages Kannada and Marathi, which generally lack extensive NLP resources.\nThird, it is just refreshing to have some papers that are not about Deep Learning.", "reasons_to_reject": "The described system already works, but it seems to be still in development,\nand some aspects of it are not yet 'fully' developed (in some cases the error rate is too high).\nAlso, it seems the applicability is still in development - it is not clear which parts of the system might be suitable for teaching of beginner-learners, and which are more suitable for advanced learners, and how to tune such aspects.\nObviously much more work needs to be done along those lines. \nHowever even the current version (as described in the manuscript) is very promising and deserves publication.\n\n", "questions_to_authors": "The system heavily depends on AutoLex (a previously published work),\nand the manuscript is difficult to understand for those who are not familiar with AutoLex.  \nI suggest to add to the manuscript a short paragraph to describe what AutoLex does and how. ", "paper_topic_and_main_contributions": "This paper deals with automated methods to prepare educational materials for L2 learners of the languages Kannada and Marathi (learners whose L1 is English).\nSpecifically, the author utilize the previously published AutoLex system. Using some NLP-preprocessed corpora, the overall system produces educational materials (rules and examples) for vocabulary, morphology, grammar and senses.  \nThe study includes reports on interviews with teachers and their ratings concerning the utility of such system for language instructors/teachers.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 396: what is \"popular adjectives\" ?  Did you mean most frequent ones based on a corpus?\n", "Justification_for_Ethical_Concerns": "I find no  ethical concerns with this study."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the applicability of NLP for creating learning materials for second language learning; evaluation with language educators from North America.

There is consensus among the reviewers that this paper presents a valuable contribution. Reviewers especially appreciate the importance and practical application of the task (education), the focus on two under-resourced languages, and the thorough human evaluation. While reviewers (and the authors) acknowledge some mixed results/limitations of the work (e.g., there is work to do before the approach is fully usable/applicable), the reviewers nevertheless find the current work promising.
Decision: Accept-Findings