Evaluating Classroom Integration for Card-it: Digital Flashcards for Learning Italian Morphology
Year: 2023

Abstract:
This paper presents Card-it, a web-based application for learning Italian verb conjugation. Card-it integrates a large-scale finite-state morphological (FSM) analyzer and a flashcard application as a user-friendly way for learners to utilize the analyzer. While Card-it can be used by individual learners, to support classroom adoption, we implemented simple classroom management functionalities such as sharing flashcards to a class and tracking students’ progression. We evaluated Card-it with teachers of Italian. Card-it was reported as engaging and supportive, especially by featuring two different quiz types combined with a verb form lookup feature. Teachers were optimistic about the potential of Card-it as a classroom supplementary tool for learners of Italian as L2. Future work includes sample sentences and a complete learners evaluation.

heading: 1 Introduction
text: Learning verb morphology plays a crucial role in the acquisition of morphologically rich languages (Slabakova, 2009), such as Italian and French. Thus, learners of Italian deal with the acquisition of a rich system of verbal inflections (e.g., Pizzuto and Caselli, 1994). Explicit morphological instructions and training have been shown to help students on acquiring new words as well as to improve their syntactic knowledge (Chen and Schwartz, 2018; Mobaraki and Jahromi, 2019). Similarly, raising meta-linguistic awareness improves the learners’ production and competence in second language (L2) acquisition (Heift, 2004; Kieseier et al., 2022). To support learners of Italian as L2, we designed, implemented, and evaluated Cardit with the help of experts: teachers of Italian as a foreign language. Card-it fosters meta-linguistic knowledge when presenting linguistic information on the analysis of verb forms (i.e., for the verb mangiare (to eat) “Prima Persona Singulare Presente Indicativo” → (io) mangio) along with additional
explanations of linguistic categories related to verb morphology that are displayed on demand. In addition, meta-linguistic information is also used to present corrective feedback (see Sec. 4.2).
Card-it is an online application for teachers and learners of Italian to create collections of digital flashcards – based on a semi-automatic approach – with which they can study and test themselves on verb morphology explicitly. Our choice for using a digital flashcard design reflects a traditional way of learning vocabulary explicitly, which has been shown to be a successful learning method that is perceived well by students (Yüksel et al., 2022). While some flashcard systems may support verb morphology with pre-defined cards and modules, they do not allow for the customization of cards or decks (e.g., Memrise1). Other systems support custom card collections, but they require manual input of the card information (e.g., Anki2). Yet, these systems do not enable teachers to track and analyze their students’ progress over time. In addition, Card-it’s learner-centred design embeds corrective feedback, meta-linguistic information, and different study modes.
This paper introduces the system’s architecture, the FSM implementation, and Card-it’s iterative design and features. Lastly, we report the results of a brief evaluation with Italian teachers which indicates Card-it’s potential for their classroom and outlines our future steps towards a learners evaluation.

heading: 2 Related Work
text: Traditionally, Natural Language Processing (NLP) tools like an FSM are a component of larger pipelines, for example, as a tokenizer (e.g., Jurafsky and Martin, 2009). As a result, using these tools is often not intuitive or easy for users unfamiliar with NLP. However, since these tools can
1https://www.memrise.com/. Accessed 05-2023. 2https://www.ankiapp.com/. Accessed 05-2023.
130
work with text, NLP has become an integral part of the field of Computer-Assisted Language Learning (CALL), with several systems using NLP tools in a language-learning context. Examples include E-Tutor (Heift, 2010), an intelligent tutoring system for learners of German that is fully incorporated into the German curriculum at Simon Fraser University; TAGARELA (Amaral and Meurers, 2011), a system for Portuguese that includes exercises on vocabulary; and FeedBook (Meurers et al., 2019), an intelligent tutoring system for English that can be fully integrated into regular classes.
Similarly, Google-Assisted Language Learning (GALL), corpus-based or data-driven learning (DDL) are increasing in popularity as language learning tools (Conroy, 2010; Pérez-Paredes, 2022). While GALL refers specifically to learners using tools provided by Google, both GALL and DDL happen when learners take advantage of online access and text processing power to use corpus tools, such as dictionaries and linguistic corpora.
Furthermore, Yoon (2016) verified that DDL was an effective cognitive tool for helping people with their lexical and grammatical problems while dealing with concordance tasks; for example, learning frequent word pairs such as to take instead of to eat a [medicine] pill. However, he suggests that some of the available resources are not user-friendly and difficult to use, such as functions for linguistic resources applied for stemming. That said, Card-it’s design uses a learner-centred approach with teacher support features; it provides a user-friendly interface to leverage an FSM to power a semi-automatic generation of flashcards that can be used to study and self-assess Italian verb conjugations. Related to using FSM in Card-it, Kaya and Eryiğit (2015) used a Finite-State Transducer to power a Turkish word synthesis system and a word-level translation system between Turkish and English. Another example is the ICALL system for two Saami languages that is based on Finite-State Transducers (Antonsen et al., 2013).

heading: 3 Card-it: System Architecture
text: Card-it is a web-based application consisting of two components: back-end and front-end.
Back-end: The FSM Analyzer. The main component of the back-end is our FSM, containing over 5000 verb lemmata and their conjugations Beesley and Karttunen (2003). It was created by extracting verb roots from free resources, the Morph-it! lexi-
con by Zanchetta and Baroni (2005) and the online dictionary provided by one of Italy’s leading news magazines, Corriere della Sera3. FSMs are usually part of a text processing pipeline within NLP tools. Here, we leveraged our FSM as a dynamic form generator and analyzer in a language-learning context. The FSM ties a verb form to its linguistic analysis: it may analyze a verb form and return its linguistic tags (analysis) or generate a verb form given its linguistic tags (generation) – see Fig. 1.
In our case, the FSM consists of a lexicon that contains verb stems, their inflectional paradigms and the appropriate morphological analysis. The lexicon of the FSM creates all verb forms following the regular pattern of concatenating stems with their respective inflectional endings. With the use of regular expressions the FSM is able to manipulate those regular forms of the lexicon on the basis of phonological rules. For example, some forms require the insertion of an -h to retain certain pronunciation patterns. Consider the verb mancare (“to miss"): the regular inflection paradigm in the lexicon creates the incorrect form manci (“you miss"), for the second person singular present indicative. However, to retain the correct pronunciation, the correct form is manchi. Whenever the FSM is run, it first creates all forms in the lexicon and then applies regular expressions to manipulate these forms based on phonological rules of the language. This architecture allows us to build a powerful and large morphological resource since it automatically creates verb forms on the basis of their stems. If we were to add new verbs to our tool, it simply requires to manually add verb stems into the FSM lexicon.
Verbs generated by the FSM, user accounts, flashcards and classroom organization are stored in a MySQL database. A Flask middleware is responsible for querying changes users request from the front-end. These changes are related to flash-
3https://dizionari.corriere.it/ dizionario_italiano/. Accessed 05-2023
card, classroom, and account organization. The main advantage of this back-end architecture is to scale the system for multiple users simultaneously; this integration approach has been taken by others (de Bernardinis et al., 2015). A set of Python scripts are responsible for parsing and updating the database with any changes to the FSM; currently, these updates are triggered manually whenever the list of verbs or morphology is altered.
Front-end: User Interface. The user interface front-end of Card-it is developed with React.js. The main function of the front-end is the flashcard design for users to study and be assessed from. Sec. 4 explains Card-it’s digital flashcards design and interaction.

heading: 4 Card-it Design and Features
text: Card-it can be used for autonomous learners who may interact with the app to study Italian conjugations on their own. In addition, Card-it can also be integrated by teachers in the classroom. In either case, learners interact with verbs and conjugations via digital flashcards.

heading: 4.1 Grouping and Organizing Flashcards
text: The flashcards reflect a traditional way of language learning. Particularly, the flashcard design reflects both directions of the FSM: one side of the card contains a verb form, the other its linguistic attributes (compare Figs. 1 and 2); learners may choose which side they want to use for studying.
Flashcards can be organized in decks; decks can be organized in collections. Both learners and teachers can organize flashcards according to their learning or teaching needs. For example, a teacher may create collections for different language classes: in a collection “Italian for Beginners”, the teacher may add a deck for present tense only, another for past tense(s), and so on.
Users can create decks of cards by searching the database for specific verbs and filtering values for the categories tense, mood, number, and person.
Alternatively, if no value is chosen, Card-it returns all forms for that category. E.g., one may search the verb amare “to love”, selecting the values present tense and indicative mood, but selecting none for person and number. Card-it returns 1st, 2nd and 3rd person singular and plural forms of amare, where each form is a flashcard. Users can select any flashcards they want to add to a specific deck.
The knowledge of the underlying linguistic concepts benefits the acquisition of a new language Heift (2004). Therefore, we made the decision to include the morphological attributes in the application to raise meta-linguistic awareness. Card-it also offers a page with definitions and explanations of all the terms used (i.e., “What is tense?”).

heading: 4.2 Studying and Self-Assessing
text: Card-it offers different study modes and ways to interact with its flashcards.
Studying with Card-it. One way is to use the flip card functionality, where Card-it presents the user with one side, and the learner can think about the content on the corresponding side. When hovering the mouse over the card, the flashcard flips to its other side, and learners can check their answer. Another mode is conjugation. Here, the flashcard presents the user with the infinitive form of a verb, a tense/mood combination, and personal pronouns for number/person configurations and prompts the user to type in the corresponding verb form. If wrong, the system returns the corrected answer as seen on the left side of Fig. 3, showing the “conjugation” study mode, with corrective feedback.
Self-assessment and corrective feedback. For testing, Card-it has two different types of quizzes, called Identify Tense, Conjugate, and a third
Mixture, a random mix of tasks from the other two types. While “Conjugate” corresponds to the above-described study mode prompting the user to type in the corresponding conjugated form, in quiz mode, it additionally contains a “Hint” button that displays multiple choice options when used (Fig. 4), otherwise hidden by default.
Studies have shown the importance of informative feedback for a positive learning trajectory as it helps learners to understand the nature of their mistakes and to improve in the future (e.g., Heift, 2004). Card-it returns informative feedback to the learner by checking whether their incorrect answer corresponds to another morphological analysis and returning that information to them, see Fig. 3. The second quiz type, “Identify Tense”, presents learners with a specific verb form, asking them to select its respective tense (Fig. 4). All quiz types may be used for self-assessment or as classroom activities.

heading: 4.3 Classroom Management and Analytics
text: To enable classroom and teacher support, we focused on 3 main tasks. The tasks supported in this category are (1) creating classrooms and generating a unique code that is shared with students allowing them to join it, (2) sharing specific collections to one or multiple classrooms, and (3) tracking the progress of students enrolled in the classroom.
After students join the classroom using the code, they can explore all collections and decks their teacher shares. Similarly, students have access to both studying and quiz modes for all decks in the classroom. Teachers can access statistical information on the students’ progress with the classroom decks. Teachers can analyze individual attempts for each student with a breakdown of correct and incorrect answers. Alternatively, teachers can see average scores per attempt for the entire group; and analyze the class’ progress over time. Lastly, Cardit shows the number of correct attempts for each card in a deck. Thus, the teacher can pinpoint the specific cards students had the most trouble with.

heading: 5 Evaluation
text: We took an iterative design approach for implementing Card-it, where we performed a preliminary expert evaluation (N = 2) with teachers of Italian at the Institute of Speech and Language at our university with an earlier version of the application. Based on this preliminary evaluation, we determined the fitness of the flashcards and the
quiz formats and iterated over the application. The teachers responded positively to Card-it as a digital version of their current classroom practices, such as verb conjugation worksheets. We also learned that Card-it could be adopted as a supplemental tool to the classroom, which led us to implement the classroom features. The following section describes our second expert evaluation.

heading: 5.1 Card-it Expert Evaluation
text: After implementing changes to reflect the feedback from the early preliminary evaluation; we reached out to Italian teachers via our professional networks. In total, 9 teachers from 2 institutions in Germany were invited to participate. Of those, 5 volunteered, but only 3 completed the study. Participants were teachers of Italian language courses; after receiving the study instructions, they had 14 days to follow to complete all steps remotely, then compensated with a $20 Amazon gift card.

heading: 5.1.1 Methodology
text: We ran our expert evaluation remotely, which allowed us to provide flexibility to participants to complete the study. Participants were asked to follow three steps to complete the study: (1) Watch a recorded video demo of Card-it’s main features; (2) Explore Card-it on their own using both teacher and student account types; (3) Answer a survey questionnaire about their experience using Card-it. In the survey, we asked 5-Point Likert Scale questions on general usability, the potential for classroom adoption, and specific questions on different features such as studying and testing modes. We also asked experts to answer a section where they give their opinions from a student perspective.

heading: 5.1.2 Results and Discussion
text: The system’s usability was rated positively, with two experts selecting easy and one expert very easy. All experts rated both quiz types, “Conjugate” and “Identify Tense”, as either appropriate or very appropriate. One expert mentioned the quizzes were their favourite features. When asked to rate the classroom management usability, two chose good and one very good. As a follow-up, we asked them about the steps to create a classroom: one expert found it difficult, and the others easy. They all mentioned that they could foresee themselves using Card-it for homework in their classes or as a tool for students to self-study at home. When asked to take on a student’s perspective, they all rated the
quiz and verb look-up features of Card-it most useful. Yet, they suggested including translations and example sentences containing the individual verbs as it would be useful for students and teachers’ perspectives.

heading: 6 Future Work and Conclusion
text: This paper discussed the power of the adequate use of NLP tools in language learning, including designing appropriate interfaces. We presented Card-it as a user-friendly app for learning Italian verb conjugation using digital flashcards; we also described Card-it’s classroom management and analytics features (more details in App. A). Lastly, we discussed our iterative approach to design, which combined expert evaluations between iterations of Card-it. The results of the expert evaluation show that according to their expertise, Card-it is an appropriate conjugation tool for autonomous learning and for classroom integration as a supplementary resource. Card-it’s usability and different quiz functions were positively evaluated. Nonetheless, we also learned that Card-it might be further improved by adding example sentences. The most promising result from the evaluation is the experts’ expression of interest in using Card-it in their classrooms.
Despite asking for the experts’ perspectives as students, it would be more reliable to run a user study with learners of Italian as a second language. We are designing a remote longitudinal study with 3 weekly sessions. At the end of each session, participants are invited to submit a Card-it quiz and a short usability survey. We also plan on testing their knowledge of a set of verb conjugations before and after their study period of 3 weeks. Other future directions may include gamification of Cardit’s quizzes and quiz modes that can support live classroom exercises such as Kahoot (Dellos, 2015).

heading: Acknowledgements
text: We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC).

heading: A Classroom Management
text: Fig. 5 shows an example classroom with two collections and the entry code for students to join the classroom:
Fig. 6 shows the statistical overview of students’ performance in a quiz. Teachers may filter for a specific collection (here: Presente Indicativo), deck (here: Regular Verbs) and quiz type (here: Conjugate). Additionally, teachers see the score for each student:
Fig. 7 illustrates how teachers can check on the groups’ performance on every single card, sorted from the least correct to the most correct:
Teachers may select one particular student to get detailed information on their performance, as in Fig. 8:
Fig. 9 shows the same example classroom as in Fig. 5 but from the students’ perspective. Here, students can select one of the three quiz types or scroll down for study mode:

heading: Future Work
text: Other future directions may include gamification of Cardit’s quizzes and quiz modes that can support live classroom exercises such as Kahoot (Dellos, 2015)