GenAI4ELab/papercli-papers huggingface dataset:
Sheffield’s Submission to the AmericasNLP Shared Task on Machine Translation into Indigenous Languages
Authors: Edward Gow-Smith; Danae Sánchez Villegas
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: The University of Sheffield took part in the shared task 2023 AmericasNLP for all eleven language pairs. Our models consist of training different variations of NLLB-200 model on data provided by the organizers and available data from various sources such as constitutions, handbooks and news articles. Our models outperform the baseline model on the development set on chrF with substantial improvements particularly for Aymara, Guarani and Quechua. On the test set, our best submission achieves the highest average chrF of all the submissions, we rank first in four of the eleven languages, and at least one of our models ranks in the top 3 for all languages.
Enhancing Translation for Indigenous Languages: Experiments with Multilingual Models
Authors: Atnafu Lambebo Tonja; Hellina Hailu Nigatu; Olga Kolesnikova; Grigori Sidorov; Alexander Gelbukh; Jugal Kalita
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: This paper describes CIC NLP’s submission to the AmericasNLP 2023 Shared Task on machine translation systems for indigenous languages of the Americas. We present the system descriptions for three methods. We used two multilingual models, namely M2M-100 and mBART50, and one bilingual (one-to-one) — Helsinki NLP Spanish-English translation model, and experimented with different transfer learning setups. We experimented with 11 languages from America and report the setups we used as well as the results we achieved. Overall, the mBART setup was able to improve upon the baseline for three out of the eleven languages.
Findings of the AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages
Authors: Abteen Ebrahimi; Manuel Mager; Shruti Rijhwani; Enora Rice; Arturo Oncevay; Claudia Baltazar; María Cortés; Cynthia Montaño; John E. Ortega; Rolando Coto-solano; Hilaria Cruz; Alexis Palmer; Katharina Kann
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: In this work, we present the results of the AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages of the Americas. This edition of the shared task featured eleven language pairs, one of which – Chatino-Spanish – uses a newly collected evaluation dataset, consisting of professionally translated text from the legal domain. Seven teams participated in the shared task, with a total of 181 submissions. Additionally, we conduct a human evaluation of the best system outputs, and compare them to the best submissions from the prior shared task. We find that this analysis agrees with the quantitative measures used to rank submissions, which shows further improvements of 9.64 ChrF on average across all languages, when compared to the prior winning system.
LFTK: Handcrafted Features in Computational Linguistics
Authors: Bruce W. Lee; Jason Lee
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: Past research has identified a rich set of handcrafted linguistic features that can potentially assist various tasks. However, their extensive number makes it difficult to effectively select and utilize existing handcrafted features. Coupled with the problem of inconsistent implementation across research works, there has been no categorization scheme or generally-accepted feature names. This creates unwanted confusion. Also, no actively-maintained open-source library extracts a wide variety of handcrafted features. The current handcrafted feature extraction practices have several inefficiencies, and a researcher often has to build such an extraction system from the ground up. We collect and categorize more than 220 popular handcrafted features grounded on past literature. Then, we conduct a correlation analysis study on several task-specific datasets and report the potential use cases of each feature. Lastly, we devise a multilingual handcrafted linguistic feature extraction system in a systematically expandable manner. We open-source our system to give the community a rich set of pre-implemented handcrafted features.
Improving Mathematics Tutoring With A Code Scratchpad
Authors: Shriyash Upadhyay; Etan Ginsberg; Chris Callison-Burch
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: Large language models can solve reasoning tasks (like math problems) more effectively when they are allowed to generate rationales. However, a good tutoring system should not just generate solutions, but should also generate explanations and should be able to correct and guide students. We show that providing a code scratchpad improves performance on each tutoring step with a gradeschool mathematics dataset. On these tutoring tasks, GPT-3 models provided with a code scratchpad significantly outperform those given only a language scratchpad (77.7% vs 48.7% cumulative accuracy).
A Transfer Learning Pipeline for Educational Resource Discovery with Application in Survey Generation
Authors: Irene Li; Thomas George; Alex Fabbri; Tammy Liao; Benjamin Chen; Rina Kawamura; Richard Zhou; Vanessa Yan; Swapnil Hingmire; Dragomir Radev
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: Effective human learning depends on a wide selection of educational materials that align with the learner’s current understanding of the topic. While the Internet has revolutionized human learning or education, a substantial resource accessibility barrier still exists. Namely, the excess of online information can make it challenging to navigate and discover high-quality learning materials in a given subject area. In this paper, we propose an automatic pipeline for building an educational resource discovery system for new domains. The pipeline consists of three main steps: resource searching, feature extraction, and resource classification. We first collect frequent queries from a set of seed documents, and search the web with these queries to obtain candidate resources such as lecture slides and introductory blog posts. Then, we process these resources for BERT-based features and meta-features. Next, we train a tree-based classifier to decide whether they are suitable learning materials. The pipeline achieves F1 scores of 0.94 and 0.82 when evaluated on two similar but novel domains. Finally, we demonstrate how this pipeline can benefit two applications: prerequisite chain learning and leading paragraph generation for surveys. We also release a corpus of 39,728 manually labeled web resources and 659 queries from NLP, Computer Vision (CV), and Statistics (STATS).
Using Learning Analytics for Adaptive Exercise Generation
Authors: Tanja Heck; Detmar Meurers
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: Single Choice exercises constitute a central exercise type for language learning in a learner’s progression from mere implicit exposure through input enhancement to productive language use in open exercises. Distractors that support learning in the individual zone of proximal development should not be derived from static analyses of learner corpora, but rely on dynamic learning analytics based on half-open exercises. We demonstrate how a system’s error diagnosis module can be re-used for automatic and dynamic generation and adaptation of distractors, as well as to inform exercise generation in terms of relevant learning goals and reasonable chunking in Jumbled Sentences exercises.
Reviewriter: AI-Generated Instructions For Peer Review Writing
Authors: Xiaotian Su; Thiemo Wambsganss; Roman Rietsche; Seyed Parsa Neshaei; Tanja Käser
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: Large Language Models (LLMs) offer novel opportunities for educational applications that have the potential to transform traditional learning for students. Despite AI-enhanced applications having the potential to provide personalized learning experiences, more studies are needed on the design of generative AI systems and evidence for using them in real educational settings. In this paper, we design, implement and evaluate \texttt{Reviewriter}, a novel tool to provide students with AI-generated instructions for writing peer reviews in German. Our study identifies three key aspects: a) we provide insights into student needs when writing peer reviews with generative models which we then use to develop a novel system to provide adaptive instructions b) we fine-tune three German language models on a selected corpus of 11,925 student-written peer review texts in German and choose German-GPT2 based on quantitative measures and human evaluation, and c) we evaluate our tool with fourteen students, revealing positive technology acceptance based on quantitative measures. Additionally, the qualitative feedback presents the benefits and limitations of generative AI in peer review writing.
Towards L2-friendly pipelines for learner corpora: A case of written production by L2-Korean learners
Authors: Hakyung Sung; Gyu-Ho Shin
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: We introduce the Korean-Learner-Morpheme (KLM) corpus, a manually annotated dataset consisting of 129,784 morphemes from second language (L2) learners of Korean, featuring morpheme tokenization and part-of-speech (POS) tagging. We evaluate the performance of four Korean morphological analyzers in tokenization and POS tagging on the L2- Korean corpus. Results highlight the analyzers’ reduced performance on L2 data, indicating the limitation of advanced deep-learning models when dealing with L2-Korean corpora. We further show that fine-tuning one of the models with the KLM corpus improves its accuracy of tokenization and POS tagging on L2-Korean dataset.
ChatBack: Investigating Methods of Providing Grammatical Error Feedback in a GUI-based Language Learning Chatbot
Authors: Kai-Hui Liang; Sam Davidson; Xun Yuan; Shehan Panditharatne; Chun-Yen Chen; Ryan Shea; Derek Pham; Yinghua Tan; Erik Voss; Luke Fryer
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: The increasing use of AI chatbots as conversation partners for second-language learners highlights the importance of providing effective feedback. To ensure a successful learning experience, it is essential for researchers and practitioners to understand the optimal timing, methods of delivery, and types of feedback that are most beneficial to learners. Synchronous grammar corrective feedback (CF) has been shown to be more effective than asynchronous methods in online writing tasks. Additionally, self-correction by language learners has proven more beneficial than teacher-provided correction, particularly for spoken language skills and non-novice learners. However, existing language-learning AI chatbots often lack synchronous CF and self-correction capabilities. To address this, we propose a synchronous conversational corrective feedback (CCF) method, which allows self-correction and provides metalinguistic explanations (ME). Our study suggests that in chatbot-driven language-learning tools, corrective feedback is more effectively delivered through means other than the social chatbot, such as a GUI interface. Furthermore, we found that guided self-correction offers a superior learning experience compared to providing explicit corrections, particularly for learners with high learning motivation or lower linguistic ability.
Enhancing Video-based Learning Using Knowledge Tracing: Personalizing Students’ Learning Experience with ORBITS
Authors: Shady Shehata; David Santandreu Calonge; Philip Purnell; Mark Thompson
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: As the world regains its footing following the COVID-19 pandemic, academia is striving to consolidate the gains made in students’ education experience. New technologies such as video-based learning have shown some early improvement in student learning and engagement. In this paper, we present ORBITS predictive engine at YOURIKA company, a video-based student support platform powered by knowledge tracing. In an exploratory case study of one master’s level Speech Processing course at the Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) in Abu Dhabi, half the students used the system while the other half did not. Student qualitative feedback was universally positive and compared the system favorably against current available methods. These findings support the use of artificial intelligence techniques to improve the student learning experience.
Enhancing Human Summaries for Question-Answer Generation in Education
Authors: Hannah Gonzalez; Liam Dugan; Eleni Miltsakaki; Zhiqi Cui; Jiaxuan Ren; Bryan Li; Shriyash Upadhyay; Etan Ginsberg; Chris Callison-Burch
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: We address the problem of generating high-quality question-answer pairs for educational materials. Previous work on this problem showed that using summaries as input improves the quality of question generation (QG) over original textbook text and that human-written summaries result in higher quality QG than automatic summaries. In this paper, a) we show that advances in Large Language Models (LLMs) are not yet sufficient to generate quality summaries for QG and b) we introduce a new methodology for enhancing bullet point student notes into fully fledged summaries and find that our methodology yields higher quality QG. We conducted a large-scale human annotation study of generated question-answer pairs for the evaluation of our methodology. In order to aid in future research, we release a new dataset of 9.2K human annotations of generated questions.
Difficulty-Controllable Neural Question Generation for Reading Comprehension using Item Response Theory
Authors: Masaki Uto; Yuto Tomikawa; Ayaka Suzuki
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: Question generation (QG) for reading comprehension, a technology for automatically generating questions related to given reading passages, has been used in various applications, including in education. Recently, QG methods based on deep neural networks have succeeded in generating fluent questions that are pertinent to given reading passages. One example of how QG can be applied in education is a reading tutor that automatically offers reading comprehension questions related to various reading materials. In such an application, QG methods should provide questions with difficulty levels appropriate for each learner’s reading ability in order to improve learning efficiency. Several difficulty-controllable QG methods have been proposed for doing so. However, conventional methods focus only on generating questions and cannot generate answers to them. Furthermore, they ignore the relation between question difficulty and learner ability, making it hard to determine an appropriate difficulty for each learner. To resolve these problems, we propose a new method for generating question–answer pairs that considers their difficulty, estimated using item response theory. The proposed difficulty-controllable generation is realized by extending two pre-trained transformer models: BERT and GPT-2.
Evaluating Classroom Potential for Card-it: Digital Flashcards for Studying and Learning Italian Morphology
Authors: Mariana Shimabukuro; Jessica Zipf; Shawn Yama; Christopher Collins
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: This paper presents Card-it, a web-based application for learning Italian verb conjugation. Card-it integrates a large-scale finite-state morphological~(FSM) analyzer and a flashcard application as a user-friendly way for learners to utilize the analyzer. While Card-it can be used by individual learners, to support classroom adoption, we implemented simple classroom management functionalities such as sharing flashcards to a class and tracking students’ progression. We evaluated Card-it with teachers of Italian. Card-it was reported as engaging and supportive, especially by featuring two different quiz types combined with a verb form look-up feature. Teachers were optimistic about the potential of Card-it as a classroom supplementary tool for learners of Italian as L2. Future work includes sample sentences and a complete learners evaluation.
Scalable and Explainable Automated Scoring for Open-Ended Constructed Response Math Word Problems
Authors: Scott Hellman; Alejandro Andrade; Kyle Habermehl
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: Open-ended constructed response math word problems (“math plus text”, or MPT) are a powerful tool in the assessment of students’ abilities to engage in mathematical reasoning and creative thinking. Such problems ask the student to compute a value or construct an expression and then explain, potentially in prose, what steps they took and why they took them. MPT items can be scored against highly structured rubrics, and we develop a novel technique for the automated scoring of MPT items that leverages these rubrics to provide explainable scoring. We show that our approach can be trained automatically and performs well on a large dataset of 34,417 responses across 14 MPT items.
Gender-Inclusive Grammatical Error Correction through Augmentation
Authors: Gunnar Lund; Kostiantyn Omelianchuk; Igor Samokhin
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: In this paper we show that GEC systems display gender bias related to the use of masculine and feminine terms and the gender-neutral singular “they”. We develop parallel datasets of texts with masculine and feminine terms, and singular “they”, and use them to quantify gender bias in three competitive GEC systems. We contribute a novel data augmentation technique for singular “they” leveraging linguistic insights about its distribution relative to plural “they”. We demonstrate that both this data augmentation technique and a refinement of a similar augmentation technique for masculine and feminine terms can generate training data that reduces bias in GEC systems, especially with respect to singular “they” while maintaining the same level of quality.
ReadAlong Studio Web Interface for Digital Interactive Storytelling
Authors: Aidan Pine; David Huggins-Daines; Eric Joanis; Patrick Littell; Marc Tessier; Delasie Torkornoo; Rebecca Knowles; Roland Kuhn; Delaney Lothian
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: We develop an interactive web-based user interface for performing textspeech alignment and creating digital interactive “read-along audio books that highlight words as they are spoken and allow users to replay individual words when clicked. We build on an existing Python library for zero-shot multilingual textspeech alignment (Littell et al., 2022), extend it by exposing its functionality through a RESTful API, and rewrite the underlying speech recognition engine to run in the browser. The ReadAlong Studio Web App is open-source, user-friendly, prioritizes privacy and data sovereignty, allows for a variety of standard export formats, and is designed to work for the majority of the world’s languages.
Labels are not necessary: Assessing peer-review helpfulness using domain adaptation based on self-training
Authors: Chengyuan Liu; Divyang Doshi; Muskaan Bhargava; Ruixuan Shang; Jialin Cui; Dongkuan Xu; Edward Gehringer
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: A peer-assessment system allows students to provide feedback on each other’s work. An effective peer assessment system urgently requires helpful reviews to facilitate students to make improvements and progress. Automated evaluation of review helpfulness, with the help of deep learning models and natural language processing techniques, gains much interest in the field of peer assessment. However, collecting labeled data with the “helpfulness” tag to build these prediction models remains challenging. A straightforward solution would be using a supervised learning algorithm to train a prediction model on a similar domain and apply it to our peer review domain for inference. But naively doing so can degrade the model performance in the presence of the distributional gap between domains. Such a distributional gap can be effectively addressed by Domain Adaptation (DA). Self-training has recently been shown as a powerful branch of DA to address the distributional gap. The first goal of this study is to evaluate the performance of self-training-based DA in predicting the helpfulness of peer reviews as well as the ability to overcome the distributional gap. Our second goal is to propose an advanced self-training framework to overcome the weakness of the existing self-training by tailoring knowledge distillation and noise injection, to further improve the model performance and better address the distributional gap.
Generating Dialog Responses with Specified Grammatical Items for Second Language Learning
Authors: Yuki Okano; Kotaro Funakoshi; Ryo Nagata; Manabu Okumura
Venue: ACL 2023 (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023))

Abstract: This paper proposes a new second language learning task of generating a response including specified grammatical items. We consider two approaches: 1) fine-tuning a pre-trained language model (DialoGPT) by reinforcement learning and 2) providing a few-shot prompt to a large language model (GPT-3). For reinforcement learning, we examine combinations of three reward functions that consider grammatical items, diversity, and fluency. Our experiments confirm that both approaches can generate responses including the specified grammatical items and that it is crucial to consider fluency rather than diversity as the reward function.