GenAI4ELab/papercli-papers huggingface dataset:
Improving Readability Assessment with Ordinal Log-Loss
Authors: Ho Hung Lim; John Lee
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: Automatic Readability Assessment (ARA) predicts the level of difficulty of a text, e.g. at Grade 1 to Grade 12. ARA is an ordinal classification task since the predicted levels follow an underlying order, from easy to difficult. However, most neural ARA models ignore the distance between the gold level and predicted level, treating all levels as independent labels. This paper investigates whether distance-sensitive loss functions can improve ARA performance. We evaluate a variety of loss functions on neural ARA models, and show that ordinal log-loss can produce statistically significant improvement over the standard cross-entropy loss in terms of adjacent accuracy in a majority of our datasets.
Automated Sentence Generation for a Spaced Repetition Software
Authors: Benjamin Paddags; Daniel Hershcovich; Valkyrie Savage
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper presents and tests AllAI, an app that utilizes state-of-the-art NLP technology to assist second language acquisition through a novel method of sentence-based spaced repetition. Diverging from current single word or fixed sentence repetition, AllAI dynamically combines words due for repetition into sentences, enabling learning words in context while scheduling them independently. This research explores various suitable NLP paradigms and finds a few-shot prompting approach and retrieval of existing sentences from a corpus to yield the best correctness and scheduling accuracy. Subsequently, it evaluates these methods on 26 learners of Danish, finding a four-fold increase in the speed at which new words are learned, compared to conventional spaced repetition. Users of the retrieval method also reported significantly higher enjoyment, hinting at a higher user engagement.
Using Large Language Models to Assess Young Students’ Writing Revisions
Authors: Tianwen Li; Zhexiong Liu; Lindsay Matsumura; Elaine Wang; Diane Litman; Richard Correnti
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: Although effective revision is the crucial component of writing instruction, few automated writing evaluation (AWE) systems specifically focus on the quality of the revisions students undertake. In this study we investigate the use of a large language model (GPT-4) with Chain-of-Thought (CoT) prompting for assessing the quality of young students’ essay revisions aligned with the automated feedback messages they received. Results indicate that GPT-4 has significant potential for evaluating revision quality, particularly when detailed rubrics are included that describe common revision patterns shown by young writers. However, the addition of CoT prompting did not significantly improve performance. Further examination of GPT-4’s scoring performance across various levels of student writing proficiency revealed variable agreement with human ratings. The implications for improving AWE systems focusing on young students are discussed.
Automatic Crossword Clues Extraction for Language Learning
Authors: Santiago Berruti; Arturo Collazo; Diego Sellanes; Aiala Rosá; Luis Chiruzzo
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: Crosswords are a powerful tool that could be used in educational contexts, but they are not that easy to build. In this work, we present experiments on automatically extracting clues from simple texts that could be used to create crosswords, with the aim of using them in the context of teaching English at the beginner level. We present a series of heuristic patterns based on NLP tools for extracting clues, and use them to create a set of 2209 clues from a collection of 400 simple texts. Human annotators labeled the clues, and this dataset is used to evaluate the performance of our heuristics, and also to create a classifier that predicts if an extracted clue is correct. Our best classifier achieves an accuracy of 84%.
Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners
Authors: Abigail Gurin Schleifer; Beata Beigman Klebanov; Moriah Ariely; Giora Alexandron
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: Unsupervised clustering of student responses to open-ended questions into behavioral and cognitive profiles using pre-trained LLM embeddings is an emerging technique, but little is known about how well this captures pedagogically meaningful information. We investigate this in the context of student responses to open-ended questions in biology, which were previously analyzed and clustered by experts into theory-driven Knowledge Profiles (KPs).Comparing these KPs to ones discovered by purely data-driven clustering techniques, we report poor discoverability of most KPs, except for the ones including the correct answers. We trace this ‘discoverability bias’ to the representations of KPs in the pre-trained LLM embeddings space.
Assessing Student Explanations with Large Language Models Using Fine-Tuning and Few-Shot Learning
Authors: Dan Carpenter; Wookhee Min; Seung Lee; Gamze Ozogul; Xiaoying Zheng; James Lester
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: The practice of soliciting self-explanations from students is widely recognized for its pedagogical benefits. However, the labor-intensive effort required to manually assess students’ explanations makes it impractical for classroom settings. As a result, many current solutions to gauge students’ understanding during class are often limited to multiple choice or fill-in-the-blank questions, which are less effective at exposing misconceptions or helping students to understand and integrate new concepts. Recent advances in large language models (LLMs) present an opportunity to assess student explanations in real-time, making explanation-based classroom response systems feasible for implementation. In this work, we investigate LLM-based approaches for assessing the correctness of students’ explanations in response to undergraduate computer science questions. We investigate alternative prompting approaches for multiple LLMs (i.e., Llama 2, GPT-3.5, and GPT-4) and compare their performance to FLAN-T5 models trained in a fine-tuning manner. The results suggest that the highest accuracy and weighted F1 score were achieved by fine-tuning FLAN-T5, while an in-context learning approach with GPT-4 attains the highest macro F1 score.
Harnessing GPT to Study Second Language Learner Essays: Can We Use Perplexity to Determine Linguistic Competence?
Authors: Ricardo Muñoz Sánchez; Simon Dobnik; Elena Volodina
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: Generative language models have been used to study a wide variety of phenomena in NLP. This allows us to better understand the linguistic capabilities of those models and to better analyse the texts that we are working with. However, these studies have mainly focused on text generated by L1 speakers of English. In this paper we study whether linguistic competence of L2 learners of Swedish (through their performance on essay tasks) correlates with the perplexity of a decoder-only model (GPT-SW3). We run two sets of experiments, doing both quantitative and qualitative analyses for each of them. In the first one, we analyse the perplexities of the essays and compare them with the CEFR level of the essays, both from an essay-wide level and from a token level. In our second experiment, we compare the perplexity of an L2 learner essay with a normalised version of it. We find that the perplexity of essays tends to be lower for higher CEFR levels and that normalised essays have a lower perplexity than the original versions. Moreover, we find that different factors can lead to spikes in perplexity, not all of them being related to L2 learner language.
BERT-IRT: Accelerating Item Piloting with BERT Embeddings and Explainable IRT Models
Authors: Kevin P. Yancey; Andrew Runge; Geoffrey LaFlair; Phoebe Mulcaire
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: Estimating item parameters (e.g., the difficulty of a question) is an important part of modern high-stakes tests. Conventional methods require lengthy pilots to collect response data from a representative population of test-takers. The need for these pilots limit item bank size and how often those item banks can be refreshed, impacting test security, while increasing costs needed to support the test and taking up the test-taker’s valuable time. Our paper presents a novel explanatory item response theory (IRT) model, BERT-IRT, that has been used on the Duolingo English Test (DET), a high-stakes test of English, to reduce the length of pilots by a factor of 10. Our evaluation shows how the model uses BERT embeddings and engineered NLP features to accelerate item piloting without sacrificing criterion validity or reliability.
Transfer Learning of Argument Mining in Student Essays
Authors: Yuning Ding; Julian Lohmann; Nils-Jonathan Schaller; Thorben Jansen; Andrea Horbach
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper explores the transferability of a cross-prompt argument mining model trained on argumentative essays authored by native English-speaking learners (EN-L1) across educational contexts and languages. Specifically, the adaptability of a multilingual transformer model is assessed through its application to comparable argumentative essays authored by English-as-a-foreign-language learners (EN-L2) for context transfer, and a dataset composed of essays written by native German learners (DE) for both language and task transfer. To separate language effects from educational context effects, we also perform experiments on a machine-translated version of the German dataset (DE-MT). Our findings demonstrate that, even under zero-shot conditions, a model trained on native English speakers exhibits satisfactory performance on the EN-L2/DE datasets. Machine translation does not substantially enhance this performance, suggesting that distinct writing styles across educational contexts impact performance more than language differences.
Building Robust Content Scoring Models for Student Explanations of Social Justice Science Issues
Authors: Allison Bradford; Kenneth Steimel; Brian Riordan; Marcia Linn
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: With increased attention to connecting science topics to real-world contexts, like issues of social justice, teachers need support to assess student progress in explaining such issues. In this work, we explore the robustness of NLP-based automatic content scoring models that provide insight into student ability to integrate their science and social justice ideas in two different environmental science contexts. We leverage encoder-only transformer models to capture the degree to which students explain a science phenomenon, understand the intersecting justice issues, and integrate their understanding of science and social justice. We developed models training on data from each of the contexts as well as from a combined dataset. We found that the models developed in one context generate educationally useful scores in the other context. The model trained on the combined dataset performed as well as or better than the models trained on separate datasets in most cases. Quadratic weighted kappas demonstrate that these models are above threshold for use in classrooms.
From Miscue to Evidence of Difficulty: Analysis of Automatically Detected Miscues in Oral Reading for Feedback Potential
Authors: Beata Beigman Klebanov; Michael Suhan; Tenaha O’Reilly; Zuowei Wang
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This research is situated in the space between an existing NLP capability and its use(s) in an educational context. We analyze oral reading data collected with a deployed automated speech analysis software and consider how the results of automated speech analysis can be interpreted and used to inform the ideation and design of a new feature – feedback to learners and teachers. Our analysis shows how the details of the system’s performance and the details of the context of use both significantly impact the ideation process.
Findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple-Choice Questions
Authors: Victoria Yaneva; Kai North; Peter Baldwin; Le An Ha; Saed Rezayi; Yiyun Zhou; Sagnik Ray Choudhury; Polina Harik; Brian Clauser
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper reports findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple-Choice Questions. The task was organized as part of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA’24), held in conjunction with NAACL 2024, and called upon the research community to contribute solutions to the problem of modeling difficulty and response time for clinical multiple-choice questions (MCQs). A set of 667 previously used and now retired MCQs from the United States Medical Licensing Examination (USMLE®) and their corresponding difficulties and mean response times were made available for experimentation. A total of 17 teams submitted solutions and 12 teams submitted system report papers describing their approaches. This paper summarizes the findings from the shared task and analyzes the main approaches proposed by the participants.
Predicting Item Difficulty and Item Response Time with Scalar-mixed Transformer Encoder Models and Rational Network Regression Heads
Authors: Sebastian Gombert; Lukas Menzel; Daniele Di Mitri; Hendrik Drachsler
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper describes a contribution to the BEA 2024 Shared Task on Automated Prediction of Item Difficulty and Response Time. The participants in this shared task are to develop models for predicting the difficulty and response time of multiple-choice items in the medical field. These items were taken from the United States Medical Licensing Examination® (USMLE®), a high-stakes medical exam. For this purpose, we evaluated multiple BERT-like pre-trained transformer encoder models, which we combined with Scalar Mixing and two custom 2-layer classification heads using learnable Rational Activations as an activation function, each for predicting one of the two variables of interest in a multi-task setup. Our best models placed first out of 43 for predicting item difficulty and fifth out of 34 for predicting Item Response Time.
UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions
Authors: Ana-Cristina Rogoz; Radu Tudor Ionescu
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This work explores a novel data augmentation method based on Large Language Models (LLMs) for predicting item difficulty and response time of retired USMLE Multiple-Choice Questions (MCQs) in the BEA 2024 Shared Task. Our approach is based on augmenting the dataset with answers from zero-shot LLMs (Falcon, Meditron, Mistral) and employing transformer-based models based on six alternative feature combinations. The results suggest that predicting the difficulty of questions is more challenging. Notably, our top performing methods consistently include the question text, and benefit from the variability of LLM answers, highlighting the potential of LLMs for improving automated assessment in medical licensing exams. We make our code available at: https://github.com/ana-rogoz/BEA-2024.
The British Council submission to the BEA 2024 shared task
Authors: Mariano Felice; Zeynep Duran Karaoz
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper describes our submission to the item difficulty prediction track of the BEA 2024 shared task. Our submission included the output of three systems: 1) a feature-based linear regression model, 2) a RoBERTa-based model and 3) a linear regression ensemble built on the predictions of the two previous models. Our systems ranked 7th, 8th and 5th respectively, demonstrating that simple models can achieve optimal results. A closer look at the results shows that predictions are more accurate for items in the middle of the difficulty range, with no other obvious relationships between difficulty and the accuracy of predictions.
ITEC at BEA 2024 Shared Task: Predicting Difficulty and Response Time of Medical Exam Questions with Statistical, Machine Learning, and Language Models
Authors: Anaïs Tack; Siem Buseyne; Changsheng Chen; Robbe D’hondt; Michiel De Vrindt; Alireza Gharahighehi; Sameh Metwaly; Felipe Kenji Nakano; Ann-Sophie Noreillie
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper presents the results of our participation in the BEA 2024 shared task on the automated prediction of item difficulty and item response time (APIDIRT), hosted by the NBME (National Board of Medical Examiners). During this task, practice multiple-choice questions from the United States Medical Licensing Examination® (USMLE®) were shared, and research teams were tasked with devising systems capable of predicting the difficulty and average response time for new exam questions.Our team, part of the interdisciplinary itec research group, participated in the task. We extracted linguistic features and clinical embeddings from question items and tested various modeling techniques, including statistical regression, machine learning, language models, and ensemble methods. Surprisingly, simplermodels such as Lasso and random forest regression, utilizing principal component features from linguistic and clinical embeddings, outperformed more complex models. In the competition, our random forest model ranked 4th out of 43 submissions for difficulty prediction, while the Lasso model secured the 2nd position out of 34 submissions for response time prediction. Further analysis suggests that had we submitted the Lasso model for difficulty prediction, we would have achieved an even higher ranking. We also observed that predicting response time is easier than predicting difficulty, with features such as item length, type, exam step, and analytical thinking influencing response time prediction more significantly.
Item Difficulty and Response Time Prediction with Large Language Models: An Empirical Analysis of USMLE Items
Authors: Okan Bulut; Guher Gorgun; Bin Tan
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper summarizes our methodology and results for the BEA 2024 Shared Task. This competition focused on predicting item difficulty and response time for retired multiple-choice items from the United States Medical Licensing Examination® (USMLE®). We extracted linguistic features from the item stem and response options using multiple methods, including the BiomedBERT model, FastText embeddings, and Coh-Metrix. The extracted features were combined with additional features available in item metadata (e.g., item type) to predict item difficulty and average response time. The results showed that the BiomedBERT model was the most effective in predicting item difficulty, while the fine-tuned model based on FastText word embeddings was the best model for predicting response time.
Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction
Authors: Rishikesh Fulari; Jonathan Rusert
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: In this paper, we present the details of ourcontribution to the BEA Shared Task on Automated Prediction of Item Difficulty and Response Time. Participants in this collaborativeeffort are tasked with developing models to predict the difficulty and response time of multiplechoice items within the medical domain. Theseitems are sourced from the United States Medical Licensing Examination® (USMLE®), asignificant medical assessment. In order toachieve this, we experimented with two featurization techniques, one using lingusitic features and the other using embeddings generated by BERT fine-tuned over MS-MARCOdataset. Further, we tried several different machine learning models such as Linear Regression, Decision Trees, KNN and Boosting models such as XGBoost and GBDT. We found thatout of all the models we experimented withRandom Forest Regressor trained on Linguisticfeatures gave the least root mean squared error.
Leveraging Physical and Semantic Features of text item for Difficulty and Response Time Prediction of USMLE Questions
Authors: Gummuluri Venkata Ravi Ram; Ashinee Kesanam; Anand Kumar M
Venue: NAACL 2024 (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024))

Abstract: This paper presents our system developed for the Shared Task on Automated Prediction of Item Difficulty and Item Response Time for USMLE questions, organized by the Association for Computational Linguistics (ACL) Special Interest Group for building Educational Applications (BEA SIGEDU). The Shared Task, held as a workshop at the North American Chapter of the Association for Computational Linguistics (NAACL) 2024 conference, aimed to advance the state-of-the-art in predicting item characteristics directly from item text, with implications for the fairness and validity of standardized exams. We compared various methods ranging from BERT for regression to Random forest, Gradient Boosting(GB), Linear Regression, Support Vector Regressor (SVR), k-nearest neighbours (KNN) Regressor, MultiLayer Perceptron(MLP) to custom-ANN using BioBERT and Word2Vec embeddings and provided inferences on which performed better. This paper also explains the importance of data augmentation to balance the data in order to get better results. We also proposed five hypotheses regarding factors impacting difficulty and response time for a question and also verified it thereby helping researchers to derive meaningful numerical attributes for accurate prediction. We achieved a RSME score of 0.315 for Difficulty prediction and 26.945 for Response Time.