ethanporcaro/eric-fulltext-meta huggingface dataset:
Knowledge Tracing over Time: A Longitudinal Analysis
Author: Lee, Morgan P.
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Bayesian Statistics,Models,Generalizability Theory,Longitudinal Studies,Mastery Learning,Learning Analytics,Prediction,Systems Approach,Reliability,Markov Processes

The use of Bayesian Knowledge Tracing (BKT) models in predicting student learning and mastery, especially in mathematics, is a well-established and proven approach in learning analytics. In this work, we report on our analysis examining the generalizability of BKT models across academic years attributed to &quot;detector rot.&quot; We compare the generalizability of Knowledge Training (KT) models by comparing model performance in predicting student knowledge within the academic year and across academic years. Models were trained on data from two popular open-source curricula available through Open Educational Resources. We observed that the models generally were highly performant in predicting student learning within an academic year, whereas certain academic years were more generalizable than other academic years. We posit that the Knowledge Tracing models are relatively stable in terms of performance across academic years yet can still be susceptible to systemic changes and underlying learner behavior. As indicated by the evidence in this paper, we posit that learning platforms leveraging KT models need to be mindful of systemic changes or drastic changes in certain user demographics. [For the complete proceedings, see ED630829. Additional funding was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN) program.]
Analysis of an Explainable Student Performance Prediction Model in an Introductory Programming Course
Author: Hoq, Muntasir
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Academic Achievement,Prediction,Models,Introductory Courses,Programming,Artificial Intelligence,Grades (Scholastic)

Prediction of student performance in introductory programming courses can assist struggling students and improve their persistence. On the other hand, it is important for the prediction to be transparent for the instructor and students to effectively utilize the results of this prediction. Explainable Machine Learning models can effectively help students and instructors gain insights into students' different programming behaviors and problem-solving strategies that can lead to good or poor performance. This study develops an explainable model that predicts students' performance based on programming assignment submission information. We extract different data-driven features from students' programming submissions and employ a stacked ensemble model to predict students' final exam grades. We use SHAP, a game-theory-based framework, to explain the model's predictions to help the stakeholders understand the impact of different programming behaviors on students' success. Moreover, we analyze the impact of important features and utilize a combination of descriptive statistics and mixture models to identify different profiles of students based on their problem-solving patterns to bolster explainability. The experimental results suggest that our model significantly outperforms other Machine Learning models, including KNN, SVM, XGBoost, Bagging, Boosting, and Linear regression. Our explainable and transparent model can help explain students' common problem-solving patterns in relationship with their level of expertise resulting in effective intervention and adaptive support to students. [For the complete proceedings, see ED630829.]
Generalizing Predictive Models of Reading Ability in Adaptive Mathematics Software
Author: Almoubayyed, Husni
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Prediction,Models,Reading Ability,Computer Software,Mathematics Instruction,Intelligent Tutoring Systems,Middle School Students,Language Arts,Academic Achievement,Standardized Tests,Grade 7,Reading Comprehension

Recent research seeks to develop more comprehensive learner models for adaptive learning software. For example, models of reading comprehension built using data from students' use of adaptive instructional software for mathematics have recently been developed. These models aim to deliver experiences that consider factors related to learning beyond performance in the target domain for instruction. We investigate the extent to which generalization is possible for a recently developed predictive model that seeks to infer students' reading comprehension ability (as measured by end-of-year standardized test scores) using an introductory learning experience in Carnegie Learning's MATHia intelligent tutoring system for mathematics. Building on a model learned on data from middle school students in a single school district in a mid-western U.S. state, using that state's end-of-year English Language Arts (ELA) standardized test score as an outcome, we consider data from a school district in a south-eastern U.S. state as well as that state's end-of-year ELA standardized test outcome. Generalization is explored by considering prediction performance when training and testing models on data from each of the individual school districts (and for their respective state's test outcomes) as well as pooling data from both districts together. We conclude with discussion of investigations of some algorithmic fairness characteristics of the learned models. The results suggest that a model trained on data from the smaller of the two school districts considered may achieve greater fairness in its predictions over models trained on data from the other district or both districts, despite broad, overall similarities in some demographic characteristics of the two school districts. This raises interesting questions for future research on generalizing these kinds of models as well as on ensuring algorithmic fairness of resulting models for use in real-world adaptive systems for learning. [For the complete proceedings, see ED630829.]
Special Education Teacher Workforce. Technical Working Group Summary
Author: Taylor, Katherine
Date: 2023-02-10
Type: Reports - Descriptive
Subjects: Special Education Teachers,Federal Aid,Elementary Secondary Education,Teacher Supply and Demand,Teacher Role,Teacher Responsibility,Interprofessional Relationship,Teacher Administrator Relationship,Educational Research,Teacher Education,Teacher Certification,Data Analysis

In February 2023, the National Center for Special Education Research (NCSER) in the Institute of Education Sciences (IES) convened a technical working group (TWG) panel of experts on the K-12 special education teacher workforce, including state- and district-level education leaders, teacher educators, researchers, and technical assistance providers. The goal of the TWG was to identify ways research can be used to better prepare, support, and retain an effective K-12 special education teacher workforce and how IES can best support this research. The discussion was centered around pressing problems, research needs, and data on the K-12 special education teacher workforce.
In Search of Negative Moments: Multi-Modal Analysis of Teacher Negativity in Classroom Observation Videos
Author: Dai, Zilin
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Teacher Behavior,Negative Attitudes,Nonverbal Communication,Teacher Student Relationship,Interaction,Video Technology,Emotional Response,Speech Communication,Classroom Environment,Classroom Observation Techniques,Artificial Intelligence,Elementary Schools,Middle Schools

We explore multi-modal machine learning-based approaches (facial expression recognition, auditory emotion recognition, and text sentiment analysis) to identify &quot;negative moments&quot; of teacher-student interaction during classroom teaching. Our analyses on a large (957 videos, each 20min) dataset of classroom observations suggest that: (1) Negative moments occur sparsely and are laborious to find by manually watching videos from start to finish; (2) Contemporary machine perception tools for emotion, speech, and text sentiment analysis show only limited ability to capture the diverse manifestations of classroom negativity in a fully automatic way; (3) Semi-automatic procedures that combine machine perception with human annotation may hold more promise for finding authentic moments of classroom negativity; Finally, (4) even short 10 sec negative moments contain rich structure in terms of the actions and behaviors that they comprise. [For the complete proceedings, see ED630829.]
Investigating the Importance of Demographic Features for EDM-Predictions
Author: Cohausz, Lea
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Information Retrieval,Data Processing,Pattern Recognition,Information Technology,At Risk Students,Prediction,Accuracy,Foreign Countries,Student Characteristics,College Students,High School Students,Academic Achievement

Demographic features are commonly used in Educational Data Mining (EDM) research to predict at-risk students. Yet, the practice of using demographic features has to be considered extremely problematic due to the data's sensitive nature, but also because (historic and representation) biases likely exist in the training data, which leads to strong fairness concerns. At the same time and despite the frequent use, the value of demographic features for prediction accuracy remains unclear. In this paper, we systematically investigate the importance of demographic features for at-risk prediction using several publicly available datasets from different countries. We find strong evidence that including demographic features does not lead to better-performing models as long as some study-related features exist, such as performance or activity data. Additionally, we show that models, nonetheless, place importance on these features when they are included in the data--although this is not necessary for accuracy. These findings, together with our discussion, strongly suggest that at-risk prediction should not include demographic features. Our code is available at: https://anonymous.4open.science/r/edm-F7D1. [For the complete proceedings, see ED630829.]
A Data Mining Approach for Detecting Collusion in Unproctored Online Exams
Author: Langerbein, Janine
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Information Retrieval,Pattern Recognition,Data Analysis,Information Technology,Computer Assisted Testing,Supervision,COVID-19,Pandemics,Electronic Learning,Foreign Countries,College Students

Due to the precautionary measures during the COVID-19 pandemic many universities offered unproctored take-home exams. We propose methods to detect potential collusion between students and apply our approach on event log data from take-home exams during the pandemic. We find groups of students with suspiciously similar exams. In addition, we compare our findings to a proctored comparison group. By this, we establish a rule of thumb for evaluating which cases are &quot;outstandingly similar&quot;, i.e., suspicious cases. [For the complete proceedings, see ED630829.]
An Analysis of Diffusion of Teacher-Curated Resources on Pinterest
Author: Karimi, Hamid
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Preschool Teachers,Early Childhood Teachers,Elementary School Teachers,Secondary School Teachers,Educational Resources,Social Media,Selection,Sharing Behavior,Information Dissemination,Shared Resources and Services

Teachers increasingly rely on online social media platforms to supplement their educational resources, greatly influencing PK-12 education through the swift and extensive diffusion of teacher-curated resources. Understanding this diffusion process is crucial, but current educational studies primarily report resource diffusion through small-scale analyses, such as teacher interviews or anecdotal accounts. To bridge this gap, we conduct a pioneering, large-scale, quantitative, and data-driven analysis of the diffusion of teacher-curated resources on Pinterest, a platform widely embraced by educators. Our study begins by defining a resource's diffusion tree, which encapsulates the cascade of resource sharing across the social network. Based on this diffusion tree, we identify three measures to characterize a resource's diffusion process: volume, virality, and velocity. Equipped with these three measures, we conduct an in-depth analysis of the diffusion of over one million resources curated by thousands of teachers on Pinterest. Our investigation concludes by examining the correlation between a resource's attributes and its curator's attributes and the diffusion of the resource. [For the complete proceedings, see ED630829.]
Evaluating Quadratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring
Author: Doewes, Afrizal
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Essays,Writing Evaluation,Evaluators,Accuracy,Models,Computer Assisted Testing,Rating Scales,High Stakes Tests,Interrater Reliability,Computer Software,Artificial Intelligence

Automated Essay Scoring (AES) tools aim to improve the efficiency and consistency of essay scoring by using machine learning algorithms. In the existing research work on this topic, most researchers agree that human-automated score agreement remains the benchmark for assessing the accuracy of machine-generated scores. To measure the performance of AES models, the Quadratic Weighted Kappa (QWK) is commonly used as the evaluation metric. However, we have identified several limitations of using QWK as the sole metric for evaluating AES model performance. These limitations include its sensitivity to the rating scale, the potential for the so-called &quot;kappa paradox&quot; to occur, the impact of prevalence, the impact of the position of agreements in the diagonal agreement matrix, and its limitation in handling a large number of raters. Our findings suggest that relying solely on QWK as the evaluation metric for AES performance may not be sufficient. We further discuss insights into additional metrics to comprehensively evaluate the performance and accuracy of AES models. [For the complete proceedings, see ED630829.]
Generating High-Precision Feedback for Programming Syntax Errors Using Large Language Models
Author: Phung, Tung
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Computational Linguistics,Feedback (Response),Programming,Computer Science Education,Programming Languages,Error Correction,Introductory Courses,Teaching Methods,Natural Language Processing,Tutors,Intelligent Tutoring Systems,Quality Control,Computer Software,Validity

Large language models (LLMs), such as Codex, hold great promise in enhancing programming education by automatically generating feedback for students. We investigate using LLMs to generate feedback for fixing syntax errors in Python programs, a key scenario in introductory programming. More concretely, given a student's buggy program, our goal is to generate feedback comprising a fixed program along with a natural language explanation describing the errors/fixes, inspired by how a human tutor would give feedback. While using LLMs is promising, the critical challenge is to ensure high precision in the generated feedback, which is imperative before deploying such technology in classrooms. The main research question we study is: &quot;Can we develop LLMs-based feedback generation techniques with a tunable precision parameter, giving educators quality control over the feedback that students receive?&quot; To this end, we introduce PyFiXV, our technique to generate high-precision feedback powered by Codex. The key idea behind PyFiXV is to use a novel run-time validation mechanism to decide whether the generated feedback is suitable for sharing with the student; notably, this validation mechanism also provides a precision knob to educators. We perform an extensive evaluation using two real-world datasets of Python programs with syntax errors and show the efficacy of PyFiXV in generating high-precision feedback. [For the complete proceedings, see ED630829.]
How to Open Science: Debugging Reproducibility within the Educational Data Mining Conference
Author: Haim, Aaron
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Conferences (Gatherings),Educational Research,Replication (Evaluation),Access to Information,Information Retrieval,Data Processing,Data Analysis

Despite increased efforts to assess the adoption rates of open science and robustness of reproducibility in sub-disciplines of education technology, there is a lack of understanding of why some research is not reproducible. Prior work has taken the first step toward assessing reproducibility of research, but has assumed certain constraints which hinder its discovery. Thus, the purpose of this study was to replicate previous work on papers within the proceedings of the &quot;International Conference on Educational Data Mining&quot; to accurately report on which papers are reproducible and why. Specifically, we examined 208 papers, attempted to reproduce them, documented reasons for reproducibility failures, and asked authors to provide additional information needed to reproduce their study. Our results showed that out of 12 papers that were potentially reproducible, only one successfully reproduced all analyses, and another two reproduced most of the analyses. The most common failure for reproducibility was failure to mention libraries needed, followed by non-seeded randomness. [For the complete proceedings, see ED630829. Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN).]
Practitioner Perspectives on Emerging Research Needs and Improving Relevance in Education Research. Technical Working Group Summary
Author: National Center for Education Research (NCER) (ED/IES)
Date: 2014-03-03
Type: Reports - Descriptive
Subjects: Researchers,Research Needs,Advisory Committees,State Departments of Education,Government Employees,School Districts,Administrator Attitudes,Teacher Attitudes,College Faculty,Specialists,Educational Research,Information Dissemination,Elementary School Students,Middle School Students,High School Students,Career Readiness,Cooperation,Evidence Based Practice,Information Sources,Special Education,Early Intervention,Early Childhood Education,College Readiness

In March 2014, The National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER), in the Institute of Education Sciences (IES) at the U.S. Department of Education, convened a Technical Working Group (TWG) of practitioners to provide input on research needs in education and how research could be more relevant and useful to the field. Fifteen expert practitioners in the field of education were invited to attend, including individuals working in state education and legislative offices, districts, and schools as well as non-profit organizations and postsecondary institutions. The TWG discussion focused on five broad sessions: (1) making research available through effective dissemination; (2) research needs for elementary, middle, and high school; (3) research needs for early intervention; (4) research needs for post-secondary/career readiness; and (5) researcher-practitioner collaborations. This meeting summary covers all sessions of the technical working group and includes both presentations and discussions among participants.
Learning Problem Decomposition-Recomposition with Data-Driven Chunky Parsons Problems within an Intelligent Logic Tutor
Author: Shabrina, Preya
Date: 2023-00-00
Type: Speeches/Meeting Papers
Subjects: Intelligent Tutoring Systems,Problem Solving,Graphs,Data Analysis,Validity,Mathematical Logic,Thinking Skills,Computer Science Education,Teaching Methods,Comparative Analysis,Difficulty Level,Correlation

Problem decomposition into sub-problems or subgoals and recomposition of the solutions to the subgoals into one complete solution is a common strategy to reduce difficulties in structured problem solving. In this study, we use a datadriven graph-mining-based method to decompose historical student solutions of logic-proof problems into Chunks. We design a new problem type where we present these chunks in a Parsons Problem fashion and asked students to reconstruct the complete solution from the chunks. We incorporated these problems within an intelligent logic tutor and called them Chunky Parsons Problems (CPP). These problems demonstrate the process of problem decomposition to students and require them to pay attention to the decomposed solution while they reconstruct the complete solution. The aim of introducing CPP was to improve students' problem-solving skills and performance by improving their decomposition-recomposition skills without significantly increasing training difficulty. Our analysis showed that CPPs could be as easy as Worked Examples (WE). And, students who received CPP with simple explanations attached to the chunks had marginally higher scores than those who received CPPs without explanation or did not receive them. Also, the normalized learning gain of these students shifted more towards the positive side than other students. Finally, as we looked into their proof-construction traces in posttest problems, we observed them to form identifiable chunks aligned with those found in historical solutions with higher efficiency. [For the complete proceedings, see ED630829.]
Researcher Perspectives on Strengthening IES&apos;s Research Grant and Training Programs. Technical Working Group Summary
Author: National Center for Special Education Research (NCSER) (ED/IES)
Date: 2014-10-16
Type: Reports - Descriptive
Subjects: Educational Researchers,Grants,Research Training,Research Needs,Educational Research,Specialists,Meetings,Educational Policy,Policy Formation,Resource Allocation,Advisory Committees,Financial Support,Educational Practices,Public Agencies,Special Education,Agency Role,Research Methodology,Program Improvement,Problems,Preschool Education,Elementary Secondary Education,Teacher Education,Leadership Role,Instructional Leadership,Student Characteristics

In October 2014, The National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER), in the Institute of Education Sciences (IES) at the U.S. Department of Education, convened a Technical Working Group (TWG). The purpose of this TWG meeting was to discuss critical education problems and issues on which high-quality research is needed, the development or adaptation of new methodological approaches for addressing these issues, the role of IES research training programs in preparing of the next generation of researchers, and ways to prioritize the use of limited funds to support credible research that impacts policy and practice. The TWG consisted of 17 experts in the field of education research who work in universities and research organizations. The TWG discussion focused on five broad sessions: (1) characteristics of influential research; (2) critical, unaddressed issues in education research; (3) advancing research methods; (4) improving IES's training programs; and (5) targeting resources. This meeting summary covers all sessions of the technical working group and includes both presentations and discussions among participants.