Reviewerly/exHarmony huggingface dataset:
Towards the Future of AI-Augmented Human Tutoring in Math Learning

One of the primary obstacles to improving middle school math achievement is lack of equitable access to high-quality learning opportunities. Human delivery of high-dosage tutoring can bring significant learning gains, but students, particularly economically disadvantaged students, have limited access to well-trained tutors. Augmenting human tutor abilities through the use of artificial intelligence (AI) technology is one way to scale up access to tutors without compromising learning quality. This workshop aims to highlight the challenges and opportunities of AI-in-the-loop math tutoring and encourage discourse in the AIED community to develop human-AI hybrid tutoring and teaching systems. We invite papers that provide clearer understanding and support the progress of human and AI-assisted personalized learning technologies. The structure of this full-day hybrid workshop will include presentations of accepted papers, small or whole group discussion, and a panel discussion focusing on common themes related to research and application, key takeaways, and findings imperative to increasing middle school math learning.
Evaluating the Rater Bias in Response Scoring in Digital Learning Platform: Analysis of Student Writing Styles

Automated essay scoring and short-answer scoring have shown tremendous potential for enhancing and promoting large-scale assessments. Challenges still remain in the equity and the implicit bias in scoring that is ingrained in the scoring system. One promising solution to mitigate the problem is the introduction of a measurement model that quantifies and evaluates the degree to which the raters are showing biases in students’ written responses. This paper presents an adoption of a generalized many-facet Rasch model (GMFRM, [8]) to evaluate the rater biases regarding students’ writing styles. We modeled students’ writing styles using a rich set of computational linguistic indices from LingFeat [7] that are empirically and theoretically associated with the text difficulty. The findings showed that the rater bias exists in scoring responses that are explicitly covering a variety of topics with less elaborative descriptions compared to the other writing styles captured in students’ responses. The rater severity was the only type of bias characteristic that was statistically significant. We discussed the implications and future applications of our findings to advance automated essay scoring and teacher professional development.
Automated Scoring of Logical Consistency of Japanese Essays

Automated essay scoring (AES) is to estimate the scores of essays automatically. Two types of AES models are commonly used: handcrafted feature-based and neural-based models. In this paper, we introduce AES systems based on the two types for evaluating the logical consistency of Japanese essays. In addition, to enhance the performance of models, we integrate the neural-based model with the handcrafted features: a hybrid AES system. In the experiment, we show the effectiveness of our hybrid AES system. Besides, most of our AES models obtained higher QWK scores than human evaluators.
Tutorial: Educational Recommender Systems

Recommender systems (RecSys) have found widespread use in a variety of applications, including e-commerce platforms like Amazon.com and eBay, online streaming services such as YouTube, Netflix, and Spotify, and social media sites like Facebook and Twitter. The success of these applications in improving user experience and decision making by providing personalized recommendations highlights the effectiveness of RecSys. Over the past few decades, RecSys has also made its way into the field of education, which results in the development of educational recommender systems (EdRec). Its applications in this field include personalized learning experiences, recommending appropriate formal or informal learning materials, suggesting learning peers, and adapting learning to context-aware or mobile environments, and so forth. Recently, the development of RecSys has been advanced by a series of interesting and promising topics, such as multi-task learning, multi-objective optimization, multi-stakeholder considerations, concerns of fairness, accountability, and transparency, etc. However, this progress made in the field of recommender systems was not adequately disseminated to the education community or the development of EdRec. In this tutorial, we will introduce the background, motivations, knowledge & skills associated with the current development of EdRec, and discuss a set of emerging topics and open challenges, along with case studies in EdRec.
A Support System to Help Teachers Design Course Plans Conforming to National Curriculum Guidelines

This study addresses the challenge of ensuring the alignment of educational materials with the objectives of public education guidelines outlined in official documents. In Japan, “Courses of Study (COS),” issued by the government, provides a detailed outline for curricula in public and private education. Teachers ought to design lessons based on this outline. This can be a time-consuming and challenging task, particularly. Accordingly, this paper proposes a new support system for designing curricula that performs an automatic search of parts of COS corresponding to teaching materials. Computational retrieval via advanced deep transfer learning methods such as BERT and DeBERTa is expected to be effective and time-saving in the implementation of COS. Using a novel dataset linking textbook text with the COS, the proposed method annotates relevant sections of the COS. Experimental results showed that the proposed method could present the corresponding sections of the COS with a statistically significantly higher mean average precision (mAP) in both unsupervised and supervised learning compared to methods that do not consider semantic factors, such as edit distance.
Building Educational Technology Quickly and Robustly with an Interactively Teachable AI

Intelligent tutoring systems (ITSs) aim to support student learning through comprehensive adaptive features, making for costly development times—about 200–300 hours of development time per hour of instruction. This proposal outlines plans to overcome several technical challenges toward building authoring tools whereby a non-programmer can build ITSs by interactively teaching simulated students. I propose both interaction design considerations and machine-learning innovations. These include a multi-modal natural language processing mechanism that mimics student learning from narrated tutorial instruction, and an active-learning mechanism that identifies training examples likely to eliminate inaccuracies in the simulated student’s induced production rules. I propose to evaluate these features over 3 user studies and evaluate the generality of this authoring method in a final open-ended authoring study. This work aims to democratize ITS authoring by opening new authoring opportunities to non-programmers by making authoring as time-efficient and natural as human-to-human tutoring.
A Unified Batch Hierarchical Reinforcement Learning Framework for Pedagogical Policy Induction with Deep Bisimulation Metrics

Intelligent Tutoring Systems (ITSs) leverage AI to adapt to individual students, and employ pedagogical policies to decide what instructional action to take next. A number of researchers applied Reinforcement Learning (RL) and Deep RL (DRL) to induce effective pedagogical policies. Most prior work, however, has been developed independently for a specific ITS and cannot directly be applied to another. In this work, we propose a Multi-Task Learning framework that combines Deep BIsimulation Metrics and DRL, named MTL-BIM, to induce a unified pedagogical policy for two different ITSs across different domains: logic and probability. Based on empirical classroom results, our unified RL policy performed significantly better than the expert-crafted policies and independently induced DQN policies on both ITSs.
A Quantitative Study of NLP Approaches to Question Difficulty Estimation

Question Difficulty Estimation from Text (QDET) received an increased research interest in recent years, but most of previous work focused on single silos, without performing quantitative comparisons between different models or across datastes from different educational domains. To fill this gap, we quantitatively analyze several approaches proposed in previous research, and compare their performance on two publicly available datasets. Specifically, we consider reading comprehension Multiple Choice Questions (MCQs) and maths questions. We find that Transformer-based models are the best performing in both educational domains; models based on linguistic features perform well on reading comprehension questions, while frequency based features and word embeddings perform better in domain knowledge assessment.
Who and How: Using Sentence-Level NLP to Evaluate Idea Completeness

Real-time feedback is very important, yet challenging to provide for free-text learner contributions in Technology-Enhanced Learning. We study whether a generic NLP pipeline can identify completeness features of learner ideas during security training. We apply PoS Tagging and Dependency Parsing on contextualised short texts, collected within a dedicated learning environment and we compare the results to an expert-annotated ground truth. We scan these contributions for the absence of responsible stakeholder (who) or featured action (how). A total of 1174 contributions in two security domains were analysed. We report precision on who ( $$PPV=0.929$$ ) and on how ( $$PPV=0.691$$ ). We consider the first result to be sufficient to provide real-time formative feedback for the case of absent who. Our results suggest that for the purposes of providing feedback in free input problem-solving exercises, generic transformer pipelines without fine-tuning can achieve good performance on stakeholder identification.
Exploring the Effect of Autoencoder Based Feature Learning for a Deep Reinforcement Learning Policy for Providing Proactive Help

Providing timely assistance to students in intelligent tutoring systems is a challenging research problem. In this study, we aim to address this problem by determining when to provide proactive help with autoencoder based feature learning and a deep reinforcement learning (DRL) model. To increase generalizability, we only use domain-independent features for the policy. The proposed pedagogical policy provides next-step proactive hints based on the prediction of the DRL model. We conduct a study to examine the effectiveness of the new policy in an intelligent logic tutor. Our findings provide insight into the use of DRL policies utilizing autoencoder based feature learning to determine when to provide proactive help to students.
Designing, Building and Evaluating Intelligent Psychomotor AIED Systems (IPAIEDS@AIED2023)

Psychomotor learning is an emerging research direction in the AIED (Artificial Intelligence in Education) field. This topic was introduced in the AIED research agenda back in 2016 in a contribution at the International Journal of AIED, where the SMDD (Sensing-Modelling-Designing-Delivering) process model to develop AIED psychomotor systems was introduced. Recently, a systematic review of the state of the art on this topic has also been published in the novel Handbook of AIED. In this context, the aim of the IPAIEDS tutorial is to motivate the AIED community to research on intelligent psychomotor systems and give tools to design, build and evaluate this kind of systems.
Learning from AI: An Interactive Learning Method Using a DNN Model Incorporating Expert Knowledge as a Teacher

Visual explanation is an approach for visualizing the grounds of judgment by deep learning, and it is possible to visually interpret the grounds of a judgment for a certain input by visualizing an attention map. As for deep-learning models that output erroneous decision-making grounds, a method that incorporates expert human knowledge in the model via an attention map in a manner that improves explanatory power and recognition accuracy is proposed. In this study, based on a deep-learning model that incorporates the knowledge of experts, a method by which a learner “learns from AI” the grounds for its decisions is proposed. An “attention branch network” (ABN), which has been fine-tuned with attention maps modified by experts, is prepared as a teacher. By using an interactive editing tool for the fine-tuned ABN and attention maps, the learner learns by editing the attention maps and changing the inference results. By repeatedly editing the attention maps and making inferences so that the correct recognition results are output, the learner can acquire the grounds for the expert’s judgments embedded in the ABN. The results of an evaluation experiment with subjects show that learning using the proposed method is more efficient than the conventional method.
Improving the Item Selection Process with Reinforcement Learning in Computerized Adaptive Testing

Item selection is the key process for computerized adaptive testing (CAT) to effectively assess examinees’ knowledge states. Existing item selection algorithms mainly rely on information metrics, suffering two issues: one is that the implicit cognitive information like relations between testing items as well as knowledge components cannot be captured by the information-based methods, and the other one is that the information-based algorithms computes item’s suitableness depending on examinees’ knowledge states which are estimated and imprecise inherently. To address these two issues, this work proposes to employ reinforcement learning technology to learn the item selection algorithm automatically in a data-driven manner. It is also able to properly capture the implicit cognitive relations between different testing items and avoid unnecessary item testing, and does not depend on examinees’ estimated knowledge states at all.
The Role of Social Presence in MOOC Students’ Behavioral Intentions and Sentiments Toward the Usage of a Learning Assistant Chatbot: A Diversity, Equity, and Inclusion Perspective Examination

Improving social presence has been popularly researched for massive open online courses (MOOCs). Despite this, little consideration has been given to how students intend to utilize new technology from a diversity, equity, and inclusion (DEI) standpoint. In this study, we examined the role of social presence in diverse MOOC students’ behavioral intentions and sentiments toward using a learning assistant chatbot. Considering potentially disparate perceptions of the technology depending on the students’ demographic factors (age, gender, region, and native language), we investigated the relationships among their social presence, age, behavioral intentions (before and after use), sentiment levels, and numbers of turn-takings with the chatbot. The investigation showed positive and moderate correlations between social presence and post-behavioral intention and between social presence and pre-behavioral intention. Also, the sentiment and social presence toward the chatbot turned out to be different according to the students’ region factor based on 13 different geographical categorizations. The native language factor was also influential in creating different sentiment levels between native and non-native English user groups. The findings implicate the importance of promoting a better understanding of the role of social presence in students’ learning experiences with AI-based applications from a DEI perspective.
Assessment in Conversational Intelligent Tutoring Systems: Are Contextual Embeddings Really Better?

This research investigates the ability of semantic text models to assess student responses during tutoring compared with expert human judges. Recent interest in text similarity has led to a proliferation of models that can potentially be used for assessing student responses; however, whether these models perform as well as traditional distributional semantic models like Latent Semantic Analysis for student response assessment in automatic short answer grading is unclear. We assessed 5166 response pairings of 219 participants across 118 electronics questions and scored each with 13 different computational text models, including models that use regular expressions, distributional semantics, word embeddings, contextual embeddings, and combinations of these features. We show a few semantic text models performing comparably to Latent Semantic Analysis, and in some cases outperforming the model. Furthermore, combination models outperformed individual models in agreement with human judges. Choosing appropriate computational techniques and optimizing the text model may continue to improve the accuracy, recall, weighted agreement and therefore, the effectiveness of conversational ITSs.
Navya3DSeg - Navya 3D Semantic Segmentation Dataset Design &amp; Split Generation for Autonomous Vehicles

Autonomous driving (AD) perception today relies heavily on deep learning based architectures requiring large scale annotated datasets with their associated costs for curation and annotation. The 3D semantic data are useful for core perception tasks such as obstacle detection and ego-vehicle localization. We propose a new dataset, Navya 3D Segmentation (Navya3DSeg), with a diverse label space corresponding to a large scale production grade operational domain, including rural, urban, industrial sites and universities from 13 countries. It contains 23 labeled sequences and 25 supplementary sequences without labels, designed to explore self-supervised and semi-supervised semantic segmentation benchmarks on point clouds. We also propose a novel method for sequential dataset split generation based on iterative multi-label stratification, and demonstrated to achieve a +1.2% <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mIoU</i> improvement over the original split proposed by SemanticKITTI dataset. A complete benchmark for semantic segmentation task was performed, with state of the art methods. Finally, we demonstrate an Active Learning (AL) based dataset distillation framework. We introduce a novel heuristic-free sampling method called ego-pose distance based sampling in the context of AL.
SUGAR: Efficient Subgraph-Level Training via Resource-Aware Graph Partitioning

Graph Neural Networks (GNNs) have demonstrated a great potential in a variety of graph-based applications, such as recommender systems, drug discovery, and object recognition. Nevertheless, resource-efficient GNN learning is a rarely explored topic despite its many benefits for edge computing and Internet of Things (IoT) applications. To improve this state of affairs, this work proposes efficient <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">su</u> b <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">g</u> raph-level tr <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</u> ining via <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</u> esource-aware graph partitioning (SUGAR). SUGAR first partitions the initial graph into a set of disjoint subgraphs and then performs local training at the subgraph-level We provide a theoretical analysis and conduct extensive experiments on five graph benchmarks to verify its efficacy in practice. Our results across five different hardware platforms demonstrate great runtime speedup and memory reduction of SUGAR on large-scale graphs. We believe SUGAR opens a new research direction towards developing GNN methods that are resource-efficient, hence suitable for IoT deployment.
STTracker: Spatio-Temporal Tracker for 3D Single Object Tracking

3D single object tracking with point clouds is a critical task in 3D computer vision. Previous methods usually input the last two frames and use the predicted box to get the template point cloud in previous frame and the search area point cloud in the current frame respectively, then use similarity-based or motion-based methods to predict the current box. Although these methods achieved good tracking performance, they ignore the historical information of the target, which is important for tracking. In this letter, compared to inputting two frames of point clouds, we input multi-frame of point clouds to encode the spatio-temporal information of the target and learn the motion information of the target implicitly, which could build the correlations among different frames to track the target in the current frame efficiently. Meanwhile, rather than directly using the point feature for feature fusion, we first crop the point cloud features into many patches and then use sparse attention mechanism to encode the patch-level similarity and finally fuse the multi-frame features. Extensive experiments show that our method achieves competitive results on challenging large-scale benchmarks (62.6% in KITTI and 49.66% in NuScenes). The code will be open soon.
Resilient Countermeasures Against Cyber-Attacks on Self-Driving Car Architecture

Self-driving cars, believed to be on track to become a seven trillion-dollar industry by 2050, are likely to be a popular mode of transportation in the near future, as the technology promises safety and efficiency. Significant advances have been made to achieve these objectives. However, the idea of a self-driving car completing its goals without human intervention in the presence of uncertainties is still posed with challenges. These challenges mostly relate to decision-making under uncertainties that could arise from transient faults or deliberate attempts to disrupt the system in favor of adversarial motives. Traditional schemes of hardening the system against known compromises will not suffice primarily because of the sophistication adversarial attempts have come to achieve and the rate at which new attacks are being devised. The problem demands a solution that is resilient to attacks and that would allow a self-driving vehicle to complete its designated task in the presence of attacks or faults. Several attempts have been made towards achieving resilience in autonomous vehicles. To the best of our knowledge, this is the first work to give a thorough review on the resilient approaches adopted in the context and presents a taxonomy of such approaches. The paper also includes a brief introduction to the architecture of self-driving vehicles and its vulnerabilities in the presented context. This paper also discusses a novel approach using N-version programming for resilience.
Private Transaction Retrieval for Lightweight Bitcoin Clients

Running a typical Bitcoin client (also called full node) needs more than 444 GB of disk space, considerable time, and computational resources to synchronize the entire blockchain, which is infeasible for resource-constrained devices. To address such concerns, the lightweight Bitcoin client proposed by Satoshi outsources most of computational and storage burdens to full nodes. Unfortunately, interacting with full nodes to query transactions leaks considerable information like addresses and transactions of lightweight client users. Thus, Bitcoin users that rely on lightweight clients are subject to de-anonymization, which defeats users privacy. Traditional schemes cannot support lightweight clients to query transactions from full nodes in an efficient yet privacy-preserving way. In this article, we propose a new efficient yet privacy-preserving transaction query scheme that specially targets the missing support for lightweight clients. We identify unique characteristics of the Bitcoin blockchain and craft a highly customized private information retrieval scheme called BIT-PIR to match the Bitcoin transaction query scenario and boost performances. Moreover, we customize a storage structure of the Bitcoin blockchain so that it further improves the query efficiency of our scheme. Finally, we develop a prototype implementation to demonstrate the feasibility of our proposed scheme.