SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zitao Liu
Guangdong Institute of Smart
Education, Jinan University
liuzitao@jnu.edu.cn
Qiongqiong Liu
TAL Education Group
liuqiongqiong1@tal.com
Teng Guo*
teng.guo@outlook.com
Jiahao Chen
chenjiahao@tal.com
Shuyan Huang
huangshuyan@tal.com
Xiangyu Zhao
Jiliang Tang
and Engineering, Michigan
State University MI, USA
tangjili@msu.edu
Weiqi Luo
lwq@jnu.edu.cn
Jian Weng
College of Cyber Security
cryptjweng@gmail.com
Knowledge tracing (KT) is a task that predicts students' future performance based on their historical learning interactions. With the rapid development of deep learning techniques, existing KT approaches follow a data-driven paradigm that uses massive problem-solving records to model students' learning processes. However, although the educational contexts contain various factors that may have an influence on student learning outcomes, existing public KT datasets mainly consist of anonymized ID-like features, which may hinder the research advances towards this field. Therefore, in this work, we present, XES3G5M, a large-scale dataset with rich auxiliary information about questions and their associated knowledge components  $(\mathrm{KCs})^2$ . The XES3G5M dataset is collected from a real-world online math learning platform, which contains 7,652 questions, and 865 KCs with 5,549,635 interactions from 18,066 students. To the best of our knowledge, the XES3G5M dataset not only has the largest number of KCs in math domain but contains the richest contextual information including tree structured KC relations, question types, textual contents and analysis and student response timestamps. Furthermore, we build a comprehensive benchmark on 19 state-of-the-art deep learning based knowledge tracing (DLKT) models. Extensive experiments demonstrate the effectiveness of leveraging the auxiliary information in our XES3G5M with DLKT models. We hope the proposed dataset can effectively facilitate the KT research work.
*The corresponding authors: Teng Guo, Qiongqiong Liu.
${}^{2}$  A KC is a generalization of everyday terms like concept,principle, fact, or skill.
Knowledge tracing (KT) is the task of using students' historical learning interaction data to model their knowledge mastery over time so as to make predictions on their future interaction performance (shown in Figure 1). Solving the KT problems may help teachers better detect students that need further attention, or recommend personalized learning materials to students which is essential for building next-generation smart and personalized education. Recently, many KT studies propose deep learning based KT (DLKT) models following the data-driven paradigm that uses lots of students' historical interaction sequences to estimate their knowledge states which have achieved promising results [25, 36, 22, 7, 18].
However, the success of existing DLKT research is heavily relied on how much information is revealed from the third-party education platforms due to the special characteristics of educational data such as data privacy. The majority of the widely used KT datasets such as ASSIST-ments [25], Peking Online Judge [23] and KDD datasets [7] only contain ID features of questions and knowledge components (KCs), which lack necessities.
Figure 1: The graphical illustration of the KT problem.
sary contextual information of questions and KCs. A KC is a description of a mental structure or process that a learner uses, alone or in combination with other KCs, to accomplish steps in a task or a problem. Furthermore, the existing KT datasets in math domain only cover a small number of KCs, which may not be able to represent the entire knowledge states even for students in a particular grade.
Therefore, in this work, we propose a large-scale dataset named XES3G5M which consists of student interaction logs collected from a K-12 online learning platform in China. Specifically, the XES3G5M dataset is made up of 7,652 questions, 865 KCs and 5,549,635 interactions from 18,066 students. The XES3G5M provides extremely rich auxiliary information from diverse educational scenes that can be summarized as follows:
- Question side information: the dataset provides plenty of question-side information including question textual content, question types, and answer analysis which provides problem-solving ideas. The information on the question side is provided in detail and is helpful to discriminate the questions with the same KC sets.
- KC side information: the KCs in XES3G5M are in a multi-level structure and each KC has a corresponding route to indicate its relation with other KCs, which can be used to capture the latent dependency of KCs.
The KC route is presented as a structural tree that explicitly introduces the prerequisite relations among all the 865 KCs involved in the proposed dataset. To the best of our knowledge, our XES3G5M dataset not only has the largest number of KCs in math domain but contains the richest contextual information including tree structured KC relations, question types, textual contents and analysis and student response timestamps. Meanwhile, we present a systematic KT benchmark on XES3G5M with 19 state-of-the-art DLKT models to evaluate the applicability and effectiveness of our dataset. We further conduct comprehensive experimental analysis to demonstrate the improvements of leveraging the auxiliary information in DLKT models. Moreover, we also conduct multi-step ahead predictions that predict students' future responses given the limited historical interactions to meet the real-world educational scenario.
2.1 Knowledge Tracing Datasets
Recently, there are lots of educational datasets that involve students' question-solving logs that have been widely used in KT task. For example, ASSISTments datasets like ASSISTments2009, ASSISTments2015 are collected from ASSISTments online learning platform, which supplies instructional
counseling by measuring the students' knowledge states [6, 24]. Junyi Academy is an online learning activity dataset that mainly solves 722 mathematics questions from an online learning platform in Taiwan<sup>3</sup>. Peking Online Judge (POJ) is derived from the Peking online platform, which offers students coding practices to help them improve their programming skills, and is made available by [23]. EdNet contains 784,309 students' learning activities for preparing the TOEIC (Test of English for International Communication<sup>®</sup>) test from an AI tutoring service platform named Santa in South Korea [5]. The Statics2011 dataset contains 333 student interactions from an online engineering statics educational course at Carnegie Mellon University during Fall 2011<sup>4</sup>.
Meanwhile, there are some high-quality students' learning sequences data released by various top data mining competitions. KDD datasets stem from the KDD Cup 2010 EDM Challenge which includes the Algebra question-solving logs of 13-14 years old students with detailed step-level student responses<sup>5</sup>. The NeurIPS2020 dataset is provided by the NeurlPS2020 Education Challenge. The recordings for Tasks 3 and 4 in NeurlPS2020 consist of student responses to mathematical questions from Eedi, which engages millions of students in daily interactions around the world<sup>6</sup>.
However, as the comparison results are shown in Table 1, all aforementioned datasets mainly contain the ID features of questions and KCs and timestamps but lack information related to educational contexts that may potentially be useful in assessing students' knowledge states [23, 31, 32, 7]. Compared to these datasets, our XES3G5M has extra question textual contents, KC relationships, question types, and answer analysis that may be utilized to enhance the modeling process of the student's learning outcomes.
Table 1: Comparisons between existing benchmark KT datasets and XES3G5M.
ASSISTments	Statistcs 2011	Junyi 2015	KDD	NeurIPS 2020	POJ	EdNet	XES3G5M
2009	2012	2015	2017	2005	2006
# of Students	4,217	46,674	19,917	1,709	333	247,606	574	1,146	4,918	22,916	1,677,583	18,066
# of Ques.	26,688	179,999	100	3,162	1,224	722	210,710	207,856	948	2,750	52,676	7,652
# of KCs	123	265	-	102	-	41	112	493	57	-	962	865
# of Interactions	346,860	6,123,270	708,631	942,816	194,947	25,925,922	809,694	3,679,199	1,382,727	996,240	372,366,720	5,549,635
Subject	Math	Math	Math	Math	Math	Math	Math	Math	Math	PL	Linguistics	Math
Language	English	English	English	English	English	Chinese	English	English	English	English	English	Chinese
Timestamp avail.	No	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Contents avail.	No	No	No	No	No	Yes	No	No	No	No	No	Yes
KC relation avail.	No	No	No	No	No	No	No	No	No	No	No	Yes
Ques. type avail.	No	No	No	No	No	No	No	No	No	No	No	Yes
Ques. analysis avail.	No	No	No	No	No	No	No	No	No	No	No	Yes
2.2 Deep Learning Based KT Models
Many KT literature apply deep learning techniques to solve the KT problem. These DLKT models can be categorized into 5 categories as follows:
C1: Deep sequential models. The models capture the students' knowledge states by using autoregressive architectures to model the historical learning interactions [3, 8, 11, 12, 19, 20, 25, 31, 35, 30, 28, 18]. For example, Piech et al. exploited an LSTM layer to estimate the student knowledge states to predict their response performances [25].
C2: Memory augmented models. The models use external memory networks to capture the latent relations between KCs to predict students' knowledge states [1, 28, 36, 34]. For example, Zhang et al. used a static key memory matrix to model the KC latent relationships and predicted the students' knowledge mastery levels via a dynamic value memory matrix [36].
C3: Adversarial based models. The models use adversarial training techniques to improve their generalization capability by performing adversarial perturbations to the original student interaction sequences [8]. For example, Guo et al. generated adversarial perturbations into interaction sequences to reduce the risk of the overfitting and limited generalization problem of DLKT models [8].
C4: Graph based models. The models use graph neural networks to model intrinsic relations among questions, KCs and interactions [21, 32, 33]. For example, Liu et al. captured the question-level and KC-level inner relations via a question-KC bipartite graph to augment the question and KC representations of DLKT model [14].
<sup>3</sup>https://www.junyiacademy.org/
4 https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=507
$^{5}$ https://kdd.org/kdd-cup/view/kdd-cup-2010-student-performance-evaluation/Data
<sup>6</sup>https://eedi.com/projects/neurips-education-challenge
C5: Attention based models. The models use attention mechanisms to capture the relevance between historical interactions and the future questions [7, 23, 26, 37, 22, 4]. For example, Pandey & Karypis leveraged a self-attention network to estimate the relation between exercises and responses [22].
Although the aforementioned DLKT approaches have achieved remarkable results, [7, 21, 25, 29, 36], recent studies [11, 12, 25, 31] seem to resemble each other with very limited nuances from the methodological perspective. Most of the existing work only provides coarse evaluation and both the contributing factors leading to the success of DLKT and how the DLKT models perform in real-world educational contexts still remain somewhat unknown. Therefore, in this work, we conduct standardized evaluations with 19 classical DLKT models on the proposed XES3G5M dataset, which ensures that all the models can be compared in a fair evaluation protocol. With the proposed benchmark, the KT researchers are able to evaluate their proposed approaches against a wide range of advanced methods on the XES3G5M dataset and the practitioners are capable of discriminating against the opportunities and challenges of the DLKT algorithms in real-world educational contexts.
3 Data Description
3.1 License
The dataset can be freely downloaded from https://github.com/ai4ed/XES3G5M and used with the MIT license<sup>7</sup>. Besides the KT task, the users can use the dataset for other custom tasks such as exercise recommendation under the license.
3.2 Data Collection and Preprocessing
To construct a high-quality educational dataset containing rich auxiliary information, we mainly collect data from two perspectives, including questions designed by teachers during class and homework assignments. First, for the questions in class, we collect educational data from an online learning system that is developed by TAL Education Group (TAL), one of the largest educational technology companies in China. For each sample, we collect question textual information and the corresponding student's answer. Second, for the questions of homework assignments, we utilize a series of auxiliary educational tools of TAL such as personalized learning applications to collect students' question-solving logs in their homework.
To this end, the dataset includes more than 5 million interactions collected from more than 18,000 third-grade students that respond to about 8,000 math questions. In XES3G5M, we only select the students' sequences with more than 200 interactions to better model students' learning processes with enough historical interaction information. Furthermore, we perform data preprocessing with the following steps to obtain the XES3G5M dataset: (1) we remove the questions with missing and incomplete content; (2) we remove the questions that miss the associated KCs; (3) the questions in the dataset are only in single choice and fill-in-the-blank types; (4) to evaluate the 19 DLKT models under a rigorous evaluation protocol, we further conduct data preprocessing following [17]. The data statistics after data preprocessing is described in Section 3.5.
3.3 Privacy
In this dataset, we have collected students' interaction sequences, including student IDs, question IDs, answers (correct or incorrect), timestamps, question contents, KCs associated with the questions, and question types. Among the selected information, the student IDs and question IDs have the potential to reveal the identity of students and questions. Therefore, to better protect students' privacy, we have employed a lightweight data encryption method based on data mapping. For each student ID and question ID, we map it into a non-reversible digital identifier. The mapping identifier generates unique IDs in the dataset to guarantee each student and question has a unique identifier to avoid any duplication or confusion. The encryption method enables data security meanwhile improving data processing efficiency.
<sup>7</sup>https://github.com/ai4ed/XES3G5M/blob/main/License
3.4 Interactions with Auxiliary Information
As shown in Table 1, XES3G5M has plentiful auxiliary information compared to existing educational question-solving logs datasets. In the following subsections, we will introduce the interaction sequences information and the corresponding auxiliary information in detail.
3.4.1 Interactions
Students' learning interactions are crucial to model their learning processes in the KT problem. In XES3G5M, each interaction has the following information: (1) question ID: the unique question ID that the student answered; (2) KC ID: the corresponding leaf KC IDs associated with the answered question; (3) response: the correctness of the response of a student, 1 denotes correct and 0 denotes incorrect; (4) timestamp: the start timestamp of the student answering the question.
3.4.2 Auxiliary Information
Besides the interaction information mentioned above, we collect additional auxiliary information to provide more fine-grained information to enhance a student's knowledge mastery estimation. Specifically, the auxiliary information is included in five types:
- Question Content: question textual content and the associated images. The examples of multi-choice questions and fill-in-the-blank questions are illustrated in Figure 2. More examples are shown in Appendix A.4. It can be seen that the image provides indispensable information for understanding multi-choice questions shown on the left of Figure 2 that are neglected in existing datasets, which is helpful to enhance the question representation during the interaction encoding. The original question textual content is stored in Chinese and we open sourced both the original Chinese version and the numerical embeddings we extract from an in-house pre-trained large-scale language model built upon math related corpus.
Figure 2: The examples of a multi-choice question and a fill-in-the-blank question in XES3G5M (questions have been translated).
- Question Answer: the standardized correct answer to the question is included in our dataset. We provide the correct options for multi-choice questions and the correct answer contents for fill-in-the-blank questions.
- Question Analysis: the detailed process of solving the question, including the principles and the particular steps required to solve the questions as shown in Figure 2.
- KC Routes: the relation among KCs. Practically, the KCs are subordinate to each other rather than independent. All the KCs in our dataset are formed hierarchically in a tree structure, and a KC route is a path from the root node of a KC to a leaf node of a KC in the tree. Since we construct KC routes from different perspectives, different KC routes may have the same leaf node. Due to the space limit, the example of KT routes is shown in Appendix A.5. With the KC routes in the dataset, we can obtain more informational representations of questions and KCs, yielding measure the relevance between questions and responses.
- Question Type: the type of question. The question type of questions in our dataset contains multi-choice and fill-in-the-blank questions.
Due to the space limit, the dataset structure and storage format are described in Appendix A.6.
3.5 Statistics
Table 1 shows the statistics of the basic information for XES3G5M. There are 5,549,635 interactions and about 18,066 students answered 7,652 questions from 865 leaf-node KCs. The average number of KCs for each question is 1.1640. There are 6,142 and 1,510 questions for fill-in-the-blank and multi-choice questions respectively. The  $82.98\%$  questions and  $92.37\%$  KCs in the dataset have more than 20 interactions to be answered by students.
Figure 3 shows the distributions of students' interaction sequence information in XES3G5M. Figure 3(a) shows the distribution of students' interaction sequence length. From the Figure 3(a), we can see that there are  $89.07\%$  student sequences containing 200 to 500 interactions. From figure 3(b), we can see that more than half of the students in the dataset  $(54.42\%)$  can answer up to  $80\%$  questions correctly among all the questions they answered. Figure 3(c) and (d) illustrate the temporal information analysis for the students' sequences. We calculate the average variance of the time span between two consecutive interactions in each student's interaction sequence. From Figure 3(c), we observe the average variance of the time span between two consecutive interactions is always 80 hours. This indicates there is a long interval time between students' consecutive interactions. We also compute the whole duration of each student responding to all the questions of a learning sequence, as shown in Figure 3(d). It can be seen that almost all students in the dataset interact with questions for more than 1 year. Due to the space limit, the distributions of questions' information are shown in Appendix A.7.
Stu Correct Ratio
(b)
Figure 3: The distributions about interaction length, correct answer ratio, the average variance of the time span between two consecutive interactions and the whole duration of each student's sequence.
Sequence Time Std
(c)
Learning Time
(d)
4 The XES3G5M Benchmark
In this section, we conduct the classical DLKT models on the proposed XES3G5M. We perform standard 5-fold cross validation for all the combinations of pairs of method and dataset. We tune the parameters on the validation set and report the performance on test set. The embedding size of all the models is set to [64, 128]. We set the learning rate as [1e-3, 1e-4, 1e-5] and the dropout rate is set to [0.05, 0.1, 0.3, 0.5]. We adopt the Adam [10] optimizer to train all models for 200 epochs. We stop the training process if the AUC scores do not improve in 10 epochs. We run all the models on NVIDIA RTX A5000 GPU devices with Pytorch. For the evaluation metric, we choose the Area Under the Curve (AUC) and accuracy to evaluate the prediction performance similar to previous DLKT studies [25, 35, 20, 30, 36, 1, 34, 8, 21, 22, 4, 7, 28, 18].
In our benchmark, we mainly select 19 state-of-the-art DLKT models in 5 categories discussed in Section 2.2 as follows:
- C1: DKT [25]: it uses an LSTM layer to model students' learning processes.
- C1: DKT+ [35]: it is a variant of DKT to address the reconstruction and inconsistent issues in the original DKT model.
- C1: DKT-F [20]: it is also an extension of DKT that considers students' forgetting behaviors to predict student performance.
- C1: KQN [11]: it is an RNN-based model that extracts the relation representation between students' learning abilities and KCs to predict students' performance.
- C1: qDKT [30]: it is a variant of DKT that models each learner's response performance on individual questions over time.
- C1: IEKT [18]: it estimates students' knowledge states via the student cognition and knowledge acquisition estimation modules.
- C1: DIMKT [27]: it considers personalized knowledge acquisition of a student and dynamically estimates the student's knowledge states by exploring the question difficulty effect.
- C1: AT-DKT [15]: it performs two auxiliary learning tasks including question tagging prediction and individualized prior knowledge prediction task to enhance the predictive capability of DKT.
- C1: QIKT [2]: it is a question-centric interpretable KT model jointly learned from a question-centric knowledge acquisition module and a question-centric problem-solving module.
- C2: DKVMN [36]: it exploits two memory networks to calculate the relationships between potential concepts and directly predicts a student's knowledge state of each concept respectively.
- C2: SKVMN [1]: it is a combination of DKVMN and LSTM that uses a hop-LSTM layer to capture sequential dependencies of exercises.
- C2: DeepIRT [34]: it incorporates DKVMN and item response theory to enhance the interpretability of the prediction output of DKVMN.
- C3: ATKT [8]: it improves the generalization of the attention-LSTM based KT model via adversarial training.
- C4: GKT [21]: it casts the knowledge structure as a graph and reformulates the KT task as a time series node-level classification problem via a graph neural network.
- C5: SAKT [22]: it uses self-attention mechanism to measure the relevance between the current KCs and the students' historical interactions.
- C5: SAINT [4]: it is a Transformer-based model that encodes exercise information and historical responses in the encoder and decoder respectively to estimate students' current responses.
- C5: AKT [7]: it uses a monotonic attention mechanism to identify the relevance of questions and responses and uses a Rasch model to capture the relation representations of KCs and questions.
- C5: simpleKT [16]: this is a strong but simple baseline method that models question-specific variations to capture the individual differences among questions covering the same set of KCs.
- C5: sparseKT [9]: it incorporates a k-selection module to select relevant historical interactions with the highest attention scores to improve the robustness and generalization of the attention based DLKT models.
4.3 Performance Analysis only with ID Features