SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2 shows the performance of all DLKT models on XES3G5M. We report the average scores of AUC and accuracy and the standard deviations (SD) across 5 folds. As we can see that: (1) the DLKT models which integrate the KC representations and question representations together such as IEKT and QIKT mostly outperform other models which only extract KCs information like DKT, DKT+, KQN, DKMVN, ATKT, GKT, SAKT, SKVMN in terms of AUC scores and accuracy. This indicates that the questions and KCs contained in our proposed dataset are both beneficial to predict the students' future performance; (2) in particular, IEKT achieves the best AUC and accuracy compared to other KT models. We believe that it is due to IEKT extra uses historical prediction results when predicting the current response of students; (3) compared to DKT, DKT-F extracts temporal information to model the students' forgetting behavior when estimating student's knowledge states which indicate the effectiveness of temporal features like timestamps in the KT datasets; (4) among attention based DLKT models including SAKT, SAINT, AKT, simpleKT and sparseKT, AKT performs achieves the best AUC and accuracy. This may be due to AKT considering the student's
learning behavior via a monotonic attention module. In general, our XES3G5M contains diverse features of students' interaction logs which can be widely used in various KT models.
4.4 Performance Analysis with Auxiliary Information
We conduct additional experiments of leveraging the auxiliary information on KT models to examine the effectiveness of augmenting KT baselines with the auxiliary information in our XES3G5M dataset. Specifically, there are three different methods for the fusion of question and KC information: (1) Add: the embeddings of question information and KC information in XES3G5M are combined together and added to the original embedding of question ID and KC ID respectively. (2) Early Fusion: keep the original embedding of question ID and KC ID unchanged. The other information of the question and KCs in XES3G5M are merged by a gate unit before as the input embedding to DLKT models. (3) Late
Table 2: The AUC and accuracy performance of KT baselines on XES3G5M dataset.
Model	Ques. ID	KC ID	Timestamp	AUC	Accuracy
DKT	No	Yes	No	0.7852±0.0006	0.8173±0.0002
DKT+	No	Yes	No	0.7861±0.0002	0.8178±0.0001
DKT-F	No	Yes	Yes	0.7940±0.0006	0.8209±0.0003
KQN	No	Yes	No	0.7793±0.0006	0.8152±0.0002
qDKT	Yes	No	No	0.8225±0.0002	0.8301±0.0000
IEKT	Yes	Yes	No	0.8280±0.0002	0.8316±0.0002
DIMKT	Yes	Yes	No	0.8220±0.0002	0.8291±0.0006
AT-DKT	Yes	Yes	No	0.7932±0.0004	0.8198±0.0004
QIKT	Yes	Yes	No	0.8222±0.0006	0.8300±0.0005
DKVMN	No	Yes	No	0.7792±0.0004	0.8155±0.0001
SKVMN	No	Yes	No	0.7514±0.0005	0.8075±0.0003
DeepIRT	No	Yes	No	0.7785±0.0005	0.8150±0.0002
ATKT	No	Yes	No	0.7783±0.0004	0.8145±0.0002
GKT	No	Yes	No	0.7727±0.0006	0.8135±0.0004
SAKT	No	Yes	No	0.7693±0.0008	0.8124±0.0002
SAINT	Yes	Yes	Yes	0.8074±0.0007	0.8177±0.0006
AKT	Yes	Yes	No	0.8207±0.0008	0.8273±0.0007
simpleKT	Yes	Yes	No	0.8163±0.0006	0.8246±0.0005
sparseKT	Yes	Yes	No	0.8165±0.0015	0.8234±0.0009
Fusion: the additional information of the question and KCs is combined via a gate unit after the output of DLKT models.
We use the abbreviations to represent different information as follows: (1) Q.ID: the question ID; (2) Q.Cont: the question content, analysis content and question type of the corresponding question; (3) KC ID: the KC ID of the leaf node in KC route for the corresponding question; (4) KC Cont: the plaintext of the corresponding KC; and (5) KC Route: the KC route of the corresponding question, the weight of each level in the KC routes are learned by the model.
Table 3: The AUC and accuracy performance with auxiliary information on XES3G5M dataset.
Model	Q.ID	KC ID	Q.Cont	KC Cont	KC Route	AUC	Accuracy
Fusion Type	Fusion Type
Add	Early Fusion	Late Fusion	Add	Early Fusion	Late Fusion
qDKT	Yes	No	No	No	No	0.8225±0.0002	-	-	0.8301±0.0000	-	-
Yes	No	Yes	No	No	0.8220±0.0004	0.8228±0.0005	-	0.8299±0.0002	0.8303±0.0003	-
simpleKT	Yes	Yes	No	No	No	0.8163±0.0006	-	-	0.8163±0.0006	-	-
Yes	Yes	Yes	Yes	No	0.8251±0.0008	0.8277±0.0003	0.8251±0.0008	0.8303±0.0007	0.8313±0.0003	0.8303±0.0003
Yes	Yes	Yes	No	Yes	0.8219±0.0006	0.8268±0.0012	0.8256±0.0006	0.8280±0.0007	0.8309±0.0003	0.8305±0.0003
Yes	Yes	Yes	Yes	Yes	0.8245±0.0006	0.8262±0.0010	0.8227±0.0029	0.8302±0.0003	0.8304±0.0007	0.8294±0.0008
AKT	Yes	Yes	No	No	No	0.8207±0.0008	-	-	0.8273±0.0007	-	-
Yes	Yes	Yes	Yes	No	0.8228±0.0004	0.8292±0.0007	0.8289±0.0008	0.8288±0.0003	0.8321±0.0005	0.8323±0.0004
Yes	Yes	Yes	No	Yes	0.8216±0.0004	0.8287±0.0007	0.8295±0.0003	0.8285±0.0003	0.8319±0.0006	0.8327±0.0003
Yes	Yes	Yes	Yes	Yes	0.8213±0.0003	0.8290±0.0007	0.8288±0.0009	0.8284±0.0004	0.8317±0.0006	0.8323±0.0005
To comprehensively evaluate the effects of auxiliary information, we select top performing methods from Table 2 in terms of AUC. Furthermore, since the IEKT has a reinforcement learning based sampling module to select students' individual cognition and acquisition representations, which is very time-consuming, we choose to use qDKT, simpleKT, and AKT as the representative approaches. Their AUC and accuracy results are shown in Table 3. We have the following three important observations described as follows: (1) augment auxiliary information in three fusion methods on simpleKT and AKT outperform the original models without auxiliary information. The simpleKT and AKT models with Q.Cont and KC Cont in the early fusion method have significant improvements of  $1.14\%$  and  $0.85\%$  in terms of the AUC scores; (2) with the comparison among three fusion methods for using the auxiliary information, the results of early fusion methods always perform best. In AKT, it gets the improvements of AUC performance in  $0.64\%$ ,  $0.71\%$  and  $0.77\%$  compared to the results of add fusion way in three selection ways of question and KC information; (3) the AUC performance of using Q.Cont and KC Cont are the majority of the best performance in three fusion methods of using auxiliary information. After using more KC Route information, the performance declined. We suppose it is because the way of encoding KC Route is too simple that leads to the
misunderstanding of KT models. Due to the space limit, we provide the visualization comparison of question embeddings both with and without the KC Route information in the Appendix A.9; (4) the augmentation results of qDKT is slightly decline compared to the original qDKT without auxiliary information, it is because qDKT estimate students' knowledge states in question level, adding more information about KC make the model become harder to discriminate the individualized of questions.
4.5 Performance on Multi-step ahead Prediction
To make the KT prediction close to real application scenarios, we make a multi-step ahead prediction to students' future responses given their historical interaction sequence following [17], shown in Figure 4. The multistep prediction will provide construc
Figure 4: The graphical illustration of the multi-step ahead prediction problem.
tive feedback to learning path selection and construction for teachers, and help them flexibly prepare the teaching materials. The multi-step ahead prediction has two prediction methods including non-accumulative prediction and accumulative prediction. The non-accumulative prediction method aims to predict all future responses all at once by only using the given historical interactions. Besides the given historical interactions, the accumulative prediction approach also uses the last predicted value to predict the current response. Specifically, we vary the observed percentages of student historical interaction length from  $20\%$  to  $90\%$  with a step size of  $10\%$ . As discussed in Section 4.4, simply augmenting the auxiliary information also improves the performance of KT models on predicting students' future performance. Hence, to further investigate the effectiveness of auxiliary information augmentation in the multi-step ahead prediction setting, we keep the same representative DLKT approaches (qDKT, simpleKT and AKT) used in Section 4.4 with their corresponding best fusion approaches reported in Table 3, i.e., qDKT with Q.ID and KC ID, simpleKT with Q.ID, KC ID, Q.Cont and KC Cont in early fusion, AKT with Q.ID, KC ID and Q.Cont in late fusion as the representative approaches.
From Figure 5, we can see that (1) the AUC and accuracy performance get higher results with the increasing percentage of historical interactions no matter whether augment DLKT models with auxiliary information or not, which matches the real-world educational scenario; (2) the majority of AUC and accuracy results of the models with auxiliary information (shown in solid lines) always outperform the baseline models without combining auxiliary information (shown in dotted lines), which indicates that with the limited historical interactions, the auxiliary information of questions provides more beneficial information to accurate the students' performance estimations; (3) the performance improvements of non-accumulative prediction with auxiliary information are higher than the results with accumulative setting compared the original baselines without auxiliary information respectively. This is because the non-accumulative prediction only considers the given historical interactions while the accumulative setting additionally uses the last prediction result to predict the next question, the limited information given leads to higher improvements from auxiliary information.
Figure 5: Accumulative (Accum) and Non-accumulative (Non-accum) predictions in the multi-step ahead scenario in terms of AUC and accuracy on XES3G5M. The results shown in dotted/solid lines are the AUC and accuracy results of the original qDKT/simpleKT/AKT without/with auxiliary information (e.g., Non-accum with aux info denotes the non-accumulative prediction of augmenting the qDKT/simpleKT/AKT with auxiliary information).
In spite of the rich auxiliary information in XES3G5M is beneficial for predicting the future performance of students, it also has some limitations:
- Data Imbalance: There are two imbalance problems in XES3G5M:
- Question Type Imbalance. From section 3, there are  $80.27\%$  fill-in-the-blank questions in XES3G5M while only  $19.73\%$  questions are multiple-choice.
- Interaction Imbalance. The number of interactions of questions are ranging from 1 to 29,504 and the interaction numbers of KCs are ranging from 1 to 150,468. Since the large number of questions and KCs, there may be some questions and KCs have limited question-solving logs of students, which lead to the model being difficult to model students' behavior due to data sparsity.
- Lack of Duration: Due to the end timestamps of students' answers to each question were not recorded in the online learning platform, XES3G5M lacks duration for each interaction which is also helpful for estimating students' knowledge states over a series of time.
In this paper, we develop a large-scale open source KT dataset named XES3G5M. The dataset has 5,549,635 interactions and about 18,066 students answered 7,652 questions associated with 865 leaf KCs. Notably, we provide the detailed information of question and knowledge side information, including question contents, question analysis, question types and the corresponding KC routes. Furthermore, we construct a DLKT benchmark on XES3G5M with 19 state-of-the-art DLKT baselines. To evaluate the effectiveness of the auxiliary information in XES3G5M, we conduct comprehensive experiments to leverage the auxiliary information to DLKT models. In the future, we will explore how to deeply model the relationship between KCs via graph neural networks based on the KC routes in XES3G5M and then improve the students' response prediction performance with the abundant information of questions.
7 Reproducibility Statement
To reproduce the experimental results, you can use the models in PYKT python library at https: //github.com/pykt-team/pykt-toolkit and download the dataset of XES3G5M in https:// github.com/ai4ed/XES3G5M. The details of data-preprocessing and the training hyper-parameters are shown in Section 3.2 and Section 4.1. Because the codes of models and dataset are all provided, all the results can be easily reproducible.
This work was supported in part by National Key R&D Program of China, under Grant No. 2020AAA0104500; in part by Key Laboratory of Smart Education of Guangdong Higher Education Institutes, Jinan University (2022LSYS003); in part by National Joint Engineering Research Center of Network Security Detection and Protection Technology and in part by APRC - CityU New Research Initiatives (No.9610565, Start-up Grant for New Faculty of City University of Hong Kong), CityU - HKIDS Early Career Research Grant (No.9360163), Hong Kong ITC Innovation and Technology Fund Midstream Research Programme for Universities Project (No.ITS/034/22MS), Hong Kong Environmental and Conservation Fund (No.88/2022), SIRG - CityU Strategic Interdisciplinary Research Grant (No.7020046, No.7020074).
[1] Ghodai Abdelrahman and Qing Wang. Knowledge tracing with sequential key-value memory networks. In Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer, editors, Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Paris, France, July 21-25, 2019, pages 175-184. ACM, 2019.
[36] Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. Dynamic Key-Value Memory Networks for Knowledge Tracing. In Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich, editors, Proceedings of the 26th International Conference on World Wide Web, Perth, Australia, April 3-7, 2017, pages 765-774. ACM, 2017.
[37] Moyu Zhang, Xinning Zhu, Chunhong Zhang, Yang Ji, Feng Pan, and Changchuan Yin. Multi-Factors Aware Dual-Attentional Knowledge Tracing. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2588–2597, 2021.