Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing

Knowledge tracing is a sequence prediction problem where the goal is to predict the outcomes of students over questions as they are interacting with a learning platform. By tracking the evolution of the knowledge of some student, one can optimize instruction. Existing methods are either based on temporal latent variable models, or factor analysis with temporal features. We here show that factorization machines (FMs), a model for regression or classification, encompasses several existing models in the educational literature as special cases, notably additive factor model, performance factor model, and multidimensional item response theory. We show, using several real datasets of tens of thousands of users and items, that FMs can estimate student knowledge accurately and fast even when student data is sparsely observed, and handle side information such as multiple knowledge components and number of attempts at item or skill level. Our approach allows to fit student models of higher dimension than existing models, and provides a testbed to try new combinations of features in order to improve existing models. Modeling student learning is key to be able to detect students that need further attention, or recommend automatically relevant learning resources. Initially, models were developed for students sitting for standardized tests, where students could read every problem statement, and missing answers could be treated as incorrect. However, in online platforms such as MOOCs, students attempt some exercises, but do not even look at other ones. Also, they may learn between different attempts. How to measure knowledge when students have attempted different questions? We want to predict the performance of a set I of students, say users, over a set J of questions, say items (we will interchangeably refer to questions as items, problems, or tasks). Each student can attempt a question multiple times, and may learn between successive attempts. We assume we observe ordered triplets (i, j, o) ∈ I × J × {0, 1} which encode the fact that student i attempted question j and got it either correct (o = 1) or incorrect (o = 0). Triplets are sorted chronologically. Then, given a new pair (i′, j′), we need to predict whether student i′ will get question j′ correct or incorrect. We can also assume extra knowledge about users, or items. So far, various models have been designed for student Copyright c © 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. modeling, either based on prediction of sequences (Piech et al. 2015), or factor analysis (Thai-Nghe et al. 2011; Lavoué et al. 2018). Most of existing techniques model students or questions with unidimensional parameters. In this paper, we generalize these models to higher dimensions and manage to train efficiently student models of dimension up to 20. Our family of models is particularly convenient when observations from students are sparse, e.g. when some students attempted few questions, or some questions were answered by few students, which is most of the data usually encountered in online platforms such as MOOCs. When fitting student models, it is better to rely on all the information available at hand. In order to get information about questions, one can identify the knowledge components (KCs) involved in each question. This side information is usually encoded under the form of a q-matrix, that maps items to knowledge components: qjk is 1 if item j involves KC k, 0 otherwise. In this paper, we will also note KC(j) the sets of skills involved by question j, i.e. KC(j) = {k|qjk = 1}. In order to model different attempts, one can keep track of how many times a student has attempted a question, or how many times a student has had the opportunity to acquire a skill, while interacting with the learning material. Our experiments show, in particular, that: • It is better to estimate a bias for each item (not only skill), which popular educational data mining (EDM) models do not. • Most existing models in EDM cannot handle side information such as multiple skills for one item, but the proposed approach does. • Side information improves performance more than increasing the latent dimension. To the best of our knowledge, this is the most generic framework that incorporates side information into a student model. For the sake of reproducibility, our implementation is available on GitHub1. The interested reader can check our code and reuse it in order to try new combinations and devise new models. In Section 2, we show related work. In Section 3, we present a family of models, knowledge tracing machines, and recover famous models of the EDM literature as special https://github.com/jilljenn/ktm ar X iv :1 81 1. 03 38 8v 2 [ cs .I R ] 1 5 N ov 2 01 8 cases. Then, in Section 4 we conduct experiments and show our results in Section 5. We conclude with further work in Section 6.