Predictive, scalable and interpretable knowledge tracing on structured domains
Authors: Hanqi Zhou
Robert Bamler
Charley M Wu
Álvaro Tejero-Cantero
Venue: ICLR 2024 spotlight (2024)
Keywords: knowledge tracing
interpretable representations
knowledge graphs
probabilistic models
variational inference
continual learning
Primary area: applications to neuroscience & cognitive science
TL;DR: Performant and scalable knowledge tracing with the interpretability that personalized education needs.

Abstract: Intelligent tutoring systems optimize the selection and timing of learning materials to enhance understanding and long-term retention. This requires estimates of both the learner's progress ("knowledge tracing"; KT), and the prerequisite structure of the learning domain ("knowledge mapping"). While recent deep learning models achieve high KT accuracy, they do so at the expense of the interpretability of psychologically-inspired models. In this work, we present a solution to this trade-off. PSI-KT is a hierarchical generative approach that explicitly models how both individual cognitive traits and the prerequisite structure of knowledge influence learning dynamics, thus achieving interpretability by design. Moreover, by using scalable Bayesian inference, PSI-KT targets the real-world need for efficient personalization even with a growing body of learners and interaction data. Evaluated on three datasets from online learning platforms, PSI-KT achieves superior multi-step **p**redictive accuracy and **s**calable inference in continual-learning settings, all while providing **i**nterpretable representations of learner-specific traits and the prerequisite structure of knowledge that causally supports learning. In sum, predictive, scalable and interpretable knowledge tracing with solid knowledge mapping lays a key foundation for effective personalized learning to make education accessible to a broad, global audience.

Published as a conference paper at ICLR 2024
PREDICTIVE, SCALABLE AND INTERPRETABLE KNOWL- EDGE TRACING ON STRUCTURED DOMAINS
Hanqi Zhou 1,2,4, Robert Bamler' Charley M. Wu & Álvaro Tejero-Cantero 1.2+
University of Tübingen, "Cluster Excellence Machine Learning, Tübingen Al Center, *IMPRS-IS (hanqi zhou, robert bamler, charley, wu, alvaro. (ejerro Buni -tuebingen. de
ABSTRACT
Intelligent tutoring systems optimize the selection and timing of learning materials to enhance understanding and long-term retention. This requires estimates of both the learner's progress ("knowledge tracing"; KT), and the prerequisite structure the learning domain ("knowledge mapping"). While recent deep learning models achieve high KT accuracy, they do at the expense of the interpretability of psychologically-inse -nnsiieed models. this work, we present a solution to this trade-off. PSI-KT is hierarchical generative approach that explicitly models how both individual cognitive traits and the prerequisite structure of knowledge influence learning dynamics, thus achieving interpretability by design. Moreover, by using scalable Bayesian inference, PSI-KT targets the real-world need for efficient personalization even with growing body of learners and learning histories. Evaluated on three datasets from online learning platforms, PSI-KT achieves superior multi-step predictive accuracy and scalable inference leaaing settings, all while providing interpretable representations of learner-specific traits and the prerequisite structure of knowledge that causally supports learning. sum, predictive, scalable and interpretable knowledge tracing with solid knowledge mapping lays a key foundation for effective personalized learning to make education accessible to broad, global audience.
INTRODUCTION
The rise of online education platforms has created new opportunities for personalization in learning, motivating a renewed interest how humans learn structured knowledge domains. Foundational theories in psychology (Ebbinghaus, 1885) have informed spaced repetition schedules (Settles & Meeder, 2016), which exploit the finding that an optimal spacing of learning sessions enhances memory retention. Yet beyond the timing of rehearsals, the sequential order of learning materials is also crucial, as evidenced by curriculum effects in learning (Dewey, 1910; Dekker et al., 2022), where exposure simpler, prerequisite concepts can facilitate the apprehension of higher-level ideas. Cognitive science and pedagogical theories have long emphasized the relational structure of knowledge in human learning (Rumelhart, 2017; Piaget, 1970), with recent research showing that mastering prerequisites enhances concept learning (Lynn & Bassett, 2020; Karuza et al., 2016; Brändle al., 2022). Yet, we still lack a predictive, scalable, and interpretable model of the structural- temporal dynamics of learning that could be used to develop future intelligent tutoring systems.
Here, we present PSI-KT, a novel approach for inferring interpretable learner-specific cognitive traits and a shared knowledge graph of prerequisite concepts. We demonstrate our approach on three real-world educational datasets covering structured domains, where our model outperforms existing baselines in terms of predictive accuracy (both within- and between- learner generalization). scalability in a continual learning setting, and interpretability of learner traits and prerequisite graphs. The paper is organized follows: We first introduce the knowledge tracing problem and summarize related work (Sec. 2). We then provide a formal description of PSI-KT and describe the inference method (Sec. 3). Experimental evaluations are organized into demonstrations of prediction performance, scalability, and interpretability (Sec. 4). Altogether, PSI-KT bridges machine learning and cognitive science, leveraging our understanding of human learning build the foundations for automated tutoring systems with broad educational applications.
"Equal contribution. Code gituhh.com/mlcolab/ssiik

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
In this section, we begin by defining the knowledge tracing problem and then review related work.
2.1 KNOWLEDGE TRACING FOR INTELLIGENT TUTORING SYSTEMS
For almost 100 years (Pressey, 1926), researchers have developed intelligent tutoring systems (ITS) to support human learning through adaptive teaching materials and feedback. More recently, knowledge tracing (KT: Corbett & Anderson, 1994) emerged as method for tracking learning progress by predicting a learner's ' performance on different knowledge components (KCs), e.g., the 'Pythagorean theorem', based on past learning interactions. Here, we focus on the KT problem, with the goal of supporting the selection of teaching materials in future ITS applications.
In this setting, a learner receives exercises or flashcards for KCs x E 1,..,..., K} at irregularly spaced times tyy whereupon the performance is recorded, often as correct/incorrect, y E {0. 1). We can formalize KT as supervised learning problem on time-series data, where the goal of the KT model is to predict future performance (e.g., UN+1) given all part of the interaction history = f% / available up to time fyy- As part of the process, a KT model may infer specific representations of learners or of the learning domain to help prediction. If these representations are interpretable, they can be valuable for downstream learning personalization.
2.2 RELATED WORK
We broadly categorize related KT approaches into psychological and deep learning methods.
Psychological methods. Focusing on interpretability, psychological methods use domain knowledge to describe the temporal decay of memory (e.g., forgetting curves; Ebbinghaus, 1885), sometimes also modeling learner-specific characteristics. Factor-based regression models use hand-crafted features based on learner interactions and KC properties (e.g., repetition counts and KC easiness; Pavlik Jr et al., 2009). While they model KC-dependent memory dynamics (Pavlik et al., 2021; Gervet et al., 2020; Lindsey et al., 2014; Lord, 2012; Ackerman, 2014), they ignore the relational structure between KCs. Half-life Regression (HLR: Settles & Meeder, 2016) from Duolingo uses both correct and incorrect counts, while the Predictive Performance Equation (PPE; Walsh et al., 2018) models the elapsed time of every past interaction with a power function to account for spacing effects. By using shallow regression models with predefined features, these models achieve interpretability. but sacrifice prediction accuracy. Latent variable models use a probabilistic two-state Hidden Markov Model (Käser et al., 2017: Sao Pedro et al., 2013; Baker al., 2008; Yudelson et al., 2013). representing either mastery or non- non-mastern mastery of a given KC. These models are limited to binary states by design, do not account for learner dynamics, and for some, their numerous parameters hinder scalability. Another probabilistic model, HKT (Wang et al., 2021) accounts for structure and dynamics by modeling knowledge evolution as multivariate Hawkes process. Close in spirit to our PSI-KT, this approach tracks KC structure but lacks any learner-specific representations.
Deep learning methods. Deep learning methods use flexible models with many parameters to achieve high prediction accuracy. However, this flexibility also makes it difficult to interpret their learned internal representations. The first deep learning methods explicitly modeled sequential interactions with recurrent neural networks to overcome the dependence on fixed summary statistics in simpler regression models, with Deep Knowledge Tracing (DKT; Piech et al., 2015) pioneering the use of Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997). A similar architecture, DKTF (Nagatani et al., 2019) incorporated additional input features, whereas Shen al. (2021) proposed an intricate modular architecture aimed at recovering interpretable learner representations, but neglecting KC relations. Structure-avare models leverage KC dependencies, accounting for the fact that human knowledge acquisition is structured by dependency relationships (i.c.. concept maps; Hill, 2005; Koponen& & Nousiainen, 2018; Lynn & Bassett, 2020). Tong et al. (2020) empirically estimate KC dependencies from the frequencies of successful transitions. AKT (Ghosh et al., 2020) relies on the attention mechanism (Vaswani et al., 2017) to implicitly capture structure (Pandey & Karypis, 2019; Choi et al., 2020; Shin et al., 2021; Liu et al., 2023), whereas GKT (Nakagawa et al., 2019) models it explicitly based graph neural networks (Kipf & Welling, 2016). Recent work towards interpretable deep learning KT uses engineered features such as learner mastery and exercise difficulty (Minn et al.. 2022), infers them with neural networks (QIKT; Chen et al., 2023, IEKT; Long et al., 2021). While diverse approaches to interpretability exist (see Chen al., 2023, for review), a comprehensive evaluation framework is still lacking.
2

---

specific cognitive traits sf E R.d (top row in Fig. Ib), and (iii) a shared static graph A of KCs whose Published as a conference paper at ICLR 2024
(4)
learners
(b) learner
prerequisite structure
correct
incorrect
dynamics +
time
Mn-1
Figure I: PSI-KT is a hierarchical probabilistic state-space model of learning. (a) Latent knowledge states for different KCs (colored eurves) are inferred from observations. (b) Full hierarchical model for single learner: cognitive traits 8n control the coupled dynamics of states z which give rise to observations yn- (c) The dynamics combine memory decay (Eq. 6) and structural influences (Eq. 5).
Here, we present our predictive, scalable and interpretable KT model (PSI-KT) as a psychologically- informed probabilistic deep learning approach, together with comprehensive evaluation framework for interpretability..
JOINT DYNAMICAL AND STRUCTURAL MODEL OF LEARNING
In this section, we describe PSI-KT, our probabilistic hierarchical state-space model of human learning (Fig. 1). Briefly, observations of learner performance y (Fig. la. filled/unfilldd boxes) provide indirect and noisy evidence about latent knowledge states = (colored curves, with matching dots in Fig. lb). These latent states evolve stochastically, in line with the psychophysics of memory (temporal decay in Fig. Ic), while simultaneously being subject to structural influences from performance on prerequisite KCs (structure in Fig. lc). We introduce a second latent level of learner-specific traits s (Fig. 1b, top), which govern the knowledge dynamics in an interpretable way.
Below, we describe the method in more detail. We start with the generative model (Sec. 3.1). Next, we discuss the joint approximate Bayesian inference of latent variables and estimation of generative parameters (Sec. 3.2). Finally, we show how to derive multi-step performance predictions (see Sec. 3.3 and Fig. 7 in Appendix A.4 for a graphical overview of inference and prediction).
3.1 PROBABILISTIC STATE-SPACE GENERATIVE MODEL.
We conceptualize observations of learner performance noisy measurements of an underlying time- dependent knowledge state, specific to each learner and KC. The evolution of knowledge states reflects the process of learning and forgetting, governed by leamer-speeific traits. Additionally, knowledge of different KCs informs one another according to learned prerequisite relationships. We translate these modeling assumptions into a generative model consisting of three main components:()) the learner _e.K E R.K (colored curves in Fig. la), (ii) learner- knowledge state across KCs, z( [ ... Zn
edges a² quantify the probability for a KC to be prerequisite for KC & (Fig. Ic).
State-space model. State-space models (SSMs) are framework for partially observable dynamical processes, They represent the inherent noise of measurements by an emission distribution p(ya 2n), separate from the stochasticity of state dynamics, modeled as first-order Markov process with transition probabilities p( 2n- The state dynamics are initiated by sampling from an initial prior p(a1) to iteratively feed the transition kernel, and predictions can be drawn at any time from the emission distribution. To represent the influence of individual cognitive traits over the knowledge dynamics, we additionally condition the z-transitions on the traits s (which also can observed only indirectly). The three-level SSM hierarchy of PSI-KT consists of:
Level 2 (latent cognitive traits):
po f 5 1 = N(s H R)
(1)
Level (latent knowledge states):
= ~ po ( 2/ 2 = II (N mm Eol w
(2)
Level 0 (observed learner performance): v% p(u > = Bern bern(sigmont (sigmoid
The choice of Gaussian initial priors (discussed below) and Gaussian transitions ensures tractability,
while the Bernoulli emissions model the observed binary outcomes. We now unpack this model and all its parameters in detail, starting with the knowledge dynamics.
3

---

Published as a conference paper at ICLR 2024
Knowledge states 2. Recent KT methods (e.g., Nagatani et al., 2019) use exponential forgetting function based on psychological theories (Ebbinghaus, 1885). Here, we augment this approach by adding stable long-term memory (Averell & Heathcote, 2011), and model the knowledge dynam- ies 2 of an isolated KC k: as a mean-reverting stochastic Constein-UUlenbern QU) process: dd" a a'' - +Ln((()
Accordingly, the state of knowledge 2 gradually reverts to long-term mean j with rate at, subject to white noise fluctuations v7(4) scaled by volatility of. To account for the influence of other KCs, we adjust the mean u using prerequisite weights a² (defined in Eq. 7 below), modulated by the learner's 's transfer ability Th:
My ph + (1% o 2..
(S)
We obtain the mean m and variance wh of the transition kernel in Eq. 2 by marginalizing the OU process over one time step of == 1 1 which can be done analytically!
L m = r 2n-1 + (1-r() p. with retention ratio r = = E (0, 1).
(6)
As the time since the last interaction T grows, the retention ratio rf decreases exponentially with rate of and the knowledge state reverts to the long-term mean _ which partly depends on the learner's mastery of prerequisite KCs (Eq. 5). This balances short-term and long-term learning, reflecting empirical findings from memory research (Averell & Heathcote, 201 The structural influences are accounted for in the dynamies of 2t,,, thus justifying the conditional independence
assumed in Eq. 2. A Gaussian initial prior po(2f ( = [N 2, w1). where ε, w1 E R are part of the generative parameters 0, completes our dynamical model of knowledge states.
Learner-specific cognitive traits The dynamics of knowledge states (Eqs, 4- 6) are parameterized by learner-specifie cognitive traits (a ph, of, ). which we collectively denote s Specifically, a' represents the forgetting rate (Ebbinghaus, 1885; Averell & Hcathcote, 2011), μ² (via " captures long-term memory consolidation (Meeter & Murre, 2004) for practiced KCs and expected performance for novel KCs, of quantifies knowledge volatility, and 7 measures transfer abillity (Bassett & Mattar, 2017) from knowledge of prerequisite KCs. These traits can develop during learning according to Eq. 1, starting from Gaussian prior pe (sf) = [s Ri) where E IR and are also part the global parameters 0.
the diagonal matrices H, R1, R E R4X4
Shared prerequisite graph A. In our model, prerequisite relations influence knowledge dynamics via the coupling introduced in Eq. 5. We now discuss an appropriate parameterization for the weight matrix of the prerequisite graph, A := =aa ],,KE..KK We assume that prerequisites are time- and dearner-indepente
so that, in the spirit of collaborative filtering (Breese et al., 2013), we can pool evidence from all learners to estimate them. To prevent a quadratic scaling in the number of KCs, we do not directly model edge weights but derive them from KC embedding vectors n² in lower dimension n² E IR P with D < K, collected in embedding matrix UKXD- A basic integrity constraint for a connected pair is that dependence of KC on KC k should trade off against that of on i, i.e., no mutual prerequisites: 0² + a 1. With this in mind, we exploit the factorization of ak introduced by Lippe al. (2021) terms of separate probability of edge existence p(i o-ok) k) and definite directionality p(i k a-o k):
ik = p(i + k oo
sigmoid( (w') Tu²)) sigmoid((w')((M (M)M)
(7)
where the skew-symmetric combination M - M of a learnable matrix M prevents mutual prerequi- sites. Having presented the generative model, we now turn to inference and prediction.
3.2 APPROXIMATE BAYESIAN INFERENCE AND AMORTIZATION WITH A NEURAL NETWORK
We now describe how we learn the generative model parameters I and how we infer the latent states s introduced in Section 3.1 using a neural network ("inference network"). Since learner- specific latent states and z are deducible solely from limited individual data, we expect non- negligible uncertainty. This motivates our probabilistic treatment of these states using approximate Bayesian inference. By contrast, the model parameters 0 (KC parameters U, M in Eq. 7, transition parameters 8, H, R,, R in Eq. 1, and E, wi in Eq. 2) can be estimated from all learners, and we thus
Saäkkkä & Solin (2019)- the variance is wh, = (of.) -- 200
4

---

Published as a conference paper at ICLR 2024
treat them point-estimated parameters as described below (detailed derivation in Appendix A.I.) Here, without loss of generality, we show the inference for a single learner.
3.2.1 INFERENCE ON FIXED LEARNING HISTORY
Here, we assume the full interaction history H is available for inferring the posterior over latents pa pa(zf.v (zf.v.55 s v..)). We approach the problem using variational inference (VI). In VI, we select a distribution family do with free parameters ø to approximate the posterior pe by minimizing their Kullback-Leible divergence, This can only done indirectly, by maximizing lower bound to the marginal probability of the data, the evidence lower bound (ELBO). Here, we adopt the mean- field approximation la (2 .v= $1,N N Vi,N) g0(21.N 94 (0(( and jointly optimize the generative and variational & parameters using variational expectation muximization (EM; Dempster et al., 1977; Beal & Ghahramani, 2003; Attias, 1999). Motivated by real-world scalability, we introduce an inference network (see Appendix A.3 for the architecture) amortize the learning of variational parameters 6 across learners, and we employ the reparametrization trick (Kingma Welling, 2014) to optimize the single-learner ELBO:
ELBO' (0, Φ) = B po((iin) [- log 20 (=f.n) + log Po + En log (
+ Bo (= [- log go (o( log po (2 (z1) + En log pa (vh. ==="")]
(8) The SSM emissions and transitions were introduced in Eqs. 1-3, along with the respective initial priors. To allow for diversity of combinations of learner traits account for the data, model the variational posterior across learners, 1 ($1:N), ), as a mixture of Gaussians (see Appendix A.4).
+ +B (=f. x) no((
N log Po 2 =
3.2.2 INFERENCE IN CONTINUAL LEARNING
In real-world educational settings, KT model must flexibly adapt its current variational param- eters on with newly available interactions (a 1+ v Retraining on a fixed, augmented history H to obtain an updated Φn+1 is possible (Eq. 8), but expensive. Instead, PSI-KT, we use the parameters on of the current posterior gon (z sy) ) to form a next-time prior,
"u, =f Iv...) (Po s(.)) Po p= +1 | 1s
(9)
Due to the Bayesian nature our model, we can now update this prior with the new evidence yh+ at time t using variational continual learning (VCL; Nguyen et al., 2017; Loo et al., 2020), i.e., by maximizing the ELBO:
ELBO (0, Φu+1) = Egg.. (ff111 (- log Jon.r (0.=1(((+1)
+ +E Ronn (#f [ log 9o.t1 (2 1) + log Pe Pa (
Beeess (#f (h (log (2 s4+1)]-
(10)
Maximizing this ELBO allows to update the parameters on+1 based new interac- tion (1 tf 1 fh directly from the previous parameters Pn, i.e, without retraining.
3.3 PREDICTIONS
To predict a learner's performance on KC wh at (441, we take the current variational distributions over s and z and transport them forward by analytically convolving them with the respective transition kernels (Eqs. 1 and 2). We then draw =n+1
_f from the resulting distribution, and predict
the outcome Uh+1 by Eq. When predicting multiple steps ahead, we repeat this procedure without conditioning on any of the previously predicted 0h+m-
EVALUATIONS
We argue above that KT for personalized education must predict accurately, scale well with new data, and provide interpretable representations. We now empirically assess these desiderata, comparing PSI-KT with up 8 baseline models across three datasets from online education plat- forms. Concretely, evaluate (i) prediction accuracy,
Table 1: Dataset characteristics
Dataset +
Assist12
Assist17
Junyi15
Learners T.
46,674
1,709
247.606
Int's 10
0.9
26
5

---

Published as a conference paper at ICLR 2024
Assist12
Assist
JanyLls
PSEET
50 Acearacy
QIKT - GIT
DKTT DET
25
100
200
300
400
500
1000
9954
100
200
300
400
500
1000
1358
100
200
300
400
500
1000
62124
*Leammens
Leaanness
#Learners
Figure 2: Within-learner prediction performance (mean 5EM as function of cohort sizes from 100 to the maximum available in each dataset (we omit HLR for legibility; see Table 2.)
quantifying both within- prediction and between-learerr generalization (Sec. 4. 4.1), (ii) scalabil- ity in a continual learning setting (Sec. 4.2), and (iii) interpretability of learner representations and prerequisite relations (Sec. 4.3).
Datasets. Assistments and Junyi Academy are non-profit online learning platforms for pre-college mathematics. We use Assistments' 2012 and 2017 datasets (Assist12 and Assist17) and Junyi's 2015 dataset' (Junyi15; Chang et al., 2015), which in addition to interaction data, provides human- numan-annotated KC relations (see Table and Appendix A.3.2 for details).
We select HLR from Duolingo and PPE two influential psychologically -infolmedd regression models. From the models that use learnable representations, we include two established deep learning benchmarks, DKT and DKTF, which capture complex dynamics via LSTM networks, well as the interpretabiliy--orineedd QIKT.
4.1 PREDICTION AND GENERALIZATION PERFORMANCE
In our evaluations, mainly focus prediction and generalization when training 10 interactions from up to 1000 learners. Good KT performance with little data is key in practical ITS to minimize the number of learners on an experimental treatment (principle of equipoise, similar to medical research; Burkholder, 2021), to mitigate the cold-start problem, and to extend the usefulness of the model to classroom-size groups. To provide ITS with basis for adaptive guidance and long-term learner assessment, we always predict the 10 next interactions. Figure 2 shows that PSI-KT's within-learner prediction performance is robustly above baselines for all but the largest cohorts (>60k learners, Junyil5), where all deep learning models perform similarly. The advantage of PSI-KT comes from its combined modeling of KC prerequisite relations and individual learner traits that evolve in time (see Appendix Fig. 13 for ablations). The between-learner generalization accuracy of the models above, when tested on 100 out-of-sample learners, shown in Table 2, where fine-tuning indicates that parameters were updated using (10-point) learning histories from the unseen learners. PSI-KT shows overall superior generalization except on Junyi 5 (when fine-tuning).
4.2 SCALABILITY CONTINUAL LEARNING
In addition to training on fixed historical data, we also conduct experiments to demonstrate PSI-KT's scalability when iteratively retraining on additional interaction data from each learner. This parallels real-world educational scenarios, where learners are continuously learning (Sec. 3.2.2). Each model initially trained on 10 interactions from 100 learners. We then incrementally provide one data point from each learner, and evaluate the training costs and prediction accuracy. Figure 3 shows PSI-KT requires the least retraining time, retains the best prediction accuracy, and thus achieves the most favorable cost-accuracy trade-off (details in Appendix A.5.3).
4.3 INTERPRETABILITE OF REPRESENTATIONS
We now evaluate the interpretability of both learner-specific cognitive traits s and the prerequisite graphs A. We first show that our model captures leamer-speccfic and disentangled traits that correlate with behavior patterns. Next, we show that our inferred graphs best align with ground truth graphs, and the edge weights predict causal support on downstream KCs.
4.3.1 LEARNER-SPEON
COGNITIVE TRAITS
For each learner, PSI-KT infers four latent traits, each with a clear dynamical role specified by the OU process (Eqs. 5-6). In contrast, high-performance baselines (AKT, DKT, and DKTF) describe learners via 16-dimensional embeddings solely constrained by network architecture and loss mini- mization. Another model QIKT constructs 3-dimensional embedddigg with each element connected
tasttp://sites poogge..co//eee... site/ assistmentsssste Prttps://pp lddatahhop.p.ooo web. cmo .emu/edd/ieee To Td=1 198

---

Published as a conference paper at ICLR 2024
Table 2: Prediction accuracy. FT indicates additional fine-tuning and indicates larger values are better. The best model performance is in bold and the 2nd best is underlined.
Dataset
Experiment
HL.
PPE
DKT
DKTP
HKT
AKT
GKT
QIKT
PSI-KT
Within f
.44
.6
66
.0.01
.7. 02
63 00
.33
.68
Assisc12
Betweea
.0.0s
.50 02
.55 02
.51.01
.5400
.58,02
.61.02
.00
.61.03
.5,002
.53.00
-8.000 00
.55.o1
.55,00
on
.62.02
03
02
Within
A5.01
.53.02 oz
.57.o2
.53.03
.52.03
.56.02
.56.01
58
.63.02
Assist
Between
.33
.51
.1.00
.48
.47
.53 oz
50
.53
w/ FT
.11.04
.51 .00
.51,08
.53.01
.51,03
.51.22
54,00
.51044
.5.02
Within Between
.55.c2
.6
77
.88.1
63
81
.78
81
.83
Junyl15
A8.02
.55 0z
76.00
.76.22
.61.11
.73.01
.27 oz
76.03
.52.00
.6
81
.84.1
83.00
.79
80
.80
Assist12
Assis117
Juny115
1e3
163
PSIKT
Time
QUKT
.65
Accungy
20-
801
GKT
DKT
:50
.50
40
70
10
50
90
10
20
Data Points
Data Points
Data Points
Figure 3: Continual learning. (Top) Cumulative training time. (Bottom) Prediction accuracy the next 10 time steps. We omit results when time is above, or accuracy is below, the range of the axes.
to scores of knowledge acquisition, knowledge mastery, and problem-solving We collectively refer to these learner-specifie variables as learner representations. Here, empirically show that PSI-KT representations provide superior interpretability. We ask that learner representations be 1) specific to individual learners, 2) consistent when trained partial learning histories, 3) disentangled (i.e., component-wer wise meaningful, as in Bengio et al., 2013), and 4) and operationally interpretable, so that they can be used to personalize future curricula. We evaluate desiderata 1-3 with information- theoretic metrics (Table 3; see Appendix A.6 for details), and desidcratum 4 with regressions against behavioral outcomes (Table 4).
Specificity, consistency, and disentanglement. Learner representations s should be maximally spe- cific about learner identity f, which can be quantified by the mutual information MI(s; E) H(s) - H(s being high, where H denotes (conditional) entropy. Ad- ditionally, when we infer representations s from different subsets of the interactions of a fixed learner, they should be consistent, i.e., each sumb should be minimally informative about the chosen subset (av- eraged across subsets), such that E,,,, MI(s buuh) = Eeg, (H(+(e) (()))))))) should be low. Note that quential subsets are unsuitable for this evaluation, since representations evolve in time track learners' pro- gression. Instead, we define subsets as groups of KCs
Table 3: Specificity, consistency, and disen- tanglement vs. best baseline.
Matric
Dataset
Baseline
PSI-KT
Specificity
Assist12
8.8
8.4
Assist17
MI( 2)
Junyi15
13.5
14.4
Consistency-1 Ee,, Ml(e (
Assist12
12.2
7.4
Assis117
6,4
6,4
Junyi15
Disentanglement (=
Assis112
2.3
7.4
Assist17
8.4
Junyi15
5.0
11.5
whose average presentation time approximately uniform over the duration of the experiment (see Appendix 1. for details). Lastly, learner representations should be disentangled, such that each dimension individually informative about learner identity. We measure disentanglement with DRi. (si( == H(s) H(s (ddiagg form of specificity that ignores correlations across s dimensions by estimating the conditional entropy only with diagonal covariances.
In empirical evaluations (Table 3), PSI-KT's representations offer competitive specificity despite being diver-dimensional, and outperform all baselines consistency and disentanglement, While disentanglement aids interpretability (Freiesleben et al., 2022), it does not itself entail domain- specific meaning for representational dimensions. We now demonstrate that PSI-KT KT representations correspond clear behavioral patterns, which is crucial for future applications in educational settings.
Operational interpretability. Having shown that PSI-KT captures specific, consistent, and disentan- gled learner features, now investigate whether these features relate meaningful aspects of future
7

---

Published as a conference paper at ICLR 2024
behavior, which would be useful for scheduling operations for ITS. We indeed find that the learner representations of PSI-KT forecast interpretable behavioral outcomes, such as performance decay or initial performance on novel KCs. Concretely, consider the observed one-step performance difference Ay, == yn - We expect it to he lower for longer intervals ,. = in - 11 due to forgetting. However, we recognize no clear trend when plotting Ayh over F for the Junyi15 dataset (Fig. 4, top right). We can explain this observation because different learners forget on different time scales. Plotting the same test data instead over scaled intervals a (Fig. 4, top center) shows clear trend against an exponential fit (solid line) with less variability, demonstrating that of, (derived from past data only) adjusts for individual learner characteristics and can be interpreted as a personalized rate of forgetting. Here, the choice of the factor af, is motivated by our inductive bias (Eq. 4). The trend is much less clear for all baselines: Fig. 4 (top left) uses the best fitting component across all learner representations from all baselines (full results in Fig. 8 in Appendix A.6.4). Analogously, when we consider initial performance on novel KC, find for PSI-KT that T (which aggrregates mastery of prerequisites for KC at time iu. see Eq. 5) explains it better than the best baseline Fig. 4 (bottom panels). Table 4 shows that these superior interpretability results are significant and hold across all datasets. In Appendix A.6.4, we discuss two more behavioral signatures (performance variability and prerequisite influence) and show they correspond to the remaining components % and on-
4.3.2 PREREQUISITE GRAPH
PSI-KT infers a prerequisite graph based on all learners' data, which helps it to generalize to un- seen learners. Beyond helping prediction, reliable prerequisite relations are an essential input for curriculum design, motivating our interest in their interpretability. Figure 5a shows an exemplary inferred subgraph with the prerequisites of a single KC. To quantitatively evaluate the graph, we (i) measure the alignment of the inferred vs. ground-truth graphs and (ii) correlate inferred prerequisite probability with Bayesian measure of causal support obtained from unseen behavioral data.
Alignment with ground-truth graphs. We analyze the Junyi15 dataset, which uniquely provides numan-annotane evaluations of prerequisite and similarity relations between KCs. We discuss here the alignment of prerequisites and leave similarity for Appendix A.7. The Junyi15 dataset provides both expert-identified prerequisite for each KC,and crowd-sourced ratings (6.6 ratings on average on a 1-9 scale). To compare with expert annotations, we compute the rank of each expert-identified prerequisite relation i > k in the relevant sorted list of inferred probabilities {a**} } and take the harmonic average (mean reciprocal rank, MRR; Yang et al., 2014). Next, we compute the negative log- likelihhood (nL.L) of inferred edges s* using a Gaussian estimate of the (rescaled) nowresourced ratings for the i - 1 KC pair. We finally calculate the Jaccard similarity (JS) between the set of inferred edges (a > 0.5) and those identified by experts as well as crowd-sourced edges with average ratings above 5. The results in Table 5 (left columns) consistently highlight PSI-KT's superior performance across all criteria (see Appendix A.7.1 for details).
Causal support across consecutive interactions. For education applications, we are interested in how KC dependencies impact learning effectiveness. If KC i is a prerequisite of KC k. mastering KC i contributes to mastering KC k. indicating causal connection. In this analysis, we show that inferred edge probabilities a² (Eq. 7) correspond to causal support, (Eq. 11), derived from behavioral data through Bayesian causal induction (Griffiths & Tenenbaum, 2009). Specifically, model the relationship between a candidate cause C' and effect E, i.e., a pair of KCs in our case, while accounting for a constant background cause B, representing the learner's overall ability and the influences of other KCs. We consider two hypothetical causal graphs, where Graph 0 G represents
Por Rerfemenc Diffreaxe
06
AKT
PSL-KT
.od
Raw Data
Behavioural signature
Dataset
Best Raseline
PSI-KT
2.8 .0
Assist12
0.01, .67
0.30. <000
Scaled Inserval C4
Scalled of
Unscaled Interval
Performance difference
Assist17
-0.03, 30
0.56, <001
Junyil5
0.03, .06
0.72, <001
Inibal Pedfemancc
Binned
Initial
Assist12
0.04, .01
0.54, <001
Regression
performance
Assist17
0.05, 001
3,70, <001
.5
Junyll5
0.04, 002
0.92, <001
95%
Best Dimensiun!?" (s')
Long tern Mea g
Figure 4: Operational interpretability of represen- tations, Junyil5 dataset. See text for axes labels and Appendix A.6.4 for additional results.
Table 4: Coefficients and p-values of regres- sions relating exp( (a 7,) and 7 to unseen behavioral data across datasets.
8

---

Published as a conference paper at ICLR 2024
Table 5: (Left) Alignment of inferred graphs with annotated graphs for the Junyi dataset. (Right) Re- gression coefficients and p-values relating causal support to inferred edge probabilities. All baseline models either lack significance or negatively predict causal support (Appendix Fig. 12).
Metric
MRR
expert 1
crowd 1
coefficient f. p-value
Datasel
Junyi15
Assist12
Assis117
Junyil5
Best Baseline
00822 .086
.015 .0119
.047 .0955
3,03 4.11
1.05, .253
0,22, .792
0.42, .593
PSI-KT
1.15, .003
0.28, <0001
0.97.<0001
(9)
multiplication 0.5
area unit cm square
trangll length of side
py thigggorenn theorem
(b)
Grmph
Graph
0.753
0.731
0.654
0,5K3
0.556
addition
node
0.842
0.712
of parallelograms
perimeter combined shapes
area
Bianed mecn +SB Regression line 99% (3)
4.5
HKT
3
AKT
HKT
Assist12
Assist17
Junyi15
Clusal Support
1.5 .35
.55
0
.14
0.2
0.3 0.1
0.5
0.6
0.7
0.8
0.9
0.3 0.4
0,5
0.6
0.7
0.0
0.1
0.2
0.3
0.1
0.5
0.6
Inferred Edge Probabiliries
Inferred Edge Prohaliiiiiiss
Inferred Edge Probabilities
Figure 5: Graph interpretability. (a) Subgraph inferred by PSI-KT on the Junyi dataset, showing prerequisites of target KC 'area of parallelograms'. (b) Hypothesized causal graphs, where Graph 1 assumes causal relationship exists from KC to KC k, while Graph 0 is the null hypothesis. (c) Regression of edge probabilities against causal supports. Insets show the best baseline model.
the null hypothesis of no causal relationship, and Graph Gi-sk assumes the causal relationship exists, i.e. correct performance on KC i causally supports correct performance on KC & (Fig. 5b). We estimate causal support for each pair of KCs behavioral data H. from KC i at time f, to KC k at time 1x+1. as a function of the difference in log-likeliihe ligliikels of the two causal graphs (see Appendix A.7.3 for details):
& based on all consecutive interactions the
(11) We then use regression to predict support,-> as function of edge probabilities a inferred from different models. The results are visualized in Figure 5c and summarized in Table 5 (right). The larger coefficients indicate that our inferred graphs possess superior operational interpretability (Sec. 4.3).
support, := log P( Gx++)) - log P(H G.-+).
DISCUSSION
We propose PSI-KT as a novel approach to knowledge tracing (KT) with compelling properties for intelligent tutoring systems: superior predictive accuracy, excellent continual-leaming scalability, and interpretable representations of learner traits and prerequisite relationships. We further find that PSI- KT has remarkable predictive performance when trained on small cohorts whereas baselines require training data from at least 60k learners to reach similar performance. An open question for future KT research is how to combine PSI-KT's unique continual learning and interpretability properties with performance that grows beyond this extreme regime. We use an analytically marginalizable -hitebbe process for knowledge states in PSI-KT, resulting in an exponential forgetting law, similar to most recent KT literature. Future work should support ongoing debates in cognition by offering alternative modeling choices for memory decay (c.g., power-law; Wixted & Ebbesen, 1997), thus facilitating empirical studies at scale. And while our model already normalizes reciprocal dependencies in the prerequisite graph, we anticipate that enforcing regional or global structural constraints, such as acyclicity, may benefit inference and interpretability. Although we designed PSI- KT with general structured domains in mind, our empirical evaluations were limited to mathematics learning by dataset availability. We highlight the need for more diverse datasets for structured KT research to strengthen representativeness in ecologically valid contexts. Overall, our work combines machine learning techniques with insights from cognitive science to derive predictive and scalable model with psychologically interpretable representations, thus laying the foundations for personalized and adaptive tutoring systems.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors thank Nathanael Bosch and Tim Xiao for their helpful discussion, and Seth Axen for code review. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Hanqi Zhou. This research was supported as part of the LEAD Graduate School & Research Network, which is funded by the Ministry of Science, Research and the Arts of the state of Baden Württemberg within the framework the sustainability funding for the projects of the Excellence Initiative II. Funded by the Deutsche Forschungsgemmiisshall (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXC number 2064/1 Project number 390727645. CMW is supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S 18039A
ETHICS STATEMENT
We evaluated our PTI-KT model on three public datasets from human learners, which all anonymize the data to protect the identities of individual learners. Although PSI-KT aims to improve personal- ized learning experiences, it infers cognitive traits from behavioral data instead of using learners' demographic characteristics (e.g., age, gender, and the name of schools provided in the Assistment 17 dataset), to avoid reinforcing existing disparities.
Evaluations of structured knowledge tracing our paper are limited by dataset availability to pre- college mathematics. To ensure broader and more ecologically valid assessment, it is essential to explore diverse datasets across various domains (e.g., biology, chemistry, linguistics) and educational stages (from primary to college level). This will allow for a more comprehensive understanding of the role structure in learning.
REFERENCES
Terry A Ackerman. Multidimensional item response theory models. Wiley StatsRef: Reference Online, 2014.
Statistics
Hagai Attias. A variational bayesian framework for graphical models. Advances in neural information processing systems, 12, 1999.
Lee Averell and Andrew Heathcote. The form of the forgetting curve and the fate of memories. Journal of Mathematical Psychology, 55:25-35, (2 2011.
Ryan SJ d Baker, Albert T Corbett, and Vincent Aleven. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Intelligent Tutoring Systems: 9th International Conference, ITS 2008, Montreal, Canada, June 23-27, 2008 Proceedings 9, pp. 406-415. Springer, 2008
Danielle S Bassett and Marcelo Mattar. A network neuroscience human learning: potential to inform quantitative theories of brain and behavior. Trends in cognitive sciences, 21(4):250-264, 2017.
Matthew J Beal and Zoubin Ghahramani. The variational bayesian em algorithm for incomplete data: with application scoring graphical model structures. Bayesian statistics, 7:453-466 2003.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new
perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 2013.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):8598877 2017.
Brändle, Binz, and E Schulz. Exploration beyond bandits. In The Drive for Knowledge: The Science of Human Information-Seeking pp. 147-168. Cambridge University Press, 2022.
John S Breese, David Heckerman, and Carl Kadie. Empirical analysis of predictive algorithms for collaborative filtering. arXiv preprint arXiv: 1301,3333 2013.
Leslie Burkholder. Equipoise and ethics in educational research. Theory and Research in Education, 1971):65-77, 2021. doi: 10. (0.1177/1477878521 1009105.
10

---

Published as a conference paper at ICLR 2024
Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen. Modeling exercise relationships in learning: A unified approach. In EDM, pp. 532-535, 2015.
Jiahao Chen, Zitao Liu, Shuyan Huang, Qiongqiong Liu, and Weiqi Luo. Improving interpretability of deep sequential knowledge tracing models with question-centric cognitive representations. arXiv preprint arXiv:2.30..00 06885, 2023.
Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Back, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae, and Jaewe Heo. Towards an appropriate query, key, and value computa- tion for knowledge tracing. In Proceedings of the seventh ACM conference on learning@ @ scale, pp. 341-344, 2020.
Albert T Corbett and John R Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4:253-278, 1994.
Ronald Dekker, Fabian Otto, and Christopher Summerfield. Curriculum learning for human compo- sitional generalization. Proceedings of the National Academy of Sciences, 119(41):2225582119 19, 2022.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (methodological), pp. 1-38, 1977.
John Dewey. The child and the curriculum. University of Chicago Press Chicago, 1910.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autocncoders, arXiv preprint arXiv:1611.02648 2016.
H. Ebbinghaus. Über das Gedächtnis: Untersuchungen zur experimentellen Psychologie. Duncker Humblot, Leipzig, 1885.
Timo Freiesleben, Gunnar König, Christoph Molnar, and Alvaro Tejero-Cantera Scientific inference with interpretable machine learning: Analyzing models to learn about real-world phenomena. arXiv preprint arXiv:2206005487. 2022.
Theophile Gervet, Ken Koedinger, Jeff Schneider, Tom Mitchell, al. When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining, 12(3):31-54, 2020.
Aritra Ghosh, Neil Heffernan, and Andrew S Lan. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2330-22 2339, 2020.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
Thomas Griffiths and Joshua B Tenenbaum. Structure and strength in causal induction. Cognitive psychology, 51(4):334-384, 2005.
Thomas Griffiths and Joshua Tenenbaum. Theory-based causal induction. Psychological review, 116(4):661, 2009.
Lilian H Hill. Concept mapping encourage meaningful student learning. Adult Learning, 16(3-4): 7-13, 2005.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Eric Jang, Shixiang Gu, and Poole. Categorical reparameterization with gumhel-softmax. arXiv preprint arXiv: 1.0/ 144, 2016.
Elisabeth A Karuza, Sharon L Thompson-Schie and Danielle S Bassett. Local patterns to global architectures: influences of network topology on human learning. Trends in cognitive sciences, 20 (8):629-640, 2016.
11

---

Published as a conference paper at ICLR 2024
Tanja Käser, Severin Klingler, Alexander G Schwing, and Markus Gross. Dynamic bayesian networks for student modeling. IEEE Transactions on Learning Technologies, 10(4):450-44 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2649-2658. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.0 6980, 2014.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint ar 10096600.2222 0099002907, 2016.
Ismo T Koponen and Maija Nousiainen. Concept networks of students' knowledge of relationships between physics concepts: Finding key concepts and their epistemic support. Applied network science, 3(1):1-21, 2018.
Robert Lindsey, Jeffery Shroyer, Harold Pashler, and Michael C Mozer. Improving students' long-term knowledge retention through personalized review. Psychological science, 25(3):639-66) 647, 2014.
Phillip Lippe, Taco Cohen, and Efstratios Gavves, Efficient neural causal discovery without acyclicity constraints. arXiv preprint arXiv:2107.10483 2021.
Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, and Weiqi Luo. simplekt: a simple but tough-to-beat baseline for knowledge tracing. arXiv preprint arXiv:2302.06888 06880. 2023.
Ting Long, Yunfci Liu, Jian Shen, Weinan Zhang, and Yong Yu. Tracing knowledge state with individual cognition and acquisition estimation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 173-182, 2021.
Noel Loo, Siddharth Swaroop, and Richard E Turner. Generalized variational continual learning. arXiv preprint arXiv:2011.12328. 2020.
Frederic M Lord. Applications of item response theory to practical testing problems. Routledge, 2012.
Christopher W Lynn and Danielle S Bassett. How humans learn and represent networks. Proceedings of the National Academy of Sciences, 117747)29407-29415, 2020.
Martijn Meeter and Jaap MJ Murre. Consolidation of long-term memory: evidence and alternatives. Psychological Bulletin, 130(6):843, 2004.
Sein Minn, Jill-Jênn Vie, Koh Takeuchi, Hisashi Kashima, and Feida Zhu. Interpretable knowledge tracing: Simple and efficient student modeling with causal relations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 12810-12818, 2022.
Koki Nagatani, Qian Zhang, Masahiro Sato, Yan-Ying Chen, Francine Chen, and Tomoko Ohkuma. Augmenting knowledge tracing by considering forgetting behavior. In The world wide web conference, pp. 3101-3107, 2019.
Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. Graph-based knowledge tracing: modeling student proficiency using graph neural network. In /EFF/WIC/ACM International Conference on Web Intelligence, pp. 156-163. 2019.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint
2017.
Shalini Pandey and George Karypis. A aelf-attentive model for knowledge tracing. arXiv preprint arXiv:1907.06833 2019.
12

---

Published as a conference paper at ICLR 2024
Philip I Pavlik, Luke G Eglington, and Leigh M constrained framework for learner modeling. IEEE Transactions on Learning Technologies, 14(5): 624-639, 2021.
Logistic knowledge tracing: A
Philip Pavlik Jr, Hao Cen, and Kenneth R Koedinger. Performance factors analysis-a new alternative to knowledge tracing. Online Submission, 2009.
Jean Piaget. Science of education and the psychology of the child. trans. coltman. 1970.
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, and Jascha Sohl-Dickster Deep knowledge tracing. Advances in neural information processing systems, 28, 2015.
Sidney Pressey. simple apparatus which gives tests and scores-and teaches. Sch. & Soc., 23: 373-376, 1926.
David E Rumelhart. Schemata: The building blocks of cognition. In Theoretical issues in reading comprehension, pp. 33-58. Routledge, 2017.
Michael Sao Pedro, Ryan Baker, and Janice Gobert. Incorporating scaffolding and tutor context into bayesian knowledge tracing to predict inquiry skill acquisition. In Educational Data Mining 2013. Citeseer, 2013.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Douglas Selent, Thanaporn Patikorn, and Neil Heffernan. Assistments dataset from multiple random- ized controlled experiments. In Proceedings of the Third (2016) ACM Conference Learning @ Scale, pp. 181-184, 2016.
Burr Settles and Brendan Meeder. A trainable spaced repetition model for language learning. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 1848-1858, 2016.
Shuanghong Shen, Qi Liu, Enhong Chen, Zhenya Huang, Wei Huang, Yu Yin, Yu Su, and Shijin Wang. Learning process-consistent knowledge tracing. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery data mining, pp. 1452-1460, 2021.
Yuge Shi, Brooks Paige, Philip Torr, al. Variational mixture-of experts autoencoders for multi- modal deep generative models. Advances in Neural Information Processing Systems, 32, 2019.
Dongmin Shin, Yugeun Shim, Hangyeol Yu, Seewoo Lee, Byungsoo Kim, and Youngduck Choi. Saint+: Integrating temporal features for ednet correctness prediction. In LAK21: 11th International Learning Analytics and Knowledge Conference, pp. 490-496, 2021.
Shiwei Tong, Qi Liu, Wei Huang, Zhenya Hunag, Enhong Chen, Chuanren Liu, Haiping Ma, and Shijin Wang. Structure-based knowledge tracing: An influence propagation view. In 2020 IEEE international conference on data mining (ICDM), pp. 541-550. IEEE, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Matthew M Walsh, Kevin Gluck, Glenn Gunzelmann, Tiffany Jastrzembski, and Michael Krusmark. Evaluating the theoretic adequacy and applied potential of computational models of the spacing effect. Cognitive science, 42:644-691, 2018.
Chenyang Wang, Weizhi Ma. Min Zhang, Chuancheng Lv, Fengyuan Wan, Huijie Lin, Taoran Tang, Yiqun Liu, and Shaoping Ma. Temporal cross-effects in knowledge tracing. In Proceedings the 14th ACM International Conference on Web Search and Data Mining, pp. 517-525, 2021.
Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel HomanddzzLLobatee Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, Simon Woodhead, and Cheng Zhang. Diagnostic questions: The neurips 2020 education challenge. arXiv preprint 200i /2061, 2020.
13

---

Published as a conference paper at ICLR 2024
John T Wixted and Ebbe B Ebbesen. Genuine power curves in forgetting: A quantitative analysis of individual subject forgetting functions. Memory cognition, 25:731-739, 1997.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575. 2014.
Michael Yudelson, Kenneth R Koedinger, and Geoffrey Gordon. Individualized bayesian knowledge tracing models. In Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA, July 9-13, 2013. Proceedings 16, pp. 171-180. Springer, 2013.
14


Reviews:
[{"review_id": "42BDtL4fT9", "reviewer": "Reviewer_rAUs", "summary": "The paper presents PSI-KT, a novel knowledge-tracing model that combines individual learning dynamics with structural influences from prerequisite relationships. PSI-KT uses Bayesian inference to model learner-specific cognitive traits and shared prerequisite graphs. Evaluated on real educational datasets, PSI-KT achieves superior predictive accuracy and scalability while also providing interpretable representations of learners and knowledge structure. The model helps advance personalized intelligent tutoring systems by combining insights from cognitive science and machine learning. PSI-KT demonstrates how explicitly modeling psychological principles within AI systems can enhance performance and interpretability.", "questions": "* Could you provide insights into the dataset limitations and discuss potential challenges in applying the model to other educational domains or datasets?\n* Have you considered exploring more complex forgetting functions based on memory research literature to improve long-term predictions?\n* Could you perform ablation studies to dissect the contributions of different model components to predictive accuracy, providing insights into the model's strengths?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The model is designed based on psychological principles and evaluated on multiple datasets. The experiments demonstrate predictive accuracy, scalability, and interpretability. The paper is technically strong in its probabilistic modeling and inference methodology.\n* The paper is well-written and provides intuitive explanations of the model components. The background gives a clear overview of knowledge tracing and related work.\n*The model advances knowledge tracing for intelligent tutoring systems by enhancing predictive accuracy, scalability, and interpretability. The interpretable representations of learners and knowledge structure provide an important basis for personalized education. The integration of cognitive science and AI is significant for developing systems that leverage psychological insights.", "weaknesses": "* The evaluations focus on three specific educational datasets. Testing on a more diverse range of datasets could better reveal the model's capabilities and limitations. The authors could discuss what other domains or data characteristics pose challenges.\n* Long-term retention modeling could be enhanced. The current exponential decay may be simplistic. Exploring more complex forgetting functions based on memory research literature could improve long-term predictions.\n* While superior overall, some accuracy metrics are comparable to certain baselines. Further ablation studies could provide insight into which model components contribute most to accuracy gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "flFsKXdqDQ", "reviewer": "Reviewer_J4qa", "summary": "This paper proposes a probabilistic state-space generative approach named PSI-KT by explicitly modeling individual cognitive traits and shared knowledge graph of prerequisite relationships to achieve predictive, scalable and interpretable knowledge tracing, inspired by cognitive science and pedagogical psychology. The author conducts extensive experiments on three datasets to demonstrate that PSI-KT can achieve superior predictive accuracy, scalable inference in continual-learning settings, and interpretability of learners’ cognitive traits and prerequisite graphs. The paper’s contributions are as follows:\n\n1.The paper proposes a novel hierarchical probabilistic state-space model for knowledge tracing by introducing individual cognitive traits and prerequisite shared knowledge graph.\n\n2.Unlike recent discriminative KT models that utilize cross-entropy loss, PKI-KT distinguishes itself by introducing a psychologically-inspired probabilistic generative model, which leverages approximate Bayesian inference and variational continual learning techniques for model optimization.\n\n3.Extensive experiments demonstrate that PKI-KT achieves impressive results in multi-step predictive accuracy and scalable inference in continual-learning settings. Moreover, novel confirmatory experiments further validate the specificity, consistency, disentanglement, and operational interpretability of individual cognitive traits, as well as the reliability of the inferred prerequisite graph.", "questions": "1.Could the authors provide some explanations about the four dimensions of cognitive traits and how they represent specific characteristics of students? It would be particularly helpful if these dimensions can be correlated with concepts from cognitive science. Additionally, I'm interested in an experimental analysis of the impact of the other two dimensions.\n\n2.Have the authors considered supplementing with essential ablation study and adding baseline models that explicitly take into account the knowledge graph structure, such as GKT[1] or SKT[2], the latter of which also considers prerequisite relationship between concepts?\n\n[1] Nakagawa, Hiromi, Yusuke Iwasawa, and Yutaka Matsuo. \"Graph-based knowledge tracing: modeling student proficiency using graph neural network.\" IEEE/WIC/ACM International Conference on Web Intelligence. 2019.\n\n[2] Tong, Shiwei, et al. \"Structure-based knowledge tracing: An influence propagation view.\" 2020 IEEE international conference on data mining (ICDM). IEEE, 2020.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Good textual expression, mathematical notation, and formula derivations. The paper provides a clear description of motivation, problem definition, and experimental setup, along with professionally presented mathematical expressions.\n\n2.The motivation is both novel and reasonable. PSI-KT takes into account students’ individual cognitive traits and the prerequisite knowledge graphs while modeling students' knowledge states.\n\n3.The proposed method is intriguing. PSI-KT applies a Probabilistic State-Space Model to model students' knowledge states in KT. It introduces a three-level hierarchical structure, utilizes approximate Bayesian inference for generating students' knowledge states and cognitive traits, and optimizes model parameters using the Evidence Lower Bound (ELBO) instead of the common cross-entropy used in recent discriminative KT models.\n\n4.The paper includes extensive confirmatory experiments with detailed and favorable results. In addition to conducting rich experiments on predicting student performance in both within-learner and between learner settings, the authors also carries out numerous analytical validation experiments concerning the representation of cognitive traits and the inferred knowledge prerequisite relationships, all of which have yielded positive outcomes.", "weaknesses": "1.The cognitive traits in the paper lack somewhat interpretability. While the authors have conducted extensive validation experiments on the representation of cognitive traits, considering that the paper introduces cognitive traits from the perspectives of cognitive science and psychology, it is advisable to explicitly state in the text which specific cognitive psychology traits the four dimensions of cognitive traits represent. This would help readers better understand the meaning and significance of these traits.\n\n2.Experiments are somewhat insufficient. Although the authors have conducted an extensive array of analytical and validation experiments, there is a notable absence of ablation study to demonstrate the effectiveness of the two proposed motivations in the paper, namely cognitive traits and the prerequisite relationship graph, on PSI-KT. Furthermore, given the mention of the use of the prerequisite graph in the paper, it seems somewhat inadequate not to include some explicit baseline models that utilize knowledge concept graphs for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NCFVktUB0W", "reviewer": "Reviewer_k3Pn", "summary": "This paper introduces PSI-KT, a generative knowledge tracing method that places emphasis on predictive accuracy, scalable inference, and interpretability. PSI-KT models both the cognitive processes of students and the underlying knowledge prerequisite structure. Extensive experimental results clearly showcase the method's superiority over various baselines from multiple angles.", "questions": "1. Why did the authors choose to experiment with only a limited portion of the datasets? The explanation provided, \"to simulate real-world data constraints in education,\" may benefit from additional clarification.\n2. Could the authors consider using more recent interpretable deep learning methods like QIKT as their baseline comparisons? Doing so could enhance the credibility of the study.\n3. Is there a specific reason why the authors did not provide case studies to visually demonstrate the model's interpretability, as has been done in previous KT research?\n4. Could the authors elaborate on the detailed rationale behind using mutual information between PSI-KT's learned parameters and the hidden vectors of baselines to measure interpretability? Further explanation would enhance the understanding of the experiments.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is designed carefully and comprehensive, focusing on mutiple perspectives of the knowledge tracing task.\n2. The motivation is meaningful that this paper focus on the pain point about interpretability of the knowledge tracing field.\n3. The paper is well-structured.", "weaknesses": "1. The method's description is not sufficiently clear. As indicated in the appendix, PSI-KT also employs neural networks to generate cognitive parameters. However, the main body of the paper only briefly touches upon this aspect, potentially leading to the misconception that PSI-KT is not a deep learning approach.\n2. The experimental setup lacks persuasiveness. As demonstrated in Table 1, two datasets contain over 10,000 learners, yet the authors chose to use only 100-1,000 learners as training data. Conducting experiments with a small dataset may unfairly disadvantage deep learning baselines, which can effectively leverage the abundance of available data. The reasoning provided, \"to simulate real-world data constraints in education,\" may not hold in the context of the vast amount of student learning data generated today.\n3. The introduction of interpretable KT methods is not comprehensive. For instance, recent approaches like IKT, ICKT, and QIKT [1, 2, 3] incorporate interpretable psychological and cognitive modules into their methods. These relevant methods are not referenced in this paper, let alone included as baselines in the experiments.\n4. The assessment of the model's interpretability is not entirely convincing. The limited dimensionality of hidden learner representations in deep learning methods (e.g., DKT, AKT) at just 16 may constrain the neural networks' capabilities. Furthermore, there is no supporting evidence indicating that the learner representations of PSI-KT and these deep learning baselines capture the same underlying student features, making direct comparisons less rational.\n5. Perhaps conducting case studies of PSI-KT could offer a more intuitive understanding of its interpretability, such as visualizing trends in students' knowledge mastery, as shown in Figure 1(a).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ojxgiCbGP", "reviewer": "Reviewer_da66", "summary": "This paper constructs a scientifically sound model for the knowledge tracing problem that takes into account past performance, prerequsite knowledge graphs, and individual learner traits. They compare this to a number of other methods for predicting learner performance using public data and exceed the baseline.", "questions": "I would also like to more details on the datasets, particularly from the perspective of diversity. Claims about educational effectiveness and knowledge graphs that do no reflect a sufficient cross section are suspect at best and can be actively harmful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Predictive Accuracy was reasonable and well evaluated. The data used in the experiments was relevant and allowed reasonable evaluation. The graphs and tables produced were helpful in following the results. \n\nIn terms of the 4 primary dimensions used for an ICLR Review\n\n- Originality: Combining knowledge tracing and knowledge mapping into one method is a nice combination of ideas into one framework. \n\n- Quality: Quality was good, useful data of a reasonable size with a good baseline of comparison to other methods. In terms of basic accuracy this was well presented.\n\n- Clarity: The presentation overall left a lot to be desired in this paper but the graphs and tables were \n\n- Significance: The primary significance of these results is in the interpretability of the results.", "weaknesses": "Most of the focus of this paper was on the accuracy. Interpretability and scalability were not well evaluated and much of that was in the form of \"correct by construction\". \n\nThe prerequisite graph was interesting, although the correctness of the graph was not well quantified. \n\nAnd although I thought the accuracy beat the provided baseline and had sufficient data to support that, I do not think the results are good, only that they are better than the baseline. For a binary problem, getting accuracy of 55-80 is not a strong result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)