simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing
Authors: Zitao Liu
Qiongqiong Liu
Jiahao Chen
Shuyan Huang
Weiqi Luo
Venue: ICLR 2023 poster (2023)
Keywords: knowledge tracing
assessment
ai for education

Abstract: Knowledge tracing (KT) is the problem of predicting students' future performance based on their historical interactions with intelligent tutoring systems. Recently, many works present lots of special methods for applying deep neural networks to KT from different perspectives like model architecture, adversarial augmentation and etc., which make the overall algorithm and system become more and more complex. Furthermore, due to the lack of standardized evaluation protocol \citep{liu2022pykt}, there is no widely agreed KT baselines and published experimental comparisons become inconsistent and self-contradictory, i.e., the reported AUC scores of DKT on ASSISTments2009 range from 0.721 to 0.821 \citep{minn2018deep,yeung2018addressing}. Therefore, in this paper, we provide a strong but simple baseline method to deal with the KT task named \textsc{simpleKT}. Inspired by the Rasch model in psychometrics, we explicitly model question-specific variations to capture the individual differences among questions covering the same set of knowledge components that are a generalization of terms of concepts or skills needed for learners to accomplish steps in a task or a problem. Furthermore, instead of using sophisticated representations to capture student forgetting behaviors, we use the ordinary dot-product attention function to extract the time-aware information embedded in the student learning interactions. Extensive experiments show that such a simple baseline is able to always rank top 3 in terms of AUC scores and achieve 57 wins, 3 ties and 16 loss against 12 DLKT baseline methods on 7 public datasets of different domains. We believe this work serves as a strong baseline for future KT research. Code is available at \url{https://github.com/pykt-team/pykt-toolkit}\footnote{We merged our model to the \textsc{pyKT} benchmark at \url{https://pykt.org/}.}.

Published as conference paper at ICLR 2023
SIMPLEKT: A SIMPLE BUT TOUGH-TO-BEAN BASE- LINE FOR KNOWLEDGE TRACING
Zitao Liu
Guangdong Institute of Smart Education, Jinan University, Guangzhou, China liuzitaoe jnu. edu, en
Qionggiong Liu, Jiahao Chen, Shuyan Huang"
TAL Education Group, Beijing, China [liugionggiongl, chenjiahao, huangshuyan)@tal.com
Weiqi Luo
Guangdong Institute of Smart Education, Jinan University, Guangzhou, China 1wge cou.edu...
ABSTRACT
Knowledge tracing (KT) is the problem of predicting students' future performance based their historical interactions with intelligent tutoring systems. Recently, many works present lots special methods for applying deep neural networks to KT from different perspectives like model architecture, adversarial augmen- tation and etc., which make the overall algorithm and system become more and more complex. Furthermore, due to the lack standardized evaluation proto- col (Liu et al. (002)). there is no widely agreed KT baselines and published ex- perimental comparisons become inconsistent and coff-contradictory. i.c., the ported AUC scores of DKT on mSS1STments20099 range from 0.721 to 0.821 (Minn et al. 2018 Yeung & Yeung 2018 Therefore, in this paper, we provide strong but simple baseline method deal with the KT task named SIMPLEKT. Inspired by the Rasch model in psychometrics, we explicitly model question- specific variations to capture the individual differences among questions cover- ing the same set of knowledge components that generalization of terms of concepts or skills needed for learners to accomplish steps in task or a problem. Furthermore, instead of using sophisticated representations to capture student for- getting behaviors,
the ordinary dot-product attention function to extract time-aware information emhedded in the student learning interactions. Ex- tensive experiments show that such a simple baseline is able to always rank top in terms of AUC scores and achieve 57 wins, ties and 16 loss against 12 DLKT baseline methods on public datasets of different domains. We believe this work serves strong baseline for future KT research. Code is available at https::/githinn cwm/pykt
http:/ykkttttteete -eatttteaamaae wtwpm///kktoooooitooo
1 INTRODUCTION
Knowledge tracing (KT) is sequential prediction task that aims to predict the outcomes students over questions by modeling their mastery of knowledge, i.e., knowledge states, as they interact with learning platforms such massive open online courses and intelligent tutoring systems, shown in Figure D Solving the problems may help teachers better detect students that need further attention, or recommend personalized learning materials to students.
The KT related research has been studied since 1990s where Corbett and Anderson, to the best of our knowledge, were the first estimate students' current knowledge with regard to each individ-
"The corresponding author: Shuyan Huang.
We merged our model to the PYKT benchmark at altttps://pytt..rr

---

Published as conference paper at ICLR 2023
ual knowledge component (KC) (Corbeut & Anderson 1994). A KC is a description of mental structure or process that learner uses, alone in combination with other KCs, to accomplish steps in a task or a problen F Since then, many attempts have been made to solve the KT problem.
such as probabilistic graphical models (Käser et al. (017) and factor analysis based models (Cen eet al. 20066] Lavoué et 2018: Thai-Nghe al.) 2012). Recently, with the rapid development of
deep neural networks, many deep learning based knowledge tracing (DLKT) models are developed,
such as auto-regressive based deep sequential KT models (Piech et al. 2015, Yeung & Yeung. 2018)
Chen et al. 2018 Wang al. (2019; Guo et al. 2021; |021,Long et | Chen al.
2023), memory-
augmented KT models (Zhang ct al. [017 Abdelrahman & Wang [2019] Yeung 2019). attention
based KT models (Pandey & Karypis, 2019 Pandey & Srivastava] 2020 Choi et al. 2020 Ghosh
et a 2020 Pu et al. 2020). and graph based KT models (Nakagawa et al.
2019 Yang et al.
2020
Tong et al. 2020).
Knowledge Components
C
C3
2
Knowledge Compunent (KC)
C
Integer Distributive Law
C2
Multi-digit Calculation
Questions
42
4
4A
45
C3
Mixed Basic Operations
Response
Student
?
C4
Common Factor in Integers
XC illustrative examples
Figure 1: Graphical illustration of the task of Knowledge Tracing." "V and is answered correctly and incorrectly.
denote the question
Although DLKT approaches have constituted new paradigms of the KT problem and achieved promising results, recently developed DLKT models seem to be more and more complex and re- semble each other with very limited nuances from the methodological perspective: applying differ- ent neural components capture student forgetting behaviors (Ghosh et all. 2020 Nagatani al..
2019), recency effects (Zhang et al. (02T)), and various auxiliary intormation including relations be-
tween questions and KCs (Tong el al. [2020/Pandde & Srivastava 2020 Liu et al, 202 Yang el al.
2020), question text content (Luu et al. 2019 Wang et 20220l, question difficulty level (L al.
2021: Shen et al., 2022), and students' learning ability (Shen et al., 2020). Furthermone, published
DLKT baseline results surprisingly diverge. For example, the reported AUC scores of DKT and
AKT on 209 range from 0.721 to 0.821 (Minn al. 2018: Yeung & Yeung 2018 and from 0.747 to 0.835 in (Ghosh et al., [2200] Wang et al. (022) respectively. Another example is that for the performance of DKT the best baselines by Ghosh et al. (2020) while Long al. (022) and Zhang al. (2021) showed that its performance is below the average. Recent survey studies by [aaraa et al. (2022) and Liu et al. (2022) summarized aforementioned inconsistencies of baseline results and showed evidence that variations in hyper-parameters and data pre-processing procedures contribute significantly to prediction performance of DLKT models. Specifically, [Sarsa et al. (2022) empirically found that even simple baselines with little predictive value may outperform DLKT models with sophisticated
the ASSISTments200
dataset, it is recognized as of
neural components. [Lu et al. (2022) built standardized DLKT benchmark platform and showed that the improvement of many DLKT approaches is minimal compared to the very first DLKT model proposed by Piech et al. (2015).
Therefore, in this paper, we propose SIMPLEKT, a simple but tough-to-beat KT baseline that is simple to implement, computationally friendly and robust to a wide range of KT datasets across different domains. Motivated by the Rasch mode 53 that is a classic yet powerful model in psy- chometrics, the proposed SIMPLEKT approach captures the individual differences among questions covering the same set of KCs by representing each question's 's embedding as an additive combina- tion of the average of its corresponding KCs' embeddings and a question-specific variation. Fur- thermore, different from many existing models that try capture various aforementioned relations and information, the SIMPLEKT purely based on the attention mechanism and uses the ordi- nary dot-product attention function to capture the contextual information embedded in the student learning interactions. To comprehensively and systematically evaluate the performance of SIM- PLEKT, we choose to use the publicly available PYKT benchmark" implementation to guarantee valid and reproducible comparisons against 12 DLKT methods on 7 popular datasets across differ-
A KC is a generalization of everyday terms like concept, principle, fact, or skill. 'The Rasch model is also known as the IPI. item response theory model. https://www..ykt.org
2

---

Published as conference paper at ICLR 2023
ent domains. Results shown that the SIMPLEKT beats wide range of modern neural KT models that based on graph neural networks, memory augmented neural networks, and adversarial ral networks. This suggests that this simple method should be used as the baseline to beat future KT research, especially when designing sophisticated neural KT architectures. To encourage repro- ducible research, all the related codes, data and the learned SIMPLEKT models are publicly available at htttt::///// /ttpt:/ com/ com/pykt: team pykt toolkit
2
RELATED WORK
Recently, deep learning technique have been widely applied into KT task for student's historical learning modeling and the future performance prediction. Existing DLKT approaches can be cate- gorized into the following 5 categories:
C1: Deep sequential models. DLKT models that use auto-regressive architectures to capture stu- dents' chronologically ordered interactions.
example, (Piech et 2015) proposed the very first DKT model that utilizes an LSTM layer to estimate the knowledge mastery. (Lee & Yeung 2019) proposed to enhance DKT with a skill encoder that combines student learning activities and KC representations.
C2: Memory augmented models. DLKT models that capture latent relations between KCs and student knowledge states via memory networks. For instance, (Zhang ct al. (017) exploited and stored the KC relationships via static key memory matrix and predict students' knowledge mastery levels with a dynamic value memory matrix.
C3: Adversarial based models. DLKT models that utilize the adversarial techniques to generate perturbations to improve model generalization capability. (Guo et al. 2021) 20iiijointly train an attentive- LSTM KT model with both original and adversarial examples.
C4: Graph based models. DLKT models that use the graph neural networks model intrinsic relations among questions. KCs and interactions. (Liu et al. (202) presented a question-Ki KC bipar- tite graph to explicitly capture question-level and KC-level inner-relations and question difficulties. (Yang et al. 2020) introduced a graph convolutional network to obtain the representation of the question-K KC correlations.
C5: Attention based models. DLKT models that capture dependencies between interactions via the attention mechanism. For example, (Pandey & Karypis, 2019) used self-attention network to capture the relevance between KCs and students' historical interactions, Choi et al. (2020) designed an encoder-decoder structure to represent the exercise and response embedding sequences. (Ghosh et al. 2020) performed three self-attention modules and explicitly model students' forgetting behav- 10rs via a monotonic attention mechanism.
Please note that the above categorizations are not exclusive and related techniques can be combined. For example, (Abdelrahman & Wang 2019) proposed sequential key-value memory network to unify the strengths recurrent modeling capacity and memory capacity. The proposed SIMPLEKT approach belongs to C5 and it purely models student interactions by using the very ordinary dot- product attention function.
3 A SIMPLE METHOD FOR KNOWLEDGE TRACING
3.1 PROBLEM STATEMENT
In this work, our objective is given an arbitrary question to predict the probability of whether student will answer q. correctly or not based on the student's historical interaction data. More specifically, for each student S, we assume that we have observed a chronologically ordered collec- tion of 7' past interactions i.e., S {s,) Each interaction is represented as 4-tuple i.e., =< q, [c).r.s r, >, where q, fc), r, represent the specific question, the associated KC set, the binary valued student response and student's response time step respectively. We would like to estimate the probability f, of the student's performance on arbitrary question q..
Response is a binary valued indicator variable where represents the student correctly answered the ques- tion, and 0 otherwise.
3

---

Published as conference paper at ICLR 2023
3.2
THE SIMPLEKT APPROACH
3.2.1
REPRESENTATIONS OF QUESTIONS, KCS AND RESPONSES.
Effectively representing student interactions is crucial to the success of the DLKT models. In real- world educational scenarios, the question bank is usually much bigger than the set of KCs. For example, the number of questions is more than 1500 times larger than the number of KCs in the Algebra2005 dataset (described in Section 4.1) Therefore, to effectively learn and fairly evaluate the DLKT models from such highly sparse cnrestion-response data, following the previous work of (Ghosh et all, (020) and (Liu et al. 2022), we artificially transform the original question-response data into KC-response data by expanding each question-level interaction into multiple KC-level interactions when the question associated with of KCs (illustrated in Figure 2
Furthermore, due to the fact that questions covering the same set of KCs may have various difficulty lev- els, students perform significantly different. As shown in Figure 2 even through questions Q and Oa have the same set of KCs, i.., and Cg. students may get 22 wrong but qa correct. Therefore, it is unreal- istic to treat every KC in the expanded KC-response sequence identical. Inspired by the very classic and simple Rasch model in psychometrics that explicitly uses a scalar to characterize the latent factor of ques- tion difficulty, we choose to use a question-specific difficulty vector capture the individual differences among questions on the same KC. More specifically, the rth representations of KC (i.e.. x() and interaction (i.e., yt) in the expanded KC sequence of concept eg are represented as follows:
41
42
cestane Original question resporse data
Expended KC-response data
Figure 2: Graphical illustration of trans- forming the original question-responss data into KC-response data.
X = Zock + M. Vegi
Y: = Zoc + Nayi
Boz W.. T, W
where Ze, denotes the latent representation of the KC ct. m. denotes the difficulty vector of ques- tion Qj and question i contains the KC cę- Ve, represents the question-centee centric variation 9; cov- ering this KC cs. re, denotes the representation of student response on gj. ec, is the n-dimensional one-hot vector indicating the corresponding KC and eq; is the 2-dimensional one-hot vector indicat- ing whether the question is answered correctly. Zeg- My. Ve, and ru, are d-dimensional learnable real-valued vectors. W. E Rdxa and W, Rdxx are learnable linear transformation operations. @ and @ are the dement-wise product and addition operators. n is the total number of KCs.
3.2.2 PREDICTION WITH ORDINARY DOT-PRODUCH ATTENTION.
Different from many existing DLKT approaches that use sophisticated neural components to model student learning and/or forgetting behaviors, we choose to use the ordinary dot-product attention function to explore and extract knowledge states from students' past learning history. Specifically, the retrieved knowledge state (h,-1) at + 1)th timestamp computed as follows:
h SelfAttention( 1, {x1,--.. x V (yi,-.. y+))-
Then we use a two-layer fully connected network to refine the knowledge state and the overall optimization function is follows:
M++1 w ReLU(W, ReLU(W, [b++1: + b1) b2)+b
L - - (r. + (1 - rı) log(1 - o(n)))
where W1, W. w. b b2 and are trainable parameters and W1 Bdx2d W2 E Rdxd w, bj, by E [gdx1 is scalar. o(-) is the sigmoid function.
3.2.3 RELATIONSHIP TO EXISTING DLKT MODELS.
Although the proposed SIMPLEKT belongs to model category C5 discussed in Section 2 it is distin- guished from attention based representative DLKT models such as AKT (Ghosh et al. 2020), SAKT
4

---

Published as conference paper at ICLR 2023
(Pandey & Karypis, 2019) and SAINT (Choi et al. 2020). The difference between SIMPLEKT and AKT threefold: first, omit the self-attentive question encoder and knowledge encoder in AKT and directly feed the representations of X;S and yrs into attention based knowledge state extractor; second, instead of using time decayed monotonic attention function to extract the initial knowledge state, choose to use the ordinary dot-product function that simple and free hyyer-parametent third, interaction representations y:s are simply computed by adding representations of KCs and re- sponses while AKT uses extra parameters to explicitly model the effects of question difficulty in interaction representations. When comparing SIMPLEK the latent question-centric difficulty when learning the KC representations while SAKT and SAINT ignore the question-level difference and treat all questions are identical if they contains the same of KCs. Furthermore, SAINT adopts the encoder-decoder architecture and utilizes Transformers to model the student interaction sequence while our SIMPLEKT only uses the dot- product attention function.
SAKT and SAINT, we explicitly model
4 EXPERIMENTS
4.1 DATASETS
In this paper, we experiment with 7 widely used datasets comprehensively evaluate the perfor- mance of models. These datasets can be divided into 2 categories: (1) D1: Datasets containing information of both questions and KCs; and (2) D2: Datasets containing information of either ques- tions or KCs. Table [ gives real samples of question and KCs from both D1 and D2 categories. The detailed statistics of each dataset are listed in Appendix A.I
4.1.1 DATASETS CONTAINING INFORMATION OF BOTH QUESTIONS KCs
2 (AS2009 This dataset about math exercises and collected from the free online tutoring ESSISTments platform in the school year 2009-2010, It is widely used as the stan-
dard benchmark for KT methods over the last decade (Feng al. 2009 Ghosh al., 2020, Zhang jet al., (017). It includes 337,4115 interactions, 4,661 sequences, 17,737 questions, 123 KCs and each question has 1. 1968 KCs on average.
Algebra2005 (AL2005 This dataset stems from KDD Cup 2010 EDM Challenge, including the detailed step-level student responses the mathematical problems (Stamper al., 2010). Similar to (Choffin et al. Ghosh et al.) [020; Zhang et al.] (017). unique question is constructed by con- catenating the problem name and step name. It has 884,098 interactions, 4,7 sequences, 173.113 questions, 112 KCs and the average KCs is 1.3521.
Bridge2006 (BD2006 7 This dataset is also from the KDD Cup 2010 EDM Challenge and its unique question construction is similar to the process used in Algebra2005. The dataset has 1,824,310 interactions 9,680 sequences, 129,263 questions, 493 KCs and the average KCs is 1.0136.
NIPS34 This dataset is provided by NeurlPS 2020 Education Challenge which contains students' answers to mathematics questions from Eedi. We use the dataset of Task 3 & Task 4 to evaluate our models (Wang et al., 2020b). There are 1,399,470 interactions, 9,401 sequences, 948 questions, 57 KCs, each question has 1.0137 KCs on average.
4.1.2 DATASETS CONTAINING INFORMATION OF EITHER QUESTIONS OR KCS
2tatics2011 This dataset is collected from an engineering statics course taught the Carnegie Mellon University during Fall 2011 1 (Steif & Bier 2014). Its unique question construction is similar to the process used in Algebra2005. The dataset has 189,292 interactions, 1,034 sequences and 1,223 questions.
// //siees.eeeeee wwoule. can www///iiaa www.e//aaaaeaooe www.aaattttmmaaa..aoa... http:///t....aacaaaa/aa/////0000 22000//00a/000000000000000 2010-0000eee casat- -astt-datta/ data/ out 2009 2010
ttps: http://psssetspppppcoooo web
/knnoop/
//edd wttp://mooetttttoooo a /neari eddoce Lenge
5

---

Published as conference paper at ICLR 2023
Similar to ASSISTments2009, this dataset is collected from the ASSISTments platform in the year of 2015, and it has the largest number of students among the other ASSISTment datasets. ends up with 682,789 interactions, 19,292 sequences and 100 KCs
ASSISTments2015 (AS2015 T0
after pre- processsinn.
PO. This dataset is collected from Peking coding practice online platform and provided by Pandey Srivastava 2020). has 987,593 interactions, 20, sequences and 2,748 questions.
Following the data pre-processing steps suggested by (Liu et al, 2022), we remove student se- quences shorter than 3 attempts and set the maximum length of student interaction history to 200 for high computational efficiency.
Table I: Examples of questions and KCs from DI and D2.
Category Datzeet
Knowledge Components
D1
NIS33 Wiich calcuiation incornee? 121-7 77 CCCTI RICPAAI 7122
Maiuplyng and Deviing Negative Numbers
DI
NIPS34
Which of the fullowing sumber ica fector of 60 and of A.3 2.2.
Flexes and Highest Commma Factor Maliples mil Lowad Commun Mulliple Not Available
D2
POJ
Given equatinn the variaiiee x and selve for and
D2
POJ
Given big integer number, you requined to find whether i number.
Not Avaibbble
4.2
BASELINES
To comprehensively and systematically evaluate the performance of SIMPLEKT, compare SIM-
PLEKT againnt 12 DLKT baseline models from aforementioned 5 categories in Section aas follows:
C1: DKT Piech et al.
2015; directly uses RNNs to model students' learning processes.
C1: DKT+ (Yeung & Yeung sistent issues.
2018): improves DKT by addressing the reconstruction and incon-
C1: DKT-F (Nagatani et al.
2019): improves DKT by considering students' forgetting behaviors.
C1: KQN (Lee & Yeung 2019): utilizes the dot product of the students' ability and KC represen- tations to predict student performance.
C1: LPKT (Shen et al.
2021) : designs the learning cell model students' learning processes.
C1: IEKT (ong et al.,
2021): estimates student knowledge state via the student cognition and
knowledge acquisition estimation modules.
C2: DKVMN ZZang et al. 2017): exploits the relationships among KCs and estimate student mastery via memory networks.
C3: ATKT (Guo et al. 2021]: I uses adversarial perturbations to enhance the generalization of the attention-LSTMM based KT model.
C4: GKT (Nakagawa et all 2019); casts the knowledge structure as a graph and reformulate the KT task as node-level classification problem.
C5: SAKT (Pandey & Karypis 2019) : uses self-attention to identify the relevance between the interactions and KCs.
C5: SAINT (Choi et al. 2020): a Transformer based model for KT that encode exercise and responses in the encoder and decoder respectively.
C5: AKT (Ghosh et al. 2020 models forgetting behaviors during the relevance computation between historical interactions and target questions.
4.3 EXPERIMENTAL SETUP
Similar to (Liu et al. 2022), we randomly withhold 20% of the students' sequences for model evaluation and perform standard 5-fold cross validation the rest 80% of each dataset. We select ADAM (Kingma & Ba 2014) as the optimizer to train our model. The maximum of the training epochs is set to 200, and an early stopping strategy is used to speed the training process. The embedding dimension, the hidden state dimension, the two dimension of the prediction layers are set [64, 128), the learning rate and dropout rate are set to [le-3, Ic-4, lc-5) and [0.05, 0, 0.3, 0.5) respectively, the number of blocks and attention heads are set to [1, 2, 4] and [4, 8), the seed
htrpas// //sit//..a./// gong www.///////ommmmmmmmmmoo///////m /a strment addetaa htope: //detvs le /ww////w/////eeeoooo//////e///e/ delders/ http://///.....oooo//ooo/e...... mote K24xBOK
www.atmenta...e. 11 11-bul der-date

---

Published as conference paper at ICLR 2023
is set to [42, 3407] for reproducing the experimental results. Our model is implemented in PyTorch and trained on NVIDIA RTX 3090 GPU device. Similar all existing DLKT research, we use the AUC as the main evaluation metric, and use accuracy as the secondary metric.
4.4 EXPERIMENTAL RESULTS
Overall Performance. Table 2 and 3 summarize the overall prediction performance of SIM- PLEKT and all baselines in terms of the average AUC and accuracy scores. Marker *, and indicates whether SIMPLEKT is statistically superior/equal/nferiio to the compared method (using paired t-test at 0.01 significance level). The last column shows the total number of win/tie/loss for SIMPLEKT against the compared method on all 7 datasets (e.g., #win is how many times SIMPLEKT significantly outperforms that method).
Table 2: Overall AUC performance of SIMPLEKT and all baselines. "_" indicates the method is inapplicable for that dataset.
DI: containing info. quesdions and
Datasets containing infu, of either questions KCs
Model
A1,2006
BD2206
Statles2011
AS2015
DKT
0:75122000119
076692000002
082228300033 Se
5X32
DRT+
0.75475000011
0815600000011
003000
0.76962000002
0.8279 0004
0.72
5152
DKT-F
0.7985-000013"
0,77333600000
0 0076
0.00 0023"
51211
KON
0.74775000111
0.802760 001.*
0,70%6e0.0011
077844000038
0.82220
0. 0.7254
0,00000000155
510/
LPKT
07614400002
0.
1.80550000000
0 00355000003a
1/1/2
EKT
0.761=000077 0.541660.0014
0.8125=0.0009
0.8045-0.0007
DKVMN
0,796160/0009
D.00560000022
AIKT
0779936600013
0,7895000005
476652000011
0.7345808007
0000
GKT
0:402400:0011 0811000000000
0 0,00050000000
47689200021
0 DICS
0,7254.5)
00360
51/1
SAKT
0.72468))00117 07780880065
0.779000.00007
0.7517200005
0.79
0.71142800001
7050
SAINT
01999200022 07775800017
0,7761800:011." 3
0.7873608807
0.759951101133 39
0.266320.0012
6/141
AKT
0.78533000117 0.820800 0,03
0007
030036010003
0.830963
0.728120)0044
0.0291=0:0015+
1/0%
simplck1
0.7341000018
0,1008000000
0.800560000000
0.3199900011
0,72185000005
0102226000000
Table 3: Overall Accuracy performance of SIMPLEKT and all baselines. "_" indicates the method is inapplicable for that dataset.
Modal
AS
DI: Datasets containing info. of both questions and KCss
AL2208
BDSNG
NIPS34
D2: Datasets containing Info, of either questions or KCs
Stalics2011
AN2015
FDJ
Fwin//ii:PPoss
SIMPLEKT
0.7344:0)0014
0.8555000015
07702501902
0,756992 D00 0,7000000004
5W52
DKT-
0,8551-000000
0'709000004
0021
DKT-F
0000
0,8536000001
077066900000
0,782200011
063771201000000
KQN
0:222800:0000
080054000000 0006
0.85320000000
022008000011
0.299810
0,700500008+
066351000017
6/11
LPKT
077355000005
0311555000000
0,35
0.704)
1853
EKT
0.7575000022 0122663000000
0183338000021
0000
1053
DKVMN
0.71992010010
0800720000007
0.8545500000
0.7016210005
0.750855)
0.33925000015
61111
ATKT
0.7208+0.0000 0.799800.00 0019*
0.851110.0004
07013-000824
0.7934-0.0011
077444h 0002
0.632H0
70500
GKT
0.715320:0032
0.00
0.85555000002
0770140000228
0,790220
0,7504010010
0611750001477
SAKT
0:7061-0:0015
0775440
0.8461-0.0005
06879-00004
0.787000.0015
074744088002
SAINT
06975100050
07771120000
0,8011800005"
071100000000
0,76
0661610000000
AKT
0.732200071 0
08882400005
07303008004
0,001100000 le
0,7521-0.0005
0.662:0.0010
SimpleKT
0.7330000012
03083000000
0.8779500000
0,752866.0001
0.7957H00000
0,750850.0004
06622400000
From Table 2 andd Table 3 we find the following results: (1) compared to other baseline methods, SIMPLEKT almost always ranks top 3 in terms of AUC scores and achieves 55 wins, 3 ties and 18 loss in total against 12 baselines
7 public datasets of different domains. This indicates the strength of SIMPLEKT as a baseline KT; (2) in general, the SIMPLEKT approach performs better on DI datasets that have both question and KC information available. When training SIMPLEKT D2 datasets, due to the lack of distinguished information about questions and KCs, the explicit question-centric difficulty modeling degrades and the KC representations become the question ag- nostic; (3) when comparing SIMPLEKT to other attentive models. i.e., SAKT, SAINT and AKT, in C5 category. our SIMPLEKT beats SAKT and SAINT on all the datasets, which indicates the effectiveness of explicit question-centric difficulty modeling. Although our SIMPLEKT approach is significantly worse than the AKT approach on 6 datasets, the performance gaps are quite minimal that are mostly within 0.5% range. On the other hand, SIMPLEKT is much more concise com- pared to the two-layer attentive architecture in the AKT approach: (4) the SIMPLEKT outperforms many deep sequential models in category C1. including DKT. DKT+, DKT-F, KQN on AS2009, AL2005, BD2006, NIPS34 and POJ. We believe this is because the above 4 sequential models use KCs to index questions cannot capture the individual differences among questions with the same KCs which is crucial to predict student future performance; (5) comparing SIMPLEKT and IEKT, see, IEKT has better prediction performance on AS2009, AL2005, and NIPS34. This is because IEKT captures both question-level and KC-level variations its representations and at the same time, it designs two specific neural modules to estimate individual cognition and acquisition
7

---

Published as conference paper at ICLR 2023
abilities: (6) compare other different types of models in C2, C3, and C4, such as DKVMN, ATKT and GKT, SIMPLEKT achieves better performance by using ordinary dot-product attention function without any memory mechanism, adversarial learning or graph constructions, which en- courages the further educational researchers and practitioners to develop effective models with design of simplicity.
0.4
2.1
3.2
348
0.0
SodorAniirry sorcnntgs.
Hoarr sancmagg..
S00eet petsrrtage
Static22011
AS2015
POJ
0.79
212
345
Stdorenitrr sartnagge.
Sedeetmiyy sarerage.
So.ent maarry paceraage.
Figure 3: Non-accumulative predictions in the multi-step ahead scenario in terms of AUC.
6.0
304
272
302
1.4
Nudder persocagg.
St.den 1802 panceragg.
2ucer: hanoyy persantags
Statios2011
A$2015
PC]
115
377
Subr Mharyanutaet
Stake perpetagee
Figure 4: Non- accuuuuullteeve predictions in the multi-step ahead scenario in terms of Accuracy.
Multi-step KT Prediction Performance. In order to make the prediction close to real application scenarios, also predict our model in multi-step prediction which predicts span of student's 's sponses given the student's 's historical interaction sequence. Practically, accurate multi-step KT pre- diction will provide constructive feedback to learning path selection and construction and help teach- adaptively adjust future teaching materials. We conduct the prediction in non-accumulative setting that predicts all future values at once to avoid accumulative prediction errors. To have fine seee-graieed analysis in the multi-ster ahead prediction scenario. we further experiment with DLKT models different portions of observed student interactions. Specifically, percentages of student interaction length from 20% to 90% with step size of 10%. Due to the space limit, we select the best baseline in each category. i.c., IEKT, DKVMN, ATKT, GKT, AKT as the representative approaches and the results in terms of AUC and accuracy
vary the observed
shown in Figure 3 and Figure 4 make the following observations: (1) with the increasing historical information, the student AUC performance prediction become more accurate in most cases, which is in line with the
8

---

Published as conference paper at ICLR 2023
real-world educational scenario; (2) the attention based model almost outperforms other KT models in Staticx2011, AS2005 and POJ according AUC scores, this is because the attention mechanism is expert in capturing the long-tenr long-term dependencies; and (3) our SIMPLEKT achieves the best pre- diction performance in BD2006, NIPS34 in terms of AUC, which indicates the proposed method is simple yet powerful.
Qualitative Visual Analysis. In this section, we qualitatively show the visualization of the predie- tion results made by SIMPLEKT in Figure 5 To better understand the model predictive behavior, compute the historical error rate (HER) per question from the data and use the HERs surro- gates of question difficulties. Due to the space limit, more illustrative and fine- finegrained results are provided in Appendix A.2 As we can see from Figure S when a student meets a certain KC for the first time, the higher HER of the question, the lower the probability that student will get it correct.
For example, the HERs for questions Q527. 4509, 9512. Q618 and Q219 are 0.35, 0,41, 0.20, 0.51, and 0.48 and the corresponding prediction probabilities of SIMPLEKT are 0.74, 0.52, 0.84, 0.45, and 0.51 respectively. Furthermore, for those questions that cover the same set of KCs, such as questions qs2e and q529, the model prediction probability decreases when the corresponding HER increases.
G Maral Mutiplicatin Divison
Multpice Lavert Common Maliple Writing Expressions
Factor and Highes Common Ficter
6 Prine Nambers and Prine
Prddetive
0.74 92 0.88
0.31
0.85
0.49
0.4
0.66
0.52
0.29
0.4
0.84
0.46
0.5
0.89
0.56
0.45
0.
0.56
Reseeac
X
X
X
X
X
X
/
X
/
/
X
/
/
/
Emn
Stankka
0.35
0.20
0.55
D61
0.22
0.4
0.5
0.30
0.41
0.56
0.47
0.30
0.53
0.46
0.19
0.47
0.51
0.35
0.51
0.48
KC IDu
9
9
9
9
13
6g
9
0.4
Quotion
400
6590
6538
05/2
0514
G512
4518
U510
6:00
Q:18
Figure 5: Visualization of student's prediction results on SIMPLEKT.
Ablation Study. We systematically examine the effect of the key component of question difficulty modeling by constructing two model variants: (1) SIMPLEKT JIMPLEKT-ScalaDiii that changes the question- that completely ignores the ques-
centric difficulty vector m, to scalar; and (2) SIMPLAKK tion difficulty modeling and simply set X2 Ze,, The prediction performance belong to D1 are reported in Table + Please note that since datasets in D2 only have either ques- tion information or KC information, SIMPLEKT, SIMPLEKT- - and SIMPLEKT-NoDD essentially become mathematically unidentifiable. From Table 4 we can easily observe that (1) the SIMPLEKK method outperforms the two model variants and especially when removing the ques- tion difficulty modeling component, the prediction performance decreases more than 2% on all DI datasets. This empirically verifies the importance of question-centrri difficulty modeling when mak- ing student performance prediction in KT scenarios; and (2) comparing SIMPLEKT and SIMPLEKT- ScalarDiff, the performance is very minimal. We believe this is because the KC representation x, is based on the simple additive assumption and scalar is expressive enough to represent question-level level difficulty under this assumption.
all datasets that
Table 4: The performance of different variants in SIMPLEKT.
AS2009
AL.2005
BD2006
NIPS34
SIMPLEKT SIMPLEK SIMPLEKT-Scalare SaaalarDiff
0.774420.0013
0.825420.000 0003
0.81602000006
0.803500.0000
0.7740+0.0021
0.8250=0.0011
0.8159=000011
0.8008+0.0012
SIMPLEKT-NoDiff
0.7411#0.0016
0.8048+0.0018
0.7922+0.0011
0.7646+0.0005
5 CONCLUSION
In this work, we propose SIMPLEKT, a simple but bough-to-beat approach to solve KT task ef- fectively. Motivated by the Rasch model in psychometrics, the SIMPLEKT approach is designed to capture individual differences among questions with the same KCs. Furthermore, the proposed SIMPLEKT approach simplifies the sophisticated student knowledge state estimation component with the ordinary dot-product attention function. Comprehensive experimental results demonstrate that SIMPLEKT is able to beat a wide range of recently proposed DL.T models on various datasets from different domains. We believe this work serves as a strong baseline for future KT research.
9

---

Published as conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The code of SIMPLEKT and its variants, i.c. SIMPLEKT-Scalare and CIMPLEKT-NoDIAE to reproduce the experimental results can be found at We give the details of data-preprocessinng and the training hyper-parameters of SIMPLEKT in Section [4.1] and Section 4.3] 'hhe code of the 12 comparison models is ac- cessible from an open-sourced PYKT python library https: /pykt. poykktorg// org/ We choose to use the same data partitions of train, validation, test sets as PYКT and hence all the re- sults can be casily reproducible. All the model training details of all baselines be found at https:/ppykt-tooliii reacthedocs
con/ pykt/tekkttttttt
pykt-tool1s
models. html
ACKNOWLEDGMENTS
This work was supported in part by National Key R&D Program of China, under Grant No. 2020AAA0104500;; in part by Beijing Nova Program (Z201 100006820068) from Beijing Munic- ipal Science & Technology Commission; in part by NFSC under Grant No. 61877029 and in part by Key Laboratory of Smart Education of Guangdong Higher Education Institutes, Jinan University (2022LSYS003).
REFERENCES
Ghodai Abdelrahman and Qing Wang. Knowledge tracing with sequential key-value memory net- works. In Proceedings of the 42nd International ACM SIGIR Conference Research and De- velopment in Information Retrieval, pp. 175-184, 2019.
Hao Cen, Kenneth Koedinger, and Brian Junker. Learning factors analysis-a general method for cognitive model evaluation and improvement. In International Conference on Intelligent Tatoring Systems, pp. 165-175. Springer, 2006.
Jiahao Chen, Zitao Liu, Shuyan Huang, Qiongqiong Liu, and Weiqi Luo. Improving interpretability of deep sequential knowledge tracing models with question-centric cognitive representations. In Proceedings of the AAAI Conference Artificial Intelligence, 2023.
Penghe Chen, Yu Lu, Vincent W 7heng, and Yang Pian. Prerequisite-driven deep knowledge tracing. In 2018 IEEE International Conference on Data Mining, pp. 39-48. IEEE, 2018
Benoît Choffin, Fabrice Popineau, Yolaine Bourda, and Jill-Iênn Vie. DAS3H: Modeling student learning and forgetting for optimally scheduling distributed practice of skills. In Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019), volume 29, pp. 38.
Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Back. Byungsoo Kim, Yeongmin Cha. Dongmin Shin, Chan Bae, and Jaewe Heo. Towards an appropriate query, key, and value com- putation for knowledge tracing. In Proceedings of the Seventh ACM Conference on Learning@ Scale, pp. 341-344, 2020.
Albert T Corbett and John R Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-adapted Interaction, 4(4):253-2) 278. 1994.
Mingyu Feng, Neil Heffernan, and Kenneth Koedinger, Addressing the assessment challenge with online system that tutors it assesses. User Modeling and User-adapted Interaction, 19(3): 243-266, 2009.
Aritra Ghosh, Neil Hefferman, and Andrew S Lan. Context-aware attentive knowledge tracing. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020.
Xiaopeng Guo, Zhijie Huang. Jie Gao, Mingyu Shang, Maojing Shu, and Jun Sun. Enhancing knowledge tracing via adversarial training. In Proceedings of the 29th ACM International Con- ference on Multimedia, 367-375, 2021.
Tanja Käser, Severin Klingler, Alexander Schwing, and Markus Gross. Dynamic bayesian net- works for student modeling. IEEE Transactions on Learning Technologies, 4504):500-462, 2017.
10

---

Published as conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.:91412.6988, 2014.
Elise Lavoué, Baptiste Monterrat, Michel Desmarais, and Sébastien George. Adaptive gamification for learning environments. IEEE Transactions on Learning Technologies, 12(1):16-28, 2018.
Jinseok Lee and Dit- Yan Yeung. Knowledge query network for knowledge tracing: How knowledge interacts with skills. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, pp. 491-500, 2019.
Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. EKT: Exercise- aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, 33(1):100- 15, 2019.
Yunfei Liu, Yang Yang, Xianyu Chen, Jian Shen, Haifeng Zhang, and Yong Yu. Improving knowl- edge tracing via pre-training question embeddings. Proceedings of the Twenty-Ninth Interna- tional Conference on International Joint Conferences on Artificial Intelligence, pp. 1556-1562,
2021.
Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Jiliang Tang, and Weiqi Luo. руКТ: A python library to benchmark deep learning based knowledge tracing models. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks., 2022.
Ting Long, Yunfei Liu, Jian Shen, Weinan Zhang, and Yong Yu. Tracing knowledge state with individual cognition and acquisition estimation. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 173-182, 2021.
Ting Long, Jiarui Qin, Jian Shen, Weinan Zhang, Wei Xia, Ruiming Tang, Xiuqiang He. and Yong
Yu. Improving knowledge tracing with collaborative information. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 599-607, 2022.
Sein Minn, Yi Yu, Michel C Desmarais, Feida Zhu, and Jill-Jenn Vie. Deep knowledge tracing and dynamic student classification for knowledge tracing. In 2018 IEEE International Conference on Data Mining, pp. 118- 187. IERE, 2018
Koki Nagatani, Qian Zhang, Masahiro Sato, Yan-Ying Chen, Francine Chen, and Tomoko Ohkuma. Augmenting knowledge tracing by considering forgetting behavior. In The World Wide Web Con- ference, pp. 3101-3107, 2019.
Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. Graph-based knowledge tracing: mod- eling student proficiency using graph neural network. In 2019 IEEE/WIC/ACM International Conference on Web Intelligence, pp. 156-163. IEEE, 2019.
Shalini Pandey and George Karypis. A self self-attentive model for knowledge tracing. In /2th Inter- national Conference on Educational Data Mining, pp. 384-389. International Educational Data Mining Society, 2019.
Shalini Pandey and Jaideep Srivastava. RKT: relation self-attenticn for knowledge tracing. In Proceedings of the 29th ACM International Conference on Information & Knowledge Manage- ment, pp. 1205-1214 1214, 2020.
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. Advances in Neural Information Processing Systems, 28, 2015.
Shi Pu, Michael Yudelson. Lu Ou, and Yuchi Huang, Deep knowledge tracing with transformers. In International Conference Artificial Intelligence in Education, pp. 252-256.65 Springer. 2020.
Sami Sarsa, Juho Leinonen, Arto Hellas, al. Empirical evaluation of deep learning models for knowledge tracing: Of hyperparameters and metrics on performance and replicability. Journal of Educational Data Mining, 14(2), 2022.
11

---

Published as conference paper at ICLR 2023
Shuanghong Shen, Qi Liu, Enhong Chen, Han Wu, Zhenya Huang, Weihao Zhao, Yu Su, Haiping Ma, and Shijin Wang. Convolutional knowledge tracing: Modeling individualization in student learning process. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1857-1860, 2020.
Shuanghong Shen, Qi Liu, Enhong Chen, Zhenya Huang, Wei Huang, Yu Yin, Yu Su, and Shijin Wang. Learning process-consisent knowledge tracing. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1452-1460, 2021.
Shuanghong Shen, Zhenya Huang, Qi Liu, Yu Su, Shijin Wang, and Enhong Chen. Assessing student's dynamic knowledge state by exploring the question difficulty effect. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 427-437, 2022.
Stamper, A Niculescu-Mizil S Ritter, G Gordon, and K Koedinger. Algebra I 2005-2006 and Bridge to Algebra 2006-2007. Development data sets from KDD Cup 2010 Educational Data
Mining Challenge, 2010.
Paul Steif and Norman Bier. Oli engineering statics fall 2011, 2014.
Nguyen Thai-Nghe, Lucas Drumond, Tomá Horváth, Artus Krohn-Grimberghe,, Alexandros Nanopoulos, and Lars Schmidt-Thieme, Factorization techniques for predicting student perfor- mance. In Educational Recommender Systems and Technologies: Practices and Challenges, pp. 129-153. IGI Global, 2012.
Hanshuang Tong. Zhen Wang, Qi Liu, Yun Zhou, and Wenyuan Han. HGKT: Introducing hierar- chical exercise graph for knowledge tracing. arXiv preprint arXiv:2006. 16915, 2020.
Chenyang Wang, Weizhi Ma, Min Zhang, Chuancheng Lv, Fengyuan Wan, Huijie Lin, Taoran Tang, Yiqun Liu, and Shaoping Ma. Temporal cross-effects in knowledge tracing. In Proceedings of the 1 14th ACM International Conference on Web Search and Data Mining, pp. 517-525, 2021.
Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang. Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6153-6161, 2020a.
Zhiwei Wang, Xiaoqin Feng, Jiliang Tang, Gale Yan Huang, and Zitao Liu. Deep knowledge tracing with side information. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, II, USA, June 25-29, 2019, Proceedings, Part 11 20, pp. 303-308. Springer, 2019.
Zichao Wang. Angus Lamb, Evgeny Saveliev, Pashmina Cameron. Yordan Zaykov, José Miguel Hornández-Lobato, Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, et al. Instructions and guide for diagnostic questions: The neurips 2020 education challenge. ArXiv preprint, abs/2007. 12061, 2020b. lattpa// //apiii///xxv....// orq/ abs /2007.000.2 12061
Yang Yang, Jian Shen, Yanru Qu, Yunfei Liu, Kerong Wang. Yaoming Zhu, Weinan Zhang, and Yong Yu. GIKT: graph-based interaction model for knowledge tracing. In Joint European Con- ference on Machine Learning and Knowledge Discovery in Databases, pp. 299-315. Springer, 2020.
Chun-Kit Yeung. Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory. In Proceedings of The 12th International Conference on Educational Data Min- ing (EDM 2019). pp. 683-686, 2019.
Chun-Kit Yeung and Dit-Yan Yeung. prediction-connsiteen regularization. Learning at Scale, pp. 1 1-10, 2018
Addressing two problems in deep knowledge tracing via In Proceedings of the Fifth Annual ACM Conference
Jiami Zhang, Xingjian Shi, Irwin King, and Dit Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web, pp. 765, 2017.
Moyu Zhang, Xinning Zhu, Chunhong Zhang, Yang Ji, Feng Pan, and Changchuan Yin. Multi- Factors Aware Dual-Attentinaal Knowledge Tracing. Proceedings of the 30th ACM Interna- tional Conference on Information & Knowledge Management, pp. 2588-2597, 2021.
12


Reviews:
[{"review_id": "wZb61jaDAwa", "reviewer": "Reviewer_nkiF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to address inconclusive findings in the literature about which models perform best for Knowledge Tracing (KT). In this paper, the authors present a baseline which convincingly outperforms most published work despite being simpler in nature. ", "summary_of_the_review": "In this paper, the authors present a baseline which convincingly outperforms most published work despite being simpler in nature. The baseline compares against 12 other models -- such a comprehensive comparison is itself valuable to the field. Additionally, the works conducts an ablation study, which provides important insight into understanding what makes the simple model nonetheless effective (e.g. how much is from the question-difficulty modeling?).", "strength_and_weaknesses": "STRENGTHS\n- This work gives a very comprehensive review of dozens of work in the KT space and how they work.\n- By comparing against 12 previous models, this weighs in on disagreements in the literature about model comparisons.\n- Demonstrating that such a simple model can outperform many deep learning-based approaches provides meaningful insight into the nature of the task, which can inspire future work and model designs.\n\nWEAKNESSES\n- I could not tell whether the 12 comparison models had publicly available code or were reimplemented by the authors (or something else). Clarification would be helpful.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f_rcKTJNNLs", "reviewer": "Reviewer_EgtS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper builds a simple yet strong baseline, named SimpleKT, for knowledge tracing (KT). The newly introduced baseline is motivated by the Rasch model in psychometrics to capture the individual differences in questions of the same KC. It only contains a few simple operations, i.e., element-wise product, addition, and the ordinary dot-product attention, which is simple to implement. Surprisingly, the proposed simple method can beat most of the 12 representative baselines on 7 public datasets.", "summary_of_the_review": "A simple-to-implement and well-motivated baseline model is presented for the Knowledge tracing task, which wins most of 12 existing methods on 7 public datasets. The newly constructed baseline can facilitate future research on KT.", "strength_and_weaknesses": "Strengths:\nA simple-to-implement and strong baseline is constructed for the knowledge tracing (TK) task, which can facilitate future KT research.\n\nWeaknesses:\nI am not familiar with the knowledge tracing task. Thus, I cannot provide useful comments and suggestions to improve this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NE4GiW1EHbz", "reviewer": "Reviewer_vuA2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SIMPLEKT, a simple but tough-to-beat KT baseline that is simple to implement, computationally friendly and robust to a wide range of KT datasets across different domains. ", "summary_of_the_review": "In summary, the topic in this work is interesting, however the overall contributions of this work are not very enough for ICLR.", "strength_and_weaknesses": "Strength\n\n1. The paper is well written and well presented.\n\n2. The source code is publicly avaiblale.\n\n3. The experimental results are good, comparing to existing baselines.\n\n\nWeakness\n1. The technical contributions of this work are very limited, simply adopting the tranformer model for their tasks. Overall contributions are below the bar of ICLR in my mind.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster