Revisiting Knowledge Tracing: A Simple and Powerful Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:18:46

Abstract: Advances in multimedia technology and its widespread application in education have made multimedia learning increasingly important. Knowledge Tracing (KT) is the key technology for achieving adaptive multimedia learning, aiming to monitor the degree of knowledge acquisition and predict students' performance during the learning process. Current KT research is dedicated to enhancing the performance of KT problems by integrating the most advanced deep learning techniques. However, this has led to increasingly complex models, which reduce model usability and divert researchers' attention away from exploring the core issues of KT. This paper aims to tackle the fundamental challenges of KT tasks, including the knowledge state representation and the core architecture design, and investigate a novel KT model that is both simple and powerful. We have revisited the KT task and propose the ReKT model. First, taking inspiration from the decision-making process of human teachers, we model the knowledge state of students from three distinct perspectives: questions, concepts, and domains. Second, building upon human cognitive development models, such as constructivism, we have designed a Forget-Response-Update (FRU) framework to serve as the core architecture for the KT task. The FRU is composed of just two linear regression units, making it an extremely lightweight framework. Extensive comparisons were conducted with 22 state-of-the-art KT models on 7 publicly available datasets. The experimental results demonstrate that ReKT outperforms all the comparative methods in question-based KT tasks, and consistently achieves the best (in most cases) or near-best performance in concept-based KT tasks. Furthermore, in comparison to other KT core architectures like Transformers or LSTMs, the FRU achieves superior prediction performance with approximately only 38\% computing resources. Through an exploration of the ReKT model that is both simple and powerful, is able to offer new insights to future KT research. The code is in the supplementary materials.

Main paper: 
 introduction: 
 Knowledge Tracing (KT) is a crucial task in online education systems. It assesses students' knowledge mastery (knowledge state) based on their interaction history with educational resources and aims to predict their future performance. Effectively addressing KT task can assist teachers in gaining insights into students' learning progress, enabling them to tailor teaching strategies and offer personalized guidance (Abdelrahman et al., 2023) . As shown in Figure 1 , the example of KT involves each question being associated with one or more concepts. Students practice with different questions, and the aim of KT is to predict the probability of them answering the next question correctly. Often, due to the lack of question information in certain datasets, KT can be further categorized into question-based KT and concept-based KT for uniformity. Additionally, to simplify the KT task, multiple concepts are sometimes amalgamated into a new concept (Xiong et al., 2016) . Many methods (Shen et al., 2021; Abdelrahman & Wang, 2022; Liu et al., 2023b) have been proposed for KT, which predict students' future performance by tracing their knowledge state. Thus, the key aspect of KT is effectively tracing and representing students' knowledge state. However, most KT methods exclusively focus on modeling either question-based KT or concept-based KT, without exploring their potential in both of these scenarios.In recent years, with the advancement of deep learning technologies, KT models based on methods such as Transformers (Ghosh et al., 2020; Cui et al., 2023) , graph neural networks (Tong et al., 2020; Wu & Ling, 2023) , and contrastive learning (Lee et al., 2022; Yin et al., 2023) have emerged one after another. Although these works have made significant contributions to in-depth research on KT, we have observed a strange phenomenon: the current mainstream of KT research seems to rely on cutting-edge technologies from other domains to build complex models, while efforts to delve deeper into the KT problem itself appear to have made limited progress. While works like LPKT (Shen et al., 2021) and LBKT (Xu et al., 2023) explore the impact of various behaviors during student learning processes, they often depend on specialized and intricate architectures, making it challenging for subsequent research to derive substantial insights from them. We hope to Revisit Knowledge Tracing (ReKT) and design a model that is as simple and powerful as possible.We start by addressing the two fundamental challenges of the KT task: (1) How to represent a student's knowledge state from learning data; (2) How to design a core architecture that is as simple as possible and suitable for KT. For the first challenge, in fact, long before we began using KT models to solve this problem, teachers had already been engaged in a similar process. When teachers assess a student's ability to solve a specific question, they consider several key factors. First, they assess the student's previous performance on the same question. Second, they consider the student's performance on similar questions previously. Finally, if the student has not encountered this question or similar ones before, the teacher consider the student's overall historical performance. Inspired by this process, we model student's knowledge state from three distinct perspectives: questions, concepts, and domains. Regarding the second challenge, drawing inspiration from human cognitive development models (Bruner, 1966) , which emphasizes that changes in human knowledge state (Response) are mainly affected by two main psychological processes: internalization and forgetting. Internalization emphasizes the updating (Update) process of knowledge state based on environmental stimuli, while forgetting (Forget) emphasizes the natural changing process of knowledge state over time. Therefore, we model these core processes and design a lightweight core architecture called FRU (Forget-Response-Update) for KT tasks.Specifically, we propose ReKT, which is both simple and powerful. It traces student's knowledge state from three distinct perspectives: (1) It traces students' question knowledge state from interaction history limited to specific questions, indicating their mastery of specific questions;(2) It traces students' concept knowledge state from interaction history limited to specific concepts (where two questions with the same concept are considered similar), indicating their mastery of questions encompassing those specific concepts; (3) It traces students' domain knowledge state from their entire interaction history, indicating their mastery of questions spanning the entire domain (i.e., overall performance). We combine these three to comprehensively represent students' knowledge state. Furthermore, we've designed a lightweight core architecture called FRU for KT tasks. It first calculates the forgetting of the student's knowledge state based on the interval time, then responses based on the knowledge state, and finally updates the knowledge state based on the learning interaction. It's worth mentioning that FRU comprises only two linear regression units. Experimental results on 7 publicly available datasets, comparing ReKT with 22 state-of-the-art KT models, including question-based KT and concept-based KT, show that in most cases, ReKT significantly outperforms other models. This demonstrates that even without relying on highly complex models or cutting-edge technologies, by delving deeply into the characteristics of the KT task, it is possible to construct a model that is both simple and powerful. We believe that ReKT has the potential to offer a wealth of new inspiration and insights for future KT research.Distinct Perspectives Analysis: In fact, purely from a model perspective, constructing the knowledge state from the entire interaction history inherently encompasses the knowledge state con-structed from limited interaction history. This is likely the reason why most KT models have failed to construct knowledge states from a more nuanced perspective. However, constrained by the small scale of KT datasets, complex model architectures like Transformers have struggled to bring significant benefits to KT (Ghosh et al., 2020) and, naturally, have been unable to achieve the precision of constructing knowledge states from a multi-perspective approach. Subsequent experiments in this paper demonstrate the effectiveness of multi-perspective modeling of knowledge states. 
related work: 
 KT was proposed by (Corbett & Anderson, 1994 ) in 1994 . Classic knowledge tracing methods can be categorized into bayesian-based methods such as BKT(Yudelson et al., 2013; Khajah et al., 2014) and factor analysis-based methods such as LFA (Cen et al., 2006b) , AFM(Cen et al., 2006a) , PFA (Pavlik Jr et al., 2009) , KTM (Vie & Kashima, 2019) . In recent years, with the advancement of deep learning, an increasing number of methods have been employed to tackle KT tasks. We categorize the existing methods into the following two types based on the nature of the KT task:Concept-based KT: This category of KT methods aims to predict the student's mastery of specific concepts. DKT (Piech et al., 2015) is regarded as a representative method for concept-based KT, utilizing an LSTM to construct the student's knowledge state. DKVMN (Zhang et al., 2017) employs a dynamic key-value memory network to capture the student's knowledge state. SAKT (Pandey & Karypis, 2019 ) models the relationship between students and concepts using self-attention mechanisms to derive the knowledge state. SKVMN (Abdelrahman & Wang, 2019) employs an enhanced LSTM for modeling students' knowledge state and updates it based on students' responses to relevant questions. GKT (Nakagawa et al., 2019) propagates a student's knowledge state over a graph structure. SKT (Tong et al., 2020) takes into account various relationships between concepts to simulate the propagation of knowledge states. ATKT (Guo et al., 2021) utilizes adversarial learning to represent students' knowledge states in a more robust manner. CL4KT (Lee et al., 2022) designs various data augmentation strategies to enhance the representational capacity of the knowledge state.Question-based KT: Building upon concept-based KT, this category of methods includes additional question information to predict a student's performance on specific questions. AKT (Ghosh et al., 2020) is considered a representative method for question-based KT, utilizing a context-aware Transformer architecture to account for forgetting behavior in tracing students' knowledge state. SAINT (Choi et al., 2020a) constructs the student's knowledge state entirely using the Transformer framework. GIKT (Yang et al., 2021) employs GCN (Kipf & Welling, 2017) to uncover relationships between questions and concepts, and incorporates an interactive module to capture students' knowledge states. DIMKT (Shen et al., 2022) extensively mines question difficulty information to model the student's knowledge state. simpleKT (Liu et al., 2023b) simplifies AKT to trace students' knowledge state. DTransformer (Yin et al., 2023) employs contrastive learning to maintain a stable knowledge state. AT-DKT (Liu et al., 2023a) enhances knowledge state representation by introducing two extra tasks related to question design.Connections and Differences: These approaches all utilize the student's entire interaction history to construct the knowledge state. Their modeling approaches focus on creating more complex models to capture effective knowledge states. ReKT also relies on the entire interaction history as one of the sources for modeling the knowledge state. However, unlike the others, ReKT's modeling approach does not involve building more complex models. Instead, ReKT deeply mines the student's interaction history information and constructs the knowledge state from different perspectives. 
method: 
 ReKT's overall framework is depicted in Figure 2 . We will present the problem formulation of KT and then detail the various modules of ReKT.The KT task can be defined as follows: Given a student's interaction history sequence L = {(q 1 , c 1 , r 1 ), (q 2 , c 2 , r 2 ), ..., (q t , c t , r t )}, where q t is the question at time t, c t signifies the concept associated with question q t , and r t ∈ {0, 1} shows if the student's response to q t is correct. KT's aim ReKT constructs the question and concept knowledge states from the interaction history limited to the current question and concept. Additionally, it constructs the domain knowledge state using the entire interaction history. is to predict the probability of the student answering the next question q t+1 correctly. For conceptbased KT, which lacks specific question data, concepts are treated as equivalent representations of questions (Zhang et al., 2017; Nagatani et al., 2019) . So, the goal is to predict the probability of the student answering the next concept c t+1 correctly. Please note that if a question involves multiple concepts, we combine them into a new concept (Xiong et al., 2016) .Effectively representing questions, concepts, and responses is important for KT, and establishing their relationships forms a highly effective approach. Let's denote the feature matrix as follows: Q ∈ R n×d for questions, C ∈ R m×d for concepts, and R ∈ R 2×d for responses. Here, n represents the total number of questions, m is the total number of concepts, and d signifies the feature dimension. Inspired by the classic psychological measurement theory of Rasch (Rasch, 1993) , which explicitly employs scalars to represent question difficulty. For the current question q t+1 and its associated concept c t+1 , their feature representationE t+1 = Q qt+1 + C ct+1 + dif f qt+1 * V ct+1 .Here, Q qt+1 denotes the q t+1 -th row of Q, C ct+1 corresponds to the c t+1 -th row of C, dif f qt+1 represents the difficulty of q t+1 and is a scalar, and V ct+1 ∈ R 1×d captures the extent of variation of the question with respect to its concept. For any given moment t within the interaction history L, where L t = (q t , c t , r t ), the feature representation X t for L t is calculated as X t = E t + R rt , where R rt corresponds to the r t -th row of R.The representation of a student's knowledge state is the most fundamental issue in KT, and ReKT constructs three types of knowledge states for a student from the interaction history L = (q 1 , c 1 , r 1 ), (q 2 , c 2 , r 2 ), ..., (q t , c t , r t ): question, concept, and domain knowledge state. For the current question q t+1 and its corresponding concept c t+1 :For the aspect of question knowledge state: We construct it from the interaction history L Q t+1 limited to the current question q t+1 . Here, L Q t+1 = j L j , if q j == q t+1 , j < t + 1. represents the union operation. The feature representation of L Q t+1 is denoted as X Q t+1 , as shown in Figure 2 , where X Q t+1 = {X 3 , X t-δ }. As shown in the example, they all include the current question 3. Thus, the question knowledge state hQ t+1 = FRU (X Q t+1 ∪ X t+1 ).The FRU architecture is shown in Figure 3 , and we'll provide a detailed explanation of it in the next section. Note the inclusion of an added term, X t+1 , here. This serves the purpose of informing the FRU about the current time, i.e., t + 1.For the aspect of concept knowledge state: We construct it from the interaction history L C t+1 limited to the current concept c t+1 . Here,L C t+1 = j L j ,if c j == c t+1 , j < t + 1. The feature repre- sentation of L Ct+1 is denoted as X C t+1 , as shown in Figure 2 , whereX C t+1 = {X 1 , X 3 , X t-δ }. Asshown in the example, they all include the current concept 2. Therefore, the concept knowledge state h C t+1 = FRU (X C t+1 ∪ X t+1 ). The inclusion of X t+1 here also aims to inform the FRU about the current moment. The FRU here is distinct from the earlier mentioned FRU parameter.For the aspect of domain knowledge state: We construct it from the entire interaction history L. The feature representation of L is denoted as X, as shown in Figure 2 , where X = {X 1 , X 2 , X 3 , . . . X t-δ , . . . , X t }. Therefore, the domain knowledge state h D t+1 = FRU (X ∪ X t+1 ). Consistent with the above, here X t+1 is used to inform the FRU of the current moment, and the FRU parameters here are independent of the previous FRU's parameters. Inspired by human cognitive development models (Bruner, 1966) , we design a lightweight core architecture called FRU (Forget-Response-Update) for KT tasks, as shown in Figure 3 .It first takes into account the process of students forgetting their knowledge state, then responses to specific questions based on knowledge state, and ultimately updates the knowledge state based on response accuracy. Specifically, let the current moment be t, the last relevant moment is t-α (this allows FRU to handle time series with varying intervals), Z t-α ∈ R 1×d represents the knowledge state at the time t -α. First, we remember that the interval time α is represented by features as I α ∈ R 1×d , then at the current moment t, the degree of forgetting of Z t-α can be calculated asf t = Sigmoid([Z t-α ⊕ I α ]W 1 + b 1 ),where W 1 and b 1 are all learnable parameters. Then the knowledge state of the current response Response t = f t * Z t-α . Finally, the student will update the knowledge state according to the feature representation X t of current interaction, then the student's current knowledge stateZ t = Response t + T anh([Response t ⊕ X t ]W 2 + b 2), Among them, W 2 and b 2 are all learnable parameters. The subsequent processing flow can be deduced by analogy.We record the interaction sequence input by FRU as X ξ t = {..., X t-α , X t }, then FRU can be abstracted as: Response t = FRU (X ξ t ). Clearly, the FRU framework is very lightweight, as it consists of only two linear regression units. However, it aligns remarkably well with students' cognitive learning processes, and subsequent experiments will demonstrate its effectiveness.For the question E t+1 that needs to be answered after feature representation and the students' three knowledge states h Q t+1 , h C t+1 and h D t+1 , then the student's current final knowledge stateH t+1 = h Q t+1 ⊕ h C t+1 ⊕ h D t+1 .It's worth noting that here we use concatenation rather than attention mechanisms to combine them, as we consider that different knowledge states play distinct roles. Finally, based on the student's current knowledge state H t+1 , we predict the student's response y t+1 for answering E t+1 , in other words,y t+1 = Sigmoid(W 4 ReLU (W 3 [E t+1 ⊕ H t+1 ] + b 3 ) + b 4 ),where W 3 , b 3 , W 4 and b 4 are all learnable parameters. The knowledge tracing loss Loss KT is computed as follows:Loss KT = - T -1 t=1 (r t+1 logy t+1 + (1 -r t+1 )log(1 -y t+1 ))Here, T represents the total number of time steps, and r t+1 denotes the actual response result from the student at time step t + 1. We use Adam (Kingma & Ba, 2015) to optimize model parameters. We evaluated the performance of ReKT on 7 publicly available commonly used datasets: ASSIST09, ASSIST12, ASSIST15, ASSIST17, Statics2011, EdNet, and Eedi. The statistical information for these datasets is provided in Table 1 , and detailed introductions and processing method for each dataset can be found in Appendix A. 
baseline: 
 In order to assess the performance of ReKT, we compared it against baseline models of conceptbased KT and question-based KT, respectively. Detailed introductions to each model can be found in Appendix B. Specific execution approaches for each model will also be clarified in Appendix B.BKT (Corbett & Anderson, 1994) . DKT (Piech et al., 2015) . DKVMN (Zhang et al., 2017) . DKT+ (Yeung & Yeung, 2018) . KQN (Lee & Yeung, 2019) .DeepIRT (Yeung, 2019) . DKT+forgetting (Nagatani et al., 2019) . SAKT (Pandey & Karypis, 2019) . GKT (Nakagawa et al., 2019) . AKT-concept (Ghosh et al., 2020) . SAINT-concept (Choi et al., 2020a) . ATKT (Guo et al., 2021) . CL4KT (Lee et al., 2022) .KTM (Vie & Kashima, 2019) . AKT (Ghosh et al., 2020) . SAINT (Choi et al., 2020a) . PEBG+DKT (Liu et al., 2020) . GIKT (Yang et al., 2021) . CDKT (Dai et al., 2022) . DIMKT (Shen et al., 2022) . QIKT (Chen et al., 2023) . simpleKT (Liu et al., 2023b) . AT-DKT (Liu et al., 2023a) . DTransformer (Yin et al., 2023) .We implemented ReKT using PyTorch (Paszke et al., 2019) . For each dataset, we split 80% of all the sequences as the training set, 20% as the test set (Yang et al., 2021; Yin et al., 2023) . The learning rate was set to 0.002, batch size to 80, and feature dimension d to 128. Additionally, we applied L2 regularization to the model weights with a decay coefficient of 1e-5. Sequences with a length of less than 3 were removed. To handle variable sequence lengths, all sequences were padded to a consistent length of 200. To mitigate overfitting, we introduced a dropout rate of 0.4. ReKT was trained on a Linux server with two 2.00GHz Intel(R) Xeon(R) CPUs and a Nvidia Tesla P100-PCIE-16GB GPU. Consistent with prior research (Song et al., 2021; 2022; Ni et al., 2023; Sun et al., 2023; Zhao et al., 2023) , we employed the Area Under the Curve (AUC) as the primary evaluation metric, with Accuracy (ACC) serving as the secondary metric. We repeated the experiment five times and reported the average performance (Su et al., 2018) . We can observe: (1) When tracing only one knowledge state, the performance of question, concept, and domain knowledge states shows an increasing trend. This is evident as the interaction history data they utilize also increases in the same order. The model evidently benefits from more data, thus leading to better performance with domain knowledge state; (2) Tracing two knowledge states, as opposed to one, results in a notable performance improvement. Furthermore, the performance of question + domain knowledge state is not always inferior to concept + domain knowledge state (as observed in ASSIST17 and EdNet). This indicates that across different datasets, each knowledge state plays a significant role;(3) Tracing all three of these knowledge states (ReKT) yields significant improvements in performance across all datasets compared to tracing only one or two knowledge states. This undoubtedly demonstrates the effectiveness of these three knowledge states and validates the effectiveness of the proposed multi-perspective modeling. Core architecture performance for KT: In this section, we will explore the performance of different core architectures on KT. Specifically, we will compare FRU with four commonly used core architectures in KT: LSTM, GRU, Transformer, and AKT-Transformer (Ghosh et al., 2020) , while only tracing domain knowledge state (as is the practice in most KT research). The results for question-based KT are shown in Table 5 , and the results for concept-based KT can be found in Appendix D. Details on the computing resources and ranking methodology are clarified in Appendix D. We can conclude the following: (1) In terms of an overall performance comparison, in question-based KT, FRU as the core architecture achieves the best performance in most cases. In concept-based KT, FRU also demonstrates commendable performance. This suggests that the proposed FRU architecture is highly suitable for KT; (2) When comparing the number of parameters and computing resources, FRU requires significantly fewer parameters and computing resources. FRU achieves excellent performance with approximately 38% of the computing resources compared to other core architectures. This undoubtedly proves the lightweight nature of FRU and highlights its simplicity and effectiveness as the core architecture; (3) Considering the average rank, FRU performs the best in question-based KT and ranks relatively well in concept-based KT. This indicates that FRU maintains strong competitiveness throughout, despite it is very simple. The time and space complexity analysis of FRU can be found in Appendix G. We also explain the differences and advantages of FRU over other architectures in Appendix H. Choice of Different Core Architectures for ReKT: In this section, we discuss the impact of different core architectures on ReKT and analyze the advantages brought by FRU. Specifically, we compare four commonly used core architectures in KT: LSTM, GRU, Transformer, and AKT-Transformer, and their performance is shown in Table 6 . The same experiments on ReKT-concept are presented in Appendix E. Please note the difference between this experiment and the previous one: this experiment, in the context of ReKT, traces students' knowledge states from multiple perspectives, while the previous experiment, like most research, only traced domain knowledge states.From Table 6 , we can draw the following conclusions: (1) Their performance follows the order of FRU, GRU, LSTM, AKT-Transformer, and Transformer in descending order, clearly demonstrating the effectiveness of FRU. In addition, simpler methods appear to be more effective. This could be attributed to the introduction of multiple perspectives, which provides more comprehensive information for these simplified models, while excluding unnecessary information or noise;(2) It also indicates that LSTM modeling outperforms AKT-Transformer or Transformer, consistent with much research (Long et al., 2021; Liu et al., 2023b) showing that architectures based on LSTM tend to perform better than those based on Transformers. This may be because LSTM aligns better with the characteristics of knowledge states: continuity and dynamic updates. Additionally, the small scale of the KT dataset may limit the benefits from Transformers;(3) AKT-Transformer consistently outperforms Transformer, highlighting the importance of considering student knowledge forgetting behavior. Transformer performs the poorest, indicating that a pure Transformer has limited improvements for KT; (4) When comparing parameters and FLOPs, ReKT achieves optimal performance with minimal resources, highlighting its simplicity and effectiveness. T+N Prediction: In order to simulate real-world scenarios and assess the stability of ReKT in tracing knowledge states, we conducted T+N prediction experiments. Assuming the current time is T , this experiment aimed to predict not only a student's performance in the next time step T + 1, but also their performance at subsequent time steps T + 2, T + 3, ..., T + N . If the model performs well in this situation, it indicates a more stable ability to trace knowledge states. As depicted in Figure 4 , we compare ReKT with models that perform well on concept-based KT and questionbased KT, respectively. The following observations were made: (1) As the number of time steps N in T + N increased, the performance of all models exhibited a downward trend. However, for smaller N values, the impact on ReKT was comparatively smaller; (2) Notably, whether in conceptbased KT or question-based KT, AKT demonstrated a rapid decline in performance, indicating an unstable ability to trace knowledge states; (3) Across all scenarios, ReKT consistently achieved the best performance. This underscores the fact that ReKT maintains a strong level of stability while retaining its simplicity and effectiveness. Appendix F provides more detailed T+N experiments. We also illustrate the advantages of ReKT in tracing students' knowledge state in Appendix I. 
conclusion: 
 In this paper, we revisit knowledge tracing and propose a simple and powerful model, ReKT. Firstly, inspired by the decision-making process of human teachers, we model students' knowledge states from three distinct perspectives: questions, concepts, and domains. Secondly, drawing inspiration from human cognitive development models, we design a lightweight FRU architecture as the core framework for KT tasks, comprising only two linear regression units. When compared to 22 state-ofthe-art KT models on 7 publicly available datasets, the results indicate that ReKT achieves optimal performance in most cases, whether in question-based KT or concept-based KT. This underscores that, even without relying on complex models or cutting-edge technology, by delving deeper into the characteristics of KT tasks, one can design models that are both simple and powerful. We believe that ReKT has the potential to offer a wealth of new inspiration and insights for future KT research.DeepIRT (Yeung, 2019) : Introduces Item Response Theory (IRT) (Wilson et al., 2016) to DKVMN for improved interpretability of predictions. We conducted experiments by running their official open-source code.DKT+forgetting (Nagatani et al., 2019) : Augments DKT by incorporating various behavioral features to consider forgetting in student knowledge states. As there was no official open-source code available, we reimplemented their model using PyTorch.SAKT (Pandey & Karypis, 2019) : The pioneering self-attention mechanism-based knowledge tracing model that attempts to capture the relationship between students and concepts to represent knowledge states. We conducted experiments by running their official open-source code.GKT (Nakagawa et al., 2019) : The first graph-based knowledge tracing model that propagates students' knowledge states within the proposed graph. As there was no official open-source code available, we reimplemented their model using PyTorch.AKT-concept (Ghosh et al., 2020) : A variant of AKT where inputs are limited to concepts (excluding questions). We conducted experiments by making modifications to the inputs of the official open-source code of AKT.SAINT-concept (Choi et al., 2020a) : A variant of SAINT where inputs are limited to concepts (excluding questions). As there was no official open-source code available, we reimplemented their model using PyTorch and conducted experiments by modifying the inputs.ATKT (Guo et al., 2021) : Enhances the generalization capability of knowledge tracing models through adversarial training. We conducted experiments using their official open-source code and during the experimentation, we identified a bug within their code. This bug resulted in unintended data leakage. We proceeded to rectify this error by implementing necessary modifications.CL4KT (Lee et al., 2022) : Introduces multiple data augmentation strategies and employs contrastive learning to alleviate sparsity in student-interaction data. We conducted experiments by running their official open-source code. 
 
 Appendix: KTM (Vie & Kashima, 2019) : Modeling students' knowledge states using Factorization Machines.We conducted experiments by running their official open-source code.AKT (Ghosh et al., 2020) : Introduces the Rasch method for representing questions and proposes a context-aware Transformer architecture to simulate students' forgetting behavior for knowledge state tracing. We conducted experiments by running their official open-source code.SAINT (Choi et al., 2020a) : Fully employs a Transformer (Vaswani et al., 2017) architecture to model students' knowledge states. As there was no official open-source code available, we reimplemented their model using PyTorch.PEBG+DKT (Liu et al., 2020) : Enhances DKT by deeply exploring the relationship between questions and concepts to obtain pre-trained question representations. We conducted experiments by running their official open-source code.GIKT (Yang et al., 2021) : Utilizes Graph Convolutional Networks (GCN) to aggregate relationships between questions and concepts, along with a historical review module to trace students' knowledge states. We referenced their official open-source code(TensorFlow) and reimplemented their model using PyTorch.CDKT (Dai et al., 2022) : Enhances DKT by employing contrastive learning to learn informative question representations. As there was no official open-source code available, we reimplemented their model using PyTorch.DIMKT (Shen et al., 2022) : Explores question difficulty information and its relationship with students' knowledge states. We referenced their official open-source code(TensorFlow) and reimplemented their model using PyTorch.QIKT (Chen et al., 2023) : Designs question-sensitive cognitive representations for modeling student knowledge states and combines IRT for improved interpretability. We reimplemented their model by referring to their official open-source code.simpleKT (Liu et al., 2023b) : A simplified version of AKT, which simplifies the architecture of AKT without sacrificing too much performance. We reimplemented their model by referring to their official open-source code.AT-DKT (Liu et al., 2023a) : Enhances DKT by introducing two additional question-related tasks. We reimplemented their model by referring to their official open-source code.DTransformer (Yin et al., 2023) : Builds a knowledge state from questions to the knowledge level and maintains stable knowledge states using contrastive learning. We conducted experiments by running their official open-source code.C PERFORMANCE COMPARISON OF REKT-CONCEPT ABLATION STUDY F T+N PREDICTION For the concept-based KT models, referring to Table 2 , it is evident that the models with better performance include DKT+, DKT+forgetting, AKT-concept, and ATKT. However, due to DKT+'s reliance on recent response data to optimize the model, it is not suitable for T+N predictions. Additionally, we excluded ATKT from the T+N comparison due to identified errors in its code. DKT serves as a crucial baseline model for comparison in this study, hence its inclusion in the T+N evaluation.For the question-based KT models, referring to Table 3 , the models demonstrating better performance are AKT, CDKT, DIMKT, and QIKT.Under review as a conference paper at ICLR 2024 For the concept-based KT, Figure 5 illustrates the T+N experiments conducted across all datasets.Likewise, for the question-based KT, Figure 6 illustrates the T+N experiments conducted across all datasets(excluding ASSIST15 due to a lack of question information). It is evident that across nearly all datasets, whether at the concept-based KT or question-based KT, as elucidated in the "T+N Prediction" section, ReKT consistently demonstrates remarkable stability in performance. Furthermore, it is worth noting that ReKT not only exhibits a slower rate of decline but also significantly outperforms other models in terms of performance, underscoring ReKT's exceptional ability to uphold simplicity and effectiveness while retaining remarkable stability.FRU is shown in Figure 3 . We denote the time step as T , and for simplicity, we set the hidden layer dimension to be consistent as d. For any time stept ∈ [1, T ], Response t ∈ R 1×d , X t ∈ R 1×d , Z t ∈ R 1×d , I α ∈ R 1×d .Learnable parameters of FRU: The parameters that FRU learn are only Space complexity of FRU: The space complexity of Response t , X t , Z t , and I α is all O(1 * d).W 1 ∈ R 2d×d , b 1 ∈ R 1×d , W 2 ∈ R 2d×d , b 2 ∈ R 1×d ,Considering the total time step, their combined space complexity is O(T * 4 * 1 * d). In addition, the number of learnable parameters of FRU is 4d 2 + 2d, so the space complexity of FRU is O(4T d + 4d 2 + 2d).H THE DIFFERENCES AND ADVANTAGES BETWEEN FRU AND MLP, RNN, LSTM, AND GRUThe differences between FRU and MLP:• FRU is a sequence model, similar to recurrent neural networks such as RNN, while MLP is a model that processes structured data. • Both linear regression units of FRU have only one change, not multiple.The differences between FRU and RNN, GRU, and LSTM:• FRU reduces the output transformation layer.• Compared with RNN, FRU considers the forgetting process at each moment; compared with LSTM and GRU, their gates (such as LSTM's forgetting and input gates) are connected in parallel; FRU's foget-response-update is in series. • When calculating the current state, LSTM additionally uses global state updates, and GRU makes additional changes. FRU, on the other hand, directly relies on the state of the previous moment without involving additional changes. • FRU considers the case of processing sequences with non-uniform time intervals.• FRU has fewer learnable parameters than LSTM and GRU.The Advantages of FRU:• Experimental results show that as the core architecture of KT, FRU is simpler than other methods and can maintain equivalent or better performance while using less computing resources. • FRU is able to handle sequences with non-uniform time intervals. One of the most interesting applications of knowledge tracing is probably tracing students' knowledge states. Once a student's knowledge state is accurately captured, teachers can provide targeted guidance to address their weaknesses in understanding. In this section, we randomly selected students' interaction sequences on the ASSIST17 dataset for 15 time steps to compare the tracing of students' knowledge states between DKT-Q and ReKT. DKT-Q is a variant of DKT where the input is changed from concepts to questions, as shown in Figure 7 and Figure 8 . The y-axis represents specific questions, differentiated by different colored circles. The x-axis represents student interactions (time step, question and question related concept). If the student answered the current question correctly, the corresponding circle is displayed. If the student answered incorrectly, an additional white circle is added to indicate the difference. The value of each element in the matrix represents the student's mastery of a specific question at the corresponding time step. As shown in Figure 7 , DKT-Q can hardly trace the mastery of question 785 by the student (indicated by minimal changes in the student's mastery of question 785). This is because there is only one interaction about it in the interaction sequence (at time step t = 13). However, considering that question 785 is related to concept 78, and the student has actually practiced numerous exercises related to concept 78 (at time steps t = 3, 5, 7, 8, 9, 11), it is intuitive to expect that the student's understanding of question 785 should improve. However, DKT-Q cannot capture this, whereas ReKT can trace the improvement in the student's mastery of question 785 as they continuously practice exercises related to concept 78 (as shown in Figure 8 ). In addition, during time steps t = 5, 7, 9, 11, the student practices question 786 continuously. Intuitively, regardless of whether the student answers correctly or incorrectly, the student should gain knowledge from it (improving the mastery of question 786). However, DKT-Q shows a decrease in the student's mastery of question 786 at time steps t = 9, 11, which goes against intuition. On the other hand, ReKT can continuously traces the improvement in the student's mastery of question 786. This shows the advantage of ReKT in tracing students' knowledge state. Through the visualization results, teachers can clearly understand the student's mastery of certain questions and provide targeted exercises. For example, if it is observed that students have a low level of mastery of question 786 in the early stages of learning, the teacher can promptly provide targeted training to the students on this question.1,837 1,050 # Interaction 207,659 717,188 683,801 153,324 91,133 1,156,254 597,124 


Decision: The paper proposes a simple yet effective model for knowledge tracing (termed ReKT). The proposed ReKT model, inspired by the decision-making processes of human teachers, provides a novel method of representing student knowledge from multiple perspectives, supported by the lightweight Forget-Response-Update (FRU) framework. This framework notably enhances performance while significantly reducing computational requirements, as demonstrated through comprehensive experiments across seven datasets. The reviewers have acknowledged the strengths of the paper, including its clear motivation, innovative approach, and extensive evaluation, which collectively validate the effectiveness and efficiency of the proposed model, while some minor limitations and suggestions for improvement have also been noted.

Reviews: review: paper_summary: paper_summary: Advances in multimedia technology have made multimedia learning crucial. Knowledge Tracing (KT) monitors knowledge acquisition and predicts student performance. While current KT research focuses on complex deep learning models, this paper addresses fundamental KT challenges: knowledge state representation and core architecture design. The proposed ReKT model, inspired by human teachers, models student knowledge from questions, concepts, and domains. Using a lightweight Forget-Response-Update (FRU) framework, ReKT not only outperforms 22 state-of-the-art KT models on seven datasets, but does so with only 38% of the computing resources of other architectures. This superior performance offers new insights for future KT research. The code is available in supplementary materials.
strength_weakness: strength_weakness: 1. Exquisite and precise diagrams in Figure 1 and Figure 2 clearly explain the concept of this task and the proposed ReKT model. 
2. Starting from the basic concepts of the task, structuring the Knowledge Tracing (KT) problem with new ideas without using the popular and influential models of today is an excellent approach in itself.
3. The main learnable parameters are within the Forget-Response-Update (FRU) framework. This paper introduces a new concept that combines the use of the FRU framework, significantly reducing inference time while improving performance.
4. A wide range of experiments were conducted to verify the effectiveness and efficiency of the proposed method and detailed framework, resulting in significant improvements across most datasets. The proposed Forget-Response-Update (FRU) was also compared with existing frameworks with similar functionalities, demonstrating notable enhancements in effectiveness and efficiency.
limitations: limitations: 1. What is the unique proposition behind the creation of a new concept for the combination of old concepts? Could this hinder the model from establishing relationships between concepts? Or is it a case where the task does not necessitate such relationships?
2. In the ablation study of replacing the core architecture, Tables 5 and 6 demonstrate the performance of the model in question-based and concept-based scenarios. The Forget-Response-Update (FRU) proposed in this work performs relatively poorly in the concept-based setting. Are there any underlying reasons for this?
paper_summary: paper_summary: The paper proposes a simple yet effective model for knowledge tracing  (termed ReKT), which starts with a newly different research perspective for KT, i.e., considering multiple levels of the student's ability representations (including the question, concept, and domain) instead of a single level.  Besides, a lightweight unit forget-response update (FRU) is devised to aggregate corresponding ability representations in three levels. The paper executed extensive experiments on seven datasets by comparing over ten KT models, and the experimental results show the proposed ReKT achieves promising performance while maintaining good efficiency and less model complexity.
strength_weakness: strength_weakness: 1) The motivation of this paper is clearly introduced
2) The proposed ReKT model is  simple  regarding model architecture yet effective  regarding model performance as well as model interpretability .
3) The sufficient and comprehensive evaluations executed in the experiments.
limitations: limitations: 1) The difference between GRU and FRU is not clarified clearly;
2) Some important KT approaches through neural architecture search are missing in the related work;
3) There are some writing issues in the manuscript affecting the paper qualiity: on line 316, the sentence lacks main statement; on lines 265, 269, a blank space should be added  before citations; on line 327, the sentence is written too colloquially; and so many others. The author may double-check their paper to revised these issues.
4) The relation between this paper and ACM MM may be highlighted.
paper_summary: paper_summary: This paper revisits knowledge tracing models and introduces the ReKT model. Drawing inspiration from the decision-making process of human teachers, the ReKT model traces knowledge states from three distinct perspectives. Additionally, inspired by constructivism, the innovative FRU architecture, tailored for knowledge tracing, is presented. Extensive experiments validate the effectiveness of this approach.
strength_weakness: strength_weakness: 1.Online education or intelligent education is an important application scenario of multimedia technology, and the estimation of learners' cognitive states is a crucial issue in the application of intelligent education. This paper inspired by the decision-making process of human teachers, represents students' knowledge states from multiple perspectives, offering an intriguing research angle.

2.The ReKT model proposed in this paper, along with the FRU architecture, has been experimentally validated on multiple datasets and compared with numerous methods, demonstrating its effectiveness convincingly. The experimental data was comprehensive and highly credible. Extensive comparisons were conducted with 22 state-of-the-art KT models on 7 publicly available datasets. 

3.This paper is concise and easy to read, with well-organized writing structure, overall presenting a high-quality work.
limitations: limitations: 1.The FRU is interesting, but it can be further improved by considering learners' learning conditions such as learning rates to enhance its completeness. The author could explore these aspects further.

2.In Question-based Knowledge Tracing, further optimization of question representation should be considered, as this often has a positive impact on the model.

3. In the realm of sequence modeling, both RNNs and LSTMs have been widely acknowledged for their simplicity and effectiveness. However, it is imperative to consider the advantages that Fully Recurrent Unit (FRU) holds over these conventional models, particularly in the context of knowledge tracking. Please give more details.

Questions：

1.Some knowledge states, such as questions, are based on limited data, making it challenge to be well represented. How does the author consider meeting this challenge? 

2.What are the limitations of the ReKT model?
paper_summary: paper_summary: This paper proposes a Knowledge Tracing (KT) model named ReKT, which aims to emphasize the representation of knowledge state and the simplicity and power of the core architecture.  First, in line with the decision-making process of human teacher, ReKT models knowledge state from question, concepts and domains and utilizes three types of subsequences. Moreover, the author designs a lightweight FRU module for ReKT. Extensive experiments on 7 datasets under different settings demonstrates the effectiveness of ReKT.
strength_weakness: strength_weakness: 1. The paper is well-written and well-organized with good textual expression, mathematical notation, and clear figures.
2. The proposed method is highly acceptable and demonstrates some innovation. Considering students' knowledge state from these three angles aligns closely with educational practice. The proposed FRU, tailored for ReKT, effectively integrates the time interval of items within the subsequence, which complements the previous method.
3. The author conducts extensive experiments across 7 datasets, comparing against 22 baseline models under both concept-based and question-based settings to analyze RKT. The experimental results are good.
limitations: limitations: 1. The motivation is not strong enough. Although I find the model framework proposed by the author intriguing, the motivation and challenges described in the introduction fails to engage me. It doesn't clearly articulate why simplifying the KT model is necessary and what problems complexity might pose for KT.
2. The article's core objective is to create a KT model that is both simple and powerful. However, the main experimental results only show the model's accuracy performance without including any metric to measure simplicity, such as parameter quantity in Table 5. I think this omission is somewhat inappropriate.
3.  Figure 4 and 5 need improvement in terms of aesthetics, as the circles in the graphs appear flattened. And some minor formatting issues need to be adjusted, such as removing spaces before citations.