Remembering is Not Applying: Interpretable Knowledge Tracing for Problem-solving Processes

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 12:30:22

Abstract: Knowledge Tracing (KT) is a critical service in distance education, predicting students' future performance based on their responses to learning resources. The reasonable assessment of the knowledge state, along with accurate response prediction, is crucial for KT. However, existing KT methods prioritize fitting results and overlook attention to the problem-solving process. They equate the knowledge students memorize before problem-solving with the knowledge that can be acquired or applied during problem-solving, leading to dramatic fluctuations in knowledge states between mastery and non-mastery, with low interpretability. This paper explores knowledge transformation in problem-solving and proposes an interpretable model, Problem-Solving Knowledge Tracing (PSKT). Specifically, we first present a knowledge-centered problem representation that enhances its expression by adjusting problem variability. Then, we meticulously designed a Sequential Neural Network (SNN) with three stages: (1) Before problem-solving, we model students' personalized problem space and simulate their acquisition of problem-related knowledge through a gating mechanism. (2) During problem-solving, we evaluate knowledge application and calculate response with a four-parameter IRT. (3) After problem-solving, we quantify student knowledge internalization and forgetting using an incremental indicator. The SNN, inspired by problem-solving and constructivist learning theories, is an interpretable model that attributes learner performance to subjective problems (difficulty, discrimination), objective knowledge (knowledge acquisition and application), and behavior (guessing and slipping). Finally, extensive experimental results demonstrate that PSKT has certain advantages in predicting accuracy, assessing knowledge states reasonably, and explaining the learning process.

Main paper: 
 introduction: 
 Digital multimedia technology enables the creation and delivery of high-quality educational content in various digital formats, promoting distance education's popularization. Consequently, online teaching modes represented by MOOCs, micro-courses, and blended learning have emerged [12, 13] . The phenomenon of knowledge reversal occurs when instructional procedures effective for beginners are ineffective for experts [17] . Therefore, educators must be able to continuously assess and monitor learners' knowledge mastery during the learning process to dynamically determine appropriate teaching resources, especially in online learning scenarios.Knowledge tracing (KT) is an influential research area in distance education, which can automatically track students' knowledge levels at various stages and has been widely used in adaptive learning systems [2, 19] . The goal of KT involves two stages. Firstly, assessing the development of students' knowledge states according to records, and secondly, predicting their performance on specific problems using the assessed knowledge states. Therefore, KT offers two key applications for intelligent educational services: dynamic adjustment of teaching strategies based on assessment states and personalized resource recommendation based on prediction results.Early KT models, such as Bayesian Knowledge Tracing (BKT) [6, 44] based on Hidden Markov Models, and Performance Factor Analysis [30] based on logistic functions, used educational parameters to estimate students' mastery probabilities. These models were characterized by high interpretability but low precision in prediction. As smart education progresses, deep learning-based approaches (DLKT) have become mainstream, significantly improving the performance of KT tasks [11, 26, 31, 45] . However, due to the black-box nature of deep learning, these models have lower transparency and interpretability. Similar to how good teachers need to understand students, a good KT model should provide highervalue educational services, such as attribution analysis, which helps understand the 'why' -not just the 'what'.Fortunately, many DLKT methods are addressing this issue by striving to enhance the interpretability of their models. Methods like Deep-IRT [42] and Deep-IRTw [38] combine KT with Item Response Theory (IRT). SAKT [28] , AKT [11] utilize the weights of the attention mechanism to explain the relationship between historical interactions. IKT [23] sets knowledge mastery, learning transfer ability, and problem difficulty as three educationally meaningful parameters. RCKT [7] , CMKT [47] enhance model interpretability through causal inference.However, the aforementioned methods primarily focus on prediction and overlook the consideration of the learning process, resulting in an unreasonable assessment of knowledge mastery patterns. LPKT [33] recognized the limitations of previous models, which assumed that learners' knowledge states would decline once they answered incorrectly. LPKT applied knowledge gain to measure the value of error interactions. From a cognitive perspective, Additionally, memorized knowledge will be forgotten over time. learners' knowledge mastery curve is relatively stable and does not fluctuate drastically in a short period. Previous work [34, 43] has noted this irrationality and mitigated the fluctuations by designing a punishment mechanism, albeit at the cost of sacrificing the model's performance.To address the irrationality of knowledge state assessment, enhance model interpretability, and improve performance, we deconstruct students' learning processes from the perspective of human problem-solving. Engaging in exercises is a typical problem-solving process. As illustrated in Figure 1 , students enhance their understanding of problems through deep thinking, continuously explore solutions, and ultimately achieve consolidation of knowledge. The entire problem-solving process involves the acquisition, application, internalization, and forgetting of memorized knowledge in the brain. However, existing KT methods overlook learners' problemsolving processes, equating memorized knowledge with acquired or applied knowledge, leading to drastic fluctuations in knowledge states. According to problem-solving theory [3, 8, 9, 27] , knowledge acquisition involves learners organizing implicit and explicit knowledge relevant to problems, while knowledge application involves applying this knowledge to seek a solution. Remembering does not equal applying. Familiarity with discrete knowledge concepts (KCs) does not imply that students can effectively organize them or convert abstract knowledge into specific applications when answering problems. The common saying that knowing the path and walking the path are two different things reflects this phenomenon. Therefore, our work delves into various stages of problem-solving, reproducing the learning process by analyzing how students understand problems, acquire, apply, internalize, and forget knowledge.In this paper, considering the effects of knowledge transformation mentioned above, we propose an interpretable knowledge tracing approach for problem-solving processes (PSKT). In PSKT, we first enhance the representation of knowledge-centered problems by learning the rich attributes of the problem. Then, we present a Sequential Neural Network (SNN) to simulate knowledge transformation in problem-solving process. Before problem-solving, PSKT models students' personalized problem space and knowledge acquisition degree. During the problem-solving process, PSKT captures learners' goal-oriented knowledge application and attributes learners' response through educational parameters, namely problem factors (difficulty, discrimination), learner knowledge factors (knowledge acquisition and application), and learner behavior factors (guessing, slipping). After problem-solving, PSKT designs an update indicator to measure the degree of learners' knowledge internalization and models the forgetting effect over time. Finally, we conducted extensive experiments on five real-world datasets, demonstrating the effectiveness of PSKT. Additionally, PSKT showed outstanding performance in modeling reasonable knowledge states, explaining prediction, and learning reliable problem difficulty. The main contributions of our paper are summarized as follows:• We quantify the learner's knowledge transformation by deconstructing the process of solving a particular problem, ensuring that knowledge state changing is reasonableness. • We propose a novel Sequential Neural Network (SNN). The design of the SNN follows the problem-solving theory and the active construction principle of constructivist learning theory, providing good interpretability. • Extensive experiments demonstrate the effectiveness of PSKT.In addition, PSKT can learn credible problem difficulty. 
related work 2.1 item response theory: 
 Item Response Theory (IRT) is a classic measurement theory rooted in education and psychology. It calculates the probability of a correct response based on the latent traits of the student and the characteristics of the problem. Although the traditional IRT model is highly interpretable, it treats students' latent traits as a set of static parameters, making it suitable only for a single assessment scenario. Recognizing the limitations of IRT, many researchers have incorporated it into KT for process assessment. [18] integrates the IRT and BKT models to model students' learning abilities. Deep-IRT [42] fits student ability through DKVMN [45] and uses IRT to compute the probability of a student's correct response. AKT [11] uses IRT to enhance the personalized embedding of questions and responses, while PKT [34] applies IRT to fit students' knowledge mastery states.PSKT is inspired by IRT and attributes answering performance to objective problem factors (difficulty, discrimination) and subjective learner factors (knowledge application, guessing and slipping). These rich educational parameters make our model interpretable while maintaining accuracy.Knowledge tracing tracks students' dynamic knowledge states by modeling their historical learning sequences. It is suitable for multiple assessment scenarios and performs process measurements on students. Early KT methods were based on hidden Markov models, which evaluated student performance by modeling four parameters: initial mastery, knowledge mastery transformation, guessing and slipping [6, 29, 44] . Neural networks have greatly improved the performance of KT and have become the current mainstream method. DKT [31] introduced deep learning into KT for the first time and applied RNN/LSTM to model students' knowledge changes. Subsequent work [21, 25, 43] improved upon this basis. [1, 42, 45] use memory networks to store potential KCs and update students' knowledge mastery. [11, 16, 28] use attention mechanism to model the relationship between interaction histories, and [26, 36, 40] apply graph neural networks to model the graph structure between KCs.Recent research has emphasized the importance of modeling the learning process. LPKT [33] and FKT [15] finely model learning and forgetting by incorporating response time and interval time. LBKT [39] highlighted the impact of answering speed, prompts, and other behaviors. However, these models suffer from poor interpretability and controllability. In contrast, our primary focus is on providing reasonable predictive explanations from an educational perspective. We disaggregate students' problem-solving processes to make multifaceted attributions of answering responses, rather than focusing on additional features of the learning process. 
problem definition: 
 Suppose that in an online learning platform, there is a set of students 𝑆 = 𝑠 1 , 𝑠 2 , ..., 𝑠 |𝑆 | , a set of problems 𝑃 = 𝑝 1 , 𝑝 2 , ..., 𝑝 |𝑃 | , and a set of knowledge concepts 𝐾 = 𝑘 1 , 𝑘 2 , ..., 𝑘 |𝐾 | . We denote the exercise records of a student as 𝑋 𝑁 = { 𝑋 1 , 𝑋 2 , ..., 𝑋 𝑁 } , where 𝑁 represents the total number of exercises. 𝑋 𝑛 = (𝑡𝑠 𝑛 , 𝑝 𝑛 , 𝑘 𝑛 , 𝑟 𝑛 ) , 𝑛 ∈ 𝑁 is the most basic practice unit, denoting that the student's response to problem 𝑝 𝑛 at timestamp 𝑡𝑠 𝑛 is 𝑟 𝑛 . Where 𝑟 𝑛 ∈ { 0, 1} , 0 denotes an incorrect answer and 1 denotes a correct answer.Problem Definition. Given a student's historical learning sequence 𝑋 𝑛 , the task of KT is to evaluate the student's knowledge states based on 𝑋 𝑛 and predict the probability of correctly answering the next problem, that is, 𝑝 (𝑟 𝑛+1 = 1|𝑝 𝑛+1 , 𝑋 𝑛 ). 
the pskt model: 
 Students increase their knowledge by practicing problems, so the core of PSKT is quantifying learners' knowledge growth by deconstructing their problem-solving process. As shown in Figure 2 , to model the learners' problem-solving process and analyze their knowledge transformation, we designed a sequence neural network (SNN). The PSKT model comprises four main modules: problem representation, knowledge acquisition, application, and updating. First, we present a knowledge-centered problem representation that enhances its expression by adjusting problem variability. Then, we model learners' personalized problem perception and knowledge acquisition degree in the knowledge acquisition module. Next, PSKT measures the knowledge application level based on knowledge acquisition and calculates problem response by IRT. Finally, the knowledge update module quantifies the degree of knowledge internalization and forgetting. The design of the SNN is inspired by problem-solving theory [9] and constructivist learning theory [10] , ensuring good interpretability.Understanding the nature, scope, and relevant factors of a problem is prerequisite for solving it. Early KT methods used KCs to replace problems, or directly concatenated problems and KCs. However, these methods simply fused information without considering the primary and secondary relationships of the problems. Since the goal of exercises is to learn knowledge, the KCs examined by the problems are of utmost importance. Moreover, problems that test the same KCs can have different effects on learning and performance due to differences in difficulty and discrimination. In the PSKT, we consider both the primary-secondary relationships and problem attributes to enhance problem representation:EQUATIONEQUATIONwhere W 𝑝𝑎 ∈ R 𝑑 ℎ ×1 , 𝑑 ℎ is embedding dimension. 𝑑 𝑛 isAccording to problem-solving theory, knowledge acquisition requires learners to first clarify the problem, ensure an understanding of critical aspects of the problem (problem perception), and then acquire knowledge about the problem (knowledge acquisition).(1) Problem Perception. The learner's solution to a problem begins with the construction of an internal representation of the external problem statement, i.e., the "problem space" (including the initial state, the goal state, and the adapted operators) [14] . To simulate the learner's personalized perception of the problem, we integrate the learner's prior knowledge H 𝑛-1 and problem representation P 𝑛 to represent the initial state and goal state in the problem space. We use the difference between the goal state EP 𝑛 and the initial state SP 𝑛 to represent the applicable operations OP 𝑛 :EQUATIONEQUATIONEQUATIONwhere⊕ is concatenate, W 𝑃𝑃 ∈ R (𝑑 ℎ +|𝐾 | ) × |𝐾 | and |𝐾 | is the total number of KCs.(2) Knowledge Acquisition. Knowledge acquisition involves the systematic generation and integration of information. This includes acquiring knowledge from the external environment as well as constructing new knowledge through internal thought processes. In the KT scenario, external knowledge comes from the initial state of the problem SP 𝑛 (known conditions), and internal knowledge is derived from the learner's previous knowledge state H 𝑛-1 . We use the tanh to generate candidate values. The specific formula is:EQUATIONsince learners have limited memory capacity, they will selectively focus on the most relevant and core aspects of the problem during the knowledge acquisition process in order to conduct more effective planning [8] . Therefore, we further design a gate to select and retain the essential parts through the problem information.EQUATIONEQUATIONwhere 𝜎 is sigmoid,W 𝐾𝐴 ∈ R 2|𝐾 | × |𝐾 | , W Γ𝐾𝐴 ∈ R 3|𝐾 | × |𝐾 | .Acquiring knowledge is only the beginning of problem-solving.What is more important is how to apply knowledge to actual problems or situations. During practice, learners transform acquired theoretical knowledge into practical abilities, propose solutions (knowledge application), and then implement actions to achieve the expected goals (problem answering).(1) Knowledge Application. Knowledge application involves transforming abstract knowledge into concrete action plans and adapting and refining these in practice. The solution is contained in the adapted operators of problem space, and we use tanh to generate possible solutions. Learners need to flexibly apply knowledge to make a series of dynamic decisions and continuously monitor the consequences of these decisions to achieve the goal as much as possible. Therefore, we designed a gate Γ 𝐾𝑃 led by problem goals and knowledge acquisition to determine the learner's final solution.EQUATIONEQUATION(2) Problem Answering. Applying knowledge is to achieve goals. IRT shows that whether students can answer problems correctly depends not only on their knowledge application but also on the problem itself. Inspired by the Four-parameter IRT and BKT, we simulate students' responses by fully considering problem factors (difficulty, discrimination), learner knowledge factors (knowledge application), and learner behavioral factors (guessing and slipping). The core functions are as follows:EQUATION) where W 1 ∈ R ( |𝐾 |+𝑑 ℎ ) ×𝑑 ℎ , W 2 ∈ R 𝑑 ℎ × |𝐾 | .The first term in Equation (11) represents the probability that the learner has not mastered the knowledge but guessed correctly, and the second term indicates that the learner can answer correctly without slipping. 𝐷 is a constant with a value of 4 × 1.7 [4] . 𝐺 𝑛 is guessing, and 𝑆 𝑛 is slipping: EQUATIONAfter practice, the learner's knowledge state changes. Constructivist learning theory [10] suggests that increasing knowledge mastery reflects knowledge internalization, while forgetting leads to a reduction in knowledge. Thus, learner state updating includes both knowledge internalization and forgetting.(1) Knowledge internalizing. Knowledge internalizing is how learners transform external knowledge, experience or information into thinking structures and cognitive patterns. In the problemsolving process, the learner's new knowledge originates from the problems, so we fuse all the information of the problems to obtain the candidate value of knowledge increment. Constructivist learning theory shows that learning is an active construction process, so we tend to think that the entire practice process positively impacts learners, and use (tanh ( ) + 1) /2 as a non-negative constraint operator to ensure that the increment is non-negative.EQUATION)Through internalization, learners do not simply accept information but integrate it into their cognitive framework. Thus, even if different learners answer the same problem, the level of knowledge they internalize will be different due to differences in the knowledge they acquire, the knowledge they apply, and the results of their answers. To this end, we propose a growth indicator to determine the learner's personalized knowledge increment.EQUATIONEQUATIONwhere R 𝑛 is response embedding,W Γ𝐾𝐼 ∈ R (2|𝐾 |+𝑑 ℎ ) × |𝐾 | .(2) Knowledge Forgetting. Limited by the human memory system, although learners may delay the occurrence of forgetting through review and reinforcement learning, forgetting is still a continuous process [22] . Therefore, current knowledge may be forgotten over time intervals. In order to model the complex forgetting effect, we design a forgetting gate as follows:EQUATIONEQUATIONwhere time interval is 𝑖𝑡 𝑛 = 𝑡𝑠 𝑛+1 -𝑡𝑠 𝑛 , in minutes. IT 𝑛 is embedding of 𝑖𝑡 𝑛 , and we set all intervals greater than 1 month to 1 month.To train all parameters Θ in PSKT, we use cross-entropy log loss between the predicted response r𝑛 and the actual response 𝑟 𝑛 as the objective function, minimized with the Adam optimizer:EQUATION 
experiments: 
 In this section, we conduct extensive experiments to answer the following questions:• RQ1. What is the predictive performance of PSKT?• RQ2. Can PSKT reasonably explain the learning process? 1 . We filtered learning records for all datasets with missing relevant KCs and removed problems with fewer than 10 occurrences and learners with fewer than 3 response records. Due to the large size of the Ednet-KT1, which contains 95,293,926 records, we randomly selected 100,000 learners [34, 41] .For datasets where a problem assesses multiple KCs, we use the average of all KC embeddings to represent the final KC embedding.We set all input sequences to a fixed length of 100. For sequences of length greater than 100, we cut them into several unique subsequences. We performed the 5-fold crossvalidation. For each fold, 80% of the learned sequences are split into training and validation sets (the ratio is 8:2), and the remaining 20% serves as the test set. The parameters 𝑑 ℎ , learning rate, 𝜆 are set to 256, 0.001, 0.5. On the smaller ASSIST12 and ASSIST17 datasets, the batch size is 64, while on other larger datasets is 512. We used an early stopping mechanism with 3 epochs of patience. For fairness, the baselines' hyper-parameters are carefully tuned to achieve optimal performance. All experiments were performed on a Linux server with an NVIDIA GeForce RTX3090. We measure model performance in terms of area under the curve (AUC), accuracy (ACC), root mean square error (RMSE), and Pearson correlation squared (R2). Consistent with previous work [15, 33, 35] , we set the accuracy threshold to 0.5. We used the average result of 5-fold cross-validation on the test set as the experimental result. Since the Eedi does not have response time features, the FKT and LPKT results on Eedi are after eliminating response time.The experimental results are shown in Table 2 , and PSKT outperforms all baseline methods on all datasets and metrics. The excellent performance of PSKT shows that comprehensively modeling the learner's problem-solving process significantly impacts understanding the learning process and predicting learning performance. Secondly, compared with the most advanced deep learning models, the superiority of the PSKT framework also shows that the factor that restricts the performance of the current KT method is not the expressive ability of the algorithm but the design of a scientific framework that conforms to students' cognitive rules.Third, the psychometric model IRT uses educational parameters to enhance the interpretability of the model. Among the models combined with IRT (AKT, Deep-IRT, PKT, PSKT), PSKT performs best, further proving the PSKT algorithm mechanism. The superiority of it takes into account both high accuracy and interpretability.To prove that PSKT can capture students' reasonable knowledge states, we conducted experiments on learning process analysis. In Figure 3 , we randomly select a student from each dataset and visualize their learning process when they practiced the same KC consecutively. We illustrate the superiority of PSKT in two aspects.(1) Knowledge state rationality. The rationality of the knowledge state includes the stability of the knowledge state curve and the rationality of the knowledge state value. First, the knowledge state curve fitted by PSKT is more stable. We quantify fluctuations based on the differences in knowledge states for all KCs between each student's adjacent two responses. As shown in the box plots in Figure 4 (a), PSKT is the most stable across the three models. Learners' knowledge mastery is directional and will not fluctuate drastically in a short period [24] . As shown in Figure 3 , the knowledge concept curves of Deep-IRT and DKT often decline and rise sharply. In contrast, PKT has more minor fluctuations because it designs a penalty mechanism for knowledge state changes. However, this mechanism can only alleviate the irrationality but does not really solve the problem and sacrifices the model's performance. On the contrary, PSKT is based on human problem-solving theory and uses a scientific and reasonable framework to make the curve more directional. Secondly, PSKT tracked more reasonable knowledge state values. DKT, Deep-IRT, and PKT obtained high knowledge values (>0.5) when learners responded correctly and decreased when incorrect. In ASSIST12, when the student answered the 4th (indices start from 0) problem incorrectly, the PKT knowledge value dropped from 0.56 to 0.30 and significantly increased when he/she answered the last three problems correctly (0.31→0.71). PSKT believes learners' knowledge states have always been relatively stable (all around 0.55) and attributes correct answers to the improvement of students' application levels. First, the 4th problem was the most difficult of the whole practice (56% error rate), so the student answered incorrectly because of a mismatch between knowledge state and difficulty rather than a decline in knowledge. The reason that learners can answer the last three exercises correctly is not because their knowledge has increased significantly but because the learners' knowledge application level has improved during the practice.70 7 7 1 6 7 1 6 2 6 1 2 2 6 1 2 2 6 1 2 2 6 7 27(2) Learning process interpretability. Compared with other methods, PSKT provides more educational parameters to explain the learning process. DKT attributes student performance entirely to knowledge mastery, Deep-IRT adds the effect of problem difficulty, and PKT considers student guessing and slipping behavior. However, these models rely heavily on a sudden drop in knowledge mastery to explain responses incorrectly, whereas PSKT provides a more reasonable explanation. For example, on the Eedi dataset, the student incorrectly answered six consecutive problems that examined 154 KC, which DKT and PKT attributed to a decrease in students' knowledge mastery. We think that the practice process positively affects learners and that students' knowledge mastery does not decline significantly during continuous answering. Specifically, although students answered problems 1 and 2 correctly, PSKT considered that learners' mastery of this knowledge was still only about 0.5 because of the low difficulty of these two problems. The student answered the last 6 problems incorrectly, though, which PSKT attributed to the high difficulty and the low level of knowledge application. At the same time, students correctly answered the first problem (more difficult) but incorrectly answered the last two problems, which PSKT attributed to the slipping.In this section, we conduct some experiments to demonstrate the reliability of the problem representation learned in PSKT, including the effectiveness of problem difficulty assessment and the interpretability of the problem representation.(1) Validity of problem difficulty assessment. Since there is no problem difficulty indicator in all datasets, to evaluate the effectiveness of the difficulty learned by PSKT, we refer to the Classical Test Theory [37] and previous KT work that considers difficulty [24, 32, 46] , and defines the statistical difficulty of the problem as the error rate. We conducted a Pearson correlation test between the two difficulties. The experimental objects are the problems involved in the test set. The experimental results are shown in subgraph (b1) in Figure 4 . PSKT learned difficulty is strongly correlated with statistical difficulty (>0.7) on all datasets, which demonstrates the reliability of its problem difficulty. In addition, the scatter plot on ASSIST17 also shows a certain degree of correlation and consistency between the two difficulties (Figure 4 (b2)).(2) Interpretability of problem representation. We visualize the correlation weights between problems by computing the cosine similarity of the problem representations, as shown in Figure 5 . We randomly selected eight questions on ASSIST17. These problems tested two KCs (9, 74) . First, the correlation weights tend to be higher between problems that test the same KC, such as problems 1, 2, and 3. Secondly, when different problems have the same KC, the closer the difficulty and discrimination of the problems are, the higher the similarity, for example, [9,0.60,0.81] and [9,0.58,0.85].We explore the impact of each module in PSKT through ablation experiments. As shown in Table 3 , we selected 6 variants, each removing an element from the PSKT. PR is replacing the problem representation with concatenate, EP denotes removing problem perception, KA means removing knowledge application, and KP denotes removing knowledge acquisition. KI is knowledge internalization without considering knowledge acquisition and Knowledge application, KF represents removing knowledge forgetting.First, knowledge forgetting is vital in the learning process, and deleting this module causes the most significant performance drop. According to the design of PSKT, if forgetting is not considered, the learner's knowledge mastery will continue to increase (or remain unchanged) as the number of exercises increases, making it challenging to identify incorrect answers. Secondly, the impact of problem perception is also significant. This is because problem perception involves modeling the problem's initial state, the target state, and the adapted operator. These three variables are throughout the entire problem-solving process. Third, both knowledge acquisition and knowledge application contribute to the performance of PSKT, proving that knowledge acquisition and knowledge application are both indispensable processes for problem-solving. 
conclusion: 
 In this paper, we propose a problem-solving process-oriented knowledge tracing model (PSKT) to explore the cognitive process of how students transform theoretical knowledge into practical skills. According to the three stages of practice, before problem-solving, we first obtain students' personalized perceptions of the problem, and simulate the students' acquisition of the problem-related knowledge.During problem-solving, we model the learner's level of knowledge application toward the problem target and use educationally meaningful parameters for performance attribution to predict the learner's response. After problem-solving, we design an update indicator to measure their level of knowledge internalization and the forgetting effect over time intervals. Finally, we conducted extensive experiments to validate the effectiveness of PSKT. In addition, PSKT not only performs well in response prediction, but also more reasonably evaluates learners' knowledge mastery patterns, provides educational insights into students' success or failure in problems. Looking forward, PSKT still has room for improvement.For example, we can consider collecting the details of students' answers to problems and using large language model technology to capture students' knowledge transformation in a more fine-grained manner from the students' answer steps. 
 
 Appendix: https://sites.google.com/site/assistmentsdata/2012-13-school-data-withaffecthttps://sites.google.com/view/assistmentsdatamining/datasethttps://github.com/riiid/ednethttps://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=1198https://eedi.com/projects/neurips-education-challenge 


Decision: All the reviewers acknowledged this paper's significant contribution and improved performance in knowledge tracing within problem-solving contexts. A common concern among the reviewers was the lack of clarity on transparency and interpretability. Upon reviewing both the paper and the authors' responses, it is clear that the paper introduces fresh ideas—integrating insights from constructivist learning theories and incorporating the Rasch model—and is well written. The authors have satisfactorily addressed the reviewers' concerns within the space limitations. One reviewer did not enter a final rating but acknowledged checking the rebuttal and decided to maintain their score of borderline accept.

Reviews: review: paper_summary: paper_summary: This paper focuses on knowledge transformation in the problem-solving process and proposes an interpretable Problem-solving Knowledge Tracing (PSKT) method. Unlike other studies, the authors first propose a knowledge-centered representation of the problem, which is enhanced by adjusting the variability of the problem. Meanwhile, the authors elaborated a Sequential Neural Network (SNN) in three stages (before problem-solving, during problem-solving, and after problem-solving).
strength_weakness: strength_weakness: 1.PSKT exhibits outstanding performance compared to existing knowledge tracing models.
2.The authors conducted sufficient experiments and provided in-depth analysis of the results.
3.The manuscript is well-written, with clear logic, a rational experimental design.
4.The figures are well-drawn and contribute to a better understanding of the model.
5. Problem solving theory as an entry point is interesting.
limitations: limitations: 1. This paper starts with the problem-solving theory to construct the model. Can you explain in detail how each component of the PSKT model fits with the problem-solving theory? You can list the corresponding theoretical basis. If you can answer my question, I am willing to give you a higher score.
2. Figure 2 could benefit from some legend explanations to clarify the meaning, as excessive letters and symbols make it difficult to discern. The inputs and outputs of each module are not very apparent.
3. As far as I know, there are many KT model studies on learning processes. What are the significant differences between the model design proposed by the author and them?
4. The font in Figures 3 and 4 is too small, and there is no necessary textual description, which makes it difficult for readers to understand the meaning conveyed by these figures. For example, what do "Knowledge Acquisition" and "Knowledge Application" mean in Figure 3?
5. The author's motivation primarily mentions that PSKT focuses on the interpretability of the model. As far as I know, the visualization of knowledge state evolution, including the visualization of problem difficulty and discrimination, has been addressed in most KT articles, and the novelty of interpretability related to problem-solving theory has not been observed.
paper_summary: paper_summary: This paper introduces the Problem-solving Knowledge Tracing (PSKT) model that adeptly captures the dynamic knowledge state of students across three distinct stages: before, during and after problem-solving, thus offering a comprehensive framework for understanding the learning process. By integrating insights from constructivist learning theories and incorporating the Rasch model, the representation of the question and the interpretability of knowledge application stage are consequently enhanced. The experimental results on several public datasets demonstrates the effectiveness of the PSKT model.
strength_weakness: strength_weakness: 1. The authors investigate the constructivist learning theories and Rasch model to enhance the PSKT’s interpretability.
2. The paper conduct experiments and analyses on the interpretability of the PSKT model including the stability of the knowledge state and the correlation between questions’ representation.
limitations: limitations: I have the following concerns and I hope the authors could make some further illustrations:
Q1: As LPKT is specifically focusing on the modelling of learning process, it’s recommended to compare the knowledge state rationality and interpretability with LPKT.
Q2: Since the authors have validated the interpretability of problem factors, e.g. assessing the generated problem difficulty’s correlation with the statistical difficulty. I hope the authors could also investigate the interpretability of the learner knowledge factors, i.e., guessing and slipping factor. This extension could involve exploring whether there exist discernible insights into potential significant differences in the guessing and slipping factors among students at varying levels of knowledge mastery. 
Q3: In Figure 3, the consistent centrality of the knowledge state of PSKT around 0.5, irrespective of whether the majority of questions are answered correctly or incorrectly, with minimal fluctuations, appears unreasonable. The provided explanation is not very convincing, and it is advisable for the authors to provide a more in-depth analysis to elucidate this observation.
paper_summary: paper_summary: The paper presents a significant advancement in the field of knowledge tracing by addressing the application of knowledge in problem-solving contexts. However, it would benefit from addressing the following weaknesses, particularly enhancing the discussion on practical implementation challenges and model generalizability. Enhancing the clarity and depth of theoretical motivations and providing a more comparative analysis of computational efficiency would also strengthen the paper.
strength_weakness: strength_weakness: 1.	Interpretable Model: The PSKT model offers high interpretability, enabling predictions of student performance and simulations of knowledge transformation during problem-solving.
2.	Problem-Solving Focus: It emphasizes problem-solving processes, enhancing its application in distance education and adaptive learning systems.
3.	Educational Parameters: PSKT utilizes educationally relevant parameters to assess learner knowledge states, promoting a deeper understanding of the learning process.
4.	Comprehensive Framework: This model incorporates multiple stages of learning, including knowledge acquisition, application, internalization, and forgetting, offering a comprehensive view.
5.	Performance: It outperforms existing knowledge transformation (KT) methods in various datasets, showcasing its predictive accuracy and effectiveness in assessing knowledge states.
6.	Extensive Evaluation: Validated through rigorous testing on five real-world datasets, demonstrating its robustness and effectiveness.
7.	Educational Insights: Provides valuable insights into students' learning processes, aiding educators in refining teaching strategies.
limitations: limitations: 1.	Challenges with the Problem-Solving Model: The paper highlights that the interpretability of problem-solving knowledge tracing (PSKT) models still faces challenges. While the models strive to interpret knowledge states and problem-solving processes, the precise mechanisms by which knowledge is transformed during problem-solving activities are not thoroughly explained or understood. This leaves a gap in how accurately the model can interpret complex learning and application processes.
2.	Modeling Knowledge Application: The paper emphasizes that current knowledge tracing methods, including PSKT, primarily focus on modeling knowledge application as a uniform process without considering the variability in how individuals apply knowledge in diverse contexts. This could lead to oversimplifications in the models, potentially misrepresenting the ways students apply learned concepts to solve problems.
3.	Need for Enhanced Data Collection: There is a suggestion for improvement in data collection methods to capture finer details of students’ problem-solving processes. The current models may not effectively capture all variations in how knowledge is applied, which is critical for improving the accuracy and utility of knowledge tracing systems.
4.	Potential for Improvement: The authors suggest that there is room for enhancing the model by collecting more detailed data on students’ responses to problems and integrating large language model technology. This indicates a recognition of the current limitations in capturing fine-grained details of students' knowledge transformation during problem-solving processes.
5.	The introduction and literature review sections could be expanded to better situate the research within the broader context of adaptive learning technologies and educational data mining.
6.	Algorithm Transparency and Interpretability: Despite the PSKT model's emphasis on interpretability, the explanations of the model's internal decision processes remain somewhat abstract. For instance, a clearer explanation of how each parameter of the model elucidates cognitive and behavioral changes during the learning process and their direct applications in educational practice could be more explicitly stated.
7.	Logical Flow and Organization: Some sections require stronger structure and logical coherence. Especially in the methodology and results sections, the way information is organized can sometimes seem disjointed, potentially hindering the reader's understanding of the entire text.
8.	Consistency in Terminology Use: The article needs more consistent definitions and use of key concepts such as "knowledge acquisition," "knowledge application," and "problem-solving." Clearer definitions could prevent confusion.