# Calibrated Q-Matrix-Enhanced Deep Knowledge Tracing with Relational Attention Mechanism
## 4 Methods
### Results and Discussion
#### 5.2.3 Knowledge State Evolution Description (RQ3)  
This section aimed at validating the performance of the QRAKT model, the DKT model, and the DKT+ model in tracking a student's knowledge state. The impacts of the QRAKT model on improving student performance are also illustrated in this section. Heat maps were applied to describe the evolution of the student's knowledge state. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **ASSIST2012** & & **Eedi** & \\ \hline  & AUC & ACC & AUC & ACC \\ PA & 0.753 & 0.689 & 0.515 & 0.541 \\ PA+CQ & 0.748 & 0.676 & 0.505 & 0.528 \\ PA+RM & 0.732 & 0.668 & 0.500 & 0.530 \\ PA+CQ+RM & 0.731 & 0.665 & 0.489 & 0.525 \\
**QRAKT** & **0.771** & **0.702** & **0.707** & **0.670** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparison of the results when the positional attention layer (PA) is removed. The best model is boldfaced. After removing the PA, the AUC of the model on the two datasets decreased by about 1.8% and 19.2%. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **ASSIST2012** & & **Eedi** & \\ \hline  & AUC & ACC & AUC & ACC \\ CQ & 0.754 & 0.695 & 0.681 & 0.633 \\ PA+CQ & 0.748 & 0.676 & 0.505 & 0.528 \\ CQ+RM & 0.735 & 0.670 & 0.684 & 0.632 \\ PA+CQ+RM & 0.731 & 0.665 & 0.489 & 0.525 \\
**QRAKT** & **0.771** & **0.702** & **0.707** & **0.670** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Comparison of the results when the calibrated Q-matrix (CQ) is removed. The best model is boldfaced. After removing the CQ, the AUC of the model on the two datasets decreased by about 1.7% and 2.6%. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **ASSIST2012** & & **Eedi** & \\ \hline  & AUC & ACC & AUC & ACC \\ RM & 0.740 & 0.678 & 0.684 & 0.637 \\ PA+RM & 0.732 & 0.668 & 0.500 & 0.530 \\ CQ+RM & 0.735 & 0.670 & 0.684 & 0.632 \\ PA+CQ+RM & 0.731 & 0.665 & 0.489 & 0.525 \\
**QRAKT** & **0.771** & **0.702** & **0.707** & **0.670** \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Comparison of the results when removing relation modeling (RM). The best model is boldfaced. After removing the RM, the AUC of the model on the two datasets decreased by approximately 3.1% and 2.3%. Figures 5-7 present the heat maps of a learner whose knowledge state constantly changes on skill sets on the Eedi dataset. These figures present the difference of the heat maps based on different knowledge tracing models, including QRAKT, DKT, and DKT+, to predict student performance on three skills, 32, 49, and 71. Figure 8 shows the evolution of student's learning process on these three knowledge points. Figure 6 illustrates, the DKT model performed worse than the other models in tracing the student knowledge state. The student failed to master Skills 32, 49, and 71 because of the low-level mastery level of the skills. Compared with the DKT model, the DKT+ model as the extension of the DKT model solved the reconstruction problem and the wavy transition problem in the DKT model to track the student's mastery level. The performance of DKT+ in tracking the student knowledge state was better than the DKT model and improved the student knowledge state performance, as seen in Figure 7. Compared with the non-hybrid methods, the DKT model and the DKT+ model, the hybrid model, QRAKT, further improved the knowledge state performance by incorporating the student behaviors and the attention mechanism to obtain more accurate results of the knowledge state prediction, as seen in Figure 7.  
Figures 5-7 only illustrate the difference in the knowledge state prediction Stability of QRAKT, DKT, and DKT+. In order to more specifically present the model performance differences in tracking the student knowledge state, the overall performance was further compared at each time step through the line chart given in Figure 9. Figure 9 shows that the QRAKT model outperformed the DKT model and the DKT+ model in the performance in tracking student knowledge state.