# Calibrated Q-Matrix-Enhanced Deep Knowledge Tracing with Relational Attention Mechanism
## 4 Methods
### Results and Discussion
#### 5.2.2 Ablation Experiments (RQ2)  
This section aims at detecting the influence of the key components proposed in this section. An ablation experiment was conducted in this section to see how those key components affected the final results of the QRAKT model. There are seven variations of QAKT, each of which takes one or more methods out of the QAKT model. Specifically, PA, CQ, and RM indicate that the positional attention layer, calibrated Q-matrix, and relation modeling are removed from QRAKT, respectively. PA+CQ, PA+RM, and CQ+RM mean that two components are removed concurrently in QRAKT, i.e., the positional attention layer and the calibrated Q-matrix, the positional attention layer and relation modeling, and the calibrated Q-matrix and relation modeling. PA+CQ+RM indicates that the calibrated Q-matrix, positional attention layer, and relation modeling are removed from QRAKT concurrently. Tables 6-8 provide some essential conclusions.  
Figure 4: The comparison of the AUC on the ASSIST2012 dataset and the Eedi dataset. The calibrated Q-matrix relational attention model presented the best results of the AUC compared to the other seven models on the two datasets of 0.771% and 0.707%, respectively.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**ASSIST2012**} & \multicolumn{3}{c}{**Eedi**} \\ \hline  & AUC & ACC & Sta & AUC & ACC & Sta \\ DKT & 0.712 & 0.679 & 0.125 & 0.489 & 0.489 & 0.125 \\ DKT+ & 0.722 & 0.685 & 0.257 & 0.584 & 0.566 & 0.358 \\ DKVMN & 0.701 & 0.686 & 0.255 & 0.698 & 0.640 & 0.683 \\ SAKT & 0.736 & 0.692 & 0.492 & 0.495 & 0.493 & 0.266 \\ EKT & 0.748 & 0.690 & 0.668 & 0.694 & 0.648 & 0.695 \\ EKPT & 0.752 & 0.693 & 0.706 & 0.698 & 0.651 & 0.704 \\ RAKT & 0.754 & 0.695 & 0.793 & 0.702 & 0.652 & 0.791 \\
**QRAKT** & **0.771** & **0.702** & 0.967 & **0.707** & **0.670** & 0.961 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison of the results of the baseline models with the relational attention knowledge tracing model (RAKT) and the calibrated Q-matrix relational attention knowledge tracing model (QRAKT). The best model is boldfaced. The RAKT model performed better than the six baseline models, including the DKT model, the DKT+ model, the DKVMN model, and the SAKT model. The QRAKT model further improved the AUC and ACC to some extent compared to the RAKT model and had the best performance in terms of the AUC and ACC.  
Firstly, the performance improved when more components were involved, which met our expectations. Secondly, on the Eedi dataset, when the positional attention layer was removed, the performance of the QRAKT model was significantly worse than when removing the other two factors. The effect of the model dropped by about 19.2%, as seen in Table 6. Therefore, the positional attention layer is a necessary part of the QRAKT model. On the ASSIST2012 datasets, relation modeling deletion leads to the most outstanding performance decline, which drops about 3.1%, as seen in Table 8. According to the results of Table 8, the relation model is an integral part of the QRAKT model. The different removal results of the two datasets are due to the fact that the number of student records on the Eedi dataset is much smaller than on the ASSIST2012 dataset. Therefore, relational modeling had relatively little impact on the results of the Eedi dataset. Lastly, the calibrated Q-matrix significantly improved the performance of the RAKT model, as seen in Table 7, which indicates that the calibrated Q-matrix is also a key component in the QRAKT model.