# Attention Transfer Reinforcement Learning for Test Case Prioritization in Continuous Integration
## 6 Results and Analysis  
### RQ1: Fault Detection Effectiveness  
Figure 5 illustrates the sorting performance of ATRL-TCP, RETECS, and PRM across the four datasets. The horizontal axis represents the number of CI cycles, while the vertical axis shows the NAPFD values of the sorting results for each method in each cycle. The black, green, and red solid curves represent the sorting performance of PRM, RETECS, and our proposed ATRL-TCP method, respectively, as the number of integration cycles increases. The dotted lines are linear fits to the curves, indicating the trend of NAPFD values over the cycles and the learning trends of the models. We observe that, compared to other datasets, RETECS performs better on the Paint Control dataset. This is because test cases in this dataset are used across multiple cycles and have a higher failure probability. Individual test cases have strong fault detection capability. It is more effective for RETECS to prioritize these datasets, because RETECS focuses on the features of individual test cases. However, RETECS is unable to effectively extract features from individual test cases in datasets with low error rates, such as GSDTSR. ATRL-TCP, on the other hand, is capable of switching its focus in cases of insufficient historical information and ranking based on the relative order between test cases. Therefore, the global ranking approach of ATRL-TCP and PRM is better suited to GSDTSR. In the Apache Drill dataset, test cases have low historical information and error rates. The relative relationships between test cases are less apparent. As a result, PRM fails to receive sufficient rewards. This causes the model's performance to gradually decline. Eventually, the overall ranking performance resembles random ranking. However, ATRL-TCP can simultaneously focus on the features of individual test cases and their relative relationships, obtaining more reward feedback and achieving better ranking performance. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Research Questions** & **Dataset** & **Methods** & **Metrics** \\ \hline RQ1 & Paint Control, IOF/ROL, GSDTSR, and Apach Drill & RETECS, PRM, ATRL-TCP & NAPFD, TTF, and Recall \\ RQ2 & Paint Control & ATRL-TCP & NAPFD \\ RQ3 & Paint Control, IOF/ROL, GSDTSR, and Apach Drill & RETECS, PRM, ATRL-TCP & Prediction time, Training time \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Correspondence between research questions, datasets, methods, and metrics. Overall, our proposed ATRL-TCP method achieves better results compared to other methods across all datasets. The fitted curves indicate a steady improvement in model performance. Initially, the model does not learn the pattern for TCP, resulting in poor performance. However, as the number of integration cycles increases, the model adjusts its strategy based on feedback from the sorting results. It learns the prioritization pattern and adapts to environmental changes. This leads to gradual improvement and eventually achieves good sorting performance. The box plots in Figure 6 show that the mean NAPFD values of ATRL-TCP in the Paint Control, IOF/ROL, GSDTSR, and Apach Drill datasets are 0.6145, 0.4576, 0.6645, and 0.4204, respectively. In comparison, the mean NAPFD values of RETECS are 0.5503, 0.3059, 0.1156, and 0.1815. PRM's mean NAPFD values are 0.4213, 0.3875, 0.6109, and 0.3169, respectively. ATRL-TCP consistently outperforms the other methods, with an improvement of up to five times compared to RETECS and up to 45.86% compared to PRM. Figure 5: Comparison of the four methods’ results. Tables 4 and 5 present the comparisons of TTF values and recall rates across various algorithms, with the optimal values highlighted in bold. As shown in Table 4, ATRL-TCP achieves smaller average TTF values in all datasets except GSDTSR, with an improvement of up to 50.81% compared to PRM. In GSDTSR, ATRL-TCP has a more pronounced advantage in TTF compared to RETECS. This indicates that ATRL-TCP effectively leverages dataset characteristics to prioritize test cases.