# Multi-Source Heterogeneous Kernel Mapping in Software Defect Prediction
## 6 Statistical Testing  
### Statistical Significance Test  
To assess the statistical significance of the predicting Pd, Pf, and GM, we conducted the non-parametric Friedman test with a post-hoc Nemenyi test at a 95% confidence level when comparing WPDP, CCA+, HDP-KS, CTKCCA, MHCPDP, and MSHKM over the 28 data sets [44]. The Friedman test compares whether the average ranks have statistical significance, and has been previously applied in heterogeneous CPDP studies [9; 13]. We ranked the values of different methods for each data set to 6, 5, 4, 3, 2, and 1, from large to small, according to Tables 4-6; for example, the ranks of five methods for Apache were 5 (WPDP), 1 (CCA+), 2 (HDP-KS), 4 (CTKCCA), 3 (MHCPDP), and 6 (MSHKM), in terms of Pd, according to Tables 4-6. It is worth noting that a rank of 6 represents the best result for Pd and GM, but represents the worst result for Pf. The results of the Friedman test, reported in Table 8, demonstrated mean rank of each method, and the p-values are far less than 0.001. To further analyze these results, we employed the Nemenyi test, which graphically visualize the comparison results for Pd, Pf, and GM in Figures 11-13, respectively. Specifically, the critical difference (CD) was used to connect methods without significant differences. In other words, there is no significant difference in the methods of connection [44].  
Figure 10: Results of different methods, in terms of GM. Among them, the square represents the mean values and “x” represents outlier values.  
In terms of the Pd mean ranks shown in Figure 11, there were four groups: {MSHKM, WPDP}, {WPDP, CTKCCA}, {CTKCCA, MHCPDP}, and {MHCPDP, CCA+, HDP-KS}. The inner methods of each group are connected. It should be noted that a higher mean rank (the left side in the axis) represents better prediction performance, which is also suitable for GM. However, for Pf, the right side indicates better performance. From Figures 10 and 11, we can see that MSHKM outperformed CTKCCA, MHCPDP, CCA+, and HDP-KS significantly for Pd and Pf, but did not significantly outperform WPDP, as they were in the same group. Similarly, for GM, from Figure 13, we can observe that MSHKM outperformed the other five baselines with significant differences.  
Overall, MSHKM provided better performance against the HDP baselines and showed comparable results when compared to WPDP, with statistical significance in Pd, Pf, and GM comparisons.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**Mean Rank**} \\ \hline  & **Pd** & **Pf** & **GM** \\ \hline WPDP & 4.536 & 2.750 & 4.643 \\ CCA+ & 2.500 & 4.929 & 2.429 \\ HDP-KS & 2.214 & 4.250 & 2.393 \\ CTKCCA & 3.679 & 2.821 & 3.179 \\ MHCPDP & 2.536 & 4.607 & 2.571 \\ MSHKM & 5.571 & 1.643 & 5.821 \\ \hline \(p\)-values & \textless{}\textless{}0.001 & \textless{}\textless{}0.001 & \textless{}\textless{}0.001 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Friedman test results.  
Figure 11: Results of the Nemenyi test for six methods, in terms of Pd.  
Figure 12: Results of the Nemenyi test for six methods, in terms of Pf.  
Figure 13: Results of the Nemenyi test for six methods, in terms of GM.