# An Improved CNN Model for Within-Project Software Defect Prediction
## 5 Results and Discussion
### Performance of the Improved CNN Model

Table 8: Performance comparison of different models on the PSC dataset. The decimal values represent the average MCC measure values. The best MCC measures for each row are highlighted in bold. To check whether our improved model outperformed other models statistically, we first performed the Friedman test. The Friedman test yielded a result of \(p=0.00042\), \(p=0.000035\), and \(p=0.005668\) for F-measure, G-measure and MCC, respectively, and they were all below the threshold 0.05, which means that the null hypothesis that all models perform equally was rejected. We then performed the Holm-Bonferroni test, and the results can be seen in Tables 9-11. As the tables indicate, the improved CNN was significantly better than the other five models in terms of F-measure and G-measure. As for MCC, the improved CNN and RF models were significantly better than other models, and there were no significant differences between the CNN and RF models. After further observation of results for each version of projects, we found that our model performed poorly on JEdit 4.3 in terms of F-measure and G-measure. This was because our model got TP = 0, which means that our model could not find buggy files on JEdit 4.3. Considering that the buggy rate of JEdit 4.3 is 2.3%, we can conclude that our model is weak at predicting ultra-low buggy rate versions of projects, and this is partly acceptable because other baseline models also performed poorly. Another finding is that our model performed poorly on Log4j 1.2 and Lucene 2.4. This was because our model got TN = 0, which means that our model could not find clean files on both versions. Considering that the buggy rate of Log4j 1.2 and Lucene 2.4 is 95.9% and 61.5%, we can conclude that our model is not robust when predicting ultra-high buggy rate version of projects in terms of G-measure. In conclusion, our model performed not as well towards extreme buggy rate conditions. Summary: Our improved CNN model outperformed the state-of-the-art machine learning models for within-version WPDP in terms of F-measure, G-measure, and MCC. Moreover, statistical hypothesis tests showed that the improved model was significantly better than the state-of-the-art machine \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **DT** & **RF** & **LR** & **NB** & **NET** \\ \hline
**RF** & 1 & - & - & - & - \\ \hline
**LR** & 0.0231 & 0.0869 & - & - & - \\ \hline
**NB** & 0.0559 & 0.1766 & 1 & - & - \\ \hline
**NET** & 0 & 0 & 0 & 0 & - \\ \hline
**Improved CNN** & 0.0231 & 0.0028 & 0 & 0 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Holm–Bonferroni test of prediction on the PSC dataset for G-measures. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **DT** & **RF** & **LR** & **NB** & **NET** \\ \hline
**RF** & 0.0069 & - & - & - & - \\ \hline
**LR** & 0.5251 & 0.0009 & - & - & - \\ \hline
**NB** & 0.2749 & 0 & 0.4681 & - & - \\ \hline
**NET** & 0.0026 & 0 & 0.0169 & 0.2749 & - \\ \hline
**Improved CNN** & 0 & 0.1102 & 0 & 0 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Holm–Bonferroni test of prediction on the PSC dataset for MCCs. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **DT** & **RF** & **LR** & **NB** & **NET** \\ \hline
**RF** & 1 & - & - & - & - \\ \hline
**LR** & 0.0857 & 0.0857 & - & - & - \\ \hline
**NB** & 0 & 0 & 0.0752 & - & - \\ \hline
**NET** & 0 & 0 & 0 & 0.0605 & - \\ \hline
**Improved CNN** & 0 & 0 & 0 & 0 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Holm–Bonferroni test of prediction on the PSC dataset for F-measures. learning models in terms of F-measure and G-measure, and our model could perform as well as RF in terms of MCC.